Your First Extension

This page builds a small DuckDB extension step by step: an aggregate function, word_count, and a scalar function, first_word. Every callback is guarded, so a panic in it fails the query with an error instead of aborting the DuckDB process. If you have not built an extension before, the Quick Start is shorter and a good place to begin.

SQLKindSignature
word_count(text)AggregateVARCHAR → BIGINT
first_word(text)ScalarVARCHAR → VARCHAR

Window functions and DuckDB before 2.0. Every aggregate registered through the C API, word_count included, crashes the DuckDB process on 1.4.x and 1.5.x when a user runs it as word_count(x) OVER (), over an explicit ROWS or RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING frame, or as word_count(x ORDER BY y). On 1.4.4, OVER (PARTITION BY p) without an ORDER BY crashes too. This is a DuckDB defect that no extension can prevent, fixed in 2.0.0; see Known Limitations. Tell the users of your own aggregates.

The repository's examples/hello-ext registers these two functions among many others. It writes them as hand-rolled unsafe extern "C" fns, which are installed as written and run with no catch_unwind; its README says it shows what the C API calls look like, not how to write new code. The code below is the guarded form.


Set up the project

Create a library crate and give it the Cargo.toml from the Quick Start, with the name word_tools:

[package]
name = "word_tools"
version = "0.1.0"
edition = "2021"
# With resolver 3, Cargo picks dependency versions that build on this.
rust-version = "1.86"
resolver = "3"

[lib]
# Must match the extension name: see Pitfall P1.
name = "word_tools"
crate-type = ["cdylib", "rlib"]

[dependencies]
quack-rs = "0.20"

[dependencies.libduckdb-sys]
version = ">=1.4.4, <1.20100"
default-features = false
features = ["loadable-extension"]

[profile.release]
panic = "unwind"
lto = true
opt-level = 3
codegen-units = 1
strip = true

Overview

An extension has four parts:

  1. State struct: holds data accumulated during aggregation (aggregate only)
  2. Callbacks: update, combine and finalize for the aggregate, written with the aggregate_*_callback! macros (ffi_state::<T>() supplies state_size, state_init and state_destroy); for the scalar, a closure
  3. Registration: wire the callbacks to DuckDB with AggregateFunctionBuilder and ScalarFunctionBuilder
  4. Entry point: DuckDB's initialization hook, generated by entry_point!

Part 1: aggregate function word_count

An aggregate function accumulates state across many rows and emits one result per group.

1a. The logic and the state struct

The counting is ordinary Rust, so it is easy to unit-test (see Unit tests):

#![allow(unused)]
fn main() {
fn count_words(s: &str) -> i64 {
    s.split_whitespace().count() as i64
}
}
#![allow(unused)]
fn main() {
use quack_rs::prelude::*;
#[derive(Default, Debug)]
struct WordCountState {
    count: i64,
}

impl AggregateState for WordCountState {}
}

AggregateState is a marker trait with no methods; its supertraits require the state to be Default + Send + Sync + 'static. FfiState<WordCountState> stores it in the bytes DuckDB allocates for each group (boxing it only when it is large or over-aligned) and manages its lifecycle (size, init, destroy).

1b. state_size, state_init and state_destroy

These three callbacks are always identical boilerplate, so you do not write them: .ffi_state::<WordCountState>() on the builder (see Part 3) installs FfiState::<WordCountState>::size_callback, init_callback and destroy_callback together. Installing all three for one type in a single call means DuckDB never allocates space sized for one state and has it initialised as another.

size_callback returns FfiState::<WordCountState>::size(): one usize tag word followed by the state. A T aligned no more strictly than usize and at most 256 bytes (like WordCountState) is stored inline in the bytes DuckDB allocates per group; a larger or more strictly aligned T is boxed, and the slot holds the Box pointer. init_callback writes WordCountState::default() into the slot (or boxes it), then sets the tag, a value salted per state type, so a destructor only drops states that were initialised for its own T.

destroy_callback drops the T in each state (and its box, if the state is too large to be stored inline), clearing the state's tag first so a second call is a no-op.

1c. update: accumulate one batch

aggregate_update_callback! generates the unsafe extern "C" fn the builder expects and runs your body under catch_unwind: a panic in it, such as an arithmetic overflow in a debug build, fails the query with the panic message and leaves the process running. See Panic-Safe Callbacks.

#![allow(unused)]
fn main() {
use quack_rs::prelude::*;
#[derive(Default, Debug)] struct WordCountState { count: i64 }
impl AggregateState for WordCountState {}
fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 }
quack_rs::aggregate_update_callback!(wc_update, |_info, input, states| {
    let reader = unsafe { VectorReader::new(input, 0) };
    for row in 0..reader.row_count() {
        if !unsafe { reader.is_valid(row) } {
            continue; // NULL input contributes no words
        }
        let words = count_words(unsafe { reader.read_str(row) });

        let state_ptr = unsafe { *states.add(row) };
        if let Some(st) = unsafe { FfiState::<WordCountState>::with_state_mut(state_ptr) } {
            st.count += words;
        }
    }
});
}

Key points:

  • Check is_valid(row) before reading. Never read an invalid (NULL) row.
  • VectorReader::new(chunk, col) gives column col from the chunk, and states[row] is the state of that row's group: rows in the same group share one state.
  • Each unsafe { … } block is its own scope: the macro body is not an unsafe context.

1d. combine: merge parallel results

Pitfall L1: DuckDB creates fresh target states before calling combine, set up by state_init (here WordCountState::default()), not copies of the source. You must copy all fields, not just the result field. In an aggregate with config fields (e.g., a histogram with a bin_width) you must also copy those, or results will be silently corrupted.

#![allow(unused)]
fn main() {
use quack_rs::prelude::*;
#[derive(Default, Debug)] struct WordCountState { count: i64 }
impl AggregateState for WordCountState {}
quack_rs::aggregate_combine_callback!(wc_combine, |_info, source, target, count| {
    for i in 0..count as usize {
        let src = unsafe { FfiState::<WordCountState>::with_state(*source.add(i)) };
        let tgt = unsafe { FfiState::<WordCountState>::with_state_mut(*target.add(i)) };
        if let (Some(s), Some(t)) = (src, tgt) {
            t.count += s.count;
            // If you add fields to WordCountState, combine them here too.
        }
    }
});
}

combine reads source and must leave it unchanged (Pitfall L15): the same source state can be combined into several targets.

1e. finalize: write output

#![allow(unused)]
fn main() {
use quack_rs::prelude::*;
#[derive(Default, Debug)] struct WordCountState { count: i64 }
impl AggregateState for WordCountState {}
quack_rs::aggregate_finalize_callback!(wc_finalize, |_info, source, result, count, offset| {
    let mut writer = unsafe { VectorWriter::new(result) };
    for i in 0..count as usize {
        let state_ptr = unsafe { *source.add(i) };
        match unsafe { FfiState::<WordCountState>::with_state(state_ptr) } {
            Some(st) => unsafe { writer.write_i64(offset as usize + i, st.count) },
            None => unsafe { writer.set_null(offset as usize + i) },
        }
    }
});
}

offset is DuckDB's output row offset: always use offset as usize + i, not just i.


Part 2: scalar function first_word

A scalar function processes one data chunk and returns one output value per row. For a function of one to three primitive or string arguments, the closure builders are the whole job: you write the per-row logic, and the builder derives the SQL types from the closure's signature, skips NULL rows and writes NULL for them, and runs the closure under catch_unwind.

#![allow(unused)]
fn main() {
use quack_rs::prelude::*;
fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") }
fn demo() -> Result<(), ExtensionError> {
let first_word_builder = ScalarFunctionBuilder::map1_str("first_word", |s: &str| {
    first_word(s).to_owned()
})?;
let _ = first_word_builder;
Ok(())
}
}

where first_word is the pure logic:

#![allow(unused)]
fn main() {
fn first_word(s: &str) -> &str {
    s.split_whitespace().next().unwrap_or("")
}
}

When the closure builders do not fit

A function over STRUCT or LIST arguments, with variable arity, or with bind-time state needs the raw chunk. Write the callback with scalar_callback!, which gives it the same guard. It is then your job to propagate NULL: if the input row is NULL, write NULL to the output and never read from an invalid row.

#![allow(unused)]
fn main() {
use quack_rs::prelude::*;
fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") }
quack_rs::scalar_callback!(first_word_raw, |_info, input, output| {
    let reader = unsafe { VectorReader::new(input, 0) };
    let mut writer = unsafe { VectorWriter::new(output) };
    for row in 0..reader.row_count() {
        if !unsafe { reader.is_valid(row) } {
            unsafe { writer.set_null(row) }; // NULL in, NULL out
            continue;
        }
        let s = unsafe { reader.read_str(row) };
        unsafe { writer.write_varchar(row, first_word(s)) };
    }
});
}

The raw builder takes the callback together with the signature you declare for it:

#![allow(unused)]
fn main() {
use libduckdb_sys::duckdb_connection;
use quack_rs::prelude::*;
quack_rs::scalar_callback!(first_word_raw, |_info, _input, _output| {});
fn register(con: duckdb_connection) -> Result<(), ExtensionError> {
// SAFETY: `con` is the live connection DuckDB passed to the entry point.
unsafe {
    ScalarFunctionBuilder::new("first_word_raw")
        .param(TypeId::Varchar)
        .returns(TypeId::Varchar)
        .function(first_word_raw)
        .register(con)?;
}
Ok(())
}
}

set_null calls duckdb_vector_ensure_validity_writable before writing the NULL flag, which DuckDB requires; VectorWriter does it for you.

Keep arithmetic in either style checked: in a release build x * 2 wraps on overflow without any error, which is why Scalar Functions and the Quick Start use checked_mul.


Part 3: registration

The aggregate builder takes the three callbacks and ffi_state; the closure builder already holds its callback. register is unsafe because it calls the C API on a connection that must be live, and the entry point is where DuckDB hands you one:

#![allow(unused)]
fn main() {
use libduckdb_sys::duckdb_connection;
use quack_rs::prelude::*;
#[derive(Default, Debug)] struct WordCountState { count: i64 }
impl AggregateState for WordCountState {}
fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 }
fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") }
quack_rs::aggregate_update_callback!(wc_update, |_info, input, states| {
    let reader = unsafe { VectorReader::new(input, 0) };
    for row in 0..reader.row_count() {
        if !unsafe { reader.is_valid(row) } { continue; }
        let words = count_words(unsafe { reader.read_str(row) });
        if let Some(st) = unsafe { FfiState::<WordCountState>::with_state_mut(*states.add(row)) } {
            st.count += words;
        }
    }
});
quack_rs::aggregate_combine_callback!(wc_combine, |_info, source, target, count| {
    for i in 0..count as usize {
        let src = unsafe { FfiState::<WordCountState>::with_state(*source.add(i)) };
        let tgt = unsafe { FfiState::<WordCountState>::with_state_mut(*target.add(i)) };
        if let (Some(s), Some(t)) = (src, tgt) { t.count += s.count; }
    }
});
quack_rs::aggregate_finalize_callback!(wc_finalize, |_info, source, result, count, offset| {
    let mut writer = unsafe { VectorWriter::new(result) };
    for i in 0..count as usize {
        match unsafe { FfiState::<WordCountState>::with_state(*source.add(i)) } {
            Some(st) => unsafe { writer.write_i64(offset as usize + i, st.count) },
            None => unsafe { writer.set_null(offset as usize + i) },
        }
    }
});
fn live_connection() -> libduckdb_sys::duckdb_connection {
    std::mem::forget(quack_rs::testing::InMemoryDb::open().unwrap());
    let (mut db, mut con) = (std::ptr::null_mut(), std::ptr::null_mut());
    unsafe {
        assert_eq!(libduckdb_sys::duckdb_open(std::ptr::null(), &mut db), libduckdb_sys::DuckDBSuccess);
        assert_eq!(libduckdb_sys::duckdb_connect(db, &mut con), libduckdb_sys::DuckDBSuccess);
    }
    con
}
/// First column of the first row, as BIGINT; `None` for NULL.
fn query_i64(con: libduckdb_sys::duckdb_connection, sql: &str) -> Option<i64> {
    let mut result = unsafe { quack_rs::query::query(con, sql) }.unwrap();
    let chunk = result.next_chunk().unwrap().unwrap();
    let reader = unsafe { chunk.reader(0) };
    unsafe { reader.is_valid(0).then(|| reader.read_i64(0)) }
}
fn register(con: duckdb_connection) -> Result<(), ExtensionError> {
    let first_word_builder = ScalarFunctionBuilder::map1_str("first_word", |s: &str| {
        first_word(s).to_owned()
    })?;

    // SAFETY: `con` is the live connection DuckDB passed to the entry point.
    unsafe {
        AggregateFunctionBuilder::new("word_count")
            .param(TypeId::Varchar)
            .returns(TypeId::BigInt)
            .ffi_state::<WordCountState>() // state_size + init + destructor
            .update(wc_update)
            .combine(wc_combine)
            .finalize(wc_finalize)
            .register(con)?;

        first_word_builder.register(con)?;
    }
    Ok(())
}
let con = live_connection();
register(con).unwrap();
assert_eq!(query_i64(con, "SELECT word_count(s) FROM (VALUES ('hello world'), (NULL), ('one two three')) t(s)"), Some(5));
assert_eq!(query_i64(con, "SELECT length(first_word('  quack rs'))::BIGINT"), Some(5));
assert_eq!(query_i64(con, "SELECT count(*) FROM (SELECT first_word(x) AS w FROM (VALUES (NULL::VARCHAR)) t(x)) WHERE w IS NULL"), Some(1));
// Many groups, so DuckDB runs combine on partial states.
assert_eq!(query_i64(con, "SELECT sum(c)::BIGINT FROM (SELECT word_count('a b') AS c FROM range(100000) GROUP BY range % 997)"), Some(200000));
// A running window frame is not one of the shapes that crash before DuckDB 2.0.
assert_eq!(query_i64(con, "SELECT max(w) FROM (SELECT word_count(s) OVER (ORDER BY i) AS w FROM (SELECT 'a b c' AS s, i FROM range(5000) t(i)))"), Some(15000));
}

register returns Err if DuckDB reports a failure, and the entry point turns that into a LOAD error the user sees.

When you test NULL handling, read the NULL from a column, as SELECT first_word(x) FROM (VALUES (NULL::VARCHAR)) t(x) does, not from a literal: DuckDB folds a literal NULL argument into the plan before it ever calls the function, so SELECT first_word(NULL) passes even for a callback that gets NULL wrong (Pitfall L8).


Part 4: entry point

#![allow(unused)]
fn main() {
use libduckdb_sys::duckdb_connection;
use quack_rs::prelude::*;
fn register(_: duckdb_connection) -> Result<(), ExtensionError> { Ok(()) }
quack_rs::entry_point!(word_tools_init_c_api, |con| register(con));
}

This one line expands to the equivalent of:

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_connection, duckdb_extension_access, duckdb_extension_info};
use quack_rs::error::ExtensionError;
fn register(_: duckdb_connection) -> Result<(), ExtensionError> { Ok(()) }
#[unsafe(no_mangle)]
pub unsafe extern "C" fn word_tools_init_c_api(
    info: duckdb_extension_info,
    access: *const duckdb_extension_access,
) -> bool {
    unsafe {
        quack_rs::entry_point::init_extension_with_policy(
            info, access, quack_rs::DUCKDB_API_VERSION,
            quack_rs::abi::AbiPolicy::Strict,
            |con| register(con),
        )
    }
}
}

Pass the full symbol name, word_tools_init_c_api here: DuckDB looks up this exact symbol when loading the extension. See The Entry Point for the full initialization sequence. Put the pieces from Parts 1 to 4 in src/lib.rs, after use libduckdb_sys::duckdb_connection; and use quack_rs::prelude::*;.


Build, stamp and load

cargo build --release

# DuckDB only loads files ending in `.duckdb_extension` that carry its
# 512-byte metadata footer; a bare `.so` is refused. Append the footer:
cargo install quack-rs --bin append_metadata   # needs Rust 1.88 or newer
append_metadata target/release/libword_tools.so word_tools.duckdb_extension \
    --abi-type C_STRUCT --extension-version v0.1.0 \
    --duckdb-version v1.2.0 --platform linux_amd64

# An unsigned local build needs -unsigned (it must be a startup flag).
duckdb -unsigned

Then in the DuckDB CLI:

LOAD './word_tools.duckdb_extension';

-- Aggregate: total words across all rows
SELECT word_count(sentence) AS words FROM (
    VALUES ('hello world'), ('one two three'), (NULL)
) t(sentence);

-- Scalar: first word of each row
SELECT first_word(sentence) AS first FROM (
    VALUES ('hello world'), ('  padded  '), (''), (NULL)
) t(sentence);
┌───────┐
│ words │
│ int64 │
├───────┤
│     5 │
└───────┘
┌─────────┐
│  first  │
│ varchar │
├─────────┤
│ hello   │
│ padded  │
│         │
│ NULL    │
└─────────┘

The 5 is 2 + 3 words; the NULL row contributes none. The empty string has no first word, so its result is the empty string (the blank row), and the NULL row gives NULL. Use the library file name and platform string for your system as the Quick Start lists them.


Unit tests (no DuckDB process needed)

Test pure logic directly:

#![allow(unused)]
fn main() {
fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 }
fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") }
#[test]
fn count_words_whitespace_variants() {
    assert_eq!(count_words("  hello  world  "), 2);
    assert_eq!(count_words("\t\nhello\tworld\n"), 2);
    assert_eq!(count_words("   "), 0); // all whitespace → 0
}

#[test]
fn first_word_empty_and_whitespace() {
    assert_eq!(first_word(""), "");
    assert_eq!(first_word("   "), "");
}
}

Test aggregate state with AggregateTestHarness:

#![allow(unused)]
fn main() {
use quack_rs::prelude::*;
use quack_rs::testing::AggregateTestHarness;
#[derive(Default, Debug)] struct WordCountState { count: i64 }
impl AggregateState for WordCountState {}
fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 }
#[test]
fn word_count_null_rows_are_skipped() {
    // DuckDB passes NULL rows to `update`; the callback's `is_valid` check
    // skips them, so they never reach the state.
    let mut h = AggregateTestHarness::<WordCountState>::new();
    h.update(|s| s.count += count_words("hello"));
    // NULL row omitted — models the callback's `is_valid` skip
    h.update(|s| s.count += count_words("world"));
    assert_eq!(h.finalize().count, 2);
}

#[test]
fn word_count_combine() {
    let mut h1 = AggregateTestHarness::<WordCountState>::new();
    h1.update(|s| s.count += count_words("hello world")); // 2

    let mut h2 = AggregateTestHarness::<WordCountState>::new();
    h2.update(|s| s.count += count_words("one two three four")); // 4

    h2.combine(&h1, |src, tgt| tgt.count += src.count);
    assert_eq!(h2.finalize().count, 6);
}
}

Run all tests with cargo test. The harness cannot exercise the DuckDB side of the callbacks (NULL handling, the window shapes above, the state layout); for that, load the built extension into a real DuckDB, or use the bundled-test features. See the Testing Guide for the full test strategy.