Your First Extension
This page builds a small DuckDB extension step by step: an aggregate function,
word_count, and a scalar function, first_word. Every callback is guarded, so a
panic in it fails the query with an error instead of aborting the DuckDB process. If
you have not built an extension before, the Quick Start is shorter
and a good place to begin.
| SQL | Kind | Signature |
|---|---|---|
word_count(text) | Aggregate | VARCHAR → BIGINT |
first_word(text) | Scalar | VARCHAR → VARCHAR |
Window functions and DuckDB before 2.0. Every aggregate registered through the C API,
word_countincluded, crashes the DuckDB process on 1.4.x and 1.5.x when a user runs it asword_count(x) OVER (), over an explicitROWSorRANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWINGframe, or asword_count(x ORDER BY y). On 1.4.4,OVER (PARTITION BY p)without anORDER BYcrashes too. This is a DuckDB defect that no extension can prevent, fixed in 2.0.0; see Known Limitations. Tell the users of your own aggregates.
The repository's examples/hello-ext
registers these two functions among many others. It writes them as hand-rolled
unsafe extern "C" fns, which are installed as written and run with no
catch_unwind; its README says it shows what the C API calls look like, not how to
write new code. The code below is the guarded form.
Set up the project
Create a library crate and give it the Cargo.toml from the
Quick Start, with the name word_tools:
[package]
name = "word_tools"
version = "0.1.0"
edition = "2021"
# With resolver 3, Cargo picks dependency versions that build on this.
rust-version = "1.86"
resolver = "3"
[lib]
# Must match the extension name: see Pitfall P1.
name = "word_tools"
crate-type = ["cdylib", "rlib"]
[dependencies]
quack-rs = "0.20"
[dependencies.libduckdb-sys]
version = ">=1.4.4, <1.20100"
default-features = false
features = ["loadable-extension"]
[profile.release]
panic = "unwind"
lto = true
opt-level = 3
codegen-units = 1
strip = true
Overview
An extension has four parts:
- State struct: holds data accumulated during aggregation (aggregate only)
- Callbacks:
update,combineandfinalizefor the aggregate, written with theaggregate_*_callback!macros (ffi_state::<T>()suppliesstate_size,state_initandstate_destroy); for the scalar, a closure - Registration: wire the callbacks to DuckDB with
AggregateFunctionBuilderandScalarFunctionBuilder - Entry point: DuckDB's initialization hook, generated by
entry_point!
Part 1: aggregate function word_count
An aggregate function accumulates state across many rows and emits one result per group.
1a. The logic and the state struct
The counting is ordinary Rust, so it is easy to unit-test (see Unit tests):
#![allow(unused)] fn main() { fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 } }
#![allow(unused)] fn main() { use quack_rs::prelude::*; #[derive(Default, Debug)] struct WordCountState { count: i64, } impl AggregateState for WordCountState {} }
AggregateState is a marker trait with no methods; its supertraits require the state to be
Default + Send + Sync + 'static.
FfiState<WordCountState> stores it in the bytes DuckDB allocates for each group
(boxing it only when it is large or over-aligned) and manages its lifecycle
(size, init, destroy).
1b. state_size, state_init and state_destroy
These three callbacks are always identical boilerplate, so you do not write them:
.ffi_state::<WordCountState>() on the builder (see Part 3)
installs FfiState::<WordCountState>::size_callback, init_callback and
destroy_callback together. Installing all three for one type in a single call means
DuckDB never allocates space sized for one state and has it initialised as another.
size_callback returns FfiState::<WordCountState>::size(): one usize tag word followed
by the state. A T aligned no more strictly than usize and at most 256 bytes (like
WordCountState) is stored inline in the bytes DuckDB allocates per group; a larger or more
strictly aligned T is boxed, and the slot holds the Box pointer. init_callback writes
WordCountState::default() into the slot (or boxes it), then sets the tag, a value salted per
state type, so a destructor only drops states that were initialised for its own T.
destroy_callback drops the T in each state (and its box, if the state is too
large to be stored inline), clearing the state's tag first so a second call is a
no-op.
1c. update: accumulate one batch
aggregate_update_callback! generates the unsafe extern "C" fn the builder expects and
runs your body under catch_unwind: a panic in it, such as an arithmetic overflow in a
debug build, fails the query with the panic message and leaves the process running. See
Panic-Safe Callbacks.
#![allow(unused)] fn main() { use quack_rs::prelude::*; #[derive(Default, Debug)] struct WordCountState { count: i64 } impl AggregateState for WordCountState {} fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 } quack_rs::aggregate_update_callback!(wc_update, |_info, input, states| { let reader = unsafe { VectorReader::new(input, 0) }; for row in 0..reader.row_count() { if !unsafe { reader.is_valid(row) } { continue; // NULL input contributes no words } let words = count_words(unsafe { reader.read_str(row) }); let state_ptr = unsafe { *states.add(row) }; if let Some(st) = unsafe { FfiState::<WordCountState>::with_state_mut(state_ptr) } { st.count += words; } } }); }
Key points:
- Check
is_valid(row)before reading. Never read an invalid (NULL) row. VectorReader::new(chunk, col)gives columncolfrom the chunk, andstates[row]is the state of that row's group: rows in the same group share one state.- Each
unsafe { … }block is its own scope: the macro body is not anunsafecontext.
1d. combine: merge parallel results
Pitfall L1: DuckDB creates fresh target states before calling
combine, set up bystate_init(hereWordCountState::default()), not copies of the source. You must copy all fields, not just the result field. In an aggregate with config fields (e.g., a histogram with abin_width) you must also copy those, or results will be silently corrupted.
#![allow(unused)] fn main() { use quack_rs::prelude::*; #[derive(Default, Debug)] struct WordCountState { count: i64 } impl AggregateState for WordCountState {} quack_rs::aggregate_combine_callback!(wc_combine, |_info, source, target, count| { for i in 0..count as usize { let src = unsafe { FfiState::<WordCountState>::with_state(*source.add(i)) }; let tgt = unsafe { FfiState::<WordCountState>::with_state_mut(*target.add(i)) }; if let (Some(s), Some(t)) = (src, tgt) { t.count += s.count; // If you add fields to WordCountState, combine them here too. } } }); }
combine reads source and must leave it unchanged (Pitfall L15):
the same source state can be combined into several targets.
1e. finalize: write output
#![allow(unused)] fn main() { use quack_rs::prelude::*; #[derive(Default, Debug)] struct WordCountState { count: i64 } impl AggregateState for WordCountState {} quack_rs::aggregate_finalize_callback!(wc_finalize, |_info, source, result, count, offset| { let mut writer = unsafe { VectorWriter::new(result) }; for i in 0..count as usize { let state_ptr = unsafe { *source.add(i) }; match unsafe { FfiState::<WordCountState>::with_state(state_ptr) } { Some(st) => unsafe { writer.write_i64(offset as usize + i, st.count) }, None => unsafe { writer.set_null(offset as usize + i) }, } } }); }
offset is DuckDB's output row offset: always use offset as usize + i, not just i.
Part 2: scalar function first_word
A scalar function processes one data chunk and returns one output value per row. For a
function of one to three primitive or string arguments, the closure builders are the
whole job: you write the per-row logic, and the builder derives the SQL types from the
closure's signature, skips NULL rows and writes NULL for them, and runs the closure under
catch_unwind.
#![allow(unused)] fn main() { use quack_rs::prelude::*; fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") } fn demo() -> Result<(), ExtensionError> { let first_word_builder = ScalarFunctionBuilder::map1_str("first_word", |s: &str| { first_word(s).to_owned() })?; let _ = first_word_builder; Ok(()) } }
where first_word is the pure logic:
#![allow(unused)] fn main() { fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") } }
When the closure builders do not fit
A function over STRUCT or LIST arguments, with variable arity, or with bind-time
state needs the raw chunk. Write the callback with scalar_callback!, which gives it the
same guard. It is then your job to propagate NULL: if the input row is NULL, write NULL to
the output and never read from an invalid row.
#![allow(unused)] fn main() { use quack_rs::prelude::*; fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") } quack_rs::scalar_callback!(first_word_raw, |_info, input, output| { let reader = unsafe { VectorReader::new(input, 0) }; let mut writer = unsafe { VectorWriter::new(output) }; for row in 0..reader.row_count() { if !unsafe { reader.is_valid(row) } { unsafe { writer.set_null(row) }; // NULL in, NULL out continue; } let s = unsafe { reader.read_str(row) }; unsafe { writer.write_varchar(row, first_word(s)) }; } }); }
The raw builder takes the callback together with the signature you declare for it:
#![allow(unused)] fn main() { use libduckdb_sys::duckdb_connection; use quack_rs::prelude::*; quack_rs::scalar_callback!(first_word_raw, |_info, _input, _output| {}); fn register(con: duckdb_connection) -> Result<(), ExtensionError> { // SAFETY: `con` is the live connection DuckDB passed to the entry point. unsafe { ScalarFunctionBuilder::new("first_word_raw") .param(TypeId::Varchar) .returns(TypeId::Varchar) .function(first_word_raw) .register(con)?; } Ok(()) } }
set_null calls duckdb_vector_ensure_validity_writable before writing the NULL flag,
which DuckDB requires; VectorWriter does it for you.
Keep arithmetic in either style checked: in a release build x * 2 wraps on overflow
without any error, which is why Scalar Functions and the Quick
Start use checked_mul.
Part 3: registration
The aggregate builder takes the three callbacks and ffi_state; the closure builder
already holds its callback. register is unsafe because it calls the C API on a
connection that must be live, and the entry point is where DuckDB hands you one:
#![allow(unused)] fn main() { use libduckdb_sys::duckdb_connection; use quack_rs::prelude::*; #[derive(Default, Debug)] struct WordCountState { count: i64 } impl AggregateState for WordCountState {} fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 } fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") } quack_rs::aggregate_update_callback!(wc_update, |_info, input, states| { let reader = unsafe { VectorReader::new(input, 0) }; for row in 0..reader.row_count() { if !unsafe { reader.is_valid(row) } { continue; } let words = count_words(unsafe { reader.read_str(row) }); if let Some(st) = unsafe { FfiState::<WordCountState>::with_state_mut(*states.add(row)) } { st.count += words; } } }); quack_rs::aggregate_combine_callback!(wc_combine, |_info, source, target, count| { for i in 0..count as usize { let src = unsafe { FfiState::<WordCountState>::with_state(*source.add(i)) }; let tgt = unsafe { FfiState::<WordCountState>::with_state_mut(*target.add(i)) }; if let (Some(s), Some(t)) = (src, tgt) { t.count += s.count; } } }); quack_rs::aggregate_finalize_callback!(wc_finalize, |_info, source, result, count, offset| { let mut writer = unsafe { VectorWriter::new(result) }; for i in 0..count as usize { match unsafe { FfiState::<WordCountState>::with_state(*source.add(i)) } { Some(st) => unsafe { writer.write_i64(offset as usize + i, st.count) }, None => unsafe { writer.set_null(offset as usize + i) }, } } }); fn live_connection() -> libduckdb_sys::duckdb_connection { std::mem::forget(quack_rs::testing::InMemoryDb::open().unwrap()); let (mut db, mut con) = (std::ptr::null_mut(), std::ptr::null_mut()); unsafe { assert_eq!(libduckdb_sys::duckdb_open(std::ptr::null(), &mut db), libduckdb_sys::DuckDBSuccess); assert_eq!(libduckdb_sys::duckdb_connect(db, &mut con), libduckdb_sys::DuckDBSuccess); } con } /// First column of the first row, as BIGINT; `None` for NULL. fn query_i64(con: libduckdb_sys::duckdb_connection, sql: &str) -> Option<i64> { let mut result = unsafe { quack_rs::query::query(con, sql) }.unwrap(); let chunk = result.next_chunk().unwrap().unwrap(); let reader = unsafe { chunk.reader(0) }; unsafe { reader.is_valid(0).then(|| reader.read_i64(0)) } } fn register(con: duckdb_connection) -> Result<(), ExtensionError> { let first_word_builder = ScalarFunctionBuilder::map1_str("first_word", |s: &str| { first_word(s).to_owned() })?; // SAFETY: `con` is the live connection DuckDB passed to the entry point. unsafe { AggregateFunctionBuilder::new("word_count") .param(TypeId::Varchar) .returns(TypeId::BigInt) .ffi_state::<WordCountState>() // state_size + init + destructor .update(wc_update) .combine(wc_combine) .finalize(wc_finalize) .register(con)?; first_word_builder.register(con)?; } Ok(()) } let con = live_connection(); register(con).unwrap(); assert_eq!(query_i64(con, "SELECT word_count(s) FROM (VALUES ('hello world'), (NULL), ('one two three')) t(s)"), Some(5)); assert_eq!(query_i64(con, "SELECT length(first_word(' quack rs'))::BIGINT"), Some(5)); assert_eq!(query_i64(con, "SELECT count(*) FROM (SELECT first_word(x) AS w FROM (VALUES (NULL::VARCHAR)) t(x)) WHERE w IS NULL"), Some(1)); // Many groups, so DuckDB runs combine on partial states. assert_eq!(query_i64(con, "SELECT sum(c)::BIGINT FROM (SELECT word_count('a b') AS c FROM range(100000) GROUP BY range % 997)"), Some(200000)); // A running window frame is not one of the shapes that crash before DuckDB 2.0. assert_eq!(query_i64(con, "SELECT max(w) FROM (SELECT word_count(s) OVER (ORDER BY i) AS w FROM (SELECT 'a b c' AS s, i FROM range(5000) t(i)))"), Some(15000)); }
register returns Err if DuckDB reports a failure, and the entry point turns that into
a LOAD error the user sees.
When you test NULL handling, read the NULL from a column, as
SELECT first_word(x) FROM (VALUES (NULL::VARCHAR)) t(x) does, not from a literal:
DuckDB folds a literal NULL argument into the plan before it ever calls the function,
so SELECT first_word(NULL) passes even for a callback that gets NULL wrong
(Pitfall L8).
Part 4: entry point
#![allow(unused)] fn main() { use libduckdb_sys::duckdb_connection; use quack_rs::prelude::*; fn register(_: duckdb_connection) -> Result<(), ExtensionError> { Ok(()) } quack_rs::entry_point!(word_tools_init_c_api, |con| register(con)); }
This one line expands to the equivalent of:
#![allow(unused)] fn main() { use libduckdb_sys::{duckdb_connection, duckdb_extension_access, duckdb_extension_info}; use quack_rs::error::ExtensionError; fn register(_: duckdb_connection) -> Result<(), ExtensionError> { Ok(()) } #[unsafe(no_mangle)] pub unsafe extern "C" fn word_tools_init_c_api( info: duckdb_extension_info, access: *const duckdb_extension_access, ) -> bool { unsafe { quack_rs::entry_point::init_extension_with_policy( info, access, quack_rs::DUCKDB_API_VERSION, quack_rs::abi::AbiPolicy::Strict, |con| register(con), ) } } }
Pass the full symbol name, word_tools_init_c_api here: DuckDB looks up this exact
symbol when loading the extension. See The Entry Point for
the full initialization sequence. Put the pieces from Parts 1 to 4 in src/lib.rs, after
use libduckdb_sys::duckdb_connection; and use quack_rs::prelude::*;.
Build, stamp and load
cargo build --release
# DuckDB only loads files ending in `.duckdb_extension` that carry its
# 512-byte metadata footer; a bare `.so` is refused. Append the footer:
cargo install quack-rs --bin append_metadata # needs Rust 1.88 or newer
append_metadata target/release/libword_tools.so word_tools.duckdb_extension \
--abi-type C_STRUCT --extension-version v0.1.0 \
--duckdb-version v1.2.0 --platform linux_amd64
# An unsigned local build needs -unsigned (it must be a startup flag).
duckdb -unsigned
Then in the DuckDB CLI:
LOAD './word_tools.duckdb_extension';
-- Aggregate: total words across all rows
SELECT word_count(sentence) AS words FROM (
VALUES ('hello world'), ('one two three'), (NULL)
) t(sentence);
-- Scalar: first word of each row
SELECT first_word(sentence) AS first FROM (
VALUES ('hello world'), (' padded '), (''), (NULL)
) t(sentence);
┌───────┐
│ words │
│ int64 │
├───────┤
│ 5 │
└───────┘
┌─────────┐
│ first │
│ varchar │
├─────────┤
│ hello │
│ padded │
│ │
│ NULL │
└─────────┘
The 5 is 2 + 3 words; the NULL row contributes none. The empty string has no first word, so its result is the empty string (the blank row), and the NULL row gives NULL. Use the library file name and platform string for your system as the Quick Start lists them.
Unit tests (no DuckDB process needed)
Test pure logic directly:
#![allow(unused)] fn main() { fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 } fn first_word(s: &str) -> &str { s.split_whitespace().next().unwrap_or("") } #[test] fn count_words_whitespace_variants() { assert_eq!(count_words(" hello world "), 2); assert_eq!(count_words("\t\nhello\tworld\n"), 2); assert_eq!(count_words(" "), 0); // all whitespace → 0 } #[test] fn first_word_empty_and_whitespace() { assert_eq!(first_word(""), ""); assert_eq!(first_word(" "), ""); } }
Test aggregate state with AggregateTestHarness:
#![allow(unused)] fn main() { use quack_rs::prelude::*; use quack_rs::testing::AggregateTestHarness; #[derive(Default, Debug)] struct WordCountState { count: i64 } impl AggregateState for WordCountState {} fn count_words(s: &str) -> i64 { s.split_whitespace().count() as i64 } #[test] fn word_count_null_rows_are_skipped() { // DuckDB passes NULL rows to `update`; the callback's `is_valid` check // skips them, so they never reach the state. let mut h = AggregateTestHarness::<WordCountState>::new(); h.update(|s| s.count += count_words("hello")); // NULL row omitted — models the callback's `is_valid` skip h.update(|s| s.count += count_words("world")); assert_eq!(h.finalize().count, 2); } #[test] fn word_count_combine() { let mut h1 = AggregateTestHarness::<WordCountState>::new(); h1.update(|s| s.count += count_words("hello world")); // 2 let mut h2 = AggregateTestHarness::<WordCountState>::new(); h2.update(|s| s.count += count_words("one two three four")); // 4 h2.combine(&h1, |src, tgt| tgt.count += src.count); assert_eq!(h2.finalize().count, 6); } }
Run all tests with cargo test. The harness cannot exercise the DuckDB side of the
callbacks (NULL handling, the window shapes above, the state layout); for that, load the
built extension into a real DuckDB, or use the bundled-test features. See the
Testing Guide for the full test strategy.