Table Functions
A DuckDB table function returns a result set rather than a single value, and is
called in the FROM clause: SELECT * FROM my_function(args). This page shows how
to write one in Rust with quack-rs. DuckDB drives a table function through three
callbacks: bind, init and scan.
quack-rs provides two layers for registering table functions:
TypedTableFunctionBuilder<S>(recommended for new extensions) — closure-based API that hides bind/init/scan trampolines behind safe Rust closures and gives every execution a fresh, typed scan state built from whatbindproduced.TableFunctionBuilder— the underlying raw builder used byTypedTableFunctionBuilderinternally. Reach for it when you need fine-grained control: parallel scans (InitInfo::set_max_threadsabove 1, usually withlocal_initfor per-thread state), projection pushdown with column filtering, or callback shapes that don't fit the "produce state in bind, mutate it in scan" model.
Both builders are backed by the helper types BindInfo, InitInfo, FunctionInfo,
FfiBindData<T>, FfiInitData<T>, and FfiLocalInitData<T>.
Lifecycle
| Phase | Callback | Called when | Typical work |
|---|---|---|---|
| bind | bind_fn | Query is planned (once per plan on DuckDB 1.x; also on every EXECUTE of a prepared statement on 2.0.0) | Extract parameters; declare output columns; store configuration in bind data |
| init | init_fn | Each execution of the plan starts | Allocate per-scan state (cursor, row index, etc.) |
| scan | scan_fn | Each output batch | Fill the output chunk with rows; set its size with DataChunk::set_size |
DuckDB calls the scan callback repeatedly until it sets the chunk size to 0, which signals the end of the results.
Bind data outlives the plan; bind itself may run again. DuckDB keeps the bind data for as long as the bound plan lives and runs
initagainst it on every execution: eachEXECUTEof a prepared statement, each iteration of a recursive CTE that references the function. Treat bind data as immutable after bind and build anything a scan consumes (cursors, open files) ininit.How often
bindruns depends on the DuckDB release. With a prepared statement that is executed three times, counters in the callbacks read 1 bind and 3 inits on DuckDB 1.4.4 and 1.5.6, and 4 binds and 3 inits on 2.0.0 (one for thePREPARE, one for eachEXECUTE). So a table function that reads a setting inbind, where a client context is available, keeps the value from the first bind on 1.x and picks up a changed setting on 2.0. Do not rely on either behaviour: onlybindis given a client context (InitInfoandFunctionInfohave none), so a table function cannot read a setting atinitor scan time.
A table function that reads a file must open it through DuckDB.
std::fsignoresSET enable_external_access = falseandallowed_directories; theFileSystemwrapper (featureduckdb-1-5) honours them.
Closure-based typed state (with_state)
For the common "take parameters at bind, stream rows until exhausted" pattern,
TypedTableFunctionBuilder<S> replaces all three callback trampolines with two
closures. With with_state, the state returned by bind is a template: every
execution of the plan scans a fresh clone() of it, so S must be Clone + Send.
#![allow(unused)] fn main() { use quack_rs::prelude::*; #[derive(Clone)] struct State { remaining: u64, } fn register(reg: &impl Registrar) -> ExtResult<()> { let builder = TableFunctionBuilder::new("count_down") .param(TypeId::BigInt) // 1. bind closure: declare the output schema, read parameters, // return the template scan state (cloned for every execution). .with_state::<State, _>(|bind| { bind.add_result_column("n", TypeId::BigInt); let raw = unsafe { bind.get_parameter_value(0) }; Ok(State { remaining: raw.as_i64_or(0).max(0) as u64 }) }) // 2. scan closure: mutate state, write rows, set chunk size. .scan(|state, chunk| { if state.remaining == 0 { unsafe { chunk.set_size(0) }; return Ok(()); } let mut writer = unsafe { chunk.writer(0) }; unsafe { writer.write_i64(0, state.remaining as i64) }; state.remaining -= 1; unsafe { chunk.set_size(1) }; Ok(()) }) .build()?; unsafe { reg.register_table(builder) } } }
Separate bind data and scan state (with_bind_init)
When the scan state is expensive or impossible to clone (it owns a file handle,
a large buffer, a connection), or the parameters and the cursor are naturally
separate, use with_bind_init. bind returns immutable bind data B
(Send + Sync); init builds a fresh scan state S from &B for every
execution:
#![allow(unused)] fn main() { use quack_rs::prelude::*; struct Params { n: i64 } struct Cursor { next: i64, end: i64 } // no Clone needed fn register(reg: &impl Registrar) -> ExtResult<()> { let builder = TableFunctionBuilder::new("count_up") .param(TypeId::BigInt) .with_bind_init( |bind| { bind.add_result_column("n", TypeId::BigInt); let n = unsafe { bind.get_parameter_value(0) }.as_i64_or(0); Ok(Params { n }) }, |params: &Params| Ok(Cursor { next: 1, end: params.n }), ) .scan(|cursor, chunk| { if cursor.next > cursor.end { unsafe { chunk.set_size(0) }; return Ok(()); } unsafe { chunk.writer(0).write_i64(0, cursor.next); chunk.set_size(1); } cursor.next += 1; Ok(()) }) .build()?; unsafe { reg.register_table(builder) } } }
What you get for free
- No hand-written
unsafe extern "C" fntrampolines.TypedTableFunctionBuildergenerates them internally. - Typed scan state. The
scanclosure receives&mut S, freshly built for each execution (a clone of thewith_statetemplate, orinit(&B)forwith_bind_init) — no manualFfiBindData/FfiInitDatashuffling, and a prepared statement can be executed any number of times. - Panic safety. User closures run inside
catch_unwind. A panic is reported as a bind, init or scan error, and the scan sets the chunk size to zero, so the query fails cleanly instead of unwinding across the FFI boundary. - Error propagation. Return
Err(ExtensionError::new("..."))from any closure to report a SQL error to DuckDB.
Trade-offs and threading
Smust beSend + 'static(plusCloneforwith_state).Syncis not required, soTypedTableFunctionBuilderforces scans to run on a single worker by callingInitInfo::set_max_threads(1)internally. Measured: atSET threads = 8a typed function's scan callbacks all ran on one thread, on DuckDB 1.5.6 and 2.0.0. A scan that is the bottleneck of a query, over a large input, needs the raw builder withset_max_threadsabove 1.- The typed builder does not offer projection pushdown: with pushdown on, the
scan's chunk holds only the projected columns and a closure written against the
declared schema would write the wrong column.
build()returns an error ifprojection_pushdown(true)was set on the raw builder beforewith_state/with_bind_init. Use the raw builder for pushdown. - The bind closure must declare at least one column. With
duckdb-1-5, a bind that declares none is reported as an ordinary bind error. Withoutduckdb-1-5the typed builder cannot check, and DuckDB itself raises anINTERNAL Errorwith a C++ stack trace (measured on 1.4.4, 1.5.6 and 2.0.0; the database stays usable). A raw bind callback gets the sameINTERNAL Erroron every feature set, because quack-rs cannot check it after it returns: callset_erroryourself. - Extensions that need multi-worker parallelism (
set_max_threadsabove 1, withlocal_init+ thread-local buffers) should use the rawTableFunctionBuilderdirectly. TypedTableFunctionBuilder::build()returns a fully configuredTableFunctionBuilder, so you can still pass it through anyRegistrar— includingMockRegistrarfor unit tests. Registration fails if you then enableprojection_pushdownon it or replace itsbind,init,local_init,scanorextra_info: the generated callbacks read one another's data.
Builder API
The raw builder takes unsafe extern "C" fn callbacks. The examples below write
them with table_bind_callback!, table_init_callback! (also used for
local_init) and table_scan_callback!. Each macro generates the function and
reports a panic in its body as a bind, init or scan error; a plain
unsafe extern "C" fn would abort the process instead. See
Panic-Safe Callbacks.
#![allow(unused)] fn main() { use libduckdb_sys::{duckdb_bind_info, duckdb_connection, duckdb_data_chunk, duckdb_function_info, duckdb_init_info}; quack_rs::table_bind_callback!(my_bind_callback, |_info| {}); quack_rs::table_init_callback!(my_init_callback, |_info| {}); quack_rs::table_scan_callback!(my_scan_callback, |_info, _output| {}); unsafe fn demo(con: duckdb_connection) -> Result<(), quack_rs::error::ExtensionError> { use quack_rs::table::{TableFunctionBuilder, BindInfo, FfiBindData, FfiInitData}; use quack_rs::types::TypeId; TableFunctionBuilder::new("my_function") .param(TypeId::BigInt) // positional parameter types .bind(my_bind_callback) // declare output columns inside bind .init(my_init_callback) .scan(my_scan_callback) .register(con)?; Ok(()) } }
Output columns are declared inside the bind callback using BindInfo::add_result_column,
not on the builder itself.
State management
Bind data
Bind data persists from the bind phase through all scan batches — and through every
later execution of the same plan (see Bind once, init many times above), possibly
read from several threads at once, so FfiBindData::set requires T: Send + Sync.
Use FfiBindData<T> to allocate it safely:
#![allow(unused)] fn main() { use libduckdb_sys::duckdb_bind_info; use quack_rs::table::{BindInfo, FfiBindData}; struct MyBindData { limit: i64, } quack_rs::table_bind_callback!(my_bind, |info| { // `get_parameter_value` returns an RAII `Value`; a NULL argument reads as the default. let n = unsafe { BindInfo::new(info).get_parameter_value(0) }.as_i64_or(0); unsafe { FfiBindData::<MyBindData>::set(info, MyBindData { limit: n }) }; }); }
FfiBindData::set stores the value and registers a destructor so DuckDB frees
it at the right time — no Box::into_raw / Box::from_raw needed.
Init (scan) state
Per-scan state (e.g., a current row index) uses FfiInitData<T> (T: Send + Sync,
since concurrent scan threads share it):
#![allow(unused)] fn main() { use libduckdb_sys::duckdb_init_info; use quack_rs::table::FfiInitData; struct MyScanState { pos: i64, } quack_rs::table_init_callback!(my_init, |info| { unsafe { FfiInitData::<MyScanState>::set(info, MyScanState { pos: 0 }) }; }); }
Complete example: generate_series_ext
The hello-ext example registers generate_series_ext(n BIGINT) which emits
integers 0 .. n-1. See examples/hello-ext/src/lib.rs for the full source.
#![allow(unused)] fn main() { use libduckdb_sys::{duckdb_bind_info, duckdb_connection, duckdb_data_chunk, duckdb_function_info, duckdb_init_info, DuckDBSuccess}; use quack_rs::data_chunk::DataChunk; use quack_rs::table::{BindInfo, FfiBindData, FfiInitData, TableFunctionBuilder}; use quack_rs::types::TypeId; struct GsBindData { total: i64 } struct GsScanState { pos: i64 } // Bind: extract `n`, register one output column quack_rs::table_bind_callback!(gs_bind, |info| { let bind_info = unsafe { BindInfo::new(info) }; // Value is RAII — automatically destroyed when dropped. // A NULL argument reads as the default rather than aborting. let n = unsafe { bind_info.get_parameter_value(0) }.as_i64_or(0); bind_info.add_result_column("value", TypeId::BigInt); unsafe { FfiBindData::<GsBindData>::set(info, GsBindData { total: n }) }; }); // Init: zero-initialise the scan cursor quack_rs::table_init_callback!(gs_init, |info| { unsafe { FfiInitData::<GsScanState>::set(info, GsScanState { pos: 0 }) }; }); // Scan: emit a batch of rows using DataChunk wrapper quack_rs::table_scan_callback!(gs_scan, |info, output| { let chunk = unsafe { DataChunk::from_raw(output) }; // Never unwrap in a callback: a missing state ends the scan instead. let bind = unsafe { FfiBindData::<GsBindData>::get_from_function(info) }; let state = unsafe { FfiInitData::<GsScanState>::get_mut(info) }; let (Some(bind), Some(state)) = (bind, state) else { unsafe { chunk.set_size(0) }; return; }; let remaining = bind.total - state.pos; // Never hard-code the chunk size: use the vector size of the running DuckDB. let capacity = quack_rs::vector::vector_size() as i64; let batch = remaining.min(capacity).max(0) as usize; let mut writer = unsafe { chunk.writer(0) }; for i in 0..batch { unsafe { writer.write_i64(i, state.pos + i as i64) }; } unsafe { chunk.set_size(batch) }; state.pos += batch as i64; }); std::mem::forget(quack_rs::testing::InMemoryDb::open().unwrap()); let (mut db, mut con) = (std::ptr::null_mut(), std::ptr::null_mut()); unsafe { assert_eq!(libduckdb_sys::duckdb_open(std::ptr::null(), &mut db), DuckDBSuccess); assert_eq!(libduckdb_sys::duckdb_connect(db, &mut con), DuckDBSuccess); TableFunctionBuilder::new("generate_series_ext") .param(TypeId::BigInt) .bind(gs_bind) .init(gs_init) .scan(gs_scan) .register(con) .unwrap(); } let sum = |sql: &str| -> i64 { let mut result = unsafe { quack_rs::query::query(con, sql) }.unwrap(); let chunk = result.next_chunk().unwrap().unwrap(); unsafe { chunk.reader(0).read_i64(0) } }; assert_eq!(sum("SELECT sum(value)::BIGINT FROM generate_series_ext(5)"), 10); assert_eq!(sum("SELECT count(*) FROM generate_series_ext(5000)"), 5000); }
Registration
#![allow(unused)] fn main() { use libduckdb_sys::{duckdb_bind_info, duckdb_connection, duckdb_data_chunk, duckdb_function_info, duckdb_init_info}; use quack_rs::table::TableFunctionBuilder; use quack_rs::types::TypeId; quack_rs::table_bind_callback!(gs_bind, |_info| {}); quack_rs::table_init_callback!(gs_init, |_info| {}); quack_rs::table_scan_callback!(gs_scan, |_info, _output| {}); unsafe fn demo(con: duckdb_connection) -> Result<(), quack_rs::error::ExtensionError> { TableFunctionBuilder::new("generate_series_ext") .param(TypeId::BigInt) .bind(gs_bind) .init(gs_init) .scan(gs_scan) .register(con)?; Ok(()) } }
Advanced features
Named parameters
Named parameters let callers pass optional arguments by name (e.g., step := 10):
#![allow(unused)] fn main() { use libduckdb_sys::{duckdb_bind_info, duckdb_connection, duckdb_data_chunk, duckdb_function_info, duckdb_init_info}; use quack_rs::table::TableFunctionBuilder; use quack_rs::types::TypeId; quack_rs::table_bind_callback!(gs_v2_bind, |_info| {}); quack_rs::table_init_callback!(gs_v2_init, |_info| {}); quack_rs::table_scan_callback!(gs_v2_scan, |_info, _output| {}); unsafe fn demo(con: duckdb_connection) -> Result<(), quack_rs::error::ExtensionError> { TableFunctionBuilder::new("gen_series_v2") .param(TypeId::BigInt) // positional: n .named_param("step", TypeId::BigInt) // named: step := <value> .bind(gs_v2_bind) .init(gs_v2_init) .scan(gs_v2_scan) .register(con)?; Ok(()) } }
In the bind callback, read the named parameter with
BindInfo::get_named_parameter_value("step"). Named parameters are optional: if the
query omits step := …, the returned Value wraps a null handle (is_null() is
true), so use a defaulting accessor such as as_i64_or(1).
Registering a name twice
The C API has no table function sets, so a second registration under a name that
already exists — your own earlier one, another extension's, or a built-in such as
range — never adds an overload and the old function keeps answering. What the call
reports differs by DuckDB release: v1.5.x drops the new function and still reports
success, while v1.4.x fails the call without a reason. register therefore checks
duckdb_functions() first and returns an error if the name already belongs to a
table function or table macro (compared case-insensitively). Table functions
registered through the C API live in the in-memory system catalog and are never
persisted, so reloading an extension into a database file never trips this check.
Registering through the entry point's Connection (con.register_table(…))
checks against a snapshot instead: the catalog is listed once, on the first
table function registration, and the snapshot then records that connection's own
registrations, so an extension with many table functions pays for one listing
per LOAD. The snapshot does not see a table function registered on the raw
connection after it was taken — by a direct TableFunctionBuilder::register,
another extension's LOAD or an autoload — so a later register_table under
that name passes the check and DuckDB answers as above: v1.5.x keeps the old
function and reports success, and v1.4.x fails the call, with an error that
names this as a possible cause.
Local init (per-thread state)
local_init allocates per-thread state for a scan that runs on several threads.
It does not make the scan parallel by itself — that is
InitInfo::set_max_threads (see Thread control):
#![allow(unused)] fn main() { use libduckdb_sys::{duckdb_bind_info, duckdb_connection, duckdb_data_chunk, duckdb_function_info, duckdb_init_info}; use quack_rs::table::TableFunctionBuilder; use quack_rs::types::TypeId; quack_rs::table_bind_callback!(gs_v2_bind, |_info| {}); quack_rs::table_init_callback!(gs_v2_init, |_info| {}); quack_rs::table_init_callback!(gs_v2_local_init, |_info| {}); quack_rs::table_scan_callback!(gs_v2_scan, |_info, _output| {}); unsafe fn demo(con: duckdb_connection) -> Result<(), quack_rs::error::ExtensionError> { TableFunctionBuilder::new("gen_series_v2") .param(TypeId::BigInt) .bind(gs_v2_bind) .init(gs_v2_init) .local_init(gs_v2_local_init) // per-thread state allocation .scan(gs_v2_scan) .register(con)?; Ok(()) } }
The local init callback receives duckdb_init_info and can use
FfiLocalInitData::<T>::set to store per-thread state (T: Send).
Thread control
Use InitInfo::set_max_threads in the global init callback to tell DuckDB how
many threads can scan concurrently. The default is 1. Above 1, DuckDB calls the
scan from that many threads at the same time whether or not local_init is
set — and all of them share the one global init data and bind data. Do not use
FfiInitData::get_mut then; keep shared mutable state behind a Mutex or
atomics and read it with FfiInitData::get:
#![allow(unused)] fn main() { use libduckdb_sys::duckdb_init_info; use quack_rs::table::{FfiInitData, InitInfo}; struct MyState { pos: i64 } quack_rs::table_init_callback!(gs_v2_init, |info| { let init_info = unsafe { InitInfo::new(info) }; init_info.set_max_threads(1); unsafe { FfiInitData::<MyState>::set(info, MyState { pos: 0 }) }; }); }
Projection pushdown
Enable projection pushdown to let DuckDB skip unrequested columns:
#![allow(unused)] fn main() { use quack_rs::table::TableFunctionBuilder; fn demo() { let _ = TableFunctionBuilder::new("my_func") .projection_pushdown(true) // ... ; } }
Caution: When projection pushdown is enabled, your scan callback must check which columns DuckDB actually needs using
InitInfo::projected_column_countandInitInfo::projected_column_index. Writing to non-projected columns causes crashes.projected_column_indexreturnsNonepast the end of the projection (the C API itself answers0there, which is indistinguishable from the first column).
See examples/hello-ext/src/lib.rs for a complete example using named_param,
local_init, and set_max_threads.
Complex parameter types
For parameterised types that TypeId cannot express (e.g. LIST(BIGINT),
MAP(VARCHAR, INTEGER), STRUCT(...)), use param_logical and
named_param_logical:
#![allow(unused)] fn main() { use libduckdb_sys::{duckdb_bind_info, duckdb_connection, duckdb_data_chunk, duckdb_function_info, duckdb_init_info}; use quack_rs::table::TableFunctionBuilder; use quack_rs::types::TypeId; quack_rs::table_bind_callback!(bind_fn, |_info| {}); quack_rs::table_init_callback!(init_fn, |_info| {}); quack_rs::table_scan_callback!(scan_fn, |_info, _output| {}); unsafe fn demo(con: duckdb_connection) -> Result<(), quack_rs::error::ExtensionError> { use quack_rs::types::LogicalType; TableFunctionBuilder::new("read_data") .param_logical(LogicalType::list(TypeId::Varchar)) // positional LIST param .named_param_logical("options", LogicalType::map( // named MAP param TypeId::Varchar, TypeId::Varchar, )) .bind(bind_fn) .init(init_fn) .scan(scan_fn) .register(con)?; Ok(()) } }
BindInfo helpers
BindInfo wraps duckdb_bind_info and exposes these methods:
| Method | Description |
|---|---|
add_result_column(name, TypeId) | Declares an output column (a type DuckDB would silently drop, like ANY, is a bind error instead) |
add_result_column_with_type(name, &LogicalType) | Output column with complex type (same check, including nested ANY/INVALID) |
set_cardinality(rows, is_exact) | Cardinality hint for the optimizer. DuckDB 1.x (through 1.5.6) treats is_exact = false as an estimate and an upper bound, true as an estimate only — the reverse of duckdb.h; DuckDB 2.0.0 follows duckdb.h (true records the bound). Pass a rows the scan never exceeds and either flag is safe on both |
set_error(message) | Report a bind-time error (an empty message is replaced by a placeholder) |
parameter_count() | Number of positional parameters |
get_parameter_value(index) | Positional parameter as an RAII Value |
get_named_parameter_value(name) | Named parameter as an RAII Value; a null handle if the query omitted it |
get_parameter(index) / get_named_parameter(name) | The same as a raw duckdb_value, which the caller must destroy |
get_extra_info() | Returns the extra-info pointer set on the function |
get_client_context() | Returns a ClientContext (duckdb-1-5) |
result_column_count() / result_column_name(i) / result_column_type(i) | The target table's columns in a COPY … FROM reader (duckdb-1-5); zero columns otherwise |
InitInfo helpers
InitInfo wraps duckdb_init_info:
| Method | Description |
|---|---|
projected_column_count() | Number of projected columns (with pushdown) |
projected_column_index(idx) | Declared column index at projection position; None when idx is out of range |
set_max_threads(n) | Maximum concurrent scan threads (default 1; shared global state above 1) |
set_error(message) | Report an init-time error (an empty message is replaced by a placeholder) |
get_extra_info() | Returns the extra-info pointer set on the function |
FunctionInfo helpers
FunctionInfo wraps duckdb_function_info (scan callbacks):
| Method | Description |
|---|---|
set_error(message) | Report a scan-time error (an empty message is replaced by a placeholder) |
get_extra_info() | Returns the extra-info pointer set on the function |
Extra info
Use TableFunctionBuilder::extra_info to attach function-level data that is
accessible from all callbacks (bind, init, and scan) via get_extra_info(). The
pointee must be Send + Sync: DuckDB passes the same pointer to callbacks running
on several threads at once, and frees it on whichever thread releases the function.
Example output
SELECT * FROM generate_series_ext(5);
-- 0
-- 1
-- 2
-- 3
-- 4
SELECT value * value AS sq FROM generate_series_ext(4);
-- 0
-- 1
-- 4
-- 9
See also
tablemodule documentationreplacement_scan— for file-path-triggered table scanshello-extREADME