pub struct FfiState<T: AggregateState> { /* private fields */ }Expand description
A generic FFI-compatible state wrapper for use with DuckDB aggregate functions.
FfiState<T> describes the bytes DuckDB allocates for each aggregate
group: size_callback tells DuckDB how many,
init_callback puts a T::default() in them,
with_state and
with_state_mut hand out the T, and
destroy_callback drops it. The type is
never constructed; it only names the layout and its callbacks.
§Known DuckDB limitation
Before DuckDB 2.0.0, two query shapes make every C-API aggregate read
out of bounds. In them DuckDB calls update with a state array holding
one state while
passing count > 1 rows, so the callback reads states[1..count] past the
end of the array — undefined behaviour in any C-API aggregate, whether
built with quack-rs or by hand:
- Window aggregates whose frame is the whole partition, e.g.
agg(x) OVER ()or an explicitROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWINGframe (OVER (PARTITION BY p)withoutORDER BYcrashed on 1.4.4 only in the shapes tried) —WindowConstantAggregator(src/function/window/window_constant_aggregator.cpp, ~lines 106 and 296–299 inDuckDB1.5.5). - Ordered aggregates, e.g.
agg(x ORDER BY y)—src/function/aggregate/sorted_aggregate_function.cpp, ~lines 630–633.
Both pass a CONSTANT_VECTOR of states because the function has no
simple_update (the C API cannot set one), and CAPIAggregateUpdate
(src/main/capi/aggregate_function-c.cpp, ~lines 92–110) hands the
vector’s data pointer to the extension without flattening it. This is a
defect in DuckDB’s C API, not in quack-rs, and it cannot be detected
from inside the callback — reading states[1] to check is itself the
out-of-bounds read. Reported upstream as
duckdb/duckdb#26109.
On DuckDB 1.4.x and 1.5.x, do not use C-API aggregates in those two
query shapes.
Fixed in DuckDB 2.0.0: CAPIAggregateUpdate flattens a non-flat
state vector into one state pointer per row before calling update
(src/main/capi/v1/aggregate_function-c.cpp, lines 104–108 at v2.0.0).
Both shapes then give the right answer, and a plain C aggregate runs them
clean under Valgrind; on 1.5.6 the same program segfaults.
§Memory layout
[ tag: usize ][ T, padded to a multiple of usize ] T stored inline
[ tag: usize ][ *mut T ] T boxedT is stored in the state bytes themselves when its alignment is at most
usize’s and it is at most 256 bytes; otherwise the slot holds a
Box<T>. The size is a multiple of usize either way, because some
DuckDB operators lay states out back to back at a stride of exactly
state_size (the window aggregators), and every state must stay aligned.
DuckDB aligns each state to 8 bytes (AlignValue in AggregateObject
and TupleDataLayout; new[]-allocated buffers elsewhere), which covers
both forms on 64- and 32-bit targets.
§States DuckDB never destroys
DuckDB 1.4.4 to 1.5.5 (not re-measured on 1.5.6) destroys a grouped
aggregate’s states as its result
scan passes them. When the scan stops early — a LIMIT above the
aggregate, an error raised above it, an interrupt — the states it did not
reach are never destroyed (docs/upstream-duckdb-reports.md, item 20):
under LIMIT 10, 2,048 of 300,000 on one thread. Their T is never
dropped. A T stored inline costs nothing more: its bytes belong to
DuckDB’s hash table, which DuckDB frees. What leaks is whatever T
itself owns on the heap (a Vec, a String, a HashMap), and, for a
boxed T, the box. DuckDB 2.0.0 still behaves this way (pinned by
tests/aggregate_leaks.rs). Before 2.0.0 the same holds for a window
frame with an EXCLUDE clause, whose segment tree initialises one extra
state per row and never destroys it (item 35; 5000 Ts undropped over a
5000-row window); 2.0.0 destroys those states
(WindowSegmentTreeLocalState::Evaluate, window_segment_tree.cpp
506–510). Until DuckDB fixes the stopped scan, prefer a state that owns
no heap memory and is small enough to be stored inline.
§States DuckDB never initialised
When a state_init call fails — a panicking T::default(), or an
allocation failure — DuckDB 1.4.4 to 1.5.5 (not re-measured on 1.5.6)
still calls the destructor
on every state row it had created, initialised or not
(RowOperations::InitializeStates is not exception-safe; see
docs/upstream-duckdb-reports.md). That includes the states of other
aggregates in the same query whose state_init never ran. Those bytes
are whatever the allocation held, and dropping them as a T would run a
destructor over garbage.
So init_callback also stores tag, a value
derived from T (and, for a boxed T, from the box’s address), and
destroy_callback drops only a slot whose tag
matches, clearing it first. A slot the destructor has already processed no
longer matches; a slot another aggregate’s type initialised does not match
either. What is left is chance: uninitialised bytes that happen to hold
the tag, one in 264 on a 64-bit target. This is a mitigation of
a DuckDB defect, not a guarantee. Reading bytes DuckDB may never have
written is, in Rust’s abstract machine, a read of uninitialised memory
whatever the load: the destructor uses volatile loads so the compiler
cannot reason from them, but no Rust construct makes such a read defined.
It is the least bad of the options this defect leaves.
The tag depends on the slot’s contents, not its address, because DuckDB
moves states: radix repartitioning of a hash aggregate copies each state’s
bytes to a new row (radix_partitioned_hashtable.cpp) and destroys it
there. The fourth audit’s tag used the address, so every moved state was
skipped and its T leaked — millions per query on eight threads. Moving
a Rust value by copying its bytes is always valid; the copy left behind is
never destroyed.
§Usage
use quack_rs::aggregate::{AggregateState, FfiState};
use libduckdb_sys::{duckdb_function_info, duckdb_aggregate_state, idx_t};
#[derive(Default)]
struct MyState { sum: i64 }
impl AggregateState for MyState {}
// In your registration code:
// .ffi_state::<MyState>()Breaking in 0.18.0: FfiState<T> was a one-word #[repr(C)] struct
with a public inner: *mut T field that always boxed T. The layout is
now private.
Implementations§
Source§impl<T: AggregateState> FfiState<T>
impl<T: AggregateState> FfiState<T>
Sourcepub const fn size() -> usize
pub const fn size() -> usize
Returns the number of bytes DuckDB allocates per state.
§Example
use quack_rs::aggregate::{AggregateState, FfiState};
#[derive(Default)]
struct MyState { val: i64 }
impl AggregateState for MyState {}
// A tag, then the `i64` itself.
assert_eq!(FfiState::<MyState>::size(), 2 * std::mem::size_of::<usize>());Sourcepub const unsafe extern "C" fn size_callback(
_info: duckdb_function_info,
) -> idx_t
pub const unsafe extern "C" fn size_callback( _info: duckdb_function_info, ) -> idx_t
The state_size callback function for use in the builder.
Returns size: the number of bytes DuckDB must
allocate per aggregate group.
§Safety
This is an unsafe extern "C" function pointer. It is safe to pass to
AggregateFunctionBuilder::state_size.
Sourcepub unsafe extern "C" fn init_callback(
info: duckdb_function_info,
state: duckdb_aggregate_state,
)
pub unsafe extern "C" fn init_callback( info: duckdb_function_info, state: duckdb_aggregate_state, )
The state_init callback function for use in the builder.
Stores a T::default() in the state at state (or a box holding one).
§Pitfall L3: T::default() is user code
DuckDB invokes this through an extern "C" function pointer, so a
panic escaping T::default() would abort the process (Rust 1.81+ turns
an unwind across extern "C" into panic_cannot_unwind). The call is
therefore run under
catch_ffi_panic and a panic is
reported through duckdb_aggregate_function_set_error — which
DuckDB’s CAPIAggregateStateInit checks, turning it into an ordinary
SQL error — instead of taking the session down.
§A failing initialiser in a window frame still crashes the host
That error report is itself the hazard when the aggregate runs as a
window function (a ROWS/RANGE frame). DuckDB’s
WindowAggregateStates fills its table of state pointers one entry per
successful initialisation and then, when one fails, destroys the whole
table, unwritten entries included: the destructor is handed pointers
that never pointed at a state. The tag check below cannot help, because
there is no slot to read a tag from. Reproduced with a plain C
aggregate, no Rust involved, on DuckDB 1.5.6 (55 of 68 pointers
garbage) and 2.0.0 (65 of 68); the host dies with SIGSEGV or SIGBUS.
Ungrouped, grouped, DISTINCT and FILTER aggregates report a clean
error. So: T::default() must not panic for an aggregate that can
be used in a window.
§Uninitialised memory
DuckDB does not promise the state allocation is zeroed, so the tag is
cleared (and a boxed slot’s pointer nulled) before any user code
runs. A slot whose T::default() panicked therefore never matches its
tag: with_state returns None for it and
destroy_callback skips it.
§Safety
statemust point tosizewritable bytes allocated byDuckDB, aligned tousize.- This function must only be called once per state allocation.
Sourcepub unsafe extern "C" fn destroy_callback(
states: *mut duckdb_aggregate_state,
count: idx_t,
)
pub unsafe extern "C" fn destroy_callback( states: *mut duckdb_aggregate_state, count: idx_t, )
The state_destroy callback function for use in the builder.
Drops the T in each state in states[0..count] (freeing its box, if
boxed), clearing the tag first so a second call is a no-op.
§Count conversion
If count (an idx_t) cannot be converted to usize, the loop iterates
zero times — no states are freed. This is a defensive choice to avoid
panicking across FFI. In practice, idx_t is u64 and usize is at
least 64 bits on all DuckDB-supported platforms, so this path is
unreachable on supported targets.
§Pitfall L3: T::drop is user code
DuckDB calls the destructor as info.destroy(states, count) — no info
handle, no return value, and therefore no error channel at all
(verified in CAPIAggregateDestructor). A panic escaping T::drop
would abort the process, so each drop runs under
catch_ffi_panic and the message is
discarded. Every remaining state is still dropped: one bad destructor
does not leak the rest of the group.
One case cannot be contained: a T::drop that panics while one of
T’s fields also panics in its own drop during that unwind. Rust
aborts the process for a panic raised while unwinding (“panic in a
destructor during cleanup”) before any catch_unwind sees it.
§Safety
statesmust point to an array ofcountvalidduckdb_aggregate_statepointers, each tosizebytes, aligned tousize, thatDuckDBallocated for this aggregate. A slotinit_callbacknever initialised is skipped (see “StatesDuckDBnever initialised” on the type).
Sourcepub unsafe fn with_state_mut<'a>(
state: duckdb_aggregate_state,
) -> Option<&'a mut T>
pub unsafe fn with_state_mut<'a>( state: duckdb_aggregate_state, ) -> Option<&'a mut T>
Provides safe mutable access to the inner T value.
Returns None if the slot holds no T — its T::default() panicked
(and DuckDB is about to fail the query), or it was destroyed.
§Pitfall L3: No panic across FFI
This method returns Option<&mut T> rather than unwrapping, so callers
can use if let Some(state) = ... patterns without panicking.
§Safety
statemust point to a stateDuckDBallocated for this aggregate andinit_callbackinitialised.- No other reference to the same
Tmust exist simultaneously. - The returned reference must not outlive the callback that received
state. The lifetime'ais the caller’s to choose and nothing ties it tostate;DuckDBdestroys the state — anddestroy_callbackdrops theT— once it is finalized or merged, and may move its bytes between callbacks.
§Example
use quack_rs::aggregate::{AggregateState, FfiState};
#[derive(Default)]
struct Counter { n: u64 }
impl AggregateState for Counter {}
// Inside your update callback:
// let ffi_state: duckdb_aggregate_state = ...;
// if let Some(state) = unsafe { FfiState::<Counter>::with_state_mut(ffi_state) } {
// state.n += 1;
// }Sourcepub unsafe fn with_state<'a>(state: duckdb_aggregate_state) -> Option<&'a T>
pub unsafe fn with_state<'a>(state: duckdb_aggregate_state) -> Option<&'a T>
Provides safe immutable access to the inner T value.
See with_state_mut for safety requirements.
§Safety
Same as with_state_mut, but only borrows immutably: no mutable
reference to the same T may exist while this one does.