Skip to main content

FfiState

Struct FfiState 

Source
pub struct FfiState<T: AggregateState> { /* private fields */ }
Expand description

A generic FFI-compatible state wrapper for use with DuckDB aggregate functions.

FfiState<T> describes the bytes DuckDB allocates for each aggregate group: size_callback tells DuckDB how many, init_callback puts a T::default() in them, with_state and with_state_mut hand out the T, and destroy_callback drops it. The type is never constructed; it only names the layout and its callbacks.

§Known DuckDB limitation

Before DuckDB 2.0.0, two query shapes make every C-API aggregate read out of bounds. In them DuckDB calls update with a state array holding one state while passing count > 1 rows, so the callback reads states[1..count] past the end of the array — undefined behaviour in any C-API aggregate, whether built with quack-rs or by hand:

  • Window aggregates whose frame is the whole partition, e.g. agg(x) OVER () or an explicit ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING frame (OVER (PARTITION BY p) without ORDER BY crashed on 1.4.4 only in the shapes tried) — WindowConstantAggregator (src/function/window/window_constant_aggregator.cpp, ~lines 106 and 296–299 in DuckDB 1.5.5).
  • Ordered aggregates, e.g. agg(x ORDER BY y) — src/function/aggregate/sorted_aggregate_function.cpp, ~lines 630–633.

Both pass a CONSTANT_VECTOR of states because the function has no simple_update (the C API cannot set one), and CAPIAggregateUpdate (src/main/capi/aggregate_function-c.cpp, ~lines 92–110) hands the vector’s data pointer to the extension without flattening it. This is a defect in DuckDB’s C API, not in quack-rs, and it cannot be detected from inside the callback — reading states[1] to check is itself the out-of-bounds read. Reported upstream as duckdb/duckdb#26109. On DuckDB 1.4.x and 1.5.x, do not use C-API aggregates in those two query shapes.

Fixed in DuckDB 2.0.0: CAPIAggregateUpdate flattens a non-flat state vector into one state pointer per row before calling update (src/main/capi/v1/aggregate_function-c.cpp, lines 104–108 at v2.0.0). Both shapes then give the right answer, and a plain C aggregate runs them clean under Valgrind; on 1.5.6 the same program segfaults.

§Memory layout

[ tag: usize ][ T, padded to a multiple of usize ]   T stored inline
[ tag: usize ][ *mut T                          ]   T boxed

T is stored in the state bytes themselves when its alignment is at most usize’s and it is at most 256 bytes; otherwise the slot holds a Box<T>. The size is a multiple of usize either way, because some DuckDB operators lay states out back to back at a stride of exactly state_size (the window aggregators), and every state must stay aligned. DuckDB aligns each state to 8 bytes (AlignValue in AggregateObject and TupleDataLayout; new[]-allocated buffers elsewhere), which covers both forms on 64- and 32-bit targets.

§States DuckDB never destroys

DuckDB 1.4.4 to 1.5.5 (not re-measured on 1.5.6) destroys a grouped aggregate’s states as its result scan passes them. When the scan stops early — a LIMIT above the aggregate, an error raised above it, an interrupt — the states it did not reach are never destroyed (docs/upstream-duckdb-reports.md, item 20): under LIMIT 10, 2,048 of 300,000 on one thread. Their T is never dropped. A T stored inline costs nothing more: its bytes belong to DuckDB’s hash table, which DuckDB frees. What leaks is whatever T itself owns on the heap (a Vec, a String, a HashMap), and, for a boxed T, the box. DuckDB 2.0.0 still behaves this way (pinned by tests/aggregate_leaks.rs). Before 2.0.0 the same holds for a window frame with an EXCLUDE clause, whose segment tree initialises one extra state per row and never destroys it (item 35; 5000 Ts undropped over a 5000-row window); 2.0.0 destroys those states (WindowSegmentTreeLocalState::Evaluate, window_segment_tree.cpp 506–510). Until DuckDB fixes the stopped scan, prefer a state that owns no heap memory and is small enough to be stored inline.

§States DuckDB never initialised

When a state_init call fails — a panicking T::default(), or an allocation failure — DuckDB 1.4.4 to 1.5.5 (not re-measured on 1.5.6) still calls the destructor on every state row it had created, initialised or not (RowOperations::InitializeStates is not exception-safe; see docs/upstream-duckdb-reports.md). That includes the states of other aggregates in the same query whose state_init never ran. Those bytes are whatever the allocation held, and dropping them as a T would run a destructor over garbage.

So init_callback also stores tag, a value derived from T (and, for a boxed T, from the box’s address), and destroy_callback drops only a slot whose tag matches, clearing it first. A slot the destructor has already processed no longer matches; a slot another aggregate’s type initialised does not match either. What is left is chance: uninitialised bytes that happen to hold the tag, one in 264 on a 64-bit target. This is a mitigation of a DuckDB defect, not a guarantee. Reading bytes DuckDB may never have written is, in Rust’s abstract machine, a read of uninitialised memory whatever the load: the destructor uses volatile loads so the compiler cannot reason from them, but no Rust construct makes such a read defined. It is the least bad of the options this defect leaves.

The tag depends on the slot’s contents, not its address, because DuckDB moves states: radix repartitioning of a hash aggregate copies each state’s bytes to a new row (radix_partitioned_hashtable.cpp) and destroys it there. The fourth audit’s tag used the address, so every moved state was skipped and its T leaked — millions per query on eight threads. Moving a Rust value by copying its bytes is always valid; the copy left behind is never destroyed.

§Usage

use quack_rs::aggregate::{AggregateState, FfiState};
use libduckdb_sys::{duckdb_function_info, duckdb_aggregate_state, idx_t};

#[derive(Default)]
struct MyState { sum: i64 }
impl AggregateState for MyState {}

// In your registration code:
// .ffi_state::<MyState>()

Breaking in 0.18.0: FfiState<T> was a one-word #[repr(C)] struct with a public inner: *mut T field that always boxed T. The layout is now private.

Implementations§

Source§

impl<T: AggregateState> FfiState<T>

Source

pub const fn size() -> usize

Returns the number of bytes DuckDB allocates per state.

§Example
use quack_rs::aggregate::{AggregateState, FfiState};

#[derive(Default)]
struct MyState { val: i64 }
impl AggregateState for MyState {}

// A tag, then the `i64` itself.
assert_eq!(FfiState::<MyState>::size(), 2 * std::mem::size_of::<usize>());
Source

pub const unsafe extern "C" fn size_callback( _info: duckdb_function_info, ) -> idx_t

The state_size callback function for use in the builder.

Returns size: the number of bytes DuckDB must allocate per aggregate group.

§Safety

This is an unsafe extern "C" function pointer. It is safe to pass to AggregateFunctionBuilder::state_size.

Source

pub unsafe extern "C" fn init_callback( info: duckdb_function_info, state: duckdb_aggregate_state, )

The state_init callback function for use in the builder.

Stores a T::default() in the state at state (or a box holding one).

§Pitfall L3: T::default() is user code

DuckDB invokes this through an extern "C" function pointer, so a panic escaping T::default() would abort the process (Rust 1.81+ turns an unwind across extern "C" into panic_cannot_unwind). The call is therefore run under catch_ffi_panic and a panic is reported through duckdb_aggregate_function_set_error — which DuckDB’s CAPIAggregateStateInit checks, turning it into an ordinary SQL error — instead of taking the session down.

§A failing initialiser in a window frame still crashes the host

That error report is itself the hazard when the aggregate runs as a window function (a ROWS/RANGE frame). DuckDB’s WindowAggregateStates fills its table of state pointers one entry per successful initialisation and then, when one fails, destroys the whole table, unwritten entries included: the destructor is handed pointers that never pointed at a state. The tag check below cannot help, because there is no slot to read a tag from. Reproduced with a plain C aggregate, no Rust involved, on DuckDB 1.5.6 (55 of 68 pointers garbage) and 2.0.0 (65 of 68); the host dies with SIGSEGV or SIGBUS. Ungrouped, grouped, DISTINCT and FILTER aggregates report a clean error. So: T::default() must not panic for an aggregate that can be used in a window.

§Uninitialised memory

DuckDB does not promise the state allocation is zeroed, so the tag is cleared (and a boxed slot’s pointer nulled) before any user code runs. A slot whose T::default() panicked therefore never matches its tag: with_state returns None for it and destroy_callback skips it.

§Safety
  • state must point to size writable bytes allocated by DuckDB, aligned to usize.
  • This function must only be called once per state allocation.
Source

pub unsafe extern "C" fn destroy_callback( states: *mut duckdb_aggregate_state, count: idx_t, )

The state_destroy callback function for use in the builder.

Drops the T in each state in states[0..count] (freeing its box, if boxed), clearing the tag first so a second call is a no-op.

§Count conversion

If count (an idx_t) cannot be converted to usize, the loop iterates zero times — no states are freed. This is a defensive choice to avoid panicking across FFI. In practice, idx_t is u64 and usize is at least 64 bits on all DuckDB-supported platforms, so this path is unreachable on supported targets.

§Pitfall L3: T::drop is user code

DuckDB calls the destructor as info.destroy(states, count) — no info handle, no return value, and therefore no error channel at all (verified in CAPIAggregateDestructor). A panic escaping T::drop would abort the process, so each drop runs under catch_ffi_panic and the message is discarded. Every remaining state is still dropped: one bad destructor does not leak the rest of the group.

One case cannot be contained: a T::drop that panics while one of T’s fields also panics in its own drop during that unwind. Rust aborts the process for a panic raised while unwinding (“panic in a destructor during cleanup”) before any catch_unwind sees it.

§Safety
  • states must point to an array of count valid duckdb_aggregate_state pointers, each to size bytes, aligned to usize, that DuckDB allocated for this aggregate. A slot init_callback never initialised is skipped (see “States DuckDB never initialised” on the type).
Source

pub unsafe fn with_state_mut<'a>( state: duckdb_aggregate_state, ) -> Option<&'a mut T>

Provides safe mutable access to the inner T value.

Returns None if the slot holds no T — its T::default() panicked (and DuckDB is about to fail the query), or it was destroyed.

§Pitfall L3: No panic across FFI

This method returns Option<&mut T> rather than unwrapping, so callers can use if let Some(state) = ... patterns without panicking.

§Safety
  • state must point to a state DuckDB allocated for this aggregate and init_callback initialised.
  • No other reference to the same T must exist simultaneously.
  • The returned reference must not outlive the callback that received state. The lifetime 'a is the caller’s to choose and nothing ties it to state; DuckDB destroys the state — and destroy_callback drops the T — once it is finalized or merged, and may move its bytes between callbacks.
§Example
use quack_rs::aggregate::{AggregateState, FfiState};

#[derive(Default)]
struct Counter { n: u64 }
impl AggregateState for Counter {}

// Inside your update callback:
// let ffi_state: duckdb_aggregate_state = ...;
// if let Some(state) = unsafe { FfiState::<Counter>::with_state_mut(ffi_state) } {
//     state.n += 1;
// }
Source

pub unsafe fn with_state<'a>(state: duckdb_aggregate_state) -> Option<&'a T>

Provides safe immutable access to the inner T value.

See with_state_mut for safety requirements.

§Safety

Same as with_state_mut, but only borrows immutably: no mutable reference to the same T may exist while this one does.

Trait Implementations§

Source§

impl<T: AggregateState> Debug for FfiState<T>

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Auto Trait Implementations§

§

impl<T> Freeze for FfiState<T>

§

impl<T> RefUnwindSafe for FfiState<T>

§

impl<T> Send for FfiState<T>

§

impl<T> Sync for FfiState<T>

§

impl<T> Unpin for FfiState<T>

§

impl<T> UnsafeUnpin for FfiState<T>

§

impl<T> UnwindSafe for FfiState<T>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.