Expand description
Arrow C Data Interface bridge (duckdb-1-5-4 feature).
DuckDB’s C API has a conversion family (already in 1.4.4) that moves data
between a duckdb_data_chunk and the Arrow C Data Interface without going through a
query result:
DuckDB C API | quack-rs |
|---|---|
duckdb_connection_get_arrow_options | ArrowOptions::from_connection, ArrowOptions::from_raw_connection |
duckdb_result_get_arrow_options | ArrowOptions::from_result |
duckdb_destroy_arrow_options | ArrowOptions’s Drop |
duckdb_to_arrow_schema | to_arrow_schema |
duckdb_data_chunk_to_arrow | data_chunk_to_arrow |
duckdb_schema_from_arrow | schema_from_arrow |
duckdb_data_chunk_from_arrow | data_chunk_from_arrow |
duckdb_destroy_arrow_converted_schema | ArrowConvertedSchema’s Drop |
out_schema->release(out_schema) | ArrowSchema’s Drop |
out_arrow_array->release(out_arrow_array) | ArrowArray’s Drop |
§No arrow crate dependency
The Arrow C Data Interface is an ABI, not a library: ArrowSchema and
ArrowArray are plain #[repr(C)] records with a release callback.
libduckdb-sys defines them directly (and asserts in its own test suite that
they match arrow-rs’s FFI_ArrowSchema / FFI_ArrowArray field-for-field),
so quack-rs bridges to Arrow without pulling in arrow, and an extension
that does use arrow-rs can hand values across with a pointer cast — see
Bridging to arrow-rs.
§Ownership, as DuckDB actually implements it
Every rule below was read out of src/main/capi/arrow-c.cpp and
src/common/arrow/arrow_converter.cpp, not inferred from the header:
duckdb_to_arrow_schema/duckdb_data_chunk_to_arrowfill a caller-allocated struct. They never release what was already there, so the destination must start out empty. Both installreleaselast, after every fallible step, so a failed conversion leaves a struct withrelease == NULLand nothing to free.to_arrow_schemaanddata_chunk_to_arrowtherefore start from a freshArrowSchema::empty/ArrowArray::emptyand only build the owning wrapper on success.duckdb_schema_from_arrowdoes not take the schema. It reads it (PopulateArrowTableSchematakesconst ArrowSchema &) and the caller still owns it, which is whyschema_from_arrowborrows.duckdb_data_chunk_from_arrowtakes the array. It setsarrow_array->release = nullptrbefore the conversion loop body, so ownership moves on the error path too.data_chunk_from_arrowtakes theArrowArrayby value for exactly that reason. The one case whereDuckDBbefore 2.0.0 does not claim it is a zero-column schema, where the loop never runs — which is handled by the same code, because the by-value array is dropped on the way out and itsDropreleases only ifreleasesurvived.DuckDB2.0.0 claims it before the loop in that case too.
§What this module refuses that DuckDB would not
duckdb_data_chunk_from_arrow indexes arrow_array->children[i] once per
column in the converted schema, with no bounds check, dereferences each
child without a null check, reads offset + length rows of each child
without comparing its length, and dereferences the array without checking
whether it has already been released. Those are segfaults or out-of-bounds
reads rather than errors. data_chunk_from_arrow checks them first and
returns an ErrorData instead — which is why ArrowConvertedSchema
remembers the column count of the schema it was built from.
It also refuses a zero-row array. duckdb_data_chunk_from_arrow passes
arrow_array->length through as the chunk’s capacity
(dchunk->Initialize(alloc, types, length)), and VectorCacheBuffer
turns a capacity of zero into Allocator::AllocateData(0), whose
D_ASSERT(size > 0) aborts a debug build of DuckDB. A release build
allocates nothing and carries on — so whether an empty batch works depends
on how the engine happens to have been compiled, which is not a contract
worth exposing. Skip empty batches, or build the empty chunk directly with
OwnedDataChunk::new, which allocates
a full-size capacity and is unaffected.
§What it still cannot check
Whether each child array’s buffers match the type its schema declares.
ArrowToDuckDBConversion::ColumnArrowToDuckDB reads array.buffers[1] for a
primitive column without testing n_buffers or the pointer, so a child that
says "i" but carries no data buffer is a null dereference inside DuckDB,
not an error. (Its sibling GetValidityMask is guarded — it tests
n_buffers > 0 && buffers[0] — so the crash comes from the data buffer, not
the validity one.) Validating that would mean reimplementing Arrow’s layout
rules for every format string, so this module does not pretend to: an array
handed to data_chunk_from_arrow must be one a conforming Arrow producer
built. Arrays that came from data_chunk_to_arrow, from arrow-rs, or from
any other real Arrow implementation qualify.
Nor whether the array conforms to the converted schema: an Arrow array
carries no type, so DuckDB reads each child as the format the schema
declares, and an int32 child imported under a utf8 schema is read as
string offsets. And length is taken on trust: DuckDB allocates the
chunk for that many rows before its error handling starts, so an absurd
length aborts the process with an allocation failure. Both are part of
data_chunk_from_arrow’s # Safety contract.
§Round trips that lose information
DuckDB’s own converters do not round-trip every type exactly: a TIMETZ
comes back as TIME with its offset dropped (01:02:03+05:30 returns as
01:02:03), and a BIT comes back as BLOB.
§Example: chunk → Arrow → chunk
use quack_rs::arrow::{
data_chunk_from_arrow, data_chunk_to_arrow, schema_from_arrow, to_arrow_schema,
ArrowOptions,
};
use quack_rs::types::{LogicalType, TypeId};
// SAFETY: `con` is a live DuckDB connection that outlives `options`.
let options = unsafe { ArrowOptions::from_raw_connection(con) }?;
let id = LogicalType::new(TypeId::Integer);
let mut schema = to_arrow_schema(&options, &[("id", &id)])?;
let array = data_chunk_to_arrow(&options, chunk)?;
// ... hand `schema` / `array` to any Arrow consumer, or convert back:
// SAFETY: `con` is a live DuckDB connection.
let converted = unsafe { schema_from_arrow(con, &mut schema) }?;
// SAFETY: same connection, and `array` was produced against `schema`.
let chunk = unsafe { data_chunk_from_arrow(con, array, &converted) }?;
assert_eq!(chunk.column_count(), 1);§Bridging to arrow-rs
RawArrowSchema and RawArrowArray are the ABI records themselves, so a
*mut FFI_ArrowSchema and a *mut RawArrowSchema address the same bytes:
// Export a quack-rs array into arrow-rs.
let ffi: FFI_ArrowArray = unsafe { std::mem::transmute(array.into_raw()) };
// Import an arrow-rs array into quack-rs, neutralising the source so only
// one side ever calls `release`.
let mut ffi = /* FFI_ArrowArray */;
let array = unsafe { ArrowArray::take_from(std::ptr::from_mut(&mut ffi).cast()) };take_from is the safer half of that pair: it moves
the record out and writes a released placeholder back, so the foreign
wrapper’s own Drop becomes a no-op instead of a double free.
§Thread safety
None of these types are Send or Sync. The Arrow C Data Interface says
nothing about which thread may call release, and duckdb_arrow_options
wraps a ClientProperties that borrows the connection’s client context.
Structs§
- Arrow
Array - An owned Arrow C Data Interface array.
- Arrow
Converted Schema - An Arrow schema translated into
DuckDB’s own type descriptors. - Arrow
Options - The Arrow production settings of a connection or a result.
- Arrow
Schema - An owned Arrow C Data Interface schema.
- RawArrow
Array - The Arrow C Data Interface
ArrowArrayABI record, re-exported fromlibduckdb-sys. - RawArrow
Schema - The Arrow C Data Interface
ArrowSchemaABI record, re-exported fromlibduckdb-sys.
Functions§
- data_
chunk_ ⚠from_ arrow - Imports an Arrow struct array as a
DuckDBdata chunk (duckdb_data_chunk_from_arrow). - data_
chunk_ to_ arrow - Exports a
DuckDBdata chunk as an Arrow struct array (duckdb_data_chunk_to_arrow). - schema_
from_ ⚠arrow - Translates an Arrow schema into
DuckDBtype descriptors (duckdb_schema_from_arrow). - to_
arrow_ schema - Renders a
DuckDBschema as an Arrow schema (duckdb_to_arrow_schema).