Skip to main content

Module arrow

Module arrow 

Source
Expand description

Arrow C Data Interface bridge (duckdb-1-5-4 feature).

DuckDB’s C API has a conversion family (already in 1.4.4) that moves data between a duckdb_data_chunk and the Arrow C Data Interface without going through a query result:

DuckDB C APIquack-rs
duckdb_connection_get_arrow_optionsArrowOptions::from_connection, ArrowOptions::from_raw_connection
duckdb_result_get_arrow_optionsArrowOptions::from_result
duckdb_destroy_arrow_optionsArrowOptions’s Drop
duckdb_to_arrow_schemato_arrow_schema
duckdb_data_chunk_to_arrowdata_chunk_to_arrow
duckdb_schema_from_arrowschema_from_arrow
duckdb_data_chunk_from_arrowdata_chunk_from_arrow
duckdb_destroy_arrow_converted_schemaArrowConvertedSchema’s Drop
out_schema->release(out_schema)ArrowSchema’s Drop
out_arrow_array->release(out_arrow_array)ArrowArray’s Drop

§No arrow crate dependency

The Arrow C Data Interface is an ABI, not a library: ArrowSchema and ArrowArray are plain #[repr(C)] records with a release callback. libduckdb-sys defines them directly (and asserts in its own test suite that they match arrow-rs’s FFI_ArrowSchema / FFI_ArrowArray field-for-field), so quack-rs bridges to Arrow without pulling in arrow, and an extension that does use arrow-rs can hand values across with a pointer cast — see Bridging to arrow-rs.

§Ownership, as DuckDB actually implements it

Every rule below was read out of src/main/capi/arrow-c.cpp and src/common/arrow/arrow_converter.cpp, not inferred from the header:

  • duckdb_to_arrow_schema / duckdb_data_chunk_to_arrow fill a caller-allocated struct. They never release what was already there, so the destination must start out empty. Both install release last, after every fallible step, so a failed conversion leaves a struct with release == NULL and nothing to free. to_arrow_schema and data_chunk_to_arrow therefore start from a fresh ArrowSchema::empty / ArrowArray::empty and only build the owning wrapper on success.
  • duckdb_schema_from_arrow does not take the schema. It reads it (PopulateArrowTableSchema takes const ArrowSchema &) and the caller still owns it, which is why schema_from_arrow borrows.
  • duckdb_data_chunk_from_arrow takes the array. It sets arrow_array->release = nullptr before the conversion loop body, so ownership moves on the error path too. data_chunk_from_arrow takes the ArrowArray by value for exactly that reason. The one case where DuckDB before 2.0.0 does not claim it is a zero-column schema, where the loop never runs — which is handled by the same code, because the by-value array is dropped on the way out and its Drop releases only if release survived. DuckDB 2.0.0 claims it before the loop in that case too.

§What this module refuses that DuckDB would not

duckdb_data_chunk_from_arrow indexes arrow_array->children[i] once per column in the converted schema, with no bounds check, dereferences each child without a null check, reads offset + length rows of each child without comparing its length, and dereferences the array without checking whether it has already been released. Those are segfaults or out-of-bounds reads rather than errors. data_chunk_from_arrow checks them first and returns an ErrorData instead — which is why ArrowConvertedSchema remembers the column count of the schema it was built from.

It also refuses a zero-row array. duckdb_data_chunk_from_arrow passes arrow_array->length through as the chunk’s capacity (dchunk->Initialize(alloc, types, length)), and VectorCacheBuffer turns a capacity of zero into Allocator::AllocateData(0), whose D_ASSERT(size > 0) aborts a debug build of DuckDB. A release build allocates nothing and carries on — so whether an empty batch works depends on how the engine happens to have been compiled, which is not a contract worth exposing. Skip empty batches, or build the empty chunk directly with OwnedDataChunk::new, which allocates a full-size capacity and is unaffected.

§What it still cannot check

Whether each child array’s buffers match the type its schema declares. ArrowToDuckDBConversion::ColumnArrowToDuckDB reads array.buffers[1] for a primitive column without testing n_buffers or the pointer, so a child that says "i" but carries no data buffer is a null dereference inside DuckDB, not an error. (Its sibling GetValidityMask is guarded — it tests n_buffers > 0 && buffers[0] — so the crash comes from the data buffer, not the validity one.) Validating that would mean reimplementing Arrow’s layout rules for every format string, so this module does not pretend to: an array handed to data_chunk_from_arrow must be one a conforming Arrow producer built. Arrays that came from data_chunk_to_arrow, from arrow-rs, or from any other real Arrow implementation qualify.

Nor whether the array conforms to the converted schema: an Arrow array carries no type, so DuckDB reads each child as the format the schema declares, and an int32 child imported under a utf8 schema is read as string offsets. And length is taken on trust: DuckDB allocates the chunk for that many rows before its error handling starts, so an absurd length aborts the process with an allocation failure. Both are part of data_chunk_from_arrow’s # Safety contract.

§Round trips that lose information

DuckDB’s own converters do not round-trip every type exactly: a TIMETZ comes back as TIME with its offset dropped (01:02:03+05:30 returns as 01:02:03), and a BIT comes back as BLOB.

§Example: chunk → Arrow → chunk

use quack_rs::arrow::{
    data_chunk_from_arrow, data_chunk_to_arrow, schema_from_arrow, to_arrow_schema,
    ArrowOptions,
};
use quack_rs::types::{LogicalType, TypeId};

// SAFETY: `con` is a live DuckDB connection that outlives `options`.
let options = unsafe { ArrowOptions::from_raw_connection(con) }?;

let id = LogicalType::new(TypeId::Integer);
let mut schema = to_arrow_schema(&options, &[("id", &id)])?;
let array = data_chunk_to_arrow(&options, chunk)?;

// ... hand `schema` / `array` to any Arrow consumer, or convert back:
// SAFETY: `con` is a live DuckDB connection.
let converted = unsafe { schema_from_arrow(con, &mut schema) }?;
// SAFETY: same connection, and `array` was produced against `schema`.
let chunk = unsafe { data_chunk_from_arrow(con, array, &converted) }?;
assert_eq!(chunk.column_count(), 1);

§Bridging to arrow-rs

RawArrowSchema and RawArrowArray are the ABI records themselves, so a *mut FFI_ArrowSchema and a *mut RawArrowSchema address the same bytes:

ⓘ
// Export a quack-rs array into arrow-rs.
let ffi: FFI_ArrowArray = unsafe { std::mem::transmute(array.into_raw()) };

// Import an arrow-rs array into quack-rs, neutralising the source so only
// one side ever calls `release`.
let mut ffi = /* FFI_ArrowArray */;
let array = unsafe { ArrowArray::take_from(std::ptr::from_mut(&mut ffi).cast()) };

take_from is the safer half of that pair: it moves the record out and writes a released placeholder back, so the foreign wrapper’s own Drop becomes a no-op instead of a double free.

§Thread safety

None of these types are Send or Sync. The Arrow C Data Interface says nothing about which thread may call release, and duckdb_arrow_options wraps a ClientProperties that borrows the connection’s client context.

Structs§

ArrowArray
An owned Arrow C Data Interface array.
ArrowConvertedSchema
An Arrow schema translated into DuckDB’s own type descriptors.
ArrowOptions
The Arrow production settings of a connection or a result.
ArrowSchema
An owned Arrow C Data Interface schema.
RawArrowArray
The Arrow C Data Interface ArrowArray ABI record, re-exported from libduckdb-sys.
RawArrowSchema
The Arrow C Data Interface ArrowSchema ABI record, re-exported from libduckdb-sys.

Functions§

data_chunk_from_arrow⚠
Imports an Arrow struct array as a DuckDB data chunk (duckdb_data_chunk_from_arrow).
data_chunk_to_arrow
Exports a DuckDB data chunk as an Arrow struct array (duckdb_data_chunk_to_arrow).
schema_from_arrow⚠
Translates an Arrow schema into DuckDB type descriptors (duckdb_schema_from_arrow).
to_arrow_schema
Renders a DuckDB schema as an Arrow schema (duckdb_to_arrow_schema).