quack_rs/arrow.rs
1// SPDX-License-Identifier: MIT
2// Copyright 2026 Tom F. <https://github.com/tomtom215/>
3// My way of giving something small back to the open source community
4// and encouraging more Rust development!
5
6//! Arrow C Data Interface bridge (`duckdb-1-5-4` feature).
7//!
8//! `DuckDB`'s C API has a conversion family (already in 1.4.4) that moves data
9//! between a `duckdb_data_chunk` and the [Arrow C Data Interface] without going through a
10//! query result:
11//!
12//! | `DuckDB` C API | quack-rs |
13//! |---|---|
14//! | `duckdb_connection_get_arrow_options` | [`ArrowOptions::from_connection`], [`ArrowOptions::from_raw_connection`] |
15//! | `duckdb_result_get_arrow_options` | [`ArrowOptions::from_result`] |
16//! | `duckdb_destroy_arrow_options` | [`ArrowOptions`]'s `Drop` |
17//! | `duckdb_to_arrow_schema` | [`to_arrow_schema`] |
18//! | `duckdb_data_chunk_to_arrow` | [`data_chunk_to_arrow`] |
19//! | `duckdb_schema_from_arrow` | [`schema_from_arrow`] |
20//! | `duckdb_data_chunk_from_arrow` | [`data_chunk_from_arrow`] |
21//! | `duckdb_destroy_arrow_converted_schema` | [`ArrowConvertedSchema`]'s `Drop` |
22//! | `out_schema->release(out_schema)` | [`ArrowSchema`]'s `Drop` |
23//! | `out_arrow_array->release(out_arrow_array)` | [`ArrowArray`]'s `Drop` |
24//!
25//! # No `arrow` crate dependency
26//!
27//! The Arrow C Data Interface is an ABI, not a library: `ArrowSchema` and
28//! `ArrowArray` are plain `#[repr(C)]` records with a `release` callback.
29//! `libduckdb-sys` defines them directly (and asserts in its own test suite that
30//! they match arrow-rs's `FFI_ArrowSchema` / `FFI_ArrowArray` field-for-field),
31//! so quack-rs bridges to Arrow without pulling in `arrow`, and an extension
32//! that *does* use arrow-rs can hand values across with a pointer cast — see
33//! [Bridging to arrow-rs](#bridging-to-arrow-rs).
34//!
35//! # Ownership, as `DuckDB` actually implements it
36//!
37//! Every rule below was read out of `src/main/capi/arrow-c.cpp` and
38//! `src/common/arrow/arrow_converter.cpp`, not inferred from the header:
39//!
40//! - **`duckdb_to_arrow_schema` / `duckdb_data_chunk_to_arrow` fill a
41//! caller-allocated struct.** They never release what was already there, so
42//! the destination must start out empty. Both install `release` **last**,
43//! after every fallible step, so a failed conversion leaves a struct with
44//! `release == NULL` and nothing to free. [`to_arrow_schema`] and
45//! [`data_chunk_to_arrow`] therefore start from a fresh
46//! [`ArrowSchema::empty`] / [`ArrowArray::empty`] and only build the owning
47//! wrapper on success.
48//! - **`duckdb_schema_from_arrow` does not take the schema.** It reads it
49//! (`PopulateArrowTableSchema` takes `const ArrowSchema &`) and the caller
50//! still owns it, which is why [`schema_from_arrow`] borrows.
51//! - **`duckdb_data_chunk_from_arrow` takes the array.** It sets
52//! `arrow_array->release = nullptr` *before* the conversion loop body, so
53//! ownership moves on the error path too. [`data_chunk_from_arrow`] takes the
54//! [`ArrowArray`] **by value** for exactly that reason. The one case where
55//! `DuckDB` before 2.0.0 does *not* claim it is a zero-column schema, where
56//! the loop never runs — which is handled by the same code, because the
57//! by-value array is dropped on the way out and its `Drop` releases only if
58//! `release` survived. `DuckDB` 2.0.0 claims it before the loop in that case
59//! too.
60//!
61//! # What this module refuses that `DuckDB` would not
62//!
63//! `duckdb_data_chunk_from_arrow` indexes `arrow_array->children[i]` once per
64//! column in the converted schema, with no bounds check, dereferences each
65//! child without a null check, reads `offset + length` rows of each child
66//! without comparing its length, and dereferences the array without checking
67//! whether it has already been released. Those are segfaults or out-of-bounds
68//! reads rather than errors. [`data_chunk_from_arrow`] checks them first and
69//! returns an [`ErrorData`][crate::error_data::ErrorData] instead — which is why [`ArrowConvertedSchema`]
70//! remembers the column count of the schema it was built from.
71//!
72//! It also refuses a **zero-row** array. `duckdb_data_chunk_from_arrow` passes
73//! `arrow_array->length` through as the chunk's *capacity*
74//! (`dchunk->Initialize(alloc, types, length)`), and `VectorCacheBuffer`
75//! turns a capacity of zero into `Allocator::AllocateData(0)`, whose
76//! `D_ASSERT(size > 0)` aborts a **debug** build of `DuckDB`. A release build
77//! allocates nothing and carries on — so whether an empty batch works depends
78//! on how the engine happens to have been compiled, which is not a contract
79//! worth exposing. Skip empty batches, or build the empty chunk directly with
80//! [`OwnedDataChunk::new`][crate::query::OwnedDataChunk::new], which allocates
81//! a full-size capacity and is unaffected.
82//!
83//! # What it still cannot check
84//!
85//! Whether each child array's **buffers** match the type its schema declares.
86//! `ArrowToDuckDBConversion::ColumnArrowToDuckDB` reads `array.buffers[1]` for a
87//! primitive column without testing `n_buffers` or the pointer, so a child that
88//! says `"i"` but carries no data buffer is a null dereference inside `DuckDB`,
89//! not an error. (Its sibling `GetValidityMask` *is* guarded — it tests
90//! `n_buffers > 0 && buffers[0]` — so the crash comes from the data buffer, not
91//! the validity one.) Validating that would mean reimplementing Arrow's layout
92//! rules for every format string, so this module does not pretend to: an array
93//! handed to [`data_chunk_from_arrow`] must be one a conforming Arrow producer
94//! built. Arrays that came from [`data_chunk_to_arrow`], from arrow-rs, or from
95//! any other real Arrow implementation qualify.
96//!
97//! Nor whether the array **conforms to the converted schema**: an Arrow array
98//! carries no type, so `DuckDB` reads each child as the format the schema
99//! declares, and an `int32` child imported under a `utf8` schema is read as
100//! string offsets. And `length` is taken on trust: `DuckDB` allocates the
101//! chunk for that many rows before its error handling starts, so an absurd
102//! length aborts the process with an allocation failure. Both are part of
103//! [`data_chunk_from_arrow`]'s `# Safety` contract.
104//!
105//! # Round trips that lose information
106//!
107//! `DuckDB`'s own converters do not round-trip every type exactly: a `TIMETZ`
108//! comes back as `TIME` with its offset dropped (`01:02:03+05:30` returns as
109//! `01:02:03`), and a `BIT` comes back as `BLOB`.
110//!
111//! # Example: chunk → Arrow → chunk
112//!
113//! ```rust,no_run
114//! use quack_rs::arrow::{
115//! data_chunk_from_arrow, data_chunk_to_arrow, schema_from_arrow, to_arrow_schema,
116//! ArrowOptions,
117//! };
118//! use quack_rs::types::{LogicalType, TypeId};
119//!
120//! # fn demo(
121//! # con: libduckdb_sys::duckdb_connection,
122//! # chunk: &quack_rs::data_chunk::DataChunk,
123//! # ) -> Result<(), Box<dyn std::error::Error>> {
124//! // SAFETY: `con` is a live DuckDB connection that outlives `options`.
125//! let options = unsafe { ArrowOptions::from_raw_connection(con) }?;
126//!
127//! let id = LogicalType::new(TypeId::Integer);
128//! let mut schema = to_arrow_schema(&options, &[("id", &id)])?;
129//! let array = data_chunk_to_arrow(&options, chunk)?;
130//!
131//! // ... hand `schema` / `array` to any Arrow consumer, or convert back:
132//! // SAFETY: `con` is a live DuckDB connection.
133//! let converted = unsafe { schema_from_arrow(con, &mut schema) }?;
134//! // SAFETY: same connection, and `array` was produced against `schema`.
135//! let chunk = unsafe { data_chunk_from_arrow(con, array, &converted) }?;
136//! assert_eq!(chunk.column_count(), 1);
137//! # Ok(())
138//! # }
139//! ```
140//!
141//! # Bridging to arrow-rs
142//!
143//! [`RawArrowSchema`] and [`RawArrowArray`] are the ABI records themselves, so a
144//! `*mut FFI_ArrowSchema` and a `*mut RawArrowSchema` address the same bytes:
145//!
146//! ```rust,ignore
147//! // Export a quack-rs array into arrow-rs.
148//! let ffi: FFI_ArrowArray = unsafe { std::mem::transmute(array.into_raw()) };
149//!
150//! // Import an arrow-rs array into quack-rs, neutralising the source so only
151//! // one side ever calls `release`.
152//! let mut ffi = /* FFI_ArrowArray */;
153//! let array = unsafe { ArrowArray::take_from(std::ptr::from_mut(&mut ffi).cast()) };
154//! ```
155//!
156//! [`take_from`][ArrowArray::take_from] is the safer half of that pair: it moves
157//! the record out and writes a released placeholder back, so the foreign
158//! wrapper's own `Drop` becomes a no-op instead of a double free.
159//!
160//! # Thread safety
161//!
162//! None of these types are `Send` or `Sync`. The Arrow C Data Interface says
163//! nothing about which thread may call `release`, and `duckdb_arrow_options`
164//! wraps a `ClientProperties` that borrows the connection's client context.
165//!
166//! [Arrow C Data Interface]: https://arrow.apache.org/docs/format/CDataInterface.html
167
168mod array;
169mod convert;
170mod converted;
171mod export_check;
172mod import_check;
173mod import_layout;
174mod options;
175mod schema;
176#[cfg(test)]
177mod tests;
178
179pub use convert::{data_chunk_from_arrow, data_chunk_to_arrow, schema_from_arrow, to_arrow_schema};
180
181use std::marker::PhantomData;
182
183use libduckdb_sys::{duckdb_arrow_converted_schema, duckdb_arrow_options};
184
185use crate::query::OwnedConnection;
186
187/// The Arrow C Data Interface `ArrowSchema` ABI record, re-exported from
188/// `libduckdb-sys`.
189///
190/// Field-for-field identical to arrow-rs's `FFI_ArrowSchema`. Prefer the owning
191/// [`ArrowSchema`] wrapper; this is here for interop with code that already
192/// speaks the raw ABI.
193pub use libduckdb_sys::ArrowSchema as RawArrowSchema;
194
195/// The Arrow C Data Interface `ArrowArray` ABI record, re-exported from
196/// `libduckdb-sys`.
197///
198/// Field-for-field identical to arrow-rs's `FFI_ArrowArray`. Prefer the owning
199/// [`ArrowArray`] wrapper; this is here for interop with code that already
200/// speaks the raw ABI.
201pub use libduckdb_sys::ArrowArray as RawArrowArray;
202
203// `libduckdb-sys` below 1.10504.0 declares both records as opaque zero-sized
204// bindgen placeholders (`_unused: [u8; 0]`), which cannot be allocated — so the
205// whole Arrow C Data Interface is unusable there. Fail with a sentence that says
206// so, rather than a pile of "no method named `empty`" errors.
207const _: () = assert!(
208 size_of::<RawArrowSchema>() > 0 && size_of::<RawArrowArray>() > 0,
209 "the `duckdb-1-5-4` feature requires libduckdb-sys >= 1.10504.0: earlier versions declare \
210 ArrowSchema/ArrowArray as opaque zero-sized types, so the caller-allocated structs the \
211 Arrow C Data Interface requires cannot be created"
212);
213
214// ─── Arrow options ───────────────────────────────────────────────────────────
215
216/// The Arrow production settings of a connection or a result.
217///
218/// `DuckDB` needs these to decide how to render its types as Arrow — the
219/// timezone to stamp on `TIMESTAMPTZ`, whether to emit large or regular string
220/// offsets, which extension types are registered. Both
221/// [`to_arrow_schema`] and [`data_chunk_to_arrow`] require one.
222///
223/// Destroyed on drop (`duckdb_destroy_arrow_options`).
224///
225/// # Lifetime
226///
227/// The handle is a copy of the connection's `ClientProperties`, and that
228/// struct keeps a **raw pointer to the connection's `ClientContext`**, which
229/// the conversion functions dereference (`DBConfig::GetConfig(context)`). Once
230/// the connection closes, every use is a heap use-after-free. So the options
231/// borrow the connection for `'conn`:
232///
233/// ```rust,compile_fail,E0505
234/// use quack_rs::arrow::{data_chunk_to_arrow, ArrowOptions};
235/// use quack_rs::query::OwnedConnection;
236///
237/// fn demo(con: OwnedConnection, chunk: &quack_rs::data_chunk::DataChunk) {
238/// let options = ArrowOptions::from_connection(&con).unwrap();
239/// drop(con); // error[E0505]: cannot move out of `con` because it is borrowed
240/// let _ = data_chunk_to_arrow(&options, chunk);
241/// }
242/// ```
243///
244/// Options read from a [`QueryResult`][crate::query::QueryResult] point at the connection that ran the
245/// query, which a `QueryResult` does not borrow, so
246/// [`from_result`][Self::from_result] is `unsafe`.
247pub struct ArrowOptions<'conn> {
248 raw: duckdb_arrow_options,
249 _conn: PhantomData<&'conn OwnedConnection>,
250}
251
252// ─── Arrow schema ────────────────────────────────────────────────────────────
253
254/// An owned Arrow C Data Interface schema.
255///
256/// Released on drop, unless it was already released or moved out with
257/// [`into_raw`][Self::into_raw].
258///
259/// # Reading a released schema
260///
261/// The Arrow specification says that once `release` has run, every other field
262/// of the record is undefined — `DuckDB`'s own release callback frees the block
263/// that `format`, `name` and `children` point into. So [`format`][Self::format],
264/// [`name`][Self::name] and [`child`][Self::child] all return `None` once
265/// [`is_released`][Self::is_released] is true, rather than handing out a
266/// dangling pointer.
267#[repr(transparent)]
268pub struct ArrowSchema(RawArrowSchema);
269
270// ─── Arrow array ─────────────────────────────────────────────────────────────
271
272/// An owned Arrow C Data Interface array.
273///
274/// Released on drop, unless it was already released, moved out with
275/// [`into_raw`][Self::into_raw], or handed to
276/// [`data_chunk_from_arrow`] — which is why that function takes it by value.
277///
278/// As with [`ArrowSchema`], every accessor returns a neutral value once
279/// [`is_released`][Self::is_released] is true: the specification leaves the
280/// other fields undefined after release.
281#[repr(transparent)]
282pub struct ArrowArray(RawArrowArray);
283
284// ─── Converted schema ────────────────────────────────────────────────────────
285
286/// An Arrow schema translated into `DuckDB`'s own type descriptors.
287///
288/// Produced by [`schema_from_arrow`] and consumed by [`data_chunk_from_arrow`].
289/// Destroyed on drop (`duckdb_destroy_arrow_converted_schema`).
290///
291/// # Why it remembers a column count
292///
293/// The C API exposes no accessor for how many columns a converted schema
294/// describes, yet `duckdb_data_chunk_from_arrow` walks
295/// `arrow_array->children[i]` once per column with no bounds check. Recording
296/// the source schema's `n_children` — which is exactly what
297/// `PopulateArrowTableSchema` iterates — lets [`data_chunk_from_arrow`] reject a
298/// mismatched array instead of reading past its children.
299///
300/// # Why it remembers the schema's shape
301///
302/// `duckdb_data_chunk_from_arrow` imports some valid layouts wrongly (see
303/// [`data_chunk_from_arrow`]), and the array alone does not say which type each
304/// node has. The converted schema keeps the formats of the schema it was built
305/// from, so the array can be checked against them before the import.
306pub struct ArrowConvertedSchema {
307 raw: duckdb_arrow_converted_schema,
308 column_count: usize,
309 shapes: Vec<import_layout::Shape>,
310}