Skip to main content

data_chunk_to_arrow

Function data_chunk_to_arrow 

Source
pub fn data_chunk_to_arrow(
    options: &ArrowOptions<'_>,
    chunk: &DataChunk,
) -> Result<ArrowArray, ErrorData>
Expand description

Exports a DuckDB data chunk as an Arrow struct array (duckdb_data_chunk_to_arrow).

The chunk is only read; it stays valid and owned by the caller. The returned array owns copies of the data, so it outlives the chunk.

Pair it with the schema to_arrow_schema produces from the same column types and the same ArrowOptions — an Arrow consumer needs both.

§Values DuckDB would export wrongly are refused

DuckDB exports these as different values, with no error (checked on 1.4.4, 1.5.0 and 1.5.5; docs/upstream-duckdb-reports.md, item 16), so the chunk is checked first and refused if it holds one, at any depth:

  • INTERVAL: Arrow’s month_day_nano counts nanoseconds in an i64, and DuckDB multiplies the microseconds by 1000 with no overflow check. An interval whose microsecond field is beyond ±i64::MAX / 1000 (about ±106,751 days, or 2,562,047 hours) would wrap: INTERVAL 2562048 HOUR would export as a negative interval.
  • UHUGEINT, and HUGEINT unless arrow_lossless_conversion is set: exported as decimal128(38, 0), which holds 38 digits. A UHUGEINT of 2^127 or more would come back negative (2^128 - 1 as -1), and a 39-digit HUGEINT would not survive either.

The check copies each column whose type contains one of these types (a chunk’s vectors may be dictionary or constant vectors), so it costs one copy of those columns.

§Errors

  • DuckDbErrorType::InvalidInput naming the column and value, for a value DuckDB would export wrongly (see above).
  • Whatever DuckDB reports for a type it cannot render as Arrow.