pub unsafe fn data_chunk_from_arrow(
connection: duckdb_connection,
array: ArrowArray,
converted: &ArrowConvertedSchema,
) -> Result<OwnedDataChunk, ErrorData>Expand description
Imports an Arrow struct array as a DuckDB data chunk
(duckdb_data_chunk_from_arrow).
§Why array is taken by value
DuckDB claims the array: it copies the record into the chunk’s owned data
and then sets arrow_array->release = nullptr — before the conversion
loop body, so the transfer happens on the error path too. Taking it by value
makes that visible in the signature, and the by-value binding is dropped on
the way out, which releases the array in the one case where DuckDB before
2.0.0 does not claim it (a zero-column converted schema, where the loop
never runs). DuckDB 2.0.0 claims it before the loop, so it releases that
array itself; the binding’s Drop then finds release null and does
nothing.
The resulting chunk keeps the Arrow buffers alive, so the data is shared, not
copied — with two exceptions, which are copied so that the chunk is flat
and every VectorReader reads it
correctly. DuckDB imports a column that is dictionary-encoded at any
depth as a dictionary vector, and an Arrow null-type column as a
constant vector; a flat read of either returns wrong values (and, for a
dictionary, reads past the end of its buffer). Those columns are copied
into fresh flat vectors before this function returns. Run-end-encoded
children are expanded by DuckDB itself.
Before DuckDB 2.0.0 the claim is held by the chunk’s first column:
DuckDB copies the record into every column’s state but nulls release
after the first (src/main/capi/arrow-c.cpp, lines 110–120 at v1.5.6), so
the producer’s buffers are released when column 0’s vector is. A vector
made to reference another column
(reference_vector) does not keep
them alive there. From 2.0.0 every column shares one owner
(src/main/capi/v1/arrow-c.cpp, lines 141–152), so a vector referencing
any column keeps them. Either way reference_vector’s contract already
requires the chunk to outlive the referencing vector.
§Errors
DuckDbErrorType::InvalidInput, checked here because DuckDB does not
check them and would read out of bounds or through a null pointer:
arrayhas already been released, has a negativelength, has a nonzerooffset(DuckDBignores the top-level offset and imports the wrong rows, seedocs/upstream-duckdb-reports.md), or has no rows (see below);- its child count does not match
converted’s column count; - its
childrenpointer, or one of the child pointers, is null; - a child is shorter than
length(the Arrow specification requires every child of a struct array to holdoffset + lengthrows); - a node anywhere in
arrayhas a valid Arrow layout thatDuckDBimports from the wrong rows or out of bounds: an offset below the top level thatDuckDBmisapplies, a dictionary under a list or with more thanduckdb_vector_size(2048) rows that can hold NULLs, a nested dictionary, overlapping or gapped list views, a sparse union with recoded type ids, or a run-end-encoded array whereDuckDBreads a plain one. The message names the node; the module docs ofsrc/arrow/import_layout.rsanddocs/upstream-duckdb-reports.md(items 9 and 24 to 28) give the details; - a node anywhere in
arraybreaks a bound the Arrow specification sets andDuckDBtakes on trust, reading past it once the chunk is read (a SIGSEGV on 1.5.6 and 2.0.0 for a list offset past its child): a list, large list or map offset that is negative, decreases, or lies past the child; a negative list-view offset or size; a struct field, list or fixed-size-list child, or union member shorter than the rows its parent reads from it; a dictionary index outside the dictionary in a row its own validity does not mark NULL (a row NULL only in an enclosing struct must still hold a valid index, as the specification requires), or indices of a non-integer type; - the array and its schema disagree on a node’s child count or dictionary encoding.
DuckDbErrorType::Internal if copying a non-flat column into a flat
vector fails (see above), which needs an allocation failure.
Every error DuckDB itself reports from the conversion also arrives as
DuckDbErrorType::InvalidInput: duckdb_data_chunk_from_arrow maps all
of them to DUCKDB_ERROR_INVALID_INPUT.
§Safety
connectionmust be a liveduckdb_connection, andconvertedmust have been produced from it (or from another connection of the same database).arraymust be a valid Arrow struct array (seeArrowArray::from_raw) that conforms toconverted’s schema: each child’s physical layout must be the one its column’s Arrow format describes.DuckDBreads a child’s buffers as that format with no check — anint32child imported under autf8schema has its values read as string offsets into a buffer that does not exist (a crash, or worse). An array exported withdata_chunk_to_arrowunder the schema thatconvertedwas made from conforms.lengthmust be the array’s true row count.DuckDBallocates a chunk of that capacity before its error handling starts (dchunk->Initialize(…, length)inarrow-c.cpp), so a length too large to allocate throws through the C API and aborts the process. A length, or a nested row count, aboveMAX_CAPACITYis refused, which on a 32-bit target also keeps every buffer’s byte size within whatmallocreceives unnarrowed; below it an allocation the system cannot satisfy still aborts. The length is not otherwise checkable: a run-end-encoded column declares any length with a few bytes of buffers.- Every validity bitmap that
DuckDBreads (one with a nonzeronull_count) must be readable for one byte past the last byte that holds its rows’ bits. When a node’s effective bit offset is not a multiple of 8,GetValidityMask(arrow_conversion.cpp) copiesceil(rows / 8) + 1bytes from the first byte it needs, which can be one more than the rows occupy: anint32column at offset 1 of length 7 with a 1-byte bitmap is read as 2 bytes (an invalid read under valgrind on 1.5.5). That byte only supplies bits for rows past the end, so the values imported are right; the read itself is the hazard. Buffers padded to a multiple of 8 or 64 bytes, as the Arrow columnar format recommends, satisfy this; the allocation size is invisible through the C Data Interface, so it cannot be checked here. - A dictionary whose values are a fixed-width type (integers, floats,
DATEin days,TIMESTAMP,DECIMAL, …) and whose indices can be NULL (a nonzeronull_count, or NULL rows in an enclosing struct) must have its values buffer readable for one element pastoffset + length.DuckDBpoints every NULL index at a sentinel entry one past the dictionary, but imports those types without copying, so the sentinel lies in the producer’s buffer; copying the column, as this function does for every dictionary-encoded column, reads it (an invalid read under valgrind on 1.4.4 to 1.5.5,docs/upstream-duckdb-reports.md, item 29). The value read is discarded: those rows are NULL. A buffer padded to a multiple of 64 bytes satisfies this unless the values fill it exactly.