Skip to main content

data_chunk_from_arrow

Function data_chunk_from_arrow 

Source
pub unsafe fn data_chunk_from_arrow(
    connection: duckdb_connection,
    array: ArrowArray,
    converted: &ArrowConvertedSchema,
) -> Result<OwnedDataChunk, ErrorData>
Expand description

Imports an Arrow struct array as a DuckDB data chunk (duckdb_data_chunk_from_arrow).

§Why array is taken by value

DuckDB claims the array: it copies the record into the chunk’s owned data and then sets arrow_array->release = nullptr — before the conversion loop body, so the transfer happens on the error path too. Taking it by value makes that visible in the signature, and the by-value binding is dropped on the way out, which releases the array in the one case where DuckDB before 2.0.0 does not claim it (a zero-column converted schema, where the loop never runs). DuckDB 2.0.0 claims it before the loop, so it releases that array itself; the binding’s Drop then finds release null and does nothing.

The resulting chunk keeps the Arrow buffers alive, so the data is shared, not copied — with two exceptions, which are copied so that the chunk is flat and every VectorReader reads it correctly. DuckDB imports a column that is dictionary-encoded at any depth as a dictionary vector, and an Arrow null-type column as a constant vector; a flat read of either returns wrong values (and, for a dictionary, reads past the end of its buffer). Those columns are copied into fresh flat vectors before this function returns. Run-end-encoded children are expanded by DuckDB itself.

Before DuckDB 2.0.0 the claim is held by the chunk’s first column: DuckDB copies the record into every column’s state but nulls release after the first (src/main/capi/arrow-c.cpp, lines 110–120 at v1.5.6), so the producer’s buffers are released when column 0’s vector is. A vector made to reference another column (reference_vector) does not keep them alive there. From 2.0.0 every column shares one owner (src/main/capi/v1/arrow-c.cpp, lines 141–152), so a vector referencing any column keeps them. Either way reference_vector’s contract already requires the chunk to outlive the referencing vector.

§Errors

DuckDbErrorType::InvalidInput, checked here because DuckDB does not check them and would read out of bounds or through a null pointer:

  • array has already been released, has a negative length, has a nonzero offset (DuckDB ignores the top-level offset and imports the wrong rows, see docs/upstream-duckdb-reports.md), or has no rows (see below);
  • its child count does not match converted’s column count;
  • its children pointer, or one of the child pointers, is null;
  • a child is shorter than length (the Arrow specification requires every child of a struct array to hold offset + length rows);
  • a node anywhere in array has a valid Arrow layout that DuckDB imports from the wrong rows or out of bounds: an offset below the top level that DuckDB misapplies, a dictionary under a list or with more than duckdb_vector_size (2048) rows that can hold NULLs, a nested dictionary, overlapping or gapped list views, a sparse union with recoded type ids, or a run-end-encoded array where DuckDB reads a plain one. The message names the node; the module docs of src/arrow/import_layout.rs and docs/upstream-duckdb-reports.md (items 9 and 24 to 28) give the details;
  • a node anywhere in array breaks a bound the Arrow specification sets and DuckDB takes on trust, reading past it once the chunk is read (a SIGSEGV on 1.5.6 and 2.0.0 for a list offset past its child): a list, large list or map offset that is negative, decreases, or lies past the child; a negative list-view offset or size; a struct field, list or fixed-size-list child, or union member shorter than the rows its parent reads from it; a dictionary index outside the dictionary in a row its own validity does not mark NULL (a row NULL only in an enclosing struct must still hold a valid index, as the specification requires), or indices of a non-integer type;
  • the array and its schema disagree on a node’s child count or dictionary encoding.

DuckDbErrorType::Internal if copying a non-flat column into a flat vector fails (see above), which needs an allocation failure.

Every error DuckDB itself reports from the conversion also arrives as DuckDbErrorType::InvalidInput: duckdb_data_chunk_from_arrow maps all of them to DUCKDB_ERROR_INVALID_INPUT.

§Safety

  • connection must be a live duckdb_connection, and converted must have been produced from it (or from another connection of the same database).
  • array must be a valid Arrow struct array (see ArrowArray::from_raw) that conforms to converted’s schema: each child’s physical layout must be the one its column’s Arrow format describes. DuckDB reads a child’s buffers as that format with no check — an int32 child imported under a utf8 schema has its values read as string offsets into a buffer that does not exist (a crash, or worse). An array exported with data_chunk_to_arrow under the schema that converted was made from conforms.
  • length must be the array’s true row count. DuckDB allocates a chunk of that capacity before its error handling starts (dchunk->Initialize(…, length) in arrow-c.cpp), so a length too large to allocate throws through the C API and aborts the process. A length, or a nested row count, above MAX_CAPACITY is refused, which on a 32-bit target also keeps every buffer’s byte size within what malloc receives unnarrowed; below it an allocation the system cannot satisfy still aborts. The length is not otherwise checkable: a run-end-encoded column declares any length with a few bytes of buffers.
  • Every validity bitmap that DuckDB reads (one with a nonzero null_count) must be readable for one byte past the last byte that holds its rows’ bits. When a node’s effective bit offset is not a multiple of 8, GetValidityMask (arrow_conversion.cpp) copies ceil(rows / 8) + 1 bytes from the first byte it needs, which can be one more than the rows occupy: an int32 column at offset 1 of length 7 with a 1-byte bitmap is read as 2 bytes (an invalid read under valgrind on 1.5.5). That byte only supplies bits for rows past the end, so the values imported are right; the read itself is the hazard. Buffers padded to a multiple of 8 or 64 bytes, as the Arrow columnar format recommends, satisfy this; the allocation size is invisible through the C Data Interface, so it cannot be checked here.
  • A dictionary whose values are a fixed-width type (integers, floats, DATE in days, TIMESTAMP, DECIMAL, …) and whose indices can be NULL (a nonzero null_count, or NULL rows in an enclosing struct) must have its values buffer readable for one element past offset + length. DuckDB points every NULL index at a sentinel entry one past the dictionary, but imports those types without copying, so the sentinel lies in the producer’s buffer; copying the column, as this function does for every dictionary-encoded column, reads it (an invalid read under valgrind on 1.4.4 to 1.5.5, docs/upstream-duckdb-reports.md, item 29). The value read is discarded: those rows are NULL. A buffer padded to a multiple of 64 bytes satisfies this unless the values fill it exactly.