Reading & Writing Vectors

DuckDB passes data to and from your extension as vectors: columnar arrays of typed values, each with a separate validity (NULL) bitmap. VectorReader and VectorWriter give typed access to these vectors from scalar, aggregate and table function callbacks; DataChunk, StructReader, StructWriter and ChunkWriter build on them.


VectorReader

Construction

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_data_chunk, duckdb_function_info, duckdb_vector};
use quack_rs::vector::{VectorReader, VectorWriter};
fn demo(input: duckdb_data_chunk, column_index: usize) {
// In a scalar function callback:
let reader = unsafe { VectorReader::new(input, column_index) };

// In an aggregate update callback:
let reader = unsafe { VectorReader::new(input, 0) };   // first column
}
}

VectorReader::new takes the duckdb_data_chunk and a zero-based column index. The reader holds raw pointers into the chunk, so it must not outlive the callback.

Row count

#![allow(unused)]
fn main() {
use quack_rs::vector::{VectorReader, VectorWriter};
fn demo(reader: &VectorReader) {
let n = reader.row_count();   // number of rows in this chunk
}
}

Chunk sizes vary. Always loop over 0..reader.row_count(); never assume a fixed size.

NULL check

#![allow(unused)]
fn main() {
use quack_rs::vector::{VectorReader, VectorWriter};
fn demo(reader: &VectorReader, writer: &mut VectorWriter) {
for row in 0..reader.row_count() {
if unsafe { !reader.is_valid(row) } {
    // row is NULL — skip or propagate NULL to output
    unsafe { writer.set_null(row) };
    continue;
}
}
}
}

Always check is_valid before reading. Reading a fixed-width value from a NULL row returns garbage data; reading a VARCHAR or BLOB from one can follow a stale pointer into freed memory (see NULL Handling & Strings).

Reading values

#![allow(unused)]
fn main() {
use quack_rs::vector::{VectorReader, VectorWriter};
fn demo(reader: &VectorReader, row: usize) {
let i: i8  = unsafe { reader.read_i8(row) };
let i: i16 = unsafe { reader.read_i16(row) };
let i: i32 = unsafe { reader.read_i32(row) };
let i: i64 = unsafe { reader.read_i64(row) };
let u: u8  = unsafe { reader.read_u8(row) };
let u: u16 = unsafe { reader.read_u16(row) };
let u: u32 = unsafe { reader.read_u32(row) };
let u: u64 = unsafe { reader.read_u64(row) };
let f: f32 = unsafe { reader.read_f32(row) };
let f: f64 = unsafe { reader.read_f64(row) };
let b: bool = unsafe { reader.read_bool(row) };   // safe: uses u8 != 0
let s: &str = unsafe { reader.read_str(row) };    // handles inline + pointer format
let iv = unsafe { reader.read_interval(row) };    // returns DuckInterval

// Temporal and binary types (v0.10.0+):
let d: i32 = unsafe { reader.read_date(row) };      // days since epoch
let ts: i64 = unsafe { reader.read_timestamp(row) }; // microseconds since epoch
let t: i64 = unsafe { reader.read_time(row) };       // microseconds since midnight
let blob: &[u8] = unsafe { reader.read_blob(row) };  // binary data
let uuid: u128 = unsafe { reader.read_uuid(row) };   // UUID's textual 128 bits
}
}

VectorWriter

Construction

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_data_chunk, duckdb_function_info, duckdb_vector};
use quack_rs::vector::{VectorReader, VectorWriter};
fn demo(output: duckdb_vector, result: duckdb_vector) {
// In a scalar function callback:
let mut writer = unsafe { VectorWriter::new(output) };

// In an aggregate finalize callback:
let mut writer = unsafe { VectorWriter::new(result) };
}
}

Writing values

#![allow(unused)]
fn main() {
use quack_rs::vector::{VectorReader, VectorWriter};
use quack_rs::interval::DuckInterval;
fn demo(writer: &mut VectorWriter, row: usize, s: &str, interval: DuckInterval,
    days_since_epoch: i32, micros_since_epoch: i64, micros_since_midnight: i64,
    bytes: Vec<u8>, uuid_bits: u128) {
unsafe { writer.write_i8(row, -8) };
unsafe { writer.write_i16(row, -16) };
unsafe { writer.write_i32(row, -32) };
unsafe { writer.write_i64(row, -64) };
unsafe { writer.write_u8(row, 8) };
unsafe { writer.write_u16(row, 16) };
unsafe { writer.write_u32(row, 32) };
unsafe { writer.write_u64(row, 64) };
unsafe { writer.write_f32(row, 3.5) };
unsafe { writer.write_f64(row, 2.5) };
unsafe { writer.write_bool(row, true) };
unsafe { writer.write_varchar(row, s) };   // &str
unsafe { writer.write_str(row, s) };       // alias for write_varchar
unsafe { writer.write_interval(row, interval) };  // DuckInterval

// Temporal and binary types (v0.10.0+):
unsafe { writer.write_date(row, days_since_epoch) };
unsafe { writer.write_timestamp(row, micros_since_epoch) };
unsafe { writer.write_time(row, micros_since_midnight) };
unsafe { writer.write_blob(row, &bytes) };
unsafe { writer.write_uuid(row, uuid_bits) };        // UUID's textual 128 bits
}
}

write_varchar and write_blob panic for a value longer than vector::string::MAX_STRING_LEN (u32::MAX bytes, DuckDB's string length limit) rather than store a truncated one. Inside scalar_callback! and the typed scalar constructors the panic becomes a SQL error. To handle the error yourself, use try_write_varchar / try_write_blob, which return Result<(), ExtensionError> and write nothing on error.

UUID is not stored as you'd expect

A UUID column is physically a HUGEINT, but the 128 bits in the vector are not the bits you see in the text form: DuckDB flips the top bit so that comparing the signed integers orders UUIDs the same way comparing their strings does.

SELECT '11111111-2222-3333-4444-555555555555'::UUID
  read_i128 (raw storage) : 0x91111111222233334444555555555555
  read_uuid (textual bits): 0x11111111222233334444555555555555

read_uuid / write_uuid apply the flip for you and speak in textual bits (u128) — the same convention as Value::uuid / Value::as_uuid and every Rust Uuid type. Reach for read_i128 / write_i128 only when you want the raw storage, and use quack_rs::vector::{uuid_from_storage, uuid_to_storage} to convert.

Writing NULL

#![allow(unused)]
fn main() {
use quack_rs::vector::{VectorReader, VectorWriter};
fn demo(writer: &mut VectorWriter, row: usize) {
unsafe { writer.set_null(row) };
}
}

Pitfall L4: set_null calls duckdb_vector_ensure_validity_writable automatically before duckdb_vector_get_validity. A vector with no NULLs yet usually has no validity mask, so without that call get_validity returns NULL and duckdb_validity_set_row_invalid silently does nothing — the row you meant to be NULL reads back as a valid value. VectorWriter::set_null handles this correctly. See Pitfall L4.

NULL rows of STRUCT and ARRAY outputs

For a STRUCT output, set_null(row) (and set_null_range, and DataChunk::propagate_nulls, which uses it) also nulls that row in every field, recursively; for an ARRAY of size n it nulls child rows row * n .. row * n + n. This mirrors DuckDB's internal FlatVector::SetNull, and it matters: struct_extract / s.a reads the field vector without looking at the parent, so a NULL struct row whose fields were left valid returns the stale field value. StructWriter::set_row_null(row) does the same from a StructWriter. LIST / MAP elements are not touched (as in DuckDB).

To reuse such a row, call set_valid(row) first: on a row that is NULL it also marks valid everything set_null nulled below it. Then write the fields, and any field NULLs after that. On a row that is already valid, set_valid leaves the fields alone, so marking a row valid after writing its fields keeps their NULLs.

Clearing NULL (v0.11.0+)

To undo a previous set_null call and mark a row as valid again:

#![allow(unused)]
fn main() {
use quack_rs::vector::{VectorReader, VectorWriter};
fn demo(writer: &mut VectorWriter, row: usize) {
unsafe { writer.set_valid(row) };
}
}

Like set_null, set_valid calls ensure_validity_writable first.


DataChunk

DataChunk wraps a duckdb_data_chunk handle and gives access to its vectors and row count without raw FFI calls:

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_data_chunk, duckdb_function_info, duckdb_vector};
use quack_rs::data_chunk::DataChunk;

unsafe extern "C" fn my_scan(info: duckdb_function_info, output: duckdb_data_chunk) {
    let chunk = unsafe { DataChunk::from_raw(output) };
    let mut writer = unsafe { chunk.writer(0) };    // VectorWriter for column 0
    unsafe { writer.write_i64(0, 42) };
    unsafe { chunk.set_size(1) };                   // set output row count
}
}

Methods:

  • size() — current row count
  • set_size(n) — set row count (0 = end of stream)
  • column_count() — number of columns
  • vector(col) — raw duckdb_vector handle
  • writer(col) — VectorWriter for a column
  • reader(col) — VectorReader for a column
  • struct_writer(col, field_count) — StructWriter for a STRUCT output column
  • struct_reader(col, field_count) — StructReader for a STRUCT input column
  • struct_field_reader(col, field) — VectorReader for a specific STRUCT field
  • any_null(row) — whether any column is NULL at row
  • propagate_nulls(&mut writer) — mark each output row NULL where any input column is NULL
  • into_chunk_writer() — convert to ChunkWriter, which calls set_size on drop

StructWriter / StructReader

For STRUCT columns, creating a VectorWriter or VectorReader for each field by hand is verbose. StructWriter and StructReader create one per field at construction:

#![allow(unused)]
fn main() {
use quack_rs::data_chunk::DataChunk;
struct Output { success: bool, data: String, count: i64, day: i32, payload: Vec<u8> }
fn demo(chunk: &DataChunk, row: usize, result: &Output) {
// Writing a 5-field STRUCT output:
let mut sw = unsafe { chunk.struct_writer(0, 5) };
unsafe {
    sw.write_bool(row, 0, result.success);
    sw.write_varchar(row, 1, &result.data);
    sw.write_i64(row, 2, result.count);
    sw.write_date(row, 3, result.day);
    sw.write_blob(row, 4, &result.payload);
}

// Reading a 3-field STRUCT input:
let sr = unsafe { chunk.struct_reader(0, 3) };
for row in 0..chunk.size() {
    let name = unsafe { sr.read_str(row, 0) };
    let age = unsafe { sr.read_i32(row, 1) };
    let active = unsafe { sr.read_bool(row, 2) };
}
}
}

ChunkWriter

ChunkWriter wraps an output duckdb_data_chunk and counts the rows handed out by next_row. It calls set_size with that count on drop, so the row count cannot be forgotten. next_row returns None once the chunk holds duckdb_vector_size() rows. The count is of rows you claimed: write every row you claim. A claimed row that is never written is published with whatever the reused output buffer held last (Pitfall L21), so check that the source has another value before you claim the row, or give the last claimed row back with release_row(row) before the writer is dropped (it panics for any other row, so the mistake is loud):

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_data_chunk, duckdb_function_info, duckdb_vector};
use quack_rs::data_chunk::DataChunk;
struct Item { name: String, value: i64 }
fn demo(output: duckdb_data_chunk, data: &[Item]) {
let mut cw = unsafe { DataChunk::from_raw(output).into_chunk_writer() };
// One writer per column, fetched once: a writer borrows nothing from `cw`, and
// building one costs several C API calls, so doing it per row is several times
// slower.
let mut names = unsafe { cw.writer(0) };
let mut values = unsafe { cw.writer(1) };
for item in data {
    let Some(row) = cw.next_row() else { break };   // chunk is full
    unsafe { names.write_varchar(row, &item.name) };
    unsafe { values.write_i64(row, item.value) };
}
// set_size called automatically when `cw` is dropped
}
}

ValidityBitmap

For advanced NULL handling beyond VectorWriter::set_null, use ValidityBitmap directly:

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_data_chunk, duckdb_function_info, duckdb_vector};
fn demo(some_vector: duckdb_vector, row: usize) {
use quack_rs::vector::ValidityBitmap;

// Writing NULLs:
let mut bitmap = unsafe { ValidityBitmap::ensure_writable(some_vector) };
unsafe { bitmap.set_row_invalid(row as u64) };   // mark as NULL
unsafe { bitmap.set_row_valid(row as u64) };     // mark as non-NULL

// Reading NULLs:
let bitmap = unsafe { ValidityBitmap::get_read_only(some_vector) };
let is_valid = unsafe { bitmap.row_is_valid(row as u64) };
}
}

ValidityBitmap is available in the prelude: use quack_rs::prelude::*.


Utility functions

The quack_rs::vector module provides two utility functions:

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_data_chunk, duckdb_function_info, duckdb_vector};
fn demo(some_vector: duckdb_vector) {
use quack_rs::vector::{vector_size, vector_get_column_type};

// Rows per data chunk: 2048 unless DuckDB was built with another STANDARD_VECTOR_SIZE.
let size: u64 = vector_size();

// Returns the LogicalType of a vector (unsafe — requires a valid duckdb_vector).
let lt = unsafe { vector_get_column_type(some_vector) };
}
}

Memory layout details

DuckDB stores vector data as flat arrays. VectorReader and VectorWriter compute element addresses as base_ptr + row * stride:

[value0][value1][value2]...[valueN]   ← typed array
[validity bitmap]                      ← separate bit array, 1 bit per row

The validity bitmap is lazily allocated — it may be null if no NULLs have been written. This is why duckdb_vector_ensure_validity_writable must be called before duckdb_vector_get_validity when writing NULLs; VectorWriter and ValidityBitmap::ensure_writable do so.


Complete scalar function pattern

#![allow(unused)]
fn main() {
use libduckdb_sys::{duckdb_data_chunk, duckdb_function_info, duckdb_vector};
use quack_rs::vector::{VectorReader, VectorWriter};
fn transform(v: i64) -> i64 { v }
unsafe extern "C" fn my_scalar(
    _info: duckdb_function_info,
    input: duckdb_data_chunk,
    output: duckdb_vector,
) {
    let reader = unsafe { VectorReader::new(input, 0) };
    let mut writer = unsafe { VectorWriter::new(output) };

    for row in 0..reader.row_count() {
        if unsafe { !reader.is_valid(row) } {
            unsafe { writer.set_null(row) };
            continue;
        }
        let value = unsafe { reader.read_i64(row) };
        unsafe { writer.write_i64(row, transform(value)) };
    }
}
}