Skip to main content

Module chunk_writer

Module chunk_writer 

Source
Expand description

Auto-sizing chunk writer for table function scan callbacks.

ChunkWriter wraps a duckdb_data_chunk and counts the rows its scan claims. On drop it calls duckdb_data_chunk_set_size with that count, which removes the error-prone manual set_size after writing rows — a common source of bugs in table functions.

§Claim a row, then write it or give it back

next_row claims the next row: it returns the row’s index and counts it, and the drop publishes every row counted. The writer cannot see whether a claimed row was then written, because the writes go through VectorWriters that do not report back. A claimed row that is never written is published anyway, holding whatever the output vector last held: DuckDB reuses one output chunk for every scan call, so that is a value from an earlier call (a wrong answer), and in a LIST or MAP column written by hand it is an earlier call’s {offset, length} entry, pointing into child data this call did not write.

So write every row you claim, or give it back with release_row before the writer drops. The natural loop over a source of unknown length claims one row more than the source fills:

use quack_rs::chunk_writer::ChunkWriter;
use libduckdb_sys::{duckdb_function_info, duckdb_data_chunk};

unsafe extern "C" fn my_scan(info: duckdb_function_info, output: duckdb_data_chunk) {
    let mut source = [("a", 1_i64), ("b", 2)].into_iter();
    let mut cw = unsafe { ChunkWriter::new(output) };
    // One writer per column, fetched once: building a writer costs several
    // C API calls, so doing it for every row is several times slower.
    let mut names = unsafe { cw.writer(0) };
    let mut values = unsafe { cw.writer(1) };
    while let Some(row) = cw.next_row() {
        let Some((name, value)) = source.next() else {
            // The source ran out after `row` was claimed: give it back, or
            // it is published with stale contents.
            cw.release_row(row);
            break;
        };
        unsafe { names.write_varchar(row, name) };
        unsafe { values.write_i64(row, value) };
    }
    // set_size is called automatically when `cw` is dropped
}

Asking the source first (for item in source { let Some(row) = cw.next_row() else { break }; ... }) never claims a row it cannot fill, but takes the item that found the chunk full out of the source: keep it for the next call if the source cannot be rewound.

§Estimated impact

Eliminates ~15 manual set_size calls and prevents off-by-one errors.

Structs§

ChunkWriter
A row-counting writer for a DuckDB output data chunk.

Constants§

DEFAULT_STANDARD_VECTOR_SIZE
DuckDB’s compile-time default vector size.