Expand description
Auto-sizing chunk writer for table function scan callbacks.
ChunkWriter wraps a duckdb_data_chunk and counts the rows its scan
claims. On drop it calls duckdb_data_chunk_set_size with that count, which
removes the error-prone manual set_size after writing rows — a common
source of bugs in table functions.
§Claim a row, then write it or give it back
next_row claims the next row: it returns the
row’s index and counts it, and the drop publishes every row counted. The
writer cannot see whether a claimed row was then written, because the
writes go through VectorWriters that do not report back. A claimed row
that is never written is published anyway, holding whatever the output
vector last held: DuckDB reuses one output chunk for every scan call, so
that is a value from an earlier call (a wrong answer), and in a LIST or
MAP column written by hand it is an earlier call’s {offset, length}
entry, pointing into child data this call did not write.
So write every row you claim, or give it back with
release_row before the writer drops. The
natural loop over a source of unknown length claims one row more than the
source fills:
use quack_rs::chunk_writer::ChunkWriter;
use libduckdb_sys::{duckdb_function_info, duckdb_data_chunk};
unsafe extern "C" fn my_scan(info: duckdb_function_info, output: duckdb_data_chunk) {
let mut source = [("a", 1_i64), ("b", 2)].into_iter();
let mut cw = unsafe { ChunkWriter::new(output) };
// One writer per column, fetched once: building a writer costs several
// C API calls, so doing it for every row is several times slower.
let mut names = unsafe { cw.writer(0) };
let mut values = unsafe { cw.writer(1) };
while let Some(row) = cw.next_row() {
let Some((name, value)) = source.next() else {
// The source ran out after `row` was claimed: give it back, or
// it is published with stale contents.
cw.release_row(row);
break;
};
unsafe { names.write_varchar(row, name) };
unsafe { values.write_i64(row, value) };
}
// set_size is called automatically when `cw` is dropped
}Asking the source first (for item in source { let Some(row) = cw.next_row() else { break }; ... }) never claims a row it cannot fill,
but takes the item that found the chunk full out of the source: keep it for
the next call if the source cannot be rewound.
§Estimated impact
Eliminates ~15 manual set_size calls and prevents off-by-one errors.
Structs§
- Chunk
Writer - A row-counting writer for a
DuckDBoutput data chunk.
Constants§
- DEFAULT_
STANDARD_ VECTOR_ SIZE DuckDB’s compile-time default vector size.