Skip to main content

Module appender

Module appender 

Source
Expand description

Bulk data appending.

Appender is an RAII wrapper around DuckDB’s appender — the fastest way to bulk-insert rows into an existing table, and considerably faster than issuing INSERT statements.

§Two ways to append

Row at a time. Call one append_* per column, then end_row — or let row call it for you, which is the difference between a forgotten end_row being a compile -time non-issue and a silently short table:

use quack_rs::appender::Appender;
// SAFETY: `con` is a valid, open connection.
let appender = unsafe { Appender::new(con, None, c"measurements") }?;
for (sensor, reading) in [("a", 1.5_f64), ("b", 2.5)] {
    appender.row(|row| {
        row.append_str(sensor)?;
        row.append_f64(reading)
    })?;
}
appender.close()?;

A chunk at a time. Build a DataChunk and hand it over with append_chunk. Fewer FFI crossings, and the natural fit when the data already lives in vectors.

§Errors and the appender’s lifecycle

Appended rows are buffered. A constraint violation therefore surfaces at flush or close, not at the append_* call that caused it, and it invalidates every buffered row.

Dropping an Appender closes it, and a failure there has nowhere to go — DuckDB’s own header is explicit that after destruction “it is no longer possible to obtain the specific error message”. Call close explicitly whenever the outcome matters.

§A row that fails half-way

DuckDB counts the values of the current row and has no way to take one back: end_row wants every column, flush wants none, and its Close flushes only in those two states — in any other it returns success and writes nothing. A row whose closure fails after its first value therefore leaves a half-written row that cannot be completed or dropped, and with it every row buffered since the last flush is lost (DuckDB also flushes on its own each time 204,800 rows accumulate; rows before that are safe).

quack-rs tracks the row itself so that this is never silent:

  • the appender becomes poisoned: every later append, row, end_row, flush and close returns an error that says how many buffered rows were not written;
  • close with a row started but not ended (by row or by hand) is an error rather than a silent no-op;
  • with duckdb-1-5, clear discards the buffered rows and the half row and makes the appender usable again. Without it, a poisoned appender stays poisoned.

A value that fails as the first of its row loses nothing (DuckDB has not counted it), and one that fails later in a row appended by hand can be retried — only an abandoned row poisons. If losing buffered rows is not acceptable, flush at the points you can afford to lose work back to.

After a successful close the appender refuses further work; DuckDB itself would accept appends and write them at the next flush.

§A streaming result open on the same connection

A flush runs an INSERT on the appender’s connection, so it is “another statement” to a streaming QueryResult still open there. Up to DuckDB 1.5.x the flush succeeds and the stream ends with an error. From 2.0.0 the flush — including the automatic one every 204,800 rows and the one in close — is refused with a “Resource In Use” error, and because a failed flush discards the buffer on 2.0.0, every buffered row is lost. Finish or drop the stream first, or append through a connection of its own.

§Schema changes while rows are buffered

Buffered rows are written by column position when they are flushed. If another connection drops a column and adds one in between, a buffered value lands in the new column (cast to its type) with no error. Flush before a concurrent ALTER TABLE if that matters.

§Feature flags

The appender is available without any feature flag: DuckDB has kept duckdb_appender_* in the frozen stable prefix of the extension API (slots 281–291 and 330–356) since v1.2.0, so using it does not push an extension onto the version-pinned unstable ABI. Three methods are the exception and are gated on duckdb-1-5: error_data, clear and append_default_to_chunk. They are not linked here because the links would not resolve when the feature is off, which is exactly when a reader most wants to know they exist.

That gate also picks the error type — see AppendError.

Structs§

Appender
RAII wrapper for a duckdb_appender.

Type Aliases§

AppendError
The error type every fallible Appender operation reports.