Skip to main content

quack_rs/
appender.rs

1// SPDX-License-Identifier: MIT
2// Copyright 2026 Tom F. <https://github.com/tomtom215/>
3// My way of giving something small back to the open source community
4// and encouraging more Rust development!
5
6//! Bulk data appending.
7//!
8//! [`Appender`] is an RAII wrapper around `DuckDB`'s appender — the fastest way
9//! to bulk-insert rows into an existing table, and considerably faster than
10//! issuing `INSERT` statements.
11//!
12//! # Two ways to append
13//!
14//! **Row at a time.** Call one `append_*` per column, then
15//! [`end_row`][Appender::end_row] — or let [`row`][Appender::row] call it for
16//! you, which is the difference between a forgotten `end_row` being a compile
17//! -time non-issue and a silently short table:
18//!
19//! ```rust,no_run
20//! use quack_rs::appender::Appender;
21//! # use libduckdb_sys::duckdb_connection;
22//! # unsafe fn demo(con: duckdb_connection) -> Result<(), quack_rs::appender::AppendError> {
23//! // SAFETY: `con` is a valid, open connection.
24//! let appender = unsafe { Appender::new(con, None, c"measurements") }?;
25//! for (sensor, reading) in [("a", 1.5_f64), ("b", 2.5)] {
26//!     appender.row(|row| {
27//!         row.append_str(sensor)?;
28//!         row.append_f64(reading)
29//!     })?;
30//! }
31//! appender.close()?;
32//! # Ok(())
33//! # }
34//! ```
35//!
36//! **A chunk at a time.** Build a [`DataChunk`][crate::data_chunk::DataChunk] and hand it over with
37//! [`append_chunk`][Appender::append_chunk]. Fewer FFI crossings, and the
38//! natural fit when the data already lives in vectors.
39//!
40//! # Errors and the appender's lifecycle
41//!
42//! Appended rows are buffered. A constraint violation therefore surfaces at
43//! [`flush`][Appender::flush] or [`close`][Appender::close], not at the
44//! `append_*` call that caused it, and it **invalidates every buffered row**.
45//!
46//! Dropping an `Appender` closes it, and a failure there has nowhere to go —
47//! `DuckDB`'s own header is explicit that after destruction "it is no longer
48//! possible to obtain the specific error message". Call
49//! [`close`][Appender::close] explicitly whenever the outcome matters.
50//!
51//! ## A row that fails half-way
52//!
53//! `DuckDB` counts the values of the current row and has no way to take one
54//! back: `end_row` wants every column, `flush` wants none, and its `Close`
55//! flushes only in those two states — in any other it returns *success* and
56//! writes nothing. A [`row`][Appender::row] whose closure fails after its first
57//! value therefore leaves a half-written row that cannot be completed or
58//! dropped, and with it **every row buffered since the last flush is lost**
59//! (`DuckDB` also flushes on its own each time 204,800 rows accumulate; rows
60//! before that are safe).
61//!
62//! quack-rs tracks the row itself so that this is never silent:
63//!
64//! - the appender becomes *poisoned*: every later append, `row`, `end_row`,
65//!   `flush` and `close` returns an error that says how many buffered rows
66//!   were not written;
67//! - `close` with a row started but not ended (by `row` or by hand) is an
68//!   error rather than a silent no-op;
69//! - with `duckdb-1-5`, `clear` discards the buffered rows and the half row
70//!   and makes the appender usable again. Without it, a poisoned appender
71//!   stays poisoned.
72//!
73//! A value that fails as the *first* of its row loses nothing (`DuckDB` has
74//! not counted it), and one that fails later in a row appended by hand can be
75//! retried — only an abandoned row poisons. If losing buffered rows is not
76//! acceptable, [`flush`][Appender::flush] at the points you can afford to
77//! lose work back to.
78//!
79//! After a successful [`close`][Appender::close] the appender refuses further
80//! work; `DuckDB` itself would accept appends and write them at the next
81//! flush.
82//!
83//! ## A streaming result open on the same connection
84//!
85//! A flush runs an `INSERT` on the appender's connection, so it is "another
86//! statement" to a streaming
87//! [`QueryResult`][crate::query::QueryResult] still open there. Up to
88//! `DuckDB` 1.5.x the flush succeeds and the stream ends with an error. From
89//! 2.0.0 the flush — including the automatic one every 204,800 rows and the
90//! one in `close` — is refused with a "Resource In Use" error, and because a
91//! failed flush discards the buffer on 2.0.0, **every buffered row is lost**.
92//! Finish or drop the stream first, or append through a connection of its
93//! own.
94//!
95//! ## Schema changes while rows are buffered
96//!
97//! Buffered rows are written by column *position* when they are flushed. If
98//! another connection drops a column and adds one in between, a buffered
99//! value lands in the new column (cast to its type) with no error. Flush
100//! before a concurrent `ALTER TABLE` if that matters.
101//!
102//! # Feature flags
103//!
104//! The appender is available **without** any feature flag: `DuckDB` has kept
105//! `duckdb_appender_*` in the frozen stable prefix of the extension API
106//! (slots 281–291 and 330–356) since v1.2.0, so using it does not push an
107//! extension onto the version-pinned unstable ABI. Three methods are the
108//! exception and are gated on `duckdb-1-5`: `error_data`, `clear` and
109//! `append_default_to_chunk`. They are not linked here because the links would
110//! not resolve when the feature is off, which is exactly when a reader most
111//! wants to know they exist.
112//!
113//! That gate also picks the error type — see [`AppendError`].
114
115mod chunk;
116mod construct;
117mod lifecycle;
118mod rows;
119mod scalars;
120
121use std::cell::Cell;
122use std::ffi::CStr;
123
124use libduckdb_sys::{
125    duckdb_appender, duckdb_appender_destroy, duckdb_appender_error, duckdb_state, DuckDBSuccess,
126};
127
128#[cfg(feature = "duckdb-1-5")]
129use crate::error_data::ErrorData;
130
131/// The error type every fallible [`Appender`] operation reports.
132///
133/// `DuckDB` exposes the appender's error two ways, and only one of them is in
134/// the stable prefix:
135///
136/// | Feature | Type | C API |
137/// |---------|------|-------|
138/// | `duckdb-1-5` on | [`ErrorData`] — message **and** machine-readable category | `duckdb_appender_error_data` (unstable slot 408) |
139/// | `duckdb-1-5` off | [`ExtensionError`][crate::error::ExtensionError] — message only | `duckdb_appender_error` (stable slot 285) |
140///
141/// Enabling `duckdb-1-5` therefore upgrades the error type in place; it does
142/// not change any method's shape.
143#[cfg(feature = "duckdb-1-5")]
144pub type AppendError = ErrorData;
145
146/// The error type every fallible [`Appender`] operation reports.
147///
148/// See the `duckdb-1-5` variant of this alias for the full explanation: without
149/// that feature the appender reports errors through the stable
150/// `duckdb_appender_error`, which carries a message but no category.
151#[cfg(not(feature = "duckdb-1-5"))]
152pub type AppendError = crate::error::ExtensionError;
153
154/// RAII wrapper for a `duckdb_appender`.
155///
156/// The appender is closed and destroyed automatically on drop. To surface any
157/// error from the final flush, call [`close`][Appender::close] explicitly
158/// beforehand.
159///
160/// See the [module docs][crate::appender] for the two append styles and the
161/// buffering rules that decide when an error appears.
162pub struct Appender {
163    handle: duckdb_appender,
164    /// Values appended to the current row: a mirror of `DuckDB`'s
165    /// `BaseAppender::column`, which advances only on a successful append.
166    column: Cell<u64>,
167    /// Rows ended or appended as chunks since the last flush this wrapper
168    /// performed — what a poisoned appender loses.
169    buffered: Cell<u64>,
170    lifecycle: Cell<Lifecycle>,
171}
172
173/// Where an [`Appender`] is in its life, as far as quack-rs can tell.
174#[derive(Clone, Copy, Debug, PartialEq, Eq)]
175enum Lifecycle {
176    Open,
177    /// A `row` was abandoned after some of its values went in; see the module
178    /// docs.
179    Poisoned,
180    /// `close` succeeded.
181    Closed,
182}
183
184impl Appender {
185    // ── Errors ──────────────────────────────────────────────────────────
186
187    /// Returns the structured error from the most recent failed operation.
188    #[cfg(feature = "duckdb-1-5")]
189    #[must_use]
190    pub fn error_data(&self) -> ErrorData {
191        // SAFETY: self.handle may be null (a failed create); DuckDB handles
192        // that and returns an owned, empty error data handle.
193        let raw = unsafe { libduckdb_sys::duckdb_appender_error_data(self.handle) };
194        // SAFETY: raw is an owned duckdb_error_data (possibly null).
195        unsafe { ErrorData::from_raw(raw) }
196    }
197
198    /// Returns the message from the most recent failed operation, if any.
199    ///
200    /// Always available; with `duckdb-1-5` prefer
201    /// `error_data` (`duckdb-1-5`), which also carries the error category.
202    #[must_use]
203    pub fn error_message(&self) -> Option<String> {
204        if self.handle.is_null() {
205            return None;
206        }
207        // SAFETY: self.handle is non-null; DuckDB returns null when there is
208        // no error, and otherwise a string it owns until the appender is
209        // destroyed — so it is copied out here rather than borrowed.
210        let ptr = unsafe { duckdb_appender_error(self.handle) };
211        if ptr.is_null() {
212            return None;
213        }
214        // SAFETY: ptr is a valid NUL-terminated string owned by the appender.
215        Some(
216            unsafe { CStr::from_ptr(ptr) }
217                .to_string_lossy()
218                .into_owned(),
219        )
220    }
221
222    /// Returns the raw handle.
223    #[inline]
224    #[must_use]
225    pub const fn as_raw(&self) -> duckdb_appender {
226        self.handle
227    }
228
229    /// Reads whichever error channel this build has.
230    fn last_error(&self) -> AppendError {
231        #[cfg(feature = "duckdb-1-5")]
232        {
233            self.error_data()
234        }
235        #[cfg(not(feature = "duckdb-1-5"))]
236        {
237            self.error_message().map_or_else(
238                || append_error("appender operation failed"),
239                crate::error::ExtensionError::new,
240            )
241        }
242    }
243
244    /// Converts a `duckdb_state` into a `Result`, reading the appender's error
245    /// on failure.
246    fn check(&self, state: duckdb_state) -> Result<(), AppendError> {
247        if state == DuckDBSuccess {
248            Ok(())
249        } else {
250            Err(self.last_error())
251        }
252    }
253
254    // ── Row and lifecycle tracking ──────────────────────────────────────
255
256    /// Wraps a handle from `duckdb_appender_create*` in a fresh, open state.
257    const fn wrap(handle: duckdb_appender) -> Self {
258        Self {
259            handle,
260            column: Cell::new(0),
261            buffered: Cell::new(0),
262            lifecycle: Cell::new(Lifecycle::Open),
263        }
264    }
265
266    /// `Ok` while the appender is open; the reason otherwise.
267    fn usable(&self) -> Result<(), AppendError> {
268        match self.lifecycle.get() {
269            Lifecycle::Open => Ok(()),
270            Lifecycle::Closed => Err(append_error(
271                "the appender is closed; create a new one to append more rows",
272            )),
273            Lifecycle::Poisoned => Err(self.lost_rows_error("append")),
274        }
275    }
276
277    /// Runs one value append if the appender is open, and counts it.
278    fn append_one(&self, append: impl FnOnce() -> duckdb_state) -> Result<(), AppendError> {
279        self.usable()?;
280        self.record_append(append())
281    }
282
283    /// Counts a value append that `DuckDB` accepted. A rejected value leaves
284    /// `DuckDB`'s column counter where it was (every append path throws before
285    /// advancing it), so the same column can be tried again.
286    fn record_append(&self, state: duckdb_state) -> Result<(), AppendError> {
287        self.check(state)?;
288        self.column.set(self.column.get().saturating_add(1));
289        Ok(())
290    }
291
292    /// Records the outcome of an operation that flushes on success.
293    fn flushed(&self, state: duckdb_state) -> Result<(), AppendError> {
294        self.check(state)?;
295        self.buffered.set(0);
296        Ok(())
297    }
298
299    /// The error for rows `DuckDB` will not write: a poisoned appender, or
300    /// `close` with an unfinished row.
301    fn lost_rows_error(&self, operation: &str) -> AppendError {
302        let rows = self.buffered.get();
303        let values = self.column.get();
304        if self.lifecycle.get() != Lifecycle::Poisoned {
305            return append_error(&format!(
306                "{operation}: the current row is unfinished ({values} value(s) without \
307                 end_row), and DuckDB writes nothing while it is: up to {rows} row(s) buffered \
308                 since the last flush are lost if the appender is dropped now; end the row and \
309                 close again"
310            ));
311        }
312        let recovery = if cfg!(feature = "duckdb-1-5") {
313            "; `clear` discards them and makes the appender usable again"
314        } else {
315            ""
316        };
317        append_error(&format!(
318            "{operation}: the appender is poisoned: up to {rows} row(s) buffered since the last \
319             flush were not written, because a row failed after {values} of its values were \
320             appended and DuckDB can neither finish nor drop a half-written row{recovery}"
321        ))
322    }
323}
324
325/// Builds an [`AppendError`] for a failure quack-rs detected itself, before
326/// `DuckDB` was ever called.
327fn append_error(message: &str) -> AppendError {
328    #[cfg(feature = "duckdb-1-5")]
329    {
330        ErrorData::new(crate::error_data::DuckDbErrorType::InvalidInput, message)
331    }
332    #[cfg(not(feature = "duckdb-1-5"))]
333    {
334        crate::error::ExtensionError::new(message)
335    }
336}
337
338impl Drop for Appender {
339    fn drop(&mut self) {
340        if !self.handle.is_null() {
341            // SAFETY: self.handle is a valid handle that we own. Destroy
342            // closes (and so flushes) it first; the state is intentionally
343            // ignored here — `close` beforehand is how a final flush error is
344            // observed, because destruction also frees the error message.
345            unsafe { duckdb_appender_destroy(&raw mut self.handle) };
346        }
347    }
348}
349
350crate::debug_repr::impl_handle_debug!(Appender.handle);