quack_rs/appender.rs
1// SPDX-License-Identifier: MIT
2// Copyright 2026 Tom F. <https://github.com/tomtom215/>
3// My way of giving something small back to the open source community
4// and encouraging more Rust development!
5
6//! Bulk data appending.
7//!
8//! [`Appender`] is an RAII wrapper around `DuckDB`'s appender — the fastest way
9//! to bulk-insert rows into an existing table, and considerably faster than
10//! issuing `INSERT` statements.
11//!
12//! # Two ways to append
13//!
14//! **Row at a time.** Call one `append_*` per column, then
15//! [`end_row`][Appender::end_row] — or let [`row`][Appender::row] call it for
16//! you, which is the difference between a forgotten `end_row` being a compile
17//! -time non-issue and a silently short table:
18//!
19//! ```rust,no_run
20//! use quack_rs::appender::Appender;
21//! # use libduckdb_sys::duckdb_connection;
22//! # unsafe fn demo(con: duckdb_connection) -> Result<(), quack_rs::appender::AppendError> {
23//! // SAFETY: `con` is a valid, open connection.
24//! let appender = unsafe { Appender::new(con, None, c"measurements") }?;
25//! for (sensor, reading) in [("a", 1.5_f64), ("b", 2.5)] {
26//! appender.row(|row| {
27//! row.append_str(sensor)?;
28//! row.append_f64(reading)
29//! })?;
30//! }
31//! appender.close()?;
32//! # Ok(())
33//! # }
34//! ```
35//!
36//! **A chunk at a time.** Build a [`DataChunk`][crate::data_chunk::DataChunk] and hand it over with
37//! [`append_chunk`][Appender::append_chunk]. Fewer FFI crossings, and the
38//! natural fit when the data already lives in vectors.
39//!
40//! # Errors and the appender's lifecycle
41//!
42//! Appended rows are buffered. A constraint violation therefore surfaces at
43//! [`flush`][Appender::flush] or [`close`][Appender::close], not at the
44//! `append_*` call that caused it, and it **invalidates every buffered row**.
45//!
46//! Dropping an `Appender` closes it, and a failure there has nowhere to go —
47//! `DuckDB`'s own header is explicit that after destruction "it is no longer
48//! possible to obtain the specific error message". Call
49//! [`close`][Appender::close] explicitly whenever the outcome matters.
50//!
51//! ## A row that fails half-way
52//!
53//! `DuckDB` counts the values of the current row and has no way to take one
54//! back: `end_row` wants every column, `flush` wants none, and its `Close`
55//! flushes only in those two states — in any other it returns *success* and
56//! writes nothing. A [`row`][Appender::row] whose closure fails after its first
57//! value therefore leaves a half-written row that cannot be completed or
58//! dropped, and with it **every row buffered since the last flush is lost**
59//! (`DuckDB` also flushes on its own each time 204,800 rows accumulate; rows
60//! before that are safe).
61//!
62//! quack-rs tracks the row itself so that this is never silent:
63//!
64//! - the appender becomes *poisoned*: every later append, `row`, `end_row`,
65//! `flush` and `close` returns an error that says how many buffered rows
66//! were not written;
67//! - `close` with a row started but not ended (by `row` or by hand) is an
68//! error rather than a silent no-op;
69//! - with `duckdb-1-5`, `clear` discards the buffered rows and the half row
70//! and makes the appender usable again. Without it, a poisoned appender
71//! stays poisoned.
72//!
73//! A value that fails as the *first* of its row loses nothing (`DuckDB` has
74//! not counted it), and one that fails later in a row appended by hand can be
75//! retried — only an abandoned row poisons. If losing buffered rows is not
76//! acceptable, [`flush`][Appender::flush] at the points you can afford to
77//! lose work back to.
78//!
79//! After a successful [`close`][Appender::close] the appender refuses further
80//! work; `DuckDB` itself would accept appends and write them at the next
81//! flush.
82//!
83//! ## A streaming result open on the same connection
84//!
85//! A flush runs an `INSERT` on the appender's connection, so it is "another
86//! statement" to a streaming
87//! [`QueryResult`][crate::query::QueryResult] still open there. Up to
88//! `DuckDB` 1.5.x the flush succeeds and the stream ends with an error. From
89//! 2.0.0 the flush — including the automatic one every 204,800 rows and the
90//! one in `close` — is refused with a "Resource In Use" error, and because a
91//! failed flush discards the buffer on 2.0.0, **every buffered row is lost**.
92//! Finish or drop the stream first, or append through a connection of its
93//! own.
94//!
95//! ## Schema changes while rows are buffered
96//!
97//! Buffered rows are written by column *position* when they are flushed. If
98//! another connection drops a column and adds one in between, a buffered
99//! value lands in the new column (cast to its type) with no error. Flush
100//! before a concurrent `ALTER TABLE` if that matters.
101//!
102//! # Feature flags
103//!
104//! The appender is available **without** any feature flag: `DuckDB` has kept
105//! `duckdb_appender_*` in the frozen stable prefix of the extension API
106//! (slots 281–291 and 330–356) since v1.2.0, so using it does not push an
107//! extension onto the version-pinned unstable ABI. Three methods are the
108//! exception and are gated on `duckdb-1-5`: `error_data`, `clear` and
109//! `append_default_to_chunk`. They are not linked here because the links would
110//! not resolve when the feature is off, which is exactly when a reader most
111//! wants to know they exist.
112//!
113//! That gate also picks the error type — see [`AppendError`].
114
115mod chunk;
116mod construct;
117mod lifecycle;
118mod rows;
119mod scalars;
120
121use std::cell::Cell;
122use std::ffi::CStr;
123
124use libduckdb_sys::{
125 duckdb_appender, duckdb_appender_destroy, duckdb_appender_error, duckdb_state, DuckDBSuccess,
126};
127
128#[cfg(feature = "duckdb-1-5")]
129use crate::error_data::ErrorData;
130
131/// The error type every fallible [`Appender`] operation reports.
132///
133/// `DuckDB` exposes the appender's error two ways, and only one of them is in
134/// the stable prefix:
135///
136/// | Feature | Type | C API |
137/// |---------|------|-------|
138/// | `duckdb-1-5` on | [`ErrorData`] — message **and** machine-readable category | `duckdb_appender_error_data` (unstable slot 408) |
139/// | `duckdb-1-5` off | [`ExtensionError`][crate::error::ExtensionError] — message only | `duckdb_appender_error` (stable slot 285) |
140///
141/// Enabling `duckdb-1-5` therefore upgrades the error type in place; it does
142/// not change any method's shape.
143#[cfg(feature = "duckdb-1-5")]
144pub type AppendError = ErrorData;
145
146/// The error type every fallible [`Appender`] operation reports.
147///
148/// See the `duckdb-1-5` variant of this alias for the full explanation: without
149/// that feature the appender reports errors through the stable
150/// `duckdb_appender_error`, which carries a message but no category.
151#[cfg(not(feature = "duckdb-1-5"))]
152pub type AppendError = crate::error::ExtensionError;
153
154/// RAII wrapper for a `duckdb_appender`.
155///
156/// The appender is closed and destroyed automatically on drop. To surface any
157/// error from the final flush, call [`close`][Appender::close] explicitly
158/// beforehand.
159///
160/// See the [module docs][crate::appender] for the two append styles and the
161/// buffering rules that decide when an error appears.
162pub struct Appender {
163 handle: duckdb_appender,
164 /// Values appended to the current row: a mirror of `DuckDB`'s
165 /// `BaseAppender::column`, which advances only on a successful append.
166 column: Cell<u64>,
167 /// Rows ended or appended as chunks since the last flush this wrapper
168 /// performed — what a poisoned appender loses.
169 buffered: Cell<u64>,
170 lifecycle: Cell<Lifecycle>,
171}
172
173/// Where an [`Appender`] is in its life, as far as quack-rs can tell.
174#[derive(Clone, Copy, Debug, PartialEq, Eq)]
175enum Lifecycle {
176 Open,
177 /// A `row` was abandoned after some of its values went in; see the module
178 /// docs.
179 Poisoned,
180 /// `close` succeeded.
181 Closed,
182}
183
184impl Appender {
185 // ── Errors ──────────────────────────────────────────────────────────
186
187 /// Returns the structured error from the most recent failed operation.
188 #[cfg(feature = "duckdb-1-5")]
189 #[must_use]
190 pub fn error_data(&self) -> ErrorData {
191 // SAFETY: self.handle may be null (a failed create); DuckDB handles
192 // that and returns an owned, empty error data handle.
193 let raw = unsafe { libduckdb_sys::duckdb_appender_error_data(self.handle) };
194 // SAFETY: raw is an owned duckdb_error_data (possibly null).
195 unsafe { ErrorData::from_raw(raw) }
196 }
197
198 /// Returns the message from the most recent failed operation, if any.
199 ///
200 /// Always available; with `duckdb-1-5` prefer
201 /// `error_data` (`duckdb-1-5`), which also carries the error category.
202 #[must_use]
203 pub fn error_message(&self) -> Option<String> {
204 if self.handle.is_null() {
205 return None;
206 }
207 // SAFETY: self.handle is non-null; DuckDB returns null when there is
208 // no error, and otherwise a string it owns until the appender is
209 // destroyed — so it is copied out here rather than borrowed.
210 let ptr = unsafe { duckdb_appender_error(self.handle) };
211 if ptr.is_null() {
212 return None;
213 }
214 // SAFETY: ptr is a valid NUL-terminated string owned by the appender.
215 Some(
216 unsafe { CStr::from_ptr(ptr) }
217 .to_string_lossy()
218 .into_owned(),
219 )
220 }
221
222 /// Returns the raw handle.
223 #[inline]
224 #[must_use]
225 pub const fn as_raw(&self) -> duckdb_appender {
226 self.handle
227 }
228
229 /// Reads whichever error channel this build has.
230 fn last_error(&self) -> AppendError {
231 #[cfg(feature = "duckdb-1-5")]
232 {
233 self.error_data()
234 }
235 #[cfg(not(feature = "duckdb-1-5"))]
236 {
237 self.error_message().map_or_else(
238 || append_error("appender operation failed"),
239 crate::error::ExtensionError::new,
240 )
241 }
242 }
243
244 /// Converts a `duckdb_state` into a `Result`, reading the appender's error
245 /// on failure.
246 fn check(&self, state: duckdb_state) -> Result<(), AppendError> {
247 if state == DuckDBSuccess {
248 Ok(())
249 } else {
250 Err(self.last_error())
251 }
252 }
253
254 // ── Row and lifecycle tracking ──────────────────────────────────────
255
256 /// Wraps a handle from `duckdb_appender_create*` in a fresh, open state.
257 const fn wrap(handle: duckdb_appender) -> Self {
258 Self {
259 handle,
260 column: Cell::new(0),
261 buffered: Cell::new(0),
262 lifecycle: Cell::new(Lifecycle::Open),
263 }
264 }
265
266 /// `Ok` while the appender is open; the reason otherwise.
267 fn usable(&self) -> Result<(), AppendError> {
268 match self.lifecycle.get() {
269 Lifecycle::Open => Ok(()),
270 Lifecycle::Closed => Err(append_error(
271 "the appender is closed; create a new one to append more rows",
272 )),
273 Lifecycle::Poisoned => Err(self.lost_rows_error("append")),
274 }
275 }
276
277 /// Runs one value append if the appender is open, and counts it.
278 fn append_one(&self, append: impl FnOnce() -> duckdb_state) -> Result<(), AppendError> {
279 self.usable()?;
280 self.record_append(append())
281 }
282
283 /// Counts a value append that `DuckDB` accepted. A rejected value leaves
284 /// `DuckDB`'s column counter where it was (every append path throws before
285 /// advancing it), so the same column can be tried again.
286 fn record_append(&self, state: duckdb_state) -> Result<(), AppendError> {
287 self.check(state)?;
288 self.column.set(self.column.get().saturating_add(1));
289 Ok(())
290 }
291
292 /// Records the outcome of an operation that flushes on success.
293 fn flushed(&self, state: duckdb_state) -> Result<(), AppendError> {
294 self.check(state)?;
295 self.buffered.set(0);
296 Ok(())
297 }
298
299 /// The error for rows `DuckDB` will not write: a poisoned appender, or
300 /// `close` with an unfinished row.
301 fn lost_rows_error(&self, operation: &str) -> AppendError {
302 let rows = self.buffered.get();
303 let values = self.column.get();
304 if self.lifecycle.get() != Lifecycle::Poisoned {
305 return append_error(&format!(
306 "{operation}: the current row is unfinished ({values} value(s) without \
307 end_row), and DuckDB writes nothing while it is: up to {rows} row(s) buffered \
308 since the last flush are lost if the appender is dropped now; end the row and \
309 close again"
310 ));
311 }
312 let recovery = if cfg!(feature = "duckdb-1-5") {
313 "; `clear` discards them and makes the appender usable again"
314 } else {
315 ""
316 };
317 append_error(&format!(
318 "{operation}: the appender is poisoned: up to {rows} row(s) buffered since the last \
319 flush were not written, because a row failed after {values} of its values were \
320 appended and DuckDB can neither finish nor drop a half-written row{recovery}"
321 ))
322 }
323}
324
325/// Builds an [`AppendError`] for a failure quack-rs detected itself, before
326/// `DuckDB` was ever called.
327fn append_error(message: &str) -> AppendError {
328 #[cfg(feature = "duckdb-1-5")]
329 {
330 ErrorData::new(crate::error_data::DuckDbErrorType::InvalidInput, message)
331 }
332 #[cfg(not(feature = "duckdb-1-5"))]
333 {
334 crate::error::ExtensionError::new(message)
335 }
336}
337
338impl Drop for Appender {
339 fn drop(&mut self) {
340 if !self.handle.is_null() {
341 // SAFETY: self.handle is a valid handle that we own. Destroy
342 // closes (and so flushes) it first; the state is intentionally
343 // ignored here — `close` beforehand is how a final flush error is
344 // observed, because destruction also frees the error message.
345 unsafe { duckdb_appender_destroy(&raw mut self.handle) };
346 }
347 }
348}
349
350crate::debug_repr::impl_handle_debug!(Appender.handle);