Copy Functions

A DuckDB copy function implements a custom file format for the COPY statement. This page shows how to register one from a Rust extension with quack-rs' CopyFunctionBuilder.

Requires the duckdb-1-5 feature flag (DuckDB 1.5.0+).

A format can support writing, reading, or both:

duckdb_register_copy_function decides which directions a format supports by looking at the sink and the reader independently, so a read-only format leaves the writing callbacks unset entirely. Set bind, sink and finalize together or not at all: register refuses a builder with only some of them, and one that implements neither direction.

Lifecycle (COPY … TO)

1. Bind — called when the statement is bound: once for a plain COPY, and again on every EXECUTE of a prepared one. Inspect output columns, configure the export.
2. Global init — called once per output file: once for a plain COPY, once per file with PER_THREAD_OUTPUT or PARTITION_BY. Open the file, allocate that file's global state. With USE_TMP_FILE the path is a temporary name that DuckDB renames afterwards.
3. Sink — called for each data chunk; with PER_THREAD_OUTPUT or PARTITION_BY, from several threads at once (see Threads). Write rows to the output.
4. Finalize — called once per output file, after its last sink. Flush buffers, close the file. It is not called when a sink reports an error, so release resources in the global state's destructor as well.

Builder API

#![allow(unused)]
fn main() {
use quack_rs::copy_function::CopyFunctionBuilder;

fn demo(
    my_bind_fn: quack_rs::copy_function::CopyBindFn,
    my_global_init_fn: quack_rs::copy_function::CopyGlobalInitFn,
    my_sink_fn: quack_rs::copy_function::CopySinkFn,
    my_finalize_fn: quack_rs::copy_function::CopyFinalizeFn,
) -> Result<(), quack_rs::error::ExtensionError> {
let builder = CopyFunctionBuilder::try_new("my_format")?
    .bind(my_bind_fn)
    .global_init(my_global_init_fn)
    .sink(my_sink_fn)
    .finalize(my_finalize_fn);

// Register on a connection, for example in the entry point:
// unsafe { builder.register(con)?; }
Ok(())
}
}

COPY … FROM

Reading is a table function, attached to the copy function rather than registered on its own. Build it with TableFunctionBuilder::build_handle, then hand it to copy_from:

#![allow(unused)]
fn main() {
use quack_rs::copy_function::CopyFunctionBuilder;
use quack_rs::table::TableFunctionBuilder;
use quack_rs::types::TypeId;

fn demo(
    bind: quack_rs::table::BindFn,
    init: quack_rs::table::InitFn,
    scan: quack_rs::table::ScanFn,
) -> Result<(), quack_rs::error::ExtensionError> {
// SAFETY: the callbacks match their declared signatures.
let reader = unsafe {
    TableFunctionBuilder::new("my_format")   // name it after the format
        .param(TypeId::Varchar)              // the file path — exactly one
        .named_param("skip_rows", TypeId::BigInt) // a COPY option
        .bind(bind)
        .init(init)
        .scan(scan)
        .build_handle()
}?;

let format = CopyFunctionBuilder::try_new("my_format")?.copy_from(reader)?;
// unsafe { format.register(con)?; }
Ok(())
}
}

Four things about the reader are not like an ordinary table function:

• The file path is positional parameter 0, always VARCHAR. duckdb.h requires the function to declare exactly that one parameter, and DuckDB does not check it — copy_from does, and returns an error naming the mismatch.
• COPY options are named parameters. (FORMAT my_format, SKIP_ROWS 1) arrives as skip_rows; matching is case-insensitive. An option the function never declared is a binder error before your bind callback runs — and the message names the table function, which is why the example gives it the format's name.
• Those options arrive uncast. DuckDB passes each value as written, not cast to the declared type: SKIP_ROWS 'abc' reaches a BIGINT parameter as the VARCHAR 'abc', and SKIP_ROWS 3 as an INTEGER. An option written without a value ((FORMAT my_format, HEADER)) is not passed at all. Check Value::type_id() before trusting a value — as_i64_or(0) on 'abc' quietly returns the default.
• The schema is already fixed, because COPY … FROM loads into an existing table. The bind callback must not call add_result_column (a typed reader built with with_state or with_bind_init fails its bind if it does). Read the target's schema instead:

#![allow(unused)]
fn main() {
use quack_rs::table::BindInfo;
fn demo(bind: &BindInfo) {
for i in 0..bind.result_column_count() {
    let name = bind.result_column_name(i);
    // SAFETY: `i` is in range, and this runs during the bind callback.
    let ty = unsafe { bind.result_column_type(i) };
    let _ = (name, ty);
}
}
}

Registering a format name twice

duckdb_register_copy_function drops a copy function whose name already exists — your earlier registration, another extension's, or a built-in format such as csv — and still reports success. register checks first and returns an error. There is no catalog view of copy functions, so the check asks the binder: it runs COPY (SELECT <missing column>) TO '' (FORMAT '<name>'), where a binder error means the format resolved and a catalog error means it did not. Nothing is written and no copy callback runs; a format name owned by an autoloadable extension (parquet, json) may be autoloaded by that lookup, exactly as the same COPY typed by a user would. Copy functions registered through the C API are never persisted, so reloading into a database file does not trip the check.

Options

CopyBindInfo::options() returns the COPY … TO options as one STRUCT value (None only if DuckDB returns a null handle). How DuckDB 1.5.5 builds it:

• Option names are upper-cased: compression 'zstd' arrives as COMPRESSION. FORMAT itself is not among them.
• With no options besides FORMAT, the value is SQL NULL, not an empty STRUCT — check is_sql_null() first.
• An option given without a value (HEADER) is a NULL field.
• Several values (LST (1, 2)) arrive as a LIST, or an unnamed STRUCT when their types differ.
• An explicit NULL value is rejected by the binder before your callback runs.
• Field order follows DuckDB's internal hash map, not the statement — look fields up by name:

#![allow(unused)]
fn main() {
use quack_rs::copy_function::CopyBindInfo;
fn demo(bind: &CopyBindInfo) -> Option<String> {
let options = bind.options()?;
if options.is_sql_null() {
    return None; // COPY ... (FORMAT my_format) with no other options
}
let names = options.struct_field_names();
let idx = names.iter().position(|n| n == "COMPRESSION")?;
options.struct_child(idx)?.as_str().ok()
}
}

Threads

The data pointers are untyped, so the compiler cannot check this for you:

• extra_info lives as long as the database and is read from every connection's thread — treat it as T: Send + Sync.
• Bind data is shared by every sink call. COPY … TO with PER_THREAD_OUTPUT or PARTITION_BY runs the sink on several threads at once — treat it as T: Send + Sync and never mutate it without a lock.
• Global state is one per output file: per thread with PER_THREAD_OUTPUT, per partition with PARTITION_BY (reachable from more than one thread). Guard any mutation with a Mutex unless you know neither option is in use.

Callback signatures

Callback info wrappers

Each phase provides an ergonomic wrapper type around its raw info handle. Wrap the handle at the top of your callback to access helper methods:

CopyBindInfo

CopyGlobalInitInfo

CopySinkInfo

CopyFinalizeInfo

All four wrappers are re-exported from quack_rs::copy_function:

#![allow(unused)]
fn main() {
use quack_rs::copy_function::{CopyBindInfo, CopyGlobalInitInfo, CopySinkInfo, CopyFinalizeInfo};
}

Related modules

• config_option — register custom settings for your format
• client_context — access the file system and catalog from callbacks
• table_description — inspect table metadata
• catalog — look up catalog entries
• table — build the table function a COPY … FROM needs