Copy Functions

A DuckDB copy function implements a custom file format for the COPY statement. This page shows how to register one from a Rust extension with quack-rs' CopyFunctionBuilder.

Requires the duckdb-1-5 feature flag (DuckDB 1.5.0+).

Open the file through DuckDB. The path of a COPY is user input. std::fs ignores SET enable_external_access = false and allowed_directories, and cannot reach s3:// or http:// paths; the FileSystem wrapper honours the settings and resolves every registered file system.

A format can support writing, reading, or both:

DirectionYou supplyDuckDB calls
COPY t TO 'f' (FORMAT my_format)bind + sink + finalize (and optionally global_init)those callbacks
COPY t FROM 'f' (FORMAT my_format)copy_from(table_function)your table function's bind, init and scan

duckdb_register_copy_function decides which directions a format supports by looking at the sink and the reader independently, so a read-only format leaves the writing callbacks unset entirely. Set bind, sink and finalize together or not at all: register refuses a builder with only some of them, and one that implements neither direction.

Lifecycle (COPY … TO)

  1. Bind — called when the statement is bound: once for a plain COPY, and again on every EXECUTE of a prepared one. Inspect output columns, configure the export.
  2. Global init — called once per output file: once for a plain COPY, once per file with PER_THREAD_OUTPUT or PARTITION_BY. Open the file, allocate that file's global state. With USE_TMP_FILE the path is a temporary name that DuckDB renames afterwards.
  3. Sink — called for each data chunk; with PER_THREAD_OUTPUT or PARTITION_BY, from several threads at once (see Threads). Write rows to the output.
  4. Finalize — called once per output file, after its last sink. Flush buffers, close the file. It is not called when a sink reports an error, so release resources in the global state's destructor as well.

Builder API

Write each COPY … TO callback with its guarded macro: copy_bind_callback!, copy_global_init_callback!, copy_sink_callback! and copy_finalize_callback!. Each one generates the unsafe extern "C" fn the builder expects and reports a panic in its body as an error on the COPY statement, instead of aborting the process. See Panic-Safe Callbacks.

#![allow(unused)]
fn main() {
use quack_rs::copy_function::{CopyBindInfo, CopyFunctionBuilder, CopySinkInfo};
use quack_rs::data_chunk::DataChunk;

quack_rs::copy_bind_callback!(my_bind, |info| {
    let bind = unsafe { CopyBindInfo::new(info) };
    if bind.column_count() == 0 {
        bind.set_error("my_format needs at least one column");
    }
});

quack_rs::copy_global_init_callback!(my_global_init, |info| {
    let _ = info; // open the output file here
});

quack_rs::copy_sink_callback!(my_sink, |info, chunk| {
    let _sink = unsafe { CopySinkInfo::new(info) };
    let chunk = unsafe { DataChunk::from_raw(chunk) };
    let _rows = chunk.size(); // write the rows here
});

quack_rs::copy_finalize_callback!(my_finalize, |info| {
    let _ = info; // flush and close the file here
});

fn demo() -> Result<(), quack_rs::error::ExtensionError> {
let builder = CopyFunctionBuilder::try_new("my_format")?
    .bind(my_bind)
    .global_init(my_global_init)
    .sink(my_sink)
    .finalize(my_finalize);

// Register on a connection, for example in the entry point:
// unsafe { builder.register(con)?; }
let _ = builder;
Ok(())
}
}

COPY … FROM

Reading is a table function, attached to the copy function rather than registered on its own. Build it with TableFunctionBuilder::build_handle, then hand it to copy_from:

#![allow(unused)]
fn main() {
use quack_rs::copy_function::CopyFunctionBuilder;
use quack_rs::table::TableFunctionBuilder;
use quack_rs::types::TypeId;

quack_rs::table_bind_callback!(bind, |_info| {});
quack_rs::table_init_callback!(init, |_info| {});
quack_rs::table_scan_callback!(scan, |_info, _output| {});
fn demo() -> Result<(), quack_rs::error::ExtensionError> {
// SAFETY: the callbacks match their declared signatures.
let reader = unsafe {
    TableFunctionBuilder::new("my_format")   // name it after the format
        .param(TypeId::Varchar)              // the file path — exactly one
        .named_param("skip_rows", TypeId::BigInt) // a COPY option
        .bind(bind)
        .init(init)
        .scan(scan)
        .build_handle()
}?;

let format = CopyFunctionBuilder::try_new("my_format")?.copy_from(reader)?;
// unsafe { format.register(con)?; }
Ok(())
}
}

Four things about the reader are not like an ordinary table function:

  • The file path is positional parameter 0, always VARCHAR. duckdb.h requires the function to declare exactly that one parameter, and DuckDB does not check it — copy_from does, and returns an error naming the mismatch.
  • COPY options are named parameters. (FORMAT my_format, SKIP_ROWS 1) arrives as skip_rows; matching is case-insensitive. An option the function never declared is a binder error before your bind callback runs — and the message names the table function, which is why the example gives it the format's name.
  • Those options arrive uncast. DuckDB passes each value as written, not cast to the declared type: SKIP_ROWS 'abc' reaches a BIGINT parameter as the VARCHAR 'abc', and SKIP_ROWS 3 as an INTEGER. An option written without a value ((FORMAT my_format, HEADER)) is not passed at all. Check Value::type_id() before trusting a value — as_i64_or(0) on 'abc' quietly returns the default.
  • The schema is already fixed, because COPY … FROM loads into an existing table. The bind callback must not call add_result_column (a typed reader built with with_state or with_bind_init fails its bind if it does). Read the target's schema instead:
#![allow(unused)]
fn main() {
use quack_rs::table::BindInfo;
fn demo(bind: &BindInfo) {
for i in 0..bind.result_column_count() {
    let name = bind.result_column_name(i);
    // SAFETY: `i` is in range, and this runs during the bind callback.
    let ty = unsafe { bind.result_column_type(i) };
    let _ = (name, ty);
}
}
}

Choosing the format from the file name

COPY t TO 'out.my_format' with no FORMAT clause picks your format when the file extension equals the copy function's name, and so does COPY t FROM 'in.my_format' for the reader. Measured on DuckDB 1.5.6 and 2.0.0: a copy function named pr_fmt was used for out.pr_fmt and was not used for out.dat. A function cannot declare a different extension (the C API v1 has no way to: see Known Limitations), so name the format after the extension you want recognised.

Registering a format name twice

duckdb_register_copy_function drops a copy function whose name already exists — your earlier registration, another extension's, or a built-in format such as csv — and still reports success. register checks first and returns an error. There is no catalog view of copy functions, so the check asks the binder: it runs COPY (SELECT <missing column>) TO '' (FORMAT '<name>'), where a binder error means the format resolved and a catalog error means it did not. Nothing is written and no copy callback runs; a format name owned by an autoloadable extension (parquet, json) may be autoloaded by that lookup, exactly as the same COPY typed by a user would. Copy functions registered through the C API are never persisted, so reloading into a database file does not trip the check.

Options

CopyBindInfo::options() returns the COPY … TO options as one STRUCT value (None only if DuckDB returns a null handle). How DuckDB 1.5.5 builds it:

  • Option names are upper-cased: compression 'zstd' arrives as COMPRESSION. FORMAT itself is not among them.
  • With no options besides FORMAT, the value is SQL NULL, not an empty STRUCT — check is_sql_null() first.
  • An option given without a value (HEADER) is a NULL field.
  • Several values (LST (1, 2)) arrive as a LIST, or an unnamed STRUCT when their types differ.
  • An explicit NULL value is rejected by the binder before your callback runs.
  • DuckDB does not validate the options of a COPY … TO format. A copy function cannot declare which options it takes, so (FORMAT my_format, bogus 1), an option without a value, a list where you expect a scalar, and a value of the wrong type all reach your bind callback and are accepted unless you reject them. (A built-in format rejects the same typo: Unrecognized option "bogus" for csv.) Check every name and type yourself and call set_error; otherwise a misspelt option is silently ignored. A COPY … FROM reader is different: its options are the table function's named parameters, which DuckDB checks.
  • Field order follows DuckDB's internal hash map, not the statement — look fields up by name:
#![allow(unused)]
fn main() {
use quack_rs::copy_function::CopyBindInfo;
fn demo(bind: &CopyBindInfo) -> Option<String> {
let options = bind.options()?;
if options.is_sql_null() {
    return None; // COPY ... (FORMAT my_format) with no other options
}
let names = options.struct_field_names();
let idx = names.iter().position(|n| n == "COMPRESSION")?;
options.struct_child(idx)?.as_str().ok()
}
}

Threads

The data pointers are untyped, so the compiler cannot check this for you:

  • extra_info lives as long as the database and is read from every connection's thread — treat it as T: Send + Sync.
  • Bind data is shared by every sink call. COPY … TO with PER_THREAD_OUTPUT or PARTITION_BY runs the sink on several threads at once — treat it as T: Send + Sync and never mutate it without a lock.
  • Global state is one per output file: per thread with PER_THREAD_OUTPUT, per partition with PARTITION_BY (reachable from more than one thread). Guard any mutation with a Mutex unless you know neither option is in use.

Typed state, without raw pointers

From 0.20.0 the three data pointers have typed forms that box the value behind a type tag and generate a destructor that contains a panic in T::drop, so no Box::into_raw and no extern "C" destructor is written by hand (all need duckdb-1-5):

SetRead
CopyFunctionBuilder::extra_info_typed(value)extra_info_ref::<T>() on each of the four info types
CopyBindInfo::set_bind_data_typed(value)bind_data_ref::<T>() in global init, sink and finalize
CopyGlobalInitInfo::set_global_state_typed(value)global_state_ref::<T>() in sink and finalize

T is Send + Sync + 'static, because the sink can run on several threads at once; state that changes needs a Mutex or atomics inside T. A read returns None when nothing was set or when the value has another type, instead of reinterpreting memory. The readers are unsafe only because they cannot tell a typed value from one set through the raw set_bind_data or set_global_state: use one form per slot. FileHandle is Send (not Sync) from 0.20.0, so a file opened in global init can be kept in the global state inside a Mutex.

Callback signatures

PhaseSignatureGuarded macro
Bindunsafe extern "C" fn(info: duckdb_copy_function_bind_info)copy_bind_callback!(name, |info| { … })
Global initunsafe extern "C" fn(info: duckdb_copy_function_global_init_info)copy_global_init_callback!(name, |info| { … })
Sinkunsafe extern "C" fn(info: duckdb_copy_function_sink_info, chunk: duckdb_data_chunk)copy_sink_callback!(name, |info, chunk| { … })
Finalizeunsafe extern "C" fn(info: duckdb_copy_function_finalize_info)copy_finalize_callback!(name, |info| { … })

A raw unsafe extern "C" fn with one of these signatures also compiles, but a panic in it aborts the process. The reader of a COPY … FROM is a table function, so its callbacks use the table_*_callback! macros.

Callback info wrappers

Each phase provides an ergonomic wrapper type around its raw info handle. Wrap the handle at the top of your callback to access helper methods:

CopyBindInfo

MethodDescription
column_count()Number of output columns
column_type(index)LogicalType of the column at index, or None if out of range. There is no column name: the C API does not give a COPY … TO bind callback the names, so a writer that wants a header needs a names option from the user
options()The COPY … TO options, as one STRUCT Value (see Options)
get_extra_info()Extra-info pointer set on the copy function
set_bind_data(data, destroy)Store bind data and its destructor
set_error(message)Report a bind-time error
get_client_context()Returns a ClientContext for catalog/config access

CopyGlobalInitInfo

MethodDescription
get_bind_data()Retrieve the bind data pointer
get_extra_info()Extra-info pointer set on the copy function
get_file_path()Output file path for the COPY operation; an error if it is not valid UTF-8
get_file_path_bytes()The same path as its exact bytes, for a path that is not UTF-8
set_global_state(state, destroy)Store global state and its destructor
set_error(message)Report an init-time error
get_client_context()Returns a ClientContext

CopySinkInfo

MethodDescription
get_bind_data()Retrieve the bind data pointer
get_extra_info()Extra-info pointer set on the copy function
get_global_state()Retrieve the global state pointer
set_error(message)Report a sink-time error
get_client_context()Returns a ClientContext

CopyFinalizeInfo

MethodDescription
get_bind_data()Retrieve the bind data pointer
get_extra_info()Extra-info pointer set on the copy function
get_global_state()Retrieve the global state pointer
set_error(message)Report a finalize-time error
get_client_context()Returns a ClientContext

All four wrappers are re-exported from quack_rs::copy_function:

#![allow(unused)]
fn main() {
use quack_rs::copy_function::{CopyBindInfo, CopyGlobalInitInfo, CopySinkInfo, CopyFinalizeInfo};
}