Copy Functions
A DuckDB copy function implements a custom file format for the COPY statement.
This page shows how to register one from a Rust extension with quack-rs'
CopyFunctionBuilder.
Requires the
duckdb-1-5feature flag (DuckDB 1.5.0+).
Open the file through DuckDB. The path of a
COPYis user input.std::fsignoresSET enable_external_access = falseandallowed_directories, and cannot reachs3://orhttp://paths; theFileSystemwrapper honours the settings and resolves every registered file system.
A format can support writing, reading, or both:
| Direction | You supply | DuckDB calls |
|---|---|---|
COPY t TO 'f' (FORMAT my_format) | bind + sink + finalize (and optionally global_init) | those callbacks |
COPY t FROM 'f' (FORMAT my_format) | copy_from(table_function) | your table function's bind, init and scan |
duckdb_register_copy_function decides which directions a format supports by
looking at the sink and the reader independently, so a read-only format leaves
the writing callbacks unset entirely. Set bind, sink and finalize together
or not at all: register refuses a builder with only some of them, and one that
implements neither direction.
Lifecycle (COPY … TO)
- Bind — called when the statement is bound: once for a plain
COPY, and again on everyEXECUTEof a prepared one. Inspect output columns, configure the export. - Global init — called once per output file: once for a plain
COPY, once per file withPER_THREAD_OUTPUTorPARTITION_BY. Open the file, allocate that file's global state. WithUSE_TMP_FILEthe path is a temporary name that DuckDB renames afterwards. - Sink — called for each data chunk; with
PER_THREAD_OUTPUTorPARTITION_BY, from several threads at once (see Threads). Write rows to the output. - Finalize — called once per output file, after its last sink. Flush buffers, close the file. It is not called when a sink reports an error, so release resources in the global state's destructor as well.
Builder API
Write each COPY … TO callback with its guarded macro: copy_bind_callback!,
copy_global_init_callback!, copy_sink_callback! and
copy_finalize_callback!. Each one generates the unsafe extern "C" fn the
builder expects and reports a panic in its body as an error on the COPY
statement, instead of aborting the process. See
Panic-Safe Callbacks.
#![allow(unused)] fn main() { use quack_rs::copy_function::{CopyBindInfo, CopyFunctionBuilder, CopySinkInfo}; use quack_rs::data_chunk::DataChunk; quack_rs::copy_bind_callback!(my_bind, |info| { let bind = unsafe { CopyBindInfo::new(info) }; if bind.column_count() == 0 { bind.set_error("my_format needs at least one column"); } }); quack_rs::copy_global_init_callback!(my_global_init, |info| { let _ = info; // open the output file here }); quack_rs::copy_sink_callback!(my_sink, |info, chunk| { let _sink = unsafe { CopySinkInfo::new(info) }; let chunk = unsafe { DataChunk::from_raw(chunk) }; let _rows = chunk.size(); // write the rows here }); quack_rs::copy_finalize_callback!(my_finalize, |info| { let _ = info; // flush and close the file here }); fn demo() -> Result<(), quack_rs::error::ExtensionError> { let builder = CopyFunctionBuilder::try_new("my_format")? .bind(my_bind) .global_init(my_global_init) .sink(my_sink) .finalize(my_finalize); // Register on a connection, for example in the entry point: // unsafe { builder.register(con)?; } let _ = builder; Ok(()) } }
COPY … FROM
Reading is a table function, attached to the copy function rather than
registered on its own. Build it with
TableFunctionBuilder::build_handle,
then hand it to copy_from:
#![allow(unused)] fn main() { use quack_rs::copy_function::CopyFunctionBuilder; use quack_rs::table::TableFunctionBuilder; use quack_rs::types::TypeId; quack_rs::table_bind_callback!(bind, |_info| {}); quack_rs::table_init_callback!(init, |_info| {}); quack_rs::table_scan_callback!(scan, |_info, _output| {}); fn demo() -> Result<(), quack_rs::error::ExtensionError> { // SAFETY: the callbacks match their declared signatures. let reader = unsafe { TableFunctionBuilder::new("my_format") // name it after the format .param(TypeId::Varchar) // the file path — exactly one .named_param("skip_rows", TypeId::BigInt) // a COPY option .bind(bind) .init(init) .scan(scan) .build_handle() }?; let format = CopyFunctionBuilder::try_new("my_format")?.copy_from(reader)?; // unsafe { format.register(con)?; } Ok(()) } }
Four things about the reader are not like an ordinary table function:
- The file path is positional parameter 0, always
VARCHAR.duckdb.hrequires the function to declare exactly that one parameter, and DuckDB does not check it —copy_fromdoes, and returns an error naming the mismatch. - COPY options are named parameters.
(FORMAT my_format, SKIP_ROWS 1)arrives asskip_rows; matching is case-insensitive. An option the function never declared is a binder error before your bind callback runs — and the message names the table function, which is why the example gives it the format's name. - Those options arrive uncast. DuckDB passes each value as written, not
cast to the declared type:
SKIP_ROWS 'abc'reaches aBIGINTparameter as theVARCHAR'abc', andSKIP_ROWS 3as anINTEGER. An option written without a value ((FORMAT my_format, HEADER)) is not passed at all. CheckValue::type_id()before trusting a value —as_i64_or(0)on'abc'quietly returns the default. - The schema is already fixed, because
COPY … FROMloads into an existing table. The bind callback must not calladd_result_column(a typed reader built withwith_stateorwith_bind_initfails its bind if it does). Read the target's schema instead:
#![allow(unused)] fn main() { use quack_rs::table::BindInfo; fn demo(bind: &BindInfo) { for i in 0..bind.result_column_count() { let name = bind.result_column_name(i); // SAFETY: `i` is in range, and this runs during the bind callback. let ty = unsafe { bind.result_column_type(i) }; let _ = (name, ty); } } }
Choosing the format from the file name
COPY t TO 'out.my_format' with no FORMAT clause picks your format when the file
extension equals the copy function's name, and so does COPY t FROM 'in.my_format' for the
reader. Measured on DuckDB 1.5.6 and 2.0.0: a copy function named pr_fmt was used for
out.pr_fmt and was not used for out.dat. A function cannot declare a different
extension (the C API v1 has no way to: see Known
Limitations),
so name the format after the extension you want recognised.
Registering a format name twice
duckdb_register_copy_function drops a copy function whose name already exists —
your earlier registration, another extension's, or a built-in format such as csv
— and still reports success. register checks first and returns an error. There is
no catalog view of copy functions, so the check asks the binder: it runs
COPY (SELECT <missing column>) TO '' (FORMAT '<name>'), where a binder error
means the format resolved and a catalog error means it did not. Nothing is written
and no copy callback runs; a format name owned by an autoloadable extension
(parquet, json) may be autoloaded by that lookup, exactly as the same COPY
typed by a user would. Copy functions registered through the C API are never
persisted, so reloading into a database file does not trip the check.
Options
CopyBindInfo::options() returns the COPY … TO options as one STRUCT value
(None only if DuckDB returns a null handle). How DuckDB 1.5.5 builds it:
- Option names are upper-cased:
compression 'zstd'arrives asCOMPRESSION.FORMATitself is not among them. - With no options besides
FORMAT, the value is SQLNULL, not an emptySTRUCT— checkis_sql_null()first. - An option given without a value (
HEADER) is aNULLfield. - Several values (
LST (1, 2)) arrive as aLIST, or an unnamedSTRUCTwhen their types differ. - An explicit
NULLvalue is rejected by the binder before your callback runs. - DuckDB does not validate the options of a
COPY … TOformat. A copy function cannot declare which options it takes, so(FORMAT my_format, bogus 1), an option without a value, a list where you expect a scalar, and a value of the wrong type all reach your bind callback and are accepted unless you reject them. (A built-in format rejects the same typo:Unrecognized option "bogus" for csv.) Check every name and type yourself and callset_error; otherwise a misspelt option is silently ignored. ACOPY … FROMreader is different: its options are the table function's named parameters, which DuckDB checks. - Field order follows DuckDB's internal hash map, not the statement — look fields up by name:
#![allow(unused)] fn main() { use quack_rs::copy_function::CopyBindInfo; fn demo(bind: &CopyBindInfo) -> Option<String> { let options = bind.options()?; if options.is_sql_null() { return None; // COPY ... (FORMAT my_format) with no other options } let names = options.struct_field_names(); let idx = names.iter().position(|n| n == "COMPRESSION")?; options.struct_child(idx)?.as_str().ok() } }
Threads
The data pointers are untyped, so the compiler cannot check this for you:
extra_infolives as long as the database and is read from every connection's thread — treat it asT: Send + Sync.- Bind data is shared by every sink call.
COPY … TOwithPER_THREAD_OUTPUTorPARTITION_BYruns the sink on several threads at once — treat it asT: Send + Syncand never mutate it without a lock. - Global state is one per output file: per thread with
PER_THREAD_OUTPUT, per partition withPARTITION_BY(reachable from more than one thread). Guard any mutation with aMutexunless you know neither option is in use.
Typed state, without raw pointers
From 0.20.0 the three data pointers have typed forms that box the value behind a type tag
and generate a destructor that contains a panic in T::drop, so no Box::into_raw and no
extern "C" destructor is written by hand (all need duckdb-1-5):
| Set | Read |
|---|---|
CopyFunctionBuilder::extra_info_typed(value) | extra_info_ref::<T>() on each of the four info types |
CopyBindInfo::set_bind_data_typed(value) | bind_data_ref::<T>() in global init, sink and finalize |
CopyGlobalInitInfo::set_global_state_typed(value) | global_state_ref::<T>() in sink and finalize |
T is Send + Sync + 'static, because the sink can run on several threads at once; state
that changes needs a Mutex or atomics inside T. A read returns None when nothing was
set or when the value has another type, instead of reinterpreting memory. The readers are
unsafe only because they cannot tell a typed value from one set through the raw
set_bind_data or set_global_state: use one form per slot. FileHandle is Send (not
Sync) from 0.20.0, so a file opened in global init can be kept in the global state inside
a Mutex.
Callback signatures
| Phase | Signature | Guarded macro |
|---|---|---|
| Bind | unsafe extern "C" fn(info: duckdb_copy_function_bind_info) | copy_bind_callback!(name, |info| { … }) |
| Global init | unsafe extern "C" fn(info: duckdb_copy_function_global_init_info) | copy_global_init_callback!(name, |info| { … }) |
| Sink | unsafe extern "C" fn(info: duckdb_copy_function_sink_info, chunk: duckdb_data_chunk) | copy_sink_callback!(name, |info, chunk| { … }) |
| Finalize | unsafe extern "C" fn(info: duckdb_copy_function_finalize_info) | copy_finalize_callback!(name, |info| { … }) |
A raw unsafe extern "C" fn with one of these signatures also compiles, but a
panic in it aborts the process. The reader of a COPY … FROM is a table
function, so its callbacks use the table_*_callback! macros.
Callback info wrappers
Each phase provides an ergonomic wrapper type around its raw info handle. Wrap the handle at the top of your callback to access helper methods:
CopyBindInfo
| Method | Description |
|---|---|
column_count() | Number of output columns |
column_type(index) | LogicalType of the column at index, or None if out of range. There is no column name: the C API does not give a COPY … TO bind callback the names, so a writer that wants a header needs a names option from the user |
options() | The COPY … TO options, as one STRUCT Value (see Options) |
get_extra_info() | Extra-info pointer set on the copy function |
set_bind_data(data, destroy) | Store bind data and its destructor |
set_error(message) | Report a bind-time error |
get_client_context() | Returns a ClientContext for catalog/config access |
CopyGlobalInitInfo
| Method | Description |
|---|---|
get_bind_data() | Retrieve the bind data pointer |
get_extra_info() | Extra-info pointer set on the copy function |
get_file_path() | Output file path for the COPY operation; an error if it is not valid UTF-8 |
get_file_path_bytes() | The same path as its exact bytes, for a path that is not UTF-8 |
set_global_state(state, destroy) | Store global state and its destructor |
set_error(message) | Report an init-time error |
get_client_context() | Returns a ClientContext |
CopySinkInfo
| Method | Description |
|---|---|
get_bind_data() | Retrieve the bind data pointer |
get_extra_info() | Extra-info pointer set on the copy function |
get_global_state() | Retrieve the global state pointer |
set_error(message) | Report a sink-time error |
get_client_context() | Returns a ClientContext |
CopyFinalizeInfo
| Method | Description |
|---|---|
get_bind_data() | Retrieve the bind data pointer |
get_extra_info() | Extra-info pointer set on the copy function |
get_global_state() | Retrieve the global state pointer |
set_error(message) | Report a finalize-time error |
get_client_context() | Returns a ClientContext |
All four wrappers are re-exported from quack_rs::copy_function:
#![allow(unused)] fn main() { use quack_rs::copy_function::{CopyBindInfo, CopyGlobalInitInfo, CopySinkInfo, CopyFinalizeInfo}; }
Related modules
config_option— register custom settings for your formatclient_context— access the file system and catalog from callbackstable_description— inspect table metadatacatalog— look up catalog entriestable— build the table function aCOPY … FROMneeds