Virtual File System

Requires the duckdb-1-5 feature flag (DuckDB 1.5.0+).

The file_system module exposes DuckDB's virtual file system (VFS) to your extension, so a custom table function, replacement scan, or copy function can read and write files through the same abstraction DuckDB uses internally. Paths then resolve through every registered file system — httpfs (s3://, http://) when it is loaded, in-memory files, and so on — where std::fs only ever sees local disk.

Obtaining a FileSystem

A FileSystem comes from a ClientContext, which most function callbacks can hand you (for example via BindInfo::get_client_context() or ScalarBindInfo::get_client_context()).

Reading a file

#![allow(unused)]
fn main() {
use quack_rs::client_context::ClientContext;
use quack_rs::file_system::{FileOpenOptions, FileSystem};

fn read_all(ctx: &ClientContext) -> Option<Vec<u8>> {
let fs = FileSystem::from_client_context(ctx)?;
let handle = fs.open(c"s3://bucket/data.csv", &FileOpenOptions::read_only()).ok()?;

let mut buf = Vec::new();
handle.read_to_end(&mut buf).ok()?;
Some(buf)
}
}

Writing a file

#![allow(unused)]
fn main() {
use quack_rs::client_context::ClientContext;
use quack_rs::file_system::{FileOpenOptions, FileSystem};

fn write_report(ctx: &ClientContext, bytes: &[u8]) -> Result<(), quack_rs::error_data::ErrorData> {
let fs = FileSystem::from_client_context(ctx).expect("file system");
let handle = fs.open(c"report.bin", &FileOpenOptions::write_create())?;
handle.write_all(bytes)?;
handle.sync()?;
handle.close()?;
Ok(())
}
}

Open options and flags

FileOpenOptions describes how a file is opened. Two convenience constructors cover the common cases; use set_flag for anything else.

Constructor / methodEffect
FileOpenOptions::read_only()Open for reading
FileOpenOptions::write_create()Open for writing, creating if absent
FileOpenOptions::new()Empty; configure with set_flag
set_flag(flag, value)Set an individual FileFlag; returns true on success

FileFlag variants: Read, Write, Create, CreateNew, Append.

  • Create, CreateNew and Append need Write as well.
  • CreateNew ("create, failing if the file exists") also sets Create. DuckDB maps it to FILE_FLAGS_EXCLUSIVE_CREATE, which only has that meaning together with FILE_FLAGS_FILE_CREATE; on its own it neither creates a missing file nor refuses an existing one. On Windows an existing file is not refused: DuckDB's local file system there ignores the exclusive flag and opens with OPEN_ALWAYS (checked in DuckDB 1.5.5), so the existing file is opened, untruncated, without an error.
  • set_flag(flag, false) does not clear a flag: the C API ORs flags in and ignores value. Build a fresh FileOpenOptions instead.

FileHandle operations

MethodReturnsDescription
read(&mut buf)Result<usize, ErrorData>Read up to buf.len() bytes (0 = EOF)
read_exact(&mut buf)Result<(), ErrorData>Read exactly buf.len() bytes, or fail
read_to_end(&mut vec)Result<usize, ErrorData>Append the rest of the file
write(&buf)Result<usize, ErrorData>Write up to buf.len() bytes
write_all(&buf)Result<(), ErrorData>Write all of buf, or fail
seek(position)Result<(), ErrorData>Seek to an absolute byte offset
tell()Result<u64, ErrorData>Current byte offset
size()Result<u64, ErrorData>Total file size in bytes
sync()Result<(), ErrorData>Flush buffered writes to durable storage
close()Result<(), ErrorData>Close the file
error_data()ErrorDataStructured error from the last failed operation

Short reads and short writes are real

duckdb_file_handle_read and duckdb_file_handle_write return "the number of bytes actually read/written" — not a promise that the whole buffer moved. On a local file the two almost always agree; over httpfs a short transfer is common. Prefer read_exact / read_to_end / write_all, which loop for you, and reach for the raw read / write only when a partial transfer is what you want.

tell() and size() are Result for the same reason: the C API reports failure with a negative return value, which is far too easy to clamp to zero and then silently treat as an empty file.

FileSystem exposes open(path, options) and error_data(). Both FileSystem and FileHandle are RAII: they are destroyed (and the handle closed) on drop.

Lifetimes

A FileSystem<'ctx> borrows the ClientContext it came from, and a FileHandle<'fs> borrows the FileSystem that opened it. A handle refers to the database's file system, so using it after the database is closed is a use-after-free (valgrind: invalid read in duckdb::FileHandle::Read); the borrow makes that a compile error instead:

#![allow(unused)]
fn main() {
use quack_rs::client_context::ClientContext;
use quack_rs::file_system::{FileOpenOptions, FileSystem};

fn demo(ctx: &ClientContext) {
    let fs = FileSystem::from_client_context(ctx).unwrap();
    let handle = fs.open(c"data.csv", &FileOpenOptions::read_only()).unwrap();
    drop(fs); // error[E0505]: cannot move out of `fs` because it is borrowed
    let _ = handle.size();
}
}