Skip to main content

DuckDB integration

DuckDB is an open-source, in-process SQL OLAP database management system designed for fast analytical query workloads.

Windmill supports seamless integration with DuckDB, allowing you to manipulate data from S3 (csv, parquet, json), Azure Blob Storage, BigQuery, PostgreSQL, and MySQL.

DuckDB in Windmill supports automatic column detection on S3 objects. You can query S3 paths directly without wrapping them in read_parquet() — for example SELECT col1, col2 FROM 's3:///file.parquet' — and the SQL parser will infer the referenced columns. The standard read_parquet(), read_csv(), and read_json() table functions also support column detection when used with S3 paths.


Integration between DuckDB and Windmill

Azure Blob Storage support​

DuckDB scripts in Windmill can read from and write to Azure Blob Storage. When Azure Blob is configured as a workspace storage, DuckDB can use the same storage paths to query and write data in Parquet, CSV, or JSON format.

This works with the same S3-compatible path syntax, and requires an Azure Blob storage resource to be configured in the workspace.

Memory and spill limits​

DuckDB runs in-process in the worker. A query that outgrows DUCKDB_MEMORY_LIMIT (by default 80% of the worker's memory) spills to disk in the job directory, and the DUCKDB_MAX_TEMP_DIRECTORY_SIZE env variable caps how much a single query may write there so one query cannot fill the worker's disk. Neither can be raised from a script: a query that needs more spill than the cap fails with an out of memory error naming the max_temp_directory_size setting.

Both take DuckDB's units (KiB, MiB, GiB, TiB, KB, MB, GB, TB); Kubernetes' 6Gi spelling is rejected and fails every DuckDB job on the worker. Windmill Cloud caps the spill at 6GiB per worker. Self-hosted instances leave it unset, which keeps DuckDB's default of 90% of the free space on the worker's temp volume.

Pipelines and macro libraries​

DuckDB is also the engine of Windmill pipelines: DuckDB steps can materialize managed DuckLake tables, and shared SQL logic can be published as workspace macro libraries callable from any DuckDB script.

To get started, check out the SQL Getting Started section.