Build production-grade data pipelines

Write the DuckDB query for one slice; Windmill owns the write into a managed Ducklake table: replace, merge by key or append, always idempotent and snapshotted, so re-runs, retries and backfills are safe by construction. Add history to turn a keyed merge into a managed SCD2 dimension with as-of joins.

Declare -- partitioned daily (or hourly, weekly, monthly, dynamic) and every run processes exactly one slice. A partition status grid shows what exists, what failed and what is missing, and range backfill re-runs the gaps in one click, each partition an idempotent run.

Every managed write is a Ducklake snapshot. Read any table as of a past version with AT (VERSION => n), roll back, or reproduce a past run exactly. Cascaded runs record the snapshot ids of their inputs, so you can always answer which version of each input produced a result.

unique, not_null, accepted_values, relationships and custom checks run against the freshly materialized slice and fail the run on violation, with a sample of the violating rows in the error. On Enterprise Edition, a failing test rolls back the whole write (write-audit-publish) so readers never see bad data.

Column-to-column lineage is inferred from the SQL itself: select a column to trace its full upstream and downstream impact across the pipeline. Output schemas are captured on every write, and saving a consumer script checks its column references against them.

Declare -- freshness 2h and every node shows a fresh or stale verdict on the graph. On Enterprise Edition, a watchdog re-runs stale scripts automatically: a backstop that stays out of the way as long as any other trigger keeps the asset current.

Share DuckDB logic as engine-native CREATE MACRO libraries: deploy once and every DuckDB script in the workspace can call them, no import needed. Real functions with signatures and autocomplete, not templated text.

Run a single step, a step and everything downstream, or a bounded slice: pick an entry point and the nodes to stop at, and Windmill runs exactly the path between them. The same bounds work from the CLI with wmill pipeline run --from --to.

Develop pipelines from your own machine: wmill pipeline dev watches your working tree and live-previews the graph in the browser as you edit, and wmill pipeline run --local runs it without deploying. The AI chat builds and edits pipelines directly, and workspace forks get an isolated Ducklake data environment, so dev runs write to dev tables and read missing ones from production through defer views.

Connect to databases, APIs and third-party services using typed resources. Credentials are stored centrally and injected at runtime. Share connections across scripts and flows without duplicating secrets.

Each step can have its own error handling strategy and configurable retries with exponential backoff. Run a custom script on failure (send a Slack alert, create a ticket), stop the pipeline early or mark non-critical steps to continue on error.

Start pipelines from cron schedules, webhooks, Postgres CDC, Kafka, SQS, or manually from the UI. Combine multiple trigger types on the same pipeline.

Built-in relational storage with zero setup via data tables. Query from Python, TypeScript, SQL or DuckDB. Credentials are managed internally and never exposed.

Promote pipelines from draft to production in one click with full deployment history and instant rollbacks. Sync your workspace with GitHub or GitLab, use your existing code review workflows and deploy via the UI, the CLI or CI/CD pipelines.

Every pipeline run is logged with inputs, outputs, duration and status. Filter by success or failure, inspect logs and re-run with one click. Track resource usage, monitor worker groups and set up alerts for failures.