Windmill vs Airflow

Python-literate data engineers running ETL and ML pipelines. A decade of community convention, books and tutorials is aimed at that persona. DAGs are Python files, and most integrations assume a Python runtime.

Non-developers can browse the DAG UI to monitor runs and trigger them manually. Authoring requires writing Python, so business users and analysts rarely edit DAGs directly.

DAGs are Python files placed in a dags/ folder and parsed by a scheduler. Tasks are built from operators (PythonOperator, BashOperator, provider-specific operators) or authored with the TaskFlow @task decorator. Data flows between tasks through XComs. Non-Python languages run as subprocesses inside a Python task.

DAGs are defined in Python. Every task is an independent execution unit: a trivial @task and a multi-minute job both go through the scheduler, executor and worker, with state transitions written to the metadata DB at each hop. The executor is pluggable (Local, Celery, Kubernetes), but the per-task overhead is fundamental to the model and is most visible on lightweight, high-volume workloads. The webserver renders DAGs (Graph, Grid, Gantt views) for monitoring, but authoring is Python-only.

No in-browser editor: authoring happens in your IDE on files in the dags/folder. airflow standalone spins up a local scheduler, webserver and SQLite metadata DB for testing; the Astronomer astro CLI adds a Docker-based local dev loop. Standard Python IDE tooling applies to DAG files. The feedback loop has a few intrinsic delays: the scheduler re-parses DAG files on its own cadence (defaults to every 30s), so new or edited DAGs appear in the UI after that window. Installing a new package requires restarting the scheduler and workers. airflow tasks test <dag> <task> runs a single task in-process for tighter iteration without going through the scheduler.

Python is first-class (PythonOperator, @task decorator). Other languages run through BashOperator or container-based operators (DockerOperator, KubernetesPodOperator). No native runtime for TypeScript, Go or other languages without wrapping them in a container.

Connections store credentials for external systems; Variables store configuration values. Both live in the metadata DB by default; external secret backends (Vault, AWS Secrets Manager, GCP Secret Manager) plug in via a config option.

DAGs are Python files, so they version naturally in Git. Deployment depends on the environment: sync the dags/ folder to a shared volume, bake it into a Docker image, or use a vendor deployment command. Connections, variables and RBAC config usually live outside Git.

Two paths. A curated catalog of 80+ provider packages (AWS, GCP, Azure, Snowflake, BigQuery, dbt, Airbyte, Databricks, Slack and most data-stack tools) bundles typed operators, hooks and sensors with managed connections and retries. Or pip install any PyPI package into the worker environment and call its SDK directly from a @task, the same way you would in Windmill.

Cron-style schedules, dataset-based triggering, sensors that poll external state, and HTTP webhook triggers via the REST API. Deferrable operators and the Triggerer handle event-like waits without holding a worker slot. No native Kafka or message-bus trigger; you build one with a deferrable sensor or operator.

Custom operator, hook or sensor as a Python class in plugins/ or a packaged provider. Authoring is straightforward; the friction is the deploy cycle: changes typically need a worker restart or an image rebuild before they pick up. Calling a Python library directly from @task sidesteps that and works for one-off logic.

If you already think in Python DAGs, onboarding is straightforward: add a file to thedags/ folder, set a schedule, pick operators. The learning curve is around Airflow-specific concepts (XComs, scheduler semantics, execution_date, catchup) and the operator catalog.

Task bodies written in plain Python port cleanly to any Python runtime. Logic that depends on Airflow-specific primitives (XComs, sensors, dataset triggers, provider operators, Jinja templating on execution_date) needs rewriting against the new platform's equivalents.

Per-task logs, Gantt chart and grid view in the UI. StatsD and OpenTelemetry metrics for scheduler and worker health. Vendor-managed Airflow typically layers richer observability dashboards on top.

Basic audit log of user actions in the UI. Longer retention and richer filtering are vendor features (Astronomer offers 7 days on Team, 90 days on Business). Vendor only.

Basic RBAC with predefined roles (Admin, Op, User, Viewer) in open source. SSO, custom RBAC, SCIM provisioning and SOC 2 / ISO certifications are supplied by commercial vendors like Astronomer or come from the underlying cloud (MWAA, Cloud Composer).Vendor only.

Single-tenant by design in open source. Isolation between teams is usually achieved by running separate Airflow deployments per team. Multi-deployment management is a vendor feature. Vendor only.

Three-hop dispatch on the critical path: scheduler (polls the metadata DB, re-parses DAG Python files every few seconds, evaluates dependencies, checks pool limits) then DB state update, then broker (Redis or RabbitMQ for Celery), then worker. Every state transition is a DB write.

Cold start depends on the executor. LocalExecutor forks a subprocess per task that imports the full DAG file plus the Airflow framework (typically 1 to 2s). KubernetesExecutor spins up a fresh pod per task (10 to 30s). Data passes between tasks through XCom, serialized to shared storage with a 48KB default limit.

Apache 2.0, free and unlimited self-hosted. Airflow itself has no proprietary tier: commercial features (SSO, custom RBAC, managed infrastructure, higher-tier support) are delivered by third-party vendors like Astronomer, AWS MWAA and Google Cloud Composer.

Astronomer publishes hourly deployment and worker pricing for its Developer and Team tiers; Business and Enterprise tiers require a sales conversation. AWS MWAA and Google Cloud Composer bill as cloud infrastructure.