Windmill Compared to Competitors
We are aware other frameworks exist out there. Some of them might suit your specific needs. We believe Windmill is the only solution to provide this comprehensive set of feature and to be fully open-source at the same time.
Out of transparency, here is our subjective impression on other players you might come across.
No-code solutions
Those names are well-known to the general public and contributed to grow awareness in the workflow building space. They are made for operational teams, have pretty UIs but are limited in terms of use cases, power & flexibility. We do not compare ourselves to them.
examples: Zapier or Make
Code-based workflow engines
On the opposite side, you could choose to go on solely code-based solutions. They give a solid foundation for building your workflows and clean up your code. However, these solutions are less intuitive as they are complex to set up and operate. Also, they do not allow to share scripts easily or build UIs. Last, they all suffer from slow cold-start and poor latency whereas Windmill is always blazing fast.
examples: Temporal, Airflow, Prefect & Dagster
More details
Airflow and Temporal are the golden standards. They are very good, battle-tested and scale well. They brought all the good abstractions for workflows and from a bird's eye perspective, we are all running DAGs of tasks, sorted in topological order, parallelized where possible and run by a fleet of workers.
Those frameworks are a great source of inspiration for us but we bring a more principled and more opinionated approach so that one can focus on writing scripts rather than becoming a workflow engine expert. The goal of Windmill is to bring the benefits of those workflow engines in a more accessible package that is fit not just for data engineers but for hybrid teams made of data scientist, ops, and software engineers with standard scripts in Python/Typescript/Go/Bash and low-code builder for the graph itself the common denominator, without any sacrifice on performance (and actually we run workflows faster than those frameworks, more on that in [1]), features, scalability/reliability and by improving the debuggability and developer experience of those. It is also possible to define workflows-as-code, in the same way as Prefect or Airflow (writing a program that defines the jobs and their dependencies, and then execute that program).
From a tech stack perspective, we rely on the ACID properties of PostgreSQL to achieve persistence and transactionality of the workflow's state. We made the simplifying assumption to be an at-least-once workflow engine, where in the exceptionally rare events of an infrastructure crash (machine shutdown, network split, etc), we will recover automatically but it is up to the application developer to implement idempotency in parts where it is critical for everything else, we support the same feature set (retries, error handler, suspend/sleep, approval steps, cancellation, inner workflows).
To do a fair comparison, looking at the quickstart is a great way to see the different orientation of the product. The Airflow quickstart is a great starting point. One has to learn what is an Airflow pipeline before being able to create or edit them. They also do not provide a great way to debug locally and iterate on those pipelines. You have to write them, deploy them, and then test them. By comparison, in Windmill you can write scripts locally, or test them either step by step or the full flow in the web UI.
Temporal is an sdk for workflows, meaning you have to code around their sdk and learn their abstractions. It is made for teams of software engineers that want to control very finely the execution of the workflow. Their documentation has good examples of that. Temporal is top notch, but it is complex and their primary language support is Go.
By comparison, in Windmill one would just write the canonical python or typescript scripts, exposing just a main function and build a dag in the low-code builder.
[1]: Windmill is not just a workflow engine, it is also a function as a service (FaaS) infrastructure where it can run arbitrary scripts in TypeScript/Python/Bash/Go. Contrary to Lambda or GCP cloud functions, we do not need the functions to be pre-packaged and deployed in advance AOT. For TypeScript, we rely on the deno runtime that leverages v8 isolates and the immutable caching capabilities of deno. For Python, we have implemented our own dependency resolver that will override the Python virtual path and create a unique virtual environment for that specific script that will respect the lockfile generated at time of saving the script/flow for reproducibility. Given that those are interpreted languages, we pay no performance penalty to interpret that code on demand. So the only limiting factor for task execution is that in the events that dependencies are not cached by the worker, they need to be installed at time of execution. With a limited number of workers, the likelihood of a cache miss is low as soon as one script/workflow is executed more than once. With a large fleet of workers, cache miss increase and hence we have implemented a global caching mechanism that relies on syncing the cache through s3. It is only available in our enterprise edition. With it in place, we run tasks and workflows with 0 overhead versus running the same scripts on bare-metal. You can even leverage hardware acceleration without any additional configuration.
Admin panels builders
Getting closer to Windmill. Those players have a blent on admin panels. Therefore they are strong on UIs and low-code features. They allow you to use code in the process. However we believe they lack flexibility for building complex workflows.
Those are great tools and since they focus solely on the UI builder part, they have made it a great and mature product. On the other hand, Windmill UI builder integrates well with the workflow and script execution engine.
If you were to choose Windmill for the workflow engine, we hope to convince you that it might be worth the switch later for the UI builder part given our high level of integrations between those layers and us having caught up with them on the UI builder part.
examples: Retool and its open source alternatives Tooljet, Appsmith & Openblocks
Workflow builder: n8n
n8n has no UI builder and is source-available but not open-source. n8n only supports javascript and is not centered around organizing your scripts, it is mostly around building workflows with pre-made integrations and inlined javascript code. n8n require pre-installation of the dependencies and does not handle per-node dependency. It is centered around integrations to external services.
Workflow builder: Pipedream
Pipedream has no UI builder and is not open-source (although their library of integrations is). Pipedream itself is neither open-source, nor self-hostable. Furthermore, it is centered around integrations to external services.
and ... Windmill
We are working to build a solution with a clear approach (targeting developers who do not want to compromise on flexibility) and the aim to solve main issues (scalability, technicity for advanced use cases, open-source).
We believe Windmill is different because:
- it allows building internal tools through code much faster, without sacrificing on one side intuitively and visibility, and on the other side, control, reliability, performance, flexibility and scalability
- it empowers semi-technical users to access and edit that code without being overwhelmed by the usual barriers to entry (git, IDE, local environments, secrets managements, etc.)
- it is compatible with senior/staff software engineers high standards for production-grade yet flexible yet customizable with code.
Have a look at our core concepts to see what makes us different.
We ourselves have our own limitations. We believe we can do better in terms of product education and having a prettier UI. All the rest is ready for scale.
Please note that this comparison is made with at least two biases: 1. we want to convince you of the power of our product and 2. we are never safe from hiding things from ourselves about the strengths of competitors.
Anyway, we are committed to the culture of transparency and of confronting the facts, so if you have any objections or suggestions, please contact us at [email protected].