Skip to main content

Launch Week Day 2 - Dedicated Workers for Scripts

· 4 min read
Ruben Fiszel

Execute jobs targeting a script much faster than normal workers.

Dedicated Workers thumbnail

Workers in Windmill

Workers serve as the backbone of Windmill. Workers are autonomous processes that run one script at a time using the full cpu and memory available to them.

To put it simply, workers are responsible for executing code in Windmill. A single normal worker in Windmill is capable of handling up to 26 million tasks a month, with each task taking roughly 100ms. You can easily scale the number of workers horizontally without incurring extra overhead.

Typically, workers fetch jobs from the job queue based on their scheduled_for datetime, provided it's in the past. Once a worker retrieves a job, it immediately changes its status to "running", processes it, streams its logs, and upon completion, archives it in the database as a "finished job". Both the final outcome and the logs are preserved indefinitely.

This simple process allows one to reliably count on a small number of workers for very different jobs. However, there is a latency between each task since the worker must perform a clean-up and a cold start between each execution (to be able to handle successive jobs but from completely different scripts). This latency amounts to around a dozen milliseconds, which can be crucial in the execution of certain priority jobs:

Dedicated Workers for scripts

Dedicated workers allow you to remove completely the cold start for selected scripts. Dedicated Workers are workers that are dedicated to a particular script. They are able to execute any job that target this script much faster than normal workers at the expense of being capable to only execute that one script. They are as fast as running the same logic in a while-loop processing requests, that is how they are implemented actually, but keep the benefit of showing separate jobs per execution:

What about AWS Lambda

It is faster than AWS lambda:

For Python and TypeScript

Dedicated workers work with TypeScript and Python scripts, they have the highest cold starts. Queries to databases such as PostgreSQL, MySQL, BigQuery, or Bash and Go scripts do not suffer from any cold starts and hence have the same benefits already without any complexity.

How to assign dedicated workers to a script

From Windmill UI's Workers page:

  1. "Edit config" of a worker group and enter script's workspace & path as worker:path.

Worker group config

The worker group will restart (assuming the pods/restart are set to restart automatically) and will now wait for step 2. below to happen:

  1. Toggle the "Dedicated Workers" option for that script in the script Settings:

Dedicated Workers in Settings

Each run will have a Worker Group Tag assigned to it. Worker Group Tags allow to assign custom worker groups to scripts and flows in Windmill for efficient execution on different machines with varying specifications.

Dedicated Workers Tag

What does it mean for flows ?

Flows steps that use scripts with a dedicated worker already benefit from the speed up of dedicated workers. However, you may wonder, could we not use dedicated workers to accelerate full flows ? Yes, you would be exactly right and we have a bigger announcement on the subject tomorrow.

Learn more

Windmill Logo
Windmill is an open-source and self-hostable serverless runtime and platform combining the power of code with the velocity of low-code. We turn your scripts into internal apps and composable steps of flows that automate repetitive workflows.

You can self-host Windmill using a docker compose up, or go with the cloud app.