Skip to main content

Launch Week Day 2 - Worker Groups Management UI & Hot Reloading

· 5 min read
Guillaume Bouvignies

Worker Groups Configuration thumbnail

Welcome to the second day of Windmill's first launch week. To follow all the announcements made this week, please go to the Launch Week #1 page.

Today we are unveiling features related to workers. Workers are the foundation of Windmill. They are autonomous processes that run one script at a time using the entire CPU and memory available to them.

We are proud to announce:

  1. Worker groups to share configurations between workers, assign them to listen to particular queues of jobs (and optionally assign priorities) or change their pre-installed binaries on the fly (this article).
  2. Dedicated workers for scripts.

Worker groups are managed from the UI and are hot reloaded.

What is the purpose of worker groups?

In Windmill, workers run the jobs. The number of workers can be horizontally scaled up or down depending on needs without any overhead. Each worker on Windmill can run up to 26 million jobs a month, where each job lasts approximatively 100ms.

By default, every worker is the same and interchangeable. However, there are often needs to assign jobs to a specific worker pool, and to configure this worker pool to behave specifically or have different pre-installed binaries. To that end, we introduce the concept of "worker groups".

You can assign groups to flows and flow steps to be executed on specific queues. The name of those queues are called tags. Worker groups listen to those tags.

Examples of configurations include:

Once the configurations are done in the Windmill UI, the workers take their configuration dynamically, without having to manually restart them.

How to have a Worker join a Worker Group

Create a worker group in your docker-compose.yml and simply pass the worker group as the env variable WORKER_GROUP=<name_of_worker_group> for it to automatically join its corresponding worker group.

Windmill's responsibility is not to spawn the worker itself but to play well with existing service orchestrator such as Kubernetes, ECS, Nomad or Docker Compose, and any IaC. In those, you define the number of replicas (which can be auto-scaled up or down), the resource to allocate to those workers and the WORKER_GROUP passed as env.

Upon start, those workers will automatically join their worker group and fetch their configurations (including init scripts). They will also listen for changes on the worker group configuration for hot reloading.

Here is an example of a worker group specification in docker-compose:

pull_policy: always
replicas: 2
cpus: '1'
memory: 4096M
restart: unless-stopped
- MODE=worker
- WORKER_GROUP=highmem

Assign replicas, resource constraints, and that's it, the worker will automatically join the worker group on start and be displayed on the Workers page in the Windmill app!

Worker only require a database URL and can thus be spawned in separate VPCs if needed (as long as there is a tunnel to the database). There is also an agent mode for situations where workers are running in an untrusted environment.

Assign different jobs to specific worker groups

Giving tags to workers will change their behavior to only pull and execute jobs with those tags from the queue. When writing a script that should be executed on a specific worker group, you have to add at least one tag the worker group is listening for.

Tags & Queues

Set an init script

Setting an init script to a worker group configuration will automatically trigger a Bash script every time the workers are started. This approach offers added convenience to pre-install binaries or set initial configurations without the need to modify the base image.

Init Script

Dedicate your worker to a specific script or flow

Dedicated Workers are workers (groups) that are dedicated to a particular script or flow. We've written another blog post dedicated to this here.

How to create worker group configurations

In a self-hosted instance, worker groups are created from your docker-compose.yml by simply passing the worker group as the env variable WORKER_GROUP=worker_group_name. It is as simple as that and means that workers can be scaled horizontally.

The configuration of worker groups is done from the Windmill UI's Workers page, pick "New worker group config" and just write the name of your worker group.

New worker group config

You can then configure it directly from the UI and in particular set tags to listen to, define an init script or dedicate it to a script.

Worker group config

Learn more

Windmill Logo
Windmill is an open-source and self-hostable serverless runtime and platform combining the power of code with the velocity of low-code. We turn your scripts into internal apps and composable steps of flows that automate repetitive workflows.

You can self-host Windmill using a docker compose up, or go with the cloud app.