Skip to main content

Security and process isolation

Windmill provides multiple layers of process isolation to protect your infrastructure from potentially malicious or buggy code execution. This page covers the isolation mechanisms available and how to configure them.

Overview​

Windmill workers execute user-provided code in various languages. To protect the worker process and the underlying infrastructure, Windmill implements multiple isolation strategies:

  1. PID Namespace Isolation - Process memory and environment variable protection (disabled by default, requires configuration)
  2. NSJAIL Sandboxing - Filesystem, network, and resource isolation (optional)
  3. Agent Workers - Workers without direct database access, communicating via the API
  4. Worker Groups - Logical separation of workers that can be used to run workers on separate clusters with different network/resource access

Why isolation matters​

Without proper isolation, user scripts could:

  • Access parent worker process memory to extract credentials and secrets
  • Read environment variables from parent processes
  • Interfere with other jobs running on the same worker
  • Access files outside their job directory
  • Make unrestricted network connections
  • Consume unlimited system resources
Important Security Notice

Job isolation is disabled by default in Windmill. Both NSJAIL and PID namespace isolation must be manually configured. Without isolation, scripts can access sensitive data from the worker process memory and environment variables.

For production deployments, you should enable at least one isolation method (PID namespace or NSJAIL) to protect your infrastructure. See the Recommended Configurations section below.

PID namespace isolation​

What is PID namespace isolation?​

PID (Process ID) namespace isolation creates a separate process namespace for each job using Linux's unshare command. This prevents jobs from:

  • Seeing parent worker processes in ps output
  • Reading parent process memory via /proc/$pid/mem
  • Accessing parent environment variables via /proc/$pid/environ
  • Accessing parent file descriptors via /proc/$pid/fd

Why PID isolation matters​

With PID isolation, the job runs in its own namespace and cannot see the parent worker process, preventing access to the worker's memory and environment variables.

Enabling PID namespace isolation​

To enable PID namespace isolation, you need to:

  1. Set the environment variable on your worker:

    ENABLE_UNSHARE_PID=true

  2. Enable privileged mode in docker-compose.yml (required for the default --mount-proc flag):

    windmill_worker:
      privileged: true

Important: PID isolation is disabled by default in the main docker-compose.yml (both settings are commented out). You must manually uncomment these settings to enable it.

Step-by-step enablement​

In your docker-compose.yml, find the worker service and uncomment these lines:

windmill_worker:
  image: ${WM_IMAGE}
  # ... other settings ...

  # Uncomment these two lines:
  # privileged: true              # <- Uncomment this
  environment:
    - DATABASE_URL=${DATABASE_URL}
    - MODE=worker
    - WORKER_GROUP=default
    # - ENABLE_UNSHARE_PID=true    # <- Uncomment this

After uncommenting:

windmill_worker:
  image: ${WM_IMAGE}
  privileged: true # Now enabled
  environment:
    - DATABASE_URL=${DATABASE_URL}
    - MODE=worker
    - WORKER_GROUP=default
    - ENABLE_UNSHARE_PID=true # Now enabled

Requirements​

  • Linux only - Not supported on Windows or macOS
  • util-linux package - Provides the unshare command (usually pre-installed)
  • Privileged mode - Required in Docker for the default --mount-proc flag
  • User namespaces - Must be enabled in kernel (check with sysctl kernel.unprivileged_userns_clone on Debian/Ubuntu)

Configuration​

Default isolation flags​

By default, PID isolation uses these flags:

UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork --mount-proc"

Important: While --user --map-root-user enables unprivileged user namespaces, the --mount-proc flag requires privileged: true in Docker. This is necessary to provide an isolated /proc filesystem.

What each flag does:

  • --user --map-root-user - Creates user namespace and maps current user to root inside it
  • --pid --fork - Creates isolated PID namespace (both flags required together)
  • --mount-proc - Mounts isolated /proc filesystem (requires privileged mode in Docker)

Note: The --fork flag is required when using --pid. Custom configurations must include --fork for PID namespace isolation to work correctly.

Custom flags​

You can customize the isolation flags:

# More restrictive: add network isolation (still requires privileged mode)
UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork --mount-proc --net"

# Truly unprivileged: without --mount-proc (no privileged mode needed)
# Note: Less isolated /proc - jobs can still see host processes in /proc
UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork"

# Alternative: Skip user namespace (requires privileged mode)
UNSHARE_ISOLATION_FLAGS="--pid --fork --mount-proc"

Recommendation: Use the default flags with privileged: true for best security. Only use truly unprivileged mode if you cannot enable privileged mode and understand the security tradeoffs.

Tini for signal handling​

When PID namespace isolation is enabled and tini is available, Windmill uses tini as PID 1 inside the namespace. Tini properly handles signal forwarding, which ensures:

  • OOM-killed processes return exit code 137 (128 + SIGKILL) instead of ambiguous errors
  • Zombie processes are properly reaped
  • Signals are correctly forwarded to child processes

Tini is included in the official Windmill Docker images. If tini is not available, Windmill falls back to running without it (with a warning about potentially incorrect OOM exit codes).

You can customize the tini path:

UNSHARE_TINI_PATH=/custom/path/to/tini

Failure behavior​

If ENABLE_UNSHARE_PID=true but unshare is unavailable or fails, the worker will panic at startup with a detailed error message:

ENABLE_UNSHARE_PID is set but unshare test failed.
Error: unshare: Operation not permitted
Flags: --user --map-root-user --pid --fork --mount-proc

Solutions:
• Check if user namespaces are enabled: 'sysctl kernel.unprivileged_userns_clone'
• For Docker: Requires 'privileged: true' in docker-compose for --mount-proc flag
• For Kubernetes/Helm: Requires 'privileged: true' in securityContext for --mount-proc flag
• Try different flags via UNSHARE_ISOLATION_FLAGS env var (remove --mount-proc for unprivileged)
• Alternative: Use NSJAIL instead
• Disable: Set ENABLE_UNSHARE_PID=false

Note: This fail-fast behavior only occurs when ENABLE_UNSHARE_PID=true. If unset or false (the default), the worker starts normally without isolation.

Platform support​

PlatformSupportedNotes
Linux (docker-compose)✅ YesRequires manual configuration (disabled by default)
Linux (bare metal)✅ YesRequires util-linux package
Docker Desktop (Linux)✅ YesWorks in Linux containers with privileged mode
Docker Desktop (Windows)⚠️ NoUse agent workers without direct DB access
Docker Desktop (macOS)⚠️ NoSet ENABLE_UNSHARE_PID=false for macOS workers
Kubernetes / Helm✅ YesRequires privileged: true in securityContext

Agent workers​

Another approach to isolation is using agent workers. Agent workers:

  • Do not have direct database access
  • Communicate with the Windmill server only via the API
  • Cannot access database credentials or other workers' data
  • Provide network-level isolation from sensitive infrastructure

This approach is particularly useful when:

  • You want to run workers in untrusted environments
  • You need workers in different network zones without database access
  • You want an additional layer of security beyond process isolation

Agent workers can be combined with PID namespace isolation or NSJAIL for defense-in-depth.

NSJAIL sandboxing​

What is NSJAIL?​

NSJAIL is a process isolation tool from Google that provides:

  • Filesystem isolation - Jobs can only access their job directory and explicitly mounted paths
  • Network restrictions - Optional network isolation
  • Resource limits - CPU, memory, and process limits
  • User namespace - Jobs run as unprivileged users even when worker runs as root

When is NSJAIL used?​

NSJAIL is disabled by default. All Windmill images include the nsjail binary, so no special image is required.

Enabling NSJAIL​

To enable NSJAIL sandboxing, set the Job isolation instance setting to Nsjail (see Force sandboxing). It applies to every worker of the instance.

The DISABLE_NSJAIL=false environment variable is the fallback: it enables NSJAIL on the workers that carry it, whatever the instance setting says. Use it to sandbox only some worker groups:

DISABLE_NSJAIL=false

When to enable NSJAIL:

  • You need filesystem isolation beyond PID namespaces
  • You want to restrict network access from jobs
  • You need resource limits per job (CPU, memory)
  • You're running untrusted code and need defense-in-depth

NSJAIL configuration​

NSJAIL behavior is controlled by configuration files for each language. You can view the default configurations:

These configurations control resource limits, mount points, network isolation, and other security settings. The configs are embedded into the Windmill binary at compile time.

NSJAIL address space limit​

NSJAIL caps the virtual address space of a jailed job at rlimit_as MiB. For Python and Ansible jobs this defaults to 4096 MiB (4 GiB). This is a cap on virtual address reservation, not physical memory: JIT runtimes such as Bun (JavaScriptCore) and the JVM reserve large virtual ranges up front, so a subprocess spawned from a jailed Python or Ansible job can crash against this cap even when its actual memory use is modest.

Two per-worker-group environment variables let you raise or lift the cap:

  • NSJAIL_PY_RLIMIT_AS_MB - Python jobs
  • NSJAIL_ANSIBLE_RLIMIT_AS_MB - Ansible jobs

Each accepts a numeric value in MiB, or unlimited, none, inf or 0 to uncap the address space entirely. Both are read once at worker startup.

# Raise the Python address space cap to 16 GiB
NSJAIL_PY_RLIMIT_AS_MB=16384

# Or uncap it entirely
NSJAIL_PY_RLIMIT_AS_MB=unlimited

Since these are set per worker group, you can lift the cap only on a dedicated pool that runs these workloads. Only the address space limit is affected: the other resource limits (CPU, file size, open files) and the mount, PID and user namespace isolation that provide the actual security boundary are unchanged.

Without NSJAIL (default):

  • Jobs have full filesystem access under the worker's user permissions
  • Jobs can read any files the worker can read
  • You can still enable PID namespace isolation separately for process/memory protection (see below)

Running nsjail without privileged: true​

A non-privileged container blocks nsjail through the runtime's default seccomp profile, its default AppArmor profile and the masked /proc. The two setups below replace privileged: true with two profiles installed on the nodes.

With the Helm chart, set isolationSecurity: userNamespaces or isolationSecurity: capabilities on a worker group and name the profiles in localhostProfiles. Without it, apply the pod settings below and turn nsjail on as described in Enabling NSJAIL.

Seccomp and AppArmor profiles​

The profiles are in the Windmill Helm chart repository. The runtime defaults (RuntimeDefault) do not work in their place.

  • windmill-nsjail.seccomp.json: Docker's default seccomp profile plus clone with namespace flags, mount, umount2, pivot_root and sethostname.
  • windmill-nsjail.apparmor: the runtime default AppArmor profile with deny mount replaced by mount, umount and pivot_root rules.

Install them on every node that runs workers:

sudo install -D -m 0644 windmill-nsjail.seccomp.json /var/lib/kubelet/seccomp/profiles/windmill-nsjail.json
# only on nodes where AppArmor is enabled
sudo install -m 0644 windmill-nsjail.apparmor /etc/apparmor.d/windmill-nsjail
sudo apparmor_parser -r /etc/apparmor.d/windmill-nsjail

A pod that references a profile missing from its node fails to start, so autoscaled node groups need these steps in their bootstrap. On nodes without AppArmor, leave appArmorProfile out of the settings below. SELinux nodes need an equivalent policy that allows the container to mount.

Out-of-memory behavior​

From Kubernetes 1.32, a job that exceeds the container's memory limit in either setup takes the worker pod down with it, instead of being killed alone as under privileged: true. Setting singleProcessOOMKill: true in the kubelet configuration of the nodes restores the per-job kill.

With Kubernetes user namespaces (no added capability)​

With a user namespace for the pod, the worker and its jobs run as a non-root user with no capability.

spec:
  hostUsers: false
  containers:
    - name: windmill-worker
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
        procMount: Unmasked
        seccompProfile:
          type: Localhost
          localhostProfile: profiles/windmill-nsjail.json
        appArmorProfile:
          type: Localhost
          localhostProfile: windmill-nsjail
  • Leave DISABLE_NUSER unset and keep the non-root user: nsjail fails as root with all capabilities dropped.
  • Requires Kubernetes 1.33 or later, nodes that support user namespaces, and volumes that support ID-mapped mounts.
  • The worker logs a warning at startup that it cannot lower its oom_score_adj.

Pod Security Standards: baseline admits this pod on Kubernetes 1.35 and later (1.33 and 1.34 need the UserNamespacesPodSecurityStandards feature gate), and restricted rejects only procMount: Unmasked.

With the SYS_ADMIN capability​

On clusters without user namespaces, disable nsjail's user namespace with DISABLE_NUSER=true and give the worker SYS_ADMIN:

env:
  - name: DISABLE_NUSER
    value: "true"
securityContext:
  runAsUser: 0
  allowPrivilegeEscalation: false
  capabilities:
    drop: [ALL]
    add:
      - SYS_ADMIN       # nsjail: namespaces, mounts, pivot_root
      - SETPCAP         # nsjail: dropping capabilities before running the job
      - SYS_RESOURCE    # optional, worker: lowering its own oom_score_adj
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/windmill-nsjail.json
  appArmorProfile:
    type: Localhost
    localhostProfile: windmill-nsjail

runAsUser: 0 is required: the nsjail binary has no file capabilities, so under a non-root UID it fails with Operation not permitted.

Pod Security Standards: baseline rejects SYS_ADMIN and SYS_RESOURCE, so this pod needs an exemption for them.

Jobs run as root (UID 0) inside the jail, with no capabilities, instead of UID 1000. A jail escape still only reaches a container confined by seccomp, AppArmor and a reduced capability set, where privileged: true would give it the host.

With Docker​

The user namespace setup for a worker in Docker, after loading the AppArmor profile on the host with apparmor_parser -r:

windmill_worker:
  user: "1000:1000"
  cap_drop:
    - ALL
  security_opt:
    - no-new-privileges:true
    - systempaths=unconfined
    - seccomp=./windmill-nsjail.seccomp.json
    - apparmor=windmill-nsjail

A cache volume previously written by a root worker is not writable by UID 1000: recreate it or change its owner first.

Recycling the worker process​

A deployment that cannot run nsjail can rely on the process lifetime instead: with EXIT_AFTER_N_JOBS set, the worker exits after that many jobs (1 for one process per job) and its supervisor restarts it. In the official images the worker is PID 1 of its container, so its exit also ends every process a job left behind. Set it on a worker process (MODE=worker) running a single worker. Worker init and periodic scripts, flow orchestration, cache hits and dedicated worker jobs do not count.

The worker name suffix, otherwise random, is then derived from the hostname so that the restarted process keeps its entry in the workers list. Only if several worker processes of the same worker group run on one host, give each of them a distinct WORKER_SUFFIX to tell them apart.

Isolation comparison​

FeatureNSJAILPID NamespaceNone
Filesystem isolation✅ Full❌ No❌ No
Network isolation⚠️ Optional❌ No❌ No
Memory protection✅ Yes✅ Yes❌ No
Process visibility✅ Hidden✅ Hidden❌ Visible
Environment protection✅ Yes✅ Yes❌ No
Resource limits✅ Yes❌ No❌ No

Isolation hierarchy​

When multiple isolation methods are available, Windmill uses them in this priority order:

  1. NSJAIL (if nsjail binary available and DISABLE_NSJAIL=false) - Provides comprehensive isolation
  2. PID Namespace (if ENABLE_UNSHARE_PID=true and unshare available) - Provides process isolation
  3. None (if neither enabled or available) - No isolation

Important: NSJAIL and PID namespace isolation are not used simultaneously. If NSJAIL is available and enabled, it takes precedence and PID namespace isolation is not used.

Force sandboxing​

Instance-level setting (job_isolation)​

The job_isolation instance setting can be set to nsjail_sandboxing to enable NSJAIL sandboxing for all jobs across the instance, regardless of the per-worker DISABLE_NSJAIL environment variable. Sandboxing is enabled when either job_isolation is set to nsjail_sandboxing or DISABLE_NSJAIL=false — neither overrides the other.

Configure it from Instance settings or via the Infrastructure as code YAML configuration.

When job_isolation is set to nsjail_sandboxing but the NSJAIL binary is not available on a worker, all jobs on that worker will fail until NSJAIL is installed or the setting is changed. NSJAIL availability is always probed at worker startup regardless of DISABLE_NSJAIL.

All language executors (Python, TypeScript/Bun, Deno, Go, Bash, Rust, C#, Java, Ansible, PHP, Ruby) respect this setting through a centralized is_sandboxing_enabled() check.

Per-script sandbox annotation​

You can enable nsjail sandboxing on a per-script basis using the sandbox annotation. This works for Python, TypeScript (Bun and Deno), and Bash scripts. When present, the script runs inside an nsjail sandbox even if sandboxing is not enabled globally.

The script will fail with a clear error if nsjail is not available on the worker.

#sandbox

echo "This script runs inside an NSJAIL sandbox"
# sandbox

def main():
    return "sandboxed Python"
// sandbox

export async function main() {
  return "sandboxed TypeScript";
}

The annotation is logged as "sandbox mode (nsjail)" in job execution logs.

The sandbox annotation can be combined with volume annotations to run scripts with persistent file storage inside a sandbox.

To enable PID namespace isolation, add to your docker-compose.yml:

windmill_worker:
  privileged: true
  environment:
    - ENABLE_UNSHARE_PID=true

Why enable this: Protects worker credentials from being accessed by jobs through process memory and environment variables. This is a critical security feature for production deployments.

NSJAIL sandboxing (maximum security)​

environment:
  - DISABLE_NSJAIL=false

Provides comprehensive isolation including filesystem, network, and resource limits.

Security best practices​

  1. Enable isolation before production - Isolation is disabled by default. You should manually enable either NSJAIL or PID isolation before deploying to production
  2. Use PID isolation at minimum - At a minimum, enable PID namespace isolation with ENABLE_UNSHARE_PID=true and privileged: true
  3. Consider NSJAIL for untrusted code - Use NSJAIL if you run code from untrusted sources or need filesystem isolation
  4. Consider agent workers - For sensitive environments, use agent workers that don't have direct database access and communicate only via the API
  5. Test your isolation - Verify isolation is working by checking worker logs for isolation status messages
  6. Use worker groups - Separate untrusted workloads onto dedicated workers with stricter isolation, or run workers on separate clusters with different network/resource access
  7. Keep systems updated - Ensure kernel and util-linux are up to date
  8. Review user permissions - Limit who can create scripts in your Windmill instance using roles and permissions

Troubleshooting​

Worker fails to start with "unshare: Operation not permitted"​

Cause: User namespaces are disabled in the kernel.

Solution:

# Check if user namespaces are enabled
sysctl kernel.unprivileged_userns_clone

# Enable if disabled (requires root)
sysctl -w kernel.unprivileged_userns_clone=1

# Make permanent (add to /etc/sysctl.conf)
echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf

Docker container: "unshare: unshare failed: Operation not permitted"​

Cause: The --mount-proc flag requires privileged mode in Docker.

Solution: Enable privileged mode in docker-compose.yml:

windmill_worker:
  privileged: true
  environment:
    - ENABLE_UNSHARE_PID=true

Alternative: If you cannot use privileged mode, remove --mount-proc from the flags (less secure):

windmill_worker:
  environment:
    - ENABLE_UNSHARE_PID=true
    - UNSHARE_ISOLATION_FLAGS=--user --map-root-user --pid --fork

Note: Without --mount-proc, jobs may still be able to see host processes in /proc.

Kubernetes/Helm: unshare fails​

Cause: The --mount-proc flag requires privileged mode.

Solution: Enable privileged mode in pod spec:

securityContext:
  privileged: true

And set the environment variable:

env:
  - name: ENABLE_UNSHARE_PID
    value: 'true'

For Helm deployments, set the appropriate values to enable privileged mode and the environment variable.

Windows workers fail to start​

Cause: PID isolation is Linux-only.

Solution: Set ENABLE_UNSHARE_PID=false for Windows workers.

Jobs fail with "Cannot allocate memory"​

Cause: Insufficient resources for isolation overhead.

Solution:

  • Increase worker memory limits
  • Reduce number of concurrent jobs
  • Use DISABLE_NSJAIL=true and rely on PID isolation only

AWS EKS with Bottlerocket AMI (including EKS Auto Mode)​

Cause: Bottlerocket sets user.max_user_namespaces=0 by default. The default PID isolation flags and nsjail's default configuration both create a user namespace, so the worker fails its unshare test at startup, or nsjail jobs fail with Couldn't launch the child process. EKS Auto Mode nodes run Bottlerocket and their kernel settings cannot be changed.

Solutions:

  1. Run nsjail without a user namespace (works on EKS Auto Mode): set DISABLE_NUSER=true and run the worker as root, either with privileged: true or with the SYS_ADMIN capability only. With the Helm chart, that is isolationSecurity: capabilities on the worker group. The Localhost profiles are optional: without them use seccompProfile: Unconfined, since RuntimeDefault blocks pivot_root. Bottlerocket's SELinux policy does not block nsjail. The user namespace setup cannot work here.

    Verified on EKS Auto Mode (Kubernetes 1.36) with bash, python3 and bun jobs, each in its own PID namespace with no capabilities.

  2. Keep PID isolation only: in a privileged worker, drop the user namespace from the flags with UNSHARE_ISOLATION_FLAGS="--pid --fork --mount-proc".

  3. Disable PID isolation: Set disableUnsharePid: true in Helm values (global or per-worker-group). Note: This reduces security isolation.

  4. Configure Bottlerocket kernel parameters (node groups you manage, not EKS Auto Mode): Use a custom launch template with user data to increase max_user_namespaces:

    [settings.kernel.sysctl]
    "user.max_user_namespaces" = "65536"

    See AWS EKS launch template documentation for details.