Security and Process Isolation

Windmill provides multiple layers of process isolation to protect your infrastructure from potentially malicious or buggy code execution. This page covers the isolation mechanisms available and how to configure them.

Overview

Windmill workers execute user-provided code in various languages. To protect the worker process and the underlying infrastructure, Windmill implements multiple isolation strategies:

1. PID Namespace Isolation - Process memory and environment protection (disabled by default, requires configuration)
2. NSJAIL Sandboxing - Filesystem, network, and resource isolation (optional, requires special image)
3. Agent Workers - Workers without direct database access, communicating via the API
4. Worker Groups - Logical separation of workers that can be used to run workers on separate clusters with different network/resource access

Why Isolation Matters

Without proper isolation, user scripts could:

• Access parent worker process memory to extract credentials and secrets
• Read environment variables from parent processes
• Interfere with other jobs running on the same worker
• Access files outside their job directory
• Make unrestricted network connections
• Consume unlimited system resources

Important Security Notice

Job isolation is disabled by default in Windmill. Both NSJAIL and PID namespace isolation must be manually configured. Without isolation, scripts can access sensitive data from the worker process memory and environment variables.

For production deployments, you should enable at least one isolation method (PID namespace or NSJAIL) to protect your infrastructure. See the Recommended Configurations section below.

NSJAIL Sandboxing

What is NSJAIL?

NSJAIL is a process isolation tool from Google that provides:

• Filesystem isolation - Jobs can only access their job directory and explicitly mounted paths
• Network restrictions - Optional network isolation
• Resource limits - CPU, memory, and process limits
• User namespace - Jobs run as unprivileged users even when worker runs as root

When is NSJAIL Used?

NSJAIL is disabled by default because:

• The default Windmill images do not include the nsjail binary
• It requires using a special -nsjail tagged image

Enabling NSJAIL

To use NSJAIL sandboxing, you need both:

1. Use the nsjail image - Switch to an image with nsjail pre-installed:

   # In docker-compose.yml or your deployment config
   image: ghcr.io/windmill-labs/windmill-ee-nsjail
   # or for community edition
   image: ghcr.io/windmill-labs/windmill-nsjail

2. Enable NSJAIL - Set the environment variable:

   DISABLE_NSJAIL=false

When to enable NSJAIL:

• You need filesystem isolation beyond PID namespaces
• You want to restrict network access from jobs
• You need resource limits per job (CPU, memory)
• You're running untrusted code and need defense-in-depth

NSJAIL Configuration

NSJAIL behavior is controlled by configuration files for each language. You can view the default configurations:

• Python configuration
• Deno configuration
• Bash configuration
• All language configs

These configurations control resource limits, mount points, network isolation, and other security settings. The configs are embedded into the Windmill binary at compile time.

Without NSJAIL (default):

• Jobs have full filesystem access under the worker's user permissions
• Jobs can read any files the worker can read
• You can still enable PID namespace isolation separately for process/memory protection (see below)

PID Namespace Isolation

What is PID Namespace Isolation?

PID (Process ID) namespace isolation creates a separate process namespace for each job using Linux's unshare command. This prevents jobs from:

• Seeing parent worker processes in ps output
• Reading parent process memory via /proc/$pid/mem
• Accessing parent environment variables via /proc/$pid/environ
• Accessing parent file descriptors via /proc/$pid/fd

Why PID Isolation Matters

With PID isolation, the job runs in its own namespace and cannot see the parent worker process, preventing access to the worker's memory and environment variables.

Enabling PID Namespace Isolation

To enable PID namespace isolation, you need to:

1. Set the environment variable on your worker:

   ENABLE_UNSHARE_PID=true

2. Enable privileged mode in docker-compose.yml (required for the default --mount-proc flag):

   windmill_worker:
     privileged: true

Important: PID isolation is disabled by default in the main docker-compose.yml (both settings are commented out). You must manually uncomment these settings to enable it.

Step-by-Step Enablement

In your docker-compose.yml, find the worker service and uncomment these lines:

windmill_worker:
  image: ${WM_IMAGE}
  # ... other settings ...

  # Uncomment these two lines:
  # privileged: true              # <- Uncomment this
  environment:
    - DATABASE_URL=${DATABASE_URL}
    - MODE=worker
    - WORKER_GROUP=default
    # - ENABLE_UNSHARE_PID=true    # <- Uncomment this

After uncommenting:

windmill_worker:
  image: ${WM_IMAGE}
  privileged: true              # Now enabled
  environment:
    - DATABASE_URL=${DATABASE_URL}
    - MODE=worker
    - WORKER_GROUP=default
    - ENABLE_UNSHARE_PID=true    # Now enabled

Requirements

• Linux only - Not supported on Windows or macOS
• util-linux package - Provides the unshare command (usually pre-installed)
• Privileged mode - Required in Docker for the default --mount-proc flag
• User namespaces - Must be enabled in kernel (check with sysctl kernel.unprivileged_userns_clone on Debian/Ubuntu)

Configuration

Default Isolation Flags

By default, PID isolation uses these flags:

UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork --mount-proc"

Important: While --user --map-root-user enables unprivileged user namespaces, the --mount-proc flag requires privileged: true in Docker. This is necessary to provide an isolated /proc filesystem.

What each flag does:

• --user --map-root-user - Creates user namespace and maps current user to root inside it
• --pid --fork - Creates isolated PID namespace
• --mount-proc - Mounts isolated /proc filesystem (requires privileged mode in Docker)

Custom Flags

You can customize the isolation flags:

# More restrictive: add network isolation (still requires privileged mode)
UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork --mount-proc --net"

# Truly unprivileged: without --mount-proc (no privileged mode needed)
# Note: Less isolated /proc - jobs can still see host processes in /proc
UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork"

# Alternative: Skip user namespace (requires privileged mode)
UNSHARE_ISOLATION_FLAGS="--pid --fork --mount-proc"

Recommendation: Use the default flags with privileged: true for best security. Only use truly unprivileged mode if you cannot enable privileged mode and understand the security tradeoffs.

Failure Behavior

If ENABLE_UNSHARE_PID=true but unshare is unavailable or fails, the worker will panic at startup with a detailed error message:

ENABLE_UNSHARE_PID is set but unshare test failed.
Error: unshare: Operation not permitted
Flags: --user --map-root-user --pid --fork --mount-proc

Solutions:
• Check if user namespaces are enabled: 'sysctl kernel.unprivileged_userns_clone'
• For Docker: Requires 'privileged: true' in docker-compose for --mount-proc flag
• For Kubernetes/Helm: Requires 'privileged: true' in securityContext for --mount-proc flag
• Try different flags via UNSHARE_ISOLATION_FLAGS env var (remove --mount-proc for unprivileged)
• Alternative: Use NSJAIL instead
• Disable: Set ENABLE_UNSHARE_PID=false

Note: This fail-fast behavior only occurs when ENABLE_UNSHARE_PID=true. If unset or false (the default), the worker starts normally without isolation.

Platform Support

Platform	Supported	Notes
Linux (docker-compose)	✅ Yes	Requires manual configuration (disabled by default)
Linux (bare metal)	✅ Yes	Requires util-linux package
Docker Desktop (Linux)	✅ Yes	Works in Linux containers with privileged mode
Docker Desktop (Windows)	⚠️ No	Use agent workers without direct DB access
Docker Desktop (macOS)	⚠️ No	Set ENABLE_UNSHARE_PID=false for macOS workers
Kubernetes / Helm	✅ Yes	Requires privileged: true in securityContext

Agent Workers

Another approach to isolation is using agent workers. Agent workers:

• Do not have direct database access
• Communicate with the Windmill server only via the API
• Cannot access database credentials or other workers' data
• Provide network-level isolation from sensitive infrastructure

This approach is particularly useful when:

• You want to run workers in untrusted environments
• You need workers in different network zones without database access
• You want an additional layer of security beyond process isolation

Agent workers can be combined with PID namespace isolation or NSJAIL for defense-in-depth.

Isolation Comparison

Feature	NSJAIL	PID Namespace	None
Filesystem isolation	✅ Full	❌ No	❌ No
Network isolation	⚠️ Optional	❌ No	❌ No
Memory protection	✅ Yes	✅ Yes	❌ No
Process visibility	✅ Hidden	✅ Hidden	❌ Visible
Environment protection	✅ Yes	✅ Yes	❌ No
Resource limits	✅ Yes	❌ No	❌ No

Isolation Hierarchy

When multiple isolation methods are available, Windmill uses them in this priority order:

1. NSJAIL (if nsjail binary available and DISABLE_NSJAIL=false) - Provides comprehensive isolation
2. PID Namespace (if ENABLE_UNSHARE_PID=true and unshare available) - Provides process isolation
3. None (if neither enabled or available) - No isolation

Important: NSJAIL and PID namespace isolation are not used simultaneously. If NSJAIL is available and enabled, it takes precedence and PID namespace isolation is not used.

Recommended Configurations

PID namespace isolation (recommended for production)

To enable PID namespace isolation, add to your docker-compose.yml:

windmill_worker:
  privileged: true
  environment:
    - ENABLE_UNSHARE_PID=true

Why enable this: Protects worker credentials from being accessed by jobs through process memory and environment variables. This is a critical security feature for production deployments.

NSJAIL sandboxing (maximum security)

# Use the nsjail image
image: ghcr.io/windmill-labs/windmill-ee-nsjail
environment:
  - DISABLE_NSJAIL=false

Provides comprehensive isolation including filesystem, network, and resource limits.

Security Best Practices

1. Enable isolation before production - Isolation is disabled by default. You should manually enable either NSJAIL or PID isolation before deploying to production
2. Use PID isolation at minimum - At a minimum, enable PID namespace isolation with ENABLE_UNSHARE_PID=true and privileged: true
3. Consider NSJAIL for untrusted code - Use NSJAIL if you run code from untrusted sources or need filesystem isolation
4. Consider agent workers - For sensitive environments, use agent workers that don't have direct database access and communicate only via the API
5. Test your isolation - Verify isolation is working by checking worker logs for isolation status messages
6. Use worker groups - Separate untrusted workloads onto dedicated workers with stricter isolation, or run workers on separate clusters with different network/resource access
7. Keep systems updated - Ensure kernel and util-linux are up to date
8. Review user permissions - Limit who can create scripts in your Windmill instance

Troubleshooting

Worker fails to start with "unshare: Operation not permitted"

Cause: User namespaces are disabled in the kernel.

Solution:

# Check if user namespaces are enabled
sysctl kernel.unprivileged_userns_clone

# Enable if disabled (requires root)
sysctl -w kernel.unprivileged_userns_clone=1

# Make permanent (add to /etc/sysctl.conf)
echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf

Docker container: "unshare: unshare failed: Operation not permitted"

Cause: The --mount-proc flag requires privileged mode in Docker.

Solution: Enable privileged mode in docker-compose.yml:

windmill_worker:
  privileged: true
  environment:
    - ENABLE_UNSHARE_PID=true

Alternative: If you cannot use privileged mode, remove --mount-proc from the flags (less secure):

windmill_worker:
  environment:
    - ENABLE_UNSHARE_PID=true
    - UNSHARE_ISOLATION_FLAGS=--user --map-root-user --pid --fork

Note: Without --mount-proc, jobs may still be able to see host processes in /proc.

Kubernetes/Helm: unshare fails

Cause: The --mount-proc flag requires privileged mode.

Solution: Enable privileged mode in pod spec:

securityContext:
  privileged: true

And set the environment variable:

env:
  - name: ENABLE_UNSHARE_PID
    value: "true"

For Helm deployments, set the appropriate values to enable privileged mode and the environment variable.

Windows workers fail to start

Cause: PID isolation is Linux-only.

Solution: Set ENABLE_UNSHARE_PID=false for Windows workers.

Jobs fail with "Cannot allocate memory"

Cause: Insufficient resources for isolation overhead.

Solution:

• Increase worker memory limits
• Reduce number of concurrent jobs
• Use DISABLE_NSJAIL=true and rely on PID isolation only

Related Documentation

• Worker Groups - Logical worker separation
• Self-Hosting - Deployment configuration
• Environment Variables - All worker configuration options
• Docker - Container-based execution