# Security and process isolation

> What are the process isolation and sandboxing options available in Windmill?

Windmill provides multiple layers of process isolation to protect your infrastructure from potentially malicious or buggy code execution. This page covers the isolation mechanisms available and how to configure them.

## Overview

Windmill [workers](../../core_concepts/9_worker_groups/index.mdx) execute user-provided code in various languages. To protect the worker process and the underlying infrastructure, Windmill implements multiple isolation strategies:

1. **PID Namespace Isolation** - Process memory and [environment variable](../../core_concepts/47_environment_variables/index.mdx) protection (disabled by default, requires configuration)
2. **NSJAIL Sandboxing** - Filesystem, network, and resource isolation (optional)
3. **[Agent Workers](../../core_concepts/28_agent_workers/index.mdx)** - Workers without direct database access, communicating via the API
4. **[Worker Groups](../../core_concepts/9_worker_groups/index.mdx)** - Logical separation of workers that can be used to run workers on separate clusters with different network/resource access

## Why isolation matters

Without proper isolation, user [scripts](../../script_editor/index.mdx) could:

- Access parent worker process memory to extract credentials and [secrets](../../core_concepts/2_variables_and_secrets/index.mdx)
- Read environment variables from parent processes
- Interfere with other [jobs](../../core_concepts/20_jobs/index.mdx) running on the same worker
- Access files outside their job directory
- Make unrestricted network connections
- Consume unlimited system resources

:::warning Important Security Notice

**Job isolation is disabled by default** in Windmill. Both NSJAIL and PID namespace isolation must be manually configured. Without isolation, scripts can access sensitive data from the worker process memory and environment variables.

**For production deployments, you should enable at least one isolation method** (PID namespace or NSJAIL) to protect your infrastructure. See the [Recommended Configurations](#recommended-configurations) section below.

:::

## PID namespace isolation

### What is PID namespace isolation?

PID (Process ID) namespace isolation creates a separate process namespace for each job using Linux's [`unshare`](https://man7.org/linux/man-pages/man1/unshare.1.html) command. This prevents jobs from:

- Seeing parent worker processes in `ps` output
- Reading parent process memory via `/proc/$pid/mem`
- Accessing parent environment variables via `/proc/$pid/environ`
- Accessing parent file descriptors via `/proc/$pid/fd`

### Why PID isolation matters

With PID isolation, the job runs in its own namespace and cannot see the parent worker process, preventing access to the worker's memory and environment variables.

### Enabling PID namespace isolation

To enable PID namespace isolation, you need to:

1. **Set the environment variable** on your worker:

   ```bash
   ENABLE_UNSHARE_PID=true
   ```

2. **Enable privileged mode** in [docker-compose.yml](../1_self_host/index.mdx) (required for the default `--mount-proc` flag):
   ```yaml
   windmill_worker:
     privileged: true
   ```

**Important**: PID isolation is **disabled by default** in the main docker-compose.yml (both settings are commented out). You must manually uncomment these settings to enable it.

#### Step-by-step enablement

In your `docker-compose.yml`, find the worker service and uncomment these lines:

```yaml
windmill_worker:
  image: ${WM_IMAGE}
  # ... other settings ...

  # Uncomment these two lines:
  # privileged: true              # <- Uncomment this
  environment:
    - DATABASE_URL=${DATABASE_URL}
    - MODE=worker
    - WORKER_GROUP=default
    # - ENABLE_UNSHARE_PID=true    # <- Uncomment this
```

After uncommenting:

```yaml
windmill_worker:
  image: ${WM_IMAGE}
  privileged: true # Now enabled
  environment:
    - DATABASE_URL=${DATABASE_URL}
    - MODE=worker
    - WORKER_GROUP=default
    - ENABLE_UNSHARE_PID=true # Now enabled
```

### Requirements

- **Linux only** - Not supported on Windows or macOS
- **util-linux package** - Provides the `unshare` command (usually pre-installed)
- **Privileged mode** - Required in Docker for the default `--mount-proc` flag
- **User namespaces** - Must be enabled in kernel (check with `sysctl kernel.unprivileged_userns_clone` on Debian/Ubuntu)

### Configuration

#### Default isolation flags

By default, PID isolation uses these flags:

```bash
UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork --mount-proc"
```

**Important**: While `--user --map-root-user` enables unprivileged user namespaces, the `--mount-proc` flag **requires `privileged: true` in Docker**. This is necessary to provide an isolated /proc filesystem.

What each flag does:

- `--user --map-root-user` - Creates user namespace and maps current user to root inside it
- `--pid --fork` - Creates isolated PID namespace (both flags required together)
- `--mount-proc` - Mounts isolated /proc filesystem (requires privileged mode in Docker)

**Note**: The `--fork` flag is required when using `--pid`. Custom configurations must include `--fork` for PID namespace isolation to work correctly.

#### Custom flags

You can customize the isolation flags:

```bash
# More restrictive: add network isolation (still requires privileged mode)
UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork --mount-proc --net"

# Truly unprivileged: without --mount-proc (no privileged mode needed)
# Note: Less isolated /proc - jobs can still see host processes in /proc
UNSHARE_ISOLATION_FLAGS="--user --map-root-user --pid --fork"

# Alternative: Skip user namespace (requires privileged mode)
UNSHARE_ISOLATION_FLAGS="--pid --fork --mount-proc"
```

**Recommendation**: Use the default flags with `privileged: true` for best security. Only use truly unprivileged mode if you cannot enable privileged mode and understand the security tradeoffs.

#### Tini for signal handling

When PID namespace isolation is enabled and tini is available, Windmill uses [tini](https://github.com/krallin/tini) as PID 1 inside the namespace. Tini properly handles signal forwarding, which ensures:

- OOM-killed processes return exit code 137 (128 + SIGKILL) instead of ambiguous errors
- Zombie processes are properly reaped
- Signals are correctly forwarded to child processes

Tini is included in the official Windmill Docker images. If tini is not available, Windmill falls back to running without it (with a warning about potentially incorrect OOM exit codes).

You can customize the tini path:

```bash
UNSHARE_TINI_PATH=/custom/path/to/tini
```

### Failure behavior

If `ENABLE_UNSHARE_PID=true` but unshare is unavailable or fails, **the worker will panic at startup** with a detailed error message:

```
ENABLE_UNSHARE_PID is set but unshare test failed.
Error: unshare: Operation not permitted
Flags: --user --map-root-user --pid --fork --mount-proc

Solutions:
• Check if user namespaces are enabled: 'sysctl kernel.unprivileged_userns_clone'
• For Docker: Requires 'privileged: true' in docker-compose for --mount-proc flag
• For Kubernetes/Helm: Requires 'privileged: true' in securityContext for --mount-proc flag
• Try different flags via UNSHARE_ISOLATION_FLAGS env var (remove --mount-proc for unprivileged)
• Alternative: Use NSJAIL instead
• Disable: Set ENABLE_UNSHARE_PID=false
```

**Note**: This fail-fast behavior only occurs when `ENABLE_UNSHARE_PID=true`. If unset or false (the default), the worker starts normally without isolation.

### Platform support

| Platform                 | Supported | Notes                                               |
| ------------------------ | --------- | --------------------------------------------------- |
| Linux (docker-compose)   | ✅ Yes    | Requires manual configuration (disabled by default) |
| Linux (bare metal)       | ✅ Yes    | Requires util-linux package                         |
| Docker Desktop (Linux)   | ✅ Yes    | Works in Linux containers with privileged mode      |
| Docker Desktop (Windows) | ⚠️ No     | Use agent workers without direct DB access          |
| Docker Desktop (macOS)   | ⚠️ No     | Set `ENABLE_UNSHARE_PID=false` for macOS workers    |
| Kubernetes / Helm        | ✅ Yes    | Requires privileged: true in securityContext        |

## Agent workers

Another approach to isolation is using [agent workers](../../core_concepts/28_agent_workers/index.mdx). Agent workers:

- Do not have direct database access
- Communicate with the Windmill server only via the API
- Cannot access database credentials or other workers' data
- Provide network-level isolation from sensitive infrastructure

This approach is particularly useful when:

- You want to run workers in untrusted environments
- You need workers in different network zones without database access
- You want an additional layer of security beyond process isolation

Agent workers can be combined with PID namespace isolation or NSJAIL for defense-in-depth.

## NSJAIL sandboxing

### What is NSJAIL?

[NSJAIL](https://github.com/google/nsjail) is a process isolation tool from Google that provides:

- **Filesystem isolation** - Jobs can only access their job directory and explicitly mounted paths
- **Network restrictions** - Optional network isolation
- **Resource limits** - CPU, memory, and process limits
- **User namespace** - Jobs run as unprivileged users even when worker runs as root

### When is NSJAIL used?

NSJAIL is **disabled by default**. All Windmill images include the nsjail binary, so no special image is required.

### Enabling NSJAIL

To enable NSJAIL sandboxing, set the environment variable:

```bash
DISABLE_NSJAIL=false
```

**When to enable NSJAIL:**

- You need filesystem isolation beyond PID namespaces
- You want to restrict network access from jobs
- You need resource limits per job (CPU, memory)
- You're running untrusted code and need defense-in-depth

### NSJAIL configuration

NSJAIL behavior is controlled by configuration files for each language. You can view the default configurations:

- [Python configuration](https://github.com/windmill-labs/windmill/blob/main/backend/windmill-worker/nsjail/run.python3.config.proto)
- [TypeScript (Bun) configuration](https://github.com/windmill-labs/windmill/blob/main/backend/windmill-worker/nsjail/run.bun.config.proto)
- [Bash configuration](https://github.com/windmill-labs/windmill/blob/main/backend/windmill-worker/nsjail/run.bash.config.proto)
- [All language configs](https://github.com/windmill-labs/windmill/tree/main/backend/windmill-worker/nsjail)

These configurations control resource limits, mount points, network isolation, and other security settings. The configs are embedded into the Windmill binary at compile time.

**Without NSJAIL (default):**

- Jobs have full filesystem access under the worker's user permissions
- Jobs can read any files the worker can read
- You can still enable PID namespace isolation separately for process/memory protection (see below)

### Running nsjail without privileged: true

By default, nsjail requires `privileged: true` because it creates user namespaces, which need unmasked `/proc` access that only privileged containers provide. However, you can run nsjail without `privileged: true` by disabling user namespaces.

**Required environment variables:**

- `DISABLE_NSJAIL=false` — enable nsjail
- `DISABLE_NUSER=true` — disable user namespace creation

**Required capabilities (minimal set):**

```yaml
securityContext:
  allowPrivilegeEscalation: true
  capabilities:
    drop: [ALL]
    add:
      - SYS_ADMIN       # nsjail: PID/mount namespaces and /proc mount
      - SYS_RESOURCE    # worker: oom_score_adj to protect worker from OOM killer
      - SETPCAP         # nsjail: prctl(PR_SET_SECUREBITS) for capability dropping
  seccompProfile:
    type: Unconfined
  appArmorProfile:
    type: Unconfined
```

Using `capabilities: add: [ALL]` also works if your security policy allows it.

**Security tradeoff:**

The only isolation you lose is UID remapping — with `privileged: true`, nsjail maps root outside to UID 1000 inside the jail, whereas with `DISABLE_NUSER=true` the jailed process runs as root (UID 0). However, this is a better tradeoff overall: `privileged: true` removes the container boundary entirely (full host access, all devices, unmasked `/proc`, no seccomp, no apparmor), so a jail escape gives full host access. With `DISABLE_NUSER=true` and limited capabilities, a jail escape only reaches the container, which remains sandboxed. You're trading an inner defense layer (UID remapping) for a much stronger outer one (container boundary).

## Isolation comparison

| Feature                | NSJAIL      | PID Namespace | None       |
| ---------------------- | ----------- | ------------- | ---------- |
| Filesystem isolation   | ✅ Full     | ❌ No         | ❌ No      |
| Network isolation      | ⚠️ Optional | ❌ No         | ❌ No      |
| Memory protection      | ✅ Yes      | ✅ Yes        | ❌ No      |
| Process visibility     | ✅ Hidden   | ✅ Hidden     | ❌ Visible |
| Environment protection | ✅ Yes      | ✅ Yes        | ❌ No      |
| Resource limits        | ✅ Yes      | ❌ No         | ❌ No      |

### Isolation hierarchy

When multiple isolation methods are available, Windmill uses them in this priority order:

1. **NSJAIL** (if nsjail binary available and `DISABLE_NSJAIL=false`) - Provides comprehensive isolation
2. **PID Namespace** (if `ENABLE_UNSHARE_PID=true` and unshare available) - Provides process isolation
3. **None** (if neither enabled or available) - No isolation

**Important**: NSJAIL and PID namespace isolation are **not used simultaneously**. If NSJAIL is available and enabled, it takes precedence and PID namespace isolation is not used.

## Force sandboxing

### Instance-level setting (`job_isolation`)

The `job_isolation` instance setting can be set to `nsjail_sandboxing` to enable NSJAIL sandboxing for all jobs across the instance, regardless of the per-worker `DISABLE_NSJAIL` environment variable. Sandboxing is enabled when **either** `job_isolation` is set to `nsjail_sandboxing` **or** `DISABLE_NSJAIL=false` — neither overrides the other.

Configure it from [Instance settings](../18_instance_settings/index.mdx) or via the [Infrastructure as code](../../core_concepts/57_infrastructure_as_code/index.mdx) YAML configuration.

When `job_isolation` is set to `nsjail_sandboxing` but the NSJAIL binary is not available on a worker, all jobs on that worker will fail until NSJAIL is installed or the setting is changed. NSJAIL availability is always probed at worker startup regardless of `DISABLE_NSJAIL`.

All language executors (Python, TypeScript/Bun, Deno, Go, Bash, Rust, C#, Java, Ansible, PHP, Ruby) respect this setting through a centralized `is_sandboxing_enabled()` check.

### Per-script sandbox annotation {#sandbox-annotation}

You can enable nsjail sandboxing on a per-[script](../../script_editor/index.mdx) basis using the `sandbox` annotation. This works for Python, TypeScript (Bun and Deno), and Bash scripts. When present, the script runs inside an nsjail sandbox even if sandboxing is not enabled globally.

The script will fail with a clear error if nsjail is not available on the worker.

```bash
#sandbox

echo "This script runs inside an NSJAIL sandbox"
```

```python
# sandbox

def main():
    return "sandboxed Python"
```

```ts
// sandbox

}
```

The annotation is logged as "sandbox mode (nsjail)" in job execution logs.

The `sandbox` annotation can be combined with [volume annotations](../../core_concepts/67_volumes/index.mdx) to run scripts with persistent file storage inside a sandbox.

## Recommended configurations

### PID namespace isolation (recommended for production)

To enable PID namespace isolation, add to your docker-compose.yml:

```yaml
windmill_worker:
  privileged: true
  environment:
    - ENABLE_UNSHARE_PID=true
```

**Why enable this**: Protects worker credentials from being accessed by jobs through process memory and environment variables. This is a critical security feature for production deployments.

### NSJAIL sandboxing (maximum security)

```yaml
environment:
  - DISABLE_NSJAIL=false
```

Provides comprehensive isolation including filesystem, network, and resource limits.

## Security best practices

1. **Enable isolation before production** - Isolation is **disabled by default**. You should manually enable either NSJAIL or PID isolation before deploying to production
2. **Use PID isolation at minimum** - At a minimum, enable PID namespace isolation with `ENABLE_UNSHARE_PID=true` and `privileged: true`
3. **Consider NSJAIL for untrusted code** - Use NSJAIL if you run code from untrusted sources or need filesystem isolation
4. **Consider agent workers** - For sensitive environments, use [agent workers](../../core_concepts/28_agent_workers/index.mdx) that don't have direct database access and communicate only via the API
5. **Test your isolation** - Verify isolation is working by checking worker logs for isolation status messages
6. **Use [worker groups](../../core_concepts/9_worker_groups/index.mdx)** - Separate untrusted workloads onto [dedicated workers](../../core_concepts/25_dedicated_workers/index.mdx) with stricter isolation, or run workers on separate clusters with different network/resource access
7. **Keep systems updated** - Ensure kernel and util-linux are up to date
8. **Review user permissions** - Limit who can create scripts in your Windmill instance using [roles and permissions](../../core_concepts/16_roles_and_permissions/index.mdx)

## Troubleshooting

### Worker fails to start with "unshare: Operation not permitted"

**Cause**: User namespaces are disabled in the kernel.

**Solution**:

```bash
# Check if user namespaces are enabled
sysctl kernel.unprivileged_userns_clone

# Enable if disabled (requires root)
sysctl -w kernel.unprivileged_userns_clone=1

# Make permanent (add to /etc/sysctl.conf)
echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf
```

### Docker container: "unshare: unshare failed: Operation not permitted"

**Cause**: The `--mount-proc` flag requires privileged mode in Docker.

**Solution**: Enable privileged mode in docker-compose.yml:

```yaml
windmill_worker:
  privileged: true
  environment:
    - ENABLE_UNSHARE_PID=true
```

**Alternative**: If you cannot use privileged mode, remove `--mount-proc` from the flags (less secure):

```yaml
windmill_worker:
  environment:
    - ENABLE_UNSHARE_PID=true
    - UNSHARE_ISOLATION_FLAGS=--user --map-root-user --pid --fork
```

Note: Without `--mount-proc`, jobs may still be able to see host processes in /proc.

### Kubernetes/Helm: unshare fails

**Cause**: The `--mount-proc` flag requires privileged mode.

**Solution**: Enable privileged mode in pod spec:

```yaml
securityContext:
  privileged: true
```

And set the environment variable:

```yaml
env:
  - name: ENABLE_UNSHARE_PID
    value: 'true'
```

For Helm deployments, set the appropriate values to enable privileged mode and the environment variable.

### Windows workers fail to start

**Cause**: PID isolation is Linux-only.

**Solution**: Set `ENABLE_UNSHARE_PID=false` for Windows workers.

### Jobs fail with "Cannot allocate memory"

**Cause**: Insufficient resources for isolation overhead.

**Solution**:

- Increase worker memory limits
- Reduce number of concurrent jobs
- Use `DISABLE_NSJAIL=true` and rely on PID isolation only

### AWS EKS with Bottlerocket AMI

**Cause**: Bottlerocket AMI sets `user.max_user_namespaces=0` by default, which prevents user namespace creation required for PID isolation and NSJAIL.

**Solutions**:

1. **Switch to a different AMI** (recommended): Use Amazon Linux 2023 or Amazon Linux 2 instead of Bottlerocket for your EKS node groups.

2. **Disable PID isolation**: Set `disableUnsharePid: true` in Helm values (global or per-worker-group). Note: This reduces security isolation.

3. **Configure Bottlerocket kernel parameters**: Use a custom launch template with user data to increase `max_user_namespaces`:
   ```toml
   [settings.kernel.sysctl]
   "user.max_user_namespaces" = "65536"
   ```
   See [AWS EKS launch template documentation](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html) for details.

## Related documentation

- [Worker Groups](../../core_concepts/9_worker_groups/index.mdx) - Logical worker separation
- [Agent Workers](../../core_concepts/28_agent_workers/index.mdx) - Workers without direct database access
- [Dedicated Workers](../../core_concepts/25_dedicated_workers/index.mdx) - Workers assigned to specific scripts
- [Self-Hosting](../1_self_host/index.mdx) - Deployment configuration
- [Environment Variables](../../core_concepts/47_environment_variables/index.mdx) - All worker configuration options
- [Docker](../7_docker/index.mdx) - Container-based execution
- [Instance Settings](../18_instance_settings/index.mdx) - Instance-level configuration
- [Roles and Permissions](../../core_concepts/16_roles_and_permissions/index.mdx) - Access control
- [Audit Logs](../../core_concepts/14_audit_logs/index.mdx) - Activity tracking and compliance
- [Workspace Secret Encryption](../../core_concepts/30_workspace_secret_encryption/index.mdx) - Secret encryption at rest