Skip to main content

OpenID Connect (OIDC)

Use Windmill's OIDC provider to authenticate from scripts to cloud providers and other APIs.

OIDC is an EE feature only.

Windmill OpenID Connect (OIDC) allows your scripts to authenticate to other APIs (such as AWS, GCP, or your own API) without having to store sensitive, long-lived credentials. Your Windmill scripts can generate short-lived ID tokens which are passed to a target API to authenticate and "prove" their identity. ID tokens contain subject and claim information specific to the script, flow or worker, allowing for fine-grained access control (e.g. allowing specific scripts to access AWS buckets). This page documents how to generate Windmill tokens and the token format.

Generating OIDC/JWT tokens​

OIDC tokens are generated at runtime and are scoped to the script that generated them. Tokens can be generated with Windmill's SDKs or from the REST API directly. Your token must be associated with an audience which identifies the intended recipient of the token. The audience is provided as a parameter when generating the token. If the audience is incorrect, the consumer will reject the token. (For AWS, this audience is sts.amazonaws.com. For your own APIs, you can specify an audience such as auth.yourcompany.com.) If you are using a TypeScript or Python scripts, you can use the Windmill SDK to generate tokens. For other like REST or shell, you should use the REST api directly:

curl -s -X POST -H "Authorization: Bearer $WM_TOKEN" "$BASE_INTERNAL_URL/api/w/$WM_WORKSPACE/oidc/token/MY_AUDIENCE"

The default expiry time is 48 hours. Set another one in seconds with the expires_in query parameter, change the default with the OIDC_TOKEN_EXPIRY_SECS environment variable, and cap what a job can request with OIDC_TOKEN_MAX_EXPIRY_SECS. A token is only issued while the job is running and not being canceled: its WM_TOKEN cannot mint one afterwards.

Generate the token​

Tokens can be generated at runtime with Windmill's SDKs. For example, to generate an OIDC token with audience sts.amazonaws.com to assume an AWS role:

import { STSClient } from 'npm:@aws-sdk/client-sts';
import { AssumeRoleWithWebIdentityCommand } from 'npm:@aws-sdk/client-sts';
import * as wmill from 'npm:windmill-client';

export async function main() {
	const token = await wmill.getIdToken('sts.amazonaws.com');

	const command = new AssumeRoleWithWebIdentityCommand({
		RoleArn: 'arn:aws:iam::000000000000:role/my_aws_role',
		WebIdentityToken: token,
		RoleSessionName: 'my_session'
	});

	const client = new STSClient({ region: 'us-east-1' });
	console.log(await client.send(command));
}

Token payload reference​

OIDC tokens generated by Windmill's OIDC provider are signed JSON web tokens (JWTs). The ID token is encoded by using RS256 and signed with a dedicated private key. The expiry time for the token is set for 48 hours. The token includes standard claims:

  • aud: Intended audience for the token. This field is set by the user when fetching the ID token and is the only customizable field.
  • iss: The issuer of the token, your base_url set in in your instance settings
  • sub: The subject claim is validated by your cloud provider. The subject is formatted as {email}::{script_path}::{flow_path}::{workspace}. For instance level, the sub claim is "instance". For SQS triggers and workspace level object storage, the sub claim is "<workspace_id>".- exp: The expiration time of the token.
  • iat: The time the token was issued.
  • alg: The algorithm used to sign the token.
  • kid: The key ID used to sign the token.

The token also includes custom claims provided by Windmill. To see the full list of claims supported by Windmill's OIDC provider, see the claims_supported entries at <base_url>/api/oidc/.well-known/openid-configuration. Note that not all cloud providers support all claims.

  • job_id: Job id
  • path: Job's path (the script path for a script or a virtual path for a step part of a flow)
  • flow_path: If job is part of a flow, contain the path of the flow
  • groups: the groups the requester is part of
  • username: the username the requester has in the given workspace
  • email: the email of the requester
  • workspace: workspace_id the job is executed in
  • fork_parent_workspace: when the job runs in a workspace fork, the workspace_id of the parent workspace. This lets OIDC consumers such as Vault write policies scoped to the parent workspace that also apply to all its forks. The claim is omitted for non-fork workspaces, and workspace-level tokens (workspace object storage, SQS triggers) do not include it.
  • end_user_email: the email of the end user when the job runs on behalf of a different user (e.g. via approval flows or operator impersonation)
  • job_kind: the kind of job that requested the token, e.g. script, preview, flow, flowscript (an inline flow step), appscript or aiagent
  • deployed, latest: whether the job runs deployed code, and its current version. See Trust only deployed code.
  • version: the script hash of a script job or the flow version id of a flow job, null for other kinds
  • digest: the SHA-256 of the job's code, null for kinds other than script, flow and flowscript. See Pin code with digests.
  • root_job_id, root_path, root_job_kind, root_version, root_digest: the same for the job at the top of the chain of parent jobs, e.g. the flow that ran this step, or the job itself when it has no parent. A flow restarted from a step is a new job, with a new root_job_id.
  • root_trigger_kind, root_trigger: how the root job was started (e.g. schedule, webhook, http) and by what, null when none was recorded, e.g. for a run started through the API. See what root_trigger holds.
  • tag, worker_group: the job's worker tag and the worker group of the worker running it. Windmill records them, the worker does not attest them: whoever can start a worker under a group name or deploy code with a tag sets them.
  • run_as_type: what email names: user, service_account, group or non_member (a superadmin outside the workspace, a built-in identity or a removed user)
  • job_token_scopes: the scopes of the restricted job token that requested the token, e.g. ["oidc:write"], absent when it is unrestricted. It reflects the token the job holds, not the script or flow setting, so a policy requiring it rejects a job whose token was minted without the restriction.
  • jti: a unique id per token, to refuse a token presented twice

The following example token was requested with the audience sts.amazonaws.com from the script u/admin/ambitious_script in the foobar workspace:

eyJhbGciOiJSUzI1NiIsImtpZCI6IndpbmRtaWxsIn0.eyJpc3MiOiJodHRwczovL215Y29tcGFueS5jb20iLCJhdWQiOlsic3RzLmFtYXpvbi5jb20iXSwiZXhwIjoxNzA1NjYwOTU1LCJpYXQiOjE3MDU0ODgxNTUsInN1YiI6ImFkbWluQHdpbmRtaWxsLmRldjo6dS9hZG1pbi9hbWJpdGlvdXNfc2NyaXB0Ojpub19mbG93Ojpmb29iYXIiLCJlbWFpbCI6ImFkbWluQHdpbmRtaWxsLmRldiIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJqb2JfaWQiOiIwMThkMTcwNC0wMGMxLTA1YmItNDE4My1lNTZhOGVkZWI4MjMiLCJwYXRoIjoidS9hZG1pbi9hbWJpdGlvdXNfc2NyaXB0IiwiZmxvd19wYXRoIjpudWxsLCJncm91cHMiOlsiYWxsIl0sInVzZXJuYW1lIjoiYWRtaW4iLCJlbWFpbCI6ImFkbWluQHdpbmRtaWxsLmRldiIsIndvcmtzcGFjZSI6ImZvb2JhciJ9.FLOAMsT7wTuCMM7hdZBGU5uMna3v-3FVXbouQ5CXJ61Vp7Ew82w9K2IFdb5rE-x-YDra-MdT8F3v1CNcvB1vQ4jSsgPLFm_fDu0aA2fI_B69Kno2KBpd5afXVhMUwSF4zyRiruZdIS1hL8PjeNsMOIfy-c_FnF_BchbarXswpo6_UjjsJg353l2C5wOqny8OT4fNGn4yyvTM5PDPaisxOrBuSQDK0GhuB1zSZ4OAGChnd5-KwopqlYmm0NlNiUUTYa5tORtNfbvDFM-hrQ1GOKCKn18nmSV5QKjzn8Au8lP8qNmoZPXdXhxr4yPx8mxWHge7ckABZfB3xGpufG436g

{
  "iss": "https://mycompany.com/api/oidc/",
  "aud": [
    "sts.amazonaws.com"
  ],
  "exp": 1705660955,
  "iat": 1705488155,
  "sub": "[email protected]::u/admin/ambitious_script::no_flow::foobar",
  "email": "[email protected]",
  "email_verified": true,
  "job_id": "018d1704-00c1-05bb-4183-e56a8edeb823",
  "path": "u/admin/ambitious_script",
  "flow_path": null,
  "groups": [
    "all"
  ],
  "username": "admin",
  "workspace": "foobar"
}

Trust only deployed code​

deployed is true only when the job and every job above it in the chain run code at a path taken from a deployed version, or a published Hub script. It is false for anything run from the script, flow or app editor, and for a deployed script called from a flow preview. latest is true only when the token is deployed and every script and flow in the chain runs the version its path currently resolves to. It is always false under an app.

deployed alone accepts every past version, which anyone who can run a script or flow can still run. Require latest so that redeploying over a bad version revokes it, or pin version and root_version. For example, to issue a credential only to the steps of one deployed flow, on a provider that checks custom claims (e.g. Vault's bound_claims), bind:

  • workspace = foobar
  • path matching f/infra_ro/* (in Vault, with "bound_claims_type": "glob" on the role)
  • root_path = f/infra_ro/agent
  • deployed = true
  • latest = true

Both are computed each time a token is requested, and turn false, even during a run, when:

  • latest: a script or flow of the chain is redeployed
  • deployed: the job's script version is deleted, its script or flow is permanently deleted, or its user folder is renamed
  • deployed: for the inline steps of a flow, the instance runs with DISABLE_FLOW_SCRIPT, or the flow was last deployed before flow nodes existed (redeploy it)

A preview declares its own path, so its token carries the path and sub of the item deployed there. Windmill therefore only issues a token to a job that is not deployed when its user has write access to the script, flow or app at every path the preview declares (its own, and any path set on a flow step that points outside the flow): otherwise the request fails with a 403. That includes deployed code that runs as another user when called from such a preview, and, unless the user is an admin, a job whose chain exceeds 100 parent jobs or leaves the workspace. A preview with no path, or under the user's own u/<username>/ folder, always gets one.

Providers that only match sub (e.g. AWS) cannot see deployed, latest or version, and sub is the same for a preview and for every version: such a role trusts everyone who can write or run the paths it names, so restrict both.

Pin code with digests​

version identifies code inside one instance only. digest and root_digest are the lowercase hex SHA-256 of the RFC 8785 canonical JSON of the item, with null members left out, so a policy can accept only code reviewed in git. Compute them with wmill digest from the root of a wmill sync checkout:

wmill digest f/infra/agent__flow f/infra/tool.ts
# 762edbbcb8c0e84744206f8efa707f10945540e3a00cb454f0dbd31520b18ddd  f/infra/agent
# abf4646174c8d6984ea9f6d9c6746b61c3d95cd0cb91a4ca89b884b0f61600ae  f/infra/agent/a
# fcf85b60c3d9aef073e6911f417fdd19bedeb6d9e0f36b2589df07900997f29d  f/infra/tool
  • a script: {"content", "lock", "codebase", "modules"}, where modules is {"<relative path>": {"content", "lock"}} for a multi-file script
  • an inline flow step: {"content", "lock"}, listed as <flow path>/<step id>, the path of its flowscript job
  • a flow: its value, as flow.yaml holds it once the !inline files are read back in

A digest covers code and lockfiles, not the language, the settings and metadata (tag, schema) or the scripts it imports. Compare against a checkout pulled with wmill sync pull after the deployment's dependency jobs completed, not the files you pushed: those jobs write the lockfiles and fill in the default settings of a flow's loops and branches.

What root_trigger holds​

root_trigger depends on root_trigger_kind:

  • schedule: the schedule's path, null for its success and recovery handlers
  • http, websocket, kafka, nats, mqtt, amqp, sqs, postgres, gcp, azure: the trigger's path, not its route, topic or queue
  • email: the email trigger's path, null for an email sent to a script's or flow's own address
  • app: the app's path
  • asset: the path of the script that produced the asset
  • nextcloud, google, github: an external id supplied by whoever calls the run endpoint, as is the kind itself: do not trust them
  • webhook: null

A job started by an HTTP route runs as the identity configured on the route: sub, email, username and groups never describe the caller.

Settings discovery​

Windmill's OIDC settings are available at the following discovery URL:

<base_url>/api/oidc/.well-known/openid-configuration

Jwks are available directly at:

<base_url>/api/oidc/jwks

Signing key​

By default, Windmill signs OIDC tokens with a 2048-bit RSA key that it generates and stores in the database, with the kid windmill. The same key signs the tokens Windmill presents to Vault, object storage, SQS and the AI proxy.

To keep the key out of the database, set OIDC_SIGNING_KEY_FILE on every server and worker to the path of the same PEM RSA private key (PKCS#1 or PKCS#8, 2048 to 4096 bits):

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out oidc-signing-key.pem

A file key's kid is its RFC 7638 thumbprint, so it changes with the key. Verifiers that fetch <base_url>/api/oidc/jwks follow on their own, while one configured with a static copy of the key has to be updated at every change.

Switching from the database key to a file invalidates the tokens signed with the database key at once: it cannot stay published as kid: windmill. Then delete it, so that a copy of the database or a backup can no longer sign tokens:

DELETE FROM global_settings WHERE name = 'rsa_keys';

Rotate the key​

The key file is re-read at every signature, and the JWKS also serves every *.pem file (private or public key) of the OIDC_SIGNING_KEY_PUBLISH_DIR directory, listed on every request. With the current public key in that directory, a rotation needs no restart:

  1. Add the new public key to the directory, and wait until every server and worker sees it (a mounted Kubernetes secret can take a minute or more) and verifiers that cache the JWKS have refetched it.
  2. Write the new private key to the signing key file, atomically (write a temporary file without the .pem extension, then rename it).
  3. Once the tokens signed with the old key have expired, delete its file from the directory. OIDC_TOKEN_MAX_EXPIRY_SECS bounds that wait.

The key can also stay in an HSM or KMS that Windmill never reads, through an external signer.

Use OIDC with AWS​

Create an identity provider with AWS for OIDC​

  1. Search for the Identity Providers tab in the console search or IAM.
  2. Add provider
  3. Select OpenID Connect
  4. Provider Url: <base_url>/api/oidc/, audience: sts.amazonaws.com

Create a role​

  1. IAM -> Roles -> Create Role
  2. Web Identity
  3. Pick provider created above, audience: sts.amazonaws.com
  4. Pick the permission policy to attach to this role. You can create as many roles as needed so it's best to be specific.
  5. fill the Trust policy:
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Principal": {
              "Federated": "arn:aws:iam::976079455550:oidc-provider/<base_url>/api/oidc/"
          },
          "Condition": {
              "StringEquals": {
                  "<base_url>/api/oidc/:aud": "sts.amazonaws.com",
                  "<base_url>/api/oidc/:email": "[email protected]",
                  ...
              }
          }
      }
  ]
}

Note that AWS only supports conditions on the aud, sub, and email claims. You can use StringLike on the sub claim to limit by job, flow, or workspace:

"StringLike": {
  "<base_url>/api/oidc/:sub": "*::<workspace>",
  "<base_url>/api/oidc/:sub": "*::<script_path>::*::*",
  "<base_url>/api/oidc/:sub": "*::*::<flow_path>::*"
}

For instance level AWS OIDC for instance object storage, the sub claim is "instance". For workspace level object storage, and SQS oidc triggersthe sub claim is "<workspace_id>".

Get AWS credentials​

import { STSClient } from 'npm:@aws-sdk/client-sts';
import { AssumeRoleWithWebIdentityCommand } from 'npm:@aws-sdk/client-sts';
import * as wmill from 'npm:windmill-client';

export async function main() {
	const token = await wmill.getIdToken('sts.amazonaws.com');

	const command = new AssumeRoleWithWebIdentityCommand({
		RoleArn: 'arn:aws:iam::000000000000:role/my_aws_role',
		WebIdentityToken: token,
		RoleSessionName: 'my_session'
	});

	const client = new STSClient({ region: 'us-east-1' });
	console.log(await client.send(command));
}

You can now get the ephemeral access key, secret key, and session token from it to use with any AWS API.

credentials = credentials["Credentials"]

aws_access_key_id=credentials["AccessKeyId"]
aws_secret_access_key=credentials["SecretAccessKey"]
aws_session_token=credentials["SessionToken"]

Use OIDC with Hashicorp Vault​

vault auth enable jwt

vault write auth/jwt/config \
  bound_issuer="<base_url>/api/oidc/" \
  oidc_discovery_url="<base_url>/api/oidc/"

vault write auth/jwt/role/myproject-production -<<EOF
{
  "role_type": "jwt",
  "bound_audiences": ["MY_AUDIENCE"],
  "bound_claims": { "email": "[email protected]"},
  "user_claim": "sub",
  "policies": ["myproject-production"],
  "ttl": "10m"
}
EOF

 vault policy write myproject-production - <<EOF
# Read-only permission on 'secret/data/production/*' path

path "secret/data/production/*" {
  capabilities = [ "read" ]
}
EOF

Test it​

Write a secret at production/foo:

vault kv put -mount=secret production/foo foo=world
export VAULT_ADDR='http://127.0.0.1:8200'

export VAULT_JWT=$(curl -s -X POST -H "Authorization: Bearer $WM_TOKEN" "$BASE_INTERNAL_URL/api/w/$WM_WORKSPACE/oidc/token/MY_AUDIENCE")
export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=myproject-production jwt=$VAULT_JWT)

vault kv get -mount=secret production/foo

External signer​

Most instances don't need this: it is for keeping the OIDC signing key in an HSM or KMS that Windmill can ask for a signature but never read.

Set up an external signer

Set OIDC_SIGNER_URL on the servers. They then sign through a small HTTP service (an adapter in front of your KMS) over mTLS, and there is no fallback to a local key: with the signer down, token requests fail with 503 while the JWKS keeps being served.

VariableDefaultDescription
OIDC_SIGNER_URLBase URL of the signer (https).
OIDC_SIGNER_CLIENT_CERT_FILE, OIDC_SIGNER_CLIENT_KEY_FILErequiredmTLS client certificate and key (PEM), re-read when they change.
OIDC_SIGNER_CA_FILEsystem rootsCA the signer's certificate must chain to. Unset, the system roots are trusted and a warning is logged at startup.
OIDC_SIGNER_MAX_RPS5Signatures per second, per server: size it as the key's cap divided by the maximum number of servers. Above it, a job's token request gets 429 with Retry-After. Windmill's own signatures (object storage, Vault) go before job tokens; they are cached, so few.
OIDC_SIGNER_TIMEOUT_MS, OIDC_SIGNER_DEADLINE_MS2000, 5000Per attempt, and for a whole signature with its retries.
OIDC_SIGNER_KEYS_REFRESH_SECS60How often the key listing is refreshed.

Only servers sign. A job that needs an OIDC token must run on an agent worker, and a normal worker refuses to sign, which includes Vault JWT authentication and object storage with AWS OIDC used from that worker. OIDC_SIGNER_URL cannot be combined with OIDC_SIGNING_KEY_FILE, and once the instance signs externally a server without OIDC_SIGNER_URL refuses to start.

Servers record in the database that the instance signs externally, so that no process falls back to a local key. To go back to a local key, stop every server that has the variables (they write the record back), delete it, then start them without the variables:

DELETE FROM global_settings WHERE name = 'oidc_external_signer';

When moving an existing instance, the database key (kid: windmill) stays in the JWKS until its row is deleted. Delete it once the tokens it signed have expired, as described above.

The signer implements two endpoints:

  • GET /v1/keys returns {"keys": [{"key_ref", "state", "alg", "public_key_pem" | "public_key_jwk"}]}. state is pending (published, does not sign), active (signs, at most one) or retired (published, no longer signs). alg is RS256 (RSA 2048 to 4096 bits) or ES256 (P-256). At most 16 keys.
  • POST /v1/sign takes {"key_ref", "alg", "digest", "request_id"}, where digest is the base64url SHA-256 of the JWS signing input, and returns {"signature"} in base64url: PKCS#1 v1.5 for RS256, raw r || s for ES256. Errors are {"error", "message"}: 409 with key_not_active or unknown_key, 429 with rate_limited and a Retry-After header in integer seconds.

Each key's kid is its RFC 7638 thumbprint. Windmill verifies every signature against the listed key before returning a token, and logs the request_id with the token's jti, or with an error class when no token was issued. request_id identifies the mint, not the signature: a retry reuses it, possibly with a different digest, so the signer must log every signature and never deduplicate on it.

Rotation is driven from the signer:

  1. List the new key as pending. Windmill publishes it in the JWKS within a refresh period. Wait until verifiers have it.
  2. Mark it active and the previous one retired.
  3. Remove the retired key from the listing once the tokens it signed have expired.

The last accepted listing is stored in the database: a server that starts while the signer is down serves those keys in the JWKS, and signs nothing until the signer answers. Since anyone who can write to the database can change that stored listing, a verifier should pin the thumbprints it accepts rather than trust every key in the JWKS.

Superadmins can see the keys Windmill serves, with their kid and state, in the instance settings under OIDC signing. That page is only listed on instances that use an external signer.

The Windmill binary checks an adapter against the protocol, with the same variables as a server, and includes a file-backed signer to try it out (not for production keys):

windmill oidc-signer conformance --burst 30
windmill oidc-signer test-signer --keys keys.json --cert server.pem --key server.key --client-ca ca.pem