Critical alerts

Get a notification everytime a job is re-run after a crash.

This feature is available in the Enterprise Edition.

If the node it which it runs halt suddenly (such as a power loss), then the job will be restarted automatically. Windmill itself doesn't crash and other softer interruptions like a pod termination involve a grace period (300s) to let the job finish.

Critical alerts are generated under the following conditions:

• Job is re-run after a crash.
• License key does not renew.
• Workspace error handler fails.
• Number of running workers in a group falls below a specified threshold (has to be configured in the worker group config).
• Number of jobs waiting in queue is above a threshold for more than a specified amount of time.
• API token is about to expire (within 7 days) or has expired (requires "Alert on token expiry" to be enabled in instance settings).

Token expiration notifications

Windmill monitors API tokens and sends notifications when they are about to expire or have expired. Token owners receive an email notification 7 days before expiration, and again when the token is deleted after expiry.

• Email notifications are always sent when SMTP is configured, regardless of the "Alert on token expiry" setting.
• Critical alerts are only sent when the "Alert on token expiry" instance setting is enabled.
• Session and ephemeral tokens are excluded from expiration notifications.

The tokens table in user settings also shows color-coded expiration badges for tokens expiring within 30 days, and a warning banner for tokens expiring within 7 days.

Critical alert channels

You just need to configure SMTP and setup a critical alert channel (aka email address) in the instance settings and/or connect your instance to Slack and Microsoft Teams and fill in a channel name.

You can also set an alert to receive notification when the number of running workers in a group falls below a given number. It's available in the worker group config.

Workers Alerts

Set an alert to receive notification when the number of running workers in a group falls below a given number.

Critical alerts in UI

Windmill itself sends critical alerts notifications through the UI.

You can disable this in the instance settings.

Visibility

Instance wide Critical Alerts are only visible to users with the superadmin or devops roles. For workspace specifc alerts, users need to have admin privilege over that workspace.