Skip to main content

Resources and resource types

Resources are rich objects in JSON that allow to store configuration and credentials.

In Windmill, Resources represent connections to third-party systems. Resources are a good way to define a connection to a frequently used third-party system such as a database. Think of Resources as a structured way to store configuration and credentials, and access them from scripts.

Each Resource has a Resource Type (RT for short) - for example MySQL, MongoDB, OpenAI, etc. - that defines the schema that the resource of this type needs to implement. Schemas implement the JSON Schema specification.

Recap Resources and Types

While resources store configuration and credentials for connections, Assets track data flows and visualize datasets. Assets automatically detect when you reference S3 objects (s3://storage/path/to/file.csv) or resources (res://path/to/resource) directly in your code, creating a visual representation of your data pipeline. This helps you understand how data moves through your workflows and identify dependencies between different data sources.

tip

Check our list of integrations (or, pre-made resource types). If one is missing, this very page details how to create your own.

List of integrations
Windmill provides a framework to easily add integrations.
JSON Schema
JSON Schema is a declarative language that allows you to annotate and validate JSON documents.

Create a resource​

To create a resource using an existing type, go to the Resources page and click "Add resource".

Add a resource

Just like most objects in Windmill, Resources have a path that defines their permissions - see ownership path prefix.

Each Resource has a Resource Type, that defines what fields that resource contains. Select one from the list and check the schema to see what fields are present.

Resources commonly need to access secrets or re-use Variables, for example, passwords or API tokens. To insert a Variable into a Resource, use Insert variable (the $ sign button) and select a Variable. The name of a Variable will look like $var:<NAME_OF_VAR>. When resources are called from a Script, the Variable reference will be replaced by its value.

Resources can be assigned a description. It supports markdown.

Resource description

tip

It's a good practice to link a script template to Resources, so that users can easily get started with it. You can use markdown in the description field to add a link, for example:

[example script with this resource](/scripts/add?template=script/template/path)

Create a resource type​

Windmill comes preloaded with some common Resource types, such as databases, apps, SMTP, etc. You can see the full list on Windmill Hub. You can also add custom Resource types by clicking "Add a resource type" on the Resources page.

Create resource type

Use the "Add Property" button to add a field to the resource type. You can specify constraints for the field (a type, making it mandatory, specifying a default, etc). You can also view the schema by toggling the "As JSON" option:

Resource type schema view

The resources types created from the Admins workspace are shared across all workspaces.

Share resource type on Hub​

You can contribute to the Windmill Hub by sharing your Resource Type. To do so, add a Resource Type on the Resources section of the Hub.

You will be asked to fill Name, Integration (the corresponding service it interacts with) and Schema (the JSON Schema of the Resource Type).

Verified Resource types on the Hub are directly added to the list of available Resource types on each new Windmill instance synced with the Hub.

Share resource type

Sync resource types with WindmillHub​

When creating a self-hosted instance, you are offered to set a schedule to regularly sync all resource types from WindmillHub. This will ensure that all the approved resource types are available in your instance. On Windmill cloud, it is done regularly by the Windmill team. Ask us if you need a specific resource type from Hub to be added.

The Bun script executed from the admin workspace is:

import * as wmill from '[email protected]';

export async function main() {
	await wmill.hubPull({
		workspace: 'admins',
		token: process.env['WM_TOKEN'],
		baseUrl: globalThis.process.env['BASE_URL']
	});
}

You can find this script on WindmillHub.

This script is probably already on your Admins workspace, as it was suggested during Windmill self-hosting setup. Having this script run on your Admins workspace will sync resources accross all workspaces of your instance.

To avoid running it manually, we recommend scheduling this script regularly.

States​

States are used by scripts to keep data persistent between runs of the same script by the same trigger (schedule or user).

In Windmill, states are considered as resources (rich objects in JSON), but they are excluded from the Workspace tab for clarity. They are displayed on the Resources menu, under a dedicated tab.

A state is an object stored as a resource of the resource type state which is meant to persist across distinct executions of the same script.

import requests
from wmill import set_state, get_state

def main():
	# Get temperature from last execution
    last_temperature = get_state()

    # Fetch the temperature in Paris from wttr.in
    response = requests.get("http://wttr.in/Paris?format=%t")

    new_temperature = response.text.strip("°F")

	# Set current temperature to state
    set_state(new_temperature)

    # Compare last_temperature and new_temperature
    if last_temperature < new_temperature:
        return "The temperature has increased."
    elif last_temperature > new_temperature:
        return "The temperature has decreased."
    else:
        return "The temperature has remained the same."

States are what enable Flows to watch for changes in most event watching scenarios (trigger scripts). The pattern is as follows:

  • Retrieve the last state or, if undefined, assume it is the first execution.
  • Retrieve the current state in the external system you are watching, e.g. the list of users having starred your repo or the maximum ID of posts on Hacker News.
  • Calculate the difference between the current state and the last internal state. This difference is what you will want to act upon.
  • Set the new state as the current state so that you do not process the elements you just processed.
  • Return the differences calculated previously so that you can process them in the next steps. You will likely want to forloop over the items and trigger one Flow per item. This is exactly the pattern used when your Flow is in the mode of "Watching changes regularly".

The convenience functions do this are:

TypeScript

  • getState() which retrieves an object of any type (internally a simple Resource) at a path determined by getStatePath, which is unique to the user currently executing the Script, the Flow in which it is currently getting called in - if any - and the path of the Script.
  • setState(value: any) which sets the new state.

Please note it requires importing the wmill client library from Deno/Bun.


Python

  • get_state() which retrieves an object of any type (internally a simple Resource) at a path determined by get_state_path, which is unique to the user currently executing the Script, the Flow in which it is currently getting called in - if any - and the path of the Script.
  • set_state(value: Any) which sets the new state.

Please note it requires importing the wmill client library from Python.


Persistent storage & databases
Ensure that your data is safely stored and easily accessible whenever required.

States path​

States are stored in a path that is unique to the script, script within a flow or schedule. The state path is defined as:

    let state_path = {
        let trigger = if schedule_path.is_some() {
            username.to_string()
        } else {
            "user".to_string()
        };

        if let Some(flow_path) = flow_path.clone() {
            format!(
                "{flow_path}/{}_{trigger}",
                step_id.clone().unwrap_or_else(|| "nostep".to_string())
            )
        } else if let Some(script_path) = path.clone() {
            let script_path = if script_path.ends_with("/") {
                "noname".to_string()
            } else {
                script_path
            };
            format!("{script_path}/{trigger}")
        } else {
            format!("u/{username}/tmp_state")
        }
    };

For example, if a script using states is used within a flow that is scheduled, the state will take a new path as: folder/flow_path/script_path/schedule_path, or more concretely: u/username/u/username/flow_name/u/username/script_name/u/username/schedule_name.

Custom flow states​

Custom flow states are a way to store data across steps in a flow. You can set and retrieve a value given a key from any step of flow and it will be available from within the flow globally. That state will be stored in the flow state itself and thus has the same lifetime as the flow job itself.

It's a powerful escape hatch when passing data as output/input is not feasible and using getResource/setResource has the issue of cluttering the workspace and inconvenient UX.

import * as wmill from '[email protected]';

export async function main(x: string) {
	await wmill.setFlowUserState('FOO', 42);
	return await wmill.getFlowUserState('FOO');
}

Using resources​

Resources can be used passed as script parameters or directly fetched within code.

Passing resources as parameters to scripts (preferred)​

Resources can be passed using the auto-generated UI.

Provided you have the right permissions and the resource type exists in the workspace, you can access resource types from scripts, flows and apps using the Windmill client or TypedDict in Python.

From the code editor's toolbar, click on the + Type button and pick the right resource type. For example, to access the u/user/my_postgresql resource of the posgtgresql Resource Type we would create a script:

type Postgresql = object;
// OR one can fully type it
type Postgresql = {
	host: string;
	port: number;
	user: string;
	dbname: string;
	sslmode: string;
	password: string;
	root_certificate_pem: string;
};

export async function main(postgres: Postgresql) {
	// Use Resource...
}

And then select the Resource in the arguments section on the right:

Select resource

tip

You can also edit the Resource or even create a new one right from the Code editor.

All details on the Add resources and variables to code editor page:

Add resources and variables to code editor
You can directly access Variables and Resources from the Code editor.

Fetching them from within a script by using the wmill client in the respective language​

By clicking on + Resource, you'll get to pick a resource from your workspace and be able to fetch it from within the script.

TypeScript:

wmill.getResource('u/user/foo');

Python:

wmill.get_resource("u/user/foo")

Go:

wmill.GetResource("u/user/foo")

Bash:

curl -s -H "Authorization: Bearer $WM_TOKEN" \
  "$BASE_INTERNAL_URL/api/w/$WM_WORKSPACE/resources/get/u/user/foo" \
    | jq -r .value

PowerShell:

$Headers = @{
  "Authorization" = "Bearer $Env:WM_TOKEN"
}
Invoke-RestMethod -Headers $Headers -Uri "$Env:BASE_INTERNAL_URL/api/w/$Env:WM_WORKSPACE/resources/get/u/user/foo"

Nu:

get_resource u/user/foo

Fetch resource

Resources in Apps​

Apps are executed on behalf of publishers and by default cannot access viewer's resources.

If the resource passed here as a reference does not come from a static Resource select component (which will be whitelisted by the auto-generated policy), you need to toggle "Resources from users allowed".

The toggle "Static resource select only / Resources from users allowed" can be found for each runnable input when the source is an eval.

Static resource select only / Resources from users allowed

Runnable editor
Learn how to create and configure Apps runnables.

Plain text file resources​

In certain scenarios it is useful to store data as a a text file, with a format such as .json, .cfg, etc. Windmill can store plain text files as a resource, which can then be used as arguments for scripts, or have special support in certain languages (e.g. Ansible).

Creating a plain text resource​

You first need to create the resource type. You will find a toggle to indicate this file is a text file, turn it on. You can then enter the file extension that will define the file format. In this example we will choose ini and create a type to represent Ansible Inventories.

Create a Text File Resource Type

Under the hood, this resource type is just like others, but the schema only has a content field where the file contents are stored. Additionally, the format that you input will be used to diplay the contents on the frontend and also when pulling with the CLI and create a type to represent Ansible Inventories.

Now you can create the resource by adding a resource and searching your newly created resource type:

Create a Text File Resource

As you will notice, the format specified will be used for syntax highlighting when editing and displaying this file. When pulling this resource on the CLI, you will find two files: <resource_name>.resource.file.ini and <resource_name>.resource.yaml.

Using the resource​

This resource only has a content field, so any language can use it and access the content like for any other resource/object. In certain languages (currenttly only Ansible), it can be preferable to have the file available on the file system instead of inlining its contents. You can ask Windmill to create these files before the execution of the script.

Mocked API files​

Simulate API interactions locally by using a JSON file to store and retrieve variables and resources.

Mocked API files
Simulate API interactions locally by using a JSON file to store and retrieve variables and resources.

Database manager​

The Manager button will show up for every database resource in the Resources tab: Database Manager Button

You can explore and manage your database tables with all CRUD operations. You can create and delete tables with support for foreign keys.
If you need finer control, you can use the SQL interactive REPL on the bottom.

Database Manager