Skip to main content

Not self-hosting critical software is irresponsible

· 6 min read
Ruben Fiszel

Nobody ever got fired for buying IBM. Nobody gets fired for picking the SaaS either. The difference is that IBM didn't keep your keys in a shared pool with 300,000 other customers.

Zeabur did. In October, an attacker read 3.2 million environment variable records across 321,810 accounts from its platform database, two months after a first breach. The company is shutting down by December 31. Its customers did nothing wrong except pick the default.

The new IBM​

"Buy IBM" was a career decision, not a technical one. If the project failed, you had picked the vendor everyone picks, so the blame went somewhere else. "Use the hosted version" works the same way today: nobody has to justify it, and nobody owns the outcome.

But the mainframe IBM sold you sat in your machine room. A bad day at IBM headquarters didn't open your machine. A hosted platform that runs your jobs, deploys your code or syncs your CRM holds your database password, your cloud keys and your OAuth tokens, next to everyone else's, behind one set of tenant checks. That pool is the most attractive target in the building, and the attackers know it: Railway said the people who breached it were after AI inference keys.

And the odds get worse every month. Models now find vulnerabilities and write working exploits, and attackers use them as much as defenders do. We made the long version of this argument last week. The short version: every platform with enough surface will have a bad day, and on a multi-tenant one, its bad day is yours.

The last 15 months​

And the one that started the genre, CircleCI, January 2023: rotate any and all secrets.

None of these teams is careless. That's the point.

The exceptions​

Some software you can't reasonably run yourself, and we're not going to pretend otherwise.

Email is the obvious one. Running a mail server that reaches inboxes and keeps spam out is a full-time job, and self-hosting doesn't make it safer by default: on-premise Exchange had ProxyLogon, and tens of thousands of servers were hit before patches landed. Use Gmail or Microsoft 365, turn on phishing-resistant MFA, and move on.

Web-scale infrastructure is the second. You can't self-host an anycast network with hundreds of points of presence, so Cloudflare or a peer it is.

Compute is the third. You have to rent servers from someone, and whether to trust AWS or a rack in your office is its own post. Note that this is renting machines, not handing your environment variables to a platform that pools them. Half the list above is hosting platforms.

What these have in common is that the alternative is either impossible or worse. A job runner, a workflow engine, a CI system or an internal tool platform is none of those. It's a container, a database and a weekend of setup.

"But you sell a cloud"​

We do, and the argument applies to Windmill Cloud exactly as much as to anyone above. We've had our own cross-workspace advisory, fixed before it was ever exploited, and we don't expect it to be the last.

So here's the line we'd draw. If the software holds credentials to production, running it yourself is the responsible default, and putting it on a multi-tenant platform is a decision someone should own in writing. If you can't run it, a single-tenant instance with its own servers and database removes the pool, and is the next best thing. For prototyping and for workflows that hold nothing that matters, use whatever gets you running. We go through where that line sits, and the layers to put around a self-hosted instance, in For security, self-host your production internal software, including Windmill.

Windmill is open source and runs on your own infrastructure with Docker or Kubernetes. So does most of what's on that list.

Windmill Logo
Windmill is an open-source and self-hostable code-first platform for workflows, apps and AI agents, running on the fastest self-hostable orchestrator. Engineers write code in 20+ languages, business teams build with AI, and engineers review and promote what ships.

You can self-host Windmill using a docker compose up, or go with the cloud app.