Test Before You Cast
Production is not a sandbox. Neither is the teacup, it turns out.
We broke the weather once. Never again.
SpellOps exists because a group of Staff Sorcerers caused a six-hour regional outage with a weather enchantment that passed every review. We built the tooling we wished we'd had that afternoon.
Our story
It started with a light drizzle.
In the spring of 2022, four sorcerers at the Northmarch Weather Office deployed a routine drizzle enchantment for a regional farmers' market. It had been tested in a teacup. It had been approved by two senior wizards and a pigeon.
Six hours later it was snowing indoors in eleven towns, and nobody could tell which version of the spell was running, where, or why. The root-cause analysis ran to 140 pages. Its conclusion fit on one line.
Nobody had proper observability.
So they quit, borrowed a tower, and started building it. Today SpellOps runs the spell infrastructure for more than 14,000 magical organizations, including the Weather Office, which has forgiven us.
INC-0001ResolvedPostmortemRegional weather enchantment outageduration 6h 12m · 11 towns · 1 farmers' market
- 09:00drizzle@1.0 deployed to Northmarch region. Tested in a teacup.
- 09:40Precipitation reported indoors at the Weston Corn Exchange.
- 11:15Snow in eleven towns. Three versions of the spell found running.
- 14:02Rollback attempted by hand. Created a small fog bank.
- 15:12Resolved by counter-charm and a very long umbrella.
- 14,000+Magical organizations
- 2022Founded, in a borrowed tower
- 340Employees and 61 familiars
- 4Realms with offices
Our values
Four rules, carved above the server-room door.
Own Your Incidents
If you summon it, you support it. Through the weekend, if necessary.
Automate the Boring Magic
Wizards should solve interesting problems, not re-chant the same ward every Tuesday.
No Blame Postmortems
Unless someone ignored the giant red warning rune. It is very large and very red.
Leadership
The people who answer the pager.
Morgana Stacktrace
Former Principal Sorcerer at the Northmarch Weather Office. Wrote the 140-page postmortem that started it all.
Albus Kubernetes
Designed the first distributed spell scheduler. Has strong opinions about sidecar familiars.
Gerald “Jerry” Doomwhistle
Spent twelve years guarding forbidden names for the Ministry. Rotates his own true name quarterly.
Samantha Fireball
Has personally talked 300 customers out of testing in production. Name is unrelated to her methods.
Careers
Help us make magic boring, in the best way.
Remote-first across four realms. Generous mana stipend, a familiar allowance, and a strict no-on-call policy during your own solstice.
- Senior Spell Reliability Engineer
- Staff Sorcerer, Observability
- Familiar Experience Designer
- Security Necromancer II