Writing the code is one job. Keeping it alive at 2am — under real users, real money, and a clock you can't pause — is a different one entirely. This simulator drops you into a live incident: a page fires, the graphs are red, and you have to act before you understand. The counter-intuitive move underneath it all is the whole lesson — stop the bleeding first, find the root cause second.
Checkout is throwing errors and customers can't pay. Every action you take costs time, and while the site is down that time burns your error budget. Work the incident the way an SRE would — the order of your moves matters far more than which tools you know.
Automated monitor: 5xx responses on /checkout crossed 5% for 3 minutes and
are still climbing. Customers are seeing failed payments.
Choose your next move — the clock is running.
The disciplined order: detect → mitigate → diagnose → resolve → learn. The pro move most people get wrong is mitigating before they root-cause — a rollback stops the pain in seconds, and the "why" can wait until users are safe.
The single biggest shift from writing software to operating it is where you think the job ends. Teams that hand the pager to someone else optimise for shipping; teams that carry their own pager optimise for the thing users actually experience — whether it works at 3am.
| The build-only mindset | You build it, you run it | |
|---|---|---|
| When is it done? | When the code merges and the feature demos. | When it's been running reliably in production for real users. |
| What's the goal? | Ship features fast. | Ship fast and keep it up — spend a budget of downtime wisely. |
| When it breaks… | Throw it over the wall to ops. "Works on my machine." | The people who wrote it get paged and fix it. Feedback loop closes. |
| The metric | Velocity, story points, features shipped. | MTTR, error budget, SLOs — how the system behaves, not just how it's built. |
| Operability is… | Someone else's problem, bolted on later. | A design input from day one: logging, rollbacks, health checks, alerts. |
Chasing perfect reliability is a trap: the last fractions of a percent cost exponentially more, and you buy them by never shipping anything.
An SLO of 99.9% is also a permission slip: it says you're allowed ~43 minutes of downtime a month. That's a budget — spend it on shipping boldly.
The instinct is to understand the bug before touching it. Under fire that's backwards — every minute spent root-causing is a minute the site stays down.
An SLO is a budget of downtime
| SLO | Allowed downtime / month | Allowed downtime / year | In plain terms |
|---|---|---|---|
| 99% | 7.2 hours | 3.65 days | "Two nines." Generous. Fine for internal tools; painful for anything customer-facing. |
| 99.9% | 43 min | 8.8 hours | "Three nines." The common bar for a serious service. |
| 99.99% | 4.3 min | 53 min | "Four nines." Elite and expensive — almost no room for a bad deploy. |
| 99.999% | 26 sec | 5.3 min | "Five nines." Rare, costly, and usually more than anyone actually needs. |
An error budget turns reliability from a moral crusade into an engineering trade-off. Move the SLO and watch how much downtime you're allowed — and how many risky changes that buys you. Then drill the reflex the simulator teaches: match the mitigation to the failure.
Not every incident wants a rollback. Pick the fastest mitigation for each — the move that stops user pain, before you worry about root cause.
The same handful of outages happen to everyone. What separates a calm team from a thrash is having the first move rehearsed — so the mitigation is muscle memory and the debate can wait for the post-mortem.
Reliability isn't a heroics problem, it's a habits problem — the order you work an incident, the budgets you set, the guardrails you build in before you need them. Leaders who've never carried a pager routinely under-invest in it until the 2am cost lands. That's the work we do with the teams we coach: turning firefighting into a calm, repeatable practice.
Talk to us about coachingUnder fire, stop the impact first — roll back, fail over, shed load — then find the root cause in the calm. The "why" is a luxury you earn after users are safe.
An SLO gives you a budget of downtime to spend on velocity. You don't need perfect reliability; you need to spend an imperfect budget deliberately.
Carrying the pager for your own code closes the feedback loop. It makes operability a design input from day one, not someone else's clean-up job.