Blameless postmortem · fictional case
Checkout outage — 47 minutes
Northstar Supply · 14 March 2026
Readout owner: A. Rivera · Incident commander: S. Chen
Written report and timeline in the appendix
Impact
47 minutes, 18,400 failed checkouts, no data loss
Timeline
From deploy to recovery
Times UTC. Full second-by-second timeline in the written report.
Root cause
A config flag mismatch, amplified by a retry storm
Trigger
4.18.0 changed the retry budget from 3 to 8 but left the payment gateway flag at the old ceiling.
Amplifier
Failed authorizations retried immediately; the gateway throttled the whole merchant account, not the single route.
Not a single bug. A deploy-safety gap let an inconsistent configuration reach 100% of traffic.
Contributing factors
What made it possible
Canary too small
2% of traffic for 4 minutes could not surface a gateway-wide throttle.
No synthetic checkout
Health checks covered the API, not the checkout path.
Runbook stale
The rollback step referenced the old deployment tool.
Handoff gap
The on-call handoff missed the deploy note by 12 minutes.
What went well
Recovery was faster because the basics held
Rollback
One command; no schema change. The last known-good artifact was available.
Comms
Status page updated at 09:26; support had a script by 09:32.
Actions
Six actions, each with an owner and a date
| Action | Owner | Due | Done when |
|---|---|---|---|
| Canary by checkout route | Platform | 28 Mar | 5% route-level canary |
| Synthetic checkout probe | SRE | 4 Apr | 1-min probe, paging |
| Retry budget validation | Payments | 21 Mar | Config test in CI |
| Rollback runbook refresh | On-call | 21 Mar | Drill passed |
| Gateway throttle isolation | Payments | 18 Apr | Per-route quota |
| Deploy note in handoff | Eng mgmt | 28 Mar | Template adopted |
Metrics
Detection was fast; recovery was the constraint
| Metric | This incident | Target |
|---|---|---|
| Time to detect | 6 min | ≤ 5 min |
| Time to mitigate | 41 min | ≤ 20 min |
| Error-budget burn | 62% of quarter | ≤ 20% |
The lesson
Deploy safety is a product, not a checklist.
The system allowed an inconsistent configuration to reach all traffic. Fix the system, not the deploy that exposed it.
Appendix
Evidence
- Second-by-second timeline and log excerpts
- Customer communication and support script
- Config diff and gateway throttle trace
- Action tracker and review dates