Guide · Reliability
Runbook Template for Production Outages
A runbook template for production outages. Database down, API down, high error rate. Copy and customize for your system.
Runbooks are Incident Readiness evidence—practiced steps beat tribal knowledge during outages. Primary control: Incident Readiness
On-call without a script invents the script under fire
A runbook is the checklist you wish you had at 3am: symptoms, checks, actions, escalation, links. One engineer burned 45 minutes rediscovering RDS checks on an outage; a five-line runbook (status → pool → restart/escalate) would have been ~10 minutes.
Fill this template once per failure mode
Title — e.g. Database Down Symptoms — user impact + which alerts fire Checks — ordered: RDS status, connection pool, recent deploys, provider status Actions — restart/scale/rollback with who can approve Escalation — when and whom Links — CloudWatch, consoles, status pages
Write the same shape for API 5xx, auth outage, and latency SEVs. Keep steps concrete ("open RDS console → check…") not "look at the database." Drill once; update when architecture changes.
Runbooks are incident readiness evidence
APRF Incident Readiness expects a known path from alert to action. Pair with PagerDuty and the IR plan so the page opens a doc, not a blank Slack thread.
Next: Incident Readiness
Open the related pillar specification for mandatory checks, artifacts, and pass conditions. Self-attest is optional.
Related
Frequently asked questions
- What should a runbook include?
- Symptoms, checks (what to look at), actions (what to try), escalation (when to call someone else), and links to dashboards and logs.
- How do I create a runbook for production outages?
- Identify common failures. For each, document: symptoms, checks, actions, escalation. Include links to dashboards. Keep it updated.
- What is a runbook template?
- A template structure: Title, Symptoms, Checks, Actions, Escalation, Links. Copy and fill in for each failure type. Customize for your system.