Purpose
Prepare people, playbooks, and tooling to detect, contain, communicate, and learn from AI-specific incidents—including abuse, quality collapse, and unsafe tool actions.
Engineering philosophy
Incidents will happen. Readiness means AI-aware detection, clear ownership, containment that includes pausing agents, and blameless learning that improves pillars.
Why it matters
Traditional SEV playbooks miss prompt injection campaigns, model outages, and agentic damage. Without AI-specific readiness, mean time to contain balloons.
Common failures
- No playbook for model abuse or data leakage via chat
- On-call cannot pause agents or roll back prompts
- Customer communication templates ignore AI failure modes
- No post-incident action tracking into evals and controls
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
AI-specific incident playbooks shall exist for abuse, leakage, bad actions, and provider outage
- Artifact
- Playbook set covering the four scenarios with owners
- Pass condition
- Four playbooks present (abuse, leakage, bad actions, provider outage), each with owner and review date ≤ 12 months
On-call shall be able to execute containment: pause agents, disable tools, roll back prompts/models
- Artifact
- Containment runbook + drill record exercising pause, disable, and rollback
- Pass condition
- Drill in last 90 days successfully demonstrated pause agents, disable tools, and roll back prompt/model within documented time budgets
Post-incident reviews shall produce tracked actions against APRF pillars
- Artifact
- Post-incident review template + sample reviews with linked actions
- Pass condition
- 100% of SEV-eligible AI incidents in last 90 days have a review with ≥1 tracked action mapped to an APRF pillar or explicit “no action” rationale
Regular tabletop exercises shall cover AI-specific incidents
- Artifact
- Tabletop plan + dated after-action report for an AI incident scenario
- Pass condition
- PASS if an AI-focused tabletop completed ≤180 days with retained actions and owners
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Page-worthy alerts for safety and quality signals, not only infra
- Artifact
- On-call alert policy export listing safety/quality pages + last 90 days of triggered incidents (or drill tickets)
- Pass condition
- At least two non-infra signals (e.g. refusal-rate spike, eval-score drop, toxicity/jailbreak hit rate) page an on-call; each has a documented threshold and owner; policy reviewed ≤90 days ago
Customer notification criteria for AI-related events
- Artifact
- Customer notification criteria for AI-related events + last drill or real notification sample
- Pass condition
- Criteria map event types (safety incident, widespread quality fail, data exposure) to notify / no-notify; last drill or incident ≤12 months followed the criteria with timestamps
Evidence required
- Playbooks and ownership roster
- Alert configuration samples
- Post-incident review examples (redacted)
Severity & risk
- Severity
- high
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Define SEV classifications for AI harm (financial, privacy, safety, reputation)
- Preserve traces for forensics under legal hold when needed
- Train support and on-call on AI failure literacy
- Link incidents to eval fixtures so regressions cannot silently return
Automatic validations
- Alert routing tests
- Automated creation of incident tickets from critical signals
- Verification that kill switches remain reachable
Manual validations
- Tabletop exercises
- After-action review quality checks
Examples
- A spike in tool denials pages on-call; the agent is paused while injection is investigated
- A postmortem adds a new adversarial eval case that fails CI until fixed
References
Crosswalks
MANAGE Manage
NIST AI Risk Management Framework · supports
§10 Improvement
ISO/IEC 42001 · aligns-with
CC7 System Operations
SOC 2 Trust Services Criteria · evidence-for
Operational Excellence Operational Excellence
AWS Well-Architected Framework · aligns-with
Reliability Reliability
AWS Well-Architected Framework · aligns-with
Future evolution
Shared AI incident taxonomies and anonymized industry learning feeds.