Purpose
Define SLIs/SLOs for AI latency, availability, and quality signals; manage error budgets so AI features meet user experience and capacity targets.
Engineering philosophy
If it is not measured with an SLO, it is not operationally owned. Stochastic systems still need latency and quality budgets.
Why it matters
Unbounded TTFT, streaming stalls, and silent quality burn destroy UX and hide regressions that “availability” alone will not catch.
Common failures
- No p95/p99 latency targets for model calls
- Quality treated as a one-time eval, not an online SLO
- No error budget for AI feature releases
- Capacity planning ignores bursty agent tool loops
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Critical AI user journeys shall have documented latency and availability SLOs
- Artifact
- SLO catalog with numeric targets per journey
- Pass condition
- 100% of journeys marked critical have availability % and latency percentile targets recorded
Online dashboards shall track AI latency, error, and at least one quality or task-success signal
- Artifact
- Dashboard URLs/config showing latency, error, and quality/task-success panels
- Pass condition
- Dashboard panels for latency, error rate, and ≥1 quality/task-success signal are live with freshness ≤ 15 minutes
Alerting shall exist when SLOs burn beyond defined thresholds
- Artifact
- Burn-rate or SLO alert policies
- Pass condition
- Alert policies exist for each critical journey SLO; alert test or documented fire demonstrates notification path works
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Error budgets formally gate release velocity for AI features
- Artifact
- Error-budget policy linking AI SLOs to release freezes + last budget burn report that gated a release
- Pass condition
- When error budget for a critical AI journey is exhausted, release velocity is blocked or requires explicit risk acceptance; ≥1 gated event or drill in 90 days
Capacity and load tests include adversarial long-prompt and agent-loop scenarios
- Artifact
- Load-test plan including long-prompt and agent-loop scenarios + latest capacity test report
- Pass condition
- Last capacity test ≤90 days includes adversarial long prompts and multi-step agent loops; p95 latency and error rate stay within SLO under documented concurrency
Streaming-specific SLIs (TTFT, inter-token latency) for streaming UIs
- Artifact
- Streaming SLI dashboard for TTFT and inter-token latency + alert config for streaming UIs
- Pass condition
- TTFT and inter-token latency SLIs exist for each streaming AI surface; alerts fire on documented thresholds; series retained ≥30 days
Evidence required
- SLO definitions for AI journeys
- Dashboard and alert configuration
- Recent burn-rate or incident examples (redacted)
Severity & risk
- Severity
- high
- Impact if violated
- Risk level
- medium
- Typical residual risk (impact × likelihood)
Engineering best practices
- Separate SLOs for AI features vs core non-AI paths
- Budget tokens and wall-clock together
- Correlate performance regressions with model/prompt version changes
Automatic validations
- Synthetic latency probes
- Burn-rate alerts
- CI performance budgets for critical paths where feasible
Manual validations
- Quarterly SLO review with product owners
- Post-incident SLO recalibration
Examples
- Chat p95 TTFT < 1.5s with error-budget policy that freezes prompt experiments when burned
- Agent task-success rate tracked as a quality SLO alongside HTTP availability
References
Crosswalks
MEASURE Measure
NIST AI Risk Management Framework · supports
§9 Performance evaluation
ISO/IEC 42001 · supports
LLM10 Unbounded Consumption
OWASP Top 10 for Large Language Model Applications · supports
A1 Availability
SOC 2 Trust Services Criteria · evidence-for
Reliability Reliability
AWS Well-Architected Framework · aligns-with
Performance Efficiency Performance Efficiency
AWS Well-Architected Framework · partial
Future evolution
Standard GenAI SLI catalogs (TTFT, tool-loop duration, citation accuracy) across ecosystems.