Guide · Observability
AI Latency and Token SLOs in Production
If you only watch uptime, AI failures look like "slow chat." APRF Performance & SLO expects latency, token, and quality budgets with burn alerts—not vibes.
Uptime alone is not enough—APRF Performance & SLO needs latency and token budgets with error-budget burn. Primary control: Performance & SLO
Green 5xx, red product
HTTP availability can look fine while p95 chat latency crawls from 2s to 12s and tokens triple. Users leave; finance notices the bill. APRF Performance & SLO treats AI journeys as promises: SLIs, targets, error budgets, and burn alerts—not "the model feels snappy."
A support bot had only uptime checks. A context-pack expansion doubled latency and tokens. Adding `ai_ttft_ms`, `ai_e2e_ms`, and `tokens_out` with a p95 SLO and burn alert caught the next regression in canary.
Pick journeys and SLIs
Name the path—e.g. "first useful token" and "turn complete." Measure:
- Latency: TTFT + end-to-end (histograms, not averages)
- Tokens in/out
- Errors / timeouts
- Optionally tool-success or groundedness
Break spans into model vs retrieval vs tools; slice dashboards by `model_id` and `prompt_version`. Tokens are a reliability metric here: they drive latency, cost, and timeouts.
Error budget is a ship policy
Document what happens when budget burns—freeze risky prompt/model releases until the path recovers (Change Management + Evaluation canaries). Alert on burn rate, not only absolute thresholds. Provider status pages are not your SLO.
Assess against APRF Core
Run the Core Profile quiz — gated pass/fail blockers for AI production readiness, not a vanity score.
Related
Frequently asked questions
- What latency should we target?
- Product-specific. Interactive chat often needs fast TTFT; batch can tolerate higher e2e. Write the number down and measure the tail, not only the mean.
- Are tokens a reliability metric?
- Yes for AI paths. Token spikes drive latency, cost, and timeout risk—APRF ties Performance & SLO to Cost Optimization ceilings.
- How does this relate to Observability?
- Observability provides the signals. Performance & SLO decides which signals are promises and when you must stop shipping.