Purpose
Institutionalize offline and online evaluation so releases are gated on measured quality, safety, and task success—not intuition or demo success.
Engineering philosophy
If you cannot measure it, you cannot claim production readiness. Evaluation is a product dependency equal to tests in traditional software, extended for stochastic systems.
Why it matters
LLMs regress silently. Without evals, teams ship prompt or model changes that increase hallucination, toxicity, tool errors, or task failure rates under real traffic.
Common failures
- Only manual spot checks before release
- Eval sets that do not match production task distribution
- No online monitoring of quality after deploy
- Safety and quality evals owned by nobody
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Critical customer journeys shall have automated offline eval suites on every relevant change
- Artifact
- Eval suite registry mapped to journeys + CI runs on prompt/model/tool changes
- Pass condition
- 100% of journeys marked critical have a versioned offline suite; 100% of relevant production changes in last 30 days triggered the suite (or documented waiver ≤ 14 days)
Release gates shall enforce numeric minimum thresholds for quality and safety metrics
- Artifact
- Gate config with metric names and thresholds + CI reports
- Pass condition
- Each critical journey has ≥1 quality and ≥1 safety metric with numeric threshold; failing gate blocks deploy in CI
Production shall have online signals for task success/failure and safety refusals
- Artifact
- Dashboard/metrics for task success and safety refusal rates
- Pass condition
- Online metrics exist and are updating for task success/failure and safety refusals; alert or review cadence defined; freshness ≤ 24 hours on the dashboard
Shadow deployment with eval comparison shall precede full cutover for high-risk AI changes
- Artifact
- Shadow/canary eval config + comparison report for the last high-risk cutover
- Pass condition
- PASS if the last high-risk AI cutover retained a shadow/canary comparison that met promotion criteria before 100% traffic
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Separate eval tracks for regression, adversarial, and distribution-shift cases
- Artifact
- Eval catalog showing separate tracks for regression, adversarial, and distribution-shift + latest CI matrix
- Pass condition
- All three tracks exist with distinct corpora and owners; each track ran successfully on the last production model/prompt promotion (≤90 days)
Human preference or expert review sampling on a defined cadence
- Artifact
- Human preference / expert-review sampling protocol + last scored sample set with inter-rater notes
- Pass condition
- Cadence and sample size are defined; last sample ≤90 days covers production-like prompts; disagreements have adjudication recorded
Evidence required
- Eval suite definitions and ownership
- Gate thresholds and recent pass/fail reports
- Dashboards for online quality metrics
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Version eval datasets; treat label quality as seriously as code quality
- Include tool-use and multi-turn scenarios, not only single-shot Q&A
- Budget for eval compute as part of CI cost
- Correlate eval failures to traces for fast debugging
Automatic validations
- CI pipelines blocking on eval gate failures
- Scheduled regression runs against production prompts/models
- Drift detection when online metrics diverge from offline baselines
Manual validations
- Periodic calibration of automated graders vs human labels
- Product review of eval coverage gaps
Examples
- A RAG support bot cannot ship a prompt change unless citation accuracy stays above threshold
- An agent fails CI when tool-selection accuracy drops on the golden set
References
Crosswalks
MEASURE Measure
NIST AI Risk Management Framework · supports
Safe Safe
NIST AI Risk Management Framework · aligns-with
Fair Fair — Harmful Bias Managed
NIST AI Risk Management Framework · partial
§6 Planning
ISO/IEC 42001 · partial
§9 Performance evaluation
ISO/IEC 42001 · supports
LLM04 Data and Model Poisoning
OWASP Top 10 for Large Language Model Applications · supports
LLM09 Misinformation
OWASP Top 10 for Large Language Model Applications · aligns-with
CC3 Risk Assessment
SOC 2 Trust Services Criteria · evidence-for
PI1 Processing Integrity
SOC 2 Trust Services Criteria · evidence-for
Generative AI Lens Generative AI Lens (themes)
AWS Well-Architected Framework · aligns-with
Future evolution
Shared public eval protocols for agentic and MCP workloads, with portable scorecards.