Guide · AI Lifecycle
AI Evaluation Gates Before Production Release
If you can change a prompt or model without a failing test suite blocking the deploy, you do not have an evaluation gate. APRF expects measurable pass conditions before production.
Shipping without a failing eval gate violates APRF Evaluation—pass conditions must block the release. Primary control: Evaluation
If the build cannot fail, you do not have a gate
Changing a system prompt to "be more helpful" on a Friday, with no suite, is how inventing refund policies ships over a weekend. Rollback becomes git archaeology under pressure. APRF Evaluation means: a new `prompt_version` or `model_id` cannot reach production unless an agreed check passes—and that check is allowed to be red.
What goes in the suite
Start with golden tasks: fixed inputs and expected properties (exact string, rubric, or calibrated judge + human sample). Add adversarial cases next: injection, corpus leakage, refusal bypass. Version the suite with the artifact under test—same discipline as Prompt Engineering pins.
Offline gates are mandatory. Online canaries (quality by `prompt_version`) catch drift after release; they do not replace the blocking suite.
Wire it into release
change prompt/model → run suite → fail pipeline on threshold → store report → canary → full rolloutThresholds must be hard fails in CI or the release train—not a dashboard someone glances at. Keep HTML/JSON reports as attestation evidence. Name an owner for suite maintenance; abandoned evals rot into theater.
You do not need an LLM-as-judge on day one. Deterministic checks and rubrics get you a real gate. Judges help later for open-ended quality, with calibration.
Assess against APRF Core
Run the Core Profile quiz — gated pass/fail blockers for AI production readiness, not a vanity score.
Related
Frequently asked questions
- Do I need an LLM-as-judge?
- Not always. Start with deterministic checks and rubrics. Judges help for open-ended quality but need calibration and human sampling.
- How big should the suite be?
- Large enough to catch known regressions and top failure modes—often dozens to hundreds of cases. Grow with incidents.
- Is offline eval enough?
- Offline gates are mandatory for APRF Evaluation. Add online canaries for drift after release.