Purpose
Promote AI artifacts (models, prompts, tools, agents, indexes) through controlled environments, and enable immediate rollback when production behavior regresses.
Engineering philosophy
If you cannot roll back, you cannot safely roll forward. AI change management treats prompts and model pins as release units with the same discipline as application code.
Why it matters
Uncontrolled hot-edits and untested rollbacks create drift, unreproducible incidents, and long mean-time-to-recover when quality or safety collapses after a change.
Common failures
- Hot-editing production prompts
- Indexes rebuilt manually with no version tag
- Tools registered in prod without staging soak
- Environment config drift for model pins
- No previous prompt version retained
- Model pin changed with no path back
- Schema migrations that break old tool clients irreversibly
- Rollback untested until an incident
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
A promotion path shall exist from non-prod to prod for prompts, models, and tools
- Artifact
- Pipeline/runbook documenting non-prod→prod promotion for prompts, models, and tools
- Pass condition
- 100% of production prompt/model/tool releases in the last 30 days flowed through the documented promotion path; 0 production hot-edits without a linked change record
Production changes shall be recorded (who/what/when) and linked to reviews
- Artifact
- Change log or ticket export for AI artifact releases
- Pass condition
- 100% of production AI artifact changes in the last 30 days have who/what/when fields and a review link (PR, ticket, or approval ID)
Infrastructure and AI config shall be defined as code or equivalent declarative config
- Artifact
- IaC/config repo paths for AI gateway, model pins, tool catalogs, and prompts
- Pass condition
- Drift check shows 0 unmanaged production AI config resources outside declarative sources; sample of live pins matches declared versions at 100%
Prior production versions of prompts and model pins shall be retained and restorable
- Artifact
- Version retention policy + registry listing of prior production versions
- Pass condition
- ≥N prior production versions retained per policy (minimum N=2); restore dry-run successfully loads the immediate prior version in staging or prod-adjacent env
Rollback procedure shall be documented and operable by on-call
- Artifact
- Rollback runbook + on-call acknowledgment or drill checklist
- Pass condition
- Runbook lists exact commands/UI steps and owners; ≥1 on-call engineer completed a walkthrough or drill in the last 90 days with recorded time-to-execute
Rollback shall be tested in a drill or real event on a defined cadence
- Artifact
- Drill or incident record with timestamps and outcome
- Pass condition
- ≥1 successful rollback (drill or real) in the last 90 days with measured time-to-restore ≤ documented RTO
Automated rollback shall trigger on quality SLO burn for AI releases
- Artifact
- Rollback automation config + sample trigger/test evidence tied to quality SLOs
- Pass condition
- PASS if quality SLO burn is wired to automated rollback (or page+runbook with measured MTTA) and a test/drill occurred ≤90 days
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Canary or progressive delivery for high-traffic AI changes
- Artifact
- Progressive delivery/canary config for high-traffic AI changes + last canary promotion log
- Pass condition
- High-traffic AI changes use canary or progressive rollout with automated rollback criteria; last such release ≤90 days includes canary metrics link
Environment parity checks for model pins and tool catalogs
- Artifact
- Environment parity checklist for model pins and tool catalogs + last parity scan across envs
- Pass condition
- Prod vs staging model pins and tool catalogs match except documented deltas; last parity scan ≤30 days with 0 unexplained drifts
Automated migration for embedding/index version upgrades
- Artifact
- Embedding/index migration runbook + last automated migration job log and validation
- Pass condition
- Index/embedding version upgrades run via automated migration with validation gates; last upgrade ≤12 months (or N/A attestation if no upgrade) succeeded without dual-write gaps
One-click or single-command rollback for AI release units
- Artifact
- Rollback runbook or one-command script for AI release units + last rollback exercise log
- Pass condition
- AI release unit can be rolled back with a single documented command/action; exercise or real rollback ≤90 days completed within documented RTO
Feature flags for new agent behaviors
- Artifact
- Feature-flag config covering new agent behaviors + sample flag change audit trail
- Pass condition
- New agent behaviors ship behind flags; flag state changes are audited; kill/disable path tested ≤90 days ago
Evidence required
- Deployment pipeline documentation
- Sample change records
- Environment inventory
- Rollback runbook
- Evidence of tested rollback
- Version retention policy
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Bundle related prompt/model/tool versions as a release unit when they interact
- Keep staging data representative without copying unnecessary production secrets
- Automate smoke tests post-deploy including one eval canary
- Document freeze windows for peak business periods
- Keep rollback independent of full application redeploy when possible
- Version tool schemas with compatibility windows
- Avoid one-way data migrations tied to prompt experiments
- Measure time-to-rollback as an operational KPI
Automatic validations
- Pipeline gates requiring eval and security checks
- Drift detection between declared and live config
- Post-deploy smoke test automation
- CI verifying previous artifacts remain fetchable
- Synthetic rollback dry-runs in staging
- Alerts recommending rollback on quality burn
Manual validations
- Release readiness review for major agent launches
- Spot audits of console-only changes
- On-call drills executing rollback
- Post-incident verification that rollback path was used or improved
Examples
- A release train ships prompt v12 + model pin + tool schema together after staging soak
- Index rebuilds produce a new version ID switched via config flip
- Quality drop after a prompt release is reversed in under five minutes via registry rollback
- A new tool schema is dual-published so clients can fall back
References
Crosswalks
MANAGE Manage
NIST AI Risk Management Framework · supports
Accountable Accountable and Transparent
NIST AI Risk Management Framework · supports
§8 Operation
ISO/IEC 42001 · supports
§10 Improvement
ISO/IEC 42001 · aligns-with
CC8 Change Management
SOC 2 Trust Services Criteria · evidence-for
PI1 Processing Integrity
SOC 2 Trust Services Criteria · evidence-for
Operational Excellence Operational Excellence
AWS Well-Architected Framework · aligns-with
Level 1 Build process documented
SLSA (Supply-chain Levels for Software Artifacts) · partial
Level 2 Hosted build + signed provenance
SLSA (Supply-chain Levels for Software Artifacts) · aligns-with
Verify-on-deploy Verification before use
SLSA (Supply-chain Levels for Software Artifacts) · supports
Future evolution
Unified release manifests covering prompts, models, tools, eval attestations, and automatic quality-triggered rollbacks.