Purpose
Manage system, developer, and tool prompts as first-class production artifacts—versioned, reviewed, tested, and rolled back like application code.
Engineering philosophy
Prompts are code. They change behavior, create regressions, and carry security and cost implications. Informal edits in dashboards without lineage are an anti-pattern for production AI.
Why it matters
Silent prompt drift is a leading cause of quality and safety regressions. Without versioning and eval gates, teams cannot answer what changed when production behavior breaks.
Common failures
- Editing production prompts directly in a vendor console with no history
- No golden-set regression tests when prompts change
- Mixing business logic, safety policy, and tone in one unmaintainable blob
- Different prompts per environment with no promotion path
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Every production prompt shall have an immutable version identifier and owner
- Artifact
- Prompt registry export
- Pass condition
- 100% of production prompts have immutable version ID + owner; 0 unversioned production prompts
Prompt changes shall go through review and be tied to evaluation results before release
- Artifact
- Change records linking prompt versions to review + eval artifacts
- Pass condition
- 100% of production prompt releases in last 30 days have review ID and eval pass artifact; promote without both is blocked
Rollback to a prior prompt version shall be possible without redeploying unrelated services
- Artifact
- Prompt rollback procedure + timed restore test
- Pass condition
- Prior prompt version restored in ≤ documented RTO without full app redeploy; demonstrated in last 90 days
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Prompt templates are parameterized; secrets and PII never hardcoded
- Artifact
- Prompt template inventory showing parameters + static analysis or review finding zero hardcoded secrets/PII
- Pass condition
- 100% of production prompt templates are parameterized for variable inputs; scan of templates finds 0 secrets and 0 hardcoded customer PII fields
Prompt linting for length, forbidden patterns, and injection-prone constructs
- Artifact
- Prompt-lint CI config (length, forbidden patterns, injection-prone constructs) + latest lint report
- Pass condition
- Lint runs on every prompt change PR; blocking rules exist for secrets patterns and unbounded user concatenation; last failing lint example retained
A/B or shadow evaluation for high-traffic prompt changes
- Artifact
- A/B or shadow-eval config for high-traffic prompt changes + last experiment report with quality/safety deltas
- Pass condition
- Last high-traffic prompt change used A/B or shadow eval with pre-registered metrics; promotion required non-inferiority (or better) on safety and primary quality SLI
Evidence required
- Prompt registry with versions, owners, and change history
- Eval reports linked to prompt releases
- Rollback procedure documentation
Severity & risk
- Severity
- high
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Store prompts in source control or a versioned registry with CI promotion
- Separate safety/policy prompts from product/UX prompts where possible
- Measure token cost impact of prompt changes alongside quality
- Document intended failure modes and refusal behavior in the prompt contract
Automatic validations
- CI blocks deploy if prompt hash lacks associated eval pass
- Diff reviews in pull requests for prompt files
- Token budget checks against historical baselines
Manual validations
- Peer review of safety-critical prompt sections
- Product review of tone and policy alignment for major releases
Examples
- A coding agent system prompt is tagged v47; a quality drop triggers rollback to v46 in minutes
- Marketing copy prompts and refund-policy prompts live in separate modules with different owners
References
Crosswalks
§8 Operation
ISO/IEC 42001 · supports
LLM01 Prompt Injection
OWASP Top 10 for Large Language Model Applications · supports
LLM07 System Prompt Leakage
OWASP Top 10 for Large Language Model Applications · aligns-with
CC8 Change Management
SOC 2 Trust Services Criteria · evidence-for
Generative AI Lens Generative AI Lens (themes)
AWS Well-Architected Framework · aligns-with
Future evolution
Industry-standard prompt manifests (SBOM-like) and signed prompt releases for regulated environments.