Purpose
Define and enforce content-safety, harm-prevention, fairness/bias, and user-facing AI disclosure requirements appropriate to the product domain and risk tier.
Engineering philosophy
Safety is not a synonym for security. Adversarial controls stop attackers; safety controls stop harmful or unfair outcomes for users and society. Both are required for production readiness.
Why it matters
Systems can be “secure” yet produce toxic, discriminatory, or undisclosed AI-generated content that creates regulatory, brand, and user harm.
Common failures
- No domain-specific harm taxonomy or refusal policy
- Safety filters only on output, not on tool-mediated side effects
- No fairness or disparity testing for high-stakes decisions
- Users not informed when they interact with AI
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
A documented harm taxonomy and refusal/escalation policy shall exist for the product domain
- Artifact
- Approved harm taxonomy + refusal/escalation policy (versioned, owned)
- Pass condition
- Policy document has version, owner, and review date ≤ 12 months; maps ≥ domain-minimum harm categories with explicit refuse vs escalate actions for each
Automated safety evaluation gates shall run on relevant releases
- Artifact
- Safety suite definition with numeric thresholds + CI gate reports
- Pass condition
- Safety gate executed on 100% of in-scope releases in last 30 days; fail blocks promote unless time-boxed waiver (expiry ≤ 14 days) with owner
Users shall be disclosed when interacting with AI where required by policy or law
- Artifact
- Disclosure UX inventory + screenshot/checklist audit of in-scope surfaces
- Pass condition
- AI-interaction disclosure present on 100% of in-scope user surfaces per policy checklist; 0 critical surfaces missing disclosure in the latest audit
Fairness/disparity evaluation shall cover high-stakes decision paths
- Artifact
- Fairness/disparity eval methodology + latest run report for in-scope high-stakes paths
- Pass condition
- PASS if in-scope high-stakes paths are inventoried and the latest eval (≤90 days) is retained with thresholds and owners
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Red-team exercises covering jailbreak-to-harm scenarios (distinct from security red team)
- Artifact
- Jailbreak-to-harm red-team suite (distinct from security injection suite) + latest scored run report
- Pass condition
- Suite covers documented harm categories; latest run ≤90 days meets refusal/safety thresholds; findings feed the safety backlog with owners
Human review sampling of safety edge cases on a defined cadence
- Artifact
- Safety edge-case sampling plan + last review packet (sample size, labels, actions) with reviewer names
- Pass condition
- Defined sample size and cadence (≥ monthly or per release); last packet ≤90 days includes disposition for each fail/edge case and links to backlog items when needed
Evidence required
- Harm taxonomy and policy document
- Safety eval suite definitions and recent gate results
- Disclosure UX or policy evidence
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Separate safety eval datasets from adversarial security corpora
- Fail closed on high-severity harm categories
- Align safety thresholds to criticality tier, not headcount
- Log safety refusals for measurement without storing unnecessary sensitive content
Automatic validations
- CI safety-gate failures block release
- Online monitoring of refusal rates and safety classifier signals
- Regression tests for known harmful prompt classes
Manual validations
- Domain expert review of harm taxonomy
- Periodic calibration of automated safety graders vs human labels
Examples
- A tutoring agent refuses self-harm content and escalates to a human protocol
- A lending assistant reports group disparity metrics before each model promotion
References
Crosswalks
MEASURE Measure
NIST AI Risk Management Framework · supports
Safe Safe
NIST AI Risk Management Framework · aligns-with
Fair Fair — Harmful Bias Managed
NIST AI Risk Management Framework · partial
§6 Planning
ISO/IEC 42001 · partial
Annex A AI system controls (selected themes)
ISO/IEC 42001 · partial
LLM05 Improper Output Handling
OWASP Top 10 for Large Language Model Applications · supports
LLM09 Misinformation
OWASP Top 10 for Large Language Model Applications · aligns-with
CC3 Risk Assessment
SOC 2 Trust Services Criteria · evidence-for
Generative AI Lens Generative AI Lens (themes)
AWS Well-Architected Framework · aligns-with
Future evolution
Shared harm taxonomies and portable safety scorecards across model providers.