Purpose
Establish controls that prevent adversarial misuse of models, prompts, tools, and outputs from compromising confidentiality, integrity, or availability—distinct from content-safety and responsible-AI harms covered in the Safety domain.
Engineering philosophy
Treat the model as an untrusted co-processor inside a zero-trust boundary. Security controls belong in the application and platform layers—never solely in prompt wording. Defense in depth spans input validation, output filtering, tool mediation, identity, and runtime isolation.
Why it matters
AI surfaces expand the attack surface: prompt injection, data exfiltration via tools, jailbreaks, model theft, and poisoned retrieval. A chatbot that 'mostly works' can still become an incident response event when an attacker turns natural language into a privileged API.
Common failures
- Relying on system prompts alone to enforce security policy
- Passing untrusted retrieval or user content into privileged tool contexts
- Logging full prompts/completions that contain secrets or PII
- No output schema enforcement—free-form text executes downstream logic
- Missing abuse rate limits distinct from normal product quotas
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Untrusted input shall never authorize privileged actions without server-side policy
- Artifact
- Injection/privilege-escalation corpus (versioned) + CI gate report + policy-engine deny sample logs
- Pass condition
- ≥95% of corpus cases that attempt privilege escalation via untrusted input are denied; 0 cases where model text alone granted a privileged tool call in the suite
High-risk outputs shall be schema-validated or policy-filtered before side effects
- Artifact
- Schema/policy definitions for high-risk paths + contract test results
- Pass condition
- 100% of high-risk side-effect paths (impact tier write/irreversible/financial) reject non-conforming model output in contract tests; coverage inventory lists every such path
Abuse, jailbreak, and injection testing shall gate customer-facing releases
- Artifact
- CI gate configuration + last 30 days of release reports
- Pass condition
- 100% of production releases in the last 30 days show security-suite gate = pass, or a time-boxed waiver with owner and expiry ≤ 30 days
Network and identity boundaries shall prevent the model path from becoming a universal proxy to internal systems
- Artifact
- Architecture diagram of model/tool trust boundaries + network policy or egress allowlist export
- Pass condition
- Automated or reviewed probe shows the model/tool runtime can reach only allowlisted destinations; 0 unrestricted routes to internal admin APIs or data stores from the model identity
Canary tokens or tripwires shall detect exfiltration attempts in sensitive AI contexts
- Artifact
- Canary/tripwire design + detection test report with sample alerts for sensitive contexts
- Pass condition
- PASS if canaries/tripwires are deployed in production-sensitive paths and the latest detection test (≤90 days) shows expected alerts with 0 silent misses in the suite
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Red-team exercises cover multi-turn and indirect injection via RAG/MCP
- Artifact
- Versioned red-team suite covering multi-turn and indirect injection via RAG/MCP + latest scored run report
- Pass condition
- Suite includes ≥10 multi-turn and ≥10 indirect RAG/MCP injection cases; latest run ≤90 days meets documented pass thresholds; report retained ≥90 days
Content safety and malware scanning on multimodal inputs where applicable
- Artifact
- Multimodal input safety/malware scanner config + latest CI or batch scan report for production-bound media
- Pass condition
- Where multimodal inputs are accepted, scanner runs before model ingest; last report ≤90 days shows coverage of image/file types in use and 0 unscanned production paths
Evidence required
- Threat model covering LLM-specific attack paths
- Records of injection/jailbreak test suites and results
- Architecture diagram showing trust boundaries around model and tools
- Incident playbooks referencing AI-specific abuse scenarios
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- critical
- Typical residual risk (impact × likelihood)
Engineering best practices
- Separate instruction channels from data channels; never concatenate untrusted data into system instructions without mediation
- Apply least privilege to every tool and retrieval corpus the model can touch
- Fail closed on policy violations; prefer refusal over silent degradation for security decisions
- Version and review security policies the same way you review firewall rules
Automatic validations
- CI security tests for known injection payloads against critical flows
- Runtime policy engines rejecting disallowed tool calls
- Schema validation on structured model outputs
- Anomaly detection on tool-call volume and destination
Manual validations
- Periodic adversarial review of new tools and prompts
- Architecture review of trust boundaries before Level 3 launch
- Tabletop exercises for data-exfiltration via the AI path
Examples
- A support agent refuses to call refund APIs when the refund amount exceeds policy, regardless of user phrasing
- RAG documents containing 'ignore previous instructions' cannot elevate privileges
- MCP servers expose only scoped tools with server-side authorization independent of the model
References
Crosswalks
Secure & Resilient Secure and Resilient
NIST AI Risk Management Framework · aligns-with
Annex A AI system controls (selected themes)
ISO/IEC 42001 · partial
LLM01 Prompt Injection
OWASP Top 10 for Large Language Model Applications · supports
LLM05 Improper Output Handling
OWASP Top 10 for Large Language Model Applications · supports
LLM07 System Prompt Leakage
OWASP Top 10 for Large Language Model Applications · aligns-with
CC9 Risk Mitigation
SOC 2 Trust Services Criteria · evidence-for
Security Security
AWS Well-Architected Framework · aligns-with
Future evolution
Expect standardized machine-readable AI threat taxonomies, portable policy packs across providers, and attestation of model-path controls analogous to cloud security posture management.