Purpose
Minimize collection and exposure of personal and sensitive data across prompts, retrieval, memory, logs, and third-party model providers.
Engineering philosophy
Privacy by design for AI means data minimization, purpose limitation, residency awareness, and contractual/technical controls on processors—not only a privacy policy page.
Why it matters
AI pipelines concentrate sensitive data. Over-collection and uncontrolled provider sharing create regulatory, contractual, and trust failures even when 'security' looks fine.
Common failures
- Sending full customer records to external models when aggregates would do
- No data classification for prompt contents
- Training or eval sets with uncleared personal data
- Ignoring residency requirements when choosing providers
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Data flowing to models shall be classified; sensitive classes shall have documented handling rules
- Artifact
- Data classification scheme + handling rules for model-bound payloads
- Pass condition
- Classification scheme covers AI payload classes; ≥1 automated or sampled audit shows 100% of sampled production model requests tagged with a class, and sensitive-class handling matches policy
Third-party model processing terms shall be reviewed for training use and retention
- Artifact
- Vendor terms review record (DPA/training/retention) with date and owner
- Pass condition
- 100% of production model providers have a review record ≤ 12 months covering training use and retention; 0 unreviewed providers in the inventory
Users or tenants shall have deletion/export paths covering AI memory and logs where required
- Artifact
- Deletion/export procedure + successful test execution record
- Pass condition
- Documented API/runbook covers AI memory and in-scope logs; test shows deletion/export completes within SLA for a sample tenant/user (measured duration recorded)
Residency-constrained routing shall be enforced for regulated workloads
- Artifact
- Routing policy config + sample routing decisions for regulated tenants/regions
- Pass condition
- PASS if regulated workloads are labeled and 100% of sampled regulated requests stay in approved regions
DPIA/PIA equivalents shall exist for major AI features before production
- Artifact
- Completed DPIA/PIA (or equivalent) with sign-off for each major in-scope AI feature
- Pass condition
- PASS if 100% of major in-scope AI features have a completed assessment with owner sign-off before production traffic
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Tokenization or redaction before model calls for high-sensitivity fields
- Artifact
- Pre-model tokenization/redaction pipeline config + sample of redacted payloads for high-sensitivity fields
- Pass condition
- Documented high-sensitivity fields never appear in cleartext in model request samples (≥50 sampled calls); pipeline fails closed when redaction errors
Evidence required
- Data flow diagrams for AI features
- Provider DPA/terms review records
- Deletion/export procedure evidence
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Default to least data in context
- Separate privacy tiers of models (e.g., self-hosted for sensitive)
- Ban pasting production data into consumer AI tools for debugging
- Review eval datasets for personal data before publication
Automatic validations
- DLP scanning on outbound model payloads where feasible
- Policy checks preventing disallowed data classes from leaving the VPC
- Automated deletion job verification
Manual validations
- Privacy review for new AI features
- Vendor assessments for new model providers
Examples
- Support transcripts are redacted before being sent to an external LLM
- A healthcare workload uses a residency-locked or self-hosted model path
References
Crosswalks
Privacy-Enhanced Privacy-Enhanced
NIST AI Risk Management Framework · aligns-with
§8 Operation
ISO/IEC 42001 · supports
Annex A AI system controls (selected themes)
ISO/IEC 42001 · partial
LLM02 Sensitive Information Disclosure
OWASP Top 10 for Large Language Model Applications · supports
C1 Confidentiality
SOC 2 Trust Services Criteria · evidence-for
P-series Privacy (selected)
SOC 2 Trust Services Criteria · partial
Generative AI Lens Generative AI Lens (themes)
AWS Well-Architected Framework · aligns-with
Future evolution
Machine-readable privacy labels for model endpoints and agent capabilities.