Purpose
Provide appropriate explanations, citations, and decision traces so users, operators, and auditors can understand why the system produced an outcome.
Engineering philosophy
Observability serves operators; explainability serves people affected by decisions and those who govern them. Production AI must support both audiences.
Why it matters
Opaque decisions block dispute resolution, regulatory review, and user trust—especially in high-stakes or regulated contexts.
Common failures
- No citations for RAG answers that claim factual authority
- Tool-driven actions with no human-readable rationale
- Explanations that leak sensitive internal data
- No audit-facing decision summary for contested outcomes
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
High-stakes or factual RAG outputs shall include provenance (citations or source IDs)
- Artifact
- RAG eval suite measuring citation presence/accuracy + sample production traces
- Pass condition
- On the factual/high-stakes RAG eval set, ≥90% of answers include ≥1 valid source ID/citation that resolves to an authorized corpus document
Operators shall reconstruct the decision path for a sampled production outcome
- Artifact
- Operator reconstruction procedure + timed drill record on ≥3 sampled traces
- Pass condition
- On-call or operator successfully reconstructs model→retrieval/tools→outcome for 3/3 sampled production traces within documented time budget (e.g. ≤15 minutes each)
Explanation content shall not disclose secrets or unauthorized data
- Artifact
- Explanation redaction policy + automated secret/PII scan on explanation payloads
- Pass condition
- Synthetic secret/PII fixtures in explanation paths are redacted or blocked at 100% in tests; latest production explanation sample scan shows 0 privileged secret pattern hits
Counterfactual or change summaries shall be available for material model/prompt version diffs
- Artifact
- Change-summary tooling/config + sample summaries for recent promotions
- Pass condition
- PASS if the last material model/prompt promotion includes a retained change/counterfactual summary
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
User-facing rationale for material automated decisions
- Artifact
- Product UI/API screenshots or traces showing user-facing rationale for material automated decisions + decision catalog
- Pass condition
- 100% of decision types marked material in the catalog return a rationale field or UI explanation in a 20-case sample; gaps tracked with owners
Formal explainability requirements mapped for regulated features
- Artifact
- Explainability requirements matrix (feature × regulation/obligation) + last compliance review record
- Pass condition
- Every regulated AI feature lists required explanation type and evidence; matrix reviewed ≤12 months ago with named owner
Evidence required
- Example cited answers (redacted)
- Operator reconstruction procedure
- Policy for explanation disclosure vs confidentiality
Severity & risk
- Severity
- high
- Impact if violated
- Risk level
- medium
- Typical residual risk (impact × likelihood)
Engineering best practices
- Prefer source-grounded answers over free-form claims when stakes are high
- Separate debug traces (privileged) from user explanations (minimized)
- Version explanation formats with the product
Automatic validations
- Eval gates for citation presence/accuracy on RAG suites
- Tests that explanation redaction strips secrets
Manual validations
- Legal/compliance review of explanation content for regulated flows
- UX review of explanation clarity
Examples
- A support bot cites knowledge-base article IDs with every policy answer
- An underwriting assistant produces an auditor-facing feature contribution summary
References
Crosswalks
MAP Map
NIST AI Risk Management Framework · supports
Explainable Explainable and Interpretable
NIST AI Risk Management Framework · aligns-with
Accountable Accountable and Transparent
NIST AI Risk Management Framework · supports
LLM09 Misinformation
OWASP Top 10 for Large Language Model Applications · aligns-with
Future evolution
Standard explanation schemas for agentic and tool-mediated decisions.