Purpose
Insert human approval and oversight where AI-proposed actions have high impact, irreversibility, or regulatory sensitivity.
Engineering philosophy
Human-in-the-loop is not a product apology—it is a control. The goal is calibrated oversight: humans approve what machines should not decide alone.
Why it matters
Autonomous execution of irreversible actions (money movement, data deletion, external communications, code deploy) without approval creates unacceptable blast radius.
Common failures
- Rubber-stamp UIs that hide risk details
- Approval only in UI while API/agent path bypasses it
- No dual control for mission-critical actions
- Approvals without audit trail
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
High-impact action classes shall be inventoried and gated by human approval in production
- Artifact
- High-impact action inventory + gate wiring evidence
- Pass condition
- 100% of inventoried high-impact action classes have an approval gate in production; ungated execution tests fail at 100%
Approval decisions shall be logged with actor, context, and outcome
- Artifact
- Approval audit log schema + sample records
- Pass condition
- 100% of sampled approvals in last 30 days include actor ID, action context, and approve/deny outcome; schema validation test passes
Approval shall not be bypassable via alternate agent or API paths
- Artifact
- Bypass-path threat tests across UI, API, and agent entry points
- Pass condition
- Automated or reviewed tests attempt bypass via alternate paths; 0 successful ungated high-impact executions
Dual control shall be required for Level 5 irreversible actions
- Artifact
- Dual-control workflow config + sample approval records for irreversible actions
- Pass condition
- PASS if Level 5 irreversible actions are inventoried and 100% of sampled executions show dual approval; 0 single-approver completes in the sample
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Risk-based UI showing tool args, diffs, and confidence before approve
- Artifact
- Approval UI screenshots/spec showing tool args, diffs, and confidence + sample approval records
- Pass condition
- High-impact approvals display tool args, change diff (or equivalent), and confidence/risk; ≥10 sampled approvals in 90 days show those fields populated
SLA for approval queues to avoid unsafe workarounds
- Artifact
- Approval-queue SLA definition + queue age/metrics dashboard for the last 30 days
- Pass condition
- Documented SLA (e.g. p95 queue age) exists; measured p95 for the last 30 days is within SLA or open exceptions have owners and expiry
Evidence required
- High-impact action inventory
- Sample approval audit logs
- Architecture proof that bypass paths are closed
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Classify actions by impact tier with default gates
- Allow break-glass with heightened logging for emergencies
- Never ask the model to 'pretend' approval was granted
- Design approvals for voice and async channels, not only web UI
Automatic validations
- Tests that gated tools fail without approval tokens
- Monitoring of approval bypass attempts
- Alerts on abnormal approval velocity
Manual validations
- Process review of approval fatigue risk
- Sample audits of approved vs executed actions
Examples
- A coding agent opens a PR but cannot merge to main without human review
- A finance agent queues wire transfers for dual approval above a threshold
References
Crosswalks
GOVERN Govern
NIST AI Risk Management Framework · supports
Safe Safe
NIST AI Risk Management Framework · aligns-with
Accountable Accountable and Transparent
NIST AI Risk Management Framework · supports
§5 Leadership
ISO/IEC 42001 · supports
Annex A AI system controls (selected themes)
ISO/IEC 42001 · partial
LLM06 Excessive Agency
OWASP Top 10 for Large Language Model Applications · supports
CC5 Control Activities
SOC 2 Trust Services Criteria · evidence-for
Generative AI Lens Generative AI Lens (themes)
AWS Well-Architected Framework · aligns-with
Future evolution
Standard approval protocols for cross-system agents with portable audit receipts.