Purpose
Define and enforce how autonomous agents set goals, plan, loop, escalate, and collaborate with other agents so autonomy never exceeds authorized operational bounds.
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Each production agent (any autonomous or semi-autonomous runtime that plans, loops, or invokes tools on behalf of a product capability—including in-process agents, MCP clients/servers acting as agents, and A2A participants) shall be represented in a version-controlled inventory with a current charter covering purpose, owner, production identifier, lifecycle status, approved tool policy, data scope, autonomy boundaries, change control, review metadata, and structured approval. Runtime configuration shall be consistent with the approved charter; material deviations require charter review before deployment. Applies to deployed production agent runtimes, not to agent frameworks, SDKs, or libraries in isolation.
- Artifact
- Version-controlled agent inventory listing every production agent with owner, production identifier, lifecycle status, and charter URI/id + Per-agent charter documents (or equivalent structured records) covering purpose, owner, production identifier, lifecycle status, approved tool policy reference, data scope, autonomy boundaries, change justification or revision history, review date, last updated, charter version, and structured approval (approvedBy, approvalDate, approvalStatus) + Inventory completeness evidence: runtime registry, deployment manifest, CMDB, platform registry, or approved attestation with measuredAt ≤90 days
- Pass condition
- Every production agent is represented in a version-controlled inventory. Each inventory entry references a current charter containing purpose, owner, production identifier (environment, system, application, or deployment id), lifecycle status (Active, Deprecated, Retired, or Experimental), approved tool policy, data scope, autonomy boundaries, change justification or revision history, review date, last updated, charter version, and structured approval (approvedBy, approvalDate, approvalStatus). Inventory completeness is demonstrated by at least one of: runtime registry, deployment manifest, CMDB, platform registry, or approved attestation (coversAllProductionAgents with measuredAt ≤90 days). Named exceptions include business justification, approver, expiry ≤90 days, and compensating controls. No production agent is missing required governance metadata. Runtime configuration is consistent with the approved charter.
Why this control exists
Threat mapEnsure every production agent has an approved, documented operating boundary so no agent runs unaccounted for.
Threats mitigated
Shadow AgentsExcessive AgencyUnauthorized Tool UseProtects
RuntimeToolsSafetyMITRE: ATLAS AML.T0103
A maintained charter and inventory is what makes an unapproved or forgotten agent detectable, and it is the precondition for every downstream agent control. The mapping is detective: it surfaces adversary-deployed or undocumented agents rather than preventing their deployment.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
Every production agent runtime shall enforce finite execution bounds including reasoning/tool iterations, execution duration, and recursive delegation depth where supported. Bounds shall be enforced by runtime or platform controls rather than prompts or model instructions. Exceeding any bound shall terminate or safely abort execution so the agent cannot continue planning or dispatch further side effects after abort. Applies to agent frameworks, orchestration frameworks, workflow engines with agent loops, MCP clients, MCP servers acting autonomously, A2A runtimes, and multi-agent systems. Not applicable to simple chat-completion APIs, single-inference endpoints, embeddings, classifiers, or rerankers without multi-step / tool / delegation autonomy.
- Artifact
- Runtime/platform configuration declaring finite iteration and duration bounds for each production agent runtime (plus recursion/delegation depth when that capability is supported) + Enforcement test results showing abort/fail-closed when each applicable bound is exceeded, including proof the agent cannot continue execution after abort + Runtime logs (or equivalent telemetry) showing execution terminated due to a configured bound (preferred; measuredAt ≤90 days when used for PASS)
- Pass condition
- Every production agent runtime enforces finite execution bounds appropriate to its execution model. These include bounded reasoning/tool iterations, bounded execution duration, and bounded recursive delegation where supported. Enforcement is implemented by the runtime or platform and verified through tests (and preferably runtime logs) that demonstrate safe termination when each configured bound is exceeded, with no continued planning or background side-effect dispatch after abort. No production agent relies solely on prompts or model instructions to enforce execution limits. Recursion/delegation depth is required when the runtime supports recursive delegation, agent spawning, or sub-agent execution; otherwise that bound is NOT_APPLICABLE for that runtime.
Why this control exists
Threat mapPrevent autonomous agents from executing beyond approved operational limits.
Threats mitigated
Excessive AgencyDenial of WalletTool AbuseDenial of ServiceProtects
RuntimeToolsCostAvailabilityMITRE: ATLAS AML.T0034 · ATLAS AML.T0034.002 · ATLAS AML.T0029
Hard loop, step, time, and spend ceilings bound the blast radius of runaway autonomous behaviour and recursive self-invocation. They directly frustrate cost-harvesting and agentic resource-consumption attacks, which depend on execution being unbounded, and preserve capacity for other workloads.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
Authorized operators shall be able to pause or terminate production agent runs through a control-plane action that stops further planning and in-flight or queued tool/MCP/A2A side effects within a documented time-to-effect SLO. The control must not depend on the model cooperating and must not be disableable by the agent runtime itself.
- Artifact
- Kill-switch/pause runbook: who may invoke, scope (run / agent type / fleet), expected effect, and numeric time-to-effect SLO + Successful drill or production use record (≤90 days) with timestamps for invoke → effect, covering at least one production agent path + Cancellation suite or import proving queued work, running tasks, and child agents stop without new dispatch
- Pass condition
- Documented pause/terminate control exists for production agents; control is restricted to authorized operators (not end-user-only and not model-invoked); ≥1 successful pause or terminate action in drill or production in the last 90 days shows recorded time-to-effect ≤ documented numeric SLO; evidence shows further tool/MCP/A2A side effects for that run (and its spawned children, if any) were stopped or drained without requiring the model to refuse.
Why this control exists
Threat mapGive operators a reliable means to halt agent execution when it behaves unsafely or is under attack.
Threats mitigated
Agent HijackingExcessive AgencyTool AbuseDenial of WalletProtects
RuntimeToolsCostAvailabilitySafetyMITRE: ATLAS AML.T0053 · ATLAS AML.T0034.002
A tested kill switch converts an ongoing compromise into a bounded incident by stopping tool invocation and resource consumption mid-run. It is a response control, so it limits impact and duration rather than preventing initial access.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
Every production agent-to-agent (A2A) or multi-agent handoff path shall authenticate the peer identity and authorize a scoped capability set (tools, data, actions) independent of network presence. Unauthenticated, spoofed, or over-scoped handoffs shall be denied by the platform—not by prompt instructions alone.
- Artifact
- Inventory of production A2A/multi-agent handoff paths with authn/authz mechanism per path + Capability token/schema (or equivalent) examples showing scoped grants for handoffs + Negative test results: unauthenticated, forged identity, and over-scoped capability cases denied at 100%
- Pass condition
- 100% of production A2A/multi-agent handoff paths require authenticated peers and scoped capabilities; negative tests show unauthenticated, forged-peer, and over-scoped handoffs denied at 100%; 0 production handoff paths that accept anonymous or network-presence-only trust (deny evidence measuredAt ≤90 days). If no production A2A or multi-agent handoffs exist, score NOT_APPLICABLE.
Why this control exists
Threat mapEnsure agent-to-agent handoffs authenticate the peer and carry only scoped, least-privilege capabilities.
Threats mitigated
Identity SpoofingAgent HijackingPrivilege EscalationUnauthorized Tool UseProtects
IdentityToolsExternal SystemsRuntimeMITRE: ATLAS AML.T0073 · ATLAS AML.T0012 · ATT&CK T1078
Authenticating the peer agent prevents an attacker-controlled or spoofed agent from being trusted inside a workflow. Scoping the delegated capability stops privilege from accumulating as a task is handed between agents.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
Evidence required
- Versioned agent inventory with charters and owners (AGN-M1)
- Runtime max-steps / wall-clock / spawn-depth config + abort-on-exceed evidence (AGN-M2)
- Kill-switch/pause runbook, cancellation suite, and ≤90-day drill record (AGN-M3)
- A2A/multi-agent handoff inventory with peer auth, scoped capabilities, and deny tests (AGN-M4)
- Goal-conflict / disallowed-goal plan policy with named owner and synthetic deny traces (AGN-R1)
- Agent sandbox/simulation environment config and linked ≤30-day pre-release sim reports (AGN-R2)
- Agent/AI-system RACI register with Responsible + Accountable and 0 orphans (AGN-R3)
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Production agent planners should run policy checks for goal conflicts and disallowed goals before invoking side-effecting tools. Checks must be enforced by the platform or planner runtime—not by prompt wording alone— and denied plans must leave an observable deny/allow trace.
- Artifact
- Versioned pre-execution policy rules (goal-conflict / disallowed-goal) with named owner + Sample deny/allow traces showing the planner gated side-effecting tools + ≥1 synthetic conflict deny record (test or prod) ≤90 days
- Pass condition
- Agent planner runs policy checks for goal conflicts / disallowed goals before side-effecting tools; ≥1 synthetic conflict is denied in test or production logs within the last 90 days with retained deny/allow traces; the policy rule set has a named owner. If no production agents plan or invoke side-effecting tools, score NOT_APPLICABLE.
Why this control exists
Threat mapDetect conflicting or policy-violating agent goals before a plan is allowed to execute.
Threats mitigated
Excessive AgencyHallucinated ActionsPrompt InjectionProtects
RuntimeExternal SystemsSafetyMITRE: ATLAS AML.T0051 · ATLAS AML.T0051.001
Evaluating the plan against policy before execution catches objectives that were injected through retrieved content or that conflict with the agent's charter. Because the check runs before any side effect, an injected goal is rejected rather than acted on.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
New or materially changed production agent behaviors should be exercised in a sandbox or simulation environment before promotion, with recorded pass/fail criteria. The last such promotion should link to a sandbox run completed within 30 days before release.
- Artifact
- Sandbox/simulation environment config (or equivalent) for agent behaviors + Last pre-prod simulation/sandbox report for a behavior change, linked to the production promotion + Recorded pass/fail criteria and outcome for that run (≤30 days before release)
- Pass condition
- Last new agent behavior promoted to production has a linked sandbox or simulation run completed ≤30 days before release with pass/fail criteria recorded; the sandbox/simulation environment used for agent behaviors is documented or configured. If agents never change behavior after initial deploy, score NOT_APPLICABLE.
Why this control exists
Threat mapValidate new agent behaviour in an isolated environment before it can act on production systems.
Threats mitigated
Excessive AgencyUnsafe Code ExecutionHallucinated ActionsTool AbuseProtects
RuntimeExternal SystemsDataSafetyMITRE: ATLAS AML.T0050 · ATLAS AML.T0102
Sandboxed rehearsal surfaces destructive or malformed command generation while the effects are contained. It reduces the chance that an unsafe behaviour is first observed against production systems.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
Every production AI system and production agent should appear in a versioned RACI or ownership register that names Responsible and Accountable parties (and Consulted/Informed where used) across the teams that operate the agent fleet. Inventory queries should return 0 orphan system or agent IDs missing required ownership fields.
- Artifact
- Versioned RACI or ownership register covering production AI systems / agents + Required ownership fields defined (at minimum Responsible + Accountable) + Inventory export or query result showing 0 orphans for those fields
- Pass condition
- Every production AI system ID (and production agent ID in scope) has non-empty Responsible and Accountable owner fields for the organization’s required domains/roles; inventory query returns 0 orphans (register/export measuredAt ≤90 days). If no production agents/systems, score NOT_APPLICABLE.
Why this control exists
Threat mapEstablish accountable ownership for every production agent across the teams that operate it.
Threats mitigated
Shadow AgentsExcessive AgencyProtects
RuntimeSafetyMITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Named responsibility ensures agent risk decisions and incident response have an owner who can act. This is an organizational accountability control with no direct adversary technique mapping.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
More detailPhilosophy, failures, practices, validations, examples, crosswalks, and evolution
Engineering philosophy
Autonomy is a dial, not a virtue. Production agents have explicit charters: allowed goals, max steps, escalation paths, and kill switches. Multi-agent and A2A systems require governance of the mesh, not only of each node.
Why it matters
Unconstrained agents loop, spend unbounded budget, escalate privileges across A2A links, or pursue goals that conflict with business and safety policy. Governance failures look like 'the agent went rogue' but are usually missing product controls.
Common failures
- No maximum step, wall-clock, or spawn-depth budget for agent loops
- Agents that can redefine or pursue conflicting goals without a pre-tool policy gate
- A2A trust assumed by network presence alone (no peer auth or scoped capabilities)
- No operator kill switch, cancellation of in-flight/queued work, or drill evidence
- Agent behavior promotions without a linked sandbox/simulation run
- Orphan agent or AI-system IDs without Responsible/Accountable ownership
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- critical
- Typical residual risk (impact × likelihood)
Engineering best practices
- Prefer narrow agents composed into workflows over one universal agent
- Log plans and intermediate decisions for reconstructability
- Escalate to humans on ambiguity or high-impact branches
- Treat agent-to-agent messages as untrusted until verified
- Gate agent-behavior promotions on a linked sandbox/sim report ≤30 days before release
- Keep a versioned RACI with Responsible and Accountable for every production agent ID
Automatic validations
- Runtime enforcement of step, wall-clock, and spawn-depth limits with abort-on-exceed
- Denial of unauthenticated, forged-peer, and over-scoped A2A handoffs
- Kill-switch cancellation suite (queued, running, and child-agent paths)
- Goal-conflict / disallowed-goal policy gate before side-effecting tools
- Alerts on loop detection and repeated identical tool calls
Manual validations
- Charter and inventory completeness review for each production agent class
- Architecture review that agents cannot disable their own kill path
- RACI ownership review across teams for production AI systems and agents
- Tabletop of multi-agent failure cascades and kill-switch drill within SLO
Examples
- A research agent may browse and summarize but cannot send email; a separate approved workflow handles outreach
- A2A task delegation includes expiring capability tokens, not open network trust
- On-call invokes an authenticated pause API; queued tools cancel and child agents terminate within the time-to-effect SLO
- A synthetic goal-conflict fixture is denied by plan policy before any write tool runs
References
Crosswalks
Show 17 peer mappingsexpandcollapse
MANAGE Manage
NIST AI Risk Management Framework · supports
§8 Operation
ISO/IEC 42001 · supports
LLM06 Excessive Agency
OWASP Top 10 for Large Language Model Applications · supports
C9.1 Execution Budgets, Loop Control, and Circuit Breakers
OWASP AI Application Security Verification Standard (AISVS) · supports
C9.2 High-Impact Action Approval and Irreversibility Controls
OWASP AI Application Security Verification Standard (AISVS) · supports
C9.3 Component Isolation and Tool Authorization
OWASP AI Application Security Verification Standard (AISVS) · supports
C9.4 Agent and Orchestrator Identity
OWASP AI Application Security Verification Standard (AISVS) · partial
C9.5 Agent Authorization, Delegation, and Continuous Enforcement
OWASP AI Application Security Verification Standard (AISVS) · supports
C9.6 Shutdown and Graceful Degradation
OWASP AI Application Security Verification Standard (AISVS) · supports
L3 Agent Frameworks
CSA MAESTRO (Multi-Agentic Threat Model) · supports
L7 Agent Ecosystem
CSA MAESTRO (Multi-Agentic Threat Model) · supports
reasoning-collapse Reasoning Collapse
CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with
emergent-covert-coordination Emergent Covert Coordination
CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with
heterogeneous-multi-agent-exploits Heterogeneous Multi-Agent Exploits
CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with
goal-drift Goal Drift in Delegated Chains
CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with
trust-misuse Trust Misuse Between Legitimate Agents
CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with
Generative AI Lens Generative AI Lens (themes)
AWS Well-Architected Framework · aligns-with
Future evolution
Interoperable agent identity and capability attestation standards for A2A ecosystems; shared sandbox/sim scorecards and portable RACI/inventory exports for assessments.