← Domains & pillars

Agents & AutonomyView domain

APRF-06

Agent Governance

Constrain goals, autonomy, loops, and escalation for agents and A2A.

Purpose

Define and enforce how autonomous agents set goals, plan, loop, escalate, and collaborate with other agents so autonomy never exceeds authorized operational bounds.

Mandatory checks

Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.

  • AGN-M1L3 · DefinedTier 2 · ProductionhybridE3

    Each production agent (any autonomous or semi-autonomous runtime that plans, loops, or invokes tools on behalf of a product capability—including in-process agents, MCP clients/servers acting as agents, and A2A participants) shall be represented in a version-controlled inventory with a current charter covering purpose, owner, production identifier, lifecycle status, approved tool policy, data scope, autonomy boundaries, change control, review metadata, and structured approval. Runtime configuration shall be consistent with the approved charter; material deviations require charter review before deployment. Applies to deployed production agent runtimes, not to agent frameworks, SDKs, or libraries in isolation.

    Artifact
    Version-controlled agent inventory listing every production agent with owner, production identifier, lifecycle status, and charter URI/id + Per-agent charter documents (or equivalent structured records) covering purpose, owner, production identifier, lifecycle status, approved tool policy reference, data scope, autonomy boundaries, change justification or revision history, review date, last updated, charter version, and structured approval (approvedBy, approvalDate, approvalStatus) + Inventory completeness evidence: runtime registry, deployment manifest, CMDB, platform registry, or approved attestation with measuredAt ≤90 days
    Pass condition
    Every production agent is represented in a version-controlled inventory. Each inventory entry references a current charter containing purpose, owner, production identifier (environment, system, application, or deployment id), lifecycle status (Active, Deprecated, Retired, or Experimental), approved tool policy, data scope, autonomy boundaries, change justification or revision history, review date, last updated, charter version, and structured approval (approvedBy, approvalDate, approvalStatus). Inventory completeness is demonstrated by at least one of: runtime registry, deployment manifest, CMDB, platform registry, or approved attestation (coversAllProductionAgents with measuredAt ≤90 days). Named exceptions include business justification, approver, expiry ≤90 days, and compensating controls. No production agent is missing required governance metadata. Runtime configuration is consistent with the approved charter.

    Why this control exists

    Threat map

    Ensure every production agent has an approved, documented operating boundary so no agent runs unaccounted for.

    Threats mitigated

    Shadow AgentsExcessive AgencyUnauthorized Tool Use

    Protects

    RuntimeToolsSafety

    MITRE: ATLAS AML.T0103

    A maintained charter and inventory is what makes an unapproved or forgotten agent detectable, and it is the precondition for every downstream agent control. The mapping is detective: it surfaces adversary-deployed or undocumented agents rather than preventing their deployment.

    Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.

  • AGN-M2L3 · DefinedTier 2 · ProductionhybridE3

    Every production agent runtime shall enforce finite execution bounds including reasoning/tool iterations, execution duration, and recursive delegation depth where supported. Bounds shall be enforced by runtime or platform controls rather than prompts or model instructions. Exceeding any bound shall terminate or safely abort execution so the agent cannot continue planning or dispatch further side effects after abort. Applies to agent frameworks, orchestration frameworks, workflow engines with agent loops, MCP clients, MCP servers acting autonomously, A2A runtimes, and multi-agent systems. Not applicable to simple chat-completion APIs, single-inference endpoints, embeddings, classifiers, or rerankers without multi-step / tool / delegation autonomy.

    Artifact
    Runtime/platform configuration declaring finite iteration and duration bounds for each production agent runtime (plus recursion/delegation depth when that capability is supported) + Enforcement test results showing abort/fail-closed when each applicable bound is exceeded, including proof the agent cannot continue execution after abort + Runtime logs (or equivalent telemetry) showing execution terminated due to a configured bound (preferred; measuredAt ≤90 days when used for PASS)
    Pass condition
    Every production agent runtime enforces finite execution bounds appropriate to its execution model. These include bounded reasoning/tool iterations, bounded execution duration, and bounded recursive delegation where supported. Enforcement is implemented by the runtime or platform and verified through tests (and preferably runtime logs) that demonstrate safe termination when each configured bound is exceeded, with no continued planning or background side-effect dispatch after abort. No production agent relies solely on prompts or model instructions to enforce execution limits. Recursion/delegation depth is required when the runtime supports recursive delegation, agent spawning, or sub-agent execution; otherwise that bound is NOT_APPLICABLE for that runtime.

    Why this control exists

    Threat map

    Prevent autonomous agents from executing beyond approved operational limits.

    Threats mitigated

    Excessive AgencyDenial of WalletTool AbuseDenial of Service

    Protects

    RuntimeToolsCostAvailability

    MITRE: ATLAS AML.T0034 · ATLAS AML.T0034.002 · ATLAS AML.T0029

    Hard loop, step, time, and spend ceilings bound the blast radius of runaway autonomous behaviour and recursive self-invocation. They directly frustrate cost-harvesting and agentic resource-consumption attacks, which depend on execution being unbounded, and preserve capacity for other workloads.

    Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.

  • AGN-M3L3 · DefinedTier 2 · ProductionhybridE3

    Authorized operators shall be able to pause or terminate production agent runs through a control-plane action that stops further planning and in-flight or queued tool/MCP/A2A side effects within a documented time-to-effect SLO. The control must not depend on the model cooperating and must not be disableable by the agent runtime itself.

    Artifact
    Kill-switch/pause runbook: who may invoke, scope (run / agent type / fleet), expected effect, and numeric time-to-effect SLO + Successful drill or production use record (≤90 days) with timestamps for invoke → effect, covering at least one production agent path + Cancellation suite or import proving queued work, running tasks, and child agents stop without new dispatch
    Pass condition
    Documented pause/terminate control exists for production agents; control is restricted to authorized operators (not end-user-only and not model-invoked); ≥1 successful pause or terminate action in drill or production in the last 90 days shows recorded time-to-effect ≤ documented numeric SLO; evidence shows further tool/MCP/A2A side effects for that run (and its spawned children, if any) were stopped or drained without requiring the model to refuse.

    Why this control exists

    Threat map

    Give operators a reliable means to halt agent execution when it behaves unsafely or is under attack.

    Threats mitigated

    Agent HijackingExcessive AgencyTool AbuseDenial of Wallet

    Protects

    RuntimeToolsCostAvailabilitySafety

    MITRE: ATLAS AML.T0053 · ATLAS AML.T0034.002

    A tested kill switch converts an ongoing compromise into a bounded incident by stopping tool invocation and resource consumption mid-run. It is a response control, so it limits impact and duration rather than preventing initial access.

    Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.

  • AGN-M4L3 · DefinedTier 2 · ProductionhybridE3

    Every production agent-to-agent (A2A) or multi-agent handoff path shall authenticate the peer identity and authorize a scoped capability set (tools, data, actions) independent of network presence. Unauthenticated, spoofed, or over-scoped handoffs shall be denied by the platform—not by prompt instructions alone.

    Artifact
    Inventory of production A2A/multi-agent handoff paths with authn/authz mechanism per path + Capability token/schema (or equivalent) examples showing scoped grants for handoffs + Negative test results: unauthenticated, forged identity, and over-scoped capability cases denied at 100%
    Pass condition
    100% of production A2A/multi-agent handoff paths require authenticated peers and scoped capabilities; negative tests show unauthenticated, forged-peer, and over-scoped handoffs denied at 100%; 0 production handoff paths that accept anonymous or network-presence-only trust (deny evidence measuredAt ≤90 days). If no production A2A or multi-agent handoffs exist, score NOT_APPLICABLE.

    Why this control exists

    Threat map

    Ensure agent-to-agent handoffs authenticate the peer and carry only scoped, least-privilege capabilities.

    Threats mitigated

    Identity SpoofingAgent HijackingPrivilege EscalationUnauthorized Tool Use

    Protects

    IdentityToolsExternal SystemsRuntime

    MITRE: ATLAS AML.T0073 · ATLAS AML.T0012 · ATT&CK T1078

    Authenticating the peer agent prevents an attacker-controlled or spoofed agent from being trusted inside a workflow. Scoping the delegated capability stops privilege from accumulating as a task is handed between agents.

    Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.

Evidence required

  • Versioned agent inventory with charters and owners (AGN-M1)
  • Runtime max-steps / wall-clock / spawn-depth config + abort-on-exceed evidence (AGN-M2)
  • Kill-switch/pause runbook, cancellation suite, and ≤90-day drill record (AGN-M3)
  • A2A/multi-agent handoff inventory with peer auth, scoped capabilities, and deny tests (AGN-M4)
  • Goal-conflict / disallowed-goal plan policy with named owner and synthetic deny traces (AGN-R1)
  • Agent sandbox/simulation environment config and linked ≤30-day pre-release sim reports (AGN-R2)
  • Agent/AI-system RACI register with Responsible + Accountable and 0 orphans (AGN-R3)
More detailPhilosophy, failures, practices, validations, examples, crosswalks, and evolution

Engineering philosophy

Autonomy is a dial, not a virtue. Production agents have explicit charters: allowed goals, max steps, escalation paths, and kill switches. Multi-agent and A2A systems require governance of the mesh, not only of each node.

Why it matters

Unconstrained agents loop, spend unbounded budget, escalate privileges across A2A links, or pursue goals that conflict with business and safety policy. Governance failures look like 'the agent went rogue' but are usually missing product controls.

Common failures

  • No maximum step, wall-clock, or spawn-depth budget for agent loops
  • Agents that can redefine or pursue conflicting goals without a pre-tool policy gate
  • A2A trust assumed by network presence alone (no peer auth or scoped capabilities)
  • No operator kill switch, cancellation of in-flight/queued work, or drill evidence
  • Agent behavior promotions without a linked sandbox/simulation run
  • Orphan agent or AI-system IDs without Responsible/Accountable ownership

Severity & risk

Severity
critical
Impact if violated
Risk level
critical
Typical residual risk (impact × likelihood)

Engineering best practices

  • Prefer narrow agents composed into workflows over one universal agent
  • Log plans and intermediate decisions for reconstructability
  • Escalate to humans on ambiguity or high-impact branches
  • Treat agent-to-agent messages as untrusted until verified
  • Gate agent-behavior promotions on a linked sandbox/sim report ≤30 days before release
  • Keep a versioned RACI with Responsible and Accountable for every production agent ID

Automatic validations

  • Runtime enforcement of step, wall-clock, and spawn-depth limits with abort-on-exceed
  • Denial of unauthenticated, forged-peer, and over-scoped A2A handoffs
  • Kill-switch cancellation suite (queued, running, and child-agent paths)
  • Goal-conflict / disallowed-goal policy gate before side-effecting tools
  • Alerts on loop detection and repeated identical tool calls

Manual validations

  • Charter and inventory completeness review for each production agent class
  • Architecture review that agents cannot disable their own kill path
  • RACI ownership review across teams for production AI systems and agents
  • Tabletop of multi-agent failure cascades and kill-switch drill within SLO

Examples

  • A research agent may browse and summarize but cannot send email; a separate approved workflow handles outreach
  • A2A task delegation includes expiring capability tokens, not open network trust
  • On-call invokes an authenticated pause API; queued tools cancel and child agents terminate within the time-to-effect SLO
  • A synthetic goal-conflict fixture is denied by plan policy before any write tool runs

References

Crosswalks

Show 17 peer mappingsexpand
  • MANAGE Manage

    NIST AI Risk Management Framework · supports

  • §8 Operation

    ISO/IEC 42001 · supports

  • LLM06 Excessive Agency

    OWASP Top 10 for Large Language Model Applications · supports

  • C9.1 Execution Budgets, Loop Control, and Circuit Breakers

    OWASP AI Application Security Verification Standard (AISVS) · supports

  • C9.2 High-Impact Action Approval and Irreversibility Controls

    OWASP AI Application Security Verification Standard (AISVS) · supports

  • C9.3 Component Isolation and Tool Authorization

    OWASP AI Application Security Verification Standard (AISVS) · supports

  • C9.4 Agent and Orchestrator Identity

    OWASP AI Application Security Verification Standard (AISVS) · partial

  • C9.5 Agent Authorization, Delegation, and Continuous Enforcement

    OWASP AI Application Security Verification Standard (AISVS) · supports

  • C9.6 Shutdown and Graceful Degradation

    OWASP AI Application Security Verification Standard (AISVS) · supports

  • L3 Agent Frameworks

    CSA MAESTRO (Multi-Agentic Threat Model) · supports

  • L7 Agent Ecosystem

    CSA MAESTRO (Multi-Agentic Threat Model) · supports

  • reasoning-collapse Reasoning Collapse

    CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with

  • emergent-covert-coordination Emergent Covert Coordination

    CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with

  • heterogeneous-multi-agent-exploits Heterogeneous Multi-Agent Exploits

    CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with

  • goal-drift Goal Drift in Delegated Chains

    CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with

  • trust-misuse Trust Misuse Between Legitimate Agents

    CSA MAESTRO (Multi-Agentic Threat Model) · aligns-with

  • Generative AI Lens Generative AI Lens (themes)

    AWS Well-Architected Framework · aligns-with

Future evolution

Interoperable agent identity and capability attestation standards for A2A ecosystems; shared sandbox/sim scorecards and portable RACI/inventory exports for assessments.