Purpose
Define and enforce how autonomous agents set goals, plan, loop, escalate, and collaborate with other agents so autonomy never exceeds authorized operational bounds.
Engineering philosophy
Autonomy is a dial, not a virtue. Production agents have explicit charters: allowed goals, max steps, escalation paths, and kill switches. Multi-agent and A2A systems require governance of the mesh, not only of each node.
Why it matters
Unconstrained agents loop, spend unbounded budget, escalate privileges across A2A links, or pursue goals that conflict with business and safety policy. Governance failures look like 'the agent went rogue' but are usually missing product controls.
Common failures
- No maximum step or time budget for agent loops
- Agents that can redefine their own goals without oversight
- A2A trust assumed by network presence alone
- No kill switch or pause for runaway agents
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Each production agent shall have a documented charter: purpose, allowed tools, data scope, and autonomy limits
- Artifact
- Agent inventory with charter fields per production agent
- Pass condition
- 100% of production agents have a charter with purpose, tool allowlist reference, data scope, and autonomy limits; inventory query returns 0 agents missing any required field
Hard limits shall exist on steps, wall-clock time, and recursive agent spawning
- Artifact
- Runtime config for max steps, wall-clock timeout, and spawn depth + enforcement tests
- Pass condition
- 100% of production agents have finite max-steps, wall-clock timeout, and spawn-depth ≤ configured bound; tests show limits are enforced (run aborts) when exceeded
Operators shall be able to pause or terminate agent runs in production
- Artifact
- Kill-switch/pause control documentation + successful drill or production use record
- Pass condition
- Documented pause/terminate control exists for production agents; ≥1 successful pause or terminate action in drill or production in last 90 days with recorded time-to-effect ≤ documented SLO
A2A and multi-agent handoffs shall authenticate peers and carry scoped capabilities
- Artifact
- A2A auth config + capability-token/schema examples + negative auth tests
- Pass condition
- 100% of production A2A/multi-agent handoff paths require authenticated peers; tests show unauthenticated or over-scoped handoffs denied at 100%
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Goal conflict detection and policy checks before plan execution
- Artifact
- Pre-execution policy checks for agent plans (goal-conflict / disallowed-goal rules) + sample deny/allow traces
- Pass condition
- Agent planner runs policy checks before side-effecting tools; ≥1 synthetic conflict is denied in test or prod logs within 90 days; rules have a named owner
Simulation/sandbox runs for new agent behaviors before production
- Artifact
- Sandbox/simulation environment config for agents + last pre-prod simulation report for a behavior change
- Pass condition
- Last new agent behavior promoted to production has a linked sandbox run ≤30 days before release with pass/fail criteria recorded
Formal RACI for agent ownership across teams
- Artifact
- RACI or ownership register entry for the system
- Pass condition
- Every production AI system ID has non-empty owner fields for required domains; inventory query returns 0 orphans
Evidence required
- Agent inventory with charters and owners
- Loop/budget configuration evidence
- Kill-switch runbook and test record
Severity & risk
- Severity
- critical
- Impact if violated
- Risk level
- critical
- Typical residual risk (impact × likelihood)
Engineering best practices
- Prefer narrow agents composed into workflows over one universal agent
- Log plans and intermediate decisions for reconstructability
- Escalate to humans on ambiguity or high-impact branches
- Treat agent-to-agent messages as untrusted until verified
Automatic validations
- Runtime enforcement of step and cost budgets
- Denial of unauthorized A2A peers
- Alerts on loop detection and repeated identical tool calls
Manual validations
- Charter review for each new agent class
- Tabletop of multi-agent failure cascades
Examples
- A research agent may browse and summarize but cannot send email; a separate approved workflow handles outreach
- A2A task delegation includes expiring capability tokens, not open network trust
References
Crosswalks
MANAGE Manage
NIST AI Risk Management Framework · supports
§8 Operation
ISO/IEC 42001 · supports
LLM06 Excessive Agency
OWASP Top 10 for Large Language Model Applications · supports
Generative AI Lens Generative AI Lens (themes)
AWS Well-Architected Framework · aligns-with
Future evolution
Interoperable agent identity and capability attestation standards for A2A ecosystems.