Guide · Agents
Agent Autonomy Limits and Kill Switches
Unbounded agents loop, spend, and escalate. Production agents need charters, autonomy caps, and a real kill switch—not hope.
Unbounded agents fail Agent Governance—charters, budgets, and kill switches are mandatory in production. Primary control: Agent Governance
Charter before code
Write one page per agent class before it touches production tools:
- Purpose — what "done" means
- Allowed tools — names, not "whatever MCP is configured"
- Data classes — what it may read/write
- Forbidden actions — e.g. prod deploy, payment capture, mass email
- Budgets — max steps, wall time, token/spend ceiling
If you cannot fill that page, you are not ready for Agent Governance—you have a demo with credentials.
Runtime bounds beat prompt manners
Caps belong in the orchestrator: step counter, timeout, spend meter, checked every turn. "Please don't loop" in the system prompt is not a control. Pair with Tool Safety allowlists and separate identities per agent class so a summarizer never shares an admin token.
A coding agent with shell + `git push` in CI, no max steps, and a README injection once opened a storm of PRs. The only interrupt was revoking the GitHub app. That is the failure mode you design against.
Kill switch drill
A kill switch is a control-plane flag (or queue drain) that:
1. Stops scheduling new steps immediately
2. Cancels in-flight tool calls where the platform allows
3. Is reachable by on-call without a redeploy
Test it in staging the way you test failover. Document the runaway-agent runbook next to infra SEVs. Ambiguous or high-impact plans should already escalate to Human Approval—kill switch is for when autonomy was wrong, not for every button click.
Assess against APRF Core
Run the Core Profile quiz — gated pass/fail blockers for AI production readiness, not a vanity score.
Related
Frequently asked questions
- Is a kill switch just "turn off the feature flag"?
- That is the minimum. Prefer a flag checked every agent step plus cancellation of in-flight tool calls where possible.
- How many steps should an agent get?
- Start low for production (tens, not thousands). Raise only with eval evidence and monitoring.
- Do chatbots need this?
- Single-turn chat needs less. Anything with tools, loops, or A2A needs Agent Governance controls.