Purpose
Keep AI spend predictable and attributable through budgets, caching, model routing, quota enforcement, and abuse prevention.
Engineering philosophy
Cost is a reliability and security property. Unbounded token spend is a denial-of-wallet attack surface and a business continuity risk.
Why it matters
Agent loops, recursive tool use, and prompt bloat can create sudden five-figure bills. Without controls, AI features become financially unsafe to operate.
Common failures
- No per-tenant or per-feature budgets
- Retry storms that multiply completions
- Sending huge contexts when retrieval would suffice
- Using frontier models for trivial classification tasks
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Production AI workloads shall have hard spend or rate limits
- Artifact
- Gateway/provider quota config
- Pass condition
- Enforced limit demonstrably denies or throttles when exceeded (automated test or production event log within last 90 days)
Cost shall be monitored with alerts on anomaly and budget burn
- Artifact
- Cost dashboard + alert policies for anomaly and budget burn
- Pass condition
- Alerts exist for budget burn and spend anomaly; synthetic or historical burn event would page/notify per runbook (alert test or documented fire in last 90 days)
Retry and loop policies shall prevent unbounded cost amplification
- Artifact
- Retry/backoff and agent-loop budget config + amplification tests
- Pass condition
- Max retries and max agent loops are finite for 100% of production AI clients; tests show cost cannot grow without bound under forced failure/retry (bounded token or $ ceiling hit)
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Caching for idempotent or repeated prompts where safe
- Artifact
- Prompt/response cache config with safety exclusions + hit-rate report for 30 days
- Pass condition
- Cache enabled for documented idempotent paths; sensitive/personalized prompts are excluded; hit-rate and savings reported for ≥30 days
Model routing: cheaper models for low-risk tasks with eval coverage
- Artifact
- Model-routing policy (cheap vs premium) + eval coverage report for low-risk routed tasks
- Pass condition
- Low-risk task classes route to cheaper models by default; eval shows quality within tolerance vs premium baseline; misroute rate monitored ≤30 days
FinOps reviews of AI unit economics per product
- Artifact
- Per-product AI unit-economics report (cost per successful task / journey) + FinOps review minutes
- Pass condition
- Each customer-facing AI product has unit-cost metrics for the last quarter; review occurred ≤90 days with owners for outliers above documented thresholds
Evidence required
- Budget and quota configuration
- Cost dashboards and alert policies
- Documented retry/loop limits
Severity & risk
- Severity
- high
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Attribute cost to tenant and feature tags
- Prefer embeddings + retrieval over stuffing large corpora
- Set timeouts on model and tool calls
- Load-test cost under adversarial long prompts
Automatic validations
- Gateway enforcement of quotas
- Anomaly detection on token spend
- CI checks estimating token cost of prompt changes
Manual validations
- Quarterly cost architecture review
- Abuse scenario walkthroughs
Examples
- A free-tier chatbot hard-stops after N tokens/day per user
- An agent aborts when projected step cost exceeds remaining budget
References
Crosswalks
§6 Planning
ISO/IEC 42001 · partial
LLM10 Unbounded Consumption
OWASP Top 10 for Large Language Model Applications · supports
Cost Optimization Cost Optimization
AWS Well-Architected Framework · supports
Sustainability Sustainability
AWS Well-Architected Framework · partial
Future evolution
Standard cost telemetry schemas for multi-provider AI gateways.