Cost Optimization

Production AI workloads shall enforce finite hard spend ceilings and/or request rate limits (provider, gateway, or application) that deny or throttle when exceeded—not soft dashboards or prompt-only budgets alone.

Artifact

Gateway/provider/application config declaring finite spend ceiling and/or rate limit (TPM/RPM/$, tokens) + Enforcement evidence: automated exceed test or ≤90-day production deny/throttle event log

Pass condition

A finite hard spend ceiling and/or rate limit is configured for production AI workloads; enforcement demonstrably denies or throttles when the limit is exceeded (automated test or production event log measuredAt ≤90 days). If no production AI/model/agent traffic exists, score NOT_APPLICABLE.

Production AI spend shall be monitored with alert policies for budget burn and spend anomaly that notify operators per runbook—not dashboards alone.

Artifact

Cost/spend dashboard or telemetry views covering production AI workloads + Alert policies for budget burn and spend anomaly (IaC, provider, or gateway) + Notify proof: alert test or documented fire ≤90 days with page/notify outcome

Pass condition

Alerts exist for both budget burn and spend anomaly covering production AI spend; a synthetic or historical burn/anomaly event would page/notify per runbook (alert test or documented fire measuredAt ≤90 days). If no production AI spend exists, score NOT_APPLICABLE.

Production AI clients shall enforce finite retry/backoff and loop budgets so forced failures cannot amplify completions without bound—covering HTTP/SDK clients and agent loops, not prompt-only “try again” guidance.

Artifact

Retry/backoff config (finite maxRetries + backoff) for production AI/model clients + Agent or client loop budget config where applicable (finite iterations/steps) + Amplification test results: forced failure/retry hits a bounded token or $ ceiling (≤90 days)

Pass condition

Max retries and max agent/client loops are finite for 100% of production AI clients; amplification tests show cost cannot grow without bound under forced failure/retry (bounded token or $ ceiling; test measuredAt ≤90 days). If no production AI/model clients exist, score NOT_APPLICABLE.

Production AI systems should cache prompt/response results for documented idempotent or repeated paths, with explicit exclusions for sensitive or personalized prompts, and report hit-rate and savings for ≥30 days.

Artifact

Prompt/response cache config naming idempotent paths and safety exclusions + Hit-rate and savings report covering ≥30 days (import or dashboard export)

Pass condition

Cache is enabled for documented idempotent/repeated prompt paths; sensitive/personalized prompts are excluded; hit-rate and savings are reported for a window of ≥30 days (report measuredAt within the last 90 days). If no production AI traffic with potentially cacheable prompts exists, score NOT_APPLICABLE.

Production AI systems should route documented low-risk task classes to cheaper models by default, with eval coverage proving quality within tolerance of a premium baseline and misroute rate monitored ≤30 days.

Artifact

Model-routing policy mapping low-risk task classes to cheap vs premium models + Eval coverage report for low-risk routed tasks vs premium baseline + Misroute-rate monitoring evidence covering ≤30 days

Pass condition

Documented low-risk task classes route to cheaper models by default; eval shows quality within tolerance versus a premium baseline; misroute rate is monitored for a window of ≤30 days (report measuredAt within the last 90 days). If a single fixed model with no routing surface, score NOT_APPLICABLE.

Each customer-facing AI product should publish unit-cost metrics (e.g. cost per successful task or journey) for the last quarter and undergo a FinOps review ≤90 days with named owners for outliers above documented thresholds.

Artifact

Per-product AI unit-economics report (cost per successful task / journey) for the last quarter + FinOps review minutes ≤90 days with outlier owners and thresholds

Pass condition

Each customer-facing AI product has unit-cost metrics for the last quarter; a FinOps review occurred within the last 90 days with named owners for outliers above documented thresholds (review measuredAt ≤90 days). If no customer-facing AI products exist, score NOT_APPLICABLE.