APRF-14 · Reliability & Continuity

Purpose

Design graceful degradation for short failures and documented continuity options for sustained disruption—so AI-dependent journeys fail safely and recover within defined RTO/RPO.

Mandatory checks

Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.

REL-M1L3 · DefinedTier 2 · ProductionhybridE3
Every production model and tool client call site shall declare a finite timeout and a finite max-retry bound—verified by static analysis or integration test—so hung providers and retry storms cannot cascade.
Artifact
Client config or code declaring finite timeout and max-retry for model/tool call sites + Static analysis report or integration test covering 100% of in-scope call sites
Pass condition
100% of production model/tool client call sites have a finite timeout and a finite max-retry bound; verified by static check or integration test (measuredAt ≤90 days). If no production model or tool clients exist, score NOT_APPLICABLE.
Why this control exists
Threat map
Prevent a slow or failing dependency from exhausting capacity or amplifying cost.
Threats mitigated
Denial of ServiceDenial of WalletExcessive Agency
Protects
AvailabilityCostRuntime
MITRE: ATLAS AML.T0029 · ATLAS AML.T0034.001
Without timeouts, a slow provider consumes connections and workers until the whole service fails; without retry bounds, one failure multiplies into many billable calls. Both limits cap the amplification available to an attacker who can induce failures, as well as to ordinary defects.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-M2L3 · DefinedTier 2 · ProductionhybridE3
Every critical user journey that depends on AI shall document degraded behavior when the AI dependency fails, and a failover test shall show a non-AI or safe fallback activates—not a blank error or silent hang.
Artifact
Degraded-mode specification covering each critical AI-dependent journey + Feature-flag, circuit-breaker, or fallback test proving safe activation when AI fails
Pass condition
100% of critical AI-dependent journeys document degraded behavior; a failover test shows non-AI or safe fallback activates when the AI dependency fails (measuredAt ≤90 days). If no critical AI-dependent user journeys exist, score NOT_APPLICABLE.
Why this control exists
Threat map
Keep critical user journeys functional when the AI path is unavailable.
Threats mitigated
Denial of Service
Protects
AvailabilityUsers
MITRE: ATLAS AML.T0029
A defined degraded mode keeps the journey working when the model path is down, whether from a provider outage or a denial-of-service attempt. It reduces the impact of the disruption rather than preventing it.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-M3L3 · DefinedTier 2 · ProductionhybridE3
Tool-using AI workflows shall detect partial tool failures (success then failure mid-sequence, truncated results, or failed side effects) and must not report overall success without explicit compensation, rollback, retry, or human approval.
Artifact
Agent/tool error-handling policy or code that detects partial tool failure + Test evidence (integration, chaos, e2e, replay, contract, simulator, or equivalent) showing no false-success without compensation/rollback/retry/approval
Pass condition
Test evidence demonstrates that when a tool partially succeeds and a later step fails, the workflow does not report overall success without explicit compensation, rollback, retry, or human approval (measuredAt ≤90 days). If no tool-using agents or workflows exist, score NOT_APPLICABLE.
Why this control exists
Threat map
Prevent tool-using workflows from reporting success when steps have silently failed.
Threats mitigated
Hallucinated ActionsMisinformationExcessive Agency
Protects
External SystemsDataUsersSafety
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Agents routinely narrate success for tool calls that actually errored, leaving downstream state inconsistent and users misinformed. Verifying real tool outcomes closes that false-success path; the failure mode is correctness rather than an adversary technique.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-M4L3 · DefinedTier 2 · ProductionhybridE3
Backup inventories shall include the AI control-plane artifacts required to restore service (prompt registry, policies, indexes, and other declared dependencies), and a restore test of a sample artifact set shall succeed within 90 days.
Artifact
Backup job or inventory covering required AI control-plane artifacts (prompt registry, policies, indexes, as applicable) + Restore test report for a sample artifact set
Pass condition
Backup inventory includes required AI control-plane artifacts; a restore test in the last 90 days succeeds for a sample artifact set (measuredAt ≤90 days). If no AI control-plane artifacts exist in scope (prompt registry, policies, vector/index state, or equivalent), score NOT_APPLICABLE.
Why this control exists
Threat map
Ensure the artifacts required to restore AI service survive a destructive event.
Threats mitigated
Denial of ServiceData and Model PoisoningSupply Chain Compromise
Protects
AvailabilityModelsPromptsData
MITRE: ATLAS AML.T0076 · ATT&CK T1485 · ATT&CK T1486
Prompts, model pins, tool catalogs, and indexes commonly fall outside conventional backup scope, which leaves the AI service unrecoverable even when the database is restored. Backing them up provides the recovery path after model corruption, data destruction, or ransomware.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-M5L4 · Quantitatively ManagedTier 3 · Mission CriticalhybridE3
Every business-critical AI service must have numeric RTO and RPO documented in continuity, service-catalog, or disaster-recovery materials and linked to a tested restore or failover procedure.
Artifact
Business continuity, service-catalog, or disaster-recovery documentation with numeric RTO and RPO for business-critical AI services + Linkage from each in-scope service to a tested restore/failover procedure
Pass condition
100% of business-critical AI services have numeric RTO and RPO documented and linked to a tested restore/failover procedure (continuity evidence measuredAt ≤90 days). If no business-critical AI services are in scope, score NOT_APPLICABLE.
Why this control exists
Threat map
Set explicit recovery time and recovery point objectives for business-critical AI services.
Threats mitigated
Denial of Service
Protects
Availability
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
RTO and RPO targets are what make backup and failover design decisions testable rather than aspirational. This is continuity planning with no adversary technique mapping.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.

Evidence required

Timeout/retry configuration
Degraded-mode UX and behavior documentation
Postmortems referencing resilience controls where applicable
Continuity plans for critical journeys
Backup/restore test records
RTO/RPO definitions

Recommended checks

Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.

REL-R1L4 · Quantitatively ManagedTier 2 · ProductionhybridE3
Production AI/provider clients should isolate failures with circuit breakers (open under sustained failure) and bulkheads (finite concurrency per dependency)—so one slow or down provider cannot exhaust the whole fleet.
Artifact
Client library/config showing circuit breakers around model/provider calls + Bulkhead or concurrency limit per provider dependency + Breaker trip evidence: induced-failure test or production trip log
Pass condition
Circuit breakers are configured around AI/provider clients and open under induced failure in test or an observed production trip within 90 days; bulkheads limit concurrent calls per provider dependency (measuredAt ≤90 days). If no production AI/provider clients exist, score NOT_APPLICABLE.
Why this control exists
Threat map
Stop a failing AI provider from cascading into total service failure.
Threats mitigated
Denial of Service
Protects
AvailabilityRuntime
MITRE: ATLAS AML.T0029
Circuit breakers cut off a failing dependency before it consumes the caller's resources, and bulkheads stop one workload from exhausting shared client capacity. Together they limit the reach of both outages and resource-exhaustion abuse.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-R2L4 · Quantitatively ManagedTier 2 · ProductionhybridE3
Production AI paths should have a configured multi-provider or multi-region fallback that was exercised within 90 days, with an eval comparing primary vs fallback quality and safety—not connectivity alone—so failover does not silently degrade outcomes.
Artifact
Multi-provider or multi-region fallback configuration + Evidence the fallback path was exercised ≤90 days + Eval report comparing primary vs fallback quality/safety against minimum bars
Pass condition
A multi-provider or multi-region fallback path is configured and was exercised ≤90 days; a retained eval comparing primary vs fallback meets documented minimum quality/safety bars (not connectivity-only; measuredAt ≤90 days). If no production AI paths that warrant fallback are in scope, score NOT_APPLICABLE.
Why this control exists
Threat map
Remove single-provider dependency for critical AI journeys without lowering the quality bar.
Threats mitigated
Denial of Service
Protects
AvailabilityUsers
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Provider-level failover eliminates a single point of failure, but only counts if the fallback path is evaluated to the same standard as the primary. Fallback routing must also respect residency constraints or it becomes a compliance exposure in its own right.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-R3L3 · DefinedTier 2 · ProductionhybridE3
Every process marked critical-AI-dependent should have at least one documented continuity option (failover, manual procedure, alternate provider, or equivalent) with a named owner—so operators know how to keep the process running when the primary AI path fails.
Artifact
Continuity options documentation per critical AI-dependent process (failover, manual, alternate provider, or equivalent) + Named owner for each continuity option
Pass condition
100% of processes marked critical-AI-dependent have ≥1 documented continuity option with a named owner (continuity evidence measuredAt ≤90 days). If no processes marked critical-AI-dependent exist, score NOT_APPLICABLE.
Why this control exists
Threat map
Document how critical AI-dependent business processes continue when AI is unavailable.
Threats mitigated
Denial of Service
Protects
Availability
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Processes that quietly became AI-dependent have no manual fallback until someone writes one down. Documented continuity options address that; no adversary technique maps.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-R4L4 · Quantitatively ManagedTier 2 · ProductionhybridE3
Production AI systems should run periodic continuity drills that include provider loss (or equivalent), measure RTO/RPO outcomes, and retain a report—so recovery objectives are proven under rehearsal, not assumed from documents alone.
Artifact
Continuity drill calendar covering provider-loss (or equivalent) scenarios + Last provider-loss drill report with RTO/RPO results (≤90 days) + Named owner + expiry for any RTO/RPO miss
Pass condition
A provider-loss (or equivalent) continuity drill completed ≤90 days with retained RTO/RPO results that met objectives or have named owners and expiry (measuredAt ≤90 days). If no production AI provider dependencies exist, score NOT_APPLICABLE.
Why this control exists
Threat map
Rehearse continuity, including full provider loss, before it is tested by a real outage.
Threats mitigated
Denial of Service
Protects
Availability
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Failover paths that are never exercised routinely fail on first real use. Drills validate them; this is readiness assurance with no adversary technique mapping.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-R5L5 · OptimizingTier 3 · Mission CriticalhybridE3
Production systems should run chaos experiments that cover AI dependency failure modes (provider, model, tool, or gateway outages) and retain an after-action with tracked actions.
Artifact
Chaos experiment plan covering AI provider/tool/gateway failure modes + Dated after-action report with retained actions
Pass condition
At least one AI-dependency chaos exercise completed in the last 180 days with retained after-action actions (attest measuredAt ≤90 days). If no production AI dependencies exist, score NOT_APPLICABLE.
Why this control exists
Threat map
Deliberately exercise AI dependency failure modes in production-like conditions.
Threats mitigated
Denial of Service
Protects
AvailabilityRuntime
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Chaos experiments reveal timeout, retry, and fallback defects that only appear under real failure. This is resilience assurance with no adversary technique mapping.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-R6L5 · OptimizingTier 3 · Mission CriticalhybridE3
Where critical workloads depend on self-hosted inference, a warm standby should exist with capacity for declared peak load and a recent failover test that met documented RTO—so self-hosted model loss does not strand the service.
Artifact
Warm-standby inference architecture for self-hosted critical workloads + Last failover test report with RTO result (≤90 days) + Standby capacity sizing covering declared peak
Pass condition
For in-scope self-hosted inference, failover to warm standby completed within documented RTO in a test ≤90 days, and standby capacity covers declared peak for critical workloads (measuredAt ≤90 days). If self-hosted inference does not serve critical workloads, score NOT_APPLICABLE.
Why this control exists
Threat map
Keep standby inference capacity ready where self-hosted models are business-critical.
Threats mitigated
Denial of Service
Protects
AvailabilityInfrastructure
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Self-hosted accelerator capacity cannot be provisioned quickly during an incident, so cold recovery can take hours. Warm standby shortens that; no adversary technique maps.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.
REL-R7L5 · OptimizingTier 3 · Mission CriticalhybridE3
Level-5 workloads should have a documented alternate provider or path (contractual and technical) and a successful failover test within 180 days.
Artifact
Provider contract summary or equivalent documenting alternate provider/path options + Technical failover design + successful failover test evidence
Pass condition
Level-5 workloads have a documented alternate provider/path and a successful failover test ≤180 days (attest measuredAt ≤90 days). If no Level-5 / mission-critical AI workloads are in scope, score NOT_APPLICABLE.
Why this control exists
Threat map
Back multi-provider continuity with contractual commitments, not just technical routing.
Threats mitigated
Denial of ServiceSupply Chain Compromise
Protects
AvailabilityModels
MITRE: no technique mapped — this control addresses governance or assurance rather than a specific adversary technique.
Technical failover does not help when the alternate provider will not supply capacity, or when a vendor withdraws a model line entirely. Contractual options address provider withdrawal and capacity denial; no adversary technique maps.
Informative threat context — mappings reduce exposure and do not guarantee mitigation; not certification.

More detailPhilosophy, failures, practices, validations, examples, crosswalks, and evolution

Engineering philosophy

Assume providers fail. Timeouts, circuit breakers, and degraded modes handle minutes; backups, failover, and human procedures handle hours and days. Safety properties must hold in every degraded mode.

Why it matters

Naive retries and single-provider dependency turn rate limits into outages; automation without continuity plans leaves the business stranded when AI is unavailable.

Common failures

No timeout on model calls
Infinite retries that worsen rate limits
No fallback when the primary model is unavailable
Agent continues after a failed critical tool as if it succeeded
No manual fallback for AI-automated workflows
Single provider with no contractual or technical alternative
Backups that exclude vector indexes and prompt registries
Untested continuity plans

Severity & risk

Severity: high; Impact if violated
Risk level: high; Typical residual risk (impact × likelihood)

Engineering best practices

Idempotent tools where retries are possible
Queue and backpressure for bursty AI workloads
Fail safe: prefer refusal over incorrect action under uncertainty
Separate availability SLOs for AI features vs core non-AI paths
Identify which AI features are critical vs optional
Design human-operated fallback procedures before full automation
Replicate prompt registries and policies, not only databases
Align continuity with resilience and rollback pillars

Automatic validations

Synthetic canaries against providers
Alerts on elevated timeout and 429 rates
Integration tests for fallback paths
Scheduled backup success checks
Restore tests in isolated environments
Health checks on standby providers

Manual validations

Game days simulating provider outage
Review of agent compensation logic after tool failure
Business impact analysis for AI features
Executive review of continuity acceptance risk

Examples

If the LLM provider is down, search falls back to keyword results with a clear banner
An agent aborts a payment flow when the ledger tool times out rather than guessing success
If the LLM provider fails, claims adjusters switch to a documented manual checklist
Prompt registry backups restore a prior known-good configuration in a DR region

References

Crosswalks

MANAGE Manage
NIST AI Risk Management Framework · supports
Secure & Resilient Secure and Resilient
NIST AI Risk Management Framework · aligns-with
LLM10 Unbounded Consumption
OWASP Top 10 for Large Language Model Applications · supports
C9.1 Execution Budgets, Loop Control, and Circuit Breakers
OWASP AI Application Security Verification Standard (AISVS) · supports
CC7 System Operations
SOC 2 Trust Services Criteria · evidence-for
CC9 Risk Mitigation
SOC 2 Trust Services Criteria · evidence-for
A1 Availability
SOC 2 Trust Services Criteria · evidence-for
Reliability Reliability
AWS Well-Architected Framework · aligns-with

Future evolution

Portable health/degradation protocols and multi-provider continuity playbooks for AI workloads.