Purpose
Provide end-to-end observability across prompts, model calls, retrieval, tools, agents, costs, and outcomes so incidents and regressions are diagnosable.
Engineering philosophy
AI systems without reconstructable traces are not operable. Observability must cover the cognitive path, not only HTTP latency—while respecting privacy redaction.
Why it matters
When quality drops or a tool fires incorrectly, teams need to replay what the model saw and did. Blind production AI creates unresolvable incidents and slow learning loops.
Common failures
- Logging only final user-visible answers
- No correlation IDs across model, tool, and app spans
- Storing raw sensitive prompts indefinitely
- Cost metrics disconnected from product features
Mandatory checks
Gate controls. Each check is pass/fail via artifact + pass condition. Expected from the annotated capability level when the system meets the minimum criticality tier.
Distributed traces shall link user request → model calls → tools → outcome
- Artifact
- Sample production traces (redacted) + tracing config
- Pass condition
- Canary or sampled traces show parent request ID linking model spans, tool spans, and outcome in ≥95% of canary requests over 24 hours
Token usage and cost shall be attributable per request/feature/tenant
- Artifact
- Cost/token metrics with request, feature, and tenant labels
- Pass condition
- ≥95% of billed model calls in a 24h sample carry request ID + feature + tenant (or equivalent) attribution labels
Sensitive fields in traces shall be redacted or access-controlled
- Artifact
- Trace redaction/ACL config + synthetic sensitive-field test
- Pass condition
- Synthetic secrets/PII in traced fields are redacted or inaccessible to unauthorized roles at 100% in tests
Recommended checks
Strengthen posture beyond the gate. Same measurable structure; non-blocking unless elevated by organizational policy.
Replay tooling for failed traces in a secure environment
- Artifact
- Secure replay tooling config + sample replay session for a failed production trace (redacted)
- Pass condition
- On-call can replay a failed AI trace in a restricted environment within the documented RTO; last drill or real replay ≤90 days old
Quality annotations attachable to traces for closed-loop eval
- Artifact
- Trace annotation schema + tooling config allowing quality labels on spans + sample annotated traces
- Pass condition
- Annotators can attach quality labels to production traces in a secure tool; ≥50 annotations in the last 90 days feed an eval or review loop
SLO dashboards for AI latency, error, and quality burn rates
- Artifact
- SLO/alert configuration export + dashboard screenshot or URL with retention note
- Pass condition
- Named SLOs with numeric targets exist for each critical AI journey; alert fires when burn exceeds documented threshold
Evidence required
- Example traces (redacted) showing full path
- Cost attribution dashboards
- Access control policy for prompt/completion logs
Severity & risk
- Severity
- high
- Impact if violated
- Risk level
- high
- Typical residual risk (impact × likelihood)
Engineering best practices
- Adopt OpenTelemetry-compatible tracing for LLM spans where possible
- Capture tool names, latency, and status codes consistently
- Separate debug retention from long-term analytics retention
- Alert on silent quality degradation, not only 5xx rates
Automatic validations
- Synthetic checks that traces are emitted for canary requests
- Alerts on missing cost telemetry
- PII scanners on log pipelines
Manual validations
- Incident retrospectives verify traces were sufficient
- Privacy review of observability data stores
Examples
- An on-call engineer opens a trace and sees which retrieved chunk caused an incorrect refund recommendation
- Finance reconciles model spend by product SKU using request-level cost tags
References
Crosswalks
MEASURE Measure
NIST AI Risk Management Framework · supports
Explainable Explainable and Interpretable
NIST AI Risk Management Framework · aligns-with
§9 Performance evaluation
ISO/IEC 42001 · supports
CC7 System Operations
SOC 2 Trust Services Criteria · evidence-for
PI1 Processing Integrity
SOC 2 Trust Services Criteria · evidence-for
Operational Excellence Operational Excellence
AWS Well-Architected Framework · aligns-with
Performance Efficiency Performance Efficiency
AWS Well-Architected Framework · partial
Future evolution
Standard LLM span semantics across providers and agent frameworks.