Traditional monitoring watches systems that behave deterministically. LLM systems are stochastic — the same prompt can produce different answers, and quality can drift between model versions without any "error" appearing. Observability for AI therefore has three layers: metrics (the numbers), traces (the path), and evals (the quality). This guide covers all three.
Layer 1 — Metrics: the numbers
Every request emits a base set:
| Metric | What it catches |
|---|---|
| Time to first token (TTFT) | Perceived latency regressions |
| Total latency | Slow completions |
| Tokens in / out | Cost per request |
| Cost per request | Budget drift, runaway prompts |
| Cache hit rate | Caching effectiveness |
| Error rate (4xx/5xx, provider) | Outages and misconfigurations |
| Eval score (rolling) | Quality regressions |
Dashboards are the easy half. The discipline is alerting on the shapes that matter: cost anomalies (a prompt loop burning tokens), latency budget breaches, error-rate spikes, and eval-score drops after any change.
Layer 2 — Traces: the prompt-to-response path
An LLM trace captures the full request journey:
input prompt
└─ retrieved context (RAG) ← which chunks, what scores?
└─ assembled prompt ← final token count
└─ model call ← model, version, params
└─ output ← tokens, finish reason
└─ post-processing / tool calls
With traces, a bad answer becomes answerable: was the context retrieval wrong, the model version different, the prompt truncated, or the output cut off? Traces are the AI equivalent of distributed traces — same principle, new spans.
Layer 3 — Evals: quality as CI
Evals are automated quality checks against a representative dataset:
eval set: 200 curated (input → expected behavior) pairs
scoring: exact match, LLM-as-judge, faithfulness, embedding similarity
gate: every model/prompt change must not regress the score
The critical habit: run evals on every change — model version bumps, prompt edits, retrieval tweaks, chunking changes. Model updates are the classic silent regression: no error, just subtly worse answers. Evals catch it before users do.
The cost dimension: observability as a budget tool
Token spend is the business number of AI. Per-request cost tracking makes it manageable:
- Cost per request by task — which features burn tokens?
- Cost per conversation — long chats with big contexts compound.
- Prompt-size distribution — context bloat is the #1 silent cost driver (compression techniques).
The privacy dimension
Observability data is sensitive by construction — prompts and outputs are user data. The rules: redact PII before storing traces, hash or truncate prompt contents where retention allows, and apply the same retention discipline as the analytics retention guide describes for click data. The AI security guide covers the full threat model.
A pragmatic starter stack
- Structured logs per request (the metrics above as JSON lines).
- A dashboard for the five core metrics.
- Alerts on cost anomaly + latency budget + error rate.
- An eval set of 50–100 cases run on every deploy.
- Monthly model review: eval scores vs cost per request, decide whether to switch or tune.
Alerting philosophy: alert on shapes, not just thresholds
Raw alerts are noise; shape alerts are signal. The patterns that deserve a pager:
- Cost anomalies. A prompt loop, a runaway retry, or a context-bloat regression burns tokens invisibly. Alert on cost-per-request spikes and token-volume anomalies, not just on fixed dollar thresholds.
- Latency budget breaches. Alert on p95 crossing the latency budget, and on TTFT regressions — the number users actually feel (inference optimization guide).
- Eval regressions. The most important alert of all: an eval-score drop after any change. Because there is no "error" when a model subtly degrades, the eval alert is the substitute. Gate changes on it in CI and page on it in production.
- Error-rate deltas. Provider 4xx/5xx spikes and cache-hit-rate cliffs indicate config or upstream issues.
- Traffic-shape surprises. Sudden QPS spikes (a viral share, an abuse attempt) need capacity attention; sudden drops need investigation too.
Every alert should have a runbook: who gets paged, what to check first, and what the rollback is.
SLOs and error budgets
Instrumentation becomes governance when it produces service-level objectives. Pick two or three that matter for AI workloads:
- Latency SLO — e.g. "p95 TTFT < 1s and p95 total latency < 4s over 30 days."
- Cost SLO — e.g. "cost per request stays within ±20% of the 30-day baseline."
- Quality SLO — e.g. "rolling eval score stays above the defined floor."
An error budget (the tolerable degradation window) tells you when you may ship a risky change and when you must hold. It turns observability from a dashboard you look at into a release control you act on.
Debugging a bad answer end to end
When a user reports "the answer is wrong," the trace collapses the possibilities:
- Was the retrieval wrong? Which chunks came back, and with what scores? If the wrong context was retrieved, the fix is chunking/embedding, not the model (RAG patterns guide).
- Was the prompt right? Was the context truncated? Was the assembled prompt the intended one? Prompt-size distribution catches bloat.
- Was the model version the one you intended? Silent model swaps are a classic cause of "nothing changed but answers changed."
- Was the output post-processed? Did a schema validator or filter alter or drop it?
The trace makes each branch answerable in minutes instead of a day of guessing. That is the entire value of the three-layer approach.
Key takeaways
- Observability for AI is three layers: metrics (numbers), traces (the path), and evals (the quality).
- Alert on shapes — cost anomalies, latency budget breaches, and eval regressions — not just fixed thresholds.
- Turn instrumentation into governance with latency, cost, and quality SLOs and an error budget.
- A trace collapses the bad-answer question into minutes: wrong retrieval, truncated prompt, silent model swap, or broken post-processing.
Instrumentation hygiene
Whatever your stack, start with structured JSON logs per request covering the base metrics, ship them to one place, and make the trace path explicit. The tooling matters less than the discipline — a team that consistently logs, traces, and evals beats a team with an expensive dashboard they do not use.
Conclusion
Conclusion
AI observability is metrics, traces, and evals — numbers, paths, and quality. Instrument the base metrics, trace the prompt-to-response journey, and gate every change with evals. That combination turns a stochastic black box into an accountable service — and it's the difference between "the AI is acting weird" and "the eval dropped 11 points when we bumped the model, here's the trace."