Traditional monitoring watches systems that behave deterministically. LLM systems are stochastic — the same prompt can produce different answers, and quality can drift between model versions without any "error" appearing. Observability for AI therefore has three layers: metrics (the numbers), traces (the path), and evals (the quality). This guide covers all three.
Layer 1 — Metrics: the numbers
Every request emits a base set:
| Metric | What it catches |
|---|---|
| Time to first token (TTFT) | Perceived latency regressions |
| Total latency | Slow completions |
| Tokens in / out | Cost per request |
| Cost per request | Budget drift, runaway prompts |
| Cache hit rate | Caching effectiveness |
| Error rate (4xx/5xx, provider) | Outages and misconfigurations |
| Eval score (rolling) | Quality regressions |
Dashboards are the easy half. The discipline is alerting on the shapes that matter: cost anomalies (a prompt loop burning tokens), latency budget breaches, error-rate spikes, and eval-score drops after any change.
Layer 2 — Traces: the prompt-to-response path
An LLM trace captures the full request journey:
input prompt
└─ retrieved context (RAG) ← which chunks, what scores?
└─ assembled prompt ← final token count
└─ model call ← model, version, params
└─ output ← tokens, finish reason
└─ post-processing / tool calls
With traces, a bad answer becomes answerable: was the context retrieval wrong, the model version different, the prompt truncated, or the output cut off? Traces are the AI equivalent of distributed traces — same principle, new spans.
Layer 3 — Evals: quality as CI
Evals are automated quality checks against a representative dataset:
eval set: 200 curated (input → expected behavior) pairs
scoring: exact match, LLM-as-judge, faithfulness, embedding similarity
gate: every model/prompt change must not regress the score
The critical habit: run evals on every change — model version bumps, prompt edits, retrieval tweaks, chunking changes. Model updates are the classic silent regression: no error, just subtly worse answers. Evals catch it before users do.
The cost dimension: observability as a budget tool
Token spend is the business number of AI. Per-request cost tracking makes it manageable:
- Cost per request by task — which features burn tokens?
- Cost per conversation — long chats with big contexts compound.
- Prompt-size distribution — context bloat is the #1 silent cost driver (compression techniques).
The privacy dimension
Observability data is sensitive by construction — prompts and outputs are user data. The rules: redact PII before storing traces, hash or truncate prompt contents where retention allows, and apply the same retention discipline as the analytics retention guide describes for click data. The AI security guide covers the full threat model.
A pragmatic starter stack
- Structured logs per request (the metrics above as JSON lines).
- A dashboard for the five core metrics.
- Alerts on cost anomaly + latency budget + error rate.
- An eval set of 50–100 cases run on every deploy.
- Monthly model review: eval scores vs cost per request, decide whether to switch or tune.
Conclusion
AI observability is metrics, traces, and evals — numbers, paths, and quality. Instrument the base metrics, trace the prompt-to-response journey, and gate every change with evals. That combination turns a stochastic black box into an accountable service — and it's the difference between "the AI is acting weird" and "the eval dropped 11 points when we bumped the model, here's the trace."