An LLM call is a latency and cost event: every request burns tokens, and every user waits for the first token. Inference optimization is the discipline of delivering the same quality with less of both. The levers are well understood — this guide ranks them by return on effort.
The levers, ranked
| Lever | Effort | Payoff | When |
|---|---|---|---|
| Caching | Low | 30–70% token savings | Repeated or similar prompts |
| Streaming | Low | Perceived 2–3x faster | Any user-facing chat |
| Model routing | Medium | 40–80% cost cut | Mixed task mix |
| Prompt compression | Medium | 20–50% fewer tokens | Long contexts, RAG |
| Batching | High | Higher throughput | Self-hosted serving |
Caching: the first lever
Two cache levels exist:
Exact-match caching — identical requests return a cached response. The win is bigger than it sounds: analytics dashboards, repetitive tool calls, and retries generate the same prompt constantly. Cache keys are the model + parameters + prompt hash.
Semantic caching — near-identical requests (same question, different phrasing) match via embeddings. This needs a vector lookup in front of the model; it's the highest-leverage optimization for support-bot workloads.
request ─► semantic cache (embedding match?)
├─ hit → return cached answer (0ms, 0 tokens)
└─ miss → model call → store answer
Streaming: the perceived-latency cheat code
Time-to-first-token is the number users feel. Streaming starts the response after the first token — typically 200–500ms — instead of after the full completion. Perceived speed roughly triples even with identical total time. Every user-facing LLM feature should stream by default; the deployment strategies guide covers the transport details.
Model routing: the cost multiplier
Not every request needs the largest model. Routing sends each request to the smallest model that meets the quality bar:
classifier (small) ─► intent/task
├─ extraction / formatting → small model (fast, cheap)
├─ summarization → medium model
└─ complex reasoning → large model
A routing classifier adds ~50ms and cuts blended cost by 40–80% on mixed workloads. The evals discipline (observability guide) is what proves the quality bar holds.
Prompt compression
Long contexts are where tokens leak. Compression techniques:
- Deduplicate retrieved passages — RAG often fetches overlapping chunks; dedupe before assembling context.
- Trim by relevance score — retrieval gives scores; use them (the RAG patterns guide has the ranking details).
- Shorten instructions — instruction bloat is real; test whether the 2-page system prompt beats the 10-line one (usually not by much).
- Summarize history — multi-turn chats: summarize old turns instead of replaying them.
The numbers that matter
Instrument every request:
| Metric | What it tells you |
|---|---|
| Tokens in / out | Cost per request |
| TTFT (time to first token) | Perceived latency |
| Total latency | Real latency |
| Cache hit rate | Caching effectiveness |
| Cost per request | The business number |
The observability for AI systems article shows the full dashboard.
Conclusion
Inference optimization is a ranked set of levers: cache first, stream always, route by task, compress context. Each one is measurable in tokens and milliseconds — instrument, apply, measure. The deployment guide shows where these levers live in the serving architecture.