Inference Optimization Guide: Making LLM Calls Fast and Cheap

An LLM call is a latency and cost event: every request burns tokens, and every user waits for the first token. Inference optimization is the discipline of delivering the same quality with less of both. The levers are well understood — this guide ranks them by return on effort.

The levers, ranked

Caching: the first lever

Two cache levels exist:

Exact-match caching — identical requests return a cached response. The win is bigger than it sounds: analytics dashboards, repetitive tool calls, and retries generate the same prompt constantly. Cache keys are the model + parameters + prompt hash.

Semantic caching — near-identical requests (same question, different phrasing) match via embeddings. This needs a vector lookup in front of the model; it's the highest-leverage optimization for support-bot workloads.

request ─► semantic cache (embedding match?)
              ├─ hit → return cached answer (0ms, 0 tokens)
              └─ miss → model call → store answer

Streaming: the perceived-latency cheat code

Time-to-first-token is the number users feel. Streaming starts the response after the first token — typically 200–500ms — instead of after the full completion. Perceived speed roughly triples even with identical total time. Every user-facing LLM feature should stream by default; the deployment strategies guide covers the transport details.

Model routing: the cost multiplier

Not every request needs the largest model. Routing sends each request to the smallest model that meets the quality bar:

classifier (small) ─► intent/task
    ├─ extraction / formatting → small model (fast, cheap)
    ├─ summarization → medium model
    └─ complex reasoning → large model

A routing classifier adds ~50ms and cuts blended cost by 40–80% on mixed workloads. The evals discipline (observability guide) is what proves the quality bar holds.

Prompt compression

Long contexts are where tokens leak. Compression techniques:

• Deduplicate retrieved passages — RAG often fetches overlapping chunks; dedupe before assembling context.
• Trim by relevance score — retrieval gives scores; use them (the RAG patterns guide has the ranking details).
• Shorten instructions — instruction bloat is real; test whether the 2-page system prompt beats the 10-line one (usually not by much).
• Summarize history — multi-turn chats: summarize old turns instead of replaying them.

The numbers that matter

Instrument every request:

The observability for AI systems article shows the full dashboard.

Quantization and distillation: shrinking the model

Once caching, streaming, routing, and prompt compression are in place, the remaining lever is making the model itself smaller and cheaper to run:

• Quantization reduces weight precision — e.g. FP16 to INT8 or INT4 — trading a small quality delta for a large speed and memory win. INT8 typically cuts memory and latency ~2x with minimal quality impact for many workloads; INT4 goes further at a more visible quality cost. Measure, don't assume: run the same eval set on the quantized and full-precision model and compare (embeddings guide makes the same point about embedding models).
• Distillation trains a small "student" model to imitate a large "teacher" on your task. A distilled model is not free — it costs training time — but it pays off permanently if you serve the same task at volume.
• When to use which. Quantization is the cheap, immediate lever on existing weights; distillation is the bigger, slower investment. Most teams do quantization first, then distill only the workloads that dominate the bill.
• Pruning and layer sparsity are more advanced and hardware-dependent; treat them as experimental until they demonstrably beat quantization on your metrics.

The measurement loop

Optimization without measurement is just churn. The loop that keeps every change honest:

1. Define the baseline. Record latency (TTFT, total), cost per request, and an eval score for the current setup on a fixed, representative workload.
2. Change one lever. One optimization at a time — never stack two and guess which mattered.
3. Measure the same numbers. Same workload, same eval set, same traffic mix.
4. Gate on quality, not just cost. A change that saves 40% but drops eval score by 8 points is a regression, not a win. The observability guide shows how to run evals on every change.
5. Promote or roll back on evidence. Record the result and move on. The team's optimization playbook is the accumulation of these evidence-backed decisions.

A practical optimization order

For a typical user-facing LLM feature, work in this sequence: enable streaming first (cheap perceived-latency win), add exact-match caching, route by task with a small classifier, compress long prompts, then batch and quantize on self-hosted serving. Re-run the measurement loop at each step. This order front-loads the lowest-effort, highest-payoff levers and defers the engineering-heavy ones until the easy wins are banked.

Key takeaways

• Bank the easy levers first: streaming, exact-match caching, model routing, prompt compression.
• Quantization is the cheap immediate win on existing weights; distillation is the bigger, slower investment.
• Change one lever at a time, measure the same numbers, and gate every optimization on eval quality — cost savings without a quality floor is a regression.
• A documented, evidence-backed optimization order turns tuning into a playbook instead of churn.

A practical starting point

If you are starting from a plain model call today, do this in an afternoon: enable streaming, add an exact-match cache for repeated prompts, and put a small classifier in front to route simple tasks to a cheaper model. That alone typically cuts latency and cost measurably with no quality change — and it gives you a measurement baseline for every later optimization.

Conclusion

Conclusion

Inference optimization is a ranked set of levers: cache first, stream always, route by task, compress context. Each one is measurable in tokens and milliseconds — instrument, apply, measure. The deployment guide shows where these levers live in the serving architecture.