Serving an LLM is a menu with five entries, and the right answer is rarely "the biggest GPU you can rent." This guide maps the options — managed APIs, serverless, dedicated GPUs, self-hosted fleets, and hybrids — with the trade-offs that actually decide outcomes: cost, latency, data control, and operational burden.
The menu
| Option | Latency | Cost shape | Control | Ops burden |
|---|---|---|---|---|
| Managed API | Good | Pay per token, zero idle | Least (data leaves) | None |
| Serverless GPU | Good (warm) | Pay per execution | Medium | Low |
| Dedicated GPU | Excellent | Fixed + idle cost | High | Medium |
| Self-hosted fleet | Best (tuned) | Capital + ops heavy | Full | High |
| Hybrid routing | Good | Blended, optimal | High | Medium |
Managed APIs: the default starting point
The case for APIs is not just convenience — it's economics. At low to medium volume, the fully-loaded cost of self-hosting (GPUs, failover, engineers, evaluation) exceeds per-token API pricing. APIs also deliver instant scale, automatic versioning, and a security boundary you don't maintain.
The costs to accept: data leaves your boundary (relevant for regulated data — see the AI security guide), per-token spend grows linearly with usage, and you depend on a vendor's uptime and roadmap.
Self-hosting: when it earns its keep
Self-hosting wins in four situations:
- Sustained high volume — above roughly 1M tokens/day, fixed GPU cost amortizes below per-token pricing.
- Data residency — the data cannot leave the boundary, full stop.
- Latency at the edge — in-region self-hosted inference beats cross-region APIs.
- Fine-tuned models — custom weights that the API vendors don't serve.
The honest costs: GPU fleet operations (orchestration guide), model versioning and rollback, and the evaluation discipline to know when a "better" model is actually worse.
The hybrid: routing as the answer
Most production systems end up hybrid — the model routing pattern:
request → router (classifier)
├─ simple task → small open model (self-hosted or serverless)
├─ standard task → managed API (medium model)
└─ hard reasoning → largest model (API or fleet)
Routing blends cost and control: sensitive data routes to the boundary, commodity traffic routes to the cheapest adequate model, and the whole thing is invisible to callers.
Choosing models: evals decide
Whatever the deployment path, model choice is an evaluation problem:
- Build a task-specific eval set (50–200 representative inputs with expected outputs).
- Run candidate models through it — including the small ones; they win more often than assumed.
- Measure quality, latency, and cost per request; choose on the blend.
- Re-run on every model version change — the observability guide covers the CI integration.
The migration path
The sequence that avoids regret:
1. API + evals + instrumentation (week 1–2)
2. Add routing; cache aggressively (month 1)
3. Pilot self-host for ONE workload (month 2–3)
4. Expand the fleet only on evidence (after the pilot proves cost/latency)
Conclusion
LLM deployment is a menu, not a religion: APIs for speed and economics, self-hosting for volume and control, hybrids for the real world. The discipline that makes any choice work is the same — evals, routing, and instrumentation. The inference optimization guide makes the chosen path fast and cheap; the security guide keeps it safe.