Serverless changed web deployment because web workloads are small, cold starts are milliseconds, and traffic is spiky. AI workloads are different: models are heavy, cold starts are seconds, and latency budgets are tight. Serverless still has a real place in AI — the skill is knowing exactly where the cut line is.
What serverless buys
- Zero idle cost — you pay for execution, not for a warm cluster.
- Infinite burst — traffic spikes don't require capacity planning.
- No fleet operations — no nodes, no patches, no bin-packing.
For webhooks, batch jobs, and spiky interactive traffic, these are decisive. The question is what the model-loading latency does to your workload.
The cold-start math
A serverless instance that has been idle must load the model before the first request:
function code + deps: ~100–300ms
model load (small, distilled): ~1–3s
model load (7B+ dense): ~5–30s
embedding model: ~1–2s
For a chat UI where users expect a response in ~2s, a 15s cold start is a product failure. For a batch job that processes 10,000 documents, a 15s cold start once per worker is noise. The workload's latency budget is the entire decision.
The fit matrix
| Workload | Serverless? | Why |
|---|---|---|
| Webhook-triggered enrichment | ✓ | Spiky, latency-tolerant |
| Batch summarization/classification | ✓ | Throughput, not latency |
| Bursty chat (few users, unpredictable) | Maybe | Cold starts vs idle cost |
| Steady interactive chat | ✗ | Persistent warm serving wins |
| Training / long evals | ✗ | Timeouts and cost |
| Tight p99 API | ✗ | Cold starts break the budget |
Warm-start engineering
Where serverless fits but cold starts still bite, the mitigations:
- Keep-warm pings — a scheduler hits the function every few minutes; costs a few executions, eliminates most cold starts.
- Smaller models at the edge — a distilled model loads in 1–2s (inference optimization); route simple tasks there.
- Provisioned concurrency — platforms let you pre-warm a minimum number of instances for a fee; use it for the p95 floor.
- Embeddings are the friendliest — small models, fast loads, and semantic caching makes them ideal serverless citizens.
The hybrid pattern
Production AI systems rarely choose one runtime. The pattern that works:
webhook / batch ─► serverless functions (embeddings, small models)
chat / interactive ─► persistent serving (warm, GPU or API)
routing ─► model router decides per request
The model routing layer makes the mix invisible to callers: same interface, different runtimes behind it.
Conclusion
Serverless AI fits spiky, latency-tolerant workloads — webhooks, batch jobs, enrichment — and struggles with steady interactive traffic where cold starts break budgets. The production answer is usually hybrid: serverless for the spikes, persistent serving for the steady stream, routing in between. The LLM deployment strategies article lays out the full menu of runtimes.