Every AI system — a chatbot, a search copilot, a document summarizer — runs on the same five pillars: compute, data, models, serving, and observability. The details differ wildly between a weekend prototype and a production service, but the pillars don't. This guide builds the mental model; the rest of this series goes deep on each pillar.
The five pillars
1. Compute. The hardware layer: GPUs for training and heavy inference, CPUs for lightweight serving, and specialized accelerators for narrow workloads. The decisions here are cost, availability, and latency — not "which card is newest".
2. Data. The pipeline that turns raw text into training or retrieval material: ingestion, cleaning, deduplication, chunking, embedding, storage. Data quality is the ceiling for everything else — models are compression of data, not magic.
3. Models. Weights plus a registry: which model, which version, which fine-tune, which license. The discipline that matters is versioning — models change behavior between releases, and your system must know exactly what it's running.
4. Serving. The runtime: loading weights, batching requests, managing context windows, streaming tokens, autoscaling. Serving is where latency budgets live — and where the LLM deployment strategies article spends its time.
5. Observability. Metrics (latency, throughput, cost per request), traces (the prompt-to-response path), and evals (automated quality checks). AI systems are stochastic — observability is how you keep them honest.
How the pillars compose
data ──► embeddings ──► vector database ──► retrieval
│
user ──► gateway ──► context assembly ──► model serving ──► response
│ │
└──────── observability ◄──────┘
The RAG architecture patterns article is this diagram expanded into variants. Every AI product you've used is some arrangement of these boxes.
The cost curve that shapes decisions
The economics of each pillar:
| Pillar | Small scale | At scale |
|---|---|---|
| Compute | API calls, pay per token | GPU fleets, utilization targets |
| Data | Notebooks, CSVs | Pipelines, catalogs, quality gates |
| Models | One API model | Several models, routed by task |
| Serving | Simple proxy | Batching, caching, autoscaling |
| Observability | Logs | Traces + evals + cost dashboards |
Most teams over-engineer compute and under-engineer data and evals — the two pillars that actually determine quality. The inference optimization guide and the embeddings guide show where the leverage really is.
The build order that works
- Prove the behavior with a managed API — no GPU, no vector DB.
- Add the data layer — a real pipeline with quality checks, because that's where quality comes from.
- Version the models — pin versions, track prompts, add evals to CI.
- Optimize serving — caching, batching, model routing, then (maybe) self-hosted GPUs.
- Instrument everything — cost per request and eval scores on one dashboard.
Skipping steps is how teams end up with a GPU cluster that makes bad answers faster.
Conclusion
AI infrastructure is five pillars, not a magic box: compute, data, models, serving, observability. The leverage is in data and evals, the cost is in compute and serving, and the discipline is versioning and measurement. The linked guides in this series — deployment, RAG, vector databases, observability — build on this foundation.