RAG (retrieval-augmented generation) is the pattern that grounds LLM answers in your own data: retrieve relevant context, feed it to the model, get answers you can trace. The gap between a RAG demo and a dependable RAG system is a pipeline — this guide builds it layer by layer, with the failure modes named.
The pipeline
documents ─► chunking ─► embeddings ─► vector index
│
query ─► embedding ─► retrieve top-k ─► re-rank ─► context assembly
│
▼
model (grounded answer)
Every box is a failure point. The model is usually the least interesting box.
Layer 1 — Chunking: the quality ceiling
Chunking decides what "relevant" can mean. Bad chunking produces retrieval that is confidently wrong.
| Strategy | Works for | Risk |
|---|---|---|
| Fixed-size (300–800 tokens) | Uniform docs, notes | Splits mid-thought |
| Paragraph-based | Articles, docs | Long paragraphs overflow |
| Semantic (boundary detection) | Mixed content | Over-segments |
| Recursive (structure-aware) | Code, markdown | Needs good structure |
The discipline: measure. Build an eval set of (query → expected chunks) pairs and tune chunk size against retrieval hit-rate. The embeddings guide covers the measurement loop.
Layer 2 — Indexing and retrieval
Embed chunks, store vectors, retrieve top-k (k=10–20 typical) via HNSW. Two upgrades that matter more than the index:
Hybrid retrieval — combine vector similarity with keyword search (BM25): exact terms, product codes, and names that embeddings mangle. Fusion (weighted merge or RRF) reliably beats pure vector search on real corpora.
Metadata filtering — date ranges, document types, tenants. Filtering before search keeps results fresh and permission-correct; unfiltered search over stale data is a classic RAG failure.
Layer 3 — Re-ranking
Embedding models measure semantic similarity; they don't rank answerability. A cross-encoder re-ranker scores (query, chunk) pairs directly and is dramatically more accurate:
retrieve top-20 (fast, bi-encoder) ─► re-rank top-5 (accurate, cross-encoder)
Re-ranking costs one model pass over the candidates — milliseconds per query — and routinely lifts retrieval quality by 10–20 points on answerability benchmarks.
Layer 4 — Context assembly
The prompt is where RAG lives or dies:
- Include provenance — chunk IDs and sources in the context so answers can cite and you can trace.
- Bound the context — top-5 re-ranked chunks, not everything retrieved; token bloat degrades the model's focus (inference optimization).
- Instruct grounding — "answer only from the context; say 'not covered' otherwise." Grounding instructions cut hallucination measurably.
- Handle empty retrieval — a system that says "I don't have this information" beats one that invents it.
Production patterns
Freshness. Data changes; the index must too. Incremental indexing, scheduled re-embedding, and deletion propagation — stale RAG answers confidently about retired products.
Caching. Identical questions hit the same context; semantic caching saves the retrieval+generation cost.
Evaluation. RAG has two quality axes: retrieval (hit-rate) and generation (faithfulness). Measure both in CI (observability guide) — retrieval regressions are the silent killer.
Guardrails. What if the top chunk is irrelevant but above threshold? Minimum-score floors and "not covered" fallbacks keep answers honest.
Conclusion
RAG is a pipeline of five layers — chunking, indexing, retrieval, re-ranking, context assembly — and production quality comes from the boring layers: measured chunking, hybrid retrieval, re-ranking, grounded prompts, and evals. Build the demo in a day; build the pipeline in a week; measure everything forever. The embeddings guide and vector database guide are the companion pieces.