RAG (retrieval-augmented generation) is the pattern that grounds LLM answers in your own data: retrieve relevant context, feed it to the model, get answers you can trace. The gap between a RAG demo and a dependable RAG system is a pipeline — this guide builds it layer by layer, with the failure modes named.
The pipeline
documents ─► chunking ─► embeddings ─► vector index
│
query ─► embedding ─► retrieve top-k ─► re-rank ─► context assembly
│
▼
model (grounded answer)
Every box is a failure point. The model is usually the least interesting box.
Layer 1 — Chunking: the quality ceiling
Chunking decides what "relevant" can mean. Bad chunking produces retrieval that is confidently wrong.
| Strategy | Works for | Risk |
|---|---|---|
| Fixed-size (300–800 tokens) | Uniform docs, notes | Splits mid-thought |
| Paragraph-based | Articles, docs | Long paragraphs overflow |
| Semantic (boundary detection) | Mixed content | Over-segments |
| Recursive (structure-aware) | Code, markdown | Needs good structure |
The discipline: measure. Build an eval set of (query → expected chunks) pairs and tune chunk size against retrieval hit-rate. The embeddings guide covers the measurement loop.
Layer 2 — Indexing and retrieval
Embed chunks, store vectors, retrieve top-k (k=10–20 typical) via HNSW. Two upgrades that matter more than the index:
Hybrid retrieval — combine vector similarity with keyword search (BM25): exact terms, product codes, and names that embeddings mangle. Fusion (weighted merge or RRF) reliably beats pure vector search on real corpora.
Metadata filtering — date ranges, document types, tenants. Filtering before search keeps results fresh and permission-correct; unfiltered search over stale data is a classic RAG failure.
Layer 3 — Re-ranking
Embedding models measure semantic similarity; they don't rank answerability. A cross-encoder re-ranker scores (query, chunk) pairs directly and is dramatically more accurate:
retrieve top-20 (fast, bi-encoder) ─► re-rank top-5 (accurate, cross-encoder)
Re-ranking costs one model pass over the candidates — milliseconds per query — and routinely lifts retrieval quality by 10–20 points on answerability benchmarks.
Layer 4 — Context assembly
The prompt is where RAG lives or dies:
- Include provenance — chunk IDs and sources in the context so answers can cite and you can trace.
- Bound the context — top-5 re-ranked chunks, not everything retrieved; token bloat degrades the model's focus (inference optimization).
- Instruct grounding — "answer only from the context; say 'not covered' otherwise." Grounding instructions cut hallucination measurably.
- Handle empty retrieval — a system that says "I don't have this information" beats one that invents it.
Production patterns
Freshness. Data changes; the index must too. Incremental indexing, scheduled re-embedding, and deletion propagation — stale RAG answers confidently about retired products.
Caching. Identical questions hit the same context; semantic caching saves the retrieval+generation cost.
Evaluation. RAG has two quality axes: retrieval (hit-rate) and generation (faithfulness). Measure both in CI (observability guide) — retrieval regressions are the silent killer.
Guardrails. What if the top chunk is irrelevant but above threshold? Minimum-score floors and "not covered" fallbacks keep answers honest.
The six most common RAG failure modes
A RAG pipeline has many moving parts, and each has a characteristic way to fail. Know the failure before it costs you:
- Retrieval miss. The right chunk never surfaces. Fixes live upstream — chunking, embedding model, hybrid retrieval — not in the prompt (embeddings guide).
- Retrieval overload. The top-k fills the context window with marginally relevant chunks, crowding out the correct one and inflating tokens. Raise the relevance bar, reduce k, or re-rank.
- Context truncation. The assembled prompt exceeds the model window and the tail (often the most relevant retrieved context) is cut. Compress and rank so the highest-value context survives (inference optimization).
- Source confusion. The model answers from the wrong document because retrieval scored a look-alike higher. Metadata filters and field-weighted scoring reduce this.
- Hallucination despite context. The model paraphrases or invents instead of grounding strictly in retrieved text. This needs faithfulness evals and, often, stricter instruction + output checking.
- Stale index. New documents are not embedded or indexed, so retrieval is quietly frozen. Freshness monitoring is an operations problem, not a model problem (observability guide).
Re-ranking done correctly
Re-ranking is a two-stage pipeline: cheap retrieval returns a broad candidate set (top 20–50), then a stronger re-ranker (a cross-encoder that scores query–chunk pairs jointly) reorders them. The properties:
- You do not need to re-rank everything. Only the retrieved candidates, not the whole corpus — that is what keeps it cheap.
- Re-ranking fixes "right-but-not-top." When the right chunk is in the candidate set but ranked too low by the first-stage ANN search, the re-ranker pulls it to the top. It cannot fix a chunk that retrieval never returned.
- Tune k and threshold together. Raising first-stage k captures more recall for the re-ranker to fix but costs latency; the threshold is the quality lever. Measure hit-rate before and after.
A/B testing retrieval changes
Retrieval changes should be treated like product experiments: pick a representative eval set, run the candidate against the current system on identical queries and chunks, and compare hit-rate@k, MRR, faithfulness, and cost per query. Promote a change only when it wins on the blend. This discipline — evaluate, compare, promote — is the difference between a RAG system that improves and one that churns.
Key takeaways
- RAG is a pipeline, not a feature; the failure modes live in retrieval, context, and freshness — not the model.
- Re-ranking fixes right-but-not-top retrieval over a broad candidate set; it cannot fix a chunk retrieval never returned.
- Treat retrieval changes like product experiments — evaluate on a fixed eval set, compare, and promote on the blend.
- Stale indexes and silent model swaps are operations problems that look like model problems.
Conclusion
Conclusion
RAG is a pipeline of five layers — chunking, indexing, retrieval, re-ranking, context assembly — and production quality comes from the boring layers: measured chunking, hybrid retrieval, re-ranking, grounded prompts, and evals. Build the demo in a day; build the pipeline in a week; measure everything forever. The embeddings guide and vector database guide are the companion pieces.