Semantic search is the retrieval engine of the AI era — and it runs on vectors. This guide explains the stack honestly: what embeddings are, how approximate nearest-neighbor indexes work, and the decision of when you actually need a vector database at all.
Embeddings: text as coordinates
An embedding model converts text into a vector — typically 384 to 3,072 numbers — where semantic similarity maps to geometric proximity:
"cat" → [0.12, -0.45, 0.88, ...] (768 dims)
"kitten" → [0.11, -0.44, 0.87, ...] (close to "cat")
"invoice" → [-0.70, 0.21, 0.05, ...] (far from both)
Similarity is measured as cosine distance (or dot product). The embeddings guide covers choosing and evaluating models; here the point is that vectors make "find things like this" a geometry problem.
The search problem
Naive search scans every vector and computes distance — fine for 10k chunks (~50ms), hopeless at 10M (~minutes). Vector databases solve this with approximate nearest neighbor (ANN) indexes, trading a tiny amount of recall for enormous speed.
HNSW: the default choice
HNSW (Hierarchical Navigable Small World) builds a multi-layer graph:
layer 3 (coarse): few nodes, long jumps
layer 2: more nodes, shorter jumps
layer 1 (fine): all nodes, local hops
query: start at top, greedily descend — logarithmic search
The properties: millisecond searches at millions of vectors, high recall (95–99%), no training phase (insert-anytime), but memory-hungry (the graph lives in RAM). For most RAG systems, HNSW is the right default.
IVF: the memory-conscious alternative
IVF (Inverted File) clusters vectors at index time:
1. K-means over the corpus → K centroids
2. Each vector assigned to nearest centroid
3. Query: check nearest centroids first, then exact scan within them
IVF uses far less memory and scales to billions with quantization (PQ), at the cost of a training step and slightly lower recall. The choice: HNSW for accuracy-first systems, IVF-PQ for corpus-scale economics.
The real bottleneck: chunking and embeddings
A vector index makes retrieval fast; it can't make it good. Retrieval quality is decided upstream:
- Chunking — how documents are split (size, overlap, semantic boundaries) determines what "nearest" means. The RAG patterns guide covers chunk strategies.
- Embedding model — domain mismatch (general model on medical text) silently degrades everything.
- Hybrid retrieval — vector + keyword (BM25) fusion beats pure vector on exact terms, IDs, and product names.
The search index on yas.sh is a reminder that classic keyword search still wins for exact-match needs — vector search is an addition, not a replacement.
Do you need one?
| Corpus size | Recommendation |
|---|---|
| < 10k chunks | No — brute-force in memory |
| 10k–100k | Maybe — evaluate first |
| 100k+ | Yes — HNSW, tune recall |
| Multi-tenant / huge | Yes — partitioned indexes, IVF |
Conclusion
Vector databases are fast similarity-search engines: embeddings turn meaning into geometry, ANN indexes (HNSW by default) make the search fast, and chunking/embedding quality decides whether the results are worth fetching. Start brute-force, add an index when the corpus earns it, and never let the index distract from the data quality upstream. The embeddings guide covers the input half of the equation.