Vector Databases Explained: Embeddings, Indexes, and Retrieval

Semantic search is the retrieval engine of the AI era — and it runs on vectors. This guide explains the stack honestly: what embeddings are, how approximate nearest-neighbor indexes work, and the decision of when you actually need a vector database at all.

Embeddings: text as coordinates

An embedding model converts text into a vector — typically 384 to 3,072 numbers — where semantic similarity maps to geometric proximity:

"cat"     → [0.12, -0.45, 0.88, ...]   (768 dims)
"kitten"  → [0.11, -0.44, 0.87, ...]   (close to "cat")
"invoice" → [-0.70, 0.21, 0.05, ...]   (far from both)

Similarity is measured as cosine distance (or dot product). The embeddings guide covers choosing and evaluating models; here the point is that vectors make "find things like this" a geometry problem.

The search problem

Naive search scans every vector and computes distance — fine for 10k chunks (~50ms), hopeless at 10M (~minutes). Vector databases solve this with approximate nearest neighbor (ANN) indexes, trading a tiny amount of recall for enormous speed.

HNSW: the default choice

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph:

layer 3 (coarse):  few nodes, long jumps
layer 2:           more nodes, shorter jumps
layer 1 (fine):    all nodes, local hops

query: start at top, greedily descend — logarithmic search

The properties: millisecond searches at millions of vectors, high recall (95–99%), no training phase (insert-anytime), but memory-hungry (the graph lives in RAM). For most RAG systems, HNSW is the right default.

IVF: the memory-conscious alternative

IVF (Inverted File) clusters vectors at index time:

1. K-means over the corpus → K centroids
2. Each vector assigned to nearest centroid
3. Query: check nearest centroids first, then exact scan within them

IVF uses far less memory and scales to billions with quantization (PQ), at the cost of a training step and slightly lower recall. The choice: HNSW for accuracy-first systems, IVF-PQ for corpus-scale economics.

The real bottleneck: chunking and embeddings

A vector index makes retrieval fast; it can't make it good. Retrieval quality is decided upstream:

1. Chunking — how documents are split (size, overlap, semantic boundaries) determines what "nearest" means. The RAG patterns guide covers chunk strategies.
2. Embedding model — domain mismatch (general model on medical text) silently degrades everything.
3. Hybrid retrieval — vector + keyword (BM25) fusion beats pure vector on exact terms, IDs, and product names.

The search index on yas.sh is a reminder that classic keyword search still wins for exact-match needs — vector search is an addition, not a replacement.

Do you need one?

Index parameters and tuning

Once you pick an index type, the knobs matter more than the choice itself:

• HNSW parameters. M (connections per node, default ~16–32) trades memory for recall; efConstruction (build-time search breadth) trades build speed for graph quality; efSearch (query-time breadth) trades latency for recall. In practice, raise efSearch first when recall dips, then M if you have memory headroom.
• IVF parameters. nlist (number of clusters) affects probe efficiency; nprobe (clusters scanned per query) is the direct recall/latency dial. Start with nlist at a few thousand and tune nprobe.
• Quantization. PQ reduces memory and speeds scans at a recall cost. Compress only when memory is the bottleneck, and validate recall on your eval set (embeddings guide) after enabling it.
• The golden rule. Tune against a fixed eval set with a target recall/latency budget — never tune by feel. A recall drop from 97% to 94% may be invisible in demos and catastrophic in production.

Operations: keeping the index honest

A vector index is a service, not a file. The operational baseline:

• Freshness. New and updated documents must be embedded and indexed on a defined schedule. "The index is stale" is a data-operations failure, and it looks like bad retrieval. Alert on index lag, not just index health (observability guide).
• Versioning. Store the embedding model version in index metadata and re-embed on model changes — mixed geometries in one index are quietly broken.
• Multi-tenancy. Partition or filter by tenant so one customer's query can never return another customer's vectors. A vector index with no tenant isolation is a data-leak machine.
• Backups and rollback. Indexes are buildable artifacts; keep the source documents and the embedding pipeline, so an index can always be rebuilt deterministically from scratch.

Evaluating your vector search

Before and after any change, measure on a representative set of real queries: hit-rate@k (does the right result land in the top-k), MRR (rank quality of the first good hit), p95 latency, and cost per query. Compare candidate configurations on identical queries and chunks. This is the only evidence that "the index is better" — a faster index that retrieves worse results is a regression with extra steps.

Key takeaways

• A vector database is a fast similarity-search engine; HNSW is the right default, IVF-PQ the corpus-scale alternative.
• Tune M, efSearch, and nprobe against a fixed eval set with a recall/latency budget — never by feel.
• Keep the index honest: monitor freshness, version the embedding model, enforce tenant isolation, and keep the rebuild path deterministic.
• Start brute-force, add an index when the corpus earns it, and never let the index distract from data quality upstream.

A decision, not a default

Adopting a vector database should be a decision made on evidence, not a trend. Measure your corpus size, query volume, and real recall requirements first; start with brute-force search and hybrid retrieval, and only introduce an ANN index — HNSW by default — when the corpus and latency budget genuinely require it. The teams that get vector search right are the ones that can also explain exactly why they need it and how they will measure it.

Conclusion

Conclusion

Vector databases are fast similarity-search engines: embeddings turn meaning into geometry, ANN indexes (HNSW by default) make the search fast, and chunking/embedding quality decides whether the results are worth fetching. Start brute-force, add an index when the corpus earns it, and never let the index distract from the data quality upstream. The embeddings guide covers the input half of the equation.