Embeddings Guide (Practical): Choosing Models and Measuring Quality

Embeddings are the input half of every semantic search and RAG system — and the half most teams choose by vibes. This guide is the practical version: how to pick a model, embed correctly, store vectors sanely, and — most importantly — measure whether your embeddings actually work on your data.

What you're actually choosing

An embedding model maps text to a vector such that similar meaning sits nearby. The practical differences between models: dimension size (384–3,072), domain fit (general vs legal/medical/code), inference cost, and the quality of the geometry. Benchmarks can't tell you which model fits your corpus — measurement can.

The selection loop

1. Start standard. Pick a mainstream model (or managed API) — don't optimize model choice before you have data.
2. Build a retrieval eval set. 50–100 queries from your real usage, each with the chunks/document that should be retrieved.
3. Measure hit-rate@k and MRR. The number that matters: does the right result land in the top-k?
4. Compare candidates on the same eval set. Same queries, same chunks, different models — the eval set decides.
5. Re-run on any model change. Embedding upgrades are regressions waiting to happen.

hit-rate@5 = fraction of queries where the known-good chunk appears in top 5
MRR        = average reciprocal rank of the first good hit

Embedding correctly

The mechanics that silently break pipelines:

Normalize. L2-normalize vectors before storage — cosine similarity becomes dot product, and index math stays stable.

Batch, don't loop. Embed in batches (32–128 texts) — per-text calls multiply latency and cost.

Check the token limit. Models cap input length (usually 256–8k tokens); long documents must be chunked before embedding (chunking guide).

Keep the model version pinned. Re-embedding the corpus with a different model produces vectors in a different geometry — mixed geometries in one index are quietly broken. Version the embedding model like you version code.

Storage: vectors are just data

Vectors live wherever your data lives: a vector database at scale, or an in-memory matrix for small corpora. The storage rules:

• Store the text alongside the vector — retrieval needs the content, not just the coordinates.
• Store the model version in the index metadata — audits and re-embedding need it.
• Normalize at write time — don't trust callers to do it.

Measuring quality on your data

The eval loop in practice:

# 1. embed eval queries and candidate chunks
# 2. search each query, record ranks
# 3. compute hit-rate@k / MRR

If hit-rate@5 sits below ~70%, the fix order is: chunking first (most common culprit), then hybrid retrieval (keyword fusion), then re-ranking, then — last — a different embedding model. Re-ordering that fix list is how teams waste weeks.

The cost angle

Embedding cost scales with corpus size and re-embedding frequency. Practical controls: embed once, store forever; incremental indexing for new documents; and schedule full re-embeds only when the model version changes.

A worked example: build the eval set

Concretely, here is how the selection loop runs on a real corpus. Suppose you have 20,000 support documents and you want semantic search over them.

1. Harvest real queries. Pull the last month of actual user searches from your logs — not imagined ones. Deduplicate to ~100–200 queries. These are your eval queries because they are what users actually type.
2. Annotate ground truth. For each query, identify the 1–5 chunks that contain the correct answer. This is manual work and it is the most valuable hour you will spend. If you cannot find the right chunk for a query, that is a retrieval gap, not an annotation error.
3. Pick candidate models. Two or three mainstream candidates (or managed API endpoints) with different dimension sizes — e.g. a 768-dim general model and a domain-tuned model if one exists for your content type.
4. Run the same queries through each. Same chunking, same normalization, same search index. The only variable is the embedding model.
5. Read the numbers. hit-rate@5 and MRR decide. If the domain model wins by 15 points on hit-rate, switch. If it ties, pick the cheaper or faster one. The eval set is the referee; you do not have to defend the choice by vibes.

Troubleshooting checklist

When retrieval quality is bad, work the fix order — chunking first, model last:

• Chunk size and boundaries. Is the answer split across two chunks? Increase chunk size or use semantic/heading-aware chunking.
• Overlap. Does context get lost at chunk seams? Add overlap (10–15%) so spans at boundaries survive.
• Metadata filters. Are you losing precision by ignoring filters (date, author, category)? Filtering before or after ANN search changes results.
• Normalization. Are vectors normalized consistently at write and query time? A mix silently degrades cosine results.
• Query phrasing. Are user queries very different in style from document text? Consider a small query-to-document paraphrase step or hybrid retrieval.
• Index freshness. Are new documents actually embedded and indexed? Stale indexes look like bad embeddings.

When not to use embeddings

Embeddings are the right tool for fuzzy, meaning-based retrieval — synonyms, paraphrase, open-ended questions. They are the wrong tool for:

• Exact identifiers — order numbers, SKUs, license keys. Keyword/BM25 or a plain database lookup beats a vector search every time.
• Tiny corpora — under ~10k chunks, brute-force in memory is fast enough (vector database guide).
• Strictly structured queries — "all invoices for customer X in March" is a SQL query, not a semantic search.
• Auditable, deterministic lookups — when you must prove exactly how a result was found, an index of keywords and rules is more defensible than a similarity score.

Knowing when not to use embeddings is part of using them well — the cheapest retrieval is the one you do not build.

Key takeaways

• Build a real retrieval eval set from actual user queries before choosing any model — it is the only referee that matters.
• Normalize vectors, batch embeddings, pin the model version, and store text alongside vectors.
• When retrieval is bad, fix chunking first, then hybrid retrieval and re-ranking, and only last swap the embedding model.
• Know when not to use embeddings: exact identifiers, tiny corpora, and structured queries belong in a keyword or SQL layer.

Conclusion

Conclusion

Embeddings are a measurement problem, not a shopping problem: start standard, normalize, pin versions, and evaluate against a retrieval eval set built from your own corpus. When retrieval underperforms, fix chunking and retrieval strategy before swapping models. The vector database guide covers the storage half; the RAG guide the pipeline half.