Hybrid Search, Tuning, and Running It in Production

Semantic search solves a real problem and creates a different one, and the failure is systematic rather than occasional.

Dense embeddings represent meaning, which is why a query about cardiac arrest retrieves a document about heart attacks. The cost is that they represent meaning approximately, and exact tokens get smoothed away.

So dense retrieval reliably struggles with product codes and SKUs, error codes, person and place names, version numbers, acronyms specific to your organisation, and any rare term the embedding model saw seldom during training. A query for error TS-4471 may retrieve documents about errors generally while missing the one page that names it.

This is not a tuning problem. It is what the representation does, and no index parameter recovers it.

Lexical search, meaning BM25 and its relatives, has the mirror properties. It matches exact terms precisely, handles rare tokens well because rarity is exactly what its weighting rewards, and fails completely when the query and document use different words for the same idea.

The two failure modes are close to complementary, which is why production systems run both and combine the results rather than choosing. That combination is the subject of the next step, and the retrieval-quality side of it is treated in more depth in the Advanced RAG cursus.

Combining a dense and a lexical result list is harder than it looks, because their scores are not comparable.

A cosine similarity of 0.82 and a BM25 score of 14.3 live on different scales with different distributions, and BM25 scores are unbounded and corpus-dependent. Adding them, or weighting and adding, requires normalisation that is itself fragile: min-max normalisation over a result set makes scores depend on which other results happened to be returned.

Reciprocal rank fusion sidesteps the problem by discarding scores and using only positions. Each document scores the sum, over the lists it appears in, of one divided by a constant k plus its rank in that list. The constant, conventionally 60, damps the influence of top positions so a document ranked well in both lists beats one ranked first in only one.

The properties that make it the default: no normalisation, no tuning beyond k, robust to wildly different score distributions, and it naturally rewards agreement between retrievers.

Its limitation is the flip side: throwing away scores discards genuine confidence information. Where the retrievers are well calibrated, weighted score fusion can beat it, at the cost of the normalisation work.

Tuning without a target is fiddling. The sequence that converges.

State the requirement first: recall at your retrieval width, and a latency budget at your real concurrency. Without both numbers there is no way to know when you are finished.

Build the ground truth. Brute-force exact neighbours for a few hundred real queries. This is the measuring instrument and everything depends on it.

Establish the ceiling. Run flat search and measure end-to-end retrieval quality. This is the best the embedding model can do, and it tells you whether your problem is the index or the embeddings. Teams routinely tune indexes for weeks when flat search shows the ceiling was the issue.

Start with defaults. HNSW at M of 16 and efConstruction of 200 is a reasonable starting point for most corpora.

Sweep the runtime dial. Vary ef, or nprobe, and plot recall against latency. This single curve answers most tuning questions and takes minutes.

If the curve cannot reach your recall target at acceptable latency, raise M or efConstruction and rebuild. Only then, since rebuilds are expensive.

If memory is the constraint, add quantization and re-measure, with rescoring if recall drops too far.

And re-measure after any change to the corpus or the embedding model, because recall is a property of the combination.

Memory is usually the binding constraint, and the arithmetic is simple enough to do before choosing a system.

Raw vector storage is the number of vectors times the dimension times bytes per component. Ten million vectors at 768 dimensions in float32 is 10,000,000 times 768 times 4, roughly 30 gigabytes.

HNSW adds edges: approximately the number of vectors times M times 2, times 4 bytes for a node identifier, since each node stores edges in both directions across layers. At M of 16 and ten million vectors that is roughly 1.3 gigabytes, so the graph overhead is modest relative to the vectors themselves.

Quantization changes the first term dramatically. Scalar quantization to int8 takes 30 gigabytes to 7.5. Product quantization at 96 bytes per vector takes it to under 1 gigabyte, at which point the graph edges dominate.

Then add: the metadata store, the original vectors if you are rescoring, replicas for availability, and headroom for the rebuild, since rebuilding usually needs the old and new index resident simultaneously.

The planning conclusion that surprises teams: at scale, whether you quantize is the decision that determines your infrastructure bill, far more than the choice between HNSW and IVF. And the rebuild headroom is the line item most often forgotten, which is how a cluster runs out of memory during a maintenance operation rather than under load.

Six problems that catch teams in production, none of which appear in a benchmark.

The embedding model changes. New model, different vector space, and every stored vector is now incomparable with new queries. Re-embedding the entire corpus is the only fix, so treat the model version as part of the index identity and plan migrations with a dual-write period.

Tombstones accumulate. A high-churn corpus fills with deleted-but-present nodes, degrading recall and latency, and requiring a rebuild nobody scheduled.

Filter selectivity shifts. A filter that matched thirty percent in testing matches two percent in production for some tenant, and the retrieval strategy that worked collapses.

Distribution drift. New documents cluster differently from the corpus the IVF centroids were trained on, lists become uneven, and recall falls with no code change.

The cold-start rebuild. Someone rebuilds during business hours and discovers it takes six hours and doubles memory.

And silent quality decay. Nobody re-measured recall after any of the above, so the system degraded for months while every dashboard looked normal.

The common structure is that all six are invisible without periodic measurement. The countermeasure is the same one the AI QA cursus prescribes: run the recall evaluation on a schedule, not only at launch, and alert on the number rather than on infrastructure metrics.

The choice is between a dedicated vector database, a vector extension to a database you already run, and a library you embed yourself.

A library such as FAISS or hnswlib gives maximum control, no additional service, and the lowest cost. You implement persistence, replication, filtering, updates and operations yourself, which is substantial work that is easy to underestimate.

An extension to an existing database, such as pgvector for PostgreSQL, is the option most teams underweight. Your data is already there, transactions and joins and filtering work normally, backups and access control already exist, and there is no new system to operate. It handles low millions of vectors comfortably. The argument against it is performance at very large scale and fewer specialised index options, both of which matter less than teams assume.

A dedicated vector database gives the best performance at scale, the most index options, purpose-built filtering, and managed operations if hosted. It costs a new system in your architecture, a new operational surface, and usually a subscription.

The honest default: if your data is already in PostgreSQL and you have under a few million vectors, use pgvector and revisit when a measurement says to. The most common architectural mistake in this area is adopting a dedicated vector database at a scale where an extension would have been simpler, cheaper and adequate, adding a system to operate for performance nobody needed.

Six claims worth carrying away.

Exact search is intractable at scale and classical indexes fail in high dimensions, so every production system trades recall for speed. That trade is a choice you make, and it has a number.

Measure recall. Brute-force ground truth on a few hundred real queries, compared against your index. Most teams never do it, which means they cannot tune, cannot detect degradation, and cannot tell an index problem from an embedding problem.

Start flat. Under a hundred thousand vectors it is fast enough, exactly correct, and has no parameters to get wrong.

Tune the runtime dial first, ef or nprobe, because it is free to change. Rebuild-requiring parameters come second.

Quantization is the decision that determines your infrastructure bill at scale, more than the index family, and rescoring is what makes aggressive compression viable.

And run hybrid search, because dense vectors systematically miss exact terms, codes and rare names, which is a property of the representation rather than a tuning failure.

The two things most likely to bite you are not algorithmic. They are filtering, which is harder than any tutorial suggests, and the day someone changes the embedding model and every stored vector becomes incomparable.