Lexical Meets Vector: Hybrid Search and Rank Fusion

By this point in the course, lexical retrieval is familiar: match query terms against indexed terms, score with BM25. Its theory of similarity is vocabulary overlap.

Vector retrieval runs on a different theory: meaning lives in geometry. An embedding model maps each document, and each query, to a point in a high-dimensional space, trained so that texts with similar meaning land near each other. Retrieval becomes nearest-neighbour search: embed the query, find the closest document vectors, return them. The approximate-nearest-neighbour machinery that makes this fast at scale, and the databases built around it, are covered in depth in the Vector Databases in Depth course; here we treat that layer as available and focus on when to use it.

The crucial property: the two theories are not better and worse versions of each other. They are orthogonal, and each is precisely blind where the other is sharp.

Key idea: lexical search fails when the same meaning wears different words. Vector search fails when exactness itself is the meaning. Every production search decision in this lesson flows from those two sentences.

Concrete failure cases sharpen the abstraction. Both columns are everyday production traffic, not corner cases.

The pattern behind the table: embeddings compress text into a fixed-size representation of its gist. Compression preserves neighbourhoods of meaning and destroys surface detail. Identifiers, error codes, part numbers and names are pure surface detail: their meaning is their exact spelling, which is exactly what the compression throws away. Meanwhile lexical matching is all surface: it keeps every character of the identifier and none of the gist.

Neither engine can be configured out of its blindness, because the blindness is the mechanism. That is why the answer is structural.

The production pattern is disarmingly simple: send every query to both retrievers in parallel, take the top candidates from each, and merge. No router tries to guess in advance whether this query is a lexical query or a semantic one.

Why not route? Because the guess is hard exactly where it matters. "XR-2000 replacement battery" is half identifier, half intent, in one query. A misrouted query gets the wrong engine's blindness applied to it silently, and routing errors are invisible in aggregate metrics because each engine looks fine on the queries it received. Running both costs one extra retrieval, cheap at top-k depths, and removes an entire class of silent failure.

The real design question is the merge. Each retriever returns a ranked list with scores, and the scores are incomparable: BM25 emits unbounded sums that vary by query length and corpus statistics; vector search emits cosine similarities in a narrow band near 1. Normalising one scale onto the other, min-max squashing per result list, is fragile: it is sensitive to outliers and to how many results each side returned, and it silently reweights whenever either distribution shifts.

The robust answer, used across Elasticsearch, OpenSearch and most modern stacks, abandons scores entirely and fuses on rank alone.

Reciprocal rank fusion, RRF, introduced by Cormack, Clarke and Buettcher, scores each document by where it ranked in each list, ignoring the raw scores completely:

The constant k, conventionally 60, damps the difference between adjacent top ranks so that one retriever's first place does not steamroll everything else.

Why this works so well is worth internalising: ranks are the one thing both retrievers express in a common currency. A document at rank 1 on either list is that engine's best answer, regardless of what its raw score was. Documents that appear high on both lists accumulate two healthy contributions and float to the top, which is precisely the behaviour you want: agreement between two orthogonal theories of relevance is the strongest signal available.

RRF is also nearly configuration-free, which is not a small virtue. Score normalisation schemes come with weights that must be tuned and re-tuned as models and corpora drift. RRF has one insensitive constant. In evaluations across many collections it fuses about as well as carefully tuned alternatives, and it cannot be silently broken by a score distribution shift, which in production is the property that matters.

One worked merge makes RRF permanent. Both retrievers answer the query "notebook won't charge", top three each, k at 60.

Lexical list: doc A (rank 1), doc B (rank 2), doc C (rank 3). Vector list: doc D (rank 1), doc A (rank 2), doc E (rank 3).

One practical footnote: fusion depth matters more than fusion formula. Fusing top-3 lists, as here, is illustrative; production fuses top-50 to top-200 from each retriever so that documents ranked moderately by both sides have the chance to accumulate their agreement bonus before the final cut.

Assembled, a modern search pipeline is a funnel with stages that get more expensive as the candidate set gets smaller.

The query fans out to both retrievers. Each produces its ranked candidates cheaply over the whole corpus, BM25 over postings, approximate nearest neighbour over vectors. RRF fuses the lists into one candidate set of perhaps a hundred documents.

Then, optionally, a re-ranker: a heavier model, typically a cross-encoder that reads the query and each candidate document together, rescores the fused short-list. Cross-encoders are far more accurate than either retriever because they model the interaction between query and document directly, and far too slow to run over a corpus, which is why they only ever see the short-list. The funnel shape is the same economics as the two-stage recommender architecture: spend almost nothing per document on millions, spend heavily on a hundred.

Business logic applies last: filters the user set, freshness boosts, diversity rules, pinned results.

Each stage is independently measurable, and when quality regresses, the first diagnostic question is always which stage lost the good document: did retrieval miss it, did fusion bury it, or did the re-ranker misjudge it?

The full pipeline is not the starting point; it is the ceiling. An honest build order, each step justified by a measured gap rather than by fashion:

1. BM25 alone, with good analysis. For catalogues dominated by identifier and known-item queries, this is often 90 percent of achievable quality at 10 percent of the complexity. Many teams should stop here longer than they do.
2. Add vector retrieval and RRF when the measured failures are vocabulary-mismatch failures: support search, documentation, conversational queries. This is the big step: two indexes to operate, an embedding model to version, reindexing on model change.
3. Add a re-ranker when fused retrieval reliably surfaces the right documents into the top hundred but orders them poorly. It attacks precision at the top, not recall.
4. Query understanding, classification, spelling, entity extraction, where the query stream itself is messy.

The discipline that makes the order work is measurement: each step should be adopted because an evaluation showed the previous stage's specific failure, and kept because the same evaluation showed the addition fixed it. Which raises the obvious question this course has so far dodged: how do you measure whether a search system got better? That is the final lesson, and it is where every serious search effort actually begins.