Hybrid Retrieval for RAG: BM25, Dense Vectors, and Cross-Encoder Reranking

Most RAG tutorials show a single vector search step: embed the query, find the top-k nearest chunks, stuff them into the prompt. In practice this breaks in predictable ways.

Dense-only retrieval fails on exact-match needs — product codes, version numbers, rare proper nouns. The embedding model smears meaning across 768+ dimensions; a query for CVE-2024-3094 returns vaguely similar security content, not the exact chunk that mentions that string.

Sparse-only retrieval (BM25) fails on semantic gaps — a document about "cardiac arrest" won't surface for the query "heart stopped beating" unless those exact tokens appear. It also can't handle multilingual or paraphrase-heavy queries.

The solution is a three-stage pipeline: (1) sparse retrieval via BM25, (2) dense retrieval via bi-encoder embeddings, (3) fusion + cross-encoder reranking. Each stage has a different cost-recall tradeoff, and stacking them gets you both recall breadth and precision at the top.

BM25 scores a document $d$ for query $q$ as:

$\text{BM25}(d,q) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t,d) \cdot (k_1 + 1)}{f(t,d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl}})}$

where $f(t,d)$ is term frequency, $k_1 \approx 1.2$ controls TF saturation, and $b \approx 0.75$ controls document-length normalization. The key insight: BM25 saturates on repeated terms (unlike raw TF-IDF), and it penalizes long documents proportionally.

For RAG, BM25 is almost free to run — it's an inverted index lookup. Use rank_bm25 in Python or Elasticsearch's built-in BM25 scorer.

from rank_bm25 import BM25Okapi

corpus = [doc.split() for doc in documents]  # tokenize
bm25 = BM25Okapi(corpus)

query_tokens = query.lower().split()
scores = bm25.get_scores(query_tokens)  # shape: (n_docs,)
top_indices = scores.argsort()[-100:][::-1]

In production, prefer Elasticsearch or OpenSearch — they handle scale, persistence, and language-specific analyzers (stemming, stopwords).

Bi-encoders embed the query and each document independently, then use cosine or dot-product similarity to find candidates. The key constraint: both sides are encoded separately, so the model can't attend to query-document token interactions.

Model choice matters a lot here. Some benchmarks on BEIR (a popular retrieval benchmark):

import voyageai

vc = voyageai.Client()

# Encode documents at index time
doc_embeddings = vc.embed(documents, model="voyage-3", input_type="document").embeddings

# Encode query at retrieval time
query_embedding = vc.embed([query], model="voyage-3", input_type="query").embeddings[0]

Heads up: always use input_type="query" vs "document" when the model supports it — they optimize the embedding direction differently and retrieval quality drops without this distinction.

You now have two ranked lists of candidate chunks — one from BM25, one from dense retrieval. How do you merge them without knowing the score scales?

Reciprocal Rank Fusion (RRF) sidesteps score normalization entirely by using only rank positions:

$\text{RRF}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \text{rank}_r(d)}$

where $k=60$ is a smoothing constant. Documents that rank highly in both lists get a much higher combined score.

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranked_list in rankings:
        for rank, doc_id in enumerate(ranked_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

fused = reciprocal_rank_fusion([bm25_ids[:100], dense_ids[:100]])
candidates = fused[:100]  # pass top-100 to reranker

RRF consistently outperforms linear score interpolation in practice because it's robust to score distribution differences. The 60-constant works well empirically — only tune it if you're doing careful ablations.

A cross-encoder takes a (query, document) pair as a single input and outputs a relevance score. Unlike bi-encoders, it can attend to every query token against every document token via full self-attention — this is why it's significantly more accurate.

The tradeoff: you can't precompute document embeddings. Every query requires a forward pass over each candidate. This is why cross-encoders are used only at the reranking stage, on a reduced candidate set (typically 50-200 docs), not over the full corpus.

Latency comparison for 100 candidates:

The accuracy gain is substantial — typically +4-8 NDCG points on hard retrieval tasks. For most RAG use cases, that precision improvement is worth the latency.

Cohere's rerank API is the simplest way to add cross-encoder reranking — one HTTP call, no model hosting.

import cohere

co = cohere.Client(api_key="YOUR_KEY")

result = co.rerank(
    model="rerank-v3.5",
    query=query,
    documents=[chunks[i] for i in candidate_indices],
    top_n=5,  # return top 5 after reranking
    return_documents=True,
)

for item in result.results:
    print(f"score={item.relevance_score:.3f}: {item.document.text[:120]}")

Cohere Rerank v3.5 supports up to 4096 tokens per document and handles multilingual content across 100+ languages — useful if your corpus mixes English and another language.

Heads up: relevance_score is not a probability. It's a log-softmax output — scores cluster between -10 and 0 for low-relevance docs and approach 0 for high-relevance. Don't use a fixed threshold like > 0.5; use rank instead.

Voyage rerank-2 is Voyage AI's cross-encoder, competitive with Cohere on code-heavy and technical corpora. It's designed to pair naturally with voyage-3 embeddings.

import voyageai

vc = voyageai.Client()
result = vc.rerank(
    query=query,
    documents=[chunks[i] for i in candidate_indices],
    model="rerank-2",
    top_k=5,
)
for r in result.results:
    print(r.relevance_score, r.document)

BGE-Reranker (BAAI) is the leading open-weight cross-encoder. BAAI/bge-reranker-v2-m3 supports 100+ languages and runs locally via sentence-transformers:

from sentence_transformers import CrossEncoder

model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
pairs = [(query, chunks[i]) for i in candidate_indices]
scores = model.predict(pairs)  # numpy array of relevance scores
top_indices = scores.argsort()[::-1][:5]

Choose BGE when you need zero marginal API cost at scale, want to fine-tune on domain data, or are operating in an air-gapped environment. Cohere/Voyage are better defaults for prototyping and when latency SLAs matter more than cost.

Here's a minimal end-to-end hybrid retrieval function that combines all three stages:

import cohere
from rank_bm25 import BM25Okapi
import numpy as np

def hybrid_retrieve(query: str, chunks: list[str], dense_embeddings: np.ndarray,
                    co: cohere.Client, top_k: int = 5) -> list[str]:
    # Stage 1: BM25
    tokenized = [c.lower().split() for c in chunks]
    bm25 = BM25Okapi(tokenized)
    bm25_scores = bm25.get_scores(query.lower().split())
    bm25_top = bm25_scores.argsort()[-100:][::-1].tolist()

    # Stage 2: Dense (assumes embeddings precomputed)
    import voyageai
    vc = voyageai.Client()
    q_emb = np.array(vc.embed([query], model="voyage-3", input_type="query").embeddings[0])
    sims = dense_embeddings @ q_emb  # cosine sim if normalized
    dense_top = sims.argsort()[-100:][::-1].tolist()

    # Stage 3: RRF fusion
    fused = reciprocal_rank_fusion([bm25_top, dense_top])[:100]
    candidates = [chunks[i] for i in fused]

    # Stage 4: Cross-encoder rerank
    result = co.rerank(model="rerank-v3.5", query=query,
                       documents=candidates, top_n=top_k)
    return [r.document.text for r in result.results]

This function keeps BM25 and dense retrieval in Python for simplicity; in production you'd replace both with Elasticsearch hybrid queries or Qdrant's built-in sparse+dense support.

Low recall before reranking. If the right chunk never makes it into the candidate pool, the reranker can't save you. Check your candidate set size — 100 is a reasonable default. Evaluate with Recall@100 on a held-out query set before optimizing NDCG@5.

Chunk size mismatch. Rerankers score individual passages. If your chunks are 2000+ tokens, the cross-encoder is scoring a wall of text against a short query — it degrades. Keep chunks to 200-500 tokens for reranking; store the full paragraph context separately to inject into the prompt after retrieval.

Query-time latency spikes. The reranker is the bottleneck. Mitigations:

• Reduce candidate pool: 50 instead of 100 often loses <1% NDCG
• Cache reranker outputs for frequent queries (query hash → ranked list)
• Use bge-reranker-v2-small locally if Cohere latency is too high

Score calibration confusion. Don't filter by absolute relevance score — it varies by model and query length. Filter by rank (keep top-k) or by score relative to the top result (drop anything below 10% of top score).

Heads up: always run an offline eval before deploying a new reranker. NDCG on BEIR doesn't always transfer to your specific domain.

Hybrid retrieval has overhead — don't add stages you don't need.

Skip BM25 when: your corpus is code (syntax is better captured by dense embeddings), you're doing multilingual retrieval with no shared vocabulary, or your index is tiny (<10k chunks) and dense search already saturates recall.

Skip reranking when: latency is under 200ms SLA and your bi-encoder quality is already good enough (run offline eval to confirm), or you're doing a non-critical first-pass retrieval that a downstream step will filter further.

Skip dense retrieval when: your use case is pure keyword search (legal citations, code identifiers, serial numbers) and your users phrase queries precisely. BM25 alone can be 90% of the way there.

A useful heuristic: if your query set is short and exact (product SKUs, error codes), bias toward BM25. If queries are long and paraphrastic ("explain why the server keeps running out of memory"), bias toward dense. Most enterprise RAG falls somewhere between — which is exactly why hybrid exists.