BM25: How Lexical Relevance Is Actually Computed

A query for "tomato soup recipe" against a big corpus might match two hundred thousand documents. Nobody reads two hundred thousand results; almost everybody reads ten. The entire commercial value of a search engine is concentrated in which ten float to the top.

That is scoring's job: assign every matching document a number expressing how well it satisfies this query, and sort. The score does not need to be meaningful in absolute terms, and is not, BM25 scores are unbounded and incomparable across queries, it only needs to order documents well.

The modern default, BM25, comes from the probabilistic retrieval tradition developed by Stephen Robertson, Karen Sparck Jones and colleagues, the Okapi BM25 line of work, and became Lucene's default scoring function, which makes it the out-of-the-box ranking of Elasticsearch, OpenSearch and Solr.

Key idea: BM25 is three intuitions with algebra attached: terms that are rare in the corpus carry more information; a term appearing repeatedly in a document matters more, but with sharply diminishing returns; and matches in long documents count for less than matches in short ones. Learn the three and the formula assembles itself.

In the query "the tomato harvest", the term "the" matches essentially every document; "tomato" matches a slice; "harvest" perhaps a sliver. A match on "harvest" is therefore evidence; a match on "the" is noise. Inverse document frequency turns that into a number: the fewer documents a term appears in, the larger its weight.

The BM25 variant, with N total documents and n(t) containing term t:

The shape is what matters: a term in half the corpus gets a weight near zero, a term in one document in a million gets a large one, and the curve moves smoothly between. The 0.5 terms smooth the edge cases, and the plus one inside the logarithm, Lucene's variant, keeps the value positive even for absurdly common terms.

Two practical consequences. IDF is why engines can often skip dedicated stop word lists: "the" earns a near-zero weight automatically, no configuration needed. And IDF is corpus-relative: "tomato" is a strong term in a general news corpus and a weak one inside a recipe database, so the same query ranks differently against different collections, exactly as it should.

A document mentioning "tomato" eight times is probably more about tomatoes than one mentioning it once. Early scoring, raw TF-IDF, took that linearly: eight mentions, eight times the credit. Linearity turned out to be both wrong and exploitable, wrong because the ninth mention of a word adds almost no new evidence about the document's topic, exploitable because stuffing a page with a keyword bought rank.

BM25's answer is saturation. The term frequency contribution passes through f times k1 plus 1, over f plus k1, which climbs steeply at first and flattens toward a ceiling.

The third correction handles an unfairness: a 40-word product title that mentions "tomato" once is intensely about tomatoes; a 4,000-word article that mentions it once, in passing, is not. Raw counting treats them identically.

BM25 discounts term frequency by how the document's length compares to the corpus average. The document's frequency denominator is inflated for long documents and deflated for short ones, scaled by the ratio of document length to average document length, so a match in a short, focused document outscores the same match diluted across a long one.

The strength of the discount is the second tuning knob, b, running from 0, no length normalisation at all, to 1, full proportional normalisation. The default 0.75 is a compromise that has survived decades of empirical testing.

When would you move it? Fields with meaningful, deliberate length differences want lower b: a title field, where every word was chosen, is often scored with b near zero, while body text keeps the default. Corpora where length just means thoroughness, encyclopaedia articles, may also want gentler discounting: a long comprehensive article is not diluted, it is complete. As with analysis, per-field configuration is the norm, not the exception.

Put the three ingredients together, summed over the query's terms:

Reading it aloud: for each query term, take its rarity weight, multiply by its saturated within-document frequency, where the saturation denominator has been stretched or shrunk by the document's relative length, and add the contributions up.

Notice what the formula does not know: word order, term proximity, which field matched, document freshness, popularity, whether anyone ever clicked this result. BM25 is a per-term, bag-of-words calculation. Everything on that missing list is added around BM25, phrase and proximity features, per-field scoring with boosts, freshness signals, and that layering is precisely the subject of the rest of the course.

A worked miniature makes the moving parts concrete. Corpus: 1,000 recipe documents, average length 100 terms. Query: "tomato bisque". Term statistics: "tomato" appears in 200 documents, "bisque" in 5. Constants at defaults, k1 1.2, b 0.75.

IDF: tomato gets ln((1000 - 200 + 0.5) / (200 + 0.5) + 1), about 1.61. Bisque gets ln((1000 - 5 + 0.5) / (5 + 0.5) + 1), about 5.20. The rare term is worth three times the common one before anyone counts occurrences.

Document A, 50 terms long, mentions tomato twice, bisque once. Its length factor is 1 - 0.75 + 0.75 times 50 over 100, which is 0.625. Tomato's contribution: 1.61 times (2 times 2.2) over (2 plus 1.2 times 0.625), about 2.57. Bisque's: 5.20 times 2.2 over (1 plus 0.75), about 6.53. Total: 9.10.

Document B, 200 terms, mentions tomato six times, no bisque. Length factor 1.75. Contribution: 1.61 times 13.2 over (6 plus 2.1), about 2.62. Total: 2.62.

Document A wins by a wide margin, and the arithmetic shows why in one sentence: one occurrence of the rare, on-topic term in a short document beat six occurrences of the common one in a long document. That single sentence is BM25's whole personality.

Knowing a tool's edges is knowing the tool, and BM25's edges are sharp and well-mapped.

• Vocabulary mismatch. BM25 scores term overlap. "Notebook" and "laptop" share no terms, so a query for one cannot rank documents about the other, however perfect the match in meaning. Synonym lists patch the known cases; the long tail of paraphrase stays broken. This is the gap embeddings exist to close, next lesson.
• No understanding of intent. The query "jaguar speed" scores identically whether the user means the cat or the car; nothing in term statistics can disambiguate it.
• Bag of words. "Dog bites man" and "man bites dog" score identically. Proximity and phrase machinery must be layered on where order matters.
• No quality signal. An authoritative page and a spam page with the same term statistics tie. Popularity, freshness, and behavioural signals live outside the formula.

Against these stands what BM25 does supremely well: exact vocabulary, identifiers, names, rare terms, no training data required, explainable scores you can decompose term by term, millisecond-cheap at any scale, and robust across domains with two knobs of tuning. Production search does not choose between BM25 and the modern alternatives; it runs BM25 as the reliable backbone and adds what the backbone cannot see, which is exactly where this course goes next.