Measuring Relevance: Judgments, NDCG, and the Click Trap

Every search team eventually lives this meeting: someone important searched for something, disliked the third result, and now wants the ranking changed. The change is made, that query improves, and nobody notices the two hundred queries that got worse.

Relevance tuning without an evaluation harness is exactly whack-a-mole, because every knob in the previous lessons, analyzer choices, BM25 parameters, fusion depths, boosts, moves thousands of queries at once. The only way to know a change helped is to measure the whole system before and after, on a fixed set of queries with known right answers.

That harness has two halves, and mature teams run both:

• Offline evaluation: a curated set of queries with human-labelled relevance judgments, scored with the metrics this lesson builds. Runs in seconds, before anything ships.
• Online evaluation: real user behaviour, clicks, reformulations, abandonment, measured in production, ideally through controlled experiments.

Key idea: offline metrics tell you whether the ranking got better against a frozen definition of good. Online metrics tell you whether users noticed. The two disagree often enough that running only one is a strategy for surprising yourself.

Offline evaluation stands on a judgment set: a sample of queries, each with documents labelled for relevance. Everything downstream is arithmetic; the judgments are the actual encoded opinion about what good search means, which makes their construction the most consequential step.

What experience says about building one:

• Sample queries from real traffic, stratified deliberately: head queries because they carry volume, tail queries because they carry the failures, and the known-item queries, SKUs, names, exact titles, that earlier lessons showed to be their own regime.
• Grade, do not binarise. A four-level scale, irrelevant, marginal, relevant, perfect, captures that the ideal result and an acceptable one differ, which binary labels erase and which NDCG, below, is built to use.
• Write a labelling rubric with examples, because two honest people disagree on raw relevance surprisingly often; the rubric is where those disagreements get settled once instead of per-label.
• Judge the document against the query, not the system. Labels must not depend on what the current engine returns, or the evaluation inherits the very biases it exists to catch. Pool candidates from several systems and sources before judging.
• Refresh on a schedule. Corpora drift, intent drifts, and a judgment set from two years ago quietly measures a search engine for a site that no longer exists.

With judgments in hand, the simple metrics come first, and each answers a different operational question.

Each has a natural home. Precision at 10 suits browsing pages where the user scans everything shown. Recall at 100 is the retrieval stage's report card in the funnel from last lesson: the re-ranker cannot promote a document that retrieval never surfaced, so recall at the fusion depth bounds everything downstream. Mean reciprocal rank fits known-item queries, where exactly one document is the answer and the only question is how far down it sits: first place scores 1, second scores 0.5, fifth scores 0.2, and the average over queries is brutally interpretable.

What none of them see: that the top of the page matters more than the bottom within the k window, and that a perfect result differs from a merely acceptable one. Both blind spots are fixed by the metric that has become the industry default.

Normalised discounted cumulative gain builds in the two missing pieces: graded labels, and the fact that rank 1 is worth more than rank 5.

Start with discounted cumulative gain. Each result contributes a gain from its relevance grade, discounted by its position:

The exponential numerator makes high grades count disproportionately, a grade-3 result contributes 7, a grade-1 contributes 1, and the logarithmic denominator taxes depth gently: position 1 divides by 1, position 3 by 2, position 7 by 3.

Raw DCG depends on how many relevant documents a query happens to have, so queries cannot be compared or averaged. The fix is normalisation: divide by the ideal DCG, the score of the same judged documents arranged in the best possible order.

The result lands in 0 to 1, where 1 means the ranking is exactly as good as the judgments allow, and 0.8 means genuine headroom remains. Averaged over the judgment set, NDCG at 10 is the single number most search teams watch, precisely because it moves when a good document climbs from rank 6 to rank 2, which is the kind of improvement users feel and coarser metrics miss.

Compute it once and the formula stops being a formula. A query returns five results with graded judgments: ranks 1 through 5 carry grades 2, 3, 0, 1, 0.

Gains: grade 2 gives 3; grade 3 gives 7; grade 0 gives 0; grade 1 gives 1; grade 0 gives 0.

Discounts by position: divide by log2 of position plus one, so 1.0, 1.585, 2.0, 2.322, 2.585.

DCG: 3 over 1, plus 7 over 1.585, plus 0, plus 1 over 2.322, plus 0. That is 3 plus 4.416 plus 0.431, totalling 7.85.

The ideal ordering sorts grades descending, 3, 2, 1, 0, 0: IDCG is 7 over 1, plus 3 over 1.585, plus 1 over 2, which is 7 plus 1.893 plus 0.5, totalling 9.39.

NDCG: 7.85 over 9.39, which is 0.84.

Offline metrics grade the ranking against frozen human opinion. Online metrics ask the users, and the users answer with behaviour: click-through rates, time to first click, reformulation rate, how often a search session ends with nothing clicked, and downstream outcomes like add-to-cart from search.

The naive reading of that behaviour fails, and it fails in one specific, well-documented way.

The consequences run deep. Click data used naively to train or tune rankers becomes a feedback loop: whatever ranks first collects the clicks that then justify ranking it first. The escape routes are deliberate: interleaving experiments, which mix two rankers' results into one list and credit the source of each click, controlled A/B experiments on ranking changes, and click models that explicitly estimate and remove the position effect before the data is trusted.

Assemble the pieces into the operating rhythm that separates search teams from search superstition.

1. Every ranking change runs offline first. The judgment set and NDCG at 10, plus recall at fusion depth for retrieval changes, gate the merge. Seconds of compute, and regressions die in review instead of production.
2. Slice before celebrating. An aggregate NDCG gain can hide a known-item collapse: the hybrid change that lifts paraphrase queries may bury exact SKUs. Report per-stratum, head, tail, known-item, at minimum, because the strata fail independently.
3. Material changes ship as experiments. Offline improvement is a hypothesis; the interleaving or A/B result is the verdict. When offline and online disagree, and they will, the judgment set just learned something: mine the disagreeing queries and fix the rubric or the sample.
4. Watch the zero-results rate and reformulations in production continuously. They are the cheapest early warnings that reality has drifted away from the judgment set.
5. Feed failures back. Queries that users reformulate, abandon, or escalate become next quarter's judgment set additions, which is how the frozen definition of good tracks a moving product.

None of this is glamorous, and all of it compounds: the team with the loop makes twenty small verified improvements a quarter, and the team without it makes twenty confident changes, of unknown sign.