RAG Evaluation in Production: Metrics, Tools, and Cadence

In classical ML, evaluation is clean: a fixed test set, a loss function, done. RAG breaks that model in three places.

First, quality has two orthogonal axes — did the retriever surface the right chunks, and did the generator use them faithfully? A perfect retriever paired with a hallucinating LLM still ships wrong answers. A faithful LLM paired with a poor retriever confidently answers from bad context.

Second, the system drifts at runtime — new documents enter the corpus, embedding models get upgraded, and user query distributions shift. A snapshot eval from launch day can become irrelevant within weeks.

Third, ground truth is expensive. Human annotations cost $5–50 per example; at scale that is not free. Production eval therefore combines cheap online signals (latency, citation rate, no-answer rate) with periodic offline sweeps against curated golden sets. The rest of this lesson builds that full picture.

Recall@k — of all the gold passages that should answer a question, what fraction appear in the top-k retrieved chunks? If your golden set says the correct chunk is passage #47 and you retrieve 5 chunks, recall@5 is either 1 or 0 for that query. Averaged over many queries, you get a number to track.

MRR (Mean Reciprocal Rank) — rewards systems that put the right chunk first. If the correct chunk appears at rank 3, its reciprocal rank is 1/3. MRR averages these across queries. Useful when LLMs attend most to the first chunks.

NDCG@k — generalises MRR to graded relevance: a score-3 chunk ranked above a score-1 chunk is better than the reverse. Overkill for simple factoid retrieval, essential when relevance has degrees (e.g., partial match vs. exact match).

Heads up: These metrics require labelled relevance judgements. Bootstrap them from your golden set (see Step 4) or use an LLM judge to label on-the-fly — but LLM judges introduce their own bias.

Once chunks are retrieved, you need to know whether the generated answer stays grounded in them.

• Faithfulness — every claim in the answer can be traced to a retrieved chunk. Measured by decomposing the answer into atomic statements, then checking each against the context. Scores near 1.0 mean no hallucination; 0.6 means 40% of claims are unverifiable.
• Answer relevance — does the answer actually address the question? An answer can be faithful but off-topic (e.g., retrieved the right document but answered a different sub-question).
• Context precision — of the retrieved chunks, how many were actually useful for generating the answer? A retriever that dumps 20 chunks to cover all bases hurts LLM attention and costs more tokens.

All three form the RAGAS triad. They can be computed reference-free (no human label needed) using an LLM-as-judge, which is their practical advantage at scale.

RAGAS (Retrieval Augmented Generation Assessment) runs LLM-as-judge pipelines for the faithfulness/relevance/precision triad with two lines of setup.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset

data = {
    "question": ["What is MVCC?"],
    "answer": ["MVCC uses snapshot isolation to avoid read locks."],
    "contexts": [["Multi-Version Concurrency Control creates snapshots..."]],
    # "ground_truth" is optional — needed only for context_recall
}

result = evaluate(
    Dataset.from_dict(data),
    metrics=[faithfulness, answer_relevancy, context_precision],
)
print(result)  # {'faithfulness': 0.92, 'answer_relevancy': 0.88, ...}

By default RAGAS calls GPT-4o for judging. You can swap in any OpenAI-compatible endpoint — Anthropic, Mistral, local Ollama — by setting llm and embeddings on the metrics. Each eval call costs roughly 3–6 LLM requests per question, so batch carefully.

TruLens wraps your RAG pipeline as a traced app and attaches feedback functions (eval metrics) that run automatically after each call. This makes it easy to log both operational data and quality scores in the same trace.

from trulens.apps.langchain import TruChain
from trulens.providers.openai import OpenAI
from trulens.core import Feedback
import numpy as np

provider = OpenAI()
f_faithfulness = Feedback(provider.groundedness_measure_with_cot_reasons,
                          name="Faithfulness").on_input_output()

tru_app = TruChain(
    my_rag_chain,
    app_name="support-rag",
    feedbacks=[f_faithfulness],
)

with tru_app as recording:
    response = my_rag_chain.invoke({"query": "What is MVCC?"})

After a session, tru.get_leaderboard() ranks app versions by metric. The built-in dashboard (port 7860) shows per-query drill-downs — which chunk was retrieved, what the judge said, where faithfulness dropped. Particularly useful when A/B-testing retriever changes.

Phoenix by Arize is designed for ongoing monitoring rather than offline evaluation sprints. It ingests OpenTelemetry-compatible traces and surfaces semantic clustering, latency percentiles, and embedding drift in one UI.

import phoenix as px
from openinference.instrumentation.langchain import LangChainInstrumentor
from opentelemetry.sdk.trace import TracerProvider

px.launch_app()  # opens at http://localhost:6006
LangChainInstrumentor().instrument(tracer_provider=TracerProvider())

# From here, every chain.invoke() is traced automatically.

Phoenix's killer feature is its embedding projector: it plots query and document embeddings in 2D (via UMAP), lets you colour by latency or quality score, and highlights queries that cluster far from any retrieved document — a visual signal that retrieval is struggling. It integrates with RAGAS for the metric layer, so you get traces + scores in the same tool.

A golden set is a curated collection of (query, expected answer or relevant passage) pairs used for offline regression testing. The temptation is to build it once and call it done — that's wrong.

Step 1 — seed from real traffic. Sample 200–500 production queries, stratified by topic cluster (use UMAP + HDBSCAN on query embeddings). Avoid sampling only the common ones; tail queries expose the worst failures.

Step 2 — annotate economically. Use LLM generation (e.g., GPT-4o) to propose an expected answer + relevant passages, then have 1–2 humans verify. Costs ~$0.02 per example in API fees plus 2–3 min human time. Target ~300 annotated pairs for a meaningful regression suite.

Step 3 — version the golden set. Store in a JSONL file under git or an artifact store. Each row: {"id": "q001", "query": "...", "relevant_passage_ids": [...], "reference_answer": "...", "added_date": "2025-06"}. Retire stale items quarterly — questions about a product that no longer exists pollute your metrics.

Retrieval drift happens when your document corpus mutates (new pages added, old pages updated or deleted) without a corresponding reindex — or when your chunking/embedding pipeline is upgraded. Both events can silently break previously-passing queries.

Detection signals:

• Recall@5 on the golden set drops by more than 5 percentage points vs. last week's baseline.
• The fraction of queries that return at least one chunk with cosine similarity > 0.75 falls.
• Mean retrieved-chunk age increases (staleness signal — your index isn't being refreshed).

Mitigation playbook:

1. Run a daily incremental reindex on any document modified in the last 24 h.
2. Gate full reindex deploys with a pre/post recall@k comparison on the golden set.
3. Emit a retrieval.top1_score metric from your retriever to your monitoring stack; alert if the 7-day moving average drops more than 8%.

Retrieval drift is the most common silent killer of RAG quality in the 3–12 month post-launch window.

Embedding drift occurs when the semantic meaning of queries or documents shifts — new terminology enters your domain (e.g., a new product name, an industry acronym) that your embedding model has never seen. The index was built with text-embedding-3-small; six months later your queries use jargon that clusters poorly in that model's space.

Measuring it: Compute the average pairwise cosine similarity between a rolling window of live query embeddings and the nearest cluster centroid from last month. A sustained drop (>0.04 over 30 days) signals vocab drift.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

def drift_score(current_embeddings, reference_centroids):
    sims = cosine_similarity(current_embeddings, reference_centroids)
    return sims.max(axis=1).mean()  # mean best-match similarity

Response options: Fine-tune or upgrade the embedding model, add hard-negative mining on the new vocabulary, or inject domain-specific terms via query expansion before embedding. Upgrading the model requires a full reindex — plan for it.

LLM-judge-based metrics cost money. You can't run RAGAS on every query at 10k QPD. Online metrics fill the gap — they're observable from logs without extra LLM calls.

Set alert thresholds on no-answer rate and citation rate first — they're the most sensitive early-warning signals. A sudden jump in no-answer rate often precedes retrieval failures that the golden-set sweep catches days later.

Running every eval every hour is financially and operationally unsustainable. Structure evals into three tiers:

Tier 1 — Always-on (free, from logs): No-answer rate, citation rate, latency, token waste. Computed from production logs with zero extra LLM calls. Refresh every 5 minutes. Alert on threshold breaches.

Tier 2 — Daily golden-set sweep (low cost): Run RAGAS faithfulness + recall@5 on the full golden set (300 examples × ~5 LLM calls ≈ 1,500 calls/day). At $0.002/call with GPT-4o-mini that's ~$3/day. Catches regressions within 24 h. Fail CI/CD gates if faithfulness drops below 0.80.

Tier 3 — Weekly deep dive (high cost, scheduled): Full RAGAS suite + human spot-check of 20 sampled answers + embedding drift score + retrieval drift. Run on weekends to avoid interfering with traffic. Budget ~$50–100/week for a mid-scale app.

Cost rule of thumb: Tier 1 is free. Tier 2 should cost < 1% of your LLM inference budget. Tier 3 < 5%. If these ratios invert, your golden set is too large or your judge model is too expensive — swap to GPT-4o-mini or a local judge.