Evaluating RAG Pipelines with RAGAS

A RAG pipeline has two moving parts: a retriever and a generator. When the final answer is wrong, you need to know which part failed. Did the retriever pull the wrong chunks? Did the LLM hallucinate despite good context? Or did the LLM ignore perfectly good context?

Standard NLP metrics like BLEU or ROUGE don't help here — they measure lexical overlap against a reference answer, which tells you almost nothing about factual faithfulness or retrieval quality. Human evaluation is reliable but doesn't scale. RAGAS (Retrieval-Augmented Generation Assessment) fills this gap with four complementary metrics that decompose RAG quality into distinct, measurable signals — all computable without a human in the loop for every sample.

Each metric probes a different failure mode. Faithfulness and Answer Relevancy evaluate the generator. Context Precision and Context Recall evaluate the retriever. You need all four to get a complete picture.

Faithfulness measures whether every claim in the generated answer is actually supported by the retrieved context. RAGAS implements this with an LLM-as-judge approach: it extracts atomic statements from the answer, then checks each statement against the context.

$\text{Faithfulness} = \frac{\text{statements supported by context}}{\text{total statements in answer}}$

A score of 1.0 means nothing in the answer contradicts or goes beyond the context. A score of 0.6 means 40% of the answer's claims were hallucinated — either invented or imported from the LLM's parametric memory.

Gotcha: Faithfulness does not check whether the answer is factually true in the world — only whether it's grounded in the context. A context full of wrong facts will still produce a high faithfulness score.

A faithful answer can still be useless if it answers the wrong question. Answer Relevancy catches this: RAGAS generates several reverse questions from the answer ("what question would this answer?"), then measures the average cosine similarity between those synthetic questions and the original question.

$\text{Answer Relevancy} = \frac{1}{n} \sum_{i=1}^{n} \cos(q, \hat{q}_i)$

High similarity means the answer maps cleanly back to the question. Low similarity means the answer drifted — perhaps answering a tangentially related question or adding irrelevant background.

Common failure modes:

• Verbosity drift: the answer pads out with correct but off-topic information.
• Paraphrase mismatch: embedding model struggles with domain-specific phrasing, deflating scores even for good answers.

Context Precision measures signal-to-noise in your retrieved chunks. Given the question and a ground-truth answer, it asks: of all the chunks you retrieved, what fraction were actually relevant?

$\text{Context Precision@K} = \frac{\sum_{k=1}^{K} (\text{Precision@}k \times \text{hit}_k)}{\text{total relevant items retrieved}}$

This is a ranking-aware metric. A relevant chunk at position 1 scores higher than the same chunk at position 5, because most LLMs pay more attention to early context. Low Context Precision means your retriever is fetching too many irrelevant chunks, which dilutes the useful signal and wastes the LLM's context window.

This is the metric to watch when you suspect your vector search is returning topically similar but factually unhelpful documents.

Context Recall checks coverage: does the retrieved context contain everything needed to answer the question? RAGAS uses the ground-truth answer as a proxy for "what the context should contain", then checks what fraction of statements in the reference answer can be attributed to the retrieved chunks.

$\text{Context Recall} = \frac{\text{reference statements supported by context}}{\text{total reference statements}}$

Low recall means your retriever is missing key chunks. This could be a chunking problem (a relevant fact spans a chunk boundary), an embedding problem (the query and document phrase the same concept differently), or a simple top-K problem (the relevant chunk is ranked 8th but you only fetch 5).

Note: Context Recall requires a ground-truth answer to compute. It's the one metric that can't be run zero-shot on live traffic without a labeled dataset.

RAGAS is a Python library that wraps OpenAI (or any LiteLLM-compatible model) for its LLM-as-judge calls.

pip install ragas datasets

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

data = {
    "question": ["What is the boiling point of ethanol?"],
    "answer": ["Ethanol boils at 78.37 °C at standard pressure."],
    "contexts": [["Ethanol (C2H5OH) has a boiling point of 78.37 °C (173.1 °F) at 1 atm."]],
    "ground_truth": ["The boiling point of ethanol is 78.37 °C."]
}

result = evaluate(
    Dataset.from_dict(data),
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)  # {'faithfulness': 1.0, 'answer_relevancy': 0.97, ...}

The contexts field is a list of lists — one list of chunk strings per question. ground_truth is only required for context_recall.

RAGAS uses GPT-4o by default, which is accurate but expensive at scale. You can swap it for any model via LangChain wrappers or the built-in LLM config.

from ragas.llms import LangchainLLMWrapper
from langchain_anthropic import ChatAnthropic
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import OpenAIEmbeddings

judge_llm = LangchainLLMWrapper(ChatAnthropic(model="claude-sonnet-4-6"))
judge_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy],
    llm=judge_llm,
    embeddings=judge_embeddings,
)

For large-scale CI runs, a smaller local model is practical. For final benchmark reports, use the strongest judge you can afford.

A golden dataset is a fixed set of (question, ground_truth_answer, expected_contexts) triples that you evaluate against on every pipeline change. Without it, you're flying blind.

How to build one:

1. Sample from real queries — pull 50-200 questions from production logs or support tickets. Real questions expose real failure modes.
2. Generate synthetic questions — RAGAS ships TestsetGenerator that uses your document corpus to create question/answer pairs automatically. Good for bootstrapping when you have no traffic yet.
3. Write adversarial cases by hand — multi-hop questions, negation questions ("what does X not do?"), and ambiguous phrasings. These are where RAG breaks most often.
4. Have a domain expert validate — spot-check at least 20% of auto-generated entries. LLM-generated ground truths carry LLM errors.

from ragas.testset import TestsetGenerator
generator = TestsetGenerator.from_langchain(llm, embeddings)
testset = generator.generate_with_langchain_docs(docs, testset_size=100)

RAGAS is a tool, not an oracle. Several failure modes evade it entirely:

• Factual correctness vs. faithfulness: a context can be wrong, the answer faithfully reproduces the wrong fact, and faithfulness scores 1.0. You need a separate factual grounding check for high-stakes domains.
• Multi-hop reasoning: if an answer requires chaining two facts from two different chunks, RAGAS doesn't verify that the chain of inference is valid — only that the final claim appears somewhere.
• Numerical precision: "the margin grew from 12% to 18%" may be grounded but the arithmetic conclusion ("a 50% improvement") can be wrong. RAGAS won't catch this.
• Tone and format: a factually perfect answer delivered in a confusing format gets a great RAGAS score but a bad user experience.
• Judge model bias: the LLM judge tends to favor verbose, confident-sounding answers. Calibrate periodically against a human-labeled subset.

Use RAGAS as your fast regression signal in CI, not as your final quality gate.

Run RAGAS on every PR that touches chunking strategy, embedding model, retriever parameters, or prompt templates. A drop of more than 0.05 in any metric should block the merge and trigger human review of the failing samples.

Metrics interact. A pipeline change that improves Context Recall (fetches more relevant chunks) often hurts Context Precision (also fetches more noise). You need to track the tradeoff, not just the individual numbers.

A useful heuristic dashboard:

Set metric floors, not targets. Decide the minimum acceptable score for each metric before deployment, and treat any score below the floor as a blocker — not a suggestion.