LLM evaluation: how to know your model output is actually good

Traditional unit tests assert exact outputs: assert f(2) == 4. LLMs produce text that's correct in many possible forms — different phrasing, different structure, different tone — all acceptable. There's no canonical equality.

You also have failure modes traditional tests don't catch: hallucinations, tone drift, format violations, partial correctness, regression-on-edge-cases. Every behaviour you care about needs its own way of being measured.

LLM evaluation is closer to product analytics than to unit testing. You're sampling behaviour over a population of inputs and watching aggregate quality move. You can't assertEqual your way through it.

Practical eval strategies, roughly in order of cost and signal:

1. Golden-set tests — a small curated dataset of inputs with expected behaviours. Cheap, deterministic, catch regressions. Doesn't generalise.
2. LLM-as-judge — use another LLM to score outputs against a rubric. Scales, but the judge has its own biases.
3. Human evaluation — paid raters or experts. Gold standard but slow and expensive.
4. Online metrics — real users in production: click-through, completion rate, thumbs-up, retention. Most ecologically valid but lagging.

A mature eval stack uses all four. Each catches what the others miss.

Build a fixture set of 20–200 (input, expected-behaviour) pairs that cover your tricky cases. Behaviours are checked by assertions, not exact match:

def test_no_refund_made_up(case):
    out = pipeline(case["input"])
    assert case["customer_id"] in out                      # cites the right customer
    assert "$" in out                                       # mentions money
    assert not re.search(r"\$[0-9]{3,}", out)               # but not big numbers we don't have
    assert any(p in out for p in ["refund", "credit"])     # mentions the action

The checks combine must-include, must-not-include, regex shape, and occasionally LLM-judge for fuzzier rules. Run on every prompt change. New failure modes you encounter in production go into the set as new cases.

An LLM scores outputs against a rubric. Looks like:

judge_prompt = """Given USER_QUESTION and ASSISTANT_REPLY, score the reply 1-5 on:
- Factuality: does it match the source?
- Relevance: does it answer what was asked?
- Tone: is it neutral and professional?
Return JSON: {"factuality": int, "relevance": int, "tone": int, "reasoning": str}"""

Known biases:

• Length bias: judges prefer longer answers, often regardless of quality.
• Position bias: when comparing A vs B, the first option scores higher more often than chance.
• Self-preference: an LLM rates outputs from its own model family higher.

Mitigations: randomise A/B order, use a different model family as judge, calibrate the judge against human scores on a small subset.

Golden + LLM-judge + occasional human spot-check.

Two kinds of drift bite even well-tested systems:

1. Model drift: the vendor updates the model. "gpt-4o" today is not exactly "gpt-4o" six months ago. Your eval scores can move without any code change.
2. Input drift: real-world usage diverges from your golden set. The set was hard six months ago; today's hard cases look different.

Mitigations: pin model versions where the API allows (Anthropic's date-stamped IDs, OpenAI's date-suffix names). Refresh golden sets quarterly with new examples from production logs — especially failure cases users complained about.

You don't have to build this from scratch:

• LangSmith — paid, polished, integrates with LangChain/LangGraph; good UI for traces + eval comparison.
• Promptfoo — open source, YAML-config evals, great for CI integration.
• Inspect AI — built by the UK AI Safety Institute, strong for capability/safety evals.
• Phoenix — Arize's open-source LLM observability + evals, runs locally.
• Helicone, Langfuse — observability-leaning; you can attach evals.

For most teams: start with golden tests in pytest + LLM-as-judge in a small script. Add a managed tool when you have multiple developers iterating on prompts.

If you're starting evals from zero, do these in order:

1. Today: collect 20 inputs that worry you. Run them on your current pipeline. Note what you see.
2. This week: write deterministic assertions for the 5 most important failure modes. Wire them into CI.
3. Next week: add an LLM-judge for the fuzzy ones (factuality, tone). Sanity-check 10 of its scores against your own judgement.
4. Once you have logs: stream a sample of production conversations into the eval pipeline. The new bugs will be there.
5. Quarterly: refresh the golden set with real failures, retire stale cases.

The single biggest mistake is waiting for the "right" eval framework. Start with pytest and a JSON fixture file. You'll learn what you actually need to measure.