Evaluating AI Agents: From Final Answers to Full Trajectories

Evaluating a language model is relatively straightforward: feed it a prompt, compare the output to a reference. Evaluating an agent is fundamentally different because an agent takes sequences of actions over time — browsing, writing code, calling APIs, reading outputs, revising — before producing a result.

This creates two failure modes that a simple pass/fail score can miss:

1. Fluke successes — the agent reached the right answer via a broken or lucky path (e.g., hardcoded the solution after a web search failed).
2. Fluke failures — the agent's reasoning was sound but a transient environment glitch caused the last step to fail.

A good eval framework needs to say something about how the agent solved the problem, not just whether it did. That distinction drives everything else in this lesson.

The simplest agent eval checks the terminal state: did the final output match the expected answer? This works well for narrow, verifiable tasks.

def eval_final_answer(agent_output: str, gold: str) -> bool:
    # Works fine for math, code with test suites, factual Q&A
    return normalize(agent_output) == normalize(gold)

Advantages:

• Cheap to run — no need to log or replay intermediate steps.
• Objective — no human or LLM judge required when gold answers exist.
• Easy to aggregate — a single accuracy number across N tasks.

Limitations:

• Doesn't penalize agents that use 40 tool calls to answer a 2-call question.
• Can't distinguish a model that understands the task from one that pattern-matched to the answer format.
• Breaks entirely for open-ended tasks where there is no single gold answer.

Final-answer evals are the right starting point — but treating them as the finish line is how you ship fragile agents.

A trajectory is the full sequence of (thought, action, observation) tuples the agent executes. Trajectory-level evals score intermediate steps, not just the outcome.

Common trajectory metrics:

Subgoal-based evaluation is especially powerful for long-horizon tasks. Instead of a single binary outcome, you define a DAG of required states — e.g., "file modified", "tests pass", "PR description accurate" — and score partial credit. This makes the benchmark resistant to agents that skip steps but still reach the correct terminal state by accident.

Heads up: Trajectory logging inflates storage costs quickly. Budget ~5-50 KB per trajectory depending on context length and number of tool calls.

SWE-bench (Princeton, 2024) sources real GitHub issues from popular Python repos (Django, Flask, scikit-learn, etc.) and asks agents to produce patches that make failing tests pass. The eval is fully automated: apply the patch, run the test suite, check the exit code.

Key numbers from the original paper:

• 2,294 tasks sampled from 12 repos.
• SWE-bench Verified (500 human-validated tasks) is now the de-facto leaderboard.
• As of early 2025, top agents solve ~40-55% of Verified tasks.

What makes SWE-bench hard:

• Tasks require reading unfamiliar codebases, not just writing code from scratch.
• Many issues involve multi-file edits; a patch touching the wrong file fails silently.
• Test suites were written by humans who assumed human understanding of context.

Common gotcha: Agents often overfit to SWE-bench by learning repo-specific patterns. A high SWE-bench score does not automatically transfer to your internal codebase.

WebArena (CMU, 2023) puts agents inside a self-hosted web environment with four realistic sites (an e-commerce store, a GitLab instance, Reddit, a map service) and asks them to complete 812 web tasks via browser actions. Success is checked by functional criteria — e.g., "cart contains exactly 3 items" — not pixel comparison.

OSWorld (2024) goes further: it runs agents inside a full desktop OS (Ubuntu, macOS, Windows snapshots) and evaluates tasks across real applications — LibreOffice, Chrome, VS Code, GIMP. Each task has a verifier script that inspects application state after the agent finishes.

Both benchmarks measure:

• Grounding accuracy — does the agent click the right element?
• Planning horizon — OSWorld tasks average ~5 steps, but hard tasks reach 15+.
• Robustness to UI variation — page layouts differ between runs.

Human baselines on OSWorld hover around 72%; top agents as of 2025 reach ~25-30%, revealing how much headroom remains.

GAIA (Meta AI / Hugging Face, 2023) evaluates assistants on 466 real-world questions that require multi-step reasoning over web, files, code, and images. Questions are split into three levels:

• Level 1 — answerable with a single web search + light reasoning.
• Level 2 — require chaining 3-5 steps across different modalities.
• Level 3 — involve >5 steps, often with ambiguous or noisy intermediates.

What distinguishes GAIA from other benchmarks is that answers are always short, unambiguous strings (a number, a name, a date) — which eliminates judge subjectivity entirely. Yet the path to that answer can be arbitrarily complex.

Human performance on GAIA is ~92%. GPT-4 with tools scored ~15% on Level 3 in the original paper; frontier agents have pushed this to ~40-50% since. The gap between human and agent performance is almost entirely in planning and error recovery, not raw knowledge.

When gold answers don't exist — open-ended writing, multi-step plans, research summaries — practitioners use a stronger LLM to score the agent's output. This works, up to a point.

def llm_judge(task: str, trajectory: str, rubric: str, model="claude-opus-4") -> dict:
    prompt = f"""Score the following agent trajectory 0-10 on this rubric:\n{rubric}
\nTask: {task}\nTrajectory:\n{trajectory}
\nReturn JSON: {{\"score\": int, \"reason\": str}}"""
    return json.loads(call_model(model, prompt))

Known failure modes:

• Self-preference bias — GPT-4 rates GPT-4 outputs higher; Claude rates Claude outputs higher (Panickssery et al., 2024).
• Verbosity bias — longer outputs often score higher regardless of quality.
• Position bias — in pairwise comparisons, the first option wins more often.
• Prompt sensitivity — rephrasing the rubric can shift scores by 2-3 points on a 10-point scale.

Mitigations: use a model from a different provider as the judge; run pairwise comparisons with position swap; include concrete rubric anchors ("score 3 means the agent completed the task but made one factual error").

Before trusting an LLM judge in production, measure its calibration against human labels on a held-out set.

from sklearn.metrics import cohen_kappa_score

human_scores = [3, 7, 5, 8, 2, 9, 4, 6]  # 0-10 scale
llm_scores   = [4, 7, 4, 9, 3, 8, 5, 7]

# Bin to 3 levels: low/mid/high
def bin_score(s): return 0 if s < 4 else (1 if s < 7 else 2)
kappa = cohen_kappa_score(
    [bin_score(s) for s in human_scores],
    [bin_score(s) for s in llm_scores]
)
print(f"Cohen's kappa: {kappa:.2f}")  # aim for > 0.6

A kappa below 0.4 means your judge is barely better than chance at distinguishing task quality levels — collect more human labels and refine the rubric before using it to compare agents.

Also track agreement on direction: if your judge says Agent A is better than Agent B, does a human agree 80%+ of the time? Directional agreement is often more actionable than absolute score correlation.

The most important thing about agent evals is not which benchmark you pick — it's treating the eval as the authoritative specification of what your agent is supposed to do.

This has practical consequences:

1. Write the eval before the agent. If you can't define success criteria before building, you don't understand the task well enough.
2. Eval failures are product bugs. An agent that scores 70% on your internal benchmark means 30% of real user sessions are broken. Treat each failure case as a filed issue.
3. Benchmark drift is real. SWE-bench Verified tasks have leaked into training data for some models; always maintain a private held-out split for trusted comparisons.
4. Coverage beats precision. A broad eval covering 200 diverse real user tasks beats a perfectly designed eval covering 20 cherry-picked tasks.

The discipline of writing evals first forces you to be concrete about edge cases, error recovery, and acceptable partial solutions — exactly the things that kill agents in production.

Public benchmarks measure the industry average; you need an eval that measures your agent on your tasks. A minimal pipeline:

@dataclass
class EvalTask:
    task_id: str
    prompt: str
    environment: str          # "web", "filesystem", "api"
    verifier: Callable        # returns (passed: bool, partial_score: float)
    subgoals: list[str]       # intermediate checkpoints
    max_steps: int = 25
    timeout_s: int = 120

def run_eval(agent, tasks: list[EvalTask]) -> EvalReport:
    results = []
    for task in tasks:
        traj = agent.run(task.prompt, max_steps=task.max_steps)
        passed, score = task.verifier(traj.final_state)
        subgoal_hits = sum(sg in traj.visited_states for sg in task.subgoals)
        results.append({
            "task_id": task.task_id,
            "passed": passed,
            "score": score,
            "steps": len(traj),
            "subgoals": subgoal_hits / len(task.subgoals),
        })
    return EvalReport(results)

Keep 20% of your tasks as a blind hold-out. Run evals on every agent version before merging, and track trends over time — a 5% regression on a 200-task eval is a real signal, not noise.