Measuring a Harness Change: Trials, Error Bars and pass^k

The word harness has a second meaning in AI, and harness engineering needs both. Anthropic's Demystifying evals for AI agents separates them:

Definition: An agent harness (or scaffold) is "the system that enables a model to act as an agent". An evaluation harness is "the infrastructure that runs evals end-to-end": it provides instructions and tools, runs tasks concurrently, records every step, grades outputs and aggregates results.

Open-source evaluation harnesses include EleutherAI's lm-evaluation-harness for model benchmarks, the UK AI Security Institute's Inspect framework for agentic and model evals, and Harbor, which runs Terminal-Bench tasks in containers against many agents.

Every change to the agent harness from the earlier lessons, a new lint rule, a shorter instruction file, a sandbox, is a hypothesis. The evaluation harness is how you test it. The benchmark landscape itself (SWE-bench, WebArena, trajectory grading) is covered in Evaluating AI Agents; this lesson is about the statistics of deciding whether your change helped.

Anthropic's post gives a compact vocabulary that maps directly onto what an evaluation harness records:

Two recommendations from the same post shape how the rest of this lesson works. Grade outcomes rather than the exact path the agent took, since agents often find valid routes you did not anticipate. And read transcripts: "You won't know if your graders are working well unless you read the transcripts and grades from many trials." A score that rises because the grader became easier to fool is not an improvement.

An agent's success on a task is a random event. Run the same harness twice and some tasks flip. Evan Miller's Adding Error Bars to Evals (2024) treats an eval as an experiment and starts from the standard error of a success rate over $n$ tasks:

For a benchmark the size of Terminal-Bench 2.0, $n = 89$, at $p = 0.5$ this gives an SE of 5.3 points, so a 95% interval of roughly plus or minus 10 points on the question "how would this agent do on tasks like these?".

That interval includes the luck of which tasks were chosen. A narrower question, "how would it do again on these same 89 tasks?", has less noise, but still some, because each trial is random. Terminal-Bench's authors ran every agent and model combination at least five times, and report 95% intervals mostly between about 2 and 3 points.

Key idea: A single eval run gives you a number with an error bar you have not measured. Before comparing two numbers, know how wide each bar is.

To see trial noise alone, the following simulation fixes 89 tasks with per-task success probabilities drawn from a Beta(0.5, 0.5) distribution (most tasks are either usually solved or usually failed). Harness B is genuinely better by 2.8 points on average. Each experiment runs both harnesses and asks which scored higher.

With one trial per task, the worse harness came out ahead in about a quarter of experiments (7.6% were exact ties), and in 13.6% it led by 3 points or more. Five trials cut that to 0.5%. The benchmark did not change; only repetition did.

Gotcha: "We tried the new prompt, the score went up 3 points, we shipped it" is, with one trial on a benchmark this size, roughly a one-in-seven chance of shipping a regression that looked like a win.

Miller's paper makes a specific recommendation for comparing two systems: when both ran on the same questions, do inference on the question-level paired differences rather than on the two overall scores. The reason is that task difficulty dominates the variance. A hard task is hard for both harnesses, so subtracting per task cancels most of that noise.

In the simulation above with five trials per task, the two methods give very different error bars for the same data:

The paired analysis is more than twice as precise at no extra cost. It still says something sobering: a true 2.8-point gap is only about 1.2 standard errors, not yet conclusive on 89 tasks. Miller's remaining recommendations follow from this: resample to reduce per-task noise, and use a power analysis before the experiment to learn whether the eval can detect the size of improvement you care about.

Running several trials per task gives you two different summary metrics, and they answer opposite questions.

pass@k is the chance that at least one of $k$ attempts succeeds: the right metric when a human or a verifier picks the best of several outputs. pass^k is the chance that all $k$ attempts succeed: the right metric when a customer-facing agent must behave correctly every single time. Anthropic's post gives the example: a 75% per-trial success rate means pass^3 of about 42%.

The pass^k metric was introduced with tau-bench (Yao et al., 2024), where a gpt-4o function-calling agent succeeded on under 50% of tasks and had pass^8 under 25% in the retail domain.

The formulas above assume one success rate for every task. Real benchmarks mix easy and hard tasks, so compute each metric per task from observed trials and then average. The unbiased pass@k estimator comes from Chen et al. (2021), the Codex paper; the pass^k version follows the same counting logic:

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    # P(at least one of k draws succeeds), from n trials with c successes
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n: int, c: int, k: int) -> float:
    # P(all k draws succeed), from n trials with c successes
    return comb(c, k) / comb(n, k)

results = {"refund-flow": 8, "rename-api": 6, "flaky-migration": 2}  # successes out of 8
for k in (1, 3):
    at = sum(pass_at_k(8, c, k) for c in results.values()) / len(results)
    hat = sum(pass_hat_k(8, c, k) for c in results.values()) / len(results)
    print(f"k={k}: pass@k={at:.2f}  pass^k={hat:.2f}")

This prints k=1: pass@k=0.67 pass^k=0.67 and k=3: pass@k=0.88 pass^k=0.45. The same eight trials per task tell two stories: most tasks are solvable, but only one of three is reliable.

Gotcha: Averaging first and exponentiating second is wrong. Half the tasks at 100% and half at 50% average 75%, yet the true pass^3 is 56%, not the 42% that $0.75^3$ suggests.

The first lesson showed that container resources alone moved Terminal-Bench 2.0 scores by 6 points in Anthropic's experiments. For your own harness experiments that makes the environment part of the experimental control. From Anthropic's infrastructure-noise post and general experimental practice:

• Pin the environment: same container image, CPU and memory allocation, timeouts and network policy for both arms. Anthropic recommends specifying both a guaranteed allocation and a hard kill threshold per task.
• Separate infrastructure failures from agent failures: at strict limits they saw 5.8% of tasks fail on pod errors unrelated to the model. Count those separately and rerun them, or they silently penalise whichever arm hit them.
• Pin the model version and sampling settings; a provider-side model update mid-experiment confounds everything.
• Randomise and interleave runs of the two arms, so a slow afternoon for an API does not land on one arm only.

In practice: Write the environment into the result. "B beats A by 4 points" is not reproducible; "B beats A by 4 points, paired SE 1.5, 5 trials, 4 CPU / 8 GB, 30-minute timeout, same model snapshot" is.

Anthropic's post distinguishes two kinds of eval suite, and a harness team needs both:

To start, the post suggests 20 to 50 simple tasks drawn from real failures, not a large synthetic benchmark. Real failures encode exactly the behaviour your users hit. When a capability task becomes reliably solved, it graduates into the regression suite.

Cost belongs in both reports. Divide total spend by tasks solved to get cost per solved task; a harness that gains 2 points while doubling tokens may be a worse product.

The loop that turns a harness idea into a decision.