Model Plus Harness: Why the Same Model Scores Differently

When people compare AI agents they usually compare models. But a model on its own only maps text to text. Everything that turns it into something that edits files, runs tests and stops when it is done lives outside the weights.

Definition: A harness is everything in an AI agent except the model itself. Birgitta Böckeler of Thoughtworks uses exactly this definition in her essay on harness engineering, published on martinfowler.com.

That "everything" is a long list:

• the system prompt and any instruction files it loads
• the tools, their names, schemas and the text they return
• the context policy: what is kept, truncated or summarised each turn
• permissions and sandboxing: what the agent may touch
• feedback: tests, linters, type checkers, screenshots
• the infrastructure: CPU, memory, timeouts, network

This course is about the craft of shaping that list. The mechanics of the control loop itself (tool registry, retries, budgets) are covered in Designing a Production Agent Harness; here the question is what surrounds the loop, and how much it matters.

Böckeler separates the harness into two layers with different owners. The inner harness ships with the agent product. The outer harness is built by the team that uses the agent, for its own codebase.

The split matters because most engineers never write an inner harness. They pick one, then spend their effort on the outer layer. That outer layer is where harness engineering as a practice lives: designing the repository, checks and guardrails so that whatever agent you run succeeds more often and fails more safely.

Key idea: You cannot change the model's weights, and you rarely change the agent's core loop. You almost always control the environment it works in.

The cleanest early evidence that the harness moves results comes from SWE-agent (Yang et al., NeurIPS 2024). The authors held the model fixed, GPT-4 Turbo, and changed only the interface it used to work on SWE-bench Lite, 300 real GitHub issues.

The shell-only agent had a real Linux shell, the interface a human engineer would use. The SWE-agent version replaced it with commands designed for a language model: a windowed file viewer, a search that summarises results, and an editor that checks syntax. Same weights, same tasks, and the resolve rate went from 11.0% to 18.0%, a 64% relative gain the paper reports directly.

The SWE-agent paper then removed one design choice at a time. Every number below is the percentage of SWE-bench Lite resolved by GPT-4 Turbo; the full interface scored 18.0.

Three lessons generalise beyond this benchmark. Guardrails pay: rejecting a syntactically broken edit was worth 3 points. Concise feedback beats raw output: summarised search beat paging through matches. Less history can be better: keeping only the last five observations outscored the full transcript.

Gotcha: An interface that is comfortable for a human (scroll one result at a time, see everything) is not automatically good for a model.

The SWE-agent file viewer showed 100 lines at a time. The authors also tried a 30-line window and showing the entire file at once.

The syntax-checking editor is easy to reproduce. The tool below refuses any edit that leaves a Python file unparseable, and tells the model why instead of silently writing broken code.

import ast
from pathlib import Path

def edit_file(path: str, start: int, end: int, new_text: str) -> str:
    # Replace lines start..end (1-based, inclusive). Reject edits that break syntax.
    p = Path(path)
    lines = p.read_text().splitlines(keepends=True)
    candidate = lines[:start - 1] + [new_text.rstrip("\n") + "\n"] + lines[end:]
    source = "".join(candidate)
    try:
        ast.parse(source)
    except SyntaxError as e:
        return (f"Edit rejected: SyntaxError at line {e.lineno}: {e.msg}. "
                "File unchanged. Re-issue the edit with corrected brackets.")
    p.write_text(source)
    window = source.splitlines()[max(0, start - 3): start + 2]
    return "Edit applied. Lines now read:\n" + "\n".join(window)

Called with an unclosed parenthesis, it returns Edit rejected: SyntaxError at line 2: '(' was never closed. File unchanged. The file on disk never breaks, and the error message is written for the model: it says what failed, where, and what to do next. On success it echoes the edited lines back, so the model sees the result without a separate read.

Terminal-Bench 2.0 (Merrill et al., 2026) runs 89 hard command-line tasks and, unusually, crosses models with several agents, at least five runs per combination. Its full results table lets you hold the model fixed and vary only the harness.

For GPT-5 the spread between its best and worst agent is 15.7 points, far outside the reported confidence intervals of roughly 3 points. For Claude Opus 4.5 it is 5.9 points. Note also that no harness wins everywhere: Terminus 2, a deliberately minimal agent with a single terminal tool, was best for Opus 4.5 and third for GPT-5.

The same paper is careful not to overclaim. Its authors write that model selection is usually more important than agent scaffold, and the table supports them:

Harness choice also moves cost, which the headline score hides. The same table reports input tokens for the full run: Opus 4.5 consumed 256.9M input tokens in Claude Code and 3.9M in Terminus 2, while scoring 52.1% and 57.8% respectively. Token counts are not dollars, since prompt caching changes the bill, but a 60-fold difference in context sent is a property of the harness, not the model.

In practice: Choose the model for its ceiling, then treat the harness as the lever you pull every week: it is where cost, reliability and a few points of accuracy are won.

Even the hardware under an agent changes its score. Anthropic ran Terminal-Bench 2.0 under six container resource settings, from strict limits to uncapped, and reported in Quantifying infrastructure noise in agentic coding evals:

• a 6 percentage point gap between the least and most resourced setups (p < 0.01)
• infrastructure failures, such as pods killed for exceeding memory, on 5.8% of tasks at strict limits versus 0.5% uncapped
• up to about 3x headroom, extra resources mostly prevented spurious crashes; beyond that, they let agents try approaches, like heavier builds, that could not run before

Their conclusion: leaderboard differences below 3 percentage points deserve skepticism until the evaluation configuration is documented and matched.

Gotcha: "Agent A beats agent B by 2 points" can be a statement about memory limits and timeouts. Resource limits, timeouts and network access belong in the harness description, next to the prompt and tools.

The harness wraps the model in layers. Most teams work on the outer ones.

The evidence in this lesson points to a working model of the field:

1. The model sets the ceiling. Terminal-Bench's 51-point spread across models inside one harness dwarfs the harness effects.
2. The harness decides how close you get to it. Interface design moved SWE-agent from 11.0% to 18.0%; harness choice moved GPT-5 by 15.7 points on Terminal-Bench.
3. No harness is universally best. Rankings flip between models, so a harness must be evaluated with the model you will run.
4. Small gaps are mostly noise. Anthropic's 3-point skepticism threshold applies to your own experiments as much as to leaderboards.

The next lesson turns to the outer harness you actually control: how to make a repository legible to an agent, and how to turn conventions into checks it cannot ignore.

Try it: Write down every element of the harness your team's coding agent runs in today, using the six-item list from the first step. Most teams find at least one (often infrastructure or feedback) that nobody owns.