Coding Agents: Productivity Studies, Benchmarks, and the METR Slowdown

There are two empirical stories about AI coding tools and you cannot understand the market by knowing only one.

Story A — Copilot lifts output. Cui, Demirer, Jaffe, Musolff, Peng, Salz (2024), The Effects of Generative AI on High-Skilled Work, three field experiments with 4,867 developers at Microsoft, Accenture, and a Fortune 100 electronics firm, SSRN 4945566, published in Management Science (2025/26). Headline: ~26% increase in completed tasks (pull requests) for developers granted GitHub Copilot access (SE ~10.3 percentage points).

Story B — frontier AI tools slow experienced developers down. METR (2025), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, arXiv:2507.09089, METR blog. RCT with 16 experienced OSS maintainers on 246 tasks; allowing AI tools increased completion time by 19% — while the same developers believed AI had made them ~20% faster.

Both are real. They study different developers, different tasks, different models. The reconciliation is the lesson.

The Cui et al. paper combines three randomised field experiments:

• Microsoft internal developers,
• Accenture developers on client engagements,
• An anonymous Fortune 100 electronics manufacturer.

In each, a randomly selected subset of developers received GitHub Copilot access. Outcome: pull requests completed, the cleanest available proxy for shipped work. Across all three, combined sample 4,867 developers, the treatment effect on completed PRs is +26.08% (SE 10.3 pp). The effect is positive in each individual experiment.

The authors also report — consistent with the Brynjolfsson et al. customer-service finding — that less experienced developers adopted Copilot more and gained more from it. The mechanism story is plausible: Copilot is strongest at boilerplate, syntax, and idiomatic patterns, which are exactly what less experienced developers spend a disproportionate share of their time on.

This is the single best causal estimate of AI coding tools on professional developer output. It is not the same thing as proof that every coding tool, on every task, with every developer, yields +26%.

PRs completed is observable, but it is a flow metric, not a quality metric. It does not directly capture:

• Whether the merged code introduces bugs that show up later.
• Whether reviewers spent more time reviewing AI-assisted PRs.
• Whether developers chose smaller, easier PRs because the tool made the easy parts faster.
• Long-term codebase entropy from AI-generated patterns.

The Cui et al. paper does check for some of these (review patterns, defect rates where measurable) and finds no large degradation, but the measurement window is months, not years. The honest reading of +26% is: more visible shipped work per developer-week, conditional on the firm's existing quality gates holding. Whether quality gates hold is a separate, harder question — and is exactly where the METR study lands its punch.

METR (Model Evaluation & Threat Research) is a non-profit research organisation in Berkeley, California, that builds evaluation methodology for frontier AI systems, particularly long-horizon autonomous task performance. Their public work includes the time-horizon metric: the length of human task that an AI system can reliably complete end-to-end. See metr.org.

The study at issue: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, posted July 2025 (arXiv:2507.09089).

Design: 16 experienced open-source maintainers, average 5 years on the project they were working on, 246 real tasks drawn from their own backlogs, randomised within-developer to AI-allowed vs AI-disallowed. The AI tools were the early-2025 frontier: Cursor Pro plus Claude 3.5/3.7 Sonnet.

Result: when AI tools were allowed, tasks took 19% longer to complete. Developers had forecast they would be ~24% faster with AI; after the study, they still believed AI had made them ~20% faster. The actual measurement said the opposite.

These do not contradict. They map onto different populations and tasks.

The Brynjolfsson pattern repeats: AI helps where the model's average suggestion is above the user's current marginal output. On a mature OSS repo where the maintainer knows every file, the model's suggestions are often below par, and the time spent reviewing/correcting them exceeds time saved. On enterprise feature work where a developer is unfamiliar with the immediate codebase, suggestions help.

The correct headline is not "AI helps coders" or "AI doesn't help coders." It is "AI's coding productivity effect is task- and developer-dependent and can flip sign."

Stylised mapping based on Cui et al. and METR.

The most-cited number from the METR study is not actually the 19% slowdown. It is the perception gap.

• Developers' prior belief: AI will speed them up by ~24%.
• Developers' posterior belief, after doing the work: AI sped them up by ~20%.
• Measured reality: AI slowed them down by 19%.

METR also collected forecasts from outside experts. Economists predicted AI would shorten tasks by ~39%. ML researchers predicted ~38%. Everyone was wrong in the same direction.

This is the most important finding for executives funding AI tooling: self-reported productivity gains from developers are not a reliable signal. The cognitive ergonomics of AI tools (less typing, more reading; less recall, more recognition) feel productive even when wall-clock time goes the wrong way. If you are buying AI tools based on developer surveys, you are buying vibes. Demand A/B measurement on real task throughput.

The standard benchmark for autonomous coding agents is SWE-bench: Jimenez, Yang, Wettig, Yao, Pei, Press, Narasimhan (Princeton), SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv:2310.06770, ICLR 2024. The benchmark presents a model with a real GitHub issue and the repository state at that time, and scores whether the patch the model produces passes the project's hidden test suite. Original SWE-bench has 2,294 tasks across 12 Python repositories.

In August 2024, OpenAI released SWE-bench Verified — a 500-task subset filtered by 93 expert software engineers to remove ambiguous problem descriptions, broken tests, and tasks underspecified for an agent. This is now the standard frontier-model scoring target.

Frontier model scores on SWE-bench Verified climbed rapidly through 2024–25, from sub-20% in early 2024 to 70%+ for top systems by 2025. By late 2025, OpenAI announced it was retiring SWE-bench Verified as a frontier metric because the benchmark was saturating and audits had found material issues in many tasks.

By 2025, a frontier model could plausibly resolve ~70% of SWE-bench Verified issues end-to-end. Yet the METR study, run in the same year on the same model class, found negative productivity on real OSS work. What gives?

Benchmark vs. real-world gaps to know about:

• Distribution. SWE-bench tasks are skewed to a fixed set of repos and historic issues already known to be resolvable. Production tasks are not.
• Specification. Benchmark tasks have a clean problem description and a hidden test oracle. Real backlog tasks are under-specified and the "test" is sometimes "does the maintainer subjectively accept the PR."
• Cost. Benchmark scores are reported without showing tokens-per-task or wall-clock-per-task. Production tools have a budget.
• Long-horizon coherence. SWE-bench is single-issue. Real engineering involves co-evolving multiple files, RFCs, and downstream consumers.
• Contamination. Many SWE-bench issues are from before model training cut-offs; models may have seen the resolutions in training data.

Benchmark scores are a ceiling under ideal conditions, not a floor under real ones. Treat them accordingly.

Widely cited but not independently audited: GitHub Copilot's installed-base figures, Cursor's user counts, Anthropic's coding-agent revenue mix.

What we can verify from public filings:

• Microsoft's 10-K and 10-Q filings reference GitHub revenue at the segment level, but typically do not break out Copilot subscription revenue with the granularity needed to estimate paid Copilot seats. See Microsoft's SEC filings on sec.gov.
• Anthropic and OpenAI are private; their disclosures are voluntary and partial.

A defensible statement: AI coding tools are deployed at scale in industry, but precise adoption metrics rely on vendor disclosures rather than independent measurement. When a deck says "X million developers use Y," check whether that number is in a 10-K or a marketing blog post. If the latter, treat as a claim, not a measurement.

If you are deploying coding agents to your engineering org, the evidence supports something like:

1. Expect heterogeneous gains. Junior engineers and engineers new to a codebase should see the largest productivity uplift. Senior engineers on mature systems should not be assumed to benefit, and may need to be measured.
2. Measure throughput, not survey. Self-reports are unreliable in both directions. Use PR throughput, cycle time, defect-escape rate. Stagger rollout if you can to recover something close to a causal estimate.
3. Buy for the task, not the brand. Autocomplete (Copilot-class) and agentic (Devin/Cursor agent / Claude Code) are different products with different evidence bases. The +26% Cui et al. number is autocomplete-era.
4. Watch the quality lag. PR throughput is observable in weeks; codebase entropy and defect-escape are observable in quarters. Do not declare victory on month-one numbers.
5. Do not extrapolate benchmark scores to ROI. A 70% SWE-bench Verified score does not mean the agent will resolve 70% of your team's tickets unsupervised.

1. The best causal estimate of AI coding-tool productivity is Cui et al. (2024): +26% completed PRs on average across 4,867 enterprise developers, with larger gains for less experienced developers.
2. The best counter-evidence is METR (2025): on 246 real tasks in mature OSS repos with experienced maintainers, AI tools increased task time by 19%, against developers' own belief that AI had sped them up.
3. Benchmark scores keep climbing. SWE-bench Verified saturated through 2024–25 and OpenAI retired it as a frontier eval. This says little about production value.
4. The perception gap is the under-appreciated finding. Developers cannot accurately self-report their AI-induced productivity in either direction. Buy on measurement, not surveys.
5. AI coding tool ROI is task- and developer-dependent. There is no single number; there is a distribution, and it can include negative outcomes.