Agentic RPA and Computer-Use Agents: Benchmarks vs. Real Deployments

Agentic RPA — the promise that AI agents will drive enterprise software the way humans do, click for click, form for form — is the splashiest enterprise AI narrative of 2024-2026. Anthropic shipped Computer Use (October 2024), OpenAI shipped Operator (January 2025), Google shipped Project Mariner (December 2024), and every major RPA incumbent (UiPath, Automation Anywhere, Microsoft Power Automate) repositioned around "agentic" workflows.

The honest question is not whether the demos work — many do. It is whether measured benchmark scores have translated into reproducible production value at scale. Two clean ways to interrogate that: (1) read the benchmark papers, which give us SOTA numbers from independent academics; (2) read what listed companies actually disclose in earnings calls and 10-Ks about deployment outcomes. We'll do both and end with the strongest published skeptic case.

The standard academic benchmark is OSWorld (Xie et al., 2024, arXiv:2404.07972) — a benchmark of 369 real-world computer tasks across Ubuntu, Windows, web apps, and desktop applications, evaluated with execution-based metrics rather than LLM-judged outcomes.

What the original paper found and what the public leaderboard shows as of 2026:

• Human baseline: ~72% task success.
• Best agents at paper time (mid-2024): ~12%.
• Best agents as of late 2025: in the 30–50% range, with newer Claude/GPT/Gemini-class models making the largest jumps.

That trajectory is what "capabilities are improving fast" looks like quantitatively. But the same numbers also tell you that best-in-class agents in 2026 still fail half or more of realistic OS-level tasks that humans handle routinely. Production-grade reliability (≥99%) is well beyond current SOTA on this benchmark.

For web-only tasks, the analogous benchmark is WebArena (Zhou et al., 2024, arXiv:2307.13854) — 812 long-horizon tasks across realistic, self-hosted versions of GitLab, Reddit, an e-commerce site, a CMS, and a mapping tool. The companion benchmark VisualWebArena (Koh et al., arXiv:2401.13649) adds visual reasoning.

Why WebArena hurts agentic claims more than its OSWorld cousin: the websites are real, the tasks are long-horizon ("find the most upvoted post in r/X from last month and create a GitLab issue summarizing it"), and the evaluation is execution-based (did the final state match?). The first paper reported best agent success rates around 14%. As of late 2025, frontier-model agents are reaching 30–40% on the original benchmark, with similar gaps to the ~78% human baseline.

The lesson: benchmarks reward narrow, well-specified tasks. Real enterprise workflows are messier. Benchmark scores set a ceiling, not a floor, on production reliability.

Even if an agent hits 60% on OSWorld, three independent gaps remain before it survives in production:

1. Tail risk. Enterprise workflows are dominated by edge cases. A 60% success rate means 40% failure — most of which is silent (wrong data entered, wrong row deleted) rather than loud (crash, refusal). Silent failures break trust faster than refusals.
2. Cost per task. Frontier reasoning models cost between roughly $0.50 and $5.00 per long-horizon task as of early 2026, depending on model and tool use. RPA bots cost effectively nothing per execution. The break-even versus offshore labor or scripted RPA is task-specific and usually narrower than the pitch implies.
3. Compliance overhead. Any agent acting on production systems needs audit logs, action-confirmation gates, role-based access, rollback, and adversarial-input hardening. That infrastructure often exceeds the agent build itself.

This is why "we have an agent that can do X" and "we have shipped agent-driven X to production" are very different claims — and why most public disclosures of the latter remain narrow.

The single most-cited corporate RPA case is JPMorgan's COIN (Contract Intelligence) program, reported by Bloomberg in February 2017 (Hugh Son). The widely repeated figure is 360,000 hours of lawyer and loan-officer time saved annually, parsing roughly 12,000 commercial-loan agreements.

What is true about this number:

• It was disclosed by JPMorgan to Bloomberg in 2017, citing internal estimates.
• It refers to a pre-LLM system (NLP + classical machine learning), not a modern agent.
• It has never been independently audited.

Why it still matters for the agentic conversation: even at face value, the COIN figure represents a very large bank automating a single document-parsing workflow over years of careful build. It is the upper end of the public evidence. There is no equivalent disclosure of a modern, autonomous-agent deployment at comparable scale that has survived independent reporting in 2024-2026. When you next hear "this AI agent saves a million hours" — ask what fraction of a JPMorgan-style commitment was needed to ship it, and what fraction of the savings is verified.

Salesforce is the largest listed company to have publicly staked a narrative on agents ("Agentforce", launched September 2024). Their fiscal Q3 FY25 earnings call (December 2024) and the Q4 FY25 call (February 2025) — transcripts available on investor.salesforce.com — are the highest-quality public source on agentic-RPA adoption inside a real enterprise software customer base.

What management actually disclosed, paraphrased to avoid quoting drift:

• Order counts in the low thousands of customers contracting Agentforce within the first two quarters.
• Revenue contribution "not material" in the immediate quarter; framed as a multi-year monetization ramp.
• Repeated framing of agents as augmenting Service Cloud / Sales Cloud rather than replacing seats.

Notably absent from the disclosures (as of early 2026): per-agent task-completion rates, customer-side ROI studies, or comparisons to non-AI deployments. The contrast with the COIN-era disclosure is instructive: in 2017 a customer disclosed an outcome figure; in 2024-2026 a vendor is disclosing an adoption figure. Those are not the same evidence.

Where the evidence actually lives, and where the claims usually outrun it:

The most-cited bear case from inside the financial industry is Goldman Sachs Global Macro Research's "Gen AI: Too Much Spend, Too Little Benefit?" (June 2024, available as a Goldman Sachs research PDF).

The note collected views from MIT's Daron Acemoglu (who modelled a ~0.5% productivity uplift over ten years), Goldman's own equity research head Jim Covello (the most aggressively skeptical voice, framing the spend-to-benefit ratio as historically anomalous), and counter-arguments from Goldman's chief economist. The skeptic argument boiled down to three claims:

1. Hyperscaler capex (~$1 trillion projected through ~2026) is large relative to plausible near-term productivity gains.
2. The cost curve of inference has not fallen as fast as proponents required for the business model to work.
3. Concrete productivity case studies are narrower than the headlines.

Whether the Covello case ages well is an open question. As a citable, sourced, dissenting view from inside a major bank, it is the cleanest counter-evidence available — and worth reading before you green-light the next agent project.

The EU AI Act — Regulation (EU) 2024/1689 — entered into force on 1 August 2024. Most provisions phase in through 2025–2027. The Act explicitly defines "AI system" in terms broad enough to capture agentic deployments and classifies systems by risk tier (unacceptable / high / limited / minimal).

What changes for agentic RPA in the EU and for any company touching EU users:

• High-risk classification triggers for AI used in employment screening, credit scoring, education access, critical-infrastructure operation, law enforcement, migration, and administration of justice (Annex III). Many enterprise "agent" use cases fall here.
• High-risk systems require risk-management systems, data governance, technical documentation, logging, human oversight, accuracy/robustness/cybersecurity, and post-market monitoring (Articles 9–15).
• General-purpose AI models (GPAI) face their own obligations (Article 51 onward).
• Penalties scale up to €35 million or 7% of global annual turnover, whichever is higher (Article 99).

The practical effect: the cost of deploying agentic systems just rose, and the burden of proof for high-risk use is on the deployer. The economics shift from "can the agent do the task" to "can the agent do the task and survive the audit".

Stitching the evidence together:

• Capability trajectory is real. OSWorld and WebArena numbers have improved substantially since 2024. The frontier is not stalled.
• Reliability is the open problem. SOTA agents fail too often, and too silently, for high-stakes single-shot production use. The gap between 50% benchmark success and 99.9% production reliability is large and not closing linearly.
• The cleanest disclosed outcome is still JPMorgan COIN. That is from 2017, pre-LLM, and concerns a single document-parsing workflow. The 2024-2026 vendor disclosures are heavier on adoption than on outcomes.
• The bear case is on the record. Goldman's Covello note and Acemoglu's modelling are publicly available; you do not need to invent the skeptic argument.
• Regulation is now a first-class cost. EU AI Act compliance, plus parallel U.S. state-level work, has changed the deployment ROI math.

The right operating posture: pilot narrowly, instrument outcomes the way you would a clinical trial, and don't deploy at scale into high-stakes workflows on the basis of a vendor's slide. The benchmarks are public — use them.