Measuring AI ROI: From Pilot to P&L

The majority of published AI ROI claims share a common flaw: they measure the treatment group without a control. A team that adopts an AI writing tool and produces more content is not evidence that the tool caused the increase. It may reflect the Hawthorne effect (people work harder when observed), selection bias (motivated teams adopted first), or a concurrent process improvement. Without a counterfactual — what would have happened without the AI — you have an anecdote, not a measurement.

Vendor case studies are structurally incapable of solving this problem. They are constructed post-hoc from successes, with no obligation to report negative deployments. The base rate of failure remains invisible. A sceptical operator treats every vendor claim as a hypothesis with an unknown prior — valuable for direction, useless for sizing budgets. The discipline is asking: what would it take to run a clean measurement here?

A credible ROI measurement requires three things established before deployment:

1. Baseline metric. A clearly defined, objectively measurable outcome — tickets resolved per hour, cycle time from draft to approval, defect-escape rate — measured for at least 4–8 weeks pre-deployment to establish variance.
2. Counterfactual design. Either a randomised holdout (some agents/teams don't get the tool in wave 1) or a staggered rollout where the timing of access is as-good-as-random. Without one of these, you have no causal estimate.
3. Pre-committed analysis plan. Decide your success criterion — effect size, duration, statistical threshold — before seeing the data. Post-hoc threshold-moving is how negative deployments become "successful pilots" in internal decks.

This discipline also forces scope clarity: which metric are you actually trying to move? Vague goals like "increase productivity" make post-hoc storytelling trivially easy.

The evidence base that actually holds up uses randomised controlled trials or natural-experiment designs. Three examples:

• Brynjolfsson, Li & Raymond (2023, NBER): Randomised 5,179 customer-support agents at a Fortune 500 software firm — some got AI-assisted chat, others didn't. Result: roughly 14% average productivity gain in resolved tickets per hour, with gains around 34% for the lowest-quartile performers. The causal interpretation is defensible because assignment was random.
• Dell'Acqua et al. / BCG-Harvard (2023): Randomised 758 BCG consultants to AI-assisted vs. unaided on structured tasks. AI-assisted consultants performed roughly 12% faster and 25% higher quality on in-frontier tasks, and measurably worse on out-of-frontier tasks — the jagged frontier. Again, randomisation makes the causal reading defensible.
• Cui et al. (2024, Management Science forthcoming): Three field experiments, 4,867 enterprise developers, randomised Copilot access. Combined effect: roughly +26% completed pull requests.

Notice: all three studies are published, peer-reviewed or under review, and all used randomisation. The positive vendor case study on your desk almost certainly did none of these things.

Most internal AI ROI spreadsheets undercount the denominator. A realistic TCO for an enterprise AI deployment includes:

A deployment that costs $50K in API fees but$400K in integration and oversight is not a $50K project. Pilot budgets almost always capture only the top line.

A common measurement trap is tracking the metric that moves first rather than the metric that matters. Leading indicators respond quickly but may not predict lasting value:

• Adoption rate (% of eligible users logging in) — moves in weeks, says nothing about impact.
• Output volume (emails drafted, tickets touched) — moves quickly, conflates speed with quality.
• User satisfaction / NPS — fast to collect; does not correlate reliably with productivity in Brynjolfsson-type studies.

Lagging indicators take longer but tell the real story:

• Error / defect rate downstream from AI-assisted work.
• Cycle time end-to-end (e.g., days from issue reported to resolved, not just first-response time).
• Cost per outcome unit (cost per resolved ticket, cost per approved contract).
• Revenue per headcount at business-unit level — the hardest but most meaningful P&L signal.

The practical guidance: design your measurement stack to include at least one lagging P&L-adjacent metric, even if it takes 6 months to accumulate signal. Early-mover pressure to declare victory at 30 days is how good deployments get mis-measured and bad deployments get approved for scale.

Even well-intentioned measurement teams fall into these four traps:

1. Contemporaneous confounds. A call-centre AI goes live in Q1. Ticket volume falls 15% — was it AI, or the new product update that reduced bugs? Isolate with a randomised holdout, not a before-after chart.
2. Survivor selection. Teams that adopted AI earliest were also the most technically capable. Measuring their outcomes and generalising to average teams overstates expected impact.
3. Substitution vs. creation. AI completes 20% more tasks — but are those tasks genuinely new value, or are they tasks that previously weren't done because they weren't worth doing manually? Output volume increases that don't connect to revenue or cost are substitution metrics, not value metrics.
4. Measurement horizon mismatch. Code quality defect-escape rates and codebase entropy take quarters to manifest; PR throughput takes days. Declaring ROI positive at week four based on throughput ignores the quality lag that METR (2025) found with experienced developers.

The honest summary: attribution is hard, and most firms don't invest enough in measurement infrastructure to do it well.

The following framework is designed to be run before a deployment begins, not after:

Step 1 — Define the target metric. One primary metric, P&L-adjacent, measurable without survey (e.g., "resolved tickets per agent-hour").

Step 2 — Establish baseline. Collect 6–8 weeks of pre-deployment data with variance band. Document any concurrent process changes.

Step 3 — Design counterfactual. Randomised holdout (preferred) or staggered rollout with documented assignment logic.

Step 4 — Pre-commit success threshold. Effect size, confidence threshold, and minimum duration. Write it down before launch.

Step 5 — Track full TCO. API costs + integration hours + oversight headcount + rework estimate.

Step 6 — Run for long enough. Minimum 8–12 weeks post-deployment before reading results. Resist the week-2 dashboard review.

Step 7 — Report honestly. Include the null result scenario. If results are ambiguous, say so. The firm's credibility in future AI investment decisions depends on the integrity of this one.

Where most enterprises break the chain and what rigorous measurement looks like.

Scenario: 200 customer-support agents, baseline throughput 8 resolved tickets per agent-hour, average cost per agent-hour $35. You deploy an AI-assist tool. API cost:$0.12 per ticket. Integration: one-time $180K. Oversight for QA sampling: 0.25 FTE at$80K/year.

Optimistic case (following Brynjolfsson et al., +14% avg uplift):

• New throughput: 9.12 tickets/agent-hour.
• Annual tickets resolved: 200 agents × 2,000 hours × 9.12 = 3.648M (vs. 3.2M baseline).
• Additional capacity: 448K tickets — equivalent to avoiding ~56 additional agent-hires at fully-loaded cost.
• Annual TCO: ($0.12 × 3.648M) +$20K monitoring + $20K oversight = roughly$478K.
• Annual value of avoided headcount (56 × $70K fully-loaded): ~$3.9M.
• Year-1 net after integration: roughly $3.9M −$478K − $180K = **~$3.24M positive**.

Honest caveat: The Brynjolfsson study was in a single firm with a specific task type. Your deployment may land at +5% rather than +14%. At +5%, the avoided headcount is 20 agents ($1.4M), and year-1 net drops to roughly$740K — still positive, but far less dramatic. The measurement framework exists precisely to find out which scenario you're in.

The measurement discipline exists to produce honest no's as well as yes's. Signals that suggest a deployment should be paused or redesigned:

• Adoption below 40% after 8 weeks despite training — usually a workflow fit problem, not a model problem.
• Output volume up but quality metrics flat or deteriorating — the tool is accelerating work, not improving it.
• Oversight cost exceeds inference cost — human-in-the-loop design needs redesign at this throughput.
• No measurable signal on the pre-committed primary metric after 12 weeks — the effect may be real but below the threshold that justifies the TCO.

A culture that can produce a measured "no" from an AI pilot is more valuable than one that can only produce a story. It preserves capital for the deployments that will actually move the needle, and it builds the measurement muscle that compounds over multiple investment cycles. The MIT GenAI Divide result (~95% of pilots with no P&L impact) implies that the organisations winning on AI are the ones running honest measurements, not the ones running the most pilots.