Customer Service AI Agents: What the Data Actually Shows

Customer service is the deployment archetype every vendor cites: bounded conversations, structured tools, measurable KPIs. It is also the area with the most hard data on what AI agents actually do to a workforce. This lesson anchors on a single landmark study and a single landmark deployment, and asks what each one really shows.

The study is Brynjolfsson, Li & Raymond (2023), Generative AI at Work, NBER Working Paper 31161 (also arXiv:2304.11771). The deployment is Klarna's GPT-4-powered customer-service assistant, launched February 2024 and quietly de-emphasised by May 2025.

These are not the only data points, but they are the cleanest. The NBER paper is a staggered-rollout natural experiment on 5,179 agents at one Fortune 500 firm; Klarna is a public company whose CEO has made public, on-the-record claims and on-the-record retractions you can cite. Most other deployment stories are vendor press releases.

The NBER 31161 study uses staggered access to a generative AI conversational assistant at a Fortune 500 software firm's customer-support business. 5,179 agents are in the panel. Because the AI rolled out to teams at different times, the authors get a difference-in-differences identification strategy that is close to an RCT in spirit, though not literally randomised at the agent level.

The outcome metric is issues resolved per hour, a clean operational KPI the firm already tracked. The authors also examine customer sentiment (from transcripts), employee retention, and the distribution of effects across the experience curve.

This matters because most published "AI productivity" numbers come from short lab studies or vendor self-report. NBER 31161 has scale (~5k agents), duration (about a year of data), an objective outcome, and a causal design. It is the single most-cited piece of empirical evidence on generative AI's effect on knowledge work — and it is one firm, one product, one industry.

Average effect: a 14% increase in issues resolved per hour for agents with access to the assistant, versus controls.

The more important finding is the heterogeneity. Splitting the sample by tenure and pre-treatment skill:

• Novice / low-skill agents: ~34% productivity gain.
• Experienced / high-skill agents: roughly zero, statistically indistinguishable from no effect.

The authors interpret this as the AI codifying the implicit knowledge of the top-decile agents — phrasing, troubleshooting trees, escalation cues — and surfacing it as suggestions to everyone else. The novices get to climb the experience curve in months instead of years. The experts already knew what the model is telling them.

Secondary findings: customer sentiment in chats improves modestly; agent retention rises (treated agents are less likely to quit); there is suggestive evidence of learning — novices show some residual skill gain even when the AI is removed.

Decomposition of the average effect by worker experience.

On 27 February 2024, Klarna and OpenAI jointly announced Klarna's AI assistant (Klarna press release, OpenAI customer page). The claimed numbers, all from Klarna:

• 2.3 million conversations in the first month.
• Two-thirds of customer service chats handled by the assistant.
• Doing "the equivalent work of 700 full-time agents".
• Customer-satisfaction scores on par with humans.
• Resolution time down from ~11 minutes to under 2 minutes.
• Estimated $40M profit improvement in 2024.

None of these were independently audited. The "700 FTEs" figure is not a layoff count — it is Klarna's internal estimate of equivalent workload. Klarna had already been shrinking headcount (~40% reduction since 2022) via a hiring freeze and attrition, which the company explicitly linked to AI productivity in subsequent communications. The framing landed as: AI is replacing the support workforce, and it works.

By May 2025, the story had changed. CEO Sebastian Siemiatkowski told Bloomberg in an interview (covered widely; see Fortune coverage) that Klarna was hiring human customer-service agents again. His framing, in his own words: cost-cutting via AI had produced "lower quality" support, and the company would now ensure customers "always have a human to talk to" when they want one.

Klarna piloted an Uber-style remote agent pool — students, rural workers, Klarna power-users — to rebuild human capacity alongside the AI. The AI was not switched off; the architecture shifted to AI-first triage with human fallback on anything nuanced.

What this is not: a refutation of the NBER finding. The NBER agents had AI as a co-pilot, not a replacement. What this is: a documented case of a public company over-rotating on the substitution interpretation, finding the quality floor, and partly reversing course. The financial impact is real either way — Klarna IPO'd in 2025 and the AI-led cost base is part of the equity story.

The NBER paper's mechanism story has two parts.

1. The model is trained on the best. The assistant's suggestion distribution is anchored on the firm's top-performing agents' resolutions. For a novice, every suggestion is potentially above their current skill ceiling. For an expert, the suggestions regress them toward the mean of the top decile — which is where they already are.

2. AI compresses the experience curve. New agents normally need 6–12 months to internalise tone, product knowledge, and edge-case routing. The assistant front-loads that learning. Brynjolfsson et al. show novices' un-assisted performance also improves over time on treatment, consistent with genuine skill acquisition rather than pure prosthetic effect.

The uncomfortable corollary for managers: the productive value of an AI co-pilot in customer service is largely a one-time levelling effect on your bottom-half workforce. After everyone has been levelled up, marginal returns drop. This is consistent with what later studies have seen — the next year does not give you another 14%.

Generalises reasonably well:

• Text-heavy customer interactions with structured knowledge bases.
• High-volume, repeatable tasks where best-practice can be codified.
• Workforces with wide skill dispersion (lots of novices, a few stars).
• English-language support (the study's language).

Does not generalise without caveats:

• One firm, one industry (enterprise software support). Different product complexity, different result.
• One model generation (a circa-2022 GPT-class assistant). Frontier models in 2025 are different beasts; gains could be larger or smaller depending on grounding and tool-use quality.
• Cooperative deployment: the AI was given to agents as a suggestion tool, not used to fire them. Sentiment and retention effects might invert in a layoff-fear context.
• Voice support, regulated industries (healthcare, finance KYC), and non-English markets are not in-sample.

If you cite NBER 31161 in a board deck, cite it as evidence of a +14% co-pilot effect at one firm, concentrated in novices, not as evidence that 14% is the universal number.

Three caveats serious readers will raise:

1. It's not a true RCT. The rollout was staggered by team, not randomised at the agent level. The authors handle this with standard staggered-DiD machinery, but unobserved team-level shocks could correlate with rollout timing.
2. Single firm, possible selection. The firm chose to deploy. Firms that don't deploy may differ systematically; firms that do may have higher-than-average AI-readiness.
3. Outcome metric is internal. "Issues resolved per hour" is a firm-defined KPI. If the AI changes what counts as a resolved issue (e.g., closing tickets faster but generating repeat contacts), the headline number overstates real productivity. The authors do check repeat-contact rates and find no degradation, which mitigates but does not eliminate this concern.

None of these invalidate the paper. They mean you should weight it as strong suggestive evidence from one firm, not as a population-level estimate.

When a vendor pitches an AI support agent, here is the evidence-based screen:

• Will it level your novices? If your support workforce is uniform-tenure veterans, expect smaller gains than 14%. If you have high turnover and a long ramp, expect larger.
• Can you measure repeat-contact rate? Demand it before and after. This is where Klarna's quality problem showed up.
• What is the human fallback latency? Klarna's reversal is essentially about how fast a customer can get to a human when the AI is stuck. Bake this into the SLA.
• Whose suggestions is it trained on? The mechanism only works if the model's suggestions reflect your best agents, not generic web text. RAG over your own resolved tickets matters.
• What does success look like in 12 months, not 1? Brynjolfsson's gains are partly persistent learning. Klarna's losses showed up only after a year. Both timescales matter.

Avoid the rhetorical move of citing "14%" without citing whose 14% and on what task.

Two more bodies of work worth knowing, briefly:

• Dell'Acqua et al. (2023), "Navigating the Jagged Technological Frontier", Harvard Business School Working Paper 24-013, on BCG consultants. Different domain, but reproduces the uneven gains finding: below-median consultants gained substantially, above-median consultants sometimes performed worse when relying on the AI for tasks outside its frontier. Useful as a triangulation of the heterogeneity story.
• Noy & Zhang (2023), "Experimental evidence on the productivity effects of generative artificial intelligence", Science 381:187–192. RCT on professional writing tasks: ~37% time reduction, with quality gains for lower-skilled writers. Same pattern: AI compresses skill dispersion.

The consistent finding across these is not "AI makes everyone X% more productive." It is "AI lifts the bottom and is roughly neutral for the top, on tasks the model can do." That is a different management problem than the 14% headline implies.

1. The strongest causal evidence on customer-service AI agents is NBER 31161: +14% on average, ~34% for novices, ~0% for experts, at one Fortune 500 firm.
2. Klarna's 2024 announcement is not independent evidence; it is a vendor-customer co-marketing piece with internal numbers. Cite it as a claim, not a measurement.
3. Klarna's 2025 walk-back is also evidence — of what happens when a firm reads the 2024 claim as "replace humans" rather than "augment them".
4. The mechanism that survives across studies is skill compression: AI raises the floor more than the ceiling. Plan workforce, training, and pricing around that.
5. Do not generalise one firm's number to your firm. Run your own measurement, ideally with staggered rollout so you can recover something causal.