The Argument: Why Learning From Us Runs Out

The paper this path follows is short, has no experiments, and is written by two people whose track record makes it hard to ignore.

Richard Sutton co-wrote the standard textbook on reinforcement learning and spent four decades arguing that general methods which scale with computation eventually beat methods that build in human knowledge. David Silver led the work behind AlphaGo and AlphaZero, the systems that beat the best human Go players and then, in AlphaZero's case, learned to do so from self-play without studying human games at all.

So when they publish an argument that the field has taken a wrong turn by relying on human data, the argument carries the weight of two people who have already demonstrated the alternative working in a narrow domain.

The paper is called Welcome to the Era of Experience. It is a position paper: a claim about where the field should go, offered as an argument rather than a result.

This path treats it as an argument to be examined. That means understanding it properly first, and then asking what would have to be true for it to be right.

The paper organises recent history into three periods, and the framing does most of the persuasive work.

The era of simulation covers systems that learned by interacting with a simulated environment. Game-playing agents trained by self-play, robotic controllers trained in physics simulators. These systems reached superhuman performance in their domains and generalised to almost nothing outside them.

The era of human data is the present. Models are trained to predict text and other artefacts that humans produced, then refined with human preferences. These systems generalise remarkably across tasks, which is what the previous era lacked, and they do it by absorbing an enormous amount of what people have already worked out.

The era of experience is what the authors argue comes next. Agents that learn primarily from their own interaction with the world, generating their own data rather than consuming ours.

Read as a narrative it is neat: the first era had learning without breadth, the second has breadth without learning from the world, and the third is supposed to combine them. Whether history actually has this shape is a fair question, and one worth holding onto.

The paper makes two distinct arguments against the current approach, and separating them matters because they are contested to different degrees.

The first is about supply. High-quality human-generated data is finite, and the authors argue it has been largely consumed. Text on the internet is not being produced at anything like the rate frontier training runs consume it, so scaling further along this axis runs into a wall that no amount of compute removes.

This claim is genuinely disputed. Multimodal data, private corpora, and synthetic data are all offered as reasons the wall is further away than it looks. The catalogue's path on synthetic training data covers why the synthetic answer is more limited than it sounds: a generator cannot supply information it does not have, which is close to the same argument in a different setting.

The second argument is about quality rather than quantity, and it is the stronger one.

Even with unlimited human data, the authors argue, you would still hit a ceiling, and that ceiling is the subject of the next step.

The deeper claim is that training a system to reproduce what humans produced sets human performance as the target, and a target is not usually exceeded by accident.

A model trained to predict the next token of human text is rewarded for producing what a human would have produced. A model refined on human preferences is rewarded for producing what a human rater prefers. In both cases the signal being optimised is agreement with people.

That is extraordinarily useful and it has a structural limit. To discover something no human has worked out, the system would have to produce an output no human would produce, and be rewarded for it. Neither training signal does that. A genuinely novel and correct answer that a rater does not recognise scores badly.

The authors' evidence for this is a demonstration rather than an argument. AlphaGo studied human games and played at the level of the best humans. AlphaZero studied no human games, learned from self-play against a hard outcome signal, and played better, producing opening moves that overturned centuries of human theory.

The difference was not scale. It was where the learning signal came from.

The whole argument reduces to one question about any learning system: what tells it that an output was good?

In supervised learning from human text, the answer is whether the output matches what a human wrote. The ceiling is what humans write.

In learning from human preferences, the answer is whether a rater prefers it. The ceiling is what a rater can recognise as good, which is lower than what humans can produce, because judging unfamiliar excellence is harder than producing familiar competence.

In learning from the world, the answer is whether the outcome actually happened. Did the code compile, did the experiment replicate, did the patient recover, did the system stay up. There is no human in that loop, so there is no human ceiling in it either.

The diagram makes the parallel visible: the first two paths route the signal through a person, and the third does not.

The authors' bet is that removing the person from that loop is what unlocks the next capability jump, in the same way it did for board games. The rest of this path is about what that costs.

The argument is less speculative than a position paper usually is, because a version of it is already deployed and working.

Reinforcement learning with verifiable rewards trains models on problems where correctness can be checked by machine. Mathematics with a checkable answer, code with a test suite, formal proofs. The model generates attempts, a checker scores them, and training reinforces what passed.

No human judged those outputs. The signal came from the world, in the narrow sense of a program that runs. And it produced the reasoning models that made 2025's most visible capability gains.

That is a partial confirmation of the thesis, and it is worth being precise about how partial. The domains involved are exactly the ones where correctness is cheap to check automatically. Silver and Sutton are arguing for something considerably broader: agents acting in open environments, over long horizons, on outcomes that take days or months to resolve and have no test suite.

So the current evidence supports the mechanism and not yet the scope. The catalogue's lesson on verifiable rewards covers how the narrow version works, and this path is about the general claim.

A consequence the authors draw, which is easy to miss, is that a shift to experience brings back a body of technique the field mostly stopped using.

When the field moved to learning from human feedback, it quietly sidestepped most of classical reinforcement learning. Human preference data is plentiful and the reward is immediate, so there was little need for the machinery built to handle long delays and sparse signals: value functions that estimate long-run return, exploration strategies that deliberately try uncertain actions, world models that predict what an action will do, and methods for assigning credit across long sequences.

In a world where an agent acts for weeks and the outcome arrives at the end, all of that becomes necessary again.

The authors treat this as a reason for optimism, since the theory already exists and was developed carefully over decades. It is also a reason for caution: those methods were developed and validated on small problems and simulators, and it is an open question how much of the theory survives contact with an agent acting in the real world over long horizons.

Either way, this is a concrete prediction the thesis makes, and it is testable.

A position paper is worth more if you can say what would falsify it, so it is worth naming the disagreements precisely.

The data-exhaustion claim would be undermined if capability keeps improving substantially from data sources that are still human in origin: video, private and specialist corpora, or better use of what already exists. Several people argue exactly this.

The imitation-ceiling claim would be undermined if models trained on human data continue to exceed what any individual human can do. There is a real counter-argument here: a system that has absorbed the output of millions of specialists is not imitating one person, and combining knowledge across fields that no single human holds together may itself produce novelty.

The scope claim would be undermined if experiential learning stays confined to domains with cheap automatic verification. That is where it works today, and the gap between checking a proof and checking whether a business decision was correct is enormous.

And the whole programme could stall on a problem the paper does not solve, which is what the third lesson of this path is about.

Holding those four in mind is the difference between reading a position paper and being persuaded by one.