Streams, Actions, Rewards, and Thinking That Is Not Ours

The first dimension is the one most visible in systems people use today.

Current assistants operate in episodes. A conversation begins, runs for some number of turns, and ends. The next conversation starts from the same place as the last one, with whatever was learned in between discarded. Memory features exist and are retrieval on top of a static model rather than learning: the weights do not change, so the system is not improving, it is looking things up.

The authors propose agents that inhabit a continuous stream of experience progressing over a long timescale, like a human life rather than like a session.

The difference this makes is about which goals are reachable. A goal like improve this person's health, or reduce this system's error rate, or advance this research question, does not resolve inside one session. It requires acting, observing what happened weeks later, and adjusting. An episodic agent cannot pursue it, not because it is not clever enough but because the outcome arrives after it has stopped existing.

So the stream is not a convenience feature. It is what makes long-horizon goals possible at all.

The second dimension concerns what the agent can do and perceive.

A language model's action space is emitting text and its observation space is receiving text. Everything else happens because a human read the text and did something. Tool use widens this, and it remains a narrow channel designed for human convenience.

The authors argue for agents acting through machine-friendly interfaces directly: calling programmatic interfaces, controlling systems, reading sensors, running experiments. Not because human interfaces are unusable, but because they are a bottleneck the agent did not need.

The consequence they emphasise is exploration. An agent that can only answer questions is confined to tasks a human posed. An agent that can act can try things nobody asked for, observe the result, and learn from it. That is where information the training data did not contain comes from.

It is also, unavoidably, where the risk profile changes. An agent that only emits text is bounded by whoever reads it. An agent that acts directly is not, and the paper is proposing exactly that expansion.

The third dimension is the one the paper is most often quoted on, and the one the next lesson argues is unresolved.

Today's reward signal is human preference: a rater compares outputs and the model is trained toward the preferred one. The authors' objection is the ceiling from the previous lesson. A rater cannot reward an answer they do not recognise as good.

Their alternative is to take rewards from measurable quantities the environment already produces. The paper lists a striking range of them: cost, error rates, hunger, productivity, health metrics, climate metrics, profit, sales, exam results, success, visits, yields, stocks, likes, income, pleasure and pain, economic indicators, accuracy, power, distance, speed, efficiency, energy consumption.

The point of the list is abundance. Once an agent is connected to the world through rich actions and observations, grounded signals are everywhere, and none of them require a human to sit and judge.

Read that list again with an engineer's eye, though, and something should be uncomfortable. Every one of those is a proxy. That discomfort is correct, and it is the subject of the next lesson.

The fourth dimension is the most philosophically loaded and the least discussed.

When a model reasons step by step, it is producing text that resembles human reasoning, because human reasoning is what it was trained on. Chain of thought is imitation of a human cognitive style, expressed in human language.

The authors argue this inherits our limitations along with our methods. Human reasoning is shaped by the constraints of a brain that thinks in language, holds a handful of items at once, and evolved for a world quite different from the one an agent operates in. There is no reason to expect that style to be optimal for a system with different constraints.

Their alternative is planning grounded in a world model: a learned predictor of what happens if the agent acts. Reasoning becomes simulating consequences and choosing, and it is optimised against whether the prediction turned out right, not against whether it reads like something a person would write.

The practical difficulty is that such reasoning would be, by construction, harder to inspect. Chain of thought is legible precisely because it imitates us, and interpretability currently leans on that. Discarding it buys performance and costs oversight.

The dimensions are not independent, and seeing how they depend on each other explains why the paper proposes all four rather than any one.

Streams are what make long-horizon goals coherent, and a long horizon is only useful if the agent can act to affect it, which is the second dimension.

Acting in the world is what makes grounded rewards available, since a signal like error rate only exists for an agent whose actions affect a running system. Without actions, there is nothing to measure.

Grounded rewards are what make non-human planning trainable, since a plan expressed in a form no human recognises can only be evaluated by whether it worked. Human preference cannot score it.

And a world model good enough to plan with can only be learned from a long stream of acting and observing, which closes the loop back to the first dimension.

So the proposal is a package. Adopting one dimension without the others tends to stall: streams without actions give a long memory and nothing to do with it, grounded rewards without streams give short-horizon metric optimisation, which is where most of the failure stories come from.

It is worth grading current systems against the four dimensions, because the gap is uneven.

Actions and observations are the furthest along. Tool use, computer control, code execution and programmatic interfaces are all deployed, and agents doing real work through them are ordinary now.

Rewards are partially there. Verifiable domains give genuine outcome-based signals, and they are the source of recent reasoning gains. Everything outside those domains still runs on human preference.

Streams are barely started. Context windows and retrieval give the appearance of continuity while the weights stay frozen, so nothing is learned. The catalogue's path on defining AGI makes the same observation from a different angle, scoring long-term memory storage at essentially zero for current models.

Planning is the least advanced. Chain of thought remains an imitation of human reasoning, and world models good enough to plan with over long horizons in open environments do not exist.

So the paper is roughly one dimension confirmed, one partially, and two open. That is a reasonable position for a position paper, and it means the interesting question is whether the two open ones are hard or merely unbuilt.

Of the two open dimensions, streams look like an engineering gap and are closer to a research problem, which is worth understanding because it constrains the whole programme.

Continual learning, updating a model's weights as experience arrives, runs into catastrophic forgetting: training on new data degrades what was previously learned, sometimes severely. This has been studied for decades and has no general solution.

The workarounds all have costs. Freeze most of the model and adapt a small part, which limits how much can be learned. Replay old data alongside new, which requires storing it and grows without bound. Periodically retrain from scratch on everything, which is not learning from a stream, it is batch training on a schedule.

There is also a stability question. A model that updates continuously from its own experience can drift, and the loop from the synthetic data path applies directly: a system training on its own output with no anchor is the configuration that degrades.

So an agent that genuinely learns from a long stream needs continual learning without forgetting and without drift, and that is not a matter of building what is known.

The value in a paper like this is not that it tells you what will happen. It is that it names the axes along which the next round of systems will differ, which makes it easier to read what arrives.

When a new agent product is announced, the four dimensions give you questions worth asking. Does it retain and learn across sessions, or does it retrieve from a frozen model? Can it act, or does it produce text a human executes? What is the training signal: a rater's preference, or an outcome that happened? Does it reason in language because that works, or because that is what it was trained to imitate?

Those questions cut through marketing more reliably than benchmark scores, because they ask about the mechanism.

The paper's largest gap is that it treats the reward question as mostly settled by pointing at the abundance of grounded signals. Several readers, including one whose critique the next lesson works through in detail, argue that the abundance is the problem rather than the solution.

That disagreement is where the real content of this debate sits.