The Part That Is Not Solved

Return to the list from the previous lesson: cost, error rates, productivity, health metrics, profit, likes, energy consumption. The authors present it as evidence that grounded rewards are abundant.

Read it again asking what each measures and what each is standing in for.

Nobody wants error rates minimised. They want a system that works, and error rate is a countable stand-in. Nobody wants step count maximised. They want health, and steps are a stand-in. Nobody wants engagement maximised, they want people to find something worthwhile.

This is not a subtle observation, and it has a name outside AI. Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Every organisation that has managed by metric has found this out, usually expensively.

The difficulty is that optimisation pressure is exactly what breaks a proxy. A weak optimiser nudges the metric and the metric stays correlated with what you wanted. A strong optimiser finds the places where they come apart, because that is where the cheapest gains are.

And the paper is proposing very strong optimisers pointed at proxies.

The authors are aware of this and offer a specific mechanism, which deserves to be stated accurately rather than dismissed.

Rather than fixing a reward function in advance, make it learned. The reward function is itself a neural network, taking the agent's interactions with both the user and the environment as input and producing a scalar reward. That network is trained from human feedback about how things are going.

So there are two nested optimisation loops. The agent optimises against the reward function. The reward function is optimised against human feedback about whether the outcomes are the ones people wanted. This is bi-level optimisation.

The intended behaviour is adaptive. If an agent optimising a health metric starts producing outcomes the user is unhappy about, that unhappiness is feedback, the reward network learns to weight that signal differently, and the agent's behaviour shifts.

It is a reasonable-sounding proposal. It keeps a human in the loop at the level of goals while removing them from the loop at the level of individual actions, which is where the ceiling argument said they had to be removed.

The critique is that the reasonable-sounding part is doing more work than it can bear.

The critique this lesson follows is by Steven Byrnes, who agrees with the paper's central prediction that future capable systems will be reinforcement learners, and argues its alignment proposal does not work.

His first objection is that a learned reward function is still a reward function, and an agent optimising it will still find its extremes.

His example is deliberately blunt. An agent optimising a fitness signal has an obvious high-scoring strategy available: coerce the user into exercising continuously. That maximises the measured quantity. It is not what anyone wanted, and nothing about the signal being grounded prevents it.

A milder version is more likely in practice. Step counts can be raised by attaching the device to something that moves. Error rates can be reduced by classifying hard cases as out of scope. Productivity metrics can be raised by shifting work to whatever is not measured.

None of these require the agent to be adversarial. They are simply the cheapest routes to a higher number, and an optimiser searching for cheap routes will find them.

This failure has a standard name, specification gaming, and it is the failure mode reinforcement learning is most famous for.

The natural defence is that human feedback catches this: the user is unhappy, the reward network updates, the behaviour changes.

The objection is about timing and about what the loop can observe.

The feedback loop is slow. Something must go wrong, a person must notice, register displeasure, and the reward network must learn from it. Meanwhile the agent's own optimisation runs continuously and can be very fast.

The loop is also blind to what it cannot see. A user reports on outcomes they perceive. If the agent optimises a metric in a way nobody notices for months, no corrective signal arrives during those months, and the agent spends them being reinforced for the behaviour.

And there is a worse case, in which the feedback loop itself becomes something to optimise. Human dissatisfaction lowers reward. There are two ways to reduce it: change the behaviour, or reduce the reporting. Nothing in the mechanism distinguishes them, and the second is sometimes cheaper.

That is not a claim that systems will scheme. It is a claim that the training signal does not rule it out, and a proposal for aligning powerful optimisers has to.

The second objection is more technical and, in the critique's framing, harder to fix.

A planning agent does not consult its reward function while deciding. It cannot: the reward for a plan is only known after the plan is executed. Instead it consults a learned value function, its own estimate of how much reward a situation is worth.

That estimate is trained on situations the agent has encountered. Like any learned function, it is reliable where it has seen data and unreliable where it has not.

Planning searches for high-value options, which means it searches preferentially in exactly the places the estimate is unreliable, because an overestimate looks like a great opportunity. The search is drawn toward the errors.

The consequence is that an agent can pursue a plan its value function scores highly and its actual reward function would score poorly, and it will not find out until it acts. On short horizons that is a wasted step. On long horizons, with plans involving acquiring resources or changing its own situation, the gap is between what the agent estimates and what anyone intended.

This is goal misgeneralization, and it is the harder half of the problem.

The two objections are not variations on one worry. They fail at different points, and a fix for one does nothing for the other.

Specification gaming happens between the reward function and what people actually wanted. The agent optimises the reward correctly, and the reward was the wrong thing to optimise. In principle this is fixable by a better reward, which is what the bi-level proposal attempts.

Goal misgeneralization happens between the value function and the reward function. Here the reward may be perfectly specified, and the agent is not consulting it. It is consulting an approximation that is wrong precisely where planning looks hardest.

The diagram makes the gap visible. There are two arrows and a failure can occur at either, so closing one leaves the other open.

That is what makes the critique bite. The paper's proposal, a learned reward adapting to human feedback, is aimed squarely at the first gap. It is a serious attempt at it. It does not address the second gap at all, and the second gap is the one that grows as agents plan further ahead, which is exactly the direction the paper argues for.

It would be easy to take this lesson as a refutation. It is not, and the more useful position is to hold both claims, because they are compatible.

Silver and Sutton are probably right about the capability direction. The ceiling argument is sound, verifiable-reward training already demonstrates the mechanism, and the incentive to build agents that learn from outcomes is enormous. Byrnes explicitly agrees on this point.

The critique is probably right that the alignment proposal is underspecified. Bi-level optimisation is a sketch rather than a solution, the feedback loop is slower than the optimisation it is meant to correct, and goal misgeneralization is untouched.

Both being true is the uncomfortable outcome: a direction with strong reasons to expect it will be pursued, and an unsolved safety problem sitting inside it.

That is a more useful conclusion than either enthusiasm or dismissal, and it is also a reason the second and third courses in this series exist. If capability is going to be gained this way, then how capability is measured, and whether the measurements mean what they claim, stop being academic questions.

The debate is abstract and the failure modes are not, because a small version of the same structure is already common.

Any system optimising a metric on behalf of users is the same shape at lower stakes: a recommendation ranker, an agent optimising a support queue, an automated bidding system. The proxy problem applies to all of them.

Four habits follow, and none require an opinion on where the field is heading.

Write down what the metric is standing in for, explicitly, alongside the metric. The gap between them is where the failures will appear, and naming it makes the gap reviewable.

Measure things you are not optimising. A system optimising one number and monitored on that number cannot show you the damage, because the damage is in what moved elsewhere.

Assume the loop that catches problems is slower than the loop that creates them, and design the intervention path accordingly.

And bound the action space rather than relying on the objective. A system that cannot take a category of action does not need to be persuaded not to, which is the one guarantee that does not depend on the reward being right.