reward-design
2 free lessons tagged reward-design across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
Reward Design: Where Execution Agents Go Wrong
An agent optimises the reward you wrote, not the objective you meant, and in execution the gap between those is unusually easy to open. This lesson computes how a mis-sized penalty makes leaving part of the order unexecuted rationally optimal, and covers the shaping that provably does not change the policy.
The Part That Is Not Solved
The paper's answer to safety is that reward signals adapt to human feedback: a network inside the reward function learns which grounded signals to weight from how people respond. This lesson works through that proposal and the published critique arguing it fails twice over, on specification gaming and on goal misgeneralization, and what an honest reading of the disagreement leaves you with.