alignment
4 free lessons tagged alignment across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
The Part That Is Not Solved
The paper's answer to safety is that reward signals adapt to human feedback: a network inside the reward function learns which grounded signals to weight from how people respond. This lesson works through that proposal and the published critique arguing it fails twice over, on specification gaming and on goal misgeneralization, and what an honest reading of the disagreement leaves you with.
Direct Preference Optimization: DPO, IPO, KTO, and SimPO
A deep dive into DPO (Rafailov et al. 2023) and its successors — how they reformulate RLHF as a classification problem, the math behind the implicit reward, and where each variant wins or loses against PPO-based pipelines.
RLHF: From Christiano 2017 to InstructGPT to the Offline Era
Trace the full arc of Reinforcement Learning from Human Feedback — preference data, Bradley-Terry reward models, PPO with KL penalty, reward hacking, and why most labs have moved to offline alternatives like DPO and RAFT.
LLM Post-Training: SFT, RLHF, DPO, and Modern Alignment Recipes
A deep dive into how raw pretrained language models become helpful assistants — from supervised fine-tuning on curated demonstrations, through reward modeling and PPO-based RLHF, to modern direct alignment methods like DPO and the recipes used in Llama 3, Llama 4, and DeepSeek.