All lessons
preference-learning
2 free lessons tagged preference-learning across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
AI
advancedDirect Preference Optimization: DPO, IPO, KTO, and SimPO
A deep dive into DPO (Rafailov et al. 2023) and its successors — how they reformulate RLHF as a classification problem, the math behind the implicit reward, and where each variant wins or loses against PPO-based pipelines.
12 steps·~18 min
AI
advancedRLHF: From Christiano 2017 to InstructGPT to the Offline Era
Trace the full arc of Reinforcement Learning from Human Feedback — preference data, Bradley-Terry reward models, PPO with KL penalty, reward hacking, and why most labs have moved to offline alternatives like DPO and RAFT.
12 steps·~18 min