dpo
3 lessons tagged dpo: free, quiz-checked micro-lessons.
Direct Preference Optimization: DPO, IPO, KTO, and SimPO
A deep dive into DPO (Rafailov et al. 2023) and its successors — how they reformulate RLHF as a classification problem, the math behind the implicit reward, and where each variant wins or loses against PPO-based pipelines.
RLHF: From Christiano 2017 to InstructGPT to the Offline Era
Trace the full arc of Reinforcement Learning from Human Feedback — preference data, Bradley-Terry reward models, PPO with KL penalty, reward hacking, and why most labs have moved to offline alternatives like DPO and RAFT.
LLM Post-Training: SFT, RLHF, DPO, and Modern Alignment Recipes
A deep dive into how raw pretrained language models become helpful assistants — from supervised fine-tuning on curated demonstrations, through reward modeling and PPO-based RLHF, to modern direct alignment methods like DPO and the recipes used in Llama 3, Llama 4, and DeepSeek.