rlhf
5 lessons tagged rlhf: free, quiz-checked micro-lessons.
RL in Reasoning Models: How o1, DeepSeek-R1, and Friends Think
A deep look at how reinforcement learning on chains-of-thought powers o1, DeepSeek-R1, Claude reasoning, and Gemini Thinking — covering GRPO, MCTS-style search, test-time compute scaling, and distillation into smaller models.
Direct Preference Optimization: DPO, IPO, KTO, and SimPO
A deep dive into DPO (Rafailov et al. 2023) and its successors — how they reformulate RLHF as a classification problem, the math behind the implicit reward, and where each variant wins or loses against PPO-based pipelines.
RLHF: From Christiano 2017 to InstructGPT to the Offline Era
Trace the full arc of Reinforcement Learning from Human Feedback — preference data, Bradley-Terry reward models, PPO with KL penalty, reward hacking, and why most labs have moved to offline alternatives like DPO and RAFT.
Reinforcement Learning in 2026: Where It Ships and Where It Stalls
An honest map of RL in 2026 — the domains where it actually reaches production (LLM post-training, robotics policies, ad bidding, RLHF, reasoning models) and the places where it still cannot reliably cross the lab-to-deployment gap.
LLM Post-Training: SFT, RLHF, DPO, and Modern Alignment Recipes
A deep dive into how raw pretrained language models become helpful assistants — from supervised fine-tuning on curated demonstrations, through reward modeling and PPO-based RLHF, to modern direct alignment methods like DPO and the recipes used in Llama 3, Llama 4, and DeepSeek.