Direct Preference Optimization: DPO, IPO, KTO, and SimPO

Reinforcement Learning from Human Feedback (RLHF) as practiced in InstructGPT has three moving parts: a supervised fine-tuned (SFT) base, a separate reward model trained on human preference pairs, and a policy updated by PPO while a KL penalty keeps it from drifting too far from the SFT base. That's three models in GPU memory simultaneously, a reward-model training loop, and PPO's notorious sensitivity to learning rate, batch size, and clipping threshold.

In practice, teams spend more time stabilizing PPO than doing actual alignment work. Reward hacking — where the policy finds high-scoring outputs the reward model didn't anticipate — is endemic. The KL coefficient has to be tuned per task. And the whole pipeline requires thousands of reward-model queries per gradient step.

DPO (Rafailov et al., 2023, Stanford / Berkeley) asks: given that PPO is just optimizing a reward signal derived from preferences, can we skip the reward model entirely and train the policy directly on preference pairs? The answer is yes — and the math is surprisingly clean.

Standard RLHF fits a reward model $r_\phi$ using the Bradley-Terry assumption: the probability that response $y_w$ is preferred over $y_l$ given prompt $x$ is

$p(y_w \succ y_l \mid x) = \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$

Then PPO maximizes $\mathbb{E}[r_\phi(x,y)] - \beta \cdot \text{KL}[\pi_\theta \| \pi_{\text{ref}}]$.

The DPO insight: the optimal policy under that KL-constrained objective has a closed form:

$r^*(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$

where $Z(x)$ is a partition function that cancels in the preference ratio. Plug this back into the Bradley-Terry likelihood and $Z(x)$ drops out, leaving a loss that depends only on log-probability ratios between the current policy and the frozen reference:

$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$

No reward model. No PPO rollouts. Just a binary cross-entropy over log-ratio differences.

DPO collapses the three-model RLHF stack into a two-model setup (policy + frozen reference) with a single supervised-style training loop. The reward model and PPO optimizer are both eliminated.

The loss is straightforward to implement. Given a batch of (prompt, chosen, rejected) triples:

import torch
import torch.nn.functional as F

def dpo_loss(policy_logps_w, policy_logps_l,
             ref_logps_w, ref_logps_l, beta=0.1):
    """All logps are per-token-averaged log-probabilities."""
    log_ratio_w = policy_logps_w - ref_logps_w
    log_ratio_l = policy_logps_l - ref_logps_l
    logits = beta * (log_ratio_w - log_ratio_l)
    loss = -F.logsigmoid(logits).mean()
    return loss

policy_logps_w is sum(log p_theta(token | context)) over the chosen response tokens, divided by length (length-normalisation avoids penalising long preferred responses). The reference model is frozen and run in torch.no_grad().

Heads up: the single most common bug is computing log-probs over the full sequence (prompt + response) rather than masking the prompt tokens out. The loss should measure only the response portion.

DPO's Bradley-Terry derivation assumes the preference data comes from a consistent reward function — if that assumption breaks (noisy labels, contradictory annotators), the policy can overfit to the noise and push the chosen/rejected log-ratio gap to infinity. Azar et al. (2023, Google DeepMind) call this the overfitting regime and propose IPO.

IPO replaces the log-sigmoid with a squared loss:

$\mathcal{L}_{\text{IPO}} = \mathbb{E}\left[ \left( \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} - \frac{1}{2\tau} \right)^2 \right]$

The $\frac{1}{2\tau}$ target acts as a margin: it limits how far the log-ratio gap is pushed, preventing the probability of chosen responses from saturating to 1.0. IPO is less aggressive than DPO on clean data but degrades more gracefully on noisy preference datasets — empirically useful when your annotation pipeline has inter-annotator agreement below ~75%.

DPO and IPO both require paired data: for every prompt you need one good and one bad completion. Ethayarajh et al. (2023, Stanford) observed that real-world feedback is often unpaired — thumbs up/down on individual responses, not comparison pairs. KTO works on single-label data.

Inspired by Kahneman & Tversky's prospect theory (losses feel roughly 2x larger than equivalent gains), KTO trains on:

$\mathcal{L}_{\text{KTO}} = \mathbb{E}\left[ \lambda_w \cdot \sigma\left(\beta \cdot h_\theta(x, y_w) - z_{\text{ref}}\right) + \lambda_l \cdot \sigma\left(- \beta \cdot h_\theta(x, y_l) + z_{\text{ref}}\right) \right]$

where $h_\theta(x,y) = \log \pi_\theta(y|x) - \log \pi_{\text{ref}}(y|x)$ and $z_{\text{ref}}$ is a running estimate of the KL-penalised expected reward. $\lambda_w$ and $\lambda_l$ are hyperparameters (default: loss-averse ratio ~$\lambda_l / \lambda_w = 1.33$).

Key advantage: you can mix independently collected "good" and "bad" completions without needing them paired per prompt. This unlocks large filtered SFT datasets as implicit preference signal.

All the variants above keep a reference model — the frozen SFT copy. SimPO (Meng et al., 2024, UIUC) eliminates it entirely. The reference log-prob terms drop out, and the margin is replaced by a fixed target $\gamma$:

$\mathcal{L}_{\text{SimPO}} = -\mathbb{E}\left[ \log \sigma\left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w \mid x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l \mid x) - \gamma \right) \right]$

The $1/|y|$ term is length normalisation applied directly to the raw log-probs rather than the log-ratio. Without a reference model you save ~50% GPU memory during training.

Catch: without the KL anchor to the SFT base, SimPO can drift toward degenerate outputs (very short or repetitive responses). The paper recommends $\gamma \in [0.5, 1.5]$ and starting from a well-tuned SFT checkpoint, not a raw base model. On AlpacaEval 2, SimPO matched or exceeded DPO with Llama-3-8B-Instruct while using one fewer GPU.

The four methods split along two axes: whether they require paired preference data and whether they require a frozen reference model.

The practical case for DPO over PPO:

• Memory: PPO needs policy + ref + reward model + value head in memory simultaneously. DPO needs policy + ref (2 models, both the same size).
• Stability: PPO's policy-gradient variance requires careful clipping, KL coefficients, advantage normalisation, and warm-up schedules. DPO is a cross-entropy loss — optimisers don't need tuning beyond learning rate.
• Reproducibility: PPO results are notoriously sensitive to implementation details (Engstrom et al., ICLR 2020). DPO results are much more consistent across runs.
• Data efficiency: DPO reuses collected preference pairs without online rollouts, so you don't need a live inference server during training.

That said, DPO trades away PPO's exploration: PPO continuously samples from the current policy and can discover high-reward completions not in the dataset. DPO is static — it can only improve over distributions present in the preference data.

DPO has real failure modes:

Distribution mismatch: The preference pairs were collected from a (usually weaker) policy. If the current policy drifts far from that distribution, the implicit reward signal becomes unreliable. PPO's online rollouts keep the policy and reward model in sync.

Long-horizon tasks: Code generation, multi-step reasoning, and agentic tasks have sparse rewards (did the code compile? did the agent succeed?). PPO can assign credit across many steps via GAE. DPO cannot — it sees the full response as a unit.

Iterative refinement: Anthropic's Constitutional AI and DeepSeek-R1 both use iterative RL loops where the model's own outputs become training signal. DPO cannot close this loop without a new round of human preference collection.

Math and reasoning benchmarks: Multiple papers (including the DPO paper itself) show PPO outperforming DPO on GSM8K and MATH when the SFT base is strong. The on-policy exploration matters for search-like reasoning.

A practical decision tree:

1. Do you have paired (chosen, rejected) data?

   • No → use KTO (single-label thumbs up/down).
   • Yes → continue.

2. Do you care about GPU memory?

   • Very constrained → SimPO (no reference model).
   • Fine with 2x model copies → continue.

3. Is your preference data noisy or from multiple annotators?

   • Yes → IPO (squared loss prevents margin collapse).
   • Clean, high-agreement labels → DPO.

4. Is your task long-horizon or reasoning-heavy?

   • Yes → consider PPO (or GRPO, a group-relative variant used by DeepSeek-R1).

For most chat/instruction-following fine-tunes on a good SFT base, DPO is the right starting point. Layer in iterative DPO (collect new preference data with the updated model, retrain) if quality plateaus.

$\beta$ (KL coefficient): Controls how closely the policy stays to the reference. Too low → the policy barely moves. Too high → aggressive updates that destabilise generation. Start at 0.1 for chat models, 0.05 for creative tasks.

Length normalisation: Without it, DPO penalises long chosen responses because their aggregate log-prob is lower. The fix is to normalise by response token count before computing the loss.

Data ratio (chosen:rejected quality gap): DPO is most effective when the chosen response is clearly better — pairs where the margin is ambiguous add noise. Filter pairs by annotator agreement score if available.

Learning rate and epochs: DPO trains for far fewer steps than SFT. 1-3 epochs at $\leq 5 \times 10^{-7}$ LR (on top of an SFT checkpoint) is typical. More epochs → reward hacking in the implicit reward space — the policy learns to boost log-ratio gaps without improving quality.