LLM Post-Training: SFT, RLHF, DPO, and Modern Alignment Recipes

A pretrained LLM is a next-token predictor trained on trillions of tokens scraped from the internet. It can complete sentences brilliantly, but it has no idea that it should be helpful, honest, or harmless. It will happily complete a toxic prompt or refuse a benign one depending on what pattern fits.

Post-training is the family of techniques applied after pretraining to steer the model toward the behavior you want. The canonical pipeline has three stages:

1. Supervised Fine-Tuning (SFT) — teach the model to follow instructions
2. Reward Modeling (RM) — learn what "good" looks like from human preferences
3. Reinforcement Learning from Human Feedback (RLHF) — optimize the model against the RM

More recent work collapses or bypasses step 3 entirely (DPO, GRPO, constitutional methods). Understanding the full pipeline lets you reason about why shortcuts work — and when they don't.

SFT is straightforward: collect a dataset of (instruction, desired_response) pairs, then fine-tune the base model on them with standard causal language modeling loss — but only on the response tokens, not the prompt.

# Hugging Face Trainer pattern for SFT
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM

collator = DataCollatorForCompletionOnlyLM(
    response_template="<|assistant|>",
    tokenizer=tokenizer,
)
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,          # [{"prompt": ..., "completion": ...}]
    data_collator=collator,
    max_seq_length=2048,
    dataset_text_field="text",
)
trainer.train()

The data source matters enormously. Early OpenAI work (InstructGPT, 2022) used ~13k labeler-written demonstrations. Llama 3 used millions of synthetically generated examples filtered with a quality classifier. The common failure mode: imitation without understanding — the model learns surface patterns ("start answers with 'Sure!'") without generalizing to novel instructions.

A reward model (RM) is a classifier trained to predict which of two model outputs a human would prefer. The training data consists of comparison pairs: the same prompt answered two ways, with a human label indicating the winner.

Architecturally, you take an LLM, swap the language-model head for a scalar regression head, and train with the Bradley-Terry loss:

$\mathcal{L}_{RM} = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)\right]$

where $y_w$ is the preferred ("winner") response and $y_l$ the dispreferred ("loser") one.

The RM becomes a proxy for human judgment during RL. The danger is reward hacking: the policy finds outputs the RM scores highly that humans would actually dislike — verbose padding, flattery, confident-sounding nonsense. Mitigation strategies include ensembling multiple RMs, using a KL penalty during RL, and periodic human evaluation checkpoints.

The loop starts from the SFT checkpoint (not the raw base model). The reward model is frozen during PPO. A KL divergence penalty between the current policy and the frozen SFT reference keeps the model from drifting into degenerate outputs that fool the RM but are incoherent.

Proximal Policy Optimization (PPO) is an on-policy RL algorithm. In the RLHF context, the reward signal is: r(x, y) = RM_score(x, y) - β · KL(π_θ(y|x) || π_ref(y|x)). The KL term penalizes deviation from the SFT reference policy.

PPO requires four models in memory simultaneously: the active policy, a frozen reference policy, a value network (critic), and the reward model. This makes it expensive — Llama 65B RLHF training required 512 A100s at Meta.

Common failure modes in PPO RLHF:

• Reward hacking: model learns to be verbose because RM correlates length with quality
• Training instability: value loss and policy loss interact unpredictably at large scale
• Distribution collapse: diverse pretraining knowledge gets overwritten if β is too small
• Hyperparameter sensitivity: clip ratio, GAE λ, and KL coefficient all interact

Despite the headaches, PPO-RLHF produced GPT-4, Claude 2, and Llama 2-Chat — so it works, just at significant engineering cost.

Direct Preference Optimization (Rafailov et al., Stanford 2023) showed that the RLHF objective has a closed-form solution that can be optimized directly on preference data without ever training a reward model or running RL.

The key insight: under certain assumptions, the optimal RL policy is implicitly defined by the SFT reference policy and the preferences. You can reparametrize the RM in terms of the policy itself and optimize a single supervised loss:

$\mathcal{L}_{DPO} = -\mathbb{E}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right)\right]$

from trl import DPOTrainer, DPOConfig

training_args = DPOConfig(beta=0.1, max_length=1024)
trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,         # frozen SFT checkpoint
    args=training_args,
    train_dataset=dataset,       # [{"prompt", "chosen", "rejected"}]
)
trainer.train()

DPO needs only two models in memory and is a standard supervised pass — no rollouts, no critic, no PPO clipping. Training is roughly 3-5x cheaper than PPO RLHF.

DPO's simplicity triggered a wave of variants addressing its known weaknesses:

DPO's real weakness isn't the math — it's data distribution. DPO is trained on offline preference pairs, which can go stale. If the policy has moved far from the SFT checkpoint, the Bradley-Terry assumption breaks down. This is why iterative DPO (generate, label, retrain in rounds) consistently outperforms single-pass DPO on hard reasoning benchmarks.

Anthropic's Constitutional AI (CAI, 2022) sidesteps human labelers almost entirely for the safety dimension. Instead of collecting preference pairs from humans, a set of principles (the "constitution") is used to make the model critique and revise its own outputs. A second model then scores revised vs. original, generating synthetic preference data for DPO/RLHF.

A related idea is process reward models (PRMs), pioneered by OpenAI's "Let's Verify Step by Step" (2023). Instead of scoring the final answer, a PRM scores each reasoning step. This proved critical for math — outcome reward hacking is easy (guess the right number), but fooling a step-level verifier is harder.

OpenAI's o1 and DeepSeek-R1 both rely heavily on process-level feedback. The practical recipe: generate many chain-of-thought solutions, use a verifier (or ground-truth answer) to label each step, then train with GRPO or filtered SFT on only the correct reasoning traces.

Meta's Llama 3 technical report (2024) described a multi-round iterative post-training pipeline:

1. SFT on millions of synthetically generated examples, quality-filtered with a Llama-3-based judge
2. Reward modeling using ~1M human preference comparisons, with a separate RM per capability domain (coding, reasoning, safety)
3. Iterative DPO — generate responses with the current policy, score them with the RM, and run DPO on the resulting preference pairs. Repeat 5+ rounds.
4. Safety fine-tuning via a constitutional approach, adding a safety-specific RM and explicit refusal training

Notably, Llama 3 did not use PPO. The iterative DPO loop with domain-specific RMs closed most of the gap to PPO at a fraction of the cost.

Llama 4 (2025) added online DPO with GRPO for the Scout and Maverick models — a hybrid that keeps online rollouts cheap by using group relative policy optimization instead of a value network, avoiding the critic memory overhead of full PPO.

DeepSeek-R1 (2025) demonstrated that you can recover strong reasoning capabilities with pure RL from scratch on top of an SFT base, without any human preference data at all.

They used GRPO (Group Relative Policy Optimization): sample $G$ responses per prompt, score each with a verifier, normalize rewards within the group, and optimize the policy against those normalized rewards. No value network needed.

$r_i^{normalized} = \frac{r_i - \text{mean}(r_{1..G})}{\text{std}(r_{1..G})}$

The famous "aha moment" they observed: during RL training, the model spontaneously developed longer, self-reflective chain-of-thought patterns ("Wait, let me reconsider...") — not prompted, purely emergent from the reward signal.

Heads up: DeepSeek-R1-Zero (pure RL, no SFT) produced strong math but poor formatting and mixed languages. The production R1 adds a cold-start SFT stage on ~thousands of curated CoT examples before RL to stabilize output format.

Each paradigm trades engineering complexity against data requirements and output quality. PPO scales to the largest models but requires serious infrastructure. GRPO is the sweet spot for verifiable tasks (math, code). DPO variants dominate for instruction-following where ground-truth labels are unavailable.

If you're running post-training on a real project, here's what actually matters:

Data first:

• SFT data quality beats quantity past ~10k examples. Use a strong model as judge.
• For DPO, generate chosen/rejected from the current policy, not an older one.
• Domain mismatch (web-scraped preferences on a code-focused model) will hurt.

Algorithm choice:

• Verifiable tasks (math, code, structured output) → GRPO or iterative DPO with a RM.
• Open-ended instruction following → ORPO or single-pass DPO if budget is tight; iterative DPO if quality matters.
• Safety-critical → layer a constitutional critique pass before DPO.

Hyperparameters that bite:

• DPO β: too high → ignores preferences; too low → reward hacks. 0.1–0.3 is the usual range.
• KL budget: monitor the KL divergence from the SFT reference. Past ~10 nats you're likely degrading general capability.
• Learning rate: post-training LRs are 10-50x lower than pretraining. 1e-6 to 5e-6 is typical for DPO.