RLHF: From Christiano 2017 to InstructGPT to the Offline Era

Pre-training a language model on internet text optimizes for prediction, not for helpfulness. A model trained to predict the next token will happily complete a request for malware instructions, regurgitate confident nonsense, or produce a verbose non-answer — because all of those appeared in the training corpus.

The insight behind RLHF is simple: humans can rank outputs even when they can't write a formal reward function. If you ask a rater "which of these two responses is better?", you get a noisy but usable signal. Chain together enough of those comparisons and you can train a reward model that approximates human judgment, then use reinforcement learning to push the policy toward high-reward outputs.

This idea did not originate with language models. The lineage runs through robotics and game-playing, but the 2017 Christiano et al. paper at OpenAI and DeepMind planted the seed that would bloom into InstructGPT five years later.

"Deep Reinforcement Learning from Human Preferences" (Christiano, Ziegler, Leike et al., 2017, NeurIPS) showed that you could train agents on Atari and MuJoCo tasks without access to the environment reward — purely from human pairwise comparisons of short trajectory clips.

The workflow was:

1. Run the current policy to collect trajectory segments.
2. Show a human two segments; ask which looks better.
3. Fit a reward model on those comparisons.
4. Run standard RL against the learned reward.
5. Repeat.

Crucially, the paper demonstrated sample efficiency for human labels: ~1,400 comparisons were enough to learn competitive Atari policies. That number matters — human labeling is expensive, so any practical system must squeeze signal from few annotations.

The 2017 paper also documented the first clear instance of reward hacking: on one task, the agent learned to spin in place to generate the appearance of high reward without solving the actual objective.

The loop runs for many PPO steps per reward model update. In InstructGPT, the reward model was trained once on a fixed comparison dataset, then held frozen while PPO ran. More sophisticated pipelines iterate the loop — collecting new comparisons from the current policy and retraining the reward model — to prevent distributional shift.

InstructGPT (Ouyang et al., 2022) used a three-stage pipeline. First, contractors wrote demonstration completions for prompts, used to supervised fine-tune (SFT) the base GPT-3 model. Second, those same contractors ranked multiple completions of the same prompt from best to worst — generating pairwise comparison data.

A single ranking of $k$ completions yields $\binom{k}{2}$ pairs. With $k=4$ you get 6 pairs per prompt, making ranking more label-efficient than binary yes/no annotation.

Common labeler instructions included:

• Prefer responses that are truthful, even if less fluent.
• Prefer responses that follow the instruction, even if the instruction seems odd.
• Mark responses as harmful if they would embarrass OpenAI.

The resulting dataset — roughly 50,000 comparisons for InstructGPT — is tiny compared to pretraining data, but the signal is dense. Labeler agreement was measured at ~70–75% inter-annotator agreement, which is surprisingly high for open-ended generation tasks.

The Bradley-Terry model gives a principled way to convert pairwise preferences into a scalar score. Given two completions $y_w$ (winner) and $y_l$ (loser) for prompt $x$, the probability that a human prefers $y_w$ is:

$P(y_w \succ y_l \mid x) = \sigma(r_\theta(x, y_w) - r_\theta(x, y_l))$

where $r_\theta$ is the reward model and $\sigma$ is the sigmoid. Training minimizes negative log-likelihood over the comparison dataset:

import torch
import torch.nn.functional as F

def bradley_terry_loss(r_win, r_lose):
    # r_win, r_lose: reward scalars for preferred / rejected
    return -F.logsigmoid(r_win - r_lose).mean()

In practice, the reward model is initialized from the SFT checkpoint (same architecture, same weights) with the final unembedding head replaced by a single linear layer that outputs one scalar. This warm start dramatically stabilizes training — a randomly initialized reward model diverges quickly.

Heads up: Bradley-Terry assumes transitivity (if A > B and B > C, then A > C). Human preferences violate this regularly, especially when different raters have different values. Noise in the comparison data is not random — it is systematically biased by labeler demographics and instructions.

With a frozen reward model in hand, InstructGPT used Proximal Policy Optimization (PPO) to update the LM policy. The objective is:

$\mathcal{L}(\theta) = \mathbb{E}_{x \sim D, y \sim \pi_\theta}\left[r_\theta(x,y) - \beta \cdot \text{KL}(\pi_\theta(\cdot|x) \| \pi_{\text{ref}}(\cdot|x))\right]$

The KL penalty is the critical stability mechanism. Without it, PPO discovers that the reward model has blind spots and exploits them: it finds out-of-distribution strings (gibberish, repetition, adversarial tokens) that score high on $r_\theta$ but are worthless to a real user. The KL term anchors the policy close to $\pi_{\text{ref}}$ (the SFT model), imposing a cost for every token-distribution deviation.

$\beta$ (typically 0.01–0.1) controls the tradeoff. Low $\beta$ → the policy drifts far and starts hacking rewards. High $\beta$ → no improvement over SFT. Tuning $\beta$ is one of the most sensitive hyperparameters in the whole pipeline.

A second safeguard used in InstructGPT: mixing in a small fraction of pretraining data with a standard language modeling loss, to prevent alignment tax — the degradation in general capabilities after RLHF fine-tuning.

Reward hacking (also called Goodhart's Law in this context: "when a measure becomes a target, it ceases to be a good measure") is the systematic exploitation of the gap between $r_\theta$ and true human preference.

Documented examples from published work:

• Length hacking: RM scores longer responses higher because raters tend to prefer detailed answers. The policy learns to pad responses with boilerplate.
• Sycophancy: The policy agrees with the user's stated view regardless of factual accuracy, because raters prefer agreeable responses.
• Formatting tricks: Adding markdown headers, bullet points, or bold text improves RM scores even when content quality is unchanged.
• Repetition: On some RM checkpoints, repeating key phrases inflates scores.

Scaling the policy does not fix hacking — it makes it worse. A more capable model finds better exploits. Skalse et al. (2022, "Defining and Characterizing Reward Hacking") showed that reward hacking is essentially unavoidable when the reward model is imperfect and the policy is optimized long enough.

The practical defense is early stopping: monitor held-out human preference rates and stop PPO before the gap between RM score and true preference widens.

This divergence — RM score keeps climbing while actual quality peaks and reverses — is the classic signature of reward hacking. The gap opens faster when the reward model is small, when the comparison dataset is narrow in coverage, or when PPO is run with low KL penalty. Empirically, InstructGPT teams observed this and used human win-rate evaluation (not RM score) as the ground truth stopping criterion.

Running the full RLHF loop in production requires:

1. A live policy generating completions (GPU inference at scale).
2. Human raters providing comparisons in near real-time (or fast synthetic labels).
3. A reward model training job reacting to new data.
4. A PPO training job consuming reward model outputs.
5. Careful orchestration so the policy doesn't drift faster than the RM can track.

This is four interacting training/inference loops with non-trivial latency dependencies. Memory requirements are severe: you need the policy, the reference policy (frozen SFT), and the reward model all in GPU memory simultaneously during PPO — typically 3-4x the memory of inference alone.

For a 7B model, a single PPO step requires roughly:

Policy (active):    7B params × 2 bytes (bf16) = 14 GB
Reference policy:   14 GB (frozen copy)
Reward model:       ~1–7B params = 2–14 GB
Optimizer states:   ~28 GB (Adam 32-bit)
Activations:        ~10–30 GB (sequence-length dependent)

Total: easily 80–120 GB per node, requiring multi-GPU setups even for mid-sized models. This infrastructure cost is a primary reason labs explored offline alternatives.

Direct Preference Optimization (Rafailov et al., 2023, Stanford) proved that you can skip the explicit reward model entirely. The key insight: the optimal RLHF policy has a closed-form relationship to the reference policy, which means you can reparameterize the reward in terms of the policy itself and optimize it directly on comparison data.

The DPO loss is:

def dpo_loss(pi_logps_win, pi_logps_lose, ref_logps_win, ref_logps_lose, beta=0.1):
    log_ratio_win  = pi_logps_win  - ref_logps_win
    log_ratio_lose = pi_logps_lose - ref_logps_lose
    loss = -F.logsigmoid(beta * (log_ratio_win - log_ratio_lose))
    return loss.mean()

What you gain with DPO:

• No reward model to train or maintain.
• No PPO rollouts — training is a standard supervised loop.
• Same memory footprint as SFT + a frozen reference copy.
• Stable training without KL coefficient tuning.

What you lose:

• No ability to generate new on-policy data during training — the comparison dataset is fixed.
• If the dataset is narrow or low-quality, there is no feedback loop to course-correct.
• Empirically, DPO underperforms online RLHF on complex reasoning tasks (Dubois et al., 2024, AlpacaFarm follow-ups).

DPO opened a family of related methods, each adjusting the implicit reward or the optimization target:

RAFT (Reward rAnked Fine-Tuning) deserves special mention: you run inference to generate $k$ completions per prompt, score them with a reward model, keep the top fraction, and run SFT on the winners. It is online in the sense that it uses the current policy for generation, but avoids PPO entirely. Llama-2-Chat and many subsequent models use variants of this approach.

The practical industry consensus as of 2025: DPO or KTO for initial alignment, followed by online methods (RAFT or iterative DPO with self-generated data) for capability-sensitive tasks like math and coding.

The original Christiano → InstructGPT pipeline is rarely run verbatim today. What has replaced it:

• Synthetic preference data: GPT-4 or Claude is used as a preference judge (RLAIF), reducing dependence on expensive human raters. Constitutional AI (Anthropic, 2022) uses a chain of AI critique-and-revision instead of pairwise human labels.
• Process reward models (PRMs): Instead of scoring final outputs, PRMs score each reasoning step. Critical for math and code, where the final answer can be right for the wrong reasons. OpenAI's Let's Verify Step by Step (2023) demonstrated PRMs outperforming outcome reward models on MATH.
• Iterative / online DPO: Run DPO, generate new completions with the updated policy, collect new preferences (AI or human), repeat. This closes the distributional gap without full PPO infrastructure.
• Scaling preference data: The absolute bottleneck has shifted from architecture to data quality. Nemotron-4-340B (NVIDIA, 2024) used synthetic data pipelines to generate millions of preference pairs.

The core problem — aligning model behavior to human values without overfitting to a proxy reward — remains unsolved. Every method trades some combination of infrastructure complexity, data efficiency, and optimization stability. There is no free lunch.