RLVR and GRPO: Training LLMs with Verifiable Rewards

Standard RLHF trains a reward model (RM) on human preference pairs, then uses PPO to push the policy toward higher RM scores. This works for open-ended generation — but the RM is a neural network that can be gamed. Give it enough optimization pressure and the policy learns to produce outputs the RM scores highly rather than outputs humans actually prefer. This is reward hacking.

For math and code, there is a better alternative: verifiable rewards. A Python unit test either passes or fails. A formal proof either type-checks or it doesn't. These are ground-truth binary signals you can compute in milliseconds with zero human labor and zero chance of gaming — the test suite doesn't care how confident the model sounds.

Reinforcement Learning with Verifiable Rewards (RLVR) replaces the learned RM with a deterministic checker. The reward function is a program, not a model. This unlocks stable training at scale and sidesteps the reward-hacking failure mode entirely.

A reward is verifiable if a program can decide correctness without human judgment. Common examples:

• Math answers — compare the model's final answer to a known solution after symbolic normalization (e.g., sympy.simplify(model_ans - gold_ans) == 0).
• Unit tests — run the model's code against a hidden test suite; reward = fraction of tests that pass.
• Lean / Coq proofs — pipe the model's proof term into the kernel; reward = 1 if the kernel accepts it, 0 otherwise.
• SQL queries — execute against a reference database and compare result sets.
• Competitive programming — compare output on hidden judge inputs.

Non-verifiable things: essay quality, stylistic preference, harmlessness judgments. Those still need a learned RM or human raters.

The key insight from DeepSeekMath (Shao et al., 2024) and DeepSeek-R1 (DeepSeek-AI, 2025) is that math competitions and coding benchmarks are almost entirely verifiable, making them ideal training domains for RL without a referee.

PPO (Proximal Policy Optimization) is the RL algorithm behind early RLHF work (InstructGPT, Claude 1). It needs four models in memory simultaneously:

1. Policy $\pi_\theta$ — the model being trained.
2. Reference policy $\pi_{\text{ref}}$ — a frozen copy used for KL regularization.
3. Reward model — a separate trained classifier.
4. Value function (critic) — estimates $V(s_t)$ to compute advantages.

For a 7B model, each copy is ~14 GB in bf16. Running all four plus optimizer states exceeds 200 GB easily. The critic also has to be trained jointly, adding gradient computation and instability.

PPO computes the advantage as $A_t = r_t + \gamma V(s_{t+1}) - V(s_t)$, which requires the critic to be accurate. If the critic lags the policy (a chronic problem early in training), advantage estimates are noisy and learning stalls.

GRPO eliminates the critic entirely — a significant engineering win when you're running thousands of GPU-hours of RL.

Group Relative Policy Optimization (introduced in DeepSeekMath, Shao et al. 2024) replaces the critic with group statistics. For each question $q$, sample a group of $G$ responses $\{o_1, \dots, o_G\}$ from the current policy. Compute rewards $\{r_1, \dots, r_G\}$ via the verifier. The advantage for response $i$ is:

$A_i = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$

This is just z-score normalization within the group. No value network, no bootstrapping, no temporal credit assignment — the group itself provides the baseline.

The policy gradient objective clips the probability ratio just like PPO:

$\mathcal{L}_{\text{GRPO}} = -\mathbb{E}\left[\min\!\left(\frac{\pi_\theta(o_i|q)}{\pi_{\text{old}}(o_i|q)} A_i,\ \text{clip}\!\left(\frac{\pi_\theta}{\pi_{\text{old}}}, 1\!\pm\!\epsilon\right) A_i\right)\right] + \beta \, D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$

$G$ is typically 8–16. Larger groups give better advantage estimates but cost proportionally more inference.

Here is a minimal GRPO reward function for math, using answer extraction and symbolic comparison:

import sympy
import re

def math_reward(response: str, gold_answer: str) -> float:
    """Return 1.0 if response contains the correct answer, else 0.0."""
    # Extract the last boxed or '=' expression
    match = re.search(r"\\boxed\{([^}]+)\}", response)
    if not match:
        return 0.0
    try:
        expr = sympy.sympify(match.group(1)) - sympy.sympify(gold_answer)
        return 1.0 if sympy.simplify(expr) == 0 else 0.0
    except Exception:
        return 0.0

def code_reward(response: str, test_cases: list[tuple]) -> float:
    """Fraction of unit tests the model's code passes."""
    code_match = re.search(r"```python\n(.*?)```", response, re.DOTALL)
    if not code_match:
        return 0.0
    namespace = {}
    try:
        exec(code_match.group(1), namespace)
    except Exception:
        return 0.0
    passed = sum(namespace.get("solution")(*inp) == out for inp, out in test_cases)
    return passed / len(test_cases)

The training loop calls these inside the rollout phase, before computing advantages. No gradient flows through the verifier — it's a pure black-box oracle.

The group baseline is the key. When all $G$ responses in a group are wrong (reward = 0), the normalized advantages are all 0 and no gradient flows. When all are correct, same thing. Gradient signal fires only when the group has variance — some responses right, some wrong. This is exactly when the model is at the boundary of a skill it's acquiring.

This property also provides automatic curriculum weighting: easy problems (policy already solves them >90% of the time) contribute almost no gradient. Hard problems (policy never solves them) also contribute nothing. The signal concentrates on problems at the current capability frontier — similar in spirit to self-paced learning, but emergent from the math.

Heads up: if $G$ is too small (say 2), variance estimates are unreliable and training is noisy. If $G$ is too large, inference cost dominates. $G = 8$ is a common sweet spot for 7B-class models.

The KL penalty $\beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ prevents the policy from collapsing to degenerate outputs that happen to pass the verifier (e.g., outputting the answer format with a random number that guesses right).

DeepSeekMath (Shao et al., 2024) was the first paper to introduce GRPO and demonstrate it at scale. Starting from a 7B math-specialized base, they applied GRPO with binary math-answer rewards on the MATH and GSM8K benchmarks.

Key results from the paper:

GRPO matched PPO's accuracy while being cheaper to run. More importantly, it scaled better — PPO's critic instability grew with model size, while GRPO's group baseline stayed well-conditioned.

The paper also showed that process reward models (PRMs, which score reasoning steps) could be used as GRPO rewards, not just outcome rewards. This set the stage for longer chain-of-thought training.

DeepSeek-R1 (DeepSeek-AI, January 2025) applied GRPO to a 671B MoE model (DeepSeek-V3 base) with a much longer horizon: the model generates extended reasoning chains before producing a final answer. Rewards remain verifiable — math answer correctness and code test pass rates.

Critical design choices in R1:

• No SFT warmup for R1-Zero — they ran GRPO directly on the base model. The model spontaneously learned to produce long reasoning chains because longer chains improved answer accuracy and thus rewards.
• Format reward — a small auxiliary reward for following <think>...</think><answer>...</answer> structure, preventing degenerate formatting.
• Cold start SFT for R1 — to avoid early instability in the pure-RL approach, the final R1 recipe uses a small set of long-CoT demonstrations before GRPO kicks in.

The result: DeepSeek-R1 matched o1 on AIME 2024 (79.8% vs 79.2%) and surpassed it on several coding benchmarks — trained primarily with programmable verifiers, not human raters.

One of the most striking findings from R1-Zero (the pure GRPO, no-SFT variant) is that extended reasoning behaviors emerged without explicit supervision. The model learned to:

• Backtrack and try alternative approaches mid-solution.
• Verify its own intermediate results.
• Increase response length on harder problems dynamically.

DeepSeek called this an "aha moment" — the model discovered that allocating more computation (more tokens) to hard problems improved verifier scores, so gradient descent reinforced longer thinking on hard inputs.

This mirrors what scaling test-time compute research (Snell et al., 2024; OpenAI o1 report) found: for reasoning tasks, more tokens at inference beats a bigger model. GRPO operationalizes this by making the reward independent of length — only correctness matters, so the model finds its own optimal length per problem.

The base model already "knows" math from pretraining. GRPO doesn't teach math — it teaches the model when and how long to think before committing to an answer.

Reward hacking is not eliminated, just harder. A model can still find degenerate strategies: outputting a huge range of candidate answers hoping one matches, or exploiting numerical precision differences in the verifier. Mitigation: diverse test cases, answer normalization, and KL penalty.

Cold-start instability. Running GRPO from scratch on a base model (no SFT) produces very high variance early — most groups have 0 or 100% success rate, killing the gradient signal. Standard fix: a small SFT phase on curated CoT examples first.

Code execution sandboxing. When the reward is unit-test pass rate, you're executing model-generated code. Use isolated containers (Docker, Firecracker) with CPU/memory limits. A model can trivially cheat by reading the test file or calling sys.exit(0) to pass all tests.

# Minimal sandboxed execution with resource limits
docker run --rm --network none --memory 256m --cpus 0.5 \
  --read-only python:3.12-slim \
  python -c "$MODEL_CODE"

Group size vs. batch throughput tradeoff. With $G=8$ and batch size 64, you're generating $64 \times 8 = 512$ rollouts per step. Profile inference time carefully — it dominates training time in GRPO.

GRPO isn't the only way to avoid a critic. A comparison of current approaches:

RLOO (Reinforce Leave-One-Out) is mathematically close to GRPO — the leave-one-out baseline $\frac{1}{G-1}\sum_{j \neq i} r_j$ is an unbiased estimator of $V$, while GRPO's mean is slightly biased but simpler to implement. In practice they perform similarly.

DPO is often preferred for preference alignment but struggles when you want to reward correct outputs rather than preferred ones — it doesn't naturally handle binary verifiable signals without synthetic preference pairs.