The Open Recipe for Reasoning RL

DeepSeek-R1 was a watershed, but it was also a black box: a landmark result with no public training recipe. The open-source community spent early 2025 trying to reproduce "RL that makes models reason" from a paper that described the destination but not the road. DAPO — from ByteDance's Seed team — is the most instructive of those reproduction attempts, because it succeeded and documented exactly which details mattered. It's a paper about the gap between "we did RL" and "here's how."

Reproducing the miracle, openly

The setup is deliberately modest: Qwen2.5-32B, a 32-billion-parameter open model, trained with a GRPO-style algorithm on math reasoning problems. The headline result: 50 points on AIME 2024, surpassing the 47 points of DeepSeek's own R1-Zero-Qwen-32B distilled baseline — and doing it with only half the training steps. Naive GRPO on the same setup managed just 30 points, which tells you something important: the algorithm name on the tin matters far less than the implementation details around it.

Those details are the paper's real contribution. The authors frame large-scale RL for reasoning as suffering from two core pathologies — entropy collapse and reward noise — plus two efficiency problems, and propose one fix for each:

• Clip-Higher. Standard PPO/GRPO clips the policy ratio symmetrically, which the authors show steadily crushes exploration: entropy collapses, the policy stops trying new things. Their fix decouples the clip bounds — keep the lower bound tight, raise the upper bound — so the policy can still move toward surprisingly good actions while being constrained from abandoning old ones. Exploration survives; stability survives.
• Dynamic sampling. Remember GRPO's failure mode: groups where every output is right or every output is wrong produce zero gradient. The fix is blunt — oversample prompts, then throw away the uninformative groups. Wasteful of generation compute, but it concentrates every gradient step on groups that actually teach something.
• Token-level policy gradient loss. Instead of averaging the loss per sequence, average per token, so long reasoning traces don't get downweighted relative to short ones. A small change with an obvious rationale: in long-CoT training, the long traces are where the learning lives.
• Overlong reward shaping. Penalize generations that hit the length limit without finishing, so the model doesn't learn that rambling forever is a viable strategy. This fights the "overthinking" pathology at its root.

Notably, the authors drop the KL penalty entirely for long-CoT training, arguing the policy diverges so far from initialization that the constraint is meaningless — a small heresy against RLHF orthodoxy that turned out to work. The paper also documents its reward design with unusual candor: a rule-based outcome reward (+1 for matching ground truth, −1 otherwise), no shaping, no learned components. And it reports its baselines honestly — naive GRPO at 30 points is the ablation that justifies every added technique. In a literature full of papers that bury their controls, DAPO reads like a lab notebook: here's what we tried, here's what moved the number, here's what didn't.

Why this paper matters more than its benchmark

Fifty points on AIME is a good result, not a historic one. What makes DAPO valuable is its genre: it's a systems paper about RL training, written for the community that needs to replicate these runs. The R1 paper told you RL could discover reasoning; DAPO tells you what breaks when you try — entropy collapse at step N, dead gradients from homogeneous groups, length pathologies — and gives you the specific knobs that fix each one.

The paper is also refreshingly honest about what it didn't need. No process reward model. No Monte Carlo tree search. No learned value function. Just outcome rewards — right or wrong, verified by rules — plus four engineering fixes around a GRPO core. That minimalism is itself a finding: it suggests the "secret sauce" of reasoning-RL is less about exotic algorithms than about unglamorous training hygiene at scale.

There's a broader point here about how AI progress actually diffuses. The R1 paper moved the field's beliefs; DAPO moved the field's capabilities, by turning a existence proof into a recipe anyone with enough GPUs could follow. Open-source RL for reasoning exists as a practical discipline largely because papers like this one wrote down the details the landmark papers omitted.

Open questions it leaves

DAPO's fixes are empirical, not principled — Clip-Higher works, but nobody has a satisfying theory of the optimal asymmetry. Dynamic sampling is computationally wasteful by construction; a smarter curriculum could get the same signal cheaper. And the whole edifice still rests on verifiable rewards: everything here is math, where correctness is checkable. Whether these techniques transfer to domains with softer rewards — code review, writing, agentic tasks — is the question the next generation of papers is trying to answer.

DAPO's thesis: the difference between RL that works and RL that doesn't is rarely the algorithm — it's a handful of unglamorous implementation choices, and those choices are the actual technology.

Where it sits in the arc

DAPO closes the loop that R1 opened. R1 proved pure RL could incentivize reasoning; Kimi k1.5 showed an independent lab could reach the same summit by a different route; DAPO showed the rest of us how to climb. Together they mark 2025 as the year reasoning-RL went from industrial secret to public infrastructure — and the current wave of RL-for-agents work is building directly on the training hygiene this paper documented.

Paper: DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Yu et al., 2025.