DAPO: An Open-Source LLM Reinforcement Learning System at Scale

We thank reviewer ZAAH for the positive comment on our paper, and address all concerns and questions below.

Q1: Lack of Evaluation:

• We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.

• We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.

Q2: Not equal comparison w/ DeepSeek-R1-Zero-Qwen-32B

• Our dataset size and batch size are presented in Section 4.1. We acknowledge that the comparison cannot be fairly apple-to-apple because DeepSeek didn't reveal any training details or recipes, while just know them train for 10000 steps from DeepSeek R1's paper. But our end2end system performance beats DeepSeek-R1-Zero-Qwen-32B.

Q3: KL Divergence

• We run comparison experiments on Qwen2.5-Math-7B for 3000 steps. The results show w/o KL achieves better performance. Besides, the w/ KL setting already saturates, yet the w/o KL setting achieves better performance and the performance has not saturated.

Q4: More clarification about $\epsilon_{high}$ and A > 0

• Thank you for your suggestion! We will edit and update our paper to clarify it.

Q5: Consider the number of samples generated when comparing training efficiency in Figure 6

• That's an insightful question! Below, we provide a comparison of training efficiency in terms of both generated samples and end-to-end wall time:
  • Number of Generated Samples: The baseline achieves 42 points after 6080 training steps, while the dynamic sampling approach achieves 43.75 points in just 2320 steps. On average, dynamic sampling generates batches twice the size of the baseline per step. Thus it effectively uses fewer total samples (2320 × 2 = 4640 batches vs. 6080 batches) while achieving superior performance.
  • Wall Time: With 128 GPUs, the baseline takes approximately 4 days to reach 42 points. In contrast, dynamic sampling reaches 43.75 points in about 2 days, significantly improving efficiency.
• We have another insight to share, about this substantial reduction in wall time. Practically, sample generation is often bottlenecked by long-tail samples, and increasing the number of samples introduces less additional overhead. Our generation engine uses the Orca scheduler [3], which replaces finished samples with new samples dynamically. Thus, during the generation of the longest sample, several samples could have been generated. In our setting with 128 GPUs, increasing the RL batch size threefold (from 512 to 1536) resulted in only a 1.5x increase in generation wall time.

Q6: Using a constant as the denominator to normalize loss across batches.

• Empirically, our concurrent work Dr. GRPO [1] proposed to use a constant as the loss denominator and they use the number of maximum tokens. We have tried that and it doesn't work, it even hurt the performance.

• Theoretically, this operation is not grounded and arguable. In the original PPO paper [2], the objective (Equation 1 and 6 of [2]) is expectation over time step t (actions or tokens in the context of LLM). Thus, for each minibatch, we should weigh the loss by dividing it by the number of tokens. This is expectation over time step t.

[1] Understanding R1-Zero-Like Training: A Critical Perspective (https://arxiv.org/pdf/2503.20783)

[2] Proximal Policy Optimization Algorithms (https://arxiv.org/pdf/1707.06347)

[3] Orca: A Distributed Serving System for Transformer-Based Generative Models (https://www.usenix.org/conference/osdi22/presentation/yu)

We thank reviewer Q4hn for the positive comment on our paper, and address all concerns and questions below.

Q1: More Evaluation and Generality

• We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.

• We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.

Q2: Details regarding computational resources

• The computation resource cost is 128 H100 GPUs for one week.

Q3: More discussion of limitations & A single random seed is a limitation.

• Thank you for kindly raising this point. We will expand our discussion on limitations as follows:
  • The outcome reward currently provides minimal guidance on the reasoning process, resulting in inefficient and prolonged reasoning chains. Future research could explore more detailed process-based reward designs to potentially shorten reasoning length.
  • The main experiments were conducted using only a single random seed due to the substantial computational resources required for large-scale LLM RL experiments. We acknowledge this as an important limitation.
  • Regarding safety, the existing verifier-based RL focuses solely on answer correctness without supervising model output safety. Future work could address this by incorporating both outcome verification and a general safety reward model.

Q4: Misinterpretation of checklist question 16:

• Thank you for pointing out this oversight :). We will correct it and notice it next time.

Q5: Could you provide more details on the data source?

• We crawl all questions from the AOPS (https://artofproblemsolving.com/) website and conducted a thorough decontamination of evaluation sets.

Q6: How many experimental runs were conducted to validate the utility of each technique, and what were the specific criteria for terminating a training run?

• Given the high computational cost, we performed only one run per technique for validation. Training runs were terminated upon reaching performance saturation.

Q7: Could you support the claim that removing KL divergence helps?

We run comparison experiments on Qwen2.5-Math-7B for 3000 steps. The results show w/o KL achieves better performance. Besides, the w/ KL setting already saturates, yet the w/o KL setting achieves better performance and the performance has not saturated.

We thank reviewer fAi3 for reviewing our paper, and address all concerns and questions below.

Q1: Evaluation of more datasets and model backbones.

• We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.

• We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.

Q2: Novelty & Clip-higher is hyperparameter tuning

• Effective tricks are important to make RL actually work, especially in large-scale LLM training. Our methods achieve a significant improvement of 20 points on AIME (the core benchmark adopted by OpenAI-O1, DeepSeek R1) and surpass DeepSeek-R1's RL. To date, few other GRPO-based methods can surpass DeepSeek-R1's performance purely using RL in the zero setting. We achieved this, and the effective algorithm details are the keys to making this happen.
• Achieving top-tier performance with a simple method is inherently a kind of innovation. Our conceptually simple method surpasses DeepSeek-level RL performance. Few open research have reached DeepSeek-level RL performance. Not only have we accomplished this, but we have also made all our detailed recipes, code, and data fully open-source, promoting transparency and reproducibility.
• Clip-Higher: it goes beyond "hyperparameter-tuning" and encodes crucial algorithm insights.
  • "Clip-Higher" is a novel approach not previously well-known or utilized by the broader research community. Specifically, few people know that Clip-Higher can effectively balance exploration and exploitation. Besides, Clip-Higher’s approach to independently setting upper and lower clip bounds is novel, where traditionally both clip bounds are identical. Despite its simplicity, it demonstrates exceptional effectiveness. We anticipate that our clear demonstration of its efficacy will inspire its adoption and broader use within the research community.
  • Pure "hyperparameter-tuning" has a vast search space and cannot lead to Clip-Higher. Our identification of Clip-Higher is a targeted, insight-driven adjustment rather than random exploration or just "tuning". RL systems contain hundreds of hyperparameters that can be tuned. To name a few, just for the entropy problem, temperature, temperature scheduler, entropy bonus weights, entropy loss weights, kl loss weights (kl helps entropy), epsilon-greedy sampling ratios, top-k/top-p threshold... These are just "hyperparameter tuning". Clip-Higher is different from these simple tuning and emerges as both novel and remarkably effective, making it an elegant and powerful solution.
• Additionally, a major contribution of our work is clearly identifying critical issues in applying RL to LLMs and providing practical solutions to make the successful application of RL possible. Our focused analysis, e.g. about entropy, will likely inspire further research in this important area.

Q3: Dynamic sampling addresses a known issue but doesn't solve the underlying exploration challenge.

• We argue that few LLM reasoning RL papers pointed this out or addressed this problem. We are the first to provide an effective solution to address this problem, which is shown to speed up training and improve performance.
• The exploration challenge is addressed by Clip-higher. Clip-higher presents a clearly effective solution to this challenge. (shown in Figure 2, it raises entropy and performance). Dynamic sampling is not designed to address the exploration challenge.

Q4: Token-Level PG Loss and Overlong Reward Shaping are not novel approaches to the core RL problems

• We acknowledge that these two techniques are not novel. However, the goal of our paper is not to propose an "entirely new method for the core RL". Instead, we aim to demonstrate how RL can be successfully applied to significantly enhance LLM reasoning practically, surpassing state-of-the-art DeepSeek-R1 level RL performance. Through our experiments, we identify them as crucial components for achieving this level of performance and transparently share these findings with the research community. Note that the community continues to face challenges in replicating DeepSeek-R1-level RL results, underscoring the practical value of our contributions.

Q5: Information of computational resources required for training

• The computation resource cost is 128 H100 GPUs for one week.

Q6: Correlation between entropy collapse and model performance:

• We show the correlation in Figure 2. We can clearly see that for the baseline trial, the entropy collapses to its minimum at around 750 steps, and the peak performance is achieved also at round 750 steps.

Q7: Epsilon lower bound's importance is emphasized but no experimental analysis

• Our proposed method is Clip-Higher and is about the epsilon upper bound. The epsilon lower bound is not our focus and we just keep it as it is.

Q8: In Figure 6, the proposed method and baseline are trained for different number of steps, where the baseline is converged.

• Our method surpasses the converged baseline using significantly fewer training steps, clearly demonstrating its superior efficiency and performance.

Thank the authors for the response. After carefully considering the authors' rebuttal, I remain unconvinced and have summarized below the specific points I still find unclear. If I’ve misunderstood anything or any of my objections are misplaced, I would greatly appreciate your clarification. The remaining concerns need to be addressed as follows.

• The authors present dynamic filtering as a contribution, but this is essentially identical to the online filtering approach already proposed in Prime[1].
• Despite the authors' defense, the Clip-Higher approach appears to be primarily a hyperparameter adjustment rather than a methodological innovation. The authors need to better articulate why this isn't simply parameter tuning.
• The reported GRPO baseline with 32B models performs significantly worse than expected. For example, your baseline scores 62 on MATH500 and 30 on Olympiad, while Qwen-2.5-7B-base models using GRPO in the VERL[2] framework in OpenR1-Math-220K dataset[3] achieved 80 on MATH500 and 41 on Olympiad with just 7B models. This suggests implementation issues that may artificially inflate the proposed method's improvements.
• Results focus too heavily on AIME. Please provide comprehensive results across all benchmarks for each model configuration, including average performance metrics.
• Based on the reward curves, the experiments appear to have converged early. Why not terminate training earlier and run multiple trials to obtain more statistically meaningful results?

These issues need to be adequately addressed to validate the paper's claims and contributions.

[1] Process reinforcement through implicit rewards.

[2] https://github.com/volcengine/verl

[3] https://huggingface.co/datasets/open-r1/OpenR1-Math-220k

Q1:

• The two techniques are fundamentally different. Online filtering doesn't improve performance because they "filter" samples, causing the effective batch size to shrink during training—this destabilizes gradients and brings no clear performance benefit. However, in dynamic sampling, the batch size is constant and yields steadier gradient estimates and more reliable learning. This is a significant difference and technical improvement. And performance improvement shows that.
• Prime is an arxiv paper and actually our contemporary work. We develop this method independently.

Q2:

• Nobody has previously shown that this hyperparameter tuning can balance exploration and exploitation, an insight that represents an innovation.
• No prior work has shown that this tuning delivers such a substantial performance gain, highlighting Clip-Higher as the true innovation.

Q3:

• The dataset is significantly different and the comparison does't hold. They use a 220K dataset while we use a 17K dataset.
• Our DAPO-32B results beat DeepSeek's RL results. This is a fact and undebateble. We will release the checkpoint for reproduction.

Q4:

• We provide more performance evaluation across different datasets and model architectures in Q1 of the initial rebuttal. We think this review just ignores the key and obvious facts presented in our initial rebuttal.

Q5:

• The evaluation score keeps growing (Figure 1) and the reward score almost keeps growing (Figure 7 b). No reason to terminate the training earlier. We think this comment just ignores the key and obvious facts presented in our paper.

Thank you for taking the time to review our work. However, several of your comments appear to disregard key evidence and facts presented in the paper and in our initial rebuttal, which has hindered productive dialogue. At the same time, all other reviewers have at least read through the contents of our paper, and offered positive feedback. We will discuss this concern with the Area Chair to identify the potentially irresponsible review.

We thank reviewer ADmi for the positive comment on our paper, and address all concerns and questions below.

Q1: Lack of novelty

• Effective tricks are important to make RL actually work, especially in large-scale LLM training. Our methods achieve a significant improvement of 20 points on AIME (the core benchmark adopted by OpenAI-O1, DeepSeek R1) and surpass DeepSeek-R1's RL. To date, few other GRPO-based methods can surpass DeepSeek-R1's performance purely using RL in the zero setting. We achieved this, and the effective algorithm details are the keys to making this happen.
• Achieving top-tier performance with a simple method is inherently a kind of innovation. Our conceptually simple method surpasses DeepSeek-level RL performance. Few open research have reached DeepSeek-level RL performance. Not only have we accomplished this, but we have also made all our detailed recipes, code, and data fully open-source, promoting transparency and reproducibility.
• Clip-Higher: it goes beyond "hyperparameter-tuning" and encodes crucial algorithm insights.
  • "Clip-Higher" is a novel approach not previously well-known or utilized by the broader research community. Specifically, few people know that Clip-Higher can effectively balance exploration and exploitation. Besides, Clip-Higher’s approach to independently setting upper and lower clip bounds is novel, where traditionally both clip bounds are identical. Despite its simplicity, it demonstrates exceptional effectiveness. We anticipate that our clear demonstration of its efficacy will inspire its adoption and broader use within the research community.
  • Pure "hyperparameter-tuning" has a vast search space and cannot lead to Clip-Higher. Our identification of Clip-Higher is a targeted, insight-driven adjustment rather than random exploration or just "tuning". RL systems contain hundreds of hyperparameters that can be tuned. To name a few, just for the entropy problem, temperature, temperature scheduler, entropy bonus weights, entropy loss weights, kl loss weights (kl helps entropy), epsilon-greedy sampling ratios, top-k/top-p threshold... These are just "hyperparameter tuning". Clip-Higher is different from these simple tuning and emerges as both novel and remarkably effective, making it an elegant and powerful solution.
• Additionally, a major contribution of our work is clearly identifying critical issues in applying RL to LLMs and providing practical solutions to make the successful application of RL possible. Our focused analysis, e.g. about entropy, will likely inspire further research in this important area.

Q2: Lack of evaluation and generality

• We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.

• We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.

Q3: Clip-Higher is purely hyperparameter tuning on math tasks and perhaps not generalizable.

• Our analysis is about the entropy dynamics of the RL process and no specific data type assumption is made. Principlely, this is a general RL trick and can generalize. We conduct a quick experiment and show that on coding tasks, clip-higher is also effective. Besides, we argue that clip-higher together with its superior performance encode algorithmic insights and the related details can be found in Q1.

Q4: How token-level loss tackle both scenarios.

• The RL training objective has two types of effects: it either encourages (increases likelihood) or punishes (decreases likelihood). Positive samples are encouraged, while negative samples are punished. When applying token-level loss, longer samples naturally receive greater weight; thus, positive samples are encouraged more strongly, and negative samples face increased punishment, significantly reducing their likelihood. This means token-level loss can naturally tackle both scenarios.

Q5: Why is a punitive overlong reward by default? If one desires longer sequences, wouldn't increase maximum length just solve the problem?

• Regarding your second point, autoregressive LLM generation is typically the primary time bottleneck of the whole system. Therefore, in practice, it is necessary to impose a maximum generation length during the rollout phase to ensure efficiency.
• The default punitive strategy for overly long responses is common practice in verifier-based LLM RL, exemplified by DeepSeek R1. Such methods use accuracy rewards, giving positive feedback only if the response is correct and is validated via robust, rule-based verification (see Sec 2.2.2 of DeepSeek R1). Overlong samples cannot pass such rule-based verification, because they cannot give a correct answer.