DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization

We thank the reviewer for dedicating the time to providing helpful feedback on our paper.

Q1: The framework only considers binary rewards (though the authors acknowledge this). It would be more compelling if it were shown to work with non-binary reward settings as well.

A: We agree that it will strengthen the work. However, the focus on the binary reward setting helps us to better illustrate the limitation of GRPO through theoretical analysis. We believe the improvement brought by DisCO is non-trivial and expect to extend its strength to non-binary rewards by considering other discriminative objectives, e.g., ranking losses.

Additionally, to make our work more compelling, we further validated our method on a model from another LLM-family, i.e., DeepSeek-R1-Distill-Llama-8B, training them for 1000 steps. The results are summarized below. We observed that our proposed DisCO methods still outperform other baselines by a large margin, demonstrating the generalizability of the proposed method to other models.

Q2: It would be helpful to understand how DisCO's performance varies with different response lengths. For example, What happens if we increase the response length (e.g., 24k or 32k)?

A: We thank the reviewer for the constructive comment. We have conducted an experiment using 16k response length for training and inference on 1.5B models. The results shown below demonstrate that increasing response length brings additional improvements, compared with 8k length for training and inference.

We thank the reviewer for providing helpful comments on improving our paper. Below we would like to answer raised questions.

Q1: In hyperparameter selection, have you accounted for the impact of different hyperparameter combinations on model generalization capabilities?

A: We tuned two hyper-parameters for DisCO, i.e., the learning rate and the $\tau$. The process of hyperparameter selection is the same as model selection, which accounts for the generalization performance. We report the performance of the best hyper-parameters in the tuning range that yields the best average test performance. We did the same model selection for all baselines. We will add more details in the revision.

Q2: Mentioning the merits and applicability of GRPO in specific scenarios.

A: We thank the reviewer for the constructive comment. In the opening paragraph of the Introduction, we highlighted the importance of GRPO in achieving performance comparable to advanced proprietary models across many reasoning benchmarks. Additionally, in the second paragraph of the Related Work section, we discussed key contributions of GRPO and its variants to provide readers with a foundational understanding of each approach.

We agree with the reviewer that each method has its own strengths. In Section 7, we acknowledged the limitation of our work in focusing on binary rewards, whereas GRPO and its variants can naturally handle non-binary reward signals without modification. We will make this clearer in the revision.

Q3: Could you quantify the individual contribution rates of each improvement in DisCO through ablation studies in detail?

A: Thank you for the helpful suggestion. The proposed method incorporates three key design choices: (1) removing question-level weight bias, (2) using a non-clipping scoring function, and (3) applying a KL-divergence constraint.

The effectiveness of (1) has been demonstrated in the comparison between GRPO and GRPO$_\text{rw}$, as shown in Fig. 1 (c, d). Additionally, we conducted new experiments on training a 1.5B model, comparing DisCO with its variant that includes question-level weight bias while keeping all other components unchanged. The results are presented in the table below.

The benefit of (2) is validated in Figure 3 (second from the left), which shows that using the non-clipping scoring functions yields significant improvements over its clipping-based counterpart, when all other components are the same as DisCO.

To evaluate the impact of (3), we conducted another experiment on the 1.5B model, comparing DisCO with a variant that replaces the KL constraint with KL-divergence regularization. The results are also summarized in the table below.

We can see that each component contributes to the improvements, where the use of non-clipping scoring function contributes the most. We will incorporate the discussion into our ablation studies.

Q4: Will the DisCO algorithm pose computational efficiency challenges as models scale further?

A: The DisCO algorithm does not introduce any extra computational overhead, compared with GRPO and its variants. They have similar computational costs.

We thank the reviewer for providing constructive comments. Below we would like to answer the raised questions.

Q1: On the math symbols that do not have explanation. For example, function $g(o,q)$ is a one time symbol that can be essentially replaced by $f(x,y)$, $f^+$ and $f^-$ can be omitted. There are also many abbreviations that show up without explanation, such as VPG, GRPO_RW in the equations.

A: We apologize for the confusion!

• $g(o,q)$ in line 134 is any function, introduced for a statement of a general fact in equation (1), which is used in the proof of Proposition 1. We will move this to the appendix in the revision.

• $f^+$ and $f^-$ in Proposition 1 are assumed to be non-decreasing functions for decomposing $f(x, y)$. Their specific forms for GRPO are actually the form in $s_{\theta}^+(o, q)$ and $s_{\theta}^-(o, q)$ in Proposition 1, i.e., $f^+(x, 1)=\min(x, 1+\epsilon)$ and $f^-(x,1)=\min(x, 1-\epsilon)$. For other methods, their specific forms can be found in Table 4. We will make it clearer in the revision.

• VPG stands for Vanilla Policy Gradient method, and GRPO_RW stands for a GRPO version with the question-level weight bias removed (line 183). We will revise the paper to make the term clearer.

Q2: The claimed motivation that previous methods have difficulty bias seems to be a bit empirical and does not have theoretical support.

A: We politely disagree. Our finding regarding the difficulty bias of GRPO is through a theoretical analysis of GRPO's objective by connecting it with a discriminative objective in Proposition 1. The empirical studies in Figure 1 are used for further supporting our claim.

Q3: In Table 3, when the initial policy is strong, the improvements of DisCO is small, and the baseline methods such as GRPO and DAPO even decrease the performance of initial policy, is there any explanation?

A: There is a misunderstanding in the interpretation of comparison with the initial policy in Table 3. It is important to compare their performance against the second entry in Table 3: DS-Distill-Qwen-7B (32k+/8k), which represents the initial policy evaluated with the same 8k response length used for GRPO, DAPO and DisCO. The first entry, DS-Distill-Qwen-7B (32k+/32k), uses a much longer 32k response length at inference and is included as a reference to highlight DisCO’s efficiency under smaller generation budgets (8k during both training and inference).

In fact, both GRPO and DAPO do improve upon the initial policy when evaluated under the same response length constraint—raising performance from 0.513 to 0.592 (GRPO) and 0.570 (DAPO), respectively. Also, the improvements of DisCO are still significant, from 0.513 to 0.627 or 0.625.

Please also note that while the performance of GRPO and DAPO has already saturated, DisCO continues to exhibit an upward trend as the number of training steps increases (Figure 2c).

Q4: The equation 9 seems to have very similar formulation as Direct Policy Optimization. Is there any essential connection and difference between the proposed DisCO and online DPO?

A: Indeed DPO's objective can be regarded as a discriminative objective and can be recovered in (9) by choosing $s_{\theta}(o,q) = \beta\log\frac{\pi_{\theta}(o|q)}{\pi_{ref}(o|q)}$, $\ell(s) = \log(1+\exp(-s))$, and removing the squared hinge penalty function. Nevertheless, the objective in (10) for DisCO is different from DPO, as it has an effect on weighting more on hard negative rollouts.

Q5: Is DisCO still effective and better than previous methods on other datasets like those provided in DAPO or OREAL papers?

A: We thank the reviewer for the constructive comment. We conducted additional experiments on DAPO-Math-17k dataset with 1.5B models, training them for 1400 steps. The results are summarized below. We can observe that DisCO methods achieve slightly better performance than those trained on DeepScaleR-Preview-Dataset, consistently outperforming GRPO and other baselines.

We thank the reviewer for providing a comprehensive review. Below we would like to address raised concerns.

Q1: In the experiments, it might be good to more closely decompose the effect of each design choice on the performance.

A: We thank the reviewer for the constructive suggestion! The proposed method incorporates three key design choices: (1) removing question-level weight bias, (2) using a non-clipping scoring function, and (3) applying a KL-divergence constraint.

The effectiveness of (1) has been demonstrated in the comparison between GRPO and GRPO$_\text{rw}$, as shown in Fig. 1 (c, d). Additionally, we conducted new experiments on training a 1.5B model, comparing DisCO with it variant that includes question-level weight bias while keeping all other components unchanged. The results are presented in the table below.

The benefit of (2) is validated in Figure 3 (second from the left), which shows that using the non-clipping scoring functions yields significant improvements over its clipping-based counterpart, when all other components are the same.

To evaluate the impact of (3), we conducted another experiment on the 1.5B model, comparing DisCO with a variant that replaces the KL constraint with KL-divergence regularization. The results are also summarized in the table below.

We can see that each component contributes to the improvements, where the use of non-clipping scoring function contributes the most. We will incorporate the discussion into our revision.

Q2: The idea of upsampling examples with the highest and lowest rewards is also explored in concurrent work [r1].

A: Thank you for pointing out the interesting work. While [r1] introduces max variance down-sampling to train on only an informative subset, we'd like to clarify that our method does not apply up-sampling or down-sampling to rollouts. Instead, we proposed a discriminative objective using all generated rollouts for more effective learning than GRPO. It would be interesting to integrate [r1]'s sampling approach into our framework.

Q3: Based on the experimental section, it looks like hyperparameters are manually tuned for their algorithm, but baseline hyperparameters are reported numbers adapted from previous papers. It's not clear whether these hyperparameters are optimal for their setup.

A: Thank you for the thoughtful comment! We would like to provide further clarification on our hyperparameter tuning process.

First, we tune the learning rate for all methods from the set $\{1\text{e}{-6}, 2\text{e}{-6}\}$ (see lines 310–311).

Second, for our proposed DisCO methods, we tune only one additional parameter, $\tau$, over a small range of three values per variant. The other hyperparameters, $\beta$ and $\delta$, are selected based on heuristics and are not tuned (lines 318-319). Our ablation study (Figure 3, right) shows that the performance is not sensitive to the choice of $\tau$.

Third, for the baseline methods GRPO, GRPO-ER, TRPA, we follow the hyper-parameter settings used in prior works [r2, r3, r4] because they have used exactly the same model and dataset as our experiments. Dr. GRPO does not introduce any additional hyperparameters beyond the learning rate. For DAPO, we used $\epsilon_{\text{high}} = 0.28$, as in the original paper. While we acknowledge that some studies may use a different value, we tested an alternative setting $\epsilon_{\text{high}} = 0.24$ to address this concern. The result, shown in the table below, shows that $\epsilon_{\text{high}} = 0.24$ performs even worse than $\epsilon_{\text{high}} = 0.28$.

In summary, the improved performance of DisCO does not stem from extensive or unfair hyperparameter tuning.

Q4: On the interplay between the expected reward and entropy collapse in your algorithm. Are there less datapoints would have extremely low or high rewards in DisCO? Would Disco focus more on these outlier examples?

A: We would like to clarify a misunderstanding regarding DisCO. We agree that entropy collapse did contribute to saturation of expected rewards for GRPO, GRPO-ER and Dr. GRPO as observed in Fig. 2. Regarding DisCO,

• (1) the generation entropy remains stable while the expected reward keeps increasing (see Fig. 2);
• (2) We verified the reward distribution for questions in our method averaged over the first 200 steps. As shown in the following table, it does not exhibit fewer datapoints with extremely low or high rewards. It is similar to that in GRPO (Fig. 1, second from the left).
• (3) It is not true that DisCO focuses more on the examples with extremely low or high rewards. Indeed, DisCO is motivated by giving the same weight to all questions regardless of their expected rewards. Our objective in (10) has an effect of weighting negative rollouts for each question, focusing more on hard negative rollouts.

Q5: If you remove the clipping for GRPO + use the same squared-hinged KL term, does the performance improve?

A: We have experimented with the suggested variant of GRPO, keeping its advantage function, but replacing its clipping-based scoring function with non-clipping based L-ratio, and using the squared-hinge KL penalty instead of its original KL regularization. This indeed is the variant of the second row (right) in the first Table of this rebuttal. We summarize the results below, from which we can see that its average performance 0.511 is better than GRPO (0.457) but still worse than DisCO (0.524).

Q6: Based on the insights provided, are there any particular datasets you would foresee Disco to be particularly effective or suboptimal in comparison to GRPO? e.g. perhaps you can try comparing these algorithms on datasets with a small number of very easy and hard examples.

A: The DeepScaleR-Preview-Dataset used in our experiments includes a wide range of questions, spanning from easy to challenging for the base models. We believe that the improvements achieved by DisCO are fundamental, rather than relying on specific properties of the dataset. To further support this claim, we conducted additional experiments on the DAPO-Math-17K dataset [r5] using 1.5B models, training them for 1400 steps. As shown below, DisCO still consistently outperforms GRPO and other baselines.

[r1] Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning

[r2] An empirical study on eliciting and improving r1-like reasoning models.

[r3] Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl.

[r4] Trust region preference approximation: A simple and stable reinforcement learning algorithm for llm reasoning.

[r5] DAPO: An Open-Source LLM Reinforcement Learning System at Scale.