We sincerely thank the reviewers for their insightful feedback. We are grateful for their recognition of our approach's clarity and simplicity, its grounding in both empirical and theoretical motivations, and the utility of our ablation study. We address your concerns below.

Responses to Weaknesses 1

Thank you reviewer for this insightful and crucial suggestion. To address this concern, in addition to the initial experiments, we conducted extensive new experiments, running our approach (ReDit) and all baselines on 5 different random seeds (42, 123, 888, 2025, 9999), and None means no random seed is set. As you suggested, in the revised manuscript, all tables will be updated with the mean and variance, and the accuracy plots will be updated with error bars. We alse show below detailed results for our ReDit (Gauss 0.05) approach on all seeds and baselines.

Setting: gsm8k, gauss,0.05

Setting: math, gauss,0.05

Setting: Geo3k, gauss,0.05

As these tables show, the standard deviation across different runs is consistently very low (typically ≤ 0.1). This provides strong empirical evidence that ReDit's performance gains are stable, consistent, and not an artifact of a specific random initialization. These new results confirm that ReDit robustly outperforms the baselines.

Responses to Weaknesses 2

We sincerely thank the reviewer for this detailed and insightful critique of our theoretical analysis in Proposition 3. Your careful examination has helped us identify an error in our original proof and allows us to clarify our argument. We agree that our initial proposition had issues and will be revised. We address the upper and lower bounds separately below.

On the Upper Bound (We agree with the reviewer)

Regarding the upper bound, you are entirely correct. We apologize for this incorrect derivation. We erroneously extrapolated the result from [37], which specifies that there exist certain inaccurate reward models that lead to faster optimization. We cannot prove that our specific noisy reward model, $r_{RM} = r + \epsilon$, satisfies this property in the general case. Action: We will remove the claim regarding the upper bound from Proposition 3 in the final version of the manuscript.

On the Lower Bound (A clarification on the objective function)

Regarding the lower bound, we would like to point out that we believe there is a misunderstanding in the understanding of the objective function in the theoretical framework of [37] and appreciate the opportunity to clarify. We apologize for not emphasizing in the paper that in the theoretical analysis (whether Proposition 1, Proposition 2 or Proposition 3), we are consistent with the assumptions in [37] and all theoretical analysis considers using exact gradients to optimize the expected objective function. The impact of ReDit on the gradient is detailed in our Propositions 1 and 2. As shown above, ReDit does not change the expectation of the gradient (i.e., the estimator is unbiased), but it significantly affects the variance of the gradient, which is directly related to the noise variance $\sigma^2$. Based on this theoretical setting, let us revisit the objective function being optimized. Following the approach of [37], the policy $\pi_\theta$ is updated to maximize the expected reward given the reward model $r_{RM}$:

$\tilde{J}(\pi\_\theta) = \mathbb{E}\_{q \sim p\_Q} \left[ \mathbb{E}\_{o \sim \pi\_\theta(\cdot|q)}r\_{RM}(q, o) \right]$

In our approach, the reward model is the noisy reward, where $r_{RM} = r + \epsilon$. This corresponds to $\tilde{R}$ in Equation (5) of our paper. The key lies in the application of Theorem 1 in [37]. Theorem 1 gives a lower bound on the optimization result of $\tilde{J}(\pi_\theta)$ based on the properties of the reward model $r_{RM}$ and its relationship to the true reward $r$. In our application, $r_{RM}$ is the reward model used by Redit, and the deterministic 0-1 reward $r$ is its true reward. Therefore, the lower bound correctly depends on quantities derived from $r_{RM}$, such as its variance $\text{Var}(r_{RM}) = \text{Var}(r) + \sigma^2$. Therefore, we maintain that the lower bound in Proposition 3 is valid under this interpretation, which is consistent with the framework of [37]. Finally, thank you for your insightful observation: in practice, ReDit also smooths batch gradient estimates, thereby improving training efficiency. This is consistent with our empirical findings.

Reference:

[37] Noam Razin et al.. What makes a reward model a good teacher? an optimization perspective.

Responses to Additional Comments

We sincerely thank the reviewer for their thorough and constructive feedback. We will address all points in the revised manuscript. We summarize our planned changes below:

1. Typos, Citations, and Minor Corrections (Comments 1, 5, 9, 10, 13): Thank you for catching these. We confirm these are errors and will make the following corrections:

• The citation in L49/81 will be corrected to [37].
• All typos will be fixed (e.g., L138 "Explosion", L195 reference to line 5, L283 grammar).

2. Presentation and Clarity Improvements (Comments 4, 7, 8, 11, 12, 15): We will improve the clarity of our presentation across the paper. This includes:

• Updating figure captions and restoring legends (Figs 1, 5, 6) to clarify reward scales and method details.
• Ensuring consistent notation throughout the paper (e.g., unifying $r_{RM}$ and $\tilde{R}$).
• Adding definitions to make all theorems self-contained (Sec 2.2.2).
• Reordering all figures and tables to match their reference order.
• Explicitly defining the "Gradient Noise" variance in Eq. 7 as the variance of the gradient's L2 norm.

3. Substantive Clarifications and Revisions (Comments 2, 3, 6, 14, 16):

• Fig. 1 Rewards: The initial rewards differ because the baseline uses the raw 0-1 signal, while ReDit uses the noisy signal ($r + \epsilon$) for gradient updates. The plot shows this noisy reward; for comparison, a plot of the reward without noise will be added to the appendix.
• Theorem 2 Text: We agree the description was imprecise and will revise it.
• Eq. 5 Expectation: We will update Equation 5 according to the formula provided in Section Responses to Weaknesses 2.
• Adding Supporting References: We agree and will add supporting references for L302-305.

We would like to thank Reviewer UG9k for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.

Sincerely, Authors

Thank you for the response, the effort in running additional experiments, and acknowledging the error in the upper bound of Proposition 3. I have read it and the other reviews carefully.

Regarding the the lower bound in Proposition 3, I can see that if one considers the objective to be the expected reward with respect to a fixed random draw of noise for each question/prompt and output, then it may be possible to derive a lower bound that is affected by the noise. However, there are a few issues remaining:

• This creates an inconsistency with Proposition 1. Namely, the expected gradient for a fixed random draw of noise is not the original gradient. Another way to look at this, as long as the expected gradient is the same (as Proposition 1 suggests), then following the analysis of [37], the lower bound should not depend on the noise since it is obtained by upper bounding the expected gradient norm and the rate at which it increases.
• I believe that if one fixes the randomness of the noise, then any dependence on the variance of the noise should be with respect to the empirical variance as opposed to the true variance. It should be possible to show that with high probability these are close to each other by standard concentration bounds, yet requires extra work and is not immediate.
• Most importantly, the current proof of Proposition 3 incorrectly invokes the lower bound from [37]. Namely, it does not establish the type of lower bound that the response above discusses. Unfortunately, I cannot recommend acceptance without a clear formal proof of the result. There is not much time in the response period, but if you are able to provide a proof that clearly specifies the conditions under which it calls the lower bound from [37] and delineates how the randomness of the noise is treated, I would be happy to consider raising my rating.

We sincerely thank you for your quick and insightful follow-up. Your continued engagement and detailed feedback are invaluable in helping us strengthen the theoretical foundations of our paper. We have structured our response to address each of your final concerns point-by-point, providing the formal justifications you requested.

1. On the Apparent Inconsistency with Proposition 1

Conclusion: There is no inconsistency. Proposition 1 analyzes the gradient in expectation over the noise distribution to show it is unbiased, ensuring the optimization is correct on average. Proposition 3, in contrast, analyzes the convergence rate of the actual stochastic process for a single noise realization, which is governed by the variance of the rewards used in that specific step.

Justification: You correctly identified that these two propositions operate at different levels of abstraction.

• Proposition 1 (Unbiasedness in Expectation): This proposition confirms that if we were to average the gradients from an infinite number of different noise realizations, the expected direction would be identical to the gradient of the original, non-perturbed objective. This is because the noise is zero-mean ($\mathbb{E}_{\epsilon}[\epsilon]=0$), and its expected effect cancels out. This ensures that our method does not introduce systematic biases.
• Optimization of a Stochastic Objective (The Actual Algorithm): We want to emphasize, as you correctly inferred, that our algorithm does not optimize the expected objective $\mathbb{E}\_{\epsilon}[\tilde{J}(\pi\_{\theta})]$. Instead, in each training step, it optimizes the objective based on a single, fixed realization of noise $\epsilon_i$ sampled for each output. For that specific step, the gradient is indeed dependent on that fixed noise draw and is not identical to the original gradient.
• Reconciling with the Lower Bound (Proposition 3): The apparent conflict with the lower bound analysis is resolved by understanding that the convergence bounds from [37], upon which our Proposition 3 is based, depend directly on the variance of the reward signal, not merely the expected direction of the gradient.
  • Theorem 1 from [37] establishes that the time required for policy improvement is inversely proportional to the reward variance.
  • Our ReDit method, by adding noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$, creates a new perturbed reward $\tilde{R}$ whose variance is explicitly increased: $Var(\tilde{R}) = Var(R) + \sigma^2$ (We will expand variance on the second point).
  • Therefore, when we analyze the optimization time for our stochastic procedure, we must consider the properties of the actual rewards being used. The lower bound on convergence time, $t_{\gamma}$, is consequently reduced by the addition of $\sigma^2$ to the variance term, as shown in Proposition 3.

In summary, there is no inconsistency:

• Proposition 1 addresses the average-case directional correctness of our approach, confirming it is unbiased.
• Proposition 3 analyzes the rate of convergence of the resulting stochastic optimization process, demonstrating that the increased reward variance—a direct consequence of using fixed noise in each step—provably accelerates convergence.

Therefore, the two propositions are consistent: one describes the average-case correctness, while the other describes the convergence rate of the practical, stochastic implementation. This theoretical speedup is precisely what we observe empirically. Thank you again for your valuable feedback.

3. On the Formal Proof and Correct Invocation of the Lower Bound from [37]

Conclusion: We provide a revised and precise Proposition 3' that correctly invokes the lower bound from [37]. The revised proposition's time bound now explicitly and formally depends on the total reward variance (including the noise term $\sigma^2$), justified by the concentration analysis above.

Proposition 3' (Revised). Let the perturbed reward be $\tilde{R}(q,o) = R(q,o) + \epsilon(o)$, where the noise $\epsilon(o) \sim \mathcal{N}(0, \sigma^2)$ is drawn independently for each output $o$. With probability at least $1-\delta$ over the realization of the noise function $\epsilon(\cdot)$, for any target reward improvement $\gamma > 0$, the time $t$ it takes for ReDit to find a policy $\pi_{\theta(t)}$ that satisfies:

is lower-bounded by:

where the inner variance term, based on the concentration bound from point 2, can be lower-bounded with probability at least $1-\delta$ as:

We believe this point-by-point response and the rigorous proof sketch fully address your concerns. We will, of course, incorporate all these clarifications—including the formal proof and the revised Proposition 3'—into the final version of the manuscript. We are immensely grateful for your detailed guidance, which has significantly improved our paper. Should any further questions arise, we would be more than happy to provide additional clarification.

It seems that there are still rather major issues with the proof.

• As I noted before, the variance decomposition $Var_{o \sim \pi_\theta} [ \tilde{R} (q, o)] = Var_{o \sim \pi_\theta} [ R (q, o)] + Var_{o \sim \pi_\theta} [ \epsilon (o)]$ is incorrect since the randomness of the noise is not with respect to their distribution, which is indeed independent to $\pi_\theta$, but rather it is taken with respect to $\pi_\theta$. In particular, in the proposed new proof, the step that is incorrect is $Cov_{y \sim \pi_\theta} (R(q,o), \epsilon (o)) = 0$. Why would this covariance be zero? The noises $\epsilon (o)$ are fixed here and can take on any value. Note that this is not the same quantity as $Cov_{y \sim \pi_\theta, \epsilon \sim N(o, \sigma^2)} (R(q,o), \epsilon)$, i.e., the covariance where the randomness of $R(q,o)$ is with respect to $\pi_\theta$ and of $\epsilon$ is with respect to the independent noise distribution, which is indeed zero.

• It is not clear what the formal definition of the optimization method is. What does "noise gradient" stand for in the differential equation? Is that a stochastic differential equation?

There may also be other issues with the remaining steps; I did not verify them since they heavily depend on these. Overall, it seems that these are not minor issues that can be easily fixed, even if a proposition similar to that proposed by the authors can be proven. I do believe the paper has potential, but after three iterations with the authors, the proof is still invalid. The discussion period is mostly ment to clarify questions and misunderstandings, rather than iterating on a completely new proof. Thus, if the authors have a simple fix to the proof and issues above, please let me know. Otherwise, I cannot recommend acceptance unless the proposition will be dropped and the authors provide an explanation of how they will modify the related text in the paper.

We sincerely thank you for your continued, detailed engagement with our manuscript. Your insightful and rigorous feedback through multiple rounds of discussion has been incredibly valuable in helping us identify and understand the deep, non-trivial issues with our theoretical proof. We truly appreciate your constructive suggestions.

After carefully considering your final comments, we have decided to follow your excellent advice and will remove Proposition 3 and its proof from the paper.

You are right that due to the limitations of Theorem 1 in [37] (which only gives a lower bound) and the fact that it is not crucial for our work, we agree that further iterating on new proofs is not the best use of discussion time.

As you suggested, we will modify the paper to discuss the benefits of ReDit on a more intuitive level, supported by our strong empirical results and the sound parts of our theory (Propositions 1 and 2). The revised discussion will focus on how ReDit's noise injection smoothes batch gradients (Proposition 2), which helps alleviate the vanishing/exploding gradient problem we identified while maintaining an average unbiased estimator (Proposition 1). We will explicitly state that a full formal proof of convergence acceleration is a challenging direction that we leave for future work.

We are confident that by following your guidance, the resulting paper will be more robust, clear, and accurately represent our contributions. We are sincerely grateful for your time and the rigorous feedback that has substantially improved our work. We hope that our responses and revisions have adequately addressed all your concerns, and we are happy to answer any further questions.

Finally, we calculate the change in the reward:

Substituting the previous results:

For $V(\theta(t_\gamma); \mathbf{q}) - V(\theta(0); \mathbf{q}) \ge \gamma$ to hold, it therefore must be the case that:

Rearranging the inequality concludes the proof:

Responses to Issue 3

Based on the proof of Issue 1, we can see that the lower bound of $t_\gamma$ is limited by the expected variance of $r_{RM}$ for all $x$. Assuming that the number of $o$ is infinite (See Definition 2 in [37]), it is strictly bounded by $\mathbb{E}\_{\mathbf{q} \sim \mathcal{S}}[\text{Var}\_{o \sim \pi\_\theta(0)(\cdot|\mathbf{q})}[r_{RM}(q,o)]] + \sigma^2$, without resorting to Bernstein's inequality.

Finally, we completely agree with your point that Proposition 3 is not crucial to this work, as it only provides a lower bound and does not indicate that Redit converges faster under all conditions. However, our absolute lower bound is also meaningful; for example, in the best case, Redit may converge faster than grpo.

In conclusion, we would like to once again express our sincere gratitude for your thorough and constructive review. Your detailed comments were instrumental in helping us improve the quality and clarity of our manuscript. We will incorporate all these clarifications—including the formal proof and the revised Proposition 3—into the final version of the manuscript. We hope that our responses and revisions have adequately addressed all your concerns, and we are happy to answer any further questions.

Responses to Issue 1

Your consideration is quite correct. We cannot directly extend Theorem 1 of [37] to Redit. Therefore, we want to prove that Proposition 3 holds regardless of whether $r_{RM}$ is a fixed function (as defined in [37]) or a variable reward function that changes with the our setting (i.e., $r_{RM}(q,o) = r_G(q,o) + \epsilon(o)$). Here, we follow the theorem of [37] and assume that the number of o that can be sampled is infinite (See Definition 2 in [37]). The detailed proof is as follows:

Following the proof structure in Appendix C.4 of [37], we replace the fixed $r_{RM}'(q, o)$ with the variable $r_{RM}(q,o) = r_G(q,o) + \epsilon(o)$. Our modified setup is as follows:

• Ground Truth Reward Function ($r_G(q, o)$): A deterministic function that defines the true optimization objective.

• Training Reward Model ($r_{RM}(q, o)$): A stochastic function, $r_{RM}(q,o) = r_G(q,o) + \epsilon(o)$, where $\epsilon(o) \sim \mathcal{N}(0, \sigma^2)$, $\epsilon(o)$ represents the noise independently sampled for different outputs $o$.

• Training Objective and Gradient at $\theta(t)$ ($\phi_{RLHF}(\theta(t)), \nabla\phi_{RLHF}(\theta(t))$): The gradient is given by $\nabla\phi_{RLHF}(\theta(t)) = \nabla\phi_G(\theta)dt + \text{noise gradient}$, where the second term represents the impact of noise in the gradient estimation.

• The term $\text{Var}\_{\mathbf{q}}^{KL}(\theta(t))$ from the proof in [37] is modified due to the change in the reward function:

  $$\text{Var}\_{\mathbf{q}}^{KL}(\theta(t)) = \text{Var}\_{o \sim \pi\_{\theta(t)}(\cdot|\mathbf{q})}[r\_{RM}^{KL}(\mathbf{q}, \mathbf{o}; \theta(t)) + \epsilon]$$

• Consequently, the term $\text{Var}^{KL}(\theta(t))$ from the proof in [37] also changes:

  $$\text{Var}^{KL}(\theta(t)) = \mathbb{E}\_{\mathbf{q} \sim \mathcal{S}}[\text{Var}\_{\mathbf{q}}^{KL}(\theta(t))] = \mathbb{E}\_{\mathbf{q} \sim \mathcal{S}}[\text{Var}\_{\mathbf{q}}^{KL}(\theta(t))]\_{\text{old}} + \mathbb{E}\_{\mathbf{q} \sim \mathcal{S}}[\text{Var}\_{o \sim \pi\_{\theta(t)}(\cdot|\mathbf{q})}[\epsilon(o)] = \mathbb{E}\_{\mathbf{q} \sim \mathcal{S}}[\text{Var}\_{\mathbf{q}}^{KL}(\theta(t))]\_{\text{old}} + \sigma^2.$$

First, we calculate the parameter movement $||\theta(T) - \theta(0)||$. The rate of change of $\theta$ with respect to $t$ is still equal to the gradient of the current objective function, $\nabla\phi_{RLHF}(\theta(t))$.

Next, we take the derivative of $\text{Var}_{\mathbf{x}}^{KL}(\theta(t))$. Observing that $r$ and $\epsilon$ are independent, we can decompose the variance as follows:

Therefore, $\nabla\text{Var}\_{\mathbf{q}}^{KL}(\theta(t))$ is consistent with the original derivative. So the derivative $\nabla\text{Var}^{KL}(\theta(t))$ (the expectation of $\nabla\text{Var}\_{\mathbf{q}}^{KL}(\theta(t))$) also remains unchanged. That is:

Here, the impact from the change in $\nabla\phi_{RLHF}(\theta(t))$ is ignored during the scaling. Substituting this into Lemma 1, we get:

where $\text{Var}^{KL}(\theta(0))$ becomes larger after the addition of noise.

Since $r_{RM}^{KL}(\mathbf{q}, \mathbf{o}; \theta(0)) = r_{RM}(\mathbf{q}, \mathbf{o}) - \lambda \cdot \ln\frac{\pi_{\theta(0)}(\mathbf{o}|\mathbf{q})}{\pi_{\theta(0)}(\mathbf{o}|\mathbf{q})} = r_{RM}(\mathbf{q}, \mathbf{o})$ for all $\mathbf{q} \in \mathcal{S}, \mathbf{o} \in \mathcal{Y}$. Thus:

Therefore, we can rewrite the bound on $||\theta(T) - \theta(0)||$ as:

Additionally, according to Proposition 2, we can find an upper bound for $||\nabla V(\theta(t); \mathbf{x})||$:

We thank the reviewers for their valuable feedback. We are pleased that they appreciated our engaging concepts, clear and accessible writing, and solid theoretical analysis. We address your concerns below.

Responses to Weaknesses 1

Thank you for raising this important point. To address the question of whether ReDit's benefits extend beyond parameter-efficient tuning, we have conducted additional experiments using a full fine-tuning approach. We can confirm that ReDit's effectiveness is not limited to LoRA and that it provides consistent performance gains in the full fine-tuning setting as well.

For these experiments, we used a setup that differs from the primary one in our main paper. Specifically, we utilized the VERL framework for training with 8 GPUs. The detailed results of this full fine-tuning evaluation on our three mathematical reasoning benchmarks are presented below.

Setting: GSM8K, Qwen2.5-7B-Instruct, Full Fine-Tuning

Setting: MATH, Qwen2.5-7B-Instruct, Full Fine-Tuning

Setting: Geo3k, Qwen2.5-VL-7B-Instruct, Full Fine-Tuning

As the results clearly demonstrate, ReDit consistently outperforms the GRPO baseline across all three benchmarks in the full fine-tuning setting. These findings confirm that ReDit is a robust and general-purpose technique, delivering benefits for both LoRA and full fine-tuning paradigms.

Responses to Weaknesses 2

We thank the reviewers for their valuable suggestions. We trained the model using the MATH training set, tested it on AIME24, and reported the results. The results are as follows:

As shown in the table, our proposed methods (Gauss 0.05 and Uniform 0.05) consistently outperform the GRPO baseline throughout the training process. They achieve a significant improvement in training efficiency and performance. In addition, Redit performs better on out-of-domain datasets than on in-domain data because Redit introduces randomness in the reward, which may lead to the model generalizing to unknown domains.

Responses to Weaknesses 3

Thank you for this suggestion regarding the choice of models for our experiments. We would like to address this in two parts.

First, we wish to clarify that all experiments presented in our main manuscript are indeed focused on reasoning tasks. Although our primary model is based on Qwen2.5, our methodology guides the model to first generate a reasoning path (e.g., chain-of-thought) before outputting the final answer. Therefore, the evaluation metrics directly reflect the model's reasoning capabilities on these tasks.

Second, to further address your concern and demonstrate the effectiveness of ReDit on specialized reasoning models, we have conducted new experiments on DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B. The results on the GSM8K benchmark are presented below.

Setting: DeepSeek-R1-Distill-Llama-8B, GSM8K

Setting: DeepSeek-R1-Distill-Qwen-7B, GSM8K

These new results clearly show that ReDit continues to outperform the GRPO baseline on DeepSeek distillation models. This confirms that ReDit's ability to stabilize training and improve performance is a general principle that holds true across various model architectures, including those specifically optimized for reasoning.

Responses to Questions 1

Thank you for this insightful question regarding the gradient behavior of other advanced RL algorithms like DAPO and VAPO. This is an important point, and we provide the following analysis based on their methodologies and our own observations.

• Regarding DAPO: DAPO is an improvement upon GRPO that introduces Dynamic Sampling to increase the number of effective samples used during training. This technique is effective at mitigating the gradient vanishing problem. However, it does not have a specific mechanism to control gradient explosion. In our own experiments, while running DAPO, we still observed instances of both gradient vanishing and gradient explosion, similar to the phenomena shown for GRPO in Figure 3. This suggests that while DAPO improves one aspect of gradient stability, it does not fully resolve the anomaly, leaving a clear need for methods like ReDit.

• Regarding VAPO: VAPO represents a different class of algorithm. It is a value-model-based method, which distinguishes it fundamentally from value-model-free methods like GRPO and DAPO. Due to this architectural difference, its gradient dynamics are likely to differ significantly from those we analyzed for GRPO. A direct comparison is further complicated by the fact that VAPO is not yet open-source. We are actively monitoring its status and plan to conduct a thorough analysis of its gradient behavior as soon as the code becomes publicly available.

In summary, existing methods like DAPO partially address but do not eliminate the gradient anomalies, while a direct, empirical comparison with VAPO is not currently feasible.

Responses to Questions 2

Thank you for pointing out this ambiguity in L222-L224. We sincerely apologize for the lack of clarity in our original wording.

You are correct that our base models are instruction-tuned. Our intention was to differentiate between two distinct stages of Supervised Fine-Tuning (SFT):

1. General Instruction-Tuning: The initial SFT performed on broad, instruction-following datasets to create the capable base instruction model. The models we use have undergone this stage [1].
2. Task-Specific SFT: A subsequent, optional SFT stage where a model is further fine-tuned on the training data of a specific downstream task (e.g., the GSM8K training set).

When we claimed we wanted to "isolate the effects of SFT," we were referring specifically to Task-Specific SFT. Our goal is to compare our RL-based approach directly against the common practice of further fine-tuning a general instruction model on a specific downstream dataset.

We understand how our original phrasing was confusing and we will revise L222-L224 in the final manuscript to make this crucial distinction explicit. Thank you again for helping us improve the clarity of the paper.

Reference:

[1] Qwen Team, Qwen2.5 Technical Report

We would like to thank Reviewer puQV for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.

Sincerely, Authors

3. Gradient Clipping vs. ReDit: Two Disparate Approaches to a Non-Smooth Landscape

Faced with this pathological landscape that violates a core assumption of gradient-based optimization, gradient clipping and ReDit employ entirely different philosophies.

Gradient clipping does not alter the pathological geometry of the loss landscape itself [4].

• It is a reactive measure: when the optimizer arrives at a "cliff" and computes an enormous gradient, clipping forcibly shortens the update step to prevent the model from "falling off" (i.e., to prevent the training from crashing).
• Limitations: While it effectively prevents divergence, it is a non-principled heuristic that can introduce significant estimation bias. More critically, it still compels the optimizer to follow the direction dictated by the pathological landscape—a direction that may be highly suboptimal for convergence—it just takes a smaller step in that poor direction.

ReDit: A Proactive Approach to Reshape the Landscape

• ReDit's core principle is to proactively fix the optimization problem by reshaping the loss landscape to better approximate L-smoothness.
• Mechanism: By dithering the discrete rewards with zero-mean noise, ReDit transforms the step-function-like reward signal into a continuous, smoothed-out signal (as visualized by the cumulative distribution in Figure 2). This is applied at the source of the problem—the reward signal itself.
• Optimization Benefits:
  1. Restoring Smoothness: Smoothing the reward signal effectively "sands down the cliffs" of the loss landscape, replacing them with manageable slopes. This lowers the landscape's effective Lipschitz constant L, allowing gradient-based methods to operate more stably and efficiently .
  2. Providing Beneficial Variance: ReDit increases reward variance within mini-batches, providing a richer, more continuous exploration signal. This is theoretically proven to reduce the lower bound on policy optimization time (Theorem 1).
  3. Maintaining an Unbiased Gradient: Crucially, our theoretical analysis in Proposition 1 proves that this process maintains an unbiased estimate of the policy gradient in expectation, ensuring the optimization proceeds in a principled direction.

4. Empirical Evidence: ReDit Outperforms GRPO with Gradient Clipping

Our experimental results (e.g., in Figure 6 and Table 2) directly answer your question: Yes, even with grad_clip=1.0 enabled for all methods, the performance of standard GRPO is still demonstrably worse than GRPO with ReDit.

5. Conclusion and Update on Proposition 3

In conclusion, gradient clipping is a necessary "symptomatic treatment" for an unstable process. In contrast, ReDit is a "root-cause cure" that improves the fundamental geometric properties of the problem itself, leading to superior performance.

Finally, regarding your comment about our discussion with Reviewer UG9k, we want to proactively provide an update. After a constructive discussion about the theoretical complexities you both identified, we have decided to remove the controversial Proposition 3 from the manuscript. The paper's core contribution is not the lower bound analysis, but rather the very topic of this discussion: ReDit as a principled method to smooth the loss landscape, resolve gradient anomalies, and thereby accelerate convergence. We believe this makes the paper stronger and more focused.

We would like to thank Reviewer puQV for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.

Reference

[1] Goyal, P. et al. Using natural language for reward shaping in reinforcement learning, IJCAI 2019.

[2] Razin, N. et al. What makes a reward model a good teacher? an optimization perspective, 2024.

[3] Razin, N. et al. Vanishing gradients in reinforcement finetuning of language models, ICLR 2024.

[4] Zhang, J. et al. Why gradient clipping accelerates training: A theoretical justification for adaptivity, ICLR 2020.

We are glad that our previous responses have addressed most of your concerns. Thank you for this insightful follow-up question. This is a critical point that touches upon the core motivation of our work, and we appreciate the opportunity to clarify the fundamental difference between ReDit and gradient clipping.

First and foremost, we must emphasize a key experimental fact: in all of our experiments, for both the vanilla GRPO baseline and our ReDit method, we consistently used actor_rollout_ref.actor.grad_clip=1.0. This is a standard practice to ensure training stability and a fair comparison. Our results directly show that ReDit outperforms GRPO even when gradient clipping is active.

Below, we elaborate on why this is the case.

1. The grad_norm in Figures 1 & 3 is the Raw Gradient Before Clipping

You astutely noticed that the gradient norm (grad_norm) of GRPO in our figures is much larger than 1.0. This is intentional and designed to expose the core issue. In these figures, we report the raw gradient norm computed before the clipping step is applied. The typical training loop sequence is:

1. Compute loss and backpropagate to get the raw gradients.
2. Record the norm of these raw gradients for monitoring (this is the value we plot).
3. Apply gradient clipping (if the norm > 1.0, it is scaled down).
4. Update the model using the (potentially clipped) gradients.

Thus, the exploding gradients in the graphs are not due to an absence of clipping; they show that GRPO's optimization is intrinsically unstable with discrete rewards, a core problem we aim to solve.

2. Root Cause: Discrete Rewards Violate the L-Smoothness of the Loss Landscape

In optimization theory, an ideal loss landscape has the property of L-smooth, that is, its gradient has L-Lipschitz continuity. This property ensures that the curvature of the landscape is bounded, thus avoiding infinitely steep "cliffs" and ensuring that gradient-based methods can converge stably with appropriately chosen learning rates.

However, using discrete, rule-based rewards (e.g., 1 for a correct answer and 0 for an incorrect answer) fundamentally violates the L-smoothness assumption.

• The resulting loss landscape is inherently non-smooth[1]. When the rewards within the group are equal, the gradient approaches zero, and the loss landscape resembles a series of broad, flat "plateaus"[3], interspersed with extremely steep "cliffs" at the decision boundaries where the rewards change (see figure 1 in [2]).

• In this pathological landscape, the L-smoothness condition is severely violated, and no "safe" learning rate can guarantee stable convergence. This is exactly why GRPO diverges immediately without gradient clipping.

Thank you for your response and for the time you have dedicated to reviewing our work throughout this discussion period. We appreciate you reconsidering our rebuttal and updating your assessment.

Your feedback has given us valuable perspectives that we will use to improve our work going forward. We remain committed to this research direction and will continue to build upon our method.

Thank you again for your engagement.

We thank the reviewers for their valuable feedback and appreciate their appreciation for the well-written work, the concise and effective methods, and the mathematical analyses that support the findings. We address your concerns below.

Responses to W1

We thank the reviewer for their question, as it allows us to clarify the principal objective and contribution of ReDit. Our method is primarily designed to address a fundamental optimization challenge: the gradient anomalies (e.g., vanishing/explosion) that impede training stability and efficiency in GRPO. By resolving these issues, ReDit yields significant gains in training efficiency, often achieving comparable or superior results with as little as one-tenth of the training steps required by the baseline. A key aspect of our contribution is that this robust solution is achieved with remarkable simplicity, typically requiring only a single line of code. Therefore, the final accuracy improvement, while a valuable and positive result, should be interpreted as a secondary consequence of this stabilized process, with the core innovation being the efficient resolution of the underlying training instability itself.

Responses to Q1

We sincerely apologize for the missing legends in Figures 5 and 6 and thank the reviewer for pointing out this oversight. The legends were inadvertently cropped during the figure generation process, and we are sorry for any confusion this may have caused. Your assumption is largely correct. To clarify, the markers in the figures correspond to the following methods:

• Red Circles: Baseline (GRPO)
• Yellow/Orange Squares: ReDit with Gaussian noise
• Blue Triangles: ReDit with Uniform noise

We will ensure that both Figures 5 and 6 are updated with clear and complete legends in the final version of the paper.

Responses to Q2

Thank you for this excellent question regarding the applicability of ReDit to domains beyond mathematical reasoning. This is a crucial point, and we are pleased to confirm that ReDit's benefits generalize effectively to coding tasks.

To validate this, we conducted a comprehensive set of experiments on three widely-used coding benchmarks: APPS, HumanEval, and CodeContests. The results consistently show that ReDit improves performance over the GRPO baseline.

Below are the detailed results for each benchmark.

Setting: APPS train data, APPS test data (pass@1)

(Note: For APPS, which has multiple difficulty levels, we report the pass@1 accuracy across all problems)

Setting: APPS train data, HumanEval test data (pass@1)

Setting: CodeContests train data, CodeContests test data (pass@1)

As these results demonstrate, both Gaussian and Uniform variants of ReDit consistently and significantly outperform the GRPO baseline across all three coding benchmarks. This suggests that the core benefit of ReDit—stabilizing the learning signal to improve optimization—is a general principle that is not limited to a single domain and applies effectively to code generation tasks as well.

We would like to thank Reviewer HRjb for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.

Sincerely, Authors

We sincerely thank you for your thoughtful review and positive engagement throughout the discussion period. We are very grateful for your recognition of our work and your agreement that ReDit successfully improves the stability of the optimization process.

We take your point regarding the performance margin to heart, and it provides a clear and valuable direction for our future research. We will continue to build upon this work to further enhance model performance.

Thank you once again for your positive assessment and support for our paper.

We thank Reviewer TpNZ for their positive assessment of our work. We appreciate their recognition of the method's key advantages, including its simplicity, effectiveness, broad applicability, and the rigorous theoretical analysis provided. We address your concerns below.

Responses to Weaknesses 1 and Questions 1

We thank the reviewer for this insightful comment and the opportunity to clarify the relationship between our proposed method, ReDit, and other established techniques.

We agree that ReDit shares a conceptual similarity with methods like Dropout, Data Augmentation, and SGD, as they all leverage randomness to improve the training process. However, the key difference and innovation of ReDit lie in what is being perturbed, how, and for what purpose. While the others are general-purpose techniques, ReDit is specifically designed to address challenges in reward-based optimization. Here is a detailed comparison:

ReDit

ReDit injects noise directly into the reward signal just before gradient computation. The primary objective is to smooth discrete rewards into a continuous (as illustrated in our Figure 2). This targeted intervention is designed to stabilize the gradient, mitigate the vanishing/exploding gradient problems common in settings with sparse rewards, and accelerate convergence (as illustrated in our Figure 1).

Comparison with Other Methods

Dropout: This method applies noise by randomly setting a fraction of the model's internal activations to zero during training. Its main goal is regularization—preventing complex co-adaptations between neurons to reduce overfitting. In contrast, ReDit perturbs the external learning signal (the reward), not the model's internal state, to directly improve the quality of the gradient for optimization.

Data Augmentation: This technique introduces randomness by applying transformations to the input data (e.g., rotating an image). This encourages the model to learn invariant features, thereby improving its ability to generalize. While Data Augmentation focuses on expanding the input space to enhance generalization, ReDit operates on the output/reward space to improve the stability and efficiency of the optimization process itself.

Stochastic Gradient Descent (SGD): The randomness in SGD comes from estimating the gradient using mini-batches of data. However, this process can fail when rewards are sparse and discrete. For example, if all samples in a mini-batch yield the same reward (e.g., all zeros), the variance of the rewards is zero, causing the gradient to vanish and halting the learning process. ReDit directly addresses this failure mode. By injecting controlled noise, ReDit ensures that the reward signal within a batch has a non-zero variance, thus always generating a usable and informative learning signal for the model to update its parameters.

In summary, while all these methods utilize stochasticity, ReDit is fundamentally distinct in its target of perturbation, its mechanism of action, and its underlying objective.

Responses to Weaknesses 2

We thank the reviewers for their valuable suggestions. We trained the model using the MATH training set, tested it on AIME24, and reported the results. The results are as follows:

As shown in the table, our proposed methods (Gauss 0.05 and Uniform 0.05) consistently outperform the GRPO baseline throughout the training process. They achieve a significant improvement in training efficiency and performance. In addition, Redit performs better on out-of-domain datasets than on in-domain data because Redit introduces randomness in the reward, which may lead to the model generalizing to unknown domains.

We would like to thank Reviewer TpNZ for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.

Sincerely, Authors

That's great to hear. Thank you for your time and positive feedback. We're glad our response addressed your concerns.