The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning

Thanks for your thoughtful feedback. We address your raised issues as follows.

While the decomposition of RLVR into PSR and NSR is insightful, the overall concept of this method is not entirely new and has been touched upon in previous studies [1].

According to NeurIPS policies, works posted after March 1st, 2025 are considered concurrent. Reference [1] was posted on arXiv on April 15 and should therefore be treated as concurrent, not prior work.

More importantly, the methodology in [1] is fundamentally different from ours. Their approach is based on prompt filtering: removing prompts whose responses are entirely correct or incorrect, this means that the training data still contains a mix of both positive and negative reward examples. In contrast, our work introduces a complete objective decomposition of RLVR into PSR and NSR, fully isolating positive and negative reward examples. This clean setup allows us to clearly examine how each reward type individually shapes model behavior.

The proposed W-REINFORCE algorithm, despite showing promise, requires careful tuning of the weighting factor

The choice of the weighting factor in W-REINFORCE seems crucial for its performance. Could the authors offer more detailed guidance or a systematic approach for selecting an appropriate value, especially for different models and tasks?

We provide hyperparameter study of W-REINFORCE with different λ as follows:

MATH (Pass@k)

RL algorithms generally require hyperparameter tuning, but our experiments show that W-REINFORCE is not highly sensitive to λ.

The claim that NSR (using only negative samples) outperforms PSR is a bit counterintuitive and confusing. The authors should provide a clearer explanation for this result (Table 1).

The explanation why NSR can outperform PSR is described in Line 142 and further supported by empirical evidence in Figure 3 and analysis in Section 4.2. As shown in Figure 3, PSR quickly overfits to the observed correct rollouts, which reduces output diversity and harms exploration, resulting in inferior pass@k performance when k is large. In contrast, NSR penalizes incorrect outputs and maintains a high entropy, preserving a broader search space. Section 4.2 further explains this with a principled gradient analysis. It reveals that PSR suppresses all possible alternatives except the highest probability token, while NSR encourages exploration by promoting all plausible candidates based on model’s prior. These results and analysis explain why NSR can outperform PSR.

Can the authors elaborate on the relationship between SFT and PSR?

SFT typically refers to off-policy training, whereas PSR is fully on-policy. Moreover, PSR follows a standard RL setting that incorporates importance sampling and clipping in its loss function.

How does the proposed W-REINFORCE algorithm compare to other RL variants in the literature that also emphasize negative reinforcement? What specific advantages does it offer over these existing methods?

To the best of our knowledge, our work is the first to find that solely penalizing mistakes through negative reinforcement learning, without relying on positive reward signals, can effectively improve LLM reasoning. If we missed any relevant work, we are happy to discuss and cite them accordingly.

The advantages are that it can preserve models’ generation diversity and strike a good balance between pass@1 and pass@k performance when k is large (i.e., more decoding budgets).

The authors mention that NSR refines the model's existing knowledge rather than introducing entirely new behaviors. Could they provide more in-depth analysis or visualization to help readers better understand this refinement process?

Section 4.2 provides a principled gradient analysis on why NSR is refining existing knowledge. It reveals that the gradients of NSR promote all plausible alternatives based on models’ prior. Figure 4 offers a visualization of this process.

Why does W-REINFORCE underperform PPO in pass@1? An explanation from the authors would be appreciated.

As shown in table 1, W-REINFORCE is either comparable to PPO (on MATH and AMC23) or outperforms PPO (on AIME 2025) measured by pass@1.

How does the proposed method generalize to OOD datasets?

We train models on the MATH dataset and evaluate them on AIME 2025 and AMC 23. MATH consists of PhD-level problems, while AIME and AMC feature Olympiad-level math questions with different formats and difficulty distributions. These test sets can be considered OOD relative to the training data. The results on various benchmarks suggest our method can generalize to OOD datasets.

What is the specific value of n in the pass@k evaluation metric?

Yes, n is 256. Pass@256 is computed by averaging over the entire evaluation set. For example, the MATH test set contains 5000 problems. For each problem, pass@256 is either 0 or 1 depending on whether at least one of the 256 generations is correct. The final pass@256 metric reported in Table 1 is the mean across all problems, which yields a number between 0 and 1.

[1] Xiong, Wei, et al. "A minimalist approach to llm reasoning: from rejection sampling to reinforce." 2025.

Thanks for your thoughtful feedback. We address your raised issues as follows.

My main concerns are related to the lack of statistical significance tests or statistical uncertainty estimates (quality, uncertain significance)...

We report the 95% confidence intervals on MATH with Qwen2.5-Math-7B in the table below:

As shown in the table, NSR/W-REINFORCE consistently matches or outperforms the state-of-the-art RL algorithms PPO and GRPO across a wide range of pass@k values. Notably, for $k > 4$, both W-REINFORCE and NSR achieve statistically significantly higher pass@k scores than GRPO. We'd like to highlight that W-REINFORCE achieves this competitive performance with a much simpler design: no value model (unlike PPO) or sophisticated group-wise advantage normalization (unlike GRPO). Given these advantages, we believe our method offers a good trade-off between performance and simplicity, and can serve as a strong baseline for future research.

The authors only tested one model (quality, uncertain significance)...

To address your concern, we conduct experiments under the same setting with a different model family. Following are results with Llama-3.1-8B-Instruct as the backbone model on MATH:

As shown in the table, NSR is also effective and comparable or outperforms other methods with Llama model.

What are the implications of splitting rewards into “only positive” and “only negative”... does encountering a positive sample every 10 negative samples impact the overall trajectory differently when comparing trajectories with only positive or negative samples?

The positive and negative rewards in RLVR are typically entangled and difficult to interpret directly, which motivates our decomposition into “only positive” and “only negative” rewards. This controlled setup enables us to isolate and understand the independent effects of each reward signal. As shown in Figure 3, we present the training trajectories for three scenarios: both positive and negative rewards (PPO/GRPO), only positive rewards (PSR), and only negative rewards (NSR).

We believe PPO and GRPO’s trajectories reflect the practical training dynamics of mixed rewards because they are designed to handle varying ratios of correct and incorrect rollouts using advantage normalization to adapt to the natural distribution of rewards. We do not study the scenario with a manually fixed positive-to-negative ratio (e.g., 1:10) as it introduces a sampling bias that deviates from the actual training dynamic and is not a realistic representation of on-policy learning.

To what extent are the results in Figure 2 explained by PPO and GRPO training encountering more negative than positive reward samples during training?

We clarify that PPO and GRPO do not encounter more negative than positive reward samples. Instead, as shown in Figure 3(c), PPO and GRPO encounter more positive than negative reward samples during training. This leads to increasing probability of generating observed correct samples and causes a reduction in model entropy, as shown in Figure 3(b). The narrowing of the output distribution limits the model’s exploration, which explains the inferior pass@k performance for larger k shown in Figure 2.

To what extent is the trend of RL models losing their advantage at larger decoding budgets solely explained by negative reward samples (as argued in lines 136-141)? Are there no other possible explanations?

The major difference between NSR and GRPO/PPO is that NSR relies solely on negative reward samples while excluding positive ones, while PSR represents the opposite extreme. This tightly controlled setup allows us to directly attribute the observed trends to the presence or absence of positive and negative rewards.

While we cannot claim with absolute certainty that no other factors contribute in any way, our controlled experimental design is a standard scientific method for isolating a primary cause (i.e., positive reward only training yields pass@k degradation). The fact that we can (a) turn the degradation effect on and off by manipulating a single variable (the positive reward signal) and (b) provide a clear, low-level mechanistic explanation for it (the gradient dynamics) makes our explanation robust and direct, minimizing the likelihood of other major confounding factors.

It is unclear to me if the authors agree with this statement “[line 265] RLVR primarily adjusts the model’s output distribution toward high-reward responses, rather than eliciting new reasoning abilities beyond the base model.”

Whether RLVR elicits new reasoning abilities or primarily adjusts the output distribution depends on the specific setup and task, and generally there isn’t a clear-cut answer. Our paper does not aim to resolve this question. Instead, our focus is to highlight a previously under-appreciated role of the negative sample reinforcement in shaping model behavior in RLVR. An in-depth investigation into the emergence of reasoning ability is orthogonal to our contributions and beyond the scope of this work.

What can we learn or hypothesize from your study about RL algorithms using dense/progress-based rewards?

In dense reward cases, where the reward may be in the range of -1~1, the findings of our paper can still transfer to such a scenario: positive reward will reinforce observed rollouts, negative reward will promote other possible candidates.

Dear Reviewer,

As the discussion deadline approaches, we kindly request that you take a moment to review our rebuttal. Thank you again for your thoughtful feedback.

Thanks for your thoughtful feedback. We address your raised issues as follows.

The conclusion of the paper heavily relies on the Qwen-Math type of "expert models" that are already extremely powerful… and thus its universality is questionable.

To address your concern, we conduct experiments under the same setting with a different model family. Following are Pass@k results with Llama-3.1-8B-Instruct as the backbone model on MATH

As shown in the table, NSR is also effective and comparable or outperforms other methods with Llama model.

This method is highly suitable for tasks in mathematics that have clear, binary (right or wrong) feedback signals. However, it is difficult to apply to tasks such as creative writing, dialogue systems, etc., where the reward signals are ambiguous, subjective or continuous. This is because it is impossible to clearly distinguish between "positive" and "negative" samples.

As stated in the paper, our focus is on reinforcement learning with verifiable rewards (RLVR), where the reward is inherently unambiguous and verifiable. This setup aligns with prior RLVR literature that uses binary rewards [1,2,3,4]. Open-ended tasks like creative writing and dialogue are typically addressed using RLHF frameworks [5] which represent a different training setting (e.g., Qwen 3 [4] has separate training stages for verifiable and non-verifiable tasks). Therefore, extending our analysis and method to non-verifiable tasks is an interesting direction for future work, but is beyond the scope of RLVR.

Long-term use of negative reinforcement alone leads to performance degradation, indicating that it is not a stable training strategy in the long run. Additionally, the new algorithm W-REINFORCE proposed by them lacks sufficient ablation analysis for the selection of key hyperparameters.

We explicitly acknowledged this training stability observation in our paper to promote transparency and to encourage further research in RLVR. That said, we also note that training instability or performance degradation over extended RL training is a well-known challenge in the broader RL-for-LLM-reasoning research, not unique to our method. For example, standard RL methods (e.g., PPO) also suffer from training instability or collapse over long horizons for LLM training, as shown in Figure 2 of [2].

As for hyperparameter of W-REINFORCE, we provide ablations with different λ as follows:

MATH (Pass@k)

As shown in the above table, we test λ=0.05, 0.1, 0.2, large λ makes the method closer to PSR and small λ makes it closer to NSR. Results show that W-REINFORCE is robust with different λ.

[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025
[2] Demystifying Long Chain-of-Thought Reasoning in LLMs, 2025
[3] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild, 2025
[4] Qwen3 Technical Report, 2025
[5] Training language models to follow instructions with human feedback, 2022

Dear Reviewer,

As the discussion deadline approaches, we kindly request that you take a moment to review our rebuttal. Thank you again for your thoughtful feedback.

Thanks for your thoughtful feedback. We address your raised issues as follows.

Personally, I find that the phenomenon observed is somewhat aligned with common intuition in the field. Additionally, the technique of reweighting the negative samples lacks novelty in my view.

To the best of our knowledge, our work is the first to find that solely penalizing mistakes through negative reinforcement learning, without relying on positive reward signals, can effectively improve LLM reasoning. If we missed any relevant work, we are happy to discuss and cite them accordingly.

The paper primarily focuses on the on-policy setting. However, off-policy settings—such as those used in popular algorithms like DPO—may exhibit different behaviors. The lack of consideration for off-policy scenarios limits the generality and significance of the findings.

On-policy RL training has been increasingly used in the latest state-of-the-art models. Notably, models such as DeepSeek-R1 [1], Kimi [2], and Qwen3 [3] all have adopted fully on-policy training without any off-policy RL components. Moreover, recent RLVR research [4,5] predominantly adopts on-policy RL algorithms like PPO and GRPO. Therefore, we believe our findings under the on-policy setting are both significant and relevant to current practice.

[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025
[2] Kimi k1.5: Scaling Reinforcement Learning with LLMs, 2025
[3] Qwen3 Technical Report, 2025
[4] Demystifying Long Chain-of-Thought Reasoning in LLMs, 2025
[5] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild, 2025

Dear Reviewer,

As the discussion deadline approaches, we kindly request that you take a moment to review our rebuttal. Thank you again for your thoughtful feedback.

Dear Reviewer,

As the discussion deadline (August 8, AOE) is approaching, we kindly remind you to take a moment to review our rebuttal. We sincerely appreciate your time and feedback.