NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation

We thank the reviewer for highlighting the strengths of our method, including the noise-driven exploration, Bayesian estimator, and lightweight design. We address the reviewer’s concerns below.

---

W1: Promote grounded answers but at the expense of descriptive breadth

---

We argue that AMBER Coverage has inherent limitations and does not fully support the claim that our method promotes grounding at the expense of descriptive breadth. In AMBER's generative setting, each case has on average 5.5 ground truth captions, and our method shows at most a 2.7% decrease in Coverage compared to the strongest baseline—equivalent to merely 0.14 missing words per case, which is a negligible trade-off considering the benefit of reduced hallucination.

Moreover, we observe that many hallucination-mitigating methods exhibit similar behavior on AMBER, where improving factual accuracy can naturally lead to slightly lower coverage scores. This suggests that the metric may not fully capture the quality of grounded yet concise descriptions. To illustrate this, we propose a CHR to measure how much the hallucination rate decreases relative to the drop in coverage. A higher value indicates more effective hallucination reduction with less sacrifice in coverage.

$\text{CHR} = \frac{|\Delta \text{Hallucination Rate}|}{|\Delta \text{Coverage}|}$

This trend is not unique to our method — as the table shows, several recent strong baselines also experience similar trade-offs between hallucination reduction and coverage."

From the table, we can see that other methods on AMBER also exhibit a decrease in hallucination accompanied by a drop in coverage, while NoisyGRPO 7B achieves the best CHR score. This suggests that NoisGRPO reduces hallucination more effectively than others, while incurring a relatively smaller drop in coverage, as reflected by the highest CHR score among all methods.

Reference:

[1] Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization, arXiv 2023, cited 148 times.

[2] CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs, ECCV 2024, cited 26 times.

---

W2: Vanilla GRPO produces more concise CoT with comparable accuracy

---

We acknowledge this observation. However, we would like to clarify that the results on the MME-CoT efficiency category do not fully support the claim that vanilla GRPO produces more concise CoT. First, MME-CoT evaluates CoT quality based on relevance and reflection, which do not comprehensively measure consciousness. Furthermore, we conducted a quantitative analysis of the actual CoT lengths during inference. As shown in the table below, NoisyGRPO consistently produces shorter CoTs across benchmarks:

We observe that NoisyGRPO generates significantly shorter CoTs compared to vanilla GRPO, while achieving better performance on most benchmarks. From the perspective of CoT length as a proxy for conciseness, we argue that NoisyGRPO demonstrates a clear advantage in generating more concise CoTs.

Moreover, we provide justification for the shorter CoT length on AMBER: since all questions are fixed as "Describe the image," NoisyGRPO typically reframes the task by posing a question about the image's main subject. This behavior is illustrated in the visualization in Figure 10 of the appendix.

---

W3: The gains are sensitive and depend on careful tuning.

---

We would like to clarify that we did not perform extensive hyperparameter tuning, and we humbly argue that introducing new hyperparameters is often inevitable when proposing a new framework. To demonstrate the robustness of our method to hyperparameter choices, we conduct a sensitivity analysis of $\gamma$, as presented in the table below.

The results show that NoisyGRPO consistently outperforms the baseline across a small but meaningful range of $\gamma$ values, demonstrating the method’s robustness and indicating that its performance gains do not rely on fine-grained hyperparameter tuning.

---

W4: Concerns about embedding bias and benchmark overfitting.

---

Embedding bias

We acknowledge the reviewer’s concern that SBERT may introduce bias as a reward function. This is indeed a key consideration in our method design. To mitigate this issue, our Bayesian advantage estimation explicitly incorporates a prior term that reflects the expected trajectory quality based on noise levels. This mechanism dynamically balances the SBERT-based semantic reward and the prior estimation, thereby reducing over-reliance on the potentially biased SBERT signal during training.

Moreover, as detailed in our response to reviewer nP6a, Weakness 4, we conducted ablation studies using a rule-based reward function. The results consistently show that SBERT outperforms the rule-based alternatives in our experiments, further supporting its utility despite its known limitations.

Benchmark Overfitting

As addressed in our response to previous W3, the parameter settings optimized on MMStar are not optimal but still yield strong results for other benchmarks. For example, $\gamma = 0.015$ achieves the best performance on MMERealworld, while $\gamma = 0.01$ performs best on AMBER. Nonetheless, our method achieves comparable and strong performance across these benchmarks, which demonstrates that NoisyGRPO does not overfit to MMStar.

---

We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.

We sincerely thank the reviewer for recognizing the conceptual novelty of introducing a principled noise-based prior in GRPO, the consistent performance improvements, the thorough ablation studies, and the transparent reporting of computational costs. We appreciate your encouraging feedback and address the remaining concerns below.

---

Q1 & W3: Comparison to DPO & R1

---

Thank you for the great suggestion. First, we clarify that VLM-R1, when applied to our VQA training data, does not introduce additional design beyond the GRPO algorithm and thus is the same as our GRPO baseline. For DPO, we empirically demonstrate that the NoisyGRPO outperforms DPO baseline. Specifically, we extracted the same number of training samples from MMRLHF and trained a DPO model using the recommended settings from LLaMA Factory. The results are summarized in the table below.

From the results, we observe a significant performance gap between the DPO baseline and NoisyGRPO across multiple metrics. This discrepancy can be attributed to the higher data efficiency of NoisyGRPO, whereas DPO requires a larger amount of high-quality preference data to achieve competitive performance.

---

Q2 & W2: Text-only generalisation

---

Although text-only generalization is beyond the current scope of our work, we conducted exploratory evaluations on MMLU and found that NoisyGRPO exhibits potential for generalization to the text domain.

Specifically, we compared GRPO, NoisyGRPO, and the pretrained MLLM.

Because the RL training prompts explicitly require the model to perform Chain-of-Thought reasoning grounded on visual input, multimodal models often fail to follow instructions properly when images are absent. This results in a high format error rate (exceeding 60%) for both GRPO and NoisyGRPO on text-only samples. To mitigate this, we only report performance on samples where the model produces valid (format-correct) answers.

We observe that, despite the high format error rates caused by the absence of visual input, NoisyGRPO consistently outperforms GRPO on valid outputs across most categories. This suggests that our method has potential for generalization beyond vision-language tasks to purely text-based scenarios.

We acknowledge that these results are preliminary, and we plan to conduct more extensive evaluations to better understand and improve text-only generalization in future work.

---

Q3 & W4: Statistics for Training

---

Thank you for the insightful question. We clarify that NoisyGRPO and GRPO sample the same number of rollouts per iteration, making their computation cost directly comparable.

The table below summarizes wall-clock time, GPU-hours, and peak memory usage measured on a single node with 8 A100 GPUs:

We acknowledge that the overall computation cost remains substantial, which is a common challenge in large-scale vision-language RL research. The 13K dataset size already pushes our computational budget limits; with additional resources, we plan to explore even larger datasets to further validate our approach.

---

Q4: Training with Fewer Data

---

Great suggestion! We conduct experiments with subsampled data (e.g., 3k and 6k) and observe that NoisyGRPO maintains consistent gains over baselines, demonstrating its scalability in leveraging more training data to improve performance.

Specifically, we sampled 3k and 6k training sets from the original 13k dataset and conducted experiments. To ensure a fair comparison, we set the number of training steps for the 3k and 6k subsets to be the same as that used for training on the full 13k dataset. The results are shown in the table below:

We observe from the results above that NoisyGRPO’s performance improves as the training data size increases. Moreover, at each data scale, NoisyGRPO consistently outperforms GRPO. Therefore, we consider NoisyGRPO to be a scalable approach.

---

W4: Prior assumes a monotonic relation between noise and difficulty

---

We would like to provide some additional clarification on this limitation. We believe this assumption holds in most current vision-language scenarios, as evidenced by the results in Table 2. For cases where the assumption may fail, such as coarse perception tasks, we observe that this failure does not cause a significant performance drop in NoisyGRPO — it achieves results comparable to the baseline (e.g., less than 1% decrease on coarse perception in Table 2). Therefore, we consider this a minor limitation that does not affect the main claims of our paper.

---

We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.

Thank you for acknowledging the methodological soundness, theoretical rigor, and practical value of our work; we will address your concerns in detail below.

---

Q1 & W1 Methodological Differentiation.

---

Thank you for raising this important question. Our method is conceptually distinct from prior RL works. Below, we provide clarification on how our approach differs from prior methods such as SNI[12], DAPO [51], and VLM-R1 [39]. We will add these discussions to the related works in the revised version:

Differentiation over Selective Noise Injection[12]

While both SNI [12] and our method leverage noise injection in policy exploration to enhance RL generalization, our approach fundamentally differs from SNI in two key aspects, as outlined below.

• Off-policy handling for noise-injected policy exploration. SNI [12] adopts a manually controlled weighted sum of gradients from noisy and clean rollouts (Eq. 7 in [12]), using a fixed mixing ratio set by a hyperparameter. In contrast, our method models the controllable noise level as a prior within a Bayesian advantage estimation framework, enabling trajectory-wise dynamic calibration of advantages and more principled off-policy correction.
• Noise Injection Strategy. SNI [12] applies noise indirectly through structural mechanisms like Dropout and Variational Information Bottleneck (VIB), which perturb intermediate feature representations. In contrast, our method injects noise directly into the input images during rollout generation—a vision-language-specific strategy that jointly enhances exploration and robustness to multimodal hallucination, which is not explored by prior methods.

Differentiation over DAPO and VLM-R1

While DAPO and VLM-R1 also build on the GRPO backbone, our method optimizes GRPO from a principled perspective. We provide justification as below:

• DAPO proposes several improvements to enhance the stability and efficiency of training with GRPO, including Clip-Higher, Dynamic Sampling, Token-level Policy Gradient Loss, and Overlong Reward Shaping.
• VLM-R1 focuses on applying GRPO to REF and OVDet tasks, designing detection-specific reward functions tailored for these domains.

In contrast, the main focus of our method lies in addressing the challenges of insufficient exploration and the absence of CoT-process rewards in multimodal RL. Beyond targeting different tasks, our approach introduces a principled framework that tackles these issues through controllable noise injection and a Bayesian modeling of advantage estimation, enabling more effective trajectory-level advantage assignment and policy exploration in multimodal settings.

---

Q2 & W4 Reward Signal Robustness

---

Thanks for the insightful question! We first clarify our claim regarding the SBERT reward bias, then we address the concerns of reward robustness.

Clarification We would like to clarify that our intention is not to suggest that the Bayesian estimator eliminates the bias inherent in the SBERT-based reward model. Rather, we aim to claim that within our experimental setting, which involves multimodal RL across verifiable and unverifiable data, the Bayesian advantage estimation alleviates the optimization bias introduced by noisy or misaligned reward signals, as emperically demonstrated in paper Section 4.6.

Reward Signal Robustness The presence of bias in rewards for unverifiable data is inevitable in open-ended RL tasks. For instance, widely adopted solutions such as LLM-as-Judge [1] produce inconsistent or unreliable signals, as highlighted in [2].

We choose SBERT as our reward model due to its strong semantic understanding and its computational efficiency. We also empirically validate its effectiveness, showing consistent gains with NoisyGRPO. We acknowledge that SBERT’s bias may lead to failure in some cases, but emphasize that SBERT is a pluggable component and can be replaced with stronger or task-specific reward models as they become available.

Furthermore, we acknowledge that reward for unverifiable data is an important future direction. We are currently investigating principled ways to diagnose and improve insufficient exploration and missing process CoT reward issues in multimodal RL settings.

Reference:

[1] Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.

[2] Scaling Laws for Reward Model Overoptimization.

---

Q3 & W5: Ablation on Scale and Efficiency

---

Good suggestion! First, we clarify that NoisyGRPO and GRPO sample the same number of rollouts. The wall-clock time, GPU-hours, and peak memory usage for both methods are summarized in the table below to support real-world deployment considerations. The numbers are tested on one node with 8 A100 GPUs.

This suggests that NoisyGRPO maintains comparable efficiency with GRPO, despite its additional sampling step, making it suitable for real-world deployment.

---

Q4 & W6 Broader Applicability

---

Thank you for the insightful suggestion. Our current work is tailored to vision-language reinforcement learning, where challenges like limited exploration and reward sparsity are prominent. While we have not yet applied NoisyGRPO to text-only or audio-visual tasks, the core idea of injecting controllable perturbation to policy exploration and leveraging the perturbation level as prior for advantage estimation is modality-agnostic and potentially transferable. We view this as promising future work and hope to validate the method’s robustness and generality in broader settings.

---

Q5 Error Analysis

---

We believe the observed limitations are due to benchmark-specific characteristics and constitute a minor issue in the broader context of our method’s performance.

Coverage in AMBER

We argue that AMBER Coverage has inherent limitations and does not provide evidence of a systematic failure in NoisyGRPO. In AMBER's generative setting, each case has on average 5.5 ground truth captions, and our method shows at most a 2.7% decrease in Coverage compared to the strongest baseline—equivalent to merely 0.14 missing words per case, which we consider a negligible trade-off given the substantial gains in hallucination reduction.

Moreover, we observe that many hallucination-mitigating methods exhibit similar behavior on AMBER, where improving factual accuracy can naturally lead to slightly lower coverage scores. This suggests that the metric may not fully capture the quality of grounded yet concise descriptions. To illustrate this, we propose a CHR to measure how much the hallucination rate decreases relative to the drop in coverage. A higher value indicates more effective hallucination reduction with less sacrifice in coverage.

$\text{CHR} = \frac{|\Delta \text{Hallucination Rate}|}{|\Delta \text{Coverage}|}$

This trend is not unique to our method — as the table shows, several recent strong baselines also experience similar trade-offs between hallucination reduction and coverage."

From the table, we can see that other methods on AMBER also exhibit a decrease in hallucination accompanied by a drop in coverage, while NoisyGRPO 7B achieves the best CHR score. This suggests that NoisGRPO reduces hallucination more effectively than others, while incurring a relatively smaller drop in coverage, as reflected by the highest CHR score among all methods.

Reference:

[1] Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization, arXiv 2023, cited 148 times.

[2] CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs, ECCV 2024, cited 26 times.

Coarse Perception

We argue that this is a minor issue of NoisyGRPO, as our method achieves performance comparable to the baseline (within ±1% accuracy). This suggests that our approach remains effective even in challenging scenarios.

The observed limitation stems from the fact that our assumption — the monotonic relation between noise and trajectory quality — does not always hold in Coarse Perception tasks. These tasks rely on high-level abstraction where fine-grained visual details are less critical, so noise injection may not sufficiently perturb the model's output to expose quality differences.

However, this does not undermine our method, as the Bayesian advantage estimator is designed to adaptively downweight the prior when its confidence is low. Specifically, when noise fails to differentiate quality, the semantic reward's variance decreases (Eq. 8), leading to higher uncertainty in the prior and a reduced weight in the final estimation (Eq. 9). This adaptive mechanism ensures stable performance even when noise injection is less effective.

---

We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.

We appreciate the reviewer for the positive feedback and for recognizing the motivation, novelty, and clarity. We appreciate your comments and will address your concerns point-by-point below.

---

W1: The Off-Policy Strategy & Lack of Justification

---

We sincerely thank the reviewer for this insightful and important question, which touches upon the core design motivation of our method. We address this concern through a three-step response:

• We clarify how the policy update is conducted in our framework.
• We justify our design by showing that it has already accounted for the mismatch between noisy rollouts and clean policy updates, in a way that aligns with the principle of importance sampling.
• We present ablation studies comparing policy updates with and without noisy inputs, which confirm that using clean inputs for policy updates leads to more stable and effective learning.

Clarification of “using clean inputs for policy updates” We provide the objective function of NoisyGRPO in Eq. (1), the policy gradient is computed based on the following objective:

$$\nabla\_{\theta} J\_{\text{NoisyGRPO}}(\theta) = \mathbb{E}\left[ \nabla\_{\theta} \log \pi\_{\theta}(o\_i^n \mid q, I) \cdot w\_i \cdot \tilde{A}\_i^k \right], \quad \text{where } w\_i = \frac{\pi\_{\theta\_{\text{old}}}(o\_i^n \mid q, I)}{\pi\_{\theta}(o\_i^n \mid q, I)}$$

For clarity, we omit auxiliary components such as the KL penalty and gradient clipping in the above expression, which are standard in practice but do not affect the core analysis here.

Here, “using clean inputs for policy updates” specifically refers to evaluating both the policy gradient term $\nabla\_{\theta} \log \pi\_{\theta}(o\_i^n \mid q, I)$ and the importance weight $w\_i$ under the clean image $I$.

Design Justification Our Bayesian Advantage Estimation accounts for the mismatch between perturbed rollouts and clean policy updates, serving a similar role to importance sampling in off-policy learning.

Specifically, our advantage $\tilde{A}^k_i$ is computed by normalizing a weighted sum of two reward estimates as shown in Eq. (9). Here, $\hat{r^n_i}$ is the group-normalized form of $r_i^n = 1-n_i$, where $n_i$ denotes the noise level of the $i^{th}$ trajectory. And $\hat{r^n_i}$ could approximate the importance sampling under our framework.

To aid understanding, we consider an extreme case for illustration, where we let $\tilde{A}^k_i = r_i^n$. In this case, the policy gradient update becomes:

$$\nabla\_{\theta} J\_{\text{NoisyGRPO}}(\theta) = \mathbb{E}\left[ \nabla\_{\theta} \log \pi\_{\theta}(o\_i^n \mid q, I) \cdot w\_i \cdot r\_i^n \right]$$

We can reinterpret $r\_i^n$ as an importance weight:

$r\_i^n = \frac {1 - n} {1} = \frac {q(x)} {p(x)}$

where $q(x)$ denotes the (clean) target distribution, and $p(x)$ denotes the noise-injected behavior distribution. This shows that $r\_i^n$ downweights noisy trajectories proportionally to their estimated noise level $n^{i}$, effectively approximating an importance sampling correction between $q(x)$ and $p(x)$.

Ablation Experiment Additionally, we perform ablation experiments where the policy is updated using noise-injected images. As shown in the table below, this leads to a performance drop. This is because the evaluation scenario assumes clean input images, requiring the target policy to generate responses based on clean visual inputs. Therefore, updating the policy with clean images during training ensures alignment with the evaluation setting. These results validate our design choice.

---

W2: The distinction between the two Bayesian advantage estimation strategies is unclear.

---

Apologies for the confusion. We would like to clarify that our method does not involve two separate Bayesian advantage estimation strategies. We only inject noise into the image inputs during the rollout collection phase. The Bayesian advantage estimation module does not involve any explicit noise injection.

We recognize two potential sources of confusion that may lead to misunderstanding our approach:

• Noise prior in line 192: The noisy prior is treated as a prior estimation of the trajectory quality decided by the noise level during rollout generation.
• Gaussian noise in line 186: In our Bayesian estimator, we model the semantic reward $\hat{r}_s$ and noisy reward $\hat{r}_n$ as being corrupted by Gaussian noise, which enables us to obtain a posterior estimation $\hat{r}_i$ through Bayesian update.

We hope this could resolve the confusion. We will also revise the manuscript accordingly in the revised version.

---

W3: Lack of comparison with the SOTA methods in multimodal mathematical reasoning benchmarks.

---

We thank the reviewer for raising this important point. Our work addresses a multimodal RL setting that is orthogonal to multimodal math reasoning, so extensive comparisons with multimodal math reasoning benchmarks are beyond the scope of our work.

Specifically, prior works [1–3] on multimodal math reasoning assume verifiable and unbiased reward signals (e.g., exact answer matching), whereas our work addresses a more realistic scenario where reinforcement learning operates under potentially noisy or biased rewards—a setting we refer to as RL beyond verifiable rewards. This setting is crucial for real-world tasks such as visual chat and image captioning, where ground-truth signals are often ambiguous or unavailable. Our method is specifically designed for this setting, incorporating Bayesian advantage estimation to effectively mitigate the impact of reward noise during policy optimization.

Furthermore, our study aims to systematically investigate whether NoisyGRPO can address two under-explored but crucial challenges in preference optimization for multimodal reasoning: (1) insufficient exploration during policy learning and (2) lack of supervision over intermediate reasoning steps. These aspects are rigorously evaluated through controlled experiments (see Table 1 and Table 2), where we measure both alignment with human preferences and improvement in reasoning consistency.

While we acknowledge that a full comparison on multimodal math benchmarks would further validate our findings, we leave comprehensive benchmarking as future work and appreciate the reviewer’s suggestion in this direction.

---

W4: Inadequate Reward Function Analysis.

---

Thank you for highlighting the importance of reward design analysis.

Comparison with rule-based reward

To validate our choice of using the text embedding model SBERT as the reward model, we compare it with a popular rule-based reward approach. Specifically, we adopt the average F1 score of ROUGE‑1, ROUGE‑2, and ROUGE‑L as the reward. The results are shown in the table below.

As shown in the table, SBERT consistently outperforms the rule-based reward in our experimental setting. This improvement is likely due to SBERT’s ability to capture semantic similarity beyond exact token overlap, which is especially important for evaluating open-ended answers where lexical variance is common.

Hyperparameter $\tau$ Analysis

We first clarify that we set $\tau$ based on a reasonable default, and we humbly argue that introducing new hyperparameters is inevitable when proposing a new framework. Furthermore, we conduct an ablation study to show the sensitivity of $\tau$, and the results are presented in the table below.

The results demonstrate that the method’s performance remains stable over a reasonable range of $\tau$ values and all of the choices of $\tau$ outperform the baseline. These indicate robustness to the hyperparameter choice.

---

We sincerely welcome the opportunity to further discuss our work during the discussion phase and thank the reviewers for their valuable feedback.