Grounded Reinforcement Learning for Visual Reasoning

Thank you for your thoughtful review. We were pleased that you recognized our work's novelty in attaching grounding signals during reasoning stages, appreciated our clear presentation, and found our contribution to the visual grounding field significant.

We have addressed concerns through additional experiments and analysis. To summarize: (1) we conducted reward ablations, (2) we validated the contribution of multi-turn visual feedback on V*Bench, (3) we clarify our state-of-the-art performance, (4) we clarify our OOD generalization, (5) we empirically validated our binary reward design over Euclidean distance, (6) compared adaptive vs. fixed cropping with experiments, and (7) we added teacher model comparisons.

We present our answers in detail below and hope we’ve addressed your concerns. Please let us know if anything remains unclear. We would be happy to provide further clarification.

Reviewer comment: Reward ablation study should be provided.

We thank the reviewer for this valuable suggestion. We have ablated all components of our reward formulations in Tables R1 and R2 below.

Specifically:

• Removing the multi-turn diversity bonus reward leads to a decrease of 3.2% on V*bench. Appendix Figure A1 shows that removing this term leads to turn collapse.
• Removing the format reward leads to a decrease of 0.6% on SAT-2.
• Removing the accuracy reward leads to a decrease of 2.3% on SAT-2.

We will include these ablations in Section 5.2 of the revised manuscript.

Table R1: Multi-turn ablations

Table R2: Additional reward ablations

Reviewer comment: The crop size for multi-turn GRPO is fixed. How to deal with irregular shapes?

We conducted additional experiments where both MCTS warm-start and RL phases in ViGoRL use predicted bounding boxes (adaptive crops) instead of fixed-size point-based crops.

As shown in Table R1 above, we observe no difference in V*Bench accuracy between the adaptive bounding box and fixed-point cropping approaches. We attribute this to two main reasons:

1. Sufficient context from fixed crops: The default crop size is typically large enough to capture both the target region and its surrounding context at a high resolution for most V*Bench examples.

2. Multi-crop coverage of larger objects: When the region of interest is too large for a single fixed crop, ViGoRL samples multiple overlapping crops in the vicinity, analogous to human multiple fixations, which suffices to infer the correct answer.

While adaptive cropping could, in principle, be more efficient, we encountered practical limitations:

• Bounding box predictions from Qwen are often imprecise.
• Inconsistent aspect ratios from adaptive crops can introduce training variability.

We agree this is a valuable design dimension and plan to further explore trade-offs in future work.

Reviewer comment: Why not use the euclidean distance between the predicted coordinates and center of bounding box?

We thank the reviewer for the suggestion. We tested the Euclidean distance reward and found it significantly reduced performance: -2.2% on ScreenSpot-V2 and -0.7% on ScreenSpot-Pro (Table R3). We will include this in the revision.

Web elements vary in size, so a distance-based reward unfairly penalizes valid clicks near the edges of large elements. For instance, clicking near the edge of an 800×60 navigation bar is functionally correct but would receive a low euclidean reward.

Table R3: Euclidean vs. binary reward ablation

Reviewer comment: The contribution of multi-turn visual feedback is unclear.

We find that multi-turn visual feedback is especially valuable when the model cannot perceive fine-grained image details in the initial input resolution. ScreenSpot-V2 has large, visible elements, so feedback offers no gain (–0.4%), while ScreenSpot-Pro features smaller, high-resolution targets, where feedback improves performance (+1.2%).

To validate this claim further, we evaluated ViGoRL on V*Bench in Table R4, which requires identifying small details in ultra high-resolution images. Multi-turn ViGoRL outperforms the single-turn variant by 2.1%, due to better disambiguating details via the zoomed-in crops. This reinforces that visual feedback is crucial when the global image tokenization blurs away fine-grained cues.

Table R4: Accuracy for visual search on V*Bench

Visual search training data: We curate 11k question-answer pairs over small objects from Segment Anything, using GPT-4o given an object mask and scripted filtering to generate fine-grained ⟨image, question, choices, answer⟩ tuples.

Reviewer comment: SOTAs performance is not demonstrated in this work.

To the best of our knowledge, ViGoRL matches or exceeds prior SOTA results across several key benchmarks, as outlined below. If we have overlooked a relevant baseline, we would welcome including it in additional comparisons.

That said, our core contribution extends beyond outperforming existing models: we demonstrate that grounded reasoning provides consistent gains over standard SFT and vanilla (ungrounded) GRPO across diverse reasoning domains, supported by extensive ablations and behavioral analyses.

• VisualWebArena: Our 7B ViGoRL model outperforms the previous 7B SOTA (ICAL) by +3.0%, without relying on human reasoning annotations or HTML inputs, which ICAL requires.
• V*Bench: As shown in Table R4 above, ViGoRL establishes SOTA among 7B-scale reasoning models (as of May 1, 2025).
• Web Grounding & Spatial Reasoning: ViGoRL approaches the performance of leading commercial models such as Kimi-VL, while being fully open-source. We release all code, models, and data in the submission supplemental.

Reviewer comment: OOD generalization should be further improved.

ViGoRL achieves strong OOD generalization compared to SFT and vanilla GRPO baselines. That said, we will add more comparisons or specific experiments if the reviewer has suggestions.

We further validated OOD generalization on benchmarks with minimal training overlap. Table R5 below (also in Supplemental Table A7) shows ViGoRL maintains or improves performance on OOD benchmarks (MMMU, RealWorldQA, V*Bench), addressing concerns about overfitting or negative transfer. We discuss potential extensions for even stronger OOD performance in Section 6 and limitations.

Table R5: Accuracy on Out-of-Distribution (OOD) Benchmarks

Reviewer comment: It is required to provide to the zero-shot performance of the teacher model.

We have added the zero-shot performance of our teacher model (Qwen2.5-VL-72B) in the revised manuscript, with results provided in Table R6 below. Despite using only 1k examples for distillation before RL, ViGoRL-7B outperforms Qwen2.5-VL-72B by 6.0% on SAT-2 while being 10× smaller, and closes the gap across most benchmarks.

Table R6: Comparison to Qwen2.5-72B-VL

Thank you for your thoughtful review and for recognizing our work's novel approach to grounding chain-of-thought reasoning in visual scenes, finding it "crucial and practical for multimodal reasoning tasks." We appreciate your positive assessment of our multi-turn reinforcement learning scheme and data generation pipeline as reasonable and well-aligned with practical needs.

We have addressed your concerns through additional experiments and analysis: (1) We conducted ablations showing that removing textual reasoning causes a performance drop, demonstrating that visual and textual CoTs serve complementary roles, (2) We clarified that our training considers seperated datasets, with experiments showing minimal impact from dataset mixing, and (3) We performed comprehensive reward ablations revealing that each component contributes meaningfully to overall performance.

We present our answers in detail below and hope we’ve addressed your concerns. Please let us know if anything remains unclear. We would be happy to provide further clarification.

Reviewer comment: Previous work, like Argus (CVPR’25), also aims to adopt visual CoT to enhance the capability of MLLMs for visual-centric tasks. What is the motivation for jointly considering the visual and textual thoughts instead of solely adopting visual CoT for visual tasks?

Thank you for the thoughtful question. To directly test the role of textual reasoning, we ran an ablation in which we removed all textual thoughts during both the SFT and RL stages, effectively reducing it to a visual CoT-style agent. As shown in Figure R2 below, this visual-only CoT resulted in a performance drop of 1.2% on V∗Bench, indicating that grounding alone, while helpful, does not fully substitute for the explicit composition and referencing afforded by textual reasoning.

Our motivation for jointly using both visual and textual chains of thought stems from the observation that they serve complementary roles. Visual grounding and cropping provide perceptual feedback -- what the model “sees” -- while textual reasoning supports binding and compositional inference over these grounded regions. Much as in human cognition, this separation allows the model not only to explore regions but also to integrate and manipulate visual evidence in a structured way, yielding more interpretable and accurate reasoning trajectories.

That said, we agree with the reviewer’s broader point: we do not claim that explicit natural language reasoning is the only or necessary mechanism for visual region binding and manipulation. In the long term, latent reasoning may serve similar purposes with less reliance on textual supervision (e.g., [1]). Nonetheless, our approach offers a practical scaffold for structured supervision, demonstrates the value of grounded reasoning, and delivers immediate performance gains. We will clarify this distinction in the revised discussion and better articulate why language remains a practical and interpretable scaffold in our current formulation.

[1] Hao et al. (2024), Training Large Language Models to Reason in a Continuous Latent Space

Table R2: Multi-turn ablations

Reviewer comment: Why would directly combining data sources of spatial reasoning and web understanding for RL training benefit all tasks?

Thank you for this question about our training strategy. To clarify, we do not combine datasets during training and each model is trained on task-specific data. We will revise lines 263-270 to clarify this point.

To quantify the effect of combining data, we conducted an experiment combining spatial and web datasets during both warm-start and RL phases. The results, shown in Table R3, reveal:

• Web grounding: Small decrease (-0.2% on ScreenSpot-Pro, -0.3% on ScreenSpot-V2)
• Spatial reasoning: Improvement of +1.1% on SAT-2
• Both approaches significantly outperform vanilla GRPO (+12-15% SAT-2)

These findings indicate that combining data has small effects on relative performance gains from our method, with slight benefits for spatial reasoning at the cost of web grounding performance.

Table R3: Training with combined vs. separate datasets

Reviewer comment: What are the contributions and effectiveness of each proposed reward item?

We thank the reviewer for this valuable suggestion. We have ablated all components of our reward formulations in Tables R4 and R5 below.

Specifically:

• Removing the multi-turn diversity bonus reward leads to a decrease of 3.2% on V*bench. Appendix Figure A1 shows that removing this term leads to turn collapse.
• Removing the format reward leads to a decrease of 0.6% on SAT-2.
• Removing the accuracy reward leads to a decrease of 2.3% on SAT-2.

We will include these ablations in Section 5.2 of the revised manuscript.

Table R4: Multi-turn reward ablation

Table R5: Additional reward ablations

Thank you for your thoughtful review and for recognizing our work as well-motivated, clearly written, and technically solid. We appreciate your positive assessment of our grounded reasoning approach and its benefits to the community.

We have carefully tried to address your concerns through additional experiments and analysis: (1) We clarify ViGoRL's scope, (2) We justify our choice of Qwen-2.5-VL, (3) We add comprehensive error analysis, (4) We provide ablation results on our diversity bonus, and (5) We detail computational costs of our SFT component.

We present our answers in detail below and hope we’ve addressed your concerns. Please let us know if anything remains unclear. We would be happy to provide further clarification.

Reviewer comment: The problem scope of the method is not clearly defined. Training may degrade performance on other tasks.

We will clarify ViGoRL's scope in Section 3.1: “ViGoRL targets improved performance on specific visual reasoning domains (e.g., spatial understanding, web element grounding) by fine-tuning domain-relevant reasoning skills, while preserving performance on related tasks by leveraging transferable grounding and reasoning mechanisms.”

To further clarify and verify the scope:

• Table R1 (also in Supplemental Table A7) shows ViGoRL maintains or improves performance on out-of-domain benchmarks (MMMU, RealWorldQA, V*Bench), suggesting that its grounded reasoning strategies are transferable and do not result in overfitting or negative transfer.

• In Table R2, we also evaluate mixed-task training (spatial + web) and find only minor trade-offs, indicating that ViGoRL’s improvements are not narrowly scoped to a single domain.

• As suggested by reviewer vmK2, we train and evaluate on the open-ended domain of V*Bench, finding ViGoRL significantly outperforms strong baselines in this additional domain (Table R3).

Table R1: Accuracy on Out-of-Distribution (OOD) Benchmarks

Table R2: Training with combined vs. separate datasets

Table R3: Accuracy for visual search on V*Bench

Visual search training data details: We curate 11k question-answer pairs over small objects from Segment Anything, using GPT-4o given an object mask and scripted filtering to generate fine-grained ⟨image, question, choices, answer⟩ tuples.

Reviewer comment: The authors should justify using Qwen-2.5-VL as a design choice.

Thank you for raising this important point about model selection. We chose Qwen2.5-VL as our primary base model based on both its technical and practical suitability for our setup. Qwen2.5-VL was one of the strongest open-source VLMs available at the time of our experiments, with strong visual grounding capabilities already incorporated during pretraining. It also comes in multiple sizes (3B, 7B, 72B), which enabled controlled scaling and distillation within a consistent architecture and tokenizer.

To address your concern about evaluating other models, we are currently running our pipeline with Kimi-VL using the EasyR1 framework, and these results will be included in the final release.

Reviewer comment: The paper lacks discussion of failure cases.

Thank you for the suggestion. We now include Table R4, which categorizes SAT-2 errors. 60% arise from perceptual or task-level challenges, while 40% reflect residual reasoning flaws (e.g., overthinking, truncation, hallucination). We believe scaling or denser rewards could address aspects of both, with further discussion in Section 6 and limitations. We will include this analysis and additional qualitative failure cases in the revised manuscript.

Table R4: ViGoRL Error Analysis by Type on SAT-2

Example R1: Overthinking

Question: “Is there a pot in the image?”

Image: <Image of stove with a large pot and pan on it>

Model response: <think> To determine if there's a pot, I'll check the regions where pots typically appear. At (300, 600), there's a pot-like object with a handle and a lid, suggesting it could be a pot. To confirm the presence of a pot, I'll check another region. At (700, 1000), there's no pot-like object present, indicating that the previous observation was accurate. Now, I'll verify if there are any other potential pots in the image. At (500, 800), there's no pot-like object present, which supports the previous findings. Next, let's examine the area at (400, 900) to ensure there's no hidden pot. There's still no pot-like object visible here, reinforcing the previous observations. I'll now check the lower region at (600, 1200) to confirm there are no additional pots. At (600, 1200), no pot-like object is visible here, further supporting the absence of a pot in the image. Lastly, I'll check the upper region at (300, 200) to ensure there's no hidden pot. At (300, 200), no pot-like object is visible here, confirming the pot is at (300, 600) in the image. </think>

Reviewer comment: How was diversity bonus determined?

The diversity bonus is treated as a hyperparameter. We ablate the diversity bonus in Table R5. Removing the multi-turn diversity bonus reward leads to a decrease of 3.2% on V*bench. We found 0.2 to encourage turn exploration while not overpowering accuracy reward.

Table R5: Multi-turn reward ablation

Reviewer comment: How expensive is the introduced SFT component?

The introduced SFT stage, including MCTS-based data generation and supervised finetuning, incurred approximately 224 GPU-hours (192 for MCTS, 32 for SFT), while the RL stage required 576 GPU-hours. Thus, SFT accounts for ~39% of the RL computational cost. It is intentially lightweight (1.5k prompts) and provides structured, grounded reasoning behaviors that RL alone fails to induce (see Table 4 and Section 5.3).

Incurring additional resources for SFT before RL is in-fact common among post-training recipes [1,2]. However, unlike prior warm-start approaches that rely on large-scale human prompt curatation or annotations [1,2], our warm start uses model+search to generate reasoning traces without human supervision.

[1] DeepSeek-AI. (2025), DeepSeek-R1

[2] Kimi Team (2025), Kimi-VL Technical Report

Table 2 is referenced before Table 1.

Thank you for the suggestions. We will revise the text to reference tables in the correct order for readability.

Limitations should be addressed in the main text.

We will move the key limitations from the appendix to the main text to ensure they are clearly visible to readers.

Thank you for your thoughtful feedback. We were encouraged you found our work novel with compelling theoretical grounding in cognitive science, appreciated our "exceptionally thorough behavioral analysis" as a "significant methodological contribution", and recognized our comprehensive experimental design and "impressive" empirical results with consistent improvements across all benchmarks.

We have carefully addressed your concerns through additional experiments and analysis. To summarize: (1) we tested ViGoRL on V*Bench as suggested, finding it significantly outperforms strong baselines, (2) compared adaptive vs. fixed cropping and found no performance difference, with further discussion on each approach, (3) we provide further justification and discussion on intermediate reward and mitigation of reward hacking, and (4) we clarify when grounded reasoning may be unnecessary.

We present our answers in detail below and hope we’ve addressed your concerns. Please let us know if anything remains unclear. We would be happy to provide further clarification.

Reviewer comment: Most evaluations are on relatively structured tasks (spatial reasoning, GUI interaction). How's the performance on V* benchmark?

Thank you for raising this point about testing the method on more open-ended visual reasoning tasks. As suggested, we have now run ViGoRL on V*Bench, which tests open-ended visual search including open-ended question answering about fine-grained image details.

We show the results in Table R1. Our key findings are:

• ViGoRL-7B achieves 86.4%, outperforming open-source reasoning models (as of May 1, 2025) and surpassing GPT-4o by 20.4%.
• Even our 3B model reaches 81.2%, matching sophisticated tool-using pipelines like IVM-Enhanced GPT-4V
• Multi-turn ViGoRL improves performance over the single-turn variant by 2.1%, demonstrating the utility of multi-turn zoomed-in feedback in this visual search settings.

This strong performance on visual search reasoning tasks demonstrates that grounded reasoning generalizes beyond spatial and web domains.

Table R1: Accuracy for visual search on V*Bench

Visual search RL training data: We curate 11k question-answer pairs over small objects from Segment Anything, using GPT-4o given an object mask and scripted filtering to generate fine-grained ⟨image, question, choices, answer⟩ tuples.

Reviewer comment: Why fixed crop size?

Thank you for this insightful question regarding adaptive cropping. We conducted an additional experiment where both MCTS warm-start and RL phases in ViGoRL use predicted bounding boxes (adaptive crops) instead of fixed-size point-based crops.

As shown in Table R2, we observe no difference in V*Bench accuracy between the adaptive bounding box and fixed-point cropping approaches. We attribute this to two main reasons:

1. Sufficient context from fixed crops: The default crop size is typically large enough to capture both the target region and its surrounding context at a high resolution for most V*Bench examples.

2. Multi-crop coverage of larger objects: When the region of interest is too large for a single fixed crop, ViGoRL naturally samples multiple overlapping crops in the vicinity, analogous to human visual exploration through multiple fixations. A combination of these partial crops often suffices to infer the correct answer, especially when supported by a global view.

While adaptive cropping could, in principle, be more efficient, we encountered practical limitations:

• Bounding box predictions from Qwen are often imprecise.
• Inconsistent aspect ratios from adaptive crops can introduce variability during training.

We agree this is a valuable design dimension and plan to further explore trade-offs between adaptive and fixed cropping in future work.

Table R2: Additional multi-turn ablations

Reviewer comment: The reliance on final-answer rewards may lead to reward hacking, as acknowledged by the authors. Have you experimented with intermediate rewards for reasoning step quality or grounding accuracy? How might this affect the training dynamics?

We agree that relying solely on final-answer rewards introduces a risk of reward hacking, and we explicitly acknowledge this limitation. To mitigate this, we use a MCTS-based warm start, a KL penalty, and a structured format rewards that enforce correct tag usage and require valid grounding coordinates, acting as a minimal but effective constraint to maintain warm-start grounded outputs during training.

To assess the quality of intermediate reasoning, we conducted a human evaluation (Section 5.4), where annotators judged reasoning steps to be both spatially accurate and helpful for understanding model behavior, demonstrating the effectiveness of our minimal design.

We agree that incorporating more nuanced intermediate rewards (e.g., region relevance or semantic alignment) is a promising direction, but non-trivial: defining automated signals that robustly reflect grounding “quality” remains an open research challenge. These additional signals would require additional supervision such as human data (e.g., eye gaze) or specialized judge models. We plan to explore this in future work and will clarify this in the main paper.

Reviewer comment: While the paper shows when grounding helps, there's insufficient analysis of when it might hurt or be unnecessary. Some tasks may not benefit from explicit spatial grounding.

We appreciate the reviewer’s point. We agree that spatial grounding may not always be necessary, particularly for tasks dominated by global scene cues or simple object recognition. Prior work shows that vision-language models like GPT-4V can perform well on such tasks without explicit region-level reasoning [1].

However, a growing body of work highlights that grounding becomes helpful for tasks involving fine-grained spatial reference, ambiguity resolution, or multi-step visual reasoning. For instance:

• Grounding improves spatial reasoning accuracy, especially when simple visual attention fails to resolve spatial relations [2].
• Referring expression grounding benefits substantially from localized object anchoring [3].
• Explicitly prompting for region references improves performance on spatial QA and tool-use tasks [4].
• Prompting grounded intermediate steps boosts both interpretability and accuracy across visual tasks [5].

We provide an example below (Example R1) where our ViGoRL model does unnecessary grounding and verification for a recognition task. While not harming performance, this behavior may lead to excessive computation. We note that this overthinking occurs with existing reasoning models [6].

We will revise Section 6 to clarify when spatial grounding is particularly helpful, and add additional qualitative results in the appendix.

[1] OpenAI, GPT-4V System Card, 2023.

[2] Liu et al., What Makes Spatial Reasoning Hard for VLMs?, 2023.

[3] Chen et al., RefCLIP: Referring with Class Name Aware Vision-Language Models, 2022.

[4] Wu et al., V*Bench: Evaluating Tool-augmented VLMs, 2023.

[5] Yang et al., Thinking in Pictures, 2024.

[6] Chen, X. Do not think that much for 2+3=? On the overthinking of o1-like LLMs. 2025.

Example R1: Overthinking

Question: “Is there a pot in the image?”

Image: <Image of stove with a large pot and pan on it>

Model response: <think> To determine if there's a pot, I'll check the regions where pots typically appear. At (300, 600), there's a pot-like object with a handle and a lid, suggesting it could be a pot. To confirm the presence of a pot, I'll check another region. At (700, 1000), there's no pot-like object present, indicating that the previous observation was accurate. Now, I'll verify if there are any other potential pots in the image. At (500, 800), there's no pot-like object present, which supports the previous findings. Next, let's examine the area at (400, 900) to ensure there's no hidden pot. There's still no pot-like object visible here, reinforcing the previous observations. I'll now check the lower region at (600, 1200) to confirm there are no additional pots. At (600, 1200), no pot-like object is visible here, further supporting the absence of a pot in the image. Lastly, I'll check the upper region at (300, 200) to ensure there's no hidden pot. At (300, 200), no pot-like object is visible here, confirming the pot is at (300, 600) in the image. </think>