OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles

Thank you for the detailed feedback. Please find our responses below. We hope that our clarifications and additional experiments resolve the concerns and any potential confusions.

---

### W1: Why term the process “iterative self‑improvement” when each cycle trains a fresh model from the base checkpoint? Please justify full restarts versus continuing fine‑tuning and explain any resulting gains.

Thank you for the suggestion. As clarified in footnote 2 (page 6, line 177) of our manuscript, restarting training from scratch at each iteration is a standard practice in iterative self-improvement methods [1, 2]. This design choice ensures training stability and prevents overfitting, thus maintaining better generalization to unseen tasks. Notably, ReST^EM [1] explicitly highlights that re-training from the base model provides comparable task-specific performance and significantly better transfer to held-out tasks compared to continued training (Section 3, page 5, Differences with ReST). We emphasize that, in our iterative re-training approach from the base model, the data is iteratively refined and improved across iterations.

To further substantiate this design choice, we conducted an additional experiment comparing our approach of re-training from scratch versus continued training from the previous checkpoint. We evaluated these approaches on three representative benchmarks: MathVista (math reasoning), EMMA (general reasoning), and HallusionBench (perception).

The results indicate that continued training from the previous iteration leads to noticeable performance degradation on HallusionBench, despite similar performance on MathVista and EMMA. We will emphasize this in our revision by elevating the footnote into the main text and including these additional results.

[1] Beyond human data: Scaling self-training for problem-solving with language models

[2] V-star: Training verifiers for self-taught reasoners

[3] Reinforced self-training (rest) for language modeling

---

### W2: Can you supply evidence that SFT’s “action‑highlighting” effect is mechanistically distinct from a simple good initialization for RL

We appreciate your comment and realize there might be a misunderstanding. We clarify that our arguments explicitly differentiate between two mechanistically distinct roles that SFT plays:

1. Highlighting reasoning actions: SFT explicitly promotes specific reasoning behaviors, such as reflections and verifications, by emphasizing reasoning-trigger keywords. Crucially, without prior highlighting from SFT or inherent reasoning tendencies from the base model, RL alone does not consistently induce these behaviors spontaneously for VL models. To substantiate this, we analyzed the keyword frequency of the RL-trained MM-Eureka (7B) model across 1,000 MathVista examples: (a) "wait": 0 occurrences, (b) "check": 52 occurrences, (c) "maybe": 0 occurrences. On the contrary, SFT effectively and efficiently influence reasoning behavior patterns, distinct from mere performance benefits.

2. Initialization for RL: Independently, the quality of SFT-generated data directly affects the model’s initial performance, facilitating more effective subsequent RL training. This quality-driven initialization is precisely the motivation behind our iterative training approach, which progressively refines the quality of SFT data while retaining the desired reasoning behaviors.

As demonstrated in Figure 6 of our manuscript, iteration 1’s SFT does not provide superior initialization in terms of accuracy; in fact, its performance is lower than the base model. Thus, the primary contribution of SFT at this iteration is not performance initialization but rather the clear distillation of reasoning behaviors, evidenced by significantly increased frequencies of reflective keywords. These behaviors persist through subsequent RL training, remaining stable while RL enhances overall model performance. Consequently, the RL-enhanced model serves as an improved reasoning demonstration source for the following iteration, retaining desired behaviors while benefiting from RL-driven performance improvements.

---

W3: Reconcile with VLAA‑Thinker’s reports that (a) the same base model already shows reflective reasoning and (b) SFT can create harmful pseudo‑reasoning paths?

Thanks for the question. We believe this perceived contradiction also roots from the misunderstanding of our iterative approach as well as our arguments toward the role of SFT. We clarify that our findings align rather than conflict with concurrent papers.

(a) We must note that several concurrent works have made similar observations as ours. VL-Rethinker [1] explicitly noted that “complex, deliberate thinking patterns, such as explicit self-correction, did not consistently emerge as a direct result of standard RL on VLMs.” This is exactly what motivated their approach in forcing the model to rethink during RL by explicitly appending the keyword “wait”. Similarly, our findings suggest these reflective behaviors are latent and require targeted approaches (like iterative SFT-RL cycles) to reliably surface and refine.

(b) We fully acknowledge VLAA-Thinker's observation regarding harmful pseudo-reasoning paths caused by SFT. Our manuscript explicitly states (lines 47–48) that CoT traces from initial SFT provide demonstrations of reasoning actions but do not immediately improve LVLM’s accuracy Additionally, we highlight (line 54) that the initial SFT step "leads to a performance drop", which is consistent with VLAA-Thinker. In Appendix C (page 18), we specifically illustrate examples of ineffective, repetitive reasoning behaviors post-SFT. Precisely this harmful initial impact motivated our iterative approach: each RL stage refines these reasoning behaviors, generating progressively improved data for subsequent SFT steps.

In short, our iterative SFT-RL methodology explicitly addresses and leverages the phenomena described by VLAA-Thinker, rather than contradicting them.

[1] VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

---

### W4: By what specific mechanism does GRPO repair the imprecise visual grounding introduced by caption‑based CoTs?

In the paper, we mentioned the imprecise visual grounding in the context of the very first SFT stage, where reasoning traces are derived from text-based LLMs (QwQ-32B) conditioned on captions generated by a vision-language model (Qwen2.5-VL-7B). Since captions inherently lose some visual information and precision, the reasoning traces from QwQ-32B may contain inferences or educated guesses based on incomplete information. For example, consider the reasoning of QwQ-32B below, based solely on captioned inputs:

Question: What is the main color of the can's top surface?

QwQ-32B: [think]\nOkay, let's tackle this question. The user is asking for the main color of the can's top surface based on the image provided. First, I need to go through the image carefully to find any shows of colors.\n\nLooking at the image details: The can's top is shown with a pull-tab mechanism. The surface is on a dark wooden table. The pull-tab is in the center with a metallic ring. The instructions on the tab are shown, but no color is specified for the text. …... Metals can be silver, but maybe the overall top is a different color.\n[/think]\n\n[answer]Silver[/answer]

GRPO improves upon this by progressively shifting the model away from caption-based inference (that it learns during SFT iteration 1) toward reasoning on direct visual input, as it allows the model explore better reasoning paths with verifiable reward. Consequently, OpenVLThinker after iteration 1 already reasons directly from visual inputs, substantially reducing the potential ambiguity in the reasoning traces it generates as compared to the QwQ generated data. Further iterations continue refining the ”teacher model” for next iteration, systematically improving reasoning demonstrations while retaining beneficial reasoning behaviors demonstrated by QwQ-32B.

We will emphasize this point in our next revision to avoid confusion.

Thank you very much for your support, careful reading and the constructive feedback that helped us improve our work. Please see our detailed response with additional experiments below.

W1. Discuss the overall computational cost of the iterative SFT → RL loop (e.g., FLOPs or GPU‑hours)

Thanks for pointing this out. In appendix B, we roughly estimated the training time on 2-4 GPUs. Here, we rigorously detail the GPU hours for the SFT and RL processes below with a 8xH100 (or equivalent) node:

The computational cost of the SFT stage is minimal, primarily due to the small dataset size of 3k examples. For RL with medium-difficulty data, we limit the number of training epochs to further manage resource usage effectively. Although the RL stage using hard data (6k examples) incurs the highest computational cost, our method's overall compute demand remains comparable to contemporary RL-based approaches. Crucially, the preceding SFT and medium RL stages expedite convergence during the final RL (hard) stage, and the relatively smaller dataset size reduces the overall computational burden. Nevertheless, we acknowledge that iterative approaches inherently introduce additional computational overhead, and we will explicitly note this in our revised manuscript and update the more precise GPU hours estimate.

---

W2. Add single‑stage SFT‑only and RL‑only (trained on the same 12 K examples) to show that iteration, not merely using both techniques, brings the lift

Thank you for suggesting these critical baselines. We conducted the requested experiments under the following settings:

• RL-only baseline: Qwen2.5-VL trained solely with GRPO on the entire 12K dataset used in OpenVLThinker.
• SFT-only baseline: Qwen2.5-VL trained solely with SFT on iteration 2 trajectories generated by OpenVLThinker for the same 12K dataset. Similar to the original setup, we filter out examples where the model fails to reach a correct answer within k=4 samplings.

Both baselines were trained until full convergence, with checkpoints selected based on validation performance. In comparison, OpenVLThinker involves both SFT and RL stages starting from the same Qwen2.5-VL initialization.

Training RL-only on the complete 12K dataset required approximately 16 hours to finish using an 8xH100 (or equivalent) node, comparable to the cumulative training time across the three iterations of OpenVLThinker. Results indicate that the RL-only approach outperforms the SFT-only baseline, especially on held-out benchmarks like HallusionBench. Nevertheless, OpenVLThinker achieves superior performance compared to both baselines.

---

W3: How does the quality of the caption‑based SFT data affect final accuracy, and can you provide a deeper analysis on the caption quality?

Our training data from QwQ-32B was selected based on final-answer correctness. Thus, the quality of captions directly influenced the quantity and richness of the initial dataset available for the first-stage SFT. Higher-quality captions increased the likelihood that the text-based reasoning model would correctly derive final answers instead of random guessing due to lack of information, thus expanding the pool of useful training samples for iteration 1.

In the original setup, we employed Qwen2.5-VL-7B to generate captions and performed rejection sampling from QwQ-32B with k=4 attempts, discarding instances if none yielded correct reasoning. To provide deeper insights, we conducted additional experiments using Qwen2.5-VL-3B, a weaker captioning model, under identical rejection sampling conditions (k=4). The comparative performance on MathVista is detailed below:

As demonstrated, better caption quality provides more precise visual grounding at the very first data generation step, enabling the reasoning model to produce more accurate and reliable traces. This in turn improves the initial training dataset and enhances the model's reasoning capabilities across iterations. Eventually, the initial data quality orthogonally complements our iterative training framework, positively influencing final model performance.

---

W4: Does the method transfer to architectures outside the Qwen family?

Thank you for raising this important point. To ensure a fair comparison with existing baselines (all trained from Qwen2.5-VL-7B-Instruct), we adhered to the same experimental setting in our paper. Currently, open-source RL frameworks for vision-language models (EasyR1, OpenRLHF, VeRL) only support models within the Qwen-VL family. Extending these frameworks to other architectures for GRPO is a non-trivial effort beyond the scope of our rebuttal timeframe. To demonstrate that our iteratively evolved dataset maintains its effectiveness beyond the Qwen-based models, we have conducted additional experiments validating the iterative SFT data (distillation) with Gemma-3-4B-IT at iterations 1, 2, and 3.

We also acknowledge the findings presented in [1] and agree that base-model capabilities significantly influence RL (GRPO) performance due to variations in pre-training quality. Consistent with this idea, we highlight SFT’s crucial role as an intermediate training step (priming) that explicitly surfaces beneficial reasoning behaviors, thereby enhancing subsequent RL effectiveness. We will incorporate this discussion with [1] into our manuscript revision.

[1] Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

---

Q1: The paper mentions an EM‑style perspective; could you formalize this connection?

As similar to ReST^EM (an iterative SFT algorithm [1]), our method also decouples data collection (E-step) and policy optimization (M-step) in one iteration. To put this formally,

• Generate (E-step): We generate a SFT dataset $D_i$ at iteration i by sampling reasoning traces from the current policy model $p$: $𝒚^{i} \sim p_{\theta} (𝒚 |𝒙_{i})$

• Improve (M-step): We leverage the new dataset $D_i$ obtained from the Generate step to improve the policy model p via minimzing the SFT (log-likelihood) loss: $𝐽(\theta) = 𝔼(𝒙,𝒚)∼D_𝑖 [𝑟(𝒙, 𝒚) \log 𝑝_{\theta} ( 𝒚 |𝒙)]$. Additionally, the model is further improved through an unified RL step via the GRPO objective on a fixed dataset $D_{RL}$.

[1] Beyond human data: Scaling self-training for problem-solving with language models

---

Q2: Table 3 shows a sharp drop for vanilla SFT. What in the distilled reasoning traces causes this degradation, and how might it be mitigated?

The challenge of small-scale LLMs (<7B) struggling to learn effectively from lengthy chains-of-thought (CoTs) generated by larger models is common to both LVLMs and text-only LLMs. [1] specifically highlights that smaller models benefit more when learning from teachers of a similar scale, rather than substantially larger models. In our observations, repetitive or overly verbose reflections become reinforced during the SFT training, causing looping or redundant behaviors, as demonstrated in Appendix C.

For the vision-language tasks examined in our paper, the degradation is further caused by the information loss when relying solely on image captions rather than direct visual inputs. Consequently, the teacher model’s reasoning may rely on incomplete or ambiguous information. Even with post-processing, this issue notably impacts performance. For example, QwQ-32B attempts to reason from captions that lack crucial visual details:

Question: What is the main color of the can's top surface?

QwQ-32B: [think]\nOkay, let's tackle this question. The user is asking for the main color of the can's top surface based on the image provided. First, I need to go through the image carefully to find any shows of colors.\n\nLooking at the image details: The can's top is shown with a pull-tab mechanism. The surface is on a dark wooden table. The pull-tab is in the center with a metallic ring. The instructions on the tab are shown, but no color is specified for the text. …... Metals can be silver, but maybe the overall top is a different color.\n[/think]\n\n[answer]Silver[/answer]

The mitigation to such performance degradation is the exact motivation of our iterative approach. By iteratively refining the teacher model through cycles of SFT and RL, we produce progressively clearer and more precise reasoning demonstrations, which in turn yield higher-quality distillation data in subsequent iterations.

We appreciate your feedback and make clarifications with additional experiments as below to address the raised questions.

---

W1: How and why were the representative keywords selected.

We selected keyword candidates by analyzing word cloud visualizations generated from reasoning outputs across various checkpoints, including QwQ-32B. Final representative keywords were chosen based on their frequency in the statistics and further categorized through manual inspection. For instance, we categorized "maybe" as indicative of the action "seeking alternatives" due to its frequent co-occurrence with reflective reasoning patterns, such as "Maybe I should consider...", "Maybe I can consider..." or “Or, maybe, …”. The co-occurence frequency of “Maybe I” is 28.1% for OpenVLThinker at iteration 1 and 7.9% for QwQ-32B at data generation. Similarly, “identify” is frequently associated with “First, I need to identify …”

This manual selection approach follows established practices in recent reasoning-model literature [1-3]. Specifically, keywords such as "wait," "check," and "maybe" are recognized reasoning indicators in studies of text-based reasoning models. Notably, [1] references the keyword "maybe" in Appendix B, while [2,3] similarly identify keyword sets through manual analysis (detailed in their appendices).

Eventually, the effectiveness of particular keywords in triggering specific reasoning behaviors depends on the reasoning model (in our case, QwQ-32B). Therefore, we prioritized the most frequent keywords highlighted by the word cloud analysis. We will clarify this in our revision.

[1] LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!

[2] SEAL: Steerable Reasoning Calibration of Large Language Models for Free

[3] Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time

---

W2: Design choice on restarting training iteration needs explanation

Thank you for the suggestion. As clarified in footnote 2 (page 6, line 177) of our manuscript, restarting training from scratch at each iteration is a standard practice in iterative self-improvement methods [1, 2]. This design choice ensures training stability and prevents overfitting, thus maintaining better generalization to unseen tasks. Notably, ReST^EM [1] explicitly highlights that re-training from the base model provides comparable task-specific performance and significantly better transfer to held-out tasks compared to continued training (Section 3, page 5, Differences with ReST). We emphasize that, in our iterative re-training approach from the base model, the SFT data is refined and improved across iterations.

To further substantiate this design choice, we conducted an additional experiment comparing our approach of re-training from scratch versus continued training from the previous checkpoint. We evaluated these approaches on three representative benchmarks: MathVista (math reasoning), EMMA (general reasoning), and HallusionBench (perception reliability).

The results clearly indicate that continued training from the previous iteration leads to noticeable performance degradation on HallusionBench, despite similar performance on MathVista and EMMA. We will emphasize this in our revision by elevating the footnote into the main text and including these additional results.

[1] Beyond human data: Scaling self-training for problem-solving with language models

[2] V-star: Training verifiers for self-taught reasoners

[3] Reinforced self-training (rest) for language modeling

---

W3: Evaluation benchmarks are relatively limited

Thank you for your suggestion. In the paper, we selected six widely-used vision-language benchmarks covering diverse reasoning aspects, including mathematical reasoning, general reasoning, and perceptual reliability. These benchmarks are consistently employed in technical reports of proprietary models (e.g., GPT, Gemini) and open-source models (e.g., Qwen-VL, Intern-VL). Our evaluation setup further aligns with concurrent works [1,2] on vision-language reasoning, which similarly employ 5–7 benchmarks with a particular emphasis on mathematical reasoning.

Notably, our evaluation benchmarks are comprehensive and all span multiple evaluation subfields. We reported the overall performance to maintain conciseness. For example, EMMA evaluates Math, Chemistry, Physics, and Code through both multiple-choice and free-form questions. To provide clarity on evaluation coverage, we include detailed subset performances on EMMA below:

Additionally, in response to your concern, we have expanded our evaluation by including two recent benchmarks: MM-Star and WeMath (both released in 2024). MM-Star is particularly comprehensive, assessing six core LVLM capabilities: fine-grained perception, coarse perception, mathematics, science & technology, logical reasoning, and instance reasoning.

We believe our expanded evaluation across these eight comprehensive benchmarks sufficiently demonstrates the effectiveness and generalizability of our approach.

[1] Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

[2] SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models