Dear reviewer iMMK,

Thank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper.

Weakness 1

Thank you for the insightful suggestion. We add an additional ablation study in the table below. Specifically, No Reason denotes a setting where the model performs direct question answering without any explicit reasoning; Direct Prompt means directly prompting VLMs to reason step by step (as in [1]); Cos corresponds to our proposed method using step-level structured reasoning with fine-grained rewards. All methods are trained using the same SFT and RL datasets to ensure a fair comparison. For No Reason, we use only the question-answer pairs (without intermediate reasoning thought) from the ShareGPT-Step-300K dataset for SFT. For Direct Prompt, we concatenate all steps of reasoning (i.e., "thoughts") from each instance into a single reasoning sequence. Since both No Reason and Direct Prompt can not output strict step-level thought, we cannot assess their intermediate reasoning quality, so we apply the outcome reward (i.e., correctness of the final answer) during RL training.

There are two key observations: i) Models trained with reasoning-augmented SFT data (i.e., Direct Prompt and Cos) clearly outperform models trained with QA-only data (No Reason) on reasoning benchmarks; ii) Our Cos yields much greater improvements during RL compared to outcome-only reward settings. This ablation study directly supports our central motivation for fine-grained reasoning and strengthens the empirical foundation of our conclusions.

Weakness 2

We chose to use GPT-4o to construct the Share-GPT-300K dataset for three main reasons: i) The primary objective of our SFT stage is not to improve the model's reasoning ability. The core goal of this paper is to explore step-level reasoning driven by fine-grained reward signals. To enable this, we first require the model to generate reasoning in a structured, step-wise format. Therefore, the main purpose of constructing the SFT dataset is to guide the model towards producing step-level reasoning trajectories that facilitate fine-grained reward assessment. At this stage, the key requirement is to generate coherent and logically sound intermediate reasoning steps; ii) Ground-truth is provided as reference. In order to ensure high-quality intermediate reasoning, we provide ground-truth answers as references when prompting GPT. This fundamentally alters the difficulty of the task, making the difference of reasoning abilities between 4o and stronger reasoning models (e.g., o1) less important . In such a setup, instruction-following ability and output consistency become more critical. In our preliminary experiments, when GT answers were provided as hints, GPT-4o did not underperform GPT-o1 in terms of reasoning quality obviously. On the other hand, GPT-4o produced more stable outputs and demonstrated stronger adherence to the target format; iii) Money cost. Using GPT-o1 would be more than 10× more expensive than GPT-4o, making the cost of money unaffordable. We also use GPT-4o to generate process annotation for the latter two reasons—better instruction-following and significantly lower cost.

Weakness 3

Thanks for the suggestion. The reason why we use a 38B PRM is that we hope to provide as accurate process rewards as possible, so that our exploration and empirical studies on fine-grained reward driven reasoning can be more precise and solid. Therefore, we use the largest model supported by our computing resources as the PRM. To address your concern, we conduct a further ablation on the rewards in RL using the 8B-PRM (i.e., expanding Table 2 in the paper). Compared with 38B-PRM, 8B-PRM has a lower accuracy of steps and a higher accuracy of answers as shown in Table 3 in the paper. Despite its lower step accuracy, the results support the same conclusion: jointly optimizing rewards for both intermediate reasoning and final answers yields superior performance. Therefore, changing the PRM of 38B to 8B will not significantly undermine the performance in Table 1, nor will it affect our core conclusions and findings.

Question 1

Currently, we have not observed obvious reward hacking phenomenon in our experiments.

Question 2

While the results in Table 3 may appear counterintuitive, they are in fact consistent with the underlying training dynamics.

Notably, evaluating intermediate reasoning steps is substantially more challenging than evaluating the final answer for three reasons: i) Higher semantic complexity: Step-level evaluation requires a deeper and more holistic understanding of the question. The model must assess whether each step contributes constructively toward the final answer. Similarly, annotating process data is also more cognitively demanding for human annotators[2]. ii) More supervision signals. As shown in equation 1 in the paper, PRM is trained to minimize the sum of cross-entropy losses over all intermediate steps and the final answer. A complete reasoning trace contains multiple steps but only a single answer, thus accurately assessing all the intermediate steps is much harder than only assessing the answer. iii) Answer memorization. As highlighted in recent studies[3,4], LLMs often exhibit "answer hacking"—producing correct answers via memorized question or patterns rather than genuine understanding. In contrast, intermediate steps exhibit higher variance across instances and are less amenable to memorization.

As we know, neural networks are inherently short-cut learners[5]. During training, smaller models (i.e., 8B PRM) tend to prioritize optimizing for final answer accuracy, as it offers a more accessible gradient path to loss reduction. Improving step-level predictions poses a greater optimization challenge. The 38B PRM, with its higher capacity, is better able to model and optimize intermediate reasoning quality. Consequently, it achieves greater improvements in step-level accuracy, which more effectively reduces the total loss—resulting in the performance pattern observed in Table 3.

[1] Large language models are zero-shot reasoners.
[2] Let's Verify Step by Step.
[3] SoK: Memorization in General-Purpose Large Language Models.
[4] Rethinking Memorization in LLMs: On Learning by Rote vs. with Understanding.
[5] Shortcut Learning in Deep Neural Networks.

Hi, As the deadline of author-reviewer discussion is approaching, could you please check the authors' rebuttal and respond accordingly?

Thanks AC

Dear Reviewer iMMK,

Thanks for replying and raising your rating to 3. For your last remaining concern, there exists a key misunderstanding that we would like to clarify.

The step-acc v.s. answer-acc discrepancy (i.e., Table 3 in our paper) is a phenomenon specific to the PRM, while our core idea is studying the fine-grained reasoning of the actor model. Research on actor model and reward model belongs to two different subfields, and our work falls into the former. The PRM, as a reward model, can be viewed as a binary classification model that outputs the probability of a step or answer being labeled as "good" or "bad." Given that the PRM does not perform reasoning at all but merely outputs classification scores, we respectfully differ that its classification discrepancy could not contradict the actor model's reasoning pattern conclusion.

Actually, we found this phenomenon of PRM counter-intuitive but interesting, which is why we reported it in the paper for open discussion. In our response to Question 2, we provided our interpretation of this phenomenon. We understand that you may reckon the explanation not convincing enough, but what we want to emphasize is that this specific phenomenon of reward model does not contradict our core idea and conclusion about the actor model. As shown in the experiments (our response to weakness 3), using PRMs with accuracy discrepancy (i.e., 8B vs. 38B) does not alter the core findings and conclusions of our study. The important thing is to use PRM to provide rewards to both intermediate steps and the final answer, while minor differences in the step or answer accuracy are less important.

In summary, this paper is a study of actor model's fine-grained reasoning, and PRM is just a tool to provide reward signals. If you think that the report of this counter-intuitive phenomenon is likely to lead readers to similar misunderstandings, it's okay for us to delete Table 3 without affecting our core conclusions and findings.

Great thanks for your engagement and suggestions to improve our paper, and we sincerely hope the above clarification can eliminate your last remaining concern and raise to a positive rating. If there is anything further we can do or clarify, please let us know.

Dear reviewer zrV1,

Thank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper.

Weakness 1 & Question 1

We utilize GPT-4o in two components of our pipeline:

1. SFT dataset construction. A key goal of this work is to explore step-level reasoning driven by fine-grained rewards. To enable this, we first require the model to generate outputs in a structured, step-level format. The SFT dataset is constructed to guide the model toward this output format. At this stage, the primary requirement is that the teacher model produces reasonably coherent targeted step-level reasoning traces; high performance is not critical. Thus, GPT-4o can be replaced by a moderately capable open-source VLM (e.g., Qwen-VL) without significantly affecting overall performance, as the purpose of SFT is to induce the output structure rather than to boost reasoning quality.
2. Step-level process annotation. This component relies on GPT-4o to evaluate the quality of intermediate reasoning steps. This requires a model with strong reasoning capability, precise instruction-following, and the ability to produce consistent judgments in a target output format. In this case, replacing GPT-4o with weaker open-source models would substantially degrade annotation accuracy, which in turn would negatively impact the training of the PRM and compromise the reliability of the empirical findings.

Dear reviewer S6Gd,

Thank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper.

Weakness 1

For the technical novelty aspect, our key innovation lies in the step-level structured design, and the consequent fine-grained RL training and test-time scaling with step-level rewards. Prior attempts on structured CoT reasoning, such as LLaVA-CoT[1], explicitly decompose the reasoning into components like summary, caption, reason, and conclusion. However, similar to traditional CoT, these methods rely solely on the correctness of the final answer to evaluate the intermediate reasoning quality, which is clearly suboptimal. In contrast, the core motivation of our work is to enable accurate assessment of intermediate reasoning steps, thereby facilitating more effective RL training and test-time scaling. For example, the DPO used in Insight-V[2] select positive and negative sample pairs based only on the answer, while we choose high-quality reason paths based on PRM scores for every single step and the answer, making the training more effective as shown in the experiments. This step-level design is simple and straightforward, but it serves as the foundation for fine-grained evaluation of reasoning quality and distinguishes our method from prior approaches from the technical perspective.

For the contribution aspect, we propose a complete and transparent framework for studying the feasibility, empirical performance, and unique training/inference properties of fine-grained step-level reasoning chains. And we plan to release all components to the community to facilitate future research in fine-grained reasoning. In the revision, we will explicitly highlight the distinctions between technical novelty and contributions to help readers better appreciate the significance of our work.

Weakness 2

While the results in Table 3 may appear counterintuitive, they are in fact consistent with the underlying training dynamics.

Notably, evaluating intermediate reasoning steps is substantially more challenging than evaluating the final answer for three reasons: i) Higher semantic complexity: Step-level evaluation requires a deeper and more holistic understanding of the question. The model must assess whether each step contributes constructively toward the final answer. Similarly, annotating process data is also more cognitively demanding for human annotators[3]. ii) More supervision signals. As shown in equation 1 in the paper, PRM is trained to minimize the sum of cross-entropy losses over all intermediate steps and the final answer. A complete reasoning trace contains multiple steps but only a single answer, thus accurately assessing all the intermediate steps is much harder than only assessing the answer. iii) Answer memorization. As highlighted in recent studies[4,5], LLMs often exhibit "answer hacking"—producing correct answers via memorized question or patterns rather than genuine understanding. In contrast, intermediate steps exhibit higher variance across instances and are less amenable to memorization.

As we know, neural networks are inherently short-cut learners[6]. During training, smaller models (i.e., 8B PRM) tend to prioritize optimizing for final answer accuracy, as it offers a more accessible gradient path to loss reduction. Improving step-level predictions poses a greater optimization challenge. The 38B PRM, with its higher capacity, is better able to model and optimize intermediate reasoning quality. Consequently, it achieves greater improvements in step-level accuracy, which more effectively reduces the total loss—resulting in the performance pattern observed in Table 3.

To verify the consistency of our core finding in this paper, we add an additional experiment using the 8B-PRM. Compared with the of 38B-PRM, it has a lower accuracy of steps and a higher accuracy of answers. Despite its limited capacity, the results support the same conclusion: jointly optimizing rewards for both intermediate reasoning and final answers yields superior performance.

Weakness 3

Thanks for your suggestion. The core focus of our work is to investigate the impact of fine-grained rewards on reasoning performance—specifically, comparing step-level reasoning with fine-grained supervision against the traditional CoT paradigm with final-answer outcomes. The optimization of the RL algorithm itself is not the primary focus of this work. However, we agree that a more relevant baseline can strengthen this paper, thus we add a new experiment comparing against more relevant baselines as suggested.

Specifically, No Reason denotes a setting where the model performs direct question answering without any explicit reasoning; Direct Prompt means directly prompting VLMs to reason step by step (as in [1]); Cos corresponds to our proposed method using step-level structured reasoning with fine-grained rewards. All methods are trained using the same SFT and RL datasets to ensure a fair comparison. For No Reason, we use only the question-answer pairs (without intermediate reasoning thought) from the ShareGPT-Step-300K dataset for SFT. For Direct Prompt, we concatenate all steps of reasoning (i.e., "thoughts") from each instance into a single reasoning sequence. Since both No Reason and Direct Prompt can not output strict step-level thought, we cannot assess their intermediate reasoning quality, so we apply the outcome reward (i.e., correctness of the final answer) during RL training. We use LLaVA-NeXt as the base model. There are two key observations: i) Models trained with reasoning-augmented SFT data (i.e., Direct Prompt and Cos) clearly outperform models trained with QA-only data (No Reason) on reasoning benchmarks; ii) Our Cos yields much greater improvements during RL compared to outcome-only reward settings.

These additional results further support our central motivation: the step-level reasoning paradigm, combined with fine-grained rewards, consistently outperforms traditional CoT approaches that rely solely on final-answer supervision. Regarding RL algorithm, we agree that including experiments with GRPO or PPO would further strengthen the work. However, both GRPO and PPO require online deployment of the PRM, which is much more computationally expensive. In contrast, DPO offers a more resource-efficient alternative by enabling offline preference modeling using PRM. We plan to add a set of GRPO experiment in the final version to further validate our findings. Thanks for your understanding of the computational and time constraints.

Weakness 4

We would like to clarify that our intention is not to demonstrate that iterative DPO is necessarily better than SFT, actually they are both effective in enhancing VLM's reasoning abilities. We conduct experiments on two base models, the weaker LLaVA-NeXt and the stronger InternVL2.5-MPO. For LLaVA-NeXt, both SFT and iterative DPO lead to performance gains and SFT gains are better than iterative DPO gains (+8.7 v.s. +3.4), indicating that for relatively weaker VLMs, SFT is more effective and direct in enhancing the model's performance. In contrast, for the stronger InternVL2.5-MPO, the performance improvement from SFT alone is relatively limited (+1.6). However, after iterative DPO, the model can still achieve a notable performance boost (+2.4), reaching state-of-the-art. This indicates that for stronger VLMs, the benefits of further improvement through SFT are marginal, while RL is more essential and effective for enhancing strong model’s reasoning capabilities. From the perspective of significance, improving over a strong baseline is more difficult and meaningful than improving over a weak baseline. For the strong baseline, iterative DPO's gains exceed SFT's gains (+2.4 v.s. +1.6), demonstrating the effectiveness of fine-grained reward RL in improving the upper limit of reasoning ability.

[1] LLaVA-CoT: Let Vision Language Models Reason Step-by-Step.
[2] Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models.
[3] Let's Verify Step by Step.
[4] SoK: Memorization in General-Purpose Large Language Models.
[5] Rethinking Memorization in LLMs: On Learning by Rote vs. with Understanding.
[6] Shortcut Learning in Deep Neural Networks.

Hi, As the deadline of author-reviewer discussion is approaching, could you please check the authors' rebuttal and respond to it accordingly?

Thanks AC

Dear Reviewer S6Gd,

Great thanks for your time and suggestions! We will revise the paper as suggested in the final version.

Dear reviewer gk2j,

Thank you for your feedback and help in improving our work. Below we have responsed to every question individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper.

Question 1

We would like to highlight two key differences between our framework and prior reasoning methods in LLMs: i) Step-level reasoning and fine-grained rewards. Existing LLM (and VLM) approaches typically generate a complete reasoning chain and evaluate its quality based solely on the correctness of the final answer. For example, in LLM-based DPO, preference pairs are constructed based on final-answer's correctness. In contrast, our approach leverages a PRM to assign scores to intermediate reasoning steps and the final answer, enabling the construction of preference pairs that reflects both intermediate reasoning quality and outcome correctness. This fine-grained supervision provides a more informative and nuanced learning signal. ii) Visual information. A key distinction between VLMs and LLMs lies in the incorporation of visual information. In VLMs, the reasoning process can reference concrete visual evidence from the image. To support this, we introduce a visual reflection mechanism into the design of reasoning steps, allowing the model to explicitly incorporate and verify visual context during reasoning. This capability is absent in standard LLM settings, where no visual modality is available as an external referent.

Question 2

Following your suggestion, we have added the latest Qwen2.5-VL as a baseline as shown in the table below. Although we build from a weaker base model (InternVL-2.5-MPO), our CoS-InternVL still have competitive performances against Qwen2.5-VL. Despite the competitive results, we would like to emphasize that achieving state-of-the-art performance is not the primary goal of this work. Instead, our focus is on exploring and validating the effectiveness of fine-grained reasoning and reward mechanisms, and on empirically characterizing their unique properties. Our core contribution lies in establishing a complete and transparent framework that we believe can serve as a foundational tool for the VL community to further investigate fine-grained reasoning chains. And we will release all the components to the public, facilitating future work on step-level reasoning and fine-grained rewards.

Question 3

Our experiments are conducted using a medium-scale 8B model, and we compare against baselines with similar parameter sizes (7B, 8B, and 11B). We appreciate the suggestion to explore larger-scale models, but training a 72B model is prohibitively expensive for a school lab. For example, even initiating SFT on a 72B model requires at least 32 A100-80G GPUs, not to mention the cost of iterative DPO and extensive empirical studies. More importantly, we would like to highlight that 8B is a standard setting for developing and validating new techniques and are commonly adopted in the VLM and reasoning literature[1,2,3]. The insights and conclusion obtained from studying models at this scale are meaningful and generalizable. Thanks for your understanding of the computational constraints.

Question 4

The splitting steps are completely determined by the model itself, that is, the model itself generates the reasoning of any number of steps according to the current question. And the name of each step is also not pre-defined manually, but generated by the model itself. This is the original intention of our step based reasoning, we do not restrict the number of steps of reasoning and let the model reason end-to-end. In section 4.3.5 and Figure 6 of the paper, we analyzed the change curves of the steps of different methods during training, and got some interesting findings such as longer reasoning steps do not necessarily lead to better performance. Please refer to section 4.3.5 for details of reasoning length.

Question 5

Thank you for the suggestion. We have added evaluation results on MMMU-Pro, EMMA, and MegaBench as suggested. Our CoS outperforms basline consistently on these new benchmarks, further demonstrating our design's effectiveness. As for MathVerse and MathVision, we do not report results on these datasets to ensure a fair comparison, as they are already included in our constructed SFT dataset Share-Step-300K.

[1] Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models.
[2] R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization.
[3] R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization.