SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning

Response to reviewer wAHj

Thank you for the encouraging and constructive feedback. We appreciate your recognition of SRPO’s strengths, including its effective self-reflection design, strong benchmark performance, and broad generalization ability. Below, we address the reviewer’s comments and questions in detail.

Response to Weakness 1. (Training Cost):

We thank the reviewer for raising the concern on training efficiency. Below we report actual wall-clock times for SRPO under 7B and 32B settings using OpenRLHF [1]:

Table a.Training Efficiency Overview

As shown in Table a, SRPO remains cost-effective despite its two-stage design. The SFT phase completes within 5 hours even for 32B models, and the RL stage is comparable to GRPO. The reflection component adds only moderate overhead, keeping training practical for academic-scale labs.

As shown in Table b, to speed up training, we used 4 nodes of 8×H100 GPUs. Importantly, SRPO is fully runnable on a single 8×H100 (80G) setup under the verl framework [2] using BF16. While this increases training time, it supports smaller hardware setups. We will open-source both single-node and multi-node training pipelines in the revised version.

Table b. Hardware Requirements on verl framework.

[1] OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

[2] HybridFlow: A Flexible and Efficient RLHF Framework

Response to Weakness 2. (Inference Overhead):

We thank the reviewer for highlighting the inference latency concern. On the MathVista test-mini set (~1k samples), we benchmarked SRPO-7B against Qwen2.5-VL-7B + GRPO using a single H100 (80G) and vLLM. SRPO incurs a 1.33× increase in wall-time and 1.41× more tokens due to the reflection step. However, our brevity-aware reward f_len​ effectively controls verbosity. As shown in Table 3, this modest overhead results in +3.5 accuracy gain on MathVista and +3.7 on average across benchmarks, highlighting a favorable trade-off between latency and reasoning quality.

Table c. Inference Efficient Overview

Response to Question 1 and weakness 5. (Qualitative & quantitative reflection analysis):

We thank the reviewer for the constructive suggestion regarding reflection quality analysis.

Due to space limitations in the main text, Figure 3 shows abbreviated samples. We provide full reflection traces from both RL training and downstream evaluation in Appendix B.4, which demonstrate the model’s ability to self-reflect and revise its reasoning steps.

Following your suggestion, we further conducted a human evaluation of reflection quality:

1. Human Expert Study for Reflection Quality:

We randomly selected 100 MathVista test samples (answered by SRPO-7B). Two additional senior PhD students (NLP/LLMs) rated each reflection on a 0–3 scale: • 3 = Highly effective • 2 = Partially effective • 1 = Redundant • 0 = Detrimental We also measured the Wrong Answer Fix Rate: how often initially incorrect answers were corrected after reflection.

Table d. Human Expert Evaluation

This shows that 70% of reflections are helpful. Out of 100 questions, the initial solution was incorrect for 33 cases. Among them, 13 were corrected after reflection and a second attempt, resulting in a wrong answer fix rate of 39%.

2. LLM-as-a-Judge Evaluation (GPT‑4o) for Reflection Quality:

Each reflection was scored on 4 dimensions (0–5 scale): logical flaws, missing assumptions, clarity, and actionable suggestions following B.3 Prompt Template. Prompt Template for Self-Reflection Generation.

Table e. LLM-as-a-Judge Evaluation (GPT‑4o)

These results confirm high-quality reflections from both human and LLM perspectives.

3. Error types analysis

We analyze the cases from MathVista where the initial answers were incorrect and compare those corrected after reflection with those that remained incorrect. The key findings are summarized below.

• Most Effectively Corrected Errors: SRPO was most effective at fixing mid-level algorithmic slips, such as arithmetic miscounts, sign errors, step-wise inconsistencies, coordinate read-offs, and option selection mistakes. These issues are internal to the reasoning chain and can be self-verified and revised by the model during reflection.

• Least Effectively Corrected Errors: Reflection struggled with errors caused by visual misinterpretation, OCR noise, or misuse of domain knowledge (e.g.,math theorems). These errors require external grounding, such as reprocessing the image or consulting factual resources, which the current reflection mechanism cannot access.

Therefore, Self-reflection serves as a reliable post-hoc validator for internal logical consistency, particularly for symbolic reasoning errors. For failures rooted in perception or knowledge, future work may integrate visual re-inspection or retrieval modules to extend SRPO’s repair capacity.

Response to Question 2. (Possible Strategies for Reducing Training and Inference Overhead with Reflection):

We thank the reviewer for the insightful question. To reduce training and inference cost, we propose two practical extensions:

• Dynamic Reflection Triggering: We introduce a lightweight classifier to decide whether to trigger reflection based on input difficulty. Trained with a reward that balances accuracy and latency, this allows easy queries to skip reflection entirely, reducing average inference time.

• Length-Aware Reflection Control: We extend SRPO with a learned reflection-length planner that predicts a token budget based on task complexity. This reduces unnecessary generation during both training and inference, while preserving reasoning quality.

These strategies make SRPO more efficient by dynamically allocating compute based on task demands.

Response to Question 3. (Domain Transfer Beyond STEM):

We thank the reviewer for this important question. Although our current evaluation focuses on STEM tasks, prior work suggests that reflective RL frameworks like SRPO can generalize to other domains. Huan et al. [1] show that reasoning skills learned from math tasks via SFT+RL transfer well to medical and general knowledge QA. Similarly, Liu et al. [2] demonstrate that reasoning trained on general-domain text can transfer to multimodal and medical tasks, even without multimodal supervision. These results suggest SRPO’s mechanisms are broadly applicable given suitable textual reasoning data. We plan to evaluate on legal, medical, and commonsense benchmarks in future work.

[1] Does Math Reasoning Improve General LLM Capabilities?

[2] X-REASONER: Towards Generalizable Reasoning Across Modalities and Domains

Response to Question 4 and Weakness 3. (Performance Gap with Closed-Source Models):

We appreciate the reviewer’s observation. While SRPO-32B outperforms many open-source and some closed-source baselines (e.g., on MathVista), it may trail larger proprietary systems like Seed1.5-VL-T. This gap is likely due to differences in capacity (e.g., >500B MoE vs. our 32B dense), scale of training data, and auxiliary tools like OCR and retrieval modules. To address this, we are (i) scaling SRPO to larger models, (ii) collecting higher-quality reflection data, potentially from human experts, and (iii) integrating high-res visual inputs and tool-augmented reflection. These are ongoing and will be included in future works.

Response to Reviewer J4Dj

We sincerely appreciate the reviewer’s positive and constructive feedback. Below, we systematically address the stated weaknesses and specific questions.

Response to Weakness (a) and Question 4. Training & inference efficiency:

We thank the reviewer for the constructive feedback regarding training efficiency. Below, we provide the measured wall‑clock time for each SRPO training stage under both 7B and 32B settings using the OpenRLHF framework.

Table a. Training Efficiency Overview

As shown in Table a, despite the two-stage design, SRPO remains cost‑efficient. The SFT stage completes in less than 5 hours even for 32B models, and the RL stage is comparable in duration to standard GRPO training. The additional rollout overhead introduced by the reflection segment is marginal, making SRPO’s overall compute footprint practical and scalable for real‑world deployment.

Table b. Inference Efficient Overview

Table b shows inference efficiency, we evaluated reasoning latency on the MathVista test‑mini split (~1k examples), comparing SRPO‑7B with Qwen2.5‑VL‑7B + GRPO trained on the same RL data. Using a single H100‑80G GPU and the vLLM framework, SRPO requires only 1.33× more wall‑time and produces 1.41× more tokens on average. This modest overhead is controlled by our brevity‑aware reward $f_{\text{len}}$, which promotes concise yet effective reflection. As shown in Table 3 of the paper, SRPO improves MathVista accuracy by +3.5 points and achieves an average +3.7 gain across benchmarks, showing that the slight inference cost is well justified by significant gains in reasoning quality.

Response to Weakness (b) and Q4. Self-reflection Training data Ablation:

We thank the reviewer for raising this important point. To verify that SRPO’s performance gains stem from our reflection-aware training design rather than simply the training data, we conducted additional ablation experiments comparing Self-Reflection SFT and Plain-CoT SFT under identical data sources and teacher models (GPT-o4-mini).

We constructed a Plain-CoT SFT baseline by removing the <reflection> segments from the same dataset and prompts. Additionally, the CoT reasoning paths were generated by GPT-4-mini given the question and the answer. We also applied GRPO to both the Plain-CoT and Reflection-SFT models. The results are summarized below:

Findings:

• Plain-CoT SFT vs. Reflection-SFT: Adding structured reflection yields an average gain of 1.3 points, indicating that performance improvements are not solely due to data coverage or quality.

• Plain-CoT SFT + GRPO vs. SRPO (full): When both variants are trained with RL, SRPO achieves an additional 2.1 gain, confirming that the reflection-aware reward design plays a critical role in enabling effective reasoning refinement.

These results validate the importance of both our reflection-format data and the self-reflection RL stage in driving SRPO’s improvements.

Response to Q1. Justification for the “first solution → reflection → second solution” structure:

We appreciate the reviewer’s thoughtful question. We clarify the rationale as follows:

• Concrete anchor for reflection: Current MLLMs cannot introspect past hidden states once tokens are emitted. Producing an initial answer exposes an explicit artifact for the reflection to critique. Without this anchor, reflections would have to operate on inaccessible or implicit reasoning, making error localization impractical.

• Stable and decomposable reward signals: Our reward design separates two components: (1) R_accuracy, which assesses the correctness of the first answer and provides immediate, low-variance feedback; and (2) I_eff, which evaluates whether the second answer improves upon the first, enabling direct supervision of reflection quality. If reflection were interleaved step-by-step, computing the reward would require aligning partial thoughts with the ground truth, leading to high credit-assignment variance and unstable RL.

• Avoiding reward hacking: Training with an explicit “first → reflection → second” structure constrains the space of possible redundant exploitation strategies, making it more robust than unconstrained step-wise reflection where the model may manipulate reasoning to game rewards.

Response to Q2. Role of the Second Solution:

Thank you for the question. The second solution is not used solely for computing the reflection reward, it serves three critical purposes:

• Final task output: During validation and benchmark evaluation, only the second answer is scored. This ensures that the reflection must result in a self-consistent and correct final answer; otherwise, the model receives punished reward. (see equation (8) and Section 4.4 Reasoning Qualitative Analysis, lines 312–318).

• Gradient carrier for reflection learning: SRPO applies token-level updates across the full sequence (answer 1 → reflection → answer 2). Although R_accuracy​ is tied to answer 1, tokens in answer 2 are still reinforced via: the shared task reward, and the I_eff​ reward when answer 2 improves or preserves correctness. Thus, answer 2 is directly optimized to be both correct and concise.

• Separation of detection and repair: Anchoring R_accuracy​ to answer 1 isolates “error detection”, while I_eff​ on answer 2 captures “repair effectiveness”. This disentanglement would be lost if only answer 2 were rewarded, obscuring whether improvements came from better initial reasoning or reflection.

Response to Q3. Determination of T_target:

Thank you for pointing this out. We define T_target​ as 2× the length of the original first-think response, and T_max​ as 2.5× the original length. This sets a soft constraint for the combined reflection and revised reasoning to remain concise while allowing sufficient flexibility. We will clarify this definition explicitly in the main paper in the revised or arxiv version.

Response to limitation:

We appreciate the reviewer’s constructive comment. As noted in our Appendix Limitation section, our current experiments focus on dense models (7B and 32B), consistent with prior works like RL-based reasoning in LLMs [1,2,3]. Extending SRPO to other architectures, such as large-scale MoE models or unified generation-understanding models, is a promising direction we plan to pursue in future work.

[1] Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaR

[2] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

[3] Rethinking Reflection in Pre-Training

Response to Paper Formatting Concerns:

We appreciate the reviewer’s kind reminder and will correct these minor typos in the revised version.

Hi, Reviewer J4Dj!

We sincerely appreciate your thoughtful and encouraging review, and acknowledged our paper's clear motivation, detailed method description, and strong empirical results. We thank you for recognizing the novelty and clarity of our framework.

We have carefully addressed all raised concerns in our rebuttal. Specifically, we have included detailed training cost breakdowns and efficiency analyses for both 7B and 32B models, as well as clarified the necessity of the “first solution → reflection → second solution” process. Additional ablations have also been added to isolate the effect of reflection structure and reward components.

We truly value your thoughtful review and hope to gain your continued support. If there are any parts of our submission that would benefit from additional clarification or elaboration to assist your evaluation, please feel free to let us know, we would be glad to provide a further detailed response.

Thank you again for your thoughtful review and support.

Best wishes and regards,

Team 12027

Response to reviewer PKxa

We thank Reviewer PKxa for the positive feedback on our reward mechanism and empirical results.

Response to Weakness 1. (Clarifying SRPO’s Contribution):

We thank Reviewer PKxa for the thoughtful comments. While the SFT → RL pipeline is widely used, SRPO introduces key innovations that extend beyond this standard scaffold:

• Explicit and Measurable Reflection in Both Stages: SRPO is, to our knowledge, the first to:(i) incorporate explicit reflection structure during SFT, teaching the multimodal LLM to critique its own reasoning; and (ii) reinforce reflection quality during RL using the reflection‑aware indicator I_eff, which directly links reflection effectiveness to answer improvement. Unlike prior work that either fine‑tunes only on CoT data or applies RL without explicitly revisiting and correcting prior reasoning, SRPO makes reflection both structurally grounded and quantitatively rewarded, elevating it to a core trainable mechanism for improving reasoning accuracy.
• Reflective Structure vs. Two‑Step Thinking: To evaluate the effectiveness of the proposed think–reflect–rethink pattern, we compared SRPO to a GRPO‑based two‑step thinking baseline that generates two consecutive <think>...</think> steps without middle reflection. As shown below, SRPO without Self‑Reflection SFT already outperforms the two‑step baseline, while the full SRPO achieves the best performance, underscoring the importance of combining Self‑Reflection SFT with reflection‑aware RL.

Table a: Comparison between SRPO and GRPO with Two‑Step Thinking

• Contribution of the Reflection‑Aware Reward I_eff

To isolate the impact of I_eff , we removed it while keeping all other hyperparameters unchanged. This resulted in a consistent drop across benchmarks, confirming that the reflection‑aware reward contributes substantially to SRPO’s improvements beyond simple brevity shaping.

Table b: Contribution of I_eff

Response to W2: Is the SFT improvement solely due to teacher distillation?

To disentangle the impact of our reflection format from generic teacher–student transfer, we conducted the following control experiments:

Table c: Ablation Study of Self‑Reflection SFT on the 7B model

Setup: All models use the same teacher (GPT‑o4‑mini), dataset size (10k), optimizer, and number of training epochs. The only differences are the inclusion of <reflection> segments and whether reflection‑aware RL is applied.

Findings:

• Plain‑CoT SFT vs. Reflection‑SFT: Adding explicit reflection segments yields an average +1.3 gain, showing that structured reflection, not just teacher distillation, contributes meaningfully to performance.

• Reflection‑SFT + GRPO vs. SRPO (full): When both undergo RL, SRPO achieves an additional +2.1 gain by aligning rewards with reflection effectiveness, underscoring the importance of our reflection‑aware reward design.

Response to Weakness 3.1. (Composite‑reward ablation and α‑sensitivity):

Below we (i) disentangle each reward component and (ii) study the robustness of the mixing coefficient α.

• (i) Ablation of Reflection-Aware RL Components.

Findings: Removing I_eff drops the 5‑task average by 1.5 points, showing it is the key driver of SRPO’s gains. Removing f_len(·) has a smaller effect (–0.4). Dropping the entire Self‑Reflection RL stage causes the largest drop (–6.9), underscoring the importance of SRPO’s integrated design.

• (ii) Sensitivity to the reflection length reward coefficient α

Due to rebuttal time and compute limits, we additionally trained with α=0.05 and α=0.3 and tested the dataset from three domains. Results show α=0.05 performs similarly to 0.1, while α=0.3 degrades accuracy, indicating that overemphasizing length reward weakens the reflection signal. Therefore, values near 0.1 offer a balanced trade-off, suggesting that length regularization primarily helps control response length and stabilize training, with limited impact on accuracy.

Response to W3.2 (Ablation study of 32B models):

We evaluate SRPO on 32B with the same ablation setup as 7B. SRPO‑32B outperforms Qwen‑32B‑VL + GRPO, confirming the value of explicit reflection in SFT and RL. Dropping either component clearly hurts performance, consistent with the 7B trends.

Response to Question 1 and weakness 5. (Qualitative & quantitative reflection analysis):

Due to space limitations in the main text, Figure 3 shows abbreviated samples. We provide full reflection traces from both RL training and downstream evaluation in Appendix B.4, which demonstrate the model’s ability to self-reflect and revise its reasoning steps. Following your suggestion, we further conducted a human evaluation of reflection quality:

1. Human Expert Study for Reflection Quality: We randomly selected 100 MathVista test samples (answered by SRPO-7B). Two additional senior PhD students (NLP/LLMs) rated each reflection on a 0–3 scale: • 3 = Highly effective • 2 = Partially effective • 1 = Redundant • 0 = Detrimental We also measured the Wrong Answer Fix Rate: how often initially incorrect answers were corrected after reflection.

Table d. Human Expert Evaluation

This shows that 70% of reflections are helpful. Out of 100 questions, the initial solution was incorrect for 33 cases. Among them, 13 were corrected after reflection and a second attempt, resulting in a wrong answer fix rate of 39%.

2. LLM-as-a-Judge Evaluation (GPT‑4o) for Reflection Quality:

Each reflection was scored on 4 dimensions (0–5 scale): logical flaws, missing assumptions, clarity, and actionable suggestions following B.3 Prompt Template. Prompt Template for Self-Reflection Generation.

Table e. LLM-as-a-Judge Evaluation (GPT‑4o)

These results confirm high-quality reflections from both human and LLM perspectives.

Response to reviewer mGVt

We thank Reviewer mGVt for the positive assessment and valuable suggestions.

Response to W1. Formatting:

We acknowledge the spacing issues that arose from automatically scaling wide figures. In the new revised version we will move Fig. 2 and Tables 1–2 to the top of the next page and re‑balance texts and figure/tables space.

Response to W2 and Q1. Training‑data transparency:

Thank you for the constructive suggestion. Due to space constraints, we briefly described the training data in the Methods section. A detailed explanation is provided in Appendix B.1, where we outline the construction of both the Self-Reflection SFT and RL datasets, including the specific open-source sources used and the prompting templates detailed in Appendix B.3.

Specifically, for the Self-Reflection SFT dataset, we follow the prompt format defined in Appendix B.3: Prompt Template for Self-Reflection Generation, and structure the data in the form of first solution → reflection → refined solution. For the RL dataset, we unify several open-source datasets into a consistent format containing image, question, and answer label.

As NeurIPS rebuttal rules prohibit modifying the PDF or adding visual content at this stage, we commit to releasing all training datasets, along with the full source code for data selection and construction pipelines for both SFT and RL, in the new-revised version.

Response to Q2. Clarifying Table 1 sources:

Thank you for the helpful question. For the results reported in Tables 1 and 2, baseline numbers are taken from the official reports whenever available. For datasets not covered in those reports, we reproduce results using the authors' released checkpoints and hyperparameters under our unified evaluation setup. These reproduced results are marked with a † symbol for clarity.

We will include this clarification in the revised version. We appreciate the reviewer’s suggestion.