Diving into Self-Evolve Training for Multimodal Reasoning

Thank you for your time and effort in reviewing our paper. We appreciate your recognition of our writing, experimental design, and research topic. We will address your concerns below.

Weakness 1: The main problem is limited experimental evaluation with respect to the number of MLLMs employed in the study. Question 1: Are the presented observations/conclusions related to MiniCPM-V 2.5 also valid for other MLLMs? If so, what is the foundation of such a claim.

Thanks for your suggestion and question, we have already added two extra models, i.e., Phi-3.5-Vision-4B and InternVL2-2B in our experiments, please see our General Response for more details.

Weakness 2: Also, the authors consider only one dataset in their experiments. …. Having a more diverse selection of these dataset (problem types) would be beneficial. Question 2: A similar question regarding the validity of conclusions for other types of problems / other reasoning domains.

Thanks for your suggestion and question. We would like to clarify that the MathVista benchmark is already an aggregation of 31 datasets that cover visual question answering, logical reasoning, statistical reasoning and many other skills – it is not math-only but rather broad.Besides, we have already added four extra benchmarks, i.e. M3CoT, MMStar, MMBench and AI2D, in our experiments, please see our General Response for more details.

Question 3: In Table 1 the results for In-Domain test samples are lower than those for OOD, which is surprising. What is the reason for a better OOD than ID performance?

There are 3 main reasons:

1. Difficulty - our in-domain testset is sampled from the MathV360K dataset, which is constructed by difficulty-aware resampling to increase the ratio of hard questions and then augment the data further. So although it is an ID test set, its difficulty is inherently higher than Mathvista which lacks such difficulty control.
2. Problem Type - Unlike MathVista, which predominantly requires simple numbers or multiple-choice options as the final answer, a significant portion of the ID MathV360K test set consists of open-ended questions that require full sentences as answers. This makes achieving an exact match— the default metric used—significantly more challenging for these types of questions.
3. Relative Improvement - a more reasonable view or our ID and OOD eval is to see the relative improvement. Compared with the base model and warmup-ed model, we have 25.2% improvement in ID testset, while only 0.2% on OOD testset.

Thank you for your time and effort in reviewing our paper. We appreciate your recognition of our paper's layout, comprehensive experimental comparisons, and in-depth analysis. We will address your concerns below.

Weakness 1: This work can be considered as a kind of grid-search process, and doesn't proposed new techniques in terms of methodology.

We would like to highlight that our main contribution is to view self-improving algorithms from the lens of RL and identify several important aspects, then we study good practices of these empirically to present an effective training recipe. We believe that the empirical contributions are meaningful and helpful to help practitioners and significant enough.

Question 1: Why is the order of focusing component be studied in the presented way? … So it would be good for the authors to provide a reasonable explanation of this point.

This is a valid point. For this type of work, it is typically challenging and computationally expensive to perform a full grid search across all possible combinations of factors in a training approach. As a result, the analysis is often conducted by examining one aspect at a time while keeping the others fixed at reasonable configurations to reduce computational cost. This approach is also commonly used in previous works [1,2]. While this method may not guarantee finding the optimal combination of factors, it is generally sufficient to identify a relatively effective training recipe that can benefit both researchers and practitioners. Regarding the specific order in which we studied the training method, reward model, and prompt variation, we chose this order based on the following reasoning: the training method is a fundamental and general aspect, and we expected that a successful training method should be broadly applicable across different settings. Therefore, we first focused on determining the training method. Next, our exploration of prompt variation involved unlabeled prompts that relied on reward signals provided by the reward models. Consequently, we studied reward models before investigating prompt variation.

[1] Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback; NeurIPS 2024

[2] What matters when building vision-language models?; NeurIPS 2024

Thank you once again for your valuable feedback and suggestions! We sincerely hope that our previous responses have addressed your concerns. If there are any remaining questions or issues, please don’t hesitate to let us know—we will do our utmost to address them further. If you feel that your concerns have been fully resolved, we would be truly grateful if you could consider revisiting your review score. We genuinely hope we’ve resolved everything to your satisfaction, but please don’t hesitate to reach out if there’s anything else we can assist you with!

Thanks for your time and effort in reviewing our paper. We appreciate your recognition of the relevance, the high technical quality and the writing clarity. We will address each of your concern as follows:

Weakness 1: Claim: The CoT warm-up phase conflicts with the paper’s definition of annotation/CoT-free self-evolving training. It is essential to clarify this reliance and provide a stronger rationale for the warm-up setting.

We would like to clarify that our paper only claims annotation-free, but not CoT-free. Our paper studies multimodal reasoning where CoT is necessary. Since human-annotated CoT data is scarce for multimodal reasoning, our approach does not use any human-annotated CoT data – In the warmup stage, the training data is synthesized from prompting the base model (policy model) itself, and neither is extra annotation needed nor other models are involved, so we do not think this warmup stage violates any self-evolving claims in this paper .

Weakness 2: Empirical results: The paper primarily evaluates MathVista, a single multimodal reasoning benchmark focused on math-based tasks. To show broader applicability, the paper would benefit from additional benchmarks such as VQA or other scientific QA. Besides, there is only one LLM used in all experiments, so it's hard to justify whether the benefit of the method can be transferred to other architectures.

1. Regarding the evaluation datasets, we would like to clarify that the MathVista benchmark is already an aggregation of 31 datasets that cover geometry reasoning, logical reasoning, statistical reasoning and many other skills – it is not math-only but rather broad. It is just that we only reported the average score on MathVista in the submission. To give more evidence on our performance on multiple tasks, we report more results on each subtask of MathVista, as shown in the General Response as well as in Appendix F of the updated PDF. We can see that most of the subtasks show improvements as we improve the configuration step by step. The final recipe, MStar, also leads to optimal results for most of them.

2. Besides MathVista, to better address your concern, we also add several new multimodal reasoning benchmarks, including M3CoT[1], MMStar[2], MMBench[3] and AI2D[4], as shown in the General Response as well as in Appendix G. We can see for all these benchmarks, M-Star gives consistent improvements.

3. For your comment on using only one model, we have also added new results on two new models, Phi-3.5-vision-4B and InternVL2-2B. The results are included as well in the General Response and Appendix F, G of the updated PDF. Similar to what we have observed on MiniCPM-V-2.5-8B, our findings hold for them in general, and M-Star can boost their performance on a wide range of benchmarks.

Weakness 3: Theoretical analysis: Since self-evolving training shares similarities with RL, a theoretical analysis comparing these methods could clarify the unique aspects of M-STAR, such as optimality, stability, and guarantees.

Thank you for your feedback and for raising these important points. We would like to further clarify our motivation and claims.

In this paper, we start from the observation that self-evolving training can be subsumed as a general reinforcement learning (RL) training process, as described in Eq. 1. However, prior works on [1, 2, 3], have primarily treated self-evolving training as an iterative training paradigm. These works have largely overlooked the deeper connection between self-evolving training and the reinforcement learning framework. Based on this observation, we aim to investigate whether the three key factors within reinforcement learning—online/offline training, reward model, and prompt variation—can contribute to improving self-evolving training. Additionally, we analyze the training dynamics of the self-evolving training process, focusing on the balance between exploration and exploitation. We believe that these perspectives set our paper unique compared to previous self-evolving training works.

[1] STaR: Bootstrapping Reasoning With Reasoning; NeurIPS 2022

[2] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models; TMLR 2024

[3] V-STaR: Training Verifiers for Self-Taught Reasoners; COLM 2024

Weakness 3: Also, the paper lacks a hypothesis-driven structure that ties the findings to the central research question, which appears more like a tech report rather than a research paper.

Our central research question is to investigate several key components of the self-improving algorithm, as motivated and outlined in the earlier sections of the paper (Section 2, Lines 120–131). Different papers may adopt different writing structures, and in this work, we follow a structure that systematically explores various aspects and draws conclusions step by step through our experiments. We do not think that this writing style constitutes a significant flaw in the paper. For instance, many prior works have also adopted similar structures to empirically study best practices, such as [1,2].

[1] Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback; NeurIPS 2024

[2] What matters when building vision-language models?; NeurIPS 2024

Weakness 4: Contribution: While the paper has explored different configurations exhaustively, the overall contribution is vague given its lack of theoretical grounding and limitations in the empirical study.

Our main contribution is to approach self-improving algorithms through the lens of reinforcement learning (RL), identify several critical aspects, and empirically study best practices to propose an effective training recipe. While we did not provide deep theoretical grounding for many of the phenomena observed, we argue that our empirical findings are both meaningful and valuable for practitioners, making the contribution significant. In particular, the additional models and evaluations presented in the General Response further strengthen this contribution.

Question 1: Ablation studies showing the model’s performance with and without the CoT phase to quantify its impact on results.

We would like to emphasize that the warmup phase with CoT training is essential because we observed that most open-weight multimodal models, such as Llava and MiniCPM-V, rarely generate CoT reasoning and typically only produce a short final answer. For self-evolving training in reasoning tasks, the key improvements are achieved by enhancing the reasoning process, specifically the CoT reasoning – it is just how this setting is defined. This follows the standard setting adopted by nearly all prior works in this area [1,2,3,4]. Thus, conducting self-evolving reasoning training when the model cannot generate CoT reasoning would be largely ineffective. This is precisely why we include the CoT warmup phase in the first place. As stated in our response to Weakness 1 of the reviewer, we do not rely on any additional annotations or external resources during the warmup phase. Therefore, we believe there is no issue or limitation associated with this approach.

[1] Scaling relationship on learning mathematical reasoning with large language models; arXiv 2308.01825

[2] STaR: Bootstrapping Reasoning With Reasoning; NeurIPS 2022

[3] V-STaR: Training Verifiers for Self-Taught Reasoners; COLM 2024

[4] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models; TMLR 2024

Question 2: What's the computation cost of the proposed method compared to baselines?

Admittedly, self-evolving training requires more computational resources compared to the SFT baseline. This is a common characteristic of self-evolving training approaches [1, 2] and reinforcement learning with human feedback (RLHF) [3, 4]. The additional computation cost mainly comes from extra data generation processes of these RL-like approaches and typically more training steps than SFT. However, when comparing different variations of self-evolving training, the computational costs are largely similar, as we are all training for the same number of steps with the same batch size for all self-evolving runs.

[1] STaR: Bootstrapping Reasoning With Reasoning; NeurIPS 2022

[2] ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search; NeurIPS 2024

[3] TÜLU 3: Pushing Frontiers in Open Language Model Post-Training; Tech Report

[4] The Llama 3 Herd of Models; arxiv 2407.21783

Question3: Can the best configurations explored in this study be applied to other task domains without losing their effectiveness?

As shown in the general response, we have applied our training recipe to other two models, Phi-3.5-Vision-4B and InternVL2-2B as well and evaluated these models on several added benchmarks and tasks. The results demonstrate that our best configurations are effective in most of these settings.

Thanks for your time and effort in reviewing our paper. We appreciate your recognition of our investigation of self-evolution in multimodal reasoning, and our systematic breakdown of each key factor in it. We will address each of your concern as follows:

Weakness1: The motivation or evidence behind the importance of the three components: the training method, the use of the reward model, and the prompt variation is insufficiently substantiated. Question 1: Could the authors elaborate on the specific motivations or additional evidence that underscore the criticality of the training method, reward model, and prompt variation in enhancing multimodal reasoning?

Thank you for the suggestion. Our selection of these three components as the main research focus in this paper is primarily based on our modeling of self-evolving training within a general RL framework (as discussed in Section 2.), which has been similarly modeled in many RL-related papers [1, 2, 3, 4, 5].

These three selected components, training methods, reward models, and prompt variation, have been recognized as essential and studied in many similar works, particularly in text-only settings. Regarding the training method, [3, 4, 5] discovered that factors such as online/offline updates and on-policy/off-policy rollouts in the training algorithm can greatly impact model performance. For the reward model, [6,7,8] investigated the impact of different reward function, reward model on the training results; and for prompt variation, [1] explored how different prompts in RL would affect the alignment RLHF training results. However, most of previous works except [1] only considers one factor instead of comprehensively investigating all these factors. Meanwhile, none of previous works study how these important factors in RL training would affect the training results of self-evolving .Since self-evolving training follows the general RL framework, we identify the three important and popular factors inside RL training following previous works in this direction. Then we systematically study these factors in a more complex and rarely discussed domain, multimodal reasoning.

We also acknowledge that other factors, such as model architecture, hyper-parameter configurations (e.g., learning rate, batch size), and the specific updating algorithms, may influence the overall pipeline. However, some of these factors are orthogonal to our RL framework. Thus, we have made practical selections, such as using an advanced LMM , MiniCPM-V-2.5, for all ablations and keeping the learning rate and batch size unchanged.

[1] Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback; NeurIPS 2024

[2] Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices; PMLR 2021

[3] RLHF Workflow: From Reward Modeling to Online RLHF; TMLR, 2024

[4] Direct Language Model Alignment from Online AI Feedback; ICML 2024

[5] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models; arxiv.2402.03300

[6] V-STaR: Training Verifiers for Self-Taught Reasoners; COLM 2024

[7] DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence; arxiv 2406.11931

[8] Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning; arxiv 2410.08146

Results with additional models and benchmarks:

MiniCPM-V-2.5

Phi-3.5-vision

InternVL2-2B

[1] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone; arXiv 2404.14219

[2] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks; CVPR 2024

[3] M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought; ACL 2024

[4] Are We on the Right Way for Evaluating Large Vision-Language Models?; NeurIPS 2024

[5] MMBench: Is Your Multi-modal Model an All-around Player?; ECCV 2024

[6] A Diagram Is Worth A Dozen Images; ECCV 2016