Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?

We are thankful for your valuable comments. Here, we address your concerns and questions point by point.

• W1: Limitations of merging different structures

As noted in the paper’s limitations, our method is designed for merging MLLMs and math LLMs that share the same base architecture and size. This ensures we can reliably identify aligned layers for merging. In practice, we believe this is not a major limitation, as most MLLMs are built on top of their corresponding LLMs. For example, LLaVA is based on LLaMA-based models, and Qwen2-VL on Qwen2-based models, making such model pairs common. Currently, this architectural constraint is shared by most existing training-free merging methods. which typically assumes the models have the same size and cater to the same modality. Our work goes a step further by enabling cross-modality merging to improve multi-modal reasoning.

• W2: More general benchmark results

We are grateful for your comments. Following your suggestion, we conduct extra experiments on TextVQA to support our claim. These results are shown below. Our method allows the models to maintain their general capabilities.

• W3&Q2: Failure / Successful case study

Thank you for the valuable advice. We illustrate two cases in the MathVista below for your reference. As shown in the Failure Case, the merged model may fail on problems where visual common-sense reasoning is required. This suggests that merging may impair the model’s ability to reason over visual information and commonsense knowledge, potentially caused by math-focused weights after merging. As shown in the successful case, the merged model is better in cases where statistical reasoning and arithmetic reasoning are demanded. In the revision, we will add one section of case study to show more qualitative results and analysis and provide a more intuitive understanding.

• W4: Scaling parameter α1 and α2

Sorry for the confusion. The scaling factors α₁ and α₂ are not hyperparameters. As mentioned in line 296 of the main paper, α₁ is set to 1, and α₂ is computed based on the matrix norm using Eq. (7). **In our paper, the main hyperparameter is the similarity threshold, which we provide in Table 9 along with the hyperparameter results in the baseline.

• Wr&Pre: Rearrange the experiments and optimize Tab 9

Thank you for the valuable suggestion. We will move the results of merging multiple models to the main paper and plot line figures to better showcase the results in the revision.

• Q1: Apply to other domains

Thank you for the thoughtful question. Math reasoning has been one crucial ability of LLMs and MLLM, hence we focus on math reasoning in our paper. Essentially, our method relies on using a specialized model and its task vector to transfer capabilities from LLMs to corresponding MLLMs. This setup is not restricted to math reasoning. While prior works have explored model merging in text-only domains, our work extends this to the more challenging multi-modal setting. Our approach could potentially be applied to other domains where paired specific LLM and MLLM models exist. In case that more benchmarks and corresponding specialized models are available, we are happy to investigate extending our method in these domains.

We are thankful for the valuable comments. Here, we address your concerns and questions point by point.

• W1: Experiments of recent reasoning models

Thanks for the suggestions. Due to the limited rebuttal window, we conduct experiments on one recent SOTA reasoning MLLM VL-Rehinker-7B [1], which still demonstrates the effectiveness of IP-Merging in enhancing the performance of reasoning MLLM on various math reasoning benchmarks. We are happy to report results on more reasoning models in our final version.

• W2&W3: Diverse baselines and benchmarks

Following your kind suggestions, we further conduct experiments using Qwen2.5-VL-7B-Instruct, InternVL3-8B-Instruct, and extend the Math-Benchmarks to MathVision, DynaMath and WeMath. As shown in the table, our merged model can further improve the math reasoning performance across different benchmarks.

• W4: Clarification on the experimental results

We are sorry for the confusion. (1) Inconsistencies in reported numbers on MathVista: For fair evaluation, we reproduce all the results using the public official evaluation code, parameters and prompts [2]. We also adopt the same setting in our merged models for fair evaluation. We applied the same setup to our merged models. Variations in hardware may also affect performance. For instance, we observed a score of 54.7 on an A800 GPU versus 55.4 on an RTX 3090. To ensure consistency, we used RTX 3090 and the same runtime environment for all experiments.

(2) Clarification for the same results on MMMU: We are aware that the MMMU subsets have the same score. However, there do exist differences if we look into the statistics of each subset. For example, in Humanities and Social Science subsets of MMMU as shown below, we find that they are different in two subsets, including History and Sociology, although their sum is the same.

Base MLLM: 'Overall-Humanities and Social Science': {'num': 120, 'acc': 66.667}, 'History': {'num': 30, 'acc': 70.0}, 'Literature': {'num': 30, 'acc': 86.667}, 'Sociology': {'num': 30, 'acc': 56.667}, 'Psychology': {'num': 30, 'acc': 53.333}

Merged Model: Overall-Humanities and Social Science': {'num': 120, 'acc': 66.667}, 'History': {'num': 30, 'acc': 66.667}, 'Literature': {'num': 30, 'acc': 86.667}, 'Sociology': {'num': 30, 'acc': 60.0}, 'Psychology': {'num': 30, 'acc': 53.333}

We suspect this might be due to the limited evaluation samples (900 samples). For your reference, we add experiments of TextVQA (5000 samples), where the base model achieves a score of 83.8 while the merged model achieves 84.0. In summary, we are confident that our results are reproducible, and all reviewers are welcome to run the code in the anonymous link to verify our results.

• W5: Limitation on architecture

As noted in the limitations of our paper, our method is designed for merging MLLMs and math LLMs that share the same base architecture and size (e.g., LLaMA-2). This is necessary to reliably identify aligned layers for merging. Nonetheless, we believe this does not pose a major constraint in practice, as Most existing MLLMs are built on top of their corresponding LLMs. For example, LLaVA is based on LLaMA models, and Qwen2-VL is based on Qwen2 models, making such model pairs common in real-world scenarios. Currently, this architectural constraint is a common limitation across train-free model merging methods, which typically assume the same size and modality. Our work takes a step further by addressing cross-modality merging to enhance multi-modal reasoning capabilities.

• Questions:

We provide extra experimental results of more recent baseline models (e.g., LLaVA and Qwen series, Fine-Tuned or RL-based training strategy) and benchmarks (e.g., math-related and general benchmarks), demonstrating the effectiveness of our approach even on stronger baseline models.

[1] Wang et.al, VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning, 2025

[2] Wang et.al, Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, 2024

Thanks a lot for your insightful comments. Here, we address your concerns and questions point by point.

• W1: Extend to other models

Yes, the method can be extended to other models like code LLM. In our paper, we tried merging CodeLlama and Tora-code with the LLaVA model, which improves MathVista performance from 25.2% to 26.3% and 28.2%, respectively. It shows that code LLMs can contribute to multi-modal reasoning tasks. The experimental results are shown in Table 10 of Appendix.

• W2: Trained visual encoder

We are thankful for your insights. In our work, we focus on developing a train-free method to enhance the reasoning performance of VLMs, therefore we do not retrain the visual encoder in the paper. Nevertheless, IP-merging also works effectively when the base MLLM has a trained vision encoder.** We test the Cambrian-8B model and merge the Deepseek-R1-distilled model (LLama-3-based reasoning LLM). We illustrate the results on MathVista, showing that IP merging can further enhance the reasoning performance.

• W3: MathGlance benchmark

Thank you for your great suggestion. We conducted experiments on the MathGlance Benchmark using Qwen2.5-VL-7B-Instruct as the base MLLM and the Deepseek-R1-distilled-Qwen-7B as reasoning LLM. After merging the LLM, the geometry reasoning performance of the base MLLM is further enhanced.

• W4: Misalignment of the text

Thanks. However, we cannot find the misalignment in Figure 2. The figure appears correctly formatted on our end (We previously found that if an old version of a browser or a PDF reader is used, it may not show the figure correctly).

We appreciate your insightful comments. Here, we address your concerns and questions point by point.

• Q1: Transfer other text-based abilities

Thank you for the inspiring question. Math reasoning has been one crucial ability of LLMs and MLLM, hence we focus on math reasoning in our paper. Essentially, our method relies on using a specialized model and its task vector to transfer capabilities from LLMs to corresponding MLLMs. This setup is not restricted to math reasoning and could be applied to other specialized domains such as code generation. However, we have not found an appropriate testing benchmark to evaluate the effect of the code generation ability of MLLMs. In our paper, we also made attempts to merge the code model to enhance math reasoning in Table 10 in the appendix, demonstrating its effectiveness. For the medical image question answering, we believe the proposed IP-merging might be useful, where both task-specific knowledge and visual understanding are required. In case that more benchmarks and corresponding specialized models are available, we are happy to investigate extending our method in these domains.

• Q2: Adaptive&fixed similarity threshold

While we fully understand your concern, we suggest setting the threshold directly (we find 0.4 performs well in our experiments). Our justifications are as follows. Our goal is to identify crucial parameters between models using similarity scores. We directly use a similarity threshold based on the two considerations: (1) A fixed similarity threshold ensures that only highly related parameters are merged. In contrast, a top-k% approach selects a fixed number of parameters regardless of their similarity. Some layers might include low-similarity parameters, which may not benefit from merging. For some model pairs, the top-k% **parameters may have high similarity, while for others, they might be relatively dissimilar. This makes the threshold harder to tune across different model pairs. (2) Similarity scores are normalized, so the fixed threshold is naturally robust across different model pairs. For your reference, we also conduct experiments of selecting the top-k percentage of parameters based on the score distribution. As shown in the following table, we find that the performance is not as robust compared to directly setting the threshold (0.4). Although the two approaches are similar, directly setting the threshold is a more effective way for our method.

• Q3: Poor performance on GPS

Thank you for your comments. Following your suggestions, we examine failure cases in MathVista, and find that the IP-merged LLaVA model may struggle to recognize key visual information in the image, leading to wrong predictions. The cases below suggest that the IP-Merged model’s performance is hindered by weakened visual understanding, particularly in tasks requiring recognition of geometric information. In the revision, we will add one section of case studies for more discussions.

• Q4-Q5: Experiments of recent MLLMs and other general multi-modal benchmarks

We are thankful for your suggestions. Following your kind suggestion, we further conduct experiments using recent MLLMs Qwen2.5-VL-7B-Instruct and InternVL3-8B-Instruct on more benchmarks including MathVision, Dynamath, Wemath and one additional general VQA benchmark TextVQA (Due to the limited rebuttal window, we do not run MMBench). We use Deepseek-R1-distilled-Qwen-7B as the math LLM. As shown in the table, our methods do enhance the reasoning performance of stronger baseline models while maintaining the general reasoning abilities.