Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging

Thanks for the comments! We address the concerns below.

W1:

The workload of experimental analysis and discovery in this paper is not enough…For examples…why not try a dynamic model merging method considering this finding?

We sincerely appreciate your feedback on our work. But we respectfully disagree with the comments on “insufficient workload” for the following reasons:

• A Simple Method Does Not Mean Less Work: We actually tried it in our early experiments but it worked similarly to the simple linear merging method [see response to Q3]. Given no clear performance gain, we opted for the simpler solution.
• We want to emphasize here again that our goals are:
  • To see whether the perception and reasoning abilities can be infused by model merging. We incorporate three VLMs of varying architectures and sizes, along with six reasoning models across diverse domains, and evaluate on five multimodal reasoning benchmarks.This evaluation process accruing costs of around $1,000.
  • To understand how perception and reasoning abilities are distributed in parameter space within a VLM. To achieve this: In Figures 4 & 5, we conducted masking experiments on each even layer (MLP/Attention) under various configurations (e.g., masking LLaVA to LLaMA or random noise), evaluating each independently—resulting in 128 runs in total. In addition, we conducted 4–5 such preliminary experiments. We use GPT-4 (via VLMEvalKit) to evaluate model outputs, accruing costs of around $3,000.
  • Although GPU hours and monetary costs do not directly equate to the quality of the work, these data points do illustrate one aspect of the hidden workload we have put into this work.

Besides, we also add results based on Qwen model [see response to Q1], hopefully these added results can help recognize the workload of this paper.

Q1/2:

include Qwen-based model.., report the results of InternVL on other benchmarks considering that LLaVA and Idefics .

Thank you for the advice! We expand our experiments for Qwen-based model here. Specifically, we merge Qwen2-VL-7B-Instruct with Qwen2-Math-7B, the results are:

From the results, we see that merging benefits certain benchmarks like MathVerse, DynaMath and MathVision. However, there is a decline in performance for benchmarks MathVista and MMStar. The unstable increase pattern is similar to what we observe with Idefics and InternVL, indicating that VLMs that have undergone math-related pure text training (as many state-of-the-art models do today [1]) benefit less from merging with reasoning models.

As for the large-scale model InternVL-76B, we have already evaluated it on the same benchmarks, as shown in the InternVL row of Figure 3 in our paper.

Q3:

Did the author try to only merge reasoning llm in the later layers? would this allow VLMs to achieve a good balance between perception and reasoning?

Thank you for the great suggestion of trying a dynamic model merging method! Like we have briefly discussed this in W1 response, we did test a dynamic approach based on our layer-wise insights.Inspired by the observation that perceptual abilities lie in early layers, we merge only the later layers to preserve them.

Although the results show clear gains over the baseline, they are only on par with linear merging. Given no clear performance gain, we opted for the simpler solution. One possible reason is that interpretation-based analysis is inherently subjective and only captures coarse-grained trends, often with substantial noise. For instance, while later layers may show stronger reasoning, earlier layers still play a role. Thus, merging only the later layers in reasoning-focused LLMs may not yield performance gains. Decomposing the two abilities may enable successful merging but doesn’t necessarily yield a better merging method. This also consists with previous findings from [2] that different merging methods behave very similarly at larger scales.

[1] What matters when building vision-language models? NeurIPS 2024.
[2] What matters for model merging at scale? Prateek et al. 2024

We appreciate your thorough review and detailed comments! Your suggestions will be helpful in improving the paper.

Q1:

Lack of baselines of other merging methods. Only TIES is provided in the appendix.

We appreciate the suggestion and have expanded our experiments to evaluate DARE merging [3] (Dare-TIES and Dare-Linear), another SOTA merging method to explore additional merging techniques. We follow the hyperparameter search strategy same as TIES, parameterized by ($\alpha_1$, $\alpha_2$), where $\alpha_1$ is tuned for the VLM and $\alpha_2$ for the Math LLM, in order to obtain the most comparable checkpoints on benchmarks that emphasize vision and textual reasoning. The results are shown below.

The results show that DARE performs comparably to linear merging and TIES, consistent with our finding that different merging methods yield similar performance (line 657), with none significantly outperforming simple averaging. This supports our choice to adopt the linear merging method in our paper. This finding is also consistent with the previous finding in [2] that different merging methods tend to exhibit similar behaviors at larger scales. Given this, we chose not to focus extensively on exploring alternative merging methods but instead to explore more about the interpretability and composition of the model’s internal abilities.

Q2:

It would be more interesting to also include other model families or larger models.

We focused on LLaVA for our interpretability experiments mainly due to resource limits. Specifically, we conducted masking-out experiments on each even layer (for MLP/Attention, respectively) across four configurations (e.g., masking LLaVA to LLaMA). These experiments, presented in Figures 4 and 5, involved a total of 128 runs. We use GPT-4 to evaluate these runs (by VLMEvalKit), which incurred a substantial cost of approximately $3,000. Due to the high computational cost, our study currently focuses on LLaVA. However, we recognize the value of extending this analysis to other model families and leave it as promising future work.

Q3:

Lack of significance test…

Thank you for your valuable feedback! We acknowledge that some of the performance improvements in Table 2 fall within a 1% margin. We conduct t-tests to compare the results of the merged model with the base LLaVA. The resulting p-values are presented in the table below, with statistically significant results (p < 0.05) shown in bold. Merging general reasoning models such as Mammoth2 and Magpie does not lead to statistically significant improvements, showing a similar pattern of marginal accuracy gains as shown in Table 1 of our paper. In contrast, merging Math LLMs, such as Dart-Prop, leads to statistically significant improvements. This aligns with our conclusion in the paper (line 204) that merging with math-related models provides the greatest benefit.

Q4:

I'm interested in how to leverage the findings that perception and reasoning abilities lie in different areas of VLMs, so that we can achieve better merging performance.

During our exploration, we wondered whether insights from the decomposition of perception and reasoning abilities could be leveraged to develop better merging methods, similar to the approach proposed by [1]. To investigate this, we experimented by keeping the early layers unchanged to preserve perceptual ability while only merging the later layers to enhance reasoning ability (see Q3 to Reviewer xnvF for details). Our results indicate that this approach does not yield any performance gains. Given no clear performance gain, we opted for the simpler solution. Our finding is also consistent with previous findings from [2] that different merging methods behave very similarly at larger scales.

[1] Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models, Lucas et al. ICLR 2025.
[2] What Matters for Model Merging at Scale?. Yadav et al. arXiv 2024.
[3] Language models are super mario: Absorbing abilities from homologous models as a free lunch. Yu et al.ICML. 2024.

We appreciate your thorough review and detailed comments! We address your questions below.

Q1:

The authors assert that "after merging, we observe that all layers..whereas the distribution of perception abilities across layers remains largely unchanged." I could not find any text discussing this observation.

Thanks for your advice! This observation is briefly discussed in Section 4 (line 376–383) in paper. After merging with Dart, nearly all layers exhibit a larger performance drop in math reasoning tasks compared to LLaVA, suggesting that reasoning capabilities are more widely integrated across layers post-merging.

The statement about “distribution of perception abilities across layers remains largely unchanged” is an implicit deduction.

• the higher influence of all layers in math reasoning task (Math subset of MathVista) implies that reasoning is layered on top of perception abilities, rather than replacing them. Since this task requires both perception and reasoning, the observation that early (perception-focused) layers continue to show sensitivity by masking supports this interpretation. If reasoning overrides perception, the influence of early layers would diminish—but it is not observed.
• Besides, in general VQA tasks (see left side of ⑤ and ⑦ vs. ⑥ and ⑧), we observe no substantial shift in the behavior of individual layers after merging. This further supports the idea that perception-related representations remain stable.

We appreciate your feedback and will revise the discussion in Section 4 of the next version to clarify this point.

Q2:

More layers show an impact on accuracy does not necessarily imply an improvement in their reasoning ability.

Thank you for your thoughtful feedback! We agree that the original phrasing may convey a stronger claim than intended. We will revise it to clarify that our analysis suggests a potential correlation, not conclusive evidence. Specifically, we hypothesize that the presence of more layers may correlate with improved reasoning performance, as the amount of reasoning-related latent knowledge stored across layers may also increase.

Q3:

it is unclear why the authors did not explore the use of other powerful logical LLMs.

Thank you for your valuable feedback. We share your interest in exploring the generalization of reasoning capabilities across domains, including logical reasoning.

• Our paper already includes diverse LLMs trained on various reasoning tasks. Specifically, we incorporated Magpie, which covers planning and coding tasks (see [1] for details), and Mammoth2, a model trained on various reasoning domains including Socratic reasoning - a form of logical reasoning. The summary of training data as shown below.

Performance results for both are at table 1 in our paper. They show only minor gains, suggesting that combining general-purpose reasoning models remains insufficient for strong performance on math-heavy benchmarks.

• To explore further on logical reasoning, we add the experiment by fine-tuning a LLaMA3-8B on LogiCoT [3] (a logical chain-of-thought dataset) and merging with LLaVA. As results shown below, this pure logic-focused approach helps the math-reasoning tasks, indicating good generalization ability, we will update these experiment results in our paper. |Model|Method|MathVista|||MathVerseBenchmarks||||||MV| |-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| |||All|General|Math|Overall|T-D|T-L|V-I|V-D|V-O||| |LLaVA|Baseline|37.4|51.7|25.4|20.1|25.9|20.8|21.1|16.5|16.0|13.8| ||+logic|37.0|49.1|26.7↑1.3|23.5↑3.4|30.5|25.0|26.4|20.3|15.1|11.2↓2.6|

Q4:

Some of the character sizes of figures appear to be too small.

Thank you for the feedback! We will enlarge the font sizes in these subfigures to ensure clarity in the revised version.

Q5:

I am curious about the performance improvements observed in Figures 4 and 5 when modifying or replacing the model parameters.

For instance, in the math-targeted VQA tasks ① and ③ (right panel) shown in Figure 4, performance notably improves after masking by LLaMA, indicating that the image SFT training—designed to equip LLaVA with image embedding comprehension—may overemphasize learning image perception, and cause forgetting issues to diminish LLaMA’s inherent mathematical reasoning skills. Restoring the original parameters recovers this capability, highlighting the need to preserve strong reasoning skills in VLMs to counteract domain shift introduced by image training.

[1] Magpie: Alignment Data Synthesis From Scratch By Prompting Aligned Llms With Nothing, Zhangchen et al. ICLR 2025.
[2] MAmmoTH2: Scaling Instructions from the Web. Xiang et al. NeurIPS 2024.
[3] LogiCoT: Logical Chain-of-Thought Instruction Tuning. Liu et al.EMNLP 2023