RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness

Dear Reviewer e3Kd:

Thank you for your effort in reviewing our paper and we are glad you are impressed by our analysis and thorough evaluation. We will respond to every question respectively.

Q1. Comparison with more fusion methods

Thank you for mentioning the valuable papers. Fisher[1] and RegMean[2] are popularly employed comparing methods for evaluation of model merging. They approximate the merging parameter by fisher matrix and finding closed-form solution for a least-squares regression problem, respectively. Both Fisher and RegMean need validation set to compute the coefficient of merged models, as shown in Tab.1, and our method refrains from validation set. BayesOpt[3] explores a new paradigm that merges checkpoints during pretraining to enhance downstream performance, while our method mainly targets at merging fine-tuned PEFT modules of different tasks. Since all models are fine-tuned with 1 epoch, [3] can not be directly applied to the setting. LORE-merging[4] iteratively solves a single-variable minimization problem with coordinate descent method to obtain better task vectors. By contrast, our method does not need the time-consuming iteration, with one straightforward calculation, which is much more efficient. As [4] does not provide its code, we can not reproduce satisfying results. Comprehensively considering reproducibility and comparability, we compare with Fisher[1] and RegMean[2] to better illustrate the effectiveness of the proposed method. Results are shown below.

It can be seen in the table that without validation set, our method still gets substantial promotion compared with Fisher and RegMean, strongly certificating the effectiveness of the proposed method. This could be attributed to in-depth analysis from the perspective of low-rank decomposition and specific merging algorithm design for PEFT components.

We totally agree with the reviewer that comparison with recent merging methods would help highlight the effectiveness of the proposed method. Following the suggestions from the reviewer, we find that checkpoint merging[5, 6] would be a promising direction for merging techniques, and we would treat it as potential future direction. We promise to properly cite and add discussion with all these six valuable works in the updated manuscript.

[1] Merging Models with Fisher-Weighted Averaging. NeurIPS 2022.

[2] Dataless Knowledge Fusion by Merging Weights of Language Models. ICLR 2023.

[3] Checkpoint Merging via Bayesian Optimization in LLM Pretraining. ICML 2025.

[4] LORE-MERGING: Exploring Low-Rank Estimation For Large Language Model Merging.

[5] Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging. EMNLP 2024.

[6] Towards enhanced LLM pretraining: Dynamic checkpoint merging via generation quality. Information Fusion.

Q2. Extension to other PEFT methods

Thank you for your constructive suggestion, comparison with more PEFT methods is significant to validate the extensibility of our method. In addition to PISSA[7], we also implement a commonly used PEFT method DoRA[8] that supports multimodal tasks. Due to time and character constraints, we conduct the experiment on a subset of 6 datasets and compare with two merging methods. Results are shown below.

It can be concluded from the results that (1) more advanced PEFT technique is beneficial to better performance of model merging, and exhibits overall better performance; (2) our method achieves consistent and substantial improvements against existing merging methods across all PEFT methods. It strongly certificates that the problem of direction robustness widespread exists in merging all kinds of PEFT modules, where attention is primarily paid to improving the performance of single task. By contrast, our method handles the issue to some extent, thereby promoting the performance when the PEFT technique varies.

[7] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024.

[8] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.

Q3. OOD results

We firmly agree with the reviewer that validating OOD ability should be highlighted in model merging to certificate the generalization ability and extensibility of merging algorithm. In Tab.1, we use seen and unseen tasks to validate the generalization ability of merging model, where we merge eight fine-tuned models, and test them on four unseen tasks (line263-274). As can be seen from the table, (1) this can be viewed as OOD generalization that existing methods perform poorly or even worse than zero-shot, demonstrating its difficulty and that merging towards specific datasets would harm the generalization ability; (2) our method outperforms existing methods by a large margin, which could be attributed to the design of the merging method. Therefore, the generalizability of our proposed method is firmly demonstrated.

Also, following the OOD setting of BayesOpt[3], we set VQAv2 as ID dataset and test merged models on three diverse OOD datasets: ScienceQA, IconQA and Grounding to further demonstrate the OOD ability. Results are shown below.

Similar conclusion can be obtained from the additional experiment that our method achieves better OOD performance with less ID dataset, strongly demonstrating the effectiveness and generalization ability of the proposed method.

Q4: Form of specific transformation

The transformation matrix $W$ is applied between $A$ and $B$ in the form of matrix multiplication. As a result, the matrix can be seen operating on $A$ and $B$ simultaneously and no matrix operates independently on B. With respect to the specific form of the transformation matrix, it is a diagonal matrix based on statistical characteristics of $A$, which is defined in Eqn.(3) with detailed analysis in line203-214. Briefly speaking, considering that explicitly operating on decomposed matrices is time-consuming, we directly adjust original low-rank matrices to achieve the same goal inspired by the asymmetry and correlation between two LoRA modules. By designing this, singular values can be enlarged as illustrated in Fig.3a and Fig.3b.

We once again thank the reviewer for time and effort in reviewing our paper with several insightful suggestions, and we hope the additional experiments and analyses could tackle your issues. We will properly cite the mentioned papers and add discussion in the updated manuscript. If you have any further questions, respond to us and we will try our best to address all your concerns!

Dear Reviewer 5p9X:

Thank you for your professional suggestions and we greatly appreciate your positive comments about our analysis and empirical results. We will respond to each question respectively.

Q1. More PEFT methods

Thank you for your valuable advice. Evaluating the method on different PEFT modules is of great significance to demonstrate the extensibility of the proposed method. We primarily test our method on LoRA since it is the most commonly used and comparable PEFT techniques. Following your suggestions, we additionally conduct experiments DoRA[1] and PISSA[2], which are two LoRA-based efficient techniques with advanced algorithms to improve the performance of PEFT learning, to demonstrate the extensibility of the proposed method. Detailed results and analyses are shown below.

It can be concluded from the results that (1) more advanced PEFT technique is beneficial to better performance of model merging, and exhibits overall better performance; (2) our method achieves consistent and substantial improvements against existing merging methods across all PEFT methods. It strongly certificates that the problem of direction robustness widespread exists in merging all kinds of PEFT modules, where attention is primarily paid to improving the performance of single task. By contrast, our method handles the issue to some extent, thereby promoting the performance when the PEFT technique varies.

[1] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.

[2] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024.

Q2. More task types

We truly appreciate the reviewer for the professional suggestion. Due to the large gap between image generation and understanding task, we do not find existing merging methods that conduct additional experiments on image generation task. To this end, we conduct experiments on vision tasks with VLM model to partially demonstrate the extensibility of our approach. With respect to image generation task, the most related work would be ZipLoRA[3], which focuses on style transfer with merging strategies. It trains a content LoRA and a style LoRA and then merges them to get one LoRA with both style and content knowledge. However, it requires post-training merged LoRAs with specific design.

As the reviewer suggests, we conduct experiments on style transfer to validate the extensibility of the proposed method in image generation task. Following the setting from ZipLoRA[3], we train a content LoRA and a style LoRA based on SDXL. We calculate the feature similarity of subject, style and text between specific and merged models to reflect the alignment and fidelity of the merged model. Results are shown below.

Due to the time limitation and merging without post-training, the model might not be that converged with overall smaller similarity in value, but the advantage of our method can still be confirmed that our method outperforms existing merging techniques in evaluation of all style, subject and text alignment, thereby showing the extensibility of the proposed merging method.

[3] ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs. ECCV 2024.

Q3. Singular value distribution of different layers

We take merging ScienceQA and ImgetNet in Fig.3 as an example to show the distribution of singular value changes with different layers. Due to the constraint of the policy, we are only able to present the results in table, so we present average singular value first layer, middle layer and last layer to represent the overall change in distribution of singular values. We select the maximum, medium and minimum of singular value to show the distribution. Results are shown in table below.

From the table above, it can be concluded that:

(1) as the position of the layer becomes higher, the maximum singular value becomes larger, and the tail singular values become smaller (reflected by both smaller minimum and medium singular values), making the distribution more stark. This shares similar observation with HiDe-LLaVA[4] that lower layers carry more general knowledge and higher layers contain more task-specific knowledge, so in the first layers, gap between top and tail values would not as large as the last layers (This could also be validated by the results of merging different layers in Q5 that naive merging of higher layers leads to more performance drop). Therefore, the distribution becomes more stark with increased layer, highlighting the necessity to properly handle direction instability during merging.

(2) Moreover, comparing our model with original model, we can conclude that our method successfully and consistently scales both top and tail singular values across different layers, thereby contributing to robust efficient merging with improved performance, which strongly demonstrates the effectiveness and rationality of the proposed method.

It would be clearer to present the results in the form of figure, and we would properly display it in the updated manuscript.

Q4. Different number of merging models

We provide results of merging different number of models in Fig.A of Appendix. It can be concluded that (1) in seen tasks, the performance undergoes slight drop, which is in consistent with the intuition that merging more models interferes specific task more; (2) in unseen tasks, the performance first improves and then declines modestly. The phenomenon could be attributed that in the first stage, seen tasks transfer knowledge and enhance unseen tasks of similar distribution; in the second stage, interference dominates merging rather than knowledge transformation with increasing merging models; (3) It is notable that under both circumstances, our method consistently and substantially outperforms existing approaches as task number varies, indicating the superiority and stability of our method.

Q5. Merging different layers

To get the influence of merging different layers, we conduct experiments on merging low, middle and high layers of the model to observe the performance variation. Specifically, we perform the proposed method on selected layers and naive average merging on the rest layers. The results are shown in table below.

It can be inferred from the results that (1) As observed from Q3 that higher layers exhibit more stark distribution, naive merging higher layers is more prone to poorer performance (reflected by the first two rows). This shares similar conclusion with results from Q3 and HiDe-LLaVA[4] that lower layers carry more general knowledge and higher layers contain more task-specific knowledge. As the specific knowledge dominates the distribution, the rest singular values are more likely to alter their directions during merging. Therefore, merging higher layers without specific design inevitably leads to direction instability. Therefore, when operating our approach on the higher layers (Last 8 layers in table), performance would be the best compared to naive merging of highest layers. This is of great significance that it gives a clue for which layers are more prone to direction instability during merging; (2) Our approach conducts from the perspective of direction robustness. and all layers merging with our method achieves the best results, furthermore certificating the effectiveness of the proposed merging method.

[4] HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model. ACL 2025.

We would like to express our sincere gratitude to the reviewer for all the thoughtful suggestions, and we hope these additional experiments could address the concerns and better strengthen the proposed method. We will update the manuscript with additional citations, experiments and analyses. If you have any remaining questions, do not hesitate to respond to us, and we are pleased to address any of your concerns!

Dear Reviewer Rk6Q:

Thank you for your professional comments about our paper and recognition about the analysis from direction robustness! We will answer each question respectively.

Q1. More PEFT methods

Our motivation is built upon matrix analysis of singular values and design specific techniques to resolve interference of direction robustness when merging LoRA components, which is based on the consideration that LoRA has been the most popular and effective parameter-efficient tuning technique across various fields. Therefore, it can not be directly applied to prompt-based methods. In this part, we do not know any existing work about merging prompts to the best of our knowledge. Nevertheless, we respectully agree with your suggestion that validating the extensibility across different PEFT methods is beneficial to further demonstrate the effectiveness of the method. Therefore, we give some additional results on other PEFT techniques like PISAA[1], DoRA[2], which are LoRA-based efficient modules with advanced techniques to improve the effectiveness of PEFT learning, to demonstrate the extensibility of the proposed method. Detailed results and analyses are shown below.

It can be concluded from the results that (1) more advanced PEFT technique is beneficial to better performance of model merging, and exhibits overall better performance; (2) our method achieves consistent and substantial improvements against existing merging methods across all PEFT methods. It strongly certificates that the problem of direction robustness widespread exists in merging all kinds of PEFT modules, where attention is primarily paid to improving the performance of single task. By contrast, our method handles the issue to some extent, thereby promoting the performance when the PEFT technique varies.

We firmly agree with your opinion and will treat exploration to more types of parameter efficient tuning merging as potential future direction. We will cite and add discussion about this in the updated manuscript.

[1] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024.

[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.

Q2. More experiments on other MLLMs

Thank you for your constructive suggestions. We firmly agree with you that validating on other MLLMs is crucial for the extensibility of the proposed method. Following your suggestion, we conduct additional experiments on Qwen-VL and the results are shown below.

It is indicated that compared with existing merging strategies, our method also achieves significant improvement on Qwen-VL (+3.76%), strongly demonstrating its effectiveness across different architectures.

Q3. More analysis on task relationships

We totally agree with you that demonstrating the phenomenon of task interference would further strengthen the effectiveness of the method. To this end, would explain it from three aspects.

1) We calculate the average similarity of each corresponding singular vector between task-specific models and merged models to reflect the direction deviation in merging. Larger similarity means the direction possesses more robustness and is not prone to change direction during merging, thereby maintaining performance of specific task. Moreover, we also incorporate ratio of singular value between merged and original models to comprehensively reflect the degree of specific knowledge learning.

Empirically taking merging ScienceQA and ImageNet in Fig.3b as an example, we present similarity of singular vector and ratio of singular value between task-specific and merged model. As is analyzed in the paper (Fig.3a and Fig.3b), the largest singular value displays more direction robustness, so we divide the value into largest and average of remaining parts for better illustration. Results are shown below.

It can be concluded that (1) during merging, the largest vector tends to be stable, while remaining vectors are extremely dissimilar (direction instability), which leads to performance drop in evaluation. By contrast, our method substantially improves the similarity of remaining vectors (x1.7), strongly promoting merging performance (line144-146); (2) Moreover, ratio of value also reflects that compared to specific model, existing method would decrease the largest singular value and fail to sufficiently strengthen smaller singular values. By contrast, our method better enhances smaller values and maintains specific knowledge during merging, which is in consistent with the paper that scaling smaller values contributes to direction robustness (line153-157).

2) From the definition established above, we additionally conduct ablation study about the similarity to certificate the effectiveness of the proposed pruning and scaling, and the results are shown below.

These analyses give a more clear explanation about the function of each component and how our method mitigates the problem of direction instability: (1) Prune is to resolove the interference between tasks while exhibiting the least influence on the direction (line189-192), and the sparsification also boosts the robustness of small values (two opposite conflicting parameters would be mitigated by setting one value to zero); (2) Equipping scale after prune aims to compensate for singular value drop raised by pruning (line 201-203), thereby enhancing the direction robustness, which can be reflected by both similarity and ratio increase.

Quantitative experiments are as follows (Tab.6 in paper) to intuitively display performance gain from the two techniques.

3) We furthermore give an analysis from the perspective of distribution of singular values in different layers in Q3 of Reviewer 5p9X to demonstrate the effectiveness of our method. The summarized conclusion is that higher layers exhibit more stark distributions, which means larger gap between task-specific knowledge with more direction variation, highlighting the need for handling the issue of direction instability, and our method designed for direction robustness well mitigates the problem.

Unfortunately, we are not able to present a picture to display it due to the constraint of the policy. We show qualitative results above and we will add extensive analyses detailedly into the updated manuscript to strengthen our method.

We express our respectful gratitude to the reviewer for your time and effort, and recognition of our paper. We hope the various additional experiments could strengthen our paper and we will revise the paper according to your valuable feedback. If you have any remaining questions or think some experiments are insufficient, feel free to respond to us at your convenience and we will do our best to address all the concerns!

Dear Reviewer f2pm:

We sincerely thank you for your time and effort in reviewing our paper with several insightful suggestions, and we will respond to each question respectively.

Q1 Evaluation of robustness of PEFT models

Thank you for your constructive suggestion. We totally agree that formulating the robustness of specific model contributes to better understanding of the effectiveness of merging method. To better illustrate this part, we use the average similarity of each corresponding singular vector between task-specific models and merged models as the criterion, which reflects the direction deviation in merging. Larger similarity means the direction possesses more robustness and is not prone to change direction during merging, thereby maintaining performance of specific task. Moreover, we also incorporate the ratio of singular value between merged and original models to comprehensively reflect the degree of specific knowledge learning.

Empirically taking merging ScienceQA and ImageNet in Fig.3b as an example, we present similarity of singular vector and ratio of singular value between task-specific model and merged model and compare two merging methods. As is analyzed in the paper (Fig.3a and Fig.3b), the largest singular value displays more direction robustness, so we divide the value into largest and average of remaining parts for better illustration. Results are shown below.

It can be concluded that (1) during merging, the largest vector tends to be stable, while remaining vectors are extremely dissimilar (direction instability), which leads to performance drop in evaluation. By contrast, our method substantially improves the similarity of remaining vectors compared to Task Arithmetic (x1.7), strongly promoting merging performance (line144-146); (2) Moreover, ratio of value also reflects that compared to specific model, existing method would decrease the largest singular value and fail to sufficiently strengthen smaller singular values. By contrast, our method better enhances smaller values and maintains knowledge during merging, which is in consistent with the paper that scaling smaller values contributes to direction robustness (line153-157).

Q2: Intuitive contribution of each component

From the definition established above, we conduct ablation study about the similarity to certificate the effectiveness of the proposed pruning and scaling, and the results are shown below.

These analyses give a more clear explanation about the function of each component: (1) Prune is to resolove the interference between tasks while exhibiting the least influence on the direction (line189-192), and the sparsification also boosts the robustness of small values (two opposite conflicting parameters would be mitigated by setting one value to zero); (2) Equipping scale after prune aims to compensate for singular value drop raised by pruning (line 201-203), thereby enhancing the direction robustness, which can be reflected by both similarity and ratio increase.

Quantitative experiments are as follows (Tab.6 in paper) to intuitively display performance gain from the two techniques.

Due to the constraint of the policy, we are only able to present table instead of better-illustrated figures. We will add the additional analysis properly and detailedly into updated manuscripts to strengthen our method.

We thank again for your insightful and professional comments. We hope the additional definition and analysis can help you have a better understanding of how our method contributes to direction robustness. If you have any remaining question that needs to be further clarified, do not hesitate to respond to us and we will try our best to address all the concerns!