Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Question 1: How can the finer-grained merging of LoRAs preserve the performance of the merged LoRA, particularly in comparison with the approach of selecting individual LoRAs?

Response: Thank you for this insightful question. We can address this from two perspectives: how LoRA-LEGO preserves the capabilities of original LoRAs, and how it compares with the individual LoRA selection approach.

Building upon our previous discussion, the goal of LoRA merging is to preserve all models' capabilities to create a more comprehensive model. However, existing methods often fail to achieve this due to parameter interference, which disrupts the internal structure and capabilities of LoRAs. Our approach addresses this by modularizing LoRAs into finer-grained MSUs, identifying minimal semantic units that can preserve LoRA's internal capabilities while enabling flexible combinations. By aligning the most similar MSUs before merging and implementing flexible rank adjustment, we ensure sufficient parameter space for preserving task-specific knowledge. This approach effectively minimizes parameter interference and preserves the original LoRA capabilities. In some cases, as shown in Table 3 of our paper, the merged model even outperforms individual LoRAs on specific tasks (e.g., RTE), demonstrating how knowledge transfer between similar tasks can enhance overall performance.

Regarding comparison with individual LoRA selection, our paper's Table 4 presents a comprehensive comparison between LoRA-LEGO and the selection approach (selecting top-1 retrieved LoRA). The results show that in IID scenarios (where input prompts have corresponding task LoRAs), while selection performs reasonably well, LoRA-LEGO achieves better average performance by leveraging knowledge transfer between similar tasks. In OOD scenarios (where input prompts lack corresponding task LoRAs), the Selection method's performance significantly degrades compared to LoRA-LEGO. These results further demonstrate LoRA-LEGO's flexibility and effectiveness in preserving and combining model capabilities across different scenarios, offering a more robust solution than individual LoRA selection.

Weakness 2: The evaluation used only 7 LoRAs. It is unclear how the proposed approach may work with more LoRAs.

Question 2: How many LoRAs can be supported by the MSU-based approach, and what performance it can achieve with more (e.g., tens of) LoRAs?

Thank you for raising this important concern about scalability. We acknowledge that LoRA merging becomes increasingly challenging with a larger number of LoRAs due to more severe parameter interference. As explained in our response to Weakness 1, our method's ability to effectively mitigate parameter interference makes it particularly robust in large-scale merging scenarios.

To thoroughly evaluate scalability, we conducted comprehensive experiments in Appendix G, where we tested merging 20 LoRAs from different tasks. The experimental results reveal several key findings in large-scale, multi-domain LoRA merging scenarios:

1. Traditional element-wise merging methods show significant performance degradation or complete failure due to severe parameter interference
2. The ensemble method failed entirely because of substantial differences in output spaces across multiple LoRA models, leading to irreconcilable conflicts
3. DELLA-Merging, while achieving the best performance among baselines through parameter pruning, still struggled to fully address parameter misalignment and knowledge conflicts
4. LoRA-LEGO demonstrated consistently superior performance by effectively structuring and integrating both LoRA knowledge and parameters through MSU, successfully resolving parameter interference issues, and achieving a significant 16% improvement over the best baseline

The detailed results across 20 different tasks are shown in the following table:

Furthermore, this large-scale evaluation validates our method's ability to handle parameter interference effectively, even when merging a substantial number of LoRAs from diverse domains. We believe these results effectively address the concerns about LoRA-LEGO's scalability and demonstrate its potential for large-scale LoRA integration.

Thank you for your insightful and valuable feedback. We will address each of your concerns with a point-by-point response.

Weakness 1: While MSU is an interesting concept, it is unclear why the finer granularity merging of LoRAs is the right approach for combing multiple LoRAs to enhance LLM capabilities: the clustering and the use of the centroid of each cluster essentially impact the performance of the merged LoRA.

Thank you for your valuable feedback. In our paper, we identified parameter misalignment and knowledge conflict as the primary factors causing parameter interference during LoRA merging and demonstrated how these factors can significantly degrade merging performance (detailed in Section 2.3 in our paper). Our MSU-based clustering approach effectively addresses both issues in a novel way.

For parameter misalignment, our clustering-based merging ensures that we combine parameters with similar semantic information. This means we only merge parameters that are as close as possible and likely carry similar semantic information, effectively aligning similar parameters before aggregation. Regarding knowledge conflict, our method's flexible rank adjustment (corresponding to the number of clusters) provides sufficient parameter space to accommodate semantic knowledge from different tasks, thereby avoiding conflicts.

Traditional element-wise merging approaches merge parameters at corresponding positions, ignoring the permutation invariance property where different parameters may represent different semantic information. This approach requires proper alignment to avoid parameter interference. Our method effectively addresses this limitation through MSU-level alignment before merging, making it more flexible and efficient.

To validate this, we conducted additional experiments in Appendix F with highly diverse tasks (RTE, CoPA, IMDB, MRPC, BoolQ, CosmosQA, and ARC) where parameter interference is more pronounced. The results show that LoRA-LEGO significantly outperforms other methods, achieving an 11% improvement over DELLA-Merging, demonstrating effective mitigation of parameter interference through MSU-level alignment and merging:

These comprehensive results demonstrate our method's superior ability to mitigate parameter interference and effectively combine different LoRAs to enhance LLM capabilities.

Continued from the previous page...

To further validate our method's effectiveness in more challenging scenarios, we conducted additional experiments detailed in Appendices F&G:

In Appendix F, we evaluated performance across highly diverse tasks (RTE, CoPA, IMDB, MRPC, BoolQ, CosmosQA, and ARC). LoRA-LEGO significantly outperforms other methods, achieving an 11% improvement over DELLA-Merging, demonstrating effective mitigation of parameter interference through MSU-level alignment and merging.

In Appendix G, we tested scalability by merging 20 LoRAs from different tasks. While traditional methods showed significant degradation or complete failure, and DELLA-Merging struggled with parameter misalignment, LoRA-LEGO maintained robust performance with a 16% improvement over the best baseline.

These comprehensive evaluations across different settings (model scales, task diversity, and large-scale merging) demonstrate our method's consistent superiority in handling parameter interference and maintaining performance in challenging scenarios. We believe these extensive comparisons thoroughly validate the effectiveness of our approach.

[1] Yu, Le, et al. "Language models are super mario: Absorbing abilities from homologous models as a free lunch." Forty-first International Conference on Machine Learning. 2024.

[2] Deep, Pala Tej, Rishabh Bhardwaj, and Soujanya Poria. "DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling." arXiv preprint arXiv:2406.11617 (2024).

Weakness 3: It does not report the overhead (e.g., in inference and training time, memory) that may introduced by this approach.

We have conducted detailed overhead analysis using an A100 GPU with a batch size of 16. For fair comparison, we measured inference time based on first-token generation, as output lengths may vary across models. Importantly, LoRA-LEGO is training-free, incurring no additional training overhead.

Here are the detailed results:

As shown, LoRA-LEGO achieves competitive inference performance with inference time comparable to other methods, similar memory consumption, and significantly higher accuracy (62.21%). These results demonstrate that LoRA-LEGO delivers superior performance without introducing significant computational overhead, making it both effective and efficient in practical applications.

Thank you for your careful and insightful feedback. We will address each of your concerns with a point-by-point response.

Weakness 1: The MSU is not related to some semantics that can be understood by humans, it would be better if we could see some analysis.

Response: While directly linking MSUs to specific human-interpretable concepts remains challenging, as it would require manually constructing concept-specific datasets and analyzing their relationships with LoRA parameters, we have conducted alternative analyses to investigate the semantic structure of MSUs.

To explore whether MSUs capture structural semantic information, we conducted a comprehensive analysis using 48 LoRA models, each trained individually on different datasets from Flan-v2 and grouped into 10 distinct task clusters. We visualized the distribution of MSUs using t-SNE projections of the first layer parameters from these LoRAs (detailed visualization provided in Appendix J).

Our analysis reveals two key findings:

1. MSUs within a single LoRA model cluster more closely together, reflecting their shared data distribution
2. LoRA models from similar task domains exhibit MSU proximity in the t-SNE space

These patterns suggest that MSUs do capture meaningful task-related information, supporting our approach of considering MSU similarity in the merging process to reduce parameter interference and preserve semantic integrity.

Weakness 2: Although the proposed method outperforms existing approaches, a more detailed comparison with a wider range of baseline methods could strengthen the claims.

Thank you for this suggestion. To ensure a comprehensive evaluation, we have strengthened our comparison by including two additional established baselines: DARE[1] and DELLA-Merging[2]. Our evaluation spans multiple scenarios and model scales to thoroughly validate our method's effectiveness.

First, we present results across multiple tasks using both Llama2-7b and Llama2-13b models (Table 3 in the revised manuscripture). For Llama2-7b, LoRA-LEGO achieves the best average performance (62.21%). This advantage extends to Llama2-13b, where our method maintains superior performance (70.27%) with significant improvements in tasks like SNLI (58.54%) and WNLI (64.29%).

Continued on next page...

Question 4: Why does the weight averaging method in Table 3 achieve identical performance to the task-specific LoRA (upper bound) for the MRPC task?

Response: Thank you for this astute observation. This interesting phenomenon can be understood as an extension of our discussion from Question 2 regarding the characteristics of the MRPC task. To investigate this further, we conducted an additional experiment to verify our hypothesis about knowledge transfer between tasks.

We performed two sets of merging experiments:

1. Original weight averaging including all LoRAs
2. Weight averaging excluding the MRPC LoRA

The results are shown below:

These results reveal several interesting insights:

• Even without including the MRPC LoRA, the merged model achieves nearly identical performance to the task-specific MRPC LoRA
• This suggests that parameters from other tasks can effectively transfer to the MRPC task
• The similarity in performance indicates that knowledge from related tasks (e.g., QQP, which also involves sentence pair similarity classification) can complement and potentially enhance MRPC task performance

This phenomenon demonstrates the potential for positive knowledge transfer between similar tasks through model merging, suggesting that well-designed merging strategies can actually improve generalization across related tasks. It also aligns with our earlier findings about MRPC's parameter structure and its amenability to parameter sharing and regularization through merging.

Weakness 2: Merging effectiveness can be highly sensitive to hyperparameter settings.

Question 3: How can the optimal number of clusters k in equation (2) be determined?

Response: Thank you for raising this important concern about hyperparameter sensitivity. We would like to emphasize that LoRA-LEGO demonstrates remarkable robustness to hyperparameter settings. The only hyperparameter in our method is the cluster number $k$, which corresponds to the rank of the merged LoRA.

This robustness stems from our discovery of a general relationship between the LoRA output scale and merged rank, as illustrated in Figure 7 of our paper. After implementing our dual-reweighting strategy (shown by the red line in Figure 7), our method maintains stable performance across different cluster numbers $k$. In contrast, without dual-reweighting, performance varies significantly due to output scale variations. This stability can be attributed to two key factors:

1. Our dual-reweighting strategy effectively normalizes the output scale of merged LoRAs, making the performance more consistent across different cluster numbers
2. The MSU-based clustering approach naturally groups similar parameters together, making the method less sensitive to the exact choice of $k$

This inherent stability allows us to use fixed cluster numbers across all experiments while maintaining consistent performance. The dual-reweighting strategy effectively addresses the scale variation issue that typically makes merging methods sensitive to hyperparameter choices. We have expanded the "Ablation of Scaling Strategies" section to include a detailed discussion of hyperparameter robustness and the role of dual-reweighting in achieving this stability.

Question 2: Why does performance improve for MRPC and RTE tasks in Table 1 following permutation merging?

Response: Thank you for this insightful question. The performance improvement in MRPC and RTE tasks can be attributed to the unique characteristics of these tasks and their internal MSU structures. We have conducted a detailed analysis to understand this phenomenon.

In Appendix I, we plotted the average Euclidean distances between MSUs in each layer across all LoRAs. The analysis reveals that MRPC and RTE exhibit notably smaller internal MSU distances compared to other tasks. This smaller distance indicates:

1. These tasks are relatively simple with smaller datasets
2. Their internal MSUs show higher similarity
3. Their parameters potentially contain redundancy and may be prone to overfitting

When applying permutation merging $(A+PA)/2, (B+BP^T)/2$, it effectively acts as a form of parameter smoothing or regularization. For tasks like MRPC and RTE, where MSUs are more closely connected (as evidenced by smaller distances), this regularization actually helps reduce overfitting, leading to performance improvements. However, for tasks where MSUs are more distant from each other, such permutation merging can cause significant performance degradation.

This observation further validates the importance of considering MSU connectivity in LoRA merging strategies. Our clustering-based approach in LoRA-LEGO specifically accounts for these connectivity patterns, merging more closely aligned MSUs to effectively mitigate parameter interference while preserving task-specific knowledge. This explains why LoRA-LEGO achieves consistent performance improvements across diverse tasks, regardless of their internal MSU structure.

Thank you for your thoughtful questions and constructive feedback. We will address each of your concerns with a point-by-point response.

Weakness 1: Performance may degrade when merging LoRAs designed for highly divergent tasks.

Question 1: How does LoRA-LEGO address performance degradation when merging LoRAs for tasks with significant divergence?

Response: We acknowledge that highly divergent tasks present a significant challenge for model merging methods due to increased parameter interference. LoRA-LEGO performs merging by clustering MSUs, which first ensures that all aggregated parameters are sufficiently aligned by grouping similar ones. Through this approach, parameter alignment is achieved. Additionally, the flexibility of the clustering algorithm allows us to freely choose the rank of the merged LoRA, ensuring that the final LoRA has enough parameter space to avoid knowledge conflict by adjusting the rank. These features provide significant advantages over traditional element-wise merging methods.

To validate our method's effectiveness with diverse tasks, we first conducted experiments in Appendix F using LoRAs trained on various tasks:

• RTE (natural language inference)
• CoPA (commonsense reasoning)
• IMDB (sentiment analysis)
• MRPC (paraphrase detection)
• BoolQ (reading comprehension)
• CosmosQA (reading comprehension with commonsense)
• ARC (closed-book question answering)

These tasks represent significantly different domains and objectives. The results are presented in Table 1:

Table 1: Performance on Diverse Tasks

To further evaluate scalability, we conducted more challenging experiments merging 20 LoRAs from diverse tasks in Appendix G. The results are shown in Table 2:

Table 2: Large-scale Integration of 20 LoRAs

These comprehensive experiments reveal several key findings:

1. In the initial diverse task setting, traditional methods suffer from severe performance degradation, while LoRA-LEGO achieves an 11% improvement over the best baseline.
2. In the 20-LoRA scenario, the challenges become more pronounced:
   • Element-wise merging methods show significant performance drops or complete failure
   • The ensemble method fails entirely due to conflicts in output spaces
   • DELLA-Merging, while performing better than other baselines by using magnitude pruning to reduce the impact of large task numbers and task diversity, still fails to fully address the fundamental challenges arising from parameter misalignment and knowledge conflict.
   • LoRA-LEGO maintains robust performance, achieving a 16% improvement over the best baseline

These results demonstrate that LoRA-LEGO effectively addresses the challenge of merging LoRAs from highly divergent tasks, maintaining strong performance even in scenarios with significant task diversity and scale.

Weakness 3: "The baselines involving LoRA composition methods are basic and are not established methods in the existing literature. It lacks citations that would align them with related work. This weakens the comparative analysis and raises questions about the validity of the selected baselines."

Response:

Thank you for raising this important concern about our baseline selection and related work discussion. We would like to clarify our baseline selection criteria and present additional comparisons.

Given that our method is training-free, we focused on comparing it with other training-free model merging methods to ensure fair comparison. We have strengthened our evaluation by including two additional methods: DARE[1] and DELLA-Merging[2]. As demonstrated in the following experimental results, LoRA-LEGO significantly outperforms these strong baselines, further validating our method's effectiveness.

We have also expanded our discussion of related work in Appendix H, providing a more detailed discussion of current model merging methods. We welcome suggestions for additional baseline methods or related work comparisons and are prepared to conduct supplementary experiments during the rebuttal period to address any specific concerns.

[1] Yu, Le, et al. "Language models are super mario: Absorbing abilities from homologous models as a free lunch." Forty-first International Conference on Machine Learning. 2024.

[2] Deep, Pala Tej, Rishabh Bhardwaj, and Soujanya Poria. "DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling." arXiv preprint arXiv:2406.11617 (2024).

Weakness 2: "The approach is not particularly strong, and this simplicity may limit the framework's performance which is reflected in the empirical results. While the average performance across tasks improves, it does not apply to all tasks. For example, performance remains poor for WNLI, an out-of-domain task for multi-task learning."

Response: Thank you for raising these concerns about our method's performance. We would like to provide additional evidence and clarification.

Our approach demonstrates substantial and statistically significant improvements over baseline methods across several benchmarks. Specifically, on the Llama2-7b model, our method outperforms the best baseline by 6.7%, and on Llama2-13b, it achieves a 1% improvement.

Furthermore, we conducted additional experiments outlined in Appendix F and Appendix G to validate our method in more diverse and scaled task settings:

1. Performance on Diverse Tasks (Appendix F): We evaluated our method's performance when integrating LoRA models from highly diverse tasks, including:

• RTE (natural language inference)
• CoPA (commonsense reasoning)
• IMDB reviews (sentiment analysis)
• MRPC (paraphrase detection)
• BoolQ (reading comprehension)
• CosmosQA (reading comprehension with commonsense)
• ARC (closed-book question answering)

In this challenging scenario with increased parameter interference due to task diversity, LoRA-LEGO significantly outperforms other methods, achieving an 11% improvement over the best baseline (DELLA-Merging). This demonstrates our method's effectiveness in mitigating parameter interference through MSU-level alignment and merging.

1. Scalability Analysis (Appendix G): We further evaluated the scalability of all methods by merging 20 LoRAs from different tasks. The results show that:

• Traditional element-wise merging methods experience significant performance degradation
• The ensemble method failed completely due to conflicts in output spaces
• DELLA-Merging, while performing best among baselines, still struggled with parameter misalignment
• LoRA-LEGO maintained superior performance, achieving a 16% improvement over the best baseline

These comprehensive evaluations demonstrate that our method is particularly effective in scenarios with high task diversity, large-scale parameter merging, and significant parameter interference. The experimental results in Appendices F and G provide strong evidence of our method's robustness and effectiveness, particularly in challenging scenarios where parameter interference is more pronounced. We believe these additional experiments address your concerns about the method's performance limitations.

We sincerely thank you for your valuable feedback, and we are grateful for the time and effort you have invested in reviewing our work. Below, we provide a point-by-point response to address each of your concerns:

Weakness 1: "The paper is generally well written with good illustrations. However, I found the approach not particularly novel. It relies on a straightforward k-means clustering method for grouping and subsequently aggregating cluster parameters."

Response: We appreciate your feedback and would like to clarify the main contributions and novelty of our approach.

The core contribution of our work lies in the further modularization of LoRA, enabled by the novel concept of MSUs and the proposed MSU clustering framework for LoRA merging. Prior work has not recognized that LoRA merging can be achieved through finer-grained parameter clustering, and such an approach offers greater flexibility and alleviates parameter interference. Moreover, our method can be used for merging heterogeneous LoRAs and applied to a singular LoRA for parameter pruning, which demonstrates the flexibility and universality of our approach across various problems.

Regarding the use of k-means clustering, we respectfully disagree with the notion that its simplicity detracts from the novelty of our approach. We believe that the true value of a contribution often stems from new insights and innovative perspectives, rather than the complexity of the techniques employed. Our choice of simple, yet effective methods was deliberate, as simplicity enhances the clarity, interpretability, and accessibility of our work, making it more adaptable for the broader community.

We also believe that our method introduces an innovative viewpoint on LoRAs, inspiring potentially impactful research directions in modular model adaptation. For example:

• Continuously adding MSUs to LoRA can expand the model's capabilities for Lifelong Learning
• Sharing and integrating different LoRA MSUs can enable more efficient LLM Federated Learning
• Model pruning can be achieved through the integration of internal LoRA MSUs
• Dynamic adjustment of LoRA MSUs can lead to more efficient LoRA training

Through this unique perspective and our novel approach to LoRA merging via MSU clustering, we believe our method offers significant novelty and inspiration to the field. We hope that this clarification highlights the novelty and potential utility of our approach and addresses your concern regarding our contribution.

Dear Reviewer 1z1o,

We sincerely appreciate the valuable suggestions and comments you provided for our paper. We have carefully considered each point and have addressed them in detail in our rebuttal.

As the Author-Review Discussion period draws to a close, we want to ensure that all your concerns have been adequately addressed. If there are any questions or unresolved issues, we are eager to provide further clarification or make necessary revisions.

Best regards,

The Authors

Dear Reviewers,

Thank you for your invaluable feedback on our paper. Your comments and suggestions have been very helpful in improving the clarity and quality of our work.

As the discussion period deadline approaches, we would like to confirm that we have addressed all your concerns and resolved any remaining issues. We are also open to any additional comments or suggestions you might have and are happy to make further clarification if needed.

Thank you once again for your time, thoughtful input, and dedication throughout this process.

Best regards,

The Authors