Ensembles of Low-Rank Expert Adapters

Dear Reviewers,

Thank you very much for your thoughtful review and valuable suggestions on our manuscript. We greatly appreciate your time and effort in evaluating our work. We have carefully considered your comments and have made corresponding revisions to address your concerns.

Weaknesses

1. Method efficiency: We understand your concern regarding the efficiency and computational cost of our model. In response, we have conducted both theoretical and empirical analyses, which are now included in Appendix G "Efficiency Analysis" of the revised manuscript. Specifically, for a 4-expert system (in addition to the base adapter), we found that the fine-tuning and inference resource consumption is approximately 2.3 times that of using the base adapter alone. While this increase is not negligible, we believe it is acceptable in practice given the performance improvements. Nonetheless, we acknowledge that efficiency is crucial, and we are actively working on methods to reduce computational costs in our ongoing research.

2. Baselines: We appreciate your suggestion to include additional baselines for a more robust comparison. In the revised Section 5 "Results and Discussion," we have incorporated results from MoLE [1] and provided an in-depth discussion of its performance relative to our model. Regarding LoraHub [3], we found that its primary objective differs from ours, as it focuses on selecting and combining adapters for downstream unseen tasks based on a few examples. Since our work assumes in-domain test samples without task-specific examples during inference, we believe LoraHub may not be a directly comparable baseline. However, we acknowledge its relevance and have included a discussion of LoraHub in the related works section.

Questions

1. Top-k experts during inference: Thank you for this insightful suggestion. Activating only the top-1 or top-4 experts during inference could indeed reduce computational costs, but it also leads to reduced performance. We illustrated these findings and provided a discussion in Figure 3(b) and the paragraph below (Lines 502--506) in the updated manuscript.

2. Clustering efficiency: We agree that clustering efficiency is an important consideration. Our clustering pipeline, as introduced in Section 3.3 "Clustering and Per-Cluster Fine-Tuning," is designed to be highly efficient. As mentioned in Appendix G "Efficiency Analysis," the clustering step requires minimal computational effort and does not significantly impact the overall training time.

We hope that our revisions and additional analyses have addressed your concerns. Your feedback has been invaluable in improving the quality of our work. We hope that our revised manuscript meets the ICLR quality standards and would be grateful if you would consider updating your evaluation accordingly.

Please do not hesitate to let us know if you have any further questions or suggestions. We are more than happy to engage in further discussions to enhance our work.

Best regards,
The Authors

[1] Wu, Xun, Shaohan Huang, and Furu Wei. "Mixture of LoRA Experts." The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=uWvKBCYh4S.
[2] Chen, Shaoxiang, Zequn Jie, and Lin Ma. "Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms." arXiv preprint arXiv:2401.16160 (2024).
[3] Huang, Chengsong, et al. "Lorahub: Efficient cross-task generalization via dynamic lora composition." arXiv preprint arXiv:2307.13269 (2023).

Dear Reviewer,

Thank you again for taking the time to review our responses and for your thoughtful consideration. We are glad that we were able to address your concerns, and we appreciate you raising your rating.

Best regards,
The Authors

Dear Reviewer,

Thank you very much for your thoughtful and constructive feedback on our work. We sincerely appreciate your support and have addressed your questions and concerns below.

Weaknesses

1.1 Discussion on the advantages of MoE: We greatly appreciate your suggestion regarding the advantages of MoE. In response, we have added a discussion on the potential benefits of MoE in the revised Section 2.3, “MoE and Ensembles.” We have also included related experimental results in the revised Section 5, “Results and Discussion” (see two paragraphs starting from Line 371). We hope that this discussion effectively emphasizes the advantages of MoE and strengthens our claims.

1.2 Trainable router: Thank you for this insightful suggestion. Your proposed method resembles the “Mixture of LoRA Experts (MoLE)” method [1] suggested by other reviewers. In the revised Section 5, "Results and Discussion," we have incorporated results from MoLE and provided an in-depth discussion of its performance relative to our model.

2. The algorithm’s time complexity: We appreciate your concern about the algorithm's time complexity. In response, we have provided theoretical analyses and empirical experiments on the resource consumption of ELREA compared to fine-tuning only the base adapter in Appendix G, "Efficiency Analysis," of the revised manuscript. Specifically, for a 4-expert system (in addition to the base adapter), we found that the fine-tuning and inference resource consumption is approximately 2.3 times that of using the base adapter alone. While this increase is not negligible, we believe it is acceptable in practice given the performance improvements. Nevertheless, we acknowledge the importance of efficiency and are actively working on methods to reduce computational costs in our ongoing research.

Questions

1. Training lower-rank LoRAs on clusters: Thank you for this practical suggestion. It is indeed possible to use lower-rank LoRAs on the clustered data for faster adaptation. Our main consideration is that, in our experiments, the LoRA experts are initialized from the base adapter, and using lower-rank experts may result in a dimension mismatch. While we can resolve this issue by applying SVD on LoRA weights, implementing this solution requires additional effort. We are committed to exploring this idea in our future work to reduce computational overhead.

2. Cluster pattern: Yes, we have observed a clear correlation of tasks within each cluster. We have added an illustration and analysis in Appendix H “Further Analysis on Data Clustering” of the revised article to provide more insight into this pattern.

We hope that our responses have satisfactorily addressed your questions and concerns. Your valuable feedback has significantly improved our work. If you find our revisions meet your expectations, we kindly hope you might consider reflecting this in your evaluation. Please do not hesitate to reach out if you have any further questions or require additional clarification; we would be delighted to continue the discussion.

Best regards,
The Authors

[1] Wu, Xun, Shaohan Huang, and Furu Wei. "Mixture of LoRA Experts." The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=uWvKBCYh4S.

Dear Reviewer,

Thank you for reviewing our revised manuscript and for your feedback. We're glad some of your concerns have been addressed. Please let us know if there are specific areas we can improve further.

Best regards,
The Authors

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. Your insightful comments are invaluable to us, and we believe they will greatly enhance the quality of our work. Below, we address your questions and concerns in detail.

Weaknesses

1. Method complexity: Thank you for highlighting the need for a detailed discussion on computational overhead. In the revised version of our manuscript, we have included a comprehensive analysis in Appendix G “Efficiency Analysis”. Briefly, we found that ELREA with four expert adapters requires approximately $2.3\times$ the computational resources needed for fine-tuning the base adapter. While this represents an increase, we consider it acceptable given the performance improvements achieved. We are also actively exploring ways to reduce computational overhead in our ongoing research to make our method more practical in real-world scenarios.

2. Comparison with baselines: We are grateful for your suggestions regarding baseline methods. To strengthen our evaluation, we have included results from the Mixture of LoRA Experts (MoLE) [2] in the revised Section 5, "Results and Discussion." MoLE differs from ELREA by fine-tuning additional layer-wise gating functions to dynamically assign weights to expert adapters. We have provided an in-depth analysis of its performance relative to our model.
   Regarding the methods mentioned in references [3,4,5], we acknowledge their relevance and have discussed them in the related work section. These methods differ from ELREA in that they simultaneously train all LoRA experts and gating functions on the entire dataset, which is outside the scope of our current setup. Implementing these baselines would require significant additional time and resources, which may not be feasible within the current discussion period. However, we are committed to including these comparisons in future versions of our work.

3. Performance gain: This point is addressed below.

[1] Fort, Stanislav, Huiyi Hu, and Balaji Lakshminarayanan. "Deep Ensembles: A Loss Landscape Perspective." arXiv preprint arXiv:1912.02757 (2019).
[2] Wu, Xun, Shaohan Huang, and Furu Wei. "Mixture of LoRA Experts." The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=uWvKBCYh4S.
[3] Li, D., Ma, Y., Wang, N., et al. "MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA Based Mixture of Experts." arXiv preprint arXiv:2404.15159, 2024.
[4] Wang, Y., Lin, Y., Zeng, X., et al. "MultiLoRA: Democratizing LoRA for Better Multi-Task Learning." arXiv preprint arXiv:2311.11501, 2023.
[5] Liu, Q., Wu, X., Zhao, X., et al. "MoELoRA: An MoE-Based Parameter Efficient Fine-Tuning Method for Multi-Task Medical Applications." arXiv preprint arXiv:2310.18339, 2023.

Questions

1. Computation overhead: We have conducted both theoretical and empirical analyses, which are now included in Appendix G "Efficiency Analysis" of the revised manuscript. Specifically, for a 4-expert system (in addition to the base adapter), we found that the fine-tuning and inference resource consumption is approximately 2.3 times that of using the base adapter alone. While this increase is not negligible, we believe it is acceptable in practice given the performance improvements. Nonetheless, we acknowledge that efficiency is crucial, and we are actively working on methods to reduce computational costs in our ongoing research.

2. Stability of clustering: In our preliminary experiments, we observed that changing the random seed for sampling had minimal impact on the clustering results. Specifically, for most random seeds (about 8 out of 10), the distribution of data points among clusters remained consistent, with only about a 5% variation in a few cases. These findings suggest that our clustering method is stable and reliable. We added related content in Section 3.3 “Clustering and Per-Cluster Fine-Tuning”. In addition, we added a set of investigations in the data pattern in each cluster in Appendix H “Further Analysis on Data Clustering”.

3. Explanation of Performance Gain: We understand your concern about the source of the performance gain. We believe you are referring to the comparison between ELREA and the Random Cluster approach when r=8 in Table 1. The Random Cluster method, as detailed in Appendix E, is similar to the Deep Ensembles approach, which has been shown to effectively explore the loss landscape by converging to different minima [1]. Since we use the standard token-wise cross-entropy loss, the training data primarily defines the loss landscape. By training different LoRA adapters that converge to diverse minima, we enhance the ensemble model's representation capability, leading to better performance.
   Our ELREA framework can be seen as a specialized ensemble model that systematically diversifies each LoRA adapter, rather than relying on the stochastic nature of Deep Ensembles. This structured approach aims to provide more consistent performance gains. To further investigate, we conducted additional experiments on the MATH-Combined dataset with four extra LoRA components, each trained on the entire dataset from scratch. Due to computational constraints, we limited the number of LoRAs. The results, now included in the revised Table 1 under "LoRA Ensembles," show that while LoRA Ensembles do provide consistent performance improvements, ELREA achieves a larger gain, aligning with our expectations.

Once again, we thank you for your thoughtful feedback. We hope our responses have addressed your concerns. Your feedback has been invaluable in improving the quality of our work. We hope that our revised manuscript meets the ICLR quality standards and would be grateful if you would consider updating your evaluation accordingly.

Please do not hesitate to let us know if you have any further questions or suggestions. We are more than happy to engage in further discussions to enhance our work.

Best regards,
The Authors

Dear Reviewer,

Thank you again for your valuable feedback on our submission. In our response, we tried our best to address your comments and hope our clarifications resolve any concerns. We kindly invite you to review our replies, and let us know if you have further questions. Thank you for your time and consideration.

Best regards,
The Authors

Dear Reviewer,

As the rebuttal session is coming to an end, we kindly ask you to review our updates and questions regarding your response, and let us know how we can further strengthen our arguments beyond the provided results. Thank you for your time and consideration.

Best regards,
The Authors

Dear Reviewer,

Thank you for your thoughtful review and supportive comments on our work. We are delighted to hear that you found our paper well-written and interesting. Our goal is to make our research accessible even to audiences who may not be deeply familiar with the latest developments in LLM fine-tuning. We sincerely appreciate your feedback and have addressed your concerns below. Please let us know if there's anything else we can do to further enhance the readability and robustness of our work.

Weaknesses

1. Method efficiency: We understand your concern regarding the computational and memory overhead of ELREA. To address this, we have conducted both theoretical and empirical analyses of the method's efficiency, which are now included in Appendix G, "Efficiency Analysis," of the revised manuscript. Specifically, for a system with four expert adapters (in addition to the base adapter), we found that the fine-tuning and inference resource consumption is approximately 2.3 times that of using the base adapter alone. While this increase is notable, we believe it is justified by the significant performance improvements ELREA offers. Nonetheless, we acknowledge the importance of efficiency and are actively exploring ways to reduce computational costs in our ongoing research.

2. The effect of clustering methods: We fully agree that the robustness of the clusters and the choice of clustering algorithm are critical to ELREA's performance. We dedicated considerable effort to selecting a clustering method that balances scalability with robustness. Through several smaller-scale experiments during the early stages of our algorithm's development, we determined that BIRCH is the best-performing clustering method for our purposes. It offers decent scalability with respect to data size and feature dimensionality and demonstrates robustness to outliers. Additionally, we experimented with different random seeds and observed consistent cluster formations, indicating that the clustering method is stable across various scenarios. We have expanded our discussion on this topic in Appendix H “Further Analysis on Data Clustering” to provide a more in-depth analysis on the data patterns in each cluster.

Questions

1. Inclusion of the base adapter: Your insight is indeed correct. When we fine-tune a low-rank adapter on a smaller, skill-specific subset, the model can lose some of its generalization capability—specifically, information related to other clusters. To mitigate this, we retain the base adapter during the inference phase to maintain access to general knowledge that may be necessary. We have included a brief discussion on this issue between Equations (9) and (10) in the revised manuscript to clarify this point.

We hope that our responses have satisfactorily addressed your questions and concerns. Your valuable feedback has significantly improved our work. If you find our revisions meet your expectations, we kindly hope you might consider reflecting this in your evaluation. Please do not hesitate to reach out if you have any further questions or require additional clarification; we would be delighted to continue the discussion.

Best regards,
The Authors

Dear Reviewer,

Thank you for your thoughtful consideration and positive feedback. We are glad to hear that our rebuttal addressed your concerns effectively. We appreciate your continued support of our work.

Best regards,
The Authors

Dear Reviewer,

Thank you again for your valuable feedback on our submission. In our response, we tried our best to address your comments and hope our clarifications resolve any concerns. We kindly invite you to review our replies, and let us know if you have further questions. Thank you for your time and consideration.

Best regards,
The Authors

Thanks for your rebuttal. It has addressed my concerns well. After carefully considering the opinions from the other reviewers, I have decided to maintain my positive score.

Dear Reviewer,

Thank you very much for your careful review of our paper and your insightful suggestions. We are grateful for your feedback, which we believe will significantly improve the quality and robustness of our work.

• Placement of the “Related Works” section: We appreciate your suggestion regarding the structure of the paper. Currently, we cited the critical works that our article is based on in the Introduction and Preliminaries and others in the “Related Works” section in Appendix A. Due to length limitations, we may not be able to put the full-fledged related works in the article body.

• Limit of novelty: Thank you for sharing your thoughts on our work. We would like to clarify that while we utilize the gradient calculation technique from [1], the rest of our approach, including the overall pipeline, clustering methodology, and inference routing weight calculation, is independently developed. Our work aims to integrate these components in a novel way to address the challenges in LLM fine-tuning. We will make sure to emphasize this distinction more clearly in the revised manuscript to avoid any potential misunderstandings.

• Per-cluster fine-tuning and LoRA: We appreciate your thoughts on the relevance of LoRA in per-cluster fine-tuning. Allow us to elaborate on our motivation for exclusively using LoRA adapters. Our primary goal was to maintain computational efficiency comparable to training a single adapter on the entire dataset. Full-parameter fine-tuning for each expert would significantly increase the computational and memory requirements, as it would necessitate storing multiple copies of the full model weights. In contrast, using LoRA adapters allows us to keep the additional memory footprint minimal, as each adapter constitutes only a small fraction (usually 0.1% – 3%) of the backbone model's size. We also added an analysis of ELREA's efficiency in Appendix G "Efficiency Analysis" in the revised manuscript.
  Regarding alternative methods like soft prompt tuning, they often require specific task indicators or prior knowledge of downstream tasks, which may not be applicable in our setup. Therefore, we believe that cluster-wise LoRA fine-tuning is the most practical and efficient approach for our scenario. We are also open to further discussion if you have additional suggestions or insights.

• Comparison with LoRA baselines: Thank you for suggesting the inclusion of additional baselines such as mixture-of-LoRA-expert methods [2,3]. We agree that incorporating these comparisons would strengthen our evaluation. In response, we have included results from MoLE [3] in the revised Section 5, "Results and Discussion," and provided an in-depth analysis of its performance relative to our model.
  Regarding HydraLoRA [2], we acknowledge its relevance and have discussed it in the related works section. However, due to differences in its setup compared to ELREA and the time constraints of the rebuttal period, implementing it for a fair comparison may not be feasible at this moment. Nevertheless, we are committed to including such comparisons in future iterations of our work to provide a more comprehensive evaluation.

• Typo and clarity issues: Thank you so much for pointing out the minor grammatical mistakes and inconsistencies. We appreciate your attention to these details and have ensured that the revised manuscript reflects these corrections.

Once again, we are grateful for your constructive feedback and believe that your suggestions will help enhance the clarity and impact of our work. We hope that our revised manuscript meets the ICLR quality standards and would be grateful if you would consider updating your evaluation accordingly. Please do not hesitate to let us know if you have any further questions or suggestions. We are more than happy to engage in further discussions to enhance our work.

Best regards,
The Authors

[1] LESS: Selecting Influential Data for Targeted Instruction Tuning. 2024 ICML.
[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. 2024 NeurIPS.
[3] Mixture of LoRA Experts. 2024 ICLR.

Dear Reviewer,

Thank you again for your valuable feedback on our submission. In our response, we tried our best to address your comments and hope our clarifications resolve any concerns. We kindly invite you to review our replies, and let us know if you have further questions. Thank you for your time and consideration.

Best regards,
The Authors