HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below：

W1: HydraLoRA still underperforms full fine-tuning.

• HydraLoRA is more efficient. HydraLoRA offers the advantage of low training overhead, allowing LLMs to adapt to specific domain tasks more efficiently than Full Fine-Tuning (FFT). Although HydraLoRA may not match FFT in model performance, as depicted in Figure 2, FFT adjusts all parameters (as shown in Table 2, FFT modifies tunable parameters about 800 times higher than HydraLoRA), which better captures downstream task features but also incurs substantial costs that may be prohibitive for end-users. FFT cannot construct efficient multi-head structures like LoRA. Meanwhile, compared to other PEFT methods, HydraLoRA minimizes this performance gap with FFT, as shown in Table 2.

• HydraLoRA is more robust and adaptive. As the downstream tasks dynamically evolve, the overhead of re-running the FFT process is significant. However, HydraLoRA easily adapts to the changes due to its plug-and-play and asymmetric architecture.

W2: Ablation where only one of the B matrices.

Thanks for your constructive comment. We add more experiments with the same setting as Table 3, to explore how the number of experts (B matrices) during the HydraLoRA inference pipeline influences performance. As shown in the table below, we find that an increase in the number of B matrices generally leads to enhanced performance in downstream tasks. In practice, user requests may belong to different tasks, while a single request potentially involves mixed tasks. This improvement can be attributed to the expanded configuration space afforded by additional LoRA modules, which allows for a more fine-grained and tailored adaptation to the diverse and mixed-task inputs encountered in the benchmark.

Table: Sensitivity analysis of the number of B matrices. ‘Base’ means vanilla Llama2-7B, Top-1 means selecting the highest-ranked (top-1) B matrix, and Top-3 means selecting three highest-ranked (top-3) B matrices.

Q1: Typos.

Thanks for pointing out them. We will correct all the typos in the updated version.

Q2: How is corpus heterogeneity measured?

Heterogeneity signifies the diversity within the dataset. To visualize the diversity, we adopt similarity between task embeddings for different tasks. We place an example heatmap figure in the overall rebuttal pdf.

Q3: How does lora_split in Table 1 classify the data?

To simulate real-world scenarios, we cannot know in advance the domains of data that require fine-tuning. Therefore, Split-LoRA, a baseline we proposed, performs k-means clustering on the data and then fine-tunes it for different categories. This approach underscores the importance of exploiting asymmetry in HydraLoRA.

Q4: The performance of Split_LoRA in Table 3.

For a single dataset, no existing studies have discussed multi-LoRA fine-tuning methods, prompting us to introduce the LoRA-Split variant. In contrast, Table 3 focuses on multi-task scenarios, where numerous methods[1,2,3,4] already exist. Therefore, we directly compared our approach with established LoRA MoE methods [1,2].

Q4: L193 “With equivalent parameters (rank=16)”

In Table 2, HydraLoRA (r=8) refers to each A/B matrices with a rank of 8, yet the total parameter count is equivalent to a single LoRA module with a rank of 16, due to multiple B matrices. Meanwhile, HydraLoRA demonstrates superior performance, further highlighting its efficiency.

Q5: What is the x-axis in Figure 7?

Figure 7 displays the dataset classification results for different methods, with the x-axis representing the number of repeated experiments, aiming to provide more representative results through a 15-fold experiment as mentioned on line 274.

References:

[1] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.

[2] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024. References:

[3] Mixture of LoRA Experts, ICLR 2024.

[4] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications, SIGIR 2024.

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below：

W1& Limitation2: HydraLoRA has multiple adapter copies.

The reason for multiple "B" modules is that, in practice, downstream tasks are often complex and multi-task. Traditional PEFT methods typically focus on optimizing for a single task. Tuning a single LoRA to serve all tasks without considering the task differences can lead to reduced performance.

Current approaches [1,2,3,4] often train multiple LoRA modules for multiple tasks, similarly overlooking the task synergies. In contrast, HydraLoRA, by sharing the "A" module and training distinct "B" modules, perfectly couples these two features, leading to superior performance. Moreover, by sharing the "A" module, HydraLoRA significantly reduces the parameter count to just 11.5% of that of existing methods [2], as detailed in Table 3. Nonetheless, the additional computation introduced is negligible compared to the parameters of LLMs themselves, as shown in Table 2 where HydraLoRA accounts for only 0.124% of the total parameters.

W2: Environmental footprint of model training.

While the vanilla LoRA method incurs higher computational overhead compared to other PEFT approaches, it also delivers significant performance improvements. HydraLoRA, an adaptation of LoRA, enhances downstream task performance with the same parameter settings (rank=16), as demonstrated in Table 2.

Moreover, as Figure 5 illustrates, HydraLoRA cuts energy consumption by 50% compared to Split-LoRA, which uses multiple LoRA modules. This underscores HydraLoRA's efficiency and its eco-friendly nature. Additionally, the carbon footprint of fine-tuning LoRA is effectively negligible when contrasted with full-parameter tuning, highlighting its environmental and computational benefits [5].

W3: Primarily on LoRA, not test other PEFT configurations.

Our core focus is on better understanding and analyzing the LoRA structure (line 35). We first perform a thorough analysis of the LoRA structure, showing that the asymmetry (Figure 3) is primarily due to the different initialization methods of the A and B matrices. However, this characteristic may not be directly transferable to other PEFT methods. We appreciate your suggestion and will consider how similar explorations might be applied to other PEFT techniques.

W4: Outside the experimental setup is not discussed.

We have validated HydraLoRA on representative single-domain datasets in General, Medical, Law, Math, and Code (line166 - line175), as well as on the multi-domain dataset Flanv2, which covers 10 distinct task clusters (line175 - line178), effectively simulating common scenarios. We hope this addresses the reviewer's question and we are willing to answer more questions about the setup.

Q1&Limitation3: HydraLoRA's asymmetric structure interpretability.

Our analysis of the LoRA module breakdown (Figure 3) revealed asymmetrical properties of the A-B modules: post-training, the A module shows similarities across tasks, whereas the B module exhibits distinct differences. This observation aligns with the synergies and differences encountered in downstream multi-task learning with LLMs. Consequently, we have refined the existing LoRA structure and introduced the HydraLoRA asymmetric architecture (Figure 1.C). In this design, the A module captures the commonalities of knowledge, while the B module captures specific characteristics. We hope this addresses the reviewer's question. Could the reviewer please clarify what is meant by “interpretability of the model”? We apologize for any confusion.

Q2&Limitation1: Initialization of K-means.

As discussed in Section 4.5, we find that the number k of clusters is NOT a sensitive parameter for HydraLoRA with a wide range of reasonable number k of clusters performing decently well in all settings in our experiments (Figure 8). We also compare K-means with sophisticated hyperparameter search approaches and find that K-means is simple but effective (Figure 7).

References:

[1] Mixture of LoRA Experts, ICLR 2024.

[2] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.

[3] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024.

[4] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications, SIGIR 2024.

[5] Carbon Footprint of LLM Fine Tuning — A Case Study.

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below：

W1：Clarify asymmetric structure and workflow.

• Asymmetric structure: Figure 3 presents the post-fine-tuning characteristics of the LoRA module within Llama-7B across four different tasks, analyzing the same submodules. Figure 3a displays the total 4x128 submodules across four LoRAs. Figure 3b illustrates the breakdown of the A matrices (even-numbered), the same submodules ( same index) overlap significantly, making them indistinguishable. Conversely, Figure 3c shows the B matrix breakdown (odd-numbered) where, after training on different tasks, the same submodules demonstrate distinct differences, facilitating clear differentiation. This analysis substantiates the HydraLoRA approach of sharing the "A" module and training distinct "B" modules to perfectly couple the synergies and differences across tasks.

• Workflow. For Section 3.2, the HydraLoRA fine-tuning first involves categorizing datasets to initialize the number of B matrices, essentially constructing asymmetric structures. Subsequently, these B matrices serve as the experts of MoE (Eq. 3). We'd like to clarify that HydraLoRA goes beyond a simple MoE for the PEFT approaches. Our core focus is on better understanding and analyzing the LoRA structure (line 35), which delivers superior model performance while maintaining the efficiency benefits of a reduced parameter footprint.

W2：Novelty of HydraLoRA & Compared with a LoRA MoE Work: MOLE [1].

• Novelty. We'd like to clarify that HydraLoRA represents an asymmetric architecture enhancement of the vanilla LoRA, while existing LoRA MoE approaches [1,2,3,4] serve as LoRA frameworks for multi-tasks. MoE plays a secondary role in HydraLoRA. We leverage it as a method to aggregate these asymmetric B-matrix modules. Thus, HydraLoRA architecture can be seamlessly adapted into existing enhancements to the LoRA MoE framework, further extending its capabilities and effectiveness.

• Compared with MOLE [1]. As the reviewer mentioned, we had noticed the LoRA MoE work MOLE [1] before, but since it is NOT open-sourced (https://github.com/yushuiwx/MoLE/issues). To be fair, in Table 3, we compare HydraLoRA with another similar LoRA MoE work ICLR 2024 [2] and COLM 2024 [3]. Meanwhile, we have attempted to reproduce MOLE [1], which is not a guaranteed fair comparison. The results are as follows, MOLE underperformances [2, 3], and HydraLoRA still achieves better performance, which further proves the strong adaptability and efficiency of HydraLoRA.

W3：Results are incremental. HydraLoRA achieves superior performance on downstream tasks with fewer parameters. Specifically,

• Compared with LoRA (r=8), HydraLoRA (r=8) demonstrates a performance gain of over 5%, as shown in Table 2;
• Compared with strategies that employ multiple LoRAs directly for Mixture of Experts (Table 3) and LoRA (r=32) (Table 2), HydraLoRA enhances efficiency by sharing the "A" module to capture task synergies and training distinct "B" modules to recognize task differences. Consequently, HydraLoRA significantly reduces about 88.5% of parameters compared to existing methods [2].

Q1: Comparison of inference speed.

For inference, the speed is primarily influenced by the base model. Since the parameters of PEFT modules constitute a small fraction of the total model parameters (ranging from 0.001% to 0.248% as shown in Table 2), the inference latency differences among various PEFT methods are minimal.

The following presents the latency and energy consumption during inference using Llama2-7B with different PEFT methods, evaluated on the WikiText2 dataset using a single NVIDIA A40 GPU. The results show nearly equal energy consumption and latency, but HydraLoRA exhibits the highest model performance.

Q2：Comparison of training overhead.

• Compared with LoRA variants and LoRA MoE methods. HydraLoRA not only enhances performance with the same parameters LoRA variants (rank=16) as shown in Table 2, but it also demonstrates substantial parameter reductions compared with LoRA MoE— reducing 88.5% compared to [2] and 72.5% to [3] as shown in Table 3. Moreover, as Figure 5 illustrates, HydraLoRA cuts energy consumption by 50% compared to Split-LoRA, which uses multiple LoRA modules. This underscores HydraLoRA's efficiency and system-friendly.

• Compared with other PEFT. While the vanilla LoRA method involves a higher computational overhead than other PEFT strategies, it offers significant performance gains, as shown in Table 2. However, LoRA's carbon footprint is negligible compared to full-parameter tuning, emphasizing its environmental and computational advantages[4]. Meanwhile, the fine-tuning is a one-time event, but inference overhead is crucial. As noted earlier, HydraLoRA boosts performance with minimal additional overhead.

References:

[1] Mixture of LoRA Experts, ICLR 2024.

[2] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.

[3] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024

[4] Carbon Footprint of LLM Fine Tuning — A Case Study.

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:

W1 & Q1: Exploring potential limitations of this design.

Thanks for the insightful question. The limitations may primarily stem from the training data. Particularly, in multi-task, extreme conditions such as contaminated or adversarial data can severely impair performance due to aggregation. The heterogeneity between tasks—differences in language, task type, and domain—means that the shared knowledge might be weaker or less noteworthy. Importantly, this challenge is not unique to HydraLoRA but is common across all multi-task frameworks. Robustness enhancement (such as data sanitization, robust aggregation, and anomaly detection) and privacy-enhancing technologies (like homomorphic encryption, differential privacy, and blockchain) may be potential solutions.

W2: Automatic routing v.s manual predefined tasks.

In Table 3, the LoRA MoE baselines [1,2] utilize existing domain knowledge (manual) to train multiple LoRA units, whereas HydraLoRA employs automatic routing. The results indicate that HydraLoRA uses fewer parameters and performs better in downstream tasks. This suggests potential coupling relationships between tasks, aligning closely with real-world conditions where we cannot anticipate the domains needing fine-tuning. Moreover, Section 4.5 reveals that the number of clusters K is not a sensitive parameter for HydraLoRA. It demonstrates the efficiency and robustness of HydraLoRA.

W3: More streamlined presentation.

Thanks for your constructive comment. We will revise the paper based on your suggestions in the updated version.

W4 & Q2: More MoE discussion.

Thanks for your constructive comment. We add more experiments with the same setting with Table 3, to explore how the number of experts (B matrices) during the HydraLoRA inference pipeline influences performance. As shown in the Table below, we find that an increase in the number of B matrices generally leads to enhanced performance in downstream tasks.

In practice, user requests may belong to different tasks, while a single request potentially involves mixed tasks. This improvement can be attributed to the expanded configuration space afforded by additional LoRA modules, which allows for a more fine-grained and tailored adaptation to the diverse and mixed-task inputs encountered in the benchmark.

Table: Sensitivity analysis of the number of B matrices. "Base" means vanilla Llama2-7B, "Top-1" means selecting the highest-ranked (top-1) B matrix, and "Top-3" means selecting three highest-ranked (top-3) B matrices.

W5：Overhead with other PEFT.

While the vanilla LoRA method incurs higher computational overhead compared to other PEFT approaches, it also delivers significant performance improvements. HydraLoRA, an adaptation of LoRA, enhances downstream task performance with the same parameter settings (rank=16), as demonstrated in Table 2.

Moreover, as Figure 5 illustrates, HydraLoRA cuts energy consumption by 50% compared to Split-LoRA, which uses multiple LoRA modules. This underscores HydraLoRA's efficiency and eco-friendly. However, LoRA's carbon footprint is negligible compared to full-parameter tuning, emphasizing its environmental and computational advantages [3]. Meanwhile, fine-tuning is a one-time event, but inference overhead is crucial. As fellows, HydraLoRA boosts performance with minimal additional overhead.

Table: latency and energy consumption during inference using Llama2-7B with different PEFT methods, evaluated on the WikiText2 dataset using a single NVIDIA A40 GPU.

References:

[1] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.

[2] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024

[3] Carbon Footprint of LLM Fine Tuning — A Case Study.