DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

We are grateful for your detailed feedback and address your points below:

1.Comparison with Different Baselines Regarding Performance on Multitask Datasets

We apologize for any misunderstanding about our contribution. Our work primarily introduces the lightweight sentence-level mini-MLP plugin that efficiently serves multiple LoRAs. As evidenced in Table 3, our method handles up to 100 LoRAs with minimal impact on inference time compared to the base LLaMA-3 8B model. We also highlight the efficiency in inference time over manually loading individual LoRAs, as shown in Table 7.

2. Inference Efficiency Comparison Between Different Baselines

   Recognizing the importance of this comparison, we have included relevant experiments in Table 7. Our DLP-LoRA demonstrates superior performance in decoding latency ratio and peak memory ratio when using LLaMA-2 7B as the backbone. The advantage stems from our sentence-level routing approach, which contrasts with token-level routing methods that increase inference time due to per-token processing.

3. LoRA Modules Training for Downstream Tasks

   We adhere to the standard LoRA fine-tuning procedure for each task. The key difference in our approach is that we select and fuse different LoRAs at the sentence level, meaning all tokens within a sentence share the same fused LoRAs.

4. Claims About Additional Fine-tuning of Other Methods for New Tasks

   To clarify:

   Methods using token-level routing, such as S-LoRA and MeteoRA, require the entire base model and all associated LoRAs to be loaded during fine-tuning for a new task, which is resource-intensive. Our approach allows for independent fine-tuning of the mini-MLP (only 5 million parameters) and a single LoRA for the new task, making it feasible on less powerful hardware.

5. Combining Single LoRAs for Different LoRA Selection

   As detailed in Figure 1 (right side) and Section 3.2, we use the mini-MLP to assign probabilities to each LoRA. We then fuse the A and B weight matrices of the selected LoRAs based on these normalized probabilities, applying the final fused weights to downstream tasks.

6. Explanation of Performance Difference Between Classification and Generation Tasks

   We acknowledge that our initial explanation was insufficient. In our updated version, we expand on this point:

   Due to the limited output space in classification tasks (typically a single token), the performance may be slightly impacted when fusing multiple LoRAs, as irrelevant information can be introduced. In contrast, generation tasks benefit from the fusion of multiple LoRAs, as the longer outputs allow the model to produce more nuanced and contextually appropriate responses.

7. Clarification in Table 4 Caption

   We apologize for any confusion. In Table 4, the inference time ratio is calculated by dividing the inference time of the standard LoRA by that of DLP-LoRA (mini-MLP). This ratio allows for consistent comparison across different hardware setups, facilitating reproducibility.

8. Explanation of Limitations of Top-k Sampling

   Thank you for this insightful question. Fixed top-k sampling lacks flexibility in multi-task settings. For classification tasks, a high top-k may introduce noise from irrelevant LoRAs, degrading performance. Conversely, for QA and generation tasks, a low top-k may omit relevant LoRAs, hindering performance. Our method employs a probability threshold p for selecting and fusing LoRAs, providing a more adaptable and task-appropriate selection mechanism.

9. Explanation for Superior Performance of LLaMA-3 8B Compared to Other LLM Backbones

   The datasets in Table 1 are predominantly from the BIG-Bench collection. According to the official Hugging Face repository (https://huggingface.co/meta-llama/Meta-Llama-3-8B), LLaMA-3 8B outperforms LLaMA-2 7B on these tasks. While direct comparisons between LLaMA-3 8B and Qwen models on BIG-Bench are not officially available, our observations indicate that Qwen models underperform on classification tasks in this benchmark compared to LLaMA-3 8B. This discrepancy may result from differences in pretraining datasets and strategies between Qwen and LLaMA-3 models.

Thank you for your valuable feedback. We address your concerns below:

1. Comparison with Different Baselines Regarding Performance on Multitask Datasets

   We apologize for any confusion regarding our main contribution. Our work primarily focuses on the lightweight sentence-level mini-MLP plugin that efficiently serves multiple LoRAs. As demonstrated in Table 6, our method can handle up to 100 LoRAs with less than twice the inference time of the base LLaMA-3 8B model. We emphasize the efficiency gains in inference time compared to manually loading single LoRAs, as presented in Table 7 of the updated version.

2. Inference Efficiency Comparison Between Different Baselines

   Recognizing the importance of this comparison, we have added quantitative experiments and compared our DLP-LoRA with several baselines in Table 7. The results show that our DLP-LoRA outperforms state-of-the-art baselines in decoding latency ratio and peak memory ratio using LLaMA-2 7B as the backbone. This improvement arises because token-level routing methods require per-token routing, leading to increased inference time. Our sentence-level routing mechanism, facilitated by the mini-MLP plugin, significantly reduces inference time by deciding which LoRAs to fuse at the sentence level.

3. Incorrect Usage of Top-p Sampling Concept

   Thank you for bringing this to our attention. We have corrected the usage of the top-p sampling term in our revised version to accurately reflect our method.

4. Claims About Additional Fine-tuning of Other Methods for New Tasks

   We apologize for any lack of clarity. To clarify:

   Token-level routing methods like S-LoRA and MeteoRA require loading the base model and all relevant LoRAs simultaneously for fine-tuning new tasks. For large models (e.g., 70B parameters), this necessitates substantial computational resources, such as multiple high-end GPUs. In contrast, our DLP-LoRA only needs to fine-tune the mini-MLP (approximately 5 million parameters) and a single LoRA for the new task. This process is resource-efficient and can be conducted on CPUs or GPUs with less memory.

5. Table 5 Comparison Between Qwen-2 1.5B with 26 DLP-LoRAs and LLaMA-2 13B

   The purpose of this comparison is to demonstrate that our method maintains strong performance across 26 tasks while offering fast inference speeds when loading multiple DLP-LoRAs simultaneously. For comparisons based on the same backbone LLMs, please refer to Tables 1, 2, 3, and 7, which showcase the superior capabilities of our DLP-LoRA.

Thank you for your thoughtful comments. We address your points below:

1. Comparison with a Single LoRA Fine-tuned on All Multitask Data with Higher Rank

   We appreciate your suggestion. We have included this new baseline in our updated version of Table 3. The experiments reveal that the single LoRA fine-tuned jointly on all tasks, even with a higher rank, performs worse and suffers from catastrophic forgetting compared to our DLP-LoRA. This indicates that a single LoRA's capacity, even when increased, is insufficient to effectively handle all 26 tasks.

2. Explanation of the Performance Difference Between Classification and Generation Tasks

   We acknowledge that our initial explanation was insufficient. In our updated version, we provide a more detailed discussion:

   Classification tasks typically require the model to output a single token representing the answer choice, offering limited linguistic variability. Consequently, performance on classification tasks may be slightly affected when fusing multiple LoRAs due to potential interference. In contrast, generation tasks involve producing longer and more varied outputs, allowing DLP-LoRA to better leverage the fused LoRAs to generate meaningful and coherent responses, as illustrated in Figure 3.

3. Missing Related Work

   We apologize for omitting relevant references. We have updated our paper to include these works and have added discussions in the related work section to contextualize our contributions within existing literature.

Dear Reviewer 4fCq, we explained each raised question during our rebuttal. We are looking forward to your further comments and valuable feedback regarding our responses.