LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

Thank you for your thoughtful review! Below, we address your questions in detail:

The github repo is not public and I believe it will be much help if the paper been accepted and public its code.

Thank you for pointing this out! We have made the code available through an anonymous GitHub repository, which can be accessed here https://anonymous.4open.science/r/LoRI-28C6.

Hugging Face has public source code for loading multiple adapters. What are the advantages of your method?

Our adapter merging is implemented using the add_weighted_adapter method from the PEFT library, leveraging the cat and linear merging methods already available there. The key advantage of LoRI lies in its reduction of interference during merging. By using independently initialized frozen A matrices, LoRI maps adapters into approximately orthogonal subspaces. This minimizes cross-task interference without requiring explicit regularization terms, unlike methods such as O-LoRA or AdaLoRA.

Can your method be applied to vision-language models?

Yes, we believe LoRI can be extended to vision-language models. This is an exciting direction, and we consider it part of our future work.

There are some existing methods working on optimizing LoRA such as CoRA and LoRA-Pro.

Thank you for pointing out these relevant works! We will make sure to include CoRA and LoRA-Pro in the related work section of a future version of our manuscript. We have added several widely adopted and representative PEFT baselines, including VeRA, IA3, LoRA-FA, AdaLoRA, rsLoRA, PiSSA, LoRA+, and DoRA. The updated results are available at our anonymous GitHub repository and anonymous Slides presentation. Due to time constraints, we were only able to include a subset of baselines, but we plan to expand our comparisons in the future version.

What will be the performance if model can be trained using a single A5000 GPU?

Thank you for raising this important point. We used 8 A5000 GPUs primarily due to cluster availability, not because LoRI requires that many GPUs. In fact, LoRI can be trained on a single A100 / H100 GPU. LoRI is also more memory-efficient than standard LoRA, as it freezes the A matrices, thereby reducing optimizer state memory and activation memory. Additionally, combining LoRI with quantization techniques like QLoRA can significantly lower the memory footprint, making it suitable for training on a single A5000. We consider LoRI + quantization a promising direction for future research.

Thank you for your thoughtful review! Below, we address your concerns in detail:

I strongly encourage a deeper study in relevant studies.

Following your suggestion, we provide a discussion of related work relevant to LoRI. Given the breadth of research in the PEFT space, a comprehensive survey is infeasible, but we aim to highlight prior works that are most closely related to our work:

• Parameter Efficiency: LoRI is motivated by reducing parameter redundancy in LoRA, which aligns with the goals of IA3, VeRA, and FourierFT. Unlike these methods, which introduce learnable vectors, we specifically target redundancy within LoRA and propose a new asymmetry: frozen A + sparse B.
• Frozen A Matrices: The concept of asymmetric LoRA variants has been explored in LoRA-FA, AsymmetryLoRA, and HydraLoRA. However, to our knowledge, none of these combine a frozen A with a sparse B.
• LoRA Pruning: Prior works such as Loraprune, LoRA-drop, RoseLoRA, and SoRA focus on pruning strategies, but do not perform targeted, asymmetric pruning solely on the B matrices. Other approaches like AdaLoRA and PiSSA prune based on singular values or vectors (via SVD), which is orthogonal to our direction.

We also note that the aforementioned papers do not consider multi-task scenarios, which are a central focus of LoRI.

• Orthogonality in Multi-Task Learning: While works like O-LoRA, AWD, and AdaLoRA enforce orthogonality via regularization, LoRI relies on the natural orthogonality of random frozen A matrices. Importantly, we leverage this property for clean adapter merging, which prior work does not address.
• Sparsity in Multi-Task Learning: Methods such as SHiRA and LoTA explore sparsity in multi-task settings. However, our approach is distinct in that we use sparsity only in B to mitigate catastrophic forgetting in continual learning.

We will revise and expand the related work section – particularly the PEFT part – in the future revision to provide a more comprehensive overview of prior work.

I strongly encourage adding further baselines.

Thank you for the suggestion! We have added several widely adopted and representative PEFT methods as baselines, including VeRA, IA3, LoRA-FA, AdaLoRA, rsLoRA, PiSSA, LoRA+, and DoRA. We believe these additions provide a more comprehensive comparison and better highlight the strengths of our approach. The updated results can be found at our anonymous GitHub repository and anonymous Slides presentation.

As shown in the updated results, LoRI-D consistently achieves the best performance on NLU and code generation tasks, and performs on par with other methods on math and safety tasks, while using 44M trainable parameters (0.5% of full fine-tuning and 50% of LoRA). LoRI-S (0.05% parameters of full fine-tuning and 5% of LoRA), while slightly behind LoRI-D due to its more aggressive sparsity, remains highly competitive among existing PEFT methods. Notably, while VeRA and IA3 are more parameter-efficient – using 0.02% of the full model parameters – their performance is significantly lower than that of LoRI-S and other PEFT approaches.

Related to the parameter-efficiency and simplicity in multi-task learning, this paper comes in my mind: Frohmann, Markus, et al. "ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale." Findings of the Association for Computational Linguistics ACL 2024. 2024.

This work also seems relevant regarding safety preservation and bias mitigation: Kumar, Deepak, et al. "Parameter-efficient Modularised Bias Mitigation via AdapterFusion." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

Thank you for sharing these relevant works! We will make sure to cite them and discuss their connections to our approach in the related work section of the future revision.

Dear Reviewer 5RGh,

Thank you again for your thoughtful feedback and questions. We hope our response has addressed your concerns clearly.

If you have any further questions or suggestions, we'd be happy to engage further. We would greatly appreciate any follow-up comments or clarification regarding your evaluation.

Best regards,

Authors

Thank you for your thoughtful review! Below, we address your concerns and questions in detail:

The ideas presented in the paper are conceptually similar to existing work.

To the best of our knowledge, no prior work combines frozen A with sparse B. Additionally, we explicitly address multi-task scenarios, which are often overlooked in prior PEFT research. LoRI leverages orthogonality to reduce interference during adapter merging and sparsity to mitigate catastrophic forgetting in continual learning. We elaborate on the differences between our approach and prior works below:

• Parameter Efficiency: LoRI is motivated by reducing parameter redundancy in LoRA, which aligns with the goals of IA3, VeRA, and FourierFT. Unlike these methods, which introduce learnable vectors, we specifically target redundancy within LoRA and propose a new asymmetry: frozen A + sparse B.
• Frozen A Matrices: The concept of asymmetric LoRA variants has been explored in LoRA-FA, AsymmetryLoRA, and HydraLoRA. However, to our knowledge, none of these combine a frozen A with a sparse B.
• LoRA Pruning: Prior works such as Loraprune, LoRA-drop, RoseLoRA, and SoRA focus on pruning strategies, but do not perform targeted, asymmetric pruning solely on the B matrices. Other approaches like AdaLoRA and PiSSA prune based on singular values or vectors (via SVD), which is orthogonal to our direction.
• Orthogonality in Multi-Task Learning: While works like O-LoRA, AWD, and AdaLoRA enforce orthogonality via regularization, LoRI relies on the natural orthogonality of random frozen A matrices. Importantly, we leverage this property for clean adapter merging, which prior work does not address.
• Sparsity in Multi-Task Learning: Methods such as SHiRA and LoTA explore sparsity in multi-task settings. However, our approach is distinct in that we use sparsity only in B to mitigate catastrophic forgetting in continual learning.

We also welcome any pointers to specific works you find closely related to ours, and we would be happy to acknowledge and discuss them in future revisions.

Related works are not in the main body of the paper.

Due to page constraints, we had to place the related work section in the appendix. If granted an additional page for the camera-ready version, we would include it in the main text for greater visibility.

The task-specific masks must first be obtained, which might introduce additional training overhead?

Yes, calibration introduces some overhead, as it does in all pruning-based methods. The cost is comparable to a single fine-tuning pass. If needed, the number of calibration steps can be reduced to minimize overhead – at the cost of a small drop in performance, as shown in Figure 4(a). Importantly, masks are calibrated only once per task and can be reused in continual learning scenarios or future adaptations.

Are there any discernible patterns in sparse masks? Do they exhibit significant overlap across different tasks?

We do not observe discernible patterns in sparse masks across tasks. In fact, task-specific masks are largely non-overlapping. At a 90% sparsity level, the average overlap is around 1%. Although enforcing strict non-overlap is possible – by preventing future updates from modifying previously used positions – we chose not to do so, as it would restrict the parameter space and reduce the reusability of calibrated masks.

Dear Reviewer cZht,

Thank you again for your thoughtful feedback and questions. We hope our response has addressed your concerns clearly.

If you have any further questions or suggestions, we'd be happy to engage further. We would greatly appreciate any follow-up comments or clarification regarding your evaluation.

Best regards,

Authors

Thank you for your thoughtful review! Below we address your concerns and questions in detail:

It minimises cross-task interference due to a shared random subspace.

We would like to clarify that in our adapter merging setup, we do not share the A matrix across adapters. Instead, each adapter uses a different random A matrix. These matrices are approximately orthogonal in high-dimensional space, which helps reduce interference. This orthogonality is achieved naturally, without the need for an explicit regularization term in the loss function used in prior works.

The authors mainly repackage prior PEFT ideas via frozen random A (as in VeRA/O-LoRA) and sparsified B (as in AdaLoRA) rather than introducing a fundamentally new algorithmic insight.

We respectfully clarify that techniques used in LoRI are distinct from prior work. Specifically:

• VeRA freezes both A and B but introduces trainable vectors; it does not use frozen A matrices in the way we do.
• O-LoRA focuses on enforcing orthogonality between LoRAs via regularization but does not leverage frozen A matrices or sparse B matrices.
• AdaLoRA performs pruning in the singular value domain, rather than applying sparsity to B.

While some works such as LoRA-FA and HydraLoRA explore frozen A matrices, to the best of our knowledge, no existing method combines frozen A with sparse B, which is the core novelty of LoRI. For a broader comparison with relevant methods, we kindly refer you to our response to Reviewer 5RGh.

It slows training and inference in practice because today’s GPU libraries are tuned for dense (or very structured-sparse) matrix multiplies.

Thank you for raising this important point! We evaluated training throughput (measured in samples per second) on GSM8K using a single H100 GPU, and present the results below:

The results show that LoRI-D achieves slightly higher training throughput than LoRA and significantly outperforms DoRA. LoRI-S introduces a small overhead due to masking gradients during the backward pass, but the difference is modest.

During inference, LoRI behaves similarly to LoRA. The low-rank updates are merged into the base model weights, meaning there is no additional inference latency compared to full fine-tuning. In practice, LoRI-D, LoRI-S, LoRA, and full fine-tuning all run inference at the same speed, as they all operate on the base model once merged.

Why doesn't the analysis include strong modern PEFT methods such as QLoRA, AdaLoRA, or HyperLoRA?

We have included AdaLoRA as a baseline in our experiments. For results on additional widely adopted PEFT baselines, we kindly refer you to our anonymous GitHub repository and anonymous Slides presentation.

QLoRA is a quantization-based method focused on reducing memory usage. It typically introduces a slight performance degradation compared to LoRA. We consider LoRI + QLoRA a promising direction for future work to further reduce memory cost. HyperLoRA is tailored toward portrait synthesis and vision-related tasks. As our experiments are focused on text generation, we did not include HyperLoRA as a baseline.

LoRA at rank 16 vs LoRI-D at rank 32

We have conducted experiments comparing LoRA at rank 16 and LoRI-D at rank 32, with results summarized below:

These results show that LoRI-D at rank 32 outperforms both LoRA variants on NLU, code generation, and safety tasks, demonstrating its strong effectiveness.

LoRI can hit a ceiling that LoRA does not have.

We agree that, in principle, LoRI-D and LoRI-S may have lower expressiveness than LoRA due to training fewer parameters. However, our aim is to highlight the surprising extent of redundancy in LoRA. Our results suggest that adaptation may not require updating A at all, and that B contains substantial redundancy. LoRI offers an efficient alternative while still achieving competitive performance across tasks.

Thank you for your thoughtful review! Below we address your questions in detail:

By sharing the Gaussian-initialized A matrix, the LoRA updates in a multi-adapter setting will be approximately orthogonal.

We would like to clarify that in our adapter merging setup, we do not share the A matrix across adapters. Instead, each adapter uses a different random A matrix. These matrices are approximately orthogonal in high-dimensional space, which helps reduce interference. This orthogonality is achieved naturally, without the need for an explicit regularization term in the loss function used in prior works.

Sensitivity analysis might be a more principled approach for pruning. Did the authors consider sensitivity analysis as a criterion for pruning B?

We chose magnitude-based masking due to its simplicity, efficiency, and strong empirical performance. Sensitivity-based approaches like those used in LQ-LoRA and TaLoS rely on the Fisher Information Matrix. To assess the effectiveness of different pruning strategies, we conducted experiments on GSM8K with LoRI-S using Llama-3-8B, comparing top-k magnitude (used in our method), Fisher score $(\nabla_B \mathcal{L})^2$, and SNIP score $|B \cdot \nabla_B \mathcal{L}|$. The results are as follows:

The results show that top-k magnitude pruning yields the best performance. We attribute this to the fact that gradient-based measures capture local sensitivity at a particular training step, whereas magnitude reflects the cumulative importance of parameters over the entire fine-tuning process.