FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts

We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.

W1: Analysis of the performance gap between merged and non-merged scenarios

Intuitively, the four tasks—general knowledge understanding (MMLU), scientific question answering (ScienceQA), mathematical reasoning (GSM8K), and code generation (HumanEval)—represent significantly different distributions, so merging their adapters is prone to substantial conflicts.

Empirically, following [1], we use centered kernel alignment (CKA) [2] to quantify the alignment between the output representations of each single-task adapter and the merged adapter. A higher CKA indicates better output alignment, which is likely the inherent reason, and therefore, results in a smaller accuracy drop after merging. Supplementary Table 1 reports both CKA and accuracy drop ($\Delta$) on Llama-3.1-8B. Since there's no apparent difference between LoRA$(r=8)$, LoRA$(r=32)$ and Split-LoRA$(4 \times 8)$ in model merging, we use LoRA$(r=8)$ as a representative to compare with FlyLoRA$(k=8)$. We observe that tasks with lower CKA (especially ScienceQA and GSM8K) suffer the largest accuracy drops. FlyLoRA consistently yields higher CKA than LoRA, which aligns with its consistently smaller $\Delta$. This micro-level analysis corroborates why FlyLoRA outperforms LoRA in heterogeneous-task merging. All of this new analysis will be included in the final version.

Supplementary Table 1: CKA and corresponding accuracy drop ($\Delta$) between single-task adapter and merged model.

W2: Evaluating on more challenging benchmarks

As suggested by the reviewer, we further evaluated single- and multi-task performance on LiveCodeBench (using CodeAlpaca checkpoints), GPQA (using MMLU checkpoints), and MATH (for advanced math reasoning, using GSM8K checkpoints) with Qwen-2.5-7B. The results in Supplementary Tables 2 and 3 indicate that FlyLoRA remains robust even on more challenging test sets.

Supplementary Table 2: Single-task accuracy comparison (Pass@1 for LiveCodeBench) using Qwen-2.5-7B.

Supplementary Table 3: Multi-task accuracy comparison (Pass@1 for LiveCodeBench) using Qwen-2.5-7B. We report the accuracy drop relative to single-task performance using weight averaging technique.

References

[1] Model merging with SVD to tie the Knots. ICLR2025.

[2] Similarity of Neural Network Representations Revisited. ICML 2019.

Thanks for the positive feedback. We are delighted that our response addressed your concerns. We will incorporate these discussions and relate new results into our final version.

We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.

W1&Q1: More sensitivity studies on initialization schemes of matrix $A$

As the reviewer suggested, we further compare three methods for generating the sparse projection $A$ with $\frac{p}{r}$ sparsity:

• Gaussian (our default): Each non-zero entry is drawn from $\mathcal{N}(0,\frac{1}{r^2})$；
• Rademacher (non-Gaussian): Each non-zero entry is $±\frac{1}{r}$ with equal probability.
• FJLT [1] (structured projection): $A=PHD$, where $D$ is a random diagonal matrix with independent Rademacher variables on its diagonal, $H$ is a normalized Hadamard matrix, and $P$ enforces the $\frac{p}{r}$ sparsity.
• Two-Phase (briefly-learned): The non-zero entries of $A$ are trainable for 5% of the total steps (warm-up), then frozen for the remainder.

The results, shown in Supplementary Table 1, indicate that almost all variants perform similarly. Non-Gaussian, structured, or briefly-learned initializations have little impact, except that the briefly-learned scheme shows a noticeable drop after merging. This demonstrates that FlyLoRA is robust to the choice of initialization scheme for matrix $A$, and that learning may break the approximate orthogonality of the random matrix, making it unsuitable.

Supplementary Table 1: Single-task and multi-task accuracy comparison on MMLU using Llama-3.1-8B.

W2&Q2: Evaluation of larger models

We further conducted experiments using the Qwen-2.5-14B model, as shown in Supplementary Tables 2 and 3, following the setting of Tables 1 and 2. The results show that FlyLoRA remains superior in accuracy (for both single-task and multi-task settings) and is more parameter-efficient. We encountered no memory or convergence bottlenecks when training FlyLoRA on the 14B model, which consistently outperforms LoRA.

Supplementary Table 2: Single-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-14B.

Supplementary Table 3: Multi-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-14B. We report the accuracy drop relative to single-task performance using weight averaging technique.

W3&Q3: More baseline into comparison

We further compare several strong and widely used baselines—AdaLoRA [2] (adaptive), SoRA [3] (sparse), and HydraLoRA [4] (MoE)—using Qwen-2.5-7B, as shown in Supplementary Tables 4 and 5. The results demonstrate that FlyLoRA remains superior in terms of accuracy and efficiency in both single-task learning and multi-task merging scenarios.

Supplementary Table 4: Single-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B.

Supplementary Table 5: Multi-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B. We report the accuracy drop relative to single-task performance using weight averaging technique.

W4&Q4: Analysis in performance drop between merge and non-merge settings

The four tasks—general knowledge understanding (MMLU), scientific question answering (ScienceQA), mathematical reasoning (GSM8K), and code generation (HumanEval)—represent significantly different distributions. As a result, merging their adapters is prone to major conflicts that can cause performance drops.

Empirically, following [5], we use centered kernel alignment (CKA) [6] to quantify the alignment between the output representations of each single-task adapter and the merged adapter. A higher CKA indicates better output alignment, which is likely the inherent reason, and therefore, results in a smaller accuracy drop after merging. Supplementary Table 1 reports both CKA and accuracy drop ($\Delta$) on Llama-3.1-8B. Since there's no apparent difference between LoRA$(r=8)$, LoRA$(r=32)$ and Split-LoRA$(4 \times 8)$ in model merging, we use LoRA$(r=8)$ as a representative to compare with FlyLoRA$(k=8)$. We observe that tasks with lower CKA (especially ScienceQA and GSM8K) suffer the largest accuracy drops. This micro-level analysis helps explain the substantial performance decreases observed on ScienceQA and GSM8K. It is important to note that such degradation is an inherent phenomenon of model merging, not a weakness of FlyLoRA. In fact, FlyLoRA still outperforms its LoRA variants (Tables 1 and 2), even in cases where accuracy falls sharply.

Supplementary Table 6: CKA and corresponding accuracy drop ($\Delta$) between single-task adapter and merged model.

W5&Q5: More ablation studies between ST+Trn and ST+Frz

We further conducted ablation studies on the GSM8K, ScienceQA, and HumanEval datasets, with results presented in Supplementary Table 7. The results show that ST+Trn does not outperform ST+Frz, with almost no difference in accuracy. Previous papers have already demonstrated that, in LoRA, updating matrix $A$ is not necessary [7,8]. Our results further validate this finding in the MoE-based LoRA setting. Thus, ST+Trn offers no clear advantage in this context, while ST+Frz can help reduce both interference during merging and the memory footprint of activation values [7].

Supplementary Table 7: Ablation study of FlyLoRA variants using Llama-3.1-8B.

References

[1] The fast Johnson-Lindenstrauss transform is even faster. ICML2023.

[2] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR2023.

[3] Sparse Low-rank Adaptation of Pre-trained Language Models. EMNLP2023.

[4] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. Neurips2024.

[5] Model merging with SVD to tie the Knots. ICLR2025.

[6] Similarity of Neural Network Representations Revisited. ICML 2019.

[7] LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. Arxiv2023.

[8] Asymmetry in Low-Rank Adapters of Foundation Models. ICML2024.

We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.

Q1: How can additive norm preservation property benefit for model merging

"Additive Norm Preservation" is a direct consequence of the "Pairwise Orthogonality" property described in Corollary 3.4. Both "Pairwise Orthogonality" and "Additive Norm Preservation" reflect the orthogonality property of FlyLoRA during model merging. According to the widely used geometric intuition (an assumption) that orthogonality benefits model merging [1–3], FlyLoRA's design aligns with this principle.

Q2: How can i.i.d sampling assumption applied to constraint on number of elements of each entry?

We will modify the description in Theorem A.2 to: "Given the matrix $A\in\mathbb{R}^{r\times n}$ with each entry i.i.d. from $\mathcal{N}(0, \frac{1}{r^2})$ with probability $\frac{r}{p}$, and set to $0$ otherwise," to make the proof more rigorous. Since this construction does not differ significantly from our original construction for $A$, we use this more analytically tractable form as a surrogate to study the distance-preserving property. We also compare the empirical performance of these two construction methods using Llama-3.1-8B in Supplementary Table 1 to support this approximation, with the difference in accuracy being negligible (all differences are within 0.2%). We will clarify this point in the final version.

Supplementary Table 1: Accuracy comparison of FlyLoRA using i.i.d versus Fixed-$p$ sparsity patterns.

Q3: How does Theorem 3.1 extend to Top-k activation?

We apologize for the confusion. The goal of an MoE router is to select $k$ out of $r$ experts that best approximate the effect of all $r$ experts. In FlyLoRA, according to Theorem 3.1 and Appendix A.2, we assume that the value of $a_i x$ for $i = 1, 2, \ldots, r$ is sufficient to serve as a surrogate for identifying the most informative experts. The top-$k$ operation, therefore, selects experts according to the magnitude of $a_i x$. We will clarify this assumption and its logical connection in the final version.

Q4: Placement of assumption

Thanks for the suggestion. We will revise our text as recommended by the reviewer.

Q5: Comparison with other merging technique

We include the comparison in Supplementary Table 2. KnOTS [4] and L-LoRA [3] are both built upon LoRA$_{(r=32)}$. The results suggest that they achieve comparable performance to FlyLoRA, with each method excelling on different datasets. It is noteworthy that FlyLoRA is not a competitor to these methods; rather, they can be used in a plug-and-play manner with FlyLoRA to further improve performance after merging.

Supplementary Table 2: Multi-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B. We report the accuracy drop relative to single-task performance.

References

[1] Editing Models with Task Arithmetic. ICLR2023.

[2] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. Neurips2023.

[3] Parameter-Efficient Multi-Task Model Fusion with Partial Linearization. ICLR2023.

[4] Model merging with SVD to tie the Knots. ICLR2025.

I appreciate the author's reply for clarification of some assumptions. The experiments part is now much stronger with additional experiments on KnoTs and L-LoRA! Glad to see this! I have a few more questions and suggestions:

"Additive Norm Preservation" is a direct consequence of the "Pairwise Orthogonality"

Although from [1-3] FlyLoRA's design aligns with this principle of orthogonality ([5,6] also theoretcially proves this), it's still a bit confusing to show the additive norm preservation property because my understanding is that it's not directly related to model merging but instead just some nice property of your FlyLoRA formulation? If it's correct, I would recommend authors to 1) cite the orthogonality intuition papers accordingly 2) clarify the implication of the second norm preservation property.

[5] Efficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach.

[6] When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers

What's the point of Theorem 3.1?

Although 3.1 is a nice property due to your random matrix construction, somehow I was thinking about your paper after the reviewing period and came up with this question: does MoE router have to preserve distance? Why is "no information loss" after projection important as a router? Why can't we just use any gating network that takes $x$ as an input and learn it like the standard MoE which I believe doesn't nessarily have this property and learn very well?

We are pleased that our response has addressed the reviewer's concerns, and we appreciate the reviewer's insightful suggestions, which have helped improve our work!

"Additive Norm Preservation" is a direct consequence of the "Pairwise Orthogonality"

To be more precise, we will modify the "Additive Norm Preservation" property into "Orthogonality's Outcome in Merging". Our original motivation for presenting "Orthogonality's Outcome in Merging" was the concern that "Pairwise Orthogonality" is only a property of LoRA components, which might not be directly linked to the computational steps of model merging. So, we added the mathematical outcome when "Pairwise Orthogonality" is directly applied to the model-merging formula. These two properties are both related to model merging.

As suggested by the reviewer, we will cite the relevant orthogonality-related papers and present the geometric intuition more clearly in the revised version.

What's the point of Theorem 3.1?

The reviewer is correct that traditional MoE routers neither have nor require the distance-preserving property, as these routers are trainable and can adjust their weights to optimize routing. In contrast, the frozen matrix $A$ in FlyLoRA is not trainable and therefore cannot adapt in the same way.

Because of the distance-preserving property of the frozen $A$, two semantically similar inputs $x$ and $y$ are projected to similar low-dimensional representations, $Ax$ and $Ay$. Points that are close in the input space remain close after projection, while dissimilar points remain far apart. Consequently, the top-$k$ operation routes similar inputs to a similar set of experts, and dissimilar inputs to different experts. This helps mitigate expert representation homogenization in MoE models, enabling each expert to focus on specialized knowledge, reducing internal conflicts, and improving sample efficiency (as supported by previous studies, e.g., [7]). This is the main point of Theorem 3.1 and the reason why the frozen $A$ is effective.

We will revise Section 3.2 according to the above discussion. The phrase "no information loss" will be removed in the revised version.

[7] Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks. ICML2023.

We appreciate the reviewer's valuable feedback and hope our responses can address the raised concerns.

Q1: Connection between FlyLoRA and orthogonality design

Both OFT [1,2] and LoReFT [3] operate on single tasks, but their orthogonal matrix $R$ is multiplied with the pre-trained weight matrix $W_0$, which differs from all LoRA variants (including FlyLoRA) that add $\Delta W$ to $W_0$. The multiplication scheme rotates the entire weight space, and [1,2] demonstrate that this approach better adjusts semantic information compared to changing the magnitude. This is why the scheme succeeds. The additive scheme lacks this property, as $W_0$ cannot be rotated. Thus, although all use an orthogonality design, FlyLoRA succeeds in a different way.

By removing the MoE part and keeping only the random matrix $A$, FlyLoRA reduces to LoRA-FA [4] / Asymmetry LoRA [5], which saves resources but does not improve performance. We also tested enforcing the orthogonality constraint ($A^\top A=I$) on an updatable $A$ (see Supplementary Table 1). The results show no significant accuracy difference compared to LoRA, confirming that orthogonal design in LoRA does not boost single-task performance.

We believe the orthogonality design in LoRA excels in "multi-task" scenarios—such as model merging (this work) and continual learning (O-LoRA [6])—because it decouples parameter interference across multiple downstream tasks when fine-tuning from the base model.

Supplementary Table 1: Single-task accuracy comparison (Pass@1 for HumanEval) using Qwen-2.5-7B.

Typos:

Thanks for pointing it out. We will fix it in our the final version.

References

[1] Controlling Text-to-Image Diffusion by Orthogonal Finetuning. Neurips2023.

[2] Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR2024.

[3] ReFT: Representation Finetuning for Language Models. Neurips2024.

[4] LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. Arxiv2023.

[5] Asymmetry in Low-Rank Adapters of Foundation Models. ICML2024.

[6] Orthogonal Subspace Learning for Language Model Continual Learning. EMNLP findings2023.