Federated Residual Low-Rank Adaptation of Large Language Models

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Does Eqs.(6) and (7) result in Information loss?

We completely AGREE that $\boldsymbol{A}_G^0$ and $\boldsymbol{B}_G^0$ retain the MOST information in the pretrained model. Actually, they are merged back into the global model (see Eq.(11)), and this does NOT result in information loss:

(1) Reinitialization at Each Round: At the start of each communication round, the low-rank matrices are reinitialized to $\boldsymbol{A}_G^0$ and $\boldsymbol{B}_G^0$, ensuring the critical information is always preserved.

(2) Final Integration: After training, $\boldsymbol{A}_G^0$ and $\boldsymbol{B}_G^0$ are merged back into the global model (Eq.(11)), fully restoring all retained information.

Q2: Motivation of FRLoRA's initialization.

We clarify it as follows:

(1) The standard method, where $\boldsymbol{B}$ is set to 0 and $\boldsymbol{A}$ is initialized with Gaussian noise, often struggles with convergence. If FRLoRA is reinitialized to such matrices in each round, it will severely hinder the convergence of the local models, thereby degrading the knowledge in the residual update $\Delta \boldsymbol{W}^t$. We have added a visualization of the loss landscape in the revision (Appendix C.7 Figure 5). The results demonstrates the effectiveness of FRLoRA's initialization strategy in addressing the convergence challenges associated with standard initialization methods, enabling faster and more stable convergence.

(2) Moreover, under data heterogeneity, standard initialization causes inconsistent convergence due to the varying data distributions across clients, further exacerbating client drift. FRLoRA's reinitialization ensures that each round of local optimization begins with a consistent principal singular value space, helping to mitigate client drift. Our ablation results (Section 4.4) on different datasets and further analysis in the revision (Appendix C.5 Figure 3), which confirm its effectiveness.

Q3: Why the global model are initialized by the residual matrice?

If the global model in Eq.(7) is initialized directly using the pretrained model, i.e., $\hat{\boldsymbol{W}}^0 =\boldsymbol{W}^0$, the model parameters $\widetilde{\boldsymbol{W}}^0$ in Eq.(11) at the initial stage becomes:

This indicates that, at the initial stage, merging the initialized low-rank matrices back into the global model does not revert it to the pretrained model. Therefore, we should initialize the global model with residual matrices instead of the pretrained model, which ensures that $\widetilde{\boldsymbol{W}}^0$ and $\boldsymbol{W}^0$ remain consistent in the initial stage.

Q5: Differences between the standard method and FRLoRA in Eq. (13).

The standard LoRA in FL can NOT be formulated as Eq.(13). If we want to express the global update of the LoRa parts in the form of Eq. (13), it only can be written as:

We can observe that $\Delta\boldsymbol{A}^{t}$ and $\Delta\boldsymbol{A}^{t-1}$, as well as $\Delta\boldsymbol{B}^{t}$ and $\Delta\boldsymbol{B}^{t-1}$, are NOT independent. They share common terms, as the standard LoRA in FL only updates the low-rank matrices and uses them as the initialization for the next training round. Since these common terms can be merged, there is NO iterative accumulative effect as shown in Eq.(13). Therefore, the final fine-tuned global model can only be expressed as Eq. (4):

In contrast, the global update process of FRLoRA can be expressed as:

Given that FRLoRA reinitializes the low-rank matrices in each training round, $\Delta\boldsymbol{W}^{t}$ and $\Delta\boldsymbol{W}^{t-1}$ are independent and can NOT be merged. Additionally, FRLoRA directly applies these global updates to the global model. As a result, the global update of FRLoRA has an iterative accumulative effect, which allows gobal updates to occur in a higher-rank parameter space, thereby effectively capturing global knowledge.

Besides, $\Delta \boldsymbol{A}^1, \ldots, \Delta\boldsymbol{A}^T$ are all $r\times d_2$ matrices, and when summing $T$ such $r \times d_2$ matrices, the resulting matrix's rank will not exceed $r$. Although $\Delta W^t$ is a $d_1 \times d_2$ matrix with rank $r$, when summing $T$ such matrices, the upper bound of rank should be $\min(rT, d_1, d_2)$, which can greater than $r$.

Q4: Explanation of residual accumulation.

We discuss this both theoretically and empirically.

(1) Theoretically, the upper bound on the rank of the global update parameter space in FRLoRA is $\min(rT, d_1, d_2)$, which is significantly larger than the upper bound $r$ in FedAvg. The iterative residual accumulation mechanism provides this higher upper bound, though we also acknowledge that it cannot strictly increase the rank as $T$ increases. However, a higher upper bound typically means a larger potential maximum value under the same conditions. This indicates that FRLoRA has the ability to explore a larger updating parameter space to capture more complex structures, allowing for better representation of the diverse knowledge learned from different clients.

(2) Empirically, we have provided results with different values of $r$ in our submitted paper (Table 10). The results are presented below:

As shown, FRLoRA ($r$ = 16) outperforms FedAvg ($r$=16) and even achieves higher performance than FedAvg (r=64). Besides, we observed that when training FRLoRA ($r=16$) on MetaMathQA after 100 rounds, the average rank of residual accumulation at each layer reached 82. These results strongly demonstrate the effectiveness of our method in expanding the parameter space of global updates.

Thanks for your valuable comment. In the following, we try our best to explain our theoretical analysis on the rank of the global update more clearly.

To begin with, we define the global update of FedAvg with LoRA as,

Here, $\boldsymbol{B}_G^T$ is a $d_1 \times r$ matrix and $\boldsymbol{A}_G^T$ is a $r \times d_2$ matrix. Based on Eq.(12), we have,

Then, we define the global update of FRLoRA as,

Obviously, $\Delta \widetilde{\boldsymbol{W}}^{T}_{FRLoRA}$ can be rewritten as,

Here, the size of $[-T\boldsymbol{B}_G^0; \boldsymbol{B}_G^1; \boldsymbol{B}_G^2; \ldots; \boldsymbol{B}_G^T]$ is $d_1 \times r(T+1)$, and the size of $[\boldsymbol{A}_G^0;\boldsymbol{A}_G^1; \boldsymbol{A}_G^2; \ldots;\boldsymbol{A}_G^T]$ is $r(T+1) \times d_2$.

Thus, we have,

and

Finally, we note that $rank(\Delta \widetilde{\boldsymbol{W}}^T_{FRLoRA})$ is greater than $rank(\Delta \widetilde{\boldsymbol{W}}^T_{FedAvg})$, only except that $\boldsymbol{B}_G^{0}\boldsymbol{A}_G^{0}, \boldsymbol{B}_G^1\boldsymbol{A}_G^1, \ldots, \boldsymbol{B}_G^{T}\boldsymbol{A}_G^{T}$ are completely linearly dependent.

We sincerely hope this theoretical analysis has effectively addressed your concerns. If you have any remaining questions or require further clarification, please do not hesitate to let us know, and we would be glad to provide further explanations.

Thanks for your valuable comment. We further clarify it as follows:

According your description, we redefine the global update of FedAvg with LoRA as,

We can observe that $\Delta \boldsymbol{W}^{t} = \boldsymbol{B}_G^t\boldsymbol{A}_G^t - \boldsymbol{B}_G^{t-1}\boldsymbol{A}_G^{t-1}$ and $\Delta \boldsymbol{W}^{t-1} = \boldsymbol{B}_G^{t-1}\boldsymbol{A}_G^{t-1}-\boldsymbol{B}_G^{t-2}\boldsymbol{A}_G^{t-2}$ are NOT independent. We note that FedAvg only updates the low-rank matrices and uses them as the initialization for the next training round. Consequently, these terms can still be merged into a low rank matrix, $\boldsymbol{B}_G^0\boldsymbol{A}_G^0 + \Delta \boldsymbol{W}^1 + \Delta \boldsymbol{W}^2 + \ldots + \boldsymbol{W}^T = \boldsymbol{B}_G^T\boldsymbol{A}_G^T$. Here, $\boldsymbol{B}_G^T$ is a $d_1 \times r$ matrix and $\boldsymbol{A}_G^T$ is a $r \times d_2$ matirx.

Thus, we have,

In contrast, the global update of FRLoRA can be expressed as,

Given that FRLoRA reinitializes the low-rank matrices in each training round and directly updates the global model using the updated global low-rank matrices, $\Delta \boldsymbol{W}^{t} = \boldsymbol{B}_G^t\boldsymbol{A}_G^t - \boldsymbol{B}_G^{0}\boldsymbol{A}_G^{0}$ and $\Delta \boldsymbol{W}^{t-1} = \boldsymbol{B}_G^{t-1}\boldsymbol{A}_G^{t-1}-\boldsymbol{B}_G^{0}\boldsymbol{A}_G^{0}$ are independent and can NOT be merged.

And $\Delta \widetilde{\boldsymbol{W}}^{T}_{FRLoRA}$ can be rewritten as,

Here, the size of $[-T\boldsymbol{B}_G^0; \boldsymbol{B}_G^1; \boldsymbol{B}_G^2; \ldots; \boldsymbol{B}_G^T]$ is $d_1 \times r(T+1)$, and the size of $[\boldsymbol{A}_G^0;\boldsymbol{A}_G^1; \boldsymbol{A}_G^2; \ldots;\boldsymbol{A}_G^T]$ is $r(T+1) \times d_2$.

Compared to FedAvg, FRLoRA extends the global update from two low-rank matrices, $\boldsymbol{B}_G^T$ and $\boldsymbol{A}_G^T$, to two high-rank matrices, $[-T\boldsymbol{B}_G^0; \boldsymbol{B}_G^1; \boldsymbol{B}_G^2; \ldots; \boldsymbol{B}_G^T]$ and $[\boldsymbol{A}_G^0;\boldsymbol{A}_G^1; \boldsymbol{A}_G^2; \ldots;\boldsymbol{A}_G^T]$.

Thus, we have,

and

Finally, we note that $rank(\Delta \widetilde{\boldsymbol{W}}^T_{FRLoRA})$ is greater than $rank(\Delta \widetilde{\boldsymbol{W}}^T_{FedAvg})$, only except that $\boldsymbol{B}_G^{0}\boldsymbol{A}_G^{0}, \boldsymbol{B}_G^1\boldsymbol{A}_G^1, \ldots, \boldsymbol{B}_G^{T}\boldsymbol{A}_G^{T}$ are completely linearly dependent.

Besides, we also recorded the rank of the global update for FedAvg (r=16) after training for 100 rounds on MetaMathQA. The average rank of $\Delta \widetilde{\boldsymbol{W}}^T_{FedAvg}$ at each layer is 16. In contrast, the average rank of $\Delta\widetilde{\boldsymbol{W}}^{T}_{FRLoRA}$ at each layer for FRLoRA (r=16) reached 82. This further proves that FRLoRA can achieve the global update with a higher rank.

We sincerely hope our response has effectively addressed your concerns. If you have any remaining questions or require further clarification, please do not hesitate to let us know, and we would be glad to provide further explanations.

Thanks for your explanation. My concerns have been addressed and thereby I raising my score. Hope you can include these clarifications in your future mannuscript.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: The distribution of cosine similarity in a single round.

Thanks for your suggestion. We analyzed the cosine similarity of local updates, $\Delta\boldsymbol{W}_k^t$, across different clients in a single round for FedAvg (non-IID), FedAvg (IID), and FRLoRA (non-IID). All methods are trained for the same number of rounds, and the results have been included in the revision (Appendix C.5 Figure 3).

The results show that the cosine similarity between clients in FedAvg (IID) (see Figure 3 (b)) is distributed significantly higher than in FedAvg (non-IID) (see Figure 3 (a)), and FRLoRA (non-IID) (see Figure 3 (c)) exhibits a higher distribution compared to FedAvg (non-IID). This further demonstrates the effectiveness of FRLoRA in reducing client drift, promoting more consistent model convergence under data heterogeneity.

Q2: Empirical analysis of the constrained parameter space.

We have conducted this empirical analysis in our submitted paper (Table 10). And we present these results below.

(1) As shown, increasing the rank $r$ in FedAvg + LoRA does alleviate the constrained parameter space issue, leading to performance improvements. However, this improvement incurs larger communication overhead.

(2) Unlike simply increasing $r$, FRLoRA effectively performs global updates in a higher-rank space with the same $r$ as FedAvg. It can be observed that FRLoRA ($r$ = 16) achieved even higher performance than FedAvg (r=64), strongly demonstrating our claim. We have clarified this in the revision (Lines 1073-1074).

Q3: Limited non-IID scenarios.

We clarify it as follows:

(1) We would like to clarify that the datasets used in our experiments, i.e., FedAya, Fed-ChatbotIT, and Fed-WildChat, are INDEED real-world datasets with data heterogeneity and suggested by FedLLM-Bench [1]. The results (Tables 3 and 4) strongly demonstrate the effectiveness of our method for NLG tasks under severe heterogeneity.

(2) We further investigate the impact of data heterogeneity on our method for NLU tasks. The results have been included in the revision (Appendix C.6 Figure 4), indicating that the accuracy of all methods increases with the increase of $\beta$, and FRLoRA significantly outperforms the other methods at different values of $\beta$. Moreover, the improvement of FRLoRA over other methods is larger when $\beta$ is small, indicating the effectiveness of FRLoRA for NLU tasks under data heterogeneity.

Q4: Add comparison with FlexLoRA.

Thanks for your suggestion. We have compared our method against FlexLoRA in the revision (Tables 1, 2, 3, and 4). The results, as presented below, show that our method consistently outperforms FlexLoRA across nine different benchmarks including both NLG amd NLU tasks.

Table 1: NLU Tasks

Table 2: NLG Tasks with MetaMathQA and Alpaca-GPT4

Table 3: Real-world NLG Tasks with Fed-Aya

Table 4: Real-world NLG Tasks with Fed-ChatbotIT and Fed-WildChat

Q5: Does FRLoRA fully address the intrinsic challenge?

We answer this question as follows:

(1) We agree that FRLoRA does not fully address the constrained optimization space at the local level. In this work, FRLoRA mainly focuses on addressing the intrinsic challenges from the global perspective. It expands the parameter space of global updates to a higher-rank space, allowing for a better representation of the diverse knowledge learned from different clients. Extensive experiments demonstrate its effectiveness.

(2) Since the global model is shared by all clients, we believe that local optimization can also benefit from the global expansion of the parameter space. We will explore this aspect in our future work.

Q6: Catastrophic forgetting at low ranks.

To directly measure catastrophic forgetting at low ranks, we conducted an exploratory experiment, i.e., fine-tuned CLIP/ViT-B32 on CIFAR-10 using FedAvg and FRLoRA. Specifically, the CIFAR-10 training set was divided into 10 clients based on a Dirichlet distribution with $\beta = 0.5$. The model was optimized using AdamW with a learning rate of 5e-5 and a batch size of 64. The fine-tuned models were evaluated on the CIFAR-100 test set in a zero-shot manner, and their performance on the CIFAR-10 test set was also reported. To measure the fine-tuning performance and catastrophic forgetting, we also reported the zero-shot accuracy of the original CLIP/ViT-B32 on the test set of CIFAR-10 and CIFAR-100. The results are presented below:

It can be observed that fine-tuning LoRA on downstream tasks leads to forgetting. However, compared to FedAvg, FRLoRA achieves better zero-shot accuracy on CIFAR-100, FRLoRA (40.32) vs. FedAvg (38.88). This indicates that FRLoRA's initialization approach can partially mitigate the forgetting issue, as it ensures that local training is consistently performed in the principal singular space of the pre-trained model. Furthermore, this work primarily aims to achieve effective federated fine-tuning rather than addressing forgetting. Our focus lies in improving model performance on downstream tasks, thereby facilitating the practical application of LLMs in real-world scenarios. The extensive experimental results presented in this paper, including the findings on CIFAR-10, provide strong evidence that our method successfully reaches this objective.

Q7: Discussion and comparison with Chain of LoRA’s （COLA） residual framework.

(1) Although FRLoRA and COLA have similar ideas, there exists differences in their residual framework. COLA directly merges the updated low-rank matrices $\boldsymbol{B}_G^t\boldsymbol{A}_G^t$ back into the model in each round, whereas FRLoRA only uses the residual updates $\boldsymbol{B}_G^t\boldsymbol{A}_G^t - \boldsymbol{B}_G^0\boldsymbol{A}_G^0$ to update the parameters of the model.

(2) If we apply the residual framework of CoLA in FL with FRLoRA’s SVD-based initialization, then Eq.(9) should be rewritten as: $\Delta\boldsymbol{W}^{t} = \boldsymbol{B}_G^t\boldsymbol{A}_G^t$

And after global update, we need to decompose the new weights of global model into a low-rank structure using Eq.(5)-(7), and use them to initialize the low-rank matrices. Acually, it is FRLoRA-v3, a variant in our ablation study (Section 4.4). We present the results as follows.

We can observe that its results are significantly worse than ours, primarily due to the frequent SVD decompositions causing oscillations in the parameter space, which leads to unstable convergence. Additionally, it also incurs extra training time for large models like LLaMA-2-7B, as shown below.

References

[1] Ye, Rui, et al. "FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models." arXiv preprint arXiv:2406.04845 (2024).

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Time and memory of SVD computation.

We have conducted an analysis of the time and memory requirements for the SVD computation. The results are summarized below:

As shown, the computational cost of SVD is minimal in terms of memory, with peak memory consumption remaining under 1 MB for both models. While the time cost scales with model size, SVD is performed only ONCE, rendering its overall impact negligible compared to the whole training phase. The results have been included in the revision (Appendix C.4 Table 15).

Q2: Further analysis on Tables 3 and 4.

Unlike NLU tasks, NLG tasks involve different datasets for training and testing. For instance, in Table 4, the model is trained on Fed-chatbotIT but evaluated on three benchmarks: MT-Bench, Vicuna, and Ref-GPT4. Due to variations in data distributions or task-specific characteristics across benchmarks, certain subtask metrics may not achieve the best performance. However, the GOAL of fine-tuned LLMs is to achieve the generalized performance across diverse downstream tasks. Consequently, average task performance holds greater importance than local advantages in individual subtasks, especially considering that real-world data distributions are typically heterogeneous and uncontrollable. FRLoRA's performance aligns MORE closely with these practical requirements.

Q3: Discussion about SVD.

We answer your question as follows:

(1) In FRLoRA, SVD is employed ONLY during the initialization phase to decompose the pre-trained weights into a low-rank structure. In our experiments across NINE different benchmarks, SVD has been empirically proven to be stable when applied to these weight matrices, as they are generally well-conditioned. Additionally, SVD is performed only ONCE during the initialization, which minimizes the risk of instability.

(2) For extreme cases, such as with ill-conditioned or noisy weight matrices, SVD may become unstable. Although such scenarios are RARE, we can replace standard SVD with randomized SVD or matrix sketching techniques, which provide more stable decompositions in challenging conditions. Furthermore, incorporating regularization methods like Tikhonov regularization or performing truncated SVD can help stabilize the decomposition process when facing instability. These alternative strategies offer flexibility in handling unstable SVD scenarios, ensuring robust and stable initialization in extreme cases.

I appreciate the dedicated work that the author invested in the rebuttal. My concerns have been addressed, and I will keep my positive rating.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Discrepancy between the uploaded matrices and the averaged updates.

We have conducted an exploratory experiment by fixing the $\boldsymbol{A}$ matrix in FRLoRA, with the results presented below.

While this variant outperforms FFA-LoRA, its performance is inferior to ours, as fixing $\boldsymbol{A}$ greatly restricts the learning capacity. This finding shows that while averaging the uploaded matrices A and B at the server differs from the averaged updates across clients, it can still serve as a suitable tradeoff between discrepancy and performance in contrast to FFA-LoRA.

Q2: Further explanation on the residual update mechanism.

We explain this from the following two aspects:

(1) In our method, the rank of the model parameters is NOT strictly improved after each residual update. Instead, the iterative accumulative effect of these residual updates (Eq.(13)), when applied to the global model’s parameters, expands the parameter space of global updates to a higher-rank space. This mechanism inherently enhances the model’s representational capacity, enabling a better representation of the diverse knowledge learned from different clients.

(2) Besides, we have provided results with different values of $r$ in our submitted paper (Table 10). The results are presented below:

As shown, FRLoRA ($r$ = 16) outperforms FedAvg ($r$=16) and even achieves higher performance than FedAvg (r=64). Besides, we observed that when training FRLoRA ($r=16$) on MetaMathQA after 100 rounds, the average rank of residual accumulation at each layer reached 82. These results strongly demonstrate the effectiveness of our method in expanding the parameter space of global updates.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

Q1: Lack of a compelling real-world use case.

We would like to clarify that the datasets used in our experiments, i.e., FedAya, Fed-ChatbotIT, and Fed-WildChat, are INDEED derived from real-world scenarios and suggested by FedLLM-Bench [1]. Their results in Tables 3 and 4 demonstrate consistently strong performance of FRLoRA across different real-world NLG tasks, providing both a compelling use case and solid empirical evidence of our method's effectiveness in real-world application scenarios.

Q2: High communication cost.

The communication cost analysis presented in the submitted paper only shows the total communication cost (Cost.ToTal) in Setting 2. This may lack key information and be misleading about the communication overhead of our method. In the revision (Appendix C.3 Table 14), we have provided a more detailed communication cost analysis, as shown below.

(1) In Setting 1, where all clients are involved in training every round, FRLoRA incurs NO additional overhead, i.e., FRLoRA (167.76 MB) vs. FedAvg (167.76 MB).

(2) In Setting 2, where only a subset of clients participates in training every round, FRLoRA adds only 75.49MB of total overhead (Cost.Inactive), all of which stems from parameter synchronization for inactive clients. Importantly, FRLoRA indeed does NOT increase the communication cost per client (Only 4.194 MB for inactive clients). As the communication channels between each client and the server are independent and parallel, Cost.Inactive does NOT affect overall communication efficiency.

(3) Besides, we further present the performance in Setting 1, where FRLoRA still achieves better performance than FedAvg, indicating that the performance improvement in Setting 2 is NOT attributable to Cost.Inactive.

References

[1] Ye, Rui, et al. "FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models." arXiv preprint arXiv:2406.04845 (2024).