FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

We greatly appreciate the reviewer's recognition of the contributions of our paper. Regarding the weaknesses raised, we address all the concerns in detail below.

W1 The paper misses some related works. For example, [1] also talks about LoRA in federated fine-tuning and optimize its efficiency and accuracy. [1] and other possible related works should be considered and added to the paper.

[1] is a quite relevant paper also discussing the error of simply averaging LoRA, and focuses on the privacy issues in federated learning to aggregate LoRA modules. We will add a detailed discussion regarding the methods and insights of [1] in our future version.

W2 The experiment results presented in Figure 4 show that some single client perform better than the server, which needs further explanation in the paper. Additionally, the subtitle 'The Impact of Heterogeneous...' here does not match the title 'Standalone...' of Figure 4, which needs further polishing.

In our m=10 experiments, the training dataset is randomly divided into ten parts and distributed to clients for local training, resulting in an uneven quality distribution of the local datasets. Many studies have pointed out that datasets often contain a large amount of useless or even harmful data. As a result, it is acceptable that some clients may train local models that perform better than the global model due to receiving higher-quality or more suitable data for the test set. We will change the title of Figure 4 to make it clearer and more accurate.

W3 The paper should explain why it uses Llama rather than SOTA models (e.g., Llama-3).

In our experiments, we observed that state-of-the-art (SOTA) models are not very sensitive to common fine-tuning datasets. Some models show no performance improvement after fine-tuning or even experience performance declines. For instance, after 3 epochs of fine-tuning on MMLU, we tested Llama-2-7b and obtained the following results:

==========================

$\quad$ $\quad$ $\quad$ $\quad$ Before $\quad$ $\quad$ After

—-------------------------------------------

MMLU $\$ $\$ $\quad$ 45.80 $\quad$ $\quad$ 45.19

HellaSwag $\$ $\$58.69 $\quad$ $\quad$59.47

==========================

We believe this occurs because stronger models like Llama-2-7b have already converged on some common benchmarks, making further fine-tuning less effective in increasing accuracy. The performance of the Llama-3 series is also similar to this. To better demonstrate the effectiveness of fine-tuning, and to provide a clearer comparison between FLoRA and baseline methods, we chose models like Llama-7b where the benefits of fine-tuning are more evident.

Still, following the advice of reviewer 7eRV, we extend our experiments to other series of models, here are some results:

===========================================

Training Set $\quad$ $\quad$ $\quad$ Gemma-2b $\quad$ Gemma-7b

—------------------------------------------------------------------------

MMLU $\quad$ $\quad$ FLoRA $\quad$ $\quad$ 3.40 $\quad$ $\quad$ $\quad$ 5.89

$\quad$ $\quad$ $\quad$ $\quad$ $\$ FedIT $\quad$ $$ $\quad$ 3.33 $\quad$ $\quad$ $\quad$ 5.72

Wizard $\quad$ $\$ $\$ FLoRA $\quad$ $\quad$ 3.54 $\quad$ $\quad$ $\quad$ 6.65

$\quad$ $\quad$ $\quad$ $\quad$ $\$ FedIT $\quad$ $$ $\quad$ 3.44 $\quad$ $\quad$ $\quad$ 6.65

===========================================

We greatly appreciate the reviewer's recognition of our proposed method's effectiveness and broad application scenarios. Regarding the weaknesses raised, we address all the concerns in detail below.

W1: Writing needs to be calibrated. The sentences in line 64 and line 70 are repeated. 2. The discussion on accelerating convergence is not indexed in the main text (Appendix B) and lacks experimental validation.

We thank the reviewer for pointing out errors in our writing and section arrangement. We will correct the repeated part and discuss more about the convergence in the final manuscript. For the experimental validation, we provide some data points here to prove that FLoRA converges faster than our baselines. We use the Wizard dataset to fine-tune a Llama-7b and test on MT-bench. The LoRA rank here we use is 8. We display the global model performance in the first 5 rounds. FLoRA converges earlier than FedIT (in 3rd round) and has higher accuracy.

=================================

Epochs $\quad$ 0 $\quad$ 1 $\quad$ $$ 2 $\quad$ $$ 3 $\quad$ $$ 4 $$ $\quad$ 5

—-------------------------------------------------------

FLoRA $\$ $$ 2.86 $$ 3.79 $$ 3.88 $$ 3.90 $$ 3.88 $$ 3.83

FedIT $\$ $$ $$ 2.86 $$ 2.93 $$ 3.12 $$ 3.47 $$ 3.66 $$ 3.62

=================================

W2: More discussion may be needed on the advantages of FLoRA compared to the implementation of 'the aggregation of local update' in Figure 2.

Figure 2 in our paper illustrates the process where the server first recovers the local updates from the LoRA modules and then aggregates them. While this method might be acceptable if the server has ample computational resources, it significantly increases communication costs. Recovering the full gradient on the server necessitates sending back the full model parameter updates to the clients rather than just the LoRA modules. As shown in Figure 6 of our paper, this results in approximately five times more communication cost compared to our FLoRA approach, which is impractical in most scenarios. Therefore, our FLoRA method offers a substantial reduction in communication overhead, making it a more efficient solution than the traditional method of recovering and aggregating local updates on the server.

W3: The foundation models chosen for the experiment are all from the Llama series. Are there other types of foundation models that can be used for validation?

We thank the reviewer for the advice. Due to the limited time of the rebuttal, we are only able to conduct the experiments on homogeneous LoRA fine-tuning on Gemma models. Here are the results of the MT-bench evaluation:

===========================================

Training Set $\quad$ $\quad$ $\quad$ Gemma-2b $\quad$ Gemma-7b

—------------------------------------------------------------------------

MMLU $\quad$ $\quad$ FLoRA $\quad$ $\quad$ 3.40 $\quad$ $\quad$ $\quad$ 5.89

$\quad$ $\quad$ $\quad$ $\quad$ $\$ FedIT $\quad$ $$ $\quad$ 3.33 $\quad$ $\quad$ $\quad$ 5.72

Wizard $\quad$ $\$ $\$ FLoRA $\quad$ $\quad$ 3.54 $\quad$ $\quad$ $\quad$ 6.65

$\quad$ $\quad$ $\quad$ $\quad$ $\$ FedIT $\quad$ $$ $\quad$ 3.44 $\quad$ $\quad$ $\quad$ 6.65

===========================================

We can see that FLoRA still outperforms the baseline in Gemma models. We will further conduct heterogeneous experiments and experiments of MMLU and ARC evaluation.

We greatly appreciate the reviewer's recognition of the effectiveness of our proposed method. Regarding the weaknesses raised, we address all the concerns in detail below.

W1: The novelty is limited. Although the proposed method is effective, the contribution of the stack-based aggregation strategy may be too limited.

The primary contribution of our paper is identifying a mathematically inaccurate aggregation noise present in nearly all existing LoRA-based federated fine-tuning algorithms. We then propose a novel and efficient stacking method to address this issue. Our paper introduces significant novelty by being the first to highlight the noise generated by the simplistic averaging process commonly used. Given that almost all current LoRA federated fine-tuning algorithms and codebases rely on this averaging method, our stacking algorithm offers a mathematically accurate and provably convergent solution. We believe this contribution will provide substantial value to the community.

W2: The motivation of the stack based aggregation is not clear. Since the computational resource of the server is usually sufficient, it is totally acceptable that recovering the local update and then aggregating them.

Recovering the local updates and then aggregating them may be acceptable from a computational cost perspective. However, this approach necessitates the server sending back the full model parameter updates to the clients, rather than just the LoRA modules. As illustrated in Figure 6 of our paper, this method results in approximately five times the communication cost compared to our FLoRA approach, which is impractical. Therefore, we argue that recovering the local updates and then aggregating them is not a feasible solution due to the prohibitive communication overhead.

Dear Reviewer,

We wanted to inform you that we have addressed the additional concerns you raised. As the rebuttal phase is coming to an end, we kindly request your prompt feedback on our rebuttal.

We appreciate your time and consideration and look forward to your response.

Best regards, Authors

Dear Reviewer 4FRM,

The rebuttal deadline is approaching soon, and we have provided detailed responses to your latest concerns. We kindly request that you review our replies at your earliest convenience. If there are any further questions or issues, please let us know. If our responses have adequately addressed your concerns, we would appreciate a higher score.

Thank you for your time and consideration.

Compared to storing the LoRA modules in a set, stack-based methods can significantly reduce computational costs due to the following advantages:

1. Reduced Memory Overhead: During the weight recovery process of LoRA, each LoRA module pair is expanded to the full parameter size. In LoRA fine-tuning with $M$ clients, if the modules are stored in a set, each module must be recovered separately, leading to a total memory usage of $(M + 1)P_{\text{full}}$, where $P_{\text{full}}$ represents the full parameter size. This includes the original parameters and $M$ newly recovered parameters. In contrast, the stack-based method employed in FLoRA allows for the recovery of the full parameters with only a single matrix multiplication, resulting in a total memory requirement of just $2P_{\text{full}}$. Given that $M > 1$ in federated fine-tuning, this approach leads to substantial computational savings.

2. Optimized GPU Utilization: Since parallel computing on a GPU is significantly faster than sequential processing, it is more efficient to compute the stacked LoRA modules collectively rather than individually. This approach better aligns with the GPU’s architecture, leading to faster computations and more efficient resource utilization. To illustrate the benefits of our method in terms of computation and communication costs, particularly in comparison to the baseline FedIT and the set-based method proposed by the reviewer, we provide a table below. For reference, the communication and computation costs of FLoRA are normalized to 1:

=========================================================================================

Method $\quad$ Communication $\quad$ Recovering Computation $\quad$ Accurate aggregation $\quad$ Applicable to inference

---

FedIT $\quad$ $\quad$ P_{full}/P_{LoRA} $\qquad$ $\qquad$ 1 $\qquad$ $\qquad$ $\quad$ $\qquad$ $\qquad$ No $\quad$ $\qquad$ $\qquad$ $\qquad$ $\qquad$ Yes

Set-saving $\qquad$ $\quad$ 1 $\qquad$ $\qquad$ $\qquad$ $(M+1)/2$ $\$ $\$ $\$ $\qquad$ $\quad$ $\qquad$ Yes $\quad$ $\qquad$ $\qquad$ $\qquad$ $\qquad$ No

FLoRA (ours) $\quad$ $\quad$ 1 $\qquad$ $\qquad$ $\qquad$ $\quad$ 1 $\quad$ $\quad$ $\qquad$ $\qquad$ $\quad$ $\qquad$ Yes $\quad$ $\qquad$ $\qquad$ $\qquad$ $\qquad$Yes

=========================================================================================

According to this table, our approach generally outperforms the baselines in important metrics.

Thanks for the response. I agree that using stack-based aggregation can improve GPU utilization. Yet, I am confused about the other advantages.

1. The simple set-based aggregation can also have a total memory usage of ( 2P_{\text{full}} ). In particular, we can compute B_iA_i one by one and add outcomes one by one.

2. I am confused about why the recovering computation of set-based aggregation is (M+1)/2. Considering an extreme case $r=1, A \in R^{d * 1}, B \in R^{1 * d}$, then the computational cost of set-based aggregation is $K*d*d$

The computational cost of stack-based computational cost is also $K*d*d$.