FedLoRA: When Personalized Federated Learning Meets Low-Rank Adaptation

Q1: ... sharing the full-rank parameters with the server might raise privacy concerns ...

Thank you for your feedback. It seems there might have been some misunderstanding. In fact, the core principle of federated learning is to ensure privacy preservation by sharing full-rank model parameters. Our proposed FedLoRA adheres to this same design, sharing full-rank model parameters with the server, and as such, it does not compromise privacy preservation.

With that said, it's important to note that some research works do recognize the potential privacy leakage risk associated with parameter transmission and propose specific solutions to enhance privacy preservation. These methods can be seamlessly integrated into our existing FedLoRA approach. However, since our study does not primarily focus on this aspect, we have not undertaken additional designs and experiments in this regard.

Q2: ... provide details on the memory cost ... compared to the general LoRA ...

Thanks for the comment. While we draw inspiration from the design principles of LoRA, it's essential to note that the application scenarios, actual methods, and objectives of our approach are significantly different from those of LoRA. Hence, we believe that making direct comparisons between FedLoRA and LoRA might not be valid.. Specifically, LoRA aims to train a low-rank part for fine-tuning a given pre-trained model (full rank). In contrast, FedLoRA aims to train a full-rank model for extracting shared knowledge and simultaneously train a low-rank personalized part for each client.

We believe that it would be more appropriate to compare FedLoRA, specifically in terms of computational cost, with other federated learning algorithms, such as FedAvg. Benefiting from our proposed alternating training, FedLoRA has a lower computation overhead than most traditional FL methods such as FedAvg. For example, if all clients in FL need to train locally for $E$ epochs before uploading parameters, then for FedAvg, each epoch requires training a standard full-rank model; but for FedLoRA, it trains $E_{lora}$ rounds of the low-rank part (fewer trainable parameters) and $E - E_{lora}$ rounds of the full-rank part. Therefore, the overall computational overhead is actually reduced compared to FedAvg.

Dear respected reviewer, we are wondering whether you have any further questions regarding our work? We posted our response regarding your concerns on 19st Nov but have not received your valuable comments yet. Any responses or score updates will be greatly appreciated.

Q1: ... benefit from more profound exposition of the underlying intuition of the proposed method. Offering analytical comparisons or insights why this method outperforms state-of-the-art approaches ...

Thanks for your review.

The insight of applying LoRA to PFL:

Benefiting from the low-rank nature of LoRA, introducing LoRA into FL for personalization has the following advantages over current PFL methods. First, constraining the personalized part to low-rank can reduce the overfitting to the local knowledge, especially when the clients' local data are very limited. Second, the low-rank structure allows the personalized part to focus its learning on the most critical aspects of the local knowledge. Third, LoRA greatly reduces the trainable parameters, it helps in retaining much of the general knowledge acquired from other clients to enhance generalization.

The insight of proposed alternating training:

The motivation for proposing alternating training is that by first training the personalized part $\tau$, the client-specific knowledge is mostly learned by $\tau$ and the shift of $\theta$ to the local minimum point is mainly done by $\tau$. Therefore, alternating training can reduce the deviation of shared parameters $\sigma$ to the local minimum point of the client, thereby reducing the influence of non-IID on general knowledge extraction. This motivation has been discussed in Section 3.4.

To validate this, we add an experiment to further prove this intuition. We add additional experiments to compare the differences in shared parameters among clients with and without alternating training. We expect that if alternating can reduce the deviation of $\sigma$, then the difference of the shared part among clients in the local training phase should be reduced. The results are shown in the table below. We can see that on both datasets, using alternating training can significantly reduce the difference in the shared part of clients.

For more detailed statistics including the trend of model difference during the training process, you can refer to Appendix C in our revised version.

We intend to incorporate these experiments into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Q2: The concern of overfitting ... of the full-rank matrix needs to be adequately addressed ...

Thanks for your review.

We feel that there might be some miscommunication here. In fact, FedLoRA has the same amount of shared parameters compared with the baseline method FedAvg. Therefore, FedLoRA does not bring additional risks of over-fitting. Here are detailed explanations:

First, we would like to clarify that our intention is not to use LoRA for training large language models (LLMs) in Federated Learning (FL). Instead, we draw inspiration from the concept of LoRA, which was originally conceived for fine-tuning LLMs, and adapt it to the context of Personalized Federated Learning (PFL). Consequently, the full-rank model employed in FedLoRA essentially resembles a traditional neural network commonly used in the FL literature. Therefore, FedLoRA has the same amount of shared parameters compared to the majority of FL methods that upload models, such as FedAvg. As a result, our FedLoRA does not bring additional risks of over-fitting.

Q3: The method's scalability seems constrained by model size ...

Thanks for your review.

This question has been partially addressed in our response to Q2. In our reply to Q2, we emphasize that FedLoRA maintains an equivalent number of shared parameters compared to the baseline method, FedAvg. Consequently, in comparison to the baseline algorithm, FedAvg, our approach does not incur additional communication costs when applied to a large-sized model.

Dear respected reviewer, we are wondering whether you have any further questions regarding our work? We posted our response regarding your concerns on 19st Nov but have not received your valuable comments yet. Any responses or score updates will be greatly appreciated.

We sincerely appreciate the comments provided by the reviewer. To clarify, we need to reiterate that our goal is not to fine-tune a given pre-trained network; rather, it is to train a model from scratch in the context of federated learning. This task is the most common task setting in the field of federated learning, and the model trained from scratch is full-rank in the vast majority of scenarios. Consequently, the appropriate benchmark for our work should not be the parameter-efficient fine-tuning of models. Rather, it should involve the comparison with similar efforts aimed at training full-rank models from scratch, which in the field of federated learning, refers to FedAvg. Additionally, in response to the reviewer's pertinent query about the computational costs involved, we list detailed numerical results in our ResNet-10 experiments here for reference:

In the above table, 'Number of parameters to update' refers to the total number of updates in each training round (i.e., the number of updated parameters $\times$ update times), which directly reflects the computational cost in each client.

In our experiments, the number of full-rank parameters (shared) is 11.24M, and the number of low-rank parameters (personalized) is 8.58M. The $E=5$ and the $E_{lora} = 3$. Hence the final number of parameters to update for FedAvg is 56.2M, and that for FedLoRA is 48.22M. Therefore, compared with the FedAvg algorithm, our computation overhead is lower and the communication overhead is the same.

Q1: ...lacks experiments and analysis about training and communication cost ... the new training strategy needs to train two times ... the cost for transmitting full-rank matrixes may be huge ...

Thank you for your feedback. We feel that there might be some miscommunication here. We would like to emphasize that FedLoRA has the same communication overhead and lower computation overhead than the baseline method FedAvg. Here are the detailed explanations:

First, we would like to clarify that our intention is not to use LoRA for training large language models (LLMs) in Federated Learning (FL). Instead, we draw inspiration from the concept of LoRA, which was originally conceived for fine-tuning LLMs, and adapt it to the context of Personalized Federated Learning (PFL). Consequently, the full-rank model employed in FedLoRA essentially resembles a traditional neural network commonly used in the FL literature. As a result, FedLoRA shares the same communication overhead as the majority of FL methods since these methods, including FedAvg, all involve uploading the full-rank model.

In terms of computational cost, it's worth noting that FedLoRA actually incurs a lower computational overhead compared to many traditional FL methods, including FedAvg. This advantage can be attributed to our proposed alternating training technique. For example, in a FL setting where all clients need to train locally for $E$ epochs before uploading parameters, then for FedAvg, each epoch requires training a standard full-rank model; but for FedLoRA, it trains $E_{lora}$ rounds of the low-rank part (fewer trainable parameters) and $E - E_{lora}$ rounds of the full-rank part. Therefore, the overall computational overhead is actually reduced compared to FedAvg.

Q2: ... may lead to sacrificing the performance of some clients for improvement.

Thank you for your valuable insights. We have performed additional experiments to demonstrate that our proposed FedLoRA, which incorporates personalized low-rank parameter matrices, does not result in any degradation of individual client performance.

The experimental details are as follows. We test FedLoRA on CIFAR-100 and Tiny-ImageNet datasets in the Dirichlet non-IID scenario with $\alpha=0.1$. Specifically, we plot the mean / minimum / maximum of accuracy improvements of FedLoRA for each client, compared with FedAvg and Individual Training (i.e., each client trains the model locally without collaboration).

The results are summarized in the following table.

For more detailed statistics for each client, please refer to Appendix D in our revised paper.

Notice that the minimum improvement value across all clients when employing FedLoRA is greater than 0, affirming that the use of FedLoRA does not lead to any deterioration in individual client performance.

We intend to incorporate these experiments into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Q3: ... no experiments and analysis ... to support the intuition in Fig 3.

Thanks for the comment. We conduct additional experiments to further validate the intuition in Fig 3. Here is the detailed experiment information.

The motivation for proposing alternating training is to reduce the impact of data heterogeneity on the shared part. That is, reduces the deviation of shared parameters to the local minimum point of the client. Thus, we expect that using alternating training should reduce the difference in shared parameters among clients during local training. We conduct experiments to analyze the change in model distance before and after using alternating training. The results are shown in the following table:

For more detailed statistics including the trend of model difference during the training process, please refer to Appendix C in our revised paper. It is evident that the utilization of the alternating training strategy results in a significantly smaller disparity in the shared components of the model. This observation aligns with our intuition, as illustrated in Fig. (3).

We intend to incorporate these experiments into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Dear respected reviewer, we are wondering whether you have any further questions regarding our work? We posted our response regarding your concerns on 19st Nov but have not received your valuable comments yet. Any responses or score updates will be greatly appreciated.

Q1: Is it possible to extend the experiments to use a ImageNet-pretrained ResNet ... we should expect that $\tau$ is learned more often than $\sigma$ ...

Thanks for your review. As requested by the reviewer, we add experiments about how the update for global part and personalized part (denoted as $\Delta \sigma$ and $\Delta \tau$) affect the final performance. Specifically, we initialize the $\sigma$ with ImageNet pre-trained weights and conduct experiment on the CIFAR-100 dataset in the Dirichlet non-IID scenario with $\alpha=0.1$.

We control the value of $E_{lora}$ (higher $E_{lora}$ means update $\tau$ more often, and results in larger $\Delta \tau$) at different levels. The results are summarized in the table below.

From this table, we can conclude that when $\Delta \tau$ is larger than $\Delta \sigma$, FedLoRA achieves better results. This indicates that with the pre-trained weights, the $\tau$ should be updated more often than $\sigma$ to learn client-specific knowledge. This aligns with the reviewer's expectation and can serve as strong support for the client-specific knowledge statement.

We intend to incorporate these experiments into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Q1

This ImageNet-pretrained ResNet experiment on CIFAR100 looks good! I don't have any other major concerns about this paper now.

I would like to maintain my score.

Dear respected reviewer, we are wondering whether you have any further questions regarding our work? We posted our response regarding your concerns on 19st Nov but have not received your valuable comments yet. Any responses or score updates will be greatly appreciated.

Thank you for your feedback.

Q1: ... novelty in applying LoRA to federated learning. ... why LoRA is a better choice for personalized FL than existing methods ...

First, recall that LoRA was initially designed for fine-tuning large language models (LLMs). Its core concept involves the integration of low-rank parameter matrices, which have fewer trainable parameters, into the pre-trained model to adapt it for downstream tasks. In a similar vein, in the context of personalized federated learning (PFL), FedLoRA plays a role by incorporating personalized low-rank matrices into the shared global model for each client. This enables the global model to acquire client-specific knowledge.

Benefiting from the low-rank nature of LoRA, introducing LoRA into FL for personalization has the following advantages over current PFL methods. First, constraining the personalized part to low-rank can reduce the overfitting to the local knowledge, especially when the clients' local data are very limited. Second, the low-rank structure allows the personalized part to focus its learning on the most critical aspects of the local knowledge. Third, LoRA greatly reduces the trainable parameters, it helps in retaining much of the general knowledge acquired from other clients to enhance generalization.

Q2: several existing works have explored this ... has the name of FedLoRA

The paper titled 'Model-Heterogeneous Personalized Federated Learning with LoRA Tuning' was made available on arXiv on October 20, which is after the submission date of our paper (e.g., September 28). Nevertheless, it's important to note that our paper differs significantly from this work in terms of objectives and methodologies. While they employ LoRA to address the challenge of model heterogeneity, our research focuses on addressing data heterogeneity. Their primary goal is to reduce computing and communication overhead, whereas our aim is to improve the separation between the learning of general knowledge and client-specific knowledge.

Q3: ... alternative training ... lacks theoretical justification.

While we reserve the theoretical justification for future work, we have taken your comment into consideration and conducted empirical experiments to validate the effectiveness of the proposed alternating training strategy.

As a quick reminder, the primary objective of alternating training is to mitigate the impact of data heterogeneity on the shared parameters, essentially reducing the deviation of shared parameters to the local minimum point of the client. Consequently, employing alternating training should lead to a reduction in the discrepancies among shared parameters across clients during their local training phases. To validate the effectiveness of alternating training in achieving this goal, we carried out additional experiments to compare the disparities in shared parameters among clients when using alternating training as opposed to not using it. The results are presented in the table below. It's evident from the data that, across both datasets, the utilization of alternating training significantly diminishes the differences in the shared parameters among clients.

For a more comprehensive set of statistics, including the trend of model differences throughout the training process, please refer to Appendix C in our revised paper.

Q4: More large datasets, especially in different modalities, should be used.

In response to your comment, we have conducted additional experiments on both a larger image dataset and a natural language processing (NLP) dataset.

• For the larger image dataset, we selected a subset from ImageNet, consisting of 400 classes with a total of 80,000 samples. We utilized the ResNet-10 model architecture, with each client having 2,000 training samples generated following the Dirichlet distribution with $\alpha=0.1$.
• For the NLP dataset, we opted for AG_NEWS, a text classification dataset with 120,000 samples. We employed the Transformer model architecture, with each client having 3,000 training samples generated following the Dirichlet distribution with $\alpha=1.0$. Additionally, for the Transformer model, we applied model decomposition to the weights in the self-attention modules and fully connected weights in the classifier module.

The table above displays the test accuracy results for these two datasets. It's evident that FedLoRA consistently outperforms other state-of-the-art methods on both datasets. We intend to incorporate these experiments on the new datasets into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Dear respected reviewer, we are wondering whether you have any further questions regarding our work? We posted our response regarding your concerns on 19st Nov but have not received your valuable comments yet. Any responses or score updates will be greatly appreciated.