Closed-Form Merging of Parameter-Efficient Modules for Federated Continual Learning

Below, we summarize reviewer AkSa's concerns and respond to each one comprehensively

1. A diagram visualizing the FCIL setting would be helpful To better clarify the Federated Class-Incremental Learning (FCIL) setting, we will include a diagram that visually represents this framework. Given the space constraints, we will reference this diagram in the main paper and provide the full illustration in the appendix.

2. More detailed caption for figure 1 We have updated the caption of Figure 1 to improve clarity. We hope that the revised version is more understandable. We report it here for convenience.

Training and aggregation procedure of LoRM. For a generic layer, at the end of the communication round $j$ or $j+1$, we obtain the global $B_M$ or $A_M$ matrix, respectively, starting from i) the distributed $A$'s or $B$'s, and ii) the gram matrices ($XX^{\top}$'s). $B_M$ or $A_M$ will serve as the fixed matrix in the next round. Finally, we apply RegMean incrementally to compute $\Delta W$.

3. Missing competitors: RanPAC and DualPrompt We conducted additional experiments involving these two methodologies on CIFAR-100 and Imagenet-R, and provide the results below.

CIFAR-100

Imagenet-R

4. Quantity-based label imbalance Additional experiments on the quantity-based label imbalance are provided in the following for the CIFAR-100 dataset. The dataset choice is motivated by CIFAR-100 being a common benchmark between our work and PILoRA. In the table, $\alpha$ indicates the number of classes in the quantity-based scenario. Following PILoRA, we used 6, 4, and 2.

CIFAR-100

5. L2P and TARGET’s relative ranking is switched w.r.t. PILoRA's experiments The reviewer correctly observed that, while L2P outperforms TARGET in our experiments, the opposite result is reported in the PILoRA paper. Although this may seem counterintuitive, we would like to highlight key differences between our experimental setup and that of PILoRA. Specifically, our setting uses fewer communication rounds (5 vs. 30) and a different pretraining checkpoint (ImageNet-21k vs. DINO). In PILoRA's scenario, the extended 30-rounds training may increase forgetting in L2P, as it lacks an explicit regularization mechanism to mitigate it. In contrast, TARGET's distillation approach likely helps preserving knowledge over the longer sequence of rounds.

6. L2P surprisingly outperforms CODA-Prompt In the original Class-Incremental Learning setting, where both methodologies were introduced, CODA-Prompt generally outperforms L2P. However, in our decentralized experimental setting, the dynamics can differ. We have no theoretical intuition for such behavior but we have thoroughly checked the source code of both methodologies, and successfully replicated the results reported in their respective original papers within their specified settings.

7. Additional results for ImageNet-A, CUB, and Cars196 We conducted additional experiments on these datasets, and report the results below.

CUB

CARS

Imagenet-A

We thank reviewer BHoB for their valuable insights. Below, we provide detailed responses to their comments and concerns.

1. Inadequate experimental diversity Regarding the experimental scope, we chose image classification as a foundational task to establish the efficacy of LoRM in a well-studied domain. We acknowledge the importance of evaluating LoRM on more diverse tasks and backbones, such as large language models (LLMs). We are actively working on additional experiments on intent classification (Out Of Scope, OOS [3]); they will be provided by the end of the rebuttal period whether possible. Otherwise, we will add them to the final version of the manuscript. Additional settings such as text-to-image models, other federated learning settings, reinforcement learning and generative tasks represent exciting avenues which we will explore in future works.

The reviewer also suggested that our contributions are backed by only empirical results. While our current work primarily focuses on image classification, we would like to emphasize that our contributions are not solely empirical. Specifically, we provide theoretical results that extend merging to different parameter-efficient fine-tuning (PEFT) techniques, including LoRA (appendix A1), VeRA (appendix A2), and IA3 (appendix A3), demonstrating the broader applicability of our method. Additionally, we prove that concatenating the incremental heads is mathematically equivalent to minimizing the RegMean regression problem across all incremental tasks (appendix A4).

2. Sensitivity to data distribution We thank the reviewer for suggesting additional regularization objectives, which could potentially improve performance in scenarios characterized by high data heterogeneity.

Even without incorporating extra regularizations -- which could increase the computational and memory footprint of LoRM -- we achieve state-of-the-art results under the highest levels of data heterogeneity reported in the literature (i.e., dirichlet distribution $\beta = 0.05$). As also requested by reviewer AkSa, we have included additional results for quantity-based label imbalance to further address this aspect and provide a clearer understanding of LoRM's performance in such conditions. The reviewer can find these results in the response to reviewer AkSa.

3. It is unclear why the alternating freezing is needed If we understand correctly, the reviewer suggests that reinitializing a new LoRA module at each round, starting from the full-rank merged module of the previous round, would achieve comparable performance while being simpler.

Although this suggestion is valid, implementing such a technique would severely increase the communication cost by a factor of $18.9\times$, as it requires sharing the full-rank updates back to the clients at each communication round. This becomes a critical limitation in Federated Learning, as would compromise the algorithm's feasibility in decentralized scenarios.

4. Comparisons to model merging [1, 2] We thank the reviewer for the suggestion, acknowledging the importance of comparing our method to existing model merging approaches such as ZipIt! [1] and TIES-Merging [2]. However, ZipIt! is not directly applicable to the Federated Learning scenario, as it requires access to model activations, which are not available for the server model (that is responsible for merging the clients' updates).

Instead, TIES-Merging operates solely on the parameters, allowing its applicability to our scenario. We will update the experimental section accordingly, including this method by the end of the rebuttal period.

[1] Stoica, G., et al. (2023). Zipit! merging models from different tasks without training. In ICLR.

[2] Yadav, P., et al (2024). Ties-merging: Resolving interference when merging models. In NeurIPS.

[3] Larson, S., et al. (2019). An evaluation dataset for intent classification and out-of-scope prediction. In EMNLP-IJCNLP.

1. Inadequate experimental diversity Thank you for the response. I'd like to point the paper claims to be a general method for parameter efficient training for "federated learning" which it underlines by showing applicability to LoRA. As we know LoRA is and has been applied to a large number of domain and modalities. Therefore, showing results only on Image Classification and on 3 conventional datasets doesn't satisfy the initial claim. It'd been a different case if the paper has claimed a narrow application to only image classification, but in that case the use of LoRA, which was initially created as a method efficiently fientune large LLMs would have been problematic.

2. Sensitivity to data distribution Although I appreciate authors' EuroSat results, as pointed out in the previous paragraph, as per the claims of the current version of the paper, I find a single experiment regarding OOD robustness not fully adequate.

3. It is unclear why the alternating freezing is needed I don't think I agree with the authors here, however if communication compression or efficiency is an issue, why not simply transfer the modification of the current LoRA matrix from a client in the form of a rank update or another low rank factorization of $\Delta A$ or $\Delta B$. Such hierarchical LoRA variants actually exist in literature and more efficient than simply transferring individual matrices.

4. Comparisons As noted in above paragraph, if the main objective is communication efficiency as the authors claim, then comparisons with model merging and more efficient PeFT methods is needed.

1. I'd like to note that my contention is with the overtly broad claims regarding federated learning made in your work. Your work is Federated Class-Incremental Learning for Image Classification only. However, your paper appears to present it as a general FCIL work, leaving the scope of the work highly ambiguous. This ambiguity is amplified by the usage of LoRA, which was initially introduced to efficiently finetune large language models. Therefore, I find that the currently written paper is too broad in its claims and need to be rewritten to adequately define the correct scope. Even after that, the question arises as to why use LoRA for a task and a backbone which is not afflicted by the problem of efficiency in the first place[1], and why claim a closed form solution for LoRA in general while only showing a "non-large" model backbone for experiments, and on datasets which do not require extensive compute or time to train. In summary, the paper makes too broad claims in its current form and does not explain well why LoRA should be used in this highly narrow, constrained manner.

2. I'd stand by my earlier comment - as per the claims of the current version of the paper, I find a single experiment regarding OOD robustness not fully adequate, and even the current one does not provide a good evidence for claims of robustness that the authors write.

3. There are number of methods which would provide more "communication efficiency" over transferring entire LoRA matrices, such as [2,3,4,5,6,7]. A few of these papers are unpublished as of now, and there is no need to compare to them - however, they underline the general idea of a whole area of very efficient low rank adapters which freeze or share different parts of the low rank matrices which can be trivially applied to the FCIL tasks especially since the "communication efficiency" here is important. These papers do not even include Mixture of Expert models, where one would only need to transfer sparse expert weights among the nodes of a federated system. The problem is that this paper introduces a solution for a problem without evaluating whether currently available methods can do similarly well or even better - there is no theoretical reasoning or empirical results as to why the closed form solution would be better than the other relevant PeFT or mixture methods for better efficiency and performance, especially since the focus is on efficiency.

4. I'd also argue that the current paper, with its only focus on image classification (relatively small datasets at that) and a single, relatively lightweight backbone does not provide any practical benefits to the community in terms of federated continual learning. Federated learning, among other motivations, is used to make learning of large models, more energy and compute efficient by using shared resources. In case of a ViT, I'd argue that more energy would be spent sending and receiving data than training the lightweight model on a single modern GPU. As such, the current experimental setup remains largely theoretical and does not hold much practical value for the community. This would change if the paper showed results on larger models, and compute extensive and complex tasks, like Natural Language Generation, Image Generation, etc.

As a general remark, I believe the current version of the paper makes unproven broad claim with regards to federated continual learning as a whole. It also uses low rank adaptation, which originally developed for finetuning large models efficiently, for a single task and backbone which arguably do not share the same efficiency problem for what LoRA was originally developed for. Further, using LoRA, worsens the problem of the unproven claims as the paper gives off an idea that this method works beyond IID image classification task when there is no evidence otherwise - the current version of the paper does not clearly disambiguate this in their paper or the title. Further since the main contribution is "communication efficiency", there exist a wide host of current methods which are good solutions to this problem setup, and the authors do not provide an effective reason or results as to why their method is better especially since the focus is on data, compute and communication efficiency. Further the current experimental setup holds little practical value for the community (point 4). Until these points are remedied, I believe my rating is a fair evaluation.

[1] Nag et al. - ViTA: A Vision Transformer Inference Accelerator for Edge Applications

[2] VeRa: Vector-based Random Matrix Adaptation

[3] Balazy et al. - LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

[4] SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors

[5] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

[6] Batched Low-Rank Adaptation of Foundation Models

[7] LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation

Below, we address reviewer H9U5's questions and concerns in detail.

1. Additional results on class-incremental Learning (Computer Vision and Natural Language processing) Our experimental analysis focuses on incremental computer vision tasks (classification) learned in a decentralized fashion. Regarding computer vision, we have conducted additional experiments on other datasets as requested by reviwer AkSa (i.e. CUB-200, CARS, Imagenet-A), and we report the results in the response to AkSa. We acknowledge that further analysis on natural language processing tasks could be valuable. We are actively working on this and will include the results by the end of the rebuttal period if possible; otherwise, they will be incorporated into the final version of the manuscript.

2. Theoretical guarantees for convergence As this point was also raised by reviewer rSBD, we have provided the derivation of the convergence proof for LoRM in response to their rebuttal. Due to space constraints, we refer the reviewer to that response without duplicating the proof here.

While LoRM has been experimentally shown to converge faster than other competitors, we do not yet have a theoretical proof to explain this behavior. We hypothesize that this improvement arises from two key factors: i) LoRM utilizes low-rank fine-tuning via LoRA, which reduces both global and local gradient magnitudes; ii) our layer-wise merging technique facilitates a better solution to the global optimization problem w.r.t. simple averaging, further decreasing the aforementioned gradients.

We sincerely thank the reviewer for raising their score and for their thoughtful feedback on our work. We appreciate their recognition of our responses to their concerns and remain available to address any further questions.

Below, we summarize reviewer rSBD's concerns and provide detailed responses.

1. Contribution (Alternative optimization vs. closed-form) We agree with the reviewer that alternating the training of the A and B LoRA matrices is a standalone technique that can be applied independently; however, in our approach, this alternation is inherently tied to the closed-form merging technique we employ for FCIL. In fact, the merging problem defined in Equation 5 cannot be solved in a single closed-form manner for both A and B simultaneously, as the system is underdetermined (Equation 6). To address this, we alternate the optimization of these two variables, treating one as constant while solving for the other, thereby finding a determinate solution. As correctly understood by the reviewer, closed-form merging is applied at each communication round (i.e. the iteration of the global algorithm) for the respective low-rank matices, and the overall optimization solver follows the mechanism of alternating optimization.

2. Convergence check for the optimization solver For what concern the convergence check, we refer the reviewer to [1], which demonstrates the convergence for a federated algorithm that applies the FedAVG technique. Differently from FedAVG, our approach changes the update rule, leveraging the optimal solution to the regression problem for each layer (equation 5 of the manuscript) instead of simple parameter averaging. Formally, for a generic even round $j$:

$

\overline{\Delta}^{j} = B_M^{j} A_M^j = B_M^{j} \left(\sum_{i = 1}^m A_i^j X_i X_i^\top \right) \left(\sum_{i = 1}^m X_i X_i^\top\right)^{-1},

$

where $m$ denotes the number of clients, $\overline{\Delta}^j$ the full-rank update for the round $j$, $B$ and $A$ represent the LoRA matrices, with the subscript $_M$ standing for merged, and $X_i X_i^\top$ corresponds to the Gram matrices of the inputs. As we are alternating the optimization, for a generic odd round $j$:

$

\overline{\Delta}^{j} = B_M^j A_M^{j} = \left(\sum_{i = 1}^m B_i^j A_M^{j} X_i X_i^\top\right) A_M^{j\top} \left(A^{j} \sum_{i = 1}^m X_i X_i^\top A_M^{j\top} \right)^{-1} A_M^{j}.

$

Following the convergence proof of [1] (appendix A.1, equation 2, second-last line), with $W$ indicating the full-rank parameters for clarity, we can rewrite the central term $\frac{\eta_L}{2m}\sum_{i=1}^{m}\sum_{k=0}^{K-1}\mathbb{E}||\nabla F_i(W_{j, k}^i) - \nabla F_i(W_j)||^2$ as:

$

\frac{\eta_L}{2m}\sum_{i=1}^{m}\sum_{k=0}^{K-1}\mathbb{E}\left|\left|B_M^{j}\left[\nabla F_i(A^i_{j, k}) - \nabla F_i(A_j)\right] X_i X_i^\top \left(\sum_{l = 1}^m X_l X_l^\top\right)^{-1}\right|\right|^2, \text{ and}

$

$

\frac{\eta_L}{2m}\sum_{i=1}^{m}\sum_{k=0}^{K-1}\mathbb{E}\left|\left|\left[\nabla F_i(B^i_{j, k}) - \nabla F_i(B_j)\right]A_M^{j}X_i X_i^\top A_M^{j} \left(\sum_{l = 1}^m A_M^{j} X_l X_l^\top A_M^{j} \right)^{-1} A_M^{j}\right|\right|^2,

$

for the even and odd rounds respectively. The same modification applies to the first term of appendix A.1 of [1], equation 3, last line. Similarly, we replace the second term of the same line $\frac{\eta^2_L}{m^2}\mathbb{E}\left|\left|\sum_{i=1}^{m}\sum_{k=0}^{K-1}\nabla F_i(W^i_{j, k})\right|\right|^2$ as:

$

\frac{\eta^2_L}{m^2}\mathbb{E}\left|\left|\sum_{i=1}^{m}\sum_{k=0}^{K-1}B_M^{j-1}\nabla F_i(A^i_{j, k})X_i X_i^\top \left(\sum_{l = 1}^m X_l X_l^\top\right)^{-1}\right|\right|^2, \text{ and}

$

$

\frac{\eta^2_L}{m^2}\mathbb{E}\left|\left|\sum_{i=1}^{m}\sum_{k=0}^{K-1}\nabla F_i(B^i_{j, k})A_M^{j}X_i X_i^\top A_M^{j} \left(\sum_{l = 1}^m A_M^{j} X_l X_l^\top A_M^{j} \right)^{-1} A_M^{j}\right|\right|^2.

$

In each round, our solution introduces a dependency on the input data of each client by leveraging the corresponding Gram matrix as a weighting factor. This factor is normalized by multiplying it with the inverse of the sum of all Gram matrices across all clients. Since the Gram matrix is positive semi-definite, this normalization ensures that the resulting norm remains bounded. Building on Assumption 3 from [1] (bounded local gradients, $\sigma_L$, and global gradients, $\sigma_G$) and leveraging Lemma 4 from [2], these expectations can be reformulated in terms of the constants $\sigma_L$ and $\sigma_G$. We will include the complete proof in a new section of the appendix.

[1] Yang, H., et al. (2021). Achieving linear speedup with partial worker participation in non-iid federated learning. In ICLR.

[2] Reddi, S., et al. (2021). Adaptive federated optimization. In ICLR.

The meanings of convergence check and some of the previously confused terms have been clarified after the rebuttal. I hope the authors will revise the paper to better explain the definitions provided in the comments. Specifically, I expect the convergence check and additional measurements to be included in the appendix or elsewhere. Believing that these points will be addressed, I have decided to increase my score.

I truly appreciate your insightful work on this topic. There are also other works that follow a somewhat similar recursive approach, which might complement and enrich the literature review. Although not focused on federated continual learning, works such as [1] and [2] adopt a closed-form solution with pre-trained CLIP, which could provide additional context to your discussion. It might be helpful to include these in the camera-ready version to complete enrich the literature review.

Thank you again for your thoughtful work on this topic.

[1] ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. NeurIPS 2022.

[2] Advancing Cross-domain Discriminability in Continual Learning of Vison-Language Models. NeurIPS 2024