Thank you for your thoughtful feedback and for recognizing the strengths of our work, including the theory, simplicity, and thorough experiments. We appreciate your positive assessment.

W1: Need for Clearer Distinction from Prior Alternating Optimization Methods

While conceptually inspired by classic methods, to the best of our knowledge, no existing algorithm both alternates the optimization of components and alternates the aggregation of the corresponding components. This alternating strategy addresses the inexact aggregation issue unique to federated LoRA, as discussed in Section 2.3. RoLoRA differs in both its specific algorithmic design and its application focus, as discussed in Appendix A2.

First, we distinguish our work from the field of matrix sensing and matrix completion. Unlike traditional methods [1,7] that assume centralized data, RoLoRA is explicitly designed for the decentralized federated learning setting. More importantly, our key algorithmic difference lies in the aggregation strategy itself. Prior work in federated matrix factorization [2] performs alternating updates only locally within each client. In contrast, RoLoRA introduces an alternating aggregation scheme at the server level, where the up- and down-projection matrices (A and B) are aggregated across communication rounds. Our algorithm is critical for mitigating destructive interference between the LoRA components, a unique challenge in this federated LoRA context.

Second, while RoLoRA shares a similar derivation framework with Multi-Task Representation Learning (MLRL) methods such as FedRep [3] and Vaswani et al. [4], it differs fundamentally in objective. MLRL aims to learn personalized models by keeping final layers diverse and unaggregated. In contrast, RoLoRA is designed to train a single, high-performing global model. This difference is reflected in Algorithm 2 (line 7) and in the theoretical analysis (Eq. 35-40), where RoLoRA's global aggregation on $b$ is captured through a union bound over each $b_i$, followed by transforming the bound to their average. Moreover, the architectural roles of components differ: in FedRep, the local head acts as a down-projection, while in RoLoRA—built on the LoRA framework—the matrix $B$ acts as an up-projection, mapping the adaptation back to the original feature space.

Finally, we differentiate RoLoRA from personalized federated learning algorithms that also use an alternating structure. These methods [5,6] are inherently designed to create separate parameters for each client, whereas RoLoRA operates entirely within the global model paradigm. This distinction leads to different motivations and theoretical approaches. Our alternating scheme is motivated by the structural decomposition of LoRA matrices, not the separation of personal and global parameters. Consequently, our proof technique is entirely novel in this context, drawing from matrix sensing theory to analyze the learning process, in stark contrast to the standard FL convergence analysis used in personalized FL literature.

[1]Jain et al., "Low-rank Matrix Completion using Alternating Minimization", STOC13

[2]Wang et al. "Federated Matrix Factorization: Algorithm Design and Application to Data Clustering", TIP22

[3]Collins et al. "Exploiting Shared Representations for Personalized Federated Learning." ICML21

[4]Vaswani et al. "Efficient federated low rank matrix recovery via alternating gd and minimization: A simple proof", TIT24

[5]Mishchenko et al. "Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity", TMLR25

[6]Pillutla et al. "Federated Learning with Partial Model Personalization", ICML22

[7]Ward et al. "Convergence of Alternating Gradient Descent for Matrix Factorization", NeurIPS23

We sincerely thank the reviewer for their thorough, thoughtful, and insightful feedback. We'll address each point below.

Q1: Client Number Selection

Our choice of 3, 20, and 50 clients was intended to comprehensively showcase RoLoRA's performance across different scales of FL scenarios, and specifically to highlight RoLoRA's advantage in achieving both faster convergence (Fig.3) and higher final accuracy (Table 1) in large-scale settings. To provide a smoother transition and further strengthen our conclusions, we have now added experimental results for 10-client setting with rank-4 adapter. The results show that RoLoRA still outperform other methods. In the 10-client setting, RoLoRA's performance gain over other methods falls between the gains observed in the 3-client and 20-client settings.

Q2: Asymmetry in LoRA and Basis Adaptation Strategy

We thank the reviewer for their deep and insightful question regarding the symmetric treatment of matrices A and B in RoLoRA. To empirically investigate whether an asymmetric update schedule might be preferred, we have conducted new experiments comparing our standard approach to two alternative strategies in a 50-client setting using rank-4 adapter on the QQP. We consider strategy 1 that prioritizes updating B, updating 3 times for every one-round update of A (e.g., B, B, B, A, ...), and strategy 2 that prioritizes updating the basis matrix A(e.g., B, A, A, A, ...). While A and B serve inherently asymmetric roles, we observe that aggressively updating either matrix without sufficiently adapting the other leads to misalignment. Furthermore, we considered even more fine-grained approaches to handle this asymmetry, such as assigning different learning rates for A and B. We adapt LoRA+'s strategy [4] into federated setting by using a larger learning rate for B. Strategy 3 and 4 corresond to $lr_B=2lr_A$ and $lr_B=4lr_A$, respectively. Our experimental results showed that a symmetric approach with equal learning rates still yielded the best performance.

It appears that a balanced, symmetric alternation between the matrices A and B yields the most stable and effective optimization trajectory. RoLoRA's design provides this balance and provides a robust strategy to update A and B matrices. We will add a remark clarifying this in the paper.

Q3: Impact of Rank on RoLoRA vs. FFA-LoRA

To investigate the reviewer's hypothesis, we've conducted a new set of experiments in a centralized setting as requested. We evaluated both methods on the MNLI and QQP using 8 LoRA adapters attached to query and value projection of last 4 layers, and trained for 50000 iterations to ensure full convergence. We note that when the rank increases from $1$ to $64$ the gap between the two schemes decreases as suggested by the reviewer. Another related technique to narrow the performance gap between the two schemes is by increasing the number of adapters, as discussed in Section 5.1 (“Effect of Number of Fine-Tuning Parameters”). With sufficient adapters, FFA-LoRA can achieve comparable peak accuracy to RoLoRA. However, in federated settings where resources are constraiend, RoLoRA is more advantageous.

Limitation: Theoretical Analysis for Autoregressive LLMs

Our theoretical analysis deliberately focuses on simplified models to provide foundational insights. The linear model provides a tractable case where different methods can be directly compared and their limitations can be formally established. We then emperically demonstrate that similar findings also apply in more complex neural networks and large langugage models. This approach aligns with standard practice in federated learning, where simplified yet representative linear models are used to isolate and understand the core mechanics of new methods [1,2]. Within this setting, we formally prove the mechanisms behind RoLoRA's improved performance and also provide a convergence analysis in non-convex loss landscapes.

It is also important to note that this simplification is common across the LoRA literature. To the best of our knowledge, no existing work on LoRA or its variants has provided a theoretical analysis that fully captures the autoregressive architecture of modern decoder-only LLMs. Prior theoretical analysis are typically limited to linear regression settings [4,6] or rely on assumptions such as rank-one shift, rank-one LoRA, or frozen B[7]. Our model is thus aligned with the current state of research [3,4,5,6], while addressing a more complex federated setting.

We view our extensive empirical evaluations on LLMs as the primary validation of our claims. The strong performance gains demonstrated in these experiments directly align with the predictions of our theory, showing that the foundational insights successfully transfer from the theoretical model to complex, practical applications.

[1]Collins, Liam, et al. "Exploiting shared representations for personalized federated learning." ICML21

[2]Collins, Liam, et al. "Fedavg with fine tuning: Local updates lead to representation learning." NeurIPS22

[3]Zhu, Jiacheng, et al. "Asymmetry in Low-Rank Adapters of Foundation Models." ICML24

[4]Hayou, Soufiane et al. "LoRA+: Efficient Low Rank Adaptation of Large Models." ICML24

[5]Zhang, Yuanhe et al. "LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently." ICML25

[6]Zhang, Fangzhao et al. "Riemannian preconditioned LoRA for Fine-Tuning Foundation Models." ICML24

[7]Dayi, Arif Kerem et al. "Gradient dynamics for low-rank fine-tuning beyond kernels."

Revisions to Paper Formatting We thank the reviewer for their careful reading and helpful suggestions for improving the paper's formatting and clarity. We will address both points in the revision. For clarity in line 30, we will revise it to succinctly state the well-known issue of applying FedAvg to LoRA and directly transition to the methods that address it.

We are deeply grateful to the reviewer for their thoughtful and encouraging reviews. Below, we address your insightful questions.

Q1: Effect of local steps

To provide a clear, empirical answer, we have conducted an ablation study on the number of local steps in a 50-client setting with rank-4 adapter. For a fair comparison, we kept the total computational budget (Local Steps * Total Rounds) constant across all settings. FFA-LoRA's performance consistently degrades on both datasets as the number of local steps increases. This indicates that with more local work per round, FFA-LoRA suffers severely from client drift, where local models overfit to their own data. RoLoRA maintains high performance across all settings.

Q2: FedAvg+LoRA Performance and Data Heterogeneity We believe the relatively good performance of FedAvg+LoRA in Table 2 is partially due to the small number of clients (10). Our findings suggest that the performance of FedAvg+LoRA is more sensitive to the number of clients than to data heterogeneity alone. With fewer clients, the model aggregation becomes more accurate. As the number of clients increases, aggregation inaccuracy becomes more pronounced.

While RoLoRA, like all federated methods, is inevitably affected by data heterogeneity, it demonstrates greater robustness to client drift induced by non-IID data distributions. Evidence for this can be found in our ablation study on the number of local epochs, where RoLoRA maintains competitive performance under increased local update steps, which is known to exacerbate drift. By contrast, FFA-LoRA exhibits a sharper performance degradation in non-IID settings, aligned with our observation in the same ablation study. Further comparisons between i.i.d. and non-i.i.d. scenarios in the 10-client setting can be drawn from Table 2 and the additional experiments provided in response to Reviewer jkws under “Client Number Selection.” These observations highlight RoLoRA's improved resilience to heterogeneity, particularly in scenarios with significant local deviation.

Limitation: privacy protection for user data

Thank you for this important and constructive suggestion. While RoLoRA was not explicitly designed with privacy mechanisms such as differential privacy, we recognize that its ability to mitigate inexactness through alternating optimization and aggregation may help mitigate a key challenge for DP-aware federated learning, where inexact model updates can be particularly problematic. We itegrated NbAFL [1] with $\epsilon=10,\delta=1e-6$ in a 3-client setting. RoLoRA outperform others across two tasks.

[1]Wei, Kang, et al. "Federated learning with differential privacy: Algorithms and performance analysis." TIFS2020

We sincerely thank the reviewer for their time and effort. We are grateful for the insightful questions, which we address below.

Q1: Clarification on Communication Rounds vs. Communication Cost

Thank you for this question. It allows us to clarify a key contribution of our work: RoLoRA's ability to achieve superior performance at a significantly reduced communication cost. Our emphasis on using the same number of communication rounds is intended to ensure a fair comparison within a typical federated learning setting. First, we note that in our paper, unless otherwise specified (as in Figure 11 in the appendix), the communication cost per round is by default different across the methods being compared, since RoLoRA and FFA-LoRA only need to transmit a single LoRA component in each round due to their freezing nature. In contrast, standard LoRA and FlexLoRA must transmit both LoRA components (A and B). This means that for a given rank, RoLoRA has half the communication cost per round compared to LoRA and FlexLoRA. Second, to make the comparison even more direct, in Figure 11 (Appendix), we compare all methods under an identical total communication budget (i.e., the same total megabytes transferred). To achieve this, we configured RoLoRA and FFA-LoRA with twice the number of LoRA adapters so their total communication cost matches that of FedAvg with LoRA. The results show that when the communication cost is equalized, RoLoRA can converge to a better accuracy.

Q2: Rank-32 results

We chose rank-4 and rank-8 in Table 1 to reflect practical scenarios in federated settings, where communication and memory constraints on edge devices often necessitate smaller ranks.

To address the reviewer's concern, we have conducted additional experiments with rank-32 adapter in the 20-client and 50-client setting. The results, averaged over three runs, are aligned with our previous observations: RoLoRA consistently outperforms others in the setting with large number of clients at higher ranks.

We will include the rank-32 results in the revision.

Q3: Error Band in Fig.3

The observed variance arises from a combination of training dynamics and task difficulty. As shown in Fig.3 (MNLI), none of the methods have fully converged—RoLoRA is nearing convergence, while LoRA and FFA-LoRA are still early in training. High variance across runs is expected in the intermediate phase due to different optimization paths from random initialization. This effect is also evident in SST-2 and QNLI.

Additionally, variance magnitude depends on task difficulty. MNLI's complexity leads to more diverse training trajectories and higher variance, whereas simpler tasks like SST-2, QNLI, or BoolQ show more stable results. We consistently observe greater variance on harder tasks like PIQA and SIQA.