Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning

We appreciate your insightful comments!

W1: Use of LoRA B for Clustering vs. Prior FL Clustering Approaches

While FedSA also recognizes the distinct roles of LoRA $A$ and $B$ matrices, its method primarily uses this insight for selective aggregation—aggregating only the $A$ matrices while keeping $B$ matrices local to preserve client-specific information.

In contrast, our approach uniquely leverages the LoRA $B$ matrices for clustering clients into distinct groups, a fundamentally different purpose from FedSA. We apply hierarchical clustering on the $B$ matrices to capture task similarity and dynamically allocate personalized LoRA experts, which enables expert-aware routing and fine-tuning in federated settings.

Beyond differentiating from FedSA, our method also offers clear advantages over traditional FL clustering methods like IFCA. Traditional clustering FL methods clusters clients using full model updates or losses, which incurs significant computational and communication cost. In contrast, FedLEASE uses only the small LoRA $B$ matrices, which are significantly more lightweight yet sufficient for capturing task-level differences. Moreover, our method couples this clustering with an adaptive top-$M$ expert selection mechanism, allowing sample-level flexibility across experts—something IFCA lacks.

As shown in our empirical results (§3.3), clustering based solely on LoRA $B$ matrices yields comparable result with LoRA $BA$-based clustering, with far less overhead. Our experiments further show that FedLEASE outperforms IFCA+LoRA by a clear margin in both performance and efficiency.

Therefore, while prior works have considered either clustering (e.g., IFCA) or LoRA-specific aggregation (e.g., FedSA), we think FedLEASE is the first to both address expert allocation and utilization challenges by LoRA $B$-based clustering with flexible top-$M$ expert selection, providing a novel and effective solution for federated LoRA fine-tuning..

W2: Justification for Task-Heterogeneous Setting

While we acknowledge that traditional federated learning often adopts Dirichlet partitioning to simulate label non-IID settings, our task-heterogeneous setup is explicitly designed to better reflect real-world scenarios of federated fine-tuning for large language models.

In practical deployments, clients typically possess data from fundamentally different domains and tasks, rather than working on the same task with skewed labels. For example:

• Customer service platforms with dialogue transcripts
• Educational institutions with question-answering datasets
• Medical research labs with clinical trial summaries
• Legal departments with contract summarization tasks
• E-commerce companies with product review classification

Such task-level heterogeneity introduces more challenging but realistic federated conditions compared to simple label imbalance, and has also been adopted in prior work such as FedDPA (NeurIPS 2024).

In our experiments, we emulate this realistic setup by using benchmarks such as:

• GLUE, which combines diverse NLU tasks including sentiment analysis (SST-2), natural language inference (QNLI), paraphrase detection (QQP), and textual similarity (MRPC);
• and the Flan Collection, which aggregates NLG tasks across question answering, summarization, reasoning, and more.

These benchmarks are not a single dataset with label skew, but rather collections of distinct tasks, validating our claim of task heterogeneity in both motivation and implementation.

Moreover, we also conducted evaluations of our method under label non-IID conditions, with results presented in Figure 9 and Table 7. The findings demonstrate that FedLEASE maintains superior performance compared to all baseline methods, even when operating under these data distribution imbalances.

W3&W4: Implement details and citation format

We thank the reviewer for these helpful suggestions. We agree that placing critical implementation details in the main paper will enhance readability and reproducibility, and we will reorganize accordingly in the final version. Additionally, we acknowledge that certain citations, such as FedDPA, should reference their official published versions. We will correct and standardize all citations in the final manuscript to ensure accuracy and clarity.

Thank you for your meaningful suggestions!

W1: System Overheads
Regarding the training and clustering overhead, the clustering step is performed only once during the initialization phase and is not repeated during iterative training. Thus, its runtime impact is negligible. As described in §3.3, using only the LoRA B matrices provides an efficient and lightweight proxy for task similarity, due to their significantly smaller size compared to full model weights or LoRA A–B combinations.

To further support this, we measured the clustering time as 3.11 seconds on an Intel Xeon Platinum 8570 CPU, which is substantially shorter than the total training time (193.49 seconds with local training on an NVIDIA B200 GPU). The detailed timing comparisons are provided below:

We also add a table summarizing convergence speed, demonstrating that FedLEASE achieves superior accuracy under the same round.

Accuracy Performance Across Training Rounds (%)

W2: Adaptivity and Sensitivity of the Cluster Count ($M$)
We clarify that $M$ is not manually fixed at first in FedLEASE. Instead, we only set an upper bound $M_{\text{max}}$ to limit computational and communication costs. The actual number of clusters $M$ is determined automatically during the initialization phase through hierarchical clustering with the silhouette score as the evaluation metric (see §4.1). This procedure allows $M$ to adapt to the underlying client distribution and similarity structure without manual tuning.

Our sensitivity analysis in Table 8 (Appendix C.2) demonstrates robustness across different $M_{\text{max}}$ values: when $M_{\text{max}}=2,3,4,8$, our method achieves 85.49%, 87.21%, 87.85%, and 87.76% average performance respectively, with the silhouette-based selection consistently identifying the optimal $M=4$ for our 16-client, 4-task setting.

These results highlight two important observations:

• The performance degradation when $M_{\text{max}} < M$ (85.49% vs 87.76%) confirms the importance of adequate expert budget. Notably, even under limited expert capacity, our method still outperforms all baselines, showcasing its effectiveness in constrained scenarios.
• Comparable performance between $M_{\text{max}} = 4$ and $M_{\text{max}} = 8$ validates our method’s ability to avoid over-clustering, even when given excess capacity.

We are grateful for the reviewer's constructive suggestions and hope our explanations resolve the raised issues.

Thank you for your valuable comments!

W1: LoRA B clustering computational overhead.
The clustering step is performed only once during the initialization phase and is not repeated during iterative training. Thus, its runtime impact is negligible. As observed in §3.3, using only the LoRA B matrices offers an efficient and lightweight proxy for task similarity, given their small size compared to full model weights or BA products.
To further support this, we measured the clustering time (3.11 seconds on Intel Xeon Platinum 8570 CPU), which is significantly shorter than the total training time (193.49 seconds with local training on the NVIDIA B200 GPU):

W2: Alternative clustering methods are not explored.
While clustering is indeed an essential component of our proposed method—playing a key role in expert allocation based on client similarity—the specific choice of clustering algorithm is not the focus of our contribution. Our observations (§3.3) suggest that as long as the method captures pairwise similarity between clients (e.g., via LoRA B matrices), the overall performance is relatively robust to the particular clustering strategy.

We adopt Agglomerative Hierarchical Clustering due to its ability to operate directly on pairwise distances without requiring pre-defined centroids. To validate the generality of our approach, we also applied Spectral Clustering, which similarly supports pairwise similarity inputs, and observed comparable performance (within 0.14% accuracy difference).

The tables below demonstrate that both clustering methods achieve similar silhouette scores and downstream performance, reinforcing that our performance gains stem primarily from the expert allocation and adaptive top-$M$ selection mechanisms, rather than the specific clustering algorithm used.

Silhouette Scores:

Final Accuracy Comparison:

W3: Comparison of computational cost and training time.
As discussed in our response to W1, FedLEASE introduces only minimal additional overhead. The clustering step is performed only once during initialization, taking only 3.11 seconds on CPU, which is negligible compared to the total training time (193.49 seconds with local training on GPU, as shown in the above Table in W1).

During training, each client updates only its assigned LoRA expert and router, keeping the per-round computation costs on par with baselines such as FedIT and FedSA. Importantly, in both GLUE and Flan experiments (Table 1 and Table 3 in the main paper), we ensured that the total number of trainable parameters in FedLEASE is comparable to or even smaller than those of baseline methods. Despite this, our method consistently achieves superior performance, validating the efficiency of our expert allocation and adaptive top-$M$ design.

Further, ablation studies (Table 2 and Figure 6) confirm that the adaptive top-$M$ mechanism provides substantial performance gains without increasing training workload, demonstrating both the computational efficiency and effectiveness of our proposed approach.

W4: Clarification on router reinitialization.
Thank you for pointing this out. The router is indeed maintained locally by each client throughout training. The statement in Algorithm 1 suggesting per-round reinitialization was a typo, in fact clients initialize their router only once at the beginning.

We will correct this in the final version of the paper. Additionally, our ablation study in Figure 6(a) explicitly demonstrates that maintaining the router locally leads to better performance, supporting our design choice.

W5: Does the optimal number of experts vary with number of clients?
In the experiments where the number of clients varies, the underlying data distribution remains fixed across four distinct datasets (SST-2, QNLI, MRPC, QQP). As a result, our clustering procedure consistently identifies 4 clusters, which aligns with the task/domain-level heterogeneity rather than the client count.

Importantly, the clustering is based on pairwise similarity of LoRA B matrices, a core component of our method motivated by prior observations that LoRA B captures task-specific information (§3.3). Overall, the number of experts is determined by the intrinsic task-level differences across clients. If additional tasks or datasets are introduced, our clustering procedure would naturally detect a different number of clusters as needed.

W6: Frequency and distribution of expert selection.
We provide additional analysis showing normalized expert selection percentage at different layers for each client. This allows us to capture relative expert usage, especially when clients assign multiple experts.

Our findings reveal several important patterns that aligns with Figure 6(c) in our paper:

• Deeper layers (e.g., Layer 23) show strong reliance on the expert corresponding to the client’s assigned cluster, indicating more specialized and task-specific behavior as depth increases.
• Shallower layers (e.g., Layer 0) tend to involve a more diverse set of experts, reflecting greater need for knowledge transfer across domains in earlier representations.
• Despite local updates of the router networks clients within the same cluster show highly similar expert selection patterns, validating the effectiveness of the initial clustering based on LoRA B similarity.

Thus we can find that the number of experts selected varies across layers—shallow layers often draw from more experts than deeper ones. This confirms the flexibility and effectiveness of our adaptive expert routing mechanism. Unlike traditional fixed top-$k$ MoE strategies—which constrain every data sample, every layer to a fixed number of experts—our method enables fine-grained adaptivity across samples and layers, offering a much more flexable approach to expert utilization in federated fine-tuning.

We thank the reviewer for the valuable comments and will incorporate these clarifications and additions in the final version.

Thank you for your thoughtful comments!

W1: Clarification on "fixed top-$k$ MoE" vs. our proposed "adaptive top-$M$ MoE".
To clarify, the term "fixed top-$k$ MoE" in our paper does not refer to the number of clusters or LoRA experts being $k$. Instead, it follows the conventional MoE paradigm where, given $M$ experts, each input is routed to the top-$k$ experts based on routing scores (i.e., the same fixed number $k$ is used for all data across all clients.)

In contrast, our proposed ''adaptive top-$M$ MoE'' introduces a more flexible and input-adaptive routing mechanism. Each client is first assigned to a primary expert via clustering. Then, the client’s router network $G_i \in \mathbb{R}^{(2M-1) \times d}$ produces $2M - 1$ routing scores:

• The first $M$ scores correspond to distinct internal components of the assigned expert, i.e., they are all routed to the same expert but model different aspects (not different experts).
• The remaining $M - 1$ scores are associated with the other experts.

These $2M - 1$ values are learned via a router network $G_i \in \mathbb{R}^{(2M-1) \times d}$, and selection is conducted via softmax followed by a top-$M$ selection.

To illustrate with an example where $M = 3$ with experts $[E1, E2, E3]$ and $E1$ is the assigned expert, the router may output scores like:
$[ω₁^{E1}, ω₂^{E1}, ω₃^{E1}, ω₁^{E2}, ω₁^{E3}]$

Here:

• $ω₁^{E1}$, $ω₂^{E1}$, and $ω₃^{E1}$ are three independently learned scores for the assigned expert E1, capturing different aspects of its contribution.
• $ω₁^{E2}$ and $ω₁^{E3}$ correspond to the other experts.

Depending on the input samples, the output scores can change and the top-M selection might yield:

• Sample 1: ${ω₁^{E1}, ω₂^{E1}, ω₁^{E2}}$ are the top-$M$ highest scores → 2 unique experts $E1, E2$ are selected
• Sample 2: ${ω₁^{E1}, ω₂^{E1}, ω₃^{E1}}$ are the top-$M$ highest scores → only 1 unique expert $E1$ is selected
• Sample 3: ${ω₂^{E1}, ω₁^{E2}, ω₁^{E3}}$ are the top-$M$ highest scores → all 3 experts are selected

This design has two main benefits:

1. It guarantees participation of the assigned expert for every client.
2. It allows fine-grained and input-dependent expert allocation, supporting different levels of specialization across inputs and across layers.

We appreciate the reviewer’s comment and acknowledge that the explanation in the current version could be confusing. In the final version, we will revise the description of this mechanism to make it explicitly clear.

W2: Choice of MoE + FL baselines.
We appreciate the reviewer’s concern regarding baseline selection. To our knowledge, there is currently no prior work that integrates MoE with federated LoRA fine-tuning. Given this gap, we selected recent state-of-the-art FL + LoRA methods for comparison, including:

• FedIT (ICASSP 2024), which uses single global LoRA
• FFA-LoRA (ICLR 2024), which freezes LoRA A to reduce server aggregation noise
• FedSA (ICLR 2025), which applies selective aggregation
• FedDPA (NeurIPS 2024), which introduces dual-personalized adapters

These methods are all published in top-tier conferences. Besides of these FL + LoRA methods, we also modified a tradtional clustered FL method (i.e. IFCA) with LoRA as an additional baseline. Our goal is to go beyond these works by addressing both (1) expert allocation (via clustering), and (2) expert utilization (via adaptive top-$M$ routing).

To further isolate and demonstrate the benefit of our adaptive top-$M$ MoE, we conducted direct comparisons in the “Ablation on Adaptive top-M Mechanism” section (see Figure 6(b)), where we compare our full method against a variant that uses fixed top-k MoE + adaptive expert allocation. This ablation serves as the closest proxy to a potential MoE + FL baseline, and results show that our adaptive routing mechanism consistently outperforms all fixed-k alternatives, regardless of the value of $k$.

Additionally, Table 2 in the main paper provides a clear breakdown of performance contributions from clustering alone (i.e., expert allocation).

W3: Runtime and clustering computational overhead.
Regarding the training and clustering overhead, the clustering step is performed only once during the initialization phase and is not repeated during iterative training. Thus, its runtime impact is negligible. As described in §3.3, using only the LoRA B matrices provides an efficient and lightweight proxy for task similarity, due to their significantly smaller size compared to full model weights or LoRA A–B combinations.

To further support this, we measured the clustering time as 3.11 seconds on the Intel Xeon Platinum 8570 CPU, which is substantially shorter than the total training time (193.49 seconds with local training on an NVIDIA B200 GPU). The detailed timing comparisons are provided below:

We will include the table in the final version to clearly demonstrate that FedLEASE maintains high performance while incurring only modest runtime overhead.

W4: Dataset partitioning: Task heterogeneity vs. label non-IID.
While we acknowledge that traditional federated learning often uses Dirichlet partitioning to simulate label non-IID settings, our task-heterogeneous experimental setup is designed to more accurately reflect real-world scenarios for large language model fine-tuning and training. In practical deployments, clients often hold data from fundamentally different domains and tasks, such as:

• Customer service platforms with dialogue transcripts
• Educational institutions with exam question-answering datasets
• Medical research labs with clinical trial summaries
• Legal departments with contract summarization tasks
• E-commerce companies with product review classification

Such task-level heterogeneity presents a more challenging and realistic federated fine-tuning setting (similar setting also considered by FedDPA) than simple label skew within the same task.

We also jointly evaluated our method under label non-IID conditions, as shown in Figure 9 and Table 7, where FedLEASE consistently outperforms all baselines even in these skewed scenarios. For these binary classification tasks (e.g., SST-2, MRPC), we followed the non-IID setup used in prior work (FedSA, FFA-LoRA) instead of applying Dirichlet partitioning, which may be less meaningful for such binary classification tasks.

We thank the reviewer for these thoughtful comments and hope our detailed clarification addresses each point clearly. We will include these clarifications and results in the final version of the paper.