Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models

Dear Reviewer CCAN:

Thank you very much for your recognition of our work. Below I will provide detailed responses to your questions. Thank you for taking your valuable time to offer suggestions for our work.

Q1:Is the proposed framework orthogonal to other LoRA training methods? (Questions For Authors)

A1: Yes, our method can be divided into two components, applied to both client and server parts. On the client side, we select important parameters based on generalization importance and special importance, and perform sparse updates on these important parameters. This is compatible with existing LoRA training methods since this process doesn't affect the training logic but simply adds a masking module. On the server side, we split and aggregate LoRA parameters, which is equivalent to first splitting LoRA into two parameter matrices on the server, applying the same training logic to both matrices separately, and then aggregating them back into a single LoRA parameter matrix at the end. Therefore, it doesn't interfere with the LoRA training process either, as it can be considered as only modifying the input/output pipeline stage of the server's LoRA training. In conclusion, our method is orthogonal to other methods for training LoRA.

Q2: Have the authors explored the communication efficiency benefits this might provide? (Other Strengths And Weaknesses & Questions For Authors)

A2: Thank you for your suggestions. Regarding FedICU, since LoRA splitting and merging can be performed locally on either the client or server, it doesn't introduce additional communication overhead compared to standard federated learning. For the Importance-Aware Updating component, we'll analyze it mathematically. Assuming there are $K$ clients, each with $N$ parameters, and in the mask built by the parameter importance selection upload component, the proportion that needs to be uploaded is $\alpha$. So the communication cost of standard federated learning would be $C_{std} = O(K * N)$ , while the communication cost of our method is $C_{imp} = O\left(\sum_{k=1}^{K} N \times \alpha_k\right), \text{ where } \alpha_k < 1$, which demonstrates the communication savings of our method.

Dear Reviewer UVcT:

Thank you very much for your time and review suggestions on our paper. We hope the following responses can address your concerns.

Q1：The claim regarding the relationship between magnitude-direction and consensus-divergence does not appear to be particularly strong. (Claims And Evidence)

A1: During model training, parameters are optimized toward both global and local optima, and direct aggregation leads to decreased general ability. We decompose parameters into consensus and divergence, where the former represents similar update behaviors across clients, while the latter represents domain-specific update characteristics. We find that during training, magnitude (consensus) updates remain relatively consistent, while directional (divergence) updates differ significantly. This shows that clients maintain general capabilities through consensus parameters and domain-specific capabilities through divergence parameters across downstream tasks. To further support our claim, we conduct an experiment with results shown in the table below. We find that consensus are similar across clients, while divergence differ more significantly, supporting our argument. We will add this experiment to revised manuscript to further substantiate our claim.

Table: Similarity of consensus matrices.

Table: Similarity of divergence matrices.

Table: Similarity of unsplit matrices.

Q2: Notation Confusion (Methods And Evaluation Criteria)

A2: We apologize for the inconsistent notation.

• In Eq.1, $\Delta\theta \in \mathbb{R}^{d\times k}$.
• For the others, the dimension of LoRA should be corrected to $d$ x $k$. In Eq.9 part, we introduce $A$ and $B$ as its decomposition matrices, with dimensions $d$ x $r$ and $r$ x $k$ respectively.

Q3: Assumption in Eq. 7, 8 is not fully supported by Figure 3. (Methods And Evaluation Criteria)

A3: In Figure 3, we use cosine similarity to support the assumption that "consensus noise" is the same. To further support it, we conduct an experiment, as shown in the table below.

Table: the $\mu$ and $\theta$ of consensus parameters' update. $\mu$ represents the mean of the consensus vector updates, and $\theta$ represents the fluctuations during the update process.

From the table, the similarity of $\mu$ and $\theta$ across different clients indicates that the consensus noise is consistent.

Q4：Need experiments to strengthen the claims in different aspects. (Experimental Designs Or Analyses)

A4: Thank you for your advice. We discuss our experiment and conduct more to further support our claims below.

• Preservation of General Knowledge: Table 1 in paper details our model's performance. We measure FedICU on MT-Bench[1] to test general capabilities. The optimal performance of ours demonstrates that it can preserve the model's general knowledge to a certain extent.
• Effectiveness of Sparse Update: According to related research [2, 3], sparse updates can help prevent catastrophic forgetting. Additionally, Table 2 in our paper shows that the model improves when including Sparse Update, proving the effectiveness.
• Convergence Improvement: We conduct an experiment measuring the rounds to reach average loss, with the table below. We can find that FedICU optimizes convergence.
• Handling Client Drift & Case Studies: In Table 1 of our paper, we test FedICU with other methods across various domains. FedICU achieves optimal average domain knowledge, meaning it effectively handles domain knowledge without excessive bias.

Table: Convergence rounds of FedICU and other methods.

We will add this discussion and experiment in revised manuscript to support our claims.

[1] Openfedllm: Training large language models on decentralized private data via federated learning. KDD 2024.
[2] A simple and effective pruning approach for large language models. arXiv 2023.
[3] LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models. arXiv 2025.

Q5: A key concern is whether FedICU could unintentionally expose user data to the server. (Other Comments Or Suggestions)

A5: FedICU follows FL protocols by only sharing parameter updates, not raw data. With Importance-Aware Update mechanism, FedICU can further reduce exposure risk by filtering parameters. Additionally FedICU follows standard FL logic and is compatible with other privacy-enhancing solutions.

Dear Reviewer fzFX:

Thank you for your valuable review of our paper. We hope our responses will address your concerns and improve our score.

Q1: Clarify how inactive parameter contribute to catastrophic forgetting. (Claims And Evidence)

A1: We apologize for the confusion in our original statement. In FedICU, we refer to inactive parameters, not inactive clients. In neural networks, only a small portion of parameters play crucial roles. To prevent clients focusing solely on local datasets and uploading all parameters that cause forgetting, FedICU calculates parameter importance and sparsely uploads parameters to balance the global model's general and domain capabilities. In the example, FedICU will upload parameters with more importance during training, balancing contributions from both clients. We conduct an experiment with results shown in the table below, demonstrating that FedICU is more stable than baseline methods and selects a substantial number of parameters from Client 1 for uploading rather than ignoring them.

Table: The accuracy of FedAvg and FedICU on CIFAR-10.

Table: The selection ratio in FedICU on CIFAR-10. The ratio means the proportion of parameters for upload. It shows that FedICU will not ignore Client 1 for the similarity with the global model.

Q2: The physical meaning of Eq.14, 16 is not intuitive. (Methods And Evaluation Criteria)

A2: The internal logic of Eq. 14, 16 is that we define parameter importance based on the magnitude of updates[1, 2]. To compare client and global, we perform normalization and introduce momentum to make it stable, deriving Eq.14, 16. We add an experiment to show the distribution of importance parameters, with results shown in the table below.

Table: Parameter distribution in FL LLM. Overlap indicates important parameters globally and locally, High-I represents parameters emphasizing general ability, and High-G represents parameters emphasizing domain capability.

The results show definition's physical significance. High-I remains stable while decreasing High-G proportions indicate better identification of core domain parameters. Declining overlap suggests clearer functional partitioning, effectively separating general and domain-specific knowledge.

Q3: The justification for Eq.17, 18 should be reinforced to strengthen validity. (Methods And Evaluation Criteria)

A3: Sparsely updating model is an approach to mitigate catastrophic forgetting [2, 3]. In FedICU, We construct masks and sparsely update models (Eq.17, 18) to mitigate forgetting. To further demonstrate the effectiveness, we conduct an experiment as shown in the table below. For Momentum, we test whether to incorporate momentum components, and for Smooth, whether to use binary or smooth updates based on the formula below. Results show our method achieves the best performance, demonstrating its effectiveness.

Formula: Smooth updates component.

W[v] = \\begin{cases} \\min{(1, \frac{G[v]}{I[v] + G[v]})} & \\text{if } G[v] > I[v] \\\\ 0 & \\text{other} \\end{cases}

Table: Experiment results of ablation study. MT-1 shows the model's general capability, while MT-2 shows the model's level of contextual understanding. MT-Final is the metric combining both of the two indicators.

[1]Learning both weights and connections for efficient neural network.NeurIPS, 2015.
[2]A simple and effective pruning approach for large language models.arXiv 2023.
[3]Finding sparse, trainable neural networks.arXiv 2018.

Q4: Some essential references are not discussed. (Essential References Not Discussed)

A4: Thank you for suggesting additional references. We will incorporate these into the revised manuscript.

Q5: Some inconsistencies in the notation. (Other Comments Or Suggestions)

A5: Thank you for pointing out the unclear expressions.

• For the second, the vectors are defined in Lines 252-261. Specifically, we perform vector-wise aggregation by processing $A_k^i$ column-by-column to obtain $r$-dimensional vectors $v$, while for $B_k^i$, we process it row-by-row.
• For the third, $j \in \{1, 2, ..., n\}$.
• For the fifth, we use model gradients for mean and variance.

We will correct all of them in the revised manuscript.

Dear Reviewer UEdC:

Thank you very much for your recognition of our paper and detailed suggestions. We will answer and explain the issues in detail below.

Q1: The method relies on LoRA, which may not achieve the same performance as full parameter fine-tuning. (Other Strengths And Weaknesses)

A1: Thank you very much for your suggestion. Using LoRA is a common method for fine-tuning large language models today, providing significant performance improvements at relatively low cost. In the future, we will further explore how to use LoRA more efficiently to achieve the ability to fine-tune nearly all parameters.

Q2: Consider using a more flexible thresholding method in the Importance-Aware Updating mechanism. (Questions For Authors)

A2: Thank you very much for your suggestions. We conduct a supplementary ablation study to investigate the impact of momentum-based parameter selection and continuous-valued parameter weighting on model generalization capability, as shown in the table below. In the Momentum column, we indicate whether we include the momentum component to smooth the mask construction process. In the Smooth column, we use a smoothing mask approach based on the formula below to filter uploaded parameters and their weights. The results validate that our binary component is a simple and effective method. We will add this supplementary experiment to the appendix of our revised manuscript to enhance the description of the effectiveness of our component.

Formula: Smooth updates component.

W[v] = \\begin{cases} \\min{(1, \\frac{G[v]}{I[v] + G[v]})} & \\text{if } G[v] > I[v] \\\\ 0 & \\text{other} \\end{cases}

Table: Experiment results of ablation study about Importance-Aware Update.

Q3: How do you expect the performance of FedICU to scale when applied to larger models or different model architectures? (Questions For Authors & Other Strengths And Weaknesses)

A3: We greatly appreciate your suggestion. We've conducted supplementary experiments with Mistral-7B to verify that our method is effective across a broader range of model architectures. The results of these supplementary experiments are as follows. We will add this experiment to our revised manuscript to further support our approach.

Table: The performance of FedICU and other methods applied in the Mistral-7B. MT-1 shows the model's general capability, while MT-2 shows the model's level of contextual understanding. MT-Final serves as a comprehensive metric that combines both of the aforementioned indicators. The results show that FedICU also demonstrates excellent performance across models with different architectures.