Towards Robust Parameter-Efficient Fine-Tuning for Federated Learning

Dear Reviewer Egmb：

Thank you for your thoughtful review and for raising key concerns regarding our work. We hope the following responses will address your concerns and update the score.

W1&Q1: Effectiveness on NLP tasks

To validate the generalizability of RFedLR, we conduct an additional experiment on NLP tasks. Following existing federated PEFT works [1][2][3], we perform an experiment on the MNLI task from the GLUE benchmark. We utilize Roberta-Base as the initial global model. The experimental setup mirrored our vision experiments, with clients holding non-IID data partitions and operating under a 20% symflip label noise. As shown in the Table R1, RFedLR achieves a accuracy of 71.42%, significantly outperforming all baseline methods. The results confirm that our framework is effective for NLP tasks in noisy federated settings.

Table R1: Comparison with state-of-the art methods on MNLI task from GLUE benchmark when the noise rate is 0.2 and noise type is symflip. We take the average test accuracy of the local models for demonstration.

Due to the time constraints of the rebuttal period, these NLP experiments were conducted for one representative task and noise setting. We will provide the full experimental results in the final version.

W2&Q2: Notational Clarity and Consistency

We thank the reviewer for pointing out this ambiguity in our notation. $S_k^i$ in Eq.5 is a scalar representing the sensitivity of a single parameter. In Eq.6, the collection of these sensitivity scores forms a sensitivity tensor $S_k$, which has the same dimensions as its corresponding parameter tensor. Therefore, $M_k$ in Eq.7 is a binary mask tensor with the same shape. We will clarify these symbols and their dimensions in the final version.

W3&Q3: Empirical validation of the AFLA weighting scheme

To provide the empirical evidence for our AFLA weighting scheme, we perform experiments in a controlled environment designed to validate its ability to mitigate noise amplification. We design a federated learning system with three clients. Client A is configured to be large but noisy (12,000 data points, 80% noise), Client B is configured to be small but clean (2,000 data points, clean), and Client C serves as a control (6,000 data points, 20% noise).

As demonstrated in the Table R2, the standard FedAvg approach, which relies solely on data size, naively assigns the highest weight to the noisy Client A. This fixed weighting leads to severe noise amplification from Client A, resulting in a poor final accuracy of only 29.54%. In contrast, our AFLA mechanism considers both importance scores and data size to give more comprehensive weight allocation. It identifies Client A as an unreliable participant and reduces its average aggregation weight to 0.35. Simultaneously, it raises the weight of the small but reliable Client B to 0.32. This dynamic re-weighting mitigate the noise propagation and significantly improves accuracy.

Table R2: Empirical validation of the AFLA weighting scheme in a controlled 3-client scenario. The weights in AFLA are dynamic each round, we show the average weight across all rounds to summarize its effective influence.

W4: Discussion of privacy restrictions

We agree that the reliance on a public proxy dataset can be a limitation. We have transparently acknowledged in the conclusion of our paper (Section 6). However, our method preserves the foundational privacy guarantee of federated learning, i.e., no private client data is ever shared. We follow the published federated learning research [4] in setting up proxy datasets, which can come from small public datasets relevant to the task, or from trusted participants in the federated learning system. In our work, we require only a mini-batch dataset to serve this purpose. Crucially, the proxy dataset does not need to be consistent with the local client data distribution, and the generated noise proxy dataset does not need to match locally specific noise patterns.

Q4: Clarification of Figure 2

We will clarify the caption of Figure 2 to indicate that “Federated LoRA” refers to the baseline FedLR method, where FedAvg is directly applied to LoRA. In addition, the accuracy difference between clean and noisy environments for federated FFT reaches 5.04%. However, the accuracy difference between Federated LoRA (FedLR) in clean and noisy environments reaches 11.48%. The presence of label noise poses a significant barrier to the robust performance of federated LoRA (FedLR). We will add specific values to Figure 2 in the final version.

Q5: Ablation experiments for the importance score

Fisher information (F) is used to approximate the importance of parameters on model decisions. Higher Fisher values indicate a greater influence of a parameter on the model output. The noise sensitivity score (S) measures the sensitivity of parameter updates to label noise. Lower sensitivity scores indicate greater stability and reliability. The formula $I = F / (S + \epsilon)$ aims to prioritize parameters that have a high impact on the task and are robust to label noise.

We supplement this with ablation experiments (Table R3) to analyze the respective contributions of F and S to the final performance. We compare the RFedLR method ("Ours") with two variants: one that removes the Fisher information component ("w/o F," using $I = 1/ (S + \epsilon)$ ) and the other that removes the noise sensitivity component ("w/o S," using $I = F$).

Table R3: Ablation study for the hybrid importance score. The table compares the average test accuracy (%) of RFedLR (Ours) with variants that exclude Fisher information (w/o F) or noise sensitivity (w/o S) from the importance calculation in Eq. (10)

The significant performance drop in the "w/o S" case confirms that accounting for noise sensitivity is crucial for robust aggregation. Using Fisher information alone is insufficient and can be misleading in noisy environments. The full method always outperforms the “w/o F” case, which indicates that Fisher information further refines the importance score.

References

[1] Improving LoRA in Privacy-preserving Federated Learning, in ICLR 2024.

[2] FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models, in ACL 2023.

[3] FedPFT: Federated Proxy Fine-Tuning of Foundation Models, in IJCAI 2024.

[4] Revisiting Weighted Aggregation in Federated Learning with Neural Networks, in ICML 2023.

Dear Reviewer yHga：

Thank you for the valuable comments. We hope our responses address your concerns and provide a clearer understanding of our work.

W1&Q1: Practicality and assumption of the proxy dataset

We acknowledge that requiring a clean proxy dataset is a significant consideration, as we mentioned in Section 6. We follow the proxy dataset settings in published federated learning work [1] to achieve robust federated learning.

In response to Q1, a proxy dataset can be sourced from small, publicly available datasets relevant to the task, or from a trusted, verifiable participant in FL systems. We agree that this is a constraint and will emphasize it in the limitations.

In response to W1, the effectiveness of our method does not require that the distribution or noise pattern of the proxy dataset must be similar to the local data.

1. Robustness to Data Distribution Mismatch: In our setup, the proxy dataset is a uniformly distributed mini-batch randomly sampled from the CIFAR-100 test set. However, the local data was explicitly configured to be non-IID. Especially in the highly data heterogeneous environments (Refer to Q4), our method performs well, proving distribution matching is unnecessary.
2. Robustness to Noise Type Mismatch: The server injects uniform symmetric flip (symflip) noise into the proxy data for its analysis. However, our method achieves its most significant performance gains when clients are affected by structured pairflip noise (Table 2&3). This demonstrates that our approach is effective even with mismatched noise types.

The proxy dataset is not used to simulate client local data, but as a common benchmark to identify parameters that are unstable to noise. This instability is a general property of model parameters, rather than their response to specific data distributions or noise patterns.

W2: Experimental diversity and system scale

To address this, we conduct experiments to simulate a more realistic, large-scale cross-device scenario. In this setup, we use 100 total clients with a 10% participation rate (10 selected per round). The results show RFedLR framework maintains its performance advantage in large-scale systems.

Table R1: Performance Comparison in a large-scale federated setting (100 total clients, 10% participation rate) with 40% pairflip noise. We take the average test accuracy for demonstration.

Furthermore, to address the reviewer's concern about the diversity of data heterogeneity, we conduct experiments under a severe non-IID setting (the Dirichlet concentration parameter is 0.1), as detailed in our response to Q4 (Table R5). These results show that RFedLR is more effective in highly heterogeneous data distribution, demonstrating its robustness.

These new results provide a more comprehensive validation of our method, confirming that RFedLR is effective in diverse federated systems.

W3: Comparison of SOTA federated noise learning methods

The experiments in Tables 2&3 focused on comparing RFedLR with other federated PEFT methods. As our work introduces a novel framework within the federated PEFT domain, we believe it is necessary to compare its performance with federated PEFT methods.

We agree that comparing with the most relevant state-of-the-art is essential. We implement and evaluate robust federated learning methods, FedCorr [2], RHFL [3], FedFixer [4]. As these methods are for conventional full-parameter FL, we adapt their core noise-handling mechanisms to our federated PEFT setting for a fair comparison. The experiments are performed under a noise rate of 0.4. These results (Table R2) demonstrate that RFedLR outperforms these robust FL methods. We analyze that these methods cannot mitigate the aggregation bias inherent in federated LoRA, while our framework is designed with SRT to counter local noise amplification and AFLA to address aggregation bias and noise propagation.

Table R2: Performance comparison of RFedLR with SOTA robust federated learning methods under a noise rate of 0.4. Values represent average test accuracy (%).

W4: Integration of each components

The process for a single round of RFedLR is as follows:

• Server-Side Preparation (AFLA): At the start of a round, the server selects the more important LoRA matrix (A or B) as trainable and distributes the global model to the participating clients.
• Client-Side Local Tuning (SRT): Each client first uses the public proxy data to calculate a noise-sensitivity mask for the trainable LoRA matrix. During local training, this mask is used to selectively update only the most noise-sensitive parameters, enhancing local robustness.
• Server-Side Aggregation (AFLA): After clients upload their updated matrices, the server calculates a hybrid aggregation weight for each client based on matrix importance and data size. It then performs a weighted average of these updates to create the new global model.

To further clarify this workflow, we will add the detailed pseudocode to the appendix of our paper.

W5&Q6: Privacy implications

We would like to clarify a potential misunderstanding. Clients do not compute or share any “parameter importance matrices”. In each round, the client performs local training on the specified LoRA matrix and calculates an importance score. In the aggregation phase, each client only uploads an updated LoRA matrix and a scalar importance score. We believe that transmitting a scalar score computed on a public proxy dataset does not introduce substantial new privacy risks. We will discuss the privacy impact in the final version.

Q2: Sensitivity analysis for $\tau$

The parameter $\tau$ controls the proportion of parameters updated in our SRT mechanism. To provide best performance, we conduct a sensitivity analysis. Our results show that performance peaks at our chosen value of $\tau=0.2$. A smaller value overly constrains the model, while a larger value diminishes the regularization effect against noise.

Table R3: Parameter analysis for $\tau$ in the SRT module. The analysis is conducted under the 20% pairflip noise setting.

Q3: The intuition behind FIM and IM

We adopt the empirical FIM to quantify the parameter importance, as it is a classic and effective measure of parameter impact on model prediction. Parameters with high Fisher Information values most affect predictive ability.

The "IM" mentioned by the reviewer likely refers to our "hybrid importance score", denoted as $I$ in Eq.10, which combines FIM with our noise sensitivity metric. Although FIM can identify important parameters under ideal conditions, it ignores the parameter stability in noisy environments. Therefore, we design the hybrid importance score "$I = F / (S + \epsilon)$" to balance the contribution and reliability of parameters. We conduct ablation experiments on these two components (Fisher information and Noise sensitivity) to justify the design.

To validate this design choice, our ablation study (Table R4) that empirically demonstrates the superiority of this hybrid score over using either Fisher Information or noise sensitivity alone. The results confirm that both components contribute positively, justifying our combined formulation.

Table R4: Ablation study for the hybrid importance score. The table compares the average test accuracy (%) of RFedLR (Ours) with variants that exclude Fisher information (w/o F) or noise sensitivity (w/o S) from the importance calculation in Eq.10

Q4: Performance under highly data heterogeneous environments

We conduct the experiments under highly data heterogeneous environments by setting the Dirichlet concentration parameter to 0.1, with a noise rate of 0.4. The results (Table R5) show that the performance advantage of RFedLR becomes even more pronounced in this severe data heterogeneous scenario. We attribute this to the AFLA mechanism, which weights clients based on the amount of data, and the estimated importance and stability of their LoRA updates, ensuring robust aggregation results in highly non-IID scenarios.

Table R5: Comparison with the state-of-the-art methods under severe data heterogeneous scenario (Dirichlet concentration parameter is 0.1) when the noise rate is 0.4. We take the average test accuracy for demonstration.

Q5: Adaptation to LoRA variants

The core modules SRT and AFLA in RFedLR focus on the LoRA matrix parameters and do not rely on their fixed rank or structure. Therefore, the RFedLR framework can be effectively combined with other LoRA variants. For example, AdaLoRA dynamically allocates optimal, heterogeneous matrix ranks to different layers, and then RFedLR performs robust local training and global aggregation of these allocated matrices.

References

[1] Revisiting Weighted Aggregation in Federated Learning with Neural Networks, in ICML 2023.

[2] Fedcorr: Multi-stage federated learning for label noise correction, in CVPR 2022.

[3] Robust federated learning with noisy and heterogeneous clients, in CVPR 2022.

[4] FedFixer: Mitigating heterogeneous label noise in federated learning, in AAAI 2024.

Dear Reviewer N5Xd：

Thank you for your insightful review and valuable feedback. We address your key concerns in detail below, aiming to clarify and demonstrate the effectiveness of our proposed approach.

W1: The Novelty of the paper

In this paper, we investigate the novel problem of achieving robust federated PEFT in noisy environments. With the rapid growth of large-scale pre-trained models, this is a realistic and challenging problem. Compared to conventional robust federated learning, federated PEFT in noisy environments presents additional challenges. The low-rank factorization structure of LoRA can lead to noise amplification in noisy environments. Noisy perturbations generate new bias terms through matrix multiplication. Furthermore, federated LoRA aggregation suffers from inherent aggregation bias, exacerbated by noisy local updates, and this bias is propagated and amplified throughout the federated system.

To address these challenges, we propose RFedLR framework, which consists of two modules: (1) SRT selectively updates noise-sensitive parameters to enhance robustness at the local fine-tuning phase, and (2) AFLA introduces an adaptive aggregation scheme at the collaborative update phase to mitigate aggregation discrepancies and curtail noise propagation.

Prior works, particularly in backdoor defense, have leverage gradient analysis to detect parameters affected by the backdoor trigger. The backdoor literature aims to detect deterministic signals from a single static attack. In contrast, our approach aims to measure the parameter stability against random label noise. Its goal is not to defend against targeted attacks, but rather at achieving generalized robust regularization to prevent overfitting. Besides, some work prunes or permanently disables outlier parameters. Unlike these approaches, SRT maintains all parameters active during forward propagation to ensure the model representability is not compromised, while selectively updating noise-sensitive parameters during backpropagation. This conditional update mechanism is a more sophisticated robust training mechanism. Furthermore, the parameter sensitivity score is combined with Fisher information to form a hybrid importance score, based on which aggregate weights are assigned to the clients.

In summary, the novelty of our work lies in studying a new problem, analyzing its unique challenges, and proposing a new framework to address these challenges.

W2: Expansion of the experimental scope

As our paper focuses on the challenges of fine-tuning large-scale Pre-trained Models (PTMs), we choose ViT-B (pre-trained on ImageNet-21K) as our backbone. Its fine-tuning on CIFAR-100 follows a standard and challenging benchmark from recent visual PEFT literature [1], ensuring our baseline comparisons are meaningful.

To demonstrate the broad applicability of our framework, we conduct experiments on a NLP fine-tuning task involving a language model (Roberta-Base model). We evaluate the performance of RFedLR on the MNLI task from GLUE benchmark with 20% symflip noise in a non-IID federated setting. The results are as follows:

Table R1: Comparison with the state-of-the art methods on MNLI task from GLUE benchmark when the noise rate is 0.2 and noise type is symflip. We take the average test accuracy of the local models for demonstration.

The results show that RFedLR significantly outperforms existing federated PEFT methods in the language domain, confirming its applicability beyond image classification. We consider extending it to the LLM fine-tuning scenario in future work.

W3: The construction of the noisy proxy dataset

To clarify, the noisy proxy dataset $D_n$ is generated by injecting symflip noise into the clean dataset $D_c$, which randomly flips a label to any other class with uniform probability. Symflip noise represents a general form of label noise that does not rely on any assumptions about class similarity. By evaluating parameter sensitivity against this high-entropy noise, we can identify parameters that are broadly unstable to any incorrect supervision signal. This ensures that our sensitivity analysis is robust and not biased towards a specific, structured noise pattern. The approach allows RFedLR to learn a more general measure of noise sensitivity, enhancing the overall robustness.

The design choice is validated by our experimental results (Tables 2&3), where RFedLR demonstrates superior performance under both symflip and pairflip noise settings, confirming the general applicability of the learned sensitivity measure. We will clarify this implementation detail in Section 4.1 of the revised manuscript.

W4: The construction of the noisy proxy dataset

The effectiveness of our method does not depend on the assumption that the distribution or noise pattern of the proxy dataset must be match to the local data.

1. Robustness to Data Distribution Mismatch: In our setup, the proxy dataset is a uniformly distributed mini-batch randomly sampled from the CIFAR-100 test set. However, the local data was explicitly configured to be non-IID. As demonstrated by our results, particularly in the highly data heterogeneous environments (Refer to Q2), our method performs well, which proves that distribution matching is unnecessary.
2. Robustness to Noise Type Mismatch: As clarified in our response to W3, our framework utilizes symflip noise to generate noisy proxy data. Table 2&3 show that RFedLR still demonstrates remarkable effectiveness when clients have structured pairflip noise. Besides, Table 1 shows that the introduction of SRT module results in a significant accuracy improvement under pairflip noise with different noise rates. This shows that a mismatch between the proxy noise and the local true noise does not undermine the effectiveness of our gradient discrepancy analysis.

The proxy dataset is not used to simulate client local data, but to serve as a common, objective benchmark for identifying parameters that are unstable to noise. This instability is a general property of model parameters, rather than their response to specific data distributions or noise patterns.

W5: Effectiveness under cross-device settings

We agree that validating the scalability and generality of our method in a setting with a larger number of clients is essential for demonstrating its practical applicability. To address this concern, we conduct experiment in a large-scale setting designed to simulate a cross-device federated learning scenario. This experiment involves 100 total clients with a 10% participation rate, meaning 10 clients are sampled for training and aggregation in each communication round. The results, presented in the table below, show that RFedLR outperforms other methods in this more challenging and realistic environment.

Table R1: Performance Comparison in a large-scale federated setting (100 total clients, 10% participation rate) with 40% pairflip noise. We take the average test accuracy of the local models for demonstration.

This result demonstrates that the effectiveness of our proposed RFedLR framework is not limited to small-scale, cross-silo settings but scales successfully to systems with more clients.

Q1: Performance of clean clients

We would like to clarify that this baseline is included in our analysis, presented in Figure 2 on page 2. The purpose of this figure is to visually demonstrate the gap in model performance between practical noisy the ideal clean scenarios. Specifically, Figure 2(b) shows the performance of Federated LoRA (FedLR), the baseline setting of our study. As can be seen from the figure, when all clients have clean datasets (shown by the gray dashed line), the final average accuracy of the model can reach 86.76%. When the client dataset contains noise (shown by the orange solid line), the accuracy drops to 75.28%. This 11.48% performance gap reveals the significant negative impact of label noise on Federated PEFT. This significant gap emphasizes the need to study a robust federated PEFT framework. Our proposed method RFedLR achieves an accuracy of 83.12% under the same noise conditions of the Figure 2. For clarity, we will add specific values to Figure 2 in the final version.

Q2: Performance under severe data heterogeneity

We have conduct the experiments in a highly non-IID setting by setting the Dirichlet concentration parameter to 0.1, with a noise rate of 0.4. The results (Table R3) show that the performance advantage of RFedLR becomes even more pronounced in this severe data heterogeneous scenario. We analyze the advantages to be attributed to the AFLA mechanism. AFLA weights clients based on the amount of data, and the estimated importance and stability of their LoRA updates, ensuring robust aggregation results in highly non-IID scenarios.

Table R3: Comparison with the state-of-the art method under severe data heterogeneous scenario (Dirichlet concentration parameter is 0.1) when the noise rate is 0.4. We take the average test accuracy of the local models for demonstration.

The small proxy dataset serves as a general benchmark for identifying which parameters are inherently unstable when exposed to label noise. This parameter-level property is largely independent of the specific data distribution.

References

[1] Sensitivity-aware visual parameter-efficient fine-tuning, in ICCV 2023.

Dear Reviewer MLqt：

Thank you for your valuable feedback and for your time reviewing our work. We hope our responses below help clarify the issues.

W1: Computational Overhead

We conduct a detailed analysis of the resource consumption of our proposed RFedLR compared to other baseline methods. As shown in Table R1, the results show that the additional memory cost of RFedLR is negligible. Specifically, its memory usage is 33642.74 MB, which is comparable to other baseline methods. Furthermore, RFedLR utilizes the fewest trainable parameters (0.1754%) and achieves the highest test accuracy among all compared methods.

We will add a table summarizing these findings to the appendix and discuss these results in the main paper.

Table R1: Comparison of trainable parameter(%) relative to FFT and memory cost(MB) for various methods.

W2: Hyperparameter Sensitivity

We include a sensitivity analysis for the key hyperparameters $\tau$ and $\lambda$. For the parameter keep rate $\tau$ in SRT (Table R2), our analysis shows that performance peaks at our chosen value of $\tau=0.2$, achieving an accuracy of 83.12%. When $\tau$ is too low, the model plasticity is limited. When $\tau$ is too high, the regularization effect of the selective update strategy is weakened and performance decline.

Table R2: Parameter analysis for $\tau$ in the SRT module. The analysis is conducted under the 20% pairflip noise setting.

For the balancing hyperparameter $\lambda$ in AFLA (Table R3), our results show that the performance of RFedLR is relatively stable for the selection of $\lambda$. When $\lambda$ is 0.4, RFedLR achieves the highest test accuracy, which can effectively balance the contribution of data scale and matrix importance.

Table R3: Parameter analysis for $\lambda$ in the AFLA module. The analysis is conducted under the 20% symflip noise setting.

We will include these results and corresponding charts in the paper to improve the integrity of the paper.

W3: Manuscript Clarifications and Corrections

We are grateful to the reviewer for the meticulous reading and for pointing out several areas for improvement in the manuscript.

We will correct all identified typos and inconsistencies. Specifically, we will revise the description of Eq. 5 and update the formula to $S_{k}^{i} = \left| \nabla_{W_{k}^{i}}L_{c} - \nabla_{W_{k}^{i}}L_{n} \right|$ to accurately reflect the calculation of the magnitude of the gradient discrepancy. For Eq.10, we need to clarify that the unnormalized sensitivity score ($S_k^i$) is used in the calculation of the hybrid importance score to avoid ambiguity. We will correct the inconsistent numerical formatting in Table 3 and the typos in the final version.