FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models

We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on the significance of topic and the clarity of the paper. We hope the following discussions can address the reviewer's remaining questions.

1. Clarity of Difference From Baseline and FL Experimental Settings

We appreciate this feedback on the clarity of the difference between FedRW and the baseline (EP-MPD), and the FL experimental settings. We want to clarify that we have detailed the key information in our manuscript:

• Section 1 (Introduction): We introduced EP-MPD, the SOTA work for federated hard deduplication, and illustrated three key challenges unresolved, followed by the introduction of our proposed method, FedRW. We also presented an overview of EP-MPD and its potential limitations in Figure 1.
• Section 2 (Related Work): In the subsection on Distributed Deduplication in existing work, we discussed the mechanism and limitations of EP-MPD.
• Section 4 (Framework): This section offered a detailed explanation of FedRW. In Figure 2, we presented the architecture framework of FedRW and provided an intuitive comparison on protocol runtime, perplexity, and third-party reliance. We consider these presentations to be sufficient in illustrating the essential differences between the two methods.
• Section 5 (Experiments): The experimental results on preprocessing and model performance clearly showed the difference between FedRW and EP-MPD.
• Section 5.1 (Experimental Settings): This section declared the datasets, client number, data preprocessing and distribution strategies, evaluation metrics, and overall experimental setup. For completeness, Appendix C includes a detailed description of the datasets, the partitioning standard, the experimental environment, and the hyperparameter configurations.

We sincerely hope that the above clarifications effectively resolve your concerns.

2. Effectiveness Across FL Settings

We appreciate your concerns for more detailed effectiveness under various FL settings. Regarding the three specific FL settings:

• Current FL settings: In this study, we followed the baseline setup for unsupervised fine-tuning. As described in Lines 203-204, "To simulate redundancy, duplicates are synthetically added into the training set at different rates and distributed uniformly across 10 clients." The data follows an IID distribution.

• Non-IID: We appreciate the suggestion to explore the Non-IID setting, and we have conducted experiments on Qwen3-0.6B to evaluate FedRW's effectiveness.

  1. Setup 1: We categorized the Rotten Tomatoes dataset by label and set up 5 clients. To simulate Quantity Skew, we distribute the data according to the proportion of the total train set as [40%, 20%, 20%, 10%, 10%]. For Label Skew, we set the distributions for the binary (0/1) labels as: [(0.5, 0.5), (0.6, 0.4), (0.4, 0.6), (0.9, 0.1), (0.1, 0.9)].

  2. Result 1 (under 25% duplication)

     Method	IID	Quantity Skew	Label Skew
     Baseline	1.71	2.02	2.44
     FedRW (Ours)	1.59	1.96	1.66

  3. Setup 2: To simulate Feature Skew, we allocated datasets Poetry, Sonnets, and Plays to separate clients. These datasets differ distinctly in terms of text structure, sentence length, and lexical and syntactic complexity.

  4. Result 2

     Method	Feature Skew
     Baseline	3.43
     FedRW (Ours)	2.70

  The experimental results indicate that FedRW consistently maintains its performance advantages in the Non-IID settings.

• Personalized FL: We appreciate the suggestion to explore personalized FL. While the current FedRW framework primarily focuses on enhancing a global LLM, we acknowledge that language models benefit significantly from personalization to adapt to individual user preferences. A deeper exploration may involve diverse model architectures or personalization strategies, beyond the scope of this foundational privacy-preserving soft deduplication framework. We consider adapting FedRW to integrate with this pattern a valuable and exciting avenue for future research.

3. Evaluations on Mainstream Models

We appreciate your valuable suggestion, and have conducted additional perplexity evaluations on Qwen3-0.6B, Qwen2.5-0.5B-Instruct, and Llama-3.2-1B-Instruct under 30% duplication.

• Qwen3-0.6B

  Method	Haiku	Short Jokes	Rotten Tomatoes	Poetry	Sonnets	Plays
  Baseline	2.47	2.61	1.71	2.54	4.07	8.21
  FedRW (Ours)	2.36	2.44	1.59	2.21	3.62	7.23

• Qwen2.5-0.5B-Instruct

  Method	Haiku	Short Jokes	Rotten Tomatoes	Poetry	Sonnets	Plays
  Baseline	2.21	2.48	1.58	2.28	4.11	11.77
  FedRW (Ours)	2.12	2.36	1.55	2.03	3.84	9.92

• Llama-3.2-1B-Instruct

  Method	Haiku	Short Jokes	Rotten Tomatoes	Poetry	Sonnets	Plays
  Baseline	2.14	2.34	1.65	2.39	4.11	18.35
  FedRW (Ours)	2.09	2.21	1.54	1.99	4.00	16.03

As shown in the tables, FedRW continues to show its advantage in addressing the impact of duplication on model performance, particularly under challenging conditions such as data sparsity or complexity.

4. Discussion on Limitations

We appreciate your concern regarding the lack of a detailed discussion on limitations. While we have discussed FedRW’s boundaries and potential extensions in Section 6 (Lines 284–286), as well as certain limitations within the Experiments section, we commit to providing a clearer and more comprehensive discussion of these aspects in the revised manuscript.

We sincerely hope our replies have clearly addressed your concerns. We are deeply grateful for the valuable feedback and the broader perspectives you offered, which have helped us better position the significance of this work within the broader research landscape.

We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on novelty, privacy and security guarantees, significance of topic, and the overall clarity of the paper. We hope the following discussions can address the reviewer's remaining questions.

1. Non-Exact Duplicates

We greatly appreciate this insightful question. Indeed, there are scenarios where non-exact duplicates are more commonly encountered. We have conducted perplexity evaluations on the dataset, Twitter Sentiment Analysis [1], which contains non-exact repeated patterns due to the nature of social media.

• Result

  Method	Qwen3-0.6B	Llama-3.2-1B-Instruct
  Baseline	3.30	3.74
  FedRW (Ours)	2.98	2.49

These results demonstrate the effectiveness of our method in addressing non-exact repetitive patterns.

2. Significance of Deduplication in Real FL Scenarios

We truly appreciate this critical concern. Exact and near-exact duplicates are indeed a common issue in large-scale real-world datasets for LLMs. The presence of duplicates among private client data stems from several prevalent mechanisms:

• Influence of Shared Public Information: Private data often contains content derived from or heavily influenced by widely shared public information. Similar to open-source datasets like Twitter Sentiment Analysis, which inherently exhibits extensive repetition due to the nature of social media, private data collected from users or organizations also tends to contain duplicated or near-duplicate patterns. Specific common patterns may appear across the private data of different clients.

• Inherent Data Collection: Beyond external influences, duplicates can occur within sensitive personal information. For instance, GPS locations in personal travel information can exhibit duplicates due to recurring travel patterns or shared high-traffic areas [2], which can be collected to analyze user behavior patterns.

• Data Augmentation: The data augmentation strategies [3] used in private datasets, including those based on generative models or synthetic data, may also produce a large number of similar or identical samples with duplicated patterns.

These mechanisms indicate that data duplication is a pervasive challenge in real-world, distributed private datasets in FL, necessitating solutions like FedRW.

References

[1] passionate-nlp, "Twitter Sentiment Analysis Dataset." Kaggle.

[2] Wu, Zhaomin, et al. "Federated transformer: Multi-party vertical federated learning on practical fuzzily linked data." Advances in Neural Information Processing Systems 37 (2024): 45791-45818.

[3] Hou, Charlie, et al. "Pre-text: Training language models on private federated data in the age of llms." arXiv preprint arXiv:2406.02958 (2024).

3. Evaluations on Mainstream Models

We appreciate your suggestion, and have conducted supplementary perplexity evaluations on Qwen3-0.6B, Qwen2.5-0.5B-Instruct, and Llama-3.2-1B-Instruct under 30% duplication.

• Qwen3-0.6B

  Method	Haiku	Short Jokes	Rotten Tomatoes	Poetry	Sonnets	Plays
  Baseline	2.47	2.61	1.71	2.54	4.07	8.21
  FedRW (Ours)	2.36	2.44	1.59	2.21	3.62	7.23

• Qwen2.5-0.5B-Instruct

  Method	Haiku	Short Jokes	Rotten Tomatoes	Poetry	Sonnets	Plays
  Baseline	2.21	2.48	1.58	2.28	4.11	11.77
  FedRW (Ours)	2.12	2.36	1.55	2.03	3.84	9.92

• Llama-3.2-1B-Instruct

  Method	Haiku	Short Jokes	Rotten Tomatoes	Poetry	Sonnets	Plays
  Baseline	2.14	2.34	1.65	2.39	4.11	18.35
  FedRW (Ours)	2.09	2.21	1.54	1.99	4.00	16.03

As shown in the tables, FedRW consistently maintains its advantage in mitigating the impact of redundancy on model performance, particularly under challenging conditions such as data sparsity or complexity.

We agree that the impact of partial repetition may vary with model advancements, while data quality remains a fundamental factor in model performance. Deduplication is still an essential preprocessing step to ensure high-quality data.

4. Hyperparameter

We appreciate this valuable feedback. Regarding epochs, we typically utilize a small number (Appendix C, Line 516) as standard practice to prevent overfitting. Hence, our primary sensitivity analysis focused on the learning rate to assess FedRW's robustness.

• Setup: We evaluated perplexity on DistilGPT2 and Qwen2.5-0.5B-Instruct under learning rates of 1e-3, 5e-4, 3e-4, 1e-4, 5e-5, and 3e-5. We selected the Plays dataset, which exhibited a significant performance gap, to investigate FedRW's generalizability.

• DistilGPT2

  Method	1e-3	5e-4	3e-4	1e-4	5e-5	3e-5
  Baseline	16.12	16.38	16.13	14.21	15.07	14.18
  FedRW (Ours)	14.42	14.79	14.74	13.17	14.50	12.76

• Qwen2.5-0.5B-Instruct

  Method	1e-3	5e-4	3e-4	1e-4	5e-5	3e-5
  Baseline	12.86	11.07	10.63	11.50	11.77	10.67
  FedRW (Ours)	11.48	9.35	8.15	8.14	9.92	9.81

The experimental results indicate that FedRW consistently maintains its performance advantages and exhibits stable training behavior across this range of learning rates.

We are deeply grateful for your thoughtful consideration, which has enriched our understanding and improved the quality of this work.

We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on novelty, efficiency and effectiveness, privacy and security guarantees, significance of topic, and the overall clarity of the paper. We hope the following discussions can address the reviewer's remaining questions.

1. Rely on Techniques

We sincerely appreciate your recognition of the innovation and originality of FedRW. We agree that FedRW draws inspiration from the centralized soft deduplication. However, the same problem remains unresolved in the FL setting, where simply applying the existing techniques introduces new privacy challenges. Compared with multiple rounds of key agreement and encryption in SOTA work, we leverage the PSI primitive as a foundational component to implement a lightweight variant, tailored for the defined 2PC functionality, while achieving the stronger, third-party-free privacy assumption. We also present an innovative parallel orchestration strategy to overcome the inherent computational overhead of MPC protocols in FL. The proposed innovations achieve up to 28.78× speedup in preprocessing.

Thus, we respectfully argue that the core novelty of our work lies in the innovative integration, adaptation, and optimization of these techniques to address a previously unexplored and challenging problem: enabling efficient and privacy-preserving soft deduplication in FL without relying on a trusted third party.

2. Future Optimization

We appreciate the reviewer's insightful comments for optimization, and we fully agree that these are vital directions for future research.

Regarding the semantic matching, this is indeed a significant issue in large-scale real-world datasets for LLMs. We may leverage the representation learning capability of transformer-based architectures to extract semantic duplication.

Regarding the dynamic client adaptability, we acknowledge the importance of dynamic scenarios in which clients may join, leave, or exhibit varying computational capabilities. A preliminary idea is to employ Updatable PSI [1] to address the changes in sample weights caused by client arrivals and departures, and this may trade off between functionality and computational overhead. We plan to explore these valuable aspects in our future research.

References

[1] Agarwal, Archita, et al. "Updatable private set intersection from structured encryption." Cryptology ePrint Archive (2024).

We sincerely appreciate your thoughtful and well-justified suggestion, as well as the valuable inspiration you have provided regarding future optimization of our work.

We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on novelty, efficiency and effectiveness, privacy and security guarantees, significance of topic, and the overall clarity and logical structure of the paper. We hope the following discussions can address the reviewer's remaining questions.

1. Convergence Comparisons.

We appreciate your suggestion and have included the average training loss of FedRW compared with the baseline under 30% duplication in the tables.

• GPT-2 Large

  Dataset	Method	Round 0	Round 1	Round 2	Round 3
  Haiku	Baseline	1.842	0.839	0.806	0.811
  	FedRW (ours)	4.671	0.835	0.725	0.705
  Rotten Tomatoes	Baseline	1.124	0.444	0.426	0.379
  	FedRW (ours)	3.038	0.425	0.378	0.375
  Short Jokes	Baseline	0.994	0.720	0.622	0.641
  	FedRW (ours)	1.263	0.679	0.595	0.547
  Sonnets	Baseline	2.835	1.466	1.366	1.730
  	FedRW (ours)	4.329	1.349	1.243	1.168

• DistilGPT2

  Dataset	Method	Round 0	Round 1	Round 2	Round 3
  Haiku	Baseline	3.225	1.283	1.267	1.274
  	FedRW (ours)	2.950	1.274	1.255	1.252
  Short Jokes	Baseline	1.511	0.751	-	-
  	FedRW (ours)	1.388	0.745	-	-
  Rotten Tomatoes	Baseline	1.771	0.607	-	-
  	FedRW (ours)	1.338	0.437	-	-
  IMDB	Baseline	4.234	1.964	-	-
  	FedRW (ours)	3.838	1.939	-	-
  Poetry	Baseline	10.858	2.989	1.313	1.107
  	FedRW (ours)	9.908	1.999	1.122	1.049
  Sonnets	Baseline	17.589	3.443	2.019	1.870
  	FedRW (ours)	15.901	3.048	1.971	1.802
  Plays	Baseline	23.848	3.379	2.885	2.815
  	FedRW (ours)	20.460	3.234	2.854	2.767

The datasets, including Short Jokes, Rotten Tomatoes, and IMDB, were trained for fewer rounds to mitigate overfitting. Notably, on GPT-2 Large, FedRW exhibited a higher initial loss compared with the baseline, whereas the DistilGPT-2 model showed a lower one. This consistency is related to the loss reweighting mechanism and the variance of model capacity. In summary, FedRW demonstrates faster model convergence and better final performance.

2. Writing-related Issues.

We sincerely appreciate your thorough reading, and we will revise the issues according to your comments in the final version.

To further clarify Figures 1 and 2, we offer the following supplementary explanation:

• Fig. 1: (a) illustrates the inherent challenges of global deduplication in a decentralized setting, where privacy constraints prevent direct data sharing.
  (b) describes the paradigm adopted by the state-of-the-art solution (also used as our baseline), which relies on a trusted third party for deduplication over encrypted data, followed by decryption. The right part of Fig.1 indicates that this pattern may lead to the loss of key training samples.
• Fig. 2: The left panel of Fig.2 illustrates the parallel framework of FedRW, where each small square represents a single 2PC protocol interaction. The middle section depicts our reweighting scheme, which preserves the integrity of the dataset. The colors reflect the different weights assigned to samples. The right panel provides a comparison between FedRW and the baseline approach.

As you pointed out, Section 3 and Appendix A.3 illustrate the security and threat model. Due to the space constraints, we were unable to illustrate adaptation to FL in detail. We will include a brief description in the revision as follows:

• FedRW operates under the semi-honest adversary model in federated learning (FL), meaning all participants will follow the protocol honestly but may attempt to infer additional information from observation. For the scope of this work, we assume that the parties do not actively collude with one another. While we acknowledge that other forms of more active or malicious attacks exist in FL – such as inference attacks, backdoor attacks, or reconstruction attacks – these threats are considered orthogonal to the primary objective of this study, which focuses on designing the first privacy-preserving soft deduplication via sample reweighting. These additional and vital security considerations will be explored in our future research to enhance FedRW's robustness.

Following your valuable suggestion, we will revise the manuscript with the above information.

3. Evaluation in Real-World Scenarios.

We appreciate your valuable suggestion, and have conducted experiments on real-world datasets with repeated patterns on two mainstream models, Qwen3-0.6B and Llama-3.2-1B-Instruct.

• databricks-dolly-15k [1] is an open-source dataset of instruction-following records generated by thousands of Databricks employees.
• Twitter Sentiment Analysis [2] is an open-source dataset of tweets labeled with sentiment categories, which inherently contains repeated patterns due to the nature of social media.

• Results (Perplexity)

  • Qwen3-0.6B

    Method	databricks-dolly-15k	Twitter Sentiment Analysis
    Baseline	5.11	3.30
    FedRW (Ours)	4.65	2.98

  • Llama-3.2-1B-Instruct

    Method	databricks-dolly-15k	Twitter Sentiment Analysis
    Baseline	4.79	3.74
    FedRW (Ours)	4.51	2.49

As presented in the table, FedRW consistently outperforms the baseline under the same experimental settings, showing the effectiveness and generalization ability of our method.

References

[1] databricks, "databricks-dolly-15k." Hugging Face.

[2] passionate-nlp, "Twitter Sentiment Analysis Dataset." Kaggle.

4. Extension to Malicious Model.

Thank you for raising the thoughtful concerns about malicious clients. Extending FedRW to resist malicious adversaries would be an interesting and important research direction.

As mentioned in 2, we acknowledge that the FedRW primarily operates under a semi-honest threat model, which is sufficient for the vast majority of real-world scenarios, given its high efficiency and practicality.

Regarding the malicious model, it is feasible to design specific verification based on FedRW’s existing architecture, such as Zero-Knowledge Proofs (ZKPs), to verify the consistency of parties’ behaviors during preprocessing and training. However, such approaches would come at a significant computational and communication cost.

These potential schemes trade off between model accuracy, data privacy, and computational overhead. Given the scope and depth of these problems, we have deferred this research to our future follow-up work.

We are deeply grateful for the thought-provoking discussions and valuable insights that have enriched our understanding and improved the quality of this work.