DictPFL: Efficient and Private Federated Learning on Encrypted Gradients

We thank Reviewer C4sK for providing constructive comments.

Q1. Why is the gradients privacy not compromised? Pruning may let the server know the distribution of large entries in the gradients. Please clarify what information is protected under the semi-honest setting.

The gradient's privacy in our DictPFL is not compromised since all model gradients shared with the server are already encrypted, and unencrypted gradients are kept local without sharing. The pruning does not impact privacy since it is performed before encryption. After encryption, the ciphertext of unpruned gradients will not reveal the values or distributions to the server. This ciphertext privacy is guaranteed by FHE. Thus, our methods share the same privacy protection as the baseline with fully encrypted gradients (privacy of gradients and model weights of clients are protected), but are more secure than the prior work FedML-HE that still shares partial unencrypted gradients with the server.

Q2: It is not sure if the evaluation is fair. The proposed method needs a pretrained weight, and it is not clear how the pretrained weight affect the utility.

Our evaluations are fairly compared. All methods (DictPFL and our baselines) are conducted on the same pre-trained weights. In the current ML settings, more and more pre-trained models are used, and it is practical to use them as initial models especially for privacy-sensitive fields like healthcare where data is scarce. Starting from a pre-trained model and fine-tuning it on privacy-sensitive data is very promising and practical. In addition, our proposed methods, like Decompose-for-Partial-Encrypt (DePE), work well for cases where no pre-trained model is used. Instead, one could use randomly initialized model parameters, just like LORA work (which decomposes W=A*B) [1] demonstrates strong performance when A is initialized randomly and fixed, and only B is trained. We performed experiments to validate that, to achieve the same level of accuracy (80% for GTSRB dataset with ViT-16), using a randomly initialized dictionary needs 11.81 minutes of training time while the baseline requires 295 minutes of training time. We will clarify this point in the next version of our manuscript.

Q3. Figure 2 only has results for existing works. How does your algorithm fit into the figure?

Thanks for asking this question. We used Figure 2 as a motivating example to analyze the execution communication and computation latency breakdown. In Figure 9 of the results section, we put our method and all prior work into one figure for comparison and show that our work significantly reduces the communication and encrypted operations. We will link Figure 2 and Figure 9 in a more explicit way in the next manuscript.

Q4. If the use of LCNN restricts the utility, as essentially we are working with a smaller model. Also, since LCNN is not a popular model structure, will this affect the applicability of the proposed method?

LCNN works more like an advanced weight decomposition technique (not a new architecture), thus when the decomposition is full-rank, it does not reduce the model size or lose representation abilities as shown in equation 1 in the paper.

As an option, users could tune the rank size to control the tradeoff between utility and efficiency. LCNN is not a new model structure or architecture, and it can be applied to any linear projections, convolutions, and Transformers[2][3]. Our experiments have shown that DictPFL works well on models of different scales (from LeNet to Llama2-7b).

Q5. How to compare your work to secure aggregation? Is goal to keep the server from knowing aggregated weights? If so, does the following simple alternation of secure aggregation work? The clients first use some crypto-safe protocol to share a random number/matrix $W_{random}$, and perform secure aggregation on $W_{random}+W$, by uploading $W_{random}/n+W_i$. In this way we may not need to perform costly homomorphic encryption.

Yes, one advantage of HE-based FL over Secure Aggregation is to keep the server from knowing aggregated weights (HE will ensure the aggregated weights are still encrypted to server).

Also, the reviewer’s proposed simple alternation of secure aggregation is insightful but introduces additional vulnerabilities. For instance, the server can compute differences between different rounds’ masked weights, i.e., $(W_2+W_{random}/n)-(W_1+W_{random}/n)=W_2-W_1$, to reveal the model updates $\Delta W$ and perform gradient inversion attacks. Additionally, client dropouts disrupt mask cancellation, as missing clients’ shares prevent proper removal of $W_{random}$, corrupting the aggregated result. While promising, this approach requires further innovations to strictly protect weight. This would be an interesting future work.

[1] Improving loRA in privacy-preserving federated learning

[2] Dictformer: Tiny transformer with shared dictionary

[3] Lite-mdetr: A lightweight multi-modal detector

We thank Reviewer Y4dC for the thorough reading of our manuscript and for providing constructive comments.

Q1. Misleading "full encryption": Dictionary D and pruning patterns may leak privacy.

Our "full encryption" refers explicitly to encrypting all information that clients share with the server. The dictionary D is static, untrainable, and globally shared only among clients—it is never shared with or accessible by the server. Reactivation patterns from pruning are based solely on the client's local gradient history, which is also never revealed to the server. The server receives only encrypted pruned gradients, and thus cannot infer pruning or reactivation patterns, ensuring no privacy leakage.

Q2. Privacy metrics are only applied to image datasets?

We used Figure 8(a) and privacy metrics (e.g., LPIPS-based recovery scores) as an example to illustrate that FedML-HE leaks sensitive information due to partially plaintext gradients vulnerable to inversion attacks. In contrast, DictPFL encrypts all gradients shared with the server, preventing such privacy leakage for any data type, including images and text.

Q3. Results for Llama2 (7B) in Figure 8b: execution details (e.g., rounds).

For the Llama2 (7B) in Figure 8b, we provided the training time for different models, where the training time is equal to the training round number multiplied by the training round time. When we compared the different methods, we calculated the training time required to achieve the same level of performance (60% accuracy on the MetaMathQA task). We will clarify these details in the new manuscript.

Q4. Efficiency comparison with non-HE sparsity/pruning methods.

We compared DictPFL's efficiency with plaintext FL in Figures 2 and 9, showing roughly a 3× overhead for privacy guarantees. Non-HE sparsity or adaptive pruning methods excel in plaintext efficiency but do not provide privacy protections, limiting direct comparison. If reviewers suggest specific methods, we can include additional comparisons.

Q5. HE parameters: Current compilers ensure the security level for the given HE parameters. Our HE parameters are secure and are detailed in Appendix A.

Q6. Table 3 statistical testing.

We performed Welch’s t-test between the 3-client and 20-client settings in Table 3. The results confirm a statistically significant difference $t(7.0) = 2.50, p = 0.040$.

Q7. Table 4 tests \alpha in [0.3, 0.9], but the main experiments’ \alpha is unclear.

The main experiments in Figure 7 were conducted under IID setting (α = ∞). These experiments ran for a maximum of 50 training rounds.

Q8: Formal proofs with lemmas and theorems.

Our DictPFL framework includes two HE-aware modules (DePE, PrME) that generalize decomposition and pruning techniques, previously grounded theoretically in plaintext FL. Our primary innovation adapts these established methods to HE constraints rather than developing new theoretical foundations. The practical benefit of our approach lies in significantly reducing ciphertext volume, thus mitigating the high training cost in HE-based FL. We will clarify explicit links to prior theoretical work in the revised manuscript.

Q9. Pretrained models: Please refer to C4sK Q2.

Q10. Study on Client numbers: Please refer to 4p9J Q2.

Q11. Text generation tasks.

In Appendix B.2, we evaluated DictPFL on text generation by fine-tuning the Tiny Llama model on the MetaMathQA mathematical reasoning task. DictPFL reduces training time by 94.2% compared to the previous state-of-the-art, FedML-HE.

Q12. The pruning strategy convergence (may be misaligned in non-IID settings due to reliance on historical gradients).

DictPFL addresses pruning misalignment in non-IID settings using Holistic Reactivation Correction (HRC), which reactivates pruned parameters by incorporating client-specific accumulated gradients. This preserves essential local information despite divergence from global patterns. Table 4 shows that HRC effectively maintains stable convergence across varying non-IID settings.

Q13. Static nature of the dictionary, the impact of pruning on convergence.

Dictionary decomposition leverages the insight that correlated model weights can be compactly represented as linear combinations of key vectors (dictionary). By using a static dictionary and training only the combination coefficients (lookup tables), we drastically reduce HE-related overhead. Our experiments confirm that this approach achieves higher accuracy and lower training costs compared to previous HE-based methods.

Q14. Dirichlet sampling may not capture real-world non-IID distributions.

Dirichlet is widely used in FL to simulate real-world non-IID. It provides reproducible conditions reflecting practical scenarios like medical imaging [1].

[1] Improving performance of federated learning based medical image analysis in non-iid settings using image augmentation.

We thank Reviewer jD6X for the thorough reading of our manuscript and for providing constructive comments.

Q1. From Section 4.1, DePE is established on the assumption that a public model is shared at the initial of FL, which may not always hold.

Please refer to the Reviewer C4sK's Q2.

Q2. TIP and HRC are empirical and lack novelty.

The proposed TIP and HRC are aiming to address novel and tricky challenges of pruning in HE-based FL. While numerous pruning methods exist in plaintext FL, their direct application to HE-based FL is fundamentally infeasible due to unique cryptographic constraints.

Key Challenges in HE-Based Pruning

1. Server-Side Pruning Limitations: Existing server-side methods [4] require access to plaintext gradients for pruning, which is impossible in HE-based FL, where the server only accesses ciphertexts.

2. Client-Side Pruning Misalignment: Client-side methods [5] allow clients to prune locally but will lead different clients to prune different positions. And because HE’s SIMD packing mechanism, which encrypts multiple plaintext gradients into a single ciphertext, demands strict alignment of pruned positions. Mismatched indices render aggregated ciphertexts unusable.

In order to address these issues to enable privacy-preserving, HE-compatible pruning, we propose TIP and HRC.

Novelty of TIP and HRC

1. Temporal Inactivity Pruning (TIP) solves the alignment challenge by leveraging global gradient history, which is identical across all clients, to derive a shared pruning mask. This ensures clients prune identical positions, maintaining HE-compatible aggregation.

2. Holistic Reactivation Correction (HRC) addresses a critical limitation of irreversible pruning: permanently excluding parameters risks losing valuable gradients. HRC dynamically reintroduces pruned parameters based on global importance, preserving model utility without compromising efficiency.

These mechanisms are the first to enable privacy-preserving, HE-compatible pruning, bridging a critical gap between plaintext efficiency techniques and encrypted FL’s constraints. Prior work cannot operate under HE’s limitations, which demand novel solutions to align pruning decisions without exposing sensitive data.

Q3. Since the experiments part considers web simulation, it is better to attach codes for more convenient reproduction.

In our submission, we have included the code for the main results as supplementary material to facilitate verification of the results. Upon acceptance, we will release a public GitHub repository with detailed documentation, step-by-step execution guides, and pre-configured environments to ensure the replication of all results.

Q4. Is it necessary to encrypt indices? What attack will be brought by transmitting encrypted content with plaintext indices?

In the proposed pruning method PrME, no indices are transmitted, plaintext or ciphertext. Pruning decisions are derived from shared global gradient history, ensuring clients prune identical positions. This eliminates the risk of attacks via plaintext indices (e.g., inferring sensitive patterns from pruned parameter locations). Thus, DictPFL’s design inherently avoids vulnerabilities associated with index transmission while maintaining full encryption of sensitive data.

Q5. In DePE, is SVD decomposition the only method to reduce matrix rank? Why is T set as V in SVD decomposition?

While other decomposition methods such as PCA and QR with Truncation can reduce matrix rank, SVD provides the better rank-k approximation according to Eckart-Young theorem, to maintain the important information in the pre-trained weights. Moreover, SVD is widely used to extract low-dimensional information-sensitive representation from model weights to compress the model [1][2][3].

The lookup table $T$ is set to $V$ (right singular vectors) to exploit the orthogonality inherent to SVD. The columns of $V$ form a decorrelated basis for the row space of the original weight matrix, where each column represents an independent direction of variation. This orthogonality minimizes redundancy and captures the most significant parameter correlations, enabling efficient gradient compression. The choice of $T = V$ and $D = U\sum$ (left singular vectors scaled by singular values) aligns with established practices in model compression [3].

[1] Asvd: Activation-aware singular value decomposition for compressing large language models.

[2] Dictformer: Tiny transformer with shared dictionary.

[3] Lite-mdetr: A lightweight multi-modal detector.

[4] Fedmef: towards memory-efficient federated dynamic pruning.

[5] Zerofl: Efficient on-device training for federated learning with local sparsity.

We thank Reviewer 4p9J for the thorough reading of our manuscript and for providing constructive comments.

Q1. Security evaluation is limited to Gradient Inversion Attacks, and the paper does not assess its resistance to other security threats. Do you plan to conduct additional security evaluations against other attack methods?

We focus on gradient inversion attacks because they represent the primary privacy risk unique to federated learning where attackers exploit gradients during training to reconstruct client data. In contrast, model inversion and membership inference attacks target the final trained model during inference, which is a post-training concern not directly related to the FL process itself. DictPFL’s encryption ensures no gradients are exposed during training, directly addressing the FL-specific threat. Other attacks are model-level risks applicable to any trained model, regardless of how it was trained. These are not influenced by FL’s training protocol. This aligns with prior SOTA HE-based FL work FedML-HE, which similarly focuses on gradient inversion. We will clarify this point in the next manuscript.

Q2. Do you have plans for additional experiments to evaluate scalability in large-scale client environments?

We appreciate the reviewer’s suggestion regarding scalability evaluation. In response, we conducted new experiments with 50, 100, and 200 clients, demonstrating that DictPFL consistently outperforms baselines in terms of efficiency. We will incorporate these new results in our new revision.

The training time is evaluated to achieve 81% accuracy on GTSRB using ViT-16 models.