PRISM: Privacy-Preserving Improved Stochastic Masking for Federated Generative Models

R4-8. Response to Concerns About Dataset and Experimental Setup [R4-W3]

We sincerely appreciate your feedback, which reflects a clear understanding of the fundamental principles of FL setups. However, we were unsure about certain aspects of your comment and would greatly appreciate any clarification regarding your concerns. You reference our statement that "distributed samples cannot be shared" to explain why centralized comparisons were excluded, but also suggest that our datasets are "mature and simple" enough for centralized training on modern laptops. This seems to conflict with the assumptions of the FL setup, where the full dataset is not accessible at each device, requiring collaboration among the clients to build a reliable model. Furthermore, your statement that "if the evaluated methods cannot even beat the central model, it is difficult to see the contribution being made" adds further ambiguity. Are you concerned that PRISM underperforms in centralized setups where the full dataset is accessible? If so, Appendix G demonstrates that PRISM performs well in such scenarios. Here, if a single laptop is assumed to have access to the entire dataset (e.g., full MNIST dataset), this scenario is considered as a centralized setting, unlike the FL scenario where each client has access to only a limited number of data samples.

First, to clarify your question, "Why do PRISM and the baselines in the paper fail to properly generate MNIST digits in Figures 2 and 3?" the observed challenges arise not from limitations of PRISM itself but from the stringent constraints imposed by the FL+DP (Federated Learning with Differential Privacy) setup we model. While generating MNIST digits is indeed trivial in centralized setups—even on a single laptop—this task becomes highly challenging under FL+DP conditions [1-4]. For example, as shown in Figure 4 of [1], generating MNIST digits while adhering to privacy constraints remains a significant challenge, even in centralized settings.

Second, regarding the evaluated datasets, we would like to emphasize that MNIST, CIFAR-10, and similar benchmarks are commonly used in FL literature to simulate challenging and constrained scenarios with limited data at each client and non-IID data distribution [5-7]. While it is true that these datasets can be fully utilized on a laptop for centralized training, the FL paradigm assumes that data is inherently distributed across multiple devices, and full datasets cannot be shared. This distinction is key: FL setups explicitly model environments where no single device or server has access to the entire dataset, which aligns with real-world applications. Thus, using subsets of datasets such as MNIST and CIFAR-10 is a standard benchmarking practice to evaluate methods under these constraints.

...

Please refer the following responses [Response 2/3]

R4-6. Difference between PRISM and prior works [R4-W1]

We acknowledge that there is some overlap with prior works. However, as R3 also noted, the value of our work lies in effectively combining these methods to address previously unsolved challenges in federated learning for generative models. PRISM demonstrates impressive performance in scenarios where existing baselines fail, successfully tackling many of the prevailing difficulties in this field. In addition, we believe MADA, which is introduced for the first time in our work, constitutes a meaningful contribution by significantly improving performance and providing a novel approach to tackling heterogeneity in federated learning environments. As observed in our experiments, MADA consistently improved performance across datasets, with the improvement being more pronounced under non-IID conditions compared to IID conditions. For example, the FID scores for Non-IID setups showed substantial improvement after applying MADA:

• MNIST: Before: 49.6273 → After: 34.2038
• FMNIST: Before: 83.0481 → After: 67.1648
• CelebA: Before: 59.4877 → After: 39.7997

These improvement is even greater than those observed under IID conditions:

• MNIST: Before: 48.5636 → After: 27.3017
• FMNIST: Before: 54.722 → After: 46.1652
• CelebA: Before: 57.0573 → After: 48.9983

These results suggest that MADA is effective in addressing the challenges posed by both IID and non-IID settings.

We hope the reviewer recognizes these contributions and the advancements they represent in this domain.

R4-7. Details in architecture and hyperparameters of PRISM [R4-W4]

Thank you for your constructive comment. We used a ResNet-based decoder and empirically observed that setting the local epoch to 100 and learning rate to 0.1 effectively optimizes the local stochastic binary masks. All of the detailed instructions have been added to Appendix.C, highlighted in “green”.

R4-1. Additional experiments on different random initialization or seed [R4-Q1]

We conducted additional experiments across various seed values to provide further performance insights. As can be seen in the table below, PRISM shows robust performance regardless of specific initialization value or random seed with identical hyperparameters.

R4-2. Clarification on the comments: “Figures 2/3 for PRISM and baselines look problematic.” & “CIFAR10 and CelebA are not complex datasets"[R4-W2, R4-W3]

We understand your concern that demonstrating performance on simpler datasets, such as MNIST, may appear less impactful as an empirical result. However, it is important to note that the field of federated generative models remains far behind the current trend of using more complex and powerful models, such as diffusion models, and large-scale datasets. This gap arises from the inherent difficulties in adapting these advanced models to FL setups.

Existing works have struggled to achieve stable performance in challenging setups, such as Non-IID and privacy-preserving cases, even with MNIST-level datasets. Even for centralized differential privacy generative models [1-2],training generative models is highly challenging and often unstable. Due to these obstacles, prior studies have primarily focused on simpler datasets, such as MNIST, and have been largely restricted to GANs. However, even under these conditions, many of these works failed to achieve desirable results. We summarize the FID scores and experimental setups reported in these works.

In contrast, PRISM is the first to consistently demonstrate stable performance under these conditions.

The purpose of our experiments was to highlight the limitations of prior approaches and showcase PRISM’s effectiveness in overcoming these challenges. Nevertheless, we acknowledge your doubts and have included additional experimental results below to further support our claims.

Still, we also recognize the importance of assessing performance on high-resolution and large-scale datasets. Considering computational resources and the rebuttal period, we conducted experiments on the CelebA 128x128 and CIFAR100 datasets under Non-IID conditions without considering privacy. The results are summarized in the table below, with qualitative results provided in Appendix.I. Please note that PRISM is the first to achieve this level of performance under the current setting, whereas existing baselines have struggled even on MNIST-level benchmarks.

[1] Dockhorn, Tim, et al. "Differentially private diffusion models." arXiv preprint arXiv:2210.09929 (2022).

[2] Jiang, Zepeng, Weiwei Ni, and Yifan Zhang. "PATE-TripleGAN: Privacy-Preserving Image Synthesis with Gaussian Differential Privacy." arXiv preprint arXiv:2404.12730 (2024).

[3] Hardy, Corentin, Erwan Le Merrer, and Bruno Sericola. "Md-gan: Multi-discriminator generative adversarial networks for distributed datasets." 2019 IEEE international parallel and distributed processing symposium (IPDPS). IEEE, 2019.

[4] Zhang, Yikai, et al. "Training federated GANs with theoretical guarantees: A universal aggregation approach." arXiv preprint arXiv:2102.04655 (2021).

[5] Amalan, Akash, et al. "Multi-flgans: multi-distributed adversarial networks for non-IID distribution." arXiv preprint arXiv:2206.12178 (2022).

...

Please refer the remaining responses in the [Respones 2/3].

We appreciate the reviewer for acknowledging our efforts and raising the score.

Below, we present additional experiments conducted with different learning rates in response to the reviewer’s comment.

R4-12. Explore various learning rates for baselines

To validate the fairness of our experiments, we explored various configurations of learning rates for the baselines. Since they utilize GANs, there are separate learning rates for the generator and discriminator, and we examined these combinations. In the tables below, the top row represents the discriminator’s lr / generator’s lr, while the subsequent row reports the FID scores. Due to the notorious instability of GAN training caused by the coupled learning dynamics between the generator and discriminator [1-3], we observed divergence in some settings. For GS-WGAN, the best performance was achieved with 1e-4 / 1e-5, but it still failed to generate MNIST images properly. We include these results on Appendix L in our main paper.

[1] Miyato, Takeru, et al. "Spectral normalization for generative adversarial networks." arXiv preprint arXiv:1802.05957 (2018). [2] Mescheder, Lars, Andreas Geiger, and Sebastian Nowozin. "Which training methods for GANs do actually converge?." International conference on machine learning. PMLR, 2018. [3] Mao, Xudong, et al. "Least squares generative adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.

R3-1. Further explanation of MADA

(1) Is MADA derived from other literature, or is this proposed here for the first time? [R3-Q1]

To the best of our knowledge, MADA, which determines the moving average ratio on the server side based on the similarity of the global model, is proposed here for the first time.

(2) Is MADA convergence due to learning rate decay? [R3-Q4]

First, we would like to clarify that we did not use learning rate decay throughout the FL process. Thus, the local model continues to update consistently using a constant learning rate. During the FL process, the decrease in $\lambda$ is due to the convergence of the binary mask, which reduces the similarity between the two global models, rather than being a result of learning rate decay.

R3-2. Communication cost savings of PRISM [R3-Q2, R3-W2]

PRISM transmits a float-type score during downlink, which, as you noted, does not support efficient communication, unlike the binary mask transmission in the uplink. In a typical FL setup, the server possesses powerful computational capabilities, whereas the clients do not. As a result, several prior studies primarily focus on addressing the limited bandwidth of clients [1-3]. Accordingly, we have concentrated on the communication cost in the downlink. We will clarify this in our manuscript (Appendix.J, highlighted in “lime”).

R3-3. Memory savings of frozen model [R3-Q3]

Our memory-saving approach is akin to ternary quantization [4], a technique that quantizes neural network weights to {1, 0, -1} through thresholding and projection, making the sign the critical factor in the process. Similar but distinct, PRISM's frozen weights are already initialized to signed constant using Kaiming normal distribution standard deviation, $\sigma_l$. Consequently, after FL concludes, each client can achieve additional memory savings by storing not only the lightweight binary mask but also the frozen weights with the scaling factor {$\sigma_1$, …, $\sigma_l$} and a 1-bit value representing the sign. We hope our clarification resolves any confusion, but let us know if further explanation is needed.

R3-4. Clarification on Non-IID Empirical Settings and Additional experiments with Dirichlet Non-IID split [R3-W3, R3-Q5]

As suggested, we conducted additional experiments under Non-IID splitting by Dirichlet distribution ($\alpha=0.005$). We report the detailed results in the table below. PRISM continues to outperform other baselines in both Non-IID with/without DP setups, demonstrating its robustness.

We would also like to address the concern (potential misunderstanding) regarding the Precision and Recall (P&R) results in our original experiments. While FID scores, which are widely regarded as a reliable metric for generative models, exhibit clear trends under non-IID settings, interpreting P&R and D&C metrics requires caution due to their inherent characteristics. For instance, fidelity metrics like Precision can yield inflated values under mode collapse, where many generated samples concentrate around a single real sample. Similarly, outliers in the generated data can disproportionately impact P&R scores, leading to variability and potentially misleading results, as highlighted in prior studies [5]. These characteristics can explain why P&R and D&C results appear similar to or even exceed those of IID settings, despite being evaluated under a Non-IID scenario.

...

remaining responses will be posted soon.

[1] Kim, Do-Yeon, et al. "Achieving Lossless Gradient Sparsification via Mapping to Alternative Space in Federated Learning." Forty-first International Conference on Machine Learning.

[2] Hu, Rui, Yuanxiong Guo, and Yanmin Gong. "Federated learning with sparsified model perturbation: Improving accuracy under client-level differential privacy." IEEE Transactions on Mobile Computing (2023).

[3] Yi, Liping, Wang Gang, and Liu Xiaoguang. "QSFL: A two-level uplink communication optimization framework for federated learning." International Conference on Machine Learning. PMLR, 2022.

[4] Liu, Dan, and Xue Liu. "Ternary Quantization: A Survey." arXiv preprint arXiv:2303.01505 (2023).

[5] Naeem, Muhammad Ferjad, et al. "Reliable fidelity and diversity metrics for generative models." International Conference on Machine Learning. PMLR, 2020.

...

Please refer the remaining responses in the [Respones 2/2].

R2-1. Scalability and performance on large-scale datasets and larger model [R2-W1]

We would like to emphasize that recent FL studies often lack discussions on high-resolution and large-scale datasets. As highlighted in the related work section of our main script [3-6] and in recent studies on privacy-preserving generative models [1-2], this reflects the inherent difficulty of training generative models in unstable FL setups. For this reason, prior works have predominantly focused on such experimental environments, with most achieving only MNIST-level generation. Nevertheless, we recognize the importance of addressing whether robust performance can be achieved on high-resolution and large-scale datasets. Considering computational resources and the rebuttal period, we conducted experiments on the CelebA 128x128 and CIFAR100 datasets under Non-IID local data distributions without privacy considerations. Quantitative results are reported in the table below, and qualitative results are provided in Appendix I. It is noteworthy that existing baselines have failed not only in these settings but also on MNIST-level benchmarks. We are the first to achieve this level of performance under the current conditions. To further address the scalability of PRISM, we will update the main text with relevant discussions and respond accordingly once the content has been revised.

R2-2. Computational cost of SLT process [R2-W2]

As the number of model parameters increases, the required score parameters also increase linearly. However, in SLT, weight parameters are not learned through gradient descent. The only additional components are the sigmoid and Bernoulli processes introduced during the binary masking procedure. While this does incur some additional computational cost, it is negligible compared to the cost of gradient descent operations on GPUs.

To further substantiate this claim, we are conducting experiments to compare the FLOPS (floating-point operations per second) with and without the binary masking process. Detailed results will be provided as soon as the experiments are completed.

[1] Dockhorn, Tim, et al. "Differentially private diffusion models." arXiv preprint arXiv:2210.09929 (2022).

[2] Jiang, Zepeng, Weiwei Ni, and Yifan Zhang. "PATE-TripleGAN: Privacy-Preserving Image Synthesis with Gaussian Differential Privacy." arXiv preprint arXiv:2404.12730 (2024).

[3] Hardy, Corentin, Erwan Le Merrer, and Bruno Sericola. "Md-gan: Multi-discriminator generative adversarial networks for distributed datasets." 2019 IEEE international parallel and distributed processing symposium (IPDPS). IEEE, 2019.

[4] Zhang, Yikai, et al. "Training federated GANs with theoretical guarantees: A universal aggregation approach." arXiv preprint arXiv:2102.04655 (2021).

[5] Amalan, Akash, et al. "Multi-flgans: multi-distributed adversarial networks for non-IID distribution." arXiv preprint arXiv:2206.12178 (2022).

[6] Li, Wei, et al. "Ifl-gan: Improved federated learning generative adversarial network with maximum mean discrepancy model aggregation." IEEE Transactions on Neural Networks and Learning Systems 34.12 (2022): 10502-10515.

R1-1. Baseline performance of purely server-side SLT [R1-Q1]

We understand that your inquiry is about the performance of PRISM under a centralized setting. In Table 6 of Appendix G in the main paper, we compared PRISM (Non-IID and privacy-preserving) with PRISM-vanilla (centralized setting). For your convenience, we present the same results in the table below. As shown in prior studies [1-3], centralized SGD typically outperforms federated settings, and our findings align with this trend.

R1-2. Would the approach work for diffusion-based generative models? [R1-Q2]

Yes, adopting diffusion models (DDPM) [9] within PRISM is conceptually feasible. However, directly applying the MMD loss-based approach used in PRISM, which requires generating a set of samples for each update, poses practical challenges due to the well-known limitations of DDPM, including slow sampling and convergence rates. These limitations lead to significant computational costs, making the approach practically infeasible. Nevertheless, given the importance of diffusion models in generative modeling, we explored a modified version of our framework. Specifically, we retained the core concept of identifying SLT but replaced the MMD loss with diffusion loss to guide score updates. We conducted preliminary experiments on the MNIST dataset under a Non-IID, privacy-free scenario, and intermediate results are provided in Appendix H. While this approach is inherently sub-optimal—since it updates scores based on the loss of individual samples rather than the sample statistics—it still demonstrates the ability to generate recognizable shapes. With further tuning, we anticipate more promising results. However, as this lies beyond the scope of our current paper, it remains an interesting direction for future research. It is worth emphasizing that, to the best of our knowledge, this is the first federated generative model capable of generating data at this level of quality in a Non-IID, privacy-preserving setting for datasets more complex than MNIST or FMNIST. We hope the reviewers recognize the significance of this achievement.

...

Please refer the remaining responses in the [Respones 2/2].

R1-3. Addressing the Concerns on Model and Dataset Complexity in Federated Learning [R1-W1]

We would like to emphasize the significant challenges of training generative models under a federated learning (FL) setup. The field of federated generative models remains far behind the current trend of using more complex and powerful models, such as diffusion models, and large-scale datasets. This gap arises from the inherent difficulties in FL setups. Even for centralized differential privacy generative models [4-5], as also highlighted in the related work of our main script, training generative models is highly challenging and often unstable. Due to these obstacles, prior studies have primarily focused on simpler datasets, such as MNIST, and have been largely restricted to GANs. However, even under these conditions, many of these works failed to achieve desirable results. We summarize the FID scores and experimental setups reported in these works.

While adapting our approach to models like diffusion models is beyond the scope of this paper (as noted in our response to R1-2, Appendix H), we recognize the importance of assessing performance on high-resolution and large-scale datasets. Considering computational resources and the rebuttal period, we conducted experiments on the CelebA 128x128 and CIFAR100 datasets under Non-IID conditions without considering privacy. The results are summarized in the table below, with qualitative results provided in Appendix.I. Please note that PRISM is the first to achieve this level of performance under the current setting, whereas existing baselines have struggled even on MNIST-level benchmarks.

[1] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.

[2] Zhao, Yue, et al. "Federated learning with non-iid data." arXiv preprint arXiv:1806.00582 (2018).

[3] Hardy, Corentin, Erwan Le Merrer, and Bruno Sericola. "Md-gan: Multi-discriminator generative adversarial networks for distributed datasets." 2019 IEEE international parallel and distributed processing symposium (IPDPS). IEEE, 2019.

[4] Dockhorn, Tim, et al. "Differentially private diffusion models." arXiv preprint arXiv:2210.09929 (2022).

[5] Jiang, Zepeng, Weiwei Ni, and Yifan Zhang. "PATE-TripleGAN: Privacy-Preserving Image Synthesis with Gaussian Differential Privacy." arXiv preprint arXiv:2404.12730 (2024).

[6] Hardy, Corentin, Erwan Le Merrer, and Bruno Sericola. "Md-gan: Multi-discriminator generative adversarial networks for distributed datasets." 2019 IEEE international parallel and distributed processing symposium (IPDPS). IEEE, 2019.

[7] Zhang, Yikai, et al. "Training federated GANs with theoretical guarantees: A universal aggregation approach." arXiv preprint arXiv:2102.04655 (2021).

[8] Amalan, Akash, et al. "Multi-flgans: multi-distributed adversarial networks for non-IID distribution." arXiv preprint arXiv:2206.12178 (2022).

[9] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.