One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models

We sincerely appreciate your recognition of our work and your valuable feedback. Below, we provide detailed responses to the key concerns you raised:

(Essential References Not Discussed) "BN loss has been widely used, for example, [Yin'2020], while the related citations are missing."

Thank you for pointing this out. In the revised version of our manuscript, we will incorporate relevant references[1,2,3] to ensure a more comprehensive discussion of BN Loss and its applications.

(Generality of BN Loss) Weaknesses #1 "BN loss only works with models with BN layers, while it does not work on others such as transformers or models for other data modalities."

We refer to the loss function in our study as BN Loss primarily because BN has been widely adopted, as demonstrated in prior works [1,2,3]. However, the core mechanism of BN Loss involves leveraging statistics, such as mean and variance, which are also present in other normalization layers such as Layer Normalization [4]. Notably, Layer Normalization has been extensively used in transformer, including models like CLIP [5]. Therefore, our method is not strictly limited to models with BN layers but can be extended to transformers and other architectures employing various normalization layers.

(Further Explanation of Theorem 1) Weaknesses #1 "The explanation of theorem 1 is not enough."

• "Why is $\mathbb{E}(\log p\_{\epsilon\_\theta}(\boldsymbol{\theta}\_k))$ a constant?"

This term is solely dependent on the parameters of the diffusion model, which remain fixed during our method. Consequently, $\mathbb{E}(\log p_{\epsilon_\theta}(\boldsymbol{\theta}\_k))$ can be considered a constant.

• "What does $p\_{\epsilon_\theta}(\boldsymbol{\theta}\_k)$ represent?"

This term represents the default data distribution of the diffusion model. After extensive pretraining, $p\_{\epsilon\_\theta}(\boldsymbol{\theta}\_k)$ approximates the data distribution of the diffusion model’s pre-train dataset.

• "Why is that minimizing the negative log-likelihood is equivalent to minimizing the cross-entropy loss?"

The negative log-likelihood and the cross-entropy loss are formally equivalent. When the target distribution is a one-hot distribution, maximizing the likelihood corresponds to minimizing the cross-entropy loss. This equivalence is widely utilized in deep learning, particularly in the loss functions of softmax-based classification tasks[6].

(Typos) Other Comments Or Suggestions

We appreciate your meticulous attention to detail in identifying typographical errors. These will be corrected in the revised manuscript. Additionally, we will conduct a thorough review to ensure the clarity and accuracy of our presentation.

Once again, thank you for your insightful review and constructive feedback. We look forward to any further comments you may have.

[1] Dreaming to distill: Data-free knowledge transfer via deepinversion, CVPR 2020.
[2] Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? NIPS 2024.
[3] Source-Free Domain Adaptation for Semantic Segmentation, CVPR 2021.
[4] Layer Normalization, NIPS 2016.
[5] Learning transferable visual models from natural language supervision, ICML 2021.
[6] Machine learning: a probabilistic perspective, MIT press 2012.

[1] FedDEO: Description-Enhanced One-Shot Federated Learning with DMs, MM 2024.
[2] One-Shot Federated Learning with Classifier-Free Diffusion Models, ICME 2025.

Thank you for recognizing our work. Below, we provide detailed responses to your concerns:

(Server Cost) Questions #3"... server computing costs."

In FL, the server is generally designed to have sufficient resources to handle the model aggregation, but clients often exhibit significant heterogeneity, necessitating constraints on costs[1]. Our method adheres to this principle by reducing client burdens.

Additionally, the following table presents a comparision of the server computation costs with other DM-based FL methods. Our method employs local models rather than additional prompts for guiding generation, eliminating the need for compositional diffusion, imposing a lower server computation cost.

(Additional Ablation Experiments) Questions #4 "… vary with dataset size."

We appreciate the reviewer’s suggestion. The following table show that increasing the number of generated images leads to an improvement in the performance. Moreover, we observe that the improvement does not saturate as the dataset size increases, further demonstrating the diversity of the synthetic dataset.

(Experimental Analysis) Weaknesses #2 "FedLMG_SD should give the best result ... no relevant analysis."

We would like to clarify that we include a dedicated analysis in line 937. We argue that due to the varying architectures of client models, some clients with more complex model structures demonstrate superior performance, allowing them to provide more accurate knowledge than the specific teacher in FedLMG_SD.

(Privacy-Related Experiments): Weaknesses #3 "… the threshold chosen is subjective."

We appreciate your suggestion, to further verify our method’s effectiveness in privacy protection, we employe [2] to ensure differential privacy and evaluate its impact on model performance. The results in the table below indicate that since our method only involves uploading local models, aligning with traditional FL, most privacy preserving methods in FL can be directly applied to our method without significantly degrading model effectiveness.

(Distribution Similarity) Questions #1 "Boundedness lacks quantitative validation ... client data distributions differ significantly..."

Quantitatively evaluating the similarity between the default distribution of a DM and client distributions is challenging, particularly given the large-scale pre-training datasets of DM. However, with the recent advancements, pre-trained DMs tailored for various domains[3,4] have become increasingly available. We believe that servers can select appropriate DMs based on the target application. Even where the data distribution is challenging, a pre-trained DM from a similar domain can be fine-tuned on the server. Thus, We assert that our method possesses practicality in diverse application scenarios.

(BN Loss) Weaknesses #1 "... BN loss lacks a strong theoretical basis."

As noted by Reviewer #xTsg, BN Loss has been widely applied[5]. Since BN Loss compares statistics, it provides an intuitive guidance and is adopted in studies such as [5,6] without additional theoretical analysis. We plan to further explore its theoretical underpinnings in future to strengthen our method’s theoretical foundation.

(Effectiveness of Guidance) Questions #2"... lacks quantitative analysis of the guidance effectiveness."

We would like to clarify that we provide quantitative analyses of the effectiveness of guidance. The Prompts Only in Table 1 represent results where no local model guidance was used, whereas FedLMG denotes results with guidance. We believe that the comparison between these settings sufficiently demonstrates the effectiveness of the guidance in our method.

[1] A survey on federated learning: challenges and applications, IJMLC 2023.
[2] Federated Learning with Differential Privacy: Algorithms and Performance Analysis, TIFS 2020.
[3] Diffusion probabilistic models for 3d point cloud generation, CVPR 2021.
[4] DiffuSeq: Sequence to Sequence Text Generation with DMs, ICLR 2023.
[5] Dreaming to distill: Data-free knowledge transfer via deepinversion, CVPR 2020.
[6] Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? NIPS 2024.

We appreciate your positive comments of our work and address each of your concerns as follows:

(Server Cost) Claims #1 & Questions #3: "… DMs on the server, which can be an additional computation cost"

In FL, the server is generally designed to have sufficient resources to handle the aggregation of client models, but clients often exhibit significant heterogeneity, necessitating constraints on computation costs[1]. Our method adheres to this principle by reducing client burdens.

Additionally, the following table presents a comparision of the server computation costs with other DM-based FL methods. Our method employs local models rather than additional prompts for guiding generation, eliminating the need for compositional diffusion, imposing a lower server computation cost.

(Uncertain Clients) Claims #2: "… we do not know if the clients’ data are in specialized domains"

The setting that the specific client tasks is uncertain for the server is Many-Task FL[2], where clients have diverse tasks simultaneously. However, this setting is not common. Most FL research presumes that the client task is known[1,3]. Therefore, we consider that our task setting is reasonable in real-world applications.

(Applicability) Claims #2 & Weaknesses #2 & Questions #2: "… may not have the data and computation resources to train a specialized DM."

Currently, pre-trained DMs have been widely applied across various domains[4,5,6]. Based on the above discussion, we posit that the server know the overall task and can select an appropriate DM. Even where the data distribution is challenging, a pre-trained DM from a similar domain can be fine-tuned on the server. We assert that our method possesses practicality in diverse application scenarios.

(References) Essential References: "... similar idea of using guidance for DM generation in OSFL are missing …"

We appreciate your suggestion. We incorporate the relevant works into our compared methods. As shown in the table below, our method achieves comparable performance without the employment of any foundational models on the clients, further demonstrating the performance of our method.

(BN Loss) Weaknesses #1: "… whether the proposed BN loss is the most suitable choice"

As noted by Reviewer #xTsg, BN Loss has been widely utilized[7]. Although there is no prior precedent for employing BN Loss in the DMs, the general method of designing task-specific loss functions and guiding the diffusion process is well established[8]. While we acknowledge that BN Loss might not be the optimal choice in all circumstances, its effectiveness has been clearly validated by the ablation experiments presented in Table 4.

(Privacy) Questions #1: "… applying noisy SGD during client model training offer stronger privacy protection"

We appreciate your suggestion. Since our method only involves uploading local models, aligning with traditional FL, most privacy preserving methods in FL can be directly applied to our method. To validate this, we adopt [8] to ensure differential privacy and evaluate its impact on model performance. The experimental results shown in the table below indicate that traditional FL privacy protection measures remain effective within our framework without significantly degrading model performance.

[1] A survey on federated learning: challenges and applications, IJMLC 2023.
[2] Many-Task Federated Learning: A New Problem Setting and A Simple Baseline, CVPR 2023.
[3] A survey on federated learning systems: Vision, hype and reality for data privacy and protection, TKDE 2021.
[4] DMs in medical imaging: A comprehensive survey, MIA 2023.
[5] DiffuSeq: Sequence to Sequence Text Generation with DMs, ICLR 2023.
[6] DiffWave: A Versatile DM for Audio Synthesis, ICLR 2021.
[7] Dreaming to distill: Data-free knowledge transfer via deepinversion, CVPR 2020.
[8] Federated Learning with Differential Privacy: Algorithms and Performance Analysis, TIFS 2020.

We sincerely appreciate your review and valuable comments and provide detailed responses to the key concerns:

(Privacy Concerns) Claims And Evidence #1 & Weaknesses "general quantitative metrics for privacy of FL, such as Gradient Leakage (GL) or Differential privacy (DP)."

Regarding GL, as discussed in [1] and [2], such attacks occur during the sharing of gradients , where attackers infer client data by analyzing gradients. However, our method does not involve sharing gradients, mitigating the risk of GL.

Regarding DP, because our method only involves uploading local models, aligning with traditional FL, most DP preserving methods in FL can be directly applied to our method. To further validate this, we incorporated the method proposed in [3] to ensure DP and assessed its impact on performance. The following table demonstrate that the DP preserving methods remain applicable to our method without significantly compromising its effectiveness.

(Server Computation Cost) Claims And Evidence #2 & Methods And Evaluation Criteria #2: "... much more burdens to the server."

In FL, the server is generally designed to have sufficient resources to handle the aggregation of client models, but clients often exhibit significant heterogeneity, necessitating constraints on computation costs[5]. Our method adheres to this principle by reducing client burdens.

Additionally, the following table presents a comparision of the server computation costs with other DM-based FL methods. Our method employs local models rather than additional prompts for guiding generation, eliminating the need for compositional diffusion, imposing a lower server computation cost.

(Additional Ablation Experiments) Experimental Designs Or Analyses #1: "More ablation experiments are needed to prove the effectiveness of the proposed method."

We appreciate the reviewer’s suggestion. The following table show that increasing the number of generated images leads to an improvement in the performance. Moreover, we observe that the improvement does not saturate as the dataset size increases, further demonstrating the diversity of the synthetic dataset.

(Multimodality) Methods And Evaluation Criteria #1: "... how the proposed method will perform on non-image and other modality datasets."

Similar to many FL studies[5], we select image as the primary modality in our paper. However, our method is not restricted to the image modality. By utilizing DMs for other modalities such as [6, 7], our method can be seamlessly adapted to other modalities.

(Dataset Heterogeneity) Experimental Designs Or Analyses #2: "The impact of the heterogeneity of datasets in the paper should be mentioned."

Dataset heterogeneity in FL primarily manifests as feature distribution skew and label distribution skew [8]. In Tables 1 and 10, we demonstrate the impact of both types of heterogeneities. Therefore, we respectfully disagree with this concern.

(Contribution of the Paper) Relation To Broader Scientific Literature: "The key contribution of the paper is in the knowledge aggregation part..."

As stated in the Introduction and acknowledged by Reviewer #A93d and #5oFb, a key advantage of our method is eliminating the need for foundation model inference on clients. We believe this characteristic significantly enhances the practicality of diffusion-based FL methods and represents a meaningful contribution to the field.

[1] Deep leakage from gradients, NIPS 2019.
[2] Understanding Deep Gradient Leakage via Inversion Influence Functions, NIPS 2023
[3] Federated Learning with Differential Privacy: Algorithms and Performance Analysis, TIFS 2020.
[4] FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models, MM 2024.
[5] A survey on federated learning: challenges and applications, IJMLC 2023.
[6] DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models, ICLR 2023.
[7] DiffWave: A Versatile Diffusion Model for Audio Synthesis, ICLR 2021.
[8] Federated Learning on Non-IID Data Silos: An Experimental Study, ICDE 2022.