How Distributed Collaboration Influences the Diffusion Model Training? A Theoretical Perspective

We thank the Reviewer YCna for the time and valuable feedback! We would try our best to address the comments one by one.

Response to “Theoretical Claims”: We agree that supplementing technical proofs with intuitive explanations greatly enhances accessibility. And we have added some explanation to the key results. For example, regarding the reverse SDE derivation, we now clarify that reversing the SDE entails “rewinding” the forward process by inverting the drift term, which is conceptually consistent with the notion of reconstructing the data distribution from noise.

Response to “Experimental Designs Or Analyses”: We have added a discussion regarding the limitations and future directions in the appendix. Specifically, we offer the following reflections: While this work provides a theoretical perspective for distributed diffusion models under resource constraints, real-world constraints are often more complex. These constraints include heterogeneity in computational power, memory availability, communication latency, and the frequency of parameter updates. Such variability can pose significant challenges for maintaining model quality. Extending our framework to explicitly model these factors—such as by incorporating asynchronous optimization or adaptive, resource-aware pruning—represents an important direction for future research.

Response to “Other Weaknesses 1” ＆ “Other Comments Or Suggestions 2”: We have added a detailed derivation for Remark 4.12 in the Appendix. For more details, you can refer to our response to Reviewer ySb9.

Response to “Other Weaknesses 2” ＆ “Other Comments Or Suggestions 1”: We have added a discussion on the important theoretical results, limitations and future directions in the appendix. The details are as follows:

Our analysis relies on several standard assumptions commonly adopted in the distributed learning literature, such as bounded gradient variance. While these assumptions facilitate tractable theoretical analysis, they may not fully capture the complexities of real-world distributed systems.

In our theoretical results, the term $w^2$ reflects the extent of pruning applied to the distributed model. As pruning becomes more aggressive (i.e., fewer parameters are retained), this term increases, leading to a looser error bound. The parameter $\\Gamma ^*$ captures the frequency with which each model parameter is updated across the distributed workers. A small $\\Gamma^{*}$ indicates that some parameters are rarely trained, which may lead to suboptimal or imbalanced updates and thus a larger bound. This motivates us to seek more effective pruning strategies that achieve a balance between resource availability and generation quality, which we leave for the future.

While this work provides a theoretical perspective for distributed diffusion models under resource constraints, real-world constraints are often more complex. These constraints include heterogeneity in computational power, memory availability, communication latency, and the frequency of parameter updates. Such variability can pose significant challenges for maintaining model quality. Extending our framework to explicitly model these factors—such as by incorporating asynchronous optimization or adaptive, resource-aware pruning—also represents an important direction for future research.

Response to “Questions For Authors 1”: While our error bound shares a structural similarity with those in single-worker settings—relying on factors such as model dimension—there are key differences in how errors accumulate. Specially, distributed systems introduce additional challenges due to inter-worker heterogeneity and uneven parameter training. These are captured in our analysis by terms like $\\sigma _ 2^2$ and $\\Gamma^*$, which do not appear in single-worker settings. Moreover, pruning may lead to imbalanced updates, which is quantified by the terms $w^2$ and $\\Gamma^*$ in our bound. In this context, the assumption of perfect score function estimation—often made in single-worker analyses—no longer holds. The key contribution of our theoretical framework is to explicitly characterize new errors that are unique to distributed generative modeling.

Response to “Questions For Authors 2”: We agree that identifying real-world applications can help illustrate our theoretical contributions. However, to maintain the paper’s main theoretical focus, we refrain from expanding on specific applications within the main text. Nonetheless, we briefly highlight several potential application areas where our theoretical findings may provide guidance: (1) Federated generative modeling in privacy-sensitive domains such as healthcare and finance. (2) Collaborative generation across edge devices for generative tasks like image synthesis or speech enhancement, where each device has limited compute/memory.

If there are any further confusions, we are happy to clarify them. Thank you again for your recognition of our work.

We thank the Reviewer ySb9 for the time and valuable feedback! We would try our best to address the comments one by one.

Response to “Essential References Not Discussed”: We appreciate the reviewer’s insightful suggestion. While our work primarily focuses on theoretical contributions, we agree that incorporating discussions of experimental works in distributed diffusion models can further underscore the practical relevance of our analysis. In response, we have cited some works on the empirical performance of distributed diffusion models, such as Phoenix and FedDM. We believe these additions enhance the manuscript.

Response to “Other Weaknesses 1” ＆ “Other Comments Or Suggestions 1”: We appreciate the reviewer’s comment on the balance between mathematical rigor and intuitive clarity. In response, we provide a few brief intuitive explanations. For the reverse SDE derivation, we now clarify that reversing the SDE involves “rewinding” the forward process by inverting the drift, which conceptually aligns with the idea of reconstructing the data distribution from noise. For the local loss bound (Lemma 4.7), it is directly derived based on the fact that the average gradient norm is non-negative. We hope these explanations address your concerns.

Response to “Other Weaknesses 2” ＆ “Other Comments Or Suggestions 2” ＆ “Questions For Authors 3”: We appreciate the reviewer’s insightful suggestion. To make it clearer how the results in Remark 4.12 are established, we have added a detailed derivation in the Appendix. Here are the details: For $T\ge 1$, $\delta < 1$, $K \ge \log(1/\delta)$, if we set $\kappa = \Theta \left(\frac{T + \log(1/\delta)}{K}\right)$, then there obviously exists a sequence $\\{t _ k\\} _ {k=0}^K$ such that $\gamma_k \le \kappa \min \\{1, T-t_{k+1}\\}$. Then, if we set $K=\Theta \big(\frac{(d+M_{n,2})(T+\log (1/\delta))^2}{F(\theta_0)}\big)$, it holds

If we set $T=\frac{1}{2}\log \big(\frac{d+M_{n,2}}{F(\theta_0)}\big)$, it holds that $(d+M_{n,2})e^{-2T}=F(\theta_0)$. Then we have $\kappa^2 d K+\kappa M_{n,2}+\kappa dT+(d+M_{n,2})e^{-2T}\lesssim F(\theta_0)$.

Similarly, if we further control the learning rate to satisfy $\eta \le\\{\frac{F(\theta_0)\Gamma^*}{SRN(\sigma_1^2+\sigma_2^2)},\frac{F(\theta_0)\Gamma^*}{SRNw^2 L},\sqrt{\frac{F(\theta_0)(\Gamma^*)^2}{SRNL\sigma_1^2}}\\}$, we have $\frac{\eta SR w^2 LN}{\Gamma^*}+\frac{\eta SRN(\sigma_1^2+\sigma_2^2)}{\Gamma^*}+\frac{\eta^2 SRLN\sigma_1^2}{(\Gamma^*)^2}\lesssim F(\theta_0)$.

These results complete the proof. And we hope these explanations address your concerns.

Response to “Questions For Authors 1-2”: Thank you for these thoughtful questions. For your concern about the constant $C_1$, we believe that the current version is correct. In fact, $C_1$ is generated due to the equivalent denoising score matching, which we have discussed in Lemma 4.8. Therefore, this constant is not included in the bound of local loss $F_n(\theta_R)$. For your concern about $\sigma_2$, it is indeed equal to zero and can be omitted. Thank you for your careful consideration, we have simplified it in the revised version. For your concern about $w^2$ and $\\Gamma^{*}$, the term $w^2$ reflects the extent of pruning applied to the distributed model. As pruning becomes more aggressive (i.e., fewer parameters are retained), this term increases, leading to a looser error bound. The parameter $\\Gamma^*$ captures the frequency with which each model parameter is updated across the distributed workers. A small $\\Gamma^*$ indicates that some parameters are rarely trained, which may lead to suboptimal updates and thus a larger bound. This motivates us to seek more effective pruning strategies that achieve a balance between resource availability and generation quality, which we leave for the future.

If there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.

We thank the Reviewer grcd for the time and valuable feedback! We would try our best to address the comments one by one.

Response to “Other Weaknesses 1” ＆ “Other Comments Or Suggestions 1”: We thank the reviewer for this constructive feedback. We agree that the proofs in Lemma 4.6 and Theorem 4.11 could benefit from additional intuition and visual aids. In response, we have enhanced the manuscript by adding more detailed explanations that clarify the logical steps. These revisions are intended to make the technical sections more accessible to a broader audience while preserving the rigor of our arguments. We believe these improvements will significantly enhance the readability and overall presentation of our work.

Response to “Other Weaknesses 2”: We appreciate the reviewer’s observation. Indeed, while assumptions like synchronized updates and bounded variance are common in theoretical analyses, we recognize that these assumptions may be overly idealized when compared to the complexities of practical distributed learning scenarios. We adopted these assumptions to align our work with the established literature on distributed learning. In the revised version, we have included an expanded discussion on potential limitations arising from these assumptions, along with directions for future work to extend our framework to settings with asynchronous updates and more realistic variance conditions. We believe these clarifications strengthen the paper and provide a balanced view of both the theoretical guarantees and practical applicability.

Response to “Other Comments Or Suggestions 2-3”: We appreciate the reviewer’s detailed suggestions. In response, we have reworded several sentences in the introduction and conclusion for improved clarity and conciseness. For example, we revised the phrase to: "The disparity in resources and data diversity undermines the accurate score function estimation assumed in single-worker models." Additionally, we conducted a thorough proofreading pass to address minor grammatical errors and ensure consistency in notation, particularly within the equations. We believe these refinements enhance the clarity and overall quality of the manuscript while maintaining its theoretical rigor.

Response to “Questions For Authors”: Regarding your first question, we would like to thank you for your carefulness. There is indeed a citation error here, that is, (15) should be (16), which we have corrected in the revised version. We next explain how to get (17) from (16). By summing (16) from $s=1$ to $S$, from $n=1$ to $N$, and from $r=1$ to $R$, we can obtain the following inequality:

By shifting term and dividing both sides of the inequality by $\Gamma^*$, we can obtain (17).

For your second question, if the initial samples across all workers are identically distributed, then $\sigma_2^2$—which quantifies the discrepancy among workers—would be zero. Consequently, the bound on $F_n(\theta_R)$ simplifies by eliminating the $\sigma_2^2$ term. Thank you for your careful consideration, we have simplified this representation in the revised version. For your concern on the third question, we avoid using Lipschitz continuity assumption on the data due to its inherent limitations. A uniform Lipschitz condition can be overly restrictive, as the associated constant may scale with the data dimension—particularly when the distribution is approximately supported on a sub-manifold. Moreover, even employing a time-varying Lipschitz constant does not fully mitigate this issue. For example, Chen et al. (2023) assume Lipschitz smoothness at $t=0$ but still obtain a quadratic dependence on the data dimension $d$. Consequently, we choose not to rely on the Lipschitz assumption with respect to the data.

If there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.

We thank the Reviewer hsmD for the time and valuable feedback! We would try our best to address the comments one by one.

In response to the concern about the outputs of diffusion models, we have provided additional Figures 4-6 in Appendix E.3, which can also be found at the anonymous link: https://anonymous.4open.science/r/Diffusion-0D86/ Specifically, we randomly selected 30 noise instances following Gaussian distribution, and then generated 30 samples from them. This procedure was applied to each pruning setting of both pruning techniques across all datasets.

The generated samples shown in Figures 4-6 further reveal that pruning influences generation quality, particularly on complex datasets. For CIFAR-10 and SVHN, the full model produces images with more details and consistent color distribution, whereas aggressive pruning leads to noticeable degradations—such as blurred features, increased noise, and color distortions. Top‑k pruning preserves critical parameters more effectively, yielding images that are more closed to those produced from the full model, particularly under moderate pruning conditions. For simpler datasets like Fashion‑MNIST, the impact of pruning is less severe; in fact, higher pruning levels sometimes result in better images due to a regularization effect that eliminates redundant details and prevents overfitting.

For the concern about the Figure 1, it serves as an illustration of distributed diffusion model training with pruning. Through Figure 1, we aim to help readers comprehend the training process as well as key aspects of the theoretical analysis, such as time discretization error, distributed training dynamics, and the impact of early stopping.

If there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.