Sparse-to-Sparse Training of Diffusion Models

Overall, the time taken to search for methods and hyperparameters seems too long compared to the performance improvements. Proposing methods to reduce the search time would be helpful.

Do you have any insights on how to effectively tune hyperparameters such as network sparsity, exploration frequency, pruning rate, and sparse method, beyond random search or grid search?

To further clarify, we are not suggesting that our study should be performed as a hyperparameter search, every time, for each task or dataset. Rather, our aim is to identify safe and effective values that could be used to perform sparse-to-sparse training of DMs. For high performance, the optimal sparsity level seems to be around 25–50%, and we suggest utilizing dynamic sparse training methods with conservative pruning rates, such as p=0.05. In this way, researchers and practitioners can benefit from our findings, without having to invest extensive computational power themselves, which could indeed defeat the purpose of our research.

In Section 4.3, line 515, a speed-up of 0.57x is mentioned, but it is unclear whether GPU inference time is improved.

In Section 4.3, line 515, could you clarify whether GPU inference speed actually improves by 0.57x as mentioned? Could you provide papers or resources that demonstrate that reducing FLOPS leads to improved inference speed on hardware?

Thanks for pointing this out. That sentence was misleading: GPU inference time is not improved. Since GPUs are not yet optimized to support unstructured sparsity at various S ratios, we mimic sparsity using sparse masks. In practice, this means that there are no actual speedups, therefore in the updated paper we refer to ‘theoretical speedup’. As mentioned in Appendix A, the hardware industry is catching up so it is a matter of time to truly leverage sparse operations.

Have you tried using structured sparsity, which removes entire layers, to reduce inference time?

Our focus was on unstructured sparsity due to its ability to maintain high performance even at very high levels of sparsity, as reported in previous work such as [1] and [2].

[1] Evci, U. et al. Rigging the Lottery: Making All Tickets Winners. ArXiv, abs/1911.11134. (2019)

[2] Frankle, J., and Carbin, M. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. arXiv: Learning. (2018)

For Table 1, would it be possible to conduct experiments that reduce the standard deviation to below 3.0 through hyperparameter tuning on the Bedrooms and Imagenette datasets? The mean + standard deviation for sparse training (for example, 28.79 + 12.65 = 41.44 for Bedrooms Static-DM) is consistently higher than the mean for dense training (31.09 for Bedrooms Dense).

We appreciate this suggestion however we believe that it is beyond the scope of our study. As previously discussed, the variance observed in the models is similar when comparing Dense and Sparse versions in all cases for a given dataset, suggesting that this variability comes from factors not related to sparsity. We have mentioned this in Section 4.1 in the revised paper.

In Section 4.1, line 376, it is mentioned that, unlike existing supervised learning and GAN models, the DM using the SST method outperforms the DST method. Additional analysis is needed to explain why this different trend is observed.

In the paper, we wonder whether the question of DST models consistently outperforming SST models, provided there is appropriate parameter exploration, might also apply to DMs. To provide further insights into this, we repeated the main experiments for RigL-Diff and MagRan-Diff using a pruning rate of 0.05 for both LatentDiffusion and ChiroDiff. We have updated the paper consequently (updated Appendix E).

The results for Latent Diffusion, presented in Table 1, show that reducing the pruning rate leads to considerable improvements in performance for RigL-DM and MagRan-DM. With the previous pruning rate p=0.5, neither of the methods are able to outperform Static-DM on all datasets. With the new pruning rate p=0.05, both methods are able to outperform it for CelebA-HQ but only MagRan-DM for LSUN-Bedrooms. In Imagenette, Static-DM remains the top method.

Table 1

The results for ChiroDiff, presented in Table 2, show a similar pattern of improvement when using the lower pruning rate, albeit not as significant. Using the new pruning rate, all DST methods were able to surpass the performance of the respective SST model, even if this was previously not the case.

Table 2

These findings do suggest that using appropriate parameter exploration, DST methods are able to outperform SST on high sparsity regimes.

Without introducing new concepts or ideas, the paper applies the existing sparse-to-sparse training method from supervised learning to Diffusion Models. It would be better to propose a new method optimized for Diffusion Models.

We would like to restate that the application of existing sparse-to-sparse training methods serves as a foundation to understand how sparsity interacts with DMs. Prior to this application, no information was known about optimal sparsity ratios, algorithms or pruning rate values. This work is a valuable research contribution by itself that can foster follow-up work like proposing new methods that are specifically tailored to DMs.

The variance of FID scores in Table 1 is overall too large, and the reduction in FLOPS is not significant for Bedrooms and Imagenette.

Actually, the variance of all sparse models is similar to the variance observed in their dense versions. The main objective of this paper is not to achieve the best reduction in FLOPS or the lowest FID scores, but to uncover new findings and insights about sparse-to-sparse training of DMs.

The efficiency gained from reducing inference speed via FLOPS reduction is dependent on hardware.

We completely agree. However, we consider this a ubiquitous challenge of sparsity research, rather than a specific weakness of our work. As the interest in unstructured sparsity research has grown in recent years, so has the development of hardware, with companies such as NVIDIA or Cerebras putting time and resources into developing sparsity-friendly hardware. We believe that hardware industry will catch-up to these research advancements.

Dear reviewer rNhu,

Thank you for your comments and constructive review. Below we address the concerns and questions mentioned.

This paper only uses a portion of the CelebA-HQ and LSUN-Bedrooms datasets for the experiments. However, I believe that the performance of sparse training may decrease on larger datasets due to the reduced expressive power of the model caused by pruning. (...) experiments on larger datasets such as the full ImageNet or the entire LSUN-Bedrooms dataset are needed.

Is sparse training effective on larger datasets such as the full LSUN-Bedrooms dataset or ImageNet1k, which are larger than CelebA-HQ?

As mentioned in Section 3.3.2, due to computing limitations, we could not consider the full versions of CelebA-HQ and LSUN-Bedrooms. However, for subsets thereof, if they are randomly sampled (as we did), then model performance on these sets can provide a reliable indication of its performance on a larger set. This is actually corroborated by our experiments in Appendix C.

Is there a specific reason for using only the FID score as the evaluation metric? If not, it would be helpful to also include the Inception Score (IS).

We decided to base our analysis on FID, as it was the metric that the original Latent Diffusion and ChiroDiff papers employed. We also thought of including Inception Score, but ultimately decided to exclude it, due to it being superseded by FID and also it being better suited to evaluate ImageNet generators [1]. Nonetheless, if the reviewer feels strongly about the inclusion of Inception Scores, we would be happy to do it.

[1] Barratt, S.T. and Sharma, R. A Note on the Inception Score. ArXiv, abs/1801.01973 (2018)

In Figure 2, only a few methods and sparse rates outperform dense training in CelebA-HQ and Imagenette. Due to the long search time for optimal settings, the reduction in training time mentioned by the authors seems insignificant.

We believe there might be some misunderstanding here. We performed several runs using different sparsity levels and algorithms on the same model and dataset, as a way of studying whether stable “safe” sparsity levels and algorithms emerged. We don’t expect/recommend practitioners to do a similar study every time they train a new diffusion model.

In general, we have found safe values that work, independent of specific characteristics of the dataset. Optimal sparsity level for high performance seems to be around 25–50%, and we recommend the utilization of dynamic sparse training methods with conservative pruning rates, such as 0.05. It is our hope that researchers will use these guidelines while training their own sparsified DMs.

Although ChiroDiff shows performance improvements with the QuickDraw dataset, it is hard to say that there are meaningful improvements in performance for KanjiVG and VMNIST. Sparse training lacks robustness across different datasets.

Could you explain why performance is strong only for QuickDraw in Table 2, but not for KanjiVG and VMNIST? Is there a particular characteristic of the datasets that leads to this?

It is true that, for ChiroDiff, the most notable improvements in FID scores are observed in the QuickDraw dataset. However we should note that beating the base model’s performance is not the main goal of this work. Still, for all datasets, most of the models with sparsity <= 50% obtain FID values that are comparable with their Dense version, using much less memory and computational cost.

In Table 2, performance is good for QuickDraw but poor for KanjiVG and VMNIST. An analysis of the reasons behind this discrepancy would be useful.

Although in the main experiments the sparse models were only able to outperform the Dense version in QuickDraw and KanjiVG, in VMNIST the performance of the best sparse model is comparable to that of the Dense model. In the new experiments using pruning rate=0.05 (mentioned previously, and included in the updated Appendix E), sparse models are also able to outperform the dense in KanjiVG. The reasons behind the difference in performance are likely due to the difference in model size and dataset difficulty.

We appreciate the time and effort in reviewing our work, and our rebuttal. We thank you for raising your score.

However, the improvement in test FLOPS remains dependent on the dataset.

We want to clarify that improvement in test FLOPs is dependent on the hardware and the architecture of the model, not the dataset.

It is suggested that the format of the references be made uniform, as there are discrepancies between different sections.

Thank you for the suggestion. We have revised all references in the updated version.

Is there any explanation regarding the design of the sparsity rates for training diffusion models, which appear to be predefined without any intuitive understanding based on specific concepts related to the models? Is it possible to design an adaptive sparsity schedule?

The application of predefined sparsity ratios is a common practice in the sparse-to-sparse training literature. They served as a solid start for our study, allowing us to observe the baseline behavior of a sparsified DM. Adaptive sparsity schedules have been applied in the context of Time Series Forecasting [1] and could be an interesting direction for a follow-up study, although we should mention that it cannot be straightforwardly applied because [1] was done in the context of supervised learning.

[1] Atashgahi, Z. et al., Adaptive Sparsity Level During Training for Efficient Time Series Forecasting with Transformers. Proc. ECML PKDD (2024)

How many GPUs were used in the training process for different DMs? Providing more details about the training settings would greatly enhance the confidence in the proposed framework.

We have trained each DM using only one GPU. We have used NVIDIA A100 and V100 SXM2 GPUs for training all models. We have added this information to Section 3.3.2 (Experimental Details).

For a given DM, how should the decision be made regarding the training strategy—whether to use static sparsity or dynamic sparsity?

Looking at our findings, dynamic sparse training is a safe choice for DMs. The new experiments using a pruning ratio of 0.05 further corroborate this (detailed in a response above and in the updated Appendix E).

Can you provide some experimental results on the text-to-image task, which is one of the most important practical applications of diffusion models?

We agree that including text-to-image experiments would be an interesting addition to the paper. However, our focus is on unconditional generation, and we hope that the reviewer can understand that the limited space precludes expanding the paper in this way.

Dear reviewer 41FH

Thank you for your comments and constructive review. Below we address the concerns and questions mentioned.

The proposed framework appears to be an incremental application with limited novelty. Furthermore, this paper seems to rely on established sparse-to-sparse strategies for optimization without any careful design.

We understand that the use of established sparse-to-sparse strategies may seem incremental, but we believe that a study like ours was necessary. Previous work could not assess whether sparse-to-sparse training would be beneficial for DMs. Our findings are proof that training DMs in a sparse-to-sparse way is feasible and beneficial, and provides valuable insights for future researchers.

Being the first paper in studying sparse-to-sparse training of DMs, we utilized already established algorithms in order to properly contextualize our study. We actually did carefully design a set of experiments, with the goal of providing insights into the performance of different sparsity levels and SoTA algorithms, and their generalization ability across different backbones and datasets.

The proposed method is not theoretically guaranteed, which may result in performance variability.

Sparse-to-sparse training has been applied to supervised learning, reinforcement learning, and continual learning settings, and it has been proven to be both stable and beneficial. We do, of course, acknowledge the empirical nature of our study, as many other DL works published at ICLR and related venues, and agree on the need for more theoretical guarantees. However, our experiments are comprehensive enough to provide a primer on sparse-to-sparse training of SoTA DMs, which can be very valuable for researchers and practitioners.

The ablation studies are lacking. The validity of the model would be better established with more experimental results provided.

One question that arose from the initial submission was the impact of the chosen pruning rate p=0.5. Thus, we decided to repeat the main experiments for RigL-DM and MagRan-DM, using a pruning rate of 0.05, for all datasets and models.

The results for Latent Diffusion, presented in Table 1, show that reducing the pruning rate leads to considerable improvements in performance for RigL-DM and MagRan-DM, in the high sparsity regime of S=0.9. With the original pruning rate, p=0.5, neither of the methods were able to outperform Static-DM on all datasets. With the new pruning rate, p=0.05, both methods are able to outperform it for CelebA-HQ but only MagRan-DM for LSUN-Bedrooms. In Imagenette, Static-DM remains the top method.

Table 1

The results for ChiroDiff, presented in Table 2, show a similar pattern of improvement when using the lower pruning rate, albeit not as significant. The only exception is RigL-DM on Kanji-VG, although the difference is very small.

Table 2

We have updated the paper with these additional experiments (updated Appendix E).

Dear Reviewer 41FH,

We would like to thank you for your positive review and detailed comments. As the end of the discussion period is approaching, we would kindly ask if our response has sufficiently addressed your concerns. We would be glad to provide further clarification to any other questions you may have.

Best regards,

Authors

Dear Reviewer mJ8a,

Thank you for your comments and constructive review. Below we address the concerns and questions mentioned.

The biggest issue is that sparse training and DMs seem not to be coupled: there's no strong desire for me to think the combination of these two is fantastic or compatible naturally, and I also didn't see any apparent problems that would prevent the two from combining easily. (...) As a result, although the paper has some contributions (of experiments and verification), it has NO core novelty.

Our paper highlights how sparse-to-sparse methods can be successfully applied to DMs, and reveals interesting insights that can inform researchers and practitioners. Critically, until now it was not clear how sparsity would impact DM training. For example, researchers have suggested that it is safe to work with high pruning rates for other models [1] however we have found that this is not the same in DMs, and it is better to be more conservative, especially in high sparsity regimes. We have conducted a new experiment (included in the updated Appendix E) to remark on the importance of this finding.

We would like to emphasize that this is the first-of-its-kind study, supported by a comprehensive set of experiments. In sum, our work creates a solid foundation that researchers and practitioners can build upon.

[1] Nowak, A. I. et al., Fantastic weights and how to find them: where to prune in dynamic sparse training. In Advances on Neural Information Processing Systems (2023)

I don't think the methods take advantage of the unique characteristics of DMs itself. (...) the backbones may have been verified of the combination with Sparse Training or pruning [1] [2].

Our work specifically focuses on the application of sparse-to-sparse methods for denoising backbones of DMs. The iterative nature of the denoising process is a characteristic that, to the best of our knowledge, has not been studied before in the literature. We would like to point out that sparse-to-sparse training is a different paradigm from what is proposed in [1], as there, the model starts out as dense, and is increasingly sparsified while training. In [2], a dense model is fully trained and then pruned, which again, is different from sparse-to-sparse training. We also should note that pruning methods are bounded by the performance of the original dense model, whereas with sparse training we can get better performance than their dense counterparts, as shown in Tables 1 and 2.

Refs (hyperlinks) can be changed to a different color (...)

We have now used a blue font color for all hyperref links.

It seems that the page number of the first page can be incorrectly hyperlinked.

Thank you for pointing this out. We have fixed it.

More introduction should be made to Latent Diffusion and ChiroDiff.

We appreciate the feedback and have expanded the description for each model.

Why choose these two DMs? AFAIK, ChiroDiff is not a well-known model.

First, LatentDiffusion is a SoTA and widely used model for image generation [1, 2]. Then, to study the generalization of sparse-to-sparse training, we decided to use a DM with a different backbone that can handle non-image generation. ChiroDiff, which was published at ICLR’23 and provided an open-source implementation, fitted well that purpose.

[1] Blattmann, A. et al., Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023) [2] Takagi, Y. and Nishimoto, S., High-resolution image reconstruction with latent diffusion models from human brain activity, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

Have experiments been done on other DMs (backbones) to test the generalization? What affects the combination of the two may not be different datasets or different generative tasks, but different network backbone architecture (e.g., U-Net v.s. Bidirectional GRU encoder). More analysis into this?

Indeed, we have conducted experiments with two different backbone architectures to test the generalization of sparse-to-sparse training of DMs. Specifically, as pointed out by the reviewer, we have tested UNet (LatentDiffusion) and Bidirectional GRU (ChiroDiff). We agree that it may be interesting to analyze more backbones, but we left this exercise as an opportunity for future work.

Dear Reviewer mJ8a,

We would like to thank you for your review and valuable comments. As the discussion period is ending soon, we would kindly ask if our response has addressed your concerns. If you have any other questions, we would be glad to provide further clarification.

Best regards,

Authors

Dear Reviewer xPfm,

Thank you for your comments and constructive review. Below we address the concerns and questions mentioned.

The biggest weakness of this paper is that there is virtually nothing new happening.

We respectfully disagree. While this paper does not present novel algorithms for sparse-to-sparse training, it is, as stated in the paper, the first study in the context of DMs, presenting novel insights that were not previously known. For example, the existence of literature that analyzes sparse-to-sparse training for GANs does not make this paper less noteworthy, since it cannot be assumed that findings in the GAN context translate directly into DMs, which have different training dynamics and challenges [1].

Our paper shows, through extensive empirical studies, that sparse-to-sparse training of DMs produces models that can match or outperform the dense counterparts, and provides new insights into optimal sparsity level, model performance, and impact of diffusion timesteps. Our results are consistent for two state-of-the-art (SoTA) DMs and over six datasets.

[1] Yang, L. et al. Diffusion Models: A Comprehensive Survey of Methods and Applications. ACM Computing Surveys (2023)

The paper needs to better motivate why exploring these specific methods is important, given that the motivation and goals are the same as other methods that can better take advantage / live up to the constraints of modern hardware.

Why are the authors specifically interested in these sparsity methods compared to other existing techniques in the literature that can actually reduce FLOP count and properly utilize hardware? The fixation on these specific sparse-to-sparse methods seems very poorly motivated, but I would welcome clarification on this.

In a nutshell, sparse models are more efficient yet equally performant as their dense counterparts. Our paper shows the theoretical advantages of sparse-to-sparse training of DMs, as the studied algorithms are currently ahead of their time in terms of hardware implementation. Although the industry is catching up, as we mentioned in Appendix A, with companies like NVIDIA or Cerebras allocating more and more resources to studying sparsity and developing sparsity-friendly hardware. By showing that sparse-to-sparse training of DMs is possible, and that it can lead to superior metrics with stability in specific sparsity levels, we hope to continue to “push” for more investment of time and resources on this topic. Most of the popular image generators available are now based on DMs, which means that plenty of resources are being poured into training these models. Showing that we can reduce these can be highly impactful for industry and academia.

Our work therefore looks more into how the future of Deep Learning could look like. We believe we all can agree on the fact that training large (and growing) models in production is expensive. But it does not need to be like that.

While we acknowledge that there are several other methods that attempt to solve the same problem, the main characteristic of sparse-to-sparse training is that it does not require specific architectural changes, and can potentially be combined with other strategies, such as the ones mentioned by the reviewer.

While the improvement of FID scores is certainly a strength of the work given that connections are being pruned, this is insufficient to demonstrate the effectiveness of any method: qualitative comparisons are key, given that the connection between FID and sample quality is not a guarantee (especially when differences are very small). This is an easy fix; the authors can provide many more samples, side-by-side with the baseline models.

Thank you for the suggestion. We have updated the paper accordingly (see new Appendix G).

Finally, while the quantity of experiments, datasets and models is appreciated by the reviewer, one less fatal but nonwithstanding a weakness, is that the datasets utilized are of very narrow domain and results may not transfer to larger settings, which are of key interest to the community.

We understand this criticism and agree that it would be interesting to test a larger dataset such as ImageNet. We plan to do it (and mention it in Section 4.4), however we believe that the experiments we report in the paper already uncover important insights that can be useful to researchers, such as the trade-off between sparsity rate and performance, and the impact of diffusion timesteps. None of this was previously known in the research literature.

Dear Reviewer xPfm,

We thank you for your time and constructive comments. As the end of the discussion period approaches, we would kindly ask if our rebuttal has adequately addressed your concerns. We would also be happy to provide further information to answer any other questions that you might have.

Best regards,

Authors