Training Diffusion-based Generative Models with Limited Data

Q1. The paper's method section could be expanded on expense of the rest of the paper.

Due to the 8-page limitation of ICML submission, we have included additional details of our proposed LD-Diffusion in the Appendix. We will extend the Sections. 4.1, 4.2, and 4.3 based on the Appendix to provide more details of our proposed LD-Diffusion in the revised version (9 pages).

Q2. The OOD regularization is presented without sufficient theoretical justification.

The OOD regularization is one of the improved training techniques in our paper, which is mainly based on empirical experiments to further avoid the leaking issue during training. Specifically, OOD regularization aims to maintain the core properties of the diffusion models while introducing the necessary adjustments to enhance performance under limited data settings. We have conducted complete experiments to demonstrate that the changes to the loss function caused by OOD regularization do not significantly affect the stability or other important properties of the diffusion models. Additionally, as highlighted in the results and ablation studies, the LD-Diffusion demonstrates further improvement with OOD regularization on low-shot datasets.

Q3. The selection of VAE should be clearly explained in the main paper.

We have already explained the reason for selecting the pre-trained encoder and decoder models from similar but slightly different VAE models (with the same latent space) for the compressing model in Lines 264-274 (left). Furthermore, we conduct an ablation study for this selection in Appendix B.2. We will add more details from the Appendix B.2 to the main paper in the revised version.

Q4. I do not see applying latent diffusion model as contribution of this work.

Although the latent diffusion model has been applied in some existing approaches, these studies primarily focus on accelerating training speed and reducing computational cost. In contrast, our work is the first to apply the latent diffusion model based on theoretical analysis (refer to Sec 3.2) to improve the training of diffusion models with limited data. We believe this presents a novel perspective distinct from existing methods and offers valuable insights for future research. Furthermore, based on Reviewer aWCW comments: The evaluation of applying these techniques (including latent diffusion model) to train diffusion models in limited data settings has never been done. This reinforces the viewpoint that applying the latent diffusion model to enhance diffusion model training with limited data can be regarded as a meaningful contribution in our paper.

Q5. The choice of NFE in the LD-Diffusion.

We have already provided a detailed ablation study of how to choose the NFEs for two cases in Table 16 in the Appendix. Specifically, the selection of NFE aligns with the denoising sampling NFE settings in EDM. Although the results in Table 16 show that different NFE selections do not significantly impact performance and are therefore not crucial for high-quality generation, it is evident from the results that setting NFE=79 for the FFHQ dataset and NFE=511 for low-shot datasets is reasonable and achieves the best practical. As shown in Table 16, if the NFE is selected as 79 for the low-shot datasets, it can decrease the performance accordingly.

Q1. One issue of the FFHQ-100 dataset.

The purpose of designing OOD regularization is to mitigate the potential leaking issue in diffusion models under limited data conditions, thus OOD regularization is deliberately designed and can only be applied to the scenarios where aiming to prevent the leaking issue. As illustrated in Sec 2.3, the evaluation of two limited data cases differs, which causes the leaking of augmentation issue to play a different role in the performance of each case (shown in Lines 278-291 and Appendix A.1). Under this situation, OOD regularization should be applied to low-shot datasets (Case 1) to mitigate the leaking problem, whereas it should not be applied to the FFHQ dataset (Case 2). Therefore, no tuning is needed for the OOD regularization when using it.

Q2. Is there any intuition behind using different VAE encoder and decoders?

We have already explained the intuition behind using slightly different encoder and decoder models (with the same latent space) in Lines 264-274. Specifically, using the same pre-trained VAE encoder and decoder can lead to a small loss of image details, which limits the variety of detailed information in generated images thus decreasing performance. By applying slightly different encoder and decoder models (with the same latent space), we intuitively alleviate this issue by providing more varied information without affecting convergence. The experiments presented in Appendix B.2 further validate this intuition.

Q3. What is novel about the theoretical insight?

Although empirical risk minimization is a well-established theory in machine learning, it has not been fully explored in diffusion models. Our paper extends the empirical risk minimization and provides pioneering theoretical insight into diffusion models with limited data. Notably, the proposed theoretical insight provides a different viewpoint compared with the commonly overfitting viewpoint in other generative models (such as GANs and IMLEs) with limited data. Therefore, we believe the proposed theoretical insight can inform future studies on diffusion models and even other generative models with limited data.

Q4. How is the performance affected if the VAE model is trained on a very different domain (not natural images, e.g.)?

The primary purpose of the pretrained VAE model in LD-Diffusion is to project images from the pixel space into a latent space while ensuring effective reconstruction back to the image domain. To achieve this, VAE models are typically trained on large-scale datasets such as LAION-Aesthetics, which provide diverse and high-quality natural images for the training of VAE models. Due to there is lack of such large-scale datasets from other domains for training the VAE model, we conduct experiments comparing EDM + DA and LD-Diffusion with non-natural image domain limited data, i.e., the BRECAHAD dataset containing 162 real-world medical images, to enable the pretrained VAE data and limited data are in the different domains. We follow the official codes of EDM for preprocessing and resizing the BRECAHAD data into 3240 partially overlapping crops with resolution $256 \times 256$ for avoiding the huge computational cost in the experiments.

The results clearly illustrate that the pretrained VAE model applied in LD-Diffusion can still achieve significant improvements on non-natural image domain limited data such as medical image datasets. This demonstrates that the effectiveness of LD-Diffusion does not depend on domain similarity, which is different from transfer learning methods.

Table 1: Comparison of LD-Diffusion and EDM + DA using FID scores on the BRECAHAD dataset.

Q5. Comparing with the transfer learning methods?

According to our response in Q4, we argue that directly comparing the proposed LD-Diffusion (a training-from-scratch method) to transfer learning approaches is not a relevant evaluation. Below, we outline three key reasons supporting this perspective: (1) LD-Diffusion is a diffusion model trained from scratch (the parameters in Pretrained VAE models are frozen during training), differing from diffusion models that rely on transfer learning for fine-tuning with limited data. (2) As stated in Q4, the effectiveness of LD-Diffusion is not dependent on the similarity between the pretrained VAE model data and the applied limited data. In contrast, diffusion models that rely on transfer learning for fine-tuning typically require the similarity between the source data domain and the target limited-data domain to achieve improvements. (3) To the best of our knowledge, there are no open-source pretrained unconditional diffusion models that have been trained on the same dataset as the pretrained VAE, such as LAION-Aesthetics. Thus, it is difficult to build up a fair comparison baseline.

Q1. How sensitive of hyperparameters related to the compressing model and MAFP?

For the hyperparameters in the compressing model, we follow the default settings from the pretrained VAE models in Stable Diffusion. Regarding MAFP, we conduct experiments using fixed probabilities $p_1$ and $p_2$, with the results shown in Tables 6 and 7. It is evident that different choices of $p_1$ and $p_2$ can influence performance, but the impact is not significant. Specifically, increasing the optimal $p_1$ and $p_2$ values 2 $\times$ only affects 2% relative performance. For comparison, in Table 6 of the APA [1] paper, increasing the threshold hyperparameter $t$ by a factor of 1.33 results in a 7% relative change in performance.

[1] Deceive d: Adaptive pseudo augmentation for gan training with limited data. NeurIPS 2021.

Q2. How were the optimal parameters selected?

Based on our response in Q1, aside from default hyperparameters, LD-Diffusion only uses $p_1$ and $p_2$ in the MAFP, with optimal values determined through experiments (Tables 6 and 7).

Q3. Scalability.

Our paper primarily focuses on research in limited data scenarios. In this case, LD-Diffusion employs the compressing model to constrain the hypothesis space, which can significantly enhance performance. For data-rich settings, Stable Diffusion has already shown that constraining the hypothesis space can significantly accelerate the training speed with minor and acceptable performance drops, demonstrating the scalability of compressing the hypothesis space.

Q4. Complexity and Overhead of MAFP.

Existing methods have already shown that both Diff-Augment and ADA introduce only negligible complexity and computational overhead in GANs. MAFP combines Diff-Augment and ADA with a fixed probability also incurs only a negligible increase in complexity and computational cost. Furthermore, MAFP is a plug-in module, non-technical can easily follow the default augmentation methods and fixed probability values provided in our paper to enhance performance under limited data settings.

Q5. Generalization Concerns.

Following our response in Q3, our paper primarily focuses on research in limited data scenarios. In Secs 2.2 and 2.3, we have shown that minimizing the total denoising score matching error in diffusion models is challenging under limited data conditions. In such cases, the high complexity and variability of the limited data further exacerbate the difficulty for diffusion models in learning meaningful information. To address this issue, LD-Diffusion employs a compressing model to constrain the hypothesis space complexity, leading to significant improvements under limited data settings. The substantial performance gains observed in our experiments demonstrate that the benefits of reducing hypothesis space complexity far outweigh its potential limitations. The insight of constraining the hypothesis space complexity can be also used to understand the effectiveness of other generative models under limited data, such as Projected GAN, further demonstrating the generalization ability of our approach.

Q6. Impact of Data Augmentation on Authenticity.

The problem you mentioned is the well-known leaking of augmentation issue, i.e., augmentation samples falling out of distribution, which leads to the generation of less authentic samples. In Lines 87 and 268-269 (right side), we have already stated that the design of MAFP aims to increase training samples while avoiding this issue. Additionally, improved training techniques such as OOD regularization can further mitigate this issue.

Q7. Computational resources required for LD-Diffusion.

Regarding the computational resources required for training and deploying LD-Diffusion, a single GPU with 24GB of memory is sufficient. This is because LD-Diffusion is benefiting from the use of the proposed compressing model.

Q8. How does the model perform in terms of training time and resource utilization compared to traditional diffusion models?

The comparison of training and sampling time between LD-Diffusion and EDM + DA can be found in Table 20. In terms of computational resources, benefiting from the use of the proposed compressing model, LD-Diffusion reduces GPU memory usage by nearly 8$\times$ under the same batch size and decreases GFLOPs by approximately 6$\times$ compared with EDM + DA.

Q9. Were any control experiments?

We have already presented ablation studies to demonstrate the effectiveness of each component in LD-Diffusion in Tables 4 and 5. By progressively removing each component (i.e., improved training techniques, compressing model and MAFP), the results help illustrate the individual contributions of the proposed components to the overall model performance, enhancing the understanding of their importance.

Due to the 5,000-character limit, The results of Inception Scores can be found in https://anonymous.4open.science/r/Inception-Scores/README.md.

Q1. Figure 4 shows how the leaking issues are different for case 1 and case 2. I think it might be helpful in understanding the proposed LD-Diffusion to explain how these differences in leaking issues lead to different strategies for cases 1 & 2, while I also think Figure 4 is critical for the motivation of the proposed method and needs to be in the main body.

Thank you for your valuable suggestions. We have already analyzed that the leaking issue affects performance differently for Cases 1 and 2 in Lines 278-291. Therefore, we introduce the patch training for Case 1 and propose a novel Out-of-distribution (OOD) regularization for Case 2 to enhance the training of diffusion models with limited data, respectively. Furthermore, due to space constraints, Sec A.1 and Figure 4 are currently included in the Appendix. In the revised version, we will reallocate space to incorporate them into the main paper, improving clarity and accessibility.