Score-of-Mixture Training: One-Step Generative Model Training Made Simple via Score Estimation of Mixture Distributions

We appreciate the effort in reviewing our work and the helpful suggestions for improving the readability of our paper. Below, we provide clarifications on the identified weaknesses and responses to the questions.

Clarifications on Weaknesses

• On stability of ScoreMix training: We appreciate the reviewer’s suggestion to include training curves to further demonstrate the claimed stability of ScoreMix. While we included the FID curve over training iterations in Figure 2, we agree that plotting additional statistics would better illustrate training stability. In Figure D of this anonymized link, we have plotted an example training trajectory from our best ImageNet model, showing training losses and gradient norms. This further illustrates the stability during training. We will also include results for different parameter settings in the revision.
• On the role of the GAN regularizer: We acknowledge that the GAN regularizer significantly helps accelerate convergence and improve FID. However, we note that we did not train the version without the GAN regularizer long enough to reach full convergence. To clarify whether the GAN regularizer is essential for achieving SOTA FID or primarily helps speed up convergence, we will conduct a training run on CIFAR10 without the GAN regularizer until convergence and report the result.
• On the stability and complexity of the GAN regularizer: Similar to DMD2, our GAN discriminator is built on top of the score network, with only a few additional MLP layers. This score-model-dependent design allows the full model to benefit from the training stability provided by denoising score matching, while the GAN discriminator loss only trains the small auxiliary MLP. (For ImageNet, the generator has 296M parameters and the discriminator has 18M.) Thus, the discriminator represents a small fraction of the overall model size and has a negligible impact on training speed. As a result, our use of the GAN regularizer is both efficient and stable.

Answers to Questions

• Additional regularization using consistency between scores of mixtures?: We note that the score of a mixture distribution can be expressed as a weighted sum of the scores of the true and fake distributions, as shown in Eq. (12), where the weight is determined by the density ratio and $\alpha$. We leverage this relation in our distillation scenario, which is in the same spirit as the reviewer’s suggestion regarding consistency. However, for training from scratch, the relation between the scores for different $\alpha$ values is more implicit. It would be very interesting if a similar consistency regularization could be achieved in this context, and we agree that this is a promising direction for future work.
• On expensive regularizers: We apologize for any confusion caused by our insufficient explanation of the term "expensive regularizers." We provide a clarification below and will address this point clearly in the revision.
  • One example of an expensive regularizer is the regression loss used in the DMD paper. To address mode collapse in the DMD framework without any regularization, the authors simulate the reverse process of a diffusion model and sample several thousand noise-image pairs to anchor the generator’s outputs. Each noise-image pair requires evaluating the diffusion denoiser 256 times for ImageNet 64×64, which is extremely costly in practice. Moreover, the cost of collecting this regression dataset scales poorly with dimensionality.
  • Another example of costly training is the approach in the CTM paper. To ensure consistency between random points along the PF ODE trajectory, the reverse diffusion sampler must be run for an arbitrary number of steps per minibatch, which also results in high computational cost. Here we note that we mistakenly referred to this expensive procedure as a "regularization" technique, and we will correct this terminology in the final revision.
  • In contrast, our method does not require sampling from a diffusion model. It relies only on score estimation of mixture distributions, making it significantly more efficient.

We thank the reviewer again for their insightful questions and helpful feedback. We will incorporate all of the above points in our revision, along with the proposed additional ablation studies.

We appreciate the reviewer’s effort in reviewing our paper and constructive comments. We will address the raised concerns in our revision as follows.

• On FID evaluation and adversarial loss: We appreciate the reviewer for the thoughtful comment on the limitation of FID evaluation and its interaction with adversarial losses (Stein et al., 2023). We will clearly explain these points in our revision. Here, we wish to clarify our standpoint.
  • While we acknowledge that FID is not a perfect metric as argued in the reference, it remains the most popular benchmark for measuring perceptual quality. Our ablation study shows that ScoreMix achieves stable and competitive performance even without the GAN regularization, indicating that the performance improvement is not solely due to adversarial losses.
  • Regarding fairness in comparison, we highlight that some recent distillation methods in Table 2 such as DMD2 use adversarial regularizer. Similar to DMD2, our GAN discriminator is built on top of the score network, with a few layers of MLP. This score-model-dependent design allows the entire model to enjoy training stability driven by denoising score matching, while the GAN discriminator loss only trains the small additional MLP. (For ImageNet, the generator has 296M params and the discriminator has 18M params.) However, for other distillation or training-from-scratch methods such as consistency training or distillation, it is nontrivial to implement the GAN discriminator assuring training stability. For example, the notable example of the consistency model framework differs conceptually from traditional distribution matching approaches. While a GAN regularizer fits within this framework, its implementation requires more than a small discriminator like ours. Designing a full discriminator network (similar to those in StyleGAN or its XL variants) would be necessary, and this could involve substantial hyperparameter tuning and careful initialization. Moreover, due to the inherent instability of the consistency training framework, introducing an adversarial loss could exacerbate these instabilities, given the well-known challenges of GAN training. Hence, investigating GAN regularization to an existing method such as consistency training is an interesting research direction, but it is beyond our current scope.
• Comparison of the proposed GAN regularization with standard GAN regularization: This will be an informative ablation study that can clarify the role of skew divergence in GAN regularizer. We will add an additional result for CIFAR10 with standard GAN regularization, i.e., only using $\\alpha=\\frac{1}{2}$ in the regularization.
• Additional ablation with standard KL or reversed KL divergence minimization: We appreciate the reviewer’s suggestion. We will add a result only using the reverse KL divergence for CIFAR10 in the revision. In the meantime, we performed an additional experiment with a toy dataset, as shared in the global response.
• Toy experiment on Swiss roll dataset: We ran a toy experiment on 2D Swiss roll dataset, to demonstrate the effectiveness of our ScoreMix framework compared to the existing schemes for a simpler setting; see Figure A in this anonymized link. In particular, our results for training from scratch and distillation are presented in Figure A (d, f, g). All three methods successfully capture the modes of the underlying distribution. While the impact of the GAN regularizer is less pronounced than in our high-dimensional experiments, we observe that enabling it reduces the number of samples in low-density regions in (g). The distillation results in (d) appear slightly noisy, likely due to the quality of the pre-trained score model. This highlights the advantage of training from scratch, as it avoids amplifying existing estimation errors in the pre-trained model. We will add this experiment to Appendix in our revision.

We thank the reviewer again for their suggestions and comments, and will certainly incorporate all the points in our revision including additional ablation study results with CIFAR10. If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.

We appreciate the effort in reviewing our manuscript and providing constructive comments. We will incorporate all the feedback in our revision to improve the manuscript.

On Weaknesses

• Missing reference and baselines: We thank the reviewer for pointing out the missing references and baselines (rectified flow models and their variants). We will certainly add and discuss these in our revision to better clarify and contextualize our contribution.
• Efficiency comparisons: We thank for the suggestion. We will include GPU hours for our experiments in the revision to highlight the efficiency of our framework. To obtain the best FID result for ImageNet reported in Table 2, for example, it took 80 hours using 7 × A100 GPUs (200k iterations with an overall batch size of 280). As a comparison, both consistency training (CT) and improved consistency training (iCT) used 800k iterations with a much larger batch size of 2048. They do not provide the number of GPU hours, making a direct comparison difficult. We do note that ECT is able to finetune a pre-trained diffusion model on 4 × H100 GPUs within 8.5 hours. However, this consistency model is initialized with a pre-trained EDM2 diffusion model that was trained on 32 × A100 GPUs with a batch size of 2048 for a total of 2500 million images. Again, the EDM2 paper does not report the number of GPU hours, but its predecessor, EDM, reports a total training time of 2 weeks with a similar computational budget on ImageNet 64×64 — which alone exceeds our entire training-from-scratch effort. This shows that our proposed method can train a one-step model from scratch with a far smaller computational budget in terms of number of GPUs, training iterations, and overall batch size. Moreover, the two-model update is not more expensive than existing SOTA methods.
• Code release: As promised, we will release our codebase and model weights upon acceptance for reproducibility.
• Typo: We appreciate the reviewer’s careful reading of our paper. We will correct the typos and thoroughly check the manuscript.

Answers to Questions

• Why α=0 for 25% of the time during the score training?: This is due to the nature of the gradient of the skew JSD in Eq. (4). In that expression, the score of the mixture distribution with $\alpha=0$ is always used, which implies its particular importance and the need for accurate estimation. We will clarify this point in our revision to avoid any confusion.
• Extension to multi-step inference?: This is a great question, and we agree that having a multi-step refinement feature would be of great interest. One potential direction is to develop a hybrid method that combines consistency models with our approach. We leave this for future work.
• Invertibility?: Our current manuscript focuses on developing a high-quality, efficient sampling scheme. Whether our model can be inverted is indeed an interesting question from a representation learning perspective, and we leave this for future investigation.
• Linear interpolation?: Thanks for the helpful suggestion. We conducted the interpolation experiment for CIFAR10 and ImageNet and uploaded the results at this anonymized link; see Figures B and C. Similar to GANs and consistency models, we found that “spherical interpolation” leads to natural interpolations.