Continuous Semi-Implicit Models

Thank you for your valuable feedback! Below are our responses to your concerns.

Q1: I checked the experimental design ... Stein et al. “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models”, NeurIPS 2023.

A1: Exactly! The paper “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models” from NeurIPS 2023 by Stein et al, directly proposed the metric FD-DINOv2. They pointed out that FID as a metric is unfairly favoring GAN-style models, and has limited numeric accuracy to handle modern advanced diffusion models, as they usually reach a very small FID value. This inspires us to use FD-DINOv2 in our experiments.

Q2: As already explained in Relation to Broader Literature ... limits the technical novelty of the paper.

A2: Here we first clarify the novelty of our method. CoSIM is derived from the framework of continuous semi-implicit variational inference. Methods such as HSIVI [1] and SiD [2] are built upon this semi-implicit training objective to accelerate diffusion models. Theoretically, We introduce a regularization framework of the SiD loss to stabilize the training and address the pathological behavior under large alpha values as demonstrated through theorem 3.1 and the ablation study on the regularization term coefficient $\mathrm{coef} = (1+\lambda) \alpha$.

Table1: Class-conditional ImageNet (512x512) for CoSIM-L given $\alpha=2.0$.

Furthermore, by using a specific parametrization, we derive a novel training method for consistency models at a distributional level. Unlike traditional consistency model fixed on the reversed ODE path, CoSIM learns the consistency map directly, without being constrained by intermediate process. Through our experiments, we show this achieves superior performance compared to the current state-of-the-art consistency models on ImageNet $512 \times 512$, using much less training budget (ours 20M compared to sCD [3] 819.2M). Table3: Sample quality measured by FID on class-conditional ImageNet (512x512).

[1] Zhou, et. al., "Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation", ICML 2024.

[2] Yu, et. al., Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration, NeurIPS 2023

[3] Lu, et al., "Simplifying, stabilizing and scaling continuous-time consistency models.", OpenAI

Q3: Do you have an opinion ... Fisher divergence ... a better model.

A3: As a novel training scheme derived from continuous semi-implicit variational inference, CoSIM differs fundamentally from traditional consistency models in the following aspects, offering distinct training advantages:

• Fisher divergence does not require strictly following the reverse ODE trajectory but only learns a consistency mapping on a distributional level. This relaxes the choice on distillation path and enables various sampling strategies. It also reduces the training budget as the mapping can be more flexible. Conversely, ECT strictly follows the ODE trajectory, and its training budget is much higher than ours on comparable performance level.
• Experimentally, our method surpasses ECT by a large margin on both FID and FD-DINOv2, details shown in Table 1 and Table 2 in paper.
• Our approach primarily focuses on few-step sampling, which is also different from SiD, diff-instruct. We show that our approach shows significantly better performance using much less total training budget in our experiments, and scales to larger models better.

Q4: Are semi-implicit models ... lacking in the paper.

A4: To clarify, our contributions over semi-implicit models are:

• We extend the framework of hierarchical semi-implicit variational inference models to the continuous settings.
• We introduce a regularization term and give theoretical analysis on why this term could further stabilize the training. This addresses the pathological behavior of SiD loss when $\alpha$ is large.
• Our formulation, under a specific parameterization, reforms to a consistency training on a distributional level. This enables a more flexible mapping rather than traditional consistency training on a fixed reverse ODE trajectory. This reduces training difficulty and reduces training budget, already validated by our experiments.
• Please refer to A2 to see that our method can surpass the performance of latest consistency models with much lower training budget.

Thank you for your valuable feedback! We will address your concerns below.

Q1: It is recommended to provide empirical evidence supporting this claim, such as FID/FD vs. training iterations or runtime comparisons.

A1: We give the details of our training iterations in Table 4 in Appendix C. For comparison with HSIVI, we give the following two tables.

Table1: Training Efficiency and Performance on Unconditioanl CIFAR10

Table2: Comparison of Training Efficiency and Performance on ImageNet $64\times 64$

Q2: Is the training stable given that multiple networks are trained simultaneously? How does the parameter lambda affect performance? Is there an ablation study?

A2: Yes, all networks are trained simultaneously and stably, which has also been observed in previous works such as SiD[1] and Diff-Instruct[2]. We provide an ablation on $\lambda$ in the form of coefficient $\mathrm{coef} := \alpha(1+\lambda)$ below.

Table 3: Sample quality measured by FD-dinov2 on class-conditional ImageNet (512x512) for CoSIM-L given $\alpha=1.2$ and NFE=$2$.

The results show that introducing regularization (i.e., increasing $\mathrm{coef}$) within a suitable range facilitates the learning of $f_\psi$, which in turn improves the training of $q_\phi$. Furthermore, since the reduction on FD-DINOv2 continues as training progresses, CoSIM demonstrates stable training across a relatively broader range of regularization coefficients, such as 1.0 and 1.5.

[1] Zhou, et. al., "Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation", ICML 2024

[2] Luo, et. al., "Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models", NeurIPS 2023

Q3: The numerical performance with a one-step generation.

A3: CoSIM is inherantly a multi-step model, trained to model the continuous transition kernel rather than a one-step deterministic generation model like GANs. One significant advantage of multi-step generation is the ability to inject more randomness in each step, enhancing sampling diversity.

For ablation, we also evaluated one-step generation quality of CoSIM on ImageNet $512 \times 512$ for M model.

Table4: Sample quality on conditional ImageNet 512x512.

Note that FID result of Moment Matching[1] is evaluated at a lower resolution 128x128. Consider the difficulty of generating in higher resolution, our method is comparable with Moment Matching[1] on one-step generation.

[1]Salimans, et. al., "Multistep distillation of diffusion models via moment matching." NeurIPS 2024.

Q4: How does CoSIM integrate ... such as classifier-free guidance?

A4: If an unconditional model $G_\phi(x_t; t,\emptyset)$ is used to modify the generation, the parameterization can be considered to integrate it into CoSIM's training.

$$G_\phi^{(w)}(x_t; t,c) = wG_\phi(x_t; t,c) + (1-w)G_\phi(x_t; t,\emptyset).$$

We also noted that SiD-LSG [1] proposed a novel guidance method by using CFG in training and inferencing with different weights $w$. This method also adapts to our formulation and provides inspiration for future work.

[1] Zhou, et. al., "Long and short guidance in score identity distillation for one-step text-to-image generation."

Q5: The numerical performance to the choice of time distribution? ... the weighting functions.

A5: We give another ablation study on the hyperparameter $R$ in the time distribution $\pi(\gamma|r_s)$.

Table5: Ablation on $R$ for CoSIM-M on class-conditional ImageNet ($512 \times 512$) with NFE=4 by FD-DINOv2

Different $R$ gives slightly different convergence rate during early stages of training. But after 38M training images, this difference is minimized.

For the weighting functions, we adopt the same weighting function as in EDM[1], this setting has been widely adopted in many works such as SiD and irrelevant to our algorithm design. Hence we keep it fixed.

[1] Karras, et. al., "Elucidating the Design Space of Diffusion-Based Generative Models", NeurIPS 2022

Q6: Typos

A6: Thank you for catching these typos! We will correct them in our revision.

Thank you for providing valuable feedback! Here are our responses.

Q1: I personally like this paper. It extends HSIVI ... paper's contributions.

A1: Thank you for your interest in our work! We acknowledge that CoSIM can be regarded as a continuous extension of HSIVI, but our contributions go beyond that and is different from traditional consistency models in the following aspects:

• Unlike traditional ODE trajectory distillation applied by normal consistency distillation methods, we train the consistency model at a distributional level using Fisher divergence. This approach eliminates the need to constrain the generator $G_\phi$ to sample along a reversed ODE path predefined by the score function, allowing the model to learn a more flexible consistency mapping on a distributional level.
• Experimentally, we find that this distinction enables CoSIM to train much faster outperform the state-of-the-art consistency model sCD [1], using only 20M total training images (compared to the 819.2M by sCD). We provide extended results of CoSIM compared with sCD on FID, demonstrating its superior performance across various model sizes.

Table1: Sample quality measured by FID on class-conditional ImageNet (512x512).

Q2: How sensitive is CoSIM to the choice of the regularization hyperparameter?

A2: We conducted an ablation study on $\lambda$ with the coefficient $\mathrm{coef} := \alpha(1+\lambda)$ in (15) for conditional ImageNet $512 \times 512$ generation of CoSIM using the $L$ model size, with fixed $\alpha=1.2$. The results demonstrate that introducing regularization (i.e., increasing $\mathrm{coef}$) significantly enhances the learning process of $f_\psi$ in a wide range.

Table2: Sample quality measured by FD-dinov2 on class-conditional ImageNet (512x512) for CoSIM-L given $\alpha=1.2$.

Q3: Computational Overhead: Could you clarify or quantify the computational overhead ... SiD and Diff Instruct?

A3: There are three networks in the training process, generator $G_\phi$, teacher $S_{\theta^*}$ and auxiliary function $f_{\psi}$. During training, teacher $S_{\theta^*}$ is frozen while generator $G_\phi$ and auxiliary function $f_{\psi}$ are simultaneously evolving. For SiD and Diff Instruct, all three networks are kept the same as the baseline methods and initialized from the same checkpoint. For our method, the auxiliary function $f_{\psi}$ is modified to incorporate the extra time parameter $s$ introduced in Section 3. In practice, we duplicate the time embedding layer, which only adds a very small number of extra parameters.

Specifically, on CIFAR, the total number of trainable parameters is 111,465,478 for SiD and Diff-Instruct. With our modification, it increases to 116,128,006, adding only 4% extra parameters. On ImageNet $64\times64$, the total number of trainable parameters is 59,1798,534 for SiD and Diff-Instruct. With our modification, it increases to 619,999,878, also adding only 4% extra parameters.

Additionally we time the computational overhead for SiD, Diff-Instruct and our method on unconditional CIFAR generation. We conduct all experiments on a single L40S GPU with all fixed hyperparameters such as batch size, learning rate, optimizer etc..

Table 3: training time for 10496 images

Compared to SiD and Diff-Instruct, our method requires 27% and 18% more training time respectively. Apart from the extra trainable parameters explained above, the regularization we introduced in Theorem 3.1 also contributes to the extra time.

We argue such overhead is worthy, as the overall training objective stabilizes and the total training budget shrinks. On conditional imagenet $64\times64$ generation, our method only requires 200M training images to reach FID 1.46, surpassing SiD which needs 1000M training images to reach FID 1.52. On the same task, Diff-Instruct reaches FID 5.57 with 4.8M training images. Since we aim for a much lower FID, we did not frequently test our model in the early stages of training. The closest one we have is that our model reaches FID 4.15 with 5.12M training images.

Thank you for your thoughtful review and valuable feedback. We address your specific questions and comments below.

Q1: It does not seem very clear to me what is the bias in the SiD loss ... on $\lambda$.

A1: Thank you for your valuable suggestion! The significance of our regularization comes in three aspects:

• Regularizing the second-stage optimization of (10) (as discussed in Theorem 3.1) helps guide the optimal $f\_{\tilde{\psi}^*(\phi)}$ towards the pretrained score function $S\_{\theta^*}(\cdot;s) \\approx p(\cdot;s)$. And the Nash equilibrium of $\\phi^*$ remains consistent as in Theorem 3.1.
• Our formulation strictly contains SiD loss as a special case, which is equivalent to $\alpha = 1.2$ and $\alpha(1+\lambda) = 0.5$. This introduces a potential bias when $f_\psi$ is perfectly trained and $\alpha$ is too large. As shown in (38), this causes $q_\phi$ to optimize in a direction that deviates from the target distribution, corresponding to the failure case in Figure 7 of SiD. Our formulation allows for a larger $\lambda$, but still ensuring positive coefficient $(1+2\lambda)$ in (38), mitigating the potential bias in SiD loss.
• The ablation study on $\mathrm{coef} := \alpha(1+\lambda)$ in (15) with $\alpha=1.2$ show that introducing regularization facilitates the learning of $f_\psi$, improving the training of $q_\phi$.

Table1: Class-conditional ImageNet (512x512) for CoSIM-L given $\alpha=1.2$.

Q2: The authors mentioned in line 167-169 that the fused loss function ... on $\alpha$ if time permits.

A2: The pathological behavior when $\alpha > 1$ refers to the SiD loss which is equivalent to $\mathrm{coef} := \alpha(1+\lambda) = 0.5$. We propose using a regularization term by increasing $\lambda$ to mitigate this issue, as discussed in A1.

The ablation study below with a larger $\alpha=2.0$ shows the effectiveness of our regularization and the pathological results of SiD loss is similar to the observations shown in Figure 7 in SiD's paper.

Table2: Class-conditional ImageNet (512x512) for CoSIM-L given $\alpha=2.0$.

Q3: My major concern lies in the statement for Proposition 3.4 and assumption 3.3 ... regularization.

A3: $\epsilon_f$ measures the gap between $f_\psi$ and its optimal $f_{\psi^*}$ in the second optimization problem described in (12). This optimization is a quadratic problem, and in the optimal case, $\epsilon_f$ approaches 0. Therefore, the introduction of regularization does not affect $\epsilon_f$'s convergence to 0. Moreover, when $f_\psi$ is initialized as $S_\theta$, regularization brings $f_\psi$ closer to the optimal $f_{\psi^*}$, as shown in (16).

Q4: I noticed that CoSIM 4-step differs by only 0.06 ... observation.

A4: FID and FD-DINOv2 are designed in different numerical scales. FD-DINOv2 was designed more suitably for diffusion model evaluation. Results below are some well-known diffusion models evaluated for conditional image synthesis on ImageNet $256\times 256$.

Furthermore, FID was introduced in 2017 and does not have enough numerical range to differentiate strong modern diffusion models. FD-DINOv2 was introduced to provide a larger numerical gap to solve this issue.

Q5: The contribution of this paper is solid yet the overall significance and novelty is moderate.

A5: Thank you for acknowledging the novelty of theoretical derivation of the fused loss and insight of regularization. We believe that CoSIM contributions go beyond SiD in the following aspects:

• We theoretically introduce a regularization framework of the SiD loss to stabilize the training and solve the pathological behavior under large alpha values.
• We propose a continuous HSIVI training scheme. With a specific parametrization, we derive a novel training method for consistency models at a distributional level.
• Experimentally, our method performs better than the state-of-the-art consistency model sCD [1], using much less training budget (ours 20M compared to sCD 819.2M).

Table3: Sample quality measured by FID on class-conditional ImageNet (512x512).

[1] Lu Cheng et al. "Simplifying, stabilizing and scaling continuous-time consistency models."