Accelerated Diffusion using Closed-form Discriminator Guidance

Thank you for your valuable feedback! Please find a detailed response to your questions below.

• Theory for discriminator-based generator process: The optimality of the GAN generator training, given the closed-form Coulomb and polyharmonic-spline kernel-based discriminator, has already been explored by Asokan and Seelamantula 2023a, 2023b. A similar analysis for neural-tangent-kernel-based discriminators was carried out by Franceschi et al. 2022. In summary, these results show that, given the optimal discriminator, the generator training converges to the optimal solution $p_g = p_d$. In consideration of your comment, to improve clarity, we have included a summary of this proof in Appendix D.1 of the revised submission. In the context of discriminator-guided Langevin diffusion, Lunz et al., 2018 have already proved the convergence of iterates involving $\nabla D_t^*({x}_t)$. We already discuss the application of this convergence result to our solution in Appendix D.3.

• Comparisons against Kim et al., 2023 [1]: We already have this comparison (LDM+$DG_\theta$ in Table 2). [1] consider discriminator guidance in both the latent space (with LSGM-G++, on CIFAR-10) and in the pixel space (with a pre-trained encoder and the ADM classifier). Since we present experiments on latent-space generation on CelebA-HQ (cf. Sec. 5), we reproduced the LSGM-G++ setting (from Table 8 in [1]) for this dataset, which is the LDM+DG$_\theta$. This is already discussed in Lines 480-485.

• Baselines for accelerated diffusion sampling: The proposed discriminator guidance term is orthogonal to baselines such as [2,3] wherein better ODE solvers are used to accelerate sampling. As such, the closed-form discriminator guidance (+DG$^*$) can be combined with these techniques as well. Given the limited rebuttal timeline, we also include experiments on CelebA-HQ, consider the suggested DPM solver [2], with and without +DG$^*$ (cf. Appendix E.2, Table 4 of the revised manuscript). We observe that including discriminator guidance allows us to further accelerate the sample generation process, with the DPM+DG$^*$ sampler achieving comparable performance in $T=15$ steps (1 discriminator step and 14 DPM-solver steps, CLIP-FID = 9.66), as the baseline DPM model with $T=20$ (CLIP-FID = 9.50). On the other hand, the DPM+DG$^*$ with $T=20$ outperforms (CLIP-FID = 9.22) the baseline for the same $T$. We will include comparisons with the Fast-ODE solver [3] in the camera-ready version of the manuscript.

• Ablation w.r.t number of samples: The closed-form discriminator is computed as a sample estimate using a batch of $M \leq B$ centers, where $B$ is the batch size, set to 80 in our experiments, and can be selected based on the available compute budget. However, since the discriminator guidance only affects the initial stages of sampling, the choice of $M$ does not affect the generated image quality. To illustrate this effect we now include Figure 21 in the Appendix, which presents images generated for varying numbers of centers $M$ considered in the closed-form discriminator.

• Closed-form discriminator guidance and accelerated sampling: As we discuss in Sec 3.1 (Lines 297-304), unlike the score, DG$^*$ provided a strong repulsive force away from the initial distribution, pushing the samples towards the target $p_d$. This repulsive force accelerated the sampling process in earlier iterations. To validate this, as we show in Fig. 6 of the Appendix, on comparing the iterate convergence of the closed-form discriminator-guided diffusion (DG$^*$) against the baseline NCSNv1, we observe that the iterate convergence (the value of $\|\mathbf{x}_{t-1} - \mathbf{x}_t\|$ as t decreases) is much quicker for DG$^*$. To add intuition, we have included Figure 19 in the Appendix, to illustrate this push-pull gradient field on toy data. In Summary, in addition to mitigating the distribution mismatch (as seen in standard discriminator guidance), the closed form discriminator also accelerates sampling by providing stronger gradients in earlier training steps. Please also see our response to Reviewer Z719 on the jump seen with DG$^*$.

• Expression for $\kappa$: We already present expressions for various choices for $\kappa$ In Table 8 of the Appendix. We have included the expression of an example kernel in Section 2.2 of the revised Manuscript.

• On memorizing the training dataset: Using DG$^*$ does not result in memorizing the training data. We have already provided ablation on this, for the MNIST, SVHN and CelebA datasets in Appendix E.2 of the submission (cf. Fig 11-13). We have also included similar comparisons on CelebA-HQ as part of the revised manuscript (cf. Fig. 20 of the revised manuscript)

Please let us know if any additional clarity is needed. We hope that we were able to address your concerns regarding our submission and that you'll consider raising your score in case the questions were resolved.

Thank you for your response and constructive feedback. We would like to provide additional clarity on the closed-form discriminator-guided sampling, and its links to Langevin Sampling.

• Does closed-form discriminator guidance truly accelerate sampling? We would like to reiterate that, as we mentioned in our response, The acceleration achieved by the proposed discriminator guidance and accelerated sampling approaches are orthogonal to one another. in the context of the discriminator guidance, acceleration is defined by means of the strong push-pull nature of the gradient field of the discriminator, which pushes the initial distribution towards $p_d$ with significantly stronger gradients as compared to the standard score -- the strength being defined via the chosen kernel. In accelerated sampling, as defined in the context of [1,2], Kim et al., etc, acceleration is in terms of improved discretization of the SDEs, or in better approximations of the score. The closed-form discriminator does not improve SDE discretization or score approximations. It provides an alternative gradient field, which is stronger than the score field when far away from the data distribution during early stages of sampling.

• Applicability of Lunz et al.: The results of Theorem 2 and Lemma 3 indicate that the convergence of a sampler (such as the GAN generator, or Line 342) with access to the closed-form optimal discriminator, and in the absence of noise, is guaranteed in 1 step, i.e., if one had access to $D_{t-1}^{\*}$ and $p_d$ and $p_{t-1}$, then $G_{t}(z_t) = x_t \sim p_t = p_d$. However, this is not achievable in practice, both due to the inaccessibility of $p_{t-1}$, and sample approximations of $D_t^{*}$. The results of Lunz et al. [3], covers a scenario wherein access to the optimal discriminator is not available, which is true in the case of DG$^{\*}$ approximated with a sample estimate. We show, via the experiments in Section 4 and ablation in Appendix E.2 (Figures 8-10), even in the case of the proposed closed-form discriminator-based Langevin sampling without the score, the optimal performance is indeed achieved in the noise-free setting.

• Provided references and discriminator guided diffusion: While we agree with the reviewer that there are primarily GAN works, as we discuss in Sec. 1, approaches such as Franceschi et al. 2023 [4], Zhang et al., 2023 [5], have extended this analysis to the diffusion/generative-particle-model settings. Additionally, [4] has already shown that score-based diffusion models, GANs, and gradient flows can be unified as a family of generative particle models, and derive convergence results for Langevin sampling with neural-network-based discriminators (Finding 8 of [4]). We already discussed these approaches in Lines 122-128). Our results can be viewed as a novel extension of their setting, wherein the convergence would be guaranteed in 1 step, given access to the closed-form $D_t^*$ and $p_{t-1}$ (Theorem 2, Lemma 3 of the submission).

• Convergence analysis of DG$^{\*}$ in Section 5.1: We agree that the neither SDE-based convergence/acceleration analysis, nor Theorem 2/Lemma 3 readily apply to the Langevin formulation presented in Section 5 and we are currently exploring approaches to extending the existing results (such as those present in Kim et al., 2023) to this setting. However, as we discuss in Sec 5 (Lines 466-472), and as shown by Li et al. (2024) [6], diffusion models contain two stages: (a) The first stage representing the movement of the Gaussian of diffusion models input towards the image distribution, during which no modes are present, and (b) the second stage, wherein patterns and structure emerge from latching onto a specific image to generate from the target distribution. The proposed DG$^*$ framework, akin to time-step shifting (Li et al., 2024), can be viewed as improving the convergence of the first stage, while the standard score-based sampling (and the associated convergence guarantees) hold for the second stage.

[4] Franceschi et al. “Unifying GANs and Score-Based Diffusion as Generative Particle Models”, NeurIPS 2023

[5] Zhang et al. “Diffflow: A unified SDE for score-based diffusion models and generative adversarial networks.” arXiv 2023

[6] Li et al., “Alleviating exposure bias in diffusion models through sampling with shifted time steps,” ICLR 2024

Please let us know if any additional clarity is needed.

I thank the authors for coming up with a proof in such a brief notice. I find that it strengthens the proposed method significantly, and I have increased the score from 3 to 5. I am not sure about increasing the score further for three reasons:

• I feel the paper would have to undergo a major restructuring in order to incorporate the additional theory,
• the provided proof does not shed any light on the acceleration effect of discriminator guided sampling,
• there is a discrepancy between theory and practice -- the provided Lemma shows one can apply discriminator guidance to all sampling steps without reduction in sample quality. However, Table 2 in the main paper shows increasing the number of discriminator steps during sampling results in worse FID.

Thank you for your valuable feedback! Please find a detailed response to your questions below.

• Experiments of CIFAR10 and ImageNet64: The experiments with discriminator guidance (DG$^*$) and EDM sampler are carried out in the image space. As already noted in the caption of Figures 15 and 16, the DG$^*$ approach leads to sub-par image quality on pixel-space generation due to challenges in computing the closed-form discriminator in high-dimensional spaces, which is why we extend DG$^*$ to work with LDMs in Sec. 5. For completeness, we now include experiments on CIFAR-10 with the DPM sampler [1] with and without DG$^*$, applied on the image space. The DPM+DG$^*$ model archives a CLIP-FID of 12.07, compared to the CLIP-FID of 11.51 achieved by the baseline DPM solver. As expected, the model is comparable to, but does not outperform the baselines in the pixel space. We also improve the clarity on the limitation of the experiments with EDM sampler in Lines 411-419 of the revised manuscript.

To further demonstrate the experimental performance of LDM+DG$^*$, we now also report FID scores on the LSUN-Churches-256 dataset, which we summarize in the table below, with additional ablations present in Table 6 of the revised manuscript.

[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022

• Score estimation: In all experiments, the score is computed using the neural network, identical to the corresponding baselines, using the provided pre-trained models. There is no Monte-Carlo sampling required to estimate the score. The closed-form discriminator is computed as a sample estimate using a batch of $M \leq B$ centers, where $B$ is the batch size, and can be selected based on the available compute budget. However, since the discriminator guidance only affects the initial stages of sampling, the choice of $M$ does not affect the generated image quality. To illustrate this effect we now include Figure 21 in the Appendix, which presents images generated for varying numbers of centers $M$ considered in the closed-form discriminator.

• WANDA vs LDM-DG$^*$: $T_D$ is unique to the WANDA algorithm, is the the application of Time-Shifted Sampling (Li et al., 2024) using wavelet-based noise estimation for of the LDM-DG$^*$ algorithm. $T_D$ denotes the time step up to which time-shifting is applied. We have improved the clarity of this distinction in Sec 5.1 of the revised manuscript.

• The choice of $T_D$: $T_D$ denotes the transition point from discriminator guidance to score-based sampling in WANDA. When $T_D=0$, there is no discriminator guidance and this corresponds to the baseline LDM algorithm, which is worse than WANDA/LDM+DG$^*$. Intuitively, it is beneficial to include discriminator guidance in the early stages of the samples process, but as we note in Fig. 6 of the Appendix, the iterate convergence (the value of $\|\mathbf{x}_{t-1} - \mathbf{x}_t\|$ as t decreases) is much quicker for DG$^*$. Therefore, a small number of closed-form discriminator guidance steps suffice to improve generation quality. Please also see our response to Reviewer #Z719, and Table 5 of the Appendix in the revised manuscript for additional details.

• On Lemma 3: Lemma 3 does not contain the data score, but draws an analogy between the data score $\nabla \ln p_d$ and the gradient field of the kernels convolved with the density difference, which is what we discuss in Lines 297-299. To illustrate this, we have included a new Figure 19 in the Appendix, comparing the data score and the discriminator kernel gradient field.

• Combining closed-form discriminator guidance and score: The score and the closed-form discriminator as combined as shown in Equations on Lines 412-416 (Lines 426-429 of the revision). The guidance strength is given by $w_{dg,t}$. In LDM-DG$^*$, the weight is decayed exponentially, as explained in Lines 417-420, with ablations given in Section 5.1 and Table 5 of the Appendix.

• RGB Color online: The note is for accessibility and is meant to indicate that the figure includes color-based disambiguation and/or is best viewed digitally, instead of on-print media.

Please let us know if any additional clarity is needed. We hope that we were able to address your concerns regarding our submission and that you'll consider raising your score in case the questions were resolved.

Thank you for your response:

• On the performance on CIFAR-10: We now also include a comparison on the images generated by the DPM+DG$^*$ sampler, compared against the baseline DPM sampler, in Figure 23 of the revised manuscript. The figure clearly shows that the images generated are visually on par with state-of-the-art performance of DPM.

• On the visual quality of the generated images: As we have shown on the FFHQ and CelebA-HQ datasets, the proposed approach (LDM+DG$^*$) does indeed result in improved visual quality. This is evident from Figure 4 of the main manuscript, and Figures 17 and 18 of the Appendix. As stated in the response above, we also show this for CIFAR-10 with the DPM sampler in Figure 23.

• On the effect of $T_D$: The baseline corresponds to the line labelled LDM in Table 1. As $T_D$ increases (increasing the number of steps for which we provide guidance, i.e., the first $T_D$ steps starting from iteration $t=T$) the performance indeed improves at first, and then deteriorates, which we also observe from the ablations in Table 5 -- We see that setting $T_D$ to about 10% of the total number of iterations $T$ is a viable choice. This behaviour has already been observed in the literature by Li et al. (2024) in the context of standard time-step shifting. Diffusion models basically contain two stages: (a) The first stage representing the movement of the Gaussian of diffusion models input towards the image distribution, during which no modes are present, and (b) the second stage, wherein patterns and structure emerge from latching onto a specific image to generate from the target distribution. Acceleration mechanisms such as time-step shifting (Li et al., 2024) and the proposed DG$^*$ operate in the first stage, which is why we focus the discriminator guidance to earlier iterations.

We hope that we were able to address your concerns.

We would like to thank you for your continued discussion. While we understand that your evaluation of our work in not centered about the GAN framework presented, we would like to emphasize that, from an experimental standpoint, we have shown the scalability of the proposed DG$^*$ to generate high-dimensional images by means of the LDM and DPM samplers, and the time-shift scaling acceleration. In these scenarios, inclusion of discriminator guidance demonstrated improvements over each corresponding baseline model, both in FID, and visual quality. Further, we would like to add that:

1. While we mention brute-force scaling of the closed-form discrimination guidance as challenging in the pixel space, we note that it is possible to achieve performance gains in the pixel space, and it requires an in depth analysis which is beyond the scope of this paper. The experiments on CIFAR-10 with the DPM sampler (Figure 23) are a proof of concept for this. For example, to scale, approaches such as improved batching/batch sampling strategies from other domains [4,5] could be employed to pick better centers used in DG$^*$, without scaling batch size $M = B_{DG^*}$ with dimensionality $n$. One could also explore clustering-based approaches to choose more representative centers. Note that $B_{DG^*}$ can be independent of the number of images being generated, $B$. These are all promising directions for future research.
2. As you rightly point out, “text-2-image generation” is a crucial direction for diffusion models and latent-space approaches, particularly LDM-based, are state-of-the-art, e.g., [1,2,3]. Given that DG$^*$ has shown performance gains with LDM, (Tables 1,4,6, Figures 4, 17 and 18), we believe that scalability is not an issue with the proposed approach and closed-form discriminator guidance can also be applied in this setting in the latent space.

[1] (SDXL) Podell et al., “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” ICLR 2024 Spotlight

[2] (Diffusion-ViT) Bao et al., “All are Worth Words: A ViT Backbone for Diffusion Models,” CVPR 2023

[3] (RAT-Diffusion) Ye et al., “Recurrent Affine Transformation for Text-to-Image Synthesis” Trans. MM, 2023

[4] Dahiya et al., “NGAME: Negative mining-aware mini-batching for extreme classification,” WSDM 2023

[5] Xiong et al., “Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval,” ICLR 2021

We hope that this is taken into consideration in rating our work positively, in terms of the experimental merit demonstrated.

Thank you for your valuable feedback! Please find a detailed response to your questions below.

• On the GAN generator optimality and discriminator optimality: The optimality of the GAN discriminator has previously been studied by Mroueh et al.,2018; Unterthiner et al., 2018; Liang, 2021; Franceschi et al. 2022, and Asokan et al., 2023. What we show in this submission, is that, given the optimal discriminator $D_{t-1}^*$ (that admits a kernel-based interpolation form) at training iteration $t-1$, the optimal generator at the subsequent iteration $G_t^*$ can be derived as a one that minimizes the value of the convolution between the density difference, and the gradient of the optimal discriminator kernel $((p_d - p_{t-1}) *\nabla\kappa)$. For most popular positive-definite kernels $\kappa$ (cf. Table 3 of the Appendix), this term would be minimized when the generator’s distribution $p_t$ moves towards the data distribution $p_d$. We have incorporated this improved discussion on this optimality of the generator and discriminator in Section 3.1 of the revision.

• On the jump post discriminator guidance: Thank you for the suggestion. To further validate our claim, we have included an experiment wherein we plot the time-step jump predicted by the noise-variance-based time-shifted sampler at each step $t$. Since the step can occur at different $t$ for different images, we plot this curve. We performed the experiment over multiple images and observed that on average, the jump is about 2-10% of the total steps. Illustrative plots of the predicted time vs the actual time $t$ of the iteration, wherein the discriminator guidance improves performance gains over the baseline time-shifting algorithm are provided in Figure 22 of the revision.

• Focussing on earlier iterations with DG$^*$: As we show in Fig. 6 of the Appendix, on comparing the iterate convergence of the closed-form discriminator-guided diffusion (DG$^*$) against the baseline NCSNv1, we observe that the iterate convergence (the value of $\|\mathbf{x}_{t-1} - \mathbf{x}_t\|$ as $t$ decreases) is much quicker for DG$^*$ -- which suggests that it is beneficial for the sampler to introduce the discriminator early on and subsequently switch to the score-based model. Furthermore, as noted by Li et al. (2024), diffusion models basically contain two stages: (a) The first stage representing the movement of the Gaussian of diffusion models input towards the image distribution, during which no modes are present, and (b) the second stage, wherein patterns and structure emerge from latching onto a specific image to generate from the target distribution. Acceleration mechanisms such as time-step shifting (Li et al., 2024) and the proposed DG$^*$ operate in the first stage, which is why we focus the discriminator guidance to earlier iterations. We have included this clarity of the revised manuscript (cf. Section 5)

• Ablations on DG$^*$: We have now included additional experiments on DG$^*$, based on the choice of $T_D$ and $w_{dg,t}$ on the FFHQ dataset. The detailed experimentation is present in Table 5 of the Appendix. In summary, we observe that discriminator guidance performs best when run for less than 20% of the overall iterations (i.e., $T_D=5$ for $T=50$ or $T_D=5,10$ for $T=100$) and with the discriminator weight $w_{dg}\in(0.5,1)$. We observe similar trends when running discriminator guidance with the DPM solver [1] (Please see our response to Reviewer QDuM).

• Ablations in the last paragraph of Sec. 5:. You are right in your observation that this experiment is w.r.t. WANDA. We have fixed this bug.

• Meaning of $w_{dg,0}$: $w_{dg,0}$ is the weight given to the discriminator guidance term at $t=0$. We have improved the clarity in Section 5.1 of the revised manuscript.

Please let us know if any additional clarity is needed. We hope that we were able to address your concerns regarding our submission and that you'll considering raising your score in case the questions were resolved.

[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022