We thank the reviewer vKuM for their feedback. Please find our detailed answer below.

...trajectories of the probability distributions for the forward process in the diffusion model and the MMD gradient flow coincide ?

This is indeed an important point. The trajectories might not coincide because in principle if one starts with particles $X_t|X_0$, one step of MMD gradient flow with $MMD_t$ (MMD which uses features conditioned on t), might produce particles Z which do not correspond to a noise level t’=t-dt, seen during training. Even if the noise levels do not coincide with the forward process, however, MMD_{t} is a valid MMD for any t in $[0,1]$ implying that we can take its corresponding MMD gradient flow. Following these gradient flows would bring us closer to the target distribution $P_0$ and the only potential problem is the speed of convergence.

Since $t$ corresponds to a noise level from the forward diffusion process, $MMD_{t}$ for $t\sim1$ was trained to distinguish between clean data and noise and $MMD_{t}$ for $t\sim0$ was trained to distinguish between clean data and slightly noisy data. Making the parallel with a Gaussian 2D example (see Figure 2), $t$ would encode the kernel scale parameter. If one follows the MMD gradient flow when $t$ is large, the gradient is only large if the samples are very noisy and is small otherwise. In order to avoid the case where the sampled noisy particles have noise level $t'$ higher than $t$ used for $MMD_{t}$, we use a two stage procedure described in Algorithm 2, where for each sampling-time noise level, we do multiple MMD gradient flow steps. This allows us to gradually decrease noise level to keep MMD gradient high.

The generation process is computationally expensive. How many steps are required for each time during sample generation (Eq 9)?

Please see Appendix F for more details. For most of the experiments, we use $100$ steps in total which do $N_{s}=10$ gradient flow steps for each of $N_{noise}=10$ noise levels.

For a general kernel, the sampling from DMMD requires access to the training data (Eq 10). Only for linear kernel, this issue can be avoided by saving the average features for each time (Eq. 12).

That's correct. But since we use a linear kernel on top of the learned Neural Network features, the overall resulting kernel is in fact not a linear kernel but rather a flexible learned kernel, see eq.15 in Appendix B. In fact, we explored using RBF and Rational Quadratic kernels instead of linear, for image generation and we did not find that it leads to the improvement in performance.

Could you provide the wall-clock time comparison to demonstrate the computational benefits of approximate sampling in Table 2?

Please see our answer above the computational complexity. As we state above, during sampling time, if we use linear kernels with ordinary witness function, the computational complexity of one sampling step is $O(N)$, where $N$ is the number of samples which we produce simultaneously (due to the interacting particles structure). When using approximate sampling, we decrease this complexity to $O(1)$.

Is this method applicable to higher-dimensional datasets, such as CelebA-HQ (256x256)?

Please see our common answer to all the reviewers.

Could you provide the potential advantages of this approach compared to other non-adversarial dynamic generative models, such as Flow Matching [1]?

Diffusion models try to reverse a Gaussian noising dynamic while flow matching methods try to match an interpolation dynamics between the data distribution and a Gaussian distribution. Both methods can be shown to be equivalent given the choice of the schedule. In particular, both methods are relying on the fact that the target distribution is Gaussian and that the forward process is given by a Ornstein-Uhlenbeck (or Brownian motion) SDE.

On the other hand, our approach is agnostic to the destruction process and does not rely on an underlying Gaussian assumption of the prior distribution. In particular, one could design destruction processes which are more adapted to the data. While extensions have been proposed for such destruction processes (see [1,2] for instance) they either rely on a soft Gaussian assumption for the noising process [2] or do not enjoy the theoretical properties of our framework [1], i.e. at optimality we recover the data distribution, up to the discretization error.

We will add this discussion on the relation to flow matching to the final version.

References:

[1] Bansal et al. – Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise

[2] Daras et al. – Soft Diffusion: Score Matching for General Corruptions

We thank reviewer xQpY for the feedback and suggestions. Please find our answers below. We would be happy to further discuss any additional questions that might arise. If we have satisfactorily addressed your questions, we would be grateful if you could reconsider your score.

prev. research utilized diffusion process in the discriminator [1]

Thank you for pointing out this connection. We would like to highlight that eq.(6) in our paper corresponds to the forward diffusion (FD) process and eq. (6) in [1] also corresponds to the FD process specified in their appendix. If your point was that we both use a FD process, then it is a correct observation.

Please note, however, that even though at first glance, the discriminator objective in [1] and in our paper look similar, they are in fact quite different, since in [1], the authors construct a discriminator to distinguish between $x_{t+1}$ produced by a trained generator (see eq.5 in [1] and eq.4 in [1]) and $x_{t}$ produced by the forward diffusion process, for a given noise level $t$. This difference arises because the purpose and algorithm in [1] are fundamentally different to ours: the authors of [1] aim to train a GAN conditional generator which can do big denoising steps (vs the small, Gaussian denoising steps in a traditional diffusion model). Our objective function is $D[x0; xt; t]$, where $x_0$ is clean data and $x_t$ is noisy data from the forward diffusion process, and $D$ is the discriminator divergence. Our objective function aims to build a discriminator that is able to distinguish clean data $x_0$ from noisy data $x_t$ produced by the forward diffusion with a given level of noise $t$. Here the noise level $t$ is roughly (see Section 4 in our paper for discussion as well as experiments on toy data) related to the quality of the GAN generator (if we were to use a GAN generator), such that $t\rightarrow1$ corresponds to a generator at the beginning of the training and $t\rightarrow0$ corresponds to a generator at the end of the training. We emphasize that this is only an intuitive correspondence, since we don’t actually use a generator!

Therefore, the optimization problem in [1] is adversarial since it involves training generator/discriminator pairs (so as to take large denoising steps), whereas the optimization problem in our work is not adversarial since we use a fixed sequence of distributions (from the forward diffusion process) to train a discriminator across corresponding noise levels. See our reply above for more details on adversarial training.

...no theoretical proof for MMD GAN can converge to more optimal points with diffusion process

If we have understood this question correctly, it addresses whether an MMD diffusion (with no generator) will be able to better match the target distribution than an MMD GAN (with a generator). This will be the case if the generator has insufficient capacity to represent the target distribution, compared with a collection of particles. For instance, if the generator is a single Gaussian, but the target is a mixture of two Gaussians, then the MMD diffusion can attain the target distribution, but the MMD GAN will not. It appears from empirical studies that limitations in generator capacity are one major reason why diffusions have now taken over as the dominant image generating paradigm (the other is that adversarial training is difficult due to its min-max nature). In all experiments on images, the MMD flow strongly outperforms the MMD GAN.

Low resolution and SOTA

Please see the answer to this point in the common answer section.

training time...

See our common answer above regarding computational complexity of DMMD.

impact of forward process...

We have not investigated the impact of the diffusion process on the DMMD model performance but took the default forward process from the DDPM paper [1]. As we explain in our common answer, investigating the impact of forward process might improve DMMD performance and is the subject of future work.

sampling time dependent on resolution?

At sampling time, we compute gradient of MMD witness function which is 2x cost of the forward pass, and the forward pass cost scales proportionally to the resolution.

There are writing errors in the article, particularly in Table 4 and Algorithm 2.

Thank you for bringing these to our attention. We will remedy these errors in a final version.

References:

[1] Denoising Diffusion Probabilistic Models, Jonathan Ho, Ajay Jain, Pieter Abbeel

[2] Johnson R, Zhang T. Composite functional gradient learning of generative adversarial models[C]//International Conference on Machine Learning. PMLR, 2018: 2371-2379.

[3] Unifying GANs and Score-Based Diffusion as Generative Particle Models Jean-Yves Franceschi, Mike Gartrell, Ludovic Dos Santos, Thibaut Issenhuth, Emmanuel de Bézenac, Mickaël Chen, Alain Rakotomamonjy, 2023

[4] Deep Generative Wasserstein Gradient Flows, Alvin Heng, Abdul Fatir Ansari, Harold Soh, 2023

Inpainting & Super-resolution?

In principle, one could use DMMD for tasks which go beyond generative modeling. In this work, we did not investigate the performance of DMMD on such tasks. We believe that in order to properly test DMMD in these scenarios, more work is required since it is typically not the case that using a model in these scenarios works out of the box. We appreciate your suggestion and think it would be worthwhile to study these use-cases in future work.

vector Z in Eq.(15) to train a generator...

We believe you refer to eq. (9) and not to eq.(15), since eq.15 refers to a kernel $k(x,y)$ in Appendix B. It is not entirely clear to us what you mean by “using vector Z in Eq.(15) to train a generator”. What generator are you referring to?

We do not have a generator in our method. We use noise-adaptive MMD gradient flow only at sampling time which essentially replaces a generator. At training time, we only use a fixed forward diffusion process. It is unclear to us what you mean when you mention using vector Z from eq.15 to train a generator, but we would be happy to discuss further if you can provide further details on this idea.

In anticipation of future questions that might arise: you might be thinking of the fact that Eq.3, Eq.4, Eq.,9 in the MMD flow are the same theoretical form as the functional gradient flow Eq.(5) in the article [2], where eq.5 is used to train a generator to approximate the sample from the target distribution. Moreover, in our paper, eq.9 employs a negative gradient from the discriminator, so you might further suppose that this would result in an adversarial process – something which is used in [2]. This is, however, not correct because there is a crucial difference between our work and [2].

In [2], there is the alternation of training of a generator and a discriminator as with Generative Adversarial Networks (GAN): see our point about adversarial training. The generator is updated at each step using a "functional" update, which involves taking the previous generator at time $t-1$ and updating it in the optimal direction indicated by the current discriminator at time $t$ (in the case of KL, the update direction is the derivative of the log density ratio given by the discriminator): see their Algorithm 1 Step 4. This means that their generator is expressed as a weighted sum of derivatives of the discriminator networks at previous steps: see their Alg. 1 Step 4.

There is a fundamental and critical difference between [2] and our method: in [2], a discriminator is trained at each step on particles produced by the generator at the previous step. This means that the procedure is adversarial: the generator moves in the direction that minimizes the discriminator loss; the discriminator then gets updated to distinguish current generator samples from real samples; and the cycle repeats. By contrast, our method is not trained adversarially. The training samples are provided by a forward diffusion, which is fixed in advance, and never changes, and is obtained simply by adding increasing levels of noise to clean samples. This is a fundamental difference with [2], where the training samples are the output of a generator, which evolves as the discriminator evolves. There is no generator in our approach.

The approach of [2] is in fact quite related to the approaches [3-4] cited in our paper, which accomplish similar adversarial updates (where this process is called "discriminator flow"). A challenge in using a discriminator flow approach, also noted in both [2] and in our paper (line 301), is that it requires maintaining a large number of intermediate discriminator networks to define the final generator at the end of the training process (see figure 1 in [2]), resulting in a generator of large size and high computational cost. Heuristics may be used to approximate this large network (as in [2]). Our approach does not suffer from this high computational overhead.

We think it's worthwhile for us to cite [2] in a revised version of our paper, and to add discussion of [2] to the literature review.

References:

[1] Denoising Diffusion Probabilistic Models, Jonathan Ho, Ajay Jain, Pieter Abbeel

[2] Johnson R, Zhang T. Composite functional gradient learning of generative adversarial models[C]//International Conference on Machine Learning. PMLR, 2018: 2371-2379.

[3] Unifying GANs and Score-Based Diffusion as Generative Particle Models Jean-Yves Franceschi, Mike Gartrell, Ludovic Dos Santos, Thibaut Issenhuth, Emmanuel de Bézenac, Mickaël Chen, Alain Rakotomamonjy, 2023

[4] Deep Generative Wasserstein Gradient Flows, Alvin Heng, Abdul Fatir Ansari, Harold Soh, 2023

High-level clarifications about adversarial training In anticipation of follow-up questions that might arise, we begin by clarifying what we mean by adversarial training. We will add this discussion in the Appendix of the revised version of our paper.

In the context of Generative Adversarial Networks (GANs), the objective function is

$F(\theta, \psi) = \mathbb{E}_{Z \sim U[-1,1]} D[X; G(Z; \psi); \theta]$,

where $G(Z, \psi)$ is a generator with parameters $\psi$ and $D[X; G(Z; \psi); \theta]$ is a discriminator divergence with parameters $\theta$.

Objective for the generator: $\psi^* \leftarrow \arg\min_{\psi}\max_{\theta} F(\theta, \psi)$

Objective for the discriminator: $\theta^* \leftarrow \arg\max_{\theta}\min_{\psi} F(\theta, \psi)$ In practice in GANs, this objective is approximated by alternating updates. Let $n$ be the iteration number.

Update generator as: $\psi^{n+1} \leftarrow \arg\min_{\psi} F(\theta^n, \psi)$

Update discriminator as: $\theta^{n+1} \leftarrow \arg\max_{\theta} F(\theta, \psi^{n+1})$

We thank reviewer KT31 for the feedback and suggestions. Please find our responses below. We would be happy to further discuss any additional questions that might arise. If we have satisfactorily addressed your questions, we would be grateful if you would consider increasing your score.

The framework's absolute performance is a concern, as DMMD shows a significant performance gap compared to DDPM and more modern methods on the selected image generation benchmarks.

Please see our common response above regarding SOTA.

Its broader application potential is limited, with empirical evaluation restricted to small datasets like MNIST and CIFAR.

Please see our common answer above.

The sampling method appears restrictive, requiring reference features from the ground truth dataset to formulate the witness function.

The sampling method comes from the fact that the witness function is by construction explicitly dependent on the data distribution of the target, since the generated particles are required to interact with the target particles, as well as with each other. Any method which applies Wasserstein Gradient Flow to MMD metric will have a similar property in the sampling method.

That being said, one of the main contributions of this work is to provide a solution to this limitation, and to remove this restriction, by pre-encoding the features of the target dataset into the average features (See Section 5, especially lines 313-315 and eq.12).

The need for dataset features during sampling seems counterintuitive. Do the authors have any insights into potential solutions for addressing this limitation?

As mentioned above, our implementation does not require dataset features through the pre-encoding procedure. See the response to the previous comment.

We thank the reviewer QpwN for the positive feedback on our work. Please find our answers below.

While the paper shows promising results, it is still outperformed by standard diffusion models, especially in terms of FID scores. Further work might be necessary to reach SOTA performance on larger datasets like ImageNet.

Please see our common answer above.

Related to the previous point, the experimental results are primarily limited to smaller datasets (CIFAR10, MNIST, CELEB-A, and LSUN Church), which may not reflect the potential scalability of DMMD to more complex, high-resolution datasets.

Please see our common answer above.

Although the method avoids adversarial training, the noise-adaptive MMD flow still introduce complexity, which may limit reproducibility.

Sampling from the noise-adaptive MMD flow introduces complexity in the sense that one must choose the hyperparameters for the sampling process. The problem of hyperparameter choice at sampling time also arises for diffusion models – i.e., the performance of diffusion models differs drastically depending on the sampling scheme chosen. Details on hyperparameter choice are provided in Appendix D.

As mentioned above, what challenges do you consider in scaling DMMD to larger datasets like ImageNet?

See above: the implementation for high resolution images like imagenet would require defining an MMD diffusion in latent space. This is an important topic for future work.

What factors limit DMMD's FID performance relative to state-of-the-art diffusion models?

Please see above for our common answer.