There and Back Again: On the relation between noises, images, and their inversions in diffusion models

We thank the Reviewer for the valuable feedback and their recognition of its originality and importance.

The description of the experiments is too hasty and does not go enough into the details, making it very hard to understand. In particular, the whole paper is very confusing in the distinction between DDPM and DDIM. [...] Therefore, I suggest the authors to:

1. clearly define DDPM and DDIM early in the paper and consistently use these terms throughout.
2. explicitly state which model (DDPM or DDIM) is being used in each experiment in Section 4.
3. explain how the inversion process is applied to DDPM models if that is indeed what they are doing.

We appreciate the Reviewer's pointing out this confusion. We agree this is misleading; hence, we want to clarify it in this comment. We use the DDIM sampler in the diffusion model inference process for all the experiments in the paper. This assumption allows us to get deterministic generations from the given noise and approximate this noise from generations with the DDIM inversion method. However, the models were trained with the standard DDPM schedulers, so we used those two terms alternately, and following [1], we used this term to distinguish pixel-space models that we called DDPMs and latent models (LDMs). We agree that this term is inacurate, so we will change the names of these models to "Pixel DMs" in the revised version of our paper.

In Section 4.4, they refer to “the smallest L2 criterion” without introducing it. Moreover, the obtained results are confusing to me since I didn't expect a discrepancy in the accuracy of $x_0\rightarrow x_T$ vs $x_T \rightarrow x_0$. In a final version of this work, I expect the authors to:

1. Formally define the "smallest L2 criterion" when it's first mentioned.
2. Explain the classification process using this criterion in more detail.

Thank you for pointing out that the initial version of the submission lacked a thorough explanation of this aspect. For calculating a distance between two objects in our experiments, we use the $L_2$ norm (euclidean distance) between two images/noises/latents calculated as follows: ${||x-y||}_2 = \sqrt{\Sigma_c\Sigma_i\Sigma_j (x_{c,i,j}-y_{c,i,j})^2}$.

For the classification experiment in Table 2, we assume a setup where we have $N$ inputs, called, in the paper, initial Gaussian noises, and their $N$ corresponding diffusion model outputs, called image generations. When assigning images to the noises, we iterate over all the $N$ noises, and for each one, we calculate its distance to all the $N$ generations. We assign the images to noises by choosing the one with lowest $L_2$ distance. For the accuracy metric, we check if, for a given noise, the predicted image is the same as the one that results from denoising this particular noise with DDIM). For the reverse problem (assignment of noises to images), the setup is the same, but we iterate over the $N$ image generations and, for each image, calculate its $L_2$-distances to all the noises and choose the one to which a distance is lowest. Please note that that even though the L2 distance is symetrical, the assignment of the closest image/noise is not the same due to the one-directional many-to-one relation (e.g. There might be several noises pointing towards the same closest image).

3. Address the apparent discrepancy between the accuracies of $x_0\rightarrow x_T$ and $x_T\rightarrow x_0$, given the symmetry of the L2 norm.

While the $L_2$ norm is symmetrical, the problem here is the many-to-one relation that we perform. When assigning images to the initial noises, there are singular generations (with large plain areas) located close to the mean of the random Gaussian noise in the set of generated images. Such generations tend to be the closest (in $L_2$-Norm) for the majority of the noises in our experiments. We show examples of such wrong assignments in Figure 7 (A) in the Appendix. In (C), we present the singular generations that lead to incorrect noise-to-image classification, along with the number of noises for which they are the closest. In (B), we sort images used in the experiment by the variance of pixels and show four with the lowest one. We observe that the set of singular generations leading to misclassification overlaps with lowest-variance generations.

In line 348, after discussing the accuracy of the metrics between image and noises, the authors say “we can observe that the distance between noise and latents accurately defines…”. Note that, in this section, the latents were not considered, since all the experiments were performed on images and noises. Therefore I do not understand what they want to say with this sentence.

Thank you to the reviewer for pointing out this error. We confirm that in the sentence we meant "the distance between the noises and their corresponding generations".

We appreciate the feedback and Reviewer's positive opinion about our experiments. We would like to clarify the remaining questions in this comment:

The analysis focuses primarily on DDPM and LDM models, leaving open the question of whether the observed phenomena generalize to other diffusion model architectures.

To show that our findings generalize over more diffusion models, we performed our experiments also on other large-scale models. To that end, we used an unconditional pixel-space U-Net trained on images from ImageNet with $256\times256$ resolution and a class-conditional Diffusion Transformer (DiT) operating in the Latent Space of the autoencoder, trained also on images from ImageNet with $256\times256$ resolution.

First, we include those two models in our experiments to compare pixel correlation in Gaussian noises and latent encodings. The latent encodings created with reverse DDIM for large-scale diffusion models also have correlated pixel values. Surprisingly, the correlation is more significant for the pixel-space model operating at $256\times256$ resolution than for the $64\times64$ model.

Next, we continue this study in the experiment for determining the most probable angles located by the vertexes of images ($x^0$), noises ($x^T$), and latents ($\hat{x}^T$), with varying diffusion steps $T$. We show that, even for large-scale diffusion models, the latents are located along the trajectory of the generated image. Our observations with angles align closely with the correlation experiment.

We appreciate the Reviewer's valuable feedback.

Missing experimental details

For instance, the image resolution at which the models were trained is missing for all datasets. Similarly, details on the network architecture used for the diffusion denoiser and training hyperparameters are missing for both pixel space diffusion models and LDMs.

In the experiments for initial submission, we leveraged three diffusion models:

1. Unconditional pixel-space Denoising Diffusion Probabilistic Model (DDPM), with a U-Net architecture as a backbone. This model was trained on the CIFAR-10 dataset at image resolution $32\times32$. We use the checkpoint from [1]. This model was trained with $T=4000$ diffusion steps, with cosine schedule and hybrid loss (composed of simplified objective and variational lower bound loss). The model was trained for 500K training steps.
2. Unconditional pixel-space Denoising Diffusion Probabilistic Model (DDPM), with a U-Net architecture as a backbone, which was trained on the ImageNet dataset with image resolution $64\times64$. We use the checkpoint from [1], which, similarly to (1), was trained with $T=4000$, cosine schedule, and hybrid loss, but for 1.5M training steps.
3. Unconditional Latent Diffusion Model (LDM) trained on the CelebA-HQ dataset with images of resolution $256\times256$. This particular model is a U-Net-based denoising diffusion model inside the $3\times64\times64$ latent space of the VQ-VAE autoencoder. We use the trained weights from [2]. The denoising model was trained with $T=1000$ diffusion steps and a linear variance schedule for 410K training steps.

Additionally, as requested, we have added experiments on two additional diffusion architectures focusing on higher resolution data:

1. Unconditional pixel-space Denoising Diffusion Probabilistic Model (DDPM), with a U-Net architecture as a backbone, that was trained on the ImageNet dataset at image resolution $256\times256$. We use the trained weights from [3]. This model was trained with $T=1000$ diffusion steps and a linear variance schedule for 1980K training steps.
2. Conditional Diffusion Transformer (DiT), leveraging Transformer architecture as the denoising diffusion backbone inside the $32\times32\times4$ latent space of Variational Autoencoder, trained on ImageNet dataset with image resolution $256\times256$. For our experiments, we skip the classifier-free guidance. We use the trained weights from [4]. This model was trained with $T = 1000$ diffusion steps and a linear variance schedule for 400K training steps.

For analyzing noise, latent and sample properties with the training progress, we train two unconditional DDPMs with the U-Net architectures - $32\times32$ CIFAR-10 (1) and $64\times64$ ImageNet (2), with the same exact hyperparameters as [1].

Moreover, it is unclear from the text how the angles between different vectors were computed in Figure 2.

For the experiment with angles (Figure 4), we first sample 1000 example generations, and calculate the angles next to the image $\angle x^0$, the Gaussian noise $\angle x^T$, and the latent encoding $\angle \hat{x}^T$ by calculating the cosine similarity between two vectors attached at a given point and converting this value from radians to degrees.

On top of that, for the visualization in Figure 2, we create histograms for each triangle's vertex and obtain the probability density function for every angle binned up to the precision of one degree. Finally, for all triples of angles that can form a triangle (adding up to 180 degrees), we calculate the probability of such a triangle as the product of the probabilities of each angle. Finally, we visualize the triangles yielding the highest probability.

Secondly, in Section 4.4, what is the minimum L2 distance criterion for the assignment of images to noise mentioned in line 321?

For calculating a distance between two objects in our experiments, we use the $L_2$ norm of the matrix being a difference between the two objects: ${||x-y||}_2 = \sqrt{\Sigma_c\Sigma_i\Sigma_j(x_{c,i,j}-y_{c,i,j})^2}$.

For the experiment in Table 2, we assume a setup where we have $N$ inputs, called, in the paper, initial Gaussian noises, and their $N$ corresponding diffusion model outputs, called image generations. When assigning images to the noises, we iterate over all the $N$ noises, and, for a particular one, we calculate its distance to all the $N$ generations. We assign the images to noises by choosing those that has the lowest L2 distance. For the accuracy metric, we calculate how accurate is the assignment described above. For the reverse problem (assignment of noises to images), the set-up is the same, but we iterate over the $N$ image generations and, for each image, calculate its $L_2$-distance to the noises.

We thank the Reviewer for valuable feedback and valuable suggestions.

For the empirical observations in Sections 4.2 and 4.3, I don’t see any clear applications based on these findings. I suggest taking it a step further; for example, how could we reduce the divergence between the inversion and the noise? What insight does it provide if the inversion lies along the trajectory from noise to image?

Overall, without applications or theoretical insights, the empirical analysis lacks clear motivation. I hope the authors can identify relevant scenarios where these observations could be put to practical use.

Thank you for this suggestion. The main goal of our work was to study a behavior of a commonly used DDIM inversion technique in order to provide explanations and in-dept understanding of its limitations. Agreeing with Prof. Black [https://perceiving-systems.blog/en/post/novelty-in-science], we follow his point of view that a novel paper does not have to come with a direct application. Our insights shed a new light on the problem, show how approximation error influences the results of the reverse DDIM procedure, and how this behavior changes during diffusion training.

The observation in Section 4.4 is interesting, but the paper doesn’t explore any theoretical insights or potential applications related to this phenomenon. One possible direction could be to link it to optimal transport, building a theoretical framework to better understand the training dynamics of diffusion models.

Thank you for pointing out this interesting direction. There are several works discussing the connection between diffusion models' training dynamics and optimal transport and to the best of our knowledge, this topic is still to be defined. In [1] authors show that DDPM encoder map (e.g. latent-sample mapping) coincides with the optimal transport map when modeling simple distributions. However, as noticed by [2,3], the proof provided by [1] cannot hold. Our experiments show that the closest-L2-based mapping in the case of pixel-space DDPMs holds only in one direction, what might be an interesting starting point for more theoretical considerations. Moreover, we also highlight that this mapping appears relatively early in the diffusion model training, sheding some light on the dynamics of diffusion models' training.

Questions: From Table 2, are there any insights into why LDM could achieve 100% accuracy for both $x^0 \rightarrow x^T$ and $x^T \rightarrow x^0$, but DDPM could only achieve high accuracy for $x^0 \rightarrow x^T$. What key differences between LDM and DDPM might explain this discrepancy?

Thank you for this interesting question. We attribute the difference to the nature of the input data provided to the diffusion in the LDM models. In this scenario the diffusion model is trained on the latent data representations extracted by the autoencoder usually trained with additional regularization either by the KL-Loss or VQ-Loss. In both cases, the application of the regularization leads to the normalization of the input data. As presented in the Appendix, the reason why DDPM does not achieve high accuracy for the $x^T \rightarrow x^0$ case is that there exist some images, and hence, some generations are by nature located closer to the mean of the input data noise, so they ''attract'' more random noises. This is not the case for the LDM model.

References:

[1] Khrulkov, Valentin, et al. "Understanding ddpm latent codes through optimal transport." arXiv preprint arXiv:2202.07477 (2022).

[2] Kim, Young-Heon, and Emanuel Milman. "A generalization of Caffarelli’s contraction theorem via (reverse) heat flow." Mathematische Annalen 354.3 (2012): 827-862.

[3] Lavenant, Hugo, and Filippo Santambrogio. "The flow map of the fokker–planck equation does not provide optimal transport." Applied Mathematics Letters 133 (2022): 108225.