Casting Light on Large Generative Networks: Taming Epistemic Uncertainty in Diffusion Models

Thank you for the review.

Weakness 1:  

We agree that the motivation behind epistemic uncertainty estimation for image generation should be better addressed in the text. To that end, we have the following text to the paper:  

Collecting data for image generation models can be a costly endeavor. Therefore, when seeking to enhance a model, leveraging epistemic uncertainty becomes a crucial factor in selecting new data points. This concept is frequently employed in Active Learning methodologies such as BALD Houlsby et al. 2011 and BatchBALD Kirsch et al. 2019. This underscores the relevance of applying our framework. It is essential to acknowledge that, at present, these models entail significant training costs. As a result, we do not offer active learning experiments. However, with the anticipation of future advancements in computational resources, there may be increased feasibility to explore these ideas.

Weakness 2:  

To clarify, when running the experiments we leverage DDIM (Song et al. 2021) to generate images more efficiently. In doing so, we take larger steps in the reverse diffusion process than DDPM (Ho et al. 2020). Hence the timestep interval changes from 1 in the reverse diffusion process to 5. Please note this is highlighted in the text, “It’s important to note that we can only take steps of 5 through the diffusion process due to the 200 DDIM steps.”

Weakness 3:  

We acknowledge the interest in your experimental setup; however, due to the timeframe involved (more than 7 days for obtaining results), we lacked sufficient time and access to compute to thoroughly evaluate the idea. Currently, we are in the midst of expanding our work for a journal submission that will encompass your suggestion. The experiments detailed in the paper focused on isolating epistemic uncertainty within an active learning framework. In this context, the emphasis was on between-class epistemic uncertainty to inform the selection of new data points.

Question 1:  

For every random noise sample, our approach generates a distinct estimate of epistemic uncertainty for each class $x_i$. To ensure the robustness of our epistemic uncertainty measure, we take the average across the 8 samples. This process is explicitly described in the paper, "We then average the ensemble’s epistemic uncertainty over these 8 random noise samples.". It's important to note that we do not analyze aleatoric uncertainty. We do not compare the images generated from different samples of random noise.

Question 2:  

In Figure 4, the plotted epistemic uncertainty on a per-pixel basis reflects the ensemble's component-wise variability derived from a single random noise sample. Aleatoric uncertainty is not considered in this figure as we consider only one sample of random noise and we are considering the variability within the ensemble.

Question 3:  

The mutual information criterion considered in the paper, is difference between total and aleatoric uncertainty (i.e. $I(X,Y) = H(Y)-E[H(Y|X)]$, EU=TU-AU) (Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning, Depeweg et al. 2018). Mutual information measures the amount of information gained about one variable from observing the other. In the context of Eq. 4, given an observation of $\theta_i$ what do we know about the distribution of $y$. One would expect that given many instances of a certain class in the training dataset that the ensemble components would produce relatively similar distributions of the output. This is conveyed in the paper, “Mutual information measures the information gained about one variable by observing the other. When all of $\theta$’s produce the same $p_{\theta} (y_0|y_T, x)$, $I(y_{t−1}, \theta|y_t, x)$ is zero, indicating no epistemic uncertainty. Conversely, when said distributions have non-overlapping supports, epistemic uncertainty is high.”

Thank you for your insightful comments and the time you devoted to reviewing our paper. Your original score seems to indicate some confusion as to how our estimates for epistemic uncertainty differed from aleatoric uncertainty. We hope that our follow up has addressed those concerns and we kindly ask for your consideration in increasing your score, as this adjustment would significantly improve the prospects of our paper's acceptance

Thank you for the review.

Number of Components:  

While the number of components can be seen from Figures 2 and 6, we recognize that this information should have been more explicitly stated in the text. Therefore, we have included the following clarification:  

We utilize an ensemble of 5 components, a number we found to be sufficient for estimating epistemic uncertainty in our context.

Uncertainty Distributions:  

Epistemic uncertainty is a relative measure, and the community lacks consensus on the definition of groundtruth uncertainty. Consequently, a small value for epistemic uncertainty is not inherently incorrect. To our knowledge, we are pioneers in considering epistemic uncertainty in the realm of large-scale generative models, we interpret the observed differences between bin distributions in Figure 3 as indicative of the expected trend (i.e. bins with lower values are more skewed towards lower epistemic uncertainty). We hope to investigate further the inter-bin distributions in future research and our current findings provide valuable insights.

Image Similarity in Table 1:  

The model encountered challenges in generating accurate images for bins 1, 10, and 100, as illustrated in Figure 4. Consequently, it is understandable that the SSIM between components for these bins would exhibit similar values, irrespective of the branching point.

PaiDEs Applied to Classification:  

PaiDEs could be employed for classification tasks; however, considering the output distribution is categorical, one can directly compute the mutual information between model outputs and weights, denoted as $I(y,\theta|x)$, when dealing with a finite number of components. Hence, in the specific context of estimating epistemic uncertainty, it does not seem necessary for one to use PaiDEs to approximate $I(y,\theta|x)$.

Additional diagram for Frozen Weights:  

To address this we have included the following text in the caption of Figure 1:  

In our ensembles, networks taking class labels as input are randomly initialized and trained, with pre-trained encoders, decoders, and UNets for each component.

Thank you for your thoughtful comments and the time you invested in reviewing our paper. We have carefully considered your concerns and believe that our responses adequately address them. We kindly request you to consider increasing your score, as it would greatly enhance the likelihood of our paper being accepted.

Thank you for the review.

Zero Covariances:  

Please be aware that covariances are intentionally set to 0 as a design choice. In Song et al. 2020, they introduce DDIM and set $\sigma_t=\eta\sqrt{((1 - \alpha_{t-1}) / (1 - \alpha_t))(1 - \alpha_t / \alpha_{t-1})}$. They highlight that setting $\eta=1$ makes it equivalent to DDPM (Ho et al. 2020), and $\eta=0$ corresponds to DDIM. To enhance clarity, we have included the following information in the text:  

Furthermore, in the DDIM implementation by Rombach et al. (2022), the covariance, $\Sigma_{\theta} (z_t, t, x)$, is intentionally set to a zero matrix, irrespective of its inputs, aligning with the approach in Song et al. (2020).

Computational Analysis:  

In the paper we mention, "This significantly reduces the number of parameters that need to be trained, 512k instead of 456M, as well as the training time (by 87%), compared to training a full latent diffusion model on Imagenet,". This indicates that one component takes 13% of the time of training a full LDM on Imagenet. Additionally, we have included the following details at the end of Appendix A.1 for more information:  

Each ensemble component was trained in parallel and required 7 days of training with the specified computational resources.

Thank you for your thoughtful comments and the time you invested in reviewing our paper. We have carefully considered your concerns and believe that our responses adequately address them. We kindly request you to consider increasing your score, as it would greatly enhance the likelihood of our paper being accepted.

Thank you for the review.

Conditioning:  

The paper focuses on estimating epistemic uncertainty in a supervised learning context, as demonstrated by our conditioning on $x_i$, defined as the class label in the paper ("... where $x_i$ represents class labels ...") at the start of section 2.1. This consistency is maintained throughout the paper, except for Equation 1, where the forward process is described. In this equation, noise is added iteratively to an image independently of the class label.

Wasserstein Distance:  

Prior works did not explore the application of PaiDEs with the Wasserstein distance. Various distributional distance metrics, including Bhattacharyya, Chernoff-$\alpha$, Hellinger, KL, and Wasserstein, could be employed for PaiDEs. In our context, we could not utilize existing ones (Bhattacharyya and KL) as they were undefined. Consequently, we introduced a different distributional distance metric for PaiDEs to ensure it could be calculated within our context.

Experiments:  

Your comment appears incomplete as it concludes with a comma.