Edge-preserving noise for diffusion models

We provide additional comparisons with the SOTA Simple Diffusion (Hoogeboom et al., 2023), being outperformed in unconditional tasks but consistently outperforming in shape-guided generation.

For shape-guided image generation, we also consistently outperform this state-of-the-art pixel-space isotropic baseline:

Although we get outperformed in unconditional image generation, we are optimistic that incorporating edge-preserving noise into Simple Diffusion will enhance its performance, as we saw with DDPM. We contend that these are reasonable expectations, as both our method and theirs align with the denoising probabilistic paradigm of known Gaussian processes outlined in paragraph 1 of the global response sent to all reviewers.

2. This adaptation seems more like adding a "weight matrix" to a specific focus, using existing work from the image processing domain.

The goal of our work is to investigate the impact of edge-preserving noise on a generative diffusion process, which is inspired by the idea of edge-preserving denoising introduced by Perona and Malik in image processing. Despite this, we had to make several efforts to make this work in the context of generative diffusion:

1. Their work considers an anisotropic diffusion process described in Eq. (10) of our submission. Unfortunately, Eq. (10) cannot be written as a linear equation of the form Eq.(1) because of its dependency on $||\nabla \mathbf{x}_t||$. By considering an alternative form of the diffusion coefficient in Eq. (11) such that it incorporates $||\nabla \mathbf{x}_0||$ instead, we are now able to use it in a linear diffusion process.

2. With our updated forward process, the forward and backward posteriors of the generative process change as well, as described in Eq. (4) to (8) in the paper.

3. Our backward process posteriors depend on non-isotropic variance (see Eq. (14) - (16) in our submission), introduced by the edge-preserving schedule. Therefore, we had to rethink the loss function in order to make sure the model learns this non-isotropic variance corresponding to the structural edge content in the data.

4. We cannot simply apply linearly varying edge-preserving noise, but needed to design a hybrid process that interpolates between edge-preserving and isotropic noise, to ensure that our process converges to the known prior $\mathcal{N}(0, I)$ from which we can start the sampling.

5. Another crucial component is to ensure smoothness in the transition between edge-preserving and isotropic noise. In the ablation in Section 5.2 of our submission we show that choosing \tau(t) to be linear instead of a steeper function like sigmoid or cosine has a great impact on performance. Additionally, by using constant values for edge sensitivity $\lambda(t)$ instead of a smoother interpolation, the model focuses on a specific granularity of structural content but fails to generate finer details.

Therefore, we would like to contend that there is more going on than simply applying a weight matrix. Our edge-preserving process fits into the two main general paradigms of generative diffusion models.

In Practice, our work shows its positive impact as follows:

• In particular, our method significantly outperforms the baselines on shape-guided generative tasks (stroke painting to image), both quantitatively but especially qualitatively (sharper reconstructions, less artifacts).
• We demonstrate that our edge-preserving model is better able to learn low-to-mid frequencies of the target data distribution.
• We show quantitative and qualitative improvements w.r.t. state-of-the-art isotropic models (DDPM, Simple Diffusion) for unconditional image generation.
• We show how it fits in 2 general mathematical frameworks: that of a well-studied Gaussian process (Eq. (4) to (8) in the paper), and that of an SDE , allowing it to be adopted in various diffusion settings. Given these 2 general interpretations we also believe that our process should be integrable in the Flow Matching (Lipman et al. 2023) framework.
• We will release minimalistic and modular source code that is easy to integrate or extend upon publication.

3. Which results support the greater robustness to visual artifacts compared to DDPM?

The visual artifacts that we mention in this statement occur ocassionally, and given that we showed uncurated samples in the submission, they might be very subtle or missing. For the shape-guided task, we have seen them occur more often and being more pronounced (as is shown in Figures 5 and 8). However, for unconditional generation, one extreme example can be found in Figure 10, row 3 in the first column for ours vs. DDPM. We plan to provide more specific examples in the revised version which we will upload later during this rebuttal phase to support this statement.

4. What are the approximate numbers for training and inference time?

Please find our response to this concern in paragraph 4 of the global response we sent to all reviewers. We will incorporate these discussions in the revised version.

5. Please clarify how $||\nabla x_0||$ is implemented.

Currently, our implementation indeed considers the approximation of $||\nabla x_0||$ made with a Sobel filter. As an alternative, we have also tried to use a Canny edge detector as a signal to determine where to suppress the noise, but this proved significantly less effective.

6. Implementing this within a diffusion framework that efficiently supports high-resolution datasets would be beneficial for quanititative evaluation

Integrating our edge-preserving model into high-resolution synthesis frameworks like Simple Diffusion (Hoogeboom et al. 2023) [6] is a promising and realistic direction for future work, given that both models are based on the same Gaussian process mathematical framework outlined in Section 3 of our paper.

Something that does need to be taken care of when moving to higher resolutions however is the computation of $||\nabla x_0||$. Our current implementation uses a Sobel filter, which is not scale-invariant. For the lower resolution data (up to 128x128) we have tested on we have found that this does not pose a real problem. For higher resolution data, this can become an issue however, and it might be better to resort to a scale-invariant "edge detector" like a Laplacian of Gaussian (LoG) with a scale-normalization strategy.

7. Can the theory be better abstracted and therefore applied inside other diffusion frameworks?

Thank you for this interesting question. In paragraph 1 of the global response to all reviewers, we show how our method fits in the two main general paradigms of generative diffusion models. In particular, we mention that we follow the probabilistic paradigm in our submission. Besides this our Eq. (13) can be reformulated as an SDE, following the score-based paradigm. We would like to refer to paragraph 1 of the global response for more concrete details on this.

Dear reviewer, thank you for your valuable feedback and suggestions. Besides this response directed to you, we have also prepared a global response to all reviewers, discussing common concerns/questions and providing further details.

1. The reported FIDs and qualitative results do not seem competitive with current state-of-the-art diffusion models.

In paragraph 3 of the global response sent to all reviewers, we provide an additional comparison with Simple Diffusion (Hoogeboom et al. 2023) [6]. We also provide CLIPScore as a new metric for these comparisons. We see that our model still consistently outperforms this baseline on the shape-guided generative task. We also discuss there why our reported FIDs seem to be on the higher side. We would like to refer you to paragraph 3 of the global response for the actual scores, as well as additional details.

2. How does the method compare to and fit in the framework of Flow Matching?

Thank you for the suggestion. We find this an exciting direction for future work. After a closer examination of the Flow Matching paper, we believe our edge-preserving noise scheduler fits naturally within their general framework for arbitrary functions $\mu_t(x_1)$ and $\sigma_t(x_1)$. Specifically, in their Equations (16)–(19), they derive a method for conditioning a vector field using Gaussian processes defined in [4] and [8]. Since our method operates within the same Gaussian process framework, the choice of $\mu_t(x_1)$ or the reversed diffusion path can remain consistent with that of the variance-preserving path, while $\sigma_t(x_1)$ would incorporate our noise-suppressing denominator inspired by Perona and Malik.

Based on your request, we also made a quantitative comparison and qualitative with Flow Matching. Given that an official implementation by the authors is unavailable, we used the popular library torchcfm, which contains an implementation for [4]. We performed an equal time training w.r.t. our edge-preserving model.

Given Flow Matching’s proven effectiveness in the results above, combining it with our edge-preserving noise scheme holds promise for further improving sample quality.

3. Can the negative log likelihood still be approximated (via variational upper bound) with this formulation?

Thank you for pointing this out, this is an interesting question. The denoising probabilistic model paradigm defined in the DDPM paper defines the loss by minimizing a variational upper bound on the negative log likelihood. Because our noise is still Gaussian, the derivation they make in Eq. (3) to (5) of their paper still holds for us. The difference however is that we're non-isotropically scaling our noise based on the image content. As a result, our methods differ on Eq. (8) in their paper. Instead, we will end up with something of the form $L_{t-1} = \mathbb{E}_q[\Sigma^{-1}(\tilde{\mathbf{\mu_t}}(\mathbf{x}_t, \mathbf{x}_0) - \mathbf{\mu_{\theta}}(\mathbf{x}_t, t)) . (\tilde{\mathbf{\mu_t}}(\mathbf{x}_t, \mathbf{x}_0) - \mathbf{\mu_{\theta}}(\mathbf{x}_t, t))]$ In essence, for our formulation that considers non-isotropic Gaussian noise, one should apply a different loss scaling for each pixel.

So yes, our formulation still provides an analytical variational upper bound to approximate the negative log-likelihood. While our heuristic loss function (Eq. 17) already proved effective for approximating non-isotropic noise corresponding to structural content in the data, a more accurate KL-divergence loss would include the scaling discussed above. We will add a more detailed discussion on this in the Appendix of the updated version.

4. The time-varying edge sensitivity $\lambda(t)$ seems arbitrary and is currently dependent on Ho et al.'s schedule.

We will add a brief note to the ablation (Section 5.2) mentioning that values $\lambda_{min}$ and $\lambda{max}$ described in Section 5 were chosen based on experimentation in the range $1e^{-1}$ and $1e^{-8}$, but had less impact on performance than other parameters. We would like to clarify however that $\lambda(t)$ does not depend on DDPM's schedule. In fact, we would like to emphasize that our method is completely independent of DDPM, as is discussed in paragraph 2 of the global response to all reviewers.

Dear reviewer, thank you for your valuable feedback and suggestions. Besides this response directed to you, we have also prepared a global response to all reviewers, discussing common concerns/questions and providing further details.

1. Does edge-aware noise affect generation diversity?

To answer your question and measure the mere impact of edge-preserving noise on diversity, we have computed precision (metric for realism) and recall (metric for diversity) scores for the isotropic model DDPM, and our edge-preserving model. We provide tables with these metrics below. For the shape-guided generative tasks (based on SDEdit (Meng et al. 2021)), we consistently outperform in terms of precision, and again closely match in terms of recall.

For unconditional image generation, while we slightly get outperformed, we find that our edge-preserving model closely matches DDPM on both metrics. therefore we would argue that edge-preserving noise minimally impacts diversity.

2. Seeming contradiction about IHDM being a(n) (non-)isotropic Gaussian model

Thanks for pointing this out. Although IHDM is a model that blurs isotropically in the image space, the diffusion is done in frequency space by taking a discrete cosine transform, and it is in this frequency space that they consider a non-isotropic form of noise. In short, the isotropic blurring in image space is equivalent to a non-isotropic addition of noise in the frequency space. We will be more explicit about this in the revised version of our paper that we will upload later during this rebuttal phase.

3. The innovation appears to be primarily an application of existing theory rather than a novel theoretical contribution.

The goal of our work is to investigate the impact of edge-preserving noise on a generative diffusion process, which is inspired by the idea of edge-preserving denoising introduced by Perona and Malik in image processing. Despite this, we had to make several efforts to make this work in the context of generative diffusion:

1. Their work considers an anisotropic diffusion process described in Eq. (10) of our submission. Unfortunately, Eq. (10) cannot be written as a linear equation of the form Eq.(1) because of its dependency on $||\nabla \mathbf{x}_t||$. By considering an alternative form of the diffusion coefficient in Eq. (11) such that it incorporates $||\nabla \mathbf{x}_0||$ instead, we are now able to use it in a linear diffusion process.

2. With our updated forward process, the forward and backward posteriors of the generative process change as well, as described in Eq. (4) to (8) in the paper.

3. Our backward process posteriors depend on non-isotropic variance (see Eq. (14) - (16) in our submission), introduced by the edge-preserving schedule. Therefore, we had to rethink the loss function in order to make sure the model learns this non-isotropic variance corresponding to the structural edge content in the data.

4. We cannot simply apply linearly varying edge-preserving noise, but needed to design a hybrid process that interpolates between edge-preserving and isotropic noise, to ensure that our process converges to the known prior $\mathcal{N}(0, I)$ from which we can start the sampling.

5. Another crucial component is to ensure smoothness in the transition between edge-preserving and isotropic noise. In the ablation in Section 5.2 of our submission we show that choosing \tau(t) to be linear instead of a steeper function like sigmoid or cosine has a great impact on performance. Additionally, by using constant values for edge sensitivity $\lambda(t)$ instead of a smoother interpolation, the model focuses on a specific granularity of structural content but fails to generate finer details.

In addition, we show how our edge-preserving diffusion process is rather general and fits into both of the two prominent "paradigms" of generative diffusion models. We refer to paragraph 1 of the global response to all reviewers for more details on this.

4. and 5. Confusing equations + ordering in Section 3

Thank you for pointing out the confusing ordering and use of equations in section 3 of the paper. We agree with this and suggest the following changes:

• Focus our presentation on Eq.(1), since this is the linear equation that forms the template for our linear diffusion process
• Remove Eq. (2) and (3), as they are not central to our work
• Engage more early with the work of Perona and Malik that served as inspiration to our work. We would like to introduce the coefficient introduced into Eq. (11) early on, and directly make the link to how we want to integrate it into Eq.(1) to introduce edge preservation into the diffusion process
• Mention that there are 2 major diffusion paradigms (score-based models and denoising probabilistic models), and how, starting from Eq. (13), our diffusion model fits into both

We are working on an updated version of the paper incorporating these and other changes, which we will upload later during this rebuttal phase.

Dear reviewer, thank you for your valuable feedback and suggestions. Besides this response directed to you, we have also prepared a global response to all reviewers, discussing common concerns/questions and providing further details.

1. The paper does not discuss the computational cost and memory usage of the proposed method.

As discussed in paragraph 4 of the global response, our approach adds no significant computation/memory overhead compared to the baseline isotropic models.

2. It is unclear how well the model generalizes to other types of models or tasks

We additionally integrate our method into an inpainting application. RePaint (Lugmayr et al. 2022) [2] is a state-of-the-art pixel-space inpainting algorithm. We made a comparative analysis between RePaint and "Edge-preserving RePaint" by performing an inpainting task over 100 images for multiple datasets. We will also add visual results for this task to the revised version of the paper, which we will upload later in this rebuttal. Note that lower FID is better, and higher CLIPScore is better. We find that our model closely matches the performance of RePaint.

We would also like to note that, since it was a question of multiple reviewers whether our model is easy to adopt in other settings/frameworks, we have dedicated a paragraph (paragraph 1) in the global response to all reviewers on this concern. In particular, we show how our edge-preserving diffusion is rather general and fits in the two major diffusion model paradigms.

3. Including newer metrics like CLIPScore, etc. could provide a more comprehensive view of performance.

The table below provides an additional comparison for our shape-guided generative task (cf. SDEdit, Meng et al., 2021) evaluated using the CLIPScore metric. Our method consistently outperforms the baselines on this metric, indicating that the generated images are more semantically aligned with the ground-truths (the original images used to generate the stroke paintings). In the submission, we showed several examples (e.g. Figure 5) where our model solves visual artifacts that are apparent with other baselines, which can improve the semantical meaning of the generated image.

4. Additional comparative analysis with newer state-of-the-art methods would be helpful, especially those that aim to preserve structural information in the images.

We do provide additional results for the state-of-the-art model Simple Diffusion (Hoogeboom et al. 2023), which can be found in paragraph 3 of the global discussion. Currently, we are not aware of any other works that are considering structure-aware noise for generative diffusion models in a way that is similar to us. A work with a similar title is "Structure Preserving Diffusion Models" (Lu et al. 2024), however, the focus of this work is very application-specific ; their model aims to be invariant w.r.t. transformations such as translations and rotations (i.e. when $\mathbf{x}_T$ is transformed, $\mathbf{x}_0$ should undergo the exact same transformation). This can be crucial for some applications, such as medical imaging. Instead, our work starts from the observation that diffusion models are denoisers, and aims to make their denoising capabilities more aware of the underlying content by investigating the impact of edge-preserving noise.

5. Please provide more insight into how the parameters affect the model's performance, and how to select them effectively

In our ablation study (Section 5.2), we discuss and aim to visually show how the hyperparameters impact the models behaviour and performance. Three important parameters significantly affect the model's performance:

• Edge sensitivity $\lambda(t)$
• Transition function $\tau(t)$
• Transition point $t_{\phi}$

The major performance improvements come from making $\lambda(t)$ (linearly) time-varying and using a linear $\tau(t)$ instead of a steeper function like a sigmoid. Additionally, the transition point ($t_{\phi}$) significantly impacts the level of detail and sharpness in the generated samples. In the submission, we recommend the point of transition should be kept at $t=\frac{T}{2}$, as this provides an effective balance between sharpness and high-frequency details.