Heavy-Tailed Diffusion with Denoising Levy Probabilistic Models

We thank the reviewer for their thoughtful feedback and for recognizing the strengths of our work. Below, we address the reviewer's concerns in detail.

• I have a doubt on the convergence property, for t>0, the marginal distribution of this new dynamic should have infinite variance. Then really it would converge to the data distribution (with finite distribution) at t=0? Can you show me more theoretical convergence proof?

This is a legitimate question raised by the reviewer, to which we can provide a positive answer. To be more precise, based on the work in [1, 2, 3] mentioned in the introduction of our paper (line 81), we can consider the following Lévy-driven SDE:

For an appropriate choice of $b$, the limiting distribution of $X_t$ when $t \to \infty$ can have finite variance. Similarly a Brownian SDE defined by

can have a limiting distribution with infinite variance with appropriate choice of $b$ and $\sigma$ [4].

Therefore, the tails of the noise do not always determine the tails of the limiting distribution, and it is indeed possible to converge to the true data distribution.

For our specific work, which relies on a discrete time structure, we can draw inspiration from existing works that establish theoretical guarantees for DDPM, particularly [5, 6].

We are confident that by adapting the approaches presented in these studies, we will be able to derive convergence bounds for DLPM. However, we would like to emphasize that these works on DDPM are relatively recent and involve complex, sophisticated techniques. Therefore, we believe that providing a complete answer to the issue you raised is beyond the scope of this paper and is best reserved for future work.

[1] Simsekli et al., 2017, Exploring Levy driven stochastic differential equations for Markov chain Monte Carlo

[2] Simsekli et al., 2020, Underdamped langevin dynamics: Retargeting sgd with momentum under heavy-tailed gradient noise

[3] Huang et al., 2020, Approximation of heavy-tailed distributions via stable-driven sdes

[4] Bresar et al., 2024, Subexponential lower bounds for f-ergodic Markov processes

[5] Li, G. and Yan, Y, 2024, O(T/d) convergence theory for diffusion probabilistic models under minimal assumptions.

[6] Z Huang, et al., 2024, Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality

• The term 'Data Augmentation' typically means increase the size of the dataset by creating new data [...]. I am quite confused [...]

The term "Data Augmentation" originates from the Markov Chain Monte Carlo (MCMC) literature, as referenced in Chapter 10 of [1]. This legacy has influenced other popular machine learning methods, such as Augmented Neural ODEs [2].

We acknowledge the potential for confusion due to the differing interpretation of the term in the deep learning context. We will clarify this distinction in the revised paper to avoid ambiguity.

[1] Chapman et al., 2011, Handbook of Markov Chain Monte Carlo

[2] Dupont et al., 2019, Augmented Neural ODEs

Dear Reviewer,

We hope this message finds you well. We are writing to kindly remind you that the discussion period for the review process will soon come to an end. As the authors, we wanted to ensure that our rebuttal has been received and addressed.

In particular, we would greatly appreciate your feedback or acknowledgment on the following key points raised in our rebuttal:

• Convergence property: We provided clarifications on the marginal distribution of stochastic processes driven by light-tailed or heavy-tailed dynamics, and how these tails do not always determine the tails of the limiting distribution. This makes it possible to converge to the true data distribution.

• Terminology (Data augmentation): We clarified our choice for the term 'Data augmentation', which originates from the MCMC literature, and which has been used in other popular machine learning methods.

If there are any remaining questions or points requiring clarification, we would be happy to provide additional responses or discuss further. Your feedback and engagement are invaluable to improving the quality of our work. Thank you very much for your time and contributions to the review process.

Best regards,

The authors

We greatly thank the reviewer for their comments, and we address their concerns below.

• Q1: When considering CIFAT10_LT, LIM performs better than the DLPM method, which can not support the discussion of this work. Could the author discuss it in detail?

While our results demonstrate the strengths of DLPM, we acknowledge that in some cases LIM shows better performance. This is due to its use of clipping techniques, which we intentionally did not include in our method. Without these clipping adjustments, LIM’s performance significantly degrades, as we see in the updated Table 3 given below. This highlights that its framework is inherently less suited for heavy-tailed noise. Our approach emphasises simplicity and theoretical grounding, providing an "out-of-the-box" method without requiring extensive hyperparameter tuning.

Table 3: FID$\downarrow$, 1000 sampling steps for LIM, 25 sampling steps for LIM-ODE.

We anticipate that incorporating additional techniques like clipping into DLPM would yield further improvements, but we leave these further methodological refinements for future work.

• Q2: As shown in the theoretical part, if $\alpha = 2$, the DLPM algorithm is equivalent to DDPM. However, the experiment results of DLPM ($\alpha=2$) are worse than DDPM. Could the author discuss this experiment phenomenon in detail?

As we mention in footnote $4$ of page 6, the DLPM algorithm with $\alpha = 2$ exactly recovers DDPM when we use $r=1$ in our loss (7).

However, in our experiments, we used $r=1/2$ in our loss, as a default setting. This choice was made for simplicity, as it provides a loss formulation that works across all $\alpha$ values. This default choice is of course less suitable for $\alpha = 2$, when all considered distributions have finite variance, leading to the observed discrepancy in performance when compared to DDPM.

An alternative approach could involve choosing $r$ dynamically, such as $r = \frac{\alpha - (2 - \alpha)c}{2}$ (with $c < 1$), which interpolates between our loss and the standard Gaussian loss, or simply setting $r = 1/2$ if $\alpha < 2$, and $r=1$ if $\alpha=2$.

If the reviewer feels this is an important point, we will include this discussion in the revised paper.

• Q3: Could the author discuss the reason for the slightly worse performance of DLPM$^{\text{ni}}$?

Several factors may contribute to this phenomenon:

• The loss function is better suited for isotropic noise distributions
• Neural network architectures may inherently favour isotropic noise (e.g., due to rotational invariance in $\ell_2$ space)
• The datasets considered in our experiments might align better with isotropic noise assumptions

While non-isotropic noise does not outperform isotropic noise in our experiments, it could be a promising approach in other datasets or contexts. We consider this a methodological question that warrants further investigation.

Dear Reviewer,

We hope this message finds you well. We are writing to kindly remind you that the discussion period for the review process will soon come to an end. As the authors, we wanted to ensure that our rebuttal has been received and addressed.

In particular, we would greatly appreciate your feedback or acknowledgment on the following key points raised in our rebuttal:

• Generative Performance of DLPM vs. LIM: We provided additional clarification on LIM’s reliance on clipping and shared results from new experiments without clipping, which highlight the robustness and suitability of DLPM for heavy-tailed settings.

• Clarification on DDPM vs DLPM: We clarified how our loss is constructed and thus how DLPM compares to DDPM when $\alpha < 2$ and when $\alpha=2$.

• Clarification on DLPM$^{\text{ni}}$: We further discussed the performance of our algorithm with non-isotropic noise, which could be a promising approach in other datasets or contexts.

If there are any remaining questions or points requiring clarification, we would be happy to provide additional responses or discuss further. Your feedback and engagement are invaluable to improving the quality of our work. Thank you very much for your time and contributions to the review process.

Best regards,

The authors

We would like to warmly thank the reviewer, who has taken great time and effort to provide in-depth comments. We address their concerns below.

• The experiment for CIFAR10_LT is not conclusive: [...] A more fair claim is that the performance of DLPM/DLIM is comparable to that of LIM/LIM-ODE. the experiments missed an important comparison [to] DDIM

As shown in Figure 4, DLPM demonstrates superior performance compared to LIM at smaller iteration counts, which aligns with one of the key advantages of heavy-tailed diffusions: improved efficiency in low-step sampling scenarios.

Regarding the generative performance of DLPM vs LIM with 1000 sampling steps, we appreciate the reviewer’s suggestion and acknowledge the need for additional clarification. In our original submission, our intent was not to present LIM in an overly negative light, so we did not mention the following point.

LIM relies on gradient and noise clipping, which introduces extra hyperparameters that must be fine-tuned for each dataset. To provide a more equitable comparison, we conducted a new set of experiments where LIM was tested without clipping. The results in table below show that LIM’s performance deteriorates significantly, raising questions about whether the framework of LIM is inherently well-suited for heavy-tailed distributions. This highlights a limitation in its robustness relative to DLPM.

Table 3: FID$\downarrow$, 1000 sampling steps for LIM, 25 sampling steps for LIM-ODE, DDIM.

We would like to add that if we applied such clipping techniques, we would certainly achieve improved results. However, our paper's philosophy prioritizes simplicity and theoretical rigor over optimization-focused interventions.

Regarding DDIM, we agree that this is an important comparison. We have reported its performance in the same table below. We will include these results and remarks in the paper revision.

• We see that the FID for DLPM at $\alpha=2$ is 21.07 while the FID for DDPM is 19.05. This is a big difference and seems to weaken the point that DLPM at $\alpha=2$ is similar to DDPM.

As we mention in footnote $4$ of page 6, the DLPM algorithm with $\alpha = 2$ exactly recovers DDPM when we use $r=1$ in our loss (7).

However, in our experiments, we used $r=1/2$ in our loss, as a default setting. This choice was made for simplicity, as it provides a loss formulation that works across all $\alpha$ values. This default choice is of course less suitable for $\alpha = 2$, when all considered distributions have finite variance, leading to the observed discrepancy in performance when compared to DDPM.

• Add experiments with $\alpha=1.5, 1.6$ in Table 1 and 2, statistical significance in Table 2

Again, we would like to thank the reviewer and his attention to details. We have updated Tables 1 and 2 with results from new experiments and computed $p$-values using Welch’s $t$-test to compare the means of DLPM with those of other methods. With the usual threshold of $p < 0.05$, the results are now statistically significant across all settings, except for two cases: when $\alpha = 2$, which is expected since all methods are comparable, and for DDPM vs DLPM$_5$ at $\alpha = 1.7$ ($p = 0.074$). These updates strengthen the evidence supporting our claims.

Upon review, we realised that, for Table 2, we only used 5,000 samples for testing. This limited sample size introduced high variance in precision and recall metrics, especially for the least-represented class, which accounts for only 1% of the samples. Since precision and recall metrics in low dimensions are sensitive to rare class occurrences, we increased the number of test samples to 25,000.

For image data, we currently lack the computational budget required to compute rigorous error bounds. However, we believe that considering the performance of unclipped LIM provides an unambiguous baseline for comparison.

Table 1: $\text{MSLE}_{\xi = 0.95} \downarrow$, averaged over 20 runs. $p$-values computed using Welch's $t$-test (unequal variance) between means of DLPM and each other method

Table 2: $F_1 \uparrow$ score, averaged over 30 runs. $p$-values computed using Welch's $t$-test (unequal variance) between means of DLPM and each other method

• It is not clear why the results for the non-isotropic noise case are mentioned

Since one of the paper’s main goal is exploration, we felt it was important to assess the performance of non-isotropic noise, which is a natural extension of our framework. Although this variant performs worse for our datasets, it may prove useful for other datasets or tasks.

We will follow the reviewer’s suggestion, and move these results to the appendix in the revised paper.

• The fact that the neural network does not have the stable variable as input seems a bit magical.

This is a great observation. In some sense this is consistent with Gaussian diffusion models, where the network takes only the noised data $X_t$ as input at each time $t$; in the classical denoising diffusion interpretation, the role of the network is then to infer the noise $G_t$ implicitly present in $X_t$.

As we mention in Appendix G (line 1556), we have actually run a lot of experiments with the network taking $A_t$ as input in a few different ways, but the performance did not improve.

If the reviewer believes it would benefit the reader, we will expand on it in the main text.

Dear Reviewer,

We hope this message finds you well. We are writing to kindly remind you that the discussion period for the review process will soon come to an end. As the authors, we wanted to ensure that our rebuttal has been received and addressed.

In particular, we would greatly appreciate your feedback or acknowledgment on the following key points raised in our rebuttal:

• Generative Performance of DLPM vs. LIM: We provided additional clarification on LIM’s reliance on clipping and shared results from new experiments without clipping, which highlight the robustness and suitability of DLPM for heavy-tailed settings.

• Statistical Significance: We included updated $p$-values using Welch’s $t$-test for key results, addressing concerns about significance for some $\alpha$ values like 1.7 and 1.8.

• Clarification on DDPM vs DLPM: We clarified how our loss is constructed and compares to DDPM when $\alpha < 2$ and when $\alpha=2$.

• Additional results (DDIM, $\alpha=1.5, 1.6$): We included DDIM in our updated experiments, and additional results on the 2d datasets for $\alpha=1.5, 1.6$. We welcome any further comments on these additional comparisons.

If there are any remaining questions or points requiring clarification, we would be happy to provide additional responses or discuss further. Your feedback and engagement are invaluable to improving the quality of our work. Thank you very much for your time and contributions to the review process.

Best regards,

The authors

• Proposition 9: [...] why this loss is not also infinite like in the LIM setting?

This is a good remark, and fortunately there is a square-root applied to $A_t$. Let us rewrite the loss in Proposition 9:

$

     \mathcal{L}^{\text{SimpleLess}}(\theta) = \sum_{t=1}^{T} \mathbb{E} \left[\mathbb{E} \left( \| \epsilon_t^{\theta}(Z_{t}) - \bar A_t^{1/2} \bar G_t \|^{2} \ | \bar A_t \right)^{1/2}\right].

$

Remark that $\mathbb{E} (\|\bar A_t^{1/2} \bar G_t \|^{2} \ | \bar A_t)^{1/2} = \bar A_t \mathbb{E} (\|\bar G_t \|^{2})^{1/2} = \bar A_t \sqrt{d}$, since $\bar G_t \sim \mathcal{N}(0, I_d)$.
Thus, setting $\epsilon_t^{\theta} = 0$, we obtain

$

\mathcal{L}^{\text{SimpleLess}}(\theta) = \sum_{t=1}^{T} \mathbb{E} \left[\mathbb{E} \left( \| \bar A_t^{1/2} \bar G_t \|^{2} \ | \bar A_t \right)^{1/2}\right] = 
\sum_{t=1}^T\mathbb{E}[ 
\bar A_t^{1/2}] < \infty.

$

Indeed, $\bar A_t \sim \mathcal{S}_{\alpha/2,1}(0, c_A )$, so all of its moments of order $< \alpha / 2$ are defined, and we always choose $1 < \alpha \leq 2$ for DLPM and use a square root in the loss (7) ($r=1/2$) as we mention in section 3.2.3 line 309.

In short, the inclusion of the square root in the loss (setting $r=1/2$ and working with $1 < \alpha \leq 2$) ensures that the loss is always finite. We appreciate the opportunity to clarify this point and if the reviewer finds it relevant, we will explicitly address this in the paper when introducing DLPM, to ensure there is no ambiguity.

We appreciated the opportunity to address these points and thank the reviewer again for their careful review.

• Extension for extension's sake. [...] It follows prior work that has already extended SDE diffusion models to heavy-tailed noises.

Unlike prior work, our framework offers better flexibility, and a principled loss that upper bounds the log-likelihood (potentially enabling further theoretical work). We believe that the reviewer acknowledged these aspects in his review.

In addition, as emphasised in our previous response, our methodology provides a simpler setup with fewer hyperparameters, as LIM requires clipping to achieve the best performance (which is what is reported in the paper). This raises the question of the relevance of their framework for heavy-tails. We have re-run experiments for LIM without clipping on both MNIST and CIFAR10_LT, with the same parameters. Here are the results:

Table 3: FID $\uparrow$ for our image datasets

We believe this constitutes a much stronger case for the DLPM framework, and we will include this distinction in the revised paper. In conclusion, all the differences between LIM and DLPM that we exhibit makes, in our opinion, our contribution both novel and valuable, from both a theoretical and practical perspectives.

• Various concerns about heavy-tails

We want to address the reviewer's concerns about the use of heavy-tails. In the paper, we have considered that the literature has already proven the benefits of heavy-tails, as further described in [1, 2, 3], and we take it as a given.

In the context of images, long-tailed data is essentially a special type of multi-class imbalanced data with a sufficiently large number of tail (minority) classes. Moreover, the combined importance of these tail classes is very significant, although each tail class by itself only has a small number of samples. Thus one can think of the data as following a sort of multi-dimensional power law. Moreover, image datasets are high-dimensional spaces, where datapoints localize in very small regions; thus, considering boundedness in this geometry does not prevent heavy-tailed behavior to emerge, especially in the case of long-tails, and in a finite data regime.

In short, the long-tailed property is analogous to the presence of heavy-tails in the data. This is why DLPM and LIM are inherently more robust to mode collapse in imbalanced data.

Long- tailed distributed data is a relatively common phenomenon in real-world scenarios, e.g. high-speed train fault diagnosis, escalator safety monitoring [1]. Heavy-tails are also naturally occurring in weather data [4], and in our introduction we cite financial data [2] (stock returns and factors in factor models, like BARRA [3], are heavy-tailed). Our framework could be especially relevant in these contexts. We will provide more details in the main body of the paper, if the reviewer believes it is relevant.

We leave experiments on other heavy-tailed datasets to further work.

[1] Zhang et al., 2024, "A Systematic Review on Long-Tailed Learning"

[2] Borak et al., 2005, "Statistical Tools for Finance and Insurance"

[3] MSCI, 2017, https://www.msci.com/eqb/methodology/meth_docs/MSCI_Barra_Factor_Indexes_Methodology_June2017.pdf

[4] Pandey et al., 2024, "Heavy-Tailed Diffusion Models"

• In section 4.2, the FIDs reported for DDPM appear to be a little too high

EDM employs a different network architecture, noise schedules, training distribution etc., all optimized and tailored for Gaussian diffusion. Our experiments aim to isolate the effect of the noise distribution on generation while keeping other aspects constant. While DLPMs may not yet achieve SOTA performance, our work provides a foundation for future improvements.

We thank the reviewer for the thoughtful feedback and for highlighting important points for discussion. We address them below.

• The training loss is unstable, poorer approximations perform better: [...] the denoising network still has to learn the conditional expectation of a heavy-tailed distribution contrary to what is said in the paper.

The reviewer makes an interesting remark which indicates that we have not been clear enough in our original submission maybe. To address this point, let us inspect the loss function again:

From this formulation, we aim to learn $\epsilon_t(Y_t,Y_0)$ conditionally to $A_{1:t}$. Contrary to what the reviewer said, the conditional distribution of $(Y_t,Y_0)$ given $A_{1:t}$ has the same tail as the data distribution. Another perspective on our approach is to see it as a reparametrization trick, as used in variational inference, to obtain more stable and efficient training. Therefore, our training loss is stable, which is supported by our numerical experiments.

If we did not address reviewer's comment sufficiently on the loss, we would be happy to provide more details on these concerns.

• approximations of the loss with a single sample perform better than approximations with 25 samples

Thank you for your this interesting comment. We suspect that, alike what happens with gradient descent vs stochastic gradient descent, the greater stochasticity is actually helping the network in some way (flatter minima, better robustness, better generalization etc.). You can think of the cases where SGD with smaller batch-size performs better than a large batch-size, even though the stochastic gradient with the small batch-size is a "poorer" approximation of the full gradient.

Furthermore, additional work is needed to determine the optimal batch sizes and other parameters, which is outside the scope of this initial exploration. We would also like to mention that our paper has a strong exploratory aspect, investigating available methods for heavy-tailed diffusion. Regarding the stability of the loss and the median-of-means method, we observed a positive trend of improvements as $M$ increases, which we believe, justified its inclusion in the paper.

If the reviewer finds it relevant, we can add this discussion in the revised version of the paper.

• heavy-tailed distributions are not a practical tool for approximating distributions

Regarding the instability for $\alpha \leq 1.5$, it is indeed possible to train models under such conditions, but this requires additional techniques such as gradient clipping, noise clipping, and extended training times. These adaptations align with the rationale that heavy-tailed noises explore a larger portion of the space as compared to Gaussian noises, particularly in high-dimensional settings, where Gaussian noise concentrates in thin shells around datapoints. This naturally impacts training.

Our goal was to present a simple and principled framework that could be used effectively "out of the box", hence our simplifying remark.

It is worth noting that, for satisfiable performance, LIM requires gradient and noise clipping introducing additional clipping hyper-parameters. These parameters have to be tuned for every $\alpha$ and for each dataset, whereas this is not required by our method. In this sense, our method, contrary to LIM, is actually the first to make heavy-tails practical, enabling straightforward manipulation at least when $\alpha \geq 1.5$.

• In section 4.1 why isn't DDPM included in the comparison?

We added DDPM to Table 1 and Table 2. For Table 2, we increased the number of test samples to 25000 to improve statistical significance.

Table 1: $\text{MSLE}_{\xi = 0.95} \downarrow$, averaged over 20 runs

Table 2: $F_1 \uparrow$ score, averaged over 30 runs

Dear reviewer,

Thank you for your thoughtful comments and for taking the time to review our rebuttal in detail. We are grateful for your clarifications and are encouraged by your increased score and acknowledgment of the additional comparisons and explanations we provided.

We acknowledge that DDPM’s performance can be improved, as seen in recent works, such as EDM. Even with the simple framework of DDPM, this requires some tailored modifications like changes to training regimes that goes beyond the scope of this study.

Once again, thank you for your valuable feedback, which has helped us strengthen the presentation and clarity of our work.

Best regards,

The authors