Stochastic Forward–Backward Deconvolution: Training Diffusion Models with Finite Noisy Datasets

Thank you very much for reviewing our paper. Below are our responses to your comments:

Q1. How many training iterations are required?

A1. In most of our experimental settings, SFBD converges within four iterations. (see [link] for results from additional iterations.) This is why we originally report the FIDs over the first four iterations.

When the noise level is fixed, SFBD converges faster with more clean samples, as the pretrained model starts closer to the true data distribution (see also Sec 5: The Importance of Pretraining). Conversely, with a fixed number of clean samples, higher noise levels also speed up convergence. We hypothesize this is because higher noise levels obscure more information in the noisy samples, prompting SFBD to rely more on the clean data and thus more rapidly adapt by combining complementary features from the noisy sources.

Q2. Regarding "training efficiency, computational overhead, or scalability compared to standard DM"

A2. While SFBD involves multiple fine-tuning steps, each step has a similar cost to standard diffusion training, as it minimizes the regular conditional score-matching loss.

Most training time is spent on pretraining and the first fine-tune, together matching the cost of training a regular diffusion model. Later fine-tuning steps are much faster—each taking less than 1/4 of standard training time in our setup. Overall, using SFBD with 4 fine-tunes takes about 1.75× the time of training a regular diffusion model.

Backward sampling adds cost but is parallelizable. With 8 RTX 6000 GPUs, each sampling step takes under 30 minutes.

Q3. Baseline model selections

A3. Our implementation follows EDM [1] with identical hyperparameters (using the FFHQ‑64×64 config for CelebA). As noted in EDM Appx F.3, the UNet backbones are nearly identical to those in DDPM and DDIM. AmbDiff, EMDiff, and TweedieDiff also use EDM backbones, so their architectures match ours. For SureScore, we use results from [2], which employs a slightly different UNet but with a similar parameter count.

Q4. The theoretical analysis seems no help for the model convergence, especially the derived lower bound.

A4. Thms 1 and 2 are not meant to describe SFBD’s convergence, but to illustrate the difficulty of estimating clean distributions from noisy data. Based on density deconvolution theory, they provide matching upper and lower bounds, implying a sample complexity of $\mathcal{O}((\log n)^{-2})$. This poor rate shows that training solely on noisy data is impractical, justifying SFBD’s use of limited clean samples for pretraining.

SFBD’s actual convergence guarantee is given in Prop 2, which shows a rate of $\mathcal{O}(1 / \sqrt{K})$, where $K$ is the number of iterations in Algorithm 1.

Note: The $K$ in Prop 2 and Alg 1 refers to the same quantity. In contrast, the $K$ in Thm 2 is an unrelated constant. To avoid confusion, we’ll revise the notation in Thm 2 to use $C’$ in the updated version.

Q5. Can you explain more about the $K$ in $\mathcal{O}(\log n)^{-2}$?

A5. In Thm 2, given $p_{\rm data}$ and the noise level $\sigma_\tau$, the constant $K$ is a fixed positive value. (Note: This quantity will be renamed to $C'$ in the revised version to avoid confusion, as it is only used in Thm 2 and is unrelated to the iteration count $K$ in Alg 1 and Prop 2.)

Q6. Regarding the data leakage issue.

A6. The SFBD algorithm does not address potential leakage from the clean images used in pretraining. However, since these images are public and copyright-free, such leakage is not a concern in our setting. We also note that:

• Some clean data is essential, as the problem is otherwise intractable;
• Clean images can be copyright-free or obtained with user consent;
• Pretrained models may be sourced from public datasets, though with potential quality trade-offs (as shown in ablation studies).

SFBD is specifically designed to prevent leakage from sensitive data by exposing the model to only one corrupted version of each sensitive sample during training. This inherently limits memorization and reconstruction of private or copyrighted content. Notably, SFBD does not require clean images or the pretrained model to be released or shared through secure channels.

To further support our privacy claims, we follow [3] by computing similarity scores between generated and sensitive samples [link]. Results show no reconstruction of sensitive content. These findings will be included in the revision.

Q7. Regarding addtional results on bigger datasets

A7. See Reviewer BKmX - A2.

[1] T. Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2023.

[2] An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations. Bai et al. 2024.

[3] Consistent Diffusion Meets Tweedie. Daras et al., 2024.

Thank you very much for the review. Below are our responses to your comments:

Q1. Regarding the non-trivial gap when $|\mathbf{u}|$ is large in Prop 2.

A1. Thank you for this insightful comment. We agree that the bound appears to grow with $\|\mathbf{u}\|$, and we would like to clarify that in practice, the behavior of characteristic functions for large $\|\mathbf{u}\|$ is typically negligible. Specifically, characteristic functions tend to decay rapidly—often at an exponential or super-polynomial rate—for distributions with smooth and bounded densities.

To make the discussion concrete, consider the 1D case: the characteristic function $\phi(u)$ is the Fourier transform of the probability density function. When the density is k-times differentiable, it is well known (e.g., Lemma 4 on page 514 of [2]) that the characteristic function satisfies $|\phi(u)| = o(|u|^{-k})$. This implies that for sufficiently large $|u|$, the magnitude of the characteristic function becomes negligible.

Therefore, under the assumption that both $p_{\rm data}$ and $p_0^{(k)}$ are smooth and have bounded support, it suffices to ensure that their characteristic functions are close within a compact domain $|u| < U$ for some $U > 0$. This local closeness in the Fourier domain translates to closeness in the original densities, and hence the distributions. We will include this clarification in the final version of the manuscript.

Q2. Regarding addtional results on bigger datasets

A2. While we acknowledge that further experiments on larger datasets would strengthen the empirical study, our ability to scale up is constrained by the limited computational resources available at our academic institution. For instance, according to EDM’s official repository [1], training a diffusion model on ImageNet requires 32×A100 GPUs for 13 days—resources that are currently beyond our reach.

To partially address your comment, we conducted an experiment on the Tiny ImageNet dataset, which contains 100,000 images. We selected 1,000 clean images as copyright-free samples for pretraining and set the noise level to 0.2. Using a reduced batch size of 256 (compared to the recommended 4096 in EDM), the pretrained model achieved an FID of 36.74. After the first fine-tuning iteration, the FID dropped to 20.41. This result reflects a trend consistent with our findings on CIFAR-10 and CelebA, further corroborating our claims.

Q3. The results are only shown for at most 4 iterations, it is better to show more iterations and specifically, how fast the results would converge to the optimal state.

A3. In most of our experimental settings, SFBD converges within four iterations. (see [link] for results from additional iterations.) This is why we focus on reporting the FID trajectories over the four few iterations in our original submission.

Moreover, when the noise level is fixed, SFBD tends to converge more quickly as the number of clean samples used for pretraining and fine-tuning increases. A larger clean dataset allows the model to start closer to the true data distribution (see also Section 5 – The Importance of Pretraining). Conversely, when the number of clean samples is fixed, increasing the noise level also accelerates convergence. We hypothesize this is because higher noise levels obscure more information in the noisy data, prompting SFBD to rely more on the clean samples, enabling faster adaptation through the fusion of complementary information from noisy samples. We will incoperate this discussion in the revision.

[1] T. Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2023.

[2] Feller, W. An Introduction to Probability Theory and Its Applications, Vol. 2. Wiley. 1971

Thank you very much for your comments. We will first clarify the distinction between our framework and standard denoising methods, and then address your comments point by point.

How our method differs from a standard denoising algorithm. SFBD alternates between denoising samples and fine-tuning the denoiser, allowing it to blend high-frequency details from clean data with global structure from noisy samples (see Section 6.3). Unlike standard denoising methods that reconstruct exact clean inputs, SFBD trains the denoiser to recover the full data distribution, resulting in more realistic samples.

To highlight SFBD’s advantages, we compare the FID of its denoised samples at each iteration ($\mathcal{E}_k$ in Algorithm 1) with the full training set, alongside results from a state-of-the-art off-the-shelf denoiser, Restormer [1].

CIFAR-10 (4% clean images)

CelebA

The results show that SFBD can produce significantly more realistic samples than regular denoisers.

Below, we provide point-by-point responses to your comments.

Q1. How does the method perform when you use a off-the-shelf denoiser trained on generic data?

A1. As shown above, Restormer-denoised images consistently yield much higher FIDs than SFBD (reported in the paper). Since generative models trained on these images can’t surpass the FID of their targets, using off-the-shelf denoisers like Restormer results in significantly worse performance. We will include this discussion in the revision.

Q2. Regarding Experimental Designs Or Analyses

A2. The upper block of Table 1 assumes full access to both clean copyright-free and sensitive samples, so the results naturally outperform those from models trained on denoised data—making our method appear weaker, not stronger.

For the algorithms in the lower block, their original designs either assume only noisy data or use clean samples in their own specific way. Adapting them to leverage denoised samples would require substantial changes and could also raise fairness concerns. To maintain a consistent and fair comparison, we used available clean data for pretraining when applicable.

As noted in A1, the reported results in the above tables serve as upper bounds for models trained on denoised samples—whether from a pretrained denoiser (SFBD-Iter 1) or an off-the-shelf one (Restormer). Importantly, final SFBD models (after multiple training iterations) still outperform these upper bounds. We will clarify this in the revision to avoid potential concern.

Q3. Regarding the case that noise variance is unknown.

A3. In our setting where noise is intentionally added to protect sensitive samples, it is reasonable to assume known noise variance (either directly controlled by the model developer or communicated to it securely).

To make the proposed framework compatible with scenarios where the noise level is unknown, it could be extended by incorporating noise level estimation techniques-such as those used in blind denoising methods. We consider this an interesting direction for future work.

Q4. How does using a denoiser trained on multiple noise levels (blindly) affect the performance?

A4. The denoiser Restormer, discussed in the tables above as well as in A1 and A2, is designed to handle multiple noise levels in a blind manner. As shown in our results and discussions, generative models trained on images denoised by Restormer cannot achieve performance comparable to those trained using SFBD.

Q5. Regarding "The approach assumes that the noise level of the noisy data is within the trajectory of the diffusion process, ..."

A5. In theory, diffusion models can handle noise levels ranging from zero to infinity. We would appreciate it if you could provide more details or clarification on this concern so we can better understand and address the issue.

Q6. Regarding "The performance of the approach degrade significantly when the clean data is out of the distribution"

A6. We included these ablation studies to support our theoretical claims and offer additional insights into the framework’s behavior. While we agree that using in-distribution clean data yields better results, the main takeaway is that pretraining on out-of-distribution data still provides a clear benefit compared to no pretraining at all.

[1] Restormer: Efficient Transformer for High-Resolution Image Restoration. S. W. Zamir et al. CVPR 2022

Thank you for reviewing our paper and for the detailed feedback.

Before addressing specific points, we would like to clarify that the remark stating "the benchmarking of this work is unfair/incomplete" is, in our view, inaccurate. Specifically,

1. We were not aware of the missing baseline [1] before our submission. We hope you would understand since [1] is very recent and there is no way we could have compared to it. We did our best to discuss its relation to our method below (in A1), which will also appear in our revision.
2. Regarding the claim of unfair implementation: our version of TweedieDiff is not the same as in Daras25 [1] but faithfully follows the original paper (see A4 for details). While we’re happy to include results from [1] and discuss the differences, we do not have any intention to “paints an unfair and very pessimistic picture of the baselines.”

Q1. Regarding a related work [1] published in ICLR 2025

A1. We agree that this very recent work [1] is relevant. We will include a detailed discussion and benchmark comparison in the revision. In particular:

Both papers show that learning the true data distribution from only noisy samples is theoretically possible but practically requires an infeasible number of samples. To address this, both incorporate a small amount of clean data to guide training.

However, they differ in approach. Our method uses the density deconvolution, while Daras et al. build on Gaussian Mixture Models (GMMs), providing more precise modeling under stronger distributional assumptions. We believe the fact that both methods reach similar conclusions—despite their different foundations—reinforces each other.

Methodologically, [1] applies Tweedie’s formula (consistency constraint) to recover the clean distribution. In contrast, ours introduces a novel forward-backward deconvolution strategy, offering a fresh perspective without the heavy computational cost of enforcing consistency.

(Benchmark Comparison) We will include the following CIFAR-10 FID results in the revision:

• 50 clean images, σ = 0.2 (Table 1 setting): Daras25: 8.05; SFBD: 13.53
• 10% clean images, σ = 0.2: Daras25: 2.81; SFBD: 2.58
• 4% clean images, σ = 0.59: Daras25: 6.75; SFBD: 6.31

Daras25 performs better with very limited clean data, likely due to overfitting during our model’s pretraining on small datasets. This overfitting is lessened as more clean data becomes available, allowing SFBD to outperform. We attribute this to SFBD’s stable fine-tuning via score-matching loss, which avoids the extra constraints of consistency-based methods.

Q2. Regarding training models in a way similar to EMU

A2. We illustrate this on CIFAR-10 by pretraining a diffusion model on noisy images (σ = 0.2), then finetuning on either 50 or 4% clean images. FID trajectories [link] show an initial drop followed by a rise—sharper with 50 clean images, slower with 4%. This is expected: finetuning on clean data improves performance at first but eventually causes the model to forget useful noisy features, leading to degradation.

Q3. Regarding ambient diffusion

A3. The reported value is from the arXiv version of [2], which states that "AmbientDiffusion is trained with the standard setting." Since it was omitted from the NeurIPS final version and its reliability is uncertain, we will remove it in the revision.

Q4. Regarding TweedieDiff

A4. Our implementation closely follows the original TweedieDiff paper. Comparing Daras25 [1] and Daras24 [3], we think the difference might be caused by the new implementation in Daras25 [link], which we were not aware of before our submission. We will include results from both versions and clarify their differences in the revision. We are happy to include the new result of Daras25 in our revision and add explanation on the different results. We want to assure the reviewer we tried our best to compare all baselines fairly (to be best of our knowledge).

Q5. Regarding memorization

A5. See Reviewer kYMA-A6.

Q6. Regarding MISE

A6. MISE is the standard metric in density estimation that evaluates error uniformly across all x, encouraging agreement both within and outside the support of $p_{\rm data}(x)$. Weighting the error by $p_{\rm data}(x)$ instead reduces the influence of low-likelihood regions, under-penalizing high-density estimates in areas where the true density is low. This can make models that generate unlikely or absurd samples appear to perform well, contradicting our goal of accurately modeling the full distribution.

Q7. Regarding computation requirement

A7. See Reviewer kYMA-A2.

[1] How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion. Daras et al., 2025.

[2] An EM Algorithm for Training Clean Diffusion Models from Corrupted Observations. Bai et al., 2024.

[3] Consistent Diffusion Meets Tweedie. Daras et al., 2024.