Ambient Diffusion Omni: Training Good Models with Bad Data

We would like to thank the Reviewer for their thoughtful feedback. The Reviewer appreciated our formalization of the problem, our theoretical results, the generality of our method, and the breadth of our experiments. The Reviewer has concerns on three fronts:

1. Missing ablation studies on the sensitivity of the classifiers and the size of the ground truth set needed for classification.
2. Missing evaluations to measure how much we can preserve fine details.
3. the applicability of our method to low-frequency corruption types.

We present numerous new ablation results addressing each one of the raised issues.

Sensitivity Results

Reviewer: "How sensitive are the final generative results to small variations in the threshold?"

Great question: short answer, results are not very sensitive to miscalibration. To test this, we report FID results for JPEG compression at 18% compression rate (Table 3 of the paper) for a fixed classifier annotating at different thresholds. The results are shown below:

Reviewer: "Performance hinges on the accuracy and thresholding of the quality and crop classifiers; mis-calibration could hurt results, yet sensitivity analyses are limited."

Beyond the threshold ablations we presented above, we further ablate the size of the dataset needed for classification and the effect of training iterations.

Training Samples Ablation

We train two CIFAR-10 classifiers for the Gaussian blur corruption at $\sigma_B=0.6$. The first classifier is trained using 25K clean samples and 25K blurry samples and the second is trained with 5K clean samples and 45K corrupted samples. The scores are shown below:

As shown, the number of samples has an effect on the performance of the classifier, but still, the performance is much better than the 8.79 FID we get by only training on clean samples. Hence, miscalibration due to small training size can hurt a bit the performance, but not to a catastrophic extent.

Training Iterations Ablation

Another way to measure sensitivity to miscalibration is to check what happens when we undertrain or overtrain the classifier. The results are shown below for the JPEG at 18% compression rate (Table 3 of the paper):

We believe the results above prove the robustness of our classifier to various types of miscalibrations.

Removing the classifier

Reviewer: "Sample-wise annotations, crop-level decisions, and dual noise schedules introduce extra engineering steps that may slow adoption in production settings"

We agree with this point; however, it might not be needed.

At its core, our method is very simple: bad data should only be used in a subset of diffusion times. Blurry images, for example, are typically useful in the high-noise regime, while out-of-distribution images might be useful in the low-noise regime. The training of the classifiers provides a rigorous way of finding the “allowed training times” (as they estimate the TV distance), but this is not necessary. For example, for our controlled experiments on blurry images, annotating all the corrupted samples with the same value already yields pretty strong results. We show these results below (annotated with Fixed Sigma):

Our paper has similar results (Fixed Sigma) for the out-of-distribution data experiment (Table 2b), too. Further, for our text-to-image results, we similarly annotated all the data from DiffDB at the same noise level without requiring the training of a classifier.

Reviewer: "In domains where no pre-trained quality estimator exists, what is a practical approach to form S_G or S_B? Could you use a small human-labeled "gold" set, or would self-supervised heuristics suffice?"

Excellent question! The problem of sampling from the "uncorrupted" distribution is not identifiable if the corruption model is unknown (as in our setting) and we don't have samples from the ground truth. Hence, a "gold" set needs to be formed, either with human supervision or with some other domain-specific knowledge. For example, in proteins, there are corrupted data in the Protein Data Bank, but they come with some "confidence level" (resolution of the protein). In robotics, there are ground truth data from humans and "corrupted data" from simulations. If we are not in such a setting, then human annotation is required to form a "gold" set. The good news is that our ablations above suggest that the "gold set" can be small without hurting the performance too much (see sensitivity analysis).

Low-frequency corruptions

Reviewer: "Adding noise cannot fully eliminate low-frequency degradations (e.g., color shifts, masking), so the approach may revert to simple filtering in those cases. Have you tried combining Gaussian noise with other synthetic corruptions to push corrupted samples into a "usable" regime at lower sigma?"

Excellent question! It is true that for certain cases, classifying under Gaussian noise corruption will essentially revert to filtering. Hence, there are two options. Option 1 is to do some preprocessing of the data, as the Reviewer proposes. This can actually take us a long way in certain cases. We experimented with two types of corruptions to show this:

• salt and pepper noise, where we set 5% of the pixels independently to either 0 or 1.
• random masking, where we delete 70% of the pixels independently.

For both of these cases, we required very high sigma to mask the corruptions -- essentially reverting to filtering. However, a simple preprocessing of the data dramatically increased performance. For salt and pepper we used a 3x3 median filter and obtained an FID of $5.70$ (compared to the filtering $8.79$) using a fixed annotation of $\sigma=3$. Similarly, for the random masking, we do a local average of the non-masked pixels (simulating the effect of blurring) and that leads to an FID of $6.72$. These experiments show that with some simple preprocessing we can actually extract more signal out of the available data.

However, this is not always possible. For some corruptions, there aren't really easy preprocessing steps we can do to make them amenable to Gaussian noising. If the dataset has a lot of samples with such corruptions, the best way forward in our opinion is to train an "exotic" diffusion model (similar to Soft Diffusion / Cold Diffusion and certain flow matching models). These diffusion generalizations learn to connect arbitrary source and terminal measures -- e.g. the fully fogged distribution to the distribution of clean images. If we are willing to train such a model, we can corrupt the samples (not by adding noise, but by e.g. adding fog) to bring them along that exotic-diffusion trajectory. That said, we think this approach will require a lot of engineering efforts and hence we did not experiment with it in the paper.

Evaluation regarding preservation of very fine features

Reviewer: "have you quantified whether Ambient-o preserves very fine-grained details better or worse than training on fully curated data? For instance, does the patch classifier inadvertently retain tiny out-of-distribution artifacts?"

Super interesting question -- we thank the Reviewer for raising this. As in the low-quality data case, using data from any distribution other than the target will have to introduce some artifacts since convergence (even for small crop sizes) is not perfect. However, these artifacts do not seem to be detectable, at least using the tests we performed during the rebuttal.

Let's take the dogs and cats example, which is the most extreme case of using out-of-distribution data. To measure to what extent we preserve fine details, we measure a per-patch size FID: i.e we compare the distribution of patches of the generated samples with the distribution of patches from the training set and we compute the FID at different patch levels. We compare two models: the baseline model trained only on dogs vs our Ambient-o model trained on the same number of dogs + crops from cats. The results for the per-patch FID are shown below:

As shown, our method not only achieves superior FID on the global images, but also achieves better FID when looking at patches. This is evidence that we have better fine-grained details compared to the model trained only on clean data.

To further assess whether this makes sense, we also computed the per-patch size FID between 1) disjoint training images of dogs and 2) training images of cats and training images of dogs. Results are shown below:

As shown, for crop sizes 4 and lower the FIDs are comparable. Our crop classifier assigns an average annotation at crop size 6 (between 8 and 4), which checks out with this experiment.

We believe the provided experiments and clarifications strongly address the Reviewer's concerns. If so, we would appreciate if the Reviewer considers increasing their rating to reflect this.

We thank the Reviewer for their time and feedback! The Reviewer appreciated our experimental setup, our theoretical proofs and the strength of our experimental results. There are some concerns regarding missing ablations. We ran numerous ablations to address these concerns:

• We show results without training classifiers.
• We provide results on additive Gaussian noise.
• We show results with truncated sampling.
• We provide baseline results for only learning with clean data on ImageNet.
• We experiment with other corruption processes with additive noise.
• We test the robustness of the thresholds.

Reviewer: "[...] the method proposed is a combination of techniques. What if the noise level were treated as a hyperparameter and optimized via grid search?"

At its core, our method is very simple: bad data should only be used in a subset of diffusion times. Blurry images, for example, are typically useful in the high-noise regime, while out-of-distribution images might be useful in the low-noise regime. The training of the classifiers provides a rigorous way of finding the “allowed training times” (as they estimate the TV distance), but this is not necessary. For example, for our controlled experiments on blurry images, annotating all the corrupted samples with the same value already yields pretty strong results. We show these results below (annotated with Fixed Sigma):

Our paper has similar results (Fixed Sigma) for the out-of-distribution data experiment (Table 2b), too. Further, for our text-to-image results, we similarly annotated all the data from DiffDB at the same noise level.

Reviewer: "It would be interesting to see the results of the current method on the additive Gaussian noise case. It can also indicate the strength of the classifier."

We agree with the Reviewer that this is a good sanity check! We corrupt our data with the noise levels $\sigma=0.05, 0.1, 0.2$, as in the paper "How much is a noisy image worth", and we annotate each datapoint with the classifier. The estimated noise levels are shown below:

As shown, the classifier does a pretty reasonable job of estimating the noise level. We trained models using the estimated noise levels and we reproduced the $\approx 2.05$ FID result reported in the paper "How much is a noisy image worth". Our approach generalizes this idea to arbitrary corruptions.

Reviewer: "Recent works [1], [3] suggest that [...] diffusion models can effectively learn clean distributions by ablating various sampling methods, e.g., truncated sampling."

We believe the Reviewer has a critical misunderstanding here. In these works, it is known that the samples have been corrupted by Gaussian noise (and the noise level is given or can be estimated). However, for unknown degradation models (like the ones we deal with in the paper) it is information-theoretically impossible to recover the uncorrupted distribution without access to clean samples from it. That might be better understood with an example. If we are just given blurry samples, it is impossible to know whether the underlying distribution has blurry samples or the underlying distribution is a clean distribution and we just happened to observe blurry observations from it. Recovery guarantees for the underlying distribution without access to clean samples are only possible if the degradation model is known – hence, the distillation method of [1, 2] wouldn’t work for our settings where the corruption model is unknown. That said, we sincerely thank the Reviewer for bringing them to our attention, and will ensure they are properly cited in the camera-ready version.

Reviewer:: "The paper claimed it is learning from bad data, but in the experiments 10% data are necessary, right? It is also interesting to see the performance without clean data. This requirement of clean data would hurt the novelty of this setup."

Given only access to blurry samples and no prior on the degradation model, the best our method can achieve is to recover the distribution of the blurred samples. The FID of the blurred training set is given below:

Our method achieves significantly better FIDs (6.16, 6.00 and 5.34 respectively), with the benefit coming from the usage of the clean data. On the other hand, just using clean data does not suffice (see Table 2a) as it gives FIDs 45.32, 28.26 and 11.42 respectively. Hence, for the success of our method the combination of a small set of clean data and a large set of corrupted data is essential. These results are similar in spirit to the results obtained by the paper How much is a Noisy Image Worth that the reviewer mentions. Our work generalizes these results to arbitrary degradations (beyond additive Gaussian noise).

Reviewer: "The paper lacks ablation studies on the two regimes—high-quality and low-quality. What happens if either regime is removed during training? It would be insightful to see which regime contributes more to performance."

See above for results when only low-quality data is available. As we explained above, the underlying distribution is not identifiable if the degradation model is not known and the best one can hope for is recovering the corrupted distribution – this leads to very poor FID scores.

For the regime where we are only given high-quality data, we actually have results in the paper and we think that the Reviewer might have missed them. This is the Only Clean baseline in Table 2a and the Additional Data = None rows in Table 2b. We further have results for fine-tuning the text-to-image model only on a curated high-quality subset in lines 318-322 of the paper, showing that our method achieves a comparable FID score with a very significant increase in diversity (>13%).

Based on the Reviewer’s recommendation, we further report results on ImageNet for the Only Clean data baseline. We report results when 10% and 50% of ImageNet is available. We show test FID below:

As shown, having access to low-quality data is critical because if there is not enough high-quality samples, performance is quite poor. We will include these results in the Camera Ready of our work – we thank the Reviewer for suggesting this.

Reviewer: "It would also be interesting to extend the results to y=Ax + w [...]"

We thank the Reviewer for their suggestion, and include results on CIFAR-10 for A = gaussian blur and w = zero-mean poisson noise, as well as salt and pepper corruption. As in the paper, we compare results using 10% clean data to using 10% clean data + 90% corrupted data.

For the salt and pepper, we pre-processed the data with a 3x3 median filter to convert the unbounded impulse noise to a blurring corruption, which is more easily confused by gaussian noise. This allowed us to obtain a best FID of 5.70 using a fixed annotation t_min=3.

Reviewer: "Provide an ablation on the thresholds"

We show FID vs threshold results in the JPEG 18% case, keeping the data and classifier constant and changing only the threshold

Reviewer: "What are the results when training in the high-noise regime with truncated sampling? Are there other techniques beyond cropping that could improve performance?"

Truncated sampling is known to deteriorate results, as the samples get more blurry. We reproduce the same finding. On ImageNet, the results with truncated sampling are shown below:

Another technique that could improve results would be the consistency loss of the CDM paper. However, this loss is expensive to run as it requires sampling trajectories during training.

We believe our numerous ablations and clarifications strongly address the Reviewer’s concerns. If so, we would be grateful if the Reviewer considers increasing their score (which is currently the lowest among all Reviewers)

We thank the Reviewer for their time and feedback!

Reviewer: "Traditional filtering methods have a significant advantage in training efficiency."

We want to point out that it is not necessarily true. For many applications, practitioners run multiple epochs during training. Hence, the important parameter is the number of training updates, not the number of datapoints. All the baseline models used in the paper obtain their peak performance after multiple epochs of training. For example, for the ImageNet generative models, the EDM-2 XS baseline sees 2,147,483,000 (2B) images during training from a dataset of size 1.2M images. So in total, there are approximately 1000 epochs being run. Similarly, our text-to-image baseline (microdiffusion) sees 686,080,000 (700M) images during training, but the dataset only has 37M images (~19 epochs being run).

Methods that leverage low-quality data can run the same number of training updates with fewer repetitions of the same data, leading to less memorization and increased diversity. For all our experiments, we trained our models for the same number of training steps as the baseline models.

Reviewer: "experiments lack diversity metrics like the commonly used precision and recall in Imagenet"

We thank the Reviewer for their recommendation! For our text-to-image results, we already report Vendi Diversity (Line 320) and we show a greater than 13% improvement compared to a baseline trained on a curated dataset. We further show qualitative results for diversity in Figure 29 of the Appendix.

For our ImageNet results, we initially did not report these metrics because we followed exactly the evaluation protocol from the EDM and the EDM-2 papers that only report FID. Following the Reviewer’s suggestion, we further report the following metrics:

• Precision (higher is better): measures the expected likelihood of fake samples against the real data manifold
• Recall (higher is better): measures the expected likelihood of real samples against the fake data manifold.
• KD (lower is better): measures the kernel distance between the real and the generated distribution
• CT (lower is better): A memorization statistic (see A Non-Parametric Test to Detect Data-Copying in Generative Models).

For all these metrics, we use the dgm-eval library that provides implementations of metrics used in the paper: Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models

We report the results for our best model (Ambient-o-XXL+crops) trained on ImageNet (Table 1 in the paper) and the baseline model trained by the EDM2 authors. We use the same training parameters, training updates and inference parameters (EMA=0.015, guidance strength=1.2 – values taken from the EDM2 paper) for both models. The results are shown below:

As shown, our method improves the FID from 2.73 to 2.53 and it also achieves a reduction in memorization (lower CT score). We further achieve mild improvements in Precision and Recall.

We further want to clarify that our ImageNet models did not use extra data -- we just treat some of the existing data as noisy, and this leads to quality improvements. The improvement in Precision/Recall is mild since these are primarily diversity metrics, and we did not have access to extra data for this experiment.

We will report all these results in the Camera Ready version, we thank the Reviewer for the recommendation.

Reviewer: "[...] the method section predominantly focuses on data corruption rather than these real-world manifestations of bad data"

We believe the Reviewer has a critical misconception about our work here. We indeed perform experiments on real-world datasets on which real corruptions (such as poor aesthetics, low lighting, and other bad data forms) appear. In Figure 4 (right) we show examples of such “bad data” on ImageNet. Similarly, in Figures 24, 25, 26, 27 in the Appendix, we show examples of “bad data” that appear on the datasets we used for our text-to-image results: CC12M, SA1B, DiffDB, and JDB, respectively.

Our framework treats these real-world manifestations of bad data as noisy samples, and by doing so, it leads to quality and diversity improvements. For example, in the text-to-image setting, we manage to significantly outperform our micro-diffusion baseline in various benchmarks, such as DrawBench and Partiprompts (Figure 6), as well as in COCO FID and GenEval (Figure 7).

We only performed synthetic corruptions for the first part of our experimental validation to verify the effectiveness of each component of our approach, prior to moving to real corruption manifestations.

Reviewer: "However, in practical applications, data are mostly in a curated form"

Our paper questions this curation/filtering process. As we show in our text-to-image results section, filtering/curation leads to a greater than 13% drop in DINO Vendi Diversity. Qualitative results of this drop in diversity are further shown in Appendix Fig. 29. Further, such filtering pipelines are often driven by heuristics and arbitrary quality cutoffs. Instead of throwing away completely some samples, we propose using the “bad” samples only for certain noise times. This paradigm leads to a significant increase in diversity while still producing samples of high-quality (see lines 314-322 in the paper).

Filtering out datapoints entirely makes sense when we are bound by compute, not by data. I.e. if we can only run K training updates and we have at least K high-quality samples, it makes sense to use those.

Reviewer: "[...] the requirement to train multiple classifiers, the paper fails to clearly demonstrate how the proposed approach can be extended to broader application scenarios."

We want to highlight that, in its simplest form, our idea does not require the training of any classifiers. The proposal is the following: instead of filtering out the “bad data”, just use them for very high noise. Similarly, for out-of-distribution data, we propose using them only for “low-noise”. The amount of “high-noise” or “low-noise” can be a single hyperparameter. Already, this very simple idea achieves significant improvements – for the dog-cats experiment, FID improves from 12.08 to 9.85 and for our text-to-image results COCO FID improves from 12.73 (Stable Diffusion v1 level) to 10.61 (DALLE-2 level).

We provide further experiments to show this by showing FID results without a classifier (fixed sigma, found by hyperparameter tuning) for blurry CIFAR-10 images:

As shown, "Fixed sigma" approaches the performance of the trained classifier.

The classifiers provide a more principled way of selecting a different noise level per sample, as they estimate the time at which the TV distance goes below a threshold. The most principled version of our framework requires the training of classifiers over two things: noise levels and crops. The training of the classifiers is computationally much cheaper compared to the training of the diffusion model. However, we acknowledge that this might add engineering complexity, and we encourage practitioners to start with the simplest version of our framework (fixed sigma) using hyperparameter tuning or domain expertise to select the allowed noise levels for each sample.

Reviewer: "detail how to train a classifier for text-to-image tasks without class label."

The training of our classifier only requires the existence of a small set of “top-tier” samples. It is impossible for any method (not just ours) to work without it. The reason is that if the degradation model is unknown and there is no ground-truth sample access, the underlying distribution of pristine samples is not identifiable.

Reviewer: "Why is the FID improvement rather small or even worse? More data causes higher GPU resource consumption [...]."

We always train for the same number of training steps as the baseline, despite the fact that we have more data. Hence, our method is as computationally intensive as the baseline.

Regarding FID: for our text-to-image results, the FID improvement on COCO is massive (from 12.37 to 10.61). That’s the equivalent of going from a Stable Diffusion v1 model to a DALLE-2 model – and this is achieved only by changing the way we use the data (no extra compute + same hyperparameters).

Further, for ImageNet, we are exposed to the same number of samples as the baseline – we just use some of the available data as low-quality. Despite being exposed to the same data and being trained for the same GPU hours, Ambient-o-XXL+crops achieves state-of-the-art FID across all evaluation metrics and guidance settings.

We also want to highlight that the test images of ImageNet might also have imperfections – so if our model only produces pristine quality images, it might get penalized in FID. To account for this, we further report the mean CLIP IQA score for our Ambient-o-XXL+crops and the baseline XXL model (EDM2) below:

• Baseline XXL model average CLIP IQA: 0.69
• Ambient-o-XXL+crops average CLIP IQA: 0.71

The increase in the average CLIP IQA corroborates our findings that our model produces samples of higher quality.

We hope that the additional experiments and the clarifications addressed the remaining concerns. If so, we would highly appreciate it if the Reviewer considers raising their score to signify stronger support for our paper.

We thank the Reviewer for their time and very encouraging feedback! We are very pleased to see that the Reviewer appreciated the novelty of our approach, the breadth of our experimental evaluations, the theoretical proofs and the generality of the method. We most certainly agree with the Reviewer that as we are running out of Internet data, methods for efficient usage of data will become increasingly relevant. The same of course holds for scientific applications where we don't have a lot of data to start with.

Regarding the question about cropping: Great question and apologies for not having it clear in the first place in the paper. During training, the classifier takes as input a crop (that can be any size) and tries to detect if the crop came from the high-quality distribution or the low-quality distribution.

At inference time, we split an image into crops of a specific size, let’s call it C, and then we see if, on average, they confuse the classifier. The bigger the C, the harder it is to confuse the classifier. If there is no confusion for the initial C, we decrease it and we try again until we find a crop size for which the classifier is (on average) confused. We start with C=64 (the whole image), and work our way down. We underline that, in principle, we could do the annotation separately for each crop of the same image, effectively leading to different diffusion times per crop. That said, for the crop experiments we did in the paper, we used an implementation that averages across crops for simplicity.

We finally map from the crop size that confused the classifier to a diffusion time, using Figures 15/17. Higher noise levels require bigger receptive fields (bigger crops) for optimal denoising. So if an image only manages to confuse the classifier at a small crop, then it is only used for a small set of diffusion times.

We clarify that the diffusion model is never trained on crops of images. We always train using the entire image, even the out-of-distribution images, but for diffusion times small enough (less than t_max) such that for the required receptive field size at that noise level, the target and out-of-distribution data is indistinguishable.

We hope this clarifies things, and we will definitely make it clearer in the camera-ready version of our work.

Regarding the typo: We thank the Reviewer for careful reading! We will fix it in the camera-ready version.

If there is anything else we can do to further strengthen the Reviewer’s support for the paper, we would be happy to do so. Once again, we thank the Reviewer for their time and feedback.