Generative Data Mining with Longtail-Guided Diffusion

We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].

We are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.

1. [igG6] is concerned that the paper’s main idea is diffusion guidance based on energy signals and requests contextualization of additional related works.

Our main idea is grounding the notion of "predictive model longtail" into measurable signals. We demonstrate that we can not only generate additional longtail examples by this definition without changing the predictive model or the diffusion model, but that these additional examples significantly improve predictive model generalization on real eval data. Diffusion guidance plays a supporting role. We also show that predictive model longtails are model-specific (Figure 9) and proactively explain a predictive model's longtail with human-interpretable text (Table 3).

We agree that existing works have already demonstrated flexible guidance schemes in data space and, in fact, compare against one such work (Universal Guidance) in Section 3 (lines 260-290). We also provide new evidence based on FID scores for why this type of guidance works in Supplemental A.8 (Figures 13-15). We will expand the related works discussion (including the additional references) with the ninth page allowed in the camera ready

Universal Guidance, which we directly compare to in Section 3 and Figure 7, already cites the Diffusion Posterior Sampling paper [1] (and its relationship to it). In short, Universal Guidance and Diffusion Posterior Sampling are related in that both use a point estimate for p(x_0 | x_t) when performing guidance. Diffusion Posterior Sampling is focused on Gaussian and Poisson measurement noise (though their framework extends to nonlinear inverse problems where gradients can be calculated). Universal Guidance represents this extension (through the use of a nonlinear observation model represented by the diffusion model’s associated VAE) and takes it two steps further with expensive backward and recurrent sampling steps discussed in our paper, lines 260-270. Longtail Guidance exists between Diffusion Posterior Sampling and Universal Guidance because it differentiably decodes from a diffusion latent to the data space through a nonlinear, differentiable observation model (the VAE), but does not perform additional sampling steps (which are expensive and cause synthetic data to fall out of distribution, see Figure 7).

Loss-Guided Diffusion [2] builds on [1] by using Monte Carlo samples from an approximating distribution to further improve guidance. We do not use it in our approach since we would have to calculate the gradient through the VAE for each sample, which we already note to be an expensive operation (lines 292-295 and Supplement A.6). We will include and contextualize [1], [2, 3, 4] to our list of related works on diffusion control methods (ControlNet, DOODL, Freedom, Edict) and distinguish by whether or not they require additional diffusion or predictor training.

We emphasize that we are primarily concerned with downstream predictive model performance – a task that neither Diffusion Posterior Sampling nor Loss-Guided Diffusion evaluate on (but which our strongest baselines, GIF, Dream-ID, and LiVT do).

2. [iGg6] is concerned about whether we address model bias, distribution shift, and trustworthiness of synthetic data.

Please see response 5. to [Yasy].

3. [iGg6] asks for clarification and inlined descriptions of our experiment setup.

We fully describe our experiments in Supplement A.2. We will inline in the main paper with the ninth page allotted for the camera ready. Please also note that the 20-30x expansion is only for one apples-to-apples evaluation (with GIF). Other evaluations use less than 1x expansion of synthetic data. See response 6. to [Yasy].

4. [igG6] is concerned that some examples with mid-to-high LTG weight look implausible

The implausibility of some synthetic data with high Longtail Guidance weights are displayed for demonstration purposes only. The guidance weights in the experiments are in the low-to-mid range of the qualitative examples. We will clarify in the camera-ready

We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].

We are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.

5. [Yasy] says, “the paper does not quantitatively verify whether the generated data truly remain in-distribution.”

Thank you for raising this concern. Longtail guidance demonstrates significant generalization improvements over strong synthetic data generation baselines across eight datasets, with as many as 1000 distinct classes (Tables 1, 2, Supplemental A.1). It is common practice to trust (real) evaluation data and, in fact, our method enables users to reserve more real data for evaluation by more heavily relying on synthetic training data.

Furthermore, in Section 4.2 and Table 3, we show that forming new diffusion prompts from VLM descriptions of longtail-guided synthetic data and then training the original predictive model on that new data outperforms the baseline where diffusion prompts are formed from VLM descriptions of baseline synthetic data (without longtail guidance). This quantitatively supports that longtail-guided data are meaningful and remain in-distribution.

We agree that there is a tension between synthetic data distributional alignment and predictive model bias. Because we are working with natural image datasets, alignment can be equated with realism. Alignment and realism are explicitly controlled by the diffusion model’s text guidance weight. Predictive model bias is explicitly controlled by the longtail guidance weight.

We address the tradeoff between alignment and bias by holding the text guidance weight constant and selecting a longtail guidance weight such that the probability of the desired class (under the current predictive model) is lower than baseline synthetic data but not so low that it drops to zero. It is a hyperparameter that was selected one time – it does not need to be done for each generation or dataset. We describe this in lines 238-258 and show in Figure 2 that there is a strong inverse relationship between longtail guidance weight and the probability of correct classification under the predictive model for generated data (before retraining).

More broadly, while predictive model bias is a concern, it is also a concern that synthetic data generated without signals from the predictive model more rapidly saturate in new concepts (see rebuttal 6 to [Yasy]), leading to weaker generalization improvements. Our experiments strongly support that giving the predictive model “a voice” in the generating process is to its benefit.

6. [Yasy] is concerned that, “the central problem in the use of diffusion models for synthesising images useful for training classifiers is the sheer amount of synthetic data needed for meaningful classifier improvements. The paper doesn't quite address this problem. The method still generates synthetic data at a scale of 20× to 30× the original dataset size.”

We note that the results in Table 2 (Dream-ID comparison) all generate longtail-guided data at less than 1x the size of the training data and that, in Supplemental A.1, we show ImageNet-LT results that generates much fewer synthetic data – just 20% of the original dataset and, yet, still yield significant generalization improvements!

The 20-30x expansion in Table 1 is for comparing apples-to-apples with the GIF baseline (this is what we mean by “for parity.”). Further inspection of our existing results shows that predictive model generalization from our longtail guidance method surpasses GIF in all cases using just 20-30% (average 25%) of the data expansion that GIF uses (caltech: 4x expansion vs 20x), cars (5x expansion vs 20x), pets (7x expansion vs 30x), flowers (6x expansion vs 20x). Training with even more longtail-guided data yields further generalization improvements as we report in the paper.

We reported the equivalent expansion numbers in the paper for direct comparison, but we will take advantage of the ninth page in the camera ready to also add a graph that demonstrates that longtail guidance drives generalization improvements in a much more data-efficient way (4x more efficient) compared to GIF!

We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].

We are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.

7. [Qmgq] asks about computational efficiency

Computational efficiency is discussed for Longtail Guidance and the Epistemic Head in Supplement A.6. In brief, we generate baseline synthetic images (no Longtail Guidance) at a rate of 6.32 images / second (50 DDIM steps, fp16, no gradient checkpointing, 8xH100). We generate Longtail-Guided synthetic images at 1.01 images / second. The largest cost is the gradient calculation through the VAE, which consumes 45GB of VRAM in fp16 for batch size 8. The Epistemic Head impacts training and inference times by less than 2% and contains less than 5% of the original predictive model’s parameters. See also our response about 4x increased data efficiency over GIF in Response 6 to Yasy.

We thank the reviewers for their time and insights. We are grateful for two solid accepts [Yasy, Qmgq] and believe that we can fully address the concerns of the weak reject [iGg6]. We respond as best as we are able to the extremely brief review of [srKG].

We are pleased to hear from reviewers that “the contributions of this paper are novel and advance the field of longtail data generation in diffusion models,” that our approach of “conditioning generation on model longtail signals is a natural and effective approach” for exposing model weaknesses, that “iterative fine-tuning for generating synthetic data throughout training is a strong design choice, as it adapts to evolving model weaknesses rather than generating all synthetic data at once,” and that “the epistemic head’s comparison with entropy and energy baselines are convincing.” We are also pleased to learn that most reviewers (Qmgq, Yasy, iGg6) agree that our evaluations are sound and compelling.

8. [srKG] states, “if the authors want to validate the scenario in practice, they need to employ a diffusion model trained on a small dataset and use it to train a predictive model for a much larger dataset,” and, “applying this [production of rare or hard concepts] to a classification task, which is easier than a generative task, is like using a sledgehammer to crack a nut.

It is frequently the case that deployed predictive models have access to much less capacity and training data than do larger foundation models (including Internet-scale generative models). This can be due to limited memory and compute budgets and limited opportunities for real data collection (as we note in lines 38-48, 69-73). This is the premise behind the substantial literature of model and data distillation, including the works we cite in the paper (Yu et al., 2023b; Gou et al., 2021), and the more nascent literature on synthetic training data, including the works we cite in the paper (Du et al., 2024; Azizi et al., 2023; Zhou et al., 2023, Zhang et al., 2023b; Li et al., 2022b).

When working with a deployed predictive model, it is critical to understand scenarios that the model struggles with. As we detail in Response 1 to [igG6], our primary contribution is towards defining, mitigating, and proactively understanding a given predictive model’s longtail. Diffusion guidance only plays a supporting role. Thus, while we appreciate the reference on minority diffusion sampling, and will include it in our camera-ready, we also note in our analysis (Section 4.3, Figure 9) that what is difficult for one model (even a foundation model like CLIP) is not necessarily the same thing as what is difficult for a given, deployed predictive model. We in fact show that Longtail Guidance demonstrates significant generalization improvements over strong synthetic data generation baselines (that use a foundation model for diffusion guidance) across eight datasets, with as many as 1000 distinct classes (Tables 1, 2, Supplemental A.1).