Boost-and-Skip: A Simple Guidance-Free Diffusion for Minority Generation

Thank you for your thoughtful feedback and for considering our work for acceptance. We appreciate your time and evaluation. If you have any further suggestions or questions, feel free to let us know. We are more than happy to address any additional points or provide further clarifications as needed.

We appreciate the reviewer for your detailed comments and valuable suggestions. Below, we provide thorough point-by-point responses to address your concerns.

1. [7UiG] expressed a concern on the sensitivity to hyperparameters.

Please refer to our response to Reviewer o8Xg (the second bullet point).

2. [7UiG] suggested comparisons with GAN-based minority generation frameworks.

To reflect your comment, we have conducted new experiments to compare our method with GAN-based minority generation frameworks. Specifically, we evaluate a class-balancing GAN approach [1] and compare its performance in generating minority samples with ours on CIFAR10-LT (a long-tailed version of CIFAR-10). See the table below for the results.

As in the experiments presented in our paper, we used real minority data (from CIFAR10-LT) as the reference for computing the metrics. Observe that B&S outperforms the GAN-based approach in [1] (i.e., CBGAN) even under this highly-biased benchmark, further highlighting its effectiveness as a minority generator.

3. [7UiG] pointed out that literature reviews on minority-conditional diffusion models are missing.

We kindly remind the reviewer that approaches with minority-conditional diffusion models are discussed in the related work section in Appendix A.1. We will move them to the main body in our revision.

4. Clarifications on Fig. 7.

We kindly remind the reviewer that the focus of minority generation is not to replicate the training data distribution but to intentionally bias generation toward minority instances, which are defined as low-density on-manifold samples. In this regard, Fig. 7 demonstrates that our framework effectively achieves this goal, as the high-valued neighborhood metric values imply the generation of low-density instances.

While the reviewer might be concerned that high neighborhood metric values indicate the presence of off-manifold (i.e., OOD) samples of poor quality, we emphasize that our method does not generate more OOD samples than state-of-the-art minority samplers such as Minority Guidance [2]. This is evidenced by our superior FID scores compared to Minority Guidance (e.g., in Table 1), where FID is computed using real minority (i.e., on-manifold, low-density) data.

5. [7UiG] suggested a sanity check of Proposition 3.2 using the toy data in Fig. 2.

Per your suggestion, we did an experiment for the sanity check in Fig. 1 (provided in the link below). Specifically, we plot generated data variance as a function of initial Gaussian noise variance for the two rings example. Blue dotted line denotes generated data variance predicted by theory, and the orange solid line illustrates actual generated data variance. The general trend of the two curves are similar, validating the ability of B&S to generate minority samples. We conjecture that the offset between theory and practice may arise from the score estimation error, as we use learned scores rather than exact scores to simulate B&S.

• Link to Fig. 1: https://docs.google.com/presentation/d/1GCZKcxbX_A7e_v_ckVCefcVxGsbm2UkZSxyUc5zuSbA/edit?usp=sharing

6. [7UiG] questioned the effectiveness of our approach on highly imbalanced benchmarks.

To address your question, we consider CIFAR10-LT, a highly-imbalanced version of CIFAR-10, and explore the performance benefit of ours. We found that B&S yields improved minority generation even under this biased setting; see the table above (included in the second bullet point of this response) for detailed results.

7. [7UiG] suggested comparisons with minority-conditional diffusion frameworks.

We gently remind the reviewer that our experiments already encompass such approaches, by incorporating ADM-ML [2] - a classifier-guided diffusion sampler conditioned on known minority labels in CelebA. See details in Table 1.

References

[1] Class Balancing GAN with a Classifier in the Loop, UAI 2021

[2] Don’t Play Favorites: Minority Guidance for Diffusion Models, ICLR 2024

We greatly appreciate Reviewer o8Xg for the strong acceptance and thoughtful feedback. Below, we provide detailed point-by-point responses to address your remaining concerns.

1. [o8Xg] questioned the effectiveness of our approach on highly biased datasets.

To address your concern, we consider CIFAR10-LT, a highly-imbalanced version of CIFAR-10, and investigate the performance benefit of ours. See the table below for the results.

"CBGAN" refers to a class-balancing GAN approach that implements minority generation with minority conditional labels [1]. Similar to the experiments in our paper, we used real minority data from CIFAR10-LT as the reference for calculating the reported metrics. Observe that B&S outperforms the considered baselines (including the GAN-based approach in [1]) under this highly-biased benchmark, further demonstrating the robustness of our framework.

2. [o8Xg] expressed concerns regarding the sensitivity to hyperparameters.

Although we acknowledge that our framework may be sensitive to hyperparameters — particularly $\gamma$ — we provide a practical heuristic for their selection in Appendix C (Lines 1250–1251), which streamlines the process of choosing both $\gamma$ and $\Delta_t$. Leveraging this approach, a simple one-dimensional grid search over $\gamma$ is sufficient to identify effective hyperparameters (see Lines 1267–1270 for details).

We also highlight that the implementation complexity of our method is significantly lower than that of existing guided minority samplers, which often involve numerous design choices. For example, the approach in [2] requires training two separate classifiers with many design options (e.g., classifier architectures), while the method in [3] necessitates the selection of six hyperparameters. In contrast, our framework only requires choosing two hyperparameters, providing substantial practical benefits over the guided minority samplers in [2,3].

3. [o8Xg] expressed concerns regarding the visual quality of minority samples.

We believe there are largely two reasons behind the quality degradation mentioned by the reviewer. First, there is a general trade-off between minority sampling performance and image quality. Second, the base diffusion model itself lacks the ability to generate minority features. We provide further explanation for each hypothesis below.

To investigate the first hypothesis, since FID and Precision are measured with respect to ground-truth minority samples, we can use FID as a proxy for how close the generated distribution is to the distribution of minority data, and Precision as a proxy for the quality of generated minority samples. In Fig. 2 (provided in the link below), we observe there is generally a trade-off between the two quantities, and B&S provides competitive trade-off performance while achieving a dramatic reduction in inference cost compared to guidance-based minority methods. The reviewer is also directed to Table 2 (c), where we again observe FID vs. Precision trade-off as we adjust boosting strength in B&S.

• Link to Fig. 2: https://docs.google.com/presentation/d/1dsMx8s5kJikQnjv6IQNvk-UyC_DgfU9xJpF-ZkaLweI/edit?usp=sharing

Also, we hypothesize that in some cases, the base diffusion model may lack the capability to synthesize minority features. For instance, on ImageNet, it is well-known that generating human faces is challenging; see Fig. 7 in [4], Fig. 15 in [5], and Fig. 5 (b) in [2]. We believe that is why human faces in Fig. 11 (c) synthesized by B&S appear unnatural and distorted. Using a better base diffusion model may yield more realistic minority samples.

References

[1] Class Balancing GAN with a Classifier in the Loop, UAI 2021

[2] Generating High Fidelity Data from Low-density Regions using Diffusion Models, CVPR 2022

[3] Self-Guided Generation of Minority Samples Using Diffusion Models, ECCV 2024

[4] Large Scale GAN Training for High Fidelity Natural Image Synthesis, ICLR 2019

[5] Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021

We thank Reviewer bs5F for the constructive feedback. Below we provide point-by-point responses on your questions and concerns.

1. [bs5F] expressed a concern that the performance is often limited compared to baselines.

We note that the superior baselines in Table 1 (e.g., [1,2]) correspond to guidance-based minority approaches that employ guidance terms to direct inference toward low-density regions. While we acknowledge that our suboptimal performance may stem from the lack of such explicit guidance for minority generation, our method significantly advances the Pareto frontier in the performance-complexity tradeoff (see Fig. 1). In particular, our framework delivers notable computational benefits over the guided minority approaches. For example, on ImageNet-64, our method achieves 65% reduction in wall-clock time and 4.5× lower peak memory usage compared to the current state-of-the-art [2] (see Table 3).

2. Clarifications on Fig. 5.

We would like to assure the reviewer that the visual attributes of our samples in Fig. 5 capture distinctive features of minority instances. For instance, jack-o’-lantern images with bright surroundings (rather than the typical dark Halloween atmosphere) are considered as low-density features of the class [3]. Also, our eagle-class images in the same figure exhibit more intricate visual details compared to the baselines, which are also known as minority features [1,2].

3. [bs5F] noted missing neighborhood metrics like AvgkNN.

As noted in L356-358 (right column), the evaluation results using AvgkNN, LOF, and Rarity Score are provided in Appendix D.1, where B&S performs consistently well across all three metrics, rivaling the state-of-the-art guided minority sampler [2]. See Fig. 7 therein for explicit details.

4. [bs5F] noted missing comparisons with guidance-based minority methods.

We would like to gently remind the reviewer that, in Tables 1,3,4, we already compare B&S with guidance-based methods such as ADM-ML [1], Minority Guidance [1], and Self-guidance for minority generation [2].

References

[1] Don’t Play Favorites: Minority Guidance for Diffusion Models, ICLR 2024

[2] Self-Guided Generation of Minority Samples Using Diffusion Models, ECCV 2024

[3] Generating High-Fidelity Data from Low-Density Regions Using Diffusion Models, CVPR 2022