Multi-Perspective Data Augmentation for Few-shot Object Detection

W1: My main concern lies in the use of Diffusion Model (DM) and ChatGPT. From the perspective of the FSOD task, data is always the most crucial element. Consequently, it is essential to employ ChatGPT and DM to generate data as a means to address this concern, as the authors did in this study. However, the following questions arise: if ChatGPT and DM are being used, why not simply generate a substantial number of samples of the corresponding categories and conduct direct training on the generated data? Alternatively, another option could be to pre-train the detector on the generated data and subsequently perform fine-tuning on few-shot categories. Currently, the authors haven't furnished a compelling rationale to either explain, oppose, or support these viewpoints, which are of utmost significance for this paper. To answer these questions, I suggest the authors: (1) conduct a comprehensive analysis comparing the performance of direct training on generated data with the proposed method; (2) explore the potential benefits and drawbacks of pretraining on generated data and then fine-tuning on few-shot categories. This could include experiments to determine the optimal amount of pretraining and the impact on final performance; (3) provide a detailed explanation of why the proposed approach of using ChatGPT and DM in a specific way is more advantageous than (1) and (2).

We thank the reviewer for your critical comment. We completely agree that data is the most crucial issue in the FSOD task. To ensure we understand your comment correctly, we restate the following points:

Why our augmentation framework (MPAD) is more advantageous than using simple prompting with ChatGPT and DM. Could using simple prompting with a substantial number of samples to train the model be better than our method?

The main difference between our MPAD and previous works (including simple prompting) is that we leverage the foreground-background relation. Simple prompting (e.g., using ChatGPT to generate prompts for DM to create images) results in a lack of diverse and hard samples for training the model. In object detection task, the background plays a crucial role. Using simple prompting, ChatGPT and DM generate data with uncontrolled backgrounds.

To confirm this point, we conducted additional experiments, and the results are reported in the table above. We simply generated data with varying sample sizes: 5, 10, 50, 100, 200, 300, 400, 600, 800, and 1000, then trained the model. The performance becomes saturated when the number of samples reaches 600 (i.e. performance does not improve when increasing #generated samples). Compared to the best performance in this experiment, our model still outperforms by $3.7\%$, even using only 300 samples. We also added the experiments to determine the number of generated images in Table 7 in the Appendix section.

Why our training scheme using both generated data and original few-shot data is better than (1) training with generated data only, and (2) pretraining with generated data, then fine-tuning with original few-shot data?

Our training scheme is superior to (1) because it leverages both few-shot samples (real and high-quality samples) and diverse generated samples in a single training time. Our MPAD is less prone to overfitting compared to scheme (2). Even when pretrained with a large amount of training data, fine-tuning with too few samples can cause the model to "forget" knowledge from the pretraining phase. Moreover, MPAD is less computationally expensive due to its simpler training process, while (2) requires training twice.

To confirm this point, we conducted further experiments with the same number of images. Using scheme (2), the performance significantly decreased to $61.6\%$, compared to $63.1\%$ of scheme (1) and $69.1\%$ of our MPAD.

We deeply appreciate reviewer B8R6 for valuable time and providing insightful feedback that has enhanced our manuscript. We will include the analysis of various training schemes in our manuscript.

W1: The framework combines multiple advanced techniques, including diffusion models, harmonic prompt scheduling, and complex background sampling, which may make it challenging for practitioners to implement effectively in real-world scenarios. At the same time, it increases the complexity of the model. Please analyze the model complexity and real-time inference of the generated model.

Our method adjusts feature embeddings in a controllable diffusion process without making any modifications to the model architecture. Specifically, instead of using text embeddings as described in PowerPaint, we use $\gamma_c^t$ as defined in Equation (3) (line 258, page 5). Additionally, base features are precomputed at the start of the generation process and computed only once. Therefore, there is no significant increase in computational cost or complexity. The inference time for the generation process primarily depends on the inference time of the diffusion model and the number of generated images.

W2: The paper demonstrates performance gains, but it would benefit from a more granular analysis comparing the effectiveness of each augmentation component (CPOS, HPAS) against similar elements in other FSOD methods. Has the pre-trained model of the generated model already seen various categories to generate more realistic new class samples when regenerated again? Can other generative models also generate samples to improve the performance of small samples, and does the proposed method have such a significant performance improvement compared to this type of method Data Efficiency in Synthetic Generation:

We use the pretrained PowerPaint on Open Images [3] and LAION-Aesthetics V2 5+ [4]. Therefore, all novel and base classes are seen during pretraining. We also compare our method with FSOD approaches using other generative models, as shown in Table 1 and Table 2. Specifically, [1] uses a VAE to generate synthetic features, while [2] uses an MAE to reconstruct new images for novel classes. Our method surpasses [1] and [2] by approximately 15% and 11%, respectively.

W3: While the method performs well, additional experiments assessing data efficiency could be valuable. Evaluating how much synthetic data is optimal or exploring the impact of different amounts of augmented data on performance could provide insights into the scalability and efficiency of the framework.

We thank the reviewer for the constructive review. We added the experiments to determine the number of generated images in Table 7 in the Appendix. We chose the number of generated images to be 300 for best results.

References:

[1] Few-Shot Object Detection via Variational Feature Aggregation, AAAI 2023.

[2] SNIDA: Unlocking Few-Shot Object Detection with Non-linear Semantic Decoupling Augmentation, CVPR 2024.

[3] Unified image classification, object detection, and visual relationship detection at scale, IJCV 2020.

[4] Laion-5b: An open large scale dataset for training next generation image-text models, NeurIPS 2022.

We hope that our response has fully addressed your concerns. Please kindly let us know if you need any further clarification or information.

We sincerely thank the reviewer for the constructive comments. There are several previous works [1, 2] using diffusion models in FSOD. However, such works do not focus on exploiting the foreground-background relation or extending the ability of diffusion models by using LLMs. To our knowledge, this is the first work to use ChatGPT to diversify prompts and embed the foreground-background relations when synthesizing diverse datasets in few-shot object detection.

W1: The description that "this work is the first ..." in Lines 81-83 is a little bit over-claimed. From my perspective, there is already a large amount of work exploring using large-scale pretrained diffusion models for object detection by generation (like general object detection and corner case generation for autonomous driving). Simply extending the setting to few-shot object detection isn't that significant to me.

As we mentioned in the above response, this is the first work using ChatGPT to diversify prompts and exploiting the foreground-background relations. Prior FSOD works [1,2] only use diffusion models with the simple prompts. We also show that simply using diffusion model is not enough to create diverse datasets in Table 4. Specifically, our MPAD (last row) can improve over 7% for all metrics in comparison with straightforward method using only controllable diffusion method (first row).

We also emphasize this point in the Experiments section (lines 442-443, page 9) : "Specifically, controllable diffusion using ICOS in the third row diversifies prompts, enhancing the diversity of the synthetic dataset and increasing detector performance by approximately $6\%$ nAP50 compared to not using ICOS (i.e., directly using PowerPaint with simple prompting, as shown in the first row)"

W2: Section 2.3 (CoT prompting for object synthesis) actually belongs to prompt engineering. The Chain-of-Thought emphasizes solving a problem step by step. For object synthesis discussed in this paper, the task simply needs to list all possible attributes of a category ( or use a pre-defined template). This can be regarded as in-context learning or prompt engineering rather than Chain-of-Thought. Meanwhile, the inpainting model is also modified from other previous works.

We thank the reviewer for your valuable suggestion. Both CoT and in-context learning share attributes of prompt engineering, such as providing detailed information in the input. We have revised "CoT" to "in-context learning" and updated the manuscript accordingly.

Dear Reviewer RKTD,

We thank the Reviewer for your time and effort in evaluating our paper and providing valuable feedback. We have carefully addressed your comments and incorporated improvements in our rebuttal. If you have had a chance to review our response, we would greatly appreciate your thoughts. Additionally, we sincerely hope you will consider updating your score if you feel our revisions have effectively addressed your concerns.

Dear Reviewer RKTD,

As the extended discussion period nears its conclusion, we would greatly appreciate it if you could take a moment to review our rebuttals and let us know if you have any further questions or concerns. Our goal is to address all concerns thoroughly during the discussion phase, and your feedback is vital to this process.

W1: The present approach heavily depends on utilizing pre-trained models to select typical and challenging samples, potentially causing interference when assessing the efficacy of augmentation strategies

We follow previous works [1, 2, 3, 4, 5, 6, 7] to use the pretrained models in our method. We also compare FSOD methods using pretrained model, which is clarified in the below response.

W2: Additional clarification is needed regarding the fairness of the experiments.

We thank the reviewer for positive suggestion. We clarified the fairness of the experiments and updated in the manuscript in the Experiments section (lines 413-417, page 8): "Considerably, our MPAD surpasses previous works Wang et al. (2024); Lin et al. (2023); Kaul et al. (2022); Li et al. (2023a); Zhu et al. (2021); Xu et al. (2023) that use pretrained CLIP, ViT, diffusion models, language models, or post-processing in detection. Meanwhile, methods Wang et al. (2024); Lin et al. (2023) are state-of-the-art data augmentation methods in FSOD"

Q1: What are the definitions of typical and hard samples? Detailed assessment criteria need to be further elaborated

In this paper, we assume that typical samples contain only features of one class and are created using augmented prompts with a diffusion model. Hard foregrounds are blended with features of two classes using HPAS. We also emphasize this point in the introduction section (lines 071-075, page 1): "In other words, in this paper, we define typical samples as those that contain features of a single class, whereas hard samples exhibit features of two classes."

Q2: In Section 2.2, lines 159-161 mention that HPAS can generate hard samples. Strategically, it may indeed involve a wider range of categories. However, for the model, the level of difficulty in recognition is not solely dependent on the number of categories in the image. Why are samples generated based on mixed prompt embedding considered hard samples in HPAS?

We agree with the review. There are other transformations that can create hard samples, such as occlusion and scale. These types of transformations are embedded during the random selection of bounding boxes with multiple scales and positions. In this paper, we consider another type of hard sample: ambiguous objects that contain features of more than one class and are hard to recognize. To generate these types of hard samples, we propose using HPAS. To the best of our knowledge, this is the first work that considers this type of hard sample in data augmentation for FSOD.

Q3: In the last paragraph on the fifth page, it mentions the concept of camouflage targets. Camouflaged objects typically refer to objects where the foreground and background have high similarity. However, in the subsequent text, the selection of data with high similarity between the background in the base stage and the classes in the novel stage as challenging data does not explicitly demonstrate the connection between selecting difficult backgrounds and camouflaged objects. Therefore, what is the relationship between the selection of challenging backgrounds and camouflaged objects?

To define the hard background, we are inspired by the concept of camouflaged objects, where the foreground and background have high similarity in appearance. To simulate that phenomenon, we encode the visual information of the background (from the base set) and a class representative (from the synthetic dataset), then compare them using a similarity metric like cosine similarity. We do not create a camouflage dataset; we simply inherit the concept of visual similarity between foreground and background.

Q4: In Equation 6, it is mentioned that typical clutter background is obtained by calculating entropy. Such selection heavily relies on the performance of the feature extractor F. If the predictive results are good, few background images may be selected. However, if the predictive results are poor, do the selected background images hold any useful value?

We agree with the reviewer. A poor $F$ extractor may evaluate all base images at the same point and degrade the performance of MPAD. Therefore, we choose a general and good enough extractor (ViT) which is also used in several previous works [2,3,4,5] in FSOD.