More datasets should be inculded

Response 4.1:

As suggested by the reviewer, we have re-executed experiments on the EuroSAT and FGVC Aircraft datasets, which cover OOD, fine-grained, and object classification. We are also working on the Country211 scene dataset. Please allow us some time to complete this. The results are presented in Table 1,2 of Section 4.1 of the manuscript. We also list them here:

As it shown, proposed DeCap method could consistently imporve baselines across these datasets. These experimental results further support the capability of proposed DeCap method in generating high-quality images for downstream few-shot learning tasks.

The OOD issues caused by resolution.

Response 4.2:

We want to kindly clarify that the image resolution is not a major issue. In practive, we could resize the images generated by the model to 32 or other resolutions without causing significant OOD issues compared to the original datasets. Additionally, the fact that our method still works well on low-resolution datasets further emphasizes the effectiveness of our approach for different types of datasets.

Moreover, the difference between a 96 resolution image and the standard 224 resolution image do not result in significant OOD effects. To illustrate this, we conducted an experiment where we resized the data from the original resolution to 96, then further resized it to 224 on the ImageNet100 dataset. The results showed that the performance only changed by about 0.1%, which we believe is entirely acceptable.

Dear Reviewer 9gWJ,

The additional discussion period will come to an end. We kindly request your feedback on whether our new responses have effectively addressed your concerns. Best regards,

The Authors

About generalization and efficiency

Response 3.1:

• Firstly, we want to kindly clarify that our method does not attempt to approximate the few-shot data domain by altering prompts. Actually, most of existing methods are primarily limited to generating data by hand-designed prompts for approximating the few-shot data domain. These approaches are prone to overfitting and may generate images that are very similar to the few-shot data. To overcome the limitations, we explore to utilize the few-shot data to evaluate the quality of generated images at the higher task(meta)-level. In other word, we pay emphasis on the classification performance of the few-shot task rather than visual similarity of the few-shot data. This could be validated in Fig.6 of Appendix D.6, where the generated images using mined prompts by DeCap method have visually distinct characteristics from the real data, but the classification accuracy of model trained with only generated images closely matches to that trained with the full real dataset. Specifically, we trained our model using PACS-Sketch dataset, but the learned prompts encompass diverse styles, such as painting and cartoon, rather than being limited to visual similarity alone.[Additionally, we compare our method with previous methods that aim to approximate few-shot data domain (also see Response 3.3)] and these experiments clearly demonstrate the superiority of task-level data generation.

• With such novel meta-learning formulation, we want to further clarify that our DeCap method is efficient and has good generalization capability. Actually, the meta-learning paradigm consists of two stage: meta-training and meta-test. Though it requires some cost to mine proper prompts during meta-training stage, the generalization and efficiency is reflected at the meta-test stage. Specifically, the generalization is reflected in the ability to transfer from the available few-shot dataset to the full dataset--just see Table 1 of Section 4.1, which we are only access to few-shot dataset during training, but successfully transferred to full dataset during testing. And the efficiency is demonstrated by the fact that the learned prompts can be directly applied to meta-test tasks without further tuning. This means when we finish training, our method can enjoy the same speed as hand-crafted methods.

In a nutshell, we admit that our method is still relatively time-consuming during the meta-training phase, while at the meta-test stage, our method could exhibits both efficiency and generalization. This is brought by the meta-learning formulation for the problem, rather than approximate the few-shot data domain by altering prompts.

Compared with image-based data curation methods

Response 3.3:

Thanks for your valuable suggestions for the related papers, we have cited them in our manuscript. And we clarify differences and connections with our method as follows.

• Current research on prompt-based data generation methods mainly focuses on how to generate images when data are scarce or unavailable for the concerned task, while data-based data augmentation methods are more focused on how to further boost the performance when data are relatively enough. We can see that both of them have their own advantages on specific problems, and they could be used simultaneously as done in the papers given by the reviewer. In this paper, we focus on prompt-based data generation methods, and thus we compare the prompt-based data generation aspects of these works for a fair comparion. These prompt-based data generation methods can be broadly categorized into two types: textual inversion for learning prompts and the use of SDEdit technology to generate images. We provide the compared results using SDEdit, textual inversion, and DeCap on the STL10 and Caltech-101 datasets as follows:

Above results show that our method has an consistent advantage over these methods under few-shot problem (i.e., scarce data). Of course, image-based data augmentation methods can be applied to further boost our approach and compared baselines. This would be explored in our future work.

• Here, we explore to use image-based data generation methods to alleviate the issue of the downstream few-shot tasks with extremely scarce data. Specifically, one could use image-based data augmentation techniques to create an augmented real dataset and then apply our method to mine proper prompts for generating high-quality images, and the experimental results are described in the response 3.4.

Improved strategy in extremely low-shot cases.

Response 3.4:

Thanks for this valuable suggestion. We addtionally provide the results for the 1-shot scenario on the STL10 dataset. As it shown, even under extremely scarce data, our method could achieve better performance than CiP method with 10-shot images. By using image-based data generation methods to alleviate the issue of scarce data, the performance of our method could be further increased. However, we would admit that, due to the limitations of scarce data in characterize few-shot task, hand-crafted prompt methods (the vanilla prompt) could yield better results by utilizing the knowledge embedded in the generative model. When given data increase to 10-shot, our method could achieve competitive results. And we will explore to better address the issue in the future work.

Addtional questions related to generalization and efficiency

Response 3.2:

• Regarding efficiency: Firstly, we want to kindly clarify that although generating images from text involves a high degree of freedom, the classification boundaries of model trained on these generating images are not as fragile as one might expect in our experiments. On the contrary, they are exceptionally robust.

To illustrate this, We provide two experiments:1) we give the classification performance on Imagenet100 dataset equipped with different generate numbers per class, in order to show classification boundaries are not sensitive to slight variations. The same finding is also found in [1]. 2) in Table 1 of our experiments, we already used different random seeds for testing and the results are quite stable (However, due to space constraints, we removed the standard deviation in the latest PDF. You can refer to earlier versions, or if you still have doubts, we can provide the data for your reference.). The results show that the model exhibits good robustness. These two points strongly support the feasibility of optimizing prompts for dataset construction.

Additionally, for the issue of too many uncontrollable components, we controlled these components during the meta-training phase, including the initialization of the downstream model, random seeds for data generation, and data augmentation methods. However, during the meta-test phase, we did not control any of these components, yet our experimental results still show that our method is effective. This may attribute to the generalization capability of our approach. As for the random seed hyperparameters you mentioned, we have already incorporated them into our experiments. If you're interested in other hyperparameters, please feel free to ask.

• Regarding generalization: To illustrate the out-of-distribution (OOD) generalization capablity of our DeCap method, we conduct the following experiment: we used CLIP's linear probe for training on the ImageNet100 dataset to eliminate any information inherent in CLIP's text encoder and then directly tested on various datasets without any further training. The results are shown below. We attribute this phenomenon to the diversity-enhancing capability of DeCap.

[1].Fake it till you make it: Learning transferable representations from synthetic ImageNet clones.CVPR,2023.

The task is limited to classification problem and datasets used for evaluation are insufficient to make this paper solid.

Response 2.2:

Thank you for recognizing the versatility of our method. As a framework, we also believe that DeCap has great flexibility and potential for extension to other domains, but this is not an easy task. In fact, our algorithm incorporates several unique designs specifically for classification that enable it to achieve classification-awareness.

For other tasks, such as object detection, how to correctly utilize the labeled information from generated data—such as understanding which pixels the text encoding is sensitive to—becomes the biggest challenge. This is not trivial, and we discuss the limitations of our method in detail in Appendix A, hoping that our paper can inspire further research in this area and contribute to other domains.

Additionally, many of the current works[1-7] on generating datasets focus on classification tasks. Extending their research ideas in this way does not seem too narrow. Lastly, our dataset already includes categories such as objects, remote sensing, fine-grained, low resolution, and specific domains, and we are preparing a scene dataset for training. If you feel that our dataset is still insufficient, we would appreciate more detailed suggestions.

[1].Synthetic Data from Diffusion Models Improves ImageNet Classification. TMLR,2023.

[2].SuS-X: Training-Free Name-Only Transfer of Vision-Language Models. ICCV,2023.

[3].Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learner. CVPR,2023.

[4].Is synthetic data from generative models ready for image recognition? ICLR,2023.

[5].Fake it till you make it: Learning transferable representations from synthetic ImageNet clones. CVPR,2023.

[6].Leaving Reality to Imagination: Robust Classification via Generated Datasets.ICLR,2023.

[7].DA-fusion: Effective Data Augmentation With Diffusion Models. ICLR,2024.

Commercially or open-sourced LLM or Multi-modal LLM will show much better performance with even not using genetic algorithm.

Response 2.1:

We want to kindly clarify that our work is different from existing commercially or open-sourced LLM or Multi-modal LLM.

• Firstly, existing commercially or open-sourced LLM or Multi-modal LLM focused on improving quality of generated images, while they pay less attention to using generated images to solve downstream tasks. As a result, although their capabilities in improving image generation are becoming better, e.g., high-resolution images, this does not mean that they could generate images that are more helpful for downstream tasks, e.g., few-shot classification tasks investigated in this paper. Previous works like [1-7] have explored to use open-sourced Stable Diffusion models to help generate images for classification task. However, existing works are relatively hard to guarantee that training models from synthetic images are efficient for downstream classification tasks. To further improve the quality of generated images for classification, we propose a novel prompt learning strategy for improve downstream few-shot classification tasks.

• Secondly, the latest commercially or open-sourced large language models could generate text prompts for text-to-image generative models to generate high-quality images. However, no matter how strong the generation ability or how high the text quality is, the generated text prompts are always less related to downstream tasks. Some generated text may be helpful for training, but others may have no or even negative impact on classification performance (Please also see Figure 3 of the manuscript). To address this issue, we propose a novel meta-learning approach to adaptively mine proper prompts tailored for the few-shot learning task. Our experimental results further support that mined prompts are attained specifically suitable to concerned few-shot learning task. Especially, our method has weak correlation to open-sourced large language models.

• Thirdly, we want to kindly clarify that the powerful large language model is not always better. Current text-to-image generation models may not fully understand complex text description, which may lead to overly detailed prompts by LLM may perform poorly. As compared, our adaptive prompt learing strategy could learn proper prompts tailored for the few-shot learning task.

• Fourthly, we additionally provide results on EuroSAT and FGVC Aircraft datasets to illustration the superiority of proposed method. The results are shown in Table 1,2 of Section 4.1 of the manuscript. These datasets present more significant challenges than commonly used datasets. We use text prompts generated by ChatGPT to construct the prompt pool, and then use our method to select the proper propmts. The results show that our method has a significant advantage over simply use ChatGPT to generate prompts. This strongly supports our viewpoint.

In a nutshell, we believe our work would inspire more researchers in the community to focus on the limitations of current LLMs, and explore more meaningful researches built upon the advancements of LLMs.

[1].Synthetic Data from Diffusion Models Improves ImageNet Classification. TMLR,2023.

[2].SuS-X: Training-Free Name-Only Transfer of Vision-Language Models. ICCV,2023.

[3].Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learner. CVPR,2023.

[4].Is synthetic data from generative models ready for image recognition? ICLR,2023.

[5].Fake it till you make it: Learning transferable representations from synthetic ImageNet clones. CVPR,2023.

[6].Leaving Reality to Imagination: Robust Classification via Generated Datasets.ICLR,2023.

[7].DA-fusion: Effective Data Augmentation With Diffusion Models. ICLR,2024.

The authors are arguing that exploiting given prompts will always show better performance. However to do so, there should be evidence about that hypothesis.

R: In fact, we do not assume that prompts will necessarily lead to better results; many prompts can even produce worse outcomes for downstream classification tasks. This is elaborated in detail in Figure 3 of the manuscript. For example, in the cat-and-dog classification task, prompts containing references to both cats and dogs (e.g., "a dog plays with a dog") can negatively impact classification performance. These results show it is challenging to directly exploit given prompts to improve downstream classification performance. To address this limitation, we propose to mine proper prompts from given prompt pool which are attained specifically suitable to concerned downstream classification task. The result in Table 1, 7, 8, and 9 support the capability of proposed method in extracting effective prompts, demonstrating that our approach achieves better results across various metrics and datasets compared to existing algorithms.

It is hard to say that a Genetic algorithm applied upon training a classification model is simple and practical.

R: Other reviewers acknowledged the effectiveness and practicality of our method. Our experimental results (Tables 1, 2, 7, 8, and 9) demonstrate this point across classification performance, generalization, and various evaluation metrics. As for simplicity, we want to kindly clarify that our method is relatively time-consuming during the meta-training phase, while is high efficiency and generalization during the meta-test stage. This point is also acknowledged by other reviewers.

Specifically, we would like to restate our method to provide a clear understanding of its generalizability and cost. Existing data generation approaches are relatively limited to approximations few-shot data domain at the data level, and thus the generating images tend to be similar to given few-shot data, which may lead to overfitting (we provide some results compared with common “approximating few-shot data” methods as follows). To address this issue, we attempt to rethink the way of utilizing few-shot data. In our implementation, we move beyond just visual similarity and focus on the task-level utilization of few-shot data to improve downstream classification performance. As validated in Fig.6 of Appendix D.6, the prompts we learned would generate images with significantly different visual features from the real few-shot data. While the classification accuracy of models trained with these synthetic data approaches that of models trained on the full dataset. Furthermore, we compare our method with previous data generation methods in Tables 1, 2, 7, 8, and 9, and the experimental results clearly show the superiority of our method in improving data generation for downstream classification performance.

To achieve this goal, meta-learning is naturally employed to provide task-level guidance for generating classification-aware images. We achieve an adaptive sample selection process that links generated data with downstream classification task. In meta-learning community, the generalization and efficiency are emphasized during the meta-test phase. In our case, generalization is demonstrated by the ability to scale from accessible few-shot datasets to full datasets, and efficiency is reflected in the fact that the learned prompts can be directly applied for generalization without any further tuning. This means when we finish meta-training process, our method can enjoy the same speed as previous hand-crafted prompt methods. Therefore, during the meta-test phase, our method tends to demonstrate high efficiency and generalization. Of course, we also acknowledge that our method is still relatively time-consuming during the meta-training phase, as discussed in Appendix A (limitations). However, the advantage of data generation for improving downstream classification task is promising to overcome the limitation of existing methods.

Besides, genetic algorithm is currently an alternative to gradient backpropagation, and has already proven the feasibility of the proposed approach. With further advancements in large-scale generative models and their using cost decreasing, our framework can be easily transformed into a white-box optimization model, which will significantly reduce runtime. Currently, we have to retrain the inner loop each time, but with white-box optimization, we can continue training with the weights from the previous loop. By that time, the time cost will no longer be an issue.

The comprehensiveness of propmt pool

Response 1.1:

In our formulation, we aim to identify an appropriate prompt set $\theta$ from the prompt pool $\Theta$ (a approximate construction of the prompt hypothesis space). Ideally, $\Theta$ contains a comprehensive set of potential prompts related to the concerned tasks. Previous methods often design prompts focusing on specific aspects. For example, hand-crafted prompts are tailored for different domains (please see Figure 1(a)), while model-generated prompts may contain rich content information (Please see Figure 1(b)). In contrast, our prompt pool construction aims to incorporate information from diverse perspectives. To enhance domain diversity, we manually curate domain-specific information. To enrich the content diversity of prompts, we utilize language enhancement techniques to generate a large number of diverse prompts. To improve task relevance, we adopt the CiP method to capture prompts closely related to the task images. Consequently, our approach significantly improves comprehensiveness of prompts compared to existing methods.

However, we admit that creating a comprehensive enough prompt pool is challenging because the optimal prompt cannot be guaranteed to exist within the constructed pool (as the prompt hypothesis space is theoretically continuous). Nonetheless, our experiments demonstrate that the constructed prompt pool sufficiently meets the needs of downstream classification tasks. From the given pool, the proposed algorithm adaptively learns the proper set of prompts, leading to performance improvements. Notably, although the constructed prompt pool is discrete, the learned optimal prompts exhibit stronger interpretability compared to those in a continuous hypothesis space. Detailed examples are provided in Fig.3 of Section 4.3 and Fig.10,11,12 of Appendix E.

About prompt selection process

Response 1.2:

• Though our method adaptively learns proper prompts from the data, the learned prompts may share a consistent pattern. Specifically, the mined prompts almost consists of two parts: hand-crafted prompts and model-generated prompts.

• Moreover, for specific categories, the ratio and content of model-generated prompts and hand-crafted prompts behaves differently. This demonstrates that our method can adaptively learn task-specific prompts for different datasets. We discuss these cases in detail in Fig 10,11 Appendix E.2.

• In Table 4,5,6 of Appendix D.2, we demonstrate the improvement of our method compared to random selection and full selection. The experimental results show that our prompt selection method is relatively more reliable, as it leads to significant performance improvements. Moreover, in Fig.6 of Appendix D.6, we explore the impact of using only domain prompts in the prompt pool. As it shown, even when the prompt pool is not comprehensive, our DeCap method could still achieve performance gains, which strongly supports the robustness of our method. All of these illustrate that our method is potentially useful for diverse few-shot learning tasks.

• While identifying prompts selection process across all datasets is challenging—since the satisfied prompts often vary depending on the specific dataset. However, the prompts selection/rejection process may emerge a general trend. On the one hand, all datasets tend to favor diverse domain prompts, such as "cartoon," "sketch," and "close-up," rather than being limited to the most common domains like "photo". On the other hand, prompts that are commonly rejected are shown in Fig.3 of Section 4.3. In summary, existing methods often face issues such as noisy class errors (e.g., both cat and dog appearing in a single image in a cat-dog classification task), model caption errors (e.g., identifying a monkey as a cat), and low-quality prompts (e.g., generating images that do not significantly aid classification). In contrast, our method rejects these prompts during the selection process.

• Since our method focuses on the impact of generated data on downstream tasks, we only show classification performance and have not provided extensive visual demonstrations of prompt selection process. As suggested by the reviewer, we provide an example in Fig.12 of Appendix E.2 showing the evolution process of prompt selection for the airplane class in the STL10 dataset during iterative optimization. As it shown, there exist some improper prompt in the early stage, while more proper prompts are gradually selected during the iteration process, showing considerable diversity and encompassing multiple methods. Finally, the prompt selection process converges, and our method could mined proper prompt set for the concerned tasks.

We believe above ‌interpretations of our prompt selection process could provide deeper understanding about our method.

About the assess of contributive prompts

Response 1.3:

To further illustrate whether or not the most contributive prompts are consistently selected, we conduct an ablation study: we randomly substitute half of the prompts DeCap learned with other prompts. The experimental results are presented as follows. It could be seen that when other prompts are substituted in learned prompts, the results show performance reduction. That is to say, the selected prompts indeed bring better classification performance on the concerned few-shot tasks.

Why don’t we use the "n-way k-shot" setting. Some other metrics for showing model’s robustness and generalization.

Response 1.4:

In the field of few-shot learning, with the emergence of foundational models like advancements in data generation techniques, the community has shifted toward directly utilizing the given few-shot datasets to improve the concerned tasks, without access to any other data as done in ``n-shot n-way" scenarios, e.g., [1-5]. We just follow previous benchmarks in our study. However, your suggestion is extremely helpful, as we indeed overlooked the computation of additional evaluation metrics. As suggested by the reviewer, we provide the results of various metrics for each dataset (precision, recall, F1-score), as shown in Table 7,8,9 of Appendix D.3 of the manuscript. The results demonstrate that our method performs well on these metrics, indicating that it not only achieves high accuracy but also excels in identifying positive samples and is more cautious when dealing with them. That is to say, our method could generate a high-quality dataset to train a robust and generalizable model for concerned few-shot tasks.

This capability is reflected not only in overall classification accuracy but also in precision, recall, and F1-score.

[1].DA-fusion: Effective Data Augmentation With Diffusion Models. ICLR,2024.

[2].SuS-X: Training-Free Name-Only Transfer of Vision-Language Models. ICCV,2023.

[3].Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learner. CVPR,2023.

[4].Is synthetic data from generative models ready for image recognition? ICLR,2023.

[5].Generating images of rare concepts using pre-trained diffusion models. AAAI,2023.