GIST: Generating Image-Specific Text for Fine-grained Object Representations

We thank the reviewer for their careful review and feedback. We appreciate that the reviewer recognizes the improvement of GIST over comparable methods. We address the weaknesses and questions raised as follows:

Weaknesses:

1. We don’t require prompt tuning or in-depth domain knowledge for the LLM prompts. Our prompts are listed in Appendix section A.3. Our prompting approach is identical for all-shot settings. We add material to the main text about prompting to clarify this.

2. We did provide experimental results for VL finetuning using image captioning models and also compare against two CVPR 2023 VL finetuning papers. Did the reviewer have a specific method in mind?

3. Could the reviewer please expand on what is meant by “full images”? GIST assumes access to an image dataset with class labels but has no restrictions on the number of images per class. The full pipeline can be applied to domains that have very few images.

4. Text summarization is not required, but does improve results (Section 4.5 of submission). Our method using the longer, original text outperforms the baselines. However, we achieve even better results when we summarize the captions. The intuition behind this is that the summarized captions remove extraneous words and are more similar to the format of the original CLIP training data. We add this discussion in the revision and thank the reviewer for the suggestion.

5. We show that even WITHOUT manual cleaning, our method can outperform the baselines. As LLMs improve, the manual cleaning step will become far less useful.

Questions:

1. This is an excellent question. We provide experimental results for using BLIP-2 and GIT to generate captions in Section 4.5: Captioning Model Comparison. The full- shot average accuracy and standard deviation is 74.17 (2.72) for GIT, 73.83 (2.75) for BLIP-2, and 75.77 (2.67) for GIST (ours). The 5-shot average accuracy and standard deviation is 42.96 (2.99) for GIT, 42.66 (3.03) for BLIP-2, and 46.96 (3.13) for GIST (ours). We show additional comparison results in the Appendix in Figure 7, Table 5, and Sections A.5 and A.6. Our method outperforms all of these other approaches. Would the reviewer suggest that we move some of the results from the Appendix into the main paper? If so, is there material that the reviewer thinks we could move from the main body into the Appendix?

2. The same caption can be matched to multiple images since the same fine-grained features can appear in many images. The average number of images matched per captions varies per dataset. For the full Fitzpatrick40 dataset, for example, each caption is matched to an average of 2.8 images. For all five-shot datasets, each matched caption is matched to an average of 1.3 images. Could the reviewer please additionally clarify which other statistics they would like?

We thank the reviewer for their careful review and feedback. We address the individual weaknesses raised below:

Weaknesses:

1, 4. We agree that prompting LLMs for visual classification is not novel. However, we believe that our approach for doing this is novel, and we show that it outperforms previous methods. Our way of matching images to LLM-generated captions is a novel part of our method that has not been used in previous methods. We show that our method outperforms three CVPR 2023 methods that use text for visual classification. We cited Maniparambil et. al. 2023 as concurrent work since the workshop proceedings were released shortly before the ICLR 2023 deadline. We will discuss it in more detail in the revision. Maniparambil aims to improve accuracy for zero-shot CLIP. In contrast, our generated image-text pairs allow us to fine tune CLIP to achieve higher overall classification accuracy.

2. We provide our text prompts in Appendix section A.3 and show that they do not require in-depth domain knowledge. We should have included some of this material in the main text. We do not require prompt tuning and only use one prompt template per domain. The user-provided prompt does not require a significant human load and the prompt can be used across many classes and datasets within the domain. We show in Figure 4 and Figure 10 that our prompting method results in more detailed and relevant text descriptions than the prompting methods used in LaBo and CaFo.

3. Our improvements over FLYP are statistically significant, using a paired t-test. They are also meaningful. For the majority of full-shot and five-shot evaluation settings in Table 1, the absolute accuracy improvement of GIST over FLYP is greater than the absolute improvement of FLYP over off-the-shelf CLIP linear probing (CLIP LP).

We thank the reviewer for their careful review and feedback. We appreciate that the reviewer recognized that GIST achieves good results across many evaluation settings and appreciate the reviewer’s suggestions to improve the paper. We address the individual weaknesses and questions raised below:

Weaknesses:

1. We agree that we are not the first to explore using LLMs to augment CLIP training. However, our novel approach shows significant improvement over previous methods, including three appearing in CVPR 2023. The image-text matching aspect of our method, leveraging CLIP encodings, is novel compared to previous approaches. We agree that [1] and [2] are relevant, and cite them as concurrent work. In our revision, we will include additional discussions on how these two works relate to GIST and make sure that the differences are made clear. Briefly, GIST, unlike [1], does not require text captions in the dataset; rather we only require an image dataset with class names, and we generate our own text for fine-grained image datasets. It would be interesting in future work to finetune our models using the pre-trained CLIP produced by [1]. [2] aims to improve accuracy for zero-shot CLIP. In contrast, our generated image-text pairs allow us to fine tune CLIP to achieve higher overall classification accuracy.
2. We include an ablation on the number of matched captions in Table 3 of the main paper. It suggests that matching more captions per image does not necessarily improve performance. The optimal number of captions per image varies per dataset.
3. We compare against CLIP LP and CLIP zero-shot and show that fine-tuning CLIP performs significantly better. We also show ablations for fine tuning CLIP on GIST captions versus fine tuning CLIP on captions from GIT and BLIP-2 or on standard text prompts (e.g.,the FLYP method). We show that fine tuning on our GIST captions outperforms fine tuning CLIP on all of these alternative text approaches. We demonstrate that fine-tuning with the full dataset of images in comparison to few-shot numbers of images (Table 1,6) improves classification performance.

Questions:

1. The text inputs to GIST are text prompts using class names. We show all of our text prompts in Appendix section A.3. We add material to the main text about prompting to clarify this.
2. In Section 4.5, we provide experimental results on fine tuning CLIP on short text descriptions versus fine tuning CLIP on the original descriptions. Both methods outperform the baselines, but fine tuning CLIP on short text descriptions does best. The text descriptions are used to finetune CLIP to improve the embedding space. No text descriptions are used to finetune the classifier.