Democratizing Fine-grained Visual Recognition with Large Language Models

Q4: Is the proposed Pokemon dataset can really fit the fine-grained patterns? Or, the LLM and VLM learnt from real-world data, can really discriminate the fine-grained pokemon? These issues may make its contribution actually limited for FGVC community.

A4: Excellent question! The second part of your question precisely captures the motivation behind creating this dataset. Our intent was exactly to further examine the ability of our method, as well as the compared methods, to discover and differentiate virtual and wild FGVC categories effectively. Therefore, we proposed this wild and virtual Pokemon dataset, which is relatively novel and challenging for foundation models. We breakdown this question into two subsequent questions and address them below:

• Fine-grained Patterns: We mannuly selected 10 highly visually similar Pokemon categories to capture the fine-grained patterns. As shown in Appendix A.2, Fig. 9, since two Pokemon categories evolved from each other, their inter-class variance is low (e.g., the subtle differences between "Pidgey" and "Pidgeotto" are only at the crown and tail.) In addition, since Pokemons are cartoon characters, they by default have high inner-class variance (e.g., different facial expressions or body decorations). Additionally, the proposed Pokemon dataset presents an extra intriguing challenge due to the close resemblance of Pokemon characters to their real-world animal counterparts (e.g., "Squirtle" closely resembles a "Turtle" and "Pidgey" a "Bird"). This cross-domain similarity introduces an additional domain gap challenge in FGVC discovery. As illustrated in Fig.7, methods based on knowledge bases often incorrectly identify the real-world counterpart of the Pokemon (e.g., identified 'Turtle' instead of 'Squirtle'). In stark contrast, our FineR system successfully discovered 7/10 ground-truth Pokemon categories.

• the LLM and VLM learnt from real-world data, can really discriminate the fine-grained Pokemons?: Yes! As demonstrated in Fig.7, our FineR system successfully reasoned and discovered 7/10 ground-truth and subsequently achieved upper-bound clustering accuracy and high semantic similarity accuracy. This suggests that LLMs and VLMs trained with Internet-scale data are able to discriminate the fine-grained virtual concepts such as Pokemons. We found the results of this experiment fascinating!

We hope our answers have adequately addressed your concerns. We hope you might consider raising your score for our pioneering work in the vocabulary-free FGVC task. Should you have any additional questions, please don't hesitate to let use know. We would be more than happy to engage in further discussion with you. Thanks again for your insightful feedbacks strengthening our paper and stimulating our thinking.

Thank you for your positive, attentive and constructive feedbacks! We are encouraged that you, as an expert in vision and FGVC, recognized our method as "novel and interesting" and "experimentally better than existing state-of-the-art methods" which provides "a new way to do training-free FGVC based on LLM". We address your thoughts point by point below.

Q1: As there is already some good VLM solution to do zero-shot FGVC and achives nearly 90% accuracy, is it necessary to solely address FGVC by pure concepts from LLM? Besides, is it necessary to solely rely on LLM instead of VLM or visual representation for FGVC?

A1: Great question! Thanks for recoginizing the novelty and technique score of our work are statisfactory for ICLR. Following your suggestion, we have now also added a discussion section in Appendix K.4 of the revision to carefully compare our setting and method with respect to VLM zero-shot transfer and prompt enrichment/tuning methods. Here, we breakdown this question into three subsequent questions and address them below:

1. Setting perspective: In fact, as recoginized by Reviewer oRxX, we would like to emphasize and clarify that the novel task we propose and study in this work is vocabulary-free FGVC with only few unlabeled samples as natural observation. We compare our setting and existing VLM solutions apple-to-apple in the Table 1 and Table 2 (in Response 2/4). To be more specific, under vocabulary-free FGVC setting, there is no target class names (vocabulary) pre-defined by domain experts (e.g., ornithologist) for VLM-based zero-shot transfer methods or VLM prompt tuning methods to construct classifiers. This makes existing VLM solutions (e.g., CLIP, CoOp[2], CoCoOp[1]) that rely on the pre-defined vocabulary fail to function in this novel task, although VLM solutions such as [1] and [2] can achieves good performance for standard zero-shot FGVC task where the class names are already known. To be best of our knowledge, vocabulary-free FGVC task is underexplored in the existing literature. We believe this is an important yet realistic novel task for the community to explore. This newly proposed vocabulary-free FGVC task is inspired by the natural case:

"An bird lover gathered several unlabeled images from a webcam set up in the Amazon jungle. Confronted with the task of identifying the fine-grained bird species existing the the region from these few unlabeled observations, and without any pre-defined species names as prior knowledge from ornithologists, the bird lover encountered a significant challenge. In this context, our proposed FineR system is functional, enabling the bird lover to not only discover the bird species but also to classify them effectively in the incoming data stream from the webcam.“

2. Method perspective: Our proposed method is not solely rely on LLM. In fact, the proposed FineR system jointly integrates the visual information translation capability of the VQA model, the reasoning capability of the LLM, and the zero-shot discrimination capability of contrastive VLM to tackle the proposed vocabulary-free FGVR task. To further clarify, FineR primarily operates in 3 steps: (i) first uses BLIP-2 to translate visual information (super-category and attribute descriptions) from the few (3 per class) unlabeled images to natural language; (ii) then uses the visual cues in text to elicit the reasoning capability of ChatGPT for discovering fine-grained categories; (iii) lastly uses CLIP to classify/recognize fine-grained categories in the test data stream based on the discovered concetps.

3. The necessity of using LLMs: Insightful question! In our work, the role of LLM is a Reasoning Engine, just like a human expert would reason fine-grained categories based on the visual input. Furthermore, to further study "Is it necessary to use LLMs?", we constructed strong baseslines that adopt different solutions: (i) Learning-based Sinkhorn; (ii) Knowledge base based WordNet (use CLIP to retrive categories from WordNet); (iii) and BLIP-2 (VQA), as well as compared with SOTA methods (SCD and CASED) based on large corpus knowledge bases. As shown in the quantitative results in Tab. 1 and qualitative results in Fig. 6 of the paper, our FineR system not only performs better but also makes more plausible mistakes. This demonstrates the advantages of using LLMs to reason in vocabulary-free FGVC task.

[1] Conditional Prompt Learning for Vision-Language Models. CVPR 2022. [2] Learning to Prompt for Vision-Language Models. IJCV 2022.

Table 1: Training setting comparison between our novel vocabulary-free FGVR task and other related machine learning settings.

Table 2: Inference setting comparison between our novel vocabulary-free FGVR task and other related machine learning settings.

---

Q2: The idea of this paper is actually not very new now. The ICCV paper [3] implements similary idea. The difference and comparison should be made and clarified.

A2: We would like to clarify that the settings and methods of our paper and [3] are very different.

• Setting. The work in [3] assumes the fine-grained categories are known. On the contrary, we do not assume any knowledge about the class names.
• Implementation. In [3], a LLM is queried to simply enhance the descriptions of the already known concepts (or class names) to improve classification. Instead, we use VQA model to first identify super-category (e.g., "Bird") from an image and then acquire useful attribute names (no description) for distinguishing the fine-grained sub-categories. Then we use a VQA model to describe the attribute values in the image. The acquired attribute-description paris are then used to invoke LLM reasoning for discovering the fine-grained categories that are unknown. In a nutshell, [3] only uses LLM to describes known classes, while our method uses the interaction between LLM and VLM to reason unknown classes.

In fact, [3] falls into the same group of LLM-based VLM prompt enrichment methods as the ICLR 2023 work [4] that we discussed in the second paragraph of Section 4. We have now also included [3] in the second paragraph Line#3 of Section 4 in the revision. Thanks for the suggestion!

[3] Learning concise and descriptive attributes for visual recognition. ICCV 2023. [4] Visual Classification via Description from Large Language Models. ICLR 2023.

Q3: So many details of the proposed dataset are missing. The specific comments have been listed in the weakness.

A3: In fact, we have included the exact number of samples, train/test splits, and statistics of the proposed Pokemon dataset in Appendix A.2. Furthermore, due to the small scale of the proposed Pokemon dataset, which consists of 10 categories with 3 images each for discovery (D_train) and 10 images per category for testing (D_test), we direcly present the entire dataset and its splits in Appendix A.2, Fig. 9. It is important to note that the Pokemon dataset was designed to delve deeper into the capabilities of vocabulary-free FGVC in the wild. Despite its small scale, it poses significant challenges for many strong methods we compared. We will elaborate on these challenges in the next question.

Q2: The effectiveness of the proposed method should be further verified. The recognition accuracy on the CUB dataset is relatively low. Besides, existing generalized zero-shot learning methods such as [a] should be compared and more corresponding analyses should be added. More corresponding experiments and analyses should be added.

A2: We first would like to clarify that it is not fair to directly compare between our approach with generalized zero-shot learning methods. It's crucial to highlight that our approach is both vocabulary-free and training-free, which fundamentally differs from GZSL methods. Directly comparing our method's recognition accuracy with that of GZSL approaches, which are supervised on numerous base seen classes with class name and attribute annotation and jointly evaluated on both seen and unseen classes, would be inappropriate. For instance, GZSL methods' accuracy on the CUB-200 dataset is jointly evaluated on 150 seen and 50 unseen classes. In contrast, all 200 classes of CUB are unseen in our evaluation. Moreover, as outlined in the setting comparison in Table 1 and Table 2 above, all the prerequisites required by GZSL during training and inference, including class names, annotated base class data, and ground-truth attribute-description pairs are not available in our setting. In our vocabulary-free FGVR task, both GZSL and standard VLM zero-shot transfer methods, which depend on predefined vocabulary, fail to function due to the absence of class names. A thorough review of [a] leads us to conclude that a comparison with GZSL approaches is not feasible in our setting due to the absence of class names. In fact, standard VLM zero-shot transfer method (CLIP) was considered and present as the upper bound in our experiments. Notably, we have achieved a significant performance improvement on the CUB-200 dataset compared to the methods we evaluated against. Nevertheless, we appreciate your suggestion and have added a discussion section in Appendix K.4 of the revised manuscript to specifically clarify the differences between our vocabulary-free FGVR setting and GZSL, along with comparisons to other related settings.

Q3: There lacks a complete recognition example for explaining the whole procedure, which should be added for better understanding.

A3: Thank you for your valuable suggestions! Indeed, we have demonstrated the entire recognition process using an unlabeled "Italian Greyhound" image as an example in Appendix B.2, from Fig.12 to Fig.15. This includes the visual input, the identified super-category ('Dog'), attribute names obtained from the LLM, attribute descriptions extracted by the VQA model, the complete LLM reasoning prompt, and the LLM's output results. In response to your advice, we have added a new figure in Appendix B.2 Fig.11 of the revised manuscript to more clearly illustrate the full pipeline with actual inputs and outputs.

We hope our responses have sufficiently addressed your concerns. With these clarifications and changes in the revision based on your suggestion, we kindly hope you might be convinced to raise your score for our pioneering work in the vocabulary-free FGVC task. Your attentive feedback has been helpful in enhancing our paper in the revision, and for that, we are truly grateful.

[a] Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning. CVPR 2023.

Q1.2: The difference from existing generalized zero-shot learning methods using visual attributes for recognition should be explained.

A1.2: In Table 1 and Table 2 below, we have compared our vocabulary-free FGVR setting with Generalized Zero-Shot Learning (GZSL) at training and inference time, respectively. Furthermore, we have now also added a discussion section in Appendix K.4 of the revision to carefully compare our setting and method with respect to GZSL.

Table 1: Training setting comparison between our novel vocabulary-free FGVR task and other related machine learning settings.

Table 2: Inference setting comparison between our novel vocabulary-free FGVR task and other related machine learning settings.

Generalized Zero-Shot Learning (GZSL) methods, like [a], rely on pre-existing ground-truth attribute-description pairs for both seen and unseen classes. GZSL methods use these attributes and seen (base) classes to train a vision model that can generalize to unseen classes based on shared attributes. Our method diverges significantly from GZSL in several ways:

• Attribute and Description Availability: As shown in Table 1 and Table 2 above, in our approach, neither ground-truth attribute names nor descriptions are available; thus, we do not use any predefined attribute information as prior knowledge, which requires expert annotations.
• Attribute and Description Acquisition: Initially, our method identifies super-category names (e.g., "Bird") using a VQA model, followed by querying an LLM to generate relevant attribute names for further distinguishing fine-grained sub-categories. The attribute descriptions are then extracted from a few unlabeled images using the VQA model.
• Attribute and Description Utilization: The generated attribute-description pairs serve as high-level semantic cues in natural language, invoking LLM reasoning in identifying fine-grained categories. This approach grants our method semantic awareness, enabling it to provide useful semantic information even in cases of incorrect predictions, as demonstrated in Fig.6.

---

Thank you for your feedback. We are particularly encouraged with your comment about our method: "is a promising way to alleviate the high need for expert annotations in fine-grained recognition". We address your feedback point by point below.

Q1.1: The novelty should be clarified.

A1.1: Thanks for the suggestions. Our work, we believe, is novel in several key aspects:

• Innovative Application/Task: To the best of our knowledge, our work is the first to tackle the novel task of vocabulary-free FGVR with only a few unlabeled samples as natural observation. To tackle this novel task we harness the reasoning capabilities of LLMs and leverage the Internet-scale expert knowledge encoded in their weights to first discover the fine-grained categories that are present in a dataset, which is otherwise not know a priori. While CaSED (Conti et al., 2023) previously addressed vocabulary-free classification by retrieving concepts from a knowledge-base, our method is tailored specifically for FGVR by leveraging the knowledge in LLM. This also resonates with the comments from Reviewer 3hkC and Reviewer WYuk who commented that utilization of LLM for FGVR as "a new way" and "a promising way", respectively.

• Methodological Advancements: We introduced FineR, a novel system that integrates the visual information translation capability of VQA models, the reasoning capability of LLMs, and the zero-shot discrimination capability of contrastive VLMs to address the vocabulary-free FGVR task. This approach is recognized by Reviewer WYuk as "novel" and "interesting". Furthermore, as shown in the preliminary experiments in Fig. 2, we observed that other LLM-based methods requiring training alignments (such as BLIP-2 learning Q-former, LLaVA, and MiniGPT-4 learning a projector) struggle to discover fine-grained categories effectively due to the limitations of alignment data. These training-based methods cannot fully utilize the reasoning capabilities of LLMs. In contrast, our FineR system translates visual information from unlabeled images into natural language in order to fully harness the reasoning power of LLMs for fine-grained concepts. Thus, we showed that visual and textual concepts can be aligned without the need of expensive training.

• Resource Efficiency and Accessibility: Our approach is both training-free and vocabulary-free, significantly reducing the computational and data resources typically required for vision model and LLM alignments. This enhances the accessibility and feasibility of our method for a broader range of users, potentially democratizing the application of FGVR across various sub-fields.

• Scalability and Generalization: Being training-free enhances the scalability and generalizability of our methods. As demonstrated in Fig. 7 (Section 3.2, page 8) and Fig. 24 (Appendix K.3, page 31), our method can effectively discover wild and novel fine-grained Pokémon categories, outperforming other methods that fail in these wild scenario.

We have summarized our novelty and contributions at the end of the Introduction section (Sec. 1) in the revision.

---

Dear Reviewer 3hkC,

We genuinely appreciate the valuable feedback you have provided.

As the deadline for the Author-Reviewer Discussion Phase approaches, we are reaching out to inquire if you might have any remaining questions or concerns regarding our submission.

We wholeheartedly thank you for your dedication and effort in evaluating our submission. Please do not hesitate to let us know if you need any clarification or have additional suggestions. Your input is highly valued.

Best regards,

Authors of Submission #2720

---

Q3: What is your view of NOVELTY of a work using LLMs without training anything?

A3: Thank you for your insightful question! Following your suggestion, we have outlined the novel aspects of our work at the end of the Introduction section (Sec. 1) in the revision. Our work, we believe, is novel in several key aspects:

• Innovative Application/Task: To the best of our knowledge, our work is the first to tackle the novel task of vocabulary-free FGVR with only a few unlabeled samples as natural observation. To tackle this novel task we harness the reasoning capabilities of LLMs and leverage the Internet-scale expert knowledge encoded in their weights to first discover the fine-grained categories that are present in a dataset, which is otherwise not know a priori. While CaSED (Conti et al., 2023) previously addressed vocabulary-free classification by retrieving concepts from a knowledge-base, our method is tailored specifically for FGVR by leveraging the knowledge in LLM. This also resonates with the comments from Reviewer 3hkC and Reviewer WYuk who commented that utilization of LLM for FGVR as "a new way" and "a promising way", respectively.

• Methodological Advancements: We introduced FineR, a novel system that integrates the visual information translation capability of VQA models, the reasoning capability of LLMs, and the zero-shot discrimination capability of contrastive VLMs to address the vocabulary-free FGVR task. This approach is recognized by Reviewer WYuk as "novel" and "interesting". Furthermore, as shown in the preliminary experiments in Fig. 2, we observed that other LLM-based methods requiring training alignments (such as BLIP-2 learning Q-former, LLaVA, and MiniGPT-4 learning a projector) struggle to discover fine-grained categories effectively due to the limitations of alignment data. These training-based methods cannot fully utilize the reasoning capabilities of LLMs. In contrast, our FineR system translates visual information from unlabeled images into natural language in order to fully harness the reasoning power of LLMs for fine-grained concepts. Thus, we showed that visual and textual concepts can be aligned without the need of expensive training.

• Resource Efficiency and Accessibility: Our approach is both training-free and vocabulary-free, significantly reducing the computational and data resources typically required for vision model and LLM alignments. This enhances the accessibility and feasibility of our method for a broader range of users, potentially democratizing the application of FGVR across various sub-fields.

• Scalability and Generalization: Being training-free enhances the scalability and generalizability of our methods. As demonstrated in Fig. 7 (Section 3.2, page 8) and Fig. 24 (Appendix K.3, page 31), our method can effectively discover wild and novel fine-grained Pokémon categories, outperforming other methods that fail in these wild scenario.

We have summarized our novelty and contributions at the end of the Introduction section (Sec. 1) in the revision.

Thank you once again for your insightful and helpful advice. We have integrated your suggestions into the revised manuscript, which we believe has led to an improved submission. With the specific changes made in response to your suggestions, such as "Adding citations of some key related works" and "Providing more examples or failure cases for Fig. 2", we hope these improvements in the revision will encourage you to raise your confidence.

We sincerely thank you for your positive feedback and helpful suggestions! We are glad that you recognize our method, which utilizes LLM for tackling vocabulary-free FGVR tasks, both "novel" and demonstrating "good performance", as well as representing a "well utilization of LLM". Furthermore, we appreciate that you found that our proposed method has potential to work in real-world applications. We address your questions below.

Q1: Add the citation of some highly related missing works.

A1: Thanks you for your helpful advice. In the revision, we have now included these two highly related papers in the second paragraph of Related Work section (Sec.4) of the main paper. Furthermore, we have now also added a discussion section in Appendix K.4 of the revision to carefully compare our setting and method with respect to related works. In response to your question, we discuss the main differences between our work and TransHP and V2L below:

• TransHP: We note that the very recent NeurIPS'23 FGVR work TransHP (arXived on 2023-October-23, which is after ICLR'24 deadline) is related to our study. Our method differs from TransHP in two ways: (i) We study the task of vocabulary-free FGVR, where neither coarse-grained class names nor the target fine-grained class names are known, unlike TransHP where coarse grained classes are known; (ii) Our method is also training-free. The only resources our method has access are few unlabeled images (3 per-class or [1,10] per class) as an observation for discovering the fine-grained class names. Under this realistic vocabulary-free FGVR setting TransHP cannot operate due to the lack of class names.

• V2L: V2L learns and complementarily ensembles vision (ResNeSt, ResNeXt, CotNet, HS-ResNet, NAT, ViT, and ViT) and vision-language models (BLIP, ALBEF, XVLM, METER and SLIP) with the supervision from coarse-grained labels and textual description. Similar to TransHP, the lack of class names and supervisory signals makes V2L inoperable in our vocabulary-free scenario.

Q2: Could you show more examples of Fig. 2? Or any failure cases of Fig. 2?

A2: In the Appendix Fig. 20 we have provided more examples of Fig. 2. In Fig. 20 we have compared our method with several baselines. We observe that our FineR predicts the correct class more number of times than the baselines.

In addition, following your suggestion we have added a thorough qualitative failure case analysis in Appendix I, Fig. 21. Specifically, Fig. 21 qualitatively presents the failure cases of our method across each dataset (including five fine-grained datasets and the proposed Pokémon dataset) and compares these with vision-language models (BLIP-2, LENS, LLaVA, MiniGPT-4). We observe that most of the baselines fail to correctly identify the finegrained categories. On the contrary our FineR, albeit not being able to predict the exact fine-grained category, it can still capture some fine-grained attributes. For eg., it predicts the dog breed Brabancon Griffon as Brussels Griffon, demonstrating that it can capture the characteristics of a Griffon dog. This suggests that our FineR makes less severe mistakes than the baselines.

Intriguingly, on the novel Pokémon dataset, almost all compared methods predict only the real-world counterparts of the Pokémon characters (e.g., 'Squirtle' as 'Turtle' and 'Pidgey' as 'Bird'), highlighting a domain gap between real-world and virtual categories. In contrast, although not completely accurate, our method offers predictions that are closer to the ground truth (evolutionized Pokemon characters, 'Pidgey' as 'Pidgeotto' and 'Squirtle' as 'Wartortle') for the two Pokémon objects. This failure case analysis further confirms that our system effectively captures fine-grained visual details from images and leverages them for reasoning. It demonstrates that FineR not only generates more precise and fine-grained predictions when correct but also displays high semantic awareness when it errs.