Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models

We thank reviewer VD1i for their remarks and their thoughtful view. We answer the questions below:

Q1: Comparison with FineR

We compare the clustering accuracy (cAcc) used in FineR with the classification accuracy used in Finedefics. With the former denoting FineR and the latter denoting Finedefics, the results are: a) Dogs-120 (48.10 vs 72.86), b) Bird-200 (51.10 vs 57.61), c) Flowers102 (63.80 vs 89.88), d) O.-Pet (72.90 vs 92.18), e) S.-Cars (49.20 vs 84.67). This shows the superiority of building an attribute-aware model compared to a multi-agent system using attributes in a zero-shot manner.

Q2: Effects of attribute description construction

We evaluate the crucial rule of our proposed construction method acquiring per-sample attribute descriptions. To this end, we compare with an upper-bound and a tag-based baseline trained solely on Bird-200 dataset. Results (Upper-bound: 52.52, Tag-based baseline: 50.48, Ours: 51.12) show that Finedefics surpasses the tag-based baseline, demonstrating the superiority of building detailed per-sample attribute descriptions. Moreover, we can observe that the accuracy of Finedefics is close to the upper-bound, showing the potential of using LLMs and VQA models to obtain large-scale attribute data for alignment without human annotations.

Q3: Performance on common object recognition

We investigate how Finedefics performs on common object recognition after pretraining and instruction tuning on FGVR datasets. The results show that training solely on FGVR tasks has a minor impact on common object recognition accuracy: a) ImageNet-adversarial (79.84 vs 75.96), b) ImageNet-rendition (93.23 vs 92.43), and c) ImageNet-sketch (68.21 vs 66.77). Therefore, training should be performed on a combined dataset comprising both coarse-grained and fine-grained recognition tasks. Developing fine-tuning methods that prevent such catastrophic forgetting to enhance FGVR capability without impacting common object recognition is a promising future research direction.

Q4: Discriminability comparison between different datasets

For each dataset, we provide a quantified analysis of intra-class variance for visual objects, as well as inter-class distance for category names, respectively. O.-Pet and Dogs-120 have the largest intra-class distance for visual objects (0.28 and 0.27), meaning that attribute values differ significantly for the same subordinate-level category of pets. Most importantly, category names have lower discriminability than visual objects in the representation space despite the super category. For example, though Flowers102 has large inter-class distance (0.41) and small intra-class variance for visual objects (0.07), the inter-class distance for category names (0.90) is not significantly different from other datasets.

Q5: Qualitative comparison and error analysis

We provide qualitative comparison and error analysis examples in Figure 6 and 7 in Appendix, respectively. Finedefics successfully captures the nuance of the object features, setting them apart from visually similar subordinate categories. This confirms that Finedefics effectively captures fine-grained visual details from images, connects them with category names in the representation space, and then generates precise, fine-grained predictions.

Q6: MLLMs for FGVR

Though supervising a strong pre-trained discriminative model (i.e., VLMs) on FGVR datasets can achieve high performance, there exist three advantages of using generative models (i.e., MLLMs):

(1) VLMs can only be used in a closed-world setting where the label set is known, while MLLMs can be used in an open-world setting where the label set is not provided and a closed-world setting where classes are concatenated in the prompt [a].

(2) Due to the characteristic of discriminative models, VLMs encounter severe catastrophic forgetting in class-incremental learning scenarios. Nevertheless, MLLMs can harness the wealth of semantic correspondences between texts and images to alleviate catastrophic forgetting, while there’s no requirement to expand the classifier with each new category, unlike in the case of VLMs [b].

(3) VLMs can only be supervised to perform FGVR task. However, recognizing an object is a prerequisite for answering complex questions about it. We attempt to further transfer the enhanced FGVR performance of MLLMs into its general capabilities, such as object-centric, knowledge-intensive question answering, where the question can only be accurately answered if the class of the object is correctly identified [a].

[a] Zhang et. al., Why are Visually-Grounded Language Models Bad at Image Classification? Arxiv 2024.

[b] Cao et. al., Generative Multi-modal Models are Good Class-Incremental Learners, CVPR 2024.

Dear Authors,

Thank you for the detailed, precise rebuttal. I have checked the author rebuttal and the revised paper. All my questions are well-addressed. I believe, with new, deeper investigation requested by me and other reviewers, the current revised paper has been significantly strengthend. I have also checked the Questions & Answers from my peer reviewers. The Authors did a really good job in the rebuttal. Kuods to the authors. The current revised paper looks really good.

Therefore, I decide to insist my initial 8: accept rating, while keep 5 confidence. This is a good paper with novel method, comprehensive experiments and ablations, where both the method and experiments present important contribution to FGVR field.

Good luck!

Best regards,

Reviewer VD1i

We thank reviewer 1pZP for their remarks and their thoughtful view. We answer the questions below:

Q1: Attribute augmented alignment on other MLLMs

To confirm the general applicability of Finedefics, we conduct attribute augmented alignment on another typical MLLM: LLaVA 1.5. After employing our proposed method, LLaVA 1.5 gains an accuracy improvement by 13.97% on average (43.24 vs 57.21), demonstrating the effectiveness and generalizability. We will include results based on more MLLMs in the final revision.

Q2: Effects of attribute types

We analyze the effects of specific attribute types in FGVR tasks. Specifically, we selectively remove typical attribute types from [color, shape, texture, size] to evaluate the contribution to performance improvement. All four types of attributes play a crucial role in distinguishing subordinate-level categories, but the contribution varies with the dataset (see Table 5 in Appendix). For example, color and texture are more critical for specific datasets, like flowers and birds.

Q3: Effects of hard negatives

We compare using hard negatives and simple negatives for contrastive learning. Specifically, we replace hard negatives with randomly sampled simple negatives, meaning that the negatives used for contrastive learning are less visually similar to positives and easier to distinguish from them. After applying contrastive learning with simple negatives, the improvement is limited (65.95 vs 74.26). With the utilization of hard negatives, the modality gap decreases further, and the model harvests a significant accuracy improvement (74.26 vs 76.84).

Q4: Effects of two-stage training stages

We analyze the effects of two-stage training by evaluating Finedefics by selectively removing specific training processes within each stage. Pretraining solely fails to follow the task instruction (0.0), while instruction tuning solely has a limited performance gain (65.95 vs 76.13). Instead, pretraining and instruction tuning are complementary to further boost the accuracy (76.13 vs 76.84), confirming the effectiveness of our two-stage training paradigm.

Q5: Effects of description quality

We first design an empirical study to evaluate the description quality, i.e, how reliable the attribute descriptions we built. Similar to the probing experiments in Section 2.2, we test the representation discriminability of our constructed attribute descriptions on the training set of Oxford-IIIT Pet-37 with a splitting ratio of 1:1. The accuracy is 68.27%, showing that the attribute descriptions can be well distinguished from each other though there exist subtle visual differences that are difficult to describe in words. Furthermore, to evaluate Finedefics’s sensitivity to the description quality, we use three different quality levels of descriptions: (1) complete descriptions, (2) noisy descriptions (i.e., replacing some attribute descriptions with incorrect ones by prompting ChatGPT), and (3) no descriptions (i.e., object-category alignment w/o attribute). Results (complete: 76.84, noisy: 75.62, none: 72.72) demonstrate Finedefics’s robustness to the description noise.

Q6: More evaluation metrics

(1) Confusion matrix analysis: Besides the classification accuracy, we conduct confusion matrix analysis on Oxford-IIIT Pet-37 to identify where the model struggles across categories (see Figure 5 in Appendix). Finedefics can recognize most of the categories correctly, while struggles in identifying a small portion of categories, such as Wheaten Terrior (52%), Staffordshire Bull Terrier (58%), Ragdoll (71%), American Bulldog (74%), and Birman (78%).

(2) Alignment quality evaluation: We provide a quantitative analysis of the object-category alignment quality for Idefics2 and Finedefics. The object-category alignment quality is calculated as the mean cosine similarity between embeddings of visual objects and their corresponding category names of each class. We observe that Finedefics significantly increases the object-category alignment quality (0.13 vs 0.27), which further demonstrates the effectiveness in boosting alignment.

Q7: Effects of levels of detailed descriptions

To analyze the impact of description length and detail on alignment effectiveness, we compare three different levels of detailed descriptions: (1) long descriptions, (2) short descriptions (i.e., generating short descriptions of input images with BLIP-2, and (3) no descriptions (i.e., object-category alignment w/o attribute). Results (long: 76.84, short: 76.11, none: 72.72) demonstrate that rich and informative category information expression plays a crucial role in boosting the alignment between visual objects and category names.

Q8: Generalization

In fact, our method does not rely on the specific architecture of Idefics2, since it conducts attribute augmented alignment on the output embeddings of LLM and the architectures of existing MLLMs are built upon LLMs.

We thank reviewer qkG7 for their remarks and their thoughtful view. We answer the questions below:

Q1: Novelty compared to prior works

Comparing [1] and our method, we both use hard negatives to enhance the performance of contrastive learning. Nevertheless, contrastive learning solely on image-caption pairs with hard negatives [1] fails to fully align visual objects and category names (65.95 vs 72.72), as shown in Table 3(b) in the manuscript. In contrast, we propose an attribute augmented contrastive learning paradigm to better alleviate the misalignment between visual objects and category names, further boosting the accuracy by a large margin (72.72 vs 76.84).

Comparing [2] and our method, we build upon the proposed Idefics2 [2] but is specifically tailored to enhance its FGVR capability. Despite the dedicated design of multi-stage pretraining and instruction fine-tuning on large-scale data [2], alignment in the distribution of visual objects and category names is still unattainable, leading to underperformance in FGVR tasks (65.95 on average). In contrast, we propose a FGVR-aware training paradigm comprising attribute augmented pretraining and classification-centered instruction tuning, improving FGVR accuracy effectively (76.84 on average).

Q2: Experimental details in Section 2.1

We feed the image token sequence into the LLM without appending any prompts. For MLLM (i.e., Idefics2), linear probing is performed after the LLM. For visual encoder (i.e., SigLiP), linear probing is performed before the connector.

Q3: Detailed explanation about Figure 2 (c+f)

We randomly sample 120 object-category pairs from Stanford Dog-120. For Idefics2, we obtain the last token embedding of the object image $\hat{o} _ {m}^{i}$ and the category name $\hat{c} _ {n}^{i}$ after the LLM. For SigLIP, we obtain projected ${[CLS]}$ embedding from the last layer of visual encoder $\hat{o} _ {\text{CLS}}^{i}$ and text encoder $\hat{c} _ {\text{CLS}}^{i}$. We normalize $\hat{o} _ {m}^{i}$ , $\hat{c} _ {m}^{i}$ for t-SNE visualization in the same representation space, shown in Figure 2(f). Similarly, we normalize $\hat{o} _ {\text{CLS}}^{i}$, $\hat{c} _ {\text{CLS}}^{i}$ for t-SNE visualization in the same representation space, shown in Figure 2(c).