RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition

Dear Reviewer nzRY,

We sincerely thank you for your thorough feedback: the paper is well-written, easy to understand with a clear methodology and achieves strong results on the proposed benchmark. We have incorporated all your feedback in our revised manuscript. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1: Could you clarify why vanilla LLaVa results are not included in Table 1?

A1: LLaVA is not designed for zero-shot fine-grained visual recognition, which requires classifying images into many specific categories. Directly using LLaVA for such tasks would yield significantly inferior performance compared to previous baselines (e.g., CaSED, FineR). Therefore, we excluded LLaVA from Table 1 to maintain a fair comparison.

Q2: Additionally, for zero-shot image classification, is there a comparison to vanilla CLIP using only category names?

A2：Thanks for your suggestion. We've included the zero-shot performance of vanilla CLIP using only category names below. While Table 2 primarily focuses on few-shot image classification, where a few sample images are provided, zero-shot classification relies solely on category names. Comparing the results, We found that on some fine-grained datasets (e.g., Eurosat, DTD, UCF101, Flowers102), our method significantly outperforms CLIP+category. This highlights the effectiveness of our approach, particularly when limited training data is available.

Dear Reviewer nzRY,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer FduB,

We sincerely thank you for your thorough feedback: the paper is well-written and the method's details are well-explained. We have incorporated all your feedback in our revised manuscript. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1: Although the paper’s title is about augmenting MLLM, it is mostly just a simple application of MLLM. The SFT and in-context learning techniques used in the thesis only improve the MLLM's ability to maintain output formats, but do not actually enhance its classification ability. This is why the SFT effects on different datasets in Table 5 are not significantly different.

A1: While our paper explores the synergy between Vision-Language Models (VLMs) and Multi-Modal Language Models (MLLMs) for visual recognition, our primary contribution lies in using MLLMs for re-ranking retrieved visual examples. We do not rely on SFT to enhance classification performance. The SFT and in-context learning techniques we employ are specifically designed to enhance MLLM's ability to maintain consistent output formats during the re-ranking process. This ensures that the final predictions align with the desired task and dataset.

As demonstrated in Table 5, in-context learning, which does not involve SFT, proves effective in improving re-ranking performance. This highlights the power of MLLMs in adapting to new tasks without extensive fine-tuning. While SFT can further refine the model's performance, our results indicate that in-context learning is a strong baseline that can be leveraged for a variety of visual recognition tasks.

Q2: The motivation was the improvement in fine-grained categories and vast vocabulary classification for CLIP and MLLM, but since no substantial optimization was done for CLIP and MLLM, and their classification abilities for this scenario are relatively weak, so the improvement of their combination is not significant, as shown in Table 1 against FineR and Table 5. The reason why the improvement was significant in Table 4 was because the object categories in LVIS were 1,000, which was not enough, and could not effectively prove the motivation.

A2: We combined the strengths of CLIP and MLLMs in handling large vocabularies and fine-grained categories. Experiments were conducted in three directions: fine-grained visual recognition tasks, few-shot image classification tasks, and zero-shot object detection tasks. The results demonstrate that our method achieves significant performance improvements across all tasks. In Table 3 and Table 4, we present the results of the LVIS and V3Det object detection datasets. The former contains over 1,000 categories, while the latter includes over 13,000 categories. We believe that such a vast vocabulary is sufficient to substantiate our claims.

Q3: The performance increase in Table 2 is all compared with CLIP+KNN, which makes people feel that RAR improves CLIP rather than MLLM, and the performance increase here is suggested to be worse than the sub-optimal result.

A3: We did not merely compare against the CLIP baseline. In Table 2, CLIP+KNN and LLaVA 1.5+finetuning represent the baselines for CLIP and LLaVA, respectively. The LLaVA model performs significantly worse than CLIP when dealing with classification datasets containing a large number of categories. However, RAR achieves substantial performance improvements over both baselines. This reflects our motivation to combine the strengths of CLIP and MLLMs, enabling better results in visual perception tasks by using the advantages of both models.

Q4: When the K of K-NN is determined, the upper limit of the method's capability is determined by CLIP (whether the prediction result is in topk), and MLLM only helps the method approach this upper limit. In some cases, even the performance of CLIP will limit the play of MLLM (if the correct category is not in topk). Also, poor MLLM performance can also undermine CLIP's capabilities (see Table 2 Flower102 dataset). Therefore, I want to see the ablation experiments under different k settings. In-context learning can be used if SFT is too time-consuming.

A4: You've raised an important point. The choice of K in K-NN significantly impacts RAR's performance, as it determines the upper limit of the method's potential.

To investigate this further, we've conducted ablation studies in Table 13 with varying K values (K = 3, 4, 5, 6, 7) and compared the results to the following baselines:

• Upper-bound: The CLIP top-7 accuracy, represents the ideal scenario where all retrieved results are ranked correctly.

• RAR with varying K: The top-a accuracy of RAR for different K values.

• CLIP (VLM only): The top-1 accuracy of CLIP, showcases the performance of the VLM alone.

• LLaVA (MLLM only): The top-1 accuracy of LLaVA, demonstrates the performance of the MLLM alone.

Our analysis reveals the following insights:

(1) Small K values (K < 5): When K is small, the correct category may not be included in the top-K retrieved results by CLIP. In such cases, RAR's performance gain over CLIP is limited, as MLLM's re-ranking capabilities are constrained by the initial retrieval.

(2) Large K values: As K increases, the challenge for MLLM becomes more significant. Re-ranking a large number of candidates accurately requires strong language understanding and reasoning abilities, which may exceed the current capabilities of MLLMs. In future work, exploring more advanced retrieval techniques or enhancing MLLM's capabilities could further improve RAR's performance, especially in scenarios with large K values.

Dear Reviewer FduB,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer FduB,

I hope this email finds you well. I’m writing to kindly follow up on the review of our paper.

We are encouraged to share that one of the reviewers, nzRY, has maintained his original score of 6 and increased the confidence level to 4, reflecting his strengthened belief in the quality and significance of our work.

We sincerely value your feedback and would greatly appreciate it if you could share your comments and scoring at your earliest convenience. Your insights are critical for the timely progress of the review process.

Thank you very much for your time and support.

Best regards,

Authors

Dear Reviewer FduB,

We are sorry that we have not received a response from you since we last addressed your questions. Do you have any further concerns? We will do our utmost to address them. Given the mixed results with two ratings of 6 and one of 5, we would be grateful if you could clarify any remaining concerns that we may not have fully addressed. We truly look forward to your response. Thank you again for your time and thoughtful feedback.

With sincere appreciation,

The Authors

Dear Reviewer Zt9H,

We sincerely thank you for your thorough feedback: RAR’s design is well-founded and thoughtfully justified. We address your questions below. We have incorporated all your feedback in our revised manuscript. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1：Incorporating a detailed error analysis on retrieval failures or ranking misclassifications would provide insights into areas where RAR may need refinement, especially regarding failure cases in subtle category differentiation.

A1: Thanks for your suggestion. A deeper understanding of RAR's error modes can guide future improvements. As shown in Fig. 13, we've analyzed two common failure cases:

• Subtle Category Differentiation: When fine-grained categories are visually similar, even powerful vision-language models like CLIP may struggle to retrieve accurate results. This limits RAR's ability to learn from the retrieved examples, leading to potential misclassifications.

• Knowledge Scope Limitations: MLLM's performance can degrade when encountering categories outside its training data distribution. This can result in incorrect predictions, especially for rare or domain-specific categories. Despite these limitations, our experimental results demonstrate that RAR significantly outperforms state-of-the-art methods in fine-grained visual recognition, few-shot image classification, and zero-shot object recognition settings, highlighting its robustness and effectiveness.

To further enhance RAR's performance, future work could explore techniques to improve the quality of retrieved examples, such as incorporating more sophisticated visual similarity measures or expanding MLLM's knowledge base.

Q2: The paper mentions, "Although the brute force method is inherently straightforward, its efficiency markedly diminishes as the dataset escalates to the magnitude of millions of embeddings." The HNSW method can significantly improve retrieval speed. So, how can we quantify the complexity of brute force retrieval and HNSW when the dataset scales to millions of embeddings? Please provide specific comparative results.

A2: Assuming the total amount of data is N.

(1) Brute-force Search: For two 512-dimensional vectors, calculating the dot product requires 512 multiplications and 511 additions, making the complexity of a single dot product O(512). Therefore, the overall complexity of brute-force search is O(N⋅d), where d=512. Simplifying, the complexity becomes O(N).

(2) HNSW Search: In HNSW, during data insertion, neighbor connections are maintained across multiple layers of the graph, with each point connecting to log⁡N neighbors and performing distance calculations. The complexity of HNSW search mainly depends on the number of layers in the hierarchical graph and the number of edges in the graph. The approximate complexity is O(log⁡N⋅d). Here, log⁡N represents the process of narrowing the search range layer by layer in the hierarchical graph, and d represents the vector dimension involved in distance calculations. For 512-dimensional feature vectors, we built an index with 64 dimensions, so the query complexity becomes O(log⁡N⋅64)=O(log⁡N).

While HNSW has a higher complexity for index construction, its query complexity is significantly lower than brute-force search. Especially when N is large, HNSW greatly improves retrieval efficiency.

Dear Reviewer Zt9H,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer Zt9H,

I hope this email finds you well. I’m writing to kindly follow up on the review of our paper.

We are encouraged to share that one of the reviewers, nzRY, has maintained his original score of 6 and increased the confidence level to 4, reflecting his strengthened belief in the quality and significance of our work.

We sincerely value your feedback and would greatly appreciate it if you could share your comments and scoring at your earliest convenience. Your insights are critical for the timely progress of the review process.

Thank you very much for your time and support.

Best regards,

Authors

Dear Reviewer Zt9H,

I hope this message finds you well. As we are in the final stages, we would greatly appreciate your valuable feedback and scoring at your earliest convenience.

We sincerely understand how busy you must be, and we are deeply grateful for the time and effort you’ve already dedicated to reviewing our paper. Your support and thoughtful feedback are extremely important to us, and we would be thankful for any final comments you can provide.

Thank you once again for your time and consideration.

Best regards,

Authors

Dear Reviewer Zt9H.

The deadline is approaching. Now two reviewers post their response to our rebuttal, and our paper received mixed results (one 6 rating and one 5 rating). We have not yet received a response from you. Could you please review our rebuttal and share your comments? Thank you.

Dear Reviewer Zt9H.

The deadline is fast approaching, and we’ve already received mixed feedback from two reviewers (one giving a rating of 6 and the other 5). However, we have not yet received your response. With only a few hours left, could you please take a moment to review our rebuttal and provide your comments? Your input would be greatly appreciated. Thank you!

Q3-1: Need more discussion of fine-tuning for re-ranking. I noticed that the re-ranking operation requires fine-tuning the MLLM to achieve satisfactory performance.

A3-1: This is not true. Our method offers two complementary approaches for re-ranking: (1) in-context learning, which uses the MLLM's capabilities without altering its weights, and (2) fine-tuning, which can further improve performance through tailored training. The results in Table 5 highlight the effectiveness of in-context learning that does not require fine-tuning.

Q3-2: Although the authors claim that "RAR is not sensitive to changes in the fine-tuning dataset for ranking," they only conducted experiments on FGVC-Aircraft and Stanford-Cars. I believe this may be insufficient. If using a dataset with severe domain gap with the test datasets, catastrophic forgetting might occur, and I suggest that the authors discuss this issue.

A3-2:

• As discussed in A3-1, our method supports in-context learning for re-ranking, which eliminates the need for fine-tuning and, consequently, the risk of catastrophic forgetting. Since the MLLM weights remain unchanged, the model retains its knowledge and can effectively handle datasets with varying domain gaps.

• Regarding fine-tuning, our SFT process primarily focuses on teaching the model a specific response format and enhancing its ability to follow instructions. This objective, rather than imparting new knowledge, minimizes the impact of potential domain gaps in the SFT dataset. Consequently, the choice of the SFT dataset is less critical, as long as it provides sufficient examples to establish the desired response format.

• Our SFT model, trained on domain-specific datasets like FGVC-Aircraft and Stanford-Cars, has been evaluated on diverse benchmarks with significant domain gaps, such as Bird-200 in Table 1 and LVIS in Table 3. Despite these challenges, our method consistently outperforms baselines, demonstrating its robustness to catastrophic forgetting.

Dear Reviewer 5xbU,

We sincerely thank you for your thorough feedback: simple yet effective framework and extensive experiments. We address your questions below. We have incorporated all your feedback in our revised manuscript. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1: Relatively Limited Novelty. The authors use off-the-shelf models, such as CLIP and LLaVA, for fine-grained vision recognition and object detection. However, “multi-query re-ranking techniques” in RAG have already been widely adopted, for example, in Re2G (Retrieve, rerank, generate) [1] and RankRAG [2]. I did not observe any specific improvements in the Retrieving or Ranking strategy tailored to the fine-grained recognition task. This limits the novelty of the proposed framework.

A1: Thank you for your valuable feedback! However, our RAR is fundamentally different from RAG and should not be treated as the same thing. RAR has the following distinct differences with RAG:

• (1) Motivation. RAG is designed to incorporate new information for text generation, whereas RAR reveals and uses the complementary strengths of VLMs' broad category coarse recognition and MLLMs' fine-grained recognition.

• (2) Framework. RAG uses retrieved text as context prompts for LLMs, whereas RAR designs a multi-modal retrieval based on VLMs and further uses MLLMs for ranking the retrieved candidates.

• (3) Tasks. RAG is for NLP, whereas RAR is designed for large-vocabulary and fine-grained visual recognition.

Our contribution lies in identifying the respective strengths and weaknesses of VLMs and MLLMs in large-vocabulary and fine-grained visual recognition. Additionally, we have explored a solution that combines the two for addressing visual recognition tasks. The Re2G series and RankRAG are both highly valuable works in the RAG field, and we have added the discussion with previous RAG works in the related work section.

Q2-1: Concern of Practical Application. Although FineR previously demonstrated the use of large models for fine-grained vision recognition, I still question the necessity of using large models. Why not use specialized/expert models with much smaller scale to accomplish this task? To my knowledge, these expert models already perform well on the tasks evaluated by the authors.

A2-1:

• Our method has the advantage of Efficiency and Generalization over using specialized/expert models. Our method uses one versatile MLLM model to effectively address diverse benchmarks (e.g., Bird-200, ImageNet) and tasks (e.g., recognition, object detection). This unified approach offers significant advantages in terms of efficiency and generalization, as opposed to requiring specialized models tailored to specific benchmarks and tasks (e.g., FineR for Bird-200, ResNet for ImageNet, Mask R-CNN for LVIS).
• Using MLLMs has better interpretability than using specialized/expert models (will be elaborated in A2-2).

Q2-2: Based on this concern, I believe that using MLLM for this task should allow for open-set responses (such as providing interpretability) rather than simply using MLLM for re-ranking final predictions.

A2-2: We appreciate your insightful suggestion. Our method primarily employs MLLMs to refine the ranking of candidate predictions, while preserving their capacity for open-set responses and interpretability. As demonstrated in Fig. 12, MLLMs use their inherent knowledge to assist in visual perception tasks during the reranking process. This aligns seamlessly with the goal of providing explicit interpretations for model decisions.

Dear Reviewer 5xbU,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer 5xbU,

I hope this email finds you well. I’m writing to kindly follow up on the review of our paper.

We are encouraged to share that one of the reviewers, nzRY, has maintained their original score of 6 and increased the confidence level to 4, reflecting their strengthened belief in the quality and significance of our work.

We sincerely value your feedback and would greatly appreciate it if you could share your comments and scoring at your earliest convenience. Your insights are critical for the timely progress of the review process.

Thank you very much for your time and support.

Best regards,

Authors

Dear Reviewer 5xbU.

The deadline is approaching. Now two reviewers post their response to our rebuttal, and our paper received mixed results (one 6 rating and one 5 rating). We have not yet received a response from you. Could you please review our rebuttal and share your comments? Thank you.

I have carefully read the author's response, which addressed some of my questions. Overall, I still maintain my original rating.