MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

We appreciate the construtive and insightful comments from you! We address your concerns in details below.

W1: The performance of current retrieval systems may not handle this task.

Our retrieval corpus contains a diverse and highly fine-grained set of objects and images, encompassing a total of 16,130 images. This extensive collection is designed to represent a realistic retrieval corpus, effectively capturing the complexity and variability of real-world scenarios.

We have demonstrated the sufficient retrieval accuracy of current retriever models in our paper’s Table 6 in our Appendix C (please see line 1199). These retrieved examples provide notable benefits to all proprietary models as shown in Table 3.

For instance, in the Angle scenario as you mentioned, the CLIP retriever achieves a Recall@5 score of 70.19, showcasing its effectiveness in retrieving relevant visual knowledge.

Moreover, the primary focus of our benchmark is to evaluate LVLMs' abilities to utilize visually augmented knowledge while also establishing a baseline for the visual RAG pipeline. If certain scenarios pose challenges for the retrieval process, we see this as an opportunity to encourage further research and advancements in retriever models to address these limitations effectively.

Suggestion 1: Maybe the benchmark can also be used to evaluate multimodal retrieval systems.

We thank you for your excellent suggestion! Indeed, our benchmark can also be utilized for the evaluation of multimodal retrieval systems. This further underscores the benefits of our benchmark and encourages more future follow-up research in this area.

We appreciate the positive and insightful comments from you! We address your concerns in details below.

W1: Difficulty of retrieving from image corpus and usefulness and text knowledge and motivation.

Please first refer to our general response for our response on motivation.

We identified there are scenarios where retrieving visual information is either more beneficial or easier to access than textual data. Testing on our benchmark across 9 distinct scenarios, we have demonstrated the critical role of incorporating visual knowledge.

Furthermore, we would like to clarify that retrieve useful image information is easier than retrieving from text corpus since we identified scenarios where image augmentation is more useful than text augmentation. Moreover, our work also motivates more research on image retrieval.

For our benchmark’s scenarios, we have demonstrated the sufficient retrieval accuracy of current retriever models in our paper’s Table 6 in our Appendix C (please see line 1199). For instance, in the Angle scenario, the CLIP retriever achieves a Recall@5 score of 70.19, showcasing its effectiveness in retrieving relevant visual knowledge. These retrieved examples provide notable benefits to all proprietary models as shown in Table 3.

W2: How is image knowledge database handled and retrieved?

The collection of image knowledge databases is introduced in our paper’s 2.3 Data Collection section (line 248, 256, and 262). All of the ground-truth images from different questions formed the image knowledge base.

We used CLIP as the multimodal retriever, which takes both the input image and the text question, and outputs the image candidates. We experimentally found the CLIP retriever worked the best as illustrated in our paper’s Table 6 and keep this retriever consistent throughout our experiments as mentioned in our paper’s line 415.

W3: Explore the possibility of CoT + Visual RAG.

We appreciate your suggestion on this valuable direction for future work. The primary focus of this paper is to introduce a benchmark specifically designed to evaluate LVLMs' abilities to utilize visually augmented knowledge, and we will leave CoT to future work.

W4: Why is GT Image RAG better than GT Text RAG in Table 4?

Please first refer to our general response. As a key contribution of this paper, our benchmark focuses on distinctive scenarios where visual knowledge is both useful and more easily accessible than textual information. For instance, consider the example in Figure 4 of our paper. In this case, the question asks for the exact car model and make. The GT text includes the context of the ground truth answer (e.g., "Chevrolet Silverado 1500 Extended Cab") and additional details such as the car's year of manufacture and engine specifications, etc. Although all these information described the information of this car model, none of them includes visual appearance of the car. Since this example represent a difficult Angle scenario, the textual information alone is insufficient—the model cannot recognize the car visually due to the challenging perspective in the image. This highlights why GT image knowledge is invaluable in such scenarios.

Q1: Public release of benchmark.

We provide an anonymous repository for evaluating the quality of our benchmark here: https://huggingface.co/datasets/mragbenchanonymous/MRAG-Bench . This repository will be made publicly available upon the acceptance of our work.

We appreciate the positive and constructive comments from you, which are crucial for improving the paper!

W1: Potential bias in data collection process and mitigation.

We thank you for raising this insightful concern. As discussed in the paper’s data collection section (Section 2.3), the data primarily originates from existing datasets and web-sourced images. Consequently, biases inherent in existing datasets, such as ImageNet, are inevitably present. For web-sourced data, as mentioned in line 256, we carefully curated keywords (listed in Appendix A.1, starting at line 924) and performed human filtering to ensure the quality of qualified images.

One potential bias arises from the process of selecting ground truth (GT) examples. Specifically, we manually selected 5 GT images to represent the necessary visual information for each answer. This introduces a potential bias, as different individuals might select different combinations of 5 images. However, this bias is mitigated to some extent by using 5 GT images instead of just one, which already covers a broad range of possible aspects of the visual object. Furthermore, the quality of our benchmark is evidenced by its positive impact on all 14 models evaluated, as shown in Table 3 of our paper.

To further mitigate this bias, a stricter control process could involve engaging more human annotators and selecting 5 GT images that reach consensus among the annotators, ensuring they are representative of the question's answer.

W2: Complex tasks when there are multiple objects in the question.

We agree that when multiple visually prominent objects are present in an image, it can be challenging for retriever models to identify and retrieve relevant visual knowledge effectively. A potential solution to address this issue is to integrate an object detector model to first identify and select specific image objects. The retriever model could then focus on finding corresponding visual knowledge for the selected objects. The primary focus of our benchmark, however, is to evaluate LVLMs' abilities to utilize visually augmented knowledge, emphasizing the importance of integrating external visual information to improve performance.

We believe it will be a fruitful future direction to explore multiple objects which is analogous to multi-hop or multi-document tasks in the text RAG community, where specific approaches are developed to address these challenges. For instance, such tasks often involve decomposing the question into smaller, manageable sub-questions and then retrieving relevant information from multiple documents. In our scenario, this approach could translate into searching for more images and selective feature extraction for each visual object, ensuring that the visual reasoning focuses on the appropriate elements. We will add more discussion about this future direction in our paper.

W3: Further discuss the implication of finding and how it might influence the development of LVLMs.

In our analysis sections 4.2 and 4.3, we have explored several potential directions for future research. These include addressing the position bias of retrieved visual examples (line 456) and optimizing the selection of the number of necessary visual knowledge images based on the complexity of the questions (line 474). For our main findings and how explicitly to help open-source LVLMs develop better abilities in handling visual knowledge, we have the following suggestions. 1) incorporating training data utilizing visual knowledge; 2) Designing Models to Assess the Utility of Noisy Visual Information: Current LVLMs tend to treat all images as useful, which is not realistic for real-world scenarios. Models should be designed to assess whether specific visual information is helpful and selectively utilize the correct visual data. We will add more discussions to our paper.

Q1: How is human evaluation conducted? How to ensure consistency and accuracy of human assessments?

As mentioned in our paper’s evaluation setup, the human evaluation details can be found at Appendix A.2, please see line 1085. Three human annotators in the domain conducted the human evaluation. The interface for human evaluation without RAG knowledge and with RAG knowledge are shown in Figure 6 and Figure 7.

To ensure the consistency and accuracy of human assessments, we specifically provided clear rules and settings to each human annotator. As detailed in Appendix A.2, the annotators are domain experts with demonstrated expertise with similar backgrounds for knowledge consistency. To further mitigate bias and ensure consistency, we averaged the final scores from all three annotators. This approach helps reduce individual bias and improves the reliability of the overall human performance evaluations.

Q2: For without RAG and with RAG setting, how to ensure human annotators do not rely on their own knowledge when answering questions? Potential bias in the human evaluation process.

Humans can rely on their own knowledge when evaluating human performance. With RAG, human performance refers to the ability to utilize combined image RAG knowledge with their inherent knowledge. This approach is consistent with the testing methodology applied to all models in Table 3. As observed, proprietary models tend to perform better in the no RAG setting due to the extensive knowledge encoded in their parameters.

The primary focus of our benchmark, however, is to evaluate LVLMs' abilities to utilize visually augmented knowledge, emphasizing the importance of integrating external visual information to improve performance.

For potential bias, we averaged the final scores from all three annotators. This approach helps reduce individual bias and improves the reliability of the overall human performance evaluations.

Q3: Whether retrieving additional visual information also enhances performance in text-intensive scenarios?

We think this is a very interesting question! In general, text-intensive benchmarks are designed to evaluate models' abilities to utilize text-specific information, such as exact dates, years, and precise numerical records from textual corpora. Visual knowledge, on the other hand, encodes information from a different perspective, which may not be directly applicable or useful in these text-focused scenarios. We leave the study of visual retrieval augmentation for text-intensive scenarios for future work.

The primary focus of our benchmark is to evaluate LVLMs' abilities to utilize visually augmented knowledge, emphasizing the importance of integrating external visual information to improve performance – a perspective that has been largely overlooked in previous studies.

Q4: How does GPT-o1 perform on this benchmark?

At the time we wrote this paper, GPT-o1 was just released which is 9/12/2024. However, due to rate limits of 50 queries per week for the o1-preview model, we were unable to test it across all 1,353 questions in our benchmark.

GPT-o1 primarily focuses on reasoning through complex tasks and solving more challenging problems in science, coding, and mathematics. Since it does not specifically mention enhanced visual reasoning abilities, we anticipate that its performance in our benchmark would be similar to that of GPT-4o.

Q5: In Section 4.3, the paper shows that further increasing the number of retrieved images does not improve performance. Could the authors explain the reasons behind this effect, and what insights does this provide for future research on retrieval-augmented multimodal models?

In section 4.3’s Figure 5, we demonstrated increasing the number of retrieved images consistently improved performance but reached a maximum. We offered one possible explanation in the paper line 470 in section 4.3:

One possible explanation could be that LLaVA-Next-Interleave may not be able to better leverage visually augmented knowledge in long context scenarios. Moreover, the complexity of questions affects the number of images needed too, one ground-truth examples sometimes help the model the most on MRAG-BENCH. We encourage the research on adaptively deciding the number of necessary images based on the complexity of questions.

Q6: Will you release this benchmark?

We provide an anonymous repository for evaluating the quality of our benchmark here: https://huggingface.co/datasets/mragbenchanonymous/MRAG-Bench . This repository will be made publicly available upon the acceptance of our work.

Q5.1: Concerning Table 3’s Human performance, if human performance is low, how is the quality of collected GT images ensured?

We would first clarify that our benchmark contains some domain-specific questions that can be challenging for the average human without relying on additional tools.

Human performance is measured by humans answering the question without seeing the ground truth answer and without using any additional resources such as web search.. Examples of the human evaluation interface are provided in Figure 6 and Figure 7 in Appendix A.2 of our paper. However, during the data collection process, annotators are required to know the ground truth answers to ensure the quality of the ground truth data.

Additionally, human experts conducted further quality checks after the initial data collection phase. Detailed descriptions of this process can be found in Section 2.3 (lines 264–270) and Appendix A.1 (lines 1076–1082).

Q5.2.1: In Table 3, Since performance decline in noisy retrieved examples, retrieval effectiveness is crucial to answer quality.

We agree that retrieval effectiveness is indeed crucial, so our work also points out a future research direction of improving image retrieval for VQA tasks. In addition, we clarify that, the focus of this table is to demonstrate the abilities of leveraging visual knowledge in LVLMs. As demonstrated in Table 3, most open-source models struggle to effectively leverage noisy retrieved examples, whereas proprietary models perform well with such data. This highlights a significant gap in current models (as discussed in lines 105–107 of our paper) and underscores the importance of continued research to address these limitations.

Q5.2.2 In Table 3, If retrieval consistently underperforms, is the entire process reasonable and valid?

We would appreciate your clarification on the comment of “retrieval consistently underperforms”. In fact, we have shown the sufficient retrieval accuracy of current retriever models in Table 6, and these retrieved examples provide notable benefits to all proprietary models.

For future work, the open-source community could focus on two main areas: Improving Model Capabilities: Enhancing open-source models' ability to better utilize noisy, visually augmented knowledge. Advancing Retrieval Models: Developing more accurate retrievers, which would benefit both open-source and proprietary models. This dual approach can help address the challenges and improve the overall process.

Q6.1: In Table 4, How were the five comparison methods implemented?

We followed the standard implementation practices for RAG systems. The code will be publicly released upon the acceptance of this paper.

Q6.2: In Table 4, What is the meaning of GT Text RAG, and how is GT Text collected?

For retrieving textual knowledge, we clarified in the paper (lines 414–416) that CLIP is used to retrieve from a Wikipedia corpus. When retrieving ground truth (GT) textual knowledge, the input to the retriever includes the ground truth answer to the question, allowing it to locate the corresponding ground truth context within Wikipedia.

Thank you for pointing this out. While this approach follows standard conventions, we recognize that it may not have been sufficiently clear for people from different backgrounds. To address this, we will incorporate these details into the revised version of our paper for improved clarity.

We sincerely thank you for your detailed review. We would first like to provide clarifications to all the points you raised by responding to each of your questions below.

After you review our responses, we will incorporate these clarifications into our paper revision as promised. This will ensure that the revised version avoids any line number conflicts between the new and old versions, thereby improving the overall clarity and coherence of our work.

W1: Data collection of GT and source of question-answer pair is unclear.

Thank you for your review. We will provide a figure in our appendix for further clarification in our revised version. The collection of image knowledge databases is introduced in our paper’s Section 2.3. Data Collection (line 248, 256, and 262). All of the ground-truth images from different questions formed the image knowledge base. As noted in line 91 of our paper, our benchmark involves human-annotated question answering. Therefore, the ground truth answers are derived either from class labels or human annotations, with a similar approach applied to ground truth images. More detailed responses to specific questions are provided below.

W2: Experimental Setup lacks details and missing Experimental Setup section.

Thank you for your review. We clarify that we have a detailed subsection titled Experimental Setup in Section 3.1 (line 318 of our paper). Regarding Table 3, the retriever model used is CLIP, as mentioned in line 415. However, we agree that including this information in Table 3’s caption would enhance clarity, and we will address this in the revised version.

To clarify further, the retriever’s input consists of the text question and the query image, while the output is the set of retrieved images. We appreciate your feedback and will incorporate these adjustments in our revisions.

W3: “How much does visual knowledge provide additional benefits over textual knowledge?" is insufficiently supported. Additional experimental analysis would strengthen it.

Thank you for your review. We would appreciate it if you can provide more details of what additional experiments could help strengthen this point. As demonstrated in Table 4and discussed in Section 4.1 (lines 412–426) of our paper, both open-source and proprietary models show notable performance improvements when leveraging image knowledge than textual knowledge, whether through retrieved knowledge or ground truth knowledge.

Q1: The meaning of “Unique number” in Table 2.

In Table 2, “Unique number” refers to a value that is distinct and not repeated within a particular context or dataset. For example, a unique number of questions means a number of questions that are not repetitive. We would clarify the wording in the revised version, thank you!

Q2: How were nine scenarios in MRAG-Bench selected?

Thank you for your question. As discussed in our paper’s introduction (see line 92-95), we focused on utilizing visually augmented knowledge in real-world scenarios and divided it into two major aspects: perspective and transformative (see paper’s section 2.1). Following their definitions, we clearly defined 9 scenarios with each scenario’s explanation detailing in our paper’s section 2.2.

Q3: Why is option A the correct answer in Figure 3’s “Others” scenario?

As mentioned in our paper’s data collection section line 259-263, the “Others” scenario sourced images from GeoDE dataset (Ramaswamy et al., 2023). This dataset has class labels for different regions such as Americas, EastAsia, and we used their class label as the ground truth answer.

Q4: How is CLIP used for text retrieval?

We used CLIP as the multimodal retriever, which takes both the input image and the text question, and outputs the image candidates. We experimentally found the CLIP retriever worked the best as illustrated in our paper’s Table 6 and keep this retriever consistent throughout our experiments as mentioned in our paper’s line 415.

Thank you for your response. I want to clarify that my primary concern regarding this paper is the lack of detailed information. Throughout the article, details on dataset collection/processing and experimental specifics are scattered across various sections. As I read the paper, I constantly flipped back to previous sections to piece together the details I was looking for, sometimes to no avail. This current layout undermines the reliability of the experimental conclusions. Also, I agree with Reviewer "U78R" that the application scenarios of this paper are limited in real-world settings.

Here are some follow-up questions, and clarifications about my concerns:

My concern:

W2: The experimental setup lacks sufficient detail. The authors should provide a complete subsection dedicated to describing the experimental setup, placed within Section 3, instead of scattering brief descriptions across various sections.

The authors' response:

We clarify that we have a detailed subsection titled Experimental Setup in Section 3.1 (line 318 of our paper).

I suggest the authors provide a complete subsection dedicated to detailing the experimental setup. Instead of only introducing the baseline models and mentioning "follow standard MCQA evaluation setup" in line 318 of the paper. This subsection should include details about default generation hyper-parameters, the collection of human performance, the implementation of Retrieved RAG and GT RAG, and so on.

Re-Q3: My real concern lies in the quality of the GT Example. I hope the author can provide a more detailed analysis of this example. What is the relationship between the current GT Example and the Query Image? How can the GT Example help identify if it is American?

Re-Q4: I suggest the author try more advanced text RAG strategies, such as INFOSEEK [1]. Directly applying text RAG to the question itself greatly limits the model performance in VQA tasks. For example, in Fig. 1, the question "Which year was this building constructed？" does not contain valid information within the question itself. Therefore, using a naive Text RAG method as a baseline method seems to be inappropriate. [1] Can pre-trained vision and language models answer visual information-seeking questions?

Re-Q5.2: A recall@5 of 60.46 cannot be considered "sufficient retrieval accuracy." As stated by reviewers Pyuc and JirS, the retrieval difficulty will be higher for broader application scenarios. Therefore, I remain concerned about the retrieval performance. This is a challenging issue to resolve within the entire pipeline of this paper.

Re-Q6.1: For the authors' statement "We followed the standard implementation practices for RAG systems," please provide specific references.

Q3 and W3: Inference cost of increasing context in multi-image models.

We thank you for raising this insightful concern. There has been significant progress in multi-image models, with many strong methods excelling at handling very long image contexts. For instance, LongVILA (Xue et al., 2024) achieves an impressive 99.8% accuracy in a 6,000-frame video (more than 1 million tokens) "needle-in-a-haystack" task.

Our approach employs only 5 additional images (fewer than 1,000 tokens) for visual augmentation. We believe this cost is negligible compared to the substantial performance improvements demonstrated in Table 3.

Moreover, while exploring more visual contexts knowledge could be an avenue for future work, current models have already demonstrated enough capabilities for incorporating long visual context.

Based on your question, we realize that our benchmark can also serve as a valuable testbed for long-context models, providing an additional avenue for research and evaluation. We will add this discussion in the future work section to enrich our paper.

We'd like to express our sincere gratitude for your thorough review of our paper. We greatly appreciate your suggestions which are crucial in improving the quality of our paper.

W1: Extensive repositories of curated and labeled images are needed.

We thank you for recognizing our visually augmented samples offer significant value. Notably, our image corpus does not require labeled data. The multimodal retriever model can search raw images and identify those that best align with the user's question and query image. Although ground-truth images can help model the most, the primary focus of our work is to encourage models to better utilize noisy retrieved images. As demonstrated in Table 3 of our paper, the retrieved images significantly enhance the performance of all proprietary LVLMs. For practical applications, our approach can easily integrate either publicly available web images or self-maintained raw image databases.

Q1: Motivation and real world application.

Please first refer to our general response. We are also glad that Reviewer JirS found that many cases in the benchmark are close to real world scenarios.

A brief example of a real-world application is the saying, "An image is worth a thousand words." Image information is inherently more diverse, capturing details, contexts, and nuances that cannot always be directly translated or recorded in a text corpus.

Although the primary focus of our work is the benchmark, and we encourage future research in this direction, the real-world applications of our visual RAG system are diverse and impactful. These include Vehicle Identification and Insurance Claims, E-commerce and Retail, Wildlife Monitoring and Conservation, Healthcare and Medical Diagnostics etc.

To illustrate one concrete example of the scenarios in our benchmark—perspective aspect—consider a real-world application: When a person spots an attractive outfit on the street and wants to find and purchase it, they might take a photo in a crowded environment. This photo could capture the clothing at an awkward angle, show only a partial view, or be taken from a distance. In such cases, an LVLM can leverage its capabilities to retrieve related images from the web, enhance its visual knowledge, and provide a more accurate and useful response to the user’s query. The real world applications can be even broader, as in all scenarios demonstrated in our paper’s Figure 3, the models can utilize and benefit from visual image knowledge.

Q2 and W2: Legal and computation restrictions. Strategies for mitigating biased and intrusive contents in image corpus.

We thank you for raising this insightful concern. As detailed in our data collection section (Section 2.3), our image corpus is sourced from both existing image datasets and web images. To ensure the quality and integrity of our benchmark, we conducted a thorough quality check (see lines 264–270) to confirm that no intrusive content is included. The keywords used for sourcing web images are provided in Appendix A.1. Furthermore, we adhered to all applicable licenses when legally downloading these images, and our benchmark will be released strictly for academic purposes.

For future works aiming to construct a similar image corpus for their applications, we propose the following mitigation strategies:

1. Filtering Keywords: When sourcing images from the web, keywords can be filtered to exclude harmful or intrusive content.
2. Content Moderation by Models: During real user interactions with LVLMs, as practiced by many current systems, models can be designed to refuse to answer queries. Similarly, our approach suggests having the model refuse to retrieve potentially harmful content, even if such content bypasses the initial filtering process.

It is important to note that addressing these challenges is not the primary focus of our work. Our benchmark is to evaluate LVLMs' abilities to utilize visually augmented knowledge while also establishing a baseline for the visual RAG pipeline. Our benchmark has demonstrated good quality as a baseline and we encourage more future work on large-scale adoption of our approach.