MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

We thank the reviewer for the feedback and provide a detailed response below.

W1. The paper only contributes data without offering any insightful analysis and lacks novelty and contribution.

We want to clarify that, for a benchmark paper, the evaluation data and framework already constitutes a substantial contribution. Prior to our work, there was no comprehensive benchmark for this specific scenario. Our benchmark fills this gap, providing a testbed for evaluating the strengths and weaknesses of models and enabling future analyses from various perspectives of multimodal LLMs.

Our paper is the first to provide a comprehensive benchmark for evaluating robust multi-image understanding capabilities in multimodal LLMs. We developed a benchmark encompassing 12 tasks and 10 image relations, offering in-depth analyses that can guide future research. Specifically, our work identifies gaps in current model capabilities and provides inspiration for researchers to expand multi-image datasets using our data collection methodology. To the best of our knowledge, this is the first paper that delivers such insights and contributions. We provide a comprehensive list of analyses and insights in the following responses.

W3/Q1. the analysis part in this paper is superficial, more insight is required.

We respectfully disagree that the analyses are superficial. We have provided in-depth analyses in Section 4 and Appendix D. All of these analyses provide valuable insights. We have analyzed model performance on the unanswerable set, error cases, and qualitative results. We have further examined the influence of image position, unanswerable type, and input format. We also analyzed the role of captioning in solving multi-image problems. Researchers will gain insight on how to advance the development of multimodal LLMs, such as (1) task, relation, and question formats to cover in the training data, (2) the tradeoff between resolution and long visual context, and (3) enhancing robustness and preventing prediction shortcuts in the context of multi-image understanding.

Below is a selected list of analyses and findings in this paper:

• In which multi-image tasks do multimodal LLMs show relative strengths and weaknesses?
  • We find that multi-image ordering and visual grounding appear to be more challenging for existing MLLMs, because these tasks require understanding the whole multi-image context and conducting more complicated reasoning processes across images and modalities afterwards. (lines 448-450)
• In which multi-image relations do multimodal LLMs show relative strengths and weaknesses?
  • We find that existing MLLMs fall short in understanding temporal relations and identifying cross-image similarity. (Table 2)
• Can models designed for single-image inputs perform multi-image tasks?
  • Our results show that models benefit from multi-image training data and learning processes to develop multi-image understanding capabilities. (lines 455-456)
• Do multimodal LLMs perform worse on the unanswerable set?
  • Our results reveals that models often fail to identify unanswerable questions, indicating that models may not comprehensively understand the multi-image context. (lines 463-465)
• Do image positions correlate with error rates?
  • Our results emphasize the necessity of training data covering diverse image positions to improve model generalizability. (lines 470-474)
• Do unanswerable types correlate with error rates?
  • This analysis demonstrate the distinct weaknesses of different MLLMs on this problem. (lines 477-480)
• Error analysis of GPT-4o
  • We identify the key weaknesses of GPT-4o in multi-image understanding, including capturing details, counting, and logical reasoning. (lines 485-500)
• Can multi-image MLLMs understand single-image input?
  • This analysis shows that concatenating images can lead to information loss and introduce coreference challenges between subimages and their textual mentions in the prompt. (lines 1065-1067)
• Can single-image MLLMs understand multi-image input?
  • The results suggest that explicitly processing multiple images as input is beneficial, as it reduces information loss and minimizes challenges in referencing images. (lines 1072-1074)

Our work identifies important problems of robust multi-image understanding in multimodal LLMs and presents a way to quantitatively evaluate the corresponding capabilities. In terms of model architecture, our analyses suggest to go beyond the current sequential input structure and attention mechanism to better capture the diverse relations across multiple images.

W2/Q2. Authors overlooked some of the newer VLMs, such as Gemini 1.5 Pro, InternVL1.5, ...

We report the results of multiple most recent VLMs, including LLaVA-OneVision (the subsequent version of LLava-Interleave; released in Aug. 2024), InternVL2 (the subsequent version of InternVL1.5; released in Jul. 2024), and Gemini-1.5-Pro (version 002 released in Sept. 2024).

W4. The difficulty of question among different tasks is imbalance.

We want to clarify that question difficulty is not the primary focus of this benchmark. Our goal is to evaluate multi-image understanding capabilities comprehensively and robustly, rather than to explicitly measure or categorize the difficulty of individual queries. Moreover, there is currently no widely accepted standard for quantifying query difficulty in the context of multimodal benchmarks. Attempting to do so would introduce additional subjectivity and complexity, potentially detracting from the primary objectives of our work.

Nevertheless, to facilitate fine-grained analysis of model performance, we have reported the performance on each task (Table 1) and each multi-image relation (Table 2) . This follows the common practice of many widely-used benchmarks, such as BLINK and MathVista.

To obtain rebalanced results, one can also report the task-balanced or relation-balanced model performance (i.e., the macro average). We provide the reweighted performance below. While the numbers differ, the trends remain roughly the same.

W5. It seems that unanswerable questions have limited practical applications in real-world scenarios.

Unanswerable questions are practical and integral to assessing model robustness, as outlined in the introduction of our paper and supported by prior research [1,2]. This aspect is previously overlooked in multi-image scenarios but is essential for understanding how models handle uncertainty and incomplete information. Our results (Figure 8) underscore the significance of this feature for a comprehensive evaluation of model performance. The significant performance gap between answerable and unanswerable sets indicates that models may achieve good performance by cheating.

Moreover, the pairwise design (answerable-unanswerable) is intended to ensure that the model cannot achieve good performance by cheating. Models may answer questions correctly by making an educated guess without fully understanding all the images. To avoid this, we pair each answerable instance with an unanswerable counterpart. This ensures robustness and reliability in evaluating model performance. According to our analysis in Figure 8, the top five models achieve significantly better performance on the answerable set than on the unanswerable set. This highlights the challenges of ensuring robust understanding rather than leveraging shortcuts.

[1] Rajpurkar, Pranav, Robin Jia, and Percy Liang. "Know What You Don’t Know: Unanswerable Questions for SQuAD." ACL 2018.

[2] Miyai, Atsuyuki, et al. Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models.

W6. As a questioner, I would consider inputting images one by one to ensure accuracy.

Note that this is a benchmark, while the reviewer is indeed discussing how to improve performance through preprocessing or inference-time techniques. This actually highlights the value of our benchmark in inspiring and supporting such research.

In reality, model developers cannot assume how users will present their questions. The reviewer’s method is only one proposal for question formatting, but questions can naturally appear in diverse formats in real-life usage. Our goal is to collect a comprehensive benchmark that measures robust multi-image understanding capabilities, regardless of how users present their questions.

Q4. How exactly are the images concatenated?

We concatenate the images following the setting of previous work [1]. The concatenation method is not the focus of our work. Instead, we seek to provide a testbed to analyze every aspect of multi-image VLMs, including the one mentioned by the reviewer. We leave specific concatenation testing analyses for future research using our benchmark.

[1] Jiang, Dongfu, et al. "Mantis: Interleaved multi-image instruction tuning." arXiv preprint arXiv:2405.01483 (2024).

We appreciate the reviewer’s active discussion and are happy to provide further clarification as follows.

For W3/Q1, W1. many of the analyses primarily describe the observed phenomena without delving into actionable points

We want to clarify that our contribution, analyses, and data size are comparable to previous widely-used multimodal benchmarks with different focuses. The key issue is that, prior to our benchmark, researchers didn’t even have a comprehensive and rigorous metric to assess the multi-image understanding capabilities of VLMs.

This paper, like any benchmark paper, focuses on identifying bottlenecks and providing a rigorous evaluation framework. The scope of analyses we provided are similar to papers of widely used benchmarks, such as BLINK (ECCV 2024), MMMU (CVPR 2024), MathVista (ICLR 2024), etc. Note that fine-grained analysis for actionable improvements is also not the primary focus of these papers.

In terms of data scale, the current data size (2,600) is sufficient to advance the development of VLMs. For example, datasets like Winoground (CVPR 2022), which contains only 400 examples, have significantly influenced VLM advancements. Similarly, although focused on different aspects, other widely adopted benchmarks—including WHOOPS! (ICCV 2023), LlaVA-Bench (NeurIPS 2023), Visit-Bench (NeurIPS 2024), ConTextual (ICML 2024), VibeEval, and Visual Riddles (NeurIPS 2024)—comprise 90, 500, 576, 500, 269, and 400 examples, respectively, and have been pivotal for evaluating VLMs.

We argue that proposing ideas for improving model capabilities and providing supporting experiments are beyond the scope of any benchmark paper, as the purpose of such work is to assess and highlight limitations, not to propose solutions. Including this would be unreasonable within the constraints of a 10-page paper, especially given our already extensive appendix making the paper 24 pages in total.

For W5. unanswerable questions can be easier and not a key contribution

We want to clarify that the randomizing answer choices mentioned by reviewer QRSh can not address the robustness concerns identified in this paper. Randomizing answer choices is only shown to be effective in reducing position bias. (The answer choices in our MuirBench are already randomly shuffled after data creation, ensuring fairness in evaluation.) There are more critical biases in model evaluation, such as overfitting to specific patterns or failing to handle unanswerable questions effectively. Our approach deliberately includes unanswerable questions to assess the model's ability to abstain when appropriate, which is crucial for robust and reliable evaluation.

Additionally, identifying the poor performance of VLMs on unanswerable questions offers further insights into their current bottlenecks. This finding highlights critical areas for improvement and emphasizes the need for more robust reasoning capabilities in VLMs. To address this problem, researchers could incorporate more realistic unanswerable data into the training process.

For W6. whether inputting images one by one can significantly enhance the performance of MLLMs

We refer the reviewer to Table 4, also presented below. In this experiment, we evaluate an alternative questioning format: first, GPT-4o extracts key information from each image independently (i.e., one by one), followed by a query to answer the question based on the extracted information. The results show that this format does not improve performance. Instead, the additional steps amplify error propagation. Furthermore, processing images independently introduces efficiency concerns, as the number of API calls scales linearly with the number of images, making it impractical in real-world applications.

For Q4. more related experimental results on different concatenation methods

We provide an analysis in Appendix D that investigates the effects of concatenating image tokens versus concatenating image pixels. The results, shown in Table 3, reveal performance variance across different methods, highlighting the value of our benchmark in facilitating such analyses.

However, we argue that while additional experiments could always be conducted, they are not the primary focus of our benchmark.

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer.

W1. The first picture of Figure5 seems to be ambiguous

The confusion arises from the simplification made for presentation purposes. The actual examples (Please check the link pointing to an anonymous figure.) for this task include six to eight images rather than just three. We want to clarify that the simplified conceptual example in the paper is to illustrate how we make the images, the question, and the options incompatible, in order to create an unanswerable instance. One way to achieve this is by designing instances where the answer is non-deterministic in terms of the options (i.e., cannot be answered with certainty based on any given option so one should choose “none of the other options”).

For quality control, as detailed in Section 3.2, we explain the process of manually filtering out errors and retaining only high-quality instances. Specifically, multiple experts reviewed all labeled instances to filter out uncertain cases. Additionally, we report human performance, which is higher than 93% accuracy, to further demonstrate the data quality.

W2. Unanswerable setting experiment may result in "cheating"

This is a very insightful question. The pairwise design (answerable-unanswerable) of MuirBench is indeed intended to ensure that the model cannot achieve good performance by cheating. Cheating can arise from two sides:

1. Answering the answerable questions correctly by making an educated guess without fully understanding all the images.
2. Correctly identifying unanswerable questions due to the trend of overly refusing to answer.

A reliable evaluation of robust multi-image understanding should avoid both problems. To address this, we ensure that each instance has a minimally revised counterpart with a different answer type (answerable <-> unanswerable). Pairing each answerable instance with an unanswerable counterpart ensures robustness and reliability in evaluating model performance.

According to our analysis in Figure 8, the current bottleneck of advanced multimodal LLMs is related to cheating on answerable questions (type 1). The top five models achieve significantly better performance on the answerable set than on the unanswerable set. This highlights the challenges of ensuring robust understanding rather than leveraging shortcuts.

Our benchmark provides a comprehensive testbed to analyze these cheating problems. We will include a discussion of this in the revised paper, incorporating findings from [1].

W3. should the author’s definition of “Multi-Image Input Models” here be models trained with multi-image data?

Our original terminology refers to the intended use of the models. We will update the terms to avoid confusion based on the reviewer's suggestion to avoid confusion. We agree with the reviewer that it is possible to apply models trained with single-image data to conduct multi-image inference. To address this, we provide an analysis in Appendix D investigating whether single-image-trained MLLMs can understand multi-image input and vice versa. The results, presented in Table 3, indicate that models trained with single-image data can transfer their capability to process multi-image data to some extent. This also demonstrates the value of our benchmark in facilitating such analyses. We additionally present the results in the next section.

W4. it is a more comprehensive way to test the "single-image input models" using both concat image and concat image token.

W5. lacks some more advanced open-sourced models for evaluation, such as LLava—interleave ...

We report the results of most recent MLLMs, including LLaVA-OneVision (the subsequent version of LLava-Interleave; released in Aug. 2024) and Gemini-1.5-Pro (version 002 released in Sept. 2024).

We appreciate the reviewer’s active discussion and are happy to provide further clarification as follows.

the description of the manual annotation

MuirBench was created by a team of more than 20 experts in this area, with annotations conducted by nine PhD students specializing in related fields. The annotated data underwent an additional review by three graduate-level students to ensure quality. We have provided the annotation details in section 3.2 and appendix A.2. If the reviewer has specific requests, we would be happy to provide further details.

Why not simply randomize the answer choices

Randomizing answer choices does not address the critical robustness concerns introduced in this paper. There are more critical biases in addition to position bias in model evaluation, such as overfitting to specific patterns or failing to handle unanswerable questions effectively. Our approach deliberately includes unanswerable questions to assess the model's ability to abstain when appropriate, which is crucial for robust and reliable evaluation.

whether the models score higher on the answerable set due to random selection

The answer choices are already randomly shuffled after data creation, ensuring fairness in evaluation.

models with poor performance on multi-image tasks

The benchmark's goal is to identify potential issues and bottlenecks in models. For weaker models, the bottleneck may simply lie in their overall performance, which is still a critical insight for understanding their limitations. This does not diminish the value of our benchmark, as its primary purpose is to systematically identify model limitations and bottlenecks.

Concat Tokens introduces the issue of long context input

The experiment was conducted following the settings requested by the reviewer. Definitely, there could be multiple causes for the model performance drop.

Notably, a comprehensive analysis of specific model issues will be an independent work, separate from the scope of this general benchmark. The purpose of this (and any) benchmark is to provide a testbed for such analyses, and we already present initial results to encourage attention to these issues.

InternVL2

In response to the reviewer's request, we evaluated InternVL2 on MuirBench. Notably, the results surpass some of larger models, such as LLaVA-v1.5-13B, highlighting the importance of the training process in addition to model size in achieving superior performance.

We will definitely include the InternVL results in our paper. We are working on the paper update and will upload a new version with revised parts highlighted within one day.

analysis of how improvements can be made (e.g., in architecture, data, or training strategies)

This paper, like any benchmark paper, focuses on identifying bottlenecks and providing a rigorous evaluation framework. The reviewer's request to propose ideas for improving model capabilities and provide supporting experiments is beyond the scope of any benchmark paper, as the purpose of such work is to assess and highlight limitations, not to propose solutions. Including this would be unreasonable within the constraints of a 10-page paper, especially given our already extensive 8-page appendix.

I appreciate the detailed response from the author, which addresses some of the potentially confusing points in the paper (such as the ambiguous images and definitions). By the way, I strongly recommend that the author highlight the changes in the updated PDF, as this would make the modifications clearer.

However, based on the author's response, I still believe the paper has considerable room for improvement:

• About Manual Annotation: The author repeatedly emphasizes that to indicate the quality of MuirBench in the rebuttal and the paper, but the description of the manual annotation seems unclear. We only see a single figure showing the interface human annotations, but there is no detailed information about the number of annotators, their backgrounds, the time spent, etc.

• About the Unanswerable Setting:

  First, regarding the statement “Pairing each answerable instance with an unanswerable counterpart ensures robustness and reliability in evaluating model performance,” I do not see the importance of the unanswerable set design. Why not simply randomize the answer choices and count as correct when the model answers both correctly? This seems to also achieve the purpose of enhancing the robustness of the evaluation. Isn’t this simpler and more direct? The unanswerable set should rather serve as an analysis tool, not as a key contribution.

  Secondly, the conclusion in Figure 8 that “models cheat on answerable questions” seems overly absolute. Did the author try asking the models to provide explanations before making a choice? This could allow for manual checks to determine whether the models score higher on the answerable set due to random selection rather than not understanding the image.

  Thirdly, I believe that for some models with poor performance on multi-image tasks, the conclusion drawn by the author may not hold, which can refer my first-round comments for further details.

• Concat Tokens for Evaluation: The author adds LLaVA-Next-34B as a single-image model to test multi-image input, concluding that “explicitly processing multiple images as input is beneficial, as it reduces information loss and minimizes challenges in referencing images.” I acknowledge that this reduces information loss, but it introduces the issue of long context input, which is likely much longer than the context length seen during training (similar to LLMs). This also brings up potential generalization issues. Therefore, the conclusion drawn from testing just one model may be biased. I think it is necessary to test additional models to reach a more comprehensive conclusion.

• More Models: Good work overall. I suggest adding to the paper, with some other new models such as InternVL2.

• Some Other Concerns: After carefully reading the paper, I share the same concern as Reviewer 1gpk. While I recognize that creating a stable benchmark for testing is a significant contribution, it is equally important to understand why certain models perform poorly and how improvements can be made (e.g., in architecture, data, or training strategies). This isn’t a request for the authors to train more models but rather to provide an analysis in this area. Although the author discusses this in Section 4.2, the focus is almost entirely on how to test. Therefore, I believe the paper lacks an analysis of how to improve multi-image capabilities, and this should be addressed.

Given these reasons and other reviewer comments, I plan to maintain my score.

Dear Reviewer QRSh,

Thank you for your time and thoughtful feedback on our paper. We hope our responses have addressed your concerns, and we kindly request that you consider updating the score accordingly. If there are any remaining issues, please let us know, and we will be happy to provide further clarification.

Thanks!

I just use InternVL2 as an example. I suggest the author to test some newer models, which will have greater reference value for the community.

We have already tested nearly 30 recent MLLMs, including the models specifically requested by the reviewer, some of which are released within two months. Additionally, we are committed to continually updating our leaderboard to include the latest models, ensuring the benchmark remains a valuable resource for the community.

Comparison with MathVerse

Below, we outline our one-to-one comparisons to emphasize the scope of our analysis relative to MathVerse.

• MathVerse: MLLMs rely more on diagram interpretation (DI) than seeing diagrams.
  MuirBench: MLLMs may rely on shortcuts and fail to identify unanswerable instances (Lines 460–465).

• MathVerse: MLLMs are moderately effective at perceiving implicit premises (IP).
  MuirBench: MLLMs perform relatively better on image-text matching and visual retrieval (Lines 429–431).

• MathVerse: MLLMs struggle to interpret explicit concepts (EC) from diagrams.
  MuirBench: Tasks requiring understanding multi-image contexts and inter-image relations are more challenging (Lines 448–450).

• MathVerse: MLLMs struggle to solve problems entirely using diagrams.
  MuirBench: MLLMs struggle to fully comprehend multi-image contexts. (Lines 372-377)

• MathVerse: Closed-source MLLMs perform better.
  MuirBench: Open-source models lag behind proprietary ones (Lines 374–375, Table 1).

• MathVerse: LLMs achieve competitive results compared to MLLMs.
  MuirBench: Multi-image understanding cannot be solved through dense captioning (Lines 1075–1079, Table 4).

• MathVerse: GPT-4(V) outperforms humans on the text-only version.
  MuirBench: Advanced MLLMs, such as GPT-4o, still fall short of satisfactory utility (Lines 372–373, Table 1).

• MathVerse: Mathematical training improves performance.
  MuirBench: Models trained with multi-image data achieves better performance (Lines 452–456, Table 1).

---

Conclusion

The above comparisons demonstrate that the scope and depth of our analysis are comparable to those in MathVerse and M3CoT, both of which are regarded as impactful benchmarks. Additionally, we provide novel insights tailored to multi-image understanding, which distinguishes MuirBench from prior works. We hope this clarifies the value and comprehensiveness of our analysis.

Thanks to the author for the detailed reply, I appreciate the author's efforts, but I still think there is large room for improvement in the article.

Concat Tokens introduces the issue of long context input：Thank you for the author's question, but I don't think that "a comprehensive analysis of specific model issues will be an independent work". Given that what the author has done is a benchmark work, it is equally important to establish a complete evaluation system and a comprehensive analysis. There seems to be no essential difference between the multi-image model and the single-image model defined by the author except for the training data level, so I think this testing method needs to be discussed in detail.

InternVL2: I just use InternVL2 as an example. I suggest the author to test some newer models, which will have greater reference value for the community.

beyond the scope of any benchmark paper : I am afraid I cannot agree with the author's such absolute description. Simply presenting a dataset that makes the model perform poorly may not be a complete contribution. Of course, I recognize the quality of the MuirBench built by the author, but the current version of the paper obviously lacks some exploration of the solution, which is also very important for benchmark papers. Here are some examples:

M3COT(ACL Oral) Section 5.4: Exploration

MMLU Section 5: Discussion

MathVerse (ECCV 2024) Section 3.2 Experimental

Dear Author,

Thank you for the explanation you provided. However, the comparison in the paper still does not fully address the limited insights provided by the article. First, the analysis presented still focuses mainly on the testing methods and simple conclusions drawn from the model performance analysis, such as "closed-source models perform better" or "multi-image training models perform better". These conclusions seem somewhat predictable and may not provide substantial new insights to readers. Since the effectiveness of closed-source models and expert models is widely accepted, I believe such a benchmark should provide more insightful conclusions.

Second, the paper currently contains only 20 LVLMs, which is less than the "nearly 30 LVLMs" claimed. While I admit that 20 is still a good number, it is obviously not up to date, and reviewer 1gpk also raised similar concerns.

I appreciate your efforts in the rebuttal stage, but like reviewer 1gpk, I believe that the current paper does not meet the standards for ICLR acceptance. Therefore, I decided to keep my score.

Best regards

These conclusions seem somewhat predictable and may not provide substantial new insights to readers.

We respectfully disagree with your assessment that the conclusions are predictable or lack substantial new insights. To the best of our knowledge, no prior work has provided the specific insights or empirical evidence presented in this paper. For example, we identify the weaknesses and strengths of multimodal LLMs on multi-image relations and tasks, which has not been addressed in any existing literature. Furthermore, the conclusions were derived through rigorous analysis of multimodal LLMs' multi-image understanding capabilities, which is novel and was explicitly designed to address the identified research gaps. These findings advance the understanding of multimodal LLMs and provide actionable implications for future research and real-world applications.

Notably, the quoted "simple conclusion", "close-source models perform better," is from the MathVerse paper you recommended. We are presenting our analysis in the same format for your understanding.

the paper currently contains only 20 LVLMs, which is less than the "nearly 30 LVLMs" claimed. it is obviously not up to date

Our original version included evaluations on 20 LVLMs. In response to your request, we have expanded our experiments to include an additional 7 models, some of which were released just days around the ICLR submission deadline.

concat images for testing is actually an unfair comparison. I think the concat tokens experiment should be used as the main test method

We respectfully disagree with your assessment that the comparison is unfair. We follow the common practice in prior work of processing multi-image inputs by concatenating images for models trained on single-image data [1]. This aligns with the intended input format of these models.

[1] Jiang, Dongfu, et al. "Mantis: Interleaved multi-image instruction tuning." arXiv preprint arXiv:2405.01483 (2024).

We appreciate the reviewer's insightful feedback. Below, we provide a detailed response to address the concerns and questions raised:

W1. detailed comparison with recent multi-image evaluations benchmarks

Thanks for pointing out MileBench. We will include it in the related work section. Different from our work centering on robust multi-image understanding, MileBench primarily focuses on multimodal long-context scenarios. We will update the paper shortly.

W2. The evaluation on MuirBench seems to require lots of computation resources

We intentionally designed MuirBench to maintain a manageable data size, ensuring that computational resources remain reasonable. Most of the evaluations reported in the paper, including those using the GPT-4 API and LLaVA/Mantis on a single A100 GPU, take less than half an hour to complete. This design choice ensures accessibility and practicality for researchers with varying computational budgets.

Q1. Do the authors have plans to expand or adapt MuirBench to include new tasks or modalities as LVLM capabilities evolve?

Yes. We agree that as multimodal LLMs evolve, new challenges and requirements will emerge. We plan to continually expand MuirBench, adding more diverse scenarios and tasks to support the ongoing development and evaluation of multimodal LLMs. This commitment ensures that MuirBench remains a relevant and valuable benchmark for the community.

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer.

W1/Q2. How is the performance on unanswerable questions measured.

“None of the other options” is included as an option for each instance. For unanswerable questions, we expect the model to select this option.

W2/Q3. the benchmark is unevenly distributed. how can one obtain one single score that can reflect the overall ability?

To facilitate fine-grained analysis of model performance, we have reported the performance on each task (Table 1) and each multi-image relation (Table 2) . This follows the common practice of many widely-used benchmarks, such as BLINK and MathVista.

To obtain rebalanced results, one can also report the task-balanced or relation-balanced model performance (i.e., the macro average). We provide the reweighted performance below. While the numbers differ, the trends remain roughly the same.

W3/Q1. how the 10 relations are chosen

We want to clarify that we did not decide or choose the multi-image relations to cover beforehand. Instead, we summarized the relations based on prior studies and the data collected, which reflect the common focus of the research community. During the development of MuirBench, we carefully reviewed the literature on multimodal models to collect multi-image data. The multi-image relations were then categorized in a bottom-up manner. Based on our expert manual observations, experience, and demonstrated examples, we believe our dataset is comprehensive enough to study common problems in existing multimodal LLMs.

Q4. it is claimed that MIRB is released later than this paper

While the paper is still under review, our benchmark has been publicly available for some time to support the development of multimodal LLMs. Our benchmark is released before MIRB.

Dear Reviewer fMCn,

Thank you for your time and thoughtful feedback on our paper. We hope our responses have addressed your concerns, and we kindly request that you consider updating the score accordingly. If there are any remaining issues, please let us know, and we will be happy to provide further clarification.

Thanks!

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer.

W1. Lack of data scale comparison

The current data size is sufficient to advance the development of VLMs. For example, datasets like Winoground (CVPR 2022), which contains only 400 examples, have significantly influenced VLM advancements. Similarly, although focused on different aspects, other widely adopted benchmarks—including WHOOPS! (ICCV 2023), LlaVA-Bench (NeurIPS 2023), Visit-Bench (NeurIPS 2024), ConTextual (ICML 2024), VibeEval, and Visual Riddles (NeurIPS 2024)—comprise 90, 500, 576, 500, 269, and 400 examples, respectively, and have been pivotal for evaluating VLMs.

We include comparisons with related benchmarks below. In summary, the size of our benchmark is comparable to prior benchmarks, and the total of 2,600 examples represents a comprehensive set. Importantly, the issue is not the size but the lack of a comprehensive benchmark for this critical problem: robust multi-image understanding. Our work provides the first comprehensive benchmark for evaluating multimodal LLMs' capabilities in this scenario.

W2. Specific explanation of comprehensiveness

During the development of MuirBench, we carefully examined over 30 related benchmarks and datasets to ensure broad coverage and relevance. The tasks and image relations in MuirBench are summarized based on prior studies and the data collected, reflecting the common focus and priorities of the research community. Guided by expert manual observations, extensive experience, and diverse demonstrated examples, we believe that MuirBench is comprehensive enough to address the most common challenges in existing multimodal models.

That said, we acknowledge that certain tasks or relations may not yet be covered. We are committed to maintaining and expanding this project, incorporating new tasks and relations as the field evolves. Continuous updates will ensure that MuirBench remains a valuable and up-to-date resource for advancing robust multi-image understanding.

W3. Are the ground-truth answers reliable?

The ground-truth labels are reliable. As detailed in Section 3.2, we implemented a rigorous manual filtering process to ensure the accuracy of the dataset. Multiple experts reviewed all labeled instances and filtered out those deemed uncertain, retaining only instances with correct answers. Notably, we reported human performance on the benchmark, which is over 93% accuracy, further demonstrating the quality of the annotations. Every instance in the benchmark includes a "no answer" option, allowing for the explicit handling of cases where the answer is genuinely uncertain. This design decision emphasizes data quality and supports fair evaluation of model capabilities.

We have provided detailed responses to address the concerns and questions raised by each reviewer. Below is a brief summary:

• Reviewer iBA5 and Yzea: The major request was to add a table comparing with prior benchmarks. We have added Table 2. Other clarification questions on computation cost and label reliability have been thoroughly addressed.

• Reviewer fMCn: The primary request was to add a balanced score. We have added it in Table 4. Other clarification questions on unanswerable options and the choice of relations have been thoroughly addressed.

• Reviewer QRSh and 1gpk:

  • Analysis scope: Compared our analysis scope with the papers named by the reviewer in a pairwise list. Added Section 5 to discuss the opportunities for model improvement.
  • More models: We added 7 models requested by the reviewer (Table 3).
  • Other minor points: We clarified the misunderstanding of Figure 5 and the unanswerable setting. We provided the balanced score in Table 4. The concat token analysis has already been presented in Table 6. The request for inputting images one by one is already presented in Table 7.