MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with purple marks based on your comments.

Q1: some writing issues

A1: Thank you for your correction! We have made the necessary changes and thoroughly reviewed the entire text again. Let us know if there is anything else we can refine or clarify further.

Q2: Are there any duplicates between the samples in your dataset and the current multi graph dataset?

A2: Compared to other multi-image benchmarks such as MuirBench [1] and MilesBench [2] , MMIU features test data that is highly distinct, establishing itself as an independently developed benchmark.

To validate this, we calculate the visual similarity between the images in MMIU and those in MuirBench and MilesBench. Specifically, we use LPIPS [3] as the similarity metric. Many image editing works adopt LPIPS as a standard measure, and based on reported results, an LPIPS value below 0.1 is generally considered indicative of relatively similar images [4,5].

As shown in Table 11 below, when using a threshold of 0.1, the image duplication rates between MMIU and MuirBench or MilesBench are only 0.1% and 1.6%, respectively. This demonstrates that MMIU is a highly independent and novel evaluation benchmark.

Let me know if this aligns with your expectations!

​ Table11: LPIPS between MMIU and MuirBench, MilesBench

[1] MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

[2] MileBench: Benchmarking MLLMs in Long Context

[3] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

[4] Diffree: Text-guided shape free object inpainting with diffusion model

[5] Inversion-Free Image Editing with Language-Guided Diffusion Models

Q3: Why do models without multi-imgae SFT...

A3: Thanks to the reviewer for his suggestion. We miss this question before and now we give a detailed answer. For the original paper, InternVL2-Pro (only used video data for "multi-image SFT") and InternVL-Chat-1.5 surpassed some multi-image SFT models, but since the sizes of these two models (78B and 26B) are large, we add more models for comparison. Specifically, Below we comprehensively discuss the effect of multi-image data at different training stages. These contents are also involved in Response to Reviewer bVMZ (Q5) and we provide a more detailed analysis in Appendix B.3: Discussion on the Effectiveness of Multi-Image Training.

We consider the two most common training stages: pretraining and SFT (supervised fine-tuning). Based on the inclusion of multi-image data at these stages, we categorized models into four groups. As shown in Table 12 below, we find the following:

1. Models exhibit some generalization to multi-image tasks without multi-image data: Models trained without multi-image data, such as Mini-InternVL1.5-chat, still achieve reasonable performance on multi-image tasks, and even surpass the model trained with multi-image SFT: XGen-MM-Interleaved, which we attribute to their strong visual encoders leading to excellent image understanding capabilities. (e.g. Mini-InternVL1.5-4B even outperforms LLaVA-Next-34B on the multimodal comprehensive evaluation benchmark MME). However, even so, their performance is still lower than that of models trained with multi-image SFT such as Mantis, highlighting the importance of multi-image SFT

2. Multi-image SFT consistently improves performance: Regardless of whether multi-image pretraining is conducted, models like LLaVA-Next-Interleave and XGen-MM-Phi3-Interleaved show significant improvements when multi-image SFT is applied.

3. Combining multi-image SFT with pretraining yields better results: Models that incorporate multi-image data at both stages, such as Mantis-idesfics2, outperform those relying solely on SFT, such as Mantis-SigCLIP.

4. Multi-image pretraining alone is insufficient: Models that only perform multi-image pretraining, like XGen-MM-Phi3-Single and idefics2, underperform compared to those that only apply multi-image SFT (e.g. Mantis-SigCLIP), highlighting the importance of multi-image SFT.

If you have further questions, please do not hesitate to let me know immediately. We sincerely hope that you will consider raising your rating and we will do our best to answer your questions.

Table12: Performance comparison of various models with multi-image training data in different stage on MMIU.

[1] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

We would like to extend our heartfelt gratitude for the invaluable suggestions provided for this paper. In light of your feedback, we have diligently provided comprehensive and elaborate explanations. If you have any further inquiries, we are more than delighted to offer additional clarifications and conduct further experiments. We eagerly anticipate your response!

As the discussion period is nearing its end, we greatly value your feedback and are more than happy to provide further clarifications if needed. If you have any additional questions, please feel free to reach out immediately. We sincerely hope you will consider revising your rating, and we will do our utmost to address any concerns you may have.

In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with orange marks based on your comments.

Q1: ... more specific insights into how different architectures impact ...

A1: We sincerely thank the reviewer for the insightful comments. Indeed, architecture plays a significant role in influencing model performance. However, given the differences in training datasets for the tested models, it is challenging to conduct detailed controlled experiments. To address this, we select some of the best-performing models and analyze commonalities in their architectures that may contribute to better performance on multi-image tasks.

1) High resolution and dynamic resolution are crucial. Research suggests that increasing resolution and incorporating dynamic resolution during training significantly improve image understanding performance (e.g., SPHINX [1]). This is because low resolutions tend to miss critical visual details, and training on a single resolution size can lead to image distortions or inefficiencies in training and inference. For multi-image tasks, we observe that top-performing models (e.g., InternVL2-8B, Mantis-idefics2, LLaVA-Next-Interleave) adopt high resolutions (448, 384, and 336, respectively) alongside dynamic resolution training strategies. This enables the models to better comprehend visual inputs.

2) A strong visual encoder makes a difference. The InternVL series demonstrates outstanding performance on MMIU. For instance, both Mini-InternVL-Chat-1.5-4B and XGen-MM-Interleave-4B [2] use Phi3-mini [3] as their LLM, yet InternVL consistently performs better. Remarkably, Mini-InternVL-Chat-1.5-4B achieves performance on par with the larger LLaVA-Next-Interleave-7B. We attribute this partly to InternViT, the encoder used in InternVL models. InternViT is a powerful vision encoder that outperforms CLIP-based alternatives across multiple benchmarks. Thus, having a robust vision encoder is instrumental in enhancing performance on multi-image tasks.

We provide a more detailed analysis in Appendix B.3.

Thank you again for your constructive comments, which have helped us refine and deepen our analysis.

[1] SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

[2] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

[3] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Q2: While MMIU successfully identifies gaps...

A2: We sincerely thank the reviewer for the valuable suggestions. Indeed, MMIU does not involve training a specific model or proposing a new method to solve multi-image understanding tasks. However, training a model is not the only way to contribute. MMIU provides a significant contribution by testing a wide range of models on comprehensive multi-image tasks, conducting detailed analyses, and summarizing potential approaches to improve multi-image performance:

1. Sufficient multi-image SFT greatly enhances performance: For example, Mantis-idesfics2, after performing multi-image SFT on the base model idefics2, achieves a remarkable 17.8% accuracy improvement. Even without multi-image pretraining, models like Mantis-SigCLIP still achieve a relatively high accuracy of 42.6%, showing the effectiveness of multi-image SFT.
2. Combining multi-image SFT with multi-image pretraining yields better results: For instance, Mantis-idesfics2 outperforms Mantis-SigCLIP because the base model of the former underwent interleaved image-text pretraining, resulting in higher accuracy.
3. Multi-image pretraining alone may not be cost-effective: Models like idefics2 and xGen-MM-Single, despite undergoing multi-image pretraining, show suboptimal performance without sufficient multi-image SFT. This highlights the importance of coupling pretraining with fine-tuning for optimal results.
4. High resolution and dynamic resolution matter: Please refer to our response to Response to Reviewer JjzH (Q1) for further details.
5. A strong visual encoder is crucial: Please refer to our response to Response to Reviewer JjzH (Q1) for further discussion.

We provide a more detailed explanation of these findings in Appendix B.3. Once again, we thank the reviewer for their insightful comments. We fully agree that proposing and validating solutions is also an important contribution, and we will prioritize this as part of our future work. Thank you for helping us improve the scope and depth of our analysis!

Q3: ... more granular examination of how and why certain models excel or struggle with specific relationship types

A3: Thank you for your question. We conduct a statistical error case analysis on four representative models across different multi-image relationships, and we summarize the findings as follows:

1）Semantic tasks are more prone to perception errors compared to temporal and spatial tasks. This is attributed to the greater diversity inherent in semantic tasks, such as image retrieval, which poses additional challenges for multi-image understanding. In contrast, temporal and spatial tasks typically involve images with inherent consistency, like changes in time or viewpoint, making them less susceptible to perception errors.

2）Reasoning errors are more common in temporal and spatial tasks. Temporal tasks often require models to infer object dynamics over time (e.g., ordering tasks or next image prediction), while spatial tasks demand reasoning about spatial relationships and cognition (e.g., Image-Captioning-with-Spatial-Context). Current models consistently struggle with these complex reasoning requirements.

3）Spatial tasks are particularly affected by capacity-related errors. Tasks requiring advanced visual understanding, such as 3D pose estimation, exhibit a significant number of errors caused by the model’s limited capacity. This is likely due to insufficient training for specialized visual challenges, resulting in irrelevant or incoherent responses.

4）Open-source models are more prone to perception errors than closed-source models. This disparity is largely due to the capabilities of the visual encoder, which is critical for multi-image understanding. Enhancing the performance of visual encoders in open-source models could effectively reduce these perception errors.

5）Smaller open-source models often fail to follow instructions, particularly in long-context multi-image tasks. Their limited language model size impairs their ability to process and understand complex multi-image inputs. This highlights the importance of improving LLM capacity, especially for tasks requiring long-context understanding, to achieve better performance in multi-image applications.

We visualize the error types for different models under various image relationships in Figure 9 and provide a detailed analysis in Appendix B.4.

Thank you very much for your insightful suggestions on this paper. We have responded to each of these in great detail. And we kindly request that you consider updating the score accordingly. If you have any other questions, we are more than happy to provide additional clarification as well as experiments, and look forward to your reply !

As the discussion period is nearing its end, we truly appreciate your feedback and are eager to provide any additional clarifications if necessary. Should you have any further questions, please don't hesitate to contact me right away. We sincerely hope you will reconsider your rating, and we will make every effort to address your concerns.

Q3: How do you ensure the high quality of incorrect options generated by GPT-4o

A3: On the one hand, we explain that although GPT-4o has only moderate multi-image reasoning capabilities, it can still generate competitive incorrect options using the approach described in A1:

1. The task is to generate incorrect options, not solve the problem: Instead of requiring GPT-4o to solve the question, we provide it with the question and the correct answer to generate incorrect options. By presenting the correct answer, the difficulty of understanding the question is effectively reduced. This is because GPT-4o can identify which image or region the question focuses on and then perform targeted reasoning based on the correct answer. Once the question is understood, GPT-4o can generate incorrect options by reasoning about semantic differences between the options and the images.
2. Task-specific few-shot prompts for incorrect option generation: We design specific few-shot prompts tailored to different tasks to enhance the quality of incorrect options. For example, given a ground truth caption such as "The woman and man were eating some street food," we provide examples of incorrect options with varying quality scores, as shown in Table 9 below. GPT-4o is then tasked with generating high-quality distractors based on these examples.

This targeted approach allows GPT-4o to produce challenging and semantically relevant incorrect options, even with its moderate multi-image reasoning abilities.

On the other hand, We demonstrate that the incorrect options in MMIU are of high quality through two experiments and manual evaluation:

1. Performance without Visual Input: As shown in Table 3 of the main text, when images are removed and LLMs are tasked with answering questions using only the questions and options, the overall accuracy of multiple LLMs approaches random guessing. This indicates that it is difficult to distinguish between correct and incorrect options solely based on the options, demonstrating that the incorrect options are sufficiently challenging.

2. Semantic Similarity Analysis: We randomly select 500 samples whose incorrect options are generated by GPT-4o. Using OpenAI’s text-embedding-3-small model, we extract embeddings for both the correct and incorrect options and compute their cosine similarity. As a baseline, we randomly select incorrect options from other samples within the same task, ensuring that these incorrect options differ from the correct option of the current sample.

   As shown in Table 8 below, the results show that the semantic similarity between GPT-generated incorrect options and the correct options is significantly higher than the baseline. This indicates that GPT-generated incorrect options are more semantically similar to the correct options, making them more challenging.

​ Table8: Comparison of semantic similarity between wrong options and correct options

3. Human Evaluation of Incorrect Options: We use the 500 samples selected in experiment 2 and recruit 5 senior undergraduate students to evaluate the quality of the incorrect options. Each sample is assigned to all 5 annotators, who score the quality of the incorrect options based on the provided guidelines (examples shown in Table 9 below). Annotators evaluate the options on a standardized scale. The cost of this manual evaluation is detailed in Table 10 below.

   The results show an average score of 0.93 (after normalization) across all annotators, with no samples receiving a score of 0. Additionally, the average variance among annotators’ scores is 0.06, indicating strong agreement among annotators. This consistency demonstrates that GPT-4o-generated incorrect options in MMIU are widely considered to be challenging and effective distractors.

​ Table9: Example of incorrect option quality annotation

​ Table10: Cost of manual inspection

Q2: Limited Discussion on Practical Applicability...

A2: Thank you for your valuable suggestion. We greatly appreciate your insights, which help us emphasize the practical relevance of MMIU. As shown in Table 7 below, we discuss this from four perspectives:

1. Embodied Systems and Robotics Unlike other benchmarks that focus primarily on semantic-related tasks, MMIU includes a rich set of spatial tasks that enable extensive exploration of the visual capabilities required for high-level planning in embodied systems and their applications in robotics. For example, in the Egocentric Video Question Answering (EVQA) task, InternVL2 outperforms other LVLMs, demonstrating its potential for physical interactions with the environment.
2. Autonomous Driving MMIU also includes spatial tasks that are critical for autonomous driving, such as 3D depth estimation, which provides vehicles with accurate, real-time spatial awareness and situational understanding in dynamic environments. Our evaluations show that most LVLMs face challenges in this task, highlighting their current limitations in spatial reasoning capabilities.
3. Surveillance Systems MMIU incorporates semantic tasks such as Vehicle Re-Identification and Person Re-Identification, which are essential for improving the effectiveness and capabilities of surveillance systems. In these tasks, InternVL2 outperforms other LVLMs, including the closed-source model GPT-4o and the multi-image SFT model Mantis, showcasing its strong potential in the surveillance domain.
4. Autonomous Control Applications MMIU includes temporal tasks like GUI app recognition and GUI next action prediction, which have significant practical value for autonomous control systems relying on visual inputs. Our results show that Claude3.5 demonstrates relatively better performance in these tasks compared to other LVLMs, highlighting its growth potential for future development.

Conclusion MMIU serves as a comprehensive benchmark encompassing diverse tasks, providing critical insights into the strengths and limitations of current LVLMs across spatial, semantic, and temporal dimensions. By doing so, it contributes to driving advancements in real-world applications.

We provide further details in Appendix B.3: Practical Applicability Discussion. Once again, thank you for your insightful suggestions, which have been instrumental in refining our analysis and highlighting the broader significance of our work.

​ Table7: The performance of some models on MMIU's practical tasks

In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with green marks based on your comments.

Thank you for raising a series of insightful questions. Considering that GPT-4o was only used to generate part of the incorrect options in the dataset, we have summarized your concerns into the following three key questions。

• Q1: What measures were taken to ensure that the generated incorrect options are genuinely incorrect and sufficiently challenging.

• Q2: Limited Discussion on Practical Applicability。

• Q3: How do you ensure the high quality of incorrect options generated by GPT-4o

Q1: What measures were taken to ensure that the generated incorrect options are genuinely incorrect and sufficiently challenging.

A1: In generating incorrect options, we employ a variety of task-specific methods, carefully designed to ensure the quality and challenge of the distractors:

1. For tasks where the ground truth is a numerical value (e.g., object tracking): We use Python scripts to introduce random perturbations to the ground truth values within a fixed standard deviation. This approach generates plausible but incorrect options that remain close to the original values, increasing the difficulty of the task.
2. For tasks where the ground truth is a long text (e.g., multi-image captioning): We use a few-shot prompting strategy with GPT-4o. The prompts are designed to help GPT-4o understand what constitutes a challenging incorrect option. For example, given a ground truth caption such as "The woman and man were eating some street food," we provide examples of incorrect options with varying quality scores, as shown in Table 9 in the response to reviewer 4Ca4 (Q3) . GPT-4o is then tasked with generating high-quality distractors based on these examples. Experiments (as discussed in our A3 response) and manual inspections confirm that this method produces high-quality incorrect options.

Throughout this process, we tailor the generation of incorrect options to the unique characteristics of each task type. This ensures that the multiple-choice questions in MMIU are not naive or trivial. Our approach aligns with methodologies used in SEED-Bench [1] and MMT-Bench [2], which demonstrate that combining human design with LLM assistance reliably produces high-quality incorrect options.

[1] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

[2] MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

We express our sincere appreciation for the valuable suggestions regarding this paper. In response, we have provided thorough and detailed explanations. And we kindly request that you consider updating the score accordingly. If there are any further questions, we are delighted to offer additional clarifications and conduct further experiments!

As the discussion period is nearing its end, we sincerely thank you for your valuable feedback and are happy to offer any further explanations if needed. If you have additional questions, please feel free to reach out at any time. We genuinely hope you will reconsider your rating, and we are committed to addressing any issues you may have.

Q2: Potential position bias in the unanswerable set is unaddressed...

A2: Thank you for your question, and we apologize for any confusion caused.

In fact, we consider this aspect not only when designing the unanswerable set but also throughout the overall benchmark. The correct options in all tasks are always randomized in order. As shown in Table 6 below, for the unanswerable set, the placement of the unanswerable option is also consistently randomized. This approach minimizes the potential position bias issue you mention to the greatest extent possible.

Additionally, we provide further clarification in Section 4.1 (Evaluation Method) to ensure this point is clearly addressed.

Thank you again for raising this important consideration.

​ Table 6: The probability of unanswerable options appearing in different positions is almost equal

In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with red marks based on your comments.

Q1: My bigger issue about this paper is regarding the presentation of experimental results ...

A2: Thank you for your valuable suggestions regarding our writing. We implement numerous revisions in the resubmitted paper to improve its readability and clarity:

1. Revised Main Results Table We replace the main results table in the main text with the average accuracy of LVLMs across seven types of image relationships. In Section 4.2, we analyze the overall trends across different models, while in Section 4.3.1, we provide a detailed performance analysis of different models under varying image relationships.
2. Enhanced Focus in Figure 4
   • We highlight the clustering results of the task map from the original Figure 4(a) by emphasizing the shared characteristics of each cluster. These details, along with annotations, are now presented in Appendix Figure 11, as our primary focus is the analysis of model performance based on clustering.
   • For the original Figure 4(b), which presents model performance, we simplify the heatmap to create the current Figure 4 by categorizing tasks into clusters and grouping models. This improves readability while maintaining essential insights. Additional analyses are provided in Appendix C.2.
3. Separated Small Tables for Independent Analysis For smaller tables, such as Appendix Table 16 and Table 17, we extract key results (e.g., representative models' performance on different image relationships or tasks) and conduct more detailed discussions in Appendix B.3.
4. Expanded Visualizations for Important Content For example, we revise Appendix Figure 9 to include more detailed visualizations categorized by image relationships. This refinement addresses the broad scope of Section 4.4 (Error Analysis) by providing a more targeted and visually interpretable breakdown, thereby improving the presentation of key findings.

We appreciate your constructive feedback, which significantly contributes to enhancing the clarity and presentation of our work.

Q2: Is there any baseline performance by human

A2: We would like to thank the reviewer for their insightful comments. These suggestions have been very helpful, and we have given them detailed consideration. Below, we address each point in turn.

1. Recruitment of Human Evaluators: We continue to invite the five students employed at Response to Reviewer 4Ca4 (Q3) to continue the task of answering questions. Given time constraints, we select 41 tasks out of the original 52 which are suitable for human evaluation. Because some tasks in MMIU, such as tracking-related tasks, are more suited for machines to answer, while human evaluation is overly labor-intensive. From each task, we sample 10 instances, resulting in a total of 410 samples for human evaluation. For each sample, we assign 5 students to answer. The average accuracy across 7 relationships is then calculated, as shown in Table 2 below. The manual effort involved in this process is recorded in Table 3 below.

2. Evaluation Results: The results show that the current model performs relatively well for Low-Level image relationships and High-Level (subjective) tasks, approaching human scores. However, the overall performance still lags behind human performance, particularly in spatial relationships, where the gap between model and human ratings is most pronounced. This highlights the current limitations of the model in handling multi-image tasks.

3. Future Work: We will expand the sample size for further human evaluation. For some visual tasks, we plan to invite more senior annotators and use specialized visual models (e.g., Depth Anything) to test the accuracy of these tasks. This will provide a more comprehensive evaluation of MMIU's theoretical upper bounds.

Once again, thank you for your valuable feedback. If the reviewer has any further questions or concerns, please do not hesitate to ask, and we will do our best to address them.

​ Table2: The cost of human annotators.

Table 3: Comparison of human results and GPT-4 results across 7 image relationships.

Thanks for your interest in our research, which have been instrumental to our work. We are deeply grateful for your thorough and constructive review! If there are any further questions, we are delighted to offer additional clarifications and conduct further experiments. We eagerly look forward to your reply!

In order to observe the changes in the PDF more clearly, we have marked the changes in the new PDF with blue marks based on your comments.

Q1: MMIU ... This may require a lot of computing resources

A1: To mitigate the computational cost caused by the large number of tasks, similar to MathVista [1], we construct a smaller test set, testmini, by randomly sampling 20 examples from each of the 52 tasks, resulting in a total of 1,040 examples. While preserving the diversity of MMIU, our experiments show that the results on testmini effectively reflect those on the full test set. These findings appear in Table 3 in the main text and Table 15 in the appendix, where we report results on both the full test set and testmini.

Additionally, we perform a quantitative analysis of the testing cost for testmini. As shown in Table 1 below, most models complete the evaluation on a single A100 GPU within one hour. Even for the closed-source model, GPT-4o, the testing cost remains manageable, requiring approximately $10 and one hour to complete.

Moving forward, we plan to evaluate more models on testmini and provide a convenient interface to enable the community to test and compare models on MMIU seamlessly.

Once again, thank you for your thoughtful suggestions, which are instrumental in improving our work.

​ Table1: Resource consumption of some models in MMIU's testmini set

[1] MathVista: Evaluating Math Reasoning in Visual Contexts

Q2: The paper shows that ... lack of practical significance

A2: Thank you for your constructive feedback. We have carefully considered your comments and would like to address them in detail below:

1）Multi-Image Training for Temporal and Spatial Relationships in LVLMs We acknowledge that earlier single-image models like LLaVA and MiniGPT-4 primarily focused on modeling semantic relationships. However, with the advancement of large vision-language models (LVLMs), many now incorporate multi-image data during training to capture temporal and spatial relationships. For instance:

• Mantis utilizes Mantis-Instruct [1] for multi-image supervised fine-tuning (SFT), which includes datasets like VIST [2] and NExT-QA [3], emphasizing temporal relationships.
• LLaVA-Next-Interleave employs M4-Instruct [4] for multi-image SFT, incorporating embodied tasks such as 3D grounding that highlight spatial reasoning.
• InternVL2 utilizes OmniCorpus [5] during pretraining, a dataset with 10 billion interleaved image-text pairs, encompassing diverse semantic, temporal, and spatial relationships. Additionally, in its Stage 2 training, it incorporates video data (e.g., EgoTaskQA [6]), enabling it to effectively handle tasks involving both spatial and temporal relationships.

These advancements demonstrate the growing focus on multi-image capabilities, aligning with the evolving needs of LVLMs.

2) Practical Significance of MMIU Task Design We also emphasize the practical value of MMIU’s task design. Beyond general high-level semantic tasks, MMIU incorporates tasks with real-world applications, relevant to fields like robotics and autonomous driving. As shown in Table 2 below, we analyzed selected tasks and model performances, highlighting the following:

• MMIU evaluates high-level visual capabilities for embodied systems, such as robotics. For example, InternVL2-Pro demonstrates strong potential in egocentric tasks like EVQA.

• MMIU includes spatial reasoning tasks critical for autonomous driving. Results reveal that current LVLMs face challenges in tasks like 3D depth estimation.

• MMIU addresses semantic tasks relevant to surveillance applications. InternVL2-Pro excels in Vehicle and Person Re-Identification, outperforming other LVLMs in these scenarios.

• MMIU incorporates temporal tasks for autonomous control, where Claude 3.5 shows potential in GUI-related applications.

We provide a more detailed discussion of these findings in Appendix B.3: Practical Applicability Discussion.

Once again, thank you for your insightful suggestions, which have been instrumental in refining our analysis and emphasizing the broader significance of our work.

​ Table2: The performance of some models on MMIU's practical tasks

[1] Mantis: Interleaved Multi-Image Instruction Tuning

[2] Visual Storytelling

[3] NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions

[4] LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

[5] OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

[6] EgoTaskQA: Understanding Human Tasks in Egocentric Videos

Q3: The authors ... data processing techniques used

A3: Thank you for pointing out this issue. We have supplemented our explanation with more detailed descriptions as follows:

In Section 3.2, we outlined the general process of data collection, which involves the following steps:

1. Defining tasks based on image relationships. We first define specific tasks corresponding to the relationships among images. Detailed explanations of these tasks and their descriptions are provided in Table 7 in the Appendix.
2. Identifying datasets that reflect the tasks. We then identify datasets suitable for each defined task, primarily through manual collection from sources such as Kaggle, GitHub, and Papers with Code. The specific data sources for each task are listed in Table 7 of the Appendix.
3. Transforming datasets into multiple-choice questions. Next, we convert the collected datasets into a unified multiple-choice format:
   • First, the datasets are standardized into a unified metadata format, with an example provided in Table 9 of the Appendix.
   • Then, we generate incorrect options based on the ground truth. For tasks where the ground truth is numerical (e.g., object detection), we use algorithmic perturbations to create distractors. For tasks involving text generation (e.g., captions), we employ a few-shot prompting approach with GPT-4o to generate high-quality distractors. For instance, given the ground truth "The woman and man were eating some street food," we provide three examples of distractors with different quality scores in Table 9 in the Response to Reviewer 4Ca4 (Q3), guiding GPT-4o to generate high-quality incorrect options.

Detailed descriptions of the key steps in data generation, such as metadata standardization and distractor generation, are provided in Appendix A.4.

We hope this additional information clarifies our data collection and processing methodology. Thank you again for your valuable feedback, which has helped us improve the clarity of our work.

Q4: ...can you give some performance of pure language models.

A4: Thank you for your suggestions. We have supplemented this part with additional experiments, as detailed below.

1. Performance without Visual Input As shown in Table 3 below, when images are excluded and only questions and options are provided, the scores of all LLMs are nearly equivalent to random guessing. This suggests that the distractors generated by GPT-4o are sufficiently challenging and meaningful. Furthermore, it highlights that MMIU tasks require visual information, demonstrating their distinct multimodal characteristics.
2. Caption-Based Reasoning We also explore an approach where GPT-4o extracted captions for each input image, and then LLMs were tasked with answering the questions based on the captions and options. As shown in Table 4 below, this approach led to significant improvement compared to random guessing. However, it still falls short compared to direct inference using LVLMs. For instance, GPT-4o’s direct inference achieves an overall accuracy of 55.5%, outperforming the caption-based reasoning accuracy of 48.7%. This finding underscores that accurately understanding the semantic information in images is essential for multi-image understanding tasks, while some tasks in MMIU require more low-level visual information to be effectively solved.

We provide a more detailed analysis of these findings in Appendix B.3.

Thank you again for your valuable feedback, which has helped enhance the depth and clarity of our evaluation.

​ Table3: Some LLMs performance on MMIU

​ Table4: Some LLMs with caption generated by GPT-4o performance on MMIU

Q5: ... add interleave data or video data in the pre-training or post-training stage.

A5: Your question is highly insightful, and we have conducted additional tests with a broader range of models to analyze the role of multi-image data at different training stages.

We consider the two most common training stages: pretraining and SFT (supervised fine-tuning). Based on the inclusion of multi-image data at these stages, we categorized models into four groups. As shown in Table 5 below, we find the following:

1. Multi-image SFT consistently improves performance: Regardless of whether multi-image pretraining is conducted, models like LLaVA-Next-Interleave and XGen-MM-Phi3-Interleaved show significant improvements when multi-image SFT is applied.
2. Combining multi-image SFT with pretraining yields better results: Models that incorporate multi-image data at both stages, such as Mantis-idesfics2, outperform those relying solely on SFT, such as Mantis-SigCLIP.
3. Multi-image pretraining alone is insufficient: Models that only perform multi-image pretraining, like XGen-MM-Phi3-Single and idefics2, underperform compared to those that only apply multi-image SFT (e.g. Mantis-SigCLIP), highlighting the importance of multi-image SFT.
4. Models exhibit some generalization to multi-image tasks without multi-image data: Models trained without multi-image data, such as Mini-InternVL1.5, still achieve reasonable performance on multi-image tasks, attributed to their strong visual encoders. However, their performance lags behind models with multi-image SFT.

We provide a more detailed analysis in Appendix B.3: Discussion on the Effectiveness of Multi-Image Training.

Thank you for your valuable feedback, which helped us deepen our analysis of multi-image training effectiveness.

​ Table5: Performance comparison of various models with multi-image training data in different stage on MMIU.

[1] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Thank you for your feedback. We have updated the latest version of the paper with additional content based on your suggestions, including the Testmini settings and an analysis of the role of multi-image data at different training stages (please refer to the Paper Update for specifics).

For models that have undergone multi-image training, such as Gemini-Flash and Qwen2-VL, we have now tested over 30 LVLMs, which provides a more comprehensive evaluation compared to other benchmarks. This includes models like LLaVA-Interleave and InternVL2, both of which have been trained with multi-image and video data. Additionally, we have included more recent models such as XGen-MM, LLaVA-Interleave, and Mini-InternVL. Absolutely, We plan to test even more models and provide convenient testing interfaces for the community to test on their own in the future.

If you have any further questions, please don’t hesitate to let us know. We will make every effort to address them. And we kindly request that you consider updating the score accordingly. Thank you again for your valuable suggestions, which we believe have significantly enhanced the quality of the paper.

We would like to express our sincere gratitude for your insightful feedback. In fact, we have already conducted tests on Gemini1.5Flash, as well as on InternVL2, which is similar to Qwen-VL2. Both models have been extensively trained on video data and interleaved image-and-text datasets. The detailed results are summarized in Table 1 below.

Notably, We have evaluated over 30 LVLMs and LLMs on MMIU, compared to other benchmark papers, we think this is a fairly comprehensive evaluation. As for Gemini and InternVL: Gemini1.5Flash, InternVL2-Pro, and InternVL2-8B have achieved superior performance compared to other models, highlighting the importance of multi-image training. However, it is also important to note that these models still fall short when compared to human scores (referencing Response to Reviewer avPS (Q2)). This suggests that there is still significant room for improvement in the models' multi-image capabilities.

In Appendix B.3, we provide a detailed analysis of both architectural and data-related approaches to enhance multi-image performance. Additionally, in Appendix B.4, we include a thorough error case analysis. We hope these findings can contribute to the development of more advanced multi-image models.

Given the contributions of our work and the insights shared, we kindly ask if you might consider raising your score accordingly. If you have any further questions, we would be more than happy to provide additional clarifications.

Table 1: Some model performance and human performance on MMIU

As the discussion period is nearing its end, we would like to sincerely thank you for your valuable feedback. If you have any further questions, please do not hesitate to contact me immediately. We are fully committed to addressing any concerns you may have.