MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models

Thank you for your constructive comments and suggestions. We have revised our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.

---

Q1: Lack of fine-grained presentation of results across fields: Although the paper evaluates several models, it does not fully present results across different fields, making it challenging to analyze performance gaps in specific domains. This limitation may restrict insights into targeted improvements for specific models in particular fields.

A1: Following your advice, as shown in Table R1, we provide detailed results for the model's performance across the following 12 fields. In the Project-based learning (PBL) category, among the seven fields, most models exhibit slightly lower performance in finance and education, while achieving better results in art, sports, and philosophy. In the Multi-step reasoning (MSR) category, across the four fields, models generally perform worse in coding and statistics but demonstrate stronger performance in mathematics and physics. We have revised the paper and put these details in Appendix A.7.

Q2: Potential Overfitting to Specific Benchmark Tasks: Since the scoring model in MMIE is fine-tuned on data specific to the benchmark, it may overfit to the types of tasks and response styles within MMIE. This could limit its generalizability to other multimodal datasets or tasks, potentially reducing its effectiveness when evaluated in new contexts outside of MMIE’s scope. Given that the scoring model is trained on data similar to MMIE (12 domains), I am curious about its generalizability to other benchmarks. Does the scoring model work on other benchmark data, such as MMLU, which includes 30 domains? This would provide insight into whether the model’s evaluation criteria are adaptable to a broader range of multimodal tasks.

A2: ​​Thank you very much for your suggestions. Our scoring model is specifically designed for the interleaved benchmark and fine-tuned on the data we collected for our specific task. ​​Following your suggestion, we apply MMIE-Score to an OOD dataset to validate its generalization capability. However, the recommended MMLU is a QA dataset in text-only format, which is not applicable to our evaluation model. Similarly, the recommended MMMU is a VQA dataset with multiple-choice questions, where accuracy can be directly calculated, making it also unsuitable for our evaluation model. Instead, we conduct an experiment using 100 examples from the COCO dataset [1], a widely used image captioning dataset for natural image scenarios. We use MMIE-Score to evaluate the generated text descriptions based on input images, using Text Quality, Image-Caption Relevance, Contextual Consistency, Diversity, Specificity and Detail, and Stylistic Consistency and Correspondence as a 6-point scoring criteria, with the Emotional Impact for penalty. We also compare the scoring quality of GPT-4o, InternVL-2 and Anole. The correlation results between all scoring methods and human evaluations are shown in Table R2. MMIE-Score demonstrates excellent performance, significantly outperforming most models and ranking only behind GPT-4o, indicating good generalization capability.

In the future version, we aim to further optimize the training of the MMIE-Score model by expanding the dataset size, incorporating techniques to improve out-of-distribution robustness, and leveraging GPT-4o for data augmentation to enhance the model's cross-task generalization capabilities. We have revised the paper and put these details in Appendix A.8.

Table R2: Comparison of scoring LVLMs in COCO benchmark.

[1] Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects in context. ECCV, 2014

Table R1: Performance comparison across various fields for different LVLMs. Since the Multi-step reasoning category (i.e., mathematics, physics, coding, statistics) do not require image outputs, the scores of integrated LVLMs for this category remain consistent.

Table R3: Comparison with integrated LVLMs, evaluated on MMIE’s 1K subset. *: LLaVA only supports single-image input and all multi-image queries are thus skipped.

---

Q4: Visual Component Importance: Is it feasible to analyze the impact of visual content within the benchmark? For instance, comparing text-only generation with generation that includes images. Some examples in Figure 1 do not rely on image content but rather use images solely for illustrative purposes.

A4: Our dataset curation and filtering process ensures that all images included in the examples contribute meaningfully to the overall task. For instance, in tasks like visual storytelling, even when images serve only an illustrative purpose, they still impact the overall output quality. We conduct a comparative experiment to evaluate the difference in performance between interleaved generation (text and images) and text-only generation for the same input(GPT-4o + SDXL) and same evaluated model. The evaluation is scored using MMIE-Score and GPT-4o. As shown in Table R4, results showed that when the model outputs included both text and images, the overall quality was superior to text-only outputs. This ensures that the inclusion of images follows reasonable and well-defined criteria. We have revised the paper and put these details in Appendix A.6.

Table R4: Comparison of GPT-4o + SDXL’s average score with and without image generation, evaluated by GPT-4o and MMIE-Score.

Q2: Content Warning Handling: The paper includes a content warning but does not elaborate on it, as Ethical Statement highlights strict guidelines and lack of bias in the dataset.

A2: We have carefully controlled the processes of dataset construction and scoring to ensure they align with the principles outlined in our ethics statement. During dataset creation, we performed multiple filtering rounds to exclude samples containing sensitive personal information, inappropriate content, or harmful material. This includes content related to graphic violence, blood, explicit sexual material, disturbing horror themes, and any other content deemed inappropriate or harmful. For the manual annotation process, we strictly adhered to the scoring criteria described in Appendix A.3. Each annotation step was governed by precise guidelines to maintain consistency and accuracy. These measures were implemented to mitigate ethical risks and minimize potential biases in the dataset, ensuring it is as fair and responsible as possible. We have revised the paper and put these details in Appendix B.

---

Q3: Dataset Size: The 20K sample benchmark is substantial and statistically robust, enabling a comprehensive evaluation. However, validating the full dataset with 20K samples may be costly. Have the authors considered designating a "mini" subset for consistent yet quicker evaluation? For example, using 800 samples selected for scoring model training.

A3: Thanks for your valuable advice. The fully 20k sample size surely ensures the comprehensiveness of our dataset, but it does come with time costs. Following your advice, we resample evenly from each category and field, and construct a subset of 1000 samples. We re-ran the models on this subset and used MMIE-Score for scoring. As shown in the Table R2 and Table R3, the performance of the models on our subset is consistent with the results from the full dataset. We have revised the paper and put these details in Appendix A.5.

Table R2: Performance of the four open-source LVLMs supporting interleaved image-and-text input and output on MMIE’s 1K subset.

Table R1: Comparison of scoring LVLMs and traditional image-text alignment metrics across different models and categories.

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we’ve revised our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.

---

Q1: Potential Biases in Scoring Model: The reliance on a LLM as a scoring model may introduce biases inherent in the base model. A more thorough analysis of these potential biases, and how they might affect evaluation outcomes across different domains or tasks, might be needed.

A1: We present several comparison results of our scoring model with other baselines across different categories and model types (Interleaved and Integrated LVLMs). As shown in Table R1, the results show that although our model exhibits slightly varying performance across different categories, for example Cosine Similarity scores are higher for Situational Analysis (SA) and Multi-Step Reasoning (MSR) category and Pearson scores are higher for Project-Based Learning (PBL) category, the biases remain little. Overall, our MMIE-Score consistently outperforms other baselines. We have revised the paper and put these details in Appendix A.4.

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: While the proposed automated evaluation metric using a fine-tuned LLM is novel and promising, details on the construction of the dataset used for fine-tuning are missing in the paper.

A1: We randomly select examples from each field for human annotation, including the original inputs (images and questions), ground truth, and responses from evaluation models. For each category, we develop comprehensive and detailed criteria with scoring standards. We first annotate 20 examples, providing specific examples for each score as references. To facilitate the annotation process, we design a graphical annotation tool. Finally, we create a dataset of 800 examples with evaluation scores through human annotation, for fine-tuning the scoring model. We have revised the paper and put these details in Appendix A.3.

---

Q2: The claim L110 “The proposed scoring model ... has proven to be comparable to human evaluation.” is not well justified in its current form. For instance, while Table 5 shows that their proposed method has better similarity with human scoring, further details on how the human annotations were obtained are missing.

A2: We use the same construction method as the fine-tuning dataset to randomly select 200 samples with human-annotated scores for evaluating MMIE-Score. We then implement traditional image-text consistency metrics (e.g., CLIPScore) and LVLM-generated scores, then evaluate their scoring quality using several correlation metrics (e.g., Cosine Similarity, MSE, MAE, Pearson) based on human-annotated scores. As shown in Table 5, MMIE-Score outperforms all the above metrics, maintaining the best performance across the board. We have revised the paper and put these details in Appendix A.3.

---

Q3: The effectiveness of the proposed evaluation metric is a function of the evaluation model. Please provide further details on the fine-tuning method and dataset utilized for the evaluation model.

A3: We randomly select 800 processed examples from each field for human annotation. As is shown above, this dataset includes the original input (images and questions), ground truth, and responses from evaluation models. To facilitate the annotation process, we design a graphical annotation tool, as shown in the figure below. In this tool, the samples are displayed on the left, while a floating panel on the right allows annotators to score the samples. Annotators can check different criteria, and the final cumulative score is calculated accordingly. Checking a criterion indicates that the sample meets that specific criterion. We have revised the paper and put these details in Appendix A.3.

---

Q4: To better understand the role that the evaluation model plays in the pipeline, please provide qualitative examples of eval model responses corresponding to Fig 7,9,10,11,12 in the appendix.

A4: The evaluation model provides detailed and comprehensive feedback for each example based on the criteria and ground truth. We present the corresponding feedback responses for all the cases. We have revised the paper and put these details in Appendix C (see Figure 8,9,10,11,12,13,14,15).

---

Q5: Please provide details on how the human annotations were obtained for Table 5 to better support the claim in L110.

A5: We design a graphical annotation tool to facilitate the process. Human annotators score the samples by checking options on the floating panel on the right and assigning a final score. Each option represents a specific criterion, and checking the box indicates that the model's response meets that criterion. Specifically, the "emotional support" criterion is checked by default; if unchecked, the score is reduced by 1 point, as explained in Appendix A.3 of the paper. The final score is determined by the number of checked criteria. We strictly follow this scoring process to build our dataset. Using 800 samples from this dataset, we fine-tune the Intern-VL2 model to enhance its scoring capabilities in our task. We then evaluate its performance by comparing the model's scores with human evaluation scores on 200 samples, demonstrating the effectiveness of our scoring model. We have revised the paper and put these details in Appendix A.3.