MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

Concern 1 However, the corresponding questions are only focused on image content recognition and simple single-step reasoning, showing limitations in task difficulty and the requirement for understanding capabilities of large models.

Thank you for your valuable feedback. You are correct that the current questions are relatively simple, primarily focused on image content recognition and basic single-step reasoning. However, the poor performance of current MLLMs on these tasks highlights that these models still have a long way to go in terms of high-resolution perception and general reasoning capabilities. Once these simpler problems are better addressed, it will be more valuable to explore more complex question forms.

We also appreciate your suggestion, and we are indeed working on designing more challenging tasks and evaluation metrics. Over the past period, we have also been considering other question formats. For example, we are preparing to release an alternative evaluation version of the dataset, where only the questions are provided without answer choices. In this setup, we will assess the models’ performance using exact match or GPT-match methods, which will prevent models from relying on the provided choices.

In our initial tests, we selected 50 samples from each task and used the prompt “Please respond to the question with a single word or phrase,” encouraging models to generate direct answers. The table below shows the results of these tests, where all models showed significant performance degradation without the choices:

As you can see, when no choices are provided, the performance significantly drops. However, we also observed that for tasks like OCR, where the answer is fairly unique, exact match methods are likely to yield correct results. For tasks with more open-ended responses (e.g., "Where is the person in yellow clothes located in the image?"), it is much more difficult to assess accuracy using exact matches, as responses like "top-right of the image" and "in front of the store" could both be correct.

To address this challenge, we have incorporated GPT-4o to align the model's responses with the correct answer in terms of meaning (shown in the last row). This alignment approach has led to a performance improvement, but it still lags behind results with the choice-based setup. Nonetheless, we believe that the "Machine match" strategy, which involves aligning model outputs with the intended meaning, will be an important evaluation approach moving forward. Under this strategy, GPT-4o’s performance only reaches about 30%, further increasing the difficulty of our questions.

We will continue to explore additional evaluation strategies in the future. Once again, thank you for your constructive suggestions!

Concern 3: The author should discuss if MME-RealWorld is sufficiently diverse or simply collecting several similar images and QAs. More comparisons on the distribution of image types, question types, and answer types between MME-RealWorld and other benchmarks are welcomed.

Thank you for this excellent suggestion.

To address your first question:

1. Diverse Data Sources: As outlined in Section B.1.1, our benchmark draws from a significantly broader range of publicly available datasets. Specifically, our data originates from over 17 distinct sources, covering a variety of scenarios, including remote sensing, surveillance, autonomous driving, charts, and natural images. This approach differs from existing benchmarks, which primarily focus on natural scenes or sample data from a limited pool. For instance, MMBench primarily samples from existing VQA benchmarks like CoCo and TextVQA, leading to substantial overlap with prior datasets. In contrast, our data sources are more varied and avoid substantial overlap with known MLLM benchmarks.

2. Uniquely Curated Content: Additionally, we include data manually sourced from the internet, incorporating unique scenarios, including purely Chinese-language settings. Such data is scarce in existing datasets and provides an element absent from current benchmarks. In Figure 14 of the revised PDF, we present the various categories of images in our dataset.

Regarding question and answer formats, we largely follow the mainstream approach by adopting multiple-choice questions, so there are no major differences in this respect. Recognizing that image diversity may not have been adequately discussed in the initial draft, we have added a related discussion to further clarify this aspect.

Concern 4: It is better to have a discussion of knowledge leakage in LVLMs on the MME-RealWorld benchmark, as in [1].

Thank you for this suggestion. We appreciate the reference to [1], which we have cited in our paper as it provides valuable insights. We are mindful of potential knowledge leakage; however, MME-RealWorld benefits from a distinct advantage due to its entirely manual annotation process, which greatly minimizes this issue. Our strict annotation guidelines require annotators to focus on subtle details within each image, as shown in Figure 1. As a result, even when presented with the image, models struggle with these questions, and without the image, their responses are essentially random (closed-source models default to selecting “E” or rejecting to respond).

We have added a related discussion to the paper to enhance readers’ understanding of our approach.

Concern 1 The holistic MME-RealWorld is too large for developing LVLMs. The author should formulate a mini set of MME-RealWorld for community convenience. I suggest maintaining a balance of task types and difficulty levels while reducing the overall size

This feedback is incredibly valuable! To facilitate the evaluation of high-resolution perception abilities in multimodal large language models, we plan to sample 50 examples from each task within each domain, resulting in a test set of 2,150 QA pairs. We believe, as per your suggestion, that this smaller-scale test set will have a greater impact.

Above are our initial tests with some models. While it is difficult to perfectly retain model performance on the original dataset due to the sampling process, the overall task difficulty has actually increased, as we’ve reduced the number of samples for simpler tasks such as OCR, charts, and basic reasoning tasks. For instance, the performance of Qwen2-VL dropped from 56.4 on the full dataset to 46.7 in this reduced test set. This demonstrates how the smaller test set not only maintains a balance of different tasks but also keeps the tests challenging, aligning with your suggestion to "maintain difficulty levels." We believe this compact test set is an excellent tool for researchers to perform smaller-scale, focused evaluations and comparisons of their models. It strikes a good balance between practicality and robustness, making it more accessible for the community while still providing meaningful insights into model performance.

Thanks for your solid response. My concerns are well solved. I will keep my rating.

Concern 2 In Lines 479-480, the authors claim that ‘This indicates that most models’ visual perception modules fail to identify the objects in the images corresponding to our questions.’ This is not clear and convincing. I am wondering how do the authors attribute the higher frequency of ‘E’ outputs to the limited image detail perception of MLLMs?

To clarify, the statement regarding the high frequency of 'E' outputs is meant to highlight that the correct answers in our dataset have a relatively low proportion of 'E' as the answer choice. Therefore, when models predominantly select 'E', it suggests that they are unable to correctly identify the relevant objects in the images corresponding to the questions. This tendency indicates a weakness in the model’s visual perception ability, as it resorts to a default or non-informative answer (such as 'E') when unable to perceive the correct object or answer. We see this as an indication of limited visual perception capacity in the model, which is unable to effectively map the visual input to the correct answer.

Concern 3 In Lines 481-485, the authors analyze the ‘limitations of MLLMs in understanding dynamic information’ without providing any qualitative or quantitative result. It would be more convincing to provide some evidences to support such statement.

To address the reviewer’s suggestion for additional evidence supporting the limitations of MLLMs in understanding dynamic information, we have included detailed results (refer to Table 6 in the revised PDF) derived from autonomous driving and monitoring tasks focused on intention prediction. These results reveal that even the strongest models perform poorly on intention prediction tasks, with average accuracies not exceeding 30%. Intention prediction is a fundamental capability in fields like autonomous driving, and it is essential for MLLMs to evolve into robust world models capable of understanding dynamic contexts. This evidence further underscores the limitations of current models in this critical area.

Concern 4 Figure 2a shows the different domains and tasks. It would be nice to add a figure to show the distributions of different domains and tasks for better readability.

We appreciate the reviewer’s suggestion to improve the readability of the distribution of domains and tasks. To address this, we have included a comprehensive figure (Figure 15 in the revised PDF) that provides a detailed breakdown of the perception and reasoning tasks, along with the sample counts for each subtask. We believe this enhancement significantly improves the clarity and readability of our paper.

Concern 5 In Lines 93- 94, the authors claim ‘we collect a total of 13, 366 high-resolution images from more than 300K public and internet sources.’ I have doubts about the number of the data sources. 300K is a huge number, and how long does it take to collect the images?

Thank you for your question. The data collection process involved five multimodal research experts working for one month. In some areas, such as autonomous driving, remote sensing, and street view data, high-resolution images were readily available, so the main task was content filtering rather than image collection, which helped reduce the time needed. The majority of the effort went into filtering images that met the resolution and content requirements.

For tasks with fewer public data sources, such as Chinese scene data, researchers had to manually search for and download the necessary images. Although this portion of the data was smaller, it still accounted for about 1/3 of the total time spent on data collection and organization.

Concern 6 For better readability, it would be better to add a paragraph title for the last paragraph in Section 3.3.

Thank you very much for your suggestion. We have added the title "Appendix Overview: Supplementary Analyses and Detailed Results" to improve readability in the last paragraph of Section 3.3.

Concern 1: Although the dataset contribution is great, it is hard to find some insightful analysis from this paper. The authors only summarize the benchmark results. It would be better to provide more in-depth analysis and highlight future directions.

Thank you for this highly relevant suggestion for enhancing the paper. As one of the few ultra-high-resolution benchmarks, we have conducted preliminary analyses on the trade-offs between computational cost and model performance to inspire future research directions. Specifically, we have highlighted the following insights:

1. The perceptual abilities of current models are limited, resulting in subpar performance on our benchmark.

2. Proficiency in Chinese is a significant limitation for existing models, though Llama 3, as a base model, demonstrates comparatively strong performance in this area.

3. Simple image-splitting techniques are insufficient to fully address high-resolution perception challenges.

4. Models still struggle with tasks involving the understanding, prediction, and inference of objects’ next actions, such as anticipating the intentions of vehicles or pedestrians in images, reflecting a gap relative to world models.

5. High-resolution models face computational limitations; excessive image patching demands considerable resources. Striking a balance between high-resolution perception and computational efficiency remains a critical issue.

Recognizing that our current analysis may be insufficient, we have extended our evaluation with additional insights into model behavior, particularly focusing on instances where models choose to refuse answering. This is directly related to model safety and utility.

1. Correct E (%): This metric measures the percentage of times a model correctly predicts "E" when the ground truth is "E," helping evaluate whether the model can recognize questions that genuinely require a refusal to respond.

2. E Ratio in Wrong Predictions (%): This indicates the proportion of wrong answers labeled as "E" among all incorrect predictions, offering insights into the model's tendency to choose "E" when uncertain.

3. # Predicted E: This counts the total number of times a model predicts "E" within the current split, providing a straightforward view of "E" predictions overall.

The results are shown in Table 21 of the revised pdf or [Response to the reviewer a7Si [concern 2]]. Below are the summarized results, from which we draw several key insights:

a) Less capable models, such as SliME-8B, rarely opt for "E" and instead tend to select answers they consider correct, leading to low rates of both correct "E" predictions and "E" frequency across domains.

b) More advanced open-source models, like InternVL, show "E" frequencies comparable to closed-source models on simpler tasks, such as OCR or Diagram and Table interpretation. In these easier tasks, most models are confident in their answers, so the frequency of selecting "E" remains low.

c) On more challenging tasks, such as Remote Sensing and Monitoring, all models have higher error rates. Notably, proprietary models like GPT-4o and Claude35 exhibit both higher accuracy in predicting "E" and a greater frequency of selecting "E" compared to InternVL, which maintains a relatively low "E" frequency (10%-35%) even on difficult tasks.

In summary, for simpler multimodal tasks, the safety profiles of advanced MLLMs and proprietary models are comparable. However, on more complex tasks, proprietary models demonstrate a significantly higher level of safety by opting for "E" when uncertain, aligning better with human values by avoiding misleading answers. Given that open-source models currently undergo limited alignment with human preferences, this presents an important direction for future research and development.

Overall, we have made every effort to provide a comprehensive analysis of model performance in high-resolution perception, Chinese-language scenarios, computational overhead, dynamic information recognition (especially critical for Embodied AI and autonomous driving), and model safety. In doing so, we have identified several key challenges in these areas and highlighted promising avenues for future research.

If the reviewers can point out specific areas where our experiments may not have been sufficiently thorough, or where a more in-depth analysis is needed, we would be happy to provide additional details or further explore those aspects.

Concern 1: The authors mentioned, "the perceptual abilities of current models are limited."

It is indeed challenging to completely separate a model’s capability to process input images from its inherent perceptual ability, as these two aspects are closely coupled. However, in the case of high-resolution images, the model's capacity to process inputs seems especially crucial. For example, as shown in the table below, Mini-Gemini-7B-HD and LLaVA1.5-7B use similar LLM architectures and have comparable training data, yet Mini-Gemini-7B-HD exhibits far superior high-resolution perception capabilities. This demonstrates the critical importance of handling higher-resolution data effectively. As a result, most modern MLLMs have incorporated various image-splitting strategies to accommodate larger maximum resolutions.

Nevertheless, simply supporting higher resolution is not a complete solution to high-resolution perception challenges. For instance, while Intern-VL2 has a higher input resolution limit than Qwen2-VL, its overall performance is slightly lower. This implies that the ability to handle larger image resolutions alone is insufficient for robust high-resolution perception. The model’s inherent capabilities (such as information extraction and comprehension) also play a vital role.

Additionally, current MLLMs do not perform well on MME-RealWorld, and as we noted in the paper, computational efficiency is an issue. Efficiently handling ultra-high-resolution images remains an open question.

Given the diversity of vision encoders (many of which are not open-sourced) and the fact that many models utilize multiple vision encoders, it is challenging to assess the quality of each encoder independently. Instead, we list the maximum resolution each method can handle in the table below, as this feature better characterizes a model’s ability to process high-resolution image inputs.

Concern 2 Is it possible that the errors (high frequency of ‘E’ outputs) are caused by some other limitations of MLLMs

This is an excellent question. In fact, as seen in Table 21, proprietary models tend to output “E” (for “none of the above”) at a slightly higher frequency, while weaker models, like SliME, are less inclined to choose “E.” We primarily attribute the “E” output to situations where the model either refuses to respond due to uncertainty or chooses “E” when it cannot select a correct answer. Importantly, this behavior is not necessarily a defect; refusing to respond when uncertain is much better than providing a potentially incorrect answer, where the latter could more confuse users. Currently, open-source models often lack robust alignment processes, making them more likely to select a specific answer instead of “E,” even if the question exceeds their capacity. Therefore, we believe that the frequency of choosing “E” is closely tied to the model's safety features.

Of course, your suggestion that “E” outputs could result from poor reasoning capabilities is also valid. If a model consistently chooses “E,” it may indeed lack relevant reasoning or perception abilities. However, we have not observed a weak model that frequently outputs “E” in our experiments. Thus, our conclusions remain focused on the points made in the first paragraph and the experiments described in Concern 1. If the reviewer has encountered cases where models with poor reasoning capabilities frequently opt for “E” or exhibit similar behavior, we would greatly appreciate any shared insights to enable a more detailed discussion.

Concern 1: why these areas are chosen over others.

Thank you for your insightful feedback. We appreciate the recognition of our dataset's value in advancing QA-focused benchmarks.

Practical Value: Instead of choosing data sources like COCO, our primary goal was to facilitate understanding of high-resolution, real-world scenarios, which are closely related to fields like autonomous driving and surveillance. In recent studies, MLLMs have increasingly been applied to these realistic domains, which pose unique challenges for multimodal large language models (MLLMs). The domains we selected specifically demand high image resolution, allowing MLLMs to demonstrate progress in these high-resolution settings. We have thus prioritized areas where MLLMs currently have the potential for meaningful impact.

Benchmark Objective: Our benchmark is not intended solely for autonomous driving or surveillance applications. Our core aim is to evaluate MLLMs’ fundamental perception and reasoning abilities within these domains. Therefore, our tasks encompass various scenarios that our researchers deemed suitable for assessment through multiple-choice QA formats. We intentionally excluded more specific or complex tasks, such as robotic arm trajectory prediction, to maintain the focus on fundamental capabilities.

Additionally, as suggested, including a broader range of embodied understanding tasks—such as those in robotics—would expand the dataset’s scope. In future iterations, we plan to extend the dataset to additional domains, including human-robot interaction and real-time monitoring applications, to promote a more balanced representation and reduce domain-specific bias.

Concern 2: "While tackling everything can be challenging, my worry is that this dataset will introduce bias toward certain domains while trying to address limitations of other benchmarks."

As mentioned in Concern 1, the tasks we included are fundamental perception and reasoning tasks designed by our expert researchers. They do not cover complex tasks such as dynamic tracking in surveillance images or trajectory prediction in autonomous driving. Thus, they remain within the realm of basic perception/reasoning capability testing for MLLMs and we try our best to make them not interfere with specific downstream tasks strongly.

Furthermore, researchers have flexibility in selecting splits for evaluation. For instance, we also included standard MLLM tasks such as OCR and chart recognition, which involve higher-resolution images and improved annotation quality compared to traditional MLLM benchmarks. We believe these tasks are also valuable for advancing foundational research in their respective fields and will have a meaningful impact on MLLM development in these areas.

Concern 3 The use of multiple-choice challenges, while popular, remains a somewhat limited way to assess model capabilities.

Thank you for your insightful suggestion. We greatly appreciate your input, as it is crucial for improving the quality of our dataset. Over the past period, we have indeed been considering alternative question formats. In response, we are preparing to release an additional evaluation version of the dataset, where only the questions are provided, and no answer choices are given. This will allow us to assess model performance using methods such as exact match or GPT-match, which can help prevent models from relying on the provided choices.

In this approach, we selected 50 samples from each task and used the prompt, “Please respond to the question with a single word or phrase,” to encourage the models to generate direct answers. We then compare the model’s output to the correct answer to evaluate its performance. The table below shows the results from our initial tests, where all models experienced a significant drop in performance when no choices were provided:

As we can see, the performance drops notably without answer choices. However, we also observe that for tasks like OCR, where the answer is more unique, an exact match method is likely to yield the correct answer. For other tasks, especially those involving more open-ended responses (e.g., "Where is the person in yellow clothes located in the image?"), it becomes more difficult to measure accuracy using exact match, since answers like "top-right of the image" and "in front of the store" might both be valid. To address this challenge, we also experimented with using GPT-4o to align the model’s response with the intended meaning of the correct answer (as shown in the last row). This alignment process did lead to a performance improvement, but it still didn’t match the results obtained when answer choices were provided.

We believe that the "Machine match" strategy will be one of the evaluation approaches we prioritize moving forward. We will also continue exploring other evaluation strategies to further improve the assessment process.

Once again, thank you for your valuable suggestion!

Concern 4 the reliance on full human annotation restricts scalability.

This is a very valid observation, and indeed, the cost of annotating this dataset has been significant. However, we believe that human annotation provides a level of control and reliability that automated methods cannot currently match. High-quality, human-annotated data is essential for accurately evaluating MLLMs, and we plan to expand it in the future to cover additional domains and task types while maintaining this standard of quality.

Concern 5: "Including more languages would strengthen the dataset’s multilingual utility."

Expanding to additional languages presents a far greater challenge than adding new tasks or domains. Notably, when creating the MME-RealWorld CN version, we made a concerted effort to control annotation quality and avoid overlap between Chinese and English evaluations. This involved collecting scenes specifically in Chinese and enlisting Chinese language experts for annotation. Expanding to other languages, especially less commonly spoken ones, would require collaboration with language-specific researchers and the collection of unique image sources for each language. While this is a formidable task, we are actively exploring the possibility of extending the dataset to include widely spoken languages, such as Korean, Japanese, and Spanish. This process will require time to ensure both data quality and proper annotation.

Regarding the reviewer's suggestion on incorporating models to improve annotation efficiency and scalability, we have considered several strategies to potentially reduce annotation costs without compromising quality.

Our dataset construction involves two key stages requiring human involvement. The first stage is image selection and filtering, where we ensure:

1. The images contain challenging or valuable small objects.
2. The images are of high quality, free from glare or noise.
3. The scenes are clear enough to minimize ambiguity in QA tasks (e.g., if a question asks about a person in a blue shirt in the upper right corner, there should be only one such person).

After ensuring high-quality and challenging images, we can provide annotators with images that are more likely to yield usable QA pairs, thus enhancing annotation efficiency.

For the second stage, human annotation involves crafting questions and answers for these complex images.

To explore whether models can help us enhance the scalability of our dataset, we have conducted a small-scale trial with the following approach:

For data filtering, we adopted a model-based strategy. We first set a minimum resolution of 1024x1024 to ensure that the images are sufficiently large. We used a Multimodal Large Language Model (MLLM) to remove images that were noisy or unclear. The MLLM was then employed with the prompt: "We provide an image; please score the scene from two aspects on a scale of [1, 2, ..., 10]:

1. Complexity: Assess whether the image contains a large number of elements, such as various objects. If the scene features only one prominent object and the foreground/background is relatively simple, the score should be 0. Conversely, if the image includes a multitude of elements and the scene is complex, the score should be higher.

2. Object Salience: Evaluate whether the image contains small objects that are difficult to observe clearly. Specifically, objects occupying less than 1/20 of the total pixel area. If all objects in the image can be observed very clearly or are relatively large, the score should be 0.

3. Objects with Observational Challenges: If there are objects in the image that are difficult to observe or smaller targets, please list these objects and their locations. “

With this strategy, our annotators can intuitively see the model's scoring of the images and the small objects for reference. Currently, we are using GPT-4o as the scoring model. In the future, we could use multiple models to vote for better quality assurance. This method has indeed reduced the workload for annotators, and we found that it allows us to quickly filter out simple images.

In the second part, manual annotation, considering the reviewer’s comments on the scalability of manual annotation, we attempted to use GPT-4o for question generation and answer construction. However, GPT-4o did not perform well in high-resolution scenarios, especially in reasoning tasks, resulting in samples that were either too easy or incorrect, making them unusable. After some consideration, we tried using Qwen2-vl-72b, Llava-ov-72b, Claude3.5-sonnect, and GPT-4o to individually generate three questions and answers for a single image. The construction standards for questions and responses were consistent with those in our article, and prompts were input to the model as a prerequisite. Finally, humans experts retained the most challenging and reasonable questions.

Through this method, we were able to obtain some good QA pairs. However, in small-scale experiments, we still observed that the task difficulty is easier and the error rate was much higher than with purely manual annotation, and the options designed manually were more reasonable. Additionally, the time spent manually reviewing multiple large model outputs was not significantly better than manual annotation.

This suggests that for our ultra-high-resolution perception tasks, there may not yet be an optimal model annotation pipeline, which is a topic for future research. Any constructive feedback from the reviewers would be highly valuable.

Dear Reviewer qNEP,

As the rebuttal ends soon, may we know if our responses have addressed your further comments?

If so, we kindly ask for your reconsideration of the score. If any aspects require additional elaboration or refinement, we will be more than happy to engage in further improvements and discussion.

Thanks again for your time.

Concern 1 The evaluation on MME-RealWorld seems to require lots of computation resources, which may limit the accessibility for researchers with fewer resources.

This feedback is incredibly valuable! To facilitate the evaluation of high-resolution perception abilities in multimodal large language models, we plan to sample 50 examples from each task within each domain, resulting in a test set of 2,150 QA pairs. We believe, as per your suggestion, that this smaller-scale test set will have a greater impact.

Above are our initial tests with some models. While it is difficult to perfectly retain model performance on the original dataset due to the sampling process, the overall task difficulty has actually increased, as we’ve reduced the number of samples for simpler tasks such as OCR, charts, and basic reasoning tasks. For instance, the performance of Qwen2-VL dropped from 56.4 on the full dataset to 46.7 in this reduced test set.

We believe this compact test set is an excellent tool for researchers to perform smaller-scale, focused evaluations and comparisons of their models. It strikes a good balance between practicality and robustness, making it more accessible for the community while still providing meaningful insights into model performance.