MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

We appreciate the reviewers’ positive feedback and recognition of our work's strengths, particularly the clarity of the paper, the reasonableness of the MMMU upgrades, and the comprehensive evaluation of existing MLLMs.

Figure 1 and Table 1 include almost 20 models and take much space

We aim to provide a comprehensive evaluation by including most of the available models, enhancing the robustness and breadth of our results, which we believe is a strength of our work. However, we understand the importance of allocating space to more insightful analyses. To address this, we remove Figure 1 in the revised version to include newly added results and deeper analyses. Thank you for this valuable suggestion.

Incremental Contribution of MMMU-Pro

We encourage reviewers to refer to the $\textrm{\color{blue}General Response}$ for a detailed explanation of our rationale and contributions. Thank you again for your thoughtful comments, which help refine and enhance our work.

CoT and OCR analysis are scratching the surface

Thank you for highlighting this point. To deepen our analysis of CoT and OCR capabilities, we have added several key experiments and insights in the revised manuscript:

1. Newly Added Analysis:

   • We conduct an extensive error analysis ($\textrm{\color{blue}Figure 8}$) to quantify the impact of reasoning errors and OCR errors, revealing that reasoning across integrated modalities poses a greater challenge but OCR is not a bottleneck for capable models like GPT-4o.
   • We investigate the influence of OCR capabilities on MMMU-Pro Vision performance ($\textrm{\color{blue}Figure 6}$) and demonstrate that good OCR performance alone does not ensure strong results in multimodal reasoning tasks.
   • A comparative study of CoT reasoning across different subjects ($\textrm{\color{blue}Figure 10}$, $\textrm{\color{blue}Table 2}$) highlights its benefits in reasoning-heavy domains like Science and Business, while showing limited effects in subjective areas like Art and Humanities.
   • We discover that in the Vision input setting, GPT-4o generates fewer CoT tokens and focuses more on descriptive tokens rather than analytical reasoning ($\textrm{\color{blue}Figure 9}$), highlighting the challenges of reasoning in integrated multimodal contexts.

2. Key Conclusions:

   • OCR errors are not a bottleneck for advanced models like GPT-4o; reasoning across integrated modalities remains the primary challenge.
   • CoT improves performance in reasoning-intensive subjects but needs refinement for integrated vision-text reasoning tasks.
   • Vision-only settings reveal weaker CoT capabilities, with a tendency to focus on description rather than analytical reasoning.

3. More Details in the General Response: These findings are elaborated in the $\textrm{\color{blue}General Response}$, which includes tables and figures for reference. We invite reviewers to review it for a more comprehensive understanding of our contributions.

New Tool Development to Facilitate Future Research

Thank you for highlighting this aspect. In our revised manuscript, we introduce a new $\textrm{\color{blue}Tool}$ designed to automatically generate datasets in the MMMU-Pro style, enabling the creation of text-rich images and reasoning-intensive multimodal tasks. This tool facilitates dataset generation and model analysis by automating question creation in real-world contexts.

This tool serves two primary purposes:

1. Dataset Augmentation: It enables efficient generation of datasets emulating real-world scenarios, supporting broader testing and fine-tuning of models.
2. Analysis Facilitation: By enabling controlled perturbations, it supports detailed analyses of model performance under varying conditions.

We believe this tool will benefit the community by supporting future multimodal research and model development.

More Insights and Fresh Perspectives

We encourage reviewers to check out our $\textrm{\color{blue}General Response}$ and the newly added results and analysis. Thank you for your valuable feedback, which has helped us significantly improve our work!

We sincerely thank the reviewers for their positive feedback and for highlighting the strengths of our work. We are particularly gratified by your recognition of our rigorous evaluation approach, including the use of LLM voting to filter text-only solvable questions, and your acknowledgment of MMMU-Pro as a more challenging and accurate benchmark for assessing multimodal integration. Additionally, we appreciate your positive remarks on our detailed findings and the future directions they suggest for improving vision-text integration strategies.

The emphasis should be on demonstrating the quality of the expanded options.

Thanks for the suggestion! We added more details in Appendix D to describe the expanded options motivation and method. Note that our motivation was not to reduce the performance of the model but instead to create a more robust evaluation setup to lower the probability of finding short-cut answers and reflect the true reasoning capability of the model.

Our option augmentation implemented a rigorous multi-stage validation process:

1. Initial Model-Based Option Augmentation and Filtering: As a first step, we utilized automated methods by leveraging an LLM GPT-4o to generate and another LLM Claude 3.5 to preliminarily filter the options that do not sound reasonable.
2. Two Rounds of Human Review:

• In the first round, individual reviewers assessed the expanded options for each question. They ensured that the options were diverse, logically distinct, and free from any ambiguity that could compromise the validity of the question. If there was anything wrong with the options, they were asked to correct or augment new ones.
• In the second round, a double-check process was conducted, where two additional human experts cross-validated each question and its options to eliminate any remaining inconsistencies or errors. This iterative process significantly enhanced the reliability and quality of the final benchmark. By combining automated and human validation steps, we ensured that each question's expanded options were robust and representative of the intended challenges.

Automatic Data Generation Tool

We think our methodology for data augmentation and screenshot generation could be used for automatic training data creation as well. To this end, we have made the scripts used for generating MMMU-Pro data publicly available here. The tool allows researchers to automate aspects of the data generation process, such as embedding questions into images and generating additional options, which can significantly reduce manual workload. By sharing these resources, we aim to empower the community to produce similar datasets more efficiently and at scale.

Technical Contribution is Limited

We appreciate the author’s feedback and re-state our technical contributions in the $\\textrm{\\color{blue}General Response}$.

Thank you for your insightful feedback! We acknowledge your concerns about the need for deeper data analysis and the perceived limitations in technical novelty. To address these points, we have significantly expanded on the insights derived from MMMU-Pro, including model rankings, OCR influence, reasoning error analysis, and the role of CoT across various subjects. These new findings, alongside our revised experiments and discussions, are elaborated in the $\\textrm{\\color{blue}General Response}$.

We invite you to review the $\\textrm{\\color{blue}General Response}$ for a comprehensive explanation of how MMMU-Pro provides valuable contributions to the evaluation of multimodal reasoning capabilities and guides the development of future models. Thank you again for your constructive comments and for helping us improve our work.

Thank the author for their response, most of my concerns have been addressed. From my part, this paper can be accepted. However, It’s not enough for me to give a higher rating. Therefore, I will keep my rating unchanged which is marginally above the acceptance threshold.

More Nuanced Evaluation and Analysis Beyond Accuracy

We agree that a detailed analysis beyond accuracy offers valuable insights. To this end, we conducted an in-depth error analysis by sampling 60 error cases from GPT-4 in the Vision input setting. We identified three major categories of errors, as shown in $\\textrm{\\color{blue}Figure 8}$:

1. Reasoning Errors (46%): These represent the largest category of errors. In these cases, the model retrieves the correct knowledge but fails to apply logical reasoning steps to reach the correct conclusion.

2. Perceptual Errors (27%): These occur when the model misinterprets visual inputs, such as misidentifying objects, texts, or relationships in the image, or overlooking key visual cues necessary for solving the problem. Notably, we manually checked for errors caused by inaccurate text recognition and found none (0%).

3. Lack of Knowledge (25%): Some errors result from insufficient subject-specific or commonsense knowledge. For instance, in tasks involving the Fourier Transform, the model struggled to apply appropriate principles, highlighting knowledge gaps in specific domains.

Key Findings

From our analysis, we derive several interesting insights:

• Text Recognition as a Non-Bottleneck: Our manual inspection confirmed that text recognition and OCR do not pose significant challenges for advanced models like GPT-4, as no errors were attributed to this factor.

• Increased Reasoning Complexity: MMMU-Pro Vision introduces significantly more reasoning challenges compared to MMMU (46% vs. 26%, as noted in the MMMU paper). This underscores that reasoning in a modality-integrated setting is notably more difficult.

New Insights and Fresh Perspective

Please check out our $\\textrm{\\color{blue}General Response}$ and newly added results and analysis.

We sincerely thank the reviewers for their thoughtful feedback and positive evaluation of our work. We are particularly encouraged by your recognition of our contributions to advancing multimodal AI benchmarking through MMMU-Pro, especially our efforts to address limitations in existing benchmarks like MMMU and to provide a more rigorous evaluation framework.

Diversity of Images in the Vision-only Setting

We have carefully accounted for diversity and realism in our dataset design! Specifically:

1. Varied Lighting and Environmental Conditions: The photo portion of our dataset was collected by over 10 annotators using a range of display devices (e.g., phones, tablets, monitors) with varying resolutions, lighting, and environmental conditions. This ensures the dataset captures realistic variability in brightness, contrast, and ambient light, reflecting real-world scenarios.
2. Annotator Variability and Subtle Rotations: Differences in annotators’ habits during data collection naturally introduced variation in camera angles and shooting styles. As shown in Figure 3, many photos in the dataset include slight rotations, which are common in casual usage scenarios. However, we intentionally excluded extreme rotations or distortions, as these are less representative of realistic interactions and could introduce challenges unrelated to the benchmark’s primary objectives. Additionally, we have updated $\\textrm{\\color{blue}Figure 2}$ to showcase a wider range of display environments, including variations in rotation, angles, and aspect ratios, further highlighting the dataset's diversity.

Breakdown of CoT’s impact

This is a great suggestion! We added the breakdown of CoT results for all the subjects in a representative proprietary model, GPT-4o, and an open-source model, LlavaOnevision 72B. Detailed scores for 30 subjects are added in Appendix $\\textrm{\\color{blue}Figure 10}$ and $\\textrm{\\color{blue}Table 2}$.

There are some interesting observations:

1. Domains with Significant CoT Improvements: The effectiveness of CoT prompting across disciplines is summarized in Table 1 and Figure 2, highlighting its varying impacts on different domains for GPT-4o and LLaVA-OneVision 72B. CoT demonstrates significant improvements in reasoning-heavy fields like Tech and Engineering, Science, and Business.

2. Domains with Limited CoT Impact: However, CoT's benefits are less pronounced or even negative in domains where subjective interpretation or direct knowledge retrieval dominates like Art, Humanities and Medicine.

I believe my concerns have been largely addressed, and I am willing to maintain a positive rating for this paper. I think the paper is overall solid, but the degree of its overall novelty prevents me from giving it a higher rating.

We thank the reviewers for their positive feedback and their recognition of the strengths of our work, particularly the significance of MMMU-Pro in addressing critical gaps in multimodal benchmarks, the rigor of our evaluations, and the insightful experiments that validate the benchmark's effectiveness.

Justification for the Motivation of MMMU-Pro

Our motivation for creating MMMU-Pro is to benchmark reasoning capabilities within an integrated multimodal context. We claim that future foundational models will increasingly adopt an integrated modalities paradigm, where input types—whether images, text, or videos—are no longer treated as separate modalities. We acknowledge the reviewer’s point that OCR challenges could obscure the true reasoning abilities of models that excel in multimodal integration but struggle with OCR. However, existing high-performing models—both open-source and closed-source—demonstrate strong OCR capabilities but still struggle with reasoning in integrated contexts (check more details below). MMMU-Pro is designed to evaluate reasoning in a more realistic and challenging way, serving as an advanced test for reasoning ability in multimodal scenarios.

Human perception seamlessly integrates and transitions between textual and visual signals through a unified interface: our eyes. If we aim for intelligent systems capable of reasoning in real-world scenarios, all forms of information—text, images, and videos—would be processed cohesively through a single interface. Such systems need to operate effectively in these more complex and dynamic environments. MMMU-Pro represents an early step toward developing intelligent systems capable of reasoning in real-world conditions.

For a more detailed justification of our motivation, please refer to our $\\textrm{\\color{blue}General Response}$.

OCR Requirement

Regarding the role of OCR in understanding MMMU-Pro style questions, we argue that MMMU-Pro operates within a vision setting that requires deeper integration of visual perception, OCR, and reasoning capabilities. We assert that OCR is a necessary but insufficient condition for success. As shown in the newly added $\\textrm{\\color{blue}Figure 6}$, high OCR accuracy does not necessarily translate to high MMMU-Pro Vision accuracy, whereas models excelling in multimodal reasoning consistently demonstrate strong OCR performance. For example, LLaVA-OneVision-72B and Pixtral-12B achieve comparable OCR accuracy (~85%) to InternVL2-Llama3-76B but significantly lower MMMU-Pro Vision accuracy (~25% vs. ~38%). Additionally, our newly added error analysis in $\\textrm{\\color{blue}Figure 10}$ reveals that text recognition and OCR are not the primary bottlenecks for capable models like GPT-4o. Among perception errors, none are caused by OCR failures. Instead, we observe a significantly higher proportion of reasoning errors compared to MMMU error analysis (46% vs 26%). This finding underscores that reasoning in a more modality-integrated setting is considerably more challenging.

Limited Impact on Model Performance Ranking

We updated $\\textrm{\\color{blue}Table 1}$ to show the detailed ranking changes of models when transitioning from MMMU (val) to MMMU-Pro Standard and MMMU-Pro Vision. On one hand, it is encouraging that rankings remain relatively stable when moving from MMMU (val) to MMMU-Pro Standard, indicating that overfitting on MMMU (val) is not a significant issue. On the other hand, we observe a notable shift in rankings when comparing MMMU (val) to MMMU-Pro Vision. This suggests that MMMU-Pro Vision demands a higher level of multimodal integration capability, which MMMU lacks. As mentioned in the $\\textrm{\\color{blue}General Response}$, while the community has yet to reach a consensus, we foresee foundational models evolving toward an integrated modalities paradigm. In light of this, we treat MMMU-Pro Vision as the primary evaluation setting, where these shifts in performance rankings underscore the challenges of reasoning in more complex, modality-integrated scenarios.

New Insights and Fresh Perspective

Please check out our $\\textrm{\\color{blue}General Response}$.