Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark

Thank you for the encouraging and thoughtful review! Below is our response to your questions and suggestions.

Q1: Why did we filter out questions that can be answered using the text and generated image captions?

Our enhanced filtering pipeline targets questions requiring deep multimodal reasoning that are difficult to solve through text reasoning or one visual pass. If a question can be solved with text and image captions, it means the necessary visual information can be compressed into text. The question is then solvable with one round of visual perception or shallow visual reasoning, with the rest handled by text reasoning. Instead, we focus on problems requiring back-and-forth textual and visual reasoning, which typically demand repeatedly referring back to the image for multi-hop reasoning or mental simulation. This filtering pipeline allows us to curate problems that assess advanced multimodal reasoning.

Q2: Counterintuitive task of "Code Choose Vis"

While "Code Choose Vis" may seem counterintuitive at first, it is tied to how humans typically write visualization code, whereby they envision their desired chart, code it, and mentally check if the code achieves their intention. The critical step of asking oneself, "is the code doing what I want it to do?" is precisely what "Code Choose Vis" problems target. This ability is also prerequisite to debugging visualizations, where one must understand what the current code produces before making corrections.

Q3: Plans to expand EMMA to more general domains

While most questions in EMMA require domain knowledge, many math questions do not. For example, simulating how a 2D shape will look after transformations or finding patterns in a sequence of shapes relies primarily on logic and general knowledge rather than math expertise.

In scoping for EMMA, we explored various disciplines for multimodal reasoning questions meeting our strict criteria. We found that most such problems are typically considered "logic" or "spatial reasoning" tasks, which we ultimately categorized under math. In other domains, such problems are much harder to source or create. Nonetheless, we are very interested in incorporating more general domain multimodal reasoning questions and are actively exploring strategies for expansion. Thank you for this great suggestion!

Q4: Why does CoT prompting not help with open-source models?

We observed that CoT prompting generally improves performance for closed-source models, but tends to hurt performance for open-source models. One possible reason is that open-source models do not effectively leverage language to assist in multimodal reasoning tasks where language could be beneficial. For example, language can often help grounding in multi-hop counting, so it should be theoretically possible to improve performance on this task with CoT, but open-source models fail to capitalize on this.

Previous work has shown that the quality of training data is crucial for CoT effectiveness. For example, [1] has found that filtering out anomalous samples (e.g., repetitive patterns) can improve CoT performance. Your point regarding the lack of CoT supervision during training aligns with this perspective: incorporating high-quality CoT data can indeed be seen as one way to improve the quality of training data for MLLMs.

Without access to the training data and pipelines of the tested models, it is difficult to pinpoint the reason behind this divergence. Nonetheless, we believe that multiple factors might contribute to this phenomenon, and a more systematic understanding of CoT prompting in multimodal settings remains an open and important direction for future research.

[1] https://arxiv.org/pdf/2412.05271

Q5: Does high Pass@16 indicate latent capability and can fine-tuning on similar data boost performance?

That is a keen observation. We agree that strong Pass@16 performance may suggest latent reasoning capabilities. However, as you noted, even if models possess relevant knowledge, they have a low probability of applying it correctly. Moreover, correct answers can sometimes stem from flawed reasoning, especially for multiple-choice questions.

The idea of finetuning on similar questions can indeed be promising. Prior work [2] has highlighted the nuanced interplay between memorization and generalization in finetuning LLMs on logical reasoning, and it would be interesting to investigate this further in the multimodal setting—an interesting direction for future work.

[2] https://arxiv.org/pdf/2410.23123

Suggestion 1: Discussing additional relevant work

Thank you for these highly valuable and relevant references. We will include and discuss them in an updated version.

Suggestion 2: Highlighting top scores in tables and including human performance breakdown on EMMA-mini

Thank you for the thoughtful suggestion. We will make sure to more clearly highlight results and include a category-level breakdown of human performance.

Thank you for your detailed review and great questions! We hope our response below helps address them.

Concern 1: Only 3 models are used in filtering. Other models might be able to solve the retained questions in the text-only setting

(1) The 3 models used were among the strongest available at the time. If none could consistently solve a question with text and captions, other models are unlikely to succeed under the same conditions.

(2) We also wish to emphasize that MLLM-based filtering is only the first step in our data pipeline (Section 3.2). This was followed by manual filtering to ensure retained questions genuinely required multimodal reasoning. In fact, 16.4% of questions were removed at this stage because they did not involve substantial multimodal reasoning.

(3) Following your suggestion, we tested more recent MLLMs in the "caption and question" setting. We focused on math to test whether our filter separates visual-reasoning-heavy questions from light ones (hereafter "heavy" and "light" questions). We selected:

• 100 "heavy" math questions from EMMA-mini
• 100 "light" math questions randomly sampled from the filtered-out pool

We evaluated 5 SOTA MLLMs using only the question and caption, and also report a full-input setting (text + image) on heavy questions:

• In the caption-only setting, all models perform significantly better on light questions than on heavy ones, indicating that the filtered-out questions can largely be solved via text-based reasoning and that our filtering pipeline effectively identifies visual-reasoning-heavy questions;
• On heavy questions, providing the image offers only marginal gains over captions. Even advanced MLLMs still struggle to utilize visual information for reasoning, reinforcing the motivation for EMMA.

Concern 2: Lack of domain knowledge can cause weak performance

We agree that a lack of domain knowledge can lead to weak performance on EMMA. However, we believe that the primary factor behind the underperformance of SOTA models on EMMA is not the lack of domain knowledge, but poor multimodal reasoning skills. Recent benchmarks suggest that SOTA models are equipped with strong domain knowledge. For example, models have achieved 87.7% accuracy on GPQA Diamond (which tests graduate-level science knowledge), 78.2% on MMMU (which spans many subjects), and 86.5% on AIME (a high school-level math competition). This suggests that the knowledge required for EMMA is largely present in current models. Hence, we believe that poor multimodal reasoning skills are the bottleneck.

Concern 3: EMMA is limited to four domains and does not include natural images

In scoping EMMA, we explored many disciplines for questions meeting our strict criteria. We found that in many domains the primary challenge lies in knowledge rather than in multimodal reasoning. For example, many biology questions involve tasks such as labeling parts of a complex diagram. These tasks often hinge more on knowledge than on reasoning over multimodal information. Constructing/curating multimodal reasoning questions in these domains is also more difficult. That said, we appreciate your suggestion and plan to involve more disciplines in future work, such as law and medicine.

We also agree that including natural images would be valuable. However, sourcing high-quality, reasoning-focused problems using natural images has proven to be challenging. Benchmarks like MMMU also include very few natural images. In math, physics, and chemistry, most problems are accompanied by simplified diagrams or sketches designed to reduce ambiguity and clarify assumptions. We will explore ways to design or curate problems involving natural imagery in future work.

Q1: Meaning of "a single visual pass"

We operationalize "a single visual pass" as generating image captions with GPT-4o. Multimodal reasoning questions often require looking at an image multiple times for multi-step/multi-hop reasoning. If a question can be solved with text and captions, its visual content can likely be compressed into text with shallow visual perception/reasoning, and the rest can be handled by text reasoning. Whereas previous work, BLINK [1], assesses perception beyond recognition, we target images that are necessary for reasoning beyond perception.

[1] https://arxiv.org/abs/2404.12390

Q2: Meaning of Pass@N

Pass@N refers to the accuracy when generating N responses per question and checking if any of them is correct. Thus, it is an upper bound for accuracy from test-time scaling.

Thank you for your encouraging review and insightful questions. We provide our responses below.

Q1: What does "organically reason" mean?

By "organically reason over and with both text and images", we refer to the integrated way humans seamlessly blend visual and textual information during reasoning processes. To measure advanced multimodal reasoning capabilities, we target questions that require both modalities working together, where neither text nor images alone are sufficient. Models must dynamically engage with both reasoning channels and effectively fuse textual and visual information to succeed on our benchmark.

Q2: The filtering pipeline seems to only retain problems that are difficult to describe in words, which can include questions that are more difficult in visual perception than in visual reasoning.

We agree that our first-step filtering, which removes questions solvable by models given only question text and image captions, may retain examples that are difficult to describe in words, but do not necessarily require strong visual reasoning. For instance, identifying a symbol in a musical staff might be retained simply because the symbol is visually subtle or hard to express textually, even though solving the task does not demand deep visual reasoning.

To address this potential bias, we installed a second filtering step. Specifically, we manually reviewed the remaining set and constructed a taxonomy that emphasizes multimodal reasoning skills. For example, for math, we identified categories such as 3D spatial simulation and pattern inference. We then used GPT-4o to categorize the questions according to this taxonomy, followed by a final round of manual verification to ensure quality and relevance. This process helped ensure that the retained questions require visual reasoning.

Here we provide an example from the MathVision dataset that is retained after the first-stage filtering but removed during manual verification. Solving the problem requires correctly perceiving all the digits in the image, which MLLMs struggle with. For instance, GPT-4o recognizes the digits at the bottom as "2" instead of "3", resulting in an incorrect answer. Although the question cleared the first-stage filter, we excluded it as it primarily tests visual perception rather than visual reasoning.

To further support our claim, we provide the following statistics from the MathVision dataset: out of 3,040 total questions, 2,195 were retained after the initial model-based filtering. However, only 668 questions (30.43%) from that set were ultimately included in EMMA, based on our taxonomy and human verification. This highlights that our filtering method applies a stricter standard to ensure a focus on multimodal reasoning. As noted in our error analysis (see Figure 5), MLLMs sometimes still fail on EMMA questions due to perceptual errors. However, we do not view these errors as contradictory to the multimodal-reasoning-based nature of the tasks. Rather, they may reflect perceptual limitations that are prerequisites for successful multimodal reasoning—an important challenge in its own right. In other words, while the immediate failure may stem from perception, the questions still require multimodal reasoning to be solved, highlighting a compound challenge that current models have yet to overcome.

In summary, we have taken careful steps to ensure that the problems included in EMMA emphasize visual reasoning over low-level visual perception. We appreciate your thoughtful comment, as it raises an important distinction and gives us the opportunity to clarify our data curation process more thoroughly.

Thank you very much for your thoughtful and careful reading!

As you have pointed out, our benchmark provides a test suite that reveals significant limitations of even the most advanced MLLMs in handling complex multimodal reasoning tasks. Although state-of-the-art MLLMs have recently achieved strong results on many challenging benchmarks, their poor performance on our benchmark highlights key gaps in their reasoning capabilities. We attribute this, in part, to our enhanced filtering pipeline, which allowed us to surface questions that genuinely require multimodal reasoning. Importantly, this pipeline is reusable and could serve as a framework for constructing future benchmarks aimed at other aspects of multimodal understanding.

In addition, we utilized our benchmark as a testbed to provide technical evaluation and insights into advanced techniques for enhancing MLLM reasoning. By comparing model performance with and without CoT, and evaluating different scaling approaches—such as majority voting, best-of-N, and tournament selection—we highlight the limitations of current prompting and inference-time methods for complex visual reasoning. These technical insights also would not have been possible without a carefully curated benchmark.

By exposing these limitations, our work points to broader issues, such as potential shortcomings in current model architectures or a mismatch between existing training paradigms and the demands of complex multimodal reasoning. We hope this will encourage further research into more robust multimodal architectures and training strategies.

We deeply appreciate your constructive suggestions. Following your advice, we plan to explore new methods to tackle these challenges in future work, and we remain committed to advancing the evaluation and development of MLLMs’ visual reasoning capabilities.

Thank you again for your insightful and encouraging feedback!