What is the Visual Cognition Gap between Humans and Multimodal LLMs?

We thank Reviewer 1KjF for your encouraging comments, especially the suggestions on more explanation for the matrix reasoning domain shift problem. Firstly, we highlight our novelty and explain our main contribution and research workflow in detail. Secondly, we discuss our observation is aligned with common knowledge in shortcut learning and provide an additional experiment on controlling the option number of RAVEN to mitigate domain shift, which we hope allow raising post-rebuttal scores.

Question: The dataset contribution is limited in originality: all the visual problems are repurposed from existing resources, leading to a marginally novel benchmark.

Thank you for your feedback. We would like to clarify the novelty and contribution of our MaRs-VQA benckmark. While the foundational visual stimuli are adapted from the established Matrix Reasoning Item Bank (Chierchia et al., 2019), our primary contribution lies in the significant and novel transformation of these items into the MaRs-VQA benckmark. This process involved more than just repurposing images; we introduced several innovative elements crucial for evaluating and advancing Multimodal LLMs:
1. Novel VQA Framework and Annotation: We are the first to convert these psychometric problems into a VQA format systematically. This involved creating language-based descriptions for all answer options (as detailed in lines 129-130 of our manuscript), providing an alternative input modality to the purely image-based options. This allows for a more diverse evaluation of MLLMs, testing their comprehension of visually grounded language. Previous benckmarks for abstract reasoning typically offer only the visual problems without this VQA structure or alternative option modalities.
2. Unique Multi-Stage Reasoning Annotations: A contribution of MaRs-VQA is the inclusion of detailed, step-by-step structured cognitive reasoning annotations for each problem (lines 160, 166-170). These annotations are designed to facilitate a deeper understanding of MLLM decision-making. Furthermore, these annotations serve as valuable data for supervised fine-tuning, enabling the development of models with improved visual cognitive abilities, as demonstrated by our Qwen2-VCog baseline (lines 161-163, 193-195).
MaRs-VQA is a psychologist-verified VQA benchmark for matrix reasoning that incorporates these rich VQA and cognitive reasoning annotations (Table 1). Such detailed annotations are essential for pushing the boundaries of MLLMs, particularly in aligning their perceptual abilities with language understanding (Huang et al., 2023) and in tackling complex visual deductive reasoning tasks where current models still exhibit limitations (Zhang et al., 2024). Therefore, while the visual source is acknowledged, the MaRs-VQA benchmark offers a substantially novel benchmark and a valuable new resource for the research community due to its unique task formulation.

Reference
Gabriele Chierchia, Delia Fuhrmann, Lisa J Knoll, Blanca Piera Pi-Sunyer, Ashok L Sakhardande, and Sarah-Jayne Blakemore. The matrix reasoning item bank (mars-ib): novel, open-access abstract reasoning items for adolescents and adults. Royal Society open science, 6(10):190232, 2019.
Huang, Shaohan, et al. "Language is not all you need: Aligning perception with language models." Advances in Neural Information Processing Systems 36 (2023): 72096-72109.
Zhang, Yizhe, et al. "How Far Are We from Intelligent Visual Deductive Reasoning?." First Conference on Language Modeling.

Dear Reviewer 1KjF:

Thank you for the follow-up and for your constructive feedback. We will incorporate your valuable suggestions on domain-shift analysis and additional experiments in the final version. Below, we address the two remaining concerns in greater detail. We hope our reply can solve your concerns. Feel free to give us more constructive feedback.

Question: The dataset contribution is limited in originality: all the visual problems are repurposed from existing resources, leading to a marginally novel benchmark.

We want to highlight that while the visual stimuli are adapted from the Matrix Reasoning Item Bank, our main contribution is transforming these items into a new benchmark designed specifically for evaluating Multimodal LLMs. Strict reuse is necessary in this domain: to preserve psychological validity, any matrix reasoning dataset must employ certified stimuli rather than simple-designed variants. Our contribution therefore lies in how the stimuli are re-framed and annotated. Besides, all datasets in this domain are built from existing source, for example, (https://openreview.net/forum?id=pYEnhZ6NAv) used 241 non-labeled IQ-test samples; (https://neurips.cc/virtual/2024/poster/97456) used 777 non-labeled government exam samples. We are the first and only one create full VQA setting and reasoning chain annotation for matrix reasoning with over 2x data than these concurrent works. The annotation process takes us 100 hours.

• First VQA reformulation of matrix-reasoning items – every choice has both an image and a natural-language description (L 126-130).

• Fine-grained cognitive chains – we supply step-wise solution paths (L 160, 165-181), enabling supervision of reasoning, not just answers.

• Demonstrated utility – models fine-tuned on these chains (Qwen2-VCog) gain much better accuracy over vanilla Qwen2 (Table 2).

Thus MaRs-VQA is the only benchmark that simultaneously (i) preserves psychometric rigor, (ii) couples vision and language, and (iii) exposes explicit reasoning supervision.

Question: The paper does not meaningfully interpret what the gap between MLLMs and human performance implies for real-world tasks or downstream applications—this would have been valuable for the COLM audience.

10% of our MaRs-VQA data is selected in Humanity's Last Exam “https://agi.safe.ai/” and it is already used by the community to evaluate MLLMs. As far as we know, the newly released Gemini-2.5-Pro and GPT-4.5 can only achieve about 20% in Humanity's Last Exam public benchmark. We hope our VQA data design can contribute to the COLM community and raise researcher interest in improving MLLM’s visual cognition ability.

These are the specific manifestations of cognitive ability:
1. Advancing Scientific and Analytical Reasoning: A primary goal for AI is to assist in complex analysis and discovery. This requires moving beyond describing known patterns to inferring the abstract principles or underlying laws from visual data, such as in experimental results or system simulations.

2. Enabling Robust Autonomy and Adaptability: For autonomous systems to be truly useful, they must operate effectively in novel environments with unfamiliar rules. This demands the ability to quickly deduce the logic of a new system from limited interaction and then adapt their behavior accordingly.

Our benchmark is used to test cognitive ability. In both settings, MaRs-VQA supplies a minimal but diagnostic proxy: if a model cannot induce the rule of a 3x3 matrix like humans, it is unlikely to induce the rule of a novel lab assay or traffic schema. By providing fine-grained reasoning chains, our benchmark also tells researchers where the reasoning pipeline fails, guiding targeted progress. The fact that even state-of-the-art models perform poorly on these tasks, as evidenced by public benchmarks like "Humanity's Last Exam" which incorporates our data, underscores the urgency of this research.

We appreciate your follow-up feedback and will incorporate these clarifications, the comparative table above, and a short section on downstream implications in the camera-ready version.

Best regards,
Authors

We thanks Reviewer 9Mv5 for your encouraging comments. We highlight our novelty and contribution and explain our main contribution and research workflow in detail, which we hope allow raising post-rebuttal scores.

Question: ultimately, relatives restricted contribution: a reformatting of an existing resource, and a couple of evaluation runs

Thank you for your feedback. We would like to clarify the novelty and contribution of our MaRs-VQA benchmark. While the foundational visual stimuli are adapted from the established Matrix Reasoning Item Bank (Chierchia et al., 2019), our primary contribution lies in the significant and novel transformation of these items into the MaRs-VQA benchmark. This process involved more than just repurposing images; we introduced several innovative elements crucial for evaluating and advancing Multimodal LLMs:
1. Novel VQA Framework and Annotation: We are the first to convert these psychometric problems into a VQA format systematically. This involved creating language-based descriptions for all answer options (as detailed in lines 129-130 of our manuscript), providing an alternative input modality to the purely image-based options. This allows for a more diverse evaluation of MLLMs, testing their comprehension of visually grounded language. Previous benchmarks for abstract reasoning typically offer only the visual problems without this VQA structure or alternative option modalities.
2. Unique Multi-step Reasoning Annotations: A contribution of MaRs-VQA is the inclusion of detailed, step-by-step structured cognitive reasoning annotations for each problem (lines 160, 166-170). These annotations are designed to facilitate a deeper understanding of MLLM decision-making. Furthermore, these annotations serve as valuable data for supervised fine-tuning, enabling the development of models with improved visual cognitive abilities, as demonstrated by our Qwen2-VCog baseline (lines 161-163, 193-195).

MaRs-VQA is a psychologist-verified VQA benchmark for matrix reasoning that incorporates these rich VQA and cognitive reasoning annotations (Table 1). Such detailed annotations are essential for pushing the boundaries of MLLMs, particularly in aligning their perceptual abilities with language understanding (Huang et al., 2023) and in tackling complex visual deductive reasoning tasks where current models still exhibit limitations (Zhang et al., 2024). Therefore, while the visual source is acknowledged, the MaRs-VQA benchmark offers a substantially novel benchmark and a valuable new resource for the research community due to its unique task formulation.

Reference
Gabriele Chierchia, Delia Fuhrmann, Lisa J Knoll, Blanca Piera Pi-Sunyer, Ashok L Sakhardande, and Sarah-Jayne Blakemore. The matrix reasoning item bank (mars-ib): novel, open-access abstract reasoning items for adolescents and adults. Royal Society open science, 6(10):190232, 2019.
Huang, Shaohan, et al. "Language is not all you need: Aligning perception with language models." Advances in Neural Information Processing Systems 36 (2023): 72096-72109.
Zhang, Yizhe, et al. "How Far Are We from Intelligent Visual Deductive Reasoning?." First Conference on Language Modeling.

We thank Reviewer dTKc for your encouraging comments and valuable experiment insights. We highlight the experimental setting of MaRs-VQA discussing why test set is small but it can make solid conclusion and conduct a additional general benchmark evaluation experiment of Qwen2-VCog and conduct an ablation experiment to explore if the failure of Qwen2-VCog is caused by vision ability limitation of MLLM, which we hope allow raising post-rebuttal scores.

Question: A dataset consisting of 1,440 image is relatively small, especially only 480 VQA are used as the test set. The benchmarking results may contain high variance.

First of all, It is pertinent to note that MaRs-VQA, with 1,440 total instances, is the largest psychologist-verified VQA dataset currently available for zero-shot matrix reasoning evaluation (Table 1 and Lines 50-51, 62-64).

Our primary benchmark (Table 2) evaluates multiple MLLMs in a zero-shot manner on the 480-instance MaRs-VQA test set. This approach assesses their inherent visual cognitive abilities without requiring task-specific fine-tuning, which was the main focus of our comparative experiments (Lines 15-17, 155-157).
The Supervised Fine-Tuning (SFT) was a distinct, secondary investigation performed on one model (Qwen2-VL) using the 960 training instances from MaRs-VQA. This set, though modest in size, is unique as it is psychologist-verified and contains specialized, cognitively-designed annotations for matrix reasoning (Lines 50-51, 63, 160, 320). The SFT aimed to probe whether targeted training on such high-quality data could enhance performance on this specific cognitive task and, crucially, to observe the model's subsequent generalization capabilities (Lines 66-68, 159-161, 261-263). For such focused investigations into adapting models to highly specific cognitive paradigms and evaluating generalization, a smaller, meticulously curated benchmark is often informative and practical.

Question: Although finetuning Qwen2-VL on MaRs-VQA obtains the best number(besides human level) in Table2, is it an overfit or does it have negative impact on Qwen2-VL's original MLLM abilities?

The experiment results on general benchmarks shows that reasoning enhanced SFT on MaRs-VQA does not have negative impacts on Qwen2-VL's original MLLM abilities. The performance is almost the same in most benchmarks.

Question: Have the authors studied the underexpressive performance of MLLM on matrix reasoning is mainly limited by its vision ability or LLM ability? For example, limitation in vision ability means it can't recognize the patters accurately, like its visual grounding ability or its small object detection is not satisfying.

Thanks for your suggestion. We explore MLLM solving matrix reasoning problems with experiments in the table above. The first row is using question image-only as the input of MLLM and let MLLM describe with language what might be the missing cell in the question image. Then we use CLIP score to select the matched option. The second row is the default setting, input the question image and 4 options and let MLLM select the best match choice. The last row is using question image and 4 options as input together with human annotated language-based description for question image and option images. Results support that vision pattern extraction is the bottleneck for GPT-4o and Qwen2-VCog (visual encoder of Qwen2-VL).

In the experiment in the first row, GPT-4o and Qwen2-VCog are almost at chance, meaning it fails to see the rule even before reasoning. Supplying the four candidate images raises accuracy of GPT-4o 9%, and after adding perfect human annotated textual cues, it gives a 2.5% improvement, confirming that once the visual relation is concluded by language, GPT-4o can follow the logic; the chief shortfall is therefore visual, not linguistic.

Similar to GPT-4o, Qwen2-VCog jumps from chance to 72.7 % as soon as option images are available, showing it already extracts most visual relations. Extra textual hints do not help, implying its visual representations embed the needed semantics and further language does not unlock additional reasoning capacity.

We thank Reviewer 1UB9 for your encouraging comments. We highlight the novelty and difference of MaRs-VQA than other works (NeurIPS-23 and COLM-24) and conduct an additional reasoning existence ablation experiment of Qwen2-VCog, which we hope allow raising post-rebuttal scores.

Question: The matrix reasoning task itself has already been used in machine learning fields.
Our work, with the introduction of the MaRs-VQA benchmark, aims to address specific gaps and enhance the evaluation of general-purpose Multimodal Large Language Models (MLLMs) in this domain.

Key distinctions from previous work, as highlighted in Table 1, include:

1. Focus on General MLLMs' Cognitive Abilities: While prior efforts often involved specialized models for matrix reasoning, MaRs-VQA aims to assess the inherent, generalized visual reasoning skills of MLLMs. This is crucial for understanding their broader cognitive capacities beyond task-specific training (Page 1, lines 38-40; Page 4, line 118).
2. Enhanced Benchmarking Rigor: MaRs-VQA offers significant improvements in scale and design. It incorporates robust human performance baselines, psychological validity, and a VQA format with diverse option modalities. This addresses limitations in some earlier benchmarks, such as smaller scale or the absence of thorough psychometric validation and direct human comparative data (Page 3, lines 101-104; Page 4, lines 136-138).
3. Probing Reasoning Processes: Our work also emphasizes understanding the cognitive gap. By employing structured reasoning evaluations (as detailed in Section 5.1, Page 5), we aim to gain insights into how MLLMs approach these complex visual reasoning tasks, rather than solely measuring task completion rates. This provides a deeper analysis of their current limitations compared to human cognition. Therefore, MaRs-VQA serves not just as another benchmark for an existing task, but as a more comprehensive public available tool to probe the fundamental visual cognitive abilities of modern MLLMs.

Question: The ablation study only covers the proprietary models. & Could you also report the ablation study results of Qwen2-VCog?

Thanks for your suggestion. We conduct additional ablation experiments to explain how incorporating reasoning helps the Qwen2-VCog’s STF training. If the reasoning steps in the training set is removed, Qwen2-VCog performs worse in MaRs-VQA test and RAVEN’s VQA subset (generalizability).