PolyMATH: A Challenging Multi-Modal Mathematical Reasoning Benchmark

Weakness 1: Benchmark Positioning: The comparison between previous VQA or read-world dataset is not enough. I think there is a bunch of benchmarks related with visual abstraction reasoning (https://arxiv.org/abs/2202.10284). Is PolyMATH different than previous dataset or If it is just a combination of previous one. Can the evaluation on PolyMATH bring new insight compared to previous ones?

We thank the reviewer for bringing this paper to our notice. We would like to highlight that most of the benchmarks in the above mentioned survey are saturated. Plus these benchmarks focus on limited reasoning capabilities and contain significantly fewer samples. We would also like to draw attention towards some of the recent benchmarks that too [puzzleqa, marvel, mathverse, mathvista] covered just a few aspects of abstract reasoning capabilities but do not completely capture it. Polymath has been created from scratch and its details are mentioned in section 3.1. Given that most of the existing benchmarks are already saturated,

Weakness 2: Pattern Design: The paper mentions the cognitive reasoning and abstract reasoning. But it seems lack a sufficient elaboration on the why do we need to evaulate these patterns on MLLMs. It will be helpful to show practical application of these pattern in real world scenraio or relavent research from human coginition literature.

Previous literature [1,2,3,4] has suggested cognitive and abstract reasoning are highly relevant for Artificial general intelligence. These skills are necessary for higher order thinking and are essential for basic reasoning abilities. They show that developed cognition and reasoning and understanding abilities are skills critical to success in a plethora of real world domains.

Weakness 3: Value Justification: From the ablation study on diagram-only subset, the result showing the major issue blocking models abstract visual reasoning ability is the diagream comprehension. Can we understand the model have difficulties on percepting diagrams? If so, it may make the evaluation less insightful if the major bottleneck is in model's perception abilities instead of abstract reasoning ability. An experiment showing how much detail and vision information MLLMs can percept from diagream will be helpful.

While we do demonstrate that models, in opposition of their highly-vaunted multimodal capabilities, still have a long way to go in terms of visual comprehension - this weakness is by means the only, or even the major, cause of the failures observed on this dataset. We conduct a further, question category-wise study into the types of errors made by models. Specifically - we look into categories which have a high proportion of diagram based questions - where, if the lack of visual reasoning skills is truly a bottleneck, we would expect to see a high proportion of spatial misunderstanding errors. We present the results of the study below

This contradicts the hypothesis, forcing us to conclude that while visual reasoning is an area with a significant scope for improvement, this dataset poses challenges in logical and abstract reasoning and that range beyond testing only visual comprehension of an image.

Citations:

1. Zhao, Jian, et al. "Cognitive psychology-based artificial intelligence review." Frontiers in Neuroscience 16 (2022): 1024316.
2. Lohman, David F., and Joni M. Lakin. "Intelligence and reasoning." The Cambridge handbook of intelligence (2011): 419-441.
3. Clement, Bradley J., Edmund H. Durfee, and Anthony C. Barrett. "Abstract reasoning for planning and coordination." Journal of Artificial Intelligence Research 28 (2007): 453-515.
4. Wang, Yingxu, et al. "Cognitive intelligence: Deep learning, thinking, and reasoning by brain-inspired systems." International Journal of Cognitive Informatics and Natural Intelligence (IJCINI) 10.4 (2016): 1-20.

Question 1: Based on the examples in Figures 1 and 3, it appears that the format of the questions and answer choices varies. For instance, in the first question of Figure 1, choices are labeled with letters (A, B, C, D) at the bottom, while in the second question, choices are labeled with numbers (1, 2, 3, 4) on the right side. I believe reorganizing all questions in a consistent format could improve the validity of the evaluation.

Thank you for your feedback. The evaluation pipeline is doing it consistently but we will take this change this in the final version of the dataset.

Question 2: The dataset is unevenly distributed, with three out of ten patterns accounting for more than 40% of the baseline data, while some patterns represent only 3% of the data. Could the imbalanced data distribution lead to misleading conclusions in Section 4.2?

We recognise the fact that the benchmark is imbalanced. We are working on increasing the benchmark as a continous effort comparable to prev benchmark datasets. We're actively involved in improving the breadth and depth of data.

Question 3: "decreased accuracy on test-img is largely due to the presence of diagram-based problems" problematic. It's possible that test-img has a different distribution compared to testmini. For example, the questions in test-img may be more challenging, prompting designers to include diagrams for better understanding.

Thank you for the question. We want to highlight that some questions are purely diagrammatic in nature, without any question that requires a clarifying diagram like the Perspective Shift category (Fig. 11 page 36 and Fig. 21 page 46). As the difficulty across categories is fairly balanced as they are meant to equitably assess different cognitive skills, test-img includes figure-based questions spanning the range from simplistic to challenging - reflecting the difficulty of the testmini and the larger test dataset. We also illustrate that spatial misunderstanding is a significant driver of errors, which further supports our observation on the limited perceptual abilities of MLLMs.

Question 4: The authors adopt four prompting strategies for evaluating the MLLMs but only mention performance differences in one sentence (Lines 418-421). Could more insights or explanations be provided regarding these four strategies?

Thank you for the suggestion. We found the following insights: Step-back and chain of thought give longer responses, are more cohesive, and have similar scores via different models. Step-back is more vulnerable to memory issues and misalignment, as the “stepping back” approach results in increased distance from the problem and possibly reduces focus on the particulars of the problem and answer options. For categories where the principle behind the question requires more thought than the specifics of the question (eg. perspective shift), step back emerges as the best strategy; for categories where the approach to the problem depends on the details of the questions themselves (eg. figure completion), CoT exhibits better overall performance. We also see model preference patterns emerge; for example, few-shot inference is particularly conducive to high performance when using Claude 3.5 Sonnet.

Question 5: As the paper presents the complete PolyMATH dataset with 5,000 images, it seems unusual to report results on only 20% of them. Could the authors also provide results for the entire dataset? This would be beneficial for comparing new MLLMs more comprehensively in future research

Thank you for your feedback. Please find the results on the entire test set. We will include them in the final draft of the paper. We followed the convention of MathVista to give the results on a smaller test set in the main paper.

I have reviewed the benchmark links you provided, and I find them acceptable. Regarding the questions about the LLMs, the authors have provided additional explanations, which are beneficial to the significance of this paper. These clarifications have addressed some of my concerns, so I will raise my score. As for comments on the novelty and depth of some conclusions in the paper, I have also referred to the feedback from other reviewers. I appreciate the authors' efforts in creating the dataset, and it would be even better if there were more impactful take-away findings.

Q1: Till the deadline of the review process, the link to the benchmark is not open-source. I hope the authors would organize the benchmark website properly during the rebuttal so that the benchmark’s composition and evaluation content could be inspected.

Please find the Anonymous Github Repo

Q2: Experiment analysis is a little superficial. I hope the authors may provide some detailed analysis of the experimental results to yield more insightful conclusions.

Thanks for the feedback. We will add the additional analysis on O1, a large language model, demonstrates varying levels of performance across different cognitive tasks, ranging from those closest to human capabilities to those furthest away. In terms of pattern recognition and mathematical reasoning, O1-Mini closely approximates human performance. When it comes to relative and logical reasoning, O1-Preview is closer to human levels. Sequence completion and spatial reasoning pose the greatest challenges for O1, with both O1-Mini and O1-Preview falling significantly short of human performance. This disparity underscores a common limitation in less-performant language models and O1: difficulties with perception and visual comprehension. Interestingly, O1-Mini outperforms O1-Preview in numerical reasoning by approximately 20%, suggesting that scaling up the model size doesn't always translate to better performance in all tasks. We also did a thorough analysis of model errors and have added additional results for response R1. We are happy to conduct any further experiments or analysis as per reviewers’ suggestions.

Q3: The paper proposes many problems of current LLM models, but does not provide possible solutions. It will make this paper more meaningful if the authors would do some explorations on it.

This dataset is intended as a benchmark to challenge researchers in the domain to improve future models by evaluating them on this dataset. We also identify errors that affect models so they can become a focus for improvement. To establish the efficacy of existing performance-improvement prompting methods, we report baseline results to provide a springboard for further advancements.

Weakness 1: The paper does not provide open-source examples from the POLYMATH dataset. Without such examples, the unique challenges and value of the dataset are not clearly conveyed, weakening the paper’s effectiveness.

Please revisit the anonymous github link which contains details of the dataset along with examples: Anonymous Github. Please download the repository as a zip since the website does not support png type format.

Weakness 2: The analysis provided is superficial and covers mostly well-known conclusions without exploring the performance differences across models in different reasoning tasks or the underlying reasons. This shallow analysis fails to offer new insights or robust support for the conclusions, limiting the paper's contribution to the field.

Thank you for your feedback. We would like to reiterate the focus of our work which is to introduce a cognitive and spatial reasoning benchmark. We showcase that even current state of the art models suffer from problems which test their abstract and cognitive reasoning capabilities. While there have been similar studies ([1], [2], [3], [4]), our work is more holistic and covers a very broad range of problems. We have also conducted experiments with 15 state of the art models with 4 well known prompting strategies and are detailed in section 4.2 and 4.3. Although we get some similar findings as compared to some previous works, we showcase these findings on significantly different problem types. We are happy to conduct any further experiments or analysis as per reviewers’ suggestions.

Weakness 3: Although the paper mentions the issue of large models’ mathematical reasoning capabilities, it lacks further exploration of this topic. The absence of an in-depth discussion or future directions weakens the paper's forward-looking impact and completeness.

In terms of future directions, we enumerate possible extensions to this work and the insights it contains in Section A of the Appendix. Primarily, the findings related to visual comprehension and comparative benchmarking invite further exploration into the differences between model capabilities, as well as concerted efforts to improve visual reasoning beyond the performance baselines we exhibit in this work. We believe that the various reasoning errors and categorical strengths and weaknesses examined in this work could provide a direction for future model development efforts. Additionally, our experiments involving generating textual descriptions and the observed improvement in performance, while exposing the shortcomings of multimodal reasoning, open up avenues for research into improving the performance of contemporary MLLMs by setting up robust inter-modal frameworks and creatively reframing problems in a way that best leverages these models’ strengths and mitigates their weaknesses. We believe these are productive/feasible lines of inquiry for future research building on this work.

Citations:

[1] Liu, Yuan, et al. "Mmbench: Is your multi-modal model an all-around player?." European Conference on Computer Vision. Springer, Cham, 2025

[2] Li, Bohao, et al. "Seed-bench: Benchmarking multimodal llms with generative comprehension." arXiv preprint arXiv:2307.16125 (2023)

[3] Fu, Chaoyou, et al. "A challenger to gpt-4v? early explorations of gemini in visual expertise." arXiv preprint arXiv:2312.12436 (2023)

[4] Sun, Keqiang, et al. "Journeydb: A benchmark for generative image understanding." Advances in Neural Information Processing Systems 36 (2024)

Q1: Table 5 and table 3's performance are not consistant, the reviewer understands they are on different subsets, but the big difference is not expected. Could the author explain why?

Test mini is a stratified sampling representation of the original benchmark which has roughly 600 questions which do not contain diagrams. On the other hand, test img has 1000 samples, all of which contain diagrams. The difference is due to the fact that MLLMs have greater difficulty in understanding questions with diagram.

Q2: For the second weakness above, could the author give hypothesis on why inference scaling doesn't help the model much? And what is expected to improve our model on PolyMath?

We conducted an additional experiment where we provided diagram description, along with diagram image and question which resulted in a better performance. The results of the additional experiment are as follows:

Q3: Since the error analysis suggest logical flaw to be the main bottle neck of MLLM's performance (which is irrelavant to image information), and all data can be converted to textual format. What make it necessary for this dataset to be made multimodal?

The main aim is to study the ability of MLLMs’ ability to comprehend visual aspects to improve their mathematical reasoning, hence the multi-modal nature of this benchmark can be helpful in this aspect since this ability is not systematically studied yet. Based on our error analysis, we draw the following insights

• Spatial misunderstanding is the second most common type of error
• Logical errors are not limited to text only questions, and can also occur in image based questions due to surface level comprehension of the image and lack of abstract reasoning over the information provided.
• Furthermore, the dataset includes some particularly complex images on which a conversion to text results in models reasoning incorrectly - hence converting the entire dataset to text cannot guarantee that the amount and integrity of information in image only questions will be preserved.

In light of this, we aim for this benchmark to be used in a multimodal fashion. Furthermore: The diagrams’ text description was human verified and the subset on which this study was performed included image based questions whose diagrams could be converted into textual description without loss in information and comprehension. This was an ablation study, which showcased that even with textual descriptions the models perform far below human capabilities. The main intent of the benchmark is to evaluate multimodal reasoning capabilities itself.

Thanks for the feedback and suggestions. We address each weakness and question below:

Weakness 1: The findings in the paper are not new. Firstly, for 'MLLMs rely more on text than image', MathVerse[1] specifically create 6 versions of their dataset with incremental reliance on visual information, showing the same trend.

We would like to highlight that the core contribution of this paper is the challenging benchmark, which includes problems related to spatial and cognitive reasoning. These problems are essential for assessing the cognitive reasoning abilities of MLLMs, in contrast to those presented in MathVerse [1], which primarily focus on problems of plane geometry, solid geometry, and functions. Our main findings (i.e., lower performance on PolyMATH) indicate the limitations of MLLMs' ability to comprehend the reasoning aspects required to solve many complex real-world problems. For the other findings, we believe they align with those from MathVerse [1], suggesting that PolyMATH also assesses mathematical reasoning abilities beyond cognitive reasoning.

Weakness 2: The error analysis suggest logical flaw to be the main bottle neck of MLLM's performance. But Table3 and 4 suggests that different inference stretagy does not help much, comparing with zero-shot. (only about 2%-6%) So it's unclear how we can improve our models on this dataset.

Thank you for this comment. We performed some preliminary experiments with various inference strategies, which resulted in marginal improvements. This suggests that off-the-shelf methods are not sufficient for achieving higher performance, highlighting the complexity of our proposed benchmark. However, we believe that our findings and insights from these experiments can help design better inference-time or alignment strategies to enhance the reasoning abilities of LLMs. Additionally, in alignment techniques involving preference optimization, preference data could be created to favor outputs with greater reasoning correctness. Furthermore, exploring newer techniques such as DPO and KTO for their potential to improve reasoning could be an interesting future direction.

Weakness 3: Insufficient discussion on related works. Since PolyMath is scoped as abstract visual reasoning benchmark, the author should discuss how PloyMath is different from and better than existing benchmarks in this field, like Puzzle-VQA[2], Marvel[3].

We thank the reviewer for bringing the above mentioned papers to our attention. We will add them to our literature review section. Puzzle- VQA focuses on visual perception on MLLMs such as colors, numbers, sizes and shape. Marvel on the other hand focuses on Object Core Knowledge, Number Core Knowledge and Geometry Core Knowledge. While PuzzleQA and MARVEL study key aspects of abstract reasoning, our work focuses on cognitive reasoning capabilities on MLLMs which have not been studied by previous works in detail. Our benchmark covers broader categories like Perspective Shift, Figure Completion, Pattern Recognition, Sequence Completion, Relative Reasoning, Mathematical Reasoning, Numerical Reasoning, Spatial Reasoning, Odd One Out and Logical Reasoning. These categories of abstract and cognitive reasoning have not been studied in previous works. We would also like to highlight that our benchmark is significantly larger than the above mentioned benchmarks (2000, 770 samples vs. 5000 samples) and foundation models obtain nearly the same scores on all three benchmarks.

[1] Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., ... & Li, H. (2025). Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?. In European Conference on Computer Vision (pp. 169-186). Springer, Cham.

[2] Chia, Y. K., Han, V. T. Y., Ghosal, D., Bing, L., & Poria, S. (2024). PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns. arXiv preprint arXiv:2403.13315.

[3] Jiang, Y., Zhang, J., Sun, K., Sourati, Z., Ahrabian, K., Ma, K., ... & Pujara, J. (2024). MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning. arXiv preprint arXiv:2404.13591.

Thanks for the feedback and suggestions. We address each weakness and question below:

Weakness 1: While the paper presents error analysis for major models like Claude-3.5 Sonnet and GPT-4o, extending this analysis to more open-source models could yield deeper insights into shared weaknesses.

Qwen2 VL (2B) Instruct (5) - Least performant open source model

LLaVA-v1.6 Mistral (7B) (15) - Best performing open source model

We extend our human-review error analysis to the most and least performant open source models - Qwen2 and LLaVa-v1.6 respectively. Similar to the more powerful closed-source models, the most frequent types of errors involve logical flaws and spatial misunderstandings, at a higher rate than the reviewed closed-source models. The rate of memory flaws, incomplete responses and misalignments is much lower, given that the models exhibit flaws in their logical and spatial reasoning well before the opportunities for other errors to present themselves.

Weakness 2 & Question 1: The paper notes a performance improvement when diagrams were replaced by textual descriptions, indicating that MLLMs may not be effectively processing visual information. Could the authors provide more qualitative examples or specific insights into how the textual descriptions differed from the diagrams and why these led to improved performance?

From Fig 19 in Pg 44 of the paper, Image description: Image 1: A rectangle with a diagonal line from the top right corner to the bottom left corner

Image 2: A hexagon with a diagonal line from the top right corner to the opposite corner, dividing the figure into halves.

Image 3: A rectangle with a diagonal line. One end of the line is on the top side, between the top corners; the other end is on the bottom side, between the bottom corners Image 4: A hexagon with a diagonal line from the top left corner to the opposite corner, dividing the figure into halves.

Solution: Image 3 is the odd one out because it is the only image that does not have a diagonal line that connects opposite corners of the shape. Instead, the diagonal line connects the midpoints of two sides of the rectangle.

Fig 22 in Pg 47 of the paper: Image description: Image 1: A circle, completely containing 2 discrete circles within it. The discrete circles do not overlap with each other.

Image 2: 2 concentric circles, as well as a separate third circle. The third circle does not overlap at all with the concentric circles.

Image 3: 2 partially overlapping circles, both contained within a single large circle

Image 4: 3 concentric circles.

Let's analyze the relationship between the categories: Animals: This is the broadest category, encompassing all living beings. Herbivores: A subset of animals that primarily consume plant matter. Lions: A specific type of animal that is carnivorous. Based on this, we can deduce: Lions and Herbivores are distinct subsets of Animals. They do not overlap. All Lions and Herbivores are Animals. Therefore, the Venn diagram that correctly represents this relationship is Image (1). In this diagram: The large circle represents Animals. The two smaller, non-overlapping circles represent Herbivores and Lions, respectively. Both smaller circles are completely contained within the larger circle, signifying that all Herbivores and Lions are Animals. This diagram accurately depicts the hierarchical and non-overlapping nature of the three categories.