MathScape: Evaluating MLLMs in Multi-modal Math Scenarios through a Hierarchical Benchmark

1. I think authors should be aware that except for those previous works you mentioned, there are many other mathematical reasoning benchmarks this year [1,2,3,4,5], especially with a similar focus on multimodal reasoning. Hence, two of your contributions (New Persepective and New Benchmark) may lack novelty. Besides, New Method (i.e., how you construct and evaluate) is a fair but not strong contribution to the MLLM community.

2. The paper indicates that the dataset primarily consists of Chinese problems. I think this will narrow the contribution as well. Besides, educational levels (i.e., primary/middle/high school) are highly different between China and Western countries. So it is better if you can address this limitation, such as including a comparison of educational standards or proposing how the benchmark could be adapted for different educational systems.

3. The analysis is not sufficient for benchmark work. For example, we need to know the proportion of diverse reasons why the best model provides incorrect answers (e.g., failure to retrieve the visual information; misunderstanding of positioning; etc.) in both the whole dataset and each dimension. Furthermore, more bad cases are needed.

4.The evaluation focuses on a set of state-of-the-art models, but it might be beneficial to include GPT-4o, which has been proven for its effectiveness for complex reasoning. Besides, math-specific MLLMs should be included as well, since you also mentioned them in your related works.

References:

[1] We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

[2] IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations

[3] CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models

[4] CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models

[5] ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection

1. The contribution of this paper is overclaimed. To the best knowledge, MathVerse contains six versions of a problem and the 'vision-only' one also contains both math figures and question in the image, similar to the contribution of this paper.

2. The two-step evaluation cannot be viewed as an important contribution, since MathVista also uses an LLM (ChatGPT) to extract answers from the free-form response of models as the first evaluation stage.

3. The evaluation of some math-domain MLLMs is missing on MathScape, for example: G-LLaVA and Math-LLaVA.

4. Human performance is needed on MathSacpe for better reference.

1. In the first challenge, the author said no existing benchmarks have both the mathematical description and figures being captured together in a single image. However, in MathVerse, one category of questions does provide descriptions and figures together. For the second challenge, the author claims existing method cannot assess long-form responses. But MathVerse proposes a method to assess the correctness of each step of a chain-of-thought response. Authors should conduct a more comprehensive literature review of the multimodal mathematical evaluation domain.
2. This paper is not well organized and written. This means it is not easy to read and understand. For example, section 3.1 is oversimplified. The authors did not mention where they collected the mathematics question and what is the original format of the question documents. Besides, it’s not clear what kinds of annotations are done. What is “knowledge-based classification”?
3. The proposed two-step evaluation method heavily relies on LLM’s ability to decompose and judge the answer. This may cause some errors in the progress. Did the authors examine how accurate LLMs are on each of the evaluation tasks?
4. For the evaluation part:
   1. The model “GLM4V” is without citation, and it is an open-sourced model from my knowledge. (https://huggingface.co/THUDM/glm-4v-9b). Besides, the open-source models in Line 278 are not cited properly. These kinds of format errors cause the paper to be hard to read.
   2. Some reference performance is not provided: e.g., frequent choice, random choice, and human performance.
   3. DeepSeekV2 is not in the evaluation setup models, did you mean DeepSeek-VL?
   4. The performance on proof questions is higher than on choice and solution questions. This is uncommon and the reason given by the authors is not convincing. They said, “The structured format and clear information in proof questions make them easier”. However, when testing models on different kinds of questions. The format of questions is supposed to be similar unless the question format (structure or non-structure) is the primary research topic.
   5. The authors provide limited insights of the performance on MathScape. Results such as “the closed-source models are more accurate than open-source ones” reveal little information.
5. MathScape claims that it is the first to combine both figures and mathematical text descriptions in a single image. What unique challenge does this format of data bring to models? Did the authors dive deep into analyzing the different challenges present by MathScape and other multimodal mathematical benchmarks?

• Insufficient Dataset Details: More comprehensive information about the dataset’s creation, sources, human annotators education level and potential biases would strengthen the paper.
• Limited Language Scope: The focus on Chinese problems limits the applicability of the benchmark to other languages and educational contexts. (Please clearly state the language scope in the abstract and/or in the introduction).
• Evaluation Method Reliance on LLMs: Using LLMs for scoring may introduce biases, as these models may share similar limitations with the models being evaluated. The judgment error is not addressed in the paper's results or case study.
• Lack of Comparative Analysis: Given that all of the problems are available in textual format, the paper will benefit from including correlation analysis between original problems and photo-converted problems solve rate.