ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection

Overall, this work proposes a good benchmark but has some issues, primarily related to the task introduction, the soundness of the experimental results, and the depth of the analysis.

1. The authors claim the benchmark addresses "more complex" mathematical reasoning, but the dataset contains only K-12 level questions. It would be beneficial for the authors to elaborate on how the error detection tasks for K-12 questions require complex reasoning, better with specific examples.

2. Although the authors provide the formulated task representation in section 2.1, it is not clear or correct enough. Specifically:

   1. From equations on line 188 and 199, The input and output for the two subtasks are not clearly defined. It lacks clarity on whether the question text, image, correct answer, and incorrect answer are part of the input for each of the subtask.
   2. The combined description of inputs and outputs for both subtasks (lines 149, 161, and 166) is confusing, suggesting the task requires training a model that complete the two subtasks simultaneously, which is not the case.
   3. The use of complex variables in Section 2.1 might be unnecessary and make the easy thing more complex, and could be simplified with natural language for better understanding. Overall, I would suggest the author re-organizing the content in section 2.1 and make it clear and simple.

3. The result presentation and metrics for error categorization may have some issues.

   1. The metric used for reason-specific results resembles recall rather than accuracy. For error categorization, using precision, recall, and F1-score would better reflect performance on biased data and biased prediction. Could the authors provide results on these metrics and explain how they might change the interpretation of the results?
   2. The red value for CAL in Table 3 is not the highest, contrary to what is indicated.

4. The conclusion on line 355 cannot be supported. The accuracies of STEP are not consistently better than those of CATE (e.g., InternVL2 actually achieves much better scores on CATE instead), and they are not hugely different even for strong MLLMs in many cases. Additionally, the two subtasks have different numbers of classification types, which may influence the accuracy values.

5. More experimental settings can be considered, such as evaluating the impact of including or excluding correct/incorrect answers in STEP, or testing a pipeline approach where the output of STEP feeds into CATE.

6. While the analysis is detailed, it lacks deeper insights or actionable findings. For example, does the output of the models explain why models miscategorize errors? Are there some fundamental ability shortage of the models that we need to further improve? What can we expect and improve for the future applications of MLLMs in education? Etc. The authors may consider providing deeper analyses and discussion on these or other issues.

Overall, I think more work is needed to establish the robustness of the benchmark.

Given the relatively low performance of the human expert baselines, it’s worth analyzing what the human errors are due to. This would help to:

1. Ensure the low human performance is not due to label noise
2. Better calibrate our expectations of MLLMs

For example, I would not have assumed current generation models would beat or come close to strong human baselines on any of these tasks (given that human educators set the baselines across educational contexts, and that LLMs have well documented flaws in mathematical reasoning).

If human expert performance is really this low on such tasks, then there is a chance that it could be due to:

1. A major, unobserved flaw in our current mathematical education system
2. Low agreement between annotators, which might challenge the establishment of this benchmark as having strong ground truth annotations (if educators can plausibly disagree to this extent)

One way of ensuring #2 does not come into play would be to provide a detailed analysis of inter-annotator agreement. However, the consensus-based process described in Appendix C might not support this (unless the initial disagreements were recorded, and some metric of agreement could be computed from these). In any case, I strongly encourage the authors to provide some reasoning as to why the human performance is where it is, and some evidence that this is not due to factors like label noise, natural and plausible disagreement, etc. I acknowledge that model performance can be evaluated against the human noise/agreement ceiling to mitigate this, but such a low ceiling (e.g. 35.3% on KNOW) makes it a bit difficult to estimate the reliability of the model performance findings.

When thinking about the value of the paper’s findings to the community:

• It’s not clear to me that Finding #1 in the main results (i.e. 3.3.1) makes a significant contribution. It’s generally known that leading closed-source models (e.g. GPT-4o) outperform open models, and this is not surprising given the commercial incentives of model providers and the large cost to train them. Is there some nuance in the results that couldn’t be obtained without this contribution? If so, it would be helpful to highlight this here.
• The extremely high performance of relatively weak MLLMs on CAL doesn’t seem adequately explained by the hypothesis in the paper that such models overfit to this category to avoid learning more complex ones. If the category is indeed simple, then stronger models should perform well on it too, unless (intuitively) doing better on CAL tends to causally lower performance on other categories. So, I think a more compelling explanation of this phenomenon is needed to trust that such a result would generalize. For example, an alternative hypothesis is that these models have seen these or similar questions during their pretraining, trivially memorizing the questions or near neighbors, and the models that do not perform well on this category have not. I notice two papers are cited as to the paper’s hypothesis for the result; if there is some more compelling explanation relating to CAL overfitting, a clear explanation and allusion to those papers’ results would be very helpful.
• Likewise, the explanation for Finding #3 doesn’t seem complete. The analogy to object detection isn’t clear; a more direct analogy would be if localization (predicting where an object is) is simpler than classification (predicting what an object is). However, is this the case empirically? Categorization seems simpler than “fully understanding [objects’] attributes or behavior” since solving a classification task doesn’t seem to require this deeper understanding (e.g. we would not claim that an ImageNet-trained classifier understands the attributes or behavior of zebras, for example, but distinguishing them from horses seems possible via pattern recognition.
• The findings in section 3.2.2 generalize across closed-source vs. open MLLMs, but is there an a priori reason to believe that the openness of a model causally affects its capabilities or confusion on different tasks? Or is this more of a commentary on scale/performance in general? I.e. could a line be drawn between low-performing or high-performing models? This seems like it would agree with the scaling law analysis presented in 3.3.4. Otherwise, again it’s not clear that this interpretation of results is robust to e.g. a low-capability closed-source model (such as local models used in things like Apple Intelligence) or a high-capability open model (such as large LLaMa models).
• Additionally, the analysis in 3.2.2 might benefit from plotting a confusion matrix, rather than highlighting only the most common mis-associations in results (this could be in an appendix). Figures 6 and 7 show the erroneously selected categories, but not what they were supposed to be, which makes it challenging to observe what kinds of nuanced reasoning errors might be driving this behavior (and where models could be improved, more specifically).

Some notes on writing and presentation:

• The abstract obscures the core contributions, and also contains several subjective claims (e.g. “comprehensive”, “rigorous”, “rich” without definition of what is meant by these). It would be helpful to simplify it to focus on the factual contributions of this work.
• Similarly, the applicability of multimodal LLMs to mathematical reasoning tasks is cited as “especially” promising, but no reason is given for this (e.g. over and above text-only LLMs, which are largely used for mathematical tasks currently). I recommend making this clearer, since it would help readers quickly orient to the potential contribution of the given work.
• Line 096: What exactly does “featuring advanced research scopes such as complex reasoning” mean?
• Line 099: It is a bit odd for the paper to refer to its own contribution as insightful. This is a determination traditionally left to external reviewers, the community, etc. Such cases occur in other parts of the paper as well (e.g. in the abstract as noted previously, or like "well-annotated” in the Figure 2 caption). I encourage the authors to carefully use neutral language when discussing their results, and focusing on the facts established in the paper.
• Re: notation in 2.1, $T$ is used to denote problem, but this is a bit confusing (given that text is a subset of a problem, and text also begins with T). Wouldn’t it be clearer to use $P$ for problem, or $Q$ for question, or similar? Considering that $A$ is answer, $S$ is step $\mathcal{M}$ is model, etc.
• Similarly the use of $Step$ for an incorrect step, such that $Step \in S$ is a bit confusing, considering that this also seems to be denoted by $S_{x,i}$. Also, shouldn’t the output be simply $x$, i.e. the step index rather than the step content itself? It might be helpful to do a pass on notation to help simplify and make it more intuitive, so the reader can quickly follow.

Issue: The paper evaluates a broad range of MLLMs, yet the chosen models are not fully representative of the diversity in architectural design or training methodologies. The focus is largely on mainstream MLLMs, which may limit insights into the broader applicability of ERRORRADAR to varied model architectures. Suggestion: Incorporating models that represent different architectural approaches, such as hybrid symbolic-neural systems or models with explicit reasoning layers, could offer a more comprehensive view of how ERRORRADAR evaluates different reasoning frameworks. This would help determine whether specific architectures consistently perform better on certain error types, which could have implications for future MLLM design.

Issue: While ERRORRADAR’s dataset is derived from real student interactions, the paper does not address how the benchmark might perform in actual deployment scenarios, such as in online tutoring systems or educational applications where students interact with AI in real-time. Suggestion: The authors could simulate a real-world deployment setting, assessing factors such as response time, user feedback integration, or adaptability in evolving learning environments. This would add practical value, helping readers understand how ERRORRADAR might function outside of controlled benchmarking scenarios and providing insights into potential deployment challenges.

Issue: Different MLLMs may exhibit varying biases toward certain error categories due to their underlying training data and design, but the paper does not delve into these biases in detail. This oversight could affect the fairness and reliability of the models in educational contexts, where bias could disadvantage certain student demographics. Suggestion: An analysis of category-specific biases in the models would add depth to the findings. For instance, are certain models disproportionately prone to labeling calculation errors over reasoning errors? Investigating these tendencies could inform guidelines for model selection and training adjustments to ensure balanced performance across error types.

Issue: The paper touches on cognitive load associated with different types of errors (e.g., reasoning errors appearing in later problem-solving steps), but it does not provide an in-depth analysis of how cognitive load might impact error detection performance across models. Suggestion: Including a more detailed cognitive load analysis, possibly with additional metrics or models designed to handle high cognitive load tasks, would provide richer insights. For example, testing models’ performance on questions with high interactivity and cognitive demand could highlight specific areas where models may benefit from cognitive load-aware training.

Issue: While the dataset annotation process is described, details regarding inter-annotator agreement, consistency checks, and the process for resolving discrepancies are sparse. This information is crucial for understanding the reliability of the dataset, especially given the complexity of categorizing mathematical errors. Suggestion: Adding inter-annotator agreement metrics, outlining quality control measures, and discussing how ambiguous cases were handled would strengthen the dataset’s credibility. It would also help readers gauge the dataset’s reliability for evaluating complex reasoning tasks.

Issue: The evaluation focuses on overall model performance, but lacks targeted baselines or ablation studies for each error type, especially for the multimodal aspects of error detection, which involve both textual and visual information. This limits the insights into specific error categories and multimodal model performance. Suggestion: Introducing error-specific baselines, such as comparing models on tasks that isolate visual or reasoning errors, would clarify which aspects of ERRORRADAR are most challenging. Ablation studies that selectively disable visual or language components could also reveal the contribution of each modality to overall performance, guiding future multimodal benchmark development.

Issue: The paper does not address the computational demands of using ERRORRADAR to evaluate large models, which could impact its adoption in academic and industry settings with limited resources. Suggestion: Discussing the benchmark’s computational requirements, particularly for larger MLLMs, and proposing strategies to improve efficiency (e.g., optimized sampling or reduced evaluation steps for preliminary testing) would make ERRORRADAR more accessible. Additionally, insights into potential trade-offs between accuracy and computational efficiency could benefit readers interested in practical deployment.

References: Key references are missing, undermining the paper’s scholarly foundation. For example, MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, a relevant multimodal benchmark is omitted and should be cited in the discussion of related work. It includes collage level mathematics questions and has a similar error classification as your paper. Besides, the mention of unrelated fields, such as e-commerce and urban planning (P1 L38-39), distracts from the focus on educational applications and should be removed.

Error Analysis: While the error analysis identifies issues and possible explanations, it would be more valuable if accompanied by actionable suggestions. For instance, in Finding #4 within the main results, the authors could propose adding domain-specific knowledge to the dataset of foundation model as a possible solution. The improvement suggestions are only observed in 3.3.5 Visual Perception Case Study, which makes the analysis feel limited in scope. Without concrete recommendations throughout, the analysis lacks depth and does not contribute substantially to advancing practical solutions in the field.

Writing and Structure: The paper’s structure is suboptimal. Placing the related work after the introduction would provide a clearer flow, allowing readers to understand the motivations and contributions of ERRORRADAR in context with prior work. Specific points also lack consistency, such as the mixed use of "Obs.2" and "Finding #2" in P9 L455 and P7 L349, which should be standardized.

Figure Presentation: The figures are challenging to interpret and require considerable improvement. For instance, P6 Figure 5 lacks clarity, especially in differentiating the small section labeled "other" for miniCPM. The pie chart format does not effectively convey comparisons between models; a diverging bar chart could better illustrate the "CAL" percentage. Additionally, the color scheme is confusing, with the inner pie chart darker than the outer one with no meaning. It is also unnecessary, as the values are reflected in the size of the pie chart. Similarly, the images of questions in P10 Figure 10 suffers from low resolution, making the table at the central and other details difficult to read. Also, the organization of 5 cases can be improved, for example, they can be presented separately. The quality and readability of both figures should be substantially enhanced.