VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning

While the paper provides a general categorization of errors, a more in-depth analysis of the types of errors made by MLLMs across different subjects could offer more targeted insights for model improvement. Suggestions for specific model enhancements or training techniques based on error analysis would be valuable additions.

This submission creates a benchmark, which is a notable contribution without a doubt. However, there are some unignorable weaknesses that, if addressed, could greatly enhance the clarity and usability of the benchmark.

Weakness #1: Ambiguity in Dataset Annotation Process

The dataset annotation process described in Section 2.2 is ambiguous, with multiple steps left unclear. Specific questions regarding this process include:

(1-1) Source of Translations: How were the English and Chinese versions of each question obtained? Which version served as the source, and how were questions and answers translated—via large language models (LLMs) or human annotators?

(1-2) Quality Control and Verification: What steps were taken to ensure quality and correctness in the annotations? The authors mention “we check xxx” and “we screen out xxx” (Lines 202-211), but it is unclear who "we" refers to. Was this verification conducted solely by the authors or by additional human annotators? Additionally, was the check single-pass, or did multiple judges perform repeated rounds of verification?

(1-3) Presence of Bilingual Images: The K12 questions listed in the Appendix indicate that many images contain Chinese characters. Are English versions of these images also available?

(1-4) Question-Image Pairing: In Table 1, the authors report 3k questions and 3k images. Does each question have an associated image, or are there cases where a single question is linked to multiple images?

Weakness #2: Lack of Clarity in Experimental Setup and Analysis

Certain aspects of the experimental setup are unclear, and additional analysis is needed for some experimental results. Specific questions include:

(2-1) Input for English-specialized MLLMs: In Lines 1052-1053, the authors state that “The English version of VisScience is designed to facilitate the evaluation of MLLMs that specialize in English, assessing their capabilities in scientific reasoning.” Were English versions of the images provided as inputs to these MLLMs during evaluation?

(2-2) Unexpected Performance of LLMs vs. MLLMs: In Table 2 and Table 3, text-only LLMs that accept text questions but no image inputs sometimes outperform MLLMs with access to both text questions and images. Why might this discrepancy occur?

(2-3) Questionable Necessity of Image Inputs: A followup question of (2-2) -- the accuracies on some text-only LLMs are higher than random guess (25% for multiple choices with 4-choice), is it because the image inputs are not necessary, or is it because the LLMs have already been posted to these questions during training that they don't even need to look at the paired image input to answer?

Weakness #3: Benchmark Score Evaluation Methodology

The evaluation method for benchmark scores is unclear, particularly in Section 3.1, Lines 316-317, where the authors mention GPT-4o as the judge. Questions related to score evaluation include:

(3-1) Handling Multiple Correct Answers: For example, in the first example in Figure 11 on page 30, a multiple-choice question has multiple correct answers (ground truth C & D). If a model selects only C, is this considered correct, partially correct, or incorrect? How is accuracy calculated in such cases?

(3-2) Accuracy for Multi-part Free-form Questions: In Figure 19 on page 38, a free-form question contains multiple sub-questions. How is accuracy determined here? Do all sub-questions carry equal weight, and how many must a model answer correctly for GPT-4o to deem the entire question correct?

(3-3) Weighting Free-form vs. Multiple-choice Questions: Free-form questions often contain multiple sub-questions. When reporting overall accuracy, do these free-form questions carry the same weight as single-answer multiple-choice questions? For instance, if a free-form question has five sub-questions, how is the overall score calculated?

1. Bilingual Dimension: As a benchmark for evaluating MLLMs, the language of the text within images is highly significant. The authors present two languages in VisScience. However, through the Appendix, it can be observed that in English questions, most images still contain Chinese text.
2. Difficulty Level Classification: The authors describe the use of LLM for classifying the difficulty levels, while VisScience is a multimodal benchamrk. I have some concerns about using text alone for classification without considering the images. For instance, in a math problem asking about an area, the complexity presented by the image can vary greatly, which cannot be detected by simply using LLM for classification.

• VisScience includes bilingual tasks in Chinese and English; however, in the case studies presented in the appendix, it is observed that the text within images corresponding to English questions might be in Chinese. This bilingual mixing situation could lead to different reasoning capabilities of the models when processing information in different languages, especially during the integration of visual and linguistic information. Therefore, it is suggested that the authors further investigate and address this issue in future research to ensure that the models can perform stably in true multilingual environments.
• The paper mentions that the data underwent multiple checks using manual review and LLM assessment, along with some screening principles. However, the details of this process are not elaborated upon in the paper. This process is crucial for ensuring the quality of the dataset and the accuracy of model evaluation. It is recommended that the authors supplement the paper with a detailed description of this process.