MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency

We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.

Q1: Accuracy concern of questions of multiple correct solutions

We believe this concern is addressed in our paper. Our methodology explicitly accounts for questions with multiple correct solutions in both annotation and evaluation phases:

1. Annotation Phase: As detailed in Lines 203-206 of the main paper, annotators are explicitly instructed to provide all key steps for all possible solution paths.
2. Evaluation Phase: Our evaluation framework accommodates multiple solutions:
   • For recall computation (Lines 245-246 and Equation 1), we compute recall scores for all possible solutions and select the maximum as the final recall score
   • For precision, relevance rate, and reflection quality, we provide GPT-4o with key step annotations for all possible solutions in the evaluation prompt, enabling proper assessment of the CoT process considering all valid approaches

Q2: How is CoT obtained in models that don't generate it autonomously?

Thanks for your advice. We have incorporated the illustration of how to obtain the CoT output in Line 257 - 261 in the right columns of the main paper. Specifically, we employ a CoT prompt to instruct the model to first perform step-by-step reasoning and finally give the answer. As illustrated in Line 355-358, the CoT prompt is:

Please generate a step-by-step answer, include all your intermediate reasoning process, and provide the final answer at the end.

We empirically find that the models that don't generate CoT automatically like LLaVA-OV could give a detailed reasoning process before providing the final answer.

GLM-4v cannot receive multiple images as input, so we only evaluate DeepSeek-vl2. The result is:

Please refer to the detailed table in Table 1 in https://anonymous.4open.science/r/reb-3538/README.md

Q3: Selection process for key step annotation and the number of key step annotation for each QA

Thanks for your advice. We want to clarify the two points below:

1. We have illustrated how to obtain the key steps in Line 187 - 205 in the main paper. We detail the process below:

   Key steps are defined as necessary steps for answering the question correcly, such as identifying critical values in math diagrams or reaching intermediate conclusions essential to determining the final answer.

   To efficiently annotate key steps for all questions, we implemented a two-phase process. First, GPT-4o generated draft annotations using the questions, images, and final answers as inputs. Including the final answers significantly improved draft quality compared to using only questions and images. Second, human annotators reviewed these drafts, correcting any errors or developing key steps independently when GPT-4o failed to provide reasonable output.

   All the key steps fall into two categories:

   1. Inference conclusions: Necessary conclusions reached through logical inference steps (including the final answer)
   2. Image captions: Identifications of critical visual information

   We reduce all steps to their simplest form, preserving only core conclusions and relevant visual element descriptions. For problems with multiple solution paths, annotators are required to provide all possible methods.

2. All QA corresponds to multiple key steps.

   1. We provide a dataset visualisation in the bottom part of Fig. 2. The key caption and key conclusion correspond to the inference conclusions and image captions of the key steps. We will make this figure more clear in the final version.

   2. We also provide the statistics of the key step annotations in Table 1. There are 837 reasoning questions with 3,865 key step annotations in total. On average, each question contains 4.6 key step annotations. This result can also be derived by the summation of the average inference conclusions and image captions listed in the table. We further look into each question and find out that all the questions have at least 3 key step annotations. We will make this more clear in the final version.

   We will explicitly add this result in the table in the final version.

We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.

Q1: Potential GPT-4o bias in evaluation

Thank you for your valuable advice. We want to address your concern from two aspects:

1. High Human-GPT-4o Alignment

   Our human alignment experiment shows strong correlation between GPT-4o evaluations and human judgment, confirming GPT-4o as a reliable tool for CoT evaluation. Specifically, we investigate two perspectives:

   1. Human Agreement Rate: A binary (yes/no) human evaluation to assess agreement with the model's per-step judgments.
   2. Hallucination Detection: We assess whether any reflection steps identified by GPT-4o are hallucinated or contain hallucinations.

   We cover four key metrics: Recall, Precision, Relevance rate, and Reflection quality. We randomly sampled 54 predictions (9 questions from each subject) from QwenVL2-72B and QvQ-72B, totaling 216 predictions and 2,368 steps. The results are as follows:

2. Best Available Automated Evaluation Method

   While acknowledging the inherent limitations, GPT-4o represents the current state-of-the-art approach for automatic CoT evaluation. GPT-4o has been widely adopted and validated for evaluation across various multimodal [1] and reasoning tasks [2]. Given our reference solutions and human alignment results, we believe GPT-4o is the most reliable option currently available for this complex evaluation task.

Q2: Issues of true direct answers

Thanks for your valuable advice. We report the actual direct answer (defined as less than 20 words) ratio in the table below:

Four models demonstrate reluctance to provide direct answers (we mark these four models with * in Table 2). Based on this, we recalculate stability and effectiveness using only questions that receive direct answers. The updated table is shown in Tab. 1 in https://anonymous.4open.science/r/reb-3538/README.md

We would like to emphasize that all conclusions in our current paper remain valid. Our analysis and conclusion do not consider any results with * to ensure validity. Thanks for pointing this out, and we will stress this further in the final version.

Q3: CoT error analysis

Thanks for your advice. We provide the analysis of CoT error types and reflection error types.

• CoT Error Types: Visual Perception, Visual Reasoning, Logical Reasoning, Calculation.
• Reflection Error Types: Ineffective Reflection, Repetition, Incompleteness, Interference.

Please refer to the example and error ratios in Fig. 3-6 in the link above.

Q4: Human verification of GPT-4o correctness

Please refer to our response to Q1.

Q5: The cost of evaluating GPT-4o

The average cost of evaluating one model is 16 dollars. We have also experimented with evaluating with GPT-4omini for all the tasks, but the human evaluation shows low agreement. With the recent development of LLM, the open-source models might be used for evaluation in the future.

We also plan to release a testmini set, which composes 200 questions with all the subjects included. Evaluation using testmini can reduce evaluation costs to around 3.5 dollars.

Q6: Most models score 100 in reflection quality

Yes, as illustrated in Line 381-382, for models not generating reflection, we define their reflection quality to be 100 since the absence of reflection can be viewed as the most efficient approach.

Q7: How does GPT-4o judge the answer for OCR tasks?

Thanks for your advice. We look into 20 predictions and their corresponding evaluation results. We observe that:

1. The tested models typically only OCR the content directly relevant to the question rather than performing comprehensive OCR on all text. The models tend to summarize other visual information briefly. This results in concise OCR content in predictions.
2. We observe that GPT-4o demonstrates high accuracy in judging OCR content, likely because the OCR content is usually highlighted in the key step annotations. Nevertheless, we will still enhance our evaluation prompt to explicitly instruct GPT-4o to follow the OCR metric design you suggested, ensuring that OCR responses are only considered correct when they exactly match the reference.

[1] Tarsier: Recipes for training and evaluating large video description models

[2] Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?

We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.

Q1: Difficulty in understanding Sec 2.2 and difference with existing metrics

Thanks for your valuable advice. We improve the clarity of Sec 2.2 below:

For CoT evaluation, we provide key steps and reference image captions for all questions.

• Key steps are defined as necessary steps for answering the question correcly.

  All the key steps fall into two categories:

  1. Inference conclusions: Necessary conclusions reached through logical inference steps (including the final answer)
  2. Image captions: Identifications of critical visual information

  To efficiently annotate key steps for all questions, we implement a two-phase process. First, GPT-4o generates initial versions of key steps annotations (containing both inference conclusions and image captions) with the questions, images, and final answers as inputs. Including the final answers significantly improves the quality of the initial versions compared to using only questions and images. Second, human annotators review these initial versions, correcting any errors or developing key steps independently when GPT-4o failed to provide reasonable output.

  We reduce all steps to the simplest form, preserving only core conclusions and relevant visual element description. For problems with multiple solution paths, annotators are required to provide all possible methods.

• Reference image captions are visual information not covered by the image caption in key steps.

  These reference captions are mainly for the calculation of precision. We use the same method as key steps to obtain the annotation: GPT-4o first generates initial version and then annotators review and correct the errors.

Difference with existing metrics:

Few works evaluate the multimodal CoT process. MathVerse [1] represents one such effort. MME-CoT differs in several key aspects:

1. MME-CoT evaluates the CoT quality in two ways: precision and recall, corresponding to CoT faithfulness and informativeness. While [1] only instructs GPT-4 to judge the correctness of step, which could be viewed as only evaluating precision.
2. MME-CoT contains ground truth key steps annotation, enabling more reliable evaluation, especially for tasks beyond GPT-4o ability. [1] contain no such GT reference.
3. MME-CoT considers both inference conclusions and image captions in CoT on all visual reasoning tasks. [1] only targets on math tasks without identifying different types of CoT steps.

Q2: Need to consider tasks involving both perception and reasoning

Sorry for the confusion caused. We'd like to clarify our task definitions:

• The perception tasks are the tasks primarily test visual recognition abilities or require very minimal reasoning.

• The reasoning tasks additionally require logical inference steps along with visual perception tasks. Therefore, tasks requiring both perception and reasoning are exactly the reasoning tasks in MME-CoT.

  Examples in Fig. 2 (bottom section) of our paper all showcase this two requirements: first perceiving visual cues, and then reasoning based on the perception.

• For your examples: Counting red balls belongs to perception task since it requires minimal reasoning. Determining relationship between people belongs to reasoning task. Similar reasoning task also occurs in MME-CoT, as shown in Fig. 1 in https://anonymous.4open.science/r/reb-3538/README.md

Q3: The reflection evaluation seems to be beyond the model's capability

Thanks for your valuable advice. We identify GPT-4o as an well-qualified evaluator for assessing the reflection quality:

1. GPT-4o shows competitive results of reflection quality evaluation in the human agreement experiments.

   We conduct additional human evaluations to verify the validity of GPT-4o assessment from two perspectives:

   1. Human Agreement: A binary (yes/no) human evaluation to assess agreement with the model's per-step judgments.
   2. Hallucination Detection: We assess whether any reflection steps identified by GPT-4o are hallucinated or contain hallucinations.

   We randomly sample 54 predictions (9 questions from each subject) from QvQ-72B. The results are below:

2. Additional instruction for reflection quality evaluation prompt.

   Acknowledging the challenges in identifying valid reflections, we incorporate specialized prompt design. We identify and list common reflection errors to better guide validity assessment (detailed in Lines 906-910).

Q4&5: Comparison with existing metrics and more examples

We provide the comparison in Fig. 7 and more examples in Fig.8-18 in the link above.

[1] Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?

We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.

Q1: Need studies on human agreement

Thank you for your advice. As you suggested, we conduct additional human evaluations to verify the validity of the GPT-4o assessment from two perspectives [1, 2]:

1. Human Agreement Rate: A binary (yes/no) human evaluation to assess agreement with the model's per-step judgments.
2. Hallucination Detection: We assess whether any steps identified by GPT-4o are hallucinated or contain hallucinations.

Our human agreement study covers four key metrics: Recall, Precision, Relevance rate, and Reflection quality. We randomly sample 54 predictions (9 questions from each subject) from QwenVL2-72B and QvQ-72B, totaling 216 predictions and 2,368 steps. The results are as follows:

These results demonstrate a high correlation between GPT-4o evaluations and human judgment, indicating that GPT-4o is a reliable tool for CoT evaluation. This result also indicates that all of our analysis and conclusion are valid.

We will incorporate references to the papers you suggested in our final version.

Q2: Confidence interval is not provided

Thanks for your advice. Our default experiment setting uses temperature 0 for GPT-4o to ensure reproducible results. To address your concern, we set the temperature to 1 and evaluate 50 predictions for 5 times. The resulting 95% confidence interval widths are below:

These narrow confidence intervals demonstrate the statistical reliability of our findings. The largest interval width is only 0.069 for Reliability, indicating that our results are stable and reproducible even with stochastic sampling.

Q3: Small separation between models of the current metric

We respectfully disagree with the reviewer's assessment for the following reasons:

1. MME-CoT separation is comparable to other benchmarks:

   The difference is listed below:

   Benchmark	Difference
   F1-score (MME-CoT)	3.05
   MMMU	0.83
   MathVista	1.43
   MathVerse	3.47

2. The robustness and efficiency scores differ from traditional metrics, so the scale of the model's separation is also different.

   • The robustness score measures the performance difference within different prompts. This difference results from the intrinsic model attributes.
   • The efficiency score specifically identifies models with excessively long CoT and reflection steps (e.g., QvQ and Virgo). As shown in Table 2, these two models score over 20 points lower in CoT efficiency compared to others, demonstrating significant separation.

Q4: Experimental setup is too simple

We respectfully disagree that our experimental setup is too simple:

1. Our experimental setup is sophisticated and comprehensive

   Compared with previous multimodal evaluation works [3, 4], our study explores multiple dimensions: different prompt strategies (CoT prompt vs. direct prompt), evaluation methods (CoT evaluation vs. direct evaluation), and diverse evaluation aspects (quality, robustness, and efficiency).

2. Our focus is assessing natural CoT behavior, not improving it

   The primary objective of our paper is to evaluate how current LMMs perform reasoning when confronted with problems and to assess the quality of their reasoning processes. We specifically examine how models behave with standard prompting approaches rather than engineering prompts to encourage specific behaviors. The latter, while valuable, is beyond our current scope and for future research.

Q5: Results of Claude and Gemini

Thanks for your advice. The results of Claude and Gemini are listed below:

Please refer to the updated table in Table 1 in https://anonymous.4open.science/r/reb-3538/README.md

[1] MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

[2] Personalized Video Comment Generation

[3] Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?

[4] Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi