MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

W2: Lack of in-context examples in Section 6.1

We apologize for the lack of clarity in Section 6.1. The reason that we summarized model behaviors instead of providing specific examples was originally due to space constraints. We have added these examples in the revised version of our paper we are uploading to the conference and appreciate your suggestion to enhance clarity. Below are the corresponding clarification.

• The "reversal curse" refers to models failing to recognize that equations presented differently (e.g., 112 - 3x = 85 vs. 112 - 85 = 3x) are equivalent.
• Errors involving units occur when models improperly combine quantities with different units, such as adding speed (5 km/h) to time (3 hours).

Clarifications on Tables 1 and 2 and the Conclusion Section

Thank you for pointing out the issues with the table captions and notations. We have also made the following clarifications in the revised version:

• Table 1: The last two columns represent the average number of solution steps and the average step number where the first error occurs.
• Table 2: We have renamed the columns to ACC_step, ACC_reason, and MCC for clarity, and provided definitions for abbreviations like TPR (True Positive Rate), TNR (True Negative Rate), and k (number of shots).
• Few-Shot Examples: We have moved the few-shot examples from the supplementary materials into the appendix for easier reference.
• Conclusions: We revised the conclusion to highlight our key findings and future directions rather than merely summarizing the paper.

We hope these revisions and clarifications address your concerns. Thank you for your constructive feedback, and we would be grateful if you could consider updating your assessment. Wishing you a wonderful day!

Dear Reviewer PXiN,

Thank you for your detailed review and thoughtful feedback. We apologize for any confusion caused and hope our responses below address your concerns.

W1: Limited novelty of the dataset construction procedure

We would like to contend that the primary contribution of our work lies in introducing the Meta-Reasoning evaluation paradigm rather than the dataset construction itself. As noted in our paper (lines 110-113 and 461-467), our paradigm transforms traditional benchmarks into more robust and comprehensive assessment tools. By shifting from result-oriented question answering to process-oriented solution scoring, our approach redefines the evaluation landscape. Models must act as "solution-scoring teachers" rather than "QA students," a fundamental shift that requires more in-depth reasoning and scrutiny.

Our paper demonstrates how this paradigm elevates benchmarks like GSM8K from being saturated and non-differentiating to becoming challenging tools that expose model weaknesses, especially concerning data contamination and memorization issues. This is evident in the struggles of several domain-finetuned expert models. Therefore, the novelty of our contribution lies in reframing the evaluation methodology, not just the dataset's problem structure.

Additionally, our work exposes significant deficiencies in the current training pipelines. By conceptualizing step-by-step reasoning as Markov Decision Process trajectories within the solution space, we highlight the limitations of conventional supervised fine-tuning (SFT), which often relies on a narrow set of success trajectories. This lack of solution space coverage and contrastive reasoning hampers the development of robust, System-2-like thinking. Our findings underscore the need for more sophisticated training methods that incorporate reflection, self-correction, and counterfactual reasoning.

To support our arguments, we hereby present the performance of the o1 series models, which utilize scaled computation budgets for reflection and backtracking:

Experiment Results

These results show that o1-mini and o1-preview models outperform previous state-of-the-art models like GPT-4 Turbo by a substantial margin, supporting our claim that models incorporating deliberate reasoning processes achieve superior performance.

Q2: Do the results of math-specialized models contradict GSM8K results?

We understand your concern and wish to clarify that the poor performance of specialized math models on MR-GSM8K does not imply that our benchmark measures an entirely different set of abilities. Rather, it highlights the potential overfitting to specific question types and formats. Specifically:

1. Format Overfitting: Specialized models, even with few-shot demonstrations, exhibited overfitting to the GSM8K response format. For example, they often used specific format indicators like "###" and tended to provide question-answering outputs, deviating from the expected solution-scoring behavior.
2. Bias Towards Correctness: These models frequently (>90% of cases) predicted solutions as "correct," regardless of the ground truth, revealing a lack of genuine evaluative reasoning.
3. Degraded Reasoning: Their explanations for errors were often nonsensical or incorrect, further indicating superficial reasoning.

Our observations align with Goodhart's Law: optimizing a specific measure (e.g., GSM8K scores) may lead to overfitting, rendering the measure ineffective at capturing true reasoning abilities. While fine-tuning on correct solutions may boost performance on traditional benchmarks, it does not necessarily foster deeper reasoning skills. As elaborated in Section 6 and illustrated by Figure-3 of our paper, we believe our paradigm indicates that a more fundamental way to enhance reasoning is to provide in depth analysis/contrast on each step of the solutions. Our hypothesis is verified from the success of the o1 series models, which, through reflective and iterative thinking, demonstrate superior performance and generalization across a broader domain.

We hope these clarifications address your concerns and illustrate the value of our meta-reasoning approach. We truly appreciate your kind review and if you find the above addresses your concerns, we would be truly grateful if you can consider adjusting your final rating for us. We wish you a wonderful day ahead. Thanks!

Dear Reviewer HcPg,

Thank you for your detailed review and thoughtful questions. We are glad to address your concerns below:

Q1: Is evaluating a solution different from producing one?

This is a very insightful question. We argue that evaluating solutions and producing them are mostly equivalent skills for several reasons.

Firstly, mathematical word problems have long been used to assess the reasoning abilities of language models. When we test LLMs on math problems, we are essentially evaluating their ability to recognize patterns, understand relationships between objects, correctly apply formulas, and logically combine conditions and assumptions. Traditionally, metrics have focused on the final computation result as a proxy for reasoning ability, mainly due to implementation convenience and lack of annotated process oriented data and corresponding evaluation methods. However, this result oriented approach overlooks the accuracy of intermediate steps, which are crucial in assessing the quality of the reasoning process.

Our meta-reasoning paradigm addresses this gap. When models are asked to score the correctness of solutions, they engage in the same fundamental skills: discerning patterns/relations/conditions, performing counterfactual reasoning that includes similar applications of formulas and computation but with an extra step of contrastive comparison. Thus, our solution-scoring benchmark serves the same overarching goal of assessing robust reasoning abilities but does so in a more holistic, challenging and process-oriented manner.

To support our claim, we present the results of the o1-mini and o1-preview models. These models reportedly combine self-reflection and self-correction processes (similar to meta-reasoning paradigm) into their reasoning procedures. They exhibited substantial improved performance on our meta-reasoning benchmark—something that is hard to observe with GSM8K due to the saturation and lack of differentiation.

Experiment Results

These results illustrate that models with effective self-reflection, self-correction, and deliberate reasoning (e.g. the same set of skills Meta-Reasoning is evaluating) excel on our benchmark (with an absolute improvement of 20% against GPT4-Turbo). This suggests that the ability to scrutinize and reason through each step is indeed a fundamental component of robust reasoning.

Dear Reviewer HcPg,

Thank you for your detailed review and valuable feedback. We have carefully addressed your concerns in our rebuttal and would greatly appreciate your engagement during the rebuttal period to discuss whether our responses sufficiently address them. Your insights are highly valuable, and we are happy to provide further clarification if needed.

Thank you again for your time and thoughtful review.

Best regards,

W2: Discussion on actionable insights

We appreciate your suggestion and agree that a discussion on actionable insights would benefit our work. Accordingly, we have conducted additional experiments, and below, we outline some key insights and potential implications.

As discussed in Section 6.2 and depicted in Figure 3, one of the core contributions of our meta-reasoning paradigm is uncovering critical weaknesses in both the evaluation and training pipelines of current models. Viewing the step-by-step reasoning of various questions as Markov Decision Process (MDP) trajectories within the solution space reveals that most supervised fine-tuning (SFT) approaches focus on a limited number of successful trajectories. This results in incomplete coverage of the solution space, lacking contrastive understanding among different solutions, discernment of assumptions, or counterfactual reasoning—all vital elements of System-2 thinking, necessary for robust, reflective reasoning.

To support our hypothesis, we evaluated our MR-GSM8K benchmark on the newly developed o1 series of models, which utilize scaled computational budgets at test time to engage in self-reflection, self-correction, backtracking, and counterfactual reasoning. These models mimic System-2 thinking, demonstrating broader knowledge and coverage of the solution space. As anticipated, models with these capabilities performed notably well in our benchmark, corroborating our hypothesis.

Experiment Results

The o1-mini and o1-preview models demonstrate significant improvements across all metrics compared to GPT4-Turbo, with an absolute performance increase of around 20%. This supports our assertion that incorporating a slow, deliberate thinking paradigm enables more robust reasoning capabilities in LLMs. While our current results are promising, we believe a more thorough investigation into scaling computational budgets at train/test time is beyond the scope of our paper and we would like to leave that in our future work.

We truly appreciate your kind review and if you find the above addresses your concerns, we would be truly grateful if you can consider adjusting your final rating for us. We wish you a wonderful day ahead. Thanks!

Dear Reviewer AEiU,

Thank you for your insightful review and constructive feedback. We hope our response below addresses your concerns adequately.

W1: Limited novelty of the dataset construction procedure

We acknowledge your observation and would like to clarify that our primary contribution lies not merely in the dataset itself but more in introducing the Meta-Reasoning evaluation paradigm. As highlighted in our paper (lines 110-113 and 461-467), this paradigm enables transforming any existing reasoning benchmark into a more robust and comprehensive assessment tool. Specifically, our approach shifts the evaluation focus from result-oriented question answering to process-oriented solution scoring. This transition is akin to changing roles from a student, who merely answers questions, to a teacher, who critically assesses the correctness and reasoning process behind a solution. Consequently, the evaluation poses a more substantial challenge to models.

We devoted significant portions of our paper to illustrating (1) how our paradigm transforms a benchmark like GSM8K from a non-differentiating, saturated metric into one that effectively distinguishes models, and (2) how this transformation enhances robustness against data contamination and memorization, which are common issues arising from opaque pretraining and fine-tuning processes. For instance, our results demonstrate how several domain-finetuned expert models struggled with our benchmark, underscoring the paradigm's strength. Thus, the framing of problems, while important, is secondary to our primary contribution: introducing a more comprehensive and effective meta-reasoning assessment methodology.

Dear Reviewer wbNK,

Thank you for your thoughtful and encouraging feedback. We are delighted to address your concerns and appreciate the opportunity to improve our work.

Clarification on Table 2

We apologize for any confusion caused by the presentation of Table 2. The bolded values indicate the best-performing models within their respective categories. Regarding the abbreviations TPR (True Positive Rate) and TNR (True Negative Rate), we explained them briefly in the main text (line 354) when introducing Table 2 in Section 5.2: “Our evaluation results are shown in Table 2... For Task 1, we also report the true positive rate and true negative rate.” Similarly, the k=0 and k=3 annotations refer to zero-shot and few-shot settings, respectively, as mentioned in Section 5.1 (line 323).

We understand that these explanations were not sufficiently clear in the table caption. We are uploading a new version of our paper and we have explicitly defined these terms in the table description and ensure the caption is more intuitive and reader-friendly.

Thank you once again for your valuable feedback. We hope this clarifies your concerns, and we wish you a wonderful day!