MathEval: A Comprehensive Benchmark for Evaluating Large Language Models on Mathematical Reasoning Capabilities

Firstly, we would like to express our sincere gratitude to the reviewers for their insightful feedback.

Q1: The contribution is limited to the aggregation of benchmarks (some of them recently introduced with less contamination, but not evaluated separately) and automated grading.

A1: MathEval offers a Comprehensive Evaluation Suite, which fills a crucial gap by providing a unified and thorough benchmark specifically designed to evaluate the mathematical reasoning capabilities of large language models (LLMs).

The significance of this contribution seems to have been overlooked, and we aim to clarify it. As shown in Figure 1, MathEval's Comprehensive Evaluation Suite consists of three parts: Math Scenarios, Prompt Adaptation, and LLM-based Evaluation. These correspond to the three issues identified in the Introduction: incomprehensiveness, inadequate adaptation, and inconsistency. Specifically: (1) Math Scenarios, which encompass languages (Chinese and English), problem types (arithmetic and math word problems), and educational levels (primary, middle, and high school), comprehensively address the challenge of incomprehensiveness; (2) Prompt Adaptation, which selects tailored datasets and model templates based on specific dataset characteristics and model information, effectively tackling the problem of inadequate adaptation; (3) LLM-based Evaluation, which utilizes GPT-4 for answer extraction and comparison to mitigate inconsistency issues, with an alternative distilled compare-answer model available for users without access to GPT-4.

Thus, MathEval can leverage this Comprehensive Evaluation Suite to adapt to new datasets and models, providing a comprehensive and fair evaluation. This is the core of MathEval as a comprehensive benchmark for evaluation. The other contributions, such as expanding the dataset, performing multidimensional evaluation result analysis, and providing a Reliable Answer Extraction Model, are merely partial outcomes in the process of building MathEval.

Q2: It's not clear whether the collection includes very difficult problems and may saturate soon. Current best results are around 70%, but not sure if there are some datasets or subdatasets where errors are quite high still. Aggregate results do not give a lot of insight.

A2: Aggregate results indeed are indeed insufficient to give a lot of insight. Given the 52 models across 22 datasets, totaling 1,144 results, it is challenging to display all the data. Therefore, evaluation is conducted using high-level dimensions. In addition to the average score, Sections 3.4 and Appendix E present several analyses from different perspectives and uncover several intriguing insights. These include dimensions such as languages, problem types, educational levels, the differences between closed-source and open-source models, and the evaluation discrepancies arising from few-shot/zero-shot settings. Furthermore, the Gaokao-2023 results in Figure 13 on page 29 of the Appendix are used to discover potential data contamination.

For further exploration of other perspectives or individual dataset-level analysis, a website will be provided to include all the specific evaluation results to support the discovery of more detailed findings. Specifically, all datasets currently covered are listed in Table 2. For particularly challenging datasets, this is an ongoing maintenance process, which includes the continuous expansion of the GAOKAO series evaluations and the newly added OlympiadBench evaluations. Current best results for these datasets are around 50% and 30%, respectively.

Q5: Figure 3 and the related text is confusing. It's not clear whether all the things in blue and green are alwasy used, but not the one in black: "[COT prompt]". Is this optional? When is it introduced?

A5: Referring to Figure 3, Figure 3(a) represents two independent processes: Model and Dataset Configuration. An example of this process is provided in Figure 19. In Figure 3(b), the prompts from Model and Dataset Configuration are wrapped for use in zero-shot and few-shot settings, with examples shown in Figures 20 and 21.

Specifically, the Dataset Configuration prompt templates include DQP, DAP, DOP, and COT prompts. The first three are clearly indicated in Figure 3(b) with their positions for concatenation. The COT prompt is more specific, as its placement depends on the dataset. Therefore, it is difficult to indicate this in the figure.

The use of the COT prompt is another issue. In a zero-shot setting, the COT prompt is always used, but in a few-shot setting, some datasets do not have a COT process, such as arithmetic problems. In those cases, the shots provided do not involve a COT process, meaning the COT prompt may not be applied. The use of [COT prompt] is therefore optional and depends on the dataset. The [] signify that this is conditional based on the dataset.

Q6: Is the configuration of prompts per model and dataset different? What if we need to explore a new model? Should we try to find the best prompts for each and every dataset in MATHBENCH?

A6: As mentioned in A3, when introducing a new model, the only thing that needs to be adjusted is the new model template, which is generally based on the prompt provided upon the model’s release.

Q7: The details about "Calculation Scheduling" and parallel processing are not part of the benchmarks, and definitely not part of the "prompts section". This is just experimental details or it could go to the appendix.

A7: Calculation Scheduling is indeed not directly related to prompt adaptation, but it is part of the answer generation process, as illustrated in Figure 3. It is an important aspect of the practical implementation of a benchmark. Reviewers, such as Reviewer Nkro, may focus on its specific implementation details.

Q8: The evaluation results are based on an arithmetic mean of all datasets. This is a common practice but requires a justification, as the different datasets are incommensurate in difficulty. Why are easy datasets weighting the same as hard datasets? Do we have models failing at easy items but succeeding at difficult ones? Averages are not the best way of comparing systems. It is telling that the paper also reflects the results for GSM8K and MATH, and they see only minor discrepancies, so what's the point then about this comprehensive dataset if the same results could have been obtained with only GSM8K and MATH?

A8: This addresses two issues. First, the arithmetic mean is indeed a single metric. As explained in A2, we have also performed analysis across three dimensions, including Grade, and provided the complete evaluation results for further analysis. Secondly, the results for GSM8K and MATH, referenced in Appendix F.3 (Table 6), are meant to help with the development of the Evaluation Suite and validate the accuracy of the evaluation, not to suggest that these benchmarks alone are sufficient.

Q9: With this aggregate results, many of the observations in Fig. 6 are confirmatory, such as the best LLMs for maths (as we knew for some other dataset collections) are the best for this benchmark, and also the effect of finetuning, but specifically parameters (perhaps FLOPS would have been a better metric).

A9: Thank you for the suggestion. In addition to the confirmatory observations, we also identified some additional findings, which are discussed in Section 3.4 and Appendix E. For example, newer model series exhibit steeper slopes, indicating that their mathematical abilities improve more effectively with an increase in parameter size.

Q10: The separation between Math word problems and arithmetic in Fig. 6 (bottom) is more insightful, but the arithmetic variability is not explained (this is partly explained by "arithmetic plugins", but isolated benchmarks with basic operations using large numbers could have been conducted to know what models are using them or not).

A10: Thank you for the suggestion. This could potentially lead to a new dataset design, and we are considering adding this to MathEval in the future.

Thank you very much for your thorough review. As the discussion period is nearing its conclusion, I wanted to follow up to ensure you’ve had the opportunity to review our detailed rebuttal. Given the additional explanations and adjustments we've incorporated, we would greatly appreciate your feedback on whether our responses have satisfactorily addressed your concerns.

Thank you once again for your time and thoughtful review. We look forward to your response.

3. Addressing the correlations between performances on all 22 datasets:

We appreciate your insightful suggestion. In response, we have conducted a comprehensive correlation analysis, which is presented in Figure 10 on page 19 (lines 850-861) of our revised submission.

• Findings from the Correlation Analysis (Figure 10): Our analysis shows that there are indeed high correlations between model performances on many of the datasets. We consider this to be a reasonable and expected outcome because mathematical abilities are interconnected. Improvements in computational skills often enhance problem-solving capabilities, and advancements in reasoning skills typically benefit performance across various mathematical tasks. This interrelated nature of mathematical competencies suggests that as models improve in one area, they tend to improve in others as well. Which proves our benchmark is reliable.

• Justification for Including Multiple Datasets:

  • Enhancing Benchmark Robustness: Including all 22 datasets increases the robustness and reliability of our benchmark. A comprehensive evaluation across diverse datasets ensures that models are not just performing well on a narrow set of problems but are demonstrating consistent mathematical reasoning abilities across a wide spectrum of topics and difficulty levels.
  • Identifying Specific Strengths and Weaknesses: Even with high overall correlations, the detailed performance on individual datasets can reveal specific strengths or weaknesses of a model.
  • Detecting Potential Data Contamination: A significant benefit of using multiple datasets is the ability to detect potential data contamination. For example, suppose Model M has high performance on Dataset A but does not show similar performance gains on other highly correlated datasets. This discrepancy might suggest that Model M has potentially memorized Dataset A due to data leakage, rather than genuinely understanding the underlying mathematical concepts.

• Measuring Orthogonal Abilities: While correlations are generally high, our analysis also reveals that certain datasets measure more specialized or orthogonal abilities. For instance, some datasets focus on advanced reasoning or problem-solving skills that may not correlate as strongly with basic computational abilities. Including these datasets allows us to assess a wider range of mathematical proficiencies.

4. Insights derived from the comprehensive evaluations:

By conducting comprehensive evaluations across all datasets, we uncovered insights that wouldn't have been apparent from analyzing just a small subset. One significant discovery was the potential data contamination identified in Figure 13 on page 29 of the Appendix.

This figure was instrumental because it highlighted discrepancies in model performance on the Gaokao-2023 dataset—a brand-new set of questions that none of the models had encountered during training. Given the high correlation in performance across datasets (as discussed in our previous answer), we wouldn't expect significant variations on Gaokao-2023 if there were no contamination. Therefore, substantial differences in rank suggest potential data contamination.

In the Upper Chart of Figure 13:

• Chinese Subsets Rank (Blue Bars): Indicates each model's ranking within Chinese mathematical datasets. A smaller rank signifies better performance.
• Gaokao-2023 Rank Increase (Orange Bars): Represents models whose rank increased (i.e., performed worse) on Gaokao-2023 compared to other datasets. A larger increase indicates poorer performance on Gaokao-2023.
• Gaokao-2023 Rank Decrease (Green Bars): Represents models whose rank decreased (i.e., performed better) on Gaokao-2023. A larger decrease signifies better performance on the new dataset.

In the Lower Chart:

Similar to the upper chart, but the blue bars represent the overall average score across all 22 datasets.

From the figure, we detected potential data contamination:

• The top two models showing a significant increase in rank (poorer performance) on Gaokao-2023 are ChatGLM3-6B and Baichuan2-13B.
• Many of the Qwen-series models display orange bars, suggesting they may have been trained on data overlapping with our evaluation sets, leading to inflated performance on those but not on Gaokao-2023.

These observations are further supported by findings in the paper "Compression Represents Intelligence Linearly", which discusses similar issues for Qwen-series model. Furthermore, most base models exhibit green bars, which suggests that chat models are more likely to have encountered similar math word problems during the instruction fine-tuning stage, increasing the probability of data contamination.

We sincerely thank the reviewer for the thoughtful feedback and insightful questions. We are pleased to hear that you find our paper easy to read and acknowledge the utility of MathEval for the community

1, Regarding the novelty and contribution of MathEval:

We appreciate reviewer's concern about the novelty aspect of our work for an ICLR research paper. While we acknowledge that benchmarks may not always introduce novel methodologies, we believe that the contributions of MathEval are significant for the following reasons:

• Comprehensive Evaluation Suite: MathEval fills a crucial gap by providing a unified and comprehensive benchmark specifically designed for evaluating the mathematical reasoning capabilities of large language models (LLMs). It encompasses 22 diverse datasets that cover various problem types (e.g., computation, application, multiple-choice), difficulty levels (elementary to high school), and languages (English and Chinese).
• Reliable Answer Extraction Model: We address the substantial challenge of answer extraction in mathematical problem-solving by introducing a fine-tuned DeepSeek-Math 7B model. This model enhances the reliability and consistency of evaluations, which is essential for fair model comparisons.
• Insights into LLM Performance: By conducting extensive experiments on 52 models, we provide valuable insights into the strengths and weaknesses of current LLMs in mathematical reasoning across different dimensions, including language proficiency, grade levels, and potential data contamination.

Furthermore, as benchmarks like MathVista and BooookScore have demonstrated in prior ICLR publications, the contribution of a well-designed benchmark lies in its ability to propel the community forward by providing a reliable and robust platform for evaluation. We believe MathEval will be instrumental for future research in this domain.

2. Regarding the potential overlap among datasets and the aspects they measure:

We appreciate reviewer's insightful concern about the potential overlap in our datasets and whether they are truly measuring different aspects of mathematical reasoning beyond multilinguality.

To address this, we have taken several steps in our revised submission:

• Detailed Dataset Classification (Appendix B.2, Table 2): We have included a comprehensive table in Appendix B.2 (pages 16-17) that lists each dataset along with its specific classification.
• Analysis of Potential Overlap Between Datasets- (Page 16, 17, Line 711-731, Appendix B.2):
  • Query Similarity Heat Map (Figure 8): To examine the overlap between datasets, we conducted a query similarity analysis using embedding techniques to represent each problem query. We computed pairwise cosine similarities between queries from different datasets and visualized the results in a heat map (Figure 8). The heat map illustrates that the average similarities between queries from different datasets are low, indicating minimal overlap in problem content.
  • Dimensionality Reduction and Clustering (Figure 9): Furthermore, we applied dimensionality reduction (e.g., t-SNE or UMAP) to the query embeddings and performed clustering to visualize the distribution of queries across datasets (Figure 9). The resulting plot shows that queries from different datasets form distinct clusters, suggesting that each dataset covers unique topics and problem types.
• Findings from the Analysis - (Page 16, 17, Line 711-731, Appendix B.2):
  • Coverage of Diverse Difficulty Levels: Our datasets collectively cover a wide range of difficulty levels, from elementary school mathematics to high school competition-level problems. This ensures a comprehensive assessment of models across varying complexities.
  • Low Inter-Dataset Similarity: The low similarity scores between queries from different datasets confirm that each dataset presents unique challenges. This indicates that the datasets are measuring fundamentally different aspects of mathematical reasoning, such as basic computation, complex problem-solving, logical reasoning, and application of mathematical concepts in various contexts.

By conducting this detailed analysis and including these findings in our revised paper, we aim to demonstrate that the datasets within MathEval are not overlapping but are instead complementary, each contributing to a holistic evaluation of mathematical reasoning in LLMs.

Thank you very much for your thorough review. As the discussion period is nearing its conclusion, I wanted to follow up to ensure you’ve had the opportunity to review our detailed rebuttal. Given the additional explanations and adjustments we've incorporated, we would greatly appreciate your feedback on whether our responses have satisfactorily addressed your concerns.

Thank you once again for your time and thoughtful review. We look forward to your response.

Thanks reviewer for the thoughtful review, we would like to address the concrens and questions you raised:

For Weakness 1:

We understand that the reviewer is pointing out that our evaluation standard may not be sufficient for assessing the mathematical process itself. In our current framework, we choose to directly evaluate whether the final output of the model is correct. This decision was made because evaluating the solution process is a complex issue that we have considered extensively.

Recent methodologies, such as Program Reward Models (PRM) or those based on attention score distribution for hallucination detection, have not yet demonstrated reliable accuracy in evaluating reasoning steps. Incorporating such methods into our benchmark could introduce unnecessary variables and potentially undermine the credibility of our results due to potential misjudgments.

Our current approach focuses on the final answer verification, which we believe offers a more straightforward and reliable measure of performance. However, we recognize the importance of evaluating the reasoning process and are actively exploring this as a research topic. It remains one of our objectives to develop and integrate robust methods for reasoning process verification in the future.

For weakness 2 and Question 1:

Reviewer bring up an important point regarding the evaluation of multi-step reasoning processes. Currently, our framework does not fully capture whether the model's reasoning aligns with the expected solution paths. We agree that evaluating the trajectory of the reasoning process is crucial for understanding a model's problem-solving abilities.

For now, we operate under the assumption that if the final answer is correct, the reasoning process is likely to be reasonable. Additionally, since we evaluate a sufficiently large number of questions, assessing mathematical reasoning ability based on the correctness of the final answer can effectively enhance the robustness of our evaluations.

Regarding whether MathEval can evaluate the reasoning process, from a benchmark perspective, it is challenging to employ a model-based method for this task without introducing a significant number of misjudgments. Even an excellent PRM model with 95% accuracy could produce many errors due to the extensive reasoning steps involved in complex problems. Therefore, to maintain the fairness and reliability of the benchmark, we have temporarily decided not to include reasoning process evaluation.

For weakness3

Thank you for highlighting this concern. We agree that refining the classification of mathematical reasoning types can improve the depth and granularity of our evaluations. In future work, we plan to conduct more detailed classifications, such as categorizing problems based on algebraic manipulation, geometric reasoning, combinatorial logic, and other specific reasoning skills.

Firstly, we would like to express our sincere gratitude to the reviewers for their insightful feedback.

Previous Work on Using Open Source LLMs in Grading Frameworks:

Thanks reviewer for highlighting the relevant work MathCAMPS (https://arxiv.org/pdf/2407.00900), which uses large language models for problem rephrasing and symbolic solvers for answer verification. While MathCAMPS focuses on generating diverse datasets, our approach stands out by addressing more natural and realistic problems across a range of difficulty levels from elementary to high school. We appreciate the reviewer's recognition of our efforts in fine-tuning a specific model for the answer comparison task.

Well-Known Findings Discussion:

We recognize that some of our discussions cover well-known findings within the domain. Due to space constraints, a more extensive discussion is included in Appendix E (Lines 824-952), where we conduct a detailed analysis that may offer more valuable insights for researchers than the content in the main text. We summarize the content as follows:

• Language Dimension: Models generally exhibit stronger mathematical performance in English than in Chinese, especially at the primary school level. This disparity is attributed to primary school math problems requiring more language comprehension, and the models trained predominantly on English datasets lack sufficient exposure to Chinese mathematical problems. Models developed by Chinese companies (e.g., WenXin 4.0, Spark-3.5) perform better in Chinese due to their training data, while those from English-speaking countries excel in English.

• Impact of Specialized Fine-Tuning: Fine-tuning models with specialized mathematical data (e.g., MAmmoTH-70B, MetaMath-70B) significantly enhances their problem-solving abilities, highlighting the importance of domain-specific fine-tuning in boosting performance beyond specific datasets.

• Grade Level Dimension: Models generally perform better on primary school math problems than on high school ones due to difficulty differences. Models like Claude-3.5-Sonnet and Gemini-1.5-Pro excel at the primary level, suggesting strong language comprehension that aids in solving word problems. In contrast, models like Llemma-7B and Llemma-34B show less pronounced advantages, possibly because their training focuses on complex concepts relevant to higher grades.

• Consistency Across Dimensions: Models often exhibit consistent performance within the same dimension; strong performance in one language or grade level typically correlates with similar performance in related areas. Evaluating models across different dimensions is crucial for identifying specific strengths, weaknesses, and potential data contamination. Significant discrepancies—such as exceptional performance at one grade level but poor performance in other tasks—may indicate contamination with data from that particular grade level.

We believe these detailed analyses provide a deeper understanding and help the community make informed decisions on model selection based on specific tasks.

Moreover, on page 29 of the appendix, in Figure 13, we present potential test set contamination for the evaluated model.

In the Upper Chart, we have three types of bars:

• Chinese Subsets Rank (Blue Bars): This indicates how each model ranks specifically within Chinese mathematical datasets. A smaller rank indicates better performance.

• Gaokao-2023 Rank Increase (Orange Bars): Represents increases in the rank of models when evaluated using Gaokao-2023 tests. A larger increase in rank signifies poorer performance on Gaokao-2023.

• Gaokao-2023 Rank Decrease (Green Bars): Represents decreases in the rank of models with the Gaokao-2023 tests. A larger decrease in rank signifies better performance on Gaokao-2023.

In the Lower Chart, we have the same three types of bars as in the upper chart; however, the blue bars represent the overall average score across 22 datasets.

We believe that Gaokao-2023 is not contained in the training dataset for all models. Thus, if a model performs very well on the blue bars but poorly on Gaokao-2023, this may indicate potential test data contamination.

From this figure, we can detect potential data contamination in the results. In the upper chart, the top two models showing an increase in rank are chatglm3-6b and Baichuan2-13b. Most of the Qwen-series models also have orange bars, indicating potential data contamination for these models. This conclusion is supported by findings in the paper "Compression Represents Intelligence Linearly". Furthermore, most base models exhibit green bars, indicating improved rankings on the Gaokao-2023 dataset. This suggests that chat models are more likely to have encountered similar math word problems during the instruction fine-tuning stage, increasing the probability of data contamination.

About Question1:

We appreciate the suggestion to use a regex-based extraction as a first step to minimize compute costs. While this approach could potentially reduce the number of instances where a more complex model like GPT-4 is needed, our experiments indicate that relying solely on regex extraction can lead to potential issues, especially for certain types of tasks.

As shown in Figure 14 of our paper, the regex-based method performs poorly on datasets like MathQA and MATH401, which involve mathematical word problems and arithmetic problems, respectively. In these cases, the complexity of the language and the variety of valid answer formats often lead to incorrect extractions or false positives—where an incorrect answer might be mistakenly marked as correct due to misextraction. This undermines the fairness and accuracy of the benchmark evaluation.

To address both the computational cost and the need for reliable verification across diverse tasks, we have developed a compare-answer model with only 7 billion parameters. This model strikes a balance by significantly reducing the computational overhead compared to GPT-4 while providing more consistent and accurate answer verification than regex-based methods.

About Question2:

Your understanding is correct. The dataset-level higher setting is defined as selecting the higher accuracy between few-shot and zero-shot prompting. Our intention in highlighting this result was to emphasize the fairness and robustness of our evaluation. Evaluating a model using only few-shot or only zero-shot prompting can lead to an underestimation of the model's true capabilities on certain tasks. By considering the higher accuracy from either prompting strategy on a per-dataset basis, we provide a more balanced and fair assessment of the model's performance.

About Question3:

Great thanks to the reviewer for pointing out this typo, we will fix this in the revised version of the paper.

Regarding Weakness 2:

We have expanded our discussion on Pages 22-23 to address this concern more comprehensively. Specifically, we have included an analysis of potential data contamination and its implications for our study. Additionally, we have refined some of our previous conclusions to provide deeper insights and to strengthen the validity of our findings.

Regarding Question 2:

We have made minor revisions on Page 9, Lines 416-419, to clarify this point.

Regarding the typos in our conclusion:

Thank you for pointing out these errors. We have corrected them accordingly to enhance the readability and professionalism of our manuscript.

Failure Modes and Other Limitations

Failure Modes or Error Patterns Due to Dataset Construction

First, we would like to clarify which stage of dataset construction is being referred to. If it pertains to prompt adaptation, we experimented with various prompt templates. We found that no Chain-of-Thought (no-CoT) prompts tend to introduce more problems compared to CoT prompts, especially in calculation problems that require step-by-step computations. For mathematical word problems like GSM8K, MATH, or datasets like OlympiadBench, the more reasoning steps required, the higher the likelihood of errors. This suggests that prompt design significantly impacts model performance, and carefully crafted CoT prompts are essential to mitigate error rates in complex reasoning tasks. If the concern is about the training data for our compare-answer model, we did not observe any significant error patterns.

Why Chat Models are Stronger

We provided a brief explanation in line 362 of the paper. Chat models generally undergo post-training based on base models, during which a substantial amount of data and methods specifically aimed at enhancing reasoning abilities are incorporated. For instance, models like DeepSeek-Math use techniques such as GRPO to improve their mathematical reasoning capabilities. Additionally, chat models are better at following instructions, which is a significant advantage over base models.

Mathematical Reasoning Not Captured by the Benchmark

Currently, our benchmark may not fully capture some advanced mathematical reasoning, such as problems involving diagrams or proof-based questions. We are actively preparing to include these two types of problems in future evaluations. While they may not be presented in this paper, we have conducted preliminary experiments. Regarding diagram-based problems, we have obtained initial results that will be part of a major update for MathEval. These results highlight significant challenges in multimodal mathematical reasoning, as models often struggle with interpreting and reasoning about visual information effectively. Here are the current performance metrics for various models on different benchmarks:

Figure-Related Issues

Q: Difficulty in Reading Figures Due to Acronyms, Color Scheme, and Flow of Components

Thank you for your insightful suggestions regarding Figures 2, 3, and 4. We agree that these improvements will significantly enhance the clarity and accessibility of our figures. Specifically:

• For Figure 2, we added intuitive icons to represent different languages and grade levels. Specifically, we introduced language icons for various languages and used school building icons to distinguish between different grade segments. This visual enhancement improves the readability and intuitiveness of the chart.
• Regarding Figure 3, we optimized the color scheme. We changed the previously hard-to-distinguish light green text to a prominent orange, while avoiding red-green combinations to ensure that individuals with color blindness can also read it easily.
• For Figure 4, we added arrows pointing from parts a and b to part c, indicating that a portion of the training data for the compare-model was extracted and validated from the model's output as well as GPT-4’s answers. This improvement illustrates the logical relationships and information flow between the components, helping readers better understand the structure and workflow of the entire framework.

These changes have been done in our revision version

Dataset and Difficulty Settings

Q: Focus on K-12 Levels and Lack of Higher-Level Mathematics

Our current focus on K-12 education levels stems from their broader applicability to our user base and the availability of extensive datasets within this range. However, we recognize that incorporating higher-level mathematics, such as undergraduate topics (College Math) and competition math (e.g., PutnamBench), would offer deeper insights into the models' capabilities across varying difficulty levels. We are actively working towards including these more challenging problems in future iterations of MathEval. MathEval is actively maintained. Two months ago, we identified a gap in our benchmarks for competition-level problems, so we integrated OlympiadBench into our dataset collection. Since then, we have continued to expand our datasets and plan to include additional ones over time. Furthermore, we are preparing to encompass multimodal mathematical evaluations. To this end, we have already collected multimodal datasets, including MATHVISTA and MATHVERSE.

We have added explatation in Appendix, Line 684-Line 688

Q: Inclusion of Dataset Summary in the Main Text

We will strive to include a dataset summary in the main text in subsequent versions of the paper.

Q: Adding More Middle School Datasets

We acknowledge that currently, Arith3K is the only middle school level dataset in MathEval. To achieve a balanced and diverse collection, we have recently collected the Zhongkao-2023 and Zhongkao-2024 datasets to compensate for the lack of middle school data. These datasets are derived from the Beijing High School Entrance Examination, which corresponds to the middle school level in China. Incorporating these datasets will enhance the diversity of MathEval and allow us to better evaluate models on mathematical abilities pertinent to middle school education.

Q: Details on Automatic Dataset Updates

From my point of view, this refers to our annual updates of the GAOKAO series datasets. As described in Appendix B.2 (line 680), these datasets are sourced from China's National College Entrance Examination (Gaokao). We have specialized educators who input the exam questions into our system shortly after the exam papers are released each year. By updating the datasets annually, we ensure that MathEval includes the most recent exam questions, which helps us evaluate models on fresh data and avoid potential data contamination from models that may have been trained on older testset.

We have added a footnote to indicate that the dataset will updated in our Github Repo in Page 15.

Q: Generation of "Ours" Labelled Parts in Table 2

We have included the details of the generation of our datasets, such as SCQ-EN-5K, SCQ-CH-5K, GAOKAO, and Arith3K in Appendix B.1. To provide additional clarity, we have also shared the corresponding GitHub repository, where we offer more information about the dataset creation process.

Evaluation and Algorithmic Details

Q: Details of Calculation Scheduling and details of the Evaluation Pipeline.

We will include the details of the calculation schedule in the appendix. This will cover dynamic dataset partitioning and automatic GPU allocation. For the evaluation pipeline, we have prepared our anonymous github, which we believe will address these issues comprehensively.

Q: Training Details for the DeepSeek-7B Model

In the final version, we will provide more information about the training process for the DeepSeek-7B model. Due to anonymity concerns, we are unable to share our Hugging Face repository that contains all the details related to the training of the compare answer model. However, we utilized a straightforward Supervised Fine-Tuning (SFT) technique with standard language model loss for finetuning.

Model and Language Choices

Q: Using Claude rather than GPT-4 for the LLM-based evaluation and limitations of using GPT-4.

We appreciate the reviewer's concern regarding our reliance on the GPT-4 model for answer comparison. Firstly, we would like to clarify our choice of GPT-4 over Claude-3.5. Our company has a collaborative agreement with the GPT-4 supplier, which grants us more efficient and cost-effective access to GPT-4, enabling extensive parallel processing capabilities. Our empirical evaluations suggest that the performance differences between GPT-4 and Claude-3.5 in answer verification tasks are minimal.

We acknowledge that even the most advanced models, such as GPT-4 or Claude-3.5, do not yet match human performance comprehensively. Another limitation of using GPT-4 is its high cost. With over 70,000 questions across 52 models, if each question averages 1000 tokens, this amounts to approximately 36.4 billion tokens, resulting in significant expenses.

Q: Choice of Chinese as the Second Language to Consider.

We chose Chinese as the second language for several reasons. First, Chinese is extensively used worldwide, and there is a wealth of rich and representative math data available in Chinese. Second, as can be seen, a significant portion of the large language models we evaluate are trained by Chinese companies. In our view, analyzing Chinese does not offer a unique advantage over other languages. We selected Chinese based on the availability of data and the variety of models.