Math for AI: On the Generalization of Learning Mathematical Problem Solving

Thanks for your thoughtful comments! We address the questions below:

W1: While the paper conducts extensive research across several models and datasets, it fails to establish a universal framework for evaluating AI's general reasoning capabilities in the context of "math for AI".

A1: Reasoning is a broad and complex topic, and, as far as we know, there is no consistent or universally agreed-upon method in the literature to evaluate LLM reasoning. In this paper, we use 16 benchmarks that are commonly adopted for evaluating reasoning. We believe that establishing a universal framework for reasoning evaluation is a valuable topic for future work.

W2: the paper does not provide effective methodologies or meaningful insights for constructing datasets that could enhance AI's general reasoning capabilities.

A2: Our paper evaluates existing methods for their generalization to reasoning, while constructing better methods and datasets is beyond its scope and worth future study.

W3: Without implementing deeper similarity detection or filtering mechanisms for the actual data content. This oversight could lead to data leakage between MPS and other tasks. The potential overlap between mathematical and general reasoning datasets might result in misleading conclusions about reasoning skill transfer.

A3: To address concerns about task classification and data leakage, we computed the is_similarity [1] metric between MMLU-STEM and GSM8K/MATH questions, which help to quantify overlap between each subset of MMLU-stem and GSM8K/MATH. Notably, the false percentage of is_similarity for all MMLU-stem samples is 100%, indicating no data leakage between MMLU-stem and GSM8K/MATH.

Q1: In Table 1 of the main experimental results (3) Instruction tuning on MPS datasets, for the MPS results, why is the COT effect better than the POT, and why does the mixed method perform much better than the 2-stage method?

A1: We want to clarify our evaluation across all benchmarks adopts a CoT format without calling Python interpreters, chosen for CoT serves as a more general and flexible format across variety benchmarks. And keeping a consistent evaluation method allows us to fairly analyze the impact of different training methods. For PoT training, our primary goal was to investigate the effect of code-based training data on reasoning tasks. Thus, it is not surprising that CoT training outperforms PoT in our setup, as CoT aligns better with the evaluation format. Previous works that report higher performance for PoT on MPS tasks typically do so by using Python interpreters during inference. Regarding why the mixed training method performs better than the 2-stage method on MPS tasks, we hypothesize this is due to the second stage using the general-purpose UltraChat dataset, which likely causes the model to forget some of the MPS skills learned during the first stage, whereas mixed training avoids this forgetting issue.

Q2: Can you provide specific examples to demonstrate how mathematical training can improve or deteriorate different reasoning abilities, which can clarify the limitations of mathematical training and offer a qualitative analysis of the reasoning patterns developed through different training methods?

A2: In the revised Appendix D, we provide some examples from logical reasoning shown in Appendix D in revision), demonstrating the detailed outputs across different training methods. We could find that the Mistral-7B and DeepSeek-Coder are more prone to logical errors, such as inconsistencies or premature assertion in reasoning. Models trained on math-related data, such as DeepSeekMath and MAmmoTH2, exhibit fewer such errors, suggesting that structured mathematical datasets not only improve mathematical performance but also enhance logical reasoning generalization, enabling more coherent and reliable conclusions.

Q3: What data do you think would really improve LLM's ability to reason over a wide range of tasks? How can this generalized reasoning capability be better assessed automatically?

A3: We think incorporating large-scale datasets spanning diverse domains, such as science, humanities, social sciences, engineering, logical reasoning, and coding may help to improve LLM’s reason abilities. In short, we believe that data diversity is the key. but measuring and creating extremely diverse data remains an open challenge. As shown in Figure 4, the training data studied in our paper only admits limited diversity. Evaluate generalized reasoning capability automatically requires comprehensive assessments across diverse benchmarks. While no consensus yet on evaluation methods or benchmark selection, we included a wide range of benchmarks in this paper. But we recognize the need for future research to develop a unified reasoning suite for more standardized and comprehensive evaluation.

[1] Zhang G, et al. Map-neo: Highly capable and transparent bilingual large language model series[J]. arXiv preprint arXiv:2405.19327, 2024.

Thanks for your detailed comments and valuable suggestions. We address the questions below:

Q1: The categorization of non-math datasets and the interpretation of results might be problematic.

A1: This is a valid point and we appreciate your suggestion on the organization of the results! We acknowledge the reviewer’s concern about treating STEM reasoning as a distinct category. To address this concern, we have taken the following steps to clarify the distinctions between MMLU-stem and math reasoning:

• Excluding Math-Related Subjects in MMLU-stem: To better distinguish STEM reasoning from math reasoning, we excluded math-related datasets within MMLU-stem and recalculated the evaluation results. These updated results are presented in the revised Table 3, showing that our conclusions remain consistent even after this adjustment.
• Measuring Question Similarity: To further differentiate MMLU-stem from math reasoning, we computed the minimum edit distance between each question in GSM8K/MATH and MMLU-STEM. Following [1], we regard the two examples are similar when the minimum edit distance is smaller than 5% of the example length. Notably, the false percentage of is_similarity for all MMLU-stem samples is 100%, indicating a very low similarity between MMLU-stem and GSM8K questions. This further supports that MMLU-stem encompasses broader reasoning skills distinct from math reasoning.

And we acknowledge that performing a discrete categorization may be both challenging and unnecessary. Instead, we have added Figure 2 (and Lines 371–398) in the revised paper, where we report results on all benchmarks, ranked by the query similarity between the benchmark data and GSM8K/MATH. Using DocMath in Figure 2 as a threshold to separate math and non-math tasks, we observe that continual pretraining and instruction pretraining consistently outperform instruction tuning that uses Math-COT SFT data. we find that continual pretraining with DeepSeekMath generally provides greater improvements on non-math tasks compared to MAmmoth2, although both of them result in performance drops sometimes.

Q2: Furthermore, the paper's main claim that "continual pretraining generally improves non-math reasoning" is drawn from improvements in 3 out of 5 domains….. Even considering all domains, showing improvement in 3 out of 5 areas probably does not suggest "significant" generalization, especially given the large performance degradation in commonsense and agent reasoning tasks.

A2: Thank you for the suggestion! We agree that the conclusions need to be expressed more precisely and carefully. Additionally, as pointed out by the reviewer, the conclusions also depend on how we categorize the domains. Therefore, as discussed in the response to Q1, we believe it is clearer to directly present the results across all benchmarks without relying on subjective categorization. Accordingly, we have revised the conclusions to state: There are no approaches that demonstrate consistent generalization for non-mathematical tasks, while continual pretraining and instruction pretraining show better generalization compared to instruction tuning. Between continual pretraining and instruction pretraining, continual pretraining generally achieves higher gains than instruction pretraining when it is effective, but both approaches result in performance drops on some benchmarks. We have updated the expression of these conclusions in multiple sections of the paper to ensure clarity and precision.

Q3: While it presents performance numbers and draws broad conclusions, it does not provide more in-depth insights. For instance, If the paper claims major improvements on specific tasks, the paper could have some analysis which specific types of data points show improvement and provide potential explanations for these gains.

A3: Thank you for the suggestion! We have added two analyzes to help readers understand some of the phenomenon:

• We have analyzed the query distribution of each benchmark and training dataset, as shown in Figure 4 in the revision. Our findings indicate that WebInstruct (the instruction pretraining dataset used for MAmmoTH2) and OpenWebMath (an example continual pretraining datasets) exhibit more overlap with many benchmarks, compared with MetaMath (the instruction tuning dataset) that mainly overlaps with the math benchmarks. As such, we believe the data diversity contributes to the effectiveness on generalization.
• In the revised version of the paper, we have also provided a clearer organization of the results in Figure 2 (Section 3.3), where we demonstrate that, in general, when benchmark queries are more similar to GSM8K/MATH queries, the overall improvements tend to be higher. This observation helps to explain some of the varied patterns observed across different benchmarks.

[1] Zhang G, et al. Map-neo: Highly capable and transparent bilingual large language model series[J]. arXiv preprint arXiv:2405.19327, 2024.

We appreciate your thoughtful comments. Below, we address the key points raised across the reviews.

W1: This work lacks of novelty. It only conducts several experiments and comes up with a conclusion which is align with my assumption.

A1: We would be grateful if the reviewer could share references to other works that have drawn similar conclusions to substantiate the comment, which would help us refine our research.

W2: The result analysis is not deep enough. Are there any cluse from the mechanical interpretability perspective, why some the instruction pretraining works while some are not, why all the instruction tuning fails.

A2: This a good point. We think the main difference among continual pretraining, instruction pretraining, and instruction tuning we studied is on the corresponding datasets rather than the specific training methods – all three methods are using the same next-token prediction loss. To respond to the reviewer’s concern, we have analyzed the query distribution of each benchmark and training dataset, as shown in Figure 4 in the revised PDF. Our findings indicate that WebInstruct (the instruction pretraining dataset used for MAmmoTH2) and OpenWebMath (an example continual pretraining datasets) exhibit more overlap with many benchmarks, compared with MetaMath (the instruction tuning dataset) that mainly overlaps with the math benchmarks. As such, we think this additional analysis helps explain why all instruction tuning fails.

We appreciate your thoughtful comments. Below, we address the key points raised across the reviews.

W1: They only attempt pretraining and finetuning-based approaches and that too with only two models.

A1: We were indeed constrained by computational resources, particularly for continual pretraining and instruction pretraining models. But to address model variety, we conducted experiments with two math SFT datasets on Llama 3.1-8B. The results (shown in Section 3.3), confirm our findings: math-specific SFT improves performance on MPS benchmarks but fails to generalize to broader reasoning tasks.

W2: The authors should try to examine methods beyond just continual pretraining and fine-tuning. For example, they should try reinforcement learning approaches

A2: This is a valid point. While RL methods could provide more insights, the pipeline for mathematical reasoning is still under development in the open-source community, with numerous loss variants (e.g., PPO[1], DPO[2], GRPO[3]) and debates around training methods and reward models[4,5]. So we think generalizing RL training warrants a separate, focused study. And this area is beyond the scope of our work, where we primarily focus on the most commonly used methods for MPS.

W3: This paper is mostly focused on “correlations” as opposed to the “causations” as to why certain training approaches and datasets succeed or fail at improving a model’s generalizability

A3: The observation about correlations versus causations is well-taken. In response, we conducted an embedding analysis (Figure 4) of training datasets and benchmarks, showing that WebInstruct (for instruction pretraining) and OpenWebMath (for continual pretraining) have greater overlap with diverse benchmarks than MetaMath (for instruction tuning), highlighting the role of topical diversity in generalization. Additionally, Figure 2 (Section 3.3) shows that benchmarks with queries more similar to GSM8K/MATH tend to see greater improvements, helping to explain varied patterns across benchmarks. We hope these analyses provide some “causations” on why certain training datasets succeed or fail across benchmarks.

Q1: Are the findings from your paper model agnostic? If not, what kinds of models benefit from this sort of training and with sorts of datasets?

A1: Fine-grained generalization depends on model architecture and weights, however, we think the high-level finding. Our findings are not entirely model agnostic. Experiments show that models like Mistral-7B and Llama 3.1-8B respond differently to math SFT training. And compare to the general-purpose model like Mistral-7B, specialized models like DeepSeek-Coder benefit boarder from math-specific datasets in STEM reasoning. This suggests that generalization also depends on the alignment between the model's architecture and pretraining corpus.

Q2:What kinds of other training-based approaches could be done to improve the score?

A2: Incorporating RL or hybrid methods may further enhance general reasoning abilities, and we see great potential for exploring these in future work.

Q3: If there is any correlation to the specific skills that an LLM has after continual pretraining which is why the model shows an improvement in general reasoning tasks? Can you explain further as to why certain training methods with their corresponding datasets succeed or fail at improving an LLM’s general reasoning ability?

A3: We think the main difference between three training paradigms lies in the datasets used rather than the training methods, as all three employ the same next-token prediction loss. To avoid repetition, we refer to our response to W3 where we highlighted the role of data diversity in enabling generalization and the correlation between query similarity to GSM8K/MATH and benchmark performance improvements.

[1] Schulman J, et al. Proximal policy optimization algorithms[J]. arXiv preprint arXiv:1707.06347, 2017.

[2] Rafailov R, et al. Direct preference optimization: Your language model is secretly a reward model[J]. Advances in Neural Information Processing Systems, 2024, 36.

[3] Shao Z, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.

[4] Tang, Yunhao, et al. "Understanding the performance gap between online and offline alignment algorithms." arXiv preprint arXiv:2405.08448, 2024.

[5] Pan, Sarah, et al. "Let's Reinforce Step by Step." arXiv preprint arXiv:2311.05821, 2023.