From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models

We thank the reviewer for their attention to detail with all the feedback! We have addressed the comments below:

Compared to pretraining, the finetuning analysis feels a bit rushed, with only the original and final checkpoints before and after finetuning. As finetuning is much lighter than pretraining, the authors may be able to conduct custom finetuning to even strengthen the paper's analysis.

Thank you for suggesting improvements to the fine-tuning analysis! We agree that controlled experiments isolating the effects of the instruction-tuning data corpus would be useful in isolating how various factors within the data impact the final performance of the model. While we were not able to conduct such experiments due to the high computational cost of fine-tuning LLMs, our current works take a first step by analyzing the impact of post-training on three publicly available models (Amber, Llama, and OLMo), each of which underwent a distinct post-training schema. This cross-model comparison highlights general trends and introduces hypotheses we hope future works can explore in more controlled manners.

Our results are also in alignment with an observation made by the very recent work of Springer et. al. [1], which considers the quantity of data seen during the pre-training of the LLM and its instruction-tuned counterpart’s performance. They show that the more tokens an LLM sees during pre-training, the less effective instruction-tuning is on it. In our case, OLMo and Llama, which each saw 15T and 4T tokens of data during pre-training respectively, saw more degradation post fine tuning compared to Amber, which saw ~1.3T tokens during pre-training. We will add a brief discussion of Springer et. al.’s paper as a possible explanation in our final manuscript.

[1] J. M. Springer et. al. Overtrained Language Models Are Harder to Fine-Tune (https://arxiv.org/abs/2503.19206)

The authors should do an ablation study on the quality of the data generation compared to directly prompting GPT-4o to generate problems on a given theme according to the common core description. It is unclear if we actually need all the complexity in the current data generation pipeline.

The main advantage of the complexity of the pipeline is that we can reliably generate high-quality problems where we trust the answer even if the generating model can't solve the problem. For example, there are many standards (e.g. 4.MD.A.2-fraction and 4.OA.A.3) where GPT-4o scores a 0.57 and 0.80, respectively. Among the 49 skills we examined, 4o scored below a 0.90 on10 of them. While this is still relatively impressive performance, not using the symbolic grounding we do would leave us with no reliable way of calculating the solution, which we currently do programmatically using the symbolic structure.

The figures should be made in vector graphics. Right now, many plots are pixelated when zoomed in. Also, I don't recommend the use of bold font in plot as they are too distractive on the normal flow of reading.

Thank you for paying such close attention to ensure the quality of our paper! We will edit all images in the final manuscript to be vector graphics, and removing bold text in the images.

We thank the reviewer for their comments on our paper!

W1. For any benchmark dataset human evaluation is necessary to check the quality of the dataset. Even though there is automatic checking of the dataset but it would have been helpful if there is also a human evaluation.

We agree that a human evaluation is necessary, and we have indeed conducted a manual evaluation of 245 problems from MathCAMPS, as discussed in Appendix F. We see that after cycle-consistency, 97.7% of the problems in our dataset are reliable (i.e. are unambiguous and are associated with a correct answer). Thus, our manual evaluation found the MathCAMPS generation pipeline to be highly reliable.

W2. The dataset contains numbers whose addition will lead 20. This could have been extended for the curation of a richer dataset.

The dataset covers 44 different skills across grades K-8, as described in our Abstract, Related Works, and Methods (Section 3.1, specifically). We will also edit the contributions to add this. Figure 1 shows the CC standard you mention simply as just a single example, but the dataset is much more comprehensive than that particular skill – the list of all Common Core standards we have implemented is given in Tables 1 through 9 in the Appendix (starting in page 18 of the submission).

W3. I would have really liked to see the performance of LLMs according to digits. Do they perform better where addition results in a single digit number like (1+5) compared to two digits (8+9).

We appreciate the reviewer’s suggestion, and have done an experiment to examine this. To explore the relationship between performance and the size of the numbers involved in the problem, we plotted the accuracy for each training checkpoint for the different numbers of digits seen in the answers.

Plot for accuracy vs. number of digits: https://ibb.co/TBdVtFhr

We will be conducting a deeper analysis in the final manuscript, but interestingly, we note how model behavior doesn’t follow exactly what is expected. For example, even though problems with 1-digit answers are easier on average than those with two digit answers, Amber seems to consistently over perform in cases where there are 2-digit answers. This holds true for the final Pythia model as well, but not for most other training checkpoints we looked at. We thank the reviewer for their suggestion again, and are looking forward to probing this deeper to include in the final manuscript.

W4. How is the effect of few-shot prompting in these cases? Would that have been helpful?

As discussed at the beginning of Section 3.2 (Lines 177-178), we employed few-shot prompting to demonstrate model behavior. We found this to be important, especially since we ran experiments on pre-training checkpoints that aren’t instruction-tuned. These were the two shots we used:

    {"role": "system", "content": "You are a mathematics teacher that solves all problems correctly and explains your reasoning. Write your final answer in the last line of your response."},

    {"role": "user", "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?"},

    {"role": "assistant", "content": "Reasoning: Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\nAnswer: 72"},

    {"role": "user", "content": "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?"},

    {"role": "assistant", "content": "Reasoning: Weng earns 12/60 = <<12/60=0.2>>0.2 per minute.\nWorking 50 minutes, she earned 0.2 x 50 = <<0.2*50=10>>10.\nAnswer: 10"}

W5. A short survey on how humans actually solve this problem would also have made the paper stronger specifically for the follow-up questions.

We agree that human experiments would be certainly interesting, especially for potential educational applications of our work. However, we believe this is a separate endeavor that is worth doing right — we would need to recruit a controlled population of humans of appropriate grade levels in order to draw informative insights (since, for instance, grade 5 students will certainly differ significantly from MTurk/Prolific workers which would be easier to recruit but perhaps less interesting for this purpose). While these evaluations would go beyond the scope of the current work, we will expand on our discussion into what these would entail for future work in our conclusion.

Q6. There are some grammatical and spelling errors (L:248, etc.).

We really appreciate you pointing this out! We will make sure to edit this error and others we've caught in the final manuscript.

You are a mathematics teacher that solves all problems correctly and explains your reasoning. Write your final answer in the last line of your response.

Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?

Reasoning: Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72

Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?

Reasoning: Weng earns 12/60 = <<12/60=0.2>>0.2 per minute.\nWorking 50 minutes, she earned 0.2 x 50 = <<0.2*50=10>>10. Answer: 10

Q7. The data is generated by GPT4-o. Chances are most of the generation follows a similar pattern. Is it covering enough stylistic variations in the problems generated?

Since the symbolic structures are generated from 44 different skills, and since we explicitly sample random themes to condition the word problem generation, we see a large variety of the types of problems in our dataset. Here are some examples of skills, associated symbolic structures that are programmatically generated, and the natural language problems they are translated into. Skill: 2.NBT.B.7 (adding and subtracting within 1000) Symbolic structure:

[[var m = 278]]
[[var f = (m + 305)]]
[[question l = ['f']]]

Problem: In one month, a fireplace company sold 278 traditional wood-burning fireplaces. In the same month, they also sold 305 more gas fireplaces than traditional ones. How many gas fireplaces did the company sell?

---

Skill: 3.MD.C.8-polygon (perimeters of polygons)

Symbolic structure:

[[var 274 = (((((54 + 51) + v) + 5) + 22) + 93)]]
[[question c = ['v']]]

Problem: A polygon has sides measuring 54cm, 51cm, 5cm, 22cm, and 93cm. If the total perimeter of the polygon is 274cm, what is the length of the sixth side?

---

Skill: 8.EE.C.8 (systems of equations with 2 variables) Symbolic structure:

[[var 37 = ((40 * s) - (91 * q))]]
[[var 121 = ((30 * s) + (12 * q))]]
[[question y = ['q', 's']]]

Problem: Let's find the values of variables q and s by solving the following system of equations:\n\n1. (40 * s) - (91 * q) = 37\n2. (30 * s) + (12 * q) = 121.

We thank the reviewer for their detailed comments and suggestions! We have responded to the questions/added some clarifications below.

Incremental with respect to existing synthetic math resources. Recent work (e.g., GSMore [1], GSM-Symbolic [2], DyVal [3]) already synthesises or perturbs math problems for robustness studies; MathCAMPS differs mainly in being tied to Common Core standards, which feels like a change of packaging rather than a conceptual advance.

We agree that our main difference, in terms of the dataset, is exactly its close tie to a human curriculum. However, that alone enables a whole suite of analyses (which we've explored in the paper) that would be impossible to conduct on any of the existing datasets. Moreover, the follow-up question generation we introduced (Section 3.3, used to answer RQ3) is not present in any of the existing mathematical reasoning datasets we know of, synthetic or otherwise, including GSMore, GSM-Symbolic and DyVal, GSM8k, MATH and many others.

Heavy reliance on GPT-4 for both generation and back-translation. Although cycle consistency helps, GPT-4 can still hallucinate or encode hidden shortcuts, and no contamination check against GPT-4’s training data is provided.

We agree that checking how much the dataset is influenced by the choice of GPT-4 as the generator model is important. Thus, to address this, we conducted an analysis where we generated a dataset using the exact same pipeline, but with Claude instead. We found that the problem difficulty seemed mostly unaffected by the model used for generation. For instance, GPT-4o achieves an accuracy of 0.910 on the GPT-4-generated dataset, and an accuracy of 0.954 on the Claude generated dataset. Meanwhile, Claude scores 0.887 on the GPT-4-generated dataset, and 0.909 on the Claude-generated dataset. Thus, in both cases, the consistent result is that GPT-4o outperformed Claude indicating that the generation model is not necessarily generating problems that are easier for itself (e.g., by copying from its training data, or introducing shortcuts). We will be adding Appendix G to the final paper to include the full analysis of this result, which will be present in the final version of the paper.

The paper infers that OLMo’s accuracy drop after tuning is due to high-quality math data in its second pre-training stage. Can the authors provide controlled experiments that vary only the instruction-tuning corpus or objective to isolate which factor causes degradation?

Thank you for the thoughtful suggestion! We agree that controlled experiments isolating the effects of the instruction-tuning data corpus would be useful in isolating how various factors within the data impact the final performance of the model. While we were not able to conduct such experiments due to the high computational cost of fine-tuning LLMs, our current works take a first step by analyzing the impact of post-training on three publicly available models (Amber, Llama, and OLMo), each of which underwent a distinct post-training schema. This cross-model comparison highlights general trends and introduces hypotheses we hope future works can explore in more controlled manners.

Our results are also in alignment with an observation made by the very recent work of Springer et. al. [1], which considers the quantity of data seen during the pre-training of the LLM and its instruction-tuned counterpart’s performance. They show that the more tokens an LLM sees during pre-training, the less effective instruction-tuning is on it. In our case, OLMo and Llama, which each saw 15T and 4T tokens of data during pre-training respectively, saw more degradation post fine tuning compared to Amber, which saw ~1.3T tokens during pre-training. We will add a brief discussion of Springer et. al.’s paper as a possible explanation in our final manuscript.

[1] J. M. Springer et. al. Overtrained Language Models Are Harder to Fine-Tune (https://arxiv.org/abs/2503.19206)