Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages

Q1: Could the authors provide more specific examples of problem-solving data used during pre-training and demonstrate how it differs from general mathematical corpora?

A1: General mathematical corpora include a large volume of text on concepts, axioms, theorems, and other descriptive content, typically derived from filtered internet corpora. Examples include open-web-math [1] and InfiMM-WebMath [2]. In contrast, problem-solving data refers to datasets composed of mathematical problems and their reasoning processes. These datasets often involve the application of concepts, axioms, and theorems. An example is provided below:

---

Question: In the numbers -1, +7, 0, -\frac{2}{3}, \frac{5}{16}, how many are positive numbers?
A. 1
B. 2
C. 3
D. 4

Solution: Among the numbers -1, +7, 0, -\frac{2}{3}, \frac{5}{16}, there are +7 and \frac{5}{16}, which are positive numbers, totaling 2. Therefore, the correct answer is: B.

---

[1] https://huggingface.co/datasets/open-web-math/open-web-math

[2] https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B

Q2: Could the authors provide more insights into the limitations of the retrospective enhancement technique? What additional methods or adjustments could make this approach more effective?

A2: Firstly, we believe that the intuitive hypothesis behind the retrospective enhancement method—"allowing models to retry upon regret"—is quite reasonable. This is also why we conducted experiments and modified the process to develop the tutorship amplification method. We think its limitations may stem from the data construction process, where subsequent steps are directly inserted into previous steps to simulate "allowing models to retry upon regret." Although this indeed saves resources, the constructed data does not reflect actual errors, leading the model to learn how to correct non-existent mistakes. To address this, we attempted to construct some real errors, which led to the subsequently proposed tutorship amplification method.

Q3: Have the authors considered applying the problem-solving data approach to other reasoning-heavy domains, such as physics or logic-based tasks? How would the methods transfer to these fields?

A3: Similar to Reviewer 3zrA's Q1, we have conducted a detailed discussion; please refer to that.

Q4: How do the proposed data synthesis methods, such as tutorship amplification, perform when applied to larger LLMs like 72B? Is the impact equally significant?

A4: We have not conducted CPT on larger models. Due to the computational cost of training a 72B model, we do not plan to carry out comparative experiments. Before training the 72B model, we plan to prioritize improving the quality of the synthetic data methods and explore more synthesis methods.

Q5: Given the success of the tutorship amplification method, do the authors plan to further refine this approach? Could it be applied in real-time tutoring systems or adaptive learning models for broader applications?

A5: We plan to continue improving the tutorship amplification method but do not have plans to apply it in real-time tutoring systems. We believe the potential areas for improvement in tutorship amplification include:

1. Enhancing the quality of synthetic data: This includes filtering out instances where the student model's answer is incorrectly judged as incorrect and optimizing the teacher model's response quality by introducing methods like process supervision.

2. Optimizing the formatting of synthetic data: This involves adjusting prompts and post-processing steps to clearly define the erroneous parts, analysis parts, and problem-solving parts in the synthetic data. This would facilitate ablation experiments and simplify subsequent applications of format shaping, process supervision, and other reasoning techniques.

Thank you for your response and clear suggestions for further updates. The first point is a very direct and actionable suggestion, and we plan to directly follow it by adding illustrations for improvement; the second, third, and fourth points are extensions to different aspects of the paper, and the design and implementation of related experiments will take time. We currently offer some simple discussions and plans in response.

First, we fully agree with the first point and consider adding illustrations in the first section to clarify the improvements brought by the methods. Due to the current revision limitations, here is a brief description of the process diagram: Content: Use problem-solving data directly to improve math reasoning skills instead of filtering the math corpus for mathematical knowledge expansion. Highlight: 1) Smaller data volume and higher training efficiency; 2) In further exploration, combining synthetic data methods, an iterative data-model update process is formed that combines model capability and data quality.

For the second point, we believe that validating and providing larger models would be beneficial, but we will not repeat experiments on several research questions with larger models. Our current plans for data and models are to: 1) Implement quality identification and enhancement on synthetic data; 2) Conduct CPT with llama3 70b; 3) Implement post-training to apply inference stage techniques, allowing the problem-solving data construction and model training to form an iterative optimization. These attempts will take time to implement and improve based on actual improvements.

For the third point, we have not validated the method's transfer due to the following two reasons: 1) We are not too concerned about the transfer of methods. Under the conclusions of the experiments in section 5.2, regarding the differences between physical reasoning and mathematical reasoning, I believe they are similar to the differences between high school and middle school math. It is relatively easy to implement experiments validating the model's transfer capabilities, and the results presented earlier have deepened our views. 2) In terms of expanding application value, expanding tasks including physics or logic is indeed what we hope for, but physical data is relatively scarce, and we have not yet begun to process the related raw data further, which will be a long-term process.

For the fourth point, continuing the discussion from previous replies, we believe the intuitive assumption of retrospective enhancement is reasonable, but the data construction process does not reflect actual errors. A more direct correction of this method is to insert a potential error process in the current step instead of using subsequent steps to achieve the purpose of reflecting actual errors. To achieve this, the direct association is to use a method combining inference process and search, such as applying MCTS to build preferred data pairs, obtaining a correct inference path, obtaining another node with a large score gap based on an intermediate node, considered as a possible error process, and inserting it before the current node of the correct reasoning path, simulating "allowing models to retry upon regret". However, although this is a more direct refinement, it loses the original advantage of saving resources, so we did not optimize and implement this improvement idea at the design stage. Currently, we consider expanding based on small-scale trials to decide whether to expand.

Thank you again for your response and suggestions for further updates and broader validations. The suggestions will be oriented towards practical implementation rather than further validation of the methods, proceeding according to the plan mentioned in the second point. For a deeper exploration of methods, we lean towards continuing exploration based on the ideas in 5.3, believing this to be important work, hoping to stimulate community interest in this exploration.

Thank you for your review!

Q1: For figure 1, have you tried ablating the percentage of the math mixture to extremes like 2:8, 1:9 or even complete problem solving data. For the tasks considered in the paper, it could be the case that simply doing CPT with the problem solving data could be sufficient.

A1: In the experimental design for Figure 1, we controlled the total amount of math data used in both the base group and the test groups to remain consistent, ensuring the full utilization of the data. Therefore, the math data mixture ratio was set at 3:7, with further discussion provided in the appendix under "Discussion on Experimental Setting."

Adjusting the ratio to extremes would alter the actual amount of newly introduced data. However, the hypothesis you mentioned—that simply conducting CPT with problem-solving data could be sufficient—seems reasonable. To investigate this, we added an experimental group, Test4, which used only problem-solving data. The detailed results of this experiment are included in the newly added appendix, "Detailed Results of Problem-Solving Data Effectiveness Experiment."

Overall, two conclusions emerged:

1. Test4 outperformed the experimental group with a math data mixture ratio of 3:7, highlighting the effectiveness of problem-solving data, which can even surpass the impact of adding a large number of new tokens.
2. When the math corpus is not used, the specific metrics of Test4 across the four evaluation datasets no longer align with those of Test1-3. This shift in data distribution undermines the improvement in GSM8K performance while enhancing the improvements in Math and Gaokao capabilities. From the perspective of ability dimensions, this may be due to the absence of the math corpus affecting the learning of general capabilities.

Although this experimental group strengthens the conclusions, it seems to introduce additional variables, such as changes in data distribution. To maintain clarity in addressing this research question, we have included this experimental group in the appendix.

Q2: For response diversification, it could happen that the model could generate incorrect logic leading to the correct answer. Do you have simple checks in place to ensure that this does not happen?

A2: The process for response diversification, as well as the other three synthesis methods, follows the procedures described in the paper and is primarily based on prompt engineering. We did not introduce reward models or other process supervision mechanisms. Since response diversification provided with process information of the seed data, occurrences of incorrect logic, while unavoidable, are significantly less severe compared to direct sampling followed by answer-based filtering.

The seed data, i.e., the original problem-solving data, incorporates process supervision techniques, including sampling with various search methods and quality checks using PRM. However, we did not employ these methods during the data synthesis process due to their high computational cost. We plan to utilize such techniques after finalizing and optimizing the synthesis methods.

Q3: For results in table 2, how will the results change if you apply inference time techniques like Best-of-N sampling or using a reward models during inference? It could be that the conclusion changes significantly. One recent work (https://arxiv.org/pdf/2410.02725) showed that simply restarting the inference can change the accuracy on math tasks significantly.

A3: The mentioned inference-time techniques are indeed significant in the current field of mathematical reasoning. Best-of-N sampling is commonly combined with methods such as self-consistency, self-evaluative beam search, reasoning-via-planning, as well as techniques like Capability-Aware Self-Evaluations and Mid-Generation Self-Evaluations mentioned in the referenced paper, which directly improve Best-of-N sampling. However, regardless of the specific method employed, a common phenomenon typically holds: while the model's performance improves as the number of samples increases, if Model A is superior to Model B when sampling only one instance, Model A generally remains superior to Model B when sampling ten instances.

Thus, we have two main concerns regarding this question. First, applying inference-time techniques in comparative experiments is rare. We could simply apply Best-of-N sampling and use metrics like pass@3 instead of pass@1 for evaluation, but it is difficult to anticipate that this would lead to a significant change in conclusions. Second, whether there is a specific connection between these techniques and the research question addressed in Table 2—namely, the impact of data distribution on the learning capabilities of CPT and SFT—or if this is merely an exploration of one experimental setting deemed important.

Could you further clarify the main focus of this question? We could then conduct an additional experiments to address it.

Q：What is the major advance of this paper? What surprised you? What did you learn that might generalize out of this narrow problem domain?

A: The results outlined in the Introduction Section hold both research and engineering value for the field of mathematical reasoning.

The major advances include significant conclusions addressing three research questions and the development of the JiuZhang-8B model:

• Result 1 (RQ1): Validates the efficiency of problem-solving data and motivates updates to math-specific model training paradigms. Specifically, it advocates for moving the more efficient problem-solving data to the CPT phase, replacing large volumes of math corpus.

• Result 2 (RQ2): The depletion of internet data has made the exploration of synthetic data methods critically important. We validated the Tutorship Amplification method, which stands apart from commonly inspired synthetic data approaches such as Response Diversification and Query Expansion. This method leverages a teacher model to identify and correct errors based on the student model's responses, aiming to equip the model with self-correction capabilities. It has demonstrated significant performance improvements.

• Results 3–5 (RQ3): Explore the distinction between CPT and SFT stages and identify the factors contributing to these differences. These findings enhance the understanding of how mathematical reasoning abilities are developed and emphasize the importance of preparing more challenging problem-solving data for the CPT phase.

• JiuZhang-8B serves three primary purposes: to validate the research conclusions presented in the paper, to act as a base model that introduces a paradigm shift in training math-specific models with features worth further exploration, and to demonstrate competitive performance as a math-specific base model within 8B parameters.

This perspective could be tested in other domain-specific models. Instead of merely providing domain-specific knowledge through corpora, introducing instruction tuning type data directly during CPT—which reflects the corresponding capabilities—might be the key to improving effectiveness. The process of defining necessary data and subsequently collecting or synthesizing it may surpass the approach of massively gathering internet corpora followed by quality and relevance filtering.

What surprised us was the results regarding the difference in learning capabilities between CPT and SFT. This difference does not stem from CPT conferring out-of-distribution (OOD) capabilities, but rather from the impact of hard data within the training datasets. The underlying mechanisms behind this phenomenon merit further investigation.

---

Finally, we would like to emphasize that conducting empirical research on the research questions based on current standard methods is not "in the weeds." On the contrary, we believe this approach provides empirical results that are significantly more meaningful compared to "conceptual" experiments, which start from a blank model and use only data related to the research questions. This is why we adopt various standard methods and begin exploring the research questions in a more complex and resource-intensive manner.

We hope the results of this paper will influence updates to training paradigms and guide data collection and synthesis directions. The empirical findings and intermediate conclusions regarding differences introduced during the training phase aim to enhance the understanding of mathematical reasoning learning and inspire further exploration. The release of JiuZhang-8B serves as a revalidation of the research conclusions and, as a competitive math-specific base model, should be beneficial to the community.

Additionally, we note that Llemma [1], which was accepted at ICLR 2024, primarily introduces the training results of a model along with some analyses. However, its empirical insights and the open-sourced model have effectively promoted subsequent explorations into LLM mathematical reasoning and have been widely cited.

[1] Llemma: An Open Language Model For Mathematics

Thank you for your review! Below are our responses to each comment:

Q1: While this paper summarizes several empirical results that are useful to the community, it makes limited efforts to advance understanding of the mechanics behind these observations. Future work in this direction would be of great value.

A1: We agree with this point: advancing understanding of the mechanics behind these observations is an important direction. Specifically, any suggestions and speculations regarding the final result in section 5.3 on why hard training data leads to differences in learning ability between CPT and SFT are helpful. Intuitive conjectures behind other conclusions are discussed to some extent in the responses to the other reviewers, as follows:

• Regarding Result 4's mention of "both SFT and CPT primarily develop capabilities aligned with their data distributions", we speculate that the generalization ability of LLMs is limited. It might be confined to simple patterns such as numerical substitution and conditional result substitution. This limitation may depend on whether the training data includes similar data pairs for learning (akin to the issue of knowledge storage and extraction discussed in "Physics of Language Models: Part 3.1, Knowledge Storage and Extraction"). The generalization of reasoning capabilities warrants further experimental design and exploration.

• Regarding Result 4's mention of "Regarding out-of-domain (OOD) capability learning, SFT is more susceptible to disturbances from data distribution compared to CPT," we speculate that the differences in OOD capability learning represented by different evaluation sets are influenced by the similarity of the evaluation set distribution to the training data; whereas the stability of CPT over SFT is due to the combined effects of generic data mixing and recall.

• For the main part of Result 4, it is noted that SFT's in-domain (IND) learning ability is weaker than that of CPT. This observation prompts the exploration and conclusions presented in Result 5.

• Regarding Result 5, why hard training data results in learning capability differences between CPT and SFT is not well-understood, and may be suited for theoretical analysis, requiring meticulous work. Similar work from a theoretical analysis perspective seems to have just begun (Understanding Chair of That in LLMs Through Information Theory).

Q2: Can the proposed synthetic data methods, especially "tutorship amplification", be effectively scaled or adapted to non-math domains?

A2: "Tutorship amplification" and "Retrospective Enhancement" are based on a similar viewpoint that models can detect errors but lack opportunities for correction. However, unlike "Retrospective Enhancement," which generates artificial errors leading to sub-optimal results, "tutorship amplification" simulates a realistic error correction process. Therefore, we believe that designing synthetic data methods that are adapted to the corresponding domain based on this viewpoint is crucial.

Q3: Given the focus on hard data and high-volume problem-solving datasets, are there any recommendations for making this approach feasible for smaller labs with limited resources?

A3: The main recommendation of the article is to shift sufficient problem-solving data to the CPT phase, replacing the need for multiple times the volume of generic mathematical corpora, which may not inherently be a research direction for smaller labs with limited resources. However, we would like to point out that for hard data, enhancing the quality and data synthesis efficiency through post-training and optimizing in conjunction with search algorithms during the inference process is crucial. This can be integrated with the capabilities developed during the CPT phase to form a multi-round iterative process, which could be an important direction for further enhancing mathematical reasoning abilities.

Q4: How well would the identified CPT benefits translate to domains requiring other forms of reasoning (e.g., logical or scientific)?

A4: We illustrate our point using the results of the mmlu-stem subtasks, as shown in the table below. We observed improvements in high school statistics, college chemistry, and conceptual physics in subtasks outside of mathematics.

This is also consistent with our expectations. The results in Section 5.2 indicate that SFT and CPT primarily enhance capabilities aligned with their respective data distributions, and the performance in out-of-distribution (OOD) scenarios is actually influenced by the similarity of those tasks to the training data. Naturally, it can be inferred that for these non-mathematical tasks, the improvements may also be attributed to their relevance to mathematical tasks.

Overall, generalization is indeed important and demonstrates a certain degree of broader applicability. However, I believe that identifying datasets that define related capabilities and using the approach proposed in the paper to transform them into in-distribution (IND) tasks could be a better option.

Thank you for your review! Below are our responses to each comment:

Q1: The paper shows promising results for mathematical reasoning but could benefit from testing on datasets that combine math with other fields, like physics. This would help assess if the model’s reasoning improvements generalize to interdisciplinary tasks requiring both math and domain knowledge, strengthening the case for broader applicability of the techniques.

A1: Thank you for your suggestion! We used the evaluation results of the mmlu-stem dataset from lm-evaluation-harness to demonstrate this point, and the results are presented in the table below. It is worth noting that the evaluation process determines outcomes based on the probabilities of the options, which might not fully represent a standard reasoning task. However, the results align with expectations, showing improvements in tasks somewhat related to mathematics. Specifically, among the subfields beyond mathematics, those more closely related—such as High School Statistics, College Chemistry, and Conceptual Physics—showed significant improvements.

This is also consistent with our expectations. The results in Section 5.2 indicate that SFT and CPT primarily enhance capabilities aligned with their respective data distributions, and the performance in out-of-distribution (OOD) scenarios is actually influenced by the similarity of those tasks to the training data. Naturally, it can be inferred that for these non-mathematical tasks, the improvements may also be attributed to their relevance to mathematical tasks.

Overall, generalization is indeed important and demonstrates a certain degree of broader applicability. However, I believe that identifying datasets that define related capabilities and using the approach proposed in the paper to transform them into in-distribution (IND) tasks could be a better option.

Q13: The paper repeatedly discusses the distinction between CPT and SFT stages, yet the insights presented do not meaningfully advance the reader's understanding. I recommend reducing redundancy and focusing on meaningful results.

A13: Thank you for the suggestion. We have streamlined the discussion of the distinction between CPT and SFT stages and consolidated the key insights into three main results to avoid redundancy while emphasizing meaningful findings:

1. Result 3 (CPT vs. SFT learning capability differences):
   This result validates the differences in learning capabilities between CPT and SFT. Combined with Result 1, it provides actionable guidance for updating training paradigms in math-specific models. Specifically, it supports the strategy of moving sufficient problem-solving data to the CPT phase, replacing multiple times the volume of generic math corpus.

2. Result 4 (CPT’s robustness to OOD capabilities):
   This result demonstrates that CPT's OOD capability is less affected by data distribution disturbances compared to SFT. It also confirms that introducing training data aligned with the target distribution remains the optimal choice for improving specific capabilities.

3. Result 5 (Primary source of CPT and SFT differences):
   This result identifies the primary source of learning capability differences between CPT and SFT. It emphasizes the importance of preparing more challenging problem-solving data for the CPT phase.

Q14: I am concerned about data quality. It seems there is heavy reliance on synthetic data and vague descriptions of the datasets. What is the data cleansing process?

A14: The methods for data synthesis have been detailed in Section 4 of the paper. The deduplication and depollution processes for the seed dataset used in synthesis are discussed in Section 2. Ensuring the quality of the seed dataset is a comprehensive task, involving both general techniques such as quality filtering and relevance filtering, as well as the integration and iterative refinement of reward models with reasoning processes.

Q15: The real-world applicability or significance of JiuZhang-8B is not discussed.

A15: JiuZhang-8B serves three primary purposes:

1. To validate the research conclusions presented in the paper.
2. To act as a foundational model that introduces a paradigm shift in training math-specific models, with features worth further exploration.
3. To demonstrate competitive performance as a math-specific base model within 8B parameters.

Q16: How do you ensure representative and high-quality data?

A16: This concern overlaps with Q14.

Q17: Can you provide more intuition or theoretical support for these empirical results?

A17: This question aligns with Q12.

Q8: Adding figures throughout would greatly enhance the argument. For example, add a main pipeline figure of the overarching framework you are implementing, and potentially new figures for Sections 3, 4, and 5.

A8: Thank you for the suggestion. We initially considered adding explanatory diagrams for the four methods described in Section 4 to improve clarity, but this was limited by space constraints.

---

Q9: The methodology appears insufficiently rigorous, particularly in the way data distribution and difficulty levels are analyzed.

A9: Thank you for the suggestion. This issue have been discussed in detail in the response to reviewer fBsg. Some clarifications and additional analyses have already been included in Appendix B of the revised PDF.

---

Q10: The experiments lack comprehensive justification, and the comparisons between different training setups (CPT vs. SFT) come across as arbitrary. What are the hypotheses leading to these decisions?

A10: We have reiterated the background of our research questions in A2 and discussed the experimental setup rationale in detail in A7. These provide clear hypotheses and motivations for the experimental comparisons.

---

Q11: Throughout the paper, there are bold claims about the effectiveness of problem-solving data and specific synthesis methods like Tutorship Amplification. I recommend relying less on superficial observations and instead adding more deep analysis of why certain methods are effective.

A11: Thank you for the suggestion. We have addressed this issue in A12, where we provide a detailed discussion of all the results.

---

Q12: I recommend adding some theoretical backing or intuition as to why these methods work.

A12: Thank you for the suggestion.We did not conduct a theoretical analysis. The following is our discussion on intuition from five conclusions.

1. Result 1 (Problem-solving data effectiveness):
   As outlined in the Introduction, the real challenge often lies not in recalling the relevant knowledge but in using this knowledge for reasoning or planning. Therefore, we should use problem solving data instead of math corpus.

2. Result 2 (Tutorship Amplification):
   The effectiveness of Tutorship Amplification can be attributed to its ability to provide synthetic data that imparts self-correction capabilities to the model. By leveraging a teacher model to identify and correct errors in the student model’s responses, the data generated by this method endows the model with the ability to detect and correct errors, thereby enhancing its reasoning ability.

3. Results 3–5 (Differences between CPT and SFT):
   These findings stem from RQ3: How do the capabilities developed from the same problem-solving data differ between the CPT and SFT stages, and what factors contribute to these differences?
   The experimental results should be reliable and provide valuable insights for further analysis. We did not delve deeper into this matter, and further discussions on the follow-up work are undoubtedly important. Here are some of our intuitive thoughts:

• Regarding Result 4's mention of "both SFT and CPT primarily develop capabilities aligned with their data distributions", we speculate that the generalization ability of LLMs is limited. It might be confined to simple patterns such as numerical substitution and conditional result substitution. This limitation may depend on whether the training data includes similar data pairs for learning (akin to the issue of knowledge storage and extraction discussed in "Physics of Language Models: Part 3.1, Knowledge Storage and Extraction"). The generalization of reasoning capabilities warrants further experimental design and exploration.

• Regarding Result 4's mention of "Regarding out-of-domain (OOD) capability learning, SFT is more susceptible to disturbances from data distribution compared to CPT," we speculate that the differences in OOD capability learning represented by different evaluation sets are influenced by the similarity of the evaluation set distribution to the training data; whereas the stability of CPT over SFT is due to the combined effects of generic data mixing and recall.

• For the main part of Result 4, it is noted that SFT's in-domain (IND) learning ability is weaker than that of CPT. This observation prompts the exploration and conclusions presented in Result 5.

• Regarding Result 5, why hard training data results in learning capability differences between CPT and SFT is not well-understood, and may be suited for theoretical analysis, requiring meticulous work. Similar work from a theoretical analysis perspective seems to have just begun (Understanding Chair of That in LLMs Through Information Theory).

Thank you for your review! Below are our responses to each comment:

Q1: I recommend briefly describing in the abstract and introduction what you mean by problem-solving data.

A1: Thank you for the suggestion. We have included a brief description in the introduction. Problem-solving data refers to datasets composed of mathematical problems and their reasoning processes.

Q2: The paper outlines three research questions but fails to present a clear rationale or context for each. Why are these research questions important to answer?

A2: In the original text, the Introduction section provided the background and the key perspective of the paper, introducing the first research question (RQ1). RQ2 and RQ3 are extensions of RQ1. Additionally, at the beginning of Sections 3, 4, and 5, we explained their respective backgrounds and connections to earlier sections. Here, we revisit and elaborate on the rationale:

We proposed RQ1 based on two observations and a perspective:
RQ1: During the CPT stage, can providing problem-solving data more effectively enhance the model's mathematical reasoning capabilities compared to using general mathematical corpora?

• Observation 1: In math-specific LLM training under the current paradigm, the CPT stage often involves a large amount of filtered mathematical corpus tokens. However, the improvements in mathematical reasoning achieved through CPT are often less significant compared to those obtained via SFT.
• Observation 2: Some other domains have attempted to introduce instruction tuning data to teach models how to utilize memorized knowledge during the pre-training stage.
• Perspective: Due to the intrinsic distinction between mathematical knowledge and general world knowledge, different strategies are required for their effective acquisition and application. Mathematical knowledge involves a relatively limited set of elements, concepts, axioms, and theorems that need to be memorized and understood. The real challenge often lies not in recalling the relevant knowledge but in using this knowledge for reasoning or planning. Therefore, we propose that alternative strategies utilizing mathematical problems and their reasoning steps—referred to as problem-solving data—during the pre-training phase, to teach the model how to apply its memorized knowledge rather than simply increasing the volume of relevant data, can potentially lead to significant improvements in mathematical reasoning capabilities.

Following the validation of the effectiveness of problem-solving data in RQ1, RQ2 addresses the challenges arising from the limited availability of such data compared to internet-scale data. Specifically, it explores the necessity of efficient data synthesis methods and examines whether additional synthesis of problem-solving data during pre-training could further enhance model performance. This forms the basis of RQ2:
RQ2: If problem-solving data can enhance mathematical reasoning capabilities, are synthetic data from the same source equally effective, and what synthesis methods are most efficient?

Finally, RQ3 seeks to resolve another dimension of RQ1. Specifically, it questions whether the effectiveness validated in RQ1 is solely due to the data itself or whether there are differences between developing mathematical reasoning skills during the CPT and SFT phases. Furthermore, if differences exist, what factors contribute to them? We believe that delving deeper into this distinction will help clarify the impact of training paradigms and provide valuable guidance for future data collection and synthesis efforts. This leads to RQ3:
RQ3: How does the effectiveness of problem-solving data during pre-training differ from its role in the SFT phase, and what factors contribute to these differences?

Thank you very much for your thorough review. As the discussion period is nearing its conclusion, I wanted to follow up to ensure you’ve had the opportunity to review our detailed rebuttal. Given the additional explanations and adjustments we've incorporated, we would greatly appreciate your feedback on whether our responses have satisfactorily addressed your concerns.

Thank you once again for your time and thoughtful review. We look forward to your response.

Firstly, we would like to thank the reviewer for their response, which includes clarifying concerns about previous issues and summarizing their opinions.

We would like to focus on the summary objection raised in Response 3. The reviewer’s reason for rejecting the submission is “unsatisfied that this is the best that could be achieved in the time provided.” We would like to clarify that we already provided a detailed response 15 days ago, which included answers to the 20 raised questions, significant revisions to the paper, and additional appendices. Given that most of the questions were succinct (often within a line), we made further expansions and discussions on certain points. However, the reviewer only provided additional clarifications on some of the concerns on the last day of the discussion period. Clearly, we could not have anticipated and addressed the unexpressed concerns beforehand. Thus, using “unsatisfied that this is the best that could be achieved in the time provided” as the core reason for rejection is manifestly unfair.

The review comments on the paper itself should serve as the basis for judgment, yet the core objection raised by the reviewer remains uncertain. For the reasons for their decision, the reviewer stated, "My decision is primarily driven by the persistent lack of rigor and coherence in addressing key concerns." However, at the beginning of Response 3, the reviewer, in reference to our initial response, remarked, "While this is valuable for the study's internal coherence, it does not address how the findings translate to practical applications or advancements in the field." This contradictory and unclear core objection reinforces our position that the main objection in Response 3 is unreasonable. As for the mentioned issue of "does not address how the findings translate to practical applications," the reviewer elaborated that this concern arises from the statement, "While the theoretical insights into CPT and SFT are valuable, the practical benefits are not well-articulated." This implies that in our design of comparative experiments to validate and discuss RQ3, we should address the practical benefits. This is illogical, as introducing additional considerations of practical benefits in strictly controlled experimental comparisons could introduce potential confounding factors, thereby undermining the validation of our conclusions.

In fact, we have already provided multiple explanations regarding the motivation of the paper, the theoretical insights offered, and the practical benefits in both the paper and the Initial Response (A2, A3, A12, A13, A15, A18). A summary of these points is as follows:

We proposed RQ1 [1] based on two observations and a perspective. We then extended RQ1 to RQ2 [2] and RQ3 [3] by considering the limitations in problem-solving data quantity and the potential learning ability differences due to the training phase. Finally, based on the research findings, we developed the competitive math-specific base model JiuZhang-8B, which, as a comprehensive validation of the research conclusions, brings about a paradigm shift in training math-specific models [4] .

[1] RQ1: During the CPT stage, can providing problem-solving data more effectively enhance the model's mathematical reasoning capabilities compared to using general mathematical corpora?

• Observation 1: In math-specific LLM training under the current paradigm, the CPT stage often involves a large amount of filtered mathematical corpus tokens. However, the improvements in mathematical reasoning achieved through CPT are often less significant compared to those obtained via SFT.
• Observation 2: Some other domains have attempted to introduce instruction tuning data to teach models how to utilize memorized knowledge during the pre-training stage.
• Perspective: Due to the intrinsic distinction between mathematical knowledge and general world knowledge, different strategies are required for their effective acquisition and application. Mathematical knowledge involves a relatively limited set of elements, concepts, axioms, and theorems that need to be memorized and understood. The real challenge often lies not in recalling the relevant knowledge but in using this knowledge for reasoning or planning. Therefore, we propose that alternative strategies utilizing mathematical problems and their reasoning steps—referred to as problem-solving data—during the pre-training phase, to teach the model how to apply its memorized knowledge rather than simply increasing the volume of relevant data, can potentially lead to significant improvements in mathematical reasoning capabilities.

Next, we will provide specific responses to the questions following the clarifying concerns. It is important to note that, despite adding further descriptions, some of the reviewer's questions still appear to be more like expressions of vague impressions rather than concrete issues, making the actual questions somewhat confusing.

The reviewer initially expressed dissatisfaction with the overall story of the paper. However, we have already provided explanations from two aspects: the motivation and the importance of the three research questions. In the earlier responses, the reviewer seemed to be caught in a dilemma—while they claimed that the motivation was not clearly articulated, they also expressed concerns about the coherence between the three research questions.

We retell the complete story again: We proposed RQ1 [1] based on two observations and a perspective. We then extended RQ1 to RQ2 [2] and RQ3 [3] by considering the limitations in problem-solving data quantity and the potential learning ability differences arising from the training phase. Finally, based on the research findings, we developed the competitive math-specific base model, JiuZhang-8B, which, as a comprehensive validation of our research conclusions, brings about a paradigm shift in training math-specific models [4]. References can be found in the previous sections.

The reviewer next expressed concerns regarding the design details of the mixture ratios.

The different math data mixture ratios used in the control groups provided validation for Result 1, specifically the finding that "Providing math problem-solving data significantly enhances the model’s mathematical capabilities compared to general mathematical corpora," as well as the discovery that "a higher proportion of problem-solving data is more effective."

We have already addressed the question Q7, "Why did you choose these data mixture ratios?" in A7 and the Appendix: Discussion on Experimental Setting. Regarding the newly raised question, "Why were these specific settings expected to yield meaningful insights?", we argue that the Result drawn from the control experiments themselves constitute the meaningful insights.

Furthermore, the reviewer’s specific query in v6B9, “Why not ablate the percentage of the math mixture to extremes like 2:8, 1:9, or even complete problem-solving data?” was addressed in our response. We explained the rationale behind not using extreme settings in the response and supplemented this with additional experiments, which were included in the Appendix under Detailed Results of Problem-Solving Data Effectiveness Experiment.

The reviewer next expressed dissatisfaction with the absence of additional figures.

The original comment raised an odd request—adding figures simply for the sake of adding them. In response, we explained that we had initially considered including a figure to illustrate the synthetic data method but ultimately decided against it due to space constraints and the lack of added clarity.

Additionally, Reviewer XMnz explicitly stated that “A more comprehensive set of practical demonstrations would significantly strengthen the argument for the proposed methods' utility.” We agree that adding relevant figures to address this concern is very reasonable. As such, we described the figures in our response, and we plan to add them once the revision is open.

In the end, the reviewer raised concerns regarding the rigor of the data distribution and difficulty level analysis.

Several new questions were introduced, such as "How were the difficulty levels of the data quantitatively defined and validated?" We believe this has already been clearly described in the manuscript, specifically in lines 439-445.

The reviewer also brought up the issue of computational costs, requesting that we address the trade-offs in our design of comparative experiments to validate and discuss RQ3, which primarily focuses on exploring the differences in model capabilities. However, we feel that this request for a discussion of trade-offs is somewhat misplaced, as the focus of the experiment was not on trade-offs but on the differences in model abilities.

Finally, we suggest referring to the more structured and coherent concerns raised by Reviewer fBsg regarding data distribution, along with our responses, as they provide a clearer foundation for addressing these issues.

Thank you for your review! We greatly appreciate your valuable feedback and the issues you have pointed out. First, we will provide an overall discussion on the experimental approach for RQ3 and clarify the three concepts that may have caused confusion. Subsequently, we will address the specific issues raised. In addition, we have reuploaded the revised manuscript and included supplementary content in Appendix B.

Clarification of the Experimental Approach and Related Results for RQ3

Thank you for your suggestions on the expression of RQ3. We originally recorded the results according to the following experimental logic and hope for clarification. Finally, based on the suggestions, we have revised the conclusion in the PDF to make it clearer.

For RQ3, we first explored the overall differences in capabilities developed during the CPT and SFT stages in Section 5.1. This led to Result 4, indicating that SFT is less effective than CPT in learning mathematical skills. In the following two subsections, we aimed to investigate the sources of this difference from various perspectives:

• Section 5.2: We hypothesized that CPT might contribute to enhanced out-of-distribution (OOD) performance. However, comparative experiments revealed that both SFT and CPT primarily develop capabilities aligned with their respective data distributions, but SFT’s in-domain (IND) learning ability is weaker than CPT's (Result 6).
• Section 5.3: Since Result 6 was more pronounced in high school training data compared to middle school, this inspired us to examine how differences in training data difficulty contribute to the learning disparities between SFT and CPT. We re-segmented the dataset based on the number of steps required to solve problems. This experimentation yielded Result 7, showing that the difference primarily arises because CPT is better at learning from hard data, which partially addresses RQ3. Result 8, in contrast to findings based on data distribution, focused on difficulty dimensions. Unlike the data distribution perspective, where providing corresponding data improves IND capabilities, in the difficulty dimension, even when hard data is provided, it does not directly enhance the ability to solve hard problems. Instead, the model primarily focuses on learning to solve simpler, fewer-step problems.

Three Concepts: Capability Dimensions, Data Distribution, and Difficulty Level

Thank you for pointing out some issues with these three concepts, which seem indeed to have caused confusion. We need to clarify that these were roughly abstracted based on necessity during the research process and were not derived from any existing standards. The absence of such standards necessitates further discussion and harmonization by future work.

Capability Dimensions

Concerning the capability dimensions of the evaluation set, we introduce these dimensions before the specific research question to define three aspects: general knowledge, math knowledge, and reasoning steps. These dimensions help to understand the model results beyond just the average accuracy.

Data Distribution

Regarding the concept of data distribution, in Section 5.2, we explore the capability training differences between SFT and CPT for in-domain and out-of-distribution data by selecting and splitting the training data to represent the data distribution of the evaluation set. This concept is generally considered to be strongly related to the evaluation dimension of math knowledge, but in reality, only when both questions belong to a very fine-grained math knowledge can they be considered to have the same data distribution. As you mentioned, introducing similarity directly from the data distribution itself might be a better approach.

Due to the large size of the training data, it is difficult to apply similarity analysis and ensure the accuracy of numerous fine-grained labels. Hence, we have opted for a high-level label, hoping to cover the data distribution of the evaluation set to explore in-domain capabilities.

Difficulty Level

The concept of difficulty is relatively clear. In Section 5.3, we re-segment the training data subsets using the number of reasoning steps in problem-solving data to explore the differences in training data of varying difficulties at different training stages.

Supplementary Information

We have supplemented Appendix B of the revised paper with:

• Examples from the evaluation set,
• Simple definitions of the three dimensions,
• Data distribution of the evaluation set problems, and
• The similarity calculated based on this distribution.

However, due to the advantage of the training data in covering a broad range of diverse data from the evaluation set, it is quite unstable to analyze through a small number of samples. We hope the added information and discussions clarify the content.