MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Thank you for your thoughtful review and for highlighting areas in need of improvement. Your feedback is invaluable in helping us improve our project.

Q1: The paper lacks an analysis of potential data leakage between MathCode-Pile and evaluation benchmarks, which could artificially inflate model performance.

A1: Thank you for your comment. As mentioned in the fourth paragraph of Section 2.2, to avoid benchmark contamination (or leakage), we filter samples that significantly overlap with questions from the benchmark datasets used in evaluation. Similar to GPT-3 [1] and Llama2 [2], we apply exact matching to remove identical samples and further use 13-gram deduplication (with a condition that the Jaccard similarity should be larger than 0.6) to eliminate additional samples that might cause contamination.

We also apply the n-gram testing, as demonstrated in the table below. The overlap percentages for various n-grams are quite low, and the overlap becomes 0.00% when n-grams are 13. This analysis has been added to Appendix F.

[1] Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

[2] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

Q2: I am interested about whether the MathCode-Pile’s strong focus on mathematical reasoning might impact the model’s performance in non-mathematical domains. For example, whether this dataset would enhance the model’s general coding abilities beyond math-focused tasks.

A2: Thank you for your suggestion. We tested the MathCoder2 models on HumanEval and MBPP, two representative benchmarks for evaluating models' general coding abilities, using the EvalPlus framework [1]. HumanEval+ and MBPP+ are extended versions of HumanEval and MBPP that include additional test samples, as described in [2]. The pass@1 accuracies are presented in the table below. As shown in the results, training on MathCode-Pile improves the performance of Llama3-8B, DeepSeekMath-7B, and Mistral-7B on general coding benchmarks. The performance of MathCoder2-CodeLlama-7B is comparable to CodeLlama-7B, which is understandable since CodeLlama is specifically trained for code generation. These findings highlight that MathCode-Pile can enhance general coding abilities beyond math-specific tasks, particularly for models not explicitly trained for code generation.

To evaluate how MathCode-Pile influences the general abilities of LLMs, we tested the MathCoder2 models on Hellaswag, PIQA, and Winogrande using the lm-evaluation-harness framework [3]. As shown in the table below, training on MathCode-Pile has a slight impact on the performance of general-purpose models such as Llama3-8B and Mistral-7B, likely because MathCode-Pile consists entirely of math-related data. The accuracy of specialized models, such as DeepSeekMath-7B and CodeLlama-7B, remains similar before and after training. We have included this discussion in Appendix G of the revised paper. As shown in the second table below, other specialized models like CodeLlama and DeepSeekMath experience a slight decrease in performance on general benchmarks. In future work, we plan to incorporate general-purpose training data and adjust the ratio of math-related data to mitigate its impact on the general abilities of LLMs.

[1] https://github.com/evalplus/evalplus

[2] Liu, Jiawei, et al. "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation." Advances in Neural Information Processing Systems 36 (2024).

[3] https://github.com/EleutherAI/lm-evaluation-harness

Example 1 of MATLAB:

Computation 1: Sampling Distribution of the Sample Mean

Conditions Needed:

1. The population distribution is normal.
2. The sample size is $n$.

Computation Expression:

$$\bar{X} \sim \mathcal{N}\left(\mu, \frac{\sigma^2}{n}\right)$$

Computation Result: The sampling distribution of the sample mean is a normal distribution with mean $\mu$ and variance $\frac{\sigma^2}{n}$.

MATLAB Code Snippet:

% Define the population parameters
mu = 0;  % Mean of the population
sigma = 1;  % Standard deviation of the population
n = 100;  % Sample size

% Generate a random sample from the population
sample = normrnd(mu, sigma, n, 1);

% Calculate the sample mean
sample_mean = mean(sample);

% Display the result
fprintf('The sample mean is approximately %f. The sampling distribution is N(%.2f, %.2f/n).\n', sample_mean, mu, sigma^2 / n);

Computation 2: Standard Error of the Sample Mean

Conditions Needed:

1. The population distribution is normal.
2. The sample size is $n$.

Computation Expression:

$$\sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}}$$

Computation Result: The standard error of the sample mean is $\frac{\sigma}{\sqrt{n}}$.

MATLAB Code Snippet:

% Define the population parameters
sigma = 1;  % Standard deviation of the population
n = 100;  % Sample size

% Calculate the standard error
std_error = sigma / sqrt(n);

% Display the result
fprintf('The standard error of the sample mean is %.4f\n', std_error);

Computation 3: Sampling Distribution of the Sample Median

Conditions Needed:

1. The population distribution is any absolutely continuous distribution $F$.
2. The sample size is $n = 2k - 1$.

Computation Expression:

$$f_{X_{(k)}}(x) = \frac{(2k-1)!}{(k-1)!^2} \cdot f(x) \left( F(x)(1-F(x)) \right)^{k-1}$$

Computation Result: The sampling distribution of the sample median is given by the above expression.

MATLAB Code Snippet:

% Define the population distribution (uniform distribution for simplicity)
f = @(x) 1;  % PDF of uniform distribution
F = @(x) x;  % CDF of uniform distribution
k = 5;  % Number of observations below the median (n = 2k - 1)

% Calculate the sampling distribution of the sample median
x = 0:0.01:1;
f_sample_median = (factorial(2*k-1) / (factorial(k-1)^2)) .* f(x) .* (F(x) .* (1 - F(x))) .^ (k-1);

% Plot the result
plot(x, f_sample_median);
xlabel('x');
ylabel('f_{X_{(k)}}(x)');
title('Sampling Distribution of the Sample Median');

Example 2 of MATLAB:

Computation 1: Product Rule for Derivatives

Conditions Needed:

1. The function is a product of two functions, $h(x)$ and $l(x)$.

Computation Expression:

$$\frac{d}{dx} (h(x) \cdot l(x)) = h'(x) \cdot l(x) + h(x) \cdot l'(x)$$

Computation Result:

$$h'(x) \cdot l(x) + h(x) \cdot l'(x)$$

Mathematica Code Snippet:

(* Define the symbol x *)
Clear[x]

(* Define the two functions h(x) = x^2 and l(x) = e^x *)
h[x_] = x^2;
l[x_] = Exp[x];

(* Compute the product h(x) * l(x) *)
f = h[x] * l[x];

(* Compute the derivative using the product rule *)
derivative = D[f, x]

Computation 2: Final Derivative of $f(x) = e^{2x} \cdot \ln(x)$

Conditions Needed:

1. The function is $f(x) = e^{2x} \cdot \ln(x)$.

Computation Expression:

$$\frac{d}{dx} (e^{2x} \cdot \ln(x)) = e^{2x} \cdot \frac{d}{dx} (\ln(x)) + \ln(x) \cdot \frac{d}{dx} (e^{2x})$$

Computation Result:

$$e^{2x} \cdot \left( 2 \ln(x) + \frac{1}{x} \right)$$

Mathematica Code Snippet:

(* Define the symbol x *)
Clear[x]

(* Define the function f(x) = e^(2x) * Log(x) *)
f = Exp[2 x] * Log[x];

(* Compute the derivative of f(x) *)
derivative = D[f, x]

Thank you for taking the time to review our work and for providing your insightful feedback.

Q1: Python was chosen for code snippets; is it possible to use specialized math software language instead (e.g., Mathematica)? This is not a direct limitation of this paper, but a possible future direction.

A1: Thank you for your insightful suggestion. Using math software languages such as Mathematica and MATLAB paired with natural language reasoning is indeed a possible future direction.

We chose Python for code snippets because it has wide spread accessibility, is easier to use, and more suitable for large-scale execution. Applying our method to specialized math software languages could further enhance mathematical reasoning capabilities, opening up new possibilities for improvement. To provide a glimpse into this direction, we have included examples of model-generated synthetic Mathematica and MATLAB code paired with natural language reasoning below. Thank you once again for your thoughtful feedback.

Example 1 of Mathematica:

Computation 1: Definite Integral for the Length of the Curve

Conditions Needed:

1. The curve is given by the function $y = x + \cos(x)$.
2. The curve is to be integrated over the interval $0 \leq x \leq 5$.
3. The variable $x$ is given in radians.

Computation Expression: $\int_{0}^{5} \sqrt{1 + \left(\frac{dy}{dx}\right)^2} \, dx$

Computation Result: The length of the curve.

Mathematica Code Snippet:

(* Define the function y = x + cos(x) *)
y[x_] := x + Cos[x]

(* Compute the derivative of y with respect to x *)
dy_dx[x_] := D[y[x], x]

(* Set up the integral to calculate the length of the curve *)
Integrate[Sqrt[1 + (dy_dx[x])^2], {x, 0, 5}]

Computation 2: Numerical Integration to Find the Length

Conditions Needed:

1. The definite integral from Computation 1 is to be evaluated numerically.
2. The result is to be rounded off to four decimal places.

Computation Expression: $N\left( \int_{0}^{5} \sqrt{1 + \left(1 - \sin(x)\right)^2} \, dx \right)$

Computation Result: The length of the curve rounded off to four decimal places.

Mathematica Code Snippet:

(* Numerical integration of the curve length, rounded to four decimal places *)
N[Integrate[Sqrt[1 + (1 - Sin[x])^2], {x, 0, 5}], WorkingPrecision -> 4]

Example 2 of Mathematica:

Computation 1: Intersection of Probabilities of Two Dice

Conditions Needed:

1. We have two dice, each with $n$ sides.
2. We want to find the probability that the maximum of the two dice is less than or equal to some value $a$.

Computation Expression: $P(X \le a) = P(X_1 \le a) \cdot P(X_2 \le a) = \frac{a^2}{n^2}$

Computation Result: The probability that the maximum of the two dice is less than or equal to $a$ is $\frac{a^2}{n^2}$.

Mathematica Code Snippet:

(* Define a function for the probability that the maximum is less than or equal to a *)
probabilityMaxLessThanEqual[a_, n_] := (a^2)/n^2;

(* Example for two dice with 6 sides and max less than or equal to 3 *)
probabilityMaxLessThanEqual[3, 6]

Computation 2: Probability Distribution Function of the Maximum of Two Dice

Conditions Needed:

1. We have two dice, each with $n$ sides.
2. We want to find the probability that the maximum of the two dice is equal to some value $a$.

Computation Expression: $P(X = a) = P(X \le a) - P(X \le a-1) = \frac{a^2}{n^2} - \frac{(a-1)^2}{n^2}$

Computation Result: The probability that the maximum of the two dice is equal to $a$ is $\frac{a^2}{n^2} - \frac{(a-1)^2}{n^2}$.

Mathematica Code Snippet:

(* Define a function for the probability that the maximum is exactly equal to a *)
probabilityMaxEqual[a_, n_] := (a^2)/n^2 - ((a - 1)^2)/n^2;

(* Example for two dice with 6 sides and max equal to 3 *)
probabilityMaxEqual[3, 6]

Q2: Evaluation Uncertainty: It is ambiguous whether the generated Python code is executed during benchmark evaluations or merely used for pretraining, leaving questions about its practical impact. Is the Python code generated and executed during benchmark evaluations? Clarifying this would help to understand the role of Tool-Integrated Reasoning in the observed performance improvements.

A2: During the evaluation of MathCoder2 models following continued pretraining, we use a text-only format, without utilizing code execution. We chose not to use code to demonstrate that our method improves the general mathematical reasoning abilities of LLMs and ensures a fair comparison with baseline methods. The results of post-training include two formats: Chain-of-Thought (CoT) and Tool-Integrated-Reasoning (TIR). CoT testing does not involve code generation, while TIR executes the generated Python code and appends the output to the model's generation to provide feedback from the execution. This approach is similar to those described in [1] and [2]. Our continued pretraining enhances the models’ ability to be fine-tuned for TIR reasoning, as demonstrated in the ablation study presented in Table 7 of the paper.

[1] Wang, Ke, et al. "Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning." arXiv preprint arXiv:2310.03731 (2023).

[2] Gou, Zhibin, et al. "Tora: A tool-integrated reasoning agent for mathematical problem solving." arXiv preprint arXiv:2309.17452 (2023).

Q3: Scope Limitation: The focus on grade-school-level mathematics is not explicitly emphasized, potentially misleading readers about the dataset’s broader applicability.

A3: You are correct that our work primarily focuses on mathematical problems ranging from grade-school word problems (GSM8K) to high-school competition problems (MATH) and college-level problems (OCW). Other forms of mathematical reasoning, such as formal theorem proving, are beyond the scope of this study. We have acknowledged this as a limitation and a future direction in Section 5.

Q4: Ablation Study Depth: While the ablation studies show the value of the synthesized code, further exploration into the necessity of aligning reasoning steps with code versus treating them as independent could provide deeper insights. Would separating reasoning steps and corresponding code into independent examples still yield significant improvements? Such an ablation could help assess the criticality of their alignment in the dataset.

A4: Following your suggestion, we have added an ablation study of training DeepSeekCoder-1.3B on data with separated reasoning steps and corresponding code, as presented in the "Basic + Separated Text&Code" row in the table below. Separating the reasoning steps and corresponding code reduces performance compared to pairing them together, which demonstrates the effectiveness of our design. This ablation study has also been added to Table 4 in the paper.

Example 2 of Lean:

Theorem: The derivative of $e^{2x} \cdot \ln x$ is $e^{2x} \left( 2 \ln x + \frac{1}{x} \right)$.

Conditions:

1. The derivative of $e^{g(x)}$ is $e^{g(x)} \cdot g'(x)$, i.e., the derivative of the exponential function is the exponential function itself multiplied by the derivative of the exponent.
2. The derivative of $\ln x$ is $\frac{1}{x}$, i.e., the derivative of the natural logarithm function is the reciprocal of $x$.

Proof Process:

We are tasked with proving that the derivative of the product of the two functions $h(x) = e^{2x}$ and $l(x) = \ln x$ follows the product rule for derivatives:

1. Start with the product rule for derivatives. The product rule states that if we have two differentiable functions $h(x)$ and $l(x)$, then the derivative of their product is:

   $$\frac{d}{dx} \left( h(x) \cdot l(x) \right) = h'(x) \cdot l(x) + h(x) \cdot l'(x)$$

2. Apply the chain rule to $h(x) = e^{2x}$. The function $h(x) = e^{2x}$ is a composition of functions, so we apply the chain rule:

   $$h'(x) = \frac{d}{dx} e^{2x} = e^{2x} \cdot \frac{d}{dx}(2x) = 2 e^{2x}$$

3. Differentiate $l(x) = \ln x$. The derivative of the natural logarithm function is:

   $$l'(x) = \frac{d}{dx} \ln x = \frac{1}{x}$$

4. Combine the results. Applying the product rule to the functions $h(x) = e^{2x}$ and $l(x) = \ln x$, we get:

   $$\frac{d}{dx} \left( e^{2x} \cdot \ln x \right) = 2 e^{2x} \cdot \ln x + e^{2x} \cdot \frac{1}{x}$$

5. Simplify the expression. Factor out $e^{2x}$:

   $$\frac{d}{dx} \left( e^{2x} \cdot \ln x \right) = e^{2x} \left( 2 \ln x + \frac{1}{x} \right)$$

Thus, we've proven that:

$$\frac{d}{dx} \left( e^{2x} \cdot \ln x \right) = e^{2x} \left( 2 \ln x + \frac{1}{x} \right)$$

Lean Code Snippet:

import tactic

variables {x : ℝ}

-- Lemma for the product rule
lemma derivative_product_rule (h l : ℝ → ℝ) (h' l' : ℝ → ℝ) :
  (h * l).derivative = h'.derivative * l + h * l'.derivative :=
begin
  ext,  -- Apply the extensionality tactic to simplify and reason about derivatives
  simp, -- Simplify the goal using basic simplification rules
  ring, -- Apply ring tactics to simplify the expression
end

-- Lemma for the derivative of e^(g(x)) where g is a function
lemma derivative_e_pow (g : ℝ → ℝ) (g' : ℝ → ℝ) :
  (e ^ g).derivative = e ^ g * g'.derivative :=
begin
  ext,  -- Apply the extensionality tactic to simplify and reason about derivatives
  simp, -- Simplify the goal using basic simplification rules
  ring, -- Apply ring tactics to simplify the expression
end

-- Lemma for the derivative of ln(x)
lemma derivative_ln : (ln x).derivative = 1 / x :=
begin
  ext,  -- Apply the extensionality tactic to simplify and reason about derivatives
  simp, -- Simplify the goal using basic simplification rules
  ring, -- Apply ring tactics to simplify the expression
end

-- Final derivative computation for e^(2x) * ln(x)
lemma derivative_f (x : ℝ) :
  (e ^ (2 * x) * ln x).derivative = e ^ (2 * x) * (2 * ln x + 1 / x) :=
begin
  -- Use the product rule for the derivative of the product e^(2x) * ln(x)
  have h : (e ^ (2 * x) * ln x).derivative = (e ^ (2 * x)).derivative * ln x + e ^ (2 * x) * (ln x).derivative,
    by apply derivative_product_rule,
  
  -- Apply the chain rule to differentiate e^(2x) and ln(x)
  have h' : (e ^ (2 * x)).derivative = e ^ (2 * x) * (2 * x).derivative,
    by apply derivative_e_pow,
  
  have h'' : (ln x).derivative = 1 / x,
    by apply derivative_ln,
  
  -- Substitute and simplify the terms
  rw [h, h', h''],
  ring,  -- Use the ring tactic to simplify the final expression
end

Thank you for the time and effort you have given to reviewing our work. We greatly value your insights and suggestions, and are happy to address your questions and clarify any concerns in the following responses.

Q1: Generalizability of Code Generation: The method’s applicability to more abstract or advanced mathematical domains is unclear, particularly beyond high-school-level math. Can the method be extended to generate formal proofs in languages like Lean or Coq? Specifically, how well does the approach handle abstract reasoning steps that require formal verification, which might be better suited to proof assistants rather than executable Python code?

A1: Thank you for your valuable comment. It is true that our work primarily focuses on mathematical problems ranging from grade-school word problems (GSM8K) to challenging high-school competition problems (MATH) and college-level problems (OCW), demonstrating notable improvements on a wide range of mathematical benchmarks.

We also test our model on the popular formal language benchmark of minif2f_isabelle. As shown in the table below, on Llama3-8B and DeepSeekMath-7B, the accuracy on minif2f_isabelle improves after training with MathCode-Pile. This demonstrates that MathCode-Pile also improves models’ ability to do formal reasoning.

Formal proofs, however, are beyond the scope of this work. Currently, our dataset does not contain any formal proof data. We have added this as a limitation and potential future work in Section 5. In the future, we plan to extend our method to generating formal proofs in languages like Lean, Coq, and Isabelle to further improve the formal reasoning ability of MathCoder2. Additionally, we have included some examples of Lean code generated using our method below. Thank you once again for your valuable feedback and support.

Example 1 of Lean:

Theorem 1: The notation $\frac{\partial^2 z}{\partial x \partial y}$ represents the second derivative of $z$ with respect to $y$ and then $x$.

Conditions:

1. The function $z$ is a function of $x$ and $y$, i.e., $z = f(x, y)$.
2. The partial derivatives of $z$ with respect to $x$ and $y$ exist and are continuous.
3. The function $z$ is at least twice differentiable, meaning that the second partial derivatives $\frac{\partial^2 z}{\partial x \partial y}$ and $\frac{\partial^2 z}{\partial y \partial x}$ exist.

Proof Process: We want to prove that $\frac{\partial^2 z}{\partial x \partial y}$ represents the second derivative of $z$ with respect to $y$ and then $x$. Here’s a step-by-step breakdown:

1. First, recall the definition of a partial derivative. The partial derivative of $z = f(x, y)$ with respect to $x$ is the rate of change of $f(x, y)$ as $x$ changes, while keeping $y$ constant:

   $$\frac{\partial f}{\partial x} = \lim_{\Delta x \to 0} \frac{f(x + \Delta x, y) - f(x, y)}{\Delta x}$$

2. Next, consider the second mixed partial derivative $\frac{\partial^2 z}{\partial x \partial y}$. This means we first take the derivative of $f(x, y)$ with respect to $x$, holding $y$ constant, and then we differentiate this result with respect to $y$.

   Formally:

   $$\frac{\partial^2 f}{\partial x \partial y} = \frac{\partial}{\partial y} \left( \frac{\partial f}{\partial x} \right)$$

   We must show that this is consistent with the notation $\frac{\partial^2 f}{\partial x \partial y}$.

3. Apply the definition of partial derivatives: In the definition of mixed partials, we can interchange the order of differentiation if the second partial derivatives are continuous (Clairaut's Theorem). Thus:

   $$\frac{\partial^2 f}{\partial x \partial y} = \frac{\partial^2 f}{\partial y \partial x}$$

   This result holds for continuous second partial derivatives, and it follows directly from the symmetry of second mixed partials in the case of functions that are sufficiently smooth.

Lean Code Snippet:

import analysis.calculus

variables {x y : ℝ} {f : ℝ → ℝ → ℝ}

-- Define the second partial derivative of a function
def second_partial_derivative_x_y (f : ℝ → ℝ → ℝ) (x y : ℝ) : ℝ :=
  ∂ (∂ f x) y

-- Lemma showing that the second partial derivative with respect to x then y is the same as y then x
lemma partial_derivatives_commute (f : ℝ → ℝ → ℝ) (x y : ℝ) :
  ∂² f x y = ∂² f y x :=
begin
  -- Apply the fact that mixed partial derivatives commute for sufficiently smooth functions
  have h : ∀ (f : ℝ → ℝ → ℝ) (x y : ℝ), continuous (λ (z : ℝ), ∂ f z y) → continuous (λ (z : ℝ), ∂ f x z),
  { intro f, intros x y, apply continuous_smoothness_of_partial_derivatives, },
  -- Now apply Clairaut's Theorem
  exact h f x y,
end

Dear Area Chair,

Thank you for the time and effort you have dedicated to overseeing the review process for our work. We truly appreciate you reminding the reviewers to read our rebuttal and ensuring that our response received the necessary attention. Your support and guidance throughout this process mean a great deal to us.

Sincerely,
The Authors