DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving

Thanks for your comments! We address your concerns below.

Q1: The proposed sampling technique is trivial and incremental …… the uniform method is used in ToRA, and the prop2diff method is used in MARIO.

A1: We respectfully disagree with the reviewer. Both ToRA and MARIO are distinct from our approach on how responses are distributed across various queries. While we adjust the distribution either to be uniform or to favor more difficult queries, ToRA&MARIO focus mainly on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easier queries as we show quantitatively in the general author rebuttal. Please check the general author rebuttal for a summary of our point on this concern.

Here we supplement more details on how we replicate the MARIO synthesis pipeline to conduct the analysis present in the general author rebuttal (the ToRA synthesis pipeline is described in response A2 to Reviewer neCQ). Below we describe in the format as “how MARIO synthesizes data -> how we replicate a simplified version of it” step by step:

---

---

1. Greedy decoding using GPT3.5 and GPT-4 each once for MATH&GSM8K, getting two responses for each query, only correct ones are kept -> We follow this but use GPT-4o mini to sample two responses for each query.
2. Sampling for 2 trials for each problem not correctly answered in 1. using GPT-4, only correct ones are kept -> We follow this with GPT-4o mini.
3. Manually correcting responses for part of the remaining problems, then tuning Llemma-34B on it to obtain a synthesis agent for next steps -> this involves human annotation and is not comparable to our approach. For simplicity, we adopt DeepSeekMath-7B-RL as the synthesis agent to align with DART-Math.
4. Sampling with 100 trials and keeping up to 4 correct responses per problem for the remaining unanswered MATH queries, achieving 93.8% coverage on MATH -> we follow this and achieve 91.3% coverage on MATH.
5. Sampling with 1 trial for new problems introduced by MetaMath and keeping correct ones -> this step introduces new prompts and would only skew the distribution of responses, if any, towards easy queries. We remove this step for simplicity, which would not affect our conclusion.

---

---

While MARIO performs more sampling trials for difficult problems, it mainly uses coverage rate as the criterion, and a high coverage rate can be obtained while the distribution is still biased towards easy queries.

We show the average # of responses per problem for different difficulty levels in the table of the general rebuttal. Our replicated ToRA and MARIO datasets obtain a similar MATH coverage ratio and total size to their original ones — While the absolute # of responses is not directly comparable between different methods, distribution-wise we can see that ToRA/MARIO still produce fewer responses for difficult problems than the easy ones. This contrasts sharply with DART-MATH-Hard, which produces, for example, 10x more responses for the MATH Level 5 queries than for the GSM8K queries.

Q2: There’s little improvement or even performance drop when tuning DeepSeekMath-7B

A2: While an improvement over the VRT baseline is smaller for Llama3-70B and DeepSeekMath-7B, we note that our results are already much better than the ones from the best open-source datasets (around 4-5 points better on average) on all 4 models. Thus the release of our datasets themselves is a significant contribution to the open-source community -- DART-Math is the SotA, public CoT datasets for math problem solving and much better than existing datasets across all the assessed models.

Q3: this dataset is somehow replaceable by math-specific continual pre-training + SFT

A3: This may be true, but we don't think it is a major issue. Strictly speaking, we suspect most carefully curated math SFT datasets are somehow replaceable by extensive continual pretraining + simpler SFT, yet this fact does not diminish the significance of these SFT datasets -- Continual pretraining typically occurs on a much larger scale, for instance, DeepSeekMath involves continual pretraining on 150B math-specific tokens, whereas DART-Math contains only ~0.2-0.3B tokens. Given this scale, the efficiency provided by SFT datasets remains crucial and non-replaceable for most developers.

Q4: even the paper claims the proposed dataset is smaller, however, the LLM used to synthesize the smaller dataset is DeepSeekMath-7B-RL, which is trained on a larger SFT dataset

A4: We don’t think using an instructed model is a major issue as we aim to create the best synthetic SFT dataset. This setting aligns with most related works that use GPT-4, an aligned model. When we claim the proposed dataset is smaller, we suggest the reviewer note that our baselines all use GPT-4/3.5 to generate data; these models are hypothetically trained on an even larger SFT&RL dataset than DeepSeekMath-7B-RL.

Q5: What’s the query coverage ratio on MATH training set constructed by Prop2Diff?

A5: 99.6%

Q6: Any figure or statistics to show the difficulty distribution of your DART dataset?

A6: Figure 1 (Right).

Q7: Any case studies to show the generated responses of your DART dataset? How do you extract answers in the raw response?

A7: The answers are mostly extractable after we improve existing regular expression implementation to be more comprehensive. Please refer to Table 1 in the PDF of the general rebuttal for several cases of responses for the hardest MATH problems.

Thanks for your reply! We would like to note that our work is not about self KD, but in general studying how to synthesize the best data from a strong model, which is like distilling from a stronger teacher model to student models, as we experimented with four different student models in the paper with the same teacher model. This setting aligns with many existing works [1,2,3,4,5,6,7], yet we replace their synthesis agent (GPT-4/3.5) with an open-weight model.

Regarding the specific concern raised about using DeepSeekMath-7B-Base as the student model, this configuration is not strictly a self-KD setting either, because DeepSeekMath-7B-RL undergoes significant training with SFT+RL from DeepSeekMath-7B-Base on potentially large-scale human data, positioning it more akin to a stronger teacher -> student distillation scenario, therefore, it is not surprising that the student is not significantly better than the stronger teacher, just as the previous works that distill from GPT-4 cannot surpass GPT-4. The self-KD case as in existing works [8,9,10,11] should correspond to synthesizing from DeepSeekMath-7B-Base and training DeepSeekMath-7B-Base, or synthesizing from DeepSeekMath-7B-RL and training DeepSeekMath-7B-RL. As the reviewer mentioned in the original review, we agree that these self-KD settings are reasonable and meaningful, we didn't adopt this setting simply because self-KD is not the focus of this paper, just as previous works that distill from GPT-4/3.5 [1,2,3,4,5,6,7] never explore the self-KD experiments with other synthesis agents.

(A minor correction: the cited “86.7 GSM8K / 58.8 MATH” results by the reviewer is the tool-integrated results that rely on external tools and are not comparable, their CoT results are 88.2 GSM8K / 51.7 MATH).

the SFT model is comparable or even worse than the original DeepSeek-RL. So why not directly use the open-sourced DeepSeek-RL?

High-quality synthetic data is of great values for the developers and open-source community, various developers may rely on those data to help strengthen their own models. Our primary goal is not to create a math model for direct use, but to develop better data synthesis methods — the roles of data synthesis and the synthetic data are not replaceable by “directly use the open-sourced DeepSeek-RL”, for example, imagine someone wants to boost the math ability of Mistral-7B during post-training while still keeping it as a generic model, they can utilize our approach to synthesize data from another math-specific model, and incorporate the data together with other SFT data as commonly practiced nowadays, but directly using DeepSeek-RL does not fulfill the goal.

As the reviewer mentioned above, self-KD is one way to synthesize data for self-improvement, yet distilling from other models is very common as well, and our paper focuses on the later where student models do not need to surpass the teacher.

[1] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.” Preprint 2023.

[2] Yu, Longhui, et al. "MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models." ICLR 2024.

[3] Liu, Haoxiong, and Andrew Chi-Chih Yao. "Augmenting math word problems via iterative question composing." Preprint 2024.

[4] Mitra, Arindam, et al. "Orca-math: Unlocking the potential of slms in grade school math.” Preprint 2024.

[5] Li, Chen, et al. "Common 7b language models already possess strong math capabilities.” Preprint 2024.

[6] Huang, Yiming, et al. "Key-point-driven data synthesis with its enhancement on mathematical reasoning." Preprint 2024.

[7] Tang, Zhengyang, et al. "MathScale: Scaling Instruction Tuning for Mathematical Reasoning." ICML 2024.

[8] Wang, Yizhong, et al. "Self-Instruct: Aligning Language Models with Self-Generated Instructions.” ACL 2023.

[9] Dong, Hanze, et al. "RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment.” TMLR.

[10] Yuan, Weizhe, et al. "Self-Rewarding Language Models.” ICML 2024.

[11] Chen, Guoxin, et al. "AlphaMath Almost Zero: process Supervision without process.” Preprint 2024.

Thanks for your reminder that DeepSeek-Math-RL can achieve “86.7 GSM8K / 58.8 MATH” and "88.2 GSM8K / 51.7 MATH" for PoT and CoT, respectively. As other partial concerns are addressed, I will increase rating from 3 to 4. However, my main concern still exists based on the 3 following facts.

1. The SFT data is generated by running the open-sourced DeepSeek-Math-RL. In addition, the modified reject sampling based data generation method is not a significant innovation, which is widely used in previous works.
2. If DeepSeek-Math-Base is used as the base model, the performance of the fine-tuned model is only comparable with the original DeepSeek-Math-RL. The improvement is not significant.
3. If a less competitive base model is used, like Llama or Mistral, the performance of the fine-tuned models are significantly worse than the original DeepSeek-Math-RL.

The standard reject sampling in the original DeepSeek paper already achieved strong results. In either way, I did not observe any advantage of using the proposed method over the open-sourced DeepSeek-Math-RL.

Thanks for your positive comments! We address your concerns below.

Q1: The success of DART relies heavily on the ability of models to generate correct responses for difficult queries, which may not always be feasible for extremely challenging problems.

A1: This is indeed a limitation for all rejection-sampling-based approaches, including both DART and the baselines. However, at least for the difficulty level of the MATH dataset, in Section 3.1 (Figure 2, Right) we have shown that DeepSeekMath-7B-RL is able to generate correct responses for nearly 100% of the queries in the MATH500 test set when sampling more.

Q2: While the focus on difficult queries is commendable, the quality of the generated responses for these queries needs to be high to truly benefit the training process. The paper does not provide a detailed analysis of the quality of these responses.

A2: This is a good point. In this work, we just use the final answer to filter responses as in most previous works, while evaluating quality of these responses is non-trivial yet potentially beneficial -- an additional reward model may be needed for assessing their quality as in the Llama3 paper. We leave exploration of this for future work.

Q3: The approach's reliance on extensive sampling for difficult queries might pose scalability issues, particularly for very large datasets or models with limited computational resources.

A3: We have conducted quantitative analysis about the synthesis cost in Section 4.3, please check it for details. While our method indeed requires extensive sampling for difficult queries, we highlight that the synthesis cost is one-time and can be amortized by distributing the obtained datasets to a large open-source community to use. Additionally, our synthesis agent is only a 7B-size model, unlike most other works that use GPT-4. Thus, our synthesis cost per example is much cheaper compared to previous works, although we do need to synthesize more examples.

Q4: How to be aware of difficulty if not labelled

A4: We sample multiple responses for each query with DeepSeekMath-7B-RL and use the proportion of incorrect responses (i.e., fail rate) as the difficulty score. We have described this in Line 158-166 of the submission.

Q5: How to choose $k_u$

A5: Generally, according to the scaling curves in Figure 3, larger $k_u$ (or $k_p$) correlates with higher accuracy. Thus, larger $k_u$ (or $k_p$) is overall preferred. However, the value of $k_u$ (or $k_p$) directly determines the final dataset size, so the choice of $k_u$ (or $k_p$) is primarily based on the desired final dataset size within resource constraints. In this paper, we choose $k_u=40$ and $k_p=192$ to ensure that both DART-Math-Uniform and DART-Math-Hard datasets end up with ~590K examples.

Q6: The details of Prop2Diff is missing. How many samples were generated for each difficulty level? What is the equation for generating numbers?

A6: As we described in Line 137-139, we compute a difficulty score for each query (see Line 158-166) and the number of correct responses per query is linearly proportional to its difficulty score, where the most difficult query receives $k_p=192$ responses — suppose the difficulty score is $r_i\in[0,1]$, number of correct responses for a certain query $i$ is $k_p * r_i$. We keep sampling until the designated number of correct responses is collected or a max sampling cap $n_{\max}$ is reached (Line 145).

Thanks for your comments! We address your concerns below:

Q1: Assigning more budget to more complex questions in data synthesis is a common practice. For instance, in [1], which successfully annotated 83.1% of MATH questions

A1: First, we note that most recent works in mathematical data synthesis, including nearly all commonly-used open-source datasets, allocate equal budgets to all questions regardless of their complexity [2,3,4,5,6]. While a few works such as the mentioned ToRA try assigning more budget to more complex questions to improve coverage, we highlight that ToRA still results in a bias towards easier queries. This is in stark contrast to our Uniform and Prop2diff strategies that focus on the distribution of responses, rather than merely improving coverage. Please see the Response to Q2 below for a detailed comparison with ToRA.

Additionally, we would like to point out that 83.1% coverage mentioned from ToRA is not an exceptionally high number -- for comparison, the MetaMathQA-MATH-AnsAug dataset achieves 82.8% of coverage on the MATH training set with evenly allocated budgets yet still admits bias towards easy queries as analyzed in Figure 2 of our submission. Below we show the coverage rate per difficulty level of different approaches. The ToRA-Corpus-16k statistics show that it only covers 68% of the Level 5 MATH queries while DART-Math datasets cover 99.6%.

Q2: Could you elaborate on how your approach differs from the rejection sampling strategy used in [1]?

A2: Please check the general rebuttal for a summary of difference of our approach from ToRA. The most important difference between our approach and ToRA is how responses are distributed across various queries -- While we adjust the distribution either to be uniform or to favor more difficult queries, ToRA focuses mainly on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easier queries as shown in the results of the general author rebuttal.

Here we supplement more details on how we replicate the ToRA synthesis pipeline to conduct the analysis present in the general author rebuttal. Below we show in the format as "ToRA's method -> how we adapt similar spirits for a simpler replication" step by step (we use CoT format rather than tool-integrated reasoning for a fairer comparison with our datasets):

---

---

1. Greedy decoding once for each problem in MATH&GSM8K with GPT-4, keeping the correct responses -> we follow this with GPT-4o mini.
2. Sampling for 10 trials for each problem not correctly answered by greedy decoding with GPT-4 and keeping up to 4 correct responses per problem (to form ToRA-Corpus-16k) -> we follow this with GPT-4o mini
3. Training CodeLlama models on ToRA-Corpus-16K to perform rejection sampling next -> to avoid additional training for a fairer comparison, we use DeepSeekMath-7B-RL to replace the trained CodeLLama models here to align with DART-Math
   1. Sampling with 64 trials for each problem in MATH&GSK8K with CodeLlama, getting 233k distinct correct responses -> we follow this with DeepSeekMath-7B-RL, getting 733k distinct correct responses
   2. correcting wrong responses by greedy decoding from the correct preceding portions (costing no more than 64 trials for each problem) with CodeLLaMA-34B, getting 69k corrected responses -> we simplify this by re-sampling another up to 64 trials per problem for all the incorrect responses, getting 225k correct samples.
   3. Randomly selecting up to 4 correct responses per problem from 3.1&3.2 -> we exactly follow this.
4. Merge ToRA-Corpus-16k and data from step 3 to form the final training dataset of 69k responses -> we exactly follow this to form the final dataset of 72k responses.

---

---

We show the average numbers of responses per problem for different difficulty levels and coverages on the MATH training set in the table of the general rebuttal — distribution-wise we can see that the ToRA pipeline will still produce fewer responses for difficult problems than the easy ones, while DART-Math-Hard produces, for example, 10x more # of responses for MATH Level 5 questions than the GSM8K questions.

[1] Gou, Zhibin, et al. "Tora: A tool-integrated reasoning agent for mathematical problem solving." ICLR 2024.

[2] Yu, Longhui, et al. "MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models." ICLR 2024.

[3] Wang, Ke, et al. "MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning." ICLR 2024.

[4] Liu, Haoxiong, et al. "Augmenting math word problems via iterative question composing." Preprint 2024.

[5] Huang, Yiming, et al. "Key-point-driven data synthesis with its enhancement on mathematical reasoning." Preprint 2024.

[6] Tang, Zhengyang, et al. "MathScale: Scaling Instruction Tuning for Mathematical Reasoning." ICML 2024.

Dear Reviewer neCQ,

Sorry to disturb you for the last time, but only one day is left until the end of the reviewer-author discussion stage. We still do not know if you have received our newest response. To address your concerns, we wrote all the responses in detail and added new experiments to support them, including:

1. A1: showing 83.1% coverage mentioned from ToRA is not an exceptionally high number;
2. A2 & Author Rebuttal: elaborately designed experiments and detailed explanations to clarify the difference between the sampling strategies between ToRA and DART.

Conducting the additional experiments within the limited rebuttal period is challenging. We would like to know whether our responses have addressed your concerns. If you still have other concerns, please give us an opportunity to clarify them.

We sincerely hope that you can take a moment to reply, as it is very important for researchers and their efforts on this work.

Best regards,

The Authors

Thank you for the positive comments! We address your concerns below.

Q1: It is unclear how the hyperparameters of the baseline, VRT, were tuned. For instance ……, sampling temperature is searched from 0.3 to 1.7 for DART.

A1: We searched temperature from 0.3 to 1.7 according to accuracies by DeepSeekMath-7B-RL on the MATH training set, where $t=1.6$ is the highest temperature that does not suffer from a significant accuracy drop. $t=1.6$ is applied for both VRT and DART-Math. This choice should be fair since the parameter search is not specifically tailored for DART method.

Q2: It is unclear if the improved performance of the proposed method is due to difficulty or the topic of the problem. … An analysis of topic-wise performance comparing DART and baseline methods may clarify this.

A2: Thanks for the advice! Difficulty and topic are naturally correlated. As shown in the following table for two models, both topic-wise and topic-macro-average scores (which assign equal weights to different topics) still show significant improvement on every topic by DART.

Q3: Another caveat is the need for extensive hyperparameter tuning compared to baselines

A3: To clarify, we apply the same training hyperparameters to all the experiments on the same base models, thus there is no additional hyperparameter tuning for DART compared to the baselines during training. For data synthesis, the only additional hyperparameters specifically chosen for DART are # responses per query (Line 174-175) and the sampling cap (Line 145), which is mainly decided by the desired final dataset size considering our resource constraints, we did not “tune” these synthesis hyperparameters according to final performance.

Q4: It is unclear how much advantage the method would provide in the case of other multi-iteration fine-tuning methods

A4: This is a good point. DART is compatible with multi-iteration fine-tuning as well and we can use the DART strategy to manipulate data synthesis for each iteration. We leave exploration on this as future work.

Q5: The data synthesis is only done using the DeepSeekMATH-7B model. It is unclear why this model was chosen.

A5: The DeepSeekMath-7B-RL model is chosen as the strongest open-source model (though with only 7B size) for mathematical problem-solving.

Q6: Previous methods using VRT-like methods typically use the same model for synthesis and generation. Thus, higher results in smaller models such as Llama-8B may partly be due to the use of stronger models' reasoning chains, making it similar to a distillation method.

A6: We agree that our approach is similar to a distillation method, and our goal is to create the best synthetic data for mathematical problem-solving. The mentioned works that use the same model for synthesis represent a different line of work that pursues" self-improvement," while our paper can be viewed as pure data synthesis. This aligns with many existing works in this field that produced SOTA synthetic datasets using GPT-4, a stronger model.

Q7: In line 138, "we sample till correct responses for each query is proportional to its difficulty score," does it mean linearly proportional?

A7: Yes, it means linearly proportional.

Q8: Do the authors have a comparison of results between using a more standard temperature (e.g., 0.7 or 1) compared to 1.6?

A8: We didn’t compare temperatures in effects on the final performance, and we only compare temperatures in a preliminary stage by evaluating DeepSeekMath-7B-RL, as explained in A1 above.

Q9: In Line 253, the authors state: "We hypothesize that this is due to these models’ extensive pretraining on mathematical content." Do the authors have more points to substantiate this?

A9: The hypothesis stems from the understanding that extended large-scale pretraining can reduce the need for meticulously curated SFT datasets. However, we did not rigorously verify this hypothesis, and we agree with the reviewer that "a slightly weaker or similar model being used to generate synthetic data" could be the reason as well.

Q10: the hypothesis: "This pretraining likely covers most skills that could be learned from the GSM8K and MATH training queries" may not be correct, since, at least for Llama2-70B, the model capacity should not be a bottleneck to achieving higher scores

A10: This is an interesting point. To clarify, we didn't mean model capacity is a bottleneck, instead, we meant the training queries are the bottleneck -- the queries in DART-Math are simply the original GSM8K & MATH training queries that limit the scope that the final dataset can generalize to. Therefore, the generalization accuracy could be bottlenecked by the training query scope no matter how we improve the synthetic responses. Our hypothesis is that math-specific pretraining may already cover most skills that could be learned from these queries, and further training with those queries could not provide significant values. A potential future direction could be synthesizing new queries.