Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis from Scratch

We sincerely appreciate your valuable feedback on our work. Your insights have provided us with an opportunity to refine and improve our paper.

To Weakness 1: The main experiments in Table 1 are somehow not very fair.

Exactly, data volume plays a significant role in shaping model performance. Achieving complete fairness in the comparisons, however, is challenging due to certain practical constraints: some datasets, such as WizardMath, MathScale, and KPMath, are closed-source, which limited our access to an equivalent number of training samples.

To ensure fairness, we have taken the following steps:

• For open-source datasets, we plotted the scaling curve in Figure 1 (Right), illustrating the effectiveness of our approach given the same volume of training data.
• We further provide evaluation results on the same number of training samples (400K) from the public dataset, i.e., MetaMath, MMIQC, DART-Math, and NuminaMath, based on the Qwen2-Math-7B model. More details are shown in Appendix C of our revised version.

To weakness 2: I am wondering if their performance is similar on OOD test sets like GSM-hard, etc

We evaluate results on two other OOD benchmarks, including GSM-hard, MathChat.

Our model achieves comparable results to Qwen-Math-7B-Ins, demonstrating its generalization capability. More details can be seen in Appendix C of our revised version.

To weakness 3: why the authors choose models like Qwen2-Math-7B, etc

Compared to GPT-4o-mini, Qwen2-Math-7B offers lower costs and more stable and predictable generation times, which we consider crucial in large-scale data synthesis.

For instance, using GPT-4o-mini to perform solvability checks on 2 million generated questions is estimated to cost around $600, and managing such a large volume of API requests introduces additional uncertainty in time consumption. In contrast, using Qwen2.5-Math-7B requires only 110 GPU hours, offering a fixed processing time and significantly reduced costs (~ $142.7).

To Question 1: Typo: Filering -> Filtering in line 215

Thank you for your feedback. We have carefully reviewed and corrected the typo in our manuscript.

To Question 2: In Figure 5, it seems that QPO is less effective, etc

We would like to correct the misunderstanding that the purpose of QPO is solely to enhance the final SFT model performance. A key objective is actually to improve the efficiency of data generation. As shown in Figure 5, after applying QPO, the solvability of the generated questions increased by 8.2%, a significant improvement, indicating that the sample utilization rate for solvable questions is correspondingly higher.

While the effect may appear minimal due to subsequent solvability filtering, our detailed analysis shows that 28.8% of unsolvable questions were filtered out in the baseline setting, whereas after QPO, only 19.4% were deemed unsolvable. This represents a 9.4% reduction in computational overhead.

To Question 3: about the effectiveness of Solvability Filtering and Difficulty sampling, etc

We agree that some of the generated questions are not perfect in terms of solvability, which can be attributed to the model’s tendency—driven by hallucination—to attempt solving problems rather than assessing their solvability.

As for difficulty sampling, inspired by DART-Math’s insight that challenging questions can drive more effective learning, we empirically fit specific difficulty distributions by observing the patterns in difficulty distribution.

To Question 1: How did the authors select the base difficulty filtering model for fine-tuning (Lines 222-239) and the reward filtering model (Lines 251-252)? etc

• Selection of difficulty filtering model: The selection was inspired by DART-Math, which uses the accuracy of DeepSeek-7B-RL on a given question as a measure of its difficulty. We experimented with different models for training (DSMath-7B-Base and DSMath-7B-RL) and found that the results were similar.
• Selection of reward filtering model: This choice was primarily guided by the model’s performance on the reasoning subset of the Reward Bench.

Thank you for your suggestion! We have updated the discussion in the revised version.

To Question 2: In Table 1, the term "Synthesis Model" in the header needs clarification, etc

Sorry for the confusion. Allow us to clarify:

• DART-Math only includes response generation using DSMath-7B-RL. Other baselines use different synthesis models for both question synthesis and response generation, such as GPT3.5, GPT-4, and GPT-4o.
• For our approach, DSMath-7B-QGen and Qwen2-Math-7B-QGen are utilized for question synthesis, with Qwen2-Math-7B-Ins used for response generation.

If multiple models are used, only the most recently released one is marked. Additional details regarding the complete synthesis models for these datasets are provided in Figure 6. This clarification has been updated in the caption of our revised version.

To Question 3: The left bar chart in Figure 5 has a confusing y-axis, etc

Thank you for your suggestion. We have updated the chart with clearer explanations for the solvable ratio and difficulty score. The difficulty score is indeed calculated based on the method described in Lines 377-406, and we have clarified this in the revised version.

To Question 4: a rigorous human evaluation on a random subset would better demonstrate ScaleQuest’s quality, etc

Thank you for pointing this out. Human evaluation is indeed a more direct approach to demonstrate the quality of ScaleQuest.

We sampled 100 examples each from NuminaMath, and ScaleQuest and evaluated them based on clarity, reasonableness, and real-world relevance, with scores ranging from [1, 5]. (Please understand that due to the complexity of mathematical tasks, we limited the sample size to 40.)

• In terms of clarity and reasonableness, our synthetic data surpasses NuminaMath but still falls short of the high-quality, real-world datasets like the training sets of GSM8K and MATH.
• Regarding real-world relevance, GSM8K leans toward practical, real-life scenarios, while MATH focuses more on theoretical mathematical derivations. Our generated data can be seen as a balance between the two.

More details have been updated in Appendix C.

To Weakness 1: This reliance on fine-tuned, domain-specific models, while effective in the tested domain, makes it challenging to adapt ScaleQuest to broader applications, potentially limiting its utility as a general-purpose data synthesis method.

We sincerely apologize for any misunderstandings and would like to clarify that our data generation method in this work is tailored to reasoning tasks, where there is a significant scarcity of high-quality data.

Many general-purpose data generation methods currently fall short in reasoning tasks such as mathematics and code [1, 2]. Reasoning has already become a crucial component of data synthesis [3], with numerous studies dedicated specifically to this area.

Apart from Mathematical Reasoning, we extended our method to the Code Reasoning task as a simple validation, and the results are listed in our response to weakness 2. By keeping the answer generation model and data volume identical, our dataset outperformed the currently popular code dataset, CodeFeedback-Filtered [4]. More details can be seen in Appendix B of the revised version.

[1] https://arxiv.org/abs/2406.08464

[2] https://arxiv.org/abs/2312.15685

[3] https://arxiv.org/abs/2404.07503

[4] https://arxiv.org/abs/2402.14658

To Weakness 2: the paper appears to make some overclaims regarding its scope and efficiency, etc

Thank you for your valuable feedback. In response, we included another important task, code reasoning, as a simple validation, and the results are as follows:

With the same amount of data, our approach outperforms the widely used dataset CodeFeedback-Filtered. Additionally, we augmented the responses for the problems in CodeFeedback-Filtered using the same response generation process as our method, creating a new dataset, CodeFeedback-Aug. The results demonstrate that our approach still achieves superior performance, highlighting the higher quality of the questions generated by our method.

More details are shown in Appendix B in our revised version.

For the QPO stage, smaller models exactly struggle to follow instructions effectively to optimize given questions (as demonstrated in Figure 4, where Qwen2-Math-7B-Ins performs poorly on this task). However, the data cost for QPO is minimal, requiring only 10K examples. If human-annotated data were available, it would further enhance the process. GPT-4o-mini was used as a substitute in the absence of available data. Additionally, we experimented with open-sourced Llama3.1-70B-Ins and found that it outperforms GPT-4o-mini in this task.

GPT-4o-mini (solvable ratio): 72.2 -> 83.7

Llama3.1-70B-Ins (solvable ratio): 72.2 -> 86.7

To Weakness 3: effectiveness of our method, etc

We would like to clarify a misunderstanding: our model is not fine-tuned on Qwen2-Math-7B-Ins but rather on the base model, Qwen2-Math-7B-Base.

Regarding the effectiveness of our method, we would like to emphasize two key points:

• Qwen2-Math-7B-Ins demonstrates strong performance on mathematical tasks; however, none of its training data has been made publicly available. This means that while the model itself can be used, the underlying data cannot be leveraged for further developments or customized, tailored applications.
• In contrast, our dataset, generation method, and implementation details are fully open-sourced. While Qwen2-Math-7B-Ins serves as a teacher model and can be viewed as an "upper bound", our work achieves comparable performance and even surpasses it on certain tasks, further demonstrating the effectiveness of our approach.

The reviewer appreciates the authors’ efforts, particularly the inclusion of additional experiments on code reasoning and human evaluation, which effectively address most of my concerns. Based on these improvements, I have raised my score from 3 to 6 to reflect the enhanced quality of the work. However, I am open to deferring to the consensus of the other reviewers regarding the final decision.

To Question 1: The authors should compare different base models in Figure 5 and Table 2.

Please see our response to weakness 3.

To Question 2: The experimental setup in the experimental module should be clearly presented, etc

We apologize for the confusion caused by our unclear descriptions. We have carefully reviewed and clarified the setup for each experiment in our revised version.

• For Table 2, we ensured that the response generation process was consistent.
• For Figure 5, we added the explanation of the solvable ratio and difficulty score.

To Question 3: The authors might discuss the effects of optimizing different question data volumes, etc

We have further explored the impact of varying training data volumes on QPO. Using Qwen2-Math-7B-Ins as an example, we conducted experiments with 5K, 10K, 15K, 20K, and 40K samples. The results are presented below, and we discuss them in detail in Appendix C of our revised version.

To Question 4: The author should probably compare the generated questions with the questions in the test set (n-grams or other methods) to prevent potential data leakage.

We appreciate the reviewer's concern about potential data leakage. To address this, we have conducted an n-gram similarity analysis between the generated questions and all test sets from both our dataset and other baseline datasets. Based on prior empirical analysis [1, 2], we set n=13 to prevent spurious collisions and calculated how much the test sets overlap with training data to assess data contamination. The table below illustrates the clean ratio across our dataset and baseline datasets, defined as the percentage of test samples containing no n-gram matches with the training set. The experiments and analysis have been updated in Appendix A of our revised version.

The results demonstrate that our dataset achieves a relatively high level of data cleanliness compared to other datasets, suggesting that our method generates novel questions instead of memorizing existing ones.

[1] https://arxiv.org/abs/2005.14165

[2] https://arxiv.org/abs/2109.01652

To Weakness 1: The proposed Question Preference Optimization (QPO) appears to be less effective; as shown in Figure 5, the difference between QPO and QFT is minimal, raising questions about the validity of QPO.

We agree that the impact of QPO on the final SFT model’s performance is limited. Allow us to provide further clarification:

• Regarding the limited SFT performance: QPO specifically targets question generation and does not directly affect the response generation process, which explains the relatively modest performance gains. However, our results demonstrate that QPO is effective, achieving consistent improvements in the solvability and difficulty of generated questions, as well as in the overall SFT effectiveness.
• Beyond SFT performance: Though the impact of QPO on SFT may be minimal, it significantly enhances the data generation efficiency. Specifically, QPO improves the solvability of generated questions from 75.4% to 83.6%, a meaningful enhancement that boosts the efficiency of data utilization. While the effect may appear minimal due to subsequent solvability filtering, our detailed analysis shows that 28.8% of unsolvable questions were filtered out in the baseline setting, whereas after QPO, only 19.4% were deemed unsolvable. This represents a 9.4% reduction in computational overhead.

To Weakness 2: the authors should conduct a more fair and detailed comparison, etc

Thank you for pointing out this unfair setting. To ensure a fair comparison and demonstrate the effectiveness of QFT, we controlled the question generator to be consistent (using DSMath-7B-RL and Qwen2-Math-7B-Ins, each generating 1M data points, i.e., 2M in total for Magpie). We then applied language, solvability, and difficulty filtering, followed by response generation and reward filtering. This process resulted in approximately 1.2M data points for SFT. The results (based on Qwen2-Math-7B) are as follows:

The corresponding results have been updated in the revised version (Figure 1 and Figure 5).

To Weakness 3: The ablation experiments are insufficient, etc

Thank you for your feedback, and we sincerely apologize for any shortcomings in our ablation experiments. We have added Figure 5 to include results using Qwen2-Math-7B as a base model, providing a broader comparison.

We hope you understand that the SFT process is quite time-consuming for us. We will try to complete the experiments for the remaining two base models and include the results before the end of rebuttal period.

To Weakness 4: with advancements in open-source models, previous sample-driven and knowledge-driven question synthesis models can also be replaced with open-source models, etc

Exactly, more advanced models tend to generate better questions and answers. Here are our key observations:

Limited effectiveness of open-source models in question sample/knowledge-driven approaches: Some math-specialized models, such as Qwen2-Math-7B, are not well-suited for sample-driven or knowledge-driven approaches. These methods demand the model to generate valuable questions under complex constraints (e.g., specific topics or knowledge points), a task that these problem-solving models often struggle with. For example, when we used Qwen2-Math-7B-Ins to optimize given questions, the quality of the generated questions even declined (as shown in the top plot of Figure 4). Similarly, general-purpose models like Llama3-8B also faced challenges in producing high-quality questions due to their lack of mathematical specialization.

This may be why many sample-driven and knowledge-driven approaches rely on highly aligned closed-source models like GPT-4. This also highlights the unique value of ScaleQuest in enabling open-source models to overcome these limitations.

Efficiency advantages of our approach: Our question generation process requires only a minimal number of tokens (e.g., a BOS token), and the generated content directly serves as the final question without any redundant steps. In contrast, sample-driven or knowledge-driven methods often rely on heavily constrained prompts and multiple rounds of verification. This will result in significantly higher consumption of input and output tokens, leading to greater computational overhead, regardless of the model used.

To Weakness 4: There are many components in the data synthesis pipeline, but the impact of each component is not clear, etc

We would like to clarify the purpose of our ablation study, which is to demonstrate the effectiveness of each submethod in the pipeline. Our ablation results, as shown in Figure 5, include evaluations with solvability and difficulty filtering, highlighting their contributions.

More analysis has been updated in Appendix C.

To Question: There are plenty of LLMs used in the data synthesis pipeline: DeepSeekMath-7B-RL , Qwen2-Math-7B-Instruct, GPT-4o-mini, GPT-4o, DeepseekMath-7B-Base, InternLM2-7B-Reward. Can you provide a Table for all the settings? Is there any specific reason to select different LLMs for different stages?

To Weakness 1: The main weakness of this paper is, that the proposed data synthesis pipeline is too complex and may be domain-specific, etc

Thank you for your feedback. We would like to provide more information about reasoning data synthesis:

• General-purpose methods struggle with reasoning tasks: Most general-purpose data generation methods face significant limitations when applied to reasoning tasks, as mentioned in works like Magpie [1].
• Reasoning data synthesis requires a more complex process to ensure high-quality data: Unlike general tasks, reasoning tasks demand higher data quality, which necessitates more sophisticated pipelines like question rephrasing [1], sub-topic extraction[2, 3], verification [1], and quality assessment [3]. Compared to previous works, our approach is more straightforward, only containing QFT, QPO, and filtering process.
• Efforts in other reasoning tasks: We extended our method to the Code Reasoning task as a simple validation, and the results are shown in the figure below. By keeping the answer generation model and data volume identical, our dataset outperformed the currently popular code dataset, CodeFeedback-Filtered [4]. More details can be seen in Appendix B of the revised version.

[1] https://arxiv.org/abs/2309.12284

[2] https://arxiv.org/abs/2403.02333

[3] https://arxiv.org/abs/2403.02884

[4] https://arxiv.org/abs/2402.14658

To Weakness 2: The proposed data synthesis method is only evaluated in the math domain.

Thank you for your valuable feedback. To address this, we made minor adjustments to the ScaleQuest method to adapt to code reasoning tasks (details in Appendix B). Our experiments demonstrate that the resulting dataset achieves higher quality compared to the open-source CodeFeedback dataset. The results are provided above (see response in Weakness 1), with additional details included in Appendix B.

To Weakness 3.1: For the baselines with different synthetic datasets, are they finetuned on the same scale of training examples?

We apologize for the confusion. The results in Table 1 do not strictly control for identical training data volumes due to practical constraints (e.g., some datasets are not publicly available).

To ensure a fair comparison, we made the following efforts:

• For open-source datasets, we plotted the scaling curve in Figure 1 (Right), which demonstrates the effectiveness of our approach with the same volume of training data.
• Additionally, we provide evaluation results using Qwen2-Math-7B fine-tuned on 400K training samples drawn from public datasets (MetaMath, DART-Math, and NuminaMath). The new results are updated in Appendix C.

To Weakness 3.2: What does the Percentage and Accuracy in Figure 5 mean? Where is the legend of the left plot of Figure 5?

We are sorry for the confusion,

• Percentage: In the solvability subplot, it indicates the proportion of generated questions judged as solvable. In the difficulty subplot, it represents the average difficulty score of the generated questions.
• Accuracy: This metric evaluates the impact of the synthesized dataset on model performance, specifically the fine-tuned model’s accuracy on the test dataset. Detailed explanations can be found in Section 3.1 under Evaluation and Metrics.

We have clarified these definitions in the figure caption of our revised version.

To Weakness 3.3: What does the question quality in Table 2 refer to?

We are sorry for the confusion. "question quality" in Table 2 refers to instruction tuning effectiveness.

The purpose of Table 2 is to compare the instruction tuning effectiveness of questions from different open-source datasets. To ensure a fair comparison, we kept the answer generation process entirely identical.

Thanks for the quick feedback. We would like to clarify some unclear points in our first round of the response. We hope the newly attached response and experiments mitigate some of your concerns.

To Concern 1: The adaptation capability of the proposed method to a new domain is cost-consuming and ineffective.

We apologize for any confusion regarding the settings. To address the concern about effectiveness in code reasoning task, we would like to provide further clarification based on the experimental results for code reasoning:

• CodeFeedback-Raw refers to the public version, which consists of a condensed collection of problems from various open-source datasets. It is one of the most frequently used datasets for code reasoning. Our method demonstrates significant improvements over this dataset (5.3% in average accuracy).
• CodeFeedback-Aug: To ensure a fair comparison, we applied the same reward filtering method to enhance the responses of existing problems, resulting in CodeFeedback-Aug. The quality of the problems synthesized by our approach is comparable to, or even surpasses, that of CodeFeedback itself, which includes both real-world questions and high-quality synthetic questions. This result further highlights the high quality of the questions generated by our method.

Regarding the concern about cost-effectiveness, we would like to provide the following clarifications:

• The training cost for the question generator is minimal, requiring only around 20 GPU hours (approximately 2.5 hours on an 8 A100-40G-PCIe server), which constitutes just 3% of the total data synthesis cost, as shown in Table 4. While using a single question generator is entirely feasible, employing multiple generators yields better results, as discussed in Multiple question generators enhance data diversity in Section 3.3. We also simplify our approach and the results can be seen in our following response to concern 2.