Exploring the Mystery of Influential Data for Mathematical Reasoning

Dear Reviewer gRx8,

Thank you for your thoughtful feedback! We are pleased to share our ideas to your concerns.

No clear motivation for math.

We emphasize our motivation below:

• Math reasoning task is a more challenging task compared to general tasks. We found that existing strategies do not perform well on reasoning tasks in paper "Figure 2", and thus focus on math reasoning.
• If there exists a right and comprehensive reasoning process in a certain sample, it should be positive to another sample. We'd like to use the positive influence of reasoning data, leading to the "one-shot inference". The theoretical support is the inherent duality between attention and gradient descent [1].

Also, we conduct experiments on two general tasks taking into account your question. Please see the table in our rebuttal for Reviewer YKSQ.

Whether different combinations could lead to improvements.

We perform ablation of the two aspects of QaDS in paper "Table 3", and we order it more clearly below:

Gains might be overshadowed by LLAMA 70B.

Firstly, we conduct experiments on LLaMA-2 70B considering your insightful comment:

Then, we also emphasize that we reach an improvement of 6.6% in paper "Figure 3" (48.8 vs 42.2) in S_{large}-4.7M.

[1] Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559 (2022).

Dear Reviewer YKSQ,

Thanks for your valuable comments of our paper! We are pleased to share our ideas on your questions.

Lack of further analysis of the selection strategy, e.g., more than one-shot.

We provide analysis from two aspects to your concern:

• Existing strategies mainly consider the data quality from some direct aspects such as input length, fluency, naturalness and so on, while not fit for reasoning. Therefore, we found that they do not perform well on reasoning tasks as shown in paper "Figure 2".
• The quality of resoning data is not easy to measure, and our solution is to measure the positive influence on each other: if there exists a right and comprehensive reasoning process in a certain sample, it should be positive to another sample, leading to the "one-shot inference". The theoretical support is the inherent duality between attention and gradient descent, and can be found in [1].

We are the first to explore data selection for reasoning, and we hope we can provide insights to the community. We will explore more strategies in our future work.

Could this strategy be applied to other tasks than the mathematical reasoning?

Yes! Although we focus on the challenging mathematical reasoning task, we also conduct experiments on two general tasks taking into account your question.

We first adopt 2000 samples of Alpaca and Dolly datasets as D_{p}, and 500 mixing samples of Alpaca and Dolly as D_{t} to perform one-shot inference. For the gap between reasoning and general tasks, we then train a new scorer for general tasks with above data. The scoring and selection process is the same as we mentioned in our paper. Finally, we fine-tune LLaMA-2 with our selected data (6.6K from 66K) by QaDS. Also, we implement IFD and Deita to select the same size 6.6K of data. We show results below:

It turns out that our QaDS also achieves the best results, showing the effectiveness of our strategy.

[1] Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559 (2022).

Dear Reviewer rr87,

Thank you for your comprehensive and meticulous review! We are pleased to address your points below.

It is not clear which dataset is referred to in Table 3.

We use 69K selected data from S_{base} in this table. A detailed composition can be found in paper "Appendix Table 6". We will add a clearer summary in future updates.

The explanation of the scoring method in Figure 1 is confusing.

The bi-directional arrows indicate the comparison between one-shot scores and corresponding zero-shot scores, showing the process of paper "Equation (3)". Also, we will polish the figure and make it clearer.

The paper does not adequately address the computational costs.

We take our S_{base}-69K setting as an example (from 428K all data to select 69K data). We will show our computational costs on LLaMA-2 (for there is no obvious gap between LLaMA-2 and Mistral) from two aspects:

• Diversity costs: It takes about 4.43 gpu hours to compute 428K data embeddings. The K-center Greedy process is implemented on CPU and there are no GPU costs.

• Quality costs: we only use a part of representative samples to perform one-shot inference, i.e. n1=2,000 and n2=100 in our paper "Appendix Section A". In this case, the computation of zero-shot and one-shot inference takes about 0.02 and 8.42 gpu hours per dataset. After that, we train a scorer and infer the 428K scores taking 2.00 and 3.32 gpu hours respectively.

We also compute the costs of another popular strategy Deita from our implement and their paper. Deita contains the process of obtaining embeddings, training quality and complexity scorers, and inferring scores. We show the gpu hours and overall accuracy in the table below.

It turns out that our costs are close to Deita. Moreover, once we finish the training of our scorer, the most expensive process, i.e. the one-shot inference, is not used. We can perform fast scoring on different reasoning tasks fully based on our scorer.

Dear Reviewer oJ7q,

Thank you for your thoughtful review! We are pleased to provide more information to your confusion.

Only the "General" is selected by QaDS.

Firstly, we do perform selection on both reasoning and general data in S_{base}, and we achieve the best results as shown in paper "Figure 2".

When coming to S_{large}, we aim to give some insights on a wide range of data sources:

• Scaling up reasoning data is helpful: In mathematical reasoning, a common law is that mathematical reasoning consistently improves with increasing data amount [1]. And our comparison between QaDS-S_{large}-4.3M and 2.6M in paper "Table 4" also validates it.
• General data can also be influential for reasoning: According to [1] and [2], other relevant tasks can also be helpful for reasoning. So we contain general data in our setting, and successfully selects the helpful general data by QaDS.

Therefore, we contain all reasoning data and a part of influential general data selected by QaDS as our OpenMathMix.

Concerning about what's really improving.

Our improvement comes from two parts:

• The direct improving mathematical ability by more mathematical reasoning data.
• The indirect improving reasoning-related ability by influential general data. We show cases in paper "Appendix Table 9 and 10".

Table 4 only shows the performance of one dataset.

We provide our reason in paper "Section Evaluation Datasets": we adopt a subset of MATH as D_{t} when computing quality scores in S_{large} , and thus choose the corresponding test set of MATH for evaluation. Also, taking into account your careful comment, we evaluate on other 6 tasks and achieve best results:

[1] How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492 (2023).

[2] Specialist or generalist? Instruction tuning for specific NLP tasks. arXiv preprint arXiv:2310.15326 (2023).