Query and Response Augmentation Cannot Help Out-of-domain Math Reasoning Generalization

Thanks for your insightful comment.

W1: We have fixed the typo in the Figure 1 caption.

W2: We have perturbed two new test sets based on the original GSM8K test set.

(A) Change-Test, is created by altering the numerical values in the GSM8K test set questions and correspondingly modifying the answers. There are 1211 query-response pairs in the Change-Test.

(B) Aug-Test, is generated by augmenting the test set in the same manner as we did for the training set. There are 1378 query-response pairs in the Aug-Test.

Upon evaluating our model on these two perturbed test sets, we found that the performance of MuggleMath consistently and significantly exceeds that of the model fine-tuned on GSM8K alone. This observation suggests that our data augmentation techniques not only enhance the model’s ability to solve the original problems but also contribute to its improved performance on varied and perturbed inputs, thereby indicating a robust generalization capability. The results are added in in Appendix H.

Thanks for your insightful comment.

W1: We have included additional comparisons with a broader range of state-of-the-art approaches in Appendix H. Part of comparisons are list here.

Further, We conduct an extended experiments on larger datasets($\mathcal{D}$ +($\sum_{i=1}^{6}{\mathcal{D}_1^i}+\mathcal{D}_1^8$)-majority voting+$\mathcal{D}_2^1+\mathcal{D}_3^1$) demonstrate to construct a new version of MuggleMath named MuggleMath-new, which achieves a better performance(70.2 for 7B size and 75.4 for 13B size on GSM8K).

W2: Regarding the impact of reasoning paths on performance, Section 3.3 of our paper examines the response effectiveness for the same problems.

We conclude that, compared to the zero-shots generation method, the response augmentation prompt we use plays a substantial role(+3.6 for LLaMA-7B, +3.7 for LLaMA-2-7B, +3.3 for LLaMA-2-13B), since the 1-shot setting stabilizes the response format. We find that, compared to GPT-3.5, the responses augmented using GPT-4 perform significantly better for SFT. Hence, the most effective way to generate the response is to use the best model and well-designed prompts.

W3: In response to the concern about the impact of data augmentation on model interpretability.

Our augmentation process involves creating variations of the original problems, which, as shown in Figure 1, results in the augmented queries from AugGSM8K having a similar distribution in the model’s latent space as the original GSM8K queries. Generally, in deep learning, more data from the same distribution tends to improve model performance, which is a key reason why models trained on AugGSM8K exhibit superior performance on the original GSM8K dataset.

In addition, we have conducted an analysis of our model’s performance on test set problems of varying difficulty. Our analysis of the 7B-2 model’s performance on the test set, shows accuracy rates of 0.55, 0.42, and 0.21 for easy, medium, and hard problems, respectively. The MuggleMath model, however, achieved higher accuracy rates of 0.73, 0.70, and 0.64 for the same problem categories.This significant performance boost on difficult questions can be attributed to the fact that the augmented problems we generated are generally more complex than the original problems.

W4: In response to the concern about computational efficiency of our methodology.

During the data generation phase, which includes the generation of queries and responses, the cost associated with using the GPT-3.5-turbo API is 0.002 dollars per 1k tokens, and for the GPT-4 API, the price is 0.03 dollars per 1k input tokens and 0.06 dollars per 1k output tokens. As shown in Table 1, the creation of the 10 versions of AugGSM8K resulted in a total expenditure of 4057.2 dollars. Specifically, in Table 4 for MuggleMath in Table 4, the cost for data generation amounted to 2508.9 dollars.

In the training phase, to train the models that achieved the results presented in Table 4 for MuggleMath, the computational resources utilized were as follows: the 7B, 7B-2, and 13B-2 models were trained on 8 NVIDIA A100 GPUs for 5.5 hours, 5.5 hours, and 13.5 hours, respectively. For the 70B-2 model, we trained on 32 NVIDIA A100 GPUs for 20 hours, which will be updated in the paper.

We appreciate the reviewer's review.

For W1, we acknowledge the work of Yu et al. 2023 “MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models”, which was indeed contemporaneous with our own research. Their paper was uploaded to arxiv on September 21, 2023, merely a day before our submission to ICLR. Consequently, our work was conducted independently and without the influence of Yu et al.'s findings. Our paper shares similar contributions with that of Yu et al. in terms of query and response enhancement methods and the analysis of scaling relationships. However, we have conducted further analysis on the generalizability of data augmentation techniques.

For W2, our paper’s title, “Query and Response Augmentation Cannot Help Out-of-Domain Math Reasoning Generalization,” is primarily based on our findings that applying query and response augmentation to the GSM8K dataset resulted in negligible and inconsistent performance improvements on the MATH dataset. There has been substantial research on Out-of-Domain (OOD) issues, and we will include a discussion of this literature in the related work section of our manuscript. Since data augmentation has demonstrated exceptional performance in large model math reasoning, but its generalizability has not been studied before, our paper provides a preliminary analysis of the generalizability of these methods. We believe this will offer some new insights to the community. To further investigate how the data augmentation performed on GSM8K affects different domains of mathematics, we conducted an additional analysis on the accuracy across various subsets of the MATH dataset.

For W3, the reasoning paths are sampled by GPT3.5 or GPT4, which is discussed in section 3.2 and section 4.1. The prompt used for reasoning path generation is listed in appendix C and a case study can be seen in Table 13.

For Q1 and Q2, while there is an overlap in the embedding space distribution between MATH and GSM8K, it is relatively small compared with that between GSM8K and AugGSM8K. In the transfer learning setting, training first on GSM8K and then on MATH with LLaMA-13B-2 does provide some benefits for certain subsets, such as Prealgebra, Algebra, and Geometry. However, if we train first on AugGSM8K and then on MATH, this benefit is not only marginal but may even lead to a decrease in performance on other subsets, like Geometry and Prealgebra, which could be related to the data proportions. Overall, the performance improvement on the MATH dataset from augmentation on GSM8K is minimal, even on subsets like Prealgebra, where there is some overlap. For 7B size and multi-task learning setting, we can draw the similar conclusion. Complete comparison will be listed in appendix H.

For Q3, in our approach to query augmentation, we took inspiration from the common pedagogical practice where human learners engage in “varied practice” training to enhance their mathematical problem-solving skills. Specifically, we employed several variation training methods commonly used in algebra to construct new problems based on the original ones.

We appreciate the reviewer’s comments. We respectfully disagree with the assessment regarding the novelty of our work and feel that the perceived overlap with “MetaMath” does not fairly represent our contributions, for the following reasons:

1. Our research was conducted in parallel with “MetaMath ...”, with our submission to ICLR taking place just one day after “MetaMath” was uploaded to arXiv, ensuring our work’s independence.

2. Our data augmentation methods in MuggleMath differ from MetaMath in nature and have demonstrated superior performance on GSM8K (MuggleMath-new-7B scored 70.2 vs. MetaMath-7B’s 66.5, and MuggleMath-new-13B scored 75.4 vs. MetaMath-13B’s 72.3). While MetaMath focuses on rewriting the question from multiple perspectives, our techniques not only alter numerical relationships but also introduce new logical content, enhancing the complexity and variety of problems.

3. Though initial, our analysis on the generalizability of data augmentation techniques provides some new insights to the community.

We appreciate the reviewer's review.

For W1, there is a trend in LLM community augmenting many more samples to race on math benchmark performance. As for the novelty and scientific contributions of our paper, we would like to highlight that our work builds on the advantages of previous works such as RFT (response augmentation) and WizardMath (query augmentation), which have been widely recognized and cited in the LLM community. Our study not only achieved the best performance among all open-source models on the GSM8K dataset but also provided an analysis of scaling laws and out-of-distribution generalization. This analysis can offer valuable insights to the community when using data augmentation methods to enhance models' mathematical reasoning abilities.

For W2, we have taken your advice and add more ML generalization paper to related works. Moreover, we have changed the conclusion to "apply augmentations on diverse math subjects".

For W3, we consider scaling laws during fine-tuning augmented dataset is useful for deliver a specific ability (like math reasoning or coding) to users. And another finding here is augmenting using model-generated samples and using human-written samples have similar scaling trend which is very interesting for who want to improve the model based on model itself.

For Q1, we use the 15-th layer of last token representation in the problem.