MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

Q: This paper has limited novelty, because it seems that it mainly combines all the math datasets together, and use GPT-4 to label the dataset, and then fine tune Llama with the new labels. This is a fairly standard pipeline, and it seems that the main improvement comes from the intelligence of GPT-4.

A: Our paper focuses on showcasing the potential of open-source models like Llama to become proficient in mathematical problem-solving, demonstrating a significant leap in their capabilities without the necessity for intensive continuous training. Prior to our study, there was a notable 'math generalization ability' gap between Llama/Falcon and GPT-4, and no comprehensive empirical research had been conducted to explore the extent to which this gap could be bridged with a manageable amount of training. Our research has revealed the latent potential of Llama, illustrating that our instruction-tuned Llama model is quite adept at generalizing to unseen math datasets such as MMLU-Math, SAT, and others. This finding is crucial for the open-source community to understand the competitive edge of GPT-4 and to develop open models that can rival it.

While Supervised Fine-Tuning (SFT) is a standard process, the distinctiveness of our work lies in the specific approach we took to construct the SFT data. We identified an effective recipe that mixes PoT and CoT data from diverse sources. It is an innovative method of data curation that significantly influences the performance of the model, marking a core innovation in our work.

Following our study, there have been other recent significant works, such as the application of MathInstruct on Mistral-7B, which further bridge the gap. This progress demonstrates the importance and impact of our contribution, particularly for those in the open-source community striving to understand and challenge the capabilities of models like GPT-4

Q: Moreover, the idea of hybrid instruction tuning is a bit confusing. According to Sec 2.4, the authors will first run PoT, and if the program cannot execute, they will switch to CoT. It seems to be a very preliminary way to combine CoT and PoT together. I was thinking a better way could be interleaving CoT and PoT in the solution.

A: There might be some confusion here. Hybrid instruction tuning refers to the training of the LM using PoT and CoT rationale. The hybrid decoding (first PoT, if fails, then CoT) aims to combine the accuracy of PoT’s accuracy and CoT’s coverage. We have made other attempts like “letting LM choose between CoT and PoT by itself” but the performance is not as satisfactory as “hybrid decoding”. We found that this approach despite its simplicity achieves the best accuracy in practice. Regarding “PoT + CoT interleave”, our current implementation already has a bit of this. For example, the PoT rationale actually starts with a line of comments to explain the solving process, which can be essentially seen as CoT. In multi-choice questions, we will execute the PoT to get the intermediate output and then call CoT to derive the final option. We admit that the current interleaving form is not flexible enough and we will continue to work on this to build a more cohesive decoding approach to combine CoT and PoT. Besides, we would like to point out we tried other decoding methods like Self-consistency. See more newly added results in the response to Reviewer z3BX.

Q: Overall speaking, I think the main contribution of this paper is "using GPT-4 to create a new dataset (which is a combination of many existing datasets), and fine-tune Llama using the created dataset". Therefore, I will say this paper has limited significance. I give weak accept mainly because I feel this is an important problem, and the authors provide a reasonably good solution.

A: Thank you for your valuable feedback. In contemporary research, the composition of a dataset, often referred to as the "data recipe," is increasingly recognized as a pivotal factor in enhancing model performance. Prior to our work with MathInstruct, the understanding of how to effectively construct a dataset to augment mathematical reasoning capabilities was limited. Our unique combination of data in this study marks a significant advancement in this area. We contend that our paper sheds light on the untapped potential of open models in addressing complex mathematical problems. The insights provided in our research have already stimulated further exploration within the academic community, aiming to develop more robust Language Learning Models (LLMs). Thus, we believe the contributions of our paper extend beyond its immediate findings, fostering a deeper understanding and exploration of advanced LLMs.

W2&3: Absence of an In-Depth Analysis of Training Data Distribution and Insufficient Ablation Studies on Training Dataset Influence.

A: We appreciate your constructive feedback and agree that a deeper analysis of training data distribution and influence is essential for a comprehensive understanding of our model’s performance. To respond, we added additional 9 model variants trained on specific subsets and combinations in Table 5:

Our key findings are summarized as follows:

• When the model is trained on individual datasets (GSM8K, MATH, or AQuA), it demonstrates strong performance in tasks closely related to the respective training dataset. However, it generally struggles to generalize this performance to other tasks.
• The datasets GSM8K, MATH, and AQuA emerge as crucial components in the training process. Training the model on a combination of these datasets leads to a marked improvement in its overall performance, indicative of the complementary nature of these datasets. Omitting any one of these datasets results in a noticeable decline in overall performance.
• TheoremQA, despite its smaller scale, exerts some influence on the model's performance. It tends to enhance the model's capabilities on datasets like Mathematics and SimulEq, yet it may slightly impede performance on GSM8K and MATH. Nevertheless, its inclusion in the training regimen is favored, as it introduces a broader variety of questions and could potentially aid the model in generalizing to other complex problems that require advanced computational skills.
• Incorporating a wider range of datasets into the training consistently improves the model's performance across most tasks. This suggests that a broad and diverse knowledge base is instrumental in enhancing the model's ability to generalize.

We have included these findings in the revision.

W1: Limited Technical Novelty.

A: Thank you for your question on the novelty of our work. First, we would like to highlight the fact that we did compare MAmmoTH directly with Orca-Platypus (a stronger variant of the original Orca) in Table 3 and 4. As can be seen from the results, MAmmoTH significantly outperform it on both in-domain and OOD settings. This comparison is essential in establishing the superiority of MAmmoTH. The results clearly show that MAmmoTH outperforms Orca in handling a variety of mathematical reasoning problems, underscoring the advancements our dataset MathInstruct and model MAmmoTH bring to the field.

Besides, we would like to highlight our contributions and novelty again:

• Hybrid Rationale Integration: A key innovation in MAmmoTH is the integration of both Chain-of-Thought (CoT) and Program-of-Thought (PoT) rationales, a feature not prevalent in models like Orca. This integration allows for flexible problem-solving strategies, enhancing computational precision and abstract reasoning capabilities in a single model​​.
• Extensive and Diverse Dataset Curation: MAmmoTH leverages the MathInstruct dataset, a comprehensive and diverse compilation of mathematical problems and rationales. This dataset covers a broader range of complexity levels and mathematical fields than typically seen, addressing a critical gap in existing math datasets and contributing to the uniqueness of our work​​.
• Superior Performance on Comprehensive Benchmarks: Our evaluations, as shown in Tables 3 and 4, demonstrate that MAmmoTH significantly outperforms advanced baselines like Orca-Platypus on both in-domain and out-of-domain datasets. This highlights MAmmoTH’s enhanced capabilities and generalization across various mathematical challenges​​.
• Detailed Ablation Studies: The ablation studies we conducted offer insights into the impact of different training subsets and model architectures, providing a deeper understanding of how various aspects of training data influence model performance. This level of analysis adds to the scientific rigor and novelty of our work​​. We hope this response adequately addresses your concerns regarding the technical novelty of our work.

W4: Lack of Comprehensive Error Analysis

A: Thank you for your insightful feedback regarding the need for a comprehensive error analysis of the MAmmoTH model. We acknowledge the importance of this aspect and have taken steps to enhance our evaluation by including an error analysis. We meticulously categorizes the types of 40 errors encountered by MAmmoTH on GSM8K and MATH, providing a clearer understanding of its current limitations and areas for improvement. The errors are classified into several distinct categories: Language Understanding Errors: These occur when the model fails to accurately comprehend the question or the context, leading to misinterpretation of the task at hand. Grounding Errors: Such errors arise when the model incorrectly translates quantitative data or percentages (e.g., interpreting 150% as “2 times”). Logic Errors: This category encompasses errors where the model's reasoning is flawed. Program Errors: These are errors where the model generate non-exectuable code due to the reasons like missing libraries or dependencies. See the image in https://imgur.com/d1imghL for breakdown.

By classifying these errors, we aim to provide a clear picture of where MAmmoTH excels and where it requires further refinement. We also included specific examples of errors that MAmmoTH in the Appendix for better illustration. These examples are chosen to represent a range of issues.

Q1: How much of the training data would be filtered after the validation?

A: We list the number of valid samples after filtering. AQuA-PoT (75%, 9772 out of 12881), MATH-PoT (51%, 7088 out of 14000), TheoremQA-PoT (44%, 703 out of 1600), TheoremQA-CoT (37%, 592 out of 1600), GSM8K (81%, 14591 out of 18000). We have included these statistics in the Appendix Table 6.

Q2: Whether the generated annotation converted from TheoremQA can be successfully executed?

A: As we answered in the last question, around 44% of TheoremQA questions were successfully executed and matched the ground truth answer.

Q: Even though MathInsutruct is already better than its competitors at generalizing, it can still remain a problem for even harder or unseen math problems.

A: We appreciate your comment on the limitations. As you rightly pointed out, and as we acknowledged in the Limitations section in the Appendix of our paper, despite the advancements brought by MathInstruct, the models might still encounter challenges with extremely complex or entirely unseen math problems. This is a common limitation when doing SFT. However, our study with MAmmoTH models demonstrates promising results in terms of performance on out-of-distribution (OOD) testing, suggesting potential pathways for enhancing model generalizability on unseen problems for future work:

Broad Coverage and Diversity in Instruction Datasets: When constructing instruction datasets like MathInstruct, ensuring diversity is crucial. This involves covering a broad range of mathematical problems from different fields and complexity levels. Such diversity helps in training models that are better equipped to handle a wide spectrum of mathematical challenges, enhancing their generalization capabilities.

Hybrid Solution Types: Incorporating a mix of different types of solutions, such as a combination of program-of-thought (PoT) and chain-of-thought (CoT) rationales, is more effective. This hybrid approach provides the models with varied perspectives and methods of problem-solving, which is particularly beneficial for tackling complex and novel mathematical problems.

Q: Is there a baseline that just finetunes all of the 13 datasets into 1 model? I think this would give a better idea of how to access MathInstruct. If not that's fine.

A: Before our study, there was no existing baseline model that had been fine-tuned on this specific collection of datasets. The MathInstruct dataset, as introduced in our paper, is an original contribution and was specifically compiled for this research.

MathInstruct is a meticulously curated dataset, comprised of 13 math datasets, including six that were newly curated by our team. This unique assembly of datasets is a novel contribution to the field of mathematical problem-solving with LLMs. The exceptional performance of our MAmmoTH models can be largely attributed to the high quality of MathInstruct. We believe that our approach of creating diverse, comprehensive datasets and incorporating a mix of solution types offers a promising direction for future research in this field.

W1: It would be useful to have numbers for some of the closed models (that have APIs) for NumGlue, Mathematics, SimulEq and MMLU-Math. I realize this is a slightly annoying request as it incurs monetary costs and the scientific value of comparing to systems with unknown training distributions and architectures is dubious, but knowing which of these datasets are the most challenging for widely-used "flagship" models still has value as a heuristic for contextualizing the contribution.

A: We have now added GPT-4's performance data on NumGlue, Mathematics, SimulEq, and MMLU-Math, enriching our results for a more comprehensive analysis. This additional data not only sheds light on the difficulty level of each dataset but also positions our open-source models’ performance (including ours) in relation to the well-regarded, proprietary model. Notably, while there remains a performance disparity between open-source models and GPT-4, it's encouraging to note that both MAmmoTH and MAmmoTH-Coder exhibit competitive, and in some cases superior, results on specific datasets (e.g., scoring 43.6 vs. 42.5 on MATH, 65.4 vs 60.8 on DeepMind Mathematics (Mat), and 74.4 vs. 74.7 on NumGLUE). We are hopeful that our investment in presenting GPT-4's results, though financially substantial, will significantly contribute to the research community's understanding of the progress in proprietary models for mathematical reasoning tasks and offer a well-rounded perspective on our contributions.

W2: The authors only experiment with one approach to hybrid prompting - it seems like a number of approaches could be viable, e.g. self-consistency across samples from both CoT+PoT, or letting the model pick which mode to operate in. If other approaches were tried but weren't effective, it would be good to see results (or at least a remark) indicating what was tried and justifying the chosen approach as the best one empirically.

A: Thanks for bringing this up. We indeed have experimented with different prompting methods. For example, we tried “letting the model decide which prompting method (PoT vs. CoT) to use” and found that the performance was not quite satisfying. In contrast, our hybrid decoding (“first PoT, if fail, then CoT”) despite its simplicity leads to better performance in practice. We are happy to put more of our exploration results to justify our choice.

Conceptually, our “hybrid decoding” is a special case of self-consistency decoding, where the ensemble size equals 2 (1 PoT + 1 CoT). We further tested other more expensive self-consistency variants like 10 PoT, 5 PoT + 5 CoT, 20 PoT, and 10 PoT + 10 CoT to see how far we could reach. We lay out some additional self-consistency experimental results on GSM and MATH for MAmmoTH-Coder-7B in the Appendix :

From our experiments, we show that self-consistency decoding can further improve performance. For the 7B MAmmoTH-coder, it seems that having a “PoT ensemble” already significantly improves the results to address the “non-executable issue“.

W1&2: I actually train a 7B-CodeLLaMA with GSM8K0PoT training set prompted from GPT-3.5-turbo myself, the performance can definitely achieve 62% (>59.4 in Table 3). I'm not sure what would be the quality of this dataset. I think the author should have more experiments to justify the datasets. For example, GSM8K PoT, we need to compare with a CodeLLaMA that trained on GSM8K PoT training set.

A: Thank you for your insightful observations and queries regarding the performance of CodeLlama - 7B on GSM8K. In response to your query, we have added additional experimental results in the Appendix Table 7:

From the results, it is evident that training the CodeLlama 7B on the individual GSM8K dataset can indeed yield better in-domain performance compared with training on MathInstruct (63.2 vs. 59.4). This performance aligns with your observations from your own training experiments, and we appreciate you providing that as a baseline for comparison.

However, we also observe that training exclusively on a single dataset like GSM8K leads to challenges in generalizing to other datasets. The average performance of the CodeLlama - 7B model is significantly higher when trained on MathInstruct (52.0%) compared to being trained solely on GSM8K (35.2%). This demonstrates a substantial improvement in the model's generalization capability when trained on the diverse and comprehensive MathInstruct dataset. We report similar results in Table 5 based on the Llama-2 7B model.

These additional results provide strong evidence for the quality and effectiveness of the MathInstruct dataset in enhancing the performance of models across a range of mathematical reasoning tasks. While dataset-specific training can be effective for targeted tasks, the diverse and comprehensive nature of MathInstruct makes it a valuable resource for developing models with broad generalization capabilities in the domain of mathematical problem-solving.

Q1: GSM8K has about 7k training data, how do you get 14k examples?

A: Regarding your question about how we obtained 14k examples from GSM8K, which originally has about 7k training data: For each GSM8K question, we prompted GPT-4 to generate up to 3 PoT solutions. We further enhanced the dataset quality by meticulously removing near-duplicated solutions, ensuring a diverse range of unique problem-solving approaches. These GPT-4 synthesized programs were then rigorously vetted by comparing their executed results with human-annotated ground truth, thereby ensuring the high quality and reliability of the added rationales. We’ve added these clarifications in the revision.

We are grateful for your valuable feedback! Should there be any further queries or discussions needed, we are more than willing to engage and provide additional clarifications.