GPT Can Solve Mathematical Problems Without a Calculator

We sincerely thank you for your valuable comments on our paper. We will explain your concerns point by point.

Weakness 1: While specializing for arithmetic is beneficial, it could compromise more general capabilities. Testing on broader math/reasoning tasks could help characterize tradeoffs.

Response: In our study, we leverage the continue training strategy to address the limitation of fine-tuning that damages the generic ability of the MathGLM.

Here, we report the generic ability of the MathGLM on Z-bench. To evaluate the quality of responses generated by the model, we score responses with the powerful GPT-4 on a scale of 1-10 based on factors such as correctness (high priority), helpfulness, relevance, depth, innovativeness, and level of detail.

Compared to GLM-10B, MathGLM-10B shows a significant improvement in handling math tasks, with an accuracy of 56.6% compared to GLM's 0%. However, it is also noticeable that there's a slight reduction in the performance on text tasks for MathGLM-10B (5.35) compared to the original GLM (5.91). While MathGLM-10B excels in mathematics, its ability to handle general text tasks is marginally impacted. However, this reduction is not drastic, indicating that the continue training using a hybrid dataset, which includes both mathematical and general text data, helps in retaining the model's overall linguistic competence.

Weakness 2: More analysis and examples demonstrating the step-by-step generation process could be useful to understand MathGLM's learned skills.

Response: We have included the impact of the step-by-step generation process in the Appendix, particularly in Section D.6. By leveraging the step-by-step generation process, the accuracy on MathGLM-500M rises from 31.96% to 89.57%, while for MathGLM-2B, it increases from 40.76% to 93.03%. This suggests that the step-by-step process is highly effective in enhancing the model's performance. This could be due to the process allowing the model to break down complex problems into smaller, more manageable steps, thereby reducing errors that can accumulate in a single-step solution.

Weakness 3: The reasoning behind curriculum learning's benefits is not fully fleshed out. Is it mainly about efficiency gains?

Response: We leverage curriculum learning to enhance inference performance. This approach involves a gradual introduction of increasingly complex tasks during training. Initially, we focus on simpler training instances, specifically within the 5-digit range. This foundational stage allows the model to efficiently grasp basic calculation rules.

Once the model achieves stable convergence and satisfactory performance on the test dataset, we then implement curriculum learning as a strategic advancement. We introduce a new set of training data that includes numbers ranging from 5 to 12 digits. This methodical escalation in complexity is designed to incrementally challenge and refine the model's capabilities.

Here, we report the performance on MathGLM-2B without using curriculum learning. From the results, it is evident that the use of curriculum learning has a significant impact on the performance of MathGLM-2B. The model, when trained without curriculum learning, achieves an accuracy of 88.70%. However, with the implementation of curriculum learning, the accuracy increases to 93.03%.

Question 4: It is not clear whether the curriculum learning strategy is beneficial since there is no comparison with a non-curriculum counterpart.

Response: We leverage curriculum learning to enhance inference performance and increase model efficiency. This approach involves a gradual introduction of increasingly complex tasks during training. Initially, we focus on simpler training instances, specifically within the 5-digit range. This foundational stage allows the model to efficiently grasp basic calculation rules.

Once the model achieves stable convergence and satisfactory performance on the test dataset, we then implement curriculum learning as a strategic advancement. We introduce a new set of training data that includes numbers ranging from 5 to 12 digits. This methodical escalation in complexity is designed to incrementally challenge and refine the model's capabilities.

Here, we report the result of the performance on MathGLM-2B without using curriculum learning. From the results, it is evident that the use of curriculum learning has a significant impact on the performance of MathGLM-2B. The model, when trained without curriculum learning, achieves an accuracy of 88.70%. However, with the implementation of curriculum learning, the accuracy increases to 93.03%.

Question 5: It is not clear how the Ape210K dataset was reconstructed. Were the step-by-step solutions generated in an automatic way? If so, how was their quality verified?

Response: In our reconstruction of the Ape210K dataset, we employ a Python script to enable a detailed, step-by-step calculation for each math problem (Cf. Figure 3). This enhancement in the dataset is specifically designed to bolster MathGLM's proficiency in grasping the intricate calculation rules essential for solving math word problems. By navigating through these step-wise computations, MathGLM is trained to not only reach the correct answer but also to comprehend the underlying mathematical reasoning. Furthermore, we plan to make the enhanced Ape210K dataset publicly available for open research, contributing to the broader academic community.

Question 6 : What is the rationale of using different models for the Arithmetic task and the Math Word Problems? Shouldn’t the same MathGLM model be able to solve both types of problems? The authors say that “our goal is to simultaneously advance both mathematical reasoning and arithmetical calculation capabilities of LLMs, addressing both aspects at the same time”, but from my understanding they trained separate models for the Arithmetic and MWP datasets (the “Training Strategy” section at pg. 5 should be expanded and described in a much clearer way).

Response: The reason why we use different models for arithmetic tasks and mathematical word problems (MWP) in our study is that the motivations we study are different.

For arithmetic tasks, we aim to address the misconception about the capabilities of large language models (LLMs) in performing arithmetic operations. MathGLM challenges the prevailing notion about LLMs' limitations in arithmetic operations, particularly with longer digits and more complex calculations like decimals and fractions. The utilized step-by-step strategy can significantly improve the arithmetic performance (Cf. Figure 10).

Conversely, for MWPs, we recognize the limitations of the Ape210K dataset, which originally provided direct answers without detailed computations. To address this, we reconstruct the Ape210K dataset using a step-by-step solution strategy and fine-tune GLM models on this reconstructed dataset, specifically for MWPs.

We apologize for the confusion and will revise the statements in the "Training Strategy" section.

Question 7: The authors should more carefully explain how GPT models were tested. Which prompting methods were used to probe these models? How did performance change when using more advanced (e.g., Chain-of-though) prompting strategies?

Response: In our evaluation, we use the standard direct prompting method. In this approach, the models are presented with straightforward questions without additional contextual or guiding prompts. This method was chosen to ensure a fair and consistent baseline for comparing the performance of different models.

We sincerely thank you for your valuable comments on our paper. We will explain your concerns point by point.

Weakness 2: The authors claim that MathGLM has a “profound understanding of the complex calculation process” and “effectively learns the underlying rules and principles of arithmetic operations”, however I do not think that its generalization abilities have been properly evaluated.

Response: Please refer to the Response in Question 2.

Weakness 4: The paper does not include any Reproducibility Statement or any pointer to source code repositories, which makes it difficult to replicate the simulations and the experimental setup.

Response: Here, we provide an anonymous repository (https://anonymous.4open.science/r/MathGLM-anonymous-3085) to release our datasets and code.

Question 1: The authors say that MathGLM learns to solve arithmetic tasks “by integrating a step-by-step strategy into its architecture”. However, it is not clear how the model architecture actually implements step-by-step reasoning process (from the description, it seems that such feature is just a property of the solution format, rather than of the architecture design). This point should be clarified.

Response: We apologize for the confusion. In this study, we do not design a new architecture to solve arithmetic tasks. Instead, we construct a pre-training dataset that includes a variety of arithmetic expressions, encompassing different types of arithmetic operations and various types of numbers.

The key to facilitating the step-by-step reasoning is in the way these expressions are structured and presented in the dataset. This structure of a pre-training dataset provides the model with examples of grasping calculation rules during training and inference.

Furthermore, to enhance the model's ability to handle arithmetic tasks, we employ a unique tokenization strategy. Each digit in an arithmetic expression is tokenized as a distinct token. This approach allows MathGLM to focus on each component of the problem individually, facilitating more accurate computations and enabling the model to follow the step-by-step reasoning pattern inherent in the dataset.

By training MathGLM on this carefully constructed dataset with its unique tokenization method, the model implicitly learns a step-by-step reasoning process, despite not having an architecture explicitly designed for this purpose. Compared with the method that directly calculates the answer of the question, the step-by-step methodology enhances the model's accuracy in arithmetic tasks (Cf. Figure 10).

Question 2: In order to properly test for generalization the authors should demonstrate that the model can solve problems outside the training distribution (e.g., involving much longer numbers, and much more operands, see for example https://arxiv.org/abs/2207.02536). At present, an alternative (and more parsimonious) explanation is simply that the larger-scale of the training data allows to the model to memorize a more consistent amount of arithmetic knowledge.

Response: We thank the reviewer for highlighting the importance of testing for generalization beyond the training distribution. To test the generalizability of MathGLM, we evaluate the generalizability ability of MathGLM on arithmetic problems that involve numbers beyond the 12-digit range.

Question 3: The authors say that “To assess the generalization ability of MathGLM beyond the 5-digit range, a set of 50,000 training records involving numbers within the 12-digit range are introduced into the training dataset”. This does not guarantee that generalization is properly assessed; it rather shows that by adding more training samples from the testing range the performance increases, which is expected (also see https://arxiv.org/abs/2306.15400).

Response: We appreciate the reviewer’s critical observation regarding our methodology for assessing the generalization ability of MathGLM. In our evaluation setting, our aim is to demonstrate MathGLM's enhanced performance in generalization with a notably smaller dataset. This dataset is significantly smaller in comparison to the volume of data used during the initial training phase of MathGLM. This approach underscores the efficiency and effectiveness of MathGLM in learning and adapting to new data.

To rigorously test the model's inherent generalization capabilities, we evaluate the generalizability ability of MathGLM on arithmetic problems that involve numbers beyond the 12-digit range. The detailed results can be found in Response to Question 3.

Thank you for your insightful and well-articulated points. We will explain your concerns point by point.

Weakness 1: Could we extend the evaluation of the new model and see the performance on non-math tasks? The math focus finetune may have reduced the performance on other tasks, and it is good to know how good / bad that would be.

Response: In fact, fine-tuning the model specifically for mathematical tasks may have implications for its performance in other domains. This specialized fine-tuning could lead to a reduction in the model's effectiveness in handling non-mathematical tasks.

In order to reduce this effect, we leverage a hybrid dataset dataset that includes both mathematical and text data for the continue training strategy. Specifically, we use a public Chinese text data Chinese-Vicuna, a rich source of public Chinese text dataset, with our reconstructed Ape210K dataset for continue training. This approach aims to maintain the model's proficiency in general language tasks while enhancing its mathematical abilities.

To evaluate the performance of the model on non-math tasks, we use the Z-bench, consisting of 63 diverse questions. To evaluate the quality of responses generated by the model, we score responses with the powerful GPT-4 on a scale of 1-10 based on factors such as correctness (high priority), helpfulness, relevance, depth, innovativeness, and level of detail. The results are as follows:

MathGLM-10B demonstrates a significant improvement in the mathematical task, achieving a 56.6% accuracy, a significant leap from GLM's 0%. However, it's important to note the slight decrease in performance on general text tasks for MathGLM-10B (5.35) compared to GLM-10B (5.91). While MathGLM-10B excels in mathematics, its ability to handle general text tasks is marginally impacted.

Nevertheless, the marginal reduction in text task performance indicates that the continue training strategy of using a hybrid training dataset has been effective in preserving the model's overall linguistic abilities. Therefore, while there is a noticeable specialization in mathematical tasks, MathGLM-10B maintains a competent level of performance in general language processing.

Question 1: Maybe a followup work would seem how to train model for middle-school level math problems, how the size of dataset / model would scale for that

Response: Thank you for the suggestion regarding follow-up work. We are indeed in the process of developing MathGLM-v2, a specialized model designed specifically for middle-school level mathematics.

The pre-training dataset with MathGLM-v2 encompasses a wide spectrum of mathematical problems relevant to middle-school education. This dataset is not just extensive in size but also diverse in content, covering areas from basic arithmetic to more complex algebraic concepts. As for the model itself, MathGLM-v2 is trained on a larger model with 12B training parameters.

Preliminary results are promising, showcasing the model's ability to grasp fundamental mathematical concepts and apply them in problem-solving. As we progress, we aim to refine MathGLM-v2 further, enhancing its accuracy and ability to generalize across a wider array of math problems.

We are working towards releasing MathGLM-v2 in the near future.

Weakness 3: The primary math word problem datasets are APE and K12, both of which are in Chinese. There were no experiments conducted on popular math datasets like MATH and GSM8K. Since GPT-4 is primarily trained in English, and MathGLM is fine-tuned for Chinese math word problems, such a comparison might not be valid. The advantage of MathGLM could be due to the language, rather than its proficiency in resolving math problems.

Response: In this study, our primary focus is indeed on Chinese language datasets. However, it's important to clarify that the effectiveness of MathGLM is not solely attributable to the language aspect.

To further verify our claim, we develop a Chinese-English mathematical model called MathGLM2-6B. Here, we report a preliminary experimental result on our MathGLM2-6B. We evaluate MathGLM2-6B with the same-scale English mathematical models on three classical inference datasets: Math23K, GSM8K, and MATH. The results indicate that MathGLM2-6B demonstrates promising cross-lingual capabilities. On Math23K, a Chinese language dataset, MathGLM2-6B shows competitive performance with a 61.8% accuracy, slightly behind GPT-4's 63.3%. On GSM8K, MathGLM2-6B achieves an accuracy of 63.00%, slightly lower than the similar-scale model MetaMath-7B. In the MATH dataset, MathGLM2-6B achieves an accuracy of 26.76%. In comparison to other same-scale models, MathGLM2-6B is ahead of WizardMath-7B, Mammonth-7B, and MetaMath-7B.

In conclusion, while language is an important aspect of our model, the primary advantage of MathGLM lies in its ability to understand and solve complex mathematical problems.

Question 1: This paper uses LLMs to conduct complex mathematical computations. This is a novel approach, but the motivation is weak because using LLMs for complex mathematical computations lacks accuracy and generalizability. Additionally, there is a lack of experimentation on MATH and GSM8K, as the primary comparisons are made using Chinese mathematical datasets.

Response: The motivation of our paper can refer to the response in Weakness 1 and the experimentation on MATH and GSM8K can refer to Weakness 3.

Thanks for your appreciation of our work and valuable suggestions! We will explain your concerns point by point.

Weakness 1: The claim regarding motivation is not robust. While MathGLM achieves a high accuracy of 93.03% on the constructed dataset for complex computations, these calculations can be done with 100% accuracy using other tools.

Response: The motivation of our paper is to address the misconception about the capabilities of large language models (LLMs) in performing arithmetic operations.

1. Redefining LLMs Capabilities on Arithmetic Tasks MathGLM challenges the prevailing notion about LLMs' limitations in arithmetic operations, particularly with longer digits and more complex calculations like decimals and fractions. This contributes to the field by expanding the understanding and capabilities of LLMs in mathematical computations.

2. Beyond Mere Calculations We acknowledge that other tools may achieve 100% accuracy. However, we clarify that the uniqueness of MathGLM lies in its ability to handle complex arithmetic operations within the framework of large language models. This allows it to not only perform calculations but also understand and process complex math word problems, which is beyond the capability of many conventional tools.

3. Enhanced Performance in Math Word Problems For math word problems, we incorporate this step-by-step strategy into the original Ape210K dataset, which significantly bolsters MathGLM's answer accuracy (42.29% performance gains).

Weakness 2: The computation is limited to addition, subtraction, multiplication, division, and exponentiation. The method probably wouldn't generalize well to more intricate computations such as log, sin, etc. Moreover, mathematics should aim for complete accuracy, so utilizing LLMs for these calculations isn't a suitable strategy, especially considering the costly pretraining involved for computations that other tools can resolve more efficiently. Instead, LLMs should concentrate on providing more insight and higher-level strategies for solving math problems.

Response: In this study, we focus on addressing the misconception about the capabilities of large language models (LLMs) in performing arithmetic operations. MathGLM is currently limited to basic operations like addition, subtraction, multiplication, division, and exponentiation.

It's important to note that functions like logarithm and sine, which are more complex, are not directly computable by humans without tools. We often rely on calculators for these function calculations, suggesting that LLMs may not be critically needed for such intricate calculations.

We agree with your claim that LLMs should concentrate on providing more insight and higher-level strategies for solving math problems. In our study, we find that calculation errors are a significant error type in solving math word problems (Cf. Figure 9). In order to enhance the performance of MathGLM-10B in solving math word problems, we incorporate an additional 5,500 calculation-specific training data into the reconstructed Ape210K, and evaluate performance on the Ape210K test dataset. From the results, we can observe that by adding calculation-specific data, MathGLM-10B's performance is significantly improved compared with the original MathGLM-10B (reported in our manuscript). Therefore, the study of calculation also plays an important role in solving math word problems.