ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

Dear reviewer Gq26,

We appreciate your thorough review and insightful comments on our paper. We are pleased that the you recognizes our efforts and the promising results of the ToRA pipeline. We will address the concerns raised and answer the questions posed to improve our work further.

Enhancing Readability with Appropriate Mathematical Fonts

Adopting complicated mathematical fonts when unnecessary only takes from the readability of the paper. There is no need to use them when the default fonts suffice.

We appreciate your feedback! We understand the reviewer's concern about the complexity of mathematical fonts making the paper less readable. Our intention was to maintain the mathematical accuracy and precision, e.g., our choice to use $\wp$ for "prompt" and $p$ for "program" was driven by a desire to avoid confusion, but we understand that this might have inadvertently complicated the presentation. To enhance readability, we will revise these symbols to more commonly used and easily comprehensible ones without losing the distinction between the two terms.

Suggestions for Further Improvements on Math Reasoning

How should we improve upon ToRA? Since the pipeline relies on a proprietary model to generate the dataset, and the final performance of the strongest ToRA is still inferior to GPT-4

Thank you for posing an insightful question about improving upon ToRA! One point that needs clarification is that the core method of ToRA is tool-integrated reasoning learning, and the data annotation in this process is model-independent for collecting tool-integrated reasoning trajectories. It can also be obtained through other open-source models or even manually annotated for higher-quality data.

Moreover, our experiments substantiated the effectiveness of this tool-integrated reasoning format on GPT-4, demonstrating its potential to further augment GPT-4's performance. As illustrated in Fig. 4 (right), the implementation of tool-integrated reasoning can yield a 9.8% improvement compared to PAL.

Looking forward, we see several main directions to further enhance mathematical reasoning based on ToRA:

• Continue Pre-training: We found in subsequent experiments that applying ToRA to models continue pre-trained on math and code data brings a more dramatic improvement! This suggests that pre-training and data engineering might deserve more attention for the research community!
• SFT Mixture Scale-up: We could explore amplifying the quality and coverage of fine-tuning data in tool-integrated reasoning format, which might further improve generalization and robustness.
• Reward Modeling: We could explore methods to enhance tool-integrated reasoning by incorporating reward modeling methods like [1], to further improve accuracy, intention alignment, interpretability, and to make it more user-friendly.

We extend our gratitude for your constructive feedback and the opportunity to clarify the nuances of our work. We believe that the proposed changes will significantly improve our paper's readability and impact.

[1] Lightman, Hunter, et al. "Let's Verify Step by Step." arXiv preprint arXiv:2305.20050 (2023).

Setting Maximum Rounds of Interaction

Ablation study of hyperparameter $n$ (maximum rounds). in experiments, $n=3$ is used. Question: is the performance of TORA sensitive to $n$?

Our preliminary experiments found that on mathematical reasoning benchmarks like GSM8k and MATH, large language models nearly always solve problems with n ≤ 3 interactions, so we set it as the default hyperparameter. Setting a higher number of interactions does not bring significant gains.

Writing

• "forward and backward reasoning, as well as result verification": references for forward/backward reasoning, verification"
• Addressing these challenges requires complex symbolic reasoning over algebraic expressions": references for symbolic reasoning
• where is the definition of $\theta$ in (4)?

We appreciate your meticulous attention to details and pointing out writing typos. We will add all the references you mentioned and the definition of $\theta$ in equation (4) in future updates.

Questions

In the Conclusion section, the authors mention that "our systematic analysis ... paving the way for the development of more advanced and versatile reasoning agents". How and why does this analysis pave the way?

We appreciate your question! As an initial exploration of constructing math reasoning agents based on tool-integrated reasoning, we demonstrated the effectiveness of combining natural language with programming language to solve challenging reasoning problems. We believe this approach has potential and is worth further exploration. Additionally, in Section 3.6, we have conducted an in-depth analysis of the areas that still require enhancement and merit additional attention. We believe that this systematic analysis and identification of areas for improvement lays the groundwork for the development of more advanced and versatile reasoning agents.

What is the difference between Tool-Integrated Reasoning (Algorithm 1) and existing methods (e.g., PAL, PoT)?

As shown in Fig. 2, PAL and PoT are Program-based methods to solve tasks with program synthesis, while our proposed Tool-integrated Reasoning format integrates natural language reasoning with program-based tool use, in order to combine the benefits of both worlds.

As shown in Fig 4, the proposed Tool-Integrated Reasoning consistently surpasses Rationale-only and Program-only approaches. Remarkably, Using LLaMA-2, Tool-Integrated Reasoning achieves substantial improvements of 29.0% and 6.7% over Rationale-only and Program-only, respectively. With the closed-source GPT-4, the improvements are 19.1% and 9.8%, respectively. This emphasizes the effectiveness of integrating natural language rationales with programs.

"For numerical values, we perform rounding, while for expressions, we employ sympy for parsing." Are these two techniques used in the baseline methods?

Sure! For a fair comparison, we adopted the same evaluation for all methods.

Are there any empirical results can support the ``Output Space Shaping improves diversity''. Furthermore, the diversity measure is not defined in the paper.

Thanks for your question! We constructed many trajectories for each problem through sampling and teacher correction, where the reasoning processes are different but the answers are correct. In other words, improving diversity means increasing the number of different solution paths for the same problem.

References

[1] Yuan, Zheng, et al. "Scaling relationship on learning mathematical reasoning with large language models." arXiv preprint arXiv:2308.01825 (2023).

[2] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct." arXiv preprint arXiv:2308.09583 (2023).

[3] Yu, Longhui, et al. "Metamath: Bootstrap your own mathematical questions for large language models." arXiv preprint arXiv:2309.12284 (2023).

[4] Yue, Xiang, et al. "Mammoth: Building math generalist models through hybrid instruction tuning." arXiv preprint arXiv:2309.05653 (2023).

[5] An, Shengnan, et al. "Learning From Mistakes Makes LLM Better Reasoner." arXiv preprint arXiv:2310.20689 (2023).

Dear Reviewer U6Sz,

Thank you for your time and thoughtful feedback on ToRA! We have addressed each of your points below.

Technical Novelty

Using imitation learning to improve the mathematical reasoning ability of open-source models has been proposed in many recent works, e.g.,

• Scaling relationship on learning mathematical reasoning with large language models, https://arxiv.org/abs/2308.01825
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct, https://arxiv.org/abs/2308.09583
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models, https://arxiv.org/abs/2309.12284
• MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, https://arxiv.org/abs/2309.05653

We appreciate your reference to concurrent works. While both ToRA and these studies employ imitation learning, the Tool-integrated Reasoning format proposed to train ToRA fundamentally differs from the CoT or PoT formats used in these works, yielding significantly better results.

Specifically, references [1-3] primarily adopt natural language rationale (CoT), focusing on augmenting CoT solutions [1] and questions [2, 3] for improvements. On the other hand, reference [4] concentrates on hybrid training using CoT and PoT.

In contrast, ToRA introduces the Tool-integrated reasoning format, which brings clear advantages. For instance, even the smallest ToRA-Code-7B outperforms MetaMath-70B [3] (44.6% vs. 26.0%) and MAmmoTH-Coder-34B [4] (44.6% vs. 43.6%).

Output Space Shaping

At the Output Space Shaping step, the authors use nucleus sampling to generate more reasoning paths and pick the corrected paths. This is a technique widely used in the works listed above. The only difference is that this paper fixes some of the preceding portions of wrong trajectories, while existing works resample the whole trajectory. However, validating which portions of trajectories are correct is very challenging, and the authors need to enumerate possible preceding portions of wrong trajectories, which is time-consuming.

We proposed the method of learning from small model errors corrected by a teacher model, which we call teacher correction, and we combined it with sampling for output space shaping.

The step-level enumeration for teacher correction is a one-time cost for constructing training data. In our experiments, compared with using sampling alone for output space shaping, introducting teacher correction does not increase training cost because we maintain the same volume of training data.

In my opinion, the existing method (i.e., re-sampling and picking the correct paths) is simpler and may be more effective.

Our research shows that integrating teacher correction contributes to higher efficacy than merely using sampling. As illustrated in Figure 5, employing the sampling strategy alone resulted in a 2.7% performance enhancement. Conversely, the incorporation of the correction strategy led to a performance improvement ranging from 3.4% to 4.0% across two datasets. This improvement was achieved while maintaining the same volume of training data as with sampling alone, thereby substantiating the effectiveness of our output space shaping strategies.

Reproducibility

Concerns on reproducibility: As training data are not provided, reviewers and readers can't check the reproducibility of this paper. Note that existing works like WizardMath, MetaMath, and MAmmoTH have released their training data for the community to reproduce their results. Moreover, checkpoints of TORA are not provided in the appendix for checking reproducibility.

We understand your concerns about reproducibility. We're currently undergoing an internal review to open-source the ToRA-Corpus and will add the open-source links to code, model checkpoints, etc., after the anonymous review period.

Thanks for the response. Some of my concerns were addressed, but some remain.

• limited novelty: Existing works have shown the effectiveness of imitation learning in improving the performance of open-source models, while this work just shows that PoT or CoT can be replaced with more powerful PoT+CoT (which is also used in MAmmoTH) to boost performance. Moreover, what are the clear advantages in the reply "In contrast, ToRA introduces the Tool-integrated reasoning format, which brings clear advantages"?
• the accuracy of LLaMA trained on TORA-CORPUS has yet to saturate (e.g., plot the accuracy w.r.t. #samples of TORA-CORPUS). A follow-up question is: Are data generated from Output Space Shaping more effective than data generated by GPT-4? If not, why not just enlarge the latter? I understand GPT-4 is costly, but how do we maintain a good trade-off?
• "Our systematic analysis ... paving the way for the development of more advanced and versatile reasoning agents" in the conclusion section might be overclaimed. Just a minor comment.
• MAmmoTH and MetaMath are missing the main table (i.e., Table 2).

For a fair comparison, we adopted the same evaluation for all methods

It seems that some of the results in the main table are copied from their publications, I checked some of them but failed to find these techniques (round, sympy), e.g., WizardMath (https://arxiv.org/pdf/2308.09583.pdf), RFT (https://arxiv.org/abs/2308.01825)

Improving diversity means increasing the number of different solution paths for the same problem

Note that different solution paths do not mean they are diverse.

btw, is the current paper the latest? (updating the paper is allowed in ICLR)

On More Significant Improvements on Smaller Models

It appears that the method proposed in this paper shows more significant improvements on smaller models, as the performance of the 7B, 13B, and 70B models shown in Figure 1 does not appear to differ significantly. How to explain this phenomenon?

Thank you for your detailed observation! According to Table 2, ToRA-Code's performance on GSM8k increased from 72.6 to 75.8 and 80.7 as the model size increased from 7B to 13B and 34B, respectively. In contrast, the performance on the MATH dataset saturated as the model size increased, with performances of 44.6, 48.1, and 50.8, respectively.

To further understand the bottleneck encountered by ToRA on the MATH dataset at around 50%, we collected the accuracy of different models on MATH problems of different difficulty levels, as shown in the following table:

From the table, it is clear that the saturation of performance with increasing model size is primarily evident in difficult problems, i.e., levels 4 and 5. This phenomenon can be attributed to the following reasons:

1. Firstly, ToRA greatly improved the learning efficiency of smaller models, and raised the starting point of the scaling curve. As shown in Fig. 4, the format of Tool-integrated reasoning is more conducive to model learning under the same amount of training data, allowing smaller models to reach a very high level.
2. Secondly, the performance of the largest ToRA model is already close to that of the GPT-4 used for data annotation, and the end point of the scaling curve is mainly limited by the bottleneck of data annotation. The data constructed using GPT-4 has a relatively low coverage of difficult samples (81.3 and 68.0% for Level 4 and Level 5 samples, respectively). Increasing the size of the model increases the model's capacity, but the bottleneck is likely to be the quality and quantity of annotated samples.

This could encourage further research to focus on challenging math problems and more effective data engineering.

On Discussing Alternating Reasoning Paradigm for Agent Reasoning

The alternating reasoning approach between natural language and tool usage proposed in this paper is fundamentally similar to the plan paradigm of Thought, Action, and Observation alternation in REACT [1]. Do you think this paradigm will become the dominant paradigm for agents to solve complex reasoning problems?

Thank you for your question! We agree with your view. Since language is the interface for LLMs to interact with tools and environments, the solving process for most LLM-Agent related tasks can be seen as interleaved rationale and interaction trajectory. Therefore, we believe alternating reasoning is a fundamental task completion paradigm based on the language interface, and we look forward to future work further promoting this method to solve more complex reasoning tasks with LLM!

Dear reviewer V2TN,

Thank you for your comprehensive and meticulous review. We appreciate your recognition of the excellent performance of ToRA, as well as the approval of our presentation.

On Output Space Shaping Results in Table 5

From Figure 5, it can be observed that the performance of the model does not significantly decrease when output space shaping is removed. More experiments are needed to demonstrate whether the performance improvement in this stage is due to this training strategy rather than additional data and more training epochs.

We appreciate your keen observation regarding Figure 5. There was a mistake in Figure 5 in the paper where the “sampling” strategy was Inappropriately referred to as “shaping”, and we will correct this in future updates. Here’s the clarification:

1. Output Space Shaping, in fact, brought a significant improvement. As shown in Table 3, this method improved the performance of the 7B, 13B, and 34B models by 4.4%, 3.5%, and 3.4% respectively. For instance, ToRA-34B increased from 47.4% to 50.8%. Furthermore, the improvement was particularly noticeable in subtopics like Precalculus, Geometry, and Algebra, with an increase of up to 5.9% ~ 6.3%.
2. Output space shaping includes two strategies: sampling and correction. In Figure 5, we conducted three experiments for comparison: pure imitation learning, using only the sampling strategy for shaping, and a combination of sampling + correction strategies for shaping. It's worth noting that all models were trained for the same number of epochs and both shaping experiments used the same amount of training data. The results showed that shaping using only sampling led to a 2.7% average improvement, while combining sampling + correction strategies led to an average improvement of 3.4% ~ 4.0% on two datasets under the same data amount. These results demonstrate the effectiveness of the sampling and correction strategies in our proposed output space shaping.

On Including More Detailed Information of ToRA-Corpus

Regarding the TORA-corpus proposed in this paper, more detailed information is needed regarding the data construction process, quality evaluation, and dataset statistics.

Thank you for your suggestion! Based on your suggestion, we will add a section in the appendix to provide a more detailed introduction to the data construction process, quality control, and report more data statistical information, beyond Sec. 2.2. Specifically:

• Data format and quality control: In our preliminary experiments, we found that the tool-integrated reasoning trajectory format generated by zero-shot prompting was somewhat chaotic. Therefore, we designed a few-shot prompting to control the reasoning format, which effectively improved data quality. On the other hand, we increased the annotation success rate by sampling, ensuring more comprehensive coverage of the training query.
• Data filtering process: For the data constructed, we filtered out paths that produced incorrect answers by matching them with standard answers. To prevent the model from learning incorrect intermediate reasoning processes, we further filtered out data samples with intermediate program execution errors.
• Dataset statistics: We compared the annotation accuracy (i.e., sample coverage) of the training set on GSM8k, MATH, and MATH subtopics of ToRA-Corpus-Greedy (Sec. 2.2) using only the greedy trajectories, and ToRA-Corpus-16k combined with sampled trajectories. Furthermore, we reported the statistical data of ToRA-Corpus-16k, such as the number of samples, average question length, average, minimum, and maximum trajectory length, as shown in the following tables.

Table 1: Accuracy of ToRA-Corpus-16k on GSM8k and MATH.

Table 2: Statistics of ToRA-Corpus-16k

Dear Reviewer mvjp,

Thank you for your thoughtful review and the recognition of our work's clarity and effectiveness. We are pleased to provide more in-depth insights into the failure modes and the impact of output space shaping, as per your suggestions.

Adding Comprehensive Failure Mode Analysis

While the authors have presented what worked well, there is a considerable amount to be gleaned from the failure modes. The authors loosely allude to failure cases including geometric problems and program timeouts, and provide single examples in the appendix, but there are surely more interesting patterns. It would be wonderful if the authors could provide more specific examples and comment on more systematic classes of errors beyond these simple categorizations. Are there certain patterns in the natural language specification of the original problem or rationale construction that fail to formalize well as programs?

We appreciate your suggestion for a more comprehensive analysis of failure modes. To this end, we manually annotated 100 randomly selected trajectories from the MATH test set, identifying and categorizing their failure modes. Here are the results:

We observed that:

1. Incorrect reasoning steps constitute the primary source of errors for ToRA on complex math reasoning tasks (38%), with hallucination issues also evident during problem interpretation and answer finalization (5%).
2. Misinterpretation of input diagrams is the second largest category of errors (21%), particularly prevalent in Geometry, Precalculus, and Intermediate Algebra. This may be due to the fact that diagrams in the MATH dataset are often specified in text with the Asymptote language [1], which poses challenges to ToRA when understanding diagrams through text alone.
3. Issues with tool usage include Inappropriate Tool Usage (10%), Syntax Error (9%), and Runtime Error (9%). These issues often manifest as ToRA being unable to correctly use tools after multiple rounds of modification and attempts. There are certain inputs that fail to formalize well as programs (3%), which require abstract reasoning rather than computation.
4. We also found that there are false negatives when using automatic indicators, i.e., correct predictions that are misjudged as wrong, but the proportion is relatively small (5%).

We will include examples for each failure mode in the Appendix of the updated paper.

Understanding the Impact of Output Space Shaping in Relation to Question Difficulty

Were the patterns informative in terms of which problems were amenable to imitation learning vs which required output space shaping in order to produce initial valid reasoning trajectories?

Your question about the patterns that determine which problems are amenable to imitation learning versus those that require output space shaping is very interesting!

Initially, we compared the results before and after output space shaping for different subtopics of MATH in Table 3 of the paper. We found that shaping has a more pronounced effect on Precalculus, Geometry, and Algebra, with improvements ranging from 5.9% to 6.3%.

In response to your question, we further compared the effects of output space shaping on MATH problems of different difficulty levels (from level 1 to level 5):

Our findings indicate:

• Across these different difficulty levels, output space shaping generally brings a significant improvement of 4.0% on average across different model sizes.
• Comparing the growth rates for different difficulty levels, we find that output space shaping brings significant improvements for difficult, long problems. For example, with ToRA-Code-13B, shaping does not significantly improve level 1 to level 2 problems, but it brings a substantial improvement of 5.4% to 5.7% for level 3 to level 5 problems.
• After using shaping, ToRA-34B outperforms GPT-4 PAL on problems from Level 1 to Level 4, but there is still a gap at Level 5 (27.3% vs. 30.0%). These problems are usually longer (average about 248.4 characters), require more reasoning steps (>1,000 characters) to solve, and more often include diagram inputs (about 20%). These findings may guide future work to focus more on solving these more difficult problems.

We will include these analyses in the updated version of the paper. Once again, thank you for your excellent suggestions!

References

[1] Hendrycks, Dan, et al. "Measuring mathematical problem solving with the math dataset." arXiv preprint arXiv:2103.03874 (2021).