Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees

Many thanks for the constructive reviews! We will revise our paper accordingly.

Q1: "The paper can benefit from some case studies. Though numbers suggest that TP-LLaMA can master tool-usage instructions better and exhibits stronger generalization capabilities, case studies comparing the reasoning trajectories between TP-LLaMA and LLaMA w/ SFT on different task scenarios is needed to prove the effectiveness of DPO."

A1: Thanks for your suggestion. Here we provide an example of the G3-Ins. scenario. First, we give the query and related API documentation:

• query: I'm organizing a film festival and I'm looking for award-winning films. Can you search for videos related to 'award-winning' on Vimeo? Additionally, fetch the related people in the 'film festival' category to invite them as judges. Finally, provide me with a streaming link for a YouTube video with the ID 'UxxajLWwzqY'.

• related API documentation (parameter information is omitted):

  • name: getrelatedchannels_for_vimeo, description: Get Related Channels.
  • name: searchvideos_for_vimeo, description: Search for videos.
  • name: getrelatedpeople_for_vimeo, description: Get a list of related people for a category.
  • name: download_stream_for_ytstream_download_youtube_videos, description: Stream or download info.
  • name: Finish, description: ...

  (Ps: during our experiment, the first three APIs are temporarily not accessible.)

Next, we briefly present the reasoning trajectories of TP-LLaMA and LLaMA w/ SFT respectively. Note that we limit a path to contain at most 12 nodes (ie. at most 4 API calls), and a tree contains at most 200 nodes. We require the model to answer as much content of the query as possible.

• The reasoning tree of TP-LLaMA includes 34 nodes and 5 paths. The successful path is:

  1. call [searchvideos_for_vimeo] with arguments {"format": "json", "query": "award-winning"}, get response "...The consumer key passed was not valid...",

  2. call [getrelatedchannels_for_vimeo] with arguments {"category": "film festival", "format": "json"}, get response "...The consumer key passed was not valid...",

  3. call [download_stream_for_ytstream_download_youtube_videos] with arguments {"is_id": "UxxajLWwzqY"}, get response "...'id': 'UxxajLWwzqY', 'title': 'Icona Pop - I Love It (feat. Charli XCX) [OFFICIAL VIDEO]', ...",

  4. call [Finish] with arguments {"return_type": "give_answer", "final_answer": "The streaming link for the YouTube video with ID UxxajLWwzqY is [link]..."}, get response "successfully giving the final answer".

     (Ps: here we use [link] to replace the real URL in the final answer, because external links are not allowed in rebuttal.)

• The reasoning tree of LLaMA w/ SFT includes 49 nodes and 8 paths. There is no successful path. Here are some of the failed paths:

  1. searchvideos_for_vimeo -> searchvideos_for_vimeo -> searchvideos_for_vimeo -> getrelatedpeople_for_vimeo

  2. searchvideos_for_vimeo -> searchvideos_for_vimeo -> searchvideos_for_vimeo -> getrelatedchannels_for_vimeo

  3. searchvideos_for_vimeo -> searchvideos_for_vimeo -> searchvideos_for_vimeo -> Finish with give up and restart

  4. ...... (omit due to space limit)

  Because the model repeatedly tried inaccessible APIs (possibly using different arguments) without accessing [download_stream_for_ytstream_download_youtube_videos] first, it failed to give part of the answer in the end.

Based on this case, we can see that TP-LLaMA can reduce unnecessary attempts of failed paths, explore uncalled APIs, and obtain accessible answers faster. Due to the space limit, we will discuss case studies in other scenarios in the revised version.

Q2: "I'd like the authors to provide a more detailed explanation of their design choices for training hyperparameters... To strengthen the analysis, it would be valuable to include ablation studies investigating the impact of these two hyperparameters on model performance."

A2: Thanks for the question. We provide an explanation of our design choices regarding training hyperparameters:

• We initially filtered 42,192 reasoning trees with branching nodes from Toolbench, which led to 69,393 DPO samples and 184,816 SFT samples (as each query may correspond to multiple samples).

• After allocating a small part as a validation set, we sampled training sets of different sizes based on these samples. The sampling methods include: by query and by sample. For sampling by query, the SFT training set ranged from 2,500 to 10,000 queries, and the DPO training set ranged from 5,000 to 32,192 queries, yielding nine combinations. For sampling by sample, SFT varied from 10,000 to 183,561, and DPO from 10,000 to 68,951, yielding seven combinations.

• We tested most of these training sets and observed that increasing the size may lead to decreased model performance in scenarios requiring generalization, such as G3_instruction (e.g., with settings {SFT: 44,412, DPO: 41,226}, the pass rate dropped to 0.36), possibly due to overfitting. Consequently, we selected the set {SFT: 11,142, DPO: 8,202} finally. We will include the ablation studies in the revised version.

Q3: "Aside from ToolBench, there are some other benchmarks that can test the tool reasoning capability, e.g. APIBench. Does TP-LLaMA still perform better than the other baselines?"

A3: Thanks for the question. Currently, TP-LLaMA relies on relevant API documentation provided for each query in instructions, which is not available in most benchmarks like APIBench. So we are temporarily unable to directly test its performance on other benchmarks. In our future work, we will add an API retriever to replace the pre-annotated relevant APIs, and then test the effect of TP-LLaMA on other benchmarks.

Many thanks for the constructive reviews! We will revise our paper accordingly.

Q1: "In the proposed ToolPreference dataset, it just uses successful path and failure path as preferred and dispreferred data. However, I think such a selection is too easy as model will never choose any dispreferred data. I think directly training on these preferrence data can also produce better performance."

A1: Thanks for the comment. Below we give an explanation of our approach.

• The "preferred" and "dispreferred" data in ToolPreference actually reflect human preferences based on whether the reasoning path is successful, rather than the model's tool preferences. Because humans always prefer paths that lead to successful reasoning and more correct answers. By constructing the preference dataset this way, we aim to align human and model preferences.
• Additionally, expert trajectory data for our SFT training comes from the same DFSDT samples as ToolPreference, meaning our SFT training includes all preferred data. However, our results indicate that training only on preferred data leads to poorer performance than utilizing ToolPreference for DPO training.

Q2: "Tool using also require model to be generalized to other scenarios. Is the proposed model also suitable for other scenarios?"

A2: Thanks for the question. ToolBench contains a wide range of tool usage scenarios across 49 categories, such as news, music, and advertising. Some categories are reserved for testing and do not appear during training. So we have evaluated our model's generalization ability based on ToolBench in our paper, specifically from three perspectives:

1. Instruction Scenario: Evaluating the model's reasoning for new instructions.
2. Tool Scenario: Assessing the model's application of unseen tools within known categories.
3. Category Scenario: Evaluating the model's ability to use tools from new categories.

Since different tool categories focus on different task scenarios, the results in our paper, particularly from the Category Scenario, offer valuable insights into our model's generalization performance, which outperformed baseline models.

Q3: "The problems in tool usage usually involve multiple aspects, mentioned by ToolQA [a], like parameters incorrection, choose incorrect tools. I think besides to train tool preference, how to enable LLMs to successfully use Tools is more important. And authors can also analyze whether the proposed method can alleviate these errors when using Tools."

A3: Thanks for the comment. We provide a brief explanation below.

• Firstly, it is important to clarify that our ToolPreference dataset construction relies on correct tool and parameter selection, as expert trajectories that succeed within a limited path length are more likely to select the right tools and input correct parameters.
• As explained in A1, ToolPreference reflects human preferences rather than a mere model preference toward tools. Our goal in optimizing based on ToolPreference is to enable models to make better choices more rapidly while minimizing unnecessary exploration.
• Our experimental results indicate that our method effectively reduces errors in tool selection and parameter generation, as our model achieves higher rates with fewer reasoning steps. However, quantitatively analyzing these error reductions is challenging due to the need for extensive human annotation, and we will try to include this in our revised version.

Q4: "I didn't see any dataset about ToolPreference in supplemental materials. I think if this dataset is considered as a contribution, it should also be reviewed."

A4: Thanks for the suggestion. We will make our dataset public after the paper is accepted.

Q5: "Can the proposed method be generalized to other benchmarks? It seems that the proposed method is just applied to ToolBench."

A5: Our method can be applied to other benchmarks. Here’s a brief overview of our process:

1. Based on the DFSDT framework, using GPT to generate annotated reasoning trees for training samples of any benchmark.
2. Using the expert annotations from the previous step, create a preference dataset through our proposed method.
3. Perform SFT and DPO training based on pre-train models. SFT uses instruction-tuning data from successful paths in the expert annotations, and DPO employs the preference dataset.

The above process is universal and does not depend on Toolbench, so it can be applied to other benchmarks. Furthermore, we plan to integrate an API retriever in future work, allowing TP-LLaMA to be used across benchmarks without additional training.

Q6: "Did you involve any human evaluation? Pass rate and win rate are just two objective metrics to evaluate whether the model can execute the answer path and whether the solution path is preferred. It cannot guarantee the obtained answer is optimal."

A6: Thanks for your suggestion, we will enhance our evaluation in the revised version accordingly. Due to the variety of APIs in ToolBench, human evaluation is complex and time-consuming. And some queries yield open answers, making it challenging for human annotators to determine the optimal answer. So we didn't involve large human evaluation but used the common method of taking ChatGPT as an expert to evaluate.

In addition, it should be emphasized that the win rate measures both the preference for solution paths and final answers. Specifically, in win rate evaluation, when both our model's answer and expert annotations are not empty, we use ChatGPT to assess whether each answer solves the problem before comparing reasoning paths. Qin et al., 2024 [1] have shown that the win rate correlates well with human preferences, making it a valid metric for our assessment.

[1] Qin, Yujia, et al. Toolllm: Facilitating large language models to master 16000+ real-world APIs. ArXiv preprint, arXiv:2307.16789v1.

I didn't have any concern about method but still have some questions about dataset construction.

If we want to apply this method other benchmarks, it also requires us to use DFSDT to get a trajectory tree and then find a solution. Such a solution is to find a way to execute API calls not from the perspective of human preference. And please note, ToolBench is also just a instruction dataset not from real-world scenarios or human data. So, it just represents a solution to find executable tools, not human perference. And it seems you did not conduct any human verification on ToolPreference data (line 187 - line 194). Therefore, I think the proposed method can help model to better find a solution to execute tools but not from human preference. As this dataset is not released and I cannot check this dataset, authors also don't provide any human evaluation in the response. So I will remain some concerns about the dataset and whether it can really align human preference. But I can increase score to 5 as I didn't have any issues about method.

Many thanks for the constructive reviews! We will revise our paper accordingly.

Q1: "Computation Time Clarification: Could you specify the computation time on the mentioned infrastructure?"

A1: Below, we specify the computation time:

• SFT training: 4.6h for 2 epochs.
• DPO training: 3.2h for 1 epoch.
• Inference: 3.4s per API call and 48.7s per query. Computation time varies due to task complexity, network conditions, and API service status.

Q2: "Error Node Handling: How do you plan to address the issue of "error" nodes in the preference pair generation method? Could you provide examples or a detailed explanation?"

A2: Thanks for the valuable insight. On the one hand, most error nodes appear on successful paths because some APIs are temporarily invalid due to their real-time nature, so the model can't get useful responses while generating correct API calls, not necessarily model decision errors. If the APIs become responsive later, the decisions would yield correct results. On the other hand, the error nodes on successful paths from incorrect API calls or parameter errors are relatively few, thus having minimal impact on results.

Nonetheless, we plan to address this issue in the revised version as follows:

• Use GPT to assess the legality of API call names and parameters, as well as evaluate the reasonableness of preference pairs.
• Filter out error nodes caused by API response failures in the preferred nodes and assess the impact on results.

Q3: "Evaluation Criteria: Can you clarify the criteria for what is considered a false positive and the details of the filtering process? This would help in understanding the robustness of the evaluation."

A3: Thanks for the question, we will add more details in the revised version. Here’s a brief explanation:

• False positives are instances where "Finish with final answer" is called but the response lacks substantive meaning (e.g., “I'm sorry, but there are no new arrivals on different platforms”). In practice, we collect common features in false positives (e.g., "sorry", "apologize") and filter responses accordingly to exclude them.

• Pass rate assesses if answers are provided and meaningful. While some scores may be inflated, removing false positives limits this impact. Additionally, in win rate evaluation, we compare the quality of our model reasoning answers with expert annotations, using ChatGPT to assess whether each answer solves the problem. Therefore, combining the results of pass rate and win rate can make a reliable evaluation of the model.

We will improve the pass rate evaluation based on your suggestions and include relevant results in the revised manuscript.

Q4: "Dataset Settings: Are the test scenarios and dataset settings identical to those used by ToolBench, or were they resampled, and what scale?"

A4: Thanks for this suggestion, we will add details in the revised version. Our test datasets come from ToolBench's open source on GitHub. Test scenarios and dataset settings are identical to theirs: G1-Ins., G1-Tool, G1-Cat., G2-Ins., G2-Cat. each contain 200 samples, G3-Ins. has 100 samples.

Q5: "ChatGPT Version Specification: The specific version of ChatGPT used in the baseline is not mentioned. Knowing the exact version is essential for replicating the results and understanding the capabilities of the baseline model."

A5: We will specify the ChatGPT version in the revised version.

• For evaluation of win rates, we used the gpt-3.5-turbo-16k-0613 version.
• The pass rate results we reported for ChatGPT in baselines are computed using test answers provided by ToolBench's open source on GitHub.

Q6: "ToolLLaMA Results Usage: How were the ToolLLaMA results used in your comparison? Were the scores used directly, or were the answers recalculated? Clarifying this would help in understanding the validity of the comparison."

A6: Sorry for our unclear expression, which may have caused difficulties in understanding. We recalculated the scores of ToolLLaMA using test answers provided by ToolBench's open source on GitHub. Due to space limitations, please check A3 to Reviewer z4Vk for more discussions.

Q7: "Same amount of post-SFT compute on different dataset: Would you consider conducting additional comparative experiments to address this? if not, could you provide specific reasons why you believe this is not a concern?"

A7: Since DPO is a preference optimization method, we can't use only successful paths from the ToolBench for preference training. And because there are no other preference datasets developed based on ToolBench, we can not directly compare how dataset choice impacts the performance, but we will try to construct preference datasets in other ways for comparative experiments in the revised version.

Currently, we can provide a preliminary comparative experiment to show that the performance improvement should not come from increased compute resources. Specifically, we created a new SFT dataset using the same original samples and size as ToolPreference, and performed one epoch SFT training based on the same post-SFT model. The pass rates we observed were lower than those of the DPO fine-tuned model and most even falling below the post-SFT model:

This may be a result of overfitting. Clearly, simply increasing the amount of post-SFT compute does not enhance the model's learning of the data.

Many thanks for the constructive reviews! We will revise our paper accordingly.

Q1. "The paper is related to the works on process reward, e.g., arxiv: 2310.10080, arxiv: 2312.08935, yet does not discuss about it in the related works section."

A1. Thank you for pointing out these important references. Both the two reference works train a process reward model to evaluate each step in the multi-step reasoning task. Our motivation is similar to theirs, but instead of explicitly performing reinforcement learning on the model with process rewards, we construct a preference dataset based on the characteristics of DFSDT to optimize the preference of the model. Their works provide inspiration for us to improve the way we construct preference data in the future. We will cite them and add more discussions in the revised version.

Q2: "Two different methods of preference dataset construction are discussed in the paper (path-wise and step-wise), but there is no result supporting that discussion."

A2: Thanks for the suggestion. Due to code difficulty and time constraints, we are unable to provide experiment results here, but we will add comparative experiments in the revised version. Below we briefly give some theoretical explanations.

• Our model employs step-wise reasoning via a depth-first search, making only one decision (i.e, API call) per interaction, and each step is based on the previously obtained API feedback. The path-wise method focuses on learning an entire reasoning path, which may cause the model to generate the entire decision sequence at once without receiving synchronous feedback from APIs. We believe this approach is fundamentally unsuitable for our step-wise reasoning paradigm. Using a path-wise preference dataset might negatively impact the model's single-step inference performance.
• The recent work of Lai et al., 2024 [1], who compared vanilla DPO and step DPO, reveals that step-wise preference optimization benefits long-chain reasoning. We are confident that similar results would be observed in the context of complex tool usage.

Q3: "The result of ToolLLaMA reported in this paper is different from the original paper. l feel like it's due to different definitions of "pass rate", but can you give an explanation on this?"

A3: Thanks for your question. The ToolLLaMA results we reported indeed differ from the original paper, but this disparity is not due to different definitions of pass rate. Below, we give an explanation.

• During our experimental process, we noticed that the original paper had been updated with a revised definition of pass rate and corresponding results. Given resource constraints, we followed the first version's definition [2], using their provided test answers of ToolLLaMA on GitHub (reproduction_data.zip) to compute pass rates. Moreover, it is important to emphasize that we could not reproduce their reported results with either definition.
• Actually, several researchers on GitHub have also reported inconsistencies between their results and the original results, without a response from the original authors yet. Potential reasons for these differences may include: (1) differences in ChatGPT APIs, despite we use the same version with them, (2) differences in the evaluation environment configurations.
• While our reported results do not match those in the original paper, it is important to emphasize that our experimental outcomes are still valid and credible. Our evaluations used a consistent setup for all models, so the relative differences between models are meaningful. Moreover, the second version's pass rate is revised upon the first version, and thus, the first version's pass rate also holds a certain reference value.

We will continue our efforts to contact the original authors to align our evaluation settings. Any updates will be included in the revised version of our paper.

Q4: "The paper lacks a thorough discussion on its limitations. For example, it seems that the performance of the method highly depends on the quality of the decision trees, while "ETO" mentioned in the paper doesn't."

A4: Thanks for the suggestion. We will include a thorough discussion of limitations in the revised version. As noted, the performance of our method relies on the quality of decision trees. In addition, we can not compare preferences between steps on unsuccessful paths, leaving room for data utilization improvement. In future work, we aim to create a general reward model for better preference data construction and filtering. Additionally, our method requires inputting all historical decision information at each inference step, which is time-consuming and increases the difficulty of understanding. To address this, we plan to add a summary step in the reasoning process to shorten interaction text and help the model extract useful information, improving reasoning efficiency.

[1] Lai, Xin, et al. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs. ArXiv preprint, arXiv:2406.18629.

[2] Qin, Yujia, et al. Toolllm: Facilitating large language models to master 16000+ real-world APIs. ArXiv preprint, arXiv:2307.16789v1.