MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

We thank the reviewer for the affirmative evaluation and the constructive suggestions on improving our work. We will reply to the concerns in detail in the following lines.

Q1. Lack of clarity regarding “solution data”. The reviewer points out that the concept of solution data is not fully introduced and suggests "defining different types of data in the experiment setup and using a consistent notation across the entire paper."

A1. A sample of solution data (or a solution path) mentioned in our paper can be defined as a sequence of actions and states $p=${$s_1, a_1, ..., s_T, a_T$}$\in P$, where $T$ is the number of steps to reach the terminal state. The solution path should lead to an objective state or satisfy the user's instruction. Note that in practice each action should include a "thought" before calling the tool with inputs, in order to elicit the reasoning ability of LLMs. For clarity, we now describe the data each baseline is trained on: LLaMA3-solution and ToolLLaMA are trained merely on the solution data $P$. MetaTool and LLaMA3-2-stage are trained with both the solution data $P$ and the meta-tasks data $M$. We have revised our paper to include the formalization (line 134) and clarification (Section 3.1.2 lines 307-311) above and kept consistent notation throughout the paper.

Q2. Lack of details about the “meta-task generation”. The reviewer kindly offers suggestions for a better demonstration of our work including "replace figure 4 with a qualitative example of meta tasks generated for real-world benchmarks" and "providing the actual prompt used for metaset construction".

A2. Thanks for the suggestion to improve our demonstration. We have added a new figure in our revised paper (Figure 4 in lines 486-514) showcasing the meta-tasks generated for ToolBench. The tool search_by_title_for_MDBList is provided on the real-world API website RapidAPI (https://rapidapi.com/hub). The parameters are named casually and we can hardly derive their function just by letters (e.g. ’s’, ’m’). The meta-tasks help the model learn the function and usage of these parameters. For example, from the QA pair of Effect meta-task the model observe that feeding ’s’ as ’friends’, ’m’ as ’movie’, and ’l’ as 1 results in a movie titled ’friends’. From the Input boundary meta-task, the model learns that ’tv’ is not a valid value for parameter ’m’. With multiple QA pairs for each tool, our model is able to learn a more robust tool understanding from actual instances besides descriptions. The tool learning benefits from this paradigm especially in real-world scenarios where the tool descriptions may be diverse and noisy.

We also provide the actual prompt for both context generation and solution path searching in Appendix A.1 (lines 706-755). Please feel free to check them out and let us know if there is any issue.

Q3. Qualitative examples of context generation. The reviewer is concerned about the impact of the complexity of the generated contextual instruction and calls for "a few qualitative examples of this context generation".

A3. We understand that the lack of examples for generating contextual instructions has led to ambiguity and concern. Here is a qualitative example of this process and we have also included more in Appendix A.2 (lines 758-788):

Input for LLMs: Tool: fixtures_for_golf_leaderboard (Lists tournament fixtures for a given tour_id and season_id). Input parameters: {"tour_id": 1, "season_id": 2023}. Result: "2023 European Tour"

Output (contextual result): Golf fixture held in 2023 season with tour_id 1 is 2023 European Tour.

As shown above (prompt provided in Appendix A.1), the LLM worker is only asked to complete the contextual information for the results returned by the tool. In the Effect meta-task, the model learns to predict the contextual results given the input parameters, which helps it better understand the tool mechanism. Otherwise asking the model to predict merely the retrieval results (e.g. 2023 European Tour) is impractical and not beneficial. No other information or prior knowledge from the LLM work is provided.

Dear reviewer vUAy:

I hope this message find you well. We have carefully considered your feedback and have made corresponding improvements to the manuscript. We truly value your insights, and your expertise has greatly contributed to enhancing the quality of our work. Could you please let us know if the revisions meet your expectations? As the deadline for discussion nears, we kindly ask if you could review our rebuttal and updated paper. We are eager to address any additional questions that may arise. Thank you for your valuable support and consideration.

Sincerely,

Authors

Thanks for the detailed questions and constructive suggestions for our work. We will reply to them in detail as follows.

Q1. Testing in live and dynamic domains. The reviewer is concerned that the experimental setup "might not fully reflect the unpredictability of real-world situations" and suggests that we should "test MetaTool in live and dynamic environments like finance or autonomous driving".

A1. While we apply simulated environments for the tool-oriented scenario, real-world APIs and live user queries are used to evaluate our method for the tool-augmented scenario. On one side, given the nature of the tool-oriented scenario that the model needs to observe the new environmental state after each action, it's more feasible and reproducible to develop a self-host environment (e.g. BW and LOG in PlanBench and some tasks in BFCL). Still, it's indeed important to expand the study from simulated environments to real-world environments. And we are looking forward to advancing this process forward. On the other side, it's worth noticing that in the tool-augmented chatbot scenario, over 16k real-world APIs are included in ToolBench and they are connected to the live and dynamic internet such as Instagram and YouTube. BFCL also considers live tasks that the instructions are contributed by real-world users. Thus, the results and evaluations on these benchmarks substantiated the applicability of MetaTool in similar scenarios.

Q2. Guidance for adding new tools and scaling up. The reviewer is concerned that "it might be tricky for people who want to apply it to new tools or unique domains without more support" and suggests "Offering some guidance or a framework for adding new tools". The reviewer also asks "How does this scale up if we’re working with very large toolsets? What’s the computational cost?"

A2. Our approach can be easily transferred to new toolsets in two optional manners: (1) Through instruction tuning on large-scale datasets with a large number of tools (similar to us training MetaTool on ToolBench), the model gains zero-shot generalizability to understand and use new tools according to their documentations. Since our method does not require human annotation, the data synthesis process is easy to scale up by filling actions or states into QA templates. (2) Generate data for new tools or new domains and train a model to master the tools. Here is a specific step-wise guide for adapting new tools in new domains: First, determine if there's solution data that includes successful paths of actions and results. If not, an easy way is to prompt advanced LLMs to trial multiple times (example prompts are showcased in Appendix A.1 in lines 728-755) and pick the paths with successful final results. Second, extract unsupervised tool-use data samples, each of which contains the tuple of action $a$, initial state $s$, and new state $s'$. Third, synthesize the self-supervised meta-task data for each sample by filling the variables into QA templates (showcased in Figure 4). Fourth, augment the solution data with the meta-task data and train the base model through supervised fine-tuning. We will also release an operable codebase to guide this practice.

Q3. Comparison to multi-task or hierarchical learning approaches. The reviewer is curious about the scalability and flexibility of our method "compared to other recent multi-task or hierarchical learning approaches, which also aim to improve model generalization"

A3. One of the most crucial challenges of tool learning is tool/task generalization. We believe multi-task learning or learning with a hierarchical framework has great potential for application in tool learning and are glad to discuss the comparison between them and our method. To address your concern more effectively, could you please specify the particular methods or papers you are referring to? We are looking forward to further discussion.

Q4. Future explorations for MetaTool. The reviewer asks about improving model understanding with more or modified meta-tasks and discusses the potential of "including tasks around probabilistic reasoning or continuous learning to help MetaTool become even more generalizable".

A4. Thanks for the inspirational idea. Developing more task-agnostic meta-tasks can be a promising exploration direction to improve tool understanding and generalization. The motivation behind our meta-tasks is to provide tool knowledge that is transferable across various tasks. A comprehensive set of meta-tasks is defined asking the model to predict each key element (e.g. actions, states, boundaries) of the tool-use process. Besides those fundamental elements, other tool knowledge can also be useful regarding different tools and scenarios. For example, we can ask the model to predict the probability of successfully changing the environmental state facing unpredictable domains and annotate the answer through repeated sampling and analyzing the results.

We thank the reviewer for the constructive concerns. We address them in detail in the following lines.

Q1. Real-world rationales behind 6 meta-tasks. The reviewer asks about the "real-world scenarios of the six settings of meta-tasks" and "specific rationales behind why we should apply this causal inference mechanism to tool use."

A1. The rationale for designing meta-tasks based on the causal theory is that there naturally exists a cause-effect relation between the tool use and its outcome. Regarding the tool-use process as a state transition (as formally defined in section 2.1), the causality can be denoted as $A \rightarrow S' \leftarrow S$, where the arrows represent the causal influences. In particular, action $A$ is an intervention in which we actively affect the state by using a tool, rather than observing the correlation between actions and outcomes, which is theoretically introduced in [5]. Understanding those causalities helps the model understand the tool mechanism better. For example in a real-world scenario (from the BFCL benchmark), taking the mv command in Linux (mv(<original file>, <target directory>)) as a tool, the action of using it actively changes the state of the file system. Learning meta-tasks such as "What's the outcome of calling tool mv with parameters of 'test.py' and '/home/codes/'?" and "What parameters should you pass to tool mv to move the 'test.py' file to directory '/home/codes'?" let the LLM understand the tool mechanism and how to actively achieve a desired outcome. We update more examples of meta-tasks in real-world scenarios in Figure 4 of our revised paper. Please feel free to browse them.

Q2. Comparison with other tool-learning methods. The reviewer suggests that "the novelty and contribution need further justification" and calls for "a comprehensive comparison with a wider range of state-of-the-art tool-learning models including those employing self-supervised learning" and "more baselines on the Berkeley Function-Calling Leaderboard to compare with. Some of them are trained from LLaMA-3.1-8B as well."

A2. We appreciate the concern for a more comprehensive comparison and now further elaborate on our contribution. Firstly, although methods like Toolformer[1] and TALM[2] also adopt the idea of self-supervised learning and do not require extra human annotation, the method we propose is essentially distinct from them. Toolformer embeds the successful tool actions and their results in the model's answer to emphasize 'when' to use tools during question answering. TALM employs an iterative self-play technique to collect successful solutions by exploring with LLMs themselves. Both methods are task-related in that they depend on a stable judgment of whether the task is successful and merely utilize valid actions. On the contrary, MetaTool aims to excavate casual knowledge of tools that exists and is transferable in various tasks, which is theoretically supported by [6]. We also make use of failed tool-use experiences such as the invalid input cases in the input boundary meta-task. This distinction enables us to apply and validate our method in various scenarios.

Secondly, the tasks and tools those methods are tested on are dramatically different from ours. While Toolformer is designed and evaluated considering several typical tools (e.g. QA language model, Wikipedia search, canlendar) and TALM considers only a text-to-text API for QA tasks in two domains, we evaluated MetaTool with over 16k tools for both multi-step planning and multi-round chatbot scenarios. Given that both of them are not officially open-sourced, it's impractical to reproduce them and transfer them to our tasks while ensuring a fair comparison with their original implementations.

Thirdly, the experiment on BFCL undoubtedly demonstrates the zero-shot generalizability of MetaTool. Essentially, we evaluate our methods on BFCL to answer the question: "Can our self-supervised method enable the zero-shot generalization to new tasks and environments?" Comparison with the LLaMA3-solution and LLaMA3-8B-instruct (shown in the table below) verifies the effectiveness of MetaTool in enhancing generalizability. Although there are other models on the BFCL, they don't share the same base model with MetaTool (i.e. LLaMA3-8B-instruct) and most haven't released their methods and datasets for tool learning, which makes the comparison less meaningful and less fair. Also, comparing with models trained from LLaMA-3.1-8B would be unfair since LLaMA3.1 has been trained for tool use based on LLaMA3. Thus we showed the performance of GPT-4-turbo (top-1), o1-mini, and Hermes-2 (the one trained based on LLaMA-3-8B) to show the relative capability of our model as reference.

Q3. Data synthesis explanation. The reviewer claims that "the self-play and tree search lack implementation details" and suggests that "using ReAct to generate thought-tool-input tuples" of solution data should be described in the method section.

A3. Thanks for the suggestion. Overall, the unsupervised data can be extracted by searching the solution data (tree search in ToolBench[3]) or merely prompting LLMs to trial (self-play in TALM[2]). We initially don't describe the implementation of the self-play or tree search approach since we extract unsupervised data from the existing solution data synthesized by ToolBench[3]. Specifically, the solution paths are searched through a Depth First Search-based Decision Tree (DFSDT), which lets GPT-4 access different reasoning paths by choosing either to continue the current node or give up and expand a new node. For clarity, we have revised the paper to formalize the solution data (sequences of thoughts, tools, and inputs) in the method section (line 134) and describe the tree search approach for ToolBench data synthesis in the experiment section (line 357).

Q4. Relation with BERT. The reviewer questions the analogy to BERT that "we are still training the LLMs under an autoregressive objective" and points out that "the model is trained on an augmented/generated dataset using BERT's 'masking' process but is not trained to predict what is missing in the context."

A4. It's true that we are still training with next-token prediction autoregressive loss instead of predicting the masked token in the context. We mentioned the masked language models including BERT and Cloze[4] to elaborate the idea of predicting masked elements in the tool-use process (through meta-tasks). Such an idea shares a similar objective with predicting masked tokens in the context and enables the learning of the lurking knowledge beneath the unsupervised materials. We have clarified the idea above in our revised paper (line 146)

Q5. Other Concerns about "the model version of ChatGPT", "full-parameter training", the Pass Rate metric based on ChatGPT, "directions of metrics", and the arrangement of tables and figures.

A5. We appreciate the questions as well as the useful suggestions. (1) We adopt GPT-3.5-turbo-16k as ChatGPT throughout the evaluation. We evaluate the results of ChatGPT, GPT-4, and Claude-2 officially provided by ToolBench, but unfortunately, we can't find the versions of them in the ToolBench paper. (2) Sorry for the misdirection. We implement Lora training instead of full-parameter training. The "full-parameter" represents targeting all modules with Lora instead of just query and value matrixes. (3) The Pass-Rate metric is also evaluated with the help of ChatGPT to determine whether the query instructions are satisfied. Although the model can call the "Finish" tool to end the task and output a response, it does not necessarily satisfy the user's request. For example, it may respond "Sorry, I'm not able to retrieve the information." and should be evaluated as a failed task. (4) We have optimized our paper regarding the directions for metrics, typos, and paper layout. Please check our revised paper and kindly leave any further suggestions.

[1] Schick, Timo, et al. "Toolformer: Language models can teach themselves to use tools." Advances in Neural Information Processing Systems 36 (2024).

[2] Parisi, Aaron, Yao Zhao, and Noah Fiedel. "Talm: Tool augmented language models." arXiv preprint arXiv:2205.12255 (2022).

[3] Qin, Yujia, et al. "Toolllm: Facilitating large language models to master 16000+ real-world apis." arXiv preprint arXiv:2307.16789 (2023).

[4] Taylor, Wilson L. "“Cloze procedure”: A new tool for measuring readability." Journalism quarterly 30.4 (1953): 415-433.

[5] Pearl, Judea. "Causal inference in statistics: An overview." (2009): 96-146.

[6] Bareinboim, Elias, and Judea Pearl. "Meta-transportability of causal effects: A formal approach." Artificial Intelligence and Statistics. PMLR, 2013.

Dear reviewer oF9y:

I hope this message find you well. We have carefully considered your feedback and have made corresponding improvements to the manuscript. We truly value your insights, and your expertise has greatly contributed to enhancing the quality of our work. Could you please let us know if the revisions meet your expectations? As the deadline for discussion nears, we kindly ask if you could review our rebuttal and updated paper. We are eager to address any additional questions that may arise. Thank you for your valuable support and consideration.

Sincerely,

Authors

We appreciate the reviewer's constructive suggestions and concerns. We address the concerns in detail in the following lines.

Q1. Ablations on hyper-settings. The reviewer suggests that ensuring updating the same parts of model parameters (for baseline LLaMA3-2-stage) and the same "selection of epoch numbers" (we choose 1 epoch for MetaTool) will make results more convincing.

A1. Thanks for the helpful suggestion to make our experiment more solid. We configured those hyper-settings based on experimental results to achieve the best performance for each baseline method. Now we update their ablation results on ToolBench below in the table. (1) Firstly, the results of both two variants of LLaMA3-2-stage (-qv targeting only query and value modules and -full targeting all parameter modules) are shown. The relatively weak performance of LLaMA3-2stage-full (-2.2%/-1.1% on average) suggests that training on metasets targeting full parameter modules may let the model overfit the QA tasks and hinder the subsequent training on solution data. We also observe some failed cases of LLaMA3-2stage-full that output meta-task answers instead of actions during testing. (2) Secondly, we show the results of LLaMA3-solution training with 1 epoch and MetaTool training with 2 epochs. The original LLaMA3-solution is trained with 2 epochs following the original configuration in ToolBench (ToolLLaMA). While early stopping for training merely on solution data harms the performance (-2.1%/-0.5% on average ), early stopping for MetaTool improves the performance (+5.6%/+4.7% on average). The contradiction is actually reasonable since the majority of the training data for MetaTool is QA data of meta-tasks (650k out of 776k). On the one hand, training too much on QA data may cause overfitting (similar to the (1) case) and weaken the ability to plan actions. On the other hand, training on meta-tasks can bring sufficient knowledge about tools. That helps the LLMs to understand the expert solutions and learn the tool-use tasks faster, thus reducing the need for the second epoch training.

In summary, when the baseline settings above are configured to be the same as MetaTool, it shows a more significant advantage over LLaMA3-2stage-full (+17.4%/+9.4% on average) and LLaMA3-solution-1epoch (+10.8%/+7.7% on average). The ablation results make our experiment more comprehensive, provide a fairer comparison, and verify that we have chosen the better hyper-settings for both LLaMA3-2-stage, LLaMA3-solution, and MetaTool. We have revised our paper to include those results in Table 4.

Q2. Training with different update steps. The reviewer poses the concern that "the LLaMA3-solution baselines are updated fewer times (10k*3) compared with other models ((10k+10k)*3)" and suggests that "It would be fairer to train the baseline with more steps and more solution data to ensure similar update steps."

A2. We insist that the comparison with this configuration is fair, since the main contribution of our method is exactly generating additional high-quality data for training without any human annotation. Both LLaMA3-solution and MetaTool are trained with supervised fine-tuning with 3 epochs, thus more training data naturally leads to more update steps. The superior performance of MetaTool verifies the effectiveness of the controlled variable (i.e. additional meta-task data) as well as our contribution. Also, it would be less fair if we train LLaMA3-solution with more solution data, since then the comparison will be 10k meta-task data against 10k solution data.

Q3. Lack of 2-stage results. The reviewer points out the "lack of 2-stage results on ToolBench and BFCL benchmarks, nor LLaMA3-8B-instuct results on BFCL".

A3. Thanks for pointing that out. We now update the results of LLaMA3-2-stage and LLaMA3-8B-instruct on BFCL below in the table and have updated them in our paper in Table 5. LLaMA3-2-stage here is trained on ToolBench first on the 650k meta-task data with 1 epoch then on the 126k solution data with 1 epoch, in order to have the same update steps with MetaTool. We adopt the results of LLaMA3-8B-instruct officially released by BFCL recently and test the LLaMA3-2-stage ourselves. The results show that the zero-shot generalizability of MetaTool is improved in most testing sets compared with its base model LLaMA3-8B-instruct (+4.3% success rate on average). We notice that MetaTool also shows retrogression in testing sets such as multi and live-parallel. That is attributed to merely training in the ToolBench scenario which is similar to the simple testing set (detailed explained in our paper lines 421-427). Training in 2 stages partly reduces the zero-shot ability of the model.

Thanks, this makes results more comprehensive. Similar results should be shown on Toolbench.

I increased soundness score.