TaskBench: Benchmarking Large Language Models for Task Automation

Thanks for your insightful comments. Below are our responses to your concerns:

[W1] Human verification on Back-Instruct Dataset

Response:

Regarding the benchmark's faithfulness and the correctness of generated instructions, we acknowledge your concern. Compared with previous methods, we further employ rule-based and LLM-based checks to filter out low-quality samples. Following your suggestions, we are exploring the integration of human verification to enhance accuracy and reliability. This combined approach will ensure a more robust and faithful benchmark. We conducted a human evaluation of the current version of the dataset and put them in Appendix A.3, please refer to [G1] in General Response.

[W2] Error Analysis on Back-Instruct Dataset

Response:

Thank you for pointing this out. We acknowledge the error in the specific instance you highlighted (id: 10949228). This example illustrates a mismatch of parameter types between VQA and the image-to-text tool: the output of image-to-text is incorrectly linked to the image input of VQA. We conducted a human verification on the constructed dataset, and the results are shown in [G2] of the general response. In our examination, we found the proportion of error cases is nearly 12%. After human verification, we found that our dataset could provide a better quality. We will continue to improve it in two ways:

• Improving the quality of tool graphs. Our analysis shows that these mismatches arise primarily when parameter types are not explicitly defined within the tool graph, leading to incorrect assumptions by the LLM about resource types. To rectify this, we propose introducing resource-type nodes in our tool graph. This will clearly delineate the types of resources each tool handles, ensuring that tool nodes are connected in a way that makes logical sense for the given query. This should significantly reduce such mismatches.
• Perform full human verification. We recognize the value of human verification in ensuring the accuracy and faithfulness of our benchmark. We are exploring ways to integrate this into our process, alongside our automated checks.

[W3] The importance of Evaluating task decomposition as textual descriptions.

Response:

Thank you for your comments and concerns regarding the necessity of evaluating task decomposition. We appreciate your perspective and it is still challenging to determine the most appropriate metric to measure task decomposition. Since no one has explored the evaluation of task decomposition when conducting task automation, we want to explore a new perspective to analyze task decomposition by using textual descriptions. Just as shown in Figure 1 in our paper, the textual descriptions from task decomposition will instruct LLMs to parse tools and predict parameters. Therefore, we think that evaluating textual descriptions can also draw some inspiration to enhance LLMs in the future and reflect the planning capability of LLMs to obtain the decomposed sub-tasks. However, we also acknowledge your concerns that current metrics based on textual descriptions are still not enough to fully reflect the accuracy of task decomposition. We will continue to design better metrics to evaluate task decomposition, including subjective and objective evaluations.

I appreciate your response. Is the dataset in the supplementary materials updated? If so, I would do another pass and will consider raising the score based on the updated dataset.

Thanks for your suggestions. We have further designed rules to filter samples based on the mismatch in the parameter number you pointed out：

• On the DailyLife APIs domain, the ratio of error to total samples is: 30 / 4350 = 0.7 %.
• On the Hugging Face Tools domain, the ratio of error to total samples is: 558 / 8104 = 6.8%.
• On the Multimedia Tools domain, the ratio of error to total samples is: 303 / 5887 = 5.1%.

We have updated the dataset in the supplementary material. We appreciate the thorough examination you provided, which has significantly enhanced the quality of our dataset. Your insights are invaluable to us, and we are committed to addressing any remaining issues.

Thanks for your continuously constructive comments on our paper. We have updated our dataset by further refining our datasets with rule-based and human verification. Since the deadline of the rebuttal is approaching, we are looking forward to your feedback. We are also willing to answer all of your concerns if possible.

Thank you again for your time and effort in reviewing.

Thanks for your insightful comments. Below are our responses to your concerns:

[W1] "task naturalness" in sampling procedure? And add more actual examples in the paper.

Response:

Thanks for your question. Here we mainly consider "task naturalness" in the instruction generation stage since currently, it is challenging to consider "task naturalness" in sampling without any human supervision. Therefore, to improve the naturalness, we adopt LLM-based and rule-based critics to filter out instances that rarely occur in the real world. We will follow your suggestions to introduce "task naturalness" into the sampling stage by using LLM critics. Besides, following your and other reviewers' suggestions, we have conducted human verification on our datasets to detect errors (Please see [G1] & [G2] of General Response) and further improve the quality of our datasets. Besides, we have presented concrete samples in [G1] of General Response and put them in Appendix A.4.1 of the revised version.

[W2] Further (manual) verification to verify the reliability of the benchmark.

Response:

Thanks for your suggestions. We admit it will be better with further human verification. We have conducted human verification and analyzed error patterns in our datasets (Please see [G1] and [G2] of General Response). Besides, compared with previous datasets (e.g., ToolBench) which all used instruction methods, we further conducted LLM-based and rule-based critics (Table 1) to filter out low-quality samples and reported the agreement of LLM rankings under human and TaskBench (Table 7). Due to the limited rebuttal period, we will continue to refine the reliability of our benchmark by using subjective and objective evaluations.

[W3] More discussions or comparisons to the concurrent work (ToolBench).

Response:

Thanks for the suggestion. Due to the page limitation, we have put some discussions between the concurrent work in the appendix in the initial version. To better understand the differences between TaskBench and other concurrent work, we have conducted in-depth comparisons in Appendix A.2 of our latest version from the perspective of datasets and evaluations. Please see [G3] in General Response to the differences and the advantages of our method.

[Q1] Is there an example of the constructed tool graph before subsampling?

Response:

Thanks for your question. In Appendix A.10 of the revised version, we have added a visualization of the constructed tool graph on Multimedia domain for reference, where the nodes denote tools, and directed edges indicate that the output type of the source tool matches the input type of the target tool.

[Q2] In Figure 3, GPT-4 almost reaches 100% in performance, Will the benchmark be saturated soon, perhaps when the next generation of LLMs comes out?

Response:

Thanks for your question. We guess this is because our evaluation dataset is also generated by GPT-4, so that its performance will be overestimated when also using GPT-4 for testing. Therefore, we also involve human evaluation to validate its quality. Besides, we believe that when more powerful LLMs come out, the dataset generated by these LLMs will be better and demonstrate better alignments to human preference when compared with GPT-4.

Thanks for your constructive suggestions on our paper. Following your suggestions and other reviewers’ comments, we have updated our dataset by further refining our datasets with rule-based and human verification. Since the deadline of the rebuttal is approaching, we are looking forward to your feedbacks and examinations on TaskBench. We are also willing to answer all your concerns if possible.

Thank you again for your time and effort in reviewing.

[W2] Evaluating a Broader Range of LLMs, Including Claude-2.

Response:

Thank you for your valuable suggestion. In response, we have incorporated the evaluation results of Claude-2 into our analysis. From Table 5-9, we observed that Claude-2 exhibited superior performance compared to gpt-3.5-turbo, yet it did not match the capabilities of GPT-4. Due to the limited time of the rebuttal period, we will add more diverse LLMs to our paper in the final version.

Table 5. Task Decomposition Performance of Claude-2.

Table 6. Tool Invocation and Parameter Prediction Performance of Claude-2 in node-structure Tasks.

Table 7. Tool Invocation and Parameter Prediction Performance of Claude-2 in chain-structure Tasks.

Table 8. Tool Invocation and Parameter Prediction Performance of Claude-2 in DAG-structure Tasks.

Table 9. Tool Invocation and Parameter Prediction Performance of Claude-2 in All Tasks.

[W3] A case study to show the performance gap among different LLMs.

Response:

Thanks for your suggestion. We added a case study to show the performance difference between different LLMs, please refer to General Response [G2] and Appendix A.4.2 in the revised version.

[W4] Discussion with ToolLLM.

Response:

Thanks for the suggestion. Due to the page limitation, we have put some discussions between the concurrent work in the appendix in the initial version. To better understand the differences between TaskBench and other concurrent work, we have conducted in-depth comparisons in Appendix A.2 of the latest version from the perspective of datasets and evaluations. Please see [G3] in General Response to the differences and the advantages of our method.

Thanks for your insightful comments. Below are our responses to your concerns:

[W1] More Analyses of evaluation results.

Response:

Thank you for your feedback. We acknowledge your concerns regarding the perceived partiality in our analyses. To resolve your concerns, we have conducted further analyses of the aspects you highlighted:

1) Factors Contributing to Task Automation Performance:

• Reasoning: The capacity for complex problem-solving and reasoning varies significantly among LLMs. This ability is crucial for understanding and executing complex tasks. In mathematical and coding tasks, GPT series models show stronger reasoning ability, therefore they also possess corresponding capabilities in task planning and tool utilization.
• Instruction Following: The proficiency in comprehending and adhering to user instructions differs, influencing task execution efficiency. Our analysis reveals that models fine-tuned with instruction-following settings, such as Vicuna-13b, WizardLLM-13b, and Nous-Hermes-13b, outperform the baseline Llama-2-13b model in task automation. Specifically, we observe that WizardLLM-13b can perform better than Vicuna-13b since it fine-tunes more complex instructions. These results also demonstrate the necessity of instruction following.

2) Intrinsic Differences of LLMs in Performing Agent Tasks:

• Code Pre-training: We note that the models (Code-Llama) with more code-pretraining outperform other open-source LLMs in task automation. Experimental results show an average improvement of 4.45% in tool prediction (n-f1) and 12.76% in parameter prediction (v-f1) across various domain datasets. We conclude that since task automation usually involves multiple stages and tools, using code-style / structural text as the interface to switch different stages is better.
• Alignment Techniques: Besides, the models (e.g., GPT series models) with human alignments (e.g., RLHF) demonstrate stronger task automation capabilities than open-source LLMs. These results also indicate that RLHF allows large language models to develop more generalized reasoning abilities, reducing overfitting to specific instructions.

Based on our analysis, we propose the following improvements for open-source LLMs:

• Enhanced Code Pre-training: Strengthening code pre-training can significantly boost the performance of LLMs in task automation.
• Quality of Instructions: Developing high-quality (complex and diverse) instructions is crucial for crafting more efficient LLMs.
• Ongoing Knowledge Acquisition: Encouraging continuous learning and knowledge updating can make LLMs more adaptable and proficient.

We have incorporated these additional analyses in Section 3.5 of the revised version to offer a more comprehensive view of the capabilities and development of LLMs in task automation. This should provide a deeper understanding and address the concerns you have raised about the initial shallowness of our insights.

Thanks for your insightful comments. Below are our responses to your concerns:

[W1] In terms of data quality, it would be more insightful to show the quality measured by humans.

Response:

Thank you for your suggestion. In the latest version, we have conducted human evaluations on TaskBench to demonstrate its quality in terms of several aspects, including the Naturalness and Complexity of the instructions, and Alignment of the tool invocation graphs. Please see Appendix A.3. The results show that our Back-Instruct strategy produces high-quality user instructions with annotated and executable tool graphs. This approach generates data that is better aligned with human preferences. For more details about the evaluation results, please refer to [G1] in General Response.

[W2] More examples (prediction and gold answers) to understand the difficulty of each task and error cases.

Response:

Thanks for your suggestions. To resolve your concerns, we have added more examples in Appendix A.4 to illustrate the difficulty and error cases in conducting task automation based on TaskBench. Please also see our response in [G2] for case studies and error analysis.

[Q1] Compared to the node prediction, the edge prediction is harder based on the evaluation results. What are the error types for edge prediction?

Response:

Thanks for your question. Yes, edge prediction is more challenging compared to node prediction because it requires accurate prediction of both the source and target nodes of an edge. Currently, we summarize the main error types in edge prediction as two parts:

• Mismatch of resource types between source node output and target node input.
• Missing edge prediction between nodes where a resource or temporal dependency exists.

We analyzed 100 randomly sampled examples of edge prediction failures and statistically 27% and 38% of edge prediction errors belong to the above two types respectively.

[Q2] Does the model make equivalent edge prediction but different connectivity?

Response:

Generally, if the model has made an equivalent edge prediction, the nodes of this edge will always keep the same connectivity. Besides, in our dataset, such a scenario is rare since we do not involve any repeated calls to a specific tool, leading to unique edge predictions at distinct tool nodes.

[G3] Comparison with ToolBench [1-2]

Response:

Following the suggestions of Reviewer BEYM and peHW, we also conducted a detailed analysis to discuss the differences between ToolBench and TaskBench (please see Appendix A.1). We summarize these differences from the perspectives of datasets and evaluations:

1. Datasets: Although both ToolBench and TaskBench have adopted instruction methods to synthesize samples, the mechanisms of these two methods are different. For ToolBench, it just randomly samples some APIs and then uses LLMs to prompt them to generate the final instructions. As a result, it cannot obtain the ground truth and then involve Solution Path Annotation which uses a depth-first search-based decision tree to obtain answers. However, such a strategy also brings new issues like costs and hallucination or bias in answer annotations. However, due to the design of Tool Graph, TaskBench demonstrates more advantages when compared with ToolBench:
   • Efficiency: Our method requires only one API call to generate a complete sample (creating instructions, generating a tool invocation graph, and checking). In contrast, ToolLLM requires one API call to generate instructions and an average of four inference steps during DFS for annotation. Additionally, ToolLLM uses few-shot learning, which consumes more tokens than our zero-shot approach.
   • Higher Quality of Instructions: Our tool graph includes potential tool invocation relationships (resource and temporal dependencies). Based on the tool graph, the instructions generated by ChatGPT are more in line with actual human behavior. As we demonstrated in our [G1] human evaluation, the generation based on a tool graph with edges significantly improves Naturalness, Complexity, and Alignment compared to generation without edges.
   • Stability: ToolLLM’s instruction generation might not cover all sampled instructions based on the API set. Our method not only covers all sampled tools but also strictly follows the dependency between tools. We can control instruction generation through the sampling process, such as the number of tools and the topology of the tool graph.
   • Reliable Annotations: In ToolLLM, instruction generation, and tool invocation annotation are independent processes. However, In our approach, the sampled tool graph can directly be utilized as the annotations for the final assessment. Hence, our Back-Instruct can directly generate instructions with better alignments to the annotations and does not need to generate answers anymore. Besides, we also utilize the consistency between the sampled tool graph and the generated tool invocation graph to further filter out low-quality annotations, ensuring high-quality tool graph annotations.
2. Evaluation: ToolBench introduces ToolEval in their paper. However, ToolEval just directly follows AlpacaEval, which uses win rate and pass rate to evaluate the capability of LLMs, and cannot manifest any characteristics of LLMs in tool utilization and planning. Our evaluation encompasses multiple aspects, including task decomposition, tool prediction, and parameter prediction. For each aspect, we have developed multiple well-designed metrics to assess the different capabilities required for task automation.

[1] Tool learning with foundation models. Qin et al. 2023.

[2] ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. Qin et al. 2023.

[G2] Need more error analysis and case studies on dataset construction and predictions

Response:

Thank you for the insightful comments. In response, we have conducted a detailed error analysis on both the TaskBench dataset and the predictions made by various LLMs. Our findings are outlined below:

1. Error Analysis on TaskBench Dataset

Despite the advanced instruction generation and labeling capabilities of GPT-4, we admit that it is challenging to guarantee the correctness of all generated samples. To better understand our dataset and assess its accuracy, following the suggestions of Reviewer LVvj and CSeK, we conduct human evaluations to provide a thorough error analysis. Here, we first randomly sampled 148 samples, and our labeling team identified 18 error samples (nearly 12%) from the sampled data. We attribute these incorrect samples to five distinct error categories, and concrete examples of each are presented in Table 3:

• Incorrect Instructions
  • Incomplete instructions: This error occurs when the instructions lack the necessary details or resources for successful completion.
  • Impractical instructions: The instructions could be irrational or impossible to execute with the capabilities of current tools.
• Parameter Errors
  • Mismatched parameter types: This error occurs when the parameters provided do not match the expected types for the used tool.
  • Incorrect parameter value: This error is evident when the values provided for the parameters are incorrect or not suitable for the task.
• Incorrect Tool Dependency: This error type refers to the incorrect linking or sequencing of tools required for a task.

Table 3. Error Analysis on Back-Instruct Dataset.

Based on these observed errors, we conclude that it is necessary to build a more elaborate prompt (e.g., more detailed tool-use specification and demonstrations) to describe tool parameters and tool dependencies when generating the tool invocation graph. Besides, we will also introduce more high-quality criteria to continuously improve our dataset in addition to our rule-based and LLM-based critics.

Moreover, we further draw some cases to intuitively show the differences between the three methods, as shown in Table 2. From these examples, we observe that our Back-Instruct can generate examples with more comprehensive and interconnected tool usage, reflecting higher naturalness and complexity in instruction generation.

Table 2. Comparative analysis of Back-Instruct, Back-Instruct w/o edges, and Self-Instruct.