CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets

We thank the reviewer for thinking our work makes an important contribution and also acknowledge the potential in our future work proposal. We are terribly sorry for giving late author responses because we are running some additional experiments, which consume a lot of time for autoregressive generation. The following is our response.

Weakness

1. Can the authors provide such results (the performance of different LLMs in creating and using tools) in the experiment?

In our experiments, we try CodeLLaMA-13B and observe that they can only achieve near-random performance on the visual tasks, as demonstrated in the table.

2. I suggest that the author should make the distinction between more specific methods more prominently in the main text.

Thanks for your advice. We summarize the distinctions between our work and previous work regarding tool creation in the following table:

The major difference between CRAFT and LATM is that LATM uses three training samples to create only one tool for one dataset, and directly apply the tool to all downstream test samples, which is neither scalable nor generalizable. Hence, LATM cannot tackle tasks that contain diverse patterns. In contrast, with the help of our retrieval method, we are able to scale up the tool set and address various problems. Experiments in Table 1 have demonstrated that CRAFT achieves better performance.

We thank reviewers for thinking that our approach is flexible to augment LLMs compared to traditional ones. The following is our response.

Weaknesses

1. The setting of the experiments is a little bit limited.

Response: Thanks for the advice. We did not include recent agent benchmarks, such as MINT and AgentBench, in our paper since they came out concurrently with CRAFT. We are also interested to see how CRAFT can benefit agents, but due to the time limits in the author response period, we are unable to adapt CRAFT to those benchmarks and finish the evaluation in time. However, since there is some overlap between our task and MINT/AgentBench, we expect the trends in our submission will translate to those in these two benchmarks.

2. The comparison with LATM is a little bit unfair.

Response: In fact, for fair comparisons, CRAFT follows exactly the same setting as LATM, and creates tools using GPT-4 and uses tools with GPT-3.5-Turbo. We will highlight this in the revision.

3. The transferability of the toolset should be discussed.

Response: Thanks for the suggestion. We have tried to mix the tools of different tasks into a single set and observe that this has little impact on the downstream task performance. We think this can be attributed to the retrieval approach in CRAFT, which will only select tools that are most useful for the problem. We will include this in the revision.

We are glad the reviewer thinks CRAFT addresses crucial challenges in augmenting LLMs. The following are our authors' responses.

Weaknesses

1. The paper could benefit from a more detailed exploration of scenarios where CRAFT might not perform as expected.

Response: Thanks for the advice. We believe that such a systematic analysis is interesting, but during this response period, we failed to find an automatic method to identify and categorize them. Therefore, it may be beyond our capacity to finish the analysis in such a short time, but we will try to figure it out in the revision when time is available.

We identify one specific failure mode in the CRAFT framework. In the abstraction process, there are some cases of successfully renaming the function names and writing a general docstring, and thus the tools look generalizable outside. However, inside the function, the specific variable names or textual inputs may fail to be converted into generic ones (expected results are shown in fig.2, replace all specific variable names with general ones, e.g., cat→animal, desk→object) and wrap textual inputs of internal function calls as arguments of the tool (e.g., date = df["date"]→date = df[column name], where the value of column name is passed in by tool users). Therefore, there are still some question-specific input-output formats. That said, the tool functions can only tackle a specific problem, instead of all problems of the same type. Such a mismatch between tool description and its implementation of the functionality could lead to errors when LLMs tend to call the tool to solve other problems. A potential solution from the tool side is to improve LLMs’ instruction-following ability so that they can abstract tools better, or introduce an extra verification process to filter out tools that “overclaim” their original functionality. In this case, we can further improve the quality of the created toolsets.

2. Expanding the comparison to include a wider range of existing tools and frameworks.

Response:

We thank the reviewer for their constructive feedback. We have added four baselines on TabMWP and MATH. VQA, a third task we consider, takes more engineering efforts. As a result, we were not able to complete this experiment on VQA given limited time. Concretely, ‘GPT-3.5-Turbo Vanilla’ directly asks GPT-3.5-Turbo model to solve the problems; Program-Of-Thought (POT)+ Chain-Of-Thought (COT), on top of POT, adds COT prompting at the beginning; POT+Rectify introduces a rectification process when error occurs in POT programs; POT+Rectify+COT combines the above two techniques.

Details on rectification: If an error occurs, then the LLM is prompted with demonstrations to correct the error. Applying a similar prompt format as before, the format of demonstrations “[EXAMPLE x]” now changes to “### Question [QST]\n ### Original [ORI]\n ### Error [ERR]\n ### Rectification [REC]”, where we provide the original tool implementation and calling decision in [ORI], offer the error tracebacks [ERR], and concatenate natural language reasoning on the error with the rectified code in [REC]. In summary, both COT and the Rectify process can enhance POT on these two tasks, and their combination may further boost POT performance on TabMWP. However, CRAFT still outperforms all of them on both tasks.

3. The paper could provide a more in-depth analysis of the tool creation and retrieval components of CRAFT.

Response: Thanks for the suggestion. For tool creation, how different types of tools contribute to performance improvements depends on what types of patterns exist in downstream tasks. As shown in our analysis section, there are tens of hundreds of types of tools in the toolset, so ablating different types of tools one by one would be extremely time-consuming and costly. Hence, we conduct a simpler experiment by removing the five most frequently retrieved tools from the MATH toolset and evaluating model performance on the reduced tool set. We found that the accuracy on MATH drops from 68.1 to 58.9, indicating that these tools are very useful to the task.

For retrieval, the high-level strategy is task-agnostic: We prompt Turbo to “describe what it needs” by inferring the name and docstring of tool functions that might be helpful for each problem, and then retrieve tools that are most similar to the expected ones from our tool sets to solve the problem. However, to improve the precision of retrieval, we design task-specific prompts as shown in Appendix B.5, mainly characterized by providing a list of “most frequently used words” in the toolset so that Tuobo-inferred names can better match the tools in our sets.