What Are Tools Anyway? A Survey from the Language Model Perspective

Thank you for your recognition of the clarity, comprehensiveness, and insightfulness of our work!

• R1: Is survey suitable for COLM As stated in the FAQ of the official COLM website (https://colmweb.org/faq.html): “We don't have a special track for survey papers. If you wish, you can submit a survey paper and we will route it to the appropriate area chairs and reviewers and based on their judgment, you will get a decision.” From this, we infer that COLM accepts survey papers and will find the most helpful ACs and reviewers for them. Therefore, we believe our survey paper is appropriate for COLM.

Thanks for your response, I will keep my score!

Thank you for your recognition of the various aspects covered by our survey!

• R1: RAG is tool We agree, and we have explicitly mentioned that “RAG is a tool” in the paper: we exemplify RAG as a “knowledge access” tool in Table 1, and we discuss RAG in Sec 4.1 the “knowledge access” part.

• R2: Merge into one category We would like to clarify that not all these tools are to obtain knowledge. E.g., make_post() tool from “interact. w/ world” takes actions to change the world instead of obtaining data. So merging these categories would not be appropriate. We group tools in a fine-grained way to highlight their uniqueness and provide more insights. E.g., readers can learn that “skilled LM” tools may not always do tasks correctly, which is not true for get_time() from “interact. w/ the world” that always perfectly solves the task. But readers may not grasp this insight if we merge these tools. Nonetheless, readers are free to further interpret based on our discussions.

• R3: Lack bottleneck discussions We discuss multiple bottlenecks of tools across the paper: tasks that tools cannot easily solve in Sec 4.2 and 5.1; tasks that no existing tools can solve in Sec 5.3; limitations in collecting/evaluating tools in Sec 6.1. Lastly, we provide an empirical analysis in Sec 7 on accuracy-efficiency performance, responding to the reviewer's “how well they perform” question.

In general, we believe our work has sufficient discussions for the above points regarded as missing by the reviewer, and has clear contributions in definition, taxonomy, evaluation, and empirical study of tools. We would appreciate it if the reviewer could reconsider these aspects. We are happy to take any more feedback!

• Q1: why programmatic tools Since programming is also a popular medium of tool use, we find it critical to clarify how program and tool relate to each other.

• Q2: Guidance to create tools Please refer to Sec 5.3 for details on various tool-making testbeds/methods and use the most suitable one. From our empirical analysis in Sec 7, we recommend TroVE when w/ data/compute limitations, and CRAFT otherwise.

• Q3: Benefits of tools As we stated in Sec1 and Figure1, tools can help LM task-solving (do math accurately) and extend LM abilities (know the current time), which the strongest LM, even with fine-tuning, is fundamentally unable to do. We would like to note that RAG is one form of tool-using, so RAG already benefits from using tools.

Thank you for your recognition of the importance and quality of our work!

• Q1: Better caption for Figure 1 Thanks for the feedback. Our figure1 intends to illustrate the text paragraphs around it, to emphasize how (calculator/api) tools come naturally to address two example limitations of current LMs (in math and in getting up-to-date information). We will cite figures1&4 and make their captions clearer in the revised version.

• Q2: Replace section sign with ‘section’ Thank you, we will update this in the revised version!

Thanks for your recognition of the importance and usefulness of our survey paper!

• R1: survey lacks some works Thank you for pointing out these works! First, the three works you pointed out were published within three months of our publication date, so we didn’t get a chance to include them all, but we are happy to add them to our revised version. Also, thanks for pointing out the newsletter source. We believe our paper has discussed most of the papers mentioned in it, but we are happy to include the earlier papers left in our revised version. (Specifically, the only tool-related earlier paper left out is Neural Programmer-Interpreters (Reed and Freitas, 2015), and we will add it)

In general, we believe we have provided a comprehensive scope of tool-related topics, and thank you again for providing these additional papers to further enrich the content of our paper!

• R2: reviewer did not learn enough Thank you for your feedback! Our survey aims to serve as a useful handbook for LM-used tools. Compared to existing methodology or survey works, our paper uniquely points out the definition of tools (section 2), a clear taxonomy of tools (section 4), a systematic summary of testbeds (section 5), and most importantly, proposes five essential evaluation aspects (section 6, along with empirical study about them in section 7) that have not yet been evaluated in tool-related literature. We particularly discuss “different tool-use methods” in section 3 and 5. We did not spend more content on it due to content length limitations and the importance of other aspects (as mentioned above). However, because tool-use methods to date are mostly homogeneous (e.g., train then test), this lack of diversity may also give the impression of a lack of pros-and-cons discussion. In addition to these aspects, if you have other specific topics of interest, feel free to point them out and we are happy to try to incorporate them in our paper.