ToolBridge: An Open-Source Dataset to Equip LLMs with External Tool Capabilities

We appreciate the reviewer’s thoughtful comments and suggestions. Below, we address each of the reviewer’s points in detail.

Comment 1: The Figure 2 needs some revision.

In the rebuttal revision, we have updated Figure 2 to include additional processing details for improved clarity and completeness. The revised figure now outlines the following key steps:

• Collection and Reformatting of SFT Datasets.
• Estimation and Ranking of Datasets.
• Valuable data entries selection: Utilizing LLama3-70B to identify appropriate data entries.
• Valuable data entries conversion: Leveraging GPT-4o-mini to insert Python code during the conversion of valuable data entries.
• Valuable Data Entries Filtering: Performing consistency checks to filter and validate the final set of valuable data entries.

These additions provide a more detailed and transparent depiction of the processing pipeline and thereby make Figure 2 more informative.

Comment 2: Only llama models are used in experiments.

In the rebuttal revision, we have also included the results of Mistral-7B in Tables 6-10. These results encompass:

• Ablation studies on GSM 8k, GSM Plus and MathBench$^*$

• Ablation studies on Stanford WebQA

• Ablation studies on TruthfulQA

• Experimental results on RandomQA

• Experimental results on FACT

Collectively, these results provide strong evidence supporting the effectiveness of ToolBridge in various scenarios and tasks.

Comment 3: Source datasets and a brief description for each.

In Appendix A.6 of the original paper, we provided a detailed list of all source datasets along with their respective download links used in constructing ToolBridge. The rebuttal revision enhances this section by including concise descriptions for each dataset.

Comment 4: Less data processing templates in RandomQA.

In Appendix A.7 of the original paper, we provided 50 templates for constructing RandomQA, comprising 20 related to data processing and 30 related to numerical computation. To address the reviewer's comment regarding the imbalance, we have added 10 additional templates focusing on data processing in the rebuttal revision. The newly added templates include:

• Filter elements in a list based on a condition,
• Merge two dictionaries into one,
• Extract all words of a specific length from a text,
• Extract email addresses from a text,
• Sort a list of strings by their length,
• Check if two strings are anagrams,
• Extract hashtags from a social media post,
• Capitalize each word in a string,
• Find the index of a substring in a string,
• Replace all vowels in a string with a specific character.

Furthermore, Table 9 has been updated to evaluate the performance of the data entries generated from the revised set of 60 templates, covering both data processing and numerical computation. The results of this evaluation are as follows:

Here, RandomQA-DP-B1/2 refers to data generated using templates related to data processing, while RandomQA-NC-B1/2 corresponds to data generated using templates related to numerical computation. These results demonstrate that LLMs effectively learn to leverage external tools, thereby enhancing their capabilities in fundamental tasks such as data processing and numerical computation when trained on ToolBridge.

Comment 5: Evaluate on existing factual datasets

In response to the reviewer's comment, we have included Table 8 in the rebuttal revision to evaluate the model's performance on TruthfulQA, a widely used factual dataset. The evaluation results are as follows:

These results demonstrate that ToolBridge can enhance the factual retrieval capabilities of LLMs, as reflected in the improved ROUGE1 and BLEURT scores.

Thank you so much for addressing my questions. I believe the score appropriately reflects the merits of the work.

We sincerely thank the reviewer for the insightful comments and constructive suggestions. Below, we have provided detailed responses to each point raised in the review.

Comment1: Comparison with Function Calling Datasets such as ToolBench

We would like to clarify that the dataset proposed in this paper is designed specifically for tool former approaches, which address a distinct problem space compared to function calling. These two paradigms serve different objectives and employ fundamentally different methodologies:

• Function calling focuses on enabling LLMs to act as agents that invoke APIs without requiring explicit training on their usage. This is achieved through curated datasets and standardized evaluation protocols that assess and compare the tool-use capabilities of LLMs. Function calling emphasizes streamlined API integration to perform predefined tasks effectively.

• Tool former, on the other hand, seeks to empower LLMs with the capability to autonomously invoke and utilize external tools during inference. This approach emphasizes advanced reasoning, enabling LLMs to independently determine which external tool to call, the optimal timing for the call, and how to parse and integrate the execution results systematically into their generated responses.

Given these differences, ToolBridge is not directly comparable to function calling datasets such as ToolBench, as their objectives and methodologies differ significantly.

The primary contribution of this paper, to the best of our knowledge, is the first open-sourcing of training data specifically tailored for the tool former domain. This dataset enables LLMs to learn how to autonomously and effectively utilize external tools. We have explicitly addressed this distinction in the rebuttal revision, particularly in lines 40–49 of the Introduction section and lines 129–161 of the Related Work section, to better highlight our contributions and mitigate potential misunderstandings.

Comment2: Only 63 Python packages included in ToolBridge

First, this paper focuses on tool former, not function calling, and thus a direct comparison with ToolBench is not appropriate or meaningful. These two approaches target fundamentally different objectives, as outlined in our response to Comment 1.

Second, the number of Python packages included in ToolBridge is not directly indicative of the dataset's breadth or functionality. A single Python package can encompass a wide range of functions with diverse capabilities. To provide a clearer perspective on the dataset’s richness, we conducted a detailed analysis and found that ToolBridge includes a total of 125,651 distinct function calls. This reflects the dataset's considerable scale and functional diversity.

Comment3: More Detailed Explanation of the Fact-checking Process

The detailed inference process of LLMs trained on ToolBridge is outlined in Algorithm 1 of the original paper. Briefly, ToolBridge enables the model to autonomously generate external link addresses required for retrieving factual information.

For example, given the query, "What county is Farmington Hills, MI in?", the model generates the following Python code:

import requests
from bs4 import BeautifulSoup

response = requests.get('https://en.wikipedia.org/wiki/Farmington_Hills,_Michigan')
soup = BeautifulSoup(response.content, 'html.parser')
county = soup.find('th', text='County').find_next_sibling('td').text.strip()

print(county)

Upon detecting special tokens <python> and </python>, the Python interpreter executes the code. The requests and BeautifulSoup libraries retrieve and parse the webpage to extract the county name, which is then returned within <result> tags (i.e., <result>Oakland</result>). This result informs subsequent text generation, demonstrating how ToolBridge enhances factual accuracy by leveraging external tools.

Comment4: Evaluate the factual retrieval abilities of LLM against established benchmarks

We appreciate the reviewer’s suggestion and have included an evaluation of the model's factual retrieval abilities using the TruthfulQA benchmark in the rebuttal revision (Table 8). The results are summarized below:

The results demonstrate that ToolBridge can improve the factual retrieval capabilities of LLMs, as reflected in the higher ROUGE-1 and BLEURT scores compared to baseline models.

We greatly value the reviewer’s thoughtful comments and helpful suggestions. Below, we offer detailed responses to each of the concerns raised.

Comment1: Run Llama 3 70B filtering twice

We employed a two-stage filtering process with Llama 3 70B to balance computational efficiency and maximize the retrieval of valuable data. In the first stage, a subset of each dataset is sampled to estimate the proportion of valuable data (i.e., suitable for invoking external tools to help LLMs’ reasoning process) within that dataset. This preliminary assessment allows us to rank datasets based on their relative quality. In the second stage, we conduct a full-scale filtering process only on the datasets identified as having a high proportion of valuable data from the first stage.

This staged approach addresses the significant computational challenges associated with the sheer volume of candidate datasets provided by the community. Running the Llama 3 70B filtering process directly on all data across all datasets would be prohibitively time-consuming. By focusing the second round of filtering on high-quality datasets, we achieve significant efficiency gains. Specifically, for the current scale of data, this approach reduces the total inference time by approximately 40%.

Comment2: Review Process of SFT Datasets

The integration of Supervised Fine-tuning (SFT) datasets presented in Table 1 was primarily carried out by conducting keyword searches (e.g., "supervised fine-tuning dataset", "SFT dataset", "instruction tuning dataset", and "LLM dataset") across platforms including Google Scholar, Hugging Face and GitHub. Representative search results include the following resources:

• https://github.com/Zjh-819/LLMDataHub,
• https://github.com/RenzeLou/awesome-instruction-learning,
• https://github.com/raunak-agarwal/instruction-datasets,
• https://github.com/zhilizju/Awesome-instruction-tuning,
• https://arxiv.org/abs/2402.18041,
• https://arxiv.org/abs/2402.06196.

Based on these resources, we conducted a manual review of all referenced datasets, including verifying whether the dataset qualified as an SFT dataset, assessing its open-source availability, identifying potential overlaps with existing collected datasets, and examining other potential concerns, such as copyright issues. At last, we derived Table 1.

To enhance transparency and reproducibility, we have included a detailed explanation of this process in Appendix A.13 of the rebuttal revision. Additionally, we have revised the phrasing in the manuscript to replace "a comprehensive review" with "a review" to prevent potential misinterpretation regarding our study.

Comment3: Sampling ratio for obtaining $\mathcal{S}_i$ in Section 3.1

To obtain $\mathcal{S}_i$, we sampled 1% of examples from $\mathcal{D}_i$ for each dataset. We have clarified this point in the rebuttal revision (line 200) to enhance transparency and ensure reproducibility of our methodology.

Comment4: Selection of $N$ in Section 3.1

By default, we randomly sample $N = 100$ data entries from $\mathcal{D}_i$ for manual review to identify the presence of irrelevant characters or content. This number has been clarified in the rebuttal revision (line 210) to enhance the transparency of our pipeline. The choice of $N$ is informed by two key considerations: (1) the cost of manual review, as excessively large values of $N$ would impose a substantial workload; and (2) the observation that datasets constructed through methods such as web crawling typically have a high likelihood of including irrelevant content. Based on our empirical experience, we find that using a relatively small $N$ introduces minimal impact on the overall results, as larger values do not significantly alter the identification of irrelevant entries.

Comment5: Different batches of FACT

The data for the three batches of FACT were generated using GPT-4o with distinct prompts. Specifically, the first five prompts listed in Appendix A.8 were utilized to construct Batch 1, the next five prompts were used for Batch 2, and the final five prompts were employed for Batch 3. This process has been explicitly clarified in lines 513 to 519 of the rebuttal revision to ensure transparency and to avoid any potential confusion regarding different batches of FACT.

Comment6: Typo Issues

We have addressed and corrected the typo issues identified in lines 187, 208, 359, and 484 of the original paper. These corrections have been incorporated into the rebuttal revision to ensure clarity and accuracy in the revised manuscript.

Comment7: Qualitative Results Section

We appreciate your insightful suggestion. In the rebuttal revision, we have removed Section 3.5 from the main text and relocated the example inputs and outputs to Appendix A.9 to improve the manuscript.

Comment6: Potential Bias Issues

We would like to clarify that the original data in ToolBridge is sourced from community-contributed datasets and not generated by GPT-4o. GPT-4o’s role is strictly limited to inserting tool calls into the original data at locations where external tools can enhance the reasoning process of LLMs. Importantly, this process is performed on existing data and does not alter the underlying content or its distribution, mitigating concerns about systematic biases being introduced.

To further address the concern, we conducted additional experiments detailed in Appendix A.11 of the rebuttal revision. Specifically, we used Google Gemini, an alternative to GPT-4o, to independently generate three batches of the FACT dataset (referred to as Gemini-FACT-B1, B2, and B3). These datasets are entirely independent of GPT-4o and ToolBridge, serving as an additional check for bias. The evaluation results on these datasets are summarized below:

The results indicate that models SFT with ToolBridge data consistently outperform their non-ToolBridge counterparts on datasets prepared independently of GPT-4o. This provides strong evidence that the observed improvements are not tied to potential biases introduced by GPT-4o and are instead attributable to the effectiveness of ToolBridge.

Comment7: Limited accuracy gaps between ToolBridge and ToolBridge$^§$ on standard benchmarks

We thank the reviewer for highlighting the small performance differences between ToolBridge and ToolBridge$^§$ on standard benchmarks. We would like to address this concern by clarifying the fundamental reasons behind these results and situating them within the broader context of our work.

• Task-Specific Focus of ToolBridge. The primary objective of ToolBridge is to enable LLMs to effectively leverage external tools to augment their reasoning processes. This focus diverges from that of many existing standard benchmarks, such as GSM8K, which prioritize evaluating a model’s intrinsic reasoning abilities, particularly in multi-step computation tasks. In these cases, tool usage often contributes only marginal gains, as the benchmarks predominantly assess a model’s reasoning aptitude rather than its tool-integration capabilities. Consequently, the observed limited performance gaps align with our expectations and underscore the distinct objectives of our work and those of standard benchmarks.

• Absence of Standardized Tool-Usage Evaluations. A significant limitation in the current landscape of LLMs evaluation is the lack of established benchmarks specifically designed to measure tool-usage capabilities (in the domain of tool former). For instance, even state-of-the-art models like Llama 3.1 are assessed on tool integration primarily through zero-shot evaluations on public datasets (sec. 5.2.7). This underscores the nascent nature of tool-usage evaluation as a research direction and highlights the need for tailored evaluation frameworks.

• Benefits of Tool Use in Multi-Step Computation. While the performance gains on existing benchmarks may appear small, enabling tool use can still provide advantages in tasks requiring multi-step computation. For instance, external tools can assist LLMs in handling intricate calculations or processing extensive data.

In conclusion, the limited performance gaps on standard benchmarks should be interpreted in the context of ToolBridge’s broader objective: pioneering the integration of external tools to enhance LLM reasoning. We view this work as an important step toward addressing the open challenge of systematically teaching LLMs to utilize tools effectively.

We are grateful to the reviewer for the valuable and insightful feedback. Below, we have provided a thorough and detailed response to each of the comments and suggestions raised.

Comment1: Relies on Existing SFT Datasets

We would like to clarify that the algorithm for constructing ToolBridge is a flexible and adaptable pipeline. This pipeline is capable of processing raw web data by transforming the data into structured QA pairs with the assistance of LLMs like GPT-4o. Here is an example of QA pairs obtained by using GPT-4o to transform the raw web data from the publicly available C4 dataset,

• Question: If a customer deposits RM100,000 into a Citibank Time Deposit account, how many partial withdrawals of RM5,000 can they make before the maturity date?
• Answer: They can make 20 partial withdrawals of RM5,000 each (RM100,000 ÷ RM5,000 = 20).

Through this transformation, the pipeline enables the subsequent selection, conversion, and filtering of data entries, ultimately ensuring the generation of high-quality entries for the ToolBridge dataset. To provide further clarity and transparency, we have included a detailed explanation of the transformation process, along with the specific prompts and example input data, in Appendix A.10 of the rebuttal revision.

Comment2: Evaluate on Tool-Use Benchmarks

We would like to clarify that the construction of ToolBridge is specifically tailored to support the tool former paradigm, which is distinct from the function calling paradigm. These paradigms differ fundamentally in their objectives and mechanisms.

• The tool former paradigm focuses on enabling large language models (LLMs) to autonomously decide which tools to use, when to use them, and how to integrate their outputs into reasoning processes.
• In contrast, the function calling paradigm emphasizes predefined API interactions, which do not require the explicit reasoning capabilities central to the tool former paradigm.

Given these distinctions, benchmarks such as ToolBench and API-Bank, which are designed to evaluate function-calling systems, are not appropriate for assessing the effectiveness of ToolBridge. To prevent potential misunderstandings, we have explicitly clarified this differentiation in the revised submission, particularly in lines 40–49 of the Introduction section and lines 129–161 of the Related Work section.

Moreover, this distinction reinforces our claim that ToolBridge, to the best of our knowledge, represents the first open-source effort aimed at enabling LLMs to autonomously learn how to utilize external tools, specifically within the framework of the tool former paradigm.

Comment3: Coding-Focused Base Models

We would like to clarify that the primary objective of ToolBridge lies in supporting the tool former paradigm, which emphasizes dynamic tool invocation rather than code generation. LLMs trained on ToolBridge are designed to handle diverse problem domains and generate textual content across a wide range of topics. In this context, Python code is dynamically generated to invoke external tools as needed, aiding the reasoning process of LLMs.

Given that coding-focused models are typically optimized for the syntactic and semantic intricacies of programming languages, their specialization would not align well with the broader, domain-agnostic objectives of ToolBridge. As such, training coding-focused base models on ToolBridge would likely yield limited benefits compared to general-purpose models like Llama, which are better suited to the diverse and dynamic requirements of the tool-former paradigm.

Comment4: Why Use WebQA and FACT for Evaluation?

The use of WebQA and FACT for evaluation is intended to test the factual retrieval capabilities of LLMs trained on ToolBridge. Specifically, during model inference, LLMs trained on ToolBridge dynamically generate links based on the given context and utilize external tools, such as Python's requests module, to access these links and retrieve factual information. This process enables the model to incorporate externally verified information into its outputs, thereby enhancing the factual accuracy of its results.

Comment5: Prompt Design and Tool Use Support

ToolBridge follows the tool-former paradigm, enabling models to autonomously decide whether to use a tool, which tool to invoke, and when to do so, without predefined tools during evaluation. This aligns with the paradigm's emphasis on dynamic, context-driven tool usage.

Regarding the prompts employed during evaluation, for GSM8K, GSM Plus, and MathBench, we adopted a fixed CoT n-shot prompt template as described in the GSM Plus paper. For all other datasets, the models were provided with the question as input, without relying on predefined prompt structures. Details are explicitly outlined in lines 384–391 of the rebuttal revision for clarity.

We appreciate the reviewer's response; here is our rebuttal to your reply.

Comment1: Limitation of the benchmark

• Differences between "tool use" and "tool former"

  • In function calling methods like Toolllm, the reasoning process mainly involves selecting the right tool and setting the correct parameters. In contrast, Toolformer not only calls the tool but also reasons about the tool’s output to guide further text generation. This two-step reasoning—first for selecting and calling the tool, then for using the tool's results to generate text—distinguishes Toolformer from methods focused solely on function calling.
  • Scope of Tool Integration: Tool use systems (e.g., Toolbench, GAIA) rely on predefined, externally structured workflows for tool invocation. Toolformer integrates tool use directly into the model's inference loop, allowing dynamic decisions without external scripting or multi-turn structures.
  • Tool Selection Granularity: Tool use methods often follow predefined logic for selecting tools based on task-specific needs (e.g., ToolComp). In contrast, Toolformer treats tool invocation as part of language modeling, enabling flexible, context-driven decisions across tasks. BTW, tool use methods rely heavily on strong priors and predefined structures, so it’s unsurprising that they achieve high performance on standard datasets.
  • Learning vs. Orchestration: Toolformer learns tool use end-to-end, including when and how to invoke tools within generation. Tool use systems rely more on external orchestration, separating reasoning and tool invocation into distinct phases.

• RandomQA: We acknowledge that RandomQA includes some simple questions; however, even these simple questions are challenging for the base model to answer correctly. This further highlights the significance of our open-sourced ToolBridge dataset.

Comment2: Applicability on web data

First, in the context of evaluating an open-source dataset, determining whether its data source originates from web text data should not be considered a primary criterion for assessing its contribution, nor is it a particularly significant factor in measuring its value. Currently, in the field of large language models (LLMs), a significant number of open-source datasets are built upon existing open-source datasets, e.g., "Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models", "MultiLegalPile: A 689GB Multilingual Legal Corpus", "IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages", "MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning", “OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset”, to name a few.

Secondly, the primary contribution of this work is to address a significant limitation in the existing literature: while prior LLMs (e.g., Llama3.1, Qwen math 2.5, GPT-4o) capable of tool use (in the tool former field) have demonstrated impressive capabilities, none have publicly released a comprehensive data construction pipeline or the associated datasets. To fill this critical gap, our study proposes a systematic methodology for constructing tool former datasets and makes the resulting dataset publicly available, thereby providing a valuable resource for advancing research in this domain.

Finally, utilizing existing SFT data is not a cost-free endeavor. It requires an initial investigation of the data, manual quality evaluation, and subsequent processing and integration before we can proceed to construct our ToolBridge dataset.

Comment3: Baselines

First, the experiments in this paper are not limited to zero-shot scenarios; they also include few-shot settings, as shown in Table 6.

Second, you can indeed construct prompts tailored to specific data, predefine certain text formats for tool invocation, and use LLMs to generate these formats to utilize the tools. However, invoking tools in this manner is essentially meaningless, as the results generated by the tool invocation do not provide positive feedback to the subsequent text generation process. Moreover, the reviewer might consider trying to use a base model like LLama3-8B to directly generate text in specific tool invocation formats. The generated content is typically unusable and demonstrates almost no generalization capability.

Finally, we would like to point out that the evaluation approach mentioned by the reviewer is more commonly used in the tool use domain rather than in the tool former paradigm. This is because tool former does not predefine which tools can be used, making such a comparison impractical.