ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models

Q7. The paper would benefit from a comparison with other works that utilize Tool graphs, such as “Taskbench: Benchmarking Large Language Models for Task Automation.”

A7. Thank you for introducing this relevant paper. To briefly summarize the similarities and differences between ToolDial and TaskBench:

Similarities

• Both papers address generating dialogue datasets that reflect tool dependency scenarios.
• Both constructed a tool graph to represent tool dependency information, and used extracted subgraphs to generate dialogues.

Differences

• TaskBench generates scenarios where a single turn can be solved by chaining tools based on the extracted subgraph. In contrast, ToolDial generates scenarios where the agent determines parameter values through interaction with the user over multiple turns. The system either requests input parameter information from the user or, if the user cannot provide it, finds and calls other APIs to obtain the necessary information. TaskBench scenarios, however, immediately chain tools without involving the user. We believe that this difference is crucial because API calls are often costly and slow, and it is more natural to ask users for information they already have.

• The evaluation of TaskBench focuses solely on assessing the system’s ability for proper tool selection and tool chain usage. In contrast, ToolDial is “dialogue data” based on real-world conversational scenarios, encompassing 16 user and system actions. Its evaluation emphasizes the skills needed to address user queries within multi-turn dialogue contexts. In both works, an API graph is essential, but obtaining this graph itself is a challenging problem. In our paper, we propose and evaluate a new method to construct an API graph, whereas TaskBench does not specify whether its graph construction is manual or automated.

We will discuss these points in the Related Works section of our paper.

Q6. It remains uncertain if DST or action prediction are essential to achieving high correctness scores. It seems models could achieve high correctness even with low DST or action prediction accuracy. The authors are encouraged to conduct an ablation study to determine the necessity and impact of DST and action prediction on overall performance.

A6. We believe that this confusion comes from the original unclarity of the term “correctness”. We have clarified the term in Q4. In short, “Correctness” assesses whether the system generates a response that aligns with the output of an API call. Therefore, the correctness score is unrelated to DST or action prediction scores. Please let us know if you have further questions about this. Additionally, you have asked about overall performance. If you could clarify what specifically you mean by this, we would be glad to provide an answer to that.

For now, we interpreted “overall performance” as a model’s ability to generate a suitable thought, action, dialogue state, and message as a reply, given the dialogue history and the most recent user utterance, all in a single generation step. To assess this capability, we designed an experiment to evaluate TD-llama's performance. In this experiment, reasoning traces were generated for a total of 5,213 user utterances from test dialogues. If both the action and dialogue state within each reasoning step were accurately generated, the result was marked as True; otherwise, it was marked as False. This evaluation yielded a performance score of 76.6%. Additionally, for 1,166 test dialogues, we measured the rate at which the reasoning trace for all turns was correctly generated from the first to the last, resulting in an accuracy rate of approximately 57.1%.

Q5. It is unclear how many test samples were used in Tables 4 and 5. The presence of integer scores is unusual, given that the test set consists of 1,166 samples. Including precise sample counts and calculation methods could enhance transparency.

A5. The current test set consists of 1,166 dialogues. Among these, there are 6,746 dialogue states recorded in the middle of dialogues, and 9,200 actions. Therefore, 6,746 instances were used for DST evaluation, and 9,200 for action prediction. For correctness (i.e., the hallucination metric), which assesses the system’s response utterance after the “response” action, 943 dialogues were evaluated for correctness, excluding those that end with a “response fail” from the total 1,166 dialogues. Additionally, we will present all experimental scores to the third decimal place.

The metrics are calculated as follows:

• DST (Dialogue State Tracking): Each turn’s label dialogue state dictionary and predicted dialogue state dictionary were converted to lowercase, and special characters were removed. The evaluation was based on whether the two dialogue states matched completely.
• Action Prediction: Each turn’s label action and predicted action were converted to lowercase, and special characters were removed. Evaluation was based on whether they matched exactly.
• Correctness: Following the same method as G-Eval, a GPT model with temperature set above 0 evaluated each response (a total of 943) 10 times. The average of the 10 results (all either 0 or 1) was used as the score. We will add this clarification in our revision.

Q4. The concept of "correctness" is unclear. Clearer definitions and criteria for correctness could improve result interpretability. It is calculated at the session level, right?

A4. We apologize for the unclarity of the term “correctness.” One of the key capabilities of TALM is to provide responses to the user that are faithful to the API results without any hallucination. The metric Correctness measures this by comparing the utterance following the system's response action (the point at which it replies to the user based on API call results). Scoring is conducted using G-Eval with GPT model, where a score of 1 is assigned if the system's response aligns with the API call results, and 0 if it does not.

We are considering renaming this metric from Correctness to Hallucination Ratio or Faithfulness to make the metric name more intuitive.

Q3. The paper lacks a detailed error analysis, missing an opportunity to discuss specific areas where TALMs underperform and how ToolDial might be used to address these issues.

A3-2. For the action prediction, we present an F1-score table as part of the error analysis.

F1-score table

The following analysis is based on the updated scores for TD-Llama and Llama Instruct as discussed in Q2 (not from the original Table 4). We will update Table 5 according to the F1-score table in our revision.

Q3. The paper lacks a detailed error analysis, missing an opportunity to discuss specific areas where TALMs underperform and how ToolDial might be used to address these issues.

A3-1. Thank you for your advice. The following are the results of our error analysis and we will include this in the appendix of our revision. The following analysis is based on the updated scores for TD-Llama and Llama Instruct as discussed in Q2 (not from the original Table 4).

DST Error analysis

• The W/O GT results for TD-Llama could not be fully entered due to length constraints, so we will present here in text form. From Generation Error to Value Error, the results are 1873, 1619, 1, 0, 103, 23, and 144, respectively.

• The DST error table shows error counts, where slot and value errors can overlap in a single prediction, causing their sum to exceed the total errors.

For better understanding, we have provided a description of each error type on Error Type. By referring to the Format of the dialogue state, you will be able to better understand exactly which aspects each error type pertains to.

Format of the dialogue state (More details are in Appendix.4 of the paper)

• When there is no confirmed API:

  • Dialogue State: {API confirmed: false, API status: none}

• When the API is confirmed and some input parameter information can be extracted from dialogue history:

  • Dialogue State: {API confirmed: true, API status: {API name: “API1”, Required parameters: {param1: “value1”, param2: “”}, Optional parameters: {param3: “value3”}}}

Here, we call the param1, param2, param3 as slots, and value1, value3 as values.

Error Type

• Generation Error: This occurs when the dialogue state dictionary is not generated at all.

• API Confirmation Error (GT = True): This error happens when API is confirmed (api_confirmed=true), but is incorrectly predicted as it is confirmed (api_confirmed=false).

• API Confirmation Error (GT = False): This error occurs when the API is not confirmed (api_confirmed=false), but the model incorrectly predicts it as confirmed (api_confirmed=true).

• Format Error: This occurs when the dialogue state does not fully generate all fields such as api_confirmed, api_status, required parameters, and optional parameters.

• Slot Error: When api_confirmed is true, this error involves generating a dialogue state that does not include all required and optional parameter slots as specified in the API documentation.

• Value Error: This error involves incorrectly extracting the slot’s value from the dialogue history, with the following types:

  • Extracting Input Value from Multiple Result Error: This error occurs when an appropriate value cannot be selected from multiple results returned by the API output (as seen in turns 6 and 7 of Figure 2).

  • Inform Intent Add Error: This occurs when there is a value within the user query that could be used as an input parameter (inform intent clear add), but the model fails to track it.

  • Other General Input Parameter Extraction Errors: Errors that occur in typical situations where the input parameter is extracted incorrectly.

Result analysis for DST error

• Various errors appear in the GPT series and Llama3 Instruct models. In the W/O GT setting, which resembles real-world scenarios, more errors occur.
• In TD-Llama, errors decrease significantly, though generation errors remain common. Other frequent errors include Extracting Input Value from Multiple Results and Inform Intent Add errors.
• To reduce errors through fine-tuning, a simple approach to address "generation errors" can be proposed. However, "Extracting Input Value from Multiple Result Error," involving noisy lists, remains a challenging problem and an important future direction.

Q2. The evaluation process has potential reliability issues: (1) There are concerns that the upper bound results (when using ground truth) are lower than expected, especially with the TD-Llama model, which performed worse than anticipated. (2) TD-Llama (finetuned on Llama3-8b) scores lower than GPT-3.5-turbo in many aspect, a result that appears counterintuitive.

A2. Thank you for raising this issue. To answer your question, we examined our current method and identified a discrepancy in the prompt format used during training and inference for the Llama-series models compared to the GPT-based models. This discrepancy critically impacted the performance of the Llama models, leading to suboptimal results. To address this, we revised the prompt format for the Llama-based models to align with the format used for the GPT-based models. We also added more baselines as per the feedback of the reviewers. The updated scores are as follows:

The scores for the GPT-series models remain unchanged. However, the scores for vanilla Llama and TD-Llama have been changed. As shown in the DST scores in the table, we observed significant performance improvements for both Vanilla Llama and TD-Llama, surpassing the performance of GPT-3.5. We will update our revision with these updated scores in Table 4 and highlight the observed performance trends accordingly.

Thank you for your time and effort to review our paper. We sincerely appreciate the recognition of our efforts in addressing the resource gap for TALMs through the ToolDial dataset, which offers extensive multi-turn interactions reflective of real-world use cases. Additionally, thank you for recognizing the value of our dataset generation approach. By leveraging an API graph with GPT models, our scalable method reduces reliance on human-curated data and provides a framework that can inspire similar methodologies in the future.

Q1. Heavy reliance on GPT-generated dialogues introduces the risk of bias and synthetic errors inherent to the model, which could undermine the dataset’s realism.

A1. Thank you for highlighting this issue. When instructing GPT to generate multi-turn dialogues between a Tool-Augmented Language Model (TALM) and a user without any guidance, the resulting dialogues indeed tend to be overly repetitive and monotonous. Specifically, certain types of APIs are disproportionately preferred, and the actions performed by both the user and system lack variety, typically following a simple "inform intent - response" pattern. This results in an overly homogeneous set of situations.

To avoid this issue, our data generation procedure incorporates several guiding strategies. First, we sampled various triples from the API graph to include a wider range of APIs and domains. Additionally, we developed dialogue scenarios based on 16 user and system actions and 23 action sequences derived from them, which were then integrated into the generated dialogues. Notably, when GPT generates dialogues, it often neglects to include turns where either the user or the system performs actions related to rejection or negation. To mitigate this, we introduced user actions like “fail_inform” and “negate”, and system actions like “response_fail”, allowing for a wider range of situations in the dialogue data.

Moreover, although not mentioned in the paper, we observed that dialogues generated without guidance often exhibit consistent speaking styles for both the user and the system. For instance, the system’s “request” action often prompts "Can you give me 'param1' and 'param2'?". To introduce more variation, we defined several distinct speaking styles for these recurring actions, including:

• user action
  • inform
    • Sure! ~
    • Ok ~
    • Certainly!
  • affirm
    • Yes, that works.,
    • That would be great.
    • Sure, that sounds good.,
    • Yes, please proceed.
  • negate
    • No, that’s not what I meant.
    • I'm good. Thank you though.
    • Umm... that's not what I want..
• System action
  • request
    • To call ~
    • I need~,
    • May I ask for ~,
    • Please tell me ~,
  • clarify
    • Could you please provide more ~,
    • I’m not sure I understand. Can you clarify ~,
    • Could you explain that in more ~,
    • Can you clarify your ~ We incorporated a mechanism to randomly select from these predefined speaking styles during the scenario generation step (Section 3.3). For each action, we included a prompt to instruct GPT to generate utterances in line with the randomly selected style for the corresponding turn. We will add this description in our revision.

Thank you again for your time and effort to review our paper and provide constructive feedback. We have made our best effort to incorporate your suggestions and address your questions in our rebuttal. We understand you're busy but we would greatly appreciate it if you could review our rebuttal and let us know if it addresses your feedback and concerns.

Dear Reviewer fu86, we apologize for urging you again to engage with our rebuttal. As the discussion phase ends in less than two days, we hope to ensure sufficient time to discuss with you to address your concerns thoroughly. We have made our best effort to respond to your questions and have incorporated all your suggestions and feedback into our revised submission (now uploaded). Below is a summary of our rebuttal, with references to the specific sections in the revision where each point has been addressed:

1. Dataset realism and GPT biases: We acknowledged the risks of GPT-generated dialogues and detailed the strategies implemented in ToolDial to diversify scenarios and reduce repetitive patterns. These include sampling diverse API triples and defining specific user/system actions and speaking styles, enhancing the dataset’s realism and variability. (Section 3.4, “Model Biases”)

2. Evaluation and model performance discrepancies: We identified a prompt format issue affecting Llama-based models and presented updated performance scores. The revised results demonstrate significant improvements for TD-Llama, now surpassing GPT-3.5. Additional baseline models were also included based on other reviewers’ suggestions. (Section 4.3, Tables 4 & 5)

3. Error analysis: We provided a detailed breakdown of error types and counts, highlighting challenges like extracting values from noisy API outputs. F1 scores for action prediction were also included for better contextualization. (Appendix A.9 and Section 4.3)

4. Clarification of correctness metric: We clarified that "Correctness" measures response faithfulness to API results and is unrelated to DST or action prediction scores. To improve clarity, we renamed this metric “faithfulness.” (Section 4.1)

5. Sample counts and metric definitions: We clarified details of the test samples (e.g., 1,166 dialogues, 6,746 DST instances, and 9,200 action predictions) and provided precise calculation methods. Scores have been updated to three decimal places. (Section 4.1)

6. Overall performance: We addressed concerns about the relationship between DST/action prediction and correctness, and shared an additional experiment evaluating TD-Llama’s “overall performance,” achieving 76.6% accuracy for individual turns and 57.1% for entire dialogues. (Section 4.3, “Overall Performance”)

7. Comparison with TaskBench: We outlined the similarities and differences between TaskBench and ToolDial, emphasizing ToolDial's focus on multi-turn dialogues and user-agent interactions, unlike TaskBench's tool-chaining approach. (Section 2)

Beyond the questions and suggestions outlined above, we have also addressed feedback from other reviewers. This includes adding more baseline models, providing statistics on the domains covered in ToolDial, adding a running example to illustrate dataset construction, clarifying the novelty of our work and its contribution to ICLR, and making structural adjustments between the main text and appendix. These updates have been well-received by the reviewers.

We appreciate your engagement and remain ready to address any further concerns. Thank you!

The author has addressed most of my concerns in the revised version, which allows me to raise my score accordingly.

However, I must highlight a significant difference between the results reported in the new version and those in the original submission. Specifically, the DST results for TD-llama with ground truth (GT) have shown an unprecedented increase from 25 to 92.7. This dramatic improvement raises questions about consistency and reproducibility that warrant careful consideration.

Please take this substantial gap into account when making the final decision.

Q3. In Figure 2, the agent response contains symbolic terms from the API (seasonId and tournamentId). I am curious about the reasons for this design. Does this mean you are also showing the APIs to the users at the same time? This will make the interactions less natural.

A3. Thank you for highlighting this issue. When the system requests a parameter value from the user, we may either use the original parameter name as-is (e.g., tournamentId) or a more intuitive description, depending on the complexity of the parameter name.

For instance, some API parameters in RapidAPI are not immediately clear based on their names alone (e.g., the parameter “X-User-Agent” expects a device type, such as “mobile” or “desktop”), which could confuse the user if requested by their names. To address this, during the Dialogue Scenario Generation step (Section 3.3), we included both the parameter name and its description in the “request” action instruction. This enables GPT to generate request utterances that are more user-friendly. For example, when the “X-User-Agent” parameter is required, our dataset prompts a response like “Please provide the device type you are using,” instead of “Please provide the X-User-Agent”.

In Figure 2, specifically, symbolic terms (seasonId and tournamentId) are used because GPT judged their names to be intuitive enough for users to understand in context. However, when there is a substantial difference between a parameter’s name and its description, as in the X-User-Agent example, the request utterance is generated with greater reliance on the parameter's description.

Q4. I also have a writing suggestion for the authors. Try to condense the paper and move some essential information from Appendix to the main paper. I am constantly scrolling to Appendix section when reading the paper.

A4. Thank you for your suggestion. We plan to strengthen the main text by moving part of the explanation of user and system actions from Appendix A.3 to Section 3.2. Additionally, we plan to incorporate part of the explanation of dialogue state and retriever status from Appendix A.4 to Section 3.3. We would greatly appreciate it if you could provide more concrete suggestions that can strengthen our paper.

Q2. Lack of details about the dialogues and APIs: I would like to (1) see more descriptions of how complex the conversations are, as this is something the authors claim as a novelty, and (2) have a clear view of the APIs—what domains are you using and what types of arguments are you employing? Is there any fundamental difference between yours and the SGD?

A2.

(1) Our dataset includes a total of 16 actions, combining both user and system actions, as shown in Table 1. This significantly surpasses the diversity of actions in other benchmark datasets, allowing us to capture a wider variety of real-world situations. Existing TALM benchmarks are typically limited to a few actions, such as the user’s “inform intent” and “inform action”, and the system’s “request” and “response” actions. However, in our dataset, we also consider the user’s action of “inform intent vague” and other actions like the user’s “fail inform”, “affirm”, and “negate” actions, as well as system actions like “clarify”, “suggest”, and “response fail”. Thanks to many combinations of these actions, ToolDial encompasses scenarios not covered in previous benchmarks and demonstrates novelty.

Furthermore, the same type of action, such as the user’s “fail inform” or the system’s “request” can lead to different conversational flows depending on the specific situation in that turn. For instance, suppose the user fails to inform a requested parameter value. The subsequent dialogue may vary depending on whether the user simply states that they do not know the requested value or instead still provides other parameter values they can (as shown in the 3th turn of Figure 2). Our dataset accounts for both scenarios.

Additionally, there are two types of request actions. The first is the typical action of asking the user for an input parameter. The second occurs when an API output returns multiple values in a list, prompting the system to ask the user to select one of these values (as shown in the 6th turn of Figure 2).

In summary, unlike most existing benchmarks that are typically structured with only inform intent → request → inform → response actions, our dataset introduces a broader range of actions. Moreover, by varying the way the same actions are performed, our dataset provides much more complex scenarios than those in conventional datasets, highlighting the novelty of our work.

(2-1) We used APIs from 28 domains:
['Business', 'Data', 'Energy', 'Finance', 'Food', 'Business_Software', 'Database', 'Entertainment', 'Events', 'Health_and_Fitness', 'Location', 'SMS', 'Science', 'Social', 'Mapping', 'Music', 'Search', 'Logistics', 'Media', 'News_Media', 'Sports', 'Travel', 'Weather', 'Transportation', 'Financial', 'Gaming', 'Payments', 'Commerce']

Data (21.6%), Sports (18.7%), Weather (16.0%), Media (10.1%), Location (7.0%) make up the majority, with the remaining 23 domains evenly distributed. The input arguments of the APIs include approximately 399 parameters in the dataset, covering both required and optional parameters. These input parameters vary widely, with the most frequently appearing types being IDs, codes, coordinates, location names, languages, and dates. The data types of these arguments include string, bool, list, date, json, number, and others.

(2-2) One of the main differences between ToolDial and SGD is that ToolDial addresses situations involving API dependencies, or "API call chains." Specifically, the user may be unable to provide an argument required to execute an API, and the agent proactively searches for and calls another API to retrieve the necessary information. In contrast, SGD does not cover such agent proactivity, and its API chains are relatively straightforward (e.g., find a restaurant -> book a restaurant). Furthermore, ToolDial encompasses more than 400 real-world APIs from RapidAPI, compared to just 44 APIs (or intents) in similar domains (e.g., Flights and Train, Hotels and Restaurants) covered by SGD. Generating complex dialogues based on a noisy pool of APIs, as in ToolDial, is inherently more challenging than in SGD, and thus we developed an automated method for API graph construction.

In addition, while the primary motivation of SGD is on evaluating a model's domain transfer performance in unseen domains that are similar but slightly different (e.g., the model is trained on the Train domain and tested on the Flight domain), ToolDial is grounded in a broader range of APIs with noisy API documentation used in real-world scenarios. As a result, ToolDial offers a more realistic setting for both training tool-augmented models and evaluating their generalizability.

Thank you for your time and effort to review our paper. We deeply appreciate the positive recognition of our work, particularly for appropriately positioning our contributions through citations and discussions, and for establishing a timely benchmark in the dialogue community. Also, thank you for recognizing our efforts to create a benchmark that stands out from existing ones such as MWOZ, SGD, and API Bank.

Q1. Although I recognize the contributions this paper makes to advance the development of dialogue systems, I am not sure the proposed methodologies for creating the dataset are adequately novel or aligned with the values of ICLR. Maybe the authors can provide more evidence and arguments in the rebuttal.

A1. Thank you for your question. Autonomous agents and their tool use are a significant research focus at ICLR, with several related studies, such as ToomLLM and ToolBench, being published. Advancing this field requires a benchmark dataset that reflects realistic and complex scenarios of user-agent interactions. Since most existing benchmarks are relatively simplistic; our work introduces ToolDial, which accounts for multi-turn dialogues, agent proactivity (see the next paragraph), diverse user and system actions, and real-world APIs with noisy API documentation from RapidAPI.

A key focus of our work is on dialogues where the agent proactively calls multiple APIs in sequence. In real-world scenarios, it is common for users to fail to provide a parameter value requested by the agent. In such cases, the agent is expected to proactively search for and call another API to retrieve the necessary information, rather than simply stating that the user request cannot be completed. Generating dialogues that account for such agent proactivity is challenging because it requires identifying which APIs can provide information that can serve as input for other APIs. To address this, we developed an automated method for API graph construction and assessed its reliability. This graph facilitates the generation of dialogues with API dependencies and could potentially be utilized for API retrieval in future research.

We believe that tool-augmented language models (TALMs) and traditional dialogue management (DM) systems share many similarities, but recent TALM research has somewhat overlooked the rich literature on DM systems. At the core of the DM research are dialogue state tracking and diverse user and system actions, which serve as explicit and interpretable methods for tracking an agent’s progress in task completion. Our work aims to bridge this gap by introducing these concepts into TALMs, thereby enriching the future research in the TALM field.

Thank you again for your time and effort to review our paper and provide constructive feedback. We have made our best effort to incorporate your suggestions and address your questions in our rebuttal. We understand you're busy but we would greatly appreciate it if you could review our rebuttal and let us know if it addresses your feedback and concerns.

Q3. The experiments in this work are still preliminary (see suggestions). Addition of more experimental details, clearly signifying the type of prompting baselines, inclusion of another open source LLM, experimenting with chain-of-thought like prompting can also be useful.

A3. Thank you for your suggestions. The baseline prompt used in our experiments was based on ReAct, with each reasoning step comprising a thought, an action, and various types of observations (e.g., retriever status, API documentation, call result). The reasoning step includes the process of thought-action based on ReAct before the system replies to the user’s utterance. Below is the prompt baseline used in the experiments.

User: Can you provide insights into which articles on NYTimes.com have been shared the most on social media platforms, particularly Facebook, to understand active engagement trends? 
System: 
- Thought: The user seems to have intent. I will call the retriever.
- Action:retriever_call
- Retriever status: {'retriever_call': 'true', 'retrieved_api': {'Business|ny_times_most_popular|/shared/{period}/{share_type}.json': 0.71372604, ….}
- API documentations: {'api_name': 'Business|ny_times_most_popular|/shared/{period}/{share_type}.json', .....}
- Thought: Based on results of the retriever, I need to call Business|ny_times_most_popular|/shared/{period}/{share_type}.json. To call it, I need to ask 'share_type' and 'period' to the user.
- Action:request
- Retriever status: {'retriever_call': 'false', 'retrieved_api': 'none'}
 - Message: Could you please provide the type of sharing on Facebook for articles and the time period for retrieving popular articles? 
- Dialogue State: <Current dialogue state>
User: Certainly! The share type is Facebook and the period is 7 days. 
System:
- Dialogue State: <Current dialogue state>
- Thought: Based on the user's response, now I can call the API Business|ny_times_most_popular|/shared/{period}/{share_type}.json
- Action:call ...

We will include additional explanations regarding the prompt format in the revised paper.

Given that our experiment was conducted using ReAct, we consider our experiment setting to be already relying on chain-of-thought.

For open-source LLM baselines, we presented the performance of more open-source LLMs (Codellama, Qwen) in the table of A1.

And for the missing reference, thank you for sharing. We will discuss these works in our revision.

Sugg9. [Future work] As API calls are similar to function calling in Python, it would be useful to also benchmark CodeLLaMa/finet-uned CodeLLaMa for the given benchmark

A9. Thank you for your suggestion. We have added the scores of Code Llama in the table in A1.

Sugg1. Please mention the language of the dataset in the paper

A1. Yes, we will mention that the dataset is in English in the Dialogue Generation section (Section 3.4).

Q6. What are the domains of the examples in the dataset?

A6. ToolDial includes the following 28 domains:

['Business', 'Data', 'Energy', 'Finance', 'Food', 'Business_Software', 'Database', 'Entertainment', 'Events', 'Health_and_Fitness', 'Location', 'SMS', 'Science', 'Social', 'Mapping', 'Music', 'Search', 'Logistics', 'Media', 'News_Media', 'Sports', 'Travel', 'Weather', 'Transportation', 'Financial', 'Gaming', 'Payments', 'Commerce']

The proportions of the domains are as follows:

• Data: 21.591%
• Sports: 18.702%
• Weather: 16.020%
• Media: 10.143%
• Location: 6.975%

These five domains make up the majority, with the remaining 23 domains evenly distributed.

Q5. Could you explain why does GPT-4o-mini perform better without ground truth in Table 4?

A5. Error analysis shows that GPT-4o-mini in the With GT setting more frequently misjudged API confirmation status compared to the W/O GT setting.

The following table shows a summary of our error analysis on DST across the models.

DST Error analysis

• The W/O GT results for TD-Llama could not be fully entered due to length constraints, so we will present here in text form. From Generation Error to Value Error, the results are 1873, 1619, 1, 0, 103, 23, and 144, respectively.)

• The DST error table shows error counts, where slot and value errors can overlap in a single prediction, causing their sum to exceed the total errors.

API Conf. Err (GT = F) refers to an error where the system incorrectly determines that the API has been confirmed when it has not, and API Conf. Err (GT = T) is the reverse. While other models show higher errors in the W/O GT setting, GPT-4o-mini does not follow this trend. This is the main reason why GPT-4o-mini performs better in the W/O GT setting compared to the With GT setting.

For better understanding, we have provided a description of each error type on Error Type. By referring to the Format of the dialogue state, you will be able to better understand exactly which aspects each error type pertains to.

Format of the dialogue state (More details are in Appendix.4 of the paper)

• When there is no confirmed API:

  • Dialogue State: {API confirmed: false, API status: none}

• When the API is confirmed and some input parameter information can be extracted from dialogue history:

  • Dialogue State: {API confirmed: true, API status: {API name: “API1”, Required parameters: {param1: “value1”, param2: “”}, Optional parameters: {param3: “value3”}}}

Here, we call the param1, param2, param3 as slots, and value1, value3 as values.

Error Type

• Generation Error: This occurs when the dialogue state dictionary is not generated at all.

• API Confirmation Error (GT = True): This error happens when API is confirmed (api_confirmed=true), but is incorrectly predicted as it is confirmed (api_confirmed=false).

• API Confirmation Error (GT = False): This error occurs when the API is not confirmed (api_confirmed=false), but the model incorrectly predicts it as confirmed (api_confirmed=true).

• Format Error: This occurs when the dialogue state does not fully generate all fields such as api_confirmed, api_status, required parameters, and optional parameters.

• Slot Error: When api_confirmed is true, this error involves generating a dialogue state that does not include all required and optional parameter slots as specified in the API documentation.

• Value Error: This error involves incorrectly extracting the slot’s value from the dialogue history, with the following types:

  • Extracting Input Value from Multiple Result Error: This error occurs when an appropriate value cannot be selected from multiple results returned by the API output (as seen in turns 6 and 7 of Figure 2).

  • Inform Intent Add Error: This occurs when there is a value within the user query that could be used as an input parameter (inform intent clear add), but the model fails to track it.

  • Other General Input Parameter Extraction Errors: Errors that occur in typical situations where the input parameter is extracted incorrectly.

Result analysis for DST error

• Various errors appear in the GPT series and Llama3 Instruct models. In the W/O GT setting, which resembles real-world scenarios, more errors occur.
• In TD-Llama, errors decrease significantly, though generation errors remain common. Other frequent errors include Extracting Input Value from Multiple Results and Inform Intent Add errors.
• To reduce errors through fine-tuning, a simple approach to address "generation errors" can be proposed. However, "Extracting Input Value from Multiple Result Error," involving noisy lists, remains a challenging problem and an important future direction.

Q4. What was the setting for the GPT-based experiments - zero-shot or in-context learning? Please add these details in the next version of the draft

A4. Thank you for your suggestion. The GPT-based experiments used zero-shot prompting. We will clarify this in the revision.

Q2. The dataset was constructed using an LLM (GPT4-o-mini) - perhaps some artifacts in generation can lead to undesirable effects . The authors back up the dataset quality through human evaluation as well, but perhaps a blind test prepared by human authors can also be helpful.

A2. Thank you for your suggestion. When instructing GPT to generate multi-turn dialogues between a Tool-Augmented Language Model (TALM) and a user without any guidance, the resulting dialogues indeed tend to be overly repetitive and monotonous. Specifically, certain types of APIs are disproportionately preferred, and the actions performed by both the user and system lack variety, typically following a simple "inform intent - response" pattern. This results in an overly homogeneous set of situations.

To avoid this issue, our data generation procedure incorporates several guiding strategies. First, we sampled various triples from the API graph to include a wider range of APIs and domains. Additionally, we developed dialogue scenarios based on 16 user and system actions and 23 action sequences derived from them, which were then integrated into the generated dialogues. Notably, when GPT generates dialogues, it often neglects to include turns where either the user or the system performs actions related to rejection or negation. To mitigate this, we introduced user actions like “fail_inform” and “negate”, and system actions like “response_fail”, allowing for a wider range of situations in the dialogue data.

Moreover, although not mentioned in the paper, we observed that dialogues generated without guidance often exhibit consistent speaking styles for both the user and the system. For instance, the system’s “request” action often prompts "Can you give me 'param1' and 'param2'?". To introduce more variation, we defined several distinct speaking styles for these recurring actions, including:

• user action
  • inform
    • Sure! ~
    • Ok ~
    • Certainly!
  • affirm
    • Yes, that works.,
    • That would be great.
    • Sure, that sounds good.,
    • Yes, please proceed.
  • negate
    • No, that’s not what I meant.
    • I'm good. Thank you though.
    • Umm... that's not what I want..
• System action
  • request
    • To call ~
    • I need~,
    • May I ask for ~,
    • Please tell me ~,
  • clarify
    • Could you please provide more ~,
    • I’m not sure I understand. Can you clarify ~,
    • Could you explain that in more ~,
    • Can you clarify your ~

We incorporated a mechanism to randomly select from these predefined speaking styles during the scenario generation step (Section 3.3). For each action, we included a prompt to instruct GPT to generate utterances in line with the randomly selected style for the corresponding turn.

The blind test you suggested is an interesting experiment, and we leave it to future work. For now, we believe that generated dialogues do not necessarily need to be indistinguishable from human-authored dialogues. Human-authored task-oriented dialogues are often error-prone (e.g., MultiWOZ has undergone multiple revisions due to errors) and tend to lack diversity (e.g., SGD proposed an automated dialogue generation method to address this limitation). We made significant efforts to mitigate potential issues arising from GPT's biases, as evidenced by the satisfactory scores achieved in both G-Eval and human evaluations.

Thank you for your time and effort to review our paper. We sincerely appreciate the positive feedback on our paper, highlighting the novelty and timeliness of introducing multi-turn tool calling to the TALM literature and the value of our benchmark in offering a more comprehensive evaluation through three smaller tasks instead of relying on a single one.

Q1. Non-LLM based baselines need to be included for completeness

A1. Thank you for your suggestion. Although we haven't included non-LLM baselines yet, we have added experiment results for more baselines, including CodeLlama-7b-Instruct-hf, Qwen2.5-Coder-7B-Instruct as per the feedback of the reviewers as shown below.

We first want to clarify that upon reviewing our original results based on Reviewer 3’s feedback, we found that the prompt format used during the inference of vanilla Llama and the fine-tuning and inference of TD-Llama was highly suboptimal. Therefore, we re-ran the experiments for the Llama3 series models using the same prompt format as in the ChatGPT model experiments. The final results are as follows.

The scores for the ChatGPT series models remain the same. However, the scores for vanilla Llama and TD-Llama have changed. We will update Table 4 accordingly in our revision.

Sugg8. The dataset construction section is a little hard to understand. Perhaps a running example may make the section clear.

A8. Using a running example would be helpful in clarifying the data construction part. We will enhance Section 3.2 by incorporating the following example based on Figure 2.

3.2 Action Sequence

One triple consisting of two nodes was sampled from the graph. Suppose this triple includes the APIs "LeagueHomeStandings" and "CategoryTournaments". LeagueHomeStandings requires two input parameters, CategoryTournaments typically returns multiple IDs as output, and one of the input parameters of LeagueHomeStandings, tournamentId, and the output component id of CategoryTournaments are connected by an edge (see the reasoning step in the 6th turn of Figure 2).

From this triple, possible action sequences include:

• inform intent clear → retriever call → request → fail inform → retriever call → request → inform → call → request → inform → call → response
• inform intent vague → clarify → inform intent clear → retriever call → request → fail inform → retriever call → request → inform → call → request → inform → call → response
• inform intent vague → suggest → affirm → request → fail inform → retriever call → request → inform → call → request → inform → call → response
• inform intent vague → suggest → negate → response fail
• And more

To enhance the diversity of conversational flows in our dataset, we generate a dialogue for each of these sequences. For instance, the dialogue in Figure 2 is based on the first action sequence above.

There are three request actions in this action sequence. The first request is for the input parameters needed to execute LeagueHomeStandings, the second is to execute CategoryTournaments, and the third is to select one ID from the multiple IDs output by CategoryTournaments. In Figure 2, CategoryTournaments is an API that returns multiple IDs, requiring an additional user prompt to select the correct value (see the 6th turn of Figure 2).

3.3 Dialogue scenario generation

During the dialogue scenario generation stage, we plan what information each user and system utterance should include in each turn to perform the corresponding action (e.g., input parameters the system would request, parameter values the user would provide, etc.). The dialogue in Figure 2 is generated from a prompt like the following:

• Inform intent clear: Instruction: the user utters a pre-constructed query related with API LeagueHomeStandings and CategoryTournament.
• (Retriever call) -> request: Instruction: the system to ask the user for seasonId and tournamentId as input parameters.
• Fail inform: Instruction: the user responds with seasonId ‘45’ but fails to provide tournamentId.
• (Retriever call) -> request: Instruction: the system prompts the user for id.
• Inform: Instruction: the user responds with the requested information.
• (call) -> request: Instruction: the system asks the user for the name variable, to select one Id from multiple results.
• Inform: Instruction: the user responds with "NBA."
• (call) -> response: Instruction: the system responds based on the results of the call.

For parameter values (e.g., 45, 264 in Figure 2), plausible inputs and outputs for the API used in dialogue generation are first generated and then utilized.

Scenario prompts are created by predefining templates for each action and filling them with the API's input parameters and values. During this process, the dialogue states used in the experiment are also automatically generated for each turn, using the same values. The prompts and dialogue states are generated by executing pre-written code.

The scenario prompt generated in this way is then provided to GPT-4o, along with simple instructions, to generate the dialogue data (in Section 3.4).

Sugg7. While it is exciting to benchmark the models on, non LLM based baselines such as BERT based/T5 based dialogue state tracking/action recognition should also be evaluated for completeness

A7. Thank you for your suggestion. For dialogue state tracking and action prediction, LLM-based models (such as SimpleTOD [1]) have proven to be superior to non-LLM models. We believe that it is already well-established in the literature that LLM models generally outperform non-LLM models, which is why we did not include non-LLM baselines in our experiments. However, if you believe it is still essential to include these baselines, we will incorporate their results in the revision.

[1] HOSSEINI-ASL, Ehsan, et al. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 2020, 33: 20179-20191.

Sugg6. While GPT based results are not reproducible (based on API changes), it may be useful to release the predictions from those runs for future work to be built on this work.

A6. Thank you for your suggestion. We will include the GPT predictions along as part of our code release.

Sugg5. Please include vanilla LLaMa results in Table 5.

A5. Thank you for your suggestion. We have added the scores of vanilla Llama as follows and we will update Table 5 in our revision accordingly.

F1-score table

Sugg4. As it is a multi-turn dataset, inclusion of task completion metric - when the model performs all the correct api calls at every turn it is marked as a complete task.

A4. Thank you for your suggestion. After each user utterance, we ensure that the system generates the appropriate dialogue state. Our DST experiment evaluates this accuracy for every turn. Therefore, if the goal is to measure whether the correct API can be called at "every turn", the DST score we reported in the experimental results reflects this measurement.

If we focus only on the turns where an API is actually called, DST scores are as follows.

Please note that the Llama scores have been updated as explained in A1.

Sugg3. Please report at least three significant digits when reporting the results

A3. Yes, we will update the numeric format in all tables accordingly.

Sugg2. Please include qualitative examples from the dataset describing (i) examples after the data collection is complete and (ii) failures of respective LLMs

A2. Yes, we will add examples of the dataset after the data collection is complete and error cases of each LLM in the paper.

Thank you again for your time and effort to review our paper and provide constructive feedback. We have made our best effort to incorporate your suggestions and address your questions in our rebuttal. We understand you're busy but we would greatly appreciate it if you could review our rebuttal and let us know if it addresses your feedback and concerns.