Tool Decoding: A Plug-and-Play Approach to Enhancing Language Models for Tool Usage

We thank Reviewer CYKv for providing constructive comments for our work. We will address your concerns in the following points.

W 1. Latency analysis

Thank you for your constructive advice! We have added latency analysis in Appendix C.

The Table below shows the inference speed (sec/sample) for Tool Decoding in comparison with greedy search and beam search. Tool Decoding with $oc \leq 1$ introduces only slight latency, while Tool Decoding with $oc \leq 6$ is faster than beam search with the same number of samples. This is because order consistency maintains multiple candidates only during the generation of the tool call, which constitutes just a portion of the entire response. Note that our current implementation applies constraints at the level of logits. For practical deployment, these constraints could be implemented at the language head layer, which would further reduce computational requirements and enhance processing speed.

W 2. Conparison with other search strategies and prompt engineering

Epsilon sampling

Following your suggestion, we perform experiments for epsilon sampling on API-Bank (Call). The results show in the Table below. Since the performance of epsilon sampling is similar to greedy search, we decided not to include it as a baseline.

Prompt engineering

As a plug-and-play method, Tool Decoding integrates seamlessly with prompt engineering. Table below presents the performance of different numbers of in-context examples on API-Bank (Call). The results demonstrate that our method effectively combines with prompt engineering, significantly enhancing the model’s tool usage capabilities. Notably, this combination even enables a 7B-level generalist model, such as deepseek-coder-6.7b, to surpass GPT-4 under the same prompt settings, as highlighted in Bold values. The results have been added in the Section 4.

W 3. Take stronger models as base models

We supply results with three stronger models, including one powerful tool-finetuned models and two 30B-level models.

In Table below, gorilla-openfunctions-v2 represents one of the most advanced tool-finetuned models, while Yi-1.5-34B and deepseek-coder-33b are both 30B-level LLMs. Tool Decoding demonstrates significant improvements across all three models, with both deepseek-coder-33b and gorilla-openfunctions-v2 outperforming GPT-4 on API-Bank. The results have been added in Appendix E.

Q 1. Why UltraTool is used in Figure 2, while API-Bank is used in Figure 3?

UltraTool is a dataset specifically designed to provide fine-grained testing for the three distinct tool-use stages individually, so we utilize it to conduct a preliminary experiment to evaluate the performance of LLMs at each stage in isolation.

However, tool usage is a continuous process across these stages rather than a series of isolated steps. Therefore, we conduct an error analysis using API-Bank, a widely used benchmark for tool-use dialogue, to better reflect the practical scenarios.

Q 2. Why the awareness error changes between the configurations with and without tool decoding in Figure 6?

Thank you for helping us identify a small bug in our error analysis. Awareness errors occur when the models fail to output the start signal for using a tool, such as [. However, we detected these errors by checking for the simultaneous absence of both [ and ], which caused a few cases to be incorrectly classified as awareness errors. We have fixed this bug and updated the corresponding results in the paper. The change is tiny and has no influence to our conclusion.

Q 3. Why LLM is used to generate optional parameters instead of supplying them directly, as is done with required parameters?

Required parameters are essential for invoking a tool, so we can directly supply them to ensure none are missing. However, optional parameters can be absent in a tool call. Whether to use them depends on the specific context, so we allow LLMs to make this decision autonomously through constrained decoding.

Q 4. Provide a more detailed explanation on how accuracy is computed in the paper?

The accuracy is computed following these steps:

1. Extract the tool call from the generated content according to the predefined format.
2. Verify whether the tool name generated is correct.
3. Check whether all the keys used are correct.
4. Confirm whether all the values used are correct.

A tool usage is considered successful only if it passes all the above steps. Otherwise, it is classified as a failure.

Q 5. Report the success rate with and without constrained decoding, specifically indicating the proportion of samples with correct awareness, selection, and tool call?

We apologize for not fully understanding your points. We have reported our success rate and error distribution across three stages with and without Tool Decoding. Please refer to Figure 5 and 6 in paper.

AC 1. Figure 5 is missing accuracy numbers for Gemma.

The accuracy numbers for Gemma are provided in Appendix G. Due to space limitations, we have reported only a subset of the results in the main text.

AC 2. Reporting accuracy numbers in Table 3 rather than error reduction might provide easier interpretation.

Thanks for your advice. We aim to emphasize the performance of Order Consistency in mitigating value errors. Additionally, since the overall distribution of error types varies across models, using the proportion of value errors, provides a clearer and more consistent basis for comparison across different models.

AC 3. re-phrase in section 2.

Thanks for your suggestion! We have re-phrased it in Section 2 according to your comment.

W 3. Conparison with other constrained decoding algorithms

There are two existing constrained decoding methods for tool usage. We have compared them with our method, and the results have been included in Appendix F.

FANTASE [1] is not a plug-and-play method, as it requires additional training of a reranker. It introduces state-tracked constrained decoding to ensure the correct format but still relies on LLMs to generate all keys, including both required and optional parameters. This approach cannot effectively address issues like missing certain parameters, as illustrated in Figure 2 of [1]. To mitigate this limitation, a separate reranker should be trained to select the optimal tool call from multiple generated samples. In contrast, our method does not require any additional training and inherently avoids parameter absence, ensuring robustness in tool usage. Considering that this approach is not training-free and the code has not been open-sourced, we did not include it in our experimental comparisons.

TOOlDEC [2] is not universally applicable due to its specific requirements for tool call formats. It employs multiple Finite-State Machines (FSMs) to perform constrained decoding, relying on a special token to signal transitions between FSMs as shown in Figure 4 of [2] . For instance, in their implementation, formats like [Action: ToolName, Action Input: {key1=value1,<0x0A>key2=value2}] are supported, where <0x0A> serves as an indicator for transition from the first value FSM to the next key FSM. Since values are generated freely, the model must independently generate this special token, which is then detected to trigger the FSM transition. This reliance introduces two key limitations:

1. If the model fails to adhere to the predefined format and omits the required special token during value generation, it remains stuck in the value mode, freely generating tokens. This disrupts the FSM transitions, rendering constrained decoding ineffective.
2. For tool-finetuned models or code models, such specialized formats may deviate from the data encountered during their fine-tuning or pretraining, potentially resulting in decreased performance.

It is important to note that punctuation marks, such as commas, spaces, and quotation marks, cannot serve as special tokens since most models encode them as part of surrounding tokens rather than as independent tokens. This makes TOOLDEC incompatible with common formats like [ToolName(key1=value1, key2=value2)]. In contrast, Tool Decoding determines transitions by verifying whether a complete variable of the specified type has been generated to assign the value, eliminating the reliance on special tokens. Table below demonstrates the robustness of our method compared to TOOLDEC across various tool-finetuned and code models.

[1] Zhuoer Wang, et al. "Fantastic sequences and where to find them: Faithful and efficient API call generation through state-tracked constrained decoding and reranking." In Findings of the Association for Computational Linguistics: EMNLP, 2024.

[2] Kexun Zhang, et al. "Syntax error-free and generalizable tool use for llms via finite-state decoding." arXiv preprint arXiv:2310.07075, 2023

Thank you for your feedback! We are pleased to know that our responses resolved your concerns. If there are any further questions or suggestions, please don’t hesitate to let us know. Have a great day!

W 1.2 Comparison with in-context learning

As a plug-and-play method, Tool Decoding integrates seamlessly with prompt engineering. Table below presents the performance of different numbers of in-context examples on API-Bank (Call). The results demonstrate that our method effectively combines with prompt engineering, significantly enhancing the model’s tool usage capabilities. Notably, this combination even enables a 7B-level generalist model, such as deepseek-coder-6.7b, to surpass GPT-4 under the same prompt settings, as highlighted in Bold values. The results have been added in the Section 4.

W 2. Larger models

Following your suggestion, we conducted experiments on two 30B-level generalist models to enhance the comprehensiveness of our evaluation. We have added the results in Appendix E. In Table below, Tool Decoding demonstrates significant improvements across these 30B-level models, with deepseek-coder-33b even outperforming GPT-4 on API-Bank.

For 70B-level models, we regret that limited computational resources prevent us from conducting more extensive experiments since Tool Decoding require deploying models locally. However, we performed a fine-grained error analysis with cloud APIs on two 70B-level models: qwen2-72b-instruct, which achieved an accuracy of 71.4%, and qwen1.5-72b-chat, which achieved 69.9%. The Table below presents the error type distribution for both models. Format and value errors remain the most prevalent, so we believe our method still works on these powerful models. The analysis has been added in Appendix D.

W 3. Computational efficiency

Thank you for your constructive advice! We have added analysis on computational efficiency in Appendix C.

The Table below shows the inference speed ((sec/sample) for Tool Decoding in comparison with greedy search and beam search. Tool Decoding with $oc \leq 1$ introduces only slight latency, while Tool Decoding with $oc \leq 6$ is faster than beam search with the same number of samples. This is because order consistency maintains multiple candidates only during the generation of the tool call, which constitutes just a portion of the entire response. Note that our current implementation applies constraints at the level of logits. For practical deployment, these constraints could be implemented at the language head layer, which would further reduce computational requirements and enhance processing speed.

We thank Reviewer fh4s for providing constructive comments for our work. We will address your concerns in the following points. All of the results below have been added in our paper.

W 1.1 Comparison with other constrained decoding methods

There are two existing constrained decoding methods for tool usage. We have compared them with our method, and the results have been included in Appendix F.

FANTASE [1] is not a plug-and-play method, as it requires additional training of a reranker. It introduces state-tracked constrained decoding to ensure the correct format but still relies on LLMs to generate all keys, including both required and optional parameters. This approach cannot effectively address issues like missing certain parameters, as illustrated in Figure 2 of [1]. To mitigate this limitation, a separate reranker should be trained to select the optimal tool call from multiple generated samples. In contrast, our method does not require any additional training and inherently avoids parameter absence, ensuring robustness in tool usage. Considering that this approach is not training-free and the code has not been open-sourced, we did not include it in our experimental comparisons.

TOOlDEC [2] is not universally applicable due to its specific requirements for tool call formats. It employs multiple Finite-State Machines (FSMs) to perform constrained decoding, relying on a special token to signal transitions between FSMs as shown in Figure 4 of [2] . For instance, in their implementation, formats like [Action: ToolName, Action Input: {key1=value1,<0x0A>key2=value2}] are supported, where <0x0A> serves as an indicator for transition from the first value FSM to the next key FSM. Since values are generated freely, the model must independently generate this special token, which is then detected to trigger the FSM transition. This reliance introduces two key limitations:

1. If the model fails to adhere to the predefined format and omits the required special token during value generation, it remains stuck in the value mode, freely generating tokens. This disrupts the FSM transitions, rendering constrained decoding ineffective.
2. For tool-finetuned models or code models, such specialized formats may deviate from the data encountered during their fine-tuning or pretraining, potentially resulting in decreased performance.

It is important to note that punctuation marks, such as commas, spaces, and quotation marks, cannot serve as special tokens since most models encode them as part of surrounding tokens rather than as independent tokens. This makes TOOLDEC incompatible with common formats like [ToolName(key1=value1, key2=value2)]. In contrast, Tool Decoding determines transitions by verifying whether a complete variable of the specified type has been generated to assign the value, eliminating the reliance on special tokens. Table below demonstrates the robustness of our method compared to TOOLDEC across various tool-finetuned and code models.

[1] Zhuoer Wang, et al. "Fantastic sequences and where to find them: Faithful and efficient API call generation through state-tracked constrained decoding and reranking." In Findings of the Association for Computational Linguistics: EMNLP, 2024.

[2] Kexun Zhang, et al. "Syntax error-free and generalizable tool use for llms via finite-state decoding." arXiv preprint arXiv:2310.07075, 2023

Thank you for raising the score! We are delighted to know that our responses addressed your concerns. If you have any additional questions or feedback, please feel free to let us know. Wishing you a wonderful day!

W&Q 2. More powerful models.

Following your suggestion, we conducted experiments on more powerful tool-finetuned models, such as gorilla-openfunctions-v2. Additionally, we included experiments on two 30B-level generalist models to enhance the comprehensiveness of our evaluation. The results have been added in Appendix E. In Table below, Tool Decoding demonstrates significant improvements across all three models, with both deepseek-coder-33b and gorilla-openfunctions-v2 outperforming GPT-4 on API-Bank.

We thank Reviewer hS7t for providing constructive comments for our work. We will address your concerns in the following points. The results below have all been added in our paper.

W&Q 1.1. Comparison with other constrained decoding methods

There are two existing constrained decoding methods for tool usage. We have compared them with our method, and the results have been included in Appendix F.

FANTASE [1] is not a plug-and-play method, as it requires additional training of a reranker. It introduces state-tracked constrained decoding to ensure the correct format but still relies on LLMs to generate all keys, including both required and optional parameters. This approach cannot effectively address issues like missing certain parameters, as illustrated in Figure 2 of [1]. To mitigate this limitation, a separate reranker should be trained to select the optimal tool call from multiple generated samples. In contrast, our method does not require any additional training and inherently avoids parameter absence, ensuring robustness in tool usage. Considering that this approach is not training-free and the code has not been open-sourced, we did not include it in our experimental comparisons.

TOOlDEC [2] is not universally applicable due to its specific requirements for tool call formats. It employs multiple Finite-State Machines (FSMs) to perform constrained decoding, relying on a special token to signal transitions between FSMs as shown in Figure 4 of [2] . For instance, in their implementation, formats like [Action: ToolName, Action Input: {key1=value1,<0x0A>key2=value2}] are supported, where <0x0A> serves as an indicator for transition from the first value FSM to the next key FSM. Since values are generated freely, the model must independently generate this special token, which is then detected to trigger the FSM transition. This reliance introduces two key limitations:

1. If the model fails to adhere to the predefined format and omits the required special token during value generation, it remains stuck in the value mode, freely generating tokens. This disrupts the FSM transitions, rendering constrained decoding ineffective.
2. For tool-finetuned models or code models, such specialized formats may deviate from the data encountered during their fine-tuning or pretraining, potentially resulting in decreased performance.

It is important to note that punctuation marks, such as commas, spaces, and quotation marks, cannot serve as special tokens since most models encode them as part of surrounding tokens rather than as independent tokens. This makes TOOLDEC incompatible with common formats like [ToolName(key1=value1, key2=value2)]. In contrast, Tool Decoding determines transitions by verifying whether a complete variable of the specified type has been generated to assign the value, eliminating the reliance on special tokens. Table below demonstrates the robustness of our method compared to TOOLDEC across various tool-finetuned and code models.

[1] Zhuoer Wang, et al. "Fantastic sequences and where to find them: Faithful and efficient API call generation through state-tracked constrained decoding and reranking." In Findings of the Association for Computational Linguistics: EMNLP, 2024.

[2] Kexun Zhang, et al. "Syntax error-free and generalizable tool use for llms via finite-state decoding." arXiv preprint arXiv:2310.07075, 2023

W 1.2. Limitation of Novelty

Thanks for your comments, but we believe that our work introduces sufficient contributions and novelty . From a methodological perspective, although a few works have introduced constrained decoding for tool usage, our method stands out as it is training-free and can be better applied to a broader range of models, as demonstrated in the comparisons above. Additionally, we propose order consistency, which can further enhance performance, building on the significant improvements achieved by constrained decoding. From an experimental perspective, we conduct detailed analyses and comprehensive experiments across a wide range of models, achieving performance gains exceeding 70% for nearly all models, as shown in Table 7 and 8. As a plug-and-play method, our approach further enhances performance when combined with tool-finetuned models and prompt engineering, highlighting its flexibility and effectiveness, as shown Table 3 and 7.

Dear Reviewer hS7t,

We have carefully prepared a response to address your concerns. Could you kindly take a moment to review it and let us know if it resolves the issues you raised? If you have any additional questions or suggestions, we would be happy to address them.

Thank you for your time and consideration. Wishing you a great day!

The Authors

Q1.1. Value errors increased in most scenarios after applying Tool Decoding.

The added value errors are not caused by Tool Decoding but are potential value errors previously hidden by format and key errors. Specifically, parameter values can only be extracted when the format and keys are correct. By addressing these format and key errors, Tool Decoding makes more values can be extracted and exposes these hidden value errors, leading to a slight increase in the observed proportion of value errors.

In contrast, our method mitigates value errors: constrained decoding does not impact value generation, while order consistency actively reduces value errors, as demonstrated in Table 4.

Q1.2. For format errors, the example in Table 1 shows a missing parenthesis. How would Tool Decoding help in this case?

Briefly speaking, we detect whether the last value has been fully generated, and once completed, the model is constrained to generate the closing parenthesis $)$.

For example, suppose tool_A holds 2 required parameters, $\alpha$ and $\beta$, and has 1 optional parameters, $\gamma$. Once the last required parameter, $\beta$, is completely assigned, we constrain the model’s vocabulary space to [$\gamma,)$]. This leads to two possible scenarios:

1. If the model outputs $)$, the tool call is considered complete.
2. If the model outputs $\gamma$, the assignment process continues for $\gamma$. After the value of $\gamma$ is fully generated, since all parameters have been completed, we directly supply $)$ to finalize the tool call.

Maybe your question is how we determine whether a value has been completely assigned. In expressions like [ToolName(key1=value1, key2=value2)], there is no special token explicitly indicating this completion. Most models encode punctuation marks, such as commas, spaces, and quote marks, as part of surrounding tokens. For instance, the expression "key1=value1, key2=value2" might be tokenized as [“key1”, “=”, “value1,”, “ key2”, “=”, “value2”].

To address this, we determine whether a value has been completely assigned by checking if a full variable of the given type has been generated. Consequently, for Tool Decoding, handling the assignment of the last parameter is no different from handling the previous ones.

Q 2. Latency analysis

Thank you for your constructive advice! We supply the results below and exhibit the low latency of our method. The analysis has been added in Appendix C.

The Table below shows the inference speed (sec/sample) for Tool Decoding in comparison with greedy search and beam search. Tool Decoding with $oc \leq 1$ introduces only slight latency, while Tool Decoding with $oc \leq 6$ is faster than beam search with the same number of samples. This is because order consistency maintains multiple candidates only during the generation of the tool call, which constitutes just a portion of the entire response. Note that our current implementation applies constraints at the level of logits. For practical deployment, these constraints could be implemented at the language head layer, which would further reduce computational requirements and enhance processing speed.

We thank Reviewer PjPG for providing constructive comments for our work. We will address your concerns in the following points. The results below have all been added in our paper.

W 1. Comparison with other methods.

Thanks for your suggestion! Since Tool Decoding is a plug-and-play method, we can combine it with many other approaches such as prompt engineering or trained models. We further expand our comparisons with more approaches as well as the combination of them and our method. As shown in the following, Tool Decoding is orthogonal to these methods and can largely improve the performance of them. And the results are also added in Section 4 and Appendix E.

In-context learning

As a plug-and-play method, Tool Decoding integrates seamlessly with prompt engineering. Table below presents the performance of different numbers of in-context examples on API-Bank (Call). The results demonstrate that our method effectively combines with prompt engineering, significantly enhancing the model’s tool usage capabilities. Notably, this combination even enables a 7B-level generalist model, such as deepseek-coder-6.7b, to surpass GPT-4 under the same prompt settings, as highlighted in Bold values.

Trained models

Please note that we have already combined our method with some trained models and compared with them in Figure 5, Table 7 and Table 8 (xLAM-7b-r and Toolformer). We cite the results on API-Bank below.

In Table below, we also conduct further experiments on gorilla-openfunctions-v2, one of the most advanced trained models. Tool Decoding demonstrates significant improvements across all three models, with both xLAM-7b-r and gorilla-openfunctions-v2 outperforming GPT-4 on API-Bank.

W 2. Inadequate acknowledgment in the error analysis part.

We apologize for this oversight. We have now adequately acknowledge it at the beginning of the Analysis of Errors part in Section 2: "While some existing works have conducted coarse error analyses, their evaluations are not sufficiently comprehensive and lack a systematic approach. For instance, the analysis in API-Bank (Li et al., 2023) overlooks value errors and includes ambiguous error types such as Has Exception, limiting both clarity and utility. In contrast, we conduct a stage-specific and comprehensive error analysis, systematically identifying errors at each stage to derive fine-grained insights."

Thank you for raising the score! We are delighted to know that our responses addressed your concerns. If you have any additional questions or feedback, please feel free to let us know. Wishing you a wonderful day!