We thank Reviewer SUbn for his/her recognition of our work and constructive comments that will surely turn our paper into a better shape.

Q1 The paper claims that the model is designed for on-device function calling; however, it lacks sufficient explanations or evaluation results to demonstrate its efficiency or performance in such scenarios. The authors did not discuss the hardware configurations required to run the model on edge devices or address potential resource constraints. Additionally, the authors did not provide evaluation results regarding inference time, which are essential for assessing the model’s practicality in real-world, resource-limited environments.

A1 We apologize for making this confusion. The demand for on-device applications is the primary motivation behind our research. Initially, our goal was to train an effective function-calling model to serve as the foundation for building on-device agents, such as mobile personal assistants. The accuracy of function-calling is critical, as it largely determines the success of task execution for on-device agents. Based on this, we focused on key aspects such as lightweight models (no larger than 7B), robustness, and balancing irrelevance detection with function-calling performance. The table below further illustrates the non-functional metrics and hardware configurations of our Hammer-7B model when deployed on mobile devices after quantization:

After quantization, the model experiences approximately 2% performance degradation, which remains within an acceptable and reasonable range. The table demonstrates that our Hammer-7B model is already capable of running on mobile devices. However, its inference speed still has significant room for improvement due to the current lack of universal and efficient edge-side inference frameworks. Nevertheless, we are confident that our community will address this issue in the near future.

We greatly appreciate the reviewer's reminder. We will include this table in the revision to further clarify the rationale behind our choice of model size and its relevance to on-device applications.

Q4 The random string generation process for masking isn't fully specified - different approaches to generating random strings could affect reproducibility.

A4 We sincerely thank the reviewer for their thoughtful reminder. We will add the following details regarding the randomization process in Section 4.1: "We first determine the length $L$ of the random string using random.randint(5, 15). Then, we randomly select $L$ characters from a character set comprising 52 uppercase and lowercase letters and 10 digits, to replace the corresponding function or parameter names."

Additionally, to ensure reproducibility, we have made all relevant code and training data publicly available in the supplementary material.

Q5 The exploration of the trade-off between masking and maintaining semantic meaning in function names is limited.

A5 Thank you very much to the reviewer for raising this concern. We also noticed this during Hammer's training. Considering the complexities of real-world scenarios, we do not apply the masking strategy to all the training samples. Instead, we set a masking ratio (We found that a masking ratio of 0.33 yields the best overall performance for Hammer across all benchmarks). Before the start of each training epoch, we randomly mask a specific ratio of the samples. This approach ensures a balance between allowing the LLM to infer functionality from names and encouraging it to focus on descriptions. The strong performance of Hammer on the Tool-Alpaca benchmark confirms the effectiveness of this trade-off since Tool-Alpaca contains function candidates in the test set with very limited descriptions.

Q6 How does the function masking technique affect the model's zero-shot generalization to entirely new API schemas or documentation formats not seen during training?

A6 Thank you for your thoughtful question. Theoretically, the function masking technique might offer a slight advantage over traditional methods in addressing zero-shot generalization to entirely new API schemas. This is because it at least circumvents the naming style drift introduced by designers' preferences in new API schemas. However, considering the overall formats, the improvement brought by directly applying function masking may be limited. The primary reason lies in the zero-shot setting, where the model may struggle to locate or distinguish among function names, parameter names, and function descriptions due to a lack of structural knowledge about the new schemas.

To address this challenge, we can consider modifying the masking technique. Specifically, by randomizing function names and parameter names while adding type tags (e.g., param: random_str, func: random_str, and desc: xxxx), the model can be trained to locate each element using the tags such as param:, func:, and desc:. Intuitively, this modified masking approach should further enhance the trained model's performance in zero-shot generalization to entirely new API schemas.

We hope this response meets your satisfaction.

We thank Reviewer 95n3 for his/her constructive comments that will surely turn our paper into a better shape.

Q1 The paper doesn't include the comparison with some recent relevant baselines (e.g., ToolBench, API-Bank).

A1 We appreciate the reviewer for recommending these relevant works. In our work, API-Bank serves as one of the primary benchmarks, and we have conducted comprehensive experiments based on it, with results presented in Tables 1, 3, 5, and Figure 5, and a detailed introduction provided in Section 5.1.

Regarding ToolBench, we also noticed this benchmark. However, we chose not to utilize it for the following reasons:

1. Access Limitations: ToolBench requires applying a specific ToolBench key for access. We attempted to request it early on, but unfortunately, we never received a response. We also observed that many other users reflected the same issue at their code repo.

2. Potential Data Contamination: The xLAM team revealed in their recently released paper [1] that they generate the xLAM-function-calling-60k dataset, which serves as our train set, with ToolBench. This poses a significant risk of data contamination if ToolBench is used as a benchmark, potentially leading to unfair advantages compared with certain baselines.

Nevertheless, based on the latest findings outlined in Point 2, we will update Section 5.1 to include a discussion on ToolBench, providing further clarification on the rationale behind our benchmark selection.

[1] Zhang, J., Lan, T., Zhu, M., Liu, Z., Hoang, T., Kokane, S., ... & Xiong, C. (2024). xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215.

Q2 There are no ablation studies on the effect of different description formats or lengths on model performance.

A2 We are very grateful for this insightful suggestion. We have conducted an ablation based on the Live dataset in BFCL-V2 to further investigate the effect of different description lengths in labels on model performance (Hammer-7B and xLAM-7b), with results shown as below:

Based on the results in the table, we observe that Hammer demonstrates a greater advantage in cases with longer descriptions, while its advantage is less pronounced in cases with shorter descriptions. However, since the number of samples with long descriptions in the test set is relatively small, the confidence of this conclusion based solely on the table may not be sufficiently robust. Nonetheless, the analysis in A5 provides supplementary evidence: Hammer's performance advantage is lower on Tool-Alpaca, which contains function candidates in the test set with very limited descriptions, compared to other benchmarks. This further corroborates our observation from a different perspective.

Q3 There’s no discussion of potential failure modes or edge cases where function masking might perform worse than traditional approaches.

A3 Thank you very much for the reviewer’s suggestion. The statement that "function masking might perform worse than traditional approaches" could be true in the following two cases.

The first case is shown in the ablation study in our Section 5.5 with result in Figure 5 and summary in lines 486-489: "A large mask ratio can impede the model’s learning speed within the same task scenario but facilitate better generalization of the model across different scenarios." This suggests that when the training and testing sets are from the same source or have similar distributions, function masking might perform worse than traditional approaches.

Another situation arises in cases where there is little description available. In such cases, due to the incomplete information in the descriptions, directly masking all names might be too aggressive, leading to a decline in performance. The comparison on the Tool-Alpaca benchmark suggests this possibility since it contains function candidates in the test set with very limited descriptions and Hammers achieve perormance very close to baselines. If the description is further simplified, function masking might perform worse than traditional approaches as well. Fortunately, we also had this concern during Hammer's training and proposed trade-off referred to A5.

Thanks again to the reviewer for their valuable feedback. We will include this discussion in the experimental section of the paper.

We thank Reviewer nBzQ for his/her recognition of our work and constructive comments that will surely turn our paper into a better shape.

Q1 The method is specifically designed for function calling problems; it is unclear whether some other domains will also benefit the tunning framework.

A1 We appreciate the reviewer's thoughtful question. While the tuning framework in this work is primarily applied to enhance the model's function-calling performance, we believe its core idea holds potential benefits for other domains as well. Meanwhile, we also extend our gratitude to Reviewer SUbn for his insight that "this method is potentially replicable and offers inspiration for ai safety practitioners in other fields who seek to improve model robustness."

Besides, the central insight of this work lies in leveraging a masking mechanism to guide the model's attention away from incomplete, potentially misleading, or redundant input elements and toward more complete and robust components, thereby improving overall model performance. This approach could benefit any task with similar characteristics. For example, when training a web agent, we could mask out website names or domain names to encourage the model to focus on the site's description. Similarly, when training LLMs to play various card games, we might guide the model to focus on the functionality of the cards rather than their names. From another perspective, a well-tuned function-calling model serves as a foundational capability for deploying LLMs across many domains, as the specific actions executed in numerous tasks can often be conceptualized as variations of "function-calls", such as the agent selection in multi-agent communication tasks.

Further, considering that different domains may prioritize different information, in the future we also plan to explore the design of more dynamic masking mechanisms as alternatives to static name masking.

Q2 I am not sure how this strategy is specifically related to on-device language models; It can also be applied to larger models, and it would be really great to see the performance of larger models such as 70B llama if they are applied with this method.

A2 We apologize for making this confusion. The demand for on-device applications is the primary motivation behind our research. Initially, our goal was to train an effective function-calling model to serve as the foundation for building on-device agents, such as mobile personal assistants. The accuracy of function-calling is critical, as it largely determines the success of task execution for on-device agents. Based on this, we focused on key aspects such as lightweight models (no larger than 7B), robustness, and balancing irrelevance detection with function-calling performance. The table below further illustrates the non-functional metrics of our Hammer-7B model when deployed on mobile devices after quantization:

After quantization, the model experiences approximately a 2% performance degradation, which remains within an acceptable and reasonable range. We will include this table in the revison to further clarify the rationale behind our choice of model size and its relevance to on-device applications.

Moreover, we agree that our strategy is applicable to larger models. In fact, we are very eager to explore the performance limits on even larger models, such as 70B, to push the boundaries of what can be achieved. However, due to our limited computational resources, it has been challenging to complete the training of a 70B model within the constraints of a limited time window (we are still actively working on this). At the same time, this limitation is also one of the key motivations for making all our materials—code, data, models, etc.—open source, since we sincerely hope that interested researchers will join us in this exploration.

Q3 For the cases where there is little description, I am not sure if we can still leverage the masking strategy.

A3 Thank you very much to the reviewer for raising this important question. We also have this concern during Hammer's training. Considering the complexities of real-world scenarios, we do not apply the masking strategy to all the training samples. Instead, we set a masking ratio (We found that a masking ratio of 0.33 yields the best overall performance for Hammer across all benchmarks). Before the start of each training epoch, we randomly mask a specific ratio of the samples. This approach ensures a balance between allowing the LLM to infer functionality from names and encouraging it to focus on descriptions. The strong performance of Hammer on the Tool-Alpaca benchmark confirms the effectiveness of this trade-off since Tool-Alpaca contains function candidates in the test set with very limited descriptions.

In addition, in Tool-Alpaca, even the performance achieved by GPT-4 is quite limited. This highlights another insight we aim to convey: in real-world scenarios requiring function calls, providing comprehensive natural language descriptions should be prioritized. That is, in practice, when descriptions are incomplete, supplementing functions in the library with detailed descriptions is often simpler, more direct, and more efficient than training a model to infer the function's purpose from the limited information.

We hope this clarifies our rationale and approach. Thanks again for this valuable feedback.

Q3 More detailed error patterns analysis would be beneficial to understand the model and baseline failure patterns across different benchmarks. For Table 1’s inconsistent performance across benchmarks, I am curious why does xLAM perform well on three benchmarks but poorly on two others? Could the authors provide some error pattern analysis and related insights?

A3 Thanks for the reviewer's suggestion. We have conducted categorical statistics for the various types of failed cases appeared in the Seal-Tools and Nexus Raven benchmarks where xLAM performs poorly, to provide readers with a more detailed and comprehensive understanding. The specific results are as follows, with "#cases with Hammer | #cases with xLAM" in each grid:

In this context, "Correct" refers to the number of cases where the outputs are entirely correct; "Func. Error" indicates cases where the function itself was incorrectly selected; "Param. Error" represents cases where the function was correctly chosen, but parameter filling was erroneous; and "Reject Error" denotes cases where the model incorrectly deemed all candidates irrelevant.

From the table above, we observe that the performance degradation of xLAM on these two benchmarks primarily stems from an overly aggressive rejection strategy, which fails to accurately assess the relevance between candidates and queries. This issue is particularly pronounced in Nexus Raven. Then, we conducted a further investigation into the test data of Nexus Raven and found that the distribution of parameter number in its function library is exceptionally broad, ranging from 0 to 27 parameters, whereas other benchmarks typically do not exceed 10 parameters. Consequently, we further analyzed the impact of varying numbers of parameters in the labels within Nexus Raven on the performance of xLAM and Hammer. The details are as follows:

From the table above, we can observe that xLAM's rejection rate significantly increases when handling functions with parameters of five or more, and all such cases are rejected when the number of parameters reaches ten or more. In contrast, the hammer method, which employs a masking technique, is still able to maintain a certain level of accuracy. We sincerely thank the reviewer for their insightful guidance. This analysis indirectly corroborates that xLAM exhibits misleading issues caused by the parameter name when assessing the relevance between candidates and queries (as mentioned in Section 3.1 of our paper).

Q2 Could you provide more insights into other factors contributing to Hammer’s performance? Have you tried using DeepSeek models as base models (just like xLAM) instead of just Qwen? This would be helpful to understand what specific aspects of the base model selection impacted results. Also, it would be beneficial to know if there are other training tricks beyond data augmentation.

A2:

Comparison of Different Base Models

As highlighted in Section 5.5 (Table 5), we have experimented with DeepSeek models as base models, similar to xLAM. The results demonstrate that our approach applied to DeepSeek also outperforms xLAM in terms of overall performance. Even more surprisingly—and here we would like to sincerely thank reviewer peJz for their insightful suggestion—upon further investigation of the training details of xLAM models from their newly released paper [1], we discovered that xLAM-fc models might still have 50% of their training data that had not been disclosed in the original paper [2]. This finding indicates that our pipeline achieves superior performance across multiple benchmarks despite utilizing only half of the available data.

Inspiration from Semantically Similar Cases

Beyond the masking technique, the reviewer’s point regarding semantically similar cases has inspired us to consider another potential factor contributing to improved performance: further balancing the ratio between semantically similar samples and masked samples. While we have not yet adjusted the ratio of semantically similar data in xLAM-function-calling-60k, we recognize this as a promising avenue for future exploration. We are deeply grateful to the reviewer for this valuable suggestion.

Future Directions in Data Augmentation

Finally, apart from data augmentation, there exist approaches like [3,4] that dynamically select different subsets of data for training across epochs to enhance performance. Beyond this method's utility in improving training outcomes, we are also planning to explore a metric-based approach to dynamically determine which data samples should be masked rather than relying on random masking. We believe this could further enhance the performance of our models.

[1] Zhang, J., Lan, T., Zhu, M., Liu, Z., Hoang, T., Kokane, S., ... & Xiong, C. (2024). xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215.

[2] Liu, Z., Hoang, T., Zhang, J., Zhu, M., Lan, T., Kokane, S., ... & Xiong, C. (2024). Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518.

[3] Yang, Y., Wang, H., Wen, M., & Zhang, W. (2024). P3: A Policy-Driven, Pace-Adaptive, and Diversity-Promoted Framework for Optimizing LLM Training. arXiv e-prints, arXiv-2408.

[4] Wang, X., Zhou, Y., Chen, H., & Zhu, W. (2024, May). Curriculum Learning: Theories, Approaches, Applications, Tools, and Future Directions in the Era of Large Language Models. In Companion Proceedings of the ACM on Web Conference 2024 (pp. 1306-1310).

We thank Reviewer zD7o for his/her constructive comments that will surely turn our paper into a better shape.

Q1 The authors replace function names with random strings, which may be overly aggressive since function names often contain valuable semantic information. Have you tried replacing function names with semantically similar ones instead of random strings? This could be more meaningful since function names are important features in practice and descriptions and parameters may sometimes be missing. In some usage scenarios LLMs need to infer functionality from names alone.

A1 Thank you very much to the reviewer for raising this important question.

In fact, our initial approach to enhancing the model's robustness also focused on leveraging semantically similar rewrites with GPT-4o for data augmentation. This approach indeed provided some improvements in robustness. However, we ultimately decided against adopting this strategy for the following reasons:

1. The xlam-function-calling-60k dataset already incorporates this aspect by including samples with semantically similar names. For instance, there are multiple function examples for retrieving addresses based on IP:
   • get_ip_location: Retrieves the latitude and longitude coordinates of a given IP address.
   • ip_lookup: Fetches the geographical information for a given IP address.
   • get_place_by_my_ip: Fetches the geographical location related to the IP address of the requester.
   • get_geolocation_data: Fetches geolocation data for a given IP address.
2. The cost of performing GPT-based rewrites for all data is significantly higher than that of random masking operations. Additionally, the pipeline for this approach is more complex. Given the dataset's inherent inclusion of semantically similar samples (as mentioned above), the marginal benefits of using GPT-based rewrites are relatively limited compared with masking operations.

Regarding the concern about overly aggressive issues, we also noticed this during Hammer's training. Considering the complexities of real-world scenarios, we do not apply the masking operatin to all the training samples. Instead, we set a masking ratio (We found that a masking ratio of 0.33 yields the best overall performance for Hammer across all benchmarks). Before the start of each training epoch, we randomly mask a specific ratio of the samples. This approach ensures a balance between allowing the LLM to infer functionality from names and encouraging it to focus on descriptions. The strong performance of Hammer on the Tool-Alpaca benchmark confirms the effectiveness of this trade-off since Tool-Alpaca contains function candidates in the test set with very limited descriptions.

We hope this clarifies our rationale and approach. Thanks again for this valuable feedback.

Q4 The presentation of results is not well-organized.

A4 We greatly appreciate reviewer's valuable feedback in helping us enhancing the paper's presentation and readability. We will make the following adjustments to the existing tables and figures:

1. For Tables 3 and 5, we will retain only the "F1 Func. + Args" entries (the final metrics) in the main text, while moving the remaining details to the appendix.
2. Table 4 will be split into two separate tables (AST and Exec) to ensure the information is easier to follow and analyze.
3. We will refine and highlight the function masking examples in Figure 3 to improve clarity and emphasis.

Q5 Could the authors provide a more detailed analysis of function name issues? Specifically, how many failed cases are due to function or parameter name errors?

A5 Thanks for the reviewer's suggestion. We have conducted categorical statistics for the various failed cases presented in Figure 2 to provide readers with a more detailed and comprehensive understanding. The specific results are as follows, with "#cases with Hammer|#cases with xLAM" in each grid:

In this context, "Correct" refers to cases where the prediction is entirely correct; "Func. Error" indicates cases where the function itself was incorrectly selected; "Param. Error" represents cases where the function was correctly chosen, but parameter filling was erroneous; and "Reject Error" denotes cases where the model incorrectly deemed all candidates irrelevant. The table confirms xLAM's overreliance on function and parameter names, as the frequency of various errors increases significantly when function and parameter names are masked. We will replace the figure 2 with this table since it provides more information.

Q6 It would be beneficial for the authors to analyze how many failed cases result from irrelevant function calls. The urgency of this issue is not apparent in the current version.

A6 Thank you for your thoughtful question. We analyzed the distribution of failed cases in the irrelevance detection scenario before and after using the irrelevance-augmented dataset for training the Hammer model. The results are shown in the tables below, where GT means "ground truth".

Hammer-7b:

Hammer-7b-without-irrelevance-augmentation:

Since xLAM has not open-sourced all its training data, the results in the table indicate that training solely on the xLAM-function-calling-60k dataset leads to poor performance in detecting irrelevant function calls (almost all ground truth irrelevant cases are predicted as relevant). We believe this strongly underscores the urgency of an open-sourced irrelevance-augmented dataset, and we will incorporate this analysis into our revision.

Q7 While Table 2 shows that Hammer achieves impressive results on AST evaluations, the comparitive performance boosting across other benchmarks in Table 3 does not show the same level of improvement. Is there any analysis to explain this discrepancy? Authors could provide discuss potential reasons for the discrepancy and propose additional experiments or analyses that could help explain the varying levels of improvement observed.

A7 Thank you for the insightful question. Regarding the performance comparison of Hammer against different baseline models across various benchmarks, we acknowledge that the level of improvement can indeed exhibit variation. Our insights into the potential reasons are as follows:

1. As demonstrated in Table 1 of our paper, the performance of baseline models inherently fluctuates across different benchmarks. This instability is exactly what our work is trying to solve.
2. Certain benchmarks, such as Tool-Alpaca, contain function candidates in the test set with very concise descriptions. In such cases, the information Hammer can extract from the descriptions is inherently limited (often overlapping significantly with the function names). Consequently, the performance improvement achieved by Hammer on Tool-Alpaca is smaller compared to other benchmarks. This observation underscores the importance of providing clear and comprehensive descriptions for each function in the library when designing function-calling tasks.

Q3 Whether the performance boosting comes from the effectiveness of the proposed pipeline or the Qwen model itself remains questionable. According to the results of ``API-Bank’’ columns in Table 5, when using the same base model, Deepseek-Coder-7B, it is not as effective as the xLAM variant. Authors could have Qwen models trained with xLAM datasets to further elaborate it.

A3 We sincerely appreciate the meticulous reviewer for identifying this potential issue and offering valuable suggestions, which makes our claim stronger. Following the reviewer’s advice, we first trained the Qwen models using the xLAM datasets without masking, i.e. "xxxx-xLAM" in the table below. Results with "F1 Func-Name|F1 Func. + Args" in each grid are presented as follows :

And below are results on BFCL-V2:

By comparing the training results with Qwen base model in the tables above, we find that the models trained using our pipeline consistently outperform those trained on the original xLAM data across nearly all settings, which confirms the effectiveness of our pipeline.

Besides, We would like to sincerely thank the reviewer for identifying this anomaly. It motivated us to further investigate the training details of xLAM-fc, which were not extensively covered in their original paper [1]. Fortunately, in their newly released paper [2] in September, they finally disclosed the data usage specifics for xLAM-fc: “To enhance the function-calling capability of xLAM-7b-fc-r and xLAM-1b-fc-r, we employ a targeted training approach, with 50% of their training data drawn from our high-quality synthetic function-calling dataset. The remaining 50% of the training data is sampled from other tasks.”

This indicates that xLAM-fc models may still have 50% of their training data undisclosed, which we believe is likely the cause of the anomaly observed. This discovery was both surprising and exciting for us, as it suggests that our pipeline achieved superior performance across multiple benchmarks despite only utilizing half of the available data. We would like to once again express our sincere gratitude to the reviewer for their valuable input!

[1] Liu, Z., Hoang, T., Zhang, J., Zhu, M., Lan, T., Kokane, S., ... & Xiong, C. (2024). Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518.

[2] Zhang, J., Lan, T., Zhu, M., Liu, Z., Hoang, T., Kokane, S., ... & Xiong, C. (2024). xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215.

We thank Reviewer peJz for his/her constructive comments that will surely turn our paper into a better shape.

Q1 The proposed methods may inherit potential biases inherent in LLMs. The sampled "Masked Data" can degrade in performance when encountering function names or parameters that differ significantly from those seen during training. This dependency may limit its effectiveness in applications where naming conventions vary widely.

A1 We sincerely apologize that our presentation might lead to misunderstanding. Indeed, the issues raised by the reviewer—"biases inherent in LLMs" and "performance degradation when naming conventions vary widely during deployment"—are critical concerns in real-world applications. However, these challenges are precisely the limitations of existing open-source models that our work aims to address or mitigate.

We pointed out this issue in lines 63-65: “Existing models tend to perform well on benchmarks that closely align with the naming conventions present in the training data but suffer notable performance declines when encountering benchmarks with differing naming styles.” Then, we proposed masking function names and parameter names to address this, as described in lines 263-265: “A direct approach to mitigate these issues involves minimizing the interference from function names and parameter names, while enforcing the model to comprehend the functionality and usage of candidates based on their descriptions.” By doing so, we aim to reduce the performance drops caused by variations in naming conventions, since function descriptions provide a more flexible and less bias-prone natural language explanation.

We greatly appreciate the reviewer for highlighting this concern, as it appears that we did not clearly articulate the logical connection between the motivation (lines 63-65) and the proposed method (lines 263-265). In the revision, we will ensure these sections are more coherently linked to minimize potential misunderstandings.

Q2 There are potential overfitting concerns. The authors do not provide a thorough data contamination analysis for the newly created dataset.

A2 We sincerely appreciate the reviewer for pointing out this issue. We believe this discussion/analysis will help readers better understand the construction process of the irrelevance-augmented dataset and its impact scope on the model. We will add clarification based on the discussion below regarding the data contamination of our newly created dataset in Section 4.2.

All queries/examples in the irrelevance-augmented dataset are sampled from the original xlam-function-calling-60k training set. For each sampled example, we simply remove the correct function from its candidate list and replace its label with an empty list, which indicates that all candidates are irrelevant. Therefore, the construction process of the irrelevance-augmented dataset should not introduce any additional data contamination beyond what already exists in the xlam-function-calling-60k dataset. Besides, xlam-function-calling-60k was released in January 2024, while the BFCL-V2 and Seal-Tools benchmarks used in our experiments were released in August and May 2024, respectively, both after the release of the training set. Therefore, Hammer's performance on these two benchmarks should, to some extent, address the overfitting concerns.

Thank you for your feedback. Most of the concerns are solved. I have raised my score from 5 to 6.