The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models

We thank the reviewer krTJ for the time to review our paper and the comments. We emphasize the motivation, purpose, and contribution of our work. We address your questions as follows.

1. Need more discussions on the comparisons and differences to existing benchmarks

A: Although in recent times quite a few benchmarks have been proposed to test LLM’s ability to perform function calling, in this work, we focus on those designed to evaluate a model’s native function-calling capabilities, rather than methods that rely on prompt-based or code-generation approaches (e.g., Nexus Raven uses a prompt-based protocol, and AppWorld emphasis code generation capability). Among benchmarks that do evaluate native function calling, many such as App Blend and API Bench focus solely on single-turn interactions. Although TinyAgent addresses nested function calls, it does so by using placeholder variables instead of letting the model see the actual execution output of earlier calls; in effect, it still operates under a single-turn framework.

For benchmarks that truly cover multi-turn interactions, most are constrained by a narrow domain scope or a limited set of functions. TauBench, for example, supports only 28 functions spanning two domains (airline and retail), and RestBench offers scenarios solely within the TMDB and Spotify domains. The narrow coverage (<150 entries) is more prone to overfitting and does not sufficiently reflect the breadth of real-world function-calling scenarios. Lastly, benchmarks like ToolSandBox and TauBench rely on LLMs to simulate user queries. Despite careful attempts to control the user simulator’s behaviour, LLM-based users remain prone to hallucination and instruction-following errors, which can confound evaluation results.

Other works, such as ToolBench, depend solely on Rapid APIs that are subject to high variance in performance, making reproducibility a challenge. While the subsequent StableToolBench version mitigates this by caching or simulating API responses, it continues to rely on LLM-based evaluators for determining response solvability, thus risking model-induced biases and undermining objectivity (e.g., GPT-family models tending to favour responses from their own model). Similar issues exist in T-Eval, which also depends on LLM-based evaluation.

Lastly, current benchmarks uniformly employ LLM-curated user queries, limiting their ability to accurately reflect genuine user interactions.

Our benchmark directly addresses these shortcomings by incorporating deterministic evaluation metrics, an extensive and diverse set of robust multi-turn interactions, and a unique multilingual dataset derived from real-world, user-contributed queries that have more than 15 languages represented in user queries, including Chinese, French, Japanese, and Korean, etc. Additionally, on top of Python and REST (which are commonly covered in existing benchmarks), we also have entries in Java and JavaScript for more diverse programming languages.

Collectively, these enhancements enable a more comprehensive, fair, and reproducible assessment of LLM function calling, setting our benchmark apart from existing efforts.

You can find all the datasets and the codebase to run them here: https://anonymous.4open.science/r/FCL. Our evaluation scripts support almost all popular models (100+ proprietary models, and 85+ open-source models).

2. Need more descriptions and examples of how different types of data instances look like, especially on multi_turn

A: A set of example test entries and corresponding ground truth outputs for each test category is available here: https://anonymous.4open.science/r/FCL/TEST_EXAMPLES.md

We thank the reviewer dutu for the time to review our paper and the comments. We address your questions as follows.

1. Need More Discussions on the Comparisons and Differences to Existing Benchmarks

A: Please see our rebuttal comment to reviewer krTJ.

2. Concern Over Potential Train/Test Data Contamination

A:

The scores reported are all run by the authors. We make sure all the scores are reproducible, and publicly archive all the results & scores we obtained, which the community finds valuable.

As the reviewer noted, the Single Turn dataset has indeed been publicly available for some time, making it difficult to prevent developers from potentially training on these test sets. Nonetheless, our Crowd-Sourced dataset was designed not only to capture more authentic user interactions but also to detect potential data contamination. By updating the Crowd-Sourced dataset on a monthly or quarterly basis, we make it considerably harder for model providers to anticipate and train on the test set. In principle, if a model has not been trained on our public test sets, it should exhibit comparable relative performance across both the Single Turn dataset and successive checkpoints of our Crowd-Sourced dataset.

Lastly, in the case of open-source models, we can calculate perplexity to identify signs of data contamination (see Section 5.5 for more details).

3. Unclear AST Substring Matching Procedure for Predicted vs. Ground Truth Sequences

A:

By definition, parallel tool calls are considered to execute simultaneously and, therefore, the order of calls does not matter. If order does matter (e.g., strictly sequential calls), the model must emit one function call at a time and await completion before issuing the next. Consequently, for a predicted sequence A = [a_1, a_2, ..., a_m] and a ground-truth sequence B = [b_1, b_2, ..., b_n], we do not require strict positional alignment. Instead, any predicted call a_i can be matched with any ground-truth call b_j. Our evaluation then adopts an “all-or-nothing” criterion: if at least one ground-truth call is not matched by any predicted call, the entire evaluation is considered a failure. This ensures the model correctly identifies all necessary calls, regardless of order.

On the parameter values, For integer, float:

• For Python tests only, we allow the use of int values for Python parameters expecting float values to accommodate the Python auto-conversion feature from int to float.
• For non-Python languages (Java and JavaScript), if the function documentation specifies float parameter, then it should be a float in the model output (such as 5.0); an int (such as 5) will not be correct.
• Supplying float value for int parameter is not allowed in any language.

For List, Tuple:

• Order matters and the elements must match exactly. For example, [1,2,3] is not equal to [2,3,1]. So for questions where the order in the list doesn't matter, permutations of the possible answer are used to accommodate this situation.
• Note that the type match extends recursively for nested data structures (List or Tuple), where both the outer type and the inner types of elements must match the specified requirements.

For String:

• The evaluation process is case-insensitive.
• All strings will be standardized before checking. This applies to both the model output and the possible answers.
  • All white space is removed.
  • A subset of punctuations ,./-_*^ are removed to make the evaluation more robust and accurate.
• For example,
• Possible Date ["20th June", "2023-06-20", "06/20/2023", "Jun.20, 2023"]
• Possible Location ["New York City", "NYC"]

For Dict:

• The evaluation focuses on the key presence and the accuracy of associated values as per the possible answers.
• Ordering within dictionaries is not considered due to they are inherently unordered.

For lists of dictionaries:

• While the ordering of dictionaries is considered (since it's a List), the order of key-value pairs within each dictionary is not.

This method works for other languages as well. For Java and JavaScript, the evaluation procedure uses tree sitters to convert the string version of a Java/JS type into their corresponding Python format, since we perform checking with Python data types. For example, a HashMap in Java will be converted to a dict in Python. During this process, the converter will also perform the type checking for those parameters; e.g. if the parameter should be a long type in Java, the converter will check to make sure that the string input does have an L at the end (because otherwise, it wouldn't be a valid Java long type).

We thank the reviewer VP8g for the time to review our paper and the comments. We address your questions as follows.

1. The Proposed Settings are Too Similar to Those From BFCL

A: In the interest of double-blind reviewing, we refrain from commenting on this.

2. Need More Discussions on the Comparisons and Differences to Existing Benchmarks

A: Please see our rebuttal comment to reviewer krTJ.

3. Insufficient Dataset Statistics and Details

A: We utilized GPT-4o-0806 throughout the data generation process and incorporated a hybrid validation strategy—combining manual, programmatic, and LLM-based methods—at various stages of data curation. For the FCL benchmark data curation across both single-turn and multi-turn categories, we provide a more detailed description of the methodology here https://anonymous.4open.science/r/FCL/DATA_CURATION_METHOD.md. Additionally, we present comprehensive multi-turn dataset statistics and visualizations https://anonymous.4open.science/r/FCL/DATASET_STATISTICS.md, including distributions of turn counts, input token usage, and other key characteristics across multi-turn categories.

4. Where to find the dataset

A: You can find all the datasets and the codebase here: https://anonymous.4open.science/r/FCL

5. Clarifying What Makes the Agentic Dataset Unique

A: Apart from the statistics that Agent Dataset has a much higher number of turns needed and that more write (as opposed to read) operations are involved, the Agentic Dataset uniquely evaluates tools frequently integrated with LLMs by specifically focusing on core agentic capabilities, independent of their narrow functional uses. Unlike general-purpose function calling evaluations, which typically assess context-free functions (e.g., a 'pause_music' command relevant primarily to virtual assistants like Siri or Alexa), the Agentic Dataset targets fundamental, broadly applicable agentic skills such as web browsing, memory management, and database querying. Take memory management as an example, the fact that it spans across multiple sessions is something you would never see in a normal/standard multi-turn dataset, yet it is critical for true agentic behaviour.

These abilities constitute the backbone of effective LLM agents but have not yet been systematically examined at scale. While there are attempts to evaluate efficient incorporation of web search knowledge or text-to-SQL, our work is unique in systematically treating these functionalities as integral tool calls within the broader agentic paradigm rather than standalone tasks.

6. Defining “Memory” and Its Role in the Memory Dataset

A: Memory refers to how the model retains and utilizes information from previous interactions so that it can provide coherent, contextually relevant responses over multiple sessions. In practical deployments (e.g., chatbots), relevant conversation data will be stored in an external database. For example, a healthcare agent might store a patient’s reported allergies or lab results for reference in future conversations. This stored information is generally not visible to the user, but they will be given a “wipe memory” command.

The Memory Dataset in our work is designed to test the model’s ability to:

• Identify which pieces of information from past interactions need to be stored.
• How to find the right balance between multiple contexts when the memory database is limited in size or entry count.
• Retrieve those details as necessary to respond to new queries in a later session.

8. Evaluating Read-Only Calls: Response-Based vs. State-Based Approaches

A: State-based evaluation is limited for read-only requests because these requests do not fundamentally alter the environment. For instance, if a user wants to list files in a directory, using commands like cd and ls repeatedly does not change the underlying filesystem state in a way that would distinguish one action from another, so a traditional state-based metric offers little insight.

Moreover, incorporating outputs of read-only calls directly into the state essentially converts the evaluation into a response-based evaluation.

We want to measure whether the model invokes the right tools at the right times, instead of relying on outdated intrinsic knowledge. By focusing on response-based evaluation for these scenarios, we hope to more accurately gauge the model’s real-world tool usage and ensure it is actually calling necessary functions rather than guessing or leveraging intrinsic information.

9. How the Model Receives and Understands Environment State

A: The model won’t be directly informed of the entire state of the environment. Instead, we equip the model with functions that can query partial or entire states of the environment, such as ‘ls’ in a file system or ‘display_tire_pressure' in a vehicle control system. We want the model to not take the state information for granted, but instead actively explore the state and retrieve as needed.

We thank the reviewer for the time to review our paper and the comments. We emphasize the motivation, purpose, and contribution of our work.

1. Need more discussions on the comparisons and differences to existing benchmarks

A: Please see our rebuttal comment to reviewer krTJ.

2. Uneven score distribution across different dataset subsets. Near-perfect scores for single turn while extremely low score for agent memory category

A: Single-turn tasks remain critical for two reasons - first, they form the foundational building blocks for more complex, multi-turn, and agentic workflows. Second, they help weed out the “weaker” models. Despite rapid progress in the LLM field, our updated results show that smaller models (e.g., 27B, 7B, 3B, 1B) still struggle on single-turn entries, indicating the ongoing need for these categories to differentiate model capabilities and guide their development. Even “flagship” models occasionally fail on certain single-turn tasks (e.g., claude-3.7 can’t do any parallel tool calls, and Llama-3.3-70B cannot refrain from tool call when the query is asking for weather information while the only function available is get_coordinate_by_city), demonstrating that near-perfect scores in one subset do not guarantee universal mastery.

As for the performance gap within the Multi Turn or Agentic category, these tasks are deliberately much more challenging, requiring context retention, dynamic planning, and advanced reasoning skills. Our findings reflect that while many models excel at basic single-turn interactions, they still struggle with multi-turn scenarios. The best model in multi_turn_base can only get 60% correct, and even lower on the more advanced miss_func or miss_param settings, and only 12% in the agentic memory subcategory.

We believe these results underscore the value of our benchmark: it captures the full spectrum from fundamental single-turn tasks to complex agentic use cases. To further illustrate this progression, we have updated Table 1 with more model sizes. And Figure 7 in the appendix shows how performance on single-turn tasks has steadily improved across multiple checkpoints for flagship models. Together, these additions demonstrate the ongoing relevance and evolving nature of our benchmark across different model scales and capabilities.

3. Insights derived seem obvious

A: The insight is not just the relative position of models (e.g., 4o better than 4o-mini), but rather to unearth interesting patterns, such as the fact that Claude-3.7 fails to handle parallel tool usage… These kinds of fine-grained observations have proven to be valuable to developers of large language models. Indeed, leading LLM providers have cited this work to identify and focus their post-training efforts based on the observations of this benchmark. For example, Google refined Gemini’s parallel tool call mechanism based on our findings, while Meta enhanced Llama’s handling of precise parameter value types by analyzing traces from our dataset. This evidence demonstrates that our work generates concrete, practical insights, rather than just confirming that larger models outperform smaller ones.