This paper proposes Tau-Bench, a benchmark for measuring the performance of models on user/agent interactions where the model has the ability to use tools. The goal of each interaction is to learn information from the user, and achieve some end state the user wants the agent to accomplish, such as re-booking a flight or initiating a return through an online store. They also introduce a pass^k (similar to pass@k) metric, measuring how often an agent succeeds at a given task when the task is re-run with slight variations in syntax/semantics/etc, simulating multiple users attempting to accomplish the same goal, to measure how reliable an agent is.

The paper proposes a new LLM-agent benchmark for two domains (retail and airline), called tau-bench. The gap that the proposed tau-bench aims to fill is LLM-agent benchmarks that focus on User and LLM agent interactions. I.e. measuring how well a LLM agent performs when interacting with a "human" user. To this end they propose to use a LLM based user simulator. Furthermore, the paper describes how the benchmark is constructed and based on databases and APIs. Finally, the authors also propose a new metric for the benchmark, pass HAT k (pass ^ k). This metric measures what the chance is that all passes are successful given k i.i.d. attempts.

The paper has a detailed evaluation of agents performance on the benchmark, detailed analysis of errors of these agents and some cost analysis of both the agent (LLM-based) and user-simulator (LLM-based). The cost of a single evaluation cost $200. There is also some analysis of user simulator performance.

This article introduces a novel benchmark called τ-bench, designed to evaluate the ability of language agents to interact and plan with human users and programmatic APIs in real-world scenarios. The main contributions of the article can be summarized as follows:

• Introduction of the τ-bench Benchmark: τ-bench simulates dynamic dialogues between users and language agents equipped with domain-specific API tools and strategy guidelines. It is built on a modular framework, featuring realistic databases and APIs, domain-specific strategy documents, and detailed instructions for various user scenarios, along with corresponding real annotations.
• New Evaluation Metric, passˆk: In addition to measuring the success rate of agents in task completion (passˆ1), the article proposes a new metric, passˆk, to assess the consistency and robustness of agent behavior across multiple independent trials. This metric is particularly important for real-world agent tasks requiring reliability and consistency, such as customer service.
• Comprehensive Experiments on Existing Language Models: The authors conduct extensive tests using τ-bench on a variety of state-of-the-art proprietary and open-source language models, including GPT, Claude, Gemini, Mistral, and Llama. The experimental results indicate that even the most advanced functional calling agents (e.g., gpt-4o) succeed in less than 50% of tasks and exhibit poor consistency (with passˆ8 < 25% in the retail domain).

the paper build a novel agent evaluation benchmark considering the complexity of domain policy, user-agent interactions with realistic databases and APIs. Based on that, the paper propose new evalaution metric pass^k to determine the robust and consistency of agent, and present insightful analysis about existing LLMs.

We appreciate the positive feedback from reviewers!

Our contributions and enabled future directions

Our work has several important contributions:

• The benchmark itself, and its detailed design principles and implementations
• The idea to combine (we acknowledge prior work for each aspect individually, but only combining all three can lead to tasks with realistic tool-agent-user interactions) real-world APIs, domain-specific policy following, and user simulation, which we believe are three key gaps of today’s agent benchmarks from real-world applications.
• The introduction of pass^k metric for reliability and robustness, which can be used in other tasks
• Comprehensive experiments across various models and various analyses

Our current work already has vast content, details, and workloads (acknowledged by all reviewers), so we could not pursue many interesting (and natural) followup directions from tau-bench, which we have to leave as future research:

• Better user simulation methods, and better ways to evaluate user simulation in tool-agent-user interactions; see the next section for more details. (EMki, p3GC)
• Better agent methods (e.g., RAG, fine-tuning, few-shot prompting) to handle tool-agent-user interactions (p3GC, D4Hj)
• Better metrics for fine-grained evaluation of tool-agent-user interactions (EMki)
• Better ways to automatically and scalably construct benchmarks for tool-agent-user interactions, beyond manual design of APIs/policy/tasks

Note that none of these directions are possible to pursue before the introduction of tau-bench. So as stated by reviewer D4Hj, our work “provides a strong foundation for future work in this area”, and we are excited to see followup research in these various exciting directions.

User simulation and its evaluation

Unfortunately, LLM-based user simulators cannot be guaranteed to always work, just like no other LLM-based systems (or even real humans) can be guaranteed to never hallucinate. Instead of formal guarantees, we believe it is much more tractable (and indeed extremely important) to establish empirical evaluations of user simulation. We have shown two approaches in our work (see Section 5.3):

1. (Main) Human examination of trajectories, where we find our user simulation currently only has a low error rate of 4%.

2. Controlled experiments with the same agent paired with different user simulators. In Table 4, we find that using slightly more complicated prompting methods (e.g., reflection, verify) slightly improves the agent-user collaborative performances, but not a lot (partly because user error rate is already low in current tau-bench). Our additional experiments using gpt-4o-mini as user simulator also shows that a weaker user simulation model will lead to lowered agent performance and increased user errors, so agent performance could be a noisy proxy to user simulation performance.

• gpt-4o: $0.399/task

  • Retail: agent pass^1 = 60.4%, user simulation error = 2%
  • Airline: agent pass^1 = 42.0%, user simulation error = 2%

• gpt-4o-mini: $0.024/task

  • Retail: agent pass^1 = 52.3%, user simulation error = 11%
  • Airline: agent pass^1 = 32.0%, user simulation error = 13.2%

In summary, for today’s tau-bench tasks, which are small-scale and relatively simple, we find that user simulation has a very low error rate, and human examination can be effectively used to evaluate user simulation. However, if future tasks evolve to be large-scale and significantly more complex, we’d expect today’s models and naive methods for user simulation to have more errors, and human examination to be less tractable for evaluation. In such a future case, we believe it is important to study better ways to construct and evaluate user simulation. We will revise our draft and incorporate such a discussion.

Overall, this work is very interesting and a good contribution.

The main concern remains the wider usability. However, we also acknowledge that this can be directed to future work.

The paper "$\tau$-bench" introduces $\tau$-bench, a benchmark for evaluating LM interactions with users and tools in domains like retail and airlines. It combines APIs, realistic databases, and policy adherence with a novel pass^k metric for reliability. Comprehensive experiments show current models struggle with consistency and rule-following, highlighting future research needs.

Strengths include the innovative benchmark design, the novel metric, and thorough experiments. Weaknesses involve overclaimed novelty, limited related work citations, and user simulation costs. Despite this, the benchmark’s utility for future work makes it recommendable for acceptance.

Accept (Poster)