Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

Thank you for your encouraging feedback! Here are the answers to your questions:

Q1.1: I cannot understand the setup of not having any information of the ground truth values at all. Also, can the bidder observe its own utility every round? If it does then it can infer the ground-truth utility right? Can the author please clarify what is the observation space for a bidder each round.

A1.1: We would like to clarify that the bidders have an estimation of the true value. This is because in the traditional auction scenario, the bidders usually have an estimated value for the item based on their prior knowledge, market research or historical prices. However, this estimated value may not exactly be the item's true value [1], due to overestimation or underestimation of the item value (in our case for example, by a percentage of 10%). We follow this setting to approach real-world cases, so that we can study the risk management skills of LLM agents (e.g., winners' curse), where overbids can lead to apparent victories that are actually strategic losses. We will clarify this detail in the revision.

Q1.2: Can the bidder observe its own utility every round? If it does then it can infer the ground-truth utility right? Can the author please clarify what is the observation space for a bidder each round.

A1.2: Yes, each bidder can observe its own utility (i.e., profit or items won) every round to infer the ground-truth utility. For each round, the observation space for each bidder includes: the remaining budget of oneself (budget information is not shared), profits so far for everyone (can be calculated based on the announced true value and the winning bid of an item), the items won so far and their winners, and the auction bidding history.

Q2: In section 3.3. Also I believe winner's curse happens when the players' beliefs of valuations are positively correlated.

A2: Yes, it seems players' beliefs of high valuations are more likely to lead to the winner's curse. However, the winner's curse doesn't necessarily happen if bidders are risk-aware and consider certain auction strategies. For example, they might take this risk into consideration to anticipate the winner's curse, and intentionally undervalue an item or limit their maximum bid (e.g., hedging strategy) to avoid paying too much for it.

Q3: In the adaptivity experiment, what is the learning rule mainly about? And why not incorporate this learning rule in previous experiments?

A3: Just to clarify, did you mean the learning module from Sec 4.3 (Figure 6)? This learning module enables agents to learn from the logs of previous auctions, and summarize some textual takeaways to guide future auctions. We didn't incorporate this since it does not bring much improvement to the agent in general, and it requires previous context for the agent to learn from, whereas most of the experiments are conducted in an "zero-shot" manner.

There are some works on non-parametric learning (i.e., no training for the model) for language agent. The results somewhat contradict those found in [2], where a similar non-parametric learning technique was applied to tasks like HotpotQA. We think this could be because the insights gained from such NLP tasks, such as "don't neglect a negation" or "decompose a sentence" are rather static. However, they might not apply to our auction arena. Unlike the static tasks, our auction environment is dynamic and unpredictable. The past experiences may not be effective in a new context influenced by fresh rounds of conversations and bids.

[1] Klemperer P. Auction theory: A guide to the literature[J]. Journal of economic surveys, 1999, 13(3): 227-286.

[2] Zhao, Andrew, et al. "Expel: Llm agents are experiential learners." arXiv preprint arXiv:2308.10144 (2023).

Thank you for the valuable comments! Please find our responses to your questions and concerns below.

Q1: While auction might be a unique setting, it is unclear what aspects of language model agents would be evaluated through this benchmark.

To emphsize again, our main focus was measuring the strategic planning and execution capabilities of LLM agents in dynamic scenarios, for which the auction setting is well suited. We conducted a detailed breakdown and assessment. A reminder of these different aspects:

• In Sec 4.1, Rationality represents the ability of strategic planning, specifically divided into whether the agent can correctly perceive the environment (measured by CFR) and whether the agent can implement the plan into the correct actions (measured by the correlation between the priority score for a specific item and the number of bids).
• Sec 4.2's Adaptability is the measure of the agent's execution ability in a "dynamic" environment, specifically including the impact of Adaptive Planning and whether resources can be allocated correctly. Additionally, we conducted a competitive analysis in a multi-agent environment, comparing the strengths and weaknesses of different LLMs.

Q2: (Re)Planning [1,2], Instruction-following [3], and belief (internal state of the agents) [4,5] is a typical design for language model agents. It does not seem as a novel proposal or evaluation of language model agents.

We acknowledge that the fundamental abilities of language agents, such as (re)planning, instruction-following, and belief tracking, are indeed well-established in the literature. However, the primary contribution of our work is not in the exhibit of these basic abilities for language agents per se, but rather about how they are applied strategically and adaptively in a real-time, dynamic setting.

Our research introduces a dynamic auction arena that uniquely challenges these agents in strategic decision-making, adaptability, and execution under competitive and resource-constrained conditions. This environment significantly differs from traditional static benchmarks and provides a more realistic and unpredictable setting, where the strategic application of these foundational abilities is critical.

Moreover, the evaluation of agents in this auction arena offers valuable insights into their strategic capabilities and limitations. It demonstrates how these agents perform in real-time, adapt to rapidly changing scenarios, and make strategic decisions to achieve their objectives. This aspect of our research is particularly relevant, as it sheds light on the current state of agent abilities and highlights areas for future development in agent design to enhance their strategic planning and execution.

In summary, the novelty of our work lies in the application of established agent abilities in a unique, competitive auction setting, providing new insights into the strategic capabilities and limitations of current language model agents. Our research contributes significantly to the field by moving beyond static evaluations and demonstrating the potential and challenges of applying these agents in dynamic, real-world scenarios.

Q3: The authors mentioned AgentBench in Sec 2 as ... are limited in terms of evaluating LLMs in dynamic environments., but in fact, they have static offline task planing evaluation (Mind2Web).

In this context, "dynamic" specifically refers to an environment that is constantly changing, shaped by the actions of other agents. This description emphasizes the "unpredictability" of the outcomes that agents face after performing an action. That is, the results are determined not only by the agent's own actions but also by the concurrent actions of other agents. In contrast, Mind2Web, while being a complex system, has a relatively predictable environment, as the outcome is constituted by information that websites provide. This differs significantly from the environment in AucArena. In AucArena, due to the uncertainty of other agents' actions, the environment and challenges faced by agents are more dynamic and unpredictable.

Q4: On the variety of bidding items.

The variety of bidding items is not the primary concern of our simulation. Instead, the choice of items valued at $2000 and$10000 is mainly to make the evaluation more controllable. This setup is intended to mimic two common scenarios encountered in reality: products that are inexpensive with limited subsequent markups/profit, and expensive items that may attract more imaginative premiums. The rationale for setting these two types of items is to prevent the complication of too many confounding factors at play.

Thank you for the detailed response.

My question on (Q7) the difference between Corrected Failure Rate and average profit becomes clear now. Here are the remaining comments:

(Q1, Q2, Q3, Q4)

After reading the response, I still concern that it is unclear what aspects of language model agents would be evaluated through this benchmark. For instance, the authors mention that "dynamic" specifically refers to an environment that is constantly changing, shaped by the actions of other agents. This description emphasizes the "unpredictability" of the outcomes that agents face after performing an action. However, other environments solved with LLMs, such as household simulators, web/computer control, MineCraft, robotic navigation, are also equally "unpredictable" for LLMs because of their open-endedness (LLMs cannot know all the possible consequence in advance). Also, I wonder if enough diverse situations happen in AucAreana to satisfy the authors arguments. In fact, (1) this simulator only has two items ($2000,$10000), (2) except for multi-agent (i.e. multi LLM) settings, the other participants are rule-based agents, (3) as far as checking Figure 5, different LLMs often achieves the similar TrueSkill Scores. If the possible situations are not diverse, we cannot test the LLM's ability of adaptive strategic thoughts. If you write tldr on the insights, that might be "GPT-4 works well and Claude sometimes well"? Even compared to other agentized LLM works, I'm not fully sure the novel insights from the paper.

(Q5, Q4)

Because the revision is allowed in this period, the update of paper would be expected. For Question 5, it is quite confusing to mix the proposed methods (based on BDI) and common belief (participants have their own desire).

Thank you for your careful reading of our paper and your valuable suggestions! We will revise the paper as suggested. Please find our responses to your questions below.

Motivation for Choosing Auctions

The major idea behind our paper is to activate and evaluate the potential and abilities of agents powered by LLMs for tasks that require strategic planning and resource management, which are general but critical in many scenarios like auctions. In the original manuscript, we discuss these general capabilities of LLMs in the context of dynamic environments. We will revise the introduction to more directly link these capabilities to our specific auction setting, and outline the unique challenges and opportunities presented by auction environments for LLMs.

The rationale for focusing on auctions, as opposed to other cooperative settings, is grounded in the unique strategic complexities auctions offer. Moreover, the convenience in quantifying its outcomes, such as total profit or number of items won, can give us clear ranks to the competitors in this arena, showing their differences in terms of these abilities. Since our auction arena is quite extensible, in future work, we will add discussion and negotiation sessions to the auction, with a specific focus to analyze deceptive and cooperative abilities of LLM agents.

Auction Design Complexity

Thank you for pointing this out. It is easy to change the setting of our auction arena by setting different parameters. However, we deliberately reduce the complexity of the auction design to facilitate analysis (which is difficult in researches on simulations), otherwise too many moving parts will add unnecessary confounders to the results. One benefit from the auction arena is that we can control the setting of the arena according to the ability we want to test LLM agents. For example, as shown in the paper, we can change the objective that drives an agent, or change the number and price of items to create more demanding and long-term auctions.

Prompting Methodology Ablations

Thank you for the suggestions. Our prompt basically shares the idea of ReAct [1] and Reflexion [2], which asks the model to reason before taking any actions, and then asks the model to reflect on the results and update for upcoming events. We have done ablation studies on the modular level, such as taking out planning/replanning/learning modules from the system to see their effects. We didn't do specific wording-level ablation study because 1) every model in our experiments shares the same, working prompt, so a fair comparison can be maintained, and 2) it's a little expensive to spend our limited API credits on such experiments while we can focus on so many intriguing possibilities from auction arena.

Justification for Using LLM Agents

Compared with language agents, our choice of LLM-powered agents is underpinned by their demonstrated flexibility and (zero-shot) generalizability in complex, language-based tasks. LLMs, with their ability to process, reason, and generate human-like language, are particularly suited for the nuanced decision-making required in many real-life scenarios. Consequently, agents powered by LLM demonstrate superior capabilities in addressing real-world tasks compared to previous specialized RL agents. These RL agents typically necessitate extensive training within specific environments and often struggle to adapt to communicative contexts during multi-agent interactions. Importantly, the challenge lies in feasibly training these RL agents in such environments, particularly given the absence of coherent training data. On the other hand, our LLMs are zero-shot agents, thus eliminating this issue. Therefore, it is imperative to understand and analyze their abilities before we actually put these LLM agents under complicated, high-stake situations.

Evaluation Metrics

Thank you for the suggestion! Quantifiable metrics is an advantage of our auction arena, compared with some other agent simulations. We have the following quantifiable metrics, which are not only about outcomes:

1. final performance, e.g., total profit for profit-driven bidders, number of items for item-driven bidders,
2. measuring bidding rationality, i.e., Corrected Failure Rate, recording belief errors,
3. behavioral faithfulness, i.e., correlation between executions and plans,
4. performance ranking between multiple agents, i.e., TrueSkill scores.

Although we agree that further digging into strategies such as intermediate reasoning process could be interesting, it is difficult to objectively do so other than manual check or case study, and using LLMs as evaluators creates unintentional biases, such as GPT-4 favors texts generated by GPT-4.

[1] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." In ICLR 2023.

[2] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." In NeurIPS 2023.

Adding an Open Sourced Model

That's a great suggestion! We add an open source model, OpenChat-3.5 7B, which proves to be better even than ChatGPT (GPT-3.5) on some benchmarks. We add it into the intra-specific multi-agent competition, with PaLM-2, GPT-3.5, GPT-4, Claude-2.0, Claude-Instant-1.2, and Rule bidder as fellow bidders. The setting is the same as that of Sec 4.3, except we have 7 bidders instead of 6. We repeat the experiments by 20 times, and the average results are as follows. The results shows that OpenChat-3.5 is at least good at numerical skills and processing facts, as shown by its less belief errors (according to CFR scores) than GPT-3.5 and Palm-2.

As for the TrueSkill score, there is still plenty room for improvement for OpenChat-3.5 in such a dynamic competition.

Thanks for your comments. Please find our responses to your questions below.

Q1: The technical contribution is limited. The fundamental mechanism for the LLM agent is quite simple.

Our prompting strategy seems simple, but our zero-shot CoT-style prompting is consistent with the current SOTA for working with these models, and novel in that we are applying it to dynamic scenarios.

Furthermore, the design of Agents in the auction context is NOT simple. This is due to the many changes in the situation as the auction progresses. Key processes like Belief Update, Replanning, and Learning from Past Experiences are highlighted as areas where LLM Agents are expected to improve toward greater intelligence.

Q2: The results are not so promising. There is no sufficient comparison among different LLMs.

For clarification, we note that our experiments involve an exhaustive set of the current best LLMs, namely PaLM-2, Claude-1.2, Claude-2, ChatGPT and GPT-4, all studied individually (Sec 4.1 & 4.2), and in combination and competition with one another (Sec. 4.3) and against human agents (Fig 6). In our view, this makes our studies fairly comprehensive, with 7 experiments.

Regarding the results, we do not take the view that The results are not so promising, in the absence of more details here we are unsure what you mean. On the contrary, while our results are nuanced, our best models such as GPT4 do indeed exhibit considerable skills in the auction arena, such as the ability to maintain coherent belief states (Fig 2), generate rational long-term plans (Fig 4) that are highly correlated with their behavior (Tab 1) and ultimately behave in a way that achieves their intended goals (Fig 3 & 5). Of course, there is still variability in the skills of different models, and models are indeed not perfect and have much room for improvement; this, in our view, is the main value of our simulation environment and our work.

Q3: The setting of the auction is quite simple. I suggest the authors test different settings (those standard ones), and try to provide more insightul conclusions, compared with human.

Our auction setting closely standard the open ascending price auction model. In the absence of more details here, we are not sure what the reviewer means by standard scenario.

Additionally, we have tried different settings in our experiments:

• In Sec. 4.1, two bidders auction for ten items with different prices.
• In Sec. 4.2, we adjusted the order of the items as a control variable.
• In Sec. 4.3, we had six different auction agents competing with each other for different items.
• In the Interspecific Competition experiment, different agents were set with two different types of objectives.
• In Figure 6, we provided a comparison with human bidding, and we indeed found that humans are not always far ahead in this kind of competitive environment. As a benchmark, our conclusions have already been written in the abstract.