Thank you for taking the time to review our work and the thoughtful questions and suggestions. We hope that our answers below can clarify some of the perceived shortcomings in our presentation. We would like to point out that the presented evaluations comprise thousands of multi-turn negotiations, covering different game settings and models from all leading LM providers. Due to space constraints and to not overwhelm the reader, we did our best to compress our findings as concisely as possible. We will add additional analyses to the Appendix.

1. About the negotiation protocol & the way it uses LLM to compose a strategy. Where do q_n/m and beta_i and context c come from? Are they fixed or sampled from some distribution during each negotiation instance? Why the negotiation strategy has to be constructed in such way, and how does it compare with other approaches? E.g., just directly input previous negotiation rounds results and output a text message.

The context, c, consists of the protocol, game, and issue descriptions. The protocol descriptions, consisting of the message and note query prompts, q_m and q_n, the negotiation rules, and termination conditions are fixed across negotiations. The rules used in our experiments are strongly inspired by those of the famous ‘Rio Copa’ negotiation game [1], a negotiating protocol commonly used to simulate negotiations for didactic purposes at leading business schools.

The queries q_m and q_n are designed to elicit structured outputs and promote measurements of metrics relevant to alignment while remaining as concise as possible. These prompts are the results of many repeated experiments on held-out data across models to minimize promoting unintended behaviors. As noted in Section 5, the need for ‘prompt engineering’ is a limitation inherent to current language models. It is thus likely that a set of prompts exists that is even more effective than the ones used in our work. However, as all models use the same prompts, the only point of concern would be if the prompts would provide an unfair example to one of the models. Between the minimality of the prompts used and the extensive tests conducted, we did not observe any evidence that this is the case. In Appendix E.1 we listed the fixed rules and prompts used for our experiments.

A game and issue descriptions and corresponding preference weights beta_i are fixed as well, based on the evaluator’s objectives. This basic formulation allows for a wide range of possible negotiations. The game and issues used in our experiments are described in Appendix E.2. For the non-integrative setting, issue preference weights beta_i are all equal, i.e. 1 / #-issues. For the integrative setting, the main requirement for the choice of preference weights is that clear collaborative bargaining opportunities become available, i.e., at least one issue is more important than the other issues. In our experiments, we therefore opted for a fixed {1/3, 2/3} v. {2/3, 1/3} split to provide clear trade-offs between issues. We will update the Appendix to include these experimental setup details.

Generally, there are two ways a negotiation can be formatted using LMs:

1. Direct dialogue between two LMs: Each LM-instance takes the ‘Human role’ to query the other LM-instance, responding in the ‘AI role’. However, a third, ‘System’ role is necessary to initialize the negotiation rules and provide per-round instructions.
2. Represent negotiations as a transcript: Each LM-instance receives messages from the other LM-instance as a transcript presented by a ‘Human’ or ‘System’ role. This same Human/System role can be used to initialize and mediate the negotiations, thus only requiring two distinct API roles.

As listed in Appendix B, not all LM-providers currently support three distinct roles (AI, Human, and System). Following preliminary experiments described in Appendix A.2 to measure the difference between the two formats, we opted to use the ‘transcript’ format to include models from all popular LM-providers.

As described in equations 2, 3, and 4, when formulating its next note/message, the LM-agent indeed observes a sequence of the previous negotiation rounds (note the t superscripts). As agents are initialized using the context c, these were omitted in these equations. However, we recognize that this notation is not clear enough and will update the revised text accordingly. Thank you for pointing this out and we apologize for the confusion.

[1] Bontempo, R., & Iyengar, S. (2008). Rio Copa: A negotiation simulation. Columbia Caseworks.

Thank you for your detailed review and for voicing your concerns. We hope that our answers to the perceived weaknesses you highlighted and the questions you posed will allow you to reassess our work. In addition to the replies below, we also included a general summary placing our efforts more squarely in the currently accepted approach to language model evaluation benchmarking.

1. empirical design with insurmountable reproducibility issues and cost issues impacting statistical validity. Unless the experiments were performed all simultaneously, it is not obvious that they are valid since these models undergo continuous improvement, meaning you might've been comparing different models across different experiments, even if API access was the same and there would be no way to know, right? For this same reason, the experiments are not necessarily reproducible. It might be better in the future to use open-source LLMs for which the models can be held frozen by selecting a checkpoint.

a. [Experimental results might not be valid and are not necessarily reproducible since closed-source models are continuously improving]

Language modeling (LM) technology has matured to a point where commercial applications are viable, leading to closed-source implementations to protect large R&D investments. The reviewer’s argument appears to be against evaluating closed-source models in general. We would argue that this strategy is dangerous. Especially now that LMs are rapidly entering the public sphere and are being used by hundreds of millions, more effort than ever should be dedicated to evaluating performance and alignment.

Any evaluation benchmark is limited by how often evaluations are performed. For example, the current industry-leading LM evaluation benchmark, ‘HELM’ [1], has the same limitation. Furthermore, closed-source models often come with external checkpoints. More importantly, they surely come with internal checkpoints. Legislative pressure could force LM providers to share evaluation metrics on several benchmarks before public releases, similar to how other models deployed in critical infrastructure are treated, e.g., risk models for consumer lending at retail banks.

[1] Liang, Percy, et al. "Holistic evaluation of language models." Transactions on Machine Learning Research (2023).

b. Cost issues [impact] statistical validity’

Machine learning models have dramatically increased in size over the past 15 years with the rise of deep learning. This increase has come at a price of reproducibility, e.g., not many labs have access to hundreds of GPUs to host a single model. The recent growth of APIs for deep learning models has increased the possibility for evaluations by lowering the thresholds of accessibility. Yet, we recognize that performing evaluations on a large suite of models comes at a hefty price (The experimental costs for this work were ~USD 10,000-). As noted in the previous point, the cost of running such evaluations between models from large providers could be imposed on the large model providers themselves before public releases. Indeed, one of our core contributions to the community is the dataset of thousands of negotiation transcripts to serve as a starting point for further research.

However, as outlined in Section 5, ‘Limitations and Ethical Considerations’, a viable alternative exists for smaller labs or individual researchers interested in benchmarking their models against third-party options. By establishing a latent-ability framework like the ELO rating system used in chess, a model could be compared to cheaper third-party models, after which results can be extrapolated.

c. It might be better in the future to use open-source LLMs for which the models can be held frozen by selecting a checkpoint

As described in Section 3.2 ‘Qualifiers’ and at the start of Section 4, we indeed attempted to benchmark the LLaMA 2 model family, widely considered the state-of-the-art in open-source LLMs at the time of research. However, we found that these models were unable to pass our Qualifier round, i.e., to successfully and consistently complete a structured negotiation.

2. no definition of agency despite it being a central concept to the paper

In the first paragraph of our introduction, we describe LM-agents as: ‘capable of completing tasks that require interactive reasoning [... and] acting over multi-step horizons.’ We will emphasize and sharpen this definition in the revised text.

3. the metrics in table 1 seem to require a lot of human checking, which makes it difficult to scale this benchmark. Are you also using LLMs to compute these benchmark values(e.g. internal faithfulness)?

The metrics displayed in all tables are fully automated, as described in Appendix A.4. We utilize a mixed strategy of regexes and an LM equipped with a concise set of hand-crafted examples. We further performed a large number of sampled spot-checks to ensure the extracted offers were reasonable and accurate.

Thank you for your positive and constructive review. We are happy to see our effort to de-bias the presented metrics and the extensive evaluations conducted being recognized. We hope the responses below can alleviate the remaining perceived weakness and answer your question.

Many of the current language models are not trained to sustain a negotiation type conversation. For instance, they don't have a framework to keep track of the issues agreement had been reached upon, or the current alternatives that are under discussion. Thus, the proposed metric measures an aspect on which the models had not been trained, and indeed their performance on it is more a side effect of some artifacts in the training data.

Thank you for highlighting this important point. Current language models (LMs) are indeed not explicitly optimized for multi-step negotiations. Yet, LMs are broadly marketed as “AGI”, which encourages users to use them as agents for tasks not seen during training (see for example the recent “personal GPTs” release by OpenAI [1]). We’re testing to what extent such (encouraged) “off-label use” is warranted.

As many of the challenges integral to successful negotiations show up in various real-world tasks of interest, negotiation performance can serve as an insightful proxy for expected behavior. Additionally, metrics like ‘internal/external faithfulness’ and instruction-following provide insight into the consistency of an LM-agent’s state of mind and the ability to stay within user-defined boundaries.

[1] https://openai.com/blog/introducing-gpts

Clearly, the performance of the LLMs in this task can be improved relatively easily, as the underlying mathematical negotiation problem is much simpler than LLM's language abilities. How would one rank a model that would have minimal language abilities, but use a specialized algorithmic plugin?

That is a great question. It is indeed possible that a new generation of specialized negotiation-algorithms will emerge, where LMs will be used as ‘sensing’ and ‘acting’ modules. Such a module would be responsible for decoding offer indications in natural language to a more restricted format for a specialized algorithmic solver, then encode back proposed solutions into natural language to communicate externally. We highlight examples of these types of ‘hybrid-LM-agents’ in our related work. Yet, even for such hybrid-models, the bar for ‘minimal language abilities’ may be quite high to accurately capture all subtleties contained in natural language.

While it is clear a specialized solver would be much more capable of finding ‘optimal’ solutions to the underlying mathematical problem, it is less clear how much this would help in finding the language required to successfully convince the opposing agent toward an acceptable agreement. Additionally, strong language understanding would be required to ‘guess’ the other agent’s payoff matrix, i.e., perform “Theory of Mind” inference.

However, the purpose of our work is not necessarily to provide a benchmark for the best ‘negotiating’ agent, but rather measure the innate, un-optimized ability of LMs to behave as agents. If it becomes the case that LM providers would find negotiating performance important enough to design specialized training/finetuning and inference routines, such advantages should quickly disappear in cross-play performance.