Don’t lie to your friends: Learning what you know from collaborative self-play

Thanks for your careful review and thoughtful questions!

• Prior work on self-play training and confidence calibration

Thanks for raising this point. We plan to expand and restructure the discussion of related work, and will add a subsection on prior work on self-play training. The main contrast with work such as the Hanabi challenge is that our focus is not on learning to play cooperative games per se, but specifically on using collaborative games to be better collaborative helpers. For this reason, we construct a game that connects much more closely with real information-seeking queries, demonstrating that self-play can help train agents in this practically-relevant user-facing setting, not just training agents to play games with other agents.

We will also add a discussion on prior work on calibration, including Tian et al (2023). Most of this work focuses on calibration with respect to parametric knowledge and not retrievals. The ICL method shown in Figure 5 is somewhat similar to the "Ling-1s" method of Tian et al, and we observe relatively poor calibration in the sense of correlation between P(Answer) and F1 of the parametric model. See also our response to reviewer GEzH, in which we compute the correlation between P(Answer) and parametric F1 as .2-.3).

• Inference costs

Inference is not a significant cost because the outputs are short. Specifically, generating the training data required 64K rollouts per round of ReST training (1000 queries per dataset, 32 rollouts per query), with 3-5 inference calls per rollout. This yields an upper bound of 1.28M inference calls, each of which resulted in 5-10 tokens. Examination of our logs show that a single ReST epoch produced less than 2M whitespace-delimited tokens, so the entire CSP training budget is <10M output tokens. At third-party prices for gemma2-9b (https://groq.com/pricing/), this would cost less than $2 US. We will add these details to the next version of the paper.

• Q1 & Q2: orchestration

Section 3 describes the setup at a very abstract level, which could apply to many possible orchestration setups. As described in section 5, the orchestrator always passes the question to the helpers in our experiments. Extension to more flexible orchestration policies is left to future work, but this is orthogonal to this paper's main focus, which is on training the helpers to be well-calibrated and efficient tool users.

• Q3: (Line 146)

The middle circle represents concatenation.

• Q4: It seems like the most important immediate direction the current work could be extended is to add search-then-hedge and guess-then-answer actions - this would also teach the models when to trust their tools or their internal knowledge, right?

Can you say more about this proposal? By decomposing search/guess and hedge/answer into separate actions, the agents can learn to hedge when the search results are disappointing. If the agent knew in advance that search was not going to yield good results, it could hedge without searching and thereby avoid paying for the tool call.

Thanks for your review, which highlights several interesting possibilities for future work.

• "The generalization to other domains or more complex interaction is leaved unproven. Besides, the study only examines small agent numbers..."

We choose a relatively focused application of collaborative self-play to isolate the key scientific questions involved in learning calibration and tool use. Scaling to other domains and interaction patterns may teach other skills, such as planning and delegation. We are excited to pursue these possibilities in future work. Similarly, extension to larger agent societies is an exciting direction, though as we show, even simple three-agent societies are sufficient for interesting and productive capabilities to emerge.

• "Have you tried other reinforcement learning (RL) algorithms? If so, what issues do you think might arise?"

We are interested to explore the applicability of other RL algorithms, but see at least two potential issues. First, these algorithms will be less efficient than ReST, which benefits by interleaving easily-parallelizable inference steps with parameter updates. Second, on-policy algorithms might be more able to hack the reward, in the sense of learning strategies that work well in the training setting but transfer poorly to interactions with human users.

Thanks for the thoughtful review and interesting questions! Responses are inline.

• "narrow performance improvements compared to the baselines" "Q8. What is the advantage of CSP vs action supervision?"

Empirically, we find that action supervision excels at training an agent to select correct actions within a given task, but is less effective at learning a policy that transfers well to new settings, as demonstrated on the heldout datasets EntQ and NQ, where it has relatively weak F1 (table 2) and poor search calibration (figure 4b).

More importantly, action supervision relies on heuristically determining the optimal action at each step of training. This will not be possible in every setting and is strictly less general than collaborative self-play training from the group reward alone. For example, it is unclear how to do action supervision in a multi-hop setting in which distinct information is needed from each agent. Even in our setting, action supervision requires an arbitrary F1 threshold, which is not required by collaborative self play.

Stepping back, CSP can be viewed as a form of outcome supervision while action supervision is process supervision. In general, we can obtain outcome supervision on a broader set of problems and techniques than those on which it is possible to obtain process supervision.

• Q1. aleatoric vs epistemic uncertainty

Thanks for raising this point. We plan to expand and restructure the discussion of related work to cover uncertainty quantification more comprehensively, including aleatoric vs epistemic uncertainty. From the perspective of the helper agents, the retrieval tool can reduce epistemic uncertainty about questions that are not in the model's parametric knowledge; from the asker's perspective, confidence markers can reduce epistemic uncertainty about which helper agent to trust.

• Q2. prior work on game theory for dialog

Thanks again, we acknowledge the related work would benefit from a deeper discussion of this literature, which we plan to add in the next version. We will include the suggested reference as well as references therein, particularly the line of research from Prashant Parikh.

• Q3. Fig 2 needs to be annotated with (a, v, t) labels, to make it more readable

Sorry but we do not understand the suggestion. Can you be more concrete?

• Q4. Confirmation bias

This comment references a paper on semi-supervised learning, in which the teacher generates answers that may be incorrect. In our setting all answers are grounded in gold references. Could you please say more about how this issue may relate to our paper?

• Q5. Hanabi challenge

Thanks for this reference. The Hanabi challenge paper does share our interest in learning to play collaborative games. However, we emphasize that our ultimate goal is not on learning to play cooperative games per se, but specifically on using collaborative games to train LMs to become better collaborative helpers. For this reason we design a new game based around realistic user information-seeking queries. We'll be sure to include this motivation in the next version of the paper.

• Q6. [minor] Table 2 needs to be annotated more with important cells highlighted for easy readability.

Thanks for the suggestion, we will restructure this table to improve readability in the next version.

Thanks for the thoughtful review! We have conducted some additional experiments to answer your questions and concerns.

• "[T]he use of the special tokens #HEDGE and #ANSWER instead of just letting models generate freeform expressions of uncertainty seems unrealistic and limits usefulness"

It is relatively easy to go from this structured representation to freeform text. As a pilot experiment, we prompted gemma2-9b to go from (question, answer, confidence) tuples to freeform generations. To increase syntactic diversity, we then asked gemma2 to paraphrase each generation zero, one, or two times. Finally, to test that the freeform generations accurately show the confidence, we then prompted gemma2-9b to predict the confidence from the free-form text. The round-trip confidence consistency on 100 examples was 91%, with all but one of the errors being a GUESS mislabeled as an ANSWER, due to the paraphrase prompt introducing overconfidence. Here are some examples of the freeform outputs:

['Terrence McNally penned the captivating play, Sweet Eros.' (ANSWER), "While I can't guarantee it, I am very confident that Marvejols is located in France." (GUESS), 'I strongly suspect, though not conclusively, that burning mouth syndrome affects post-menopausal women more frequently.' (GUESS), 'John le Carré is the creative force responsible for the novel The Dancer Upstairs.' (ANSWER), 'All evidence points to David Lynch as the probable producer of BE.' (GUESS)]

This experiment suggests that our approach could be applied to free-text communication between the agents, and could provide the user with free-text representations that combine the confidence and the answer.

• "I was expecting to see some calibration measure, e.g., ECE."

Thanks for this comment, we agree this would be helpful. We have computed the Spearman correlation between P(answer) and the F1 of the prompted model (parametric knowledge), which corresponds to a statistical summary of the relationship shown in Figure 5. Rank correlation is more appropriate than ECE or Brier for this case, because the evaluation metric is F1 and not accuracy.

In-distribution (popqa, bioasq)

Out-of-distribution (nq, entq)

By the Fisher transform all correlations are statistically significant at p<.05 (z=2.1 for the correlation \rho=.069). CSP and action supervision significantly improve on in-context learning, and as discussed in the text, deanonymization dramatically reduces the calibration of CSP.

• Prior work

Thanks for these pointers. We plan to add significantly more discussion of prior work in the paper, including these two references. A key contrast is that our focus is not on learning to play cooperative games per se, but specifically on using collaborative games to be better collaborative helpers. For this reason we design a new game based around realistic user information-seeking queries, rather than automatically generated social tasks (Sotopia) or negotiation games (Tomlin et al).

• "Does the finding that models learn to use the retrieval tool less frequently over the course of training require a multi-agent setup?"

The retrieval rate decreases across rest epochs for both multi-agent and single-agent training (CSP and Action Supervision; see figure 8 in the appendix). This is likely an artifact of the very high search rate of the prompted model.

We hope the reviewer will consider raising their score given the new results provided, explanations and commitment to improve the discussion of related work.