Efficient Human-AI Coordination via Preparatory Language-based Convention

Concern 4 RL Action Space: Unfair comparison with RL agent

A: Sorry for our unclear statements. To make sure that the comparison with RL agents is fair, the data utilized in our experiments to learn low-level skills are not complete task trajectories; rather, it only consists of trajectory segments that accomplish specific skills, such as delivering onions to the pot in the top left corner. Thus, these demonstration data is purely functional and does not reflect any human behavior preference. Besides, these trajectory segments are actually collected by our authors, and not provided to the participants.

References

[1] Hu, Hengyuan, and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordination[J]. arXiv preprint arXiv:2304.07297, 2023.

[2] Wang, Xihuai, et al. Quantifying Zero-shot Coordination Capability with Behavior Preferring Partners[J]. arXiv preprint arXiv:2310.05208, 2023.

[3] Zhou, Siyuan, et al. Adaptive Online Replanning with Diffusion Models[C]. NeurIPS, 2023.

[4] Guo, Yanjiang, et al. DoReMi: Grounding language model by detecting and recovering from plan-execution misalignment[J]. arXiv preprint arXiv:2307.00329, 2023.

[5] Ahn, Michael, et al. Do as i can, not as i say: Grounding language in robotic affordances[J]. arXiv preprint arXiv:2204.01691, 2022.

We sincerely thanks for you thoughtful comments. Below we have provided some clarifications to your main concerns. Further questions are always welcome!

Concern 1 Clarity Issues: Unclear usage of items and concepts, especially regarding user preference and sequencing strategy.

C1-1 Unclear about what exactly is human-preference

A: We sincerely apologize for any confusion caused by our unclear statements. Actually, the human-preference here means that the humans can have certain preference priorities regarding how to collaboratively accomplish the task and what specific roles they undertake. The same concept has also been adopted in existing works, such as [1, 2].

More specifically, on the Overcooked, human-preference can be that the human prefers to utilize the left pot instead of the right pot, or the human prefers to make onion soup instead of tomato soup. We have added explanations about this concept in our revised version.

C1-2 The paper talks about problem decomposition … is applicable to all possible extended-form Markov games?

A: We apologize for the unclear statement. In original statement, we intended to explain that our approach to task decomposition is relatively abstract and not strongly coupled to the Overcooked task, instead of claiming that it is applicable to all possible extended-form Markov games. We have modified our statement and incorporated it in Appendix F.

C1-3 What happens when there is concurrency required between different tasks? Can each task be refined in isolation?

A: Although the Overcooked environment does not involve concurrency between different tasks, for new environment with this requirement, it is entirely feasible for HAPLAN to support this functionality. Actually, we only need to design the prompts to make the derived convention express between which tasks there exist concurrency.

C1-4 How does stochasticity affect this process?

A: The planning of LLMs disregards stochasticity as the resulting high-level conventions are insensitive to it. Instead, we expect the low-level skill policies can be robust to the stochasticity.

On Overcooked environment, there does not exist stochasticity in the transition and reward functions. However, there can exist stochasticity in the agent’s observation because of the uncertainty in the partner’s state. We address this issue by collecting the demonstration segments with random-policy partner. How to train low-level skill policies that are robust to the stochasticity of the environment itself is an important future research direction.

C1-5 How do you account for task failure?

A: When encountered with task failure, we would inspect the high-level convention and low-level skill policies respectively. Firstly, we can check whether there exist issues in the high-level convention. If so, we will regenerate the convention through providing human feedback. Similar adaptive replanning methods can be referenced in [3, 4]. If not, it indicates that there is something wrong with the low-level skill policies, and we will re-train the low-level skills to address this issue.

Concern 2 User Studies: Concerns about the user study

A: We sincerely appreciate this detailed feedback. We have provided clarification in our Common Response to Q1.

Concern 3 Assumptions: Some assumptions should be stated clearly

A: Thanks to your comment.

• (Two assumptions are central to the success of the task) Thanks for this kind reminder. We acknowledge that these two assumptions are significant for HAPLAN and we have added this discussion in our revision. As mentioned by the reviewer, systems like SayCan [5] also requires the first assumption, which is kindly feasible in many scenarios. For the second assumption, despite the skill actions in SayCan are not conveyed to humans, they are also represented in the form of natural language, which implicitly indicates that humans are likely to be able to understand these skills.
• (Access to a high-level abstraction of human actions) The high-level abstraction of human actions is related to the task itself. Since we focus on human-AI coordination tasks, we assume that we can have access to this portion of information and provide it in the examples preceding the prompt. We have stressed this assumption in our revision.
• (Refinement of high-level conventions will be free of conflicts) We appreciate your thoughtful comment. However, we actually do not require this assumption. It is possible that conflicts arise during the refinement of high-level conventions. When such situation occurs, we consider these conventions as undesirable and expect to regenerate them through human feedback. For complex scenarios beyond immediate human recognition, humans can also avoid similar situations after an episode of failure.

Dear Reviewer fw9C:

We thank you again for your comments and hope our responses could address your questions. As the response system will end soon, please let us know if we missed anything. More questions on our paper are always welcomed. If there are no more questions, we will appreciate it if you can kindly raise the score.

Sincerely yours,

Authors of Paper3413

We are really delighted to see your new feedback! Firstly, we would like to clarify that the purpose of our work is NOT to provide a general task decomposition approach for various problems. Our main contribution is to propose a new paradigm for human-AI coordination through the integration of LLMs, as also pointed out by Reviewer nsKW. (Sorry for not highlighting it to you earlier)

While the current decomposition method applies to all layouts on Overcooked, it is true that we would need to modify it when facing some new cases, like cases with concurrency requirement or stochasticity. However, we do not see this as a fundamental weakness but as an opportunity to enhance task coordination performance by incorporating our task priors. This capability is a direct result of introducing LLMs, which is clearly not allowed in the traditional Zero-Shot Coordination methods [1, 2, 3].

Once again, we extend our gratitude for your feedback.

References

[1] Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. ICML, 2015.

[2] Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-based training for zero-shot human-ai coordination. AAAI, 2023.

[3] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. ICLR, 2023.

Weaknesses

W1 Unclear writing of Section 5.3 and 5.4

A: We apologize for the unclear writing of two subsections of 5.3 and 5.4. Actually, the experiments in these two subsections are both aimed to validate the effectiveness of utilizing multiple LLM sessions. Detailed clarification is provided in the response to Q3. We have improved the writing in our revision.

W2 Concern about the requirement for human to decompose problems and to evaluate & execute the plans generated by LLM

A: We acknowledge that it is a practical problem. Currently, we follow the common mechanisms using LLMs like in [1, 2], which have a reliance on human manual work. In the future, we will explore utilizing AI to decompose problems automatically and help evaluate the generated plans. Recently, there have been some efforts [3, 4] to explore related problems.

W3 The usage of multiple sessions are expected

A: Firstly, we sincerely appreciate the reviewer’s acknowledgement of the feasibility of utilizing multiple LLM sessions. Although this finding is somewhat expected, we have not come across any statements regarding this conclusion in the relevant literature. To our knowledge, we are the first to formally propose this finding. We hope it can inspire more research in the future.

W4 Vague details regarding the additional benchmark results

A: Apologize for the vague descriptions in this part. Actually, the additional benchmark results have nothing to do with the coordination aspect but to validate that utilizing multiple sessions can also help enhance the planning capability of LLMs in the reasoning tasks. We have made it clearer in our revision.

W5 Human evaluation may be unreliable

A: We acknowledge that this may be a limitation of our approach at this stage. As stated in our response to W2, in the future we will explore to utilize AI to help evaluate the plans in complex domains, as [3, 4] did.

References

[1] Zhou D, Schärli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models[J]. arXiv preprint arXiv:2205.10625, 2022.

[2] Yang, Xinyi, et al. Human-in-the-loop Machine Translation with Large Language Model[J]. arXiv preprint arXiv:2310.08908, 2023.

[3] Bai, Yuntao, et al. Constitutional ai: Harmlessness from ai feedback[J]. arXiv preprint arXiv:2212.08073, 2022.

[4] Lee, Harrison, et al. RLAIF: Scaling reinforcement learning from human feedback with ai feedback[J]. arXiv preprint arXiv:2309.00267*,* 2023.

Questions

Q1 Any cases that the LLM provided non-executable high-level actions?

A: Yes, there exist this kind of cases. Such as in the case shown in Figure 5(b), the initial plan generated by LLMs instructs the human player to fetch the onions from the bottom left corner, but this is not achievable due to the obstacle in the middle. When encountered this situation, the human can provide further feedback to help LLMs modify the plans.

Q2 Figure 2 is hard to comprehend. What are the boxes in blue supposed to convey?

A: We are sorry for any confusion in Figure 2. The boxes in blue means the questions that the LLM is expected to answer. We appreciate your feedback and we have provided a clearer depiction in the revised version.

Q3 Do the additional benchmark results have human-AI coordination as well? How are the respective tasks being divided between humans and the AI?

A: Actually, the additional benchmark results are unrelated to the human-AI coordination, thus without need to divide the tasks between humans and AI. We apologize for this unclear statement again, and we have made modifications in our revised manuscript.

Dear Reviewer i2dJ:

We thank you again for your comments and hope our responses could address your questions. As the response system will end soon, please let us know if we missed anything. More questions on our paper are always welcomed.

Sincerely yours,

Authors of Paper3413

Questions

Q1, 2 Am I correct in my understanding of HAPLAN? Am I correct in my understanding of the development/design and engineering work needed to adapt HAPLAN to a new domain?

A: Please refer to our reponse to W1.

Q3 What was the methodology of the human subject study?

A: Please refer to our common response to human subject study.

Q4 Is the improvement of HAPLAN significant?

A: To check whether the improvement of HAPLAN is significant, we compare the percentage of performance improvement of HAPLAN over the second-best baseline with both human proxy models and real humans. The results are presented in the following two tables.

From the above tables, we see that: When tested with human proxy models, HAPLAN achieved a performance improvement of more than 10% compared to the second-best baseline in half of the scenarios, and a significant performance improvement of more than 40% in Distant Tomato; when tested with real humans, HAPLAN surpassed all other baselines in all of the layouts.

W5 To what extent do the conventions help?

A: We appreciate your thoughtful comment. Our response is two-fold:

• First, the baselines in our experiments do not utilize conventions, as they are unable to explicitly incorporate feedback. The performance improvement of them across three rounds is mainly because that humans become more familiar with the task and optimize their own decision-making in the game.
• To further check to what extent the convention is helping, we externally include one baseline denoted as HAPLAN w/o convention, which disables developing conventions via human feedback. The result is shown in the subsequent table, where we can see that HAPLAN w/o convention suffers from a significant performance decline especially at the final round, indicating the effectiveness of convention.

References

[1] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824-24837.

[2] Zhou D, Schärli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models[J]. arXiv preprint arXiv:2205.10625, 2022.

[3] Yao S, Yu D, Zhao J, et al. Tree of thoughts: Deliberate problem solving with large language models[J]. arXiv preprint arXiv:2305.10601, 2023.

[4] OpenAI. Prompt engineering. https://platform.openai.com/docs/guides/prompt-engineering.

[5] Isa Fulford, Andrew Ng. ChatGPT Prompt Engineering for Developers**.** https://learn.deeplearning.ai/chatgpt-prompt-eng/

[6] Lilian Weng. Prompt Engineering. https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/

[7] Guestrin, Carlos, et al. Efficient solution algorithms for factored MDPs[J]. Journal of Artificial Intelligence Research*,* 2003.

[8] Pateria, Shubham, et al. Hierarchical reinforcement learning: A comprehensive survey[J]. ACM Computing Surveys, 2021.

[9] Zhang, Chongjie. Scaling multi-agent learning in complex environments. University of Massachusetts Amherst, 2011.

[10] Yuan, Lei, et al. Multi-Agent Concentrative Coordination with Decentralized Task Representation[C]. IJCAI, 2022.

[11] Samvelyan, Mikayel, et al. The starcraft multi-agent challenge[J]. arXiv preprint arXiv:1902.04043, 2019.

[12] Kurach, Karol, et al. Google research football: A novel reinforcement learning environment[C]. AAAI, 2020.

[13] Ahn, Michael, et al. Do as i can, not as i say: Grounding language in robotic affordances[J]. arXiv preprint arXiv:2204.01691, 2022.

[14] Zhang F, Jia C, Li Y C, et al. Discovering generalizable multi-agent coordination skills from multi-task offline data[C], ICLR, 2022.

Weaknesses

W1 Concerns about the development and engineering work needed to adapt HAPLAN to a new domain

A: Thanks for your insightful comment. We admit that it may be inevitable to design new prompts when facing a brand-new domain. However, we claim that it is not a fatal drawback of HAPLAN:

• Firstly, we would like to re-claim that the purpose of our work is NOT to provide a general prompt suitable for every human-AI coordination task. Instead, just as pointed out by Reviewer nsKW, our main contribution is that we propose a new paradigm for human-AI coordination that leverages the strengths of LLMs.
• Secondly, we agree that to guide LLMs to perform in a manner aligned with human expectations needs carefully designed prompts. However, with the publications of some recent works on prompting LLMs, such as Chain-of-Thought [1], Least-to-Most [2], Tree-of-Thought [3], etc. and tutorials on the Internet, such as [4,5,6], we can anticipate that prompt engineering is becoming progressively easier.
• Moreover, we have provided the complete prompts in Appendix F used in our experiments, which we believe can serve as good references for new domains. Providing clear descriptions of task details and rules, incorporating human preferences, and leveraging techniques like Chain-of-Thought and using multiple sessions proposed in our work can all be helpful.

W2 Whether it is always possible to break a problem into sequences or tasks or into low-level skills

A: We appreciate your thoughtful comment. We assume that the task can be divided into sequences due to the sequential property of Dec-POMDPs. Moreover, we argue that the task can be divided into sub-tasks or low-level skills is not a strong assumption. Many problems could be divided into sub-tasks [7] or hierarchical structure [8], as the complex world is decomposable or nearly decomposable[9, 10]. Below are more evidences:

• On one hand, most existing collaborative problems can be structurally decomposed, such as SMAC [11] which can be divided based on which specific target to attack; Google Research Football [12] which can be divided on multiple subtasks or skills, like interception, passing and shooting; not to mention the Overcooked we use.
• On the other hand, this assumption is commonly used in existing studies. In SayCan [13], as pointed out by Reviewer fw9C, the authors assume that we can pre-train low-level reusable skills, like “pick up the apple”. Least-to-most [2] decomposes a hard problem into several easier subproblems to facilitate the reasoning of LLMs. Moreover, a large body of existing multi-agent hierarchical reinforcement learning methods, such as [14], dealing with coordination tasks also make similar assumptions.

For some non-decomposable setting, we leave it as an significant future work.

W3 Novelty of the approach

A: Our main contribution, as mentioned by Reviewer nsKW, lies in proposing a novel paradigm for human-AI coordination. To our best knowledge, this is the first work that utilizes LLMs to enable natural language interaction as part of the coordination process itself.

Through our proposed decomposing the problem, solving subproblems via multiple LLM sessions, and constructing conventions via human feedback, HAPLAN significantly enhances the efficiency of human-AI coordination.

W4 Clarification on the methodology of the human subject study?

A: We apologize for any confusion caused by our unclear statement. We have provided clarification to human subject study in Common Response Q1. Hope it will help.

Weaknesses

W1 Analysis on performance improvement of HAPLAN

A: Due to the strong hierarchical planning capabilities of LLMs and our proposed establishing conventions via feedback, HAPLAN can coordinate with diverse partners. To further analyze the performance improvement of HAPLAN, we additionally conducted two experiments:

• Performance of jointly learned policy: We train a joint policy on Overcooked via MAPPO [1], the performance of which we denote as “Oracle” since both sides are best responses to each other. The results are listed in the following table. We can see that HAPLAN can achieve 75%~89% of the oracle performance thanks to planning by LLMs and human feedback. Although it is currently slightly less than 90%, HAPLAN still outperforms other baselines significantly. We hypothesize a significant factor affecting the overall performance is that the human participants are new to Overcooked task. We will conduct further research in the future to enhance performance.

  	Counter Circle	Asymmetric Advantages	Soup Coordination	Distant Tomato	Many Orders
  HAPLAN (First Round)	135.00$\pm$8.66	345.00$\pm$16.58	195.00$\pm$8.66	325.00$\pm$29.58	350.00$\pm$53.85
  HAPLAN (Second Round)	160.00$\pm$14.14	360.00$\pm$24.49	205.00$\pm$8.66	355.00$\pm$16.58	380.00$\pm$50.99
  HAPLAN (Real Human)	170.00$\pm$17.32	385.00$\pm$21.79	215.00$\pm$16.58	370.00$\pm$22.36	410.00$\pm$51.96
  Oracle	225.19$\pm$18.31	447.63$\pm$8.11	240.67$\pm$4.13	481.51$\pm$54.08	462.76$\pm$12.71

• ``diversity'' of strategies: We also visualize the human behaviors from human subject study. The result is shown in Figure 10 of our revised manuscript, where we can see that the participants encompass enough diversity. Considering that HAPLAN achieves the best performance in experiments, we conclude that our method can coordinate with diverse teammates.

W2 Constraints under certain problem setting

A: We greatly appreciate your suggestion and have made corresponding clarifications in the newest revision.

Questions

Q1 How was the textual plan generated by HAPLAN ``stepped'' during execution?

A: The textual plan generated by HAPLAN is a sequence of skills for both human and AI. Humans can act seamlessly based on their natural language understanding of the plan. For AI, the execution requires to implement two additional functions, namely, index function and completion function. Here, the index function determines which low-level skill policy to call according to the text, while the completion function determines whether the current skill has been completed.

We implement both the index function and complete function as rules in Overcooked, with the details provided in the supplementary materials. Investigating learning based implementation is an interesting direction, and we plan to leave it for future work.

Q2 Was there any real-time synchronization between the human and the AI?

A: Yes. Overcooked is a real-time synchronous environment, which means that the feedback ($s', r)$ for the next time step is provided only when both human and AI’s actions are determined.

Q3 How long were individual episodes of interaction?

A: The horizon of the individual episodes of interaction is 400 timesteps, and the agents are allowed to complete multiple dishes.

Q9 How would a plan generated by HAPLAN compare to a joint policy trained via a cooperative MARL method?

A: Overall, HAPLAN can attain a performance ranging from 75% to 89% compared to a jointly trained policy using cooperative MARL methods. More details are provided in our response to W1.

Q10 How much prompt engineering was required here? The prompts themselves are quite complex; are any results available with elements of these prompts removed?

A: We thanks for your valuable feedback. However, we believe we have already used the minimal possible version of prompts and it is hard to further reduce them.

Considering the complexity of the problem itself, we decompose the whole problem into several sub-problems and design specific prompts for a separate LLM session to solve each sub-problem. This kind of task decomposition has been demonstrated effective to improve the reasoning quality of the LLMs in Least-to-Most [2]. The prompts for each session contributes to the overall completion of the entire task and are indispensable.

Besides, it is also inappropriate to remove any element in the prompts for one specific session. Taking the prompts for Session 1 in our Appendix G as an example, the first part In a collaborative cooking game, ... the pot is onion introduces the task background knowledge to the LLM, the second part For Example 1: The instructions ... the middle pot provides some demonstrations for the subproblem, while the third part Now, the instructions ... examples proposes the concrete question to the LLM.

Actually, part 1 and part 3 are obviously necessary for the LLM session to solve the subproblem, while part 2 as demonstrations have already been proved effective in OpenAI’s official document https://platform.openai.com/docs/guides/prompt-engineering/strategy-write-clear-instructions. Moreover, we claim that we have already made the prompts design as concise as possible while ensuring the functionality of these prompts, as redundancy is unnecessary.

References

[1] Yu C, Velu A, Vinitsky E, et al. The surprising effectiveness of ppo in cooperative multi-agent games[C]. NeurIPS, 2022.

[2] Zhou D, Schärli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models[J]. arXiv preprint arXiv:2205.10625, 2022.

Q4 For the scripted agents (Table 1), what information about the specific scripted policy was provided via the prompts?

A: The prompts will convey the behavioral preference of each script agent to the LLMs. For example, for Onion Placement scripted agent, we will mention ``I will continuously place onions or tomatoes into the pot'' in the prompt.

Q5 Were humans allowed to construct their own initial prompts? Rather than providing the instruction given to the human, could the human provide their own description of the role they planned to take?

A: Yes, humans are allowed to construct their own initial prompts and provide their own description of the role they planned to take, all of which will be part of the prompts to the LLMs.

Q6 How closely did human’s observed behavior match the joint strategy they finally agreed upon?

A: Since both human and AI strictly follow the convention to take their actions, the human’s observed behavior fully match the convention (or the so-called joint strategy here) they finally agreed upon. The supplementary materials also provide a video demo of human-AI coordination.

Q7 It wasn’t immediately clear, but were humans allowed to provide text-based feedback between episodes, or only in advance of episodes?

A: Text-based feedback between episodes is allowed. Actually, in our real human study, we had each human participant cooperate with HAPLAN for three episodes, requiring human feedback at the beginning of each episode. To help humans focus more on the task, we currently do not allow them to provide feedback in an episode.

Q8 How much variance was there in the types of feedback humans provided?

A: Actually, the human feedbacks can reflect quite different human preferences. Taking the layout of Many Orders as an example, we can list the following different types of human preferences:

Taking the first type of human preference as an example, the human feedback can take various forms. We have compiled the occurrences observed in our experiments, which are as follows:

Dear Reviewer nsKW:

We thank you again for your comments and hope our responses could address your questions. As the response system will end soon, please let us know if we missed anything. More questions on our paper are always welcomed.

Sincerely yours,

Authors of Paper3413