MindAgent: Emergent Gaming Interaction

The CuisineWorld, based on the video game Overcooked which is popular for measuring multi-agent cooperation and has many existing environments already[1][2]. The benchmark introduces more cuisines but falls short in diversifying kitchen maps when compared to previous Overcooked environments.

We are grateful for your insightful comment regarding the design of our CuisineWorld benchmark in relation to existing environments based on the popular video game Overcooked. Our approach, indeed, differs from the environments presented in references [1] and [2]. While these existing environments offer a variety of kitchen maps, our benchmark is specifically tailored to explore high-level multi-agent planning and coordination using Large Language Model-based Multi-agent Systems (MAS).

To achieve this, instead of focusing on diversifying kitchen maps, which often necessitates low-level control of agent motion, we have introduced several unique elements to enhance the complexity and diversity of the tasks:

• Enhanced Kitchen Layouts: We have diversified the layout of kitchens by incorporating a wider range of tools (both in types and quantities) and ingredients. This approach differs from [1] and [2], where the emphasis is not on the variety of kitchen tools and ingredients. For example, in [1], there are only two types of ingredients and two types of tools. In [2], there are two types of ingredients, and two types of tools. In comparision, there are 27 unique ingredients, and 10 tools in CuisineWorld.

• Complex Task Design: Our benchmark includes a broader spectrum of recipes, varying significantly in difficulty levels. This variation is not just in terms of the number of tools and ingredients required but also in the intermediate steps involved in each recipe. We invite you to refer to Figure 23 for a detailed illustration. This aspect of task complexity, particularly in the context of high-level planning, is not extensively explored in [1] and [2]. In [1,2] there are very limited number of dishes, 1 and 3 respectively. However, in CuisineWorld, there are 33 unique dishes.

• Multi-Dish Episodes and Collaborative Strategy Assessment: We require agents to complete multiple dishes within a single episode, with the types of dishes varying to challenge and assess the collaborative strategies of the agents. Our level design ensures that there are shared intermediate steps among the types of dishes in a single episode. The system is tasked with multiple different goals at the same time. This approach allows us to use metrics like 'Collaborative Score' (CoS) to evaluate how agents collaborate to achieve higher dish throughput. This dynamic aspect of collaboration, especially in the context of dish expiration and shared tasks, offers a new dimension to the study of multi-agent cooperation, which is distinct from the environments in [1] and [2]. In [1] the goal is to finish as many dishes as possible in a limited amount of time. In [2], the goal is to finish one dish in the least amount of time. Both of them do not consider the density of the tasks (interval between dish orders coming to the kitchen) and its effect on coordination. As we mentioned earlier, this concpet (changing density of the tasks and measuring collaboration proficiency upon it) is at heart of CuisineWorld and our CoS metric, and it has demonstrated its effectiveness of benchmarking collaboration between LLMs and human-NPCs (as indicated in the abstract).

Table 12 made a good summary of related benchmarks though some related embodied multi-agent cooperation benchmarks with most of the features in the table are missing, such as [3][4].

Thanks for pointing this out. We updated the paper and added the missing references.

From C.3.4, level 3 has only two similar recipes which differ only with the words "salmon" and "tuna", only one demonstration may decrease the task difficulty significantly for the LMs, raising doubts about the benchmark formation.

We appreciate your concern about the perceived task simplicity in Level 3 of CuisineWorld, where the recipes differ only in terms of "salmon" and "tuna." However, we respectfully disagree with the notion that a single demonstration significantly decreases task difficulty for LLMs. Our benchmark's design is not centered around creating complex recipes for LLMs to master, but rather on introducing and evaluating the complexity of collaboration through carefully designed recipe combinations.

In Level 3, the challenge for LLMs is to effectively coordinate agents around shared tasks and bottleneck resources. A strategic collaboration example could involve assigning one agent to prepare cooked rice—a common component in all dishes—while others handle different kitchen stations, such as mixers or chopping boards. This requires the LLMs to identify and manage shared resources and task bottlenecks, and to establish an efficient “pipeline” for task execution, which significantly enhances throughput. Such strategic planning and coordination cannot be trivially inferred from a single demonstration.

Furthermore, it is important to note that the demonstrations provided to the LLMs are based on Level 0 tasks, which are fundamentally different from those in Level 3. Therefore, the LLMs are essentially working on novel tasks that require adaptive and strategic thinking beyond replicating demonstrated actions.

This aspects of our design underscores the benchmark's focus on assessing high-level planning and collaboration, rather than recipe complexity.

In section 5.1, The paper claims "more agents will lead to higher collaboration efficiencies. Thus, indicating that the LLM dispatcher can coordinate additional agents to execute tasks more efficiently". However, from Table 1, 4 agents perform worse than 3 agents in most scenarios, and even 2 agents achieve the highest score in levels 2 and 4, which contradicts the claim. It may provide more insights if the paper can provide more analysis on these contradictory results instead of only ablating on level_3, which seems to be the only level that "looks normal" (a.k.a 2 agents < 3 agents < 4 agents with somewhat clear gaps)

Thank you for highlighting the discrepancies in our results concerning the relationship between the number of agents and collaboration efficiency. We have revised our claim to more accurately reflect that while generally increasing the number of agents can lead to higher efficiency, there are scenarios where adding more agents may actually hinder performance.

Our revised claim now acknowledges that the relationship between the number of agents and task efficiency is not always linear and can be influenced by various factors, including task complexity and the nature of the agents' collaboration. We have added detailed discussions around this revised claim to explore these nuances and provide a more comprehensive understanding of when and why more agents may or may not lead to better performance.

In our initial study, we chose Level 3 for our ablation study because it showed a clear pattern where increasing the number of agents improved performance. We've updated our paper to explain that this choice was based on the observation that the LLM performed well on this level, offering a stable platform to assess the impact of varying agent numbers. When LLMs do not perform well on certain levels, additional unknown factors might confound the results, making it challenging to isolate the effects of ablated factors on performance.

In response to your suggestion, we have conducted a preliminary qualitative analysis and found the following failure mode:

1. Inability to prioritize the task order. Sometimes LLM will skip the task that is on the top of the task queue and leads to task expiration. Therefore, the success rate will drop.

2. Unable to understand occupy() state instruction. In our CuisineWorld, agents need to wait for a varying timesteps before the cooking is done. The wait time depends on the specific tool. When agents attempt to take out ingredients immediately after activating the tool, actions will fail. Instead of keeping waiting, the agents might work on other task components which will slow down the overall progress.

3. Unable to allocate agents to the correct subgoals when there are multiple concurrent dishes. This is especially the case when there are 4 agents.

Thanks for the detailed responses! I highly appreciate it and some of my concerns are addressed. However, my main concern remains: the claims made in this paper are not well supported by the experiments and results provided, and there are not many new efforts in the revision to improve this. Therefore I'm keeping my score for this version of the paper.

Thanks for the thoughtful comments. Furthermore, we also conducted additional reinforcement learning experiments. Results and comparison with our MindAgent results are be added in the Appendix Table 12 for supporting our claims.

We really appreciate your very helpful suggestion and all your constructive feedback. We addressed each of your comments in our revised manuscript for your reference.

"Thanks for the detailed responses! I highly appreciate it and some of my concerns are addressed. However, my main concern remains: the claims made in this paper are not well supported by the experiments and results provided, there are not many new efforts in the revision to improve this."

We thank you for the constructive comments and helpful suggestions again. We address your main concerns below and describe every change that was made to the manuscript in our revision.

The CuisineWorld, based on the video game Overcooked which is popular for measuring multi-agent cooperation and has many existing environments already[1][2]. The benchmark introduces more cuisines but falls short in diversifying kitchen maps when compared to previous Overcooked environments.

We addressed this comment in red-ink of Appendix Sec. E.1 in the new revision paper.

In section D.2, the statement "In this game, the actions of human players can be obtained from an inverse dynamic model by checking pre-conditions and post-effects" is confusing to me. How was Figure 22 obtained? It seems to be in real-time with human players using the keyboard to move, how's the time step defined here and the "goto" action for the LLM agent implemented?

We addressed your comments in red-ink of Appendix Sec. E.2. (we add one more section in the appendix of the revision, so the Sec. E.2 maps to the previous Sec. D.2 as per your comments.)

The benchmark predominantly employs a "new auto-metric collaboration score CoS for assessing the collaboration efficiency". However, this metric is defined as the average task success rate with different time intervals for each dish that appears highly tailored to the specific environment and lacks a clear connection to "cooperation efficiency".

We addressed these comments in red-ink of Appendix Sec. E.6 in our revision.

In section 5.1, The paper claims "more agents will lead to higher collaboration efficiencies. Thus, indicating that the LLM dispatcher can coordinate additional agents to execute tasks more efficiently". However, from Table 1, 4 agents perform worse than 3 agents in most scenarios, and even 2 agents achieve the highest score in levels 2 and 4, which contradicts the claim. It may provide more insights if the paper can provide more analysis on these contradictory results instead of only ablating on level_3, which seems to be the only level that "looks normal" (a.k.a 2 agents < 3 agents < 4 agents with somewhat clear gaps)

We addressed your comments in red-ink of Section 5.1 findings.

Table 4, "For a fair comparison, all tests employed identical prompt inputs" using "identical" prompt for different LM families may not be "fair", especially if the prompt is "tuned" specifically for one model. More details on the prompt engineering process may help clarify these concerns.

We describe the detailed prompt engineering process in red-ink Section B of our revision.

Table 3 presents a perplexing scenario where four agents using a two-agent demo outperform four agents using a four-agent demo, albeit marginally. This result could benefit from a clearer explanation.

We addressed your comments in red-ink of Section 6.2 of our revision.

Novel game adaptation of Minecraft seems very promising, but the details are extremely limited. It would be more convincing if there were more details and formal experiments on it. For example, how is the "adaptation" conducted? What's the additional human effort required (such as prompt engineering and rounds of re-playing for prompt tuning)?

We addressed your comments in red-ink of Appendix Sec. F.1 for our revision.

The method employed in the paper primarily relies on intensive prompt engineering, without introducing novel designs for multi-agent cooperation. .... ....... Providing only screenshots of part of the prompts as in Figures 6 and 7 is not enough. More details on the full prompt and game episode may help clarify these concerns.

We addressed this issue in Appendix Section H and Section B. We add parts of the log as "The full interaction history for finishing one dish in level 9", which is shown in Sec. H. and we provide the full game log here for your reference. Due to the page limitaion, we can't attach the full game log in our revised paper. Hope for your understanding.

Table 12 made a good summary of related benchmarks though some related embodied multi-agent cooperation benchmarks with most of the features in the table are missing, such as [3][4].

We compare our work vs [3][4] in the Table 13 of Appendix Section B1 with red-ink, and we cited the papers and discussed the differences with our work in line 3, line4, line 5, line 11, and line 12 in the section 2 as well as our revised manuscript .

Technical novelty: The technical novelty of this work is lacking because the multi-agent setting is really just adding an LLM call to allocate what different agents should do. Concretely, it is a simple prompting and what is supposed to be an optimization problem is all packed into one LLM call. This doesn't seem to be a very principled way of studying the agent allocation.

Thank you for the comments. First of all, we should point out that the motivation of this paper is not about proposing a set of agent allocation tasks to compare planning proficiency between LLMs and other canonical planning methods (ex. what [1] did), but the following (in our abstract):

However, despite the introduction of numerous gaming frameworks, the community lacks adequate benchmarks that support the implementation of a general multi-agent infrastructure encompassing **collaboration between LLMs and human-NPCs**. We propose a novel infrastructure-MindAgent—for evaluating planning and coordination capabilities in the context of gaming interaction. 

That is, we simply would like to explore whether LLMs, which enjoys a powerful and native natural language interface that can interact with humans, can support sohphsticated multi-agent collaboration and human-NPC collaboration tasks, and this results in our benchmark. We've made this very clear -- the subject of study is collaboration between LLMs and human-NPCs.

That being said, we do agree that showing some results of the canonical planning approach, including RL, could offer a better understanding of the dynamics of the collaboration tasks. Below, we provide some results of using a canonical RL algorithm on the proposed benchmarks for LLMs:

We implemented a masked version of Proximal Policy Optimization (PPO) approach. In additon, we provide heavily engineered dense reward. Despite leveraging privileged information (masked action, what action is valid during the execution), the RL strategies show limited success, even in two-agent settings. For level 1, PPO achieve 45.14% success rate and for level 4, PPO achieves 59.8% for \tau=2.5 after training for 30M steps or 500, 000 episodes. In comparison, GPT-4 agent achieves 95% success rate on both settings.

Further the trained RL policy cannot adapt to varying numbers of agents and to new tasks, or even a change in the number of ingredients. This limitation underscores the complexity and dynamic nature of our environment.

[1] https://arxiv.org/abs/2302.06706

Problem setting: the multi-agent collaboration problem is reduced to a top-down allocation: a distributor distributes tasks for agents to do. But for a collaboration problem to work, there should also be communication between the agents, which this work does not study.

We thank the reviewer for pointing out that our method is a top-down allocation. The multi-agent task planning problem [1], is a very difficult problem when they are multiple dependencies. We want to address that is different from previous settings as in [1], where the task is known priori. In our settings, the tasks are not known prior to the agents simulating the experience of a real kitchen, dish orders keep flooding in. This modification will only push the original problem harder.

We acknowledge that communication between agents is indeed a critical aspect of collaboration, and our current model does not explicitly study inter-agent communication. This is a limitation of our study, and we believe it represents an exciting avenue for future research. The integration of direct agent-to-agent communication would undoubtedly add a rich layer of complexity and realism to the collaboration model, offering insights into emergent behaviors, negotiation strategies, and decentralized decision-making processes.

However, this addition will also add communication cost which might not be ideal for gaming applications where a faster response is desired. Therefore, in this specific work (MindAgent) that focuses on collaboration between LLMs and human-NPCs in gaming scenarios, we choose to anchor our study on a centralized planning scheme.

[1] Korsah et al. A comprehensive taxonomy for multi-robot task allocation.

Have the authors considered incorporating a human-agent communication module in MindAgent? As suggested by Gao et al. [1], introducing interpretable human-agent communication into collaborative games can effectively improve human-agent collaboration performance and human subjective preferences. Natural language is the best medium for human-agent communication, which is also a natural advantage of LLM-based agents in human-agent collaboration.

Thank you for highlighting the potential of incorporating human-agent communication in MindAgent, as suggested by Gao et al. [1]. We agree that introducing interpretable human-agent communication in collaborative games is a promising direction for enhancing human-agent collaboration. Accordingly, we have updated our paper to include citations and discussions related to [1], acknowledging its relevance to our work.

We adopted a straightforward method for facilitating communication between the human user and the LLM-based agent. Participants in the experiment were instructed to verbally express their intents or comments. These verbal inputs were then transcribed and integrated into the LLM’s context as “current human instruction: <instruction>”.

Implementation and Results: This simple yet effective approach allowed for real-time adjustments in the behavior of the LLM-based agents based on human inputs. For instance, in our demo videos can be found in the supplementary videos, when a user suggested a specific strategy or highlighted a priority task, the agent was able to promptly adapt its actions accordingly. This led to a more dynamic and responsive collaboration between humans and agents, showcasing the potential of natural language communication in enhancing the collaborative experience.

In summary, while our initial trials in human-agent communication are relatively basic, they clearly demonstrate the effectiveness of integrating natural language communication in LLM-based collaborative environments. We plan to further develop and refine this aspect of our system in future work, drawing inspiration from the findings of Gao et al. [1] and other seminal works in the field.

Can the design of MindAgent support collaborative games that require more domain-specific knowledge? For example, Multiplayer Online Battle Arena (MOBA) games [1], First-person Shooter (FPS) games [2], and Diplomacy [3] have very complex gameplay and their outcomes heavily depend on the planning and collaboration capabilities of the agents. The authors could discuss how the infrastructure needs to be modified when extending it to other games, which would enhance the generalisability of the infrastructure.

We thank the reviewer for suggesting the exploration of domain-specific knowledge in collaborative games like MOBA, FPS, and Diplomacy, and its implications for the design of MindAgent. Our current work in CuisineWorld has indeed laid the groundwork for such adaptations, primarily through the use of In-Context Learning (ICL).

In CuisineWorld and Minecraft, we inject domain-specific knowledge (such as recipes) directly into the context, which the Large Language Model System utilizes to inform its decision-making and collaboration strategies. This demonstrates the feasibility of adapting MindAgent to other domains where specific knowledge plays a crucial role. In addition, techniques like Retrieval-Augmented Generation (RAG) and Fine Tuning could be pivotal in further developing MindAgent's capabilities to handle such domain-specific complexities.

Thanks for the suggestions, and we have updated the paper accordingly.

The paper claims that the LLM can seamlessly adapt to new planning problems across different domains, but this assertion is questionable considering the dependence on context length. While the simplicity of the current setting allows all environment information to be described within 1K tokens, this approach may prove inadequate for more challenging environments. Additionally, engaging in manual prompt engineering to enhance performance may entail significant monetary costs.

We thank the reviewer for pointing out that context length might be a issue for general planning. Indeed, there could be more sophisticated tasks with more complicated and long interaction. But:

1. First of all, MindAgent aims at benchmarking the multi-agent gaming interaction planning of LLMs and between LLMs and humans. Per our results suggested, even for this relatively simplified scenarios with shorter interaction history and limited context-length requirement, the collaboration score can still low, meaning that the context-length is not the major concern given the current collaboration proficiency of LLMs.

2. Moreover, even from the perspective of how can LLMs handle the context length, the context length is becoming less of a issue as the current version of GPT-4 can handle 128k context tokens in a single prompt,the approximately equivalent of 300 pages of text as suggested by OpenAI[6], which could allow sufficient interaction history. Moreover, we don’t need to look at the full interaction history as is the case in our work. History trimming is a common practice as in [4,5] and can further mitigate the context length limit. In addition, for tasks with interaction that could be extremely complicated, we can optional use memory-based approach as in [1,2,3].

[1] Retrieval augmented generation for knowledge-intensive nlp tasks

[2] Generation-augmented retrieval for open-domain question answering

[3] JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models

[4] Compressing Context to Enhance Inference Efficiency of Large Language Models

[5] LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

[6] https://openai.com/blog/new-models-and-developer-products-announced-at-devday

The description of the environment setting requires further clarification. Does \tau_int,(1) mean that a new task will be added at every timestep? What is the maximum horizon of an episode?

We apologize for the confusion. \tau_int,(1) means a new task will be added after a fixed time step. The specific time value of \tau_int(1) depends on the environment complexity. We compute this value based on the minimum number of steps to complete the task in a single agent setting.

Here are more details on how the task complexity and subsequently, \tau_int is determined, these can also be found in Appendix Section E.4

In our approach, we initially construct a task graph delineating subgoals, which serves as the foundation for our computations. We then apply breadth-first search in a single-agent context to determine the optimal task sequence. This sequence is a key component for calculating task intervals in multi-agent collaboration scenarios. For each tool requiring activation, we incorporate its activation wait time into the respective task intervals. Additionally, for each new connection in the task graph, we increase the total task interval time by tripling the edge time. We assume each subgoal requires at least, goto, get and put 3 actions. The cumulative task interval is subsequently adjusted by a scaling factor of 0.3 and the variable $\tau$. We pick the value $tau$ ranging from 1.0, 1.5, 2.0, 2.5 and 3.0 to represent different task difficulties. This process enables us to effectively compute task intervals tailored for multi-agent collaborative environments of varied difficulties.

We have updated the manuscript to make this more clear.

The paper only highlights the emergent behavior of GPT-4 while neglecting to discuss its potential drawbacks. A more balanced perspective could be achieved by including examples of GPT-4's failure cases.

We thank the reviewer for pointing this out. We have updated the paper accordingly. Here we list the common failure mode:

1. GPT-4 heavily rely on feedback, without feedback its performance drops significantly as indicated in our Table 2.

2. GPT-4 is sensitive to prompt inputs as indicated in Table3, without inference knowledge, the performance will drop. We have updated the paper with these new discussions.

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

The proposed LLM coordination system, MineAgent, lacks novelty as its approach of leveraging scratchpad or memory has been extensively explored in prior research.

First, we want to emphasize that compare to the generative agents, we are using a centralized coordination planning. In our work, we have a complex combinatorial action space, each agent has a action space of 221 due to the presence of multiple ingredients, multiple tools available in the environment., and with four agents, the action space becomes 2385443281 due to exponential effect of adding more agents. In addition, our work explored sophisticated spatial-temporal reasoning. To efficiently collaborate, agents need to work on multiple different tasks at the same time, manage to avoid space conflict (two or more agents cannot work on the same tools), etc.

In addition, compared with concurrent work focus on planning (ex. [1]), we have more agents, more environments, and our environments are extensible and adaptable.

Thirdly, we created a new benchmark with extensive experiment. We even perform cross-domain experiments to adapt our settings to Minecraft.

[1] JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models

The proposed environment features limited state and action spaces and furnishes all the requisite recipes to solve tasks, possibly oversimplifying the challenge. In this situation, heuristic or RL planners could be readily employed. However, the paper does not provide comparisons with these approaches.

Thank you for the comments. First of all, we should point out that the motivation of this paper is not about proposing a set of planning tasks to compare planning proficiency between LLMs and other canonical planning methods (ex. what [1] did), but the following (in our abstract):

However, despite the introduction of numerous gaming frameworks, the community lacks adequate benchmarks that support the implementation of a general multi-agent infrastructure encompassing collaboration between LLMs and human-NPCs. We propose a novel infrastructure-MindAgent—for evaluating planning and coordination capabilities in the context of gaming interaction. 

That is, we simply would like to explore whether LLMs, which enjoys a powerful and native natural language interface that can interact with humans, can support sohphsticated multi-agent collaboration and human-NPC collaboration tasks, and this results in our benchmark. We've made this very clear -- the subject of study is collaboration between LLMs and human-NPCs.

That being said, we do agree that showing some results of the canonical planning approach, including RL, could offer a better understanding of the dynamics of the collaboration tasks. Below, we try to address your concern on action space, alternative methods, etc:

We respectfully disagree the reviewer's observation regarding the "limited state and action spaces" of our proposed environment. However, we argue that the complexity of our environment is not trivial due to the presence of multiple and duplicated tools and ingredients. In a single-agent scenario, the action space expands to 221, and with four distinct agents, it grows exponentially to approximately 2.39 billion. This immense action space presents a unique challenge specific to multi-agent research.

Addressing the enormous action space, we implemented a masked version of Proximal Policy Optimization (PPO) approach. In additon, we provide heavily engineered dense reward. Despite leveraging privileged information (masked action, what action is valid during the execution), the RL strategies show limited success, even in two-agent settings. For level 1, PPO achieve 45.14% success rate and for level 4, PPO achieves 59.8% for \tau=2.5 after training for 30M steps or 500, 000 episodes. In comparison, GPT-4 agent achieves 95% success rate on both settings.

Further the trained RL policy cannot adapt to varying numbers of agents and to new tasks, or even a change in the number of ingredients. This limitation underscores the complexity and dynamic nature of our environment.

[1] https://arxiv.org/abs/2302.06706

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

The font size of Figure 4 is too small.

Thanks for the suggestion, we will update the paper accordingly.

The paper seems not provide a clear definition of the terms .

Thanks for pointing this out. We apologize for the confusions. Here q refers to the utility/rewards the system will generate if the agent perform the specified action. C refers to the cost of the action. We have updated the paper to make this point more clear.