EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents

We really appreciate the reviewer for the effort in review and advice. And we do apologize for not clarifying the papers’ focus enough.

Regarding to Detailed Discription

Although we have mentioned a big topic in this paper, as we described in the introduction (section 1, line 100), our focus is the embodiment-aware task planning using the LLM multi-agent methodology, which is related to the high-level task planning. For detailed high-level decision making process in our framework, we have illustrated in section 3.4, line 302.

We do not prominently emphasize the contribution of the entire system in low-level motion planning, as we have mentioned in section 4.1, line 390, the collision has been disabled with Pybullet physics simulation in this benchmark. Besides, we acknowledge the detailed description is not discussed enough. Thus we add section B.4 Detailed Implementation of Robot Low-level Control in Benchmark for clarity.

Regarding to Communication Graph

We really thank you for your suggestion. We discribe the communication graph and discussion process in section 3.4 but without visualization. And we have added the communication graph in the appendix A.2 MAS Communication Graph for graph visualization and detailed discription to address this concern.

Regarding to Error in Video

We are really sorry for the confusion caused by the video. For the drone at the 16th second in the video, in fact the drone passed through the room door following the routine, which can be seen from this screenshot that drone passes through the door. Please refer to this picture CLICK, in which the door that drone flying through can be clearly seen. We’ve rechecked the demo videos on the website and confirmed that all the robots in the video behave correctly.

We sincerely express our appreciation for reviewer's valuable advice, and we'll update the draft according to those suggestions and questions as soon as possible. We do hope the refinement and explaination can win our reviewer back and look forward to further discussion.

Dear reviewer,

We hope this message finds you well. We are writing to request your response to our rebuttal comment kindly. We would appreciate it if you could review our response and we could have helpful discussions here.

We have revised the draft as you suggested:

• We reorganized the Section 2. Related Works and their relations to our paper as suggested by Reviewer jjgY and Reviewer xZUU.
• We added Appendix A.2 MULTI-AGENT SYSTEM DESIGN AND COMMUNICATION according to the request by Reviewer xZUU to add the agents' communication graph
• We added Appendix B.4 DETAILED IMPLEMENTATION OF ROBOT LOW-LEVEL CONTROL IN BENCHMARK according to the request by Reviewer xZUU.

Please let us know if you have any more questions and we are happy for further discussions.

Kind regards,

The authors

We sincerely appreciate the reviewer's thorough and insightful feedback on our paper. The reviewer has provided valuable comments on the strengths of our approach, particularly highlighting the innovative concept of "robot resume" for dynamic role determination in LLM-based multi-robot systems. We are grateful for their recognition of the compelling nature of our work. Since there are a lot of questions and discussions from the reviewer’s feedback, and the number of characters in a comment box is limited, we will respond in multiple parts:

Concern about degree of freedom for the system

We appreciate the reviewer's insightful question about the additional degrees of freedom introduced in our decision-making and planning process. Specifically, we agree with the reviewer’s point of “ ’who does what’ is partially predetermined, simplifying the planning”. In fact, we follow the conclusion in [1] which explored four different LLM multi-agent architectures for group discussion. Their conclusion is that the centralized decision making process greatly helps all agents to make agreement and the discussion “converges” to a concrete plan, i.e. sequence of sub-tasks and task assignment. For a constrained-free LLM multi-agent discussion, the system has a high probability of divergence, in which no agreement can be made or the discussion falls into a loop or mode collapse. We do acknowledge the limitation of this constrained task planning here, trading a bit of the freedom for the convergence and performance. We will add the section of “Acknowledgment for Limitation” to discuss the potential limitations in the design for the extensive systems.

Suggestion on comparing our approach with other methods with predefined roles

We are really sorry that the writeup has led to some confusion. In the ablation study itself, we have already compared our framework with other similar baseline methods. For example,  w/o. Robot resume refers to Meta GPT[2] and w/o. Discussion refers to CMAS[1]. It is worth mentioning that Meta GPT[2] is a role-playing method that assigns different roles (managers, developers, test engineers) manually to LLM agents as we discussed in the section 4.4 in the submission, lines [475 - 478]. Our method’s success rate, sub-goal success rate, and time efficiency all constantly outperform the role-playing method with human-created profiles. We really thank the reviewers for pointing out this ambiguity, and we will revise the draft as soon as possible.

Question of modularity and experiment on adding battery life

Thank you for your thoughtful question about the modularity and extensibility of our system. Our implementation leverages modularity through a RobotResume Python class, which stores robot capabilities in a JSON file. Different capabilities are inferred via separate modules, such as manipulation, perception, and navigation, each utilizing specific information like URDF files and camera parameters. To add new capabilities, such as battery life, we update the defaults.py file with the relevant descriptions and regenerate the robot resumes in JSON format. The new capability can be integrated into task planning by refining the agents' prompts to focus on the added feature, demonstrating the modularity and extensibility of our system.

At the reviewer's request, and to also better illustrate the extensibility of the robot resume, we conducted experiments by adding battery life designed based on real-world conditions as a new capability. We tested the system on 5 episodes of perception tasks and 5 episodes of manipulation tasks. The results showed a 100% success rate in avoiding the allocation of robots whose task execution time would exceed their battery life, demonstrating the effective integration of the new capability into the task planning and allocation process. The result is expected thanks to the strong prior of common sense reasoning stored in LLMs. The chat history of the experiment is uploaded to our repository

Considering the limited time for the rebuttal and limited comment length, this just tries to address part of the concerns and questions. We will upload the updated draft and the rest parts of the rebuttal comments as soon as possible. We really thank the reviewer's understanding.

[1] Chen, Yongchao, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. "Scalable multi-robot collaboration with large language models: Centralized or decentralized systems?." In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4311-4317. IEEE, 2024.

[2] Sirui Hong, undefined., et al, "MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework," in The Twelfth International Conference on Learning Representations, 2024.

Suggestion on using prior knowledge to improve planning and decision-making

We appreciate the reviewer's insightful suggestion about incorporating prior knowledge to improve planning efficiency. We want to clarify that our EMOS framework already implements this idea through our novel "robot resume" approach (Sec 3.3), but in a more comprehensive and flexible way than using simple context/class vectors. Here's why:

1. Hardware-specific capabilities: Our robot resume captures each robot's physical capabilities by analyzing their URDF files and using forward kinematics to generate numerical specifications. For example, for a quadcopter, its resume would indicate no manipulation capability, automatically excluding it from relevant tasks.
2. Context representation: The robot resume provides both textual summaries for common-sense reasoning and numerical specifications for precise spatial calculations. This dual representation allows agents to quickly filter out infeasible tasks while maintaining the ability to perform detailed geometric verification when needed.
3. Handling complex tasks: Simple context vectors would struggle with complex, compositional tasks. For example, consider a task like "Find the toy in the bedroom on the second floor and place it on the high shelf in the living room." While a context vector might indicate a robot has manipulation capability, our resume-based approach can:
   • Use numerical workspace analysis to verify if the robot arm can reach the high shelf height
   • Check mobility constraints for multi-floor navigation
   • Consider perception capabilities to ensure the robot can visually locate the toy This level of detailed capability matching would be difficult to encode in simple class vectors.

Regarding robot availability, while our current implementation focuses on physical capabilities, the framework can be naturally extended to include dynamic state information (e.g., current task status) in the robot resume.

Finally, we really appreciate the reviewer for the careful review of this paper, and we will correct the grammar errors in the paper and update the revised version of the paper as soon as possible.

Question on how the system handles partial observable environment

We thank the reviewer for this visionary question. Firstly, we admit that we assume the multi-robot system is equipped with a perfect multi-agent SLAM system, providing a full observability of the environment, as we mentioned in section 3.1, lines 203-206. While our current implementation focuses on fully observable scenarios, we also recognize the need to address partial observability for real-world applications. For this purpose, we propose several extensions that could extend our current system to handle partial observability:

• Probabilistic State Estimation: We could incorporate probabilistic methods like Bayesian filtering or particle filters to estimate the full state based on partial observations. This would allow agents to reason about uncertainty in their knowledge of the environment.
• Active Perception: We could incorporate active perception strategies, where agents actively seek information to reduce uncertainty about critical aspects of the environment.
• Belief-Space Planning: Instead of planning in the state space, we could adapt our planning algorithms to operate in the belief space, accounting for uncertainty in the current state and future outcomes of actions.

There are two challenges for this adaptation we think matter in this dynamic decision process. Firstly, agents need to synchronize observation with global representation to update the state, and this could be very costly and slow with LLMs. There could be some classical distributed SLAM algorithm to reduce this overhead. Secondly, reasoning about partial observability typically increases the computational burden and instability, which may require optimizations to maintain real-time performance.

Scalability of the framework

We recognize the importance of scalability of a multi-agent system and we are grateful to the reviewer for this question. Now we are emergently adding a new experiment to study the system's performance with the increase of robot numbers in the environment. We have started these two experiments:

• In the first experiment, we are scaling the number of robots performing the same task. In our experiments, we sample 10 episodes from the manipulation task and evaluate performance across different numbers of Fetch robots (2, 4, 6, 10, and 20) to assess communication efficiency and success rate. However, due to computational limitations and the limited time of the rebuttal session, this experiment is still ongoing.
• In the second experiment, we plan to study the impact of increasing task complexity by scaling up the number of objects to manipulate. While maintaining a fixed number of Fetch robots (2), we evaluate the system's performance with varying numbers of objects (1, 2, 3, 5, and 10).

Some existing results from the first ongoing experiment indicate that, with the key design of leader allocation and group discussion, the system can successfully assign a minimal yet sufficient number of robots to complete the manipulation tasks.

We will report the experiment results with analysis in the revised paper and comment boxes as soon as possible.

Dear reviewer,

We hope this message finds you well. We are writing to request your response to our rebuttal comment kindly. We would appreciate it if you could review our response and we could have helpful discussions here.

We have revised the draft as you suggested:

• We reorganized the Section 2. Related Works and their relations to our paper as suggested by Reviewer jjgY and Reviewer xZUU. Specifically, we divided the "Multi-Agent System" subsection into two sections "LLM-Based Multi-Agent System" and "Heterogeneous Multi-Agent Learning". All the mentioned related works by these two reviewers have been added to the reference list.

We are looking forward to your responses and we also welcome further discussions.

Kind regards,

The authors

We really thank the reviewer for the time and efforts invested in this thoughtful evaluation of our paper. We would also appreciate the recognition of our work's strengths, particularly the acknowledgment of the innovative use of robot resumes generated from URDF files to enhance inter-robot communication and the effective leveraging of LLMs for heterogeneous multi-robot collaboration challenges. Apart from the positive comments, we would like to address the concerns and questions raised by the reviewer:

Comparison with similar frameworks and role-playing methods

We are really sorry that the writeup has led to some confusion. In the ablation study itself, we have already compared our framework with other similar baseline methods. For example,  w/o. Robot resume refers to Meta GPT[1] and w/o. Discussion refers to CMAS[2]. It is worth mentioning that Meta GPT[1] is a role-playing method that assigns different roles (managers, developers, test engineers) manually to LLM agents as we discussed in the section 4.4 in the submission, lines [475 - 478]. Our method’s success rate, sub-goal success rate, and time efficiency all constantly outperform the role-playing method with human-created profiles. We really thank the reviewers for pointing out this ambiguity, and we will revise the draft as soon as possible.

Performance on various scenarios and different robot configurations

We greatly appreciate this question, though we feel that the scope of this question could be somewhat broad.

First, we highlight that our Habitat-MAS benchmark is one of the most diverse and comprehensive datasets for involving 1) large-scale multi-floor multi-room indoor scenarios from Matterport3D [3] and HSSD [4]; 2) and diverse types of robots including drones, wheeled robot, legged robot, and different types of arms including revolute arms and prismatic arm, as we introduced in paper section 4.1, lines [372-379]. We believe our problem settings are representative and can cover a large ratio of indoor multi-robot scenarios and robot configurations that are on the commercial market.

Yet we agree that this setting is still not the general scenarios nor general multi-robot configurations. However, due to the limited time for the rebuttal session and massive engineering efforts in designing new environments, tasks and robot teams, we cannot provide experimental results to answer this question. We also believe that this problem can be better addressed by large corporations and institutes with their resources and labor.

Question about extension to game relationship

Thank you for your insightful question regarding the potential extension of our framework to scenarios involving game relationships. While our current implementation focuses on a fully interconnected, collaborative robot system, we recognize the importance of addressing more complex scenarios.

To extend our framework to game-theoretic situations, we propose two potential approaches:

• Introducing a group concept: We could incorporate the notion of groups or teams within the multi-robot system. This would allow us to model gaming relationships between different groups of interests, similar to scenarios found in competitive robotics applications like RoboMaster or robot soccer.
• Implementing a more complex communication graph: Instead of a fully connected graph, we could design a layered communication topology. This approach would better represent scenarios where information flow is restricted or strategic, as often seen in game-theoretic contexts.

However, we believe that the fundamental principles of our framework, particularly the embodiment-aware reasoning, could be adapted to handle these more complex interaction dynamics.

[1] Sirui Hong, undefined., et al, "MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework," in The Twelfth International Conference on Learning Representations, 2024.

[2] Chen, Yongchao, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. "Scalable multi-robot collaboration with large language models: Centralized or decentralized systems?." In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4311-4317. IEEE, 2024.

[3] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017

[4] Khanna, Mukul, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. "Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We really thank the reviewer's insightful question on the scalability of the current framework. As promised before, we are now updating the results of the scalability experiments:

Table 1: Scalability with Increasing Agent Numbers

In the first experiments of scaling up the robot number, we found that as the number scales up, the multi-agent system will face problems like hallucinations (in the setting of 10 agents) and the average success rate will decline. This is as expected since the hallucination problem in LLM is prevalent and it becomes worse with the increase of context length. This could be alleviated with more powerful LLM models as we have witnessed amazing progress in LLM model capability in the past year. Or designs like hierarchical communication with smaller sub-group discussions and larger group aggregation (delegate meeting) could help to solve the scalability problem in multi-agent discussion.

Table 2: Scalability with Increasing Task Complexity

In the second experiment, which involved scaling up task complexity, we found that our system demonstrates robustness to a certain extent. As the number of objects increases—representing greater task complexity—the system maintains a relatively high and stable success rate (above 70%). This indicates that our communication structure is effective and scalable, capable of handling more complex problems.

For the experiments raw data, you can check this file: exp_scalability.zip

Explanation of changing experiment setting

We are conducting this experiment on a workstation with a single RTX 4090. Due to the limitations of the simulator and computational constraints, rendering camera sensors for 20 robots always caused the program to crash. If time permits, there might be a workaround for the simulation to reduce the resource consumption. As a result, we simplified the experimental settings and removed the setting for 20 robots.

We thanks again for your thoughtful question.

We thank the reviewer for the time and effort you have invested in evaluating our work. We are grateful for the positive comments on the soundness and presentation of our paper. Although the reviewer thinks differently and “would recommend the authors to submit it for a Robotics conference”, we would like to address the reviewers’ concerns and do more clarifications:

Relevance to ICLR scope

Firstly, we would like to clarify why this paper is relevant to ICLR scope, in response to the reviewer's comment to submit to the Robotics conference. Due to the length limitation of the comment box, we have written a comprehensive relevance statement, available at Relevance Statement. You can also check all the open-source data and data processing script in https://github.com/EMOS-project/EMOS-project.github.io/tree/main/relevance_statement

In short, our work aligns closely with ICLR's growing focus on robotics, embodied AI, and benchmarking, as evidenced by the increasing number of accepted papers in these areas over the past few years.

(CLICK) statistics plot

We highlight that, only in ICLR 2024, there are 83 robotics and 103 benchmark papers ACCEPTED. EMOS addresses key challenges in embodiment-aware task planning that are currently missing in the Embodied AI community, making it highly relevant to ICLR's scope.

Regarding the novelty of our contribution

While we understand the reviewer’ s perspective, we believe our work presents significant innovations in the field of embodied AI and multi-robot systems. EMOS addresses a crucial gap in embodiment-aware task planning for heterogeneous multi-robot systems, which has not been adequately explored in previous research. Our approach of leveraging large language models (LLMs) for this purpose is novel and aligns with the growing focus on robotics and embodied AI at ICLR, as evidenced by recent accepted papers in this domain.

Suggestion of fine-tuning the LLM

We appreciate your suggestion to explore custom fine-tuning of the LLM to be tailored for our application. However, the focus of our paper is leveraging the strong priors and reasoning capabilities of pre-trained LLMs and design of LLM multi-agent systems to solve way too complex problems for a single LLM model, i.e. in the context of “LLM agents”. We agree that further adaptation could enhance performance, and we plan to explore this direction in future work.

Contribution of Habitat-MAS simulation

The purpose of this benchmark platform extends beyond mere implementation. It is for the broader Embodied AI community, enabling researchers to study and develop methods for further automation of complex multi-robot systems in a more general problem setting. As we stated in lines [92-96] of the submission paper, our benchmark tasks are processed such that not all robots or random agents can finish a specific task. The LLM agents need to understand their physical capabilities for the task planning.

Finally, we really thank the reviewer's time and effort in reviewing this paper. We also value the different perspectives the reviewer held at the beginning. We do hope the statistics and clarifications above can help us win the reviewer back. We would really appreciate it if the reviewer is willing to have more discussions here and we will respond with our best effort.

Dear reviewer,

We hope this message finds you well. We are writing to request your response to our rebuttal comment kindly. We would appreciate it if you could review our response and we could have helpful discussions here.

Kind regards,

The authors

Dear reviewers,

We greatly appreciate all the constructive comments from the reviewers. We have revised our paper according to the suggestions. Based on the revision V1, we have these extra updates:

• We added C.6 EXTRA EXPERIMENTS ON SCALABILITY OF EMOS. Thanks to insightful questions by Reviewer 79K2 and Reviewer jjgY about the scalability of the EMOS framework, we had helpful discussions about this issue. In this revision, we summarized the extra experiments and discussions about scalability to a new section in appendix to discuss this issue.

We have highlighted all the revised/added parts with blue color for the reviewers' convenience. We will remove the colorization in the formal version. We are also looking forward to reviewers' feedback on this revision.