Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication

We thank the reviewer for appreciating our work, and we seek to clarify our experiment design with the following responses.

Q1: Figure 1 can be better presented

Your suggestion is well received. Thank you for pointing it out. We have remade the framework illustration in Figure D in the PDF and will add it to the final paper.

Q2: Zero-shot generalization experiment

We apologize for the confusion. We seek to clarify the design of the zero-shot generalization experiment to address your concern.

It's worth noting that there are two levels of zero-shot generalization in our work. The first level is communication alignment, where LangGround agents generate human-interpretable communication in task states they have never been grounded on during training. The second level is task completion, where LangGround agents are trained on environment A and are still able to complete the task in unseen environment B.

In Section 6.3.1, we remove a subset of prey spawn locations (i.e., 4 out of 25 cells) during training and evaluate LangGround agents with those unseen states in a zero-shot manner. This experiment requires both task-level and communication-level generalizability because those removed states are 1) unseen during training, and 2) not grounded with human language. Table 3 shows the inclusive list of 4 unseen states and the measurements of communication alignment with unseen ground truth. This is equivalent to your suggested experiment on a smaller scale, and we indeed found that "the communication messages can handle new coordinates, e.g., (1,1), (1,3), (3,1), (3,3), which were previously unseen in the training scenarios".

We acknowledge your suggestion about training on a 5x5 gridworld and testing on a 10x10 gridworld. However, such task-level generalization depends more on the backbone RL model structure and specific environment design, which is out of the scope of this work. Therefore, we present an alternative experiment focusing on communication-level generalization. We train LangGround agents on a 10x10 gridworld but only provide language grounding in a subset of states. The learning curves of LangGround agents with different levels of language grounding are presented in Fig. B and Fig. C in the PDF file. As shown in the figures, the more grounded states, the better the team performance, as well as the better the communication alignment. In addition, Table A in the PDF shows the results of communication alignment measurements in un-grounded states. Similarly, the more grounded states, the better alignment we observe in communication-level zero-shot generalization.

To summarize, the communication-level generalizability of LangGround stems from the assumption that we could ground the agent communication space with the word embedding space of human language on a limited number of instances, and expect LangGround to output interpretable messages in un-grounded states via topographic similarity of the aligned space. In practice, this assumption depends on many factors such as the number of grounded states, dimension of the communication space, scale of the problem, etc. The above experiment illustrates the impact of language grounding percentage (25%, 50%, 75%, 100%), and we leave further investigation to future work.

Q3: Ad-hoc teamwork experiment

We apologize for the confusion. Brief descriptions about ad-hoc teamwork experiments are presented in Section 6.3.2 and Appendix A.3.2 of the original paper. We will add more details to the revised paper.

Ad-hoc teamwork refers to situations where agents collaborate with unseen teammates without pre-coordination. In this work, we use embodied LLM agents to emulate human behaviors in human-agent ad-hoc teams. We match 2 MARL agents with 1 unseen LLM agent in a team and ask them to complete the collaborative task in predator-prey and USAR environments. The LLM agent is powered by GPT-4-turbo and prompted to output action selection and communication messages given observation inputs, similar to the data collection process introduced in Section 4.1. The Gym environment is wrapped with a text interface to decode observations into English descriptions and encode the LLM agent's output into concrete action selection. Both MARL agents and LLM agents interact with the same task environment in sequence. Natural language communication messages from LLMs are embedded using OpenAI's word embedding API and sent to MARL agents. The communication vectors from MARL agents are translated to English sentences via cosine similarity matching in dataset $\mathcal{D}$. Their team performance is measured by the number of steps spent in completing the task.

The main reason for only evaluating each ad-hoc team composition on 24 episodes is the cost of calling OpenAI's API for LLM agents. In our case, each agent consumes 2k - 5k tokens per round, and each episode takes 20 - 50 rounds to complete. Evaluating 5 team compositions on 3 environments over 24 episodes costs around $500. We plan to conduct more comprehensive evaluations in the future with different backbone models and human participants.

Q4: Realistic environments

Your suggestion is well received. As for future work, we plan to extend LangGround into more realistic task environments such as ALFWorld, RoCoBench, and ThreeDWorld Transport Challenge, where agents must physically interact with real-world objects based on visual and text inputs. Our proposed pipeline is agnostic to environments and backbone LLM and RL models, therefore it can be easily generalized to those scenarios.

We sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.

We thank the reviewer for appreciating the motivation and generalizability of our work, and we seek to clarify the framing of our research.

Q1: Missing comparison with CICERO

Thank you for pointing out this paper. We indeed came across it but decided not to include it in the comparison because the motivation and technical approach of CICERO are inherently different from our work. CICERO has an RL module for strategic reasoning and a language model for generating messages. The two modules are trained separately on different datasets, namely self-play trajectories and conversation data collected from human players. During inference, the strategic reasoning model outputs actions and intent estimations conditioned on game state and dialogue history, while the language model generates messages based on game state, dialogue history, and intent estimation. It is clear that the two modules in CICERO function independently, with the only connection being that the language model takes intention estimation from the planning module as input.

However, in our work, both action and communication are generated by individual RL agents that are trained end-to-end with a combination of RL loss and supervised learning loss. As shown in Fig. 1, each LangGround agent is controlled by an LSTM policy, and its hidden state $h_t^i$ is used to generate vector $c_t^i$ for inter-agent communication. The gated average communication vector is taken by each agent's policy to output the action. We do not use an LLM to generate communication in our pipeline. Instead, a dataset of example messages is used to calculate the supervised learning loss.

Although CICERO and LangGround both result in agents capable of communicating with humans in competitive/collaborative tasks, their major contributions are fundamentally different. CICERO focuses on empirically evaluating the proposed hierarchical framework in real-world scenarios, while our work focuses on proving the concept of aligning MARL agents' emergent communication space with the semantic space of human language. We hope our explanations address your concern.

Q2: Does RL determine useful information to share or not?

Please refer to our answer to Q3 in the overall rebuttal.

Q3: Concerns about Cosine similarity and BLUE socre

We apologize for the confusion and seek to clarify the design of the zero-shot generalization experiment in the context of the previous argument about our model structure.

In Section 6.3.1, we remove a subset of prey spawn locations during training and evaluate LangGround agents with those unseen states in a zero-shot manner. To accurately retrieve unseen English sentences from the dataset, the agent needs to generalize their communication vector in the learned semantic space in a similar way as in the word embedding space.

For example, in a hypothetical scenario, the agent was only trained on prey locations (0,0) and (2,2) but needs to communicate about (1,1) during evaluation. It must generate an interpolated vector that lies between vectors referring to known locations in the high-dimensional communication space. Because this space is aligned with the word-embedding space, the interpolated vector is likely to be close to the corresponding unseen messages about (1,1) in the dataset. In practice, the generalization might not be done via linear interpolation, depending on the properties of the aligned high-dimensional space.

The above-mentioned process is the prerequisite for achieving high cosine similarity and BLEU scores, as well as retrieving unseen sentences correctly in zero-shot generalization. It is clear that this process is not guaranteed 'by construction', but rather depends on the alignment between the agent communication space and word embedding space of human language. As mentioned earlier, the LangGround agent is solving a multi-objective optimization problem with a single LSTM model. The agent policy must trade off completing the task and aligning communication, and neither of them can be solved easily. Therefore, we design different criterias to evaluate the agents in both task performance and communication alignment respectively.

We also present additional zero-shot generalization experiment results in the overall rebuttal. Please refer to our answer to Q4 for more information.

Q4: Compare GPT-4 with LangGround

The comparison results are presented in Table 4 in the original paper. The LLM row corresponds to the team performance of 3 embodied agents powered by GPT-4-turbo. As analyzed in Section 6.4.2, LLM-only teams perform worse than MARL-comm teams, and better than ad-hoc teams.

Q5: LangGround relies on LLMs' performance

Please refer to our answer to Q2 in the overall rebutall.

We sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.

Compare LangGround with LLM embodied agents

Although comparing LLM embodied agents with MARL agents (e.g., LangGround) is not the main focus of this paper, we provide our analysis in three parts:

• Many studies evaluate LLM's planning capability for embodied agents. It is widely acknowledged that embodied LLM agents may generate infeasible action plans in interactive tasks due to hallucinations and bias from training data [1]. Consequently, MARL agents are generally more efficient in completing tasks than pre-trained LLM agents [2]. Specifically in our case, LLM and LangGround perform similarly because 1) LangGround is not 100% optimized for task reward, 2) LangGround does not use SOTA MARL models as the backbone (e.g., MAPPO), and 3) LLM embodied agents are heavily prompt-engineered for specific environments.

• LLMs have a large number of parameters and thus take longer to run. In our case, calling OpenAI's API is ~50 times slower than running local inference of MARL models and costs ~5 dollars for each episode. Taking the massive time and resources required to train LLMs into consideration, MARL models are more affordable and eco-friendly in solving specific tasks.

• The effort required for generalizing LangGround to different scenarios is smaller compared to embodied LLM methods. Because LangGround only uses effective communication messages from LLMs, a mere amount of prompt engineering is required compared to alternative methods that require LLMs to generate actionable plans for each step.

Gating experiment

We ran this ablation study to answer your original question about attributing RL to communication. We came to similar conclusions that MARL indeed learns to selectively communicate with the help of language grounding, i.e., mediator effect. We also find the results interesting and plan to further explore them in future work. It would be helpful for us to improve the work if you could specify the potential research directions you have in mind.

Generalization

We would like to clarify the experimental settings to help you better evaluate the zero-shot generalization results. The dataset $\mathcal{D}$ we used in the evaluation is constructed from limited trajectories of LLM agents, thus usually does not cover all (obs, action) combinations. For example, the full dataset of $pp_{v0}$ contains 1893 data points which covers only 45 out of 50 possible observations. The additional zero-shot generalization experiments were conducted on a 10 by 10 map with an even smaller $\mathcal{D}$ of size ~1000 data points, due to time and resource limitation. This means the 100% condition in the attached PDF represents roughly 12.5% (the map size is 4 times bigger, and the dataset is half the size) of all possible (o, a) combinations. With this being said, the generalization performance we observe in Table A is approximately between 12.5% to 3% states grounded. We will revise the description to make the experimental setting clearer. We hope this could help you calibrate your evaluation of the scalability of LangGround in more complex scenarios.

[1] LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks. arXiv 2024

[2] Theory of Mind for Multi-Agent Collaboration via Large Language Models. EMNLP 2023

We are sorry to see you have changed your overall rating from '5 = borderline accept' to '3 = reject' after reading our rebuttal and additional experiment results. We seek to provide additional clarifications for your remaining concerns.

GPT-4 comparison

We acknowledge your evaluation of the results in Table 4 that the LLM-only team performs on par with MARL-only teams. However, as pointed out in Section 6.3.2, the comparison among homogeneous teams is not the focus of ad-hoc teamwork experiments. Instead, we try to quantify the performance loss when introducing an unseen teammate into the team, and the degree of human-agent communication achieved via LangGround compared to random baselines, by comparing different ad-hoc teams (e.g., LLM+ aeComm, LLM+LangGround). Therefore, whether pure LangGround teams perform better than pure GPT-4 teams does not impact the conclusion we draw in the paper. We do not see this as a weakness of our work by any means.

Communication protocol is a distillation of example messages

We acknowledge your hypothesis about "distillation," as it is a possible analogy of our proposed pipeline. However, as mentioned in the overall rebuttal, it is impossible to fully decouple the contribution of RL and SL with our current model structure. We have no way to verify the hypothesis about "interpretation" and can only attribute both communication and action to RL and SL as a multi-objective optimization problem.

More importantly, whether the hypothesis is true or false does not diminish the value of our work. Many previous works in multi-agent communication learn a latent representation of observation and directly use the encoding as a communication vector [1, 2]. Our method works similarly, but we use word embeddings of observation-action descriptions as the latent representation. Whether RL determines information to share or "interprets" messages are merely different technical approaches, rather than critical concerns that limit our pipeline.

Communication sparsity

We seek to clarify the misunderstanding you might have regarding the experiment settings and the value of sparse communication.

1. We used the semi-cooperative setting for predator-prey (i.e., mixed mode) where agents receive individual rewards. The original statement in IC3Net you cited only applies to fully-cooperative settings where agents receive a global shared reward. In fact, the communication probability learned by LangGround agents is around 80%, which aligns with the results reported in the IC3Net paper.
2. The value of sparse communication is well acknowledged by the research community. For example, IC3Net outperforms CommNet in mixed predator-prey by introducing the gating function to allow agents to learn to communicate when necessary. Previous research proves that a sparse communication protocol can be learned with little to no performance loss [3]. The statement that 'learned communications are often uninformative' is exactly the motivation for exploring sparse communication. We see this as the nature of agent and human communication, rather than a weakness of our work. We leave further exploration to future work because communication sparsity is out of the scope of this paper.

LangGround relies on LLMs

We acknowledge your statement that LangGround's communication depends on the quality of the example dataset. However, the collection of this dataset is fairly accessible. LLMs are known to have good linguistic capabilities (e.g., describing) while struggling with formal reasoning (e.g., planning). Our pipeline actually allocates the appropriate task to LLMs, i.e., describing the obs-action pairs, and leaving the planning part for RL. In more complicated scenarios in which communication is beyond describing observation and action, we could still expect LLMs to generate reasonable outputs since they were trained on massive conversations and dialogue data. Compared to alternative methods in embodied agents where LLMs must make correct action planning at every timestamp, it is more feasible to collecte semantically meaningful messages from either LLMs or any other sources.

To be continued.

[1] Learning to ground multi-agent communication with autoencoders. NeurIPS 2022

[2] Learning to Communicate using Contrastive Learning. ICLR 2024

[3] Towards True Lossless Sparse Communication in Multi-Agent Systems. ICRA 2023

We thank the reviewer for acknowledging the originality and soundness of LangGround, our proposed method for MARL agents with human-interpretable communication. We seek to clarify your questions with the following responses and additional experimental results.

Q1: No theoretical proof

We do not provide a proof for our work because the theories we used in deriving the proposed method are from previous literature in emergent communication and MARL-comm. However, we indeed provided analysis to explain why our method works in practice in the discussion section.

The core idea of LangGround is providing an auxiliary loss to directly regulate intermediate hidden layers of the RL agent's policy network. This technique has been proven effective in stabilizing the learning process via additional supervision [1]. In MARL-comm, previous work has used representation learning methods (e.g., autoencoder [2], contrastive learning [3]) to construct communication messages solely based on encoding observations. Our pipeline combines these two techniques by using language priors to regulate RL agent's communication via supervised learning. Our dataset $\mathcal{D}$ consists of expert trajectories from LLM embodied agents with a well-established grounding on the task. We believe our method helps agent learn a semantic representation of task-related information grounded on word embeddings of example messages.

The other relevant theory is the Information Bottleneck Theory. In emergent communication, a speaker and a listener jointly learn a communication protocol to maximize the task reward. But this may compromise other properties of the learned language such as task-agnostic informativeness and human interpretability [4]. To overcome this issue, we align the agent communication space with the semantic space of human language by grounding them on a small set of data, and add the supervised learning loss to the task loss [5]. This explain the trade-off between utility and informativeness we observe across task environments, representing different solutions of the multi-objective optimization problem's Pareto front.

Q2: Evaluate LangGround in scaled environments

Please refer to our answer to Q1 in the overall rebutall.

Q3: LangGround relies on LLMs' performance

Please refer to our answer to Q2 in the overall rebutall.

Q4: Details about MARL implementation and ad-hoc teamwork are missing

We apologize for the confusion. Descriptions about ad-hoc teamwork experiments are presented in Section 6.3.2 and Appendix A.3.2 of the original paper. Implementation details of MARL agents are presented in Section 4.2 and Appendix A.1.2. We provide additional descriptions and a re-made framework illustration in Figure D in the PDF. We will add them to the final paper.

Ad-hoc teamwork refers to situations where agents collaborate with unseen teammates without pre-coordination. In this work, we use embodied LLM agents to emulate human behaviors in human-agent ad-hoc teams. We match 2 MARL agents with 1 unseen LLM agent in a team and ask them to complete the collaborative task in predator-prey and USAR environments. The LLM agent is powered by GPT-4-turbo and prompted to output action selection and communication messages given observation inputs, similar to the data collection process introduced in Section 4.1. The Gym environment is wrapped with a text interface to decode observations into English descriptions and encode LLM agent's output into concrete action selection. Both MARL agents and LLM agents interact with the same task environment in sequence. Natural language communication messages from LLMs are embedded using OpenAI's word embedding API and sent to MARL agents. The communication vectors from MARL agents are translated to English sentences via cosine similarity matching in dataset $\mathcal{D}$. Their team performance is measured by the number of steps spent in completing the task.

We sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.

[1] Lee, C. Y., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2015, February). Deeply-supervised nets. In Artificial intelligence and statistics (pp. 562-570). Pmlr.

[2] Lin, T., Huh, J., Stauffer, C., Lim, S. N., & Isola, P. (2021). Learning to ground multi-agent communication with autoencoders. Advances in Neural Information Processing Systems, 34, 15230-15242.

[3] Lo, Y. L., & Sengupta, B. (2022). Learning to ground decentralized multi-agent communication with contrastive learning. arXiv preprint arXiv:2203.03344.

[4] Tucker, M., Levy, R., Shah, J. A., & Zaslavsky, N. (2022). Trading off utility, informativeness, and complexity in emergent communication. Advances in neural information processing systems, 35, 22214-22228.

[5] Tucker, M., Li, H., Agrawal, S., Hughes, D., Sycara, K., Lewis, M., & Shah, J. A. (2021). Emergent discrete communication in semantic spaces. Advances in Neural Information Processing Systems, 34, 10574-10586.

We thank the reviewer for acknowledging the novelty and soundness of LangGround, our proposed method for MARL agents with human-interpretable communication. We appreciate your constructive comments and suggestions and seek to clarify them with the following responses and additional experiment results.

Q1: Human Subjects experiment

We acknowledge the necessity of evaluating our method with real human participants given the ultimate goal of learning human-interpretable communication. However, we decide to leave human subject experiments for future work based on the following considerations:

• We want to focus the scope of this paper on the technical details of the proposed pipeline and evaluation of aligned communication space. Human experiment results might be influenced by many factors (e.g. demographics, background, attitude to AI) therefore need more considerations.
• Many works have proved the effectiveness of LLMs in simulating human behaviors and replicating data collection from human participants [1]. This is our original motivation for using LLMs to generate synthetic data for LangGround. It is a reasonable first step to conduct a sanity check by evaluating LangGround in ad-hoc teamwork with embodied LLM agents, before directly testing it with human subjects.
• Our method is the only model that learns an aligned communication space with natural language, facilitating direct conversation with humans. Alternative models either do not have any interpretability (e.g. IC3Net, aeComm), or only learn a semantically meaningful space independent from human language (e.g. protoComm, VQ-VIB). In the ad-hoc teamwork experiments, we actually give baseline models an advantage by allowing them to use the offline dataset of LangGround and OpenAI's word embedding model. Otherwise, no alternative models are able to communicate with embodied LLM agents or humans in natural language.

Q2: Evaluate LangGround in scaled environments

Please refer to our answer to Q1 in the overall rebutall.

Q3: What is the primary difference between LangGround and [2]

Thank you for pointing out potential directions for improving the presentation of our work. We will add more details to explain the method used in previous work [2] and compare it with LangGround.

Essentially, [2] proposes a prompt engineering technique to improve the performance of embodied LLM agents in collaborative tasks, by allowing them to keep track of key task-related information. They evaluated this method in USAR environment and proved its effectiveness. While in this paper we propose a pipeline to improve the robustness and interpretability of emergent communication learned by Multi-Agent Reinforcement Learning agents. Therefore, the motivations and technical approaches of [2] and LangGround are inherently different, although both result in artificial agents capable of collaborating and communicating with humans.

Q4: Compare MARL-comm agents with embodied LLM agents in general

Regarding this question, we provide the argument in two-fold:

• MARL and LLM are trained with different learning objectives: MARL is optimized for maximizing the expected reward from environment, while LLMs are optimized to predict the next word given the context. Embodied LLM agents may generate infeasible action plans in interactive tasks due to hallucinations and bias from training data. Consequently, MARL agents are generally more efficient in completing tasks than pre-trained LLM agents [2].
• LLMs have a large size of parameters thus take longer to run. In our case, calling OpenAI's API is ~50 times slower than running local inference of MARL models, and costs ~5 dollars for each eposide. Taking the massive time and resources required to train LLMs into consideration, MARL models are more affordable and eco-friendly in solving specific tasks.

Q5: Impact of LLM's hallucination and performance on LangGround

Please refer to our answer to Q2 in the overall rebutall.

Q6: Prompt engineering is required for generalization to new environments

In practice, yes. But the amount of effort required for generalization is much smaller when compared to alternative methods in either embodied LLM agents or MARL-comm literature. The rationale is similar to the argument we made for Q5, because LangGround only uses effective communication messages from LLMs, a mere amount of prompt engineering is required compared to alternative methods that require LLMs to generate actionable plans for each step. Learning communication protocol from scratch is known to be challenging in the MARL community. Our method stabilizes the learning process and reduces the amount of engineering work (e.g. hyper-parameter searching) when generalizing to new domains, by introducing language priors from general knowledge base, LLMs.

We sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.

[1] Using large language models to simulate multiple humans and replicate human subject studies. ICML 2023

[2] Theory of Mind for Multi-Agent Collaboration via Large Language Models. EMNLP 2023