LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination

We sincerely appreciate your recognition and the valuable comments! Extra experimental results can be found in this link.

Q1: The performance of population-based baselines with the multi-head network architecture.

A: We train the agents using a multi-head architecture (denoted as {}-mh) with the generated teammates. For testing with unseen teammates, we evaluate the baselines by selecting the best-performing head among all learned heads (the same as the -R1 variants in the paper), which requires extensive interactions. As shown in Table 11 in the link. the multi-head architecture significantly improves performance in LBF but yields no gains in the more complex SMACv2 environment. In both cases, a substantial performance gap remains between the baselines and SemDiv, highlighting the necessity of semantic-level diverse teammate generation.

Q2: Discuss essential references.

A: Thanks for pointing out these essential related work. We will add the discussion in the paper.

• The N-AHT framework [1] does align with our problem formulation, and we extend it by introducing natural language descriptions about testing teammates, making it more suitable for multi-agent systems with communication and intention sharing.
• InstructRL [2] is a notable work that leverages LLMs to guide RL by using their decisions to regularize policy learning. This approach offers an alternative to reward-generation methods like those discussed in our paper. However, InstructRL requires querying the LLM at every step, which is computationally expensive and limits its applicability to relatively simple tasks. Due to these constraints, we did not include it as a baseline in our study.
• ADVERSITY [3] and CoMeDi [4] propose novel techniques for cross-play based diversity, which can be categorized into the “policy-level” methods in our paper and introduced as strong baselines.

Q3: Human user studies and more complex games.

A: Both directions present valuable opportunities, particularly in bridging the gap between simulated coordination and human-AI teamwork, as well as extending the framework to richer strategic environments. We plan to investigate these challenges in future work.

Q4: Design choices for continual learning.

A: We adopt a continual learning paradigm instead of a two-stage framework (like FCP or MEP) because it allows us to leverage previously generated grounded behaviors as positive examples, guiding the LLMs toward progressively better behavior generation. In contrast, a two-stage approach, where behaviors are generated first and then used to train the teammate population, isolates the learning process of different teammates and may reduce diversity.

Q5: Design choices for agents’ RL instead of BC.

A: While behavior cloning (BC) from the complementary policy is a viable approach, we choose RL for two key reasons: 1. Surpassing complementary policy performance: BC merely imitates the complementary policy, capping performance at the quality of the training data. In contrast, RL enables exploration and optimization beyond demonstrations, potentially discovering superior coordination strategies. 2. Mitigating distribution shift: BC may struggle with OOD teammates due to its inability to recover from unfamiliar situations. Since our focus is on generalizing to unseen teammates, BC’s reliance on static datasets may lead to failure.

We apologize for unclear expressions and typos, and will correct them in the paper.

• Relationship between Section 3.3 and Macop: For training, we remove the head merging technique of Macop, since the novelty of new teammates if confirmed in the policy verification process. For testing, we utilize language-based reasoning to select optimal heads, avoiding Macop’s need for trial-and-error interactions to improving efficiency. We will rephrase the first sentence of the second paragraph in Section 3.3: To address these challenges, SemDiv adopts a multi-head network architecture \cite{owl,macpro} similar with Macop \cite{macop} and empowers the agents with continual learning ability.
• We report the 95% confidence intervals (CI) of SemDiv and the next best performing method (Macop-R1) in the link.
• Testing teammate generation: We train these teammates using hand-designed reward functions and manually verify that they exhibit the desired behaviors. These reward functions are simple and sparse (returning either 1 or 0), allowing for direct derivation of the R2 values.

References

[1] Wang et al. N-Agent Ad Hoc Teamwork. NeurIPS 2024.

[2] Hu et al. Language Instructed Reinforcement Learning for Human-AI Coordination. ICML 2023.

[3] Cui et al. Adversarial Diversity in Hanabi. ICLR 2023.

[4] Sarkar et al. Diverse Conventions for Human-AI Collaboration. NeurIPS 2023.

We sincerely thank you for the valuable comments! Extra experimental results can be found in this link.

Q1: Experiments with more agents and with continuous action space.

A: SemDiv is agnostic to team sizes and action spaces, still achieving the best performance compared with baselines, indicating its scalability and potential to solve more complex tasks. The results are in Table 8 in the link.

First, we extend the original LBF task to a 3-agent setting, where it takes 3 agents to collect a food at the same step. During testing, one unseen agent joins the team. Training details (like MARL methods, prompts for LLMs, etc), and behaviors of testing teammates, are similar with the 2-agent experiments.

Next, we include the Spread task based on the HARL codebase [1], in which 3 agents with continuous action space need to approach three different landmarks. For training, we select MAPPO similar with GRF. For LLM prompts, we slightly modify the ones used in PP since both of them are MPE tasks. During testing, 6 unseen teammates trained in teams with different approaching strategies (different agent-landmark pairs) are joined.

Q2: Testing with lazy agents.

A: We build lazy agents that take the noop action or move back and forth in the initial position, in all four environments used in the paper. We set their descriptions as “I am a lazy agent and I will do nothing” when testing SemDiv. The results are shown in Table 9.

• LBF & SMACv2: All methods fail as ≥2 agents are needed for food/enemy tasks.
• PP: Baselines match SemDiv by ignoring lazy teammates (reward for catching a rabbit = 0.5).
• GRF: SemDiv excels by learning attack strategies without passing and selecting these policy heads.

Q3: Discuss several assumptions and limitations.

A: (1) Environment APIs, and language descriptions of episodes. Both assumptions are important considerations, which are well-established in prior LLM-assisted RL research [2,3,4]. However, we acknowledge that relaxing these assumptions is valuable future work. (2) Language descriptions of teammates' behavior during execution. Please refer to Q1 for reviewer EFqS. (3) Dependence on LLMs. Like prior work, our method relies on LLMs, which may hallucinate. Potential solutions include using more advanced models or human verification. (4) Generalization scope. Our evaluation focuses on close games with clear rewards. Extending to real-world tasks (e.g., embodied AI) introduces challenges like real-time perception and physical constraints. We will discuss these limitations explicitly in the paper.

Q4: How does SemDiv handle contradictory behaviors across teammates?

A: SemDiv uses a multi-head architecture (Sec. 3.3) to mitigate catastrophic forgetting in multi-agent settings [5,6]. Testing confirms the final agents successfully coordinate with all 6 learned teammates, including those with contradictory behaviors (Table 10).

We apologize for these unclear expressions, and will improve them in the paper.

• Complementary policy: we aim to train a teammate policy $\pi^{tm}$ (e.g., player A in a two-player game) to form a team and train with agent $\pi^{ag}$ (e.g., the other player B). To achieve this, $\pi^{tm}$ must first learn coordination by training alongside a complementary policy $\tilde{\pi}^{tm}$ (which also controls Player B) using MARL algorithms. This training phase with $\tilde{\pi}^{tm}$ ensures that $\pi^{tm}$ develops specific coordination behaviors before interacting with $\pi^{ag}$.
• $\lambda$ configurations in Eq. 2: The equation ensures the new teammate $\pi^{tm}_m$ differs from previous ones $\pi^{tm}_j$. When $\lambda_1 = 1, \lambda_2 = 0$, it verifies that agents trained with $\pi^{tm}_j$ cannot achieve comparable original task rewards. When $\lambda_1 = 0, \lambda_2 = 1$, it checks the same for shaped rewards. Together, these configurations strictly guarantee the new teammate's distinctiveness.
• Explanations to the figures and SemDiv-R1/R2: Please refer to the reply for reviewer XbFR.
• Second paragraph of Section 4.1: The five teammates mentioned here are the ones for testing different methods, rather than the six teammates generated during training. We again apologize for the unclear expressions.

References

[1] Liu et al. Maximum entropy heterogeneous-agent reinforcement learning. ICLR 2024.

[2] Xie et al. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. ICLR 2024.

[3] Ma et al. Eureka: Human-Level Reward Design via Coding Large Language Models. ICLR 2024.

[4] Ma et al. DrEureka: Language Model Guided Sim-To-Real Transfer. RSS 2024.

[5] Yuan et al. Multi-agent Continual Coordination via Progressive Task Contextualization. TNNLS 2024.

[6] Yuan et al. Learning to Coordinate with Anyone. DAI 2023.

We sincerely thank you for the valuable comments!

Q1: Difference between “policy level” and “semantic level”, and why optimizing policy level differences is bad.

A: Policy-level methods mainly optimize policy-space diversity without explicitly modeling the corresponding semantics. In this approach, the search space grows exponentially with the number of agents, leading to inefficient exploration. In contrast, our semantic-level method explicitly models human-interpretable coordination behaviors through language-guided generation. By clustering similar policies into semantically meaningful behaviors, our approach significantly reduces the complexity of the search space.

Prior works [1,2] have provided analysis and conducted experiments to demonstrate that traditional policy-level methods (e.g., FCP and LIPO) struggle to efficiently explore teammate policy spaces. We further validate this claim through comprehensive performance evaluations and trajectory analysis, highlighting the advantages of semantic-level optimization over policy-level approaches.

Q2: More results on t-SNE visualization, cross-play matrices, etc.

A: We report these important results in this link, and will add them in the paper.

Q3: Requirement of natural language behavior descriptions from unknown teammates.

A: Please refer to Q1 for reviewer EFqS.

Q4: The performance of Macop.

A: We would like to clarify that we have actually included the performance of Macop in Table 1, though presented as its upper bound versions that directly report the results of the best policy heads. However, there is still a gap between Macop-{R1, R2} and SemDiv, proving the impact of SemDiv’s semantically diverse teammates.

Q5: [Clarification Question] How SemDiv selects policy heads and the SemDiv-Dist baseline.

A: During training, all methods are unaware of testing teammates' behavior descriptions. SemDiv agents only select heads for the corresponding training teammates for policy verification. During testing, SemDiv agents select heads according to the testing teammates’ behavior descriptions. The SemDiv-Dist baseline also requires these descriptions during testing to select heads. Instead of using LLMs for reasoning, it selects the head that minimizes Distance(embed($b$), embed($b_{test}$)), where $b$ is the behavior learned by the head, $b_{test}$ is the behavior of the testing teammate. To improve SemDiv-Dist, we can finetune the language model that computes the embeddings with task-specific data, enhancing its understanding of the task.

Q6: The reward shaping problem.

A: In our work, we empirically set a fixed step limit for inner-loop training, a practical trade-off that may limit the "deepness" of learnable behaviors. While this suffices for simpler tasks, we fully acknowledge the challenge for more complex behaviors. Future work will adopt Neural Architecture Search (NAS) [3] to automate hyperparameter setting, enabling efficient training of deeper behaviors under shaped rewards. We will clarify this limitation and solution in the paper.

We apologize for these unclear expressions, and will improve them in the paper.

• Number of testing teammates and controlled agents: We have modified the Abstract and Introduction to explicitly state: “Evaluation with five unseen representative teammates per environment”, and we will add an explanation in the Problem Formulation that we focus on scenarios where N=2.
• Related work of Section 3.3: Please refer to the reply for reviewer Q2F9.
• More detailed figure captions:
  • Figure 1: An overview of the training and testing process of SemDiv. Left: During training, SemDiv proposes novel coordination behaviors in natural language and transform them into teammate policies for agent learning. Right: During testing, SemDiv takes as input the description of the unseen teammates and selects the optimal learned policy for coordination.
  • Figure 2: The overall workflow of SemDiv. (a) Generating coordination behavior. SemDiv iteratively generates of semantically diverse coordination behaviors, enabling efficient exploration of the teammate policy space. (b) Training aligned teammate policy. For each coordination behavior described in natural language, a teammate policy is trained to align with that behavior. (c) Training agents. Agents are continually trained with these teammates, developing strong coordination ability.
• Upper bounds: {SemDiv, Macop}-R1, R2 are only upper bounds of {SemDiv, Macop}, which directly report the results of the best policy heads to investigate the impact of the head selection module. We again apologize for these unclear expressions.

References

[1] Yuan et al. Learning to Coordinate with Anyone. DAI 2023.

[2] Rahman et al. Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents. AAAI 2024.

[3] Elsken et al. Neural Architecture Search: A Survey. JMLR 2019.

We sincerely thank you for the valuable comments! Extra experimental results can be found in this link.

Q1: The motivation for unseen teammate communicating behaviors before testing.

A: Communication and intention sharing are the basic and core techniques in multi-agent systems [1,2], mitigating partial observability. Following these techniques, our approach leverages one-time natural language communication to boost coordination efficiency. For situations where communication is entirely infeasible, SemDiv can still infer teammate behaviors from interactions using techniques in [3,4]. We note this as a future direction to extend our framework to zero-comm environments.

Q2: Compare more baselines to support the claims in Section 4.4.

A: As shown in Table 1 in the link, even with extra policy-level diversity techniques and larger population sizes, policy-level baselines still achieve limited performance, further indicating the necessity and efficiency of semantic-level diversity.

Q3: Compare baselines with Policy Verification.

A: Even with Policy Verification, population-based baselines show limited improvement (Table 2). The Policy Verification process: (1) Train 12 teammates, verifying all can complete the task. (2) Select the 6 most diverse policies:

• MEP: Farthest from population mean.
• LIPO: Iteratively pick worst-performing partners.

No behavior verification is needed as baselines lack semantic info. Results confirm LLM-assisted diversity drives SemDiv's success, not just Policy Verification.

Q4: Pass rate of Policy Verification in SemDiv.

A: We report the results of Policy Verification (Table 3), including (1) the number of occurrences of different issues averaged over 3 random seeds and (2) total pass rate. SemDiv efficiently generates diverse teammates while verifying their quality. We will add these results in the paper.

Q5: Similarity information and agent behavior text representation.

A: We simply represent info_sim as: “Not-novel example: {behavior}. This behavior is the same as {similar_behavior}“. For behavior texts, we write light-weight functions to extract information from trajectory data and fill in text templates, aligning with previous works [5,6]. For example, in SMACv2, info = “Agents killed enemy {enemy_id} …”. We will add more details of text representation in the paper.

Q6: Whether SemDiv’s improvement is stemmed from the instability of baselines.

A: SemDiv’s improvement is stemmed from better semantic-level diversity rather than exploting the instability of baselines. We report the performance of all generated teammates in Table 4. The teammate stability and quality of baselines is similar to that of SemDiv.

Q7: Substantiate previous methods‘ inefficient exploration of the teammate policy space.

Efficiently exploring diverse high-quality teammate policies is a fundamental challenge in MARL for open environments [7]. Early methods primarily imposed regularization at the action level, requiring exhaustive traversal of the joint policy space, which becomes computationally prohibitive in complex scenarios. Recent works like [4,8] have demonstrated the limitations of classic policy-level methods like FCP and LIPO. In contrast, our approach abstracts the joint policy space into a higher-level semantic space, where a single semantic behavior can correspond to a wide range of policies. This abstraction explicitly reduces exploration difficulty by decoupling diversity from low-level action redundancy. Empirical results validate that SemDiv achieves more efficient and scalable policy exploration compared to prior works, representing a significant technical advancement in this direction.

Q8: Mention related work in Eq. 2.

A: We apologize for not properly citing the work [9] when introducing Eq. 2 and explaining its relevance for policy similarity verification. We will correct this by adding the citation at Eq. 2 and providing a clearer discussion.

References

[1] Albrecht et al. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press, 2024.

[2] Zhu et al. A survey of multi-agent deep reinforcement learning with communication. Auton. Agents Multi Agent Syst. 38(1): 4 (2024).

[3] Yuan et al. Multi-agent Continual Coordination via Progressive Task Contextualization. TNNLS 2024.

[4] Yuan et al. Learning to Coordinate with Anyone. DAI 2023.

[5] Xie et al. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. ICLR 2024.

[6] Ma et al. Eureka: Human-Level Reward Design via Coding Large Language Models. ICLR 2024.

[7] Yuan et al. A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environment. 2023.

[8] Rahman et al. Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents. AAAI 2024.

[9] Charakorn et al. Generating diverse cooperative agents by learning incompatible policies. ICLR 2023.