N-agent Ad Hoc Teamwork

Novelty - Encoder/Decoder-based Agent Modelling

While we understand the reviewer’s reservations on the prevalent use of the encoder-decoder architectures for agent modeling, we believe it should not be the sole basis for assessing the novelty of our work.

Despite using encoder-decoders for agent modeling, whose use was proposed as early as 2016 [1], some important recent works in AHT research [2, 3, 4] have contributed to the community by proposing creative ways to use representations produced by the encoder-decoder network to tackle unaddressed problems/settings in AHT teamwork. Therefore, we argue that how the agent modeling component is used to address existing issues in AHT research should also be considered when measuring novelty.

Here, our innovation is our use of the agent modeling component’s representations during joint training for all controlled agents, for addressing NAHT problem. We show through our motivating example (Section 4) and experimental baseline of POAM-AHT (Fig. 3) that solely relying on agent modeling without joint training will yield controlled agents whose joint policy is suboptimal for addressing even the simplest NAHT problems. In the end, we believe that the AHT community could build upon this insight to extend current AHT methods to NAHT methods. We expect this insight to lead towards better solutions to the NAHT problem, which may come from improvements to the agent modeling component or the joint training process.

Convergence & Performance Guarantees, other theoretical analysis

From a theoretical perspective, POAM would inherit IPPO’s convergence guarantees for a nonstationary learning environment, which have recently been established [6].

In POAM, the gradient corresponding to the encoder-decoder (ED) embeddings is detached when the encoder-decoder embeddings are computed in the actor and critic networks. Thus, the actor/critic updates do not change the weights of the ED networks. Conversely, as the ED update is fully independent of the actor/critic networks, updates to the ED do not change the weights of the actor/critic networks.

The effect of this architecture and update scheme is that the ED updates cause a small amount of nonstationarity for the PPO backbone learning algorithm. Empirically, we address this issue by using a small learning rate for the ED. Further, as POAM uses an independent learning scheme, another source of nonstationarity is the updates of the controlled agents themselves.

Scalability to Increasing Number of Agents & Task Complexity

In practice, the paper demonstrates that POAM is effective on tasks with a variety of team sizes. The task with the most agents is the 10v11 task, which has 10 agents on the allied agent team, and 11 enemies controlled by the game AI, which is a relatively large number of agents—especially in the context of AHT research, which typically considers only 2 agents.

We employ several techniques to help POAM scale to a larger number of agents. Multi-agent Reinforcement Learning: the backbone multi-agent learning algorithm is independent learning with parameter sharing [5], which aids scalability in the homogeneous agent setting.

Agent Modeling: In the agent modeling problem, the prediction target dimension increases linearly with the number of agents. To prevent the output dimension of the decoder network from similarly scaling with the number of agents, the decoder employs parameter sharing as well.

References

[1] He et al. Opponent modeling in deep reinforcement learning. ICML 2016.

[2] Papoudakis et al. Agent modelling under partial observability for deep reinforcement learning. NeurIPS 2021.

[3] Zintgraf et al. Deep Interactive Bayesian Reinforcement Learning via Meta-Learning. AAMAS 2021.

[4] Gu et al. Online ad hoc teamwork under partial observability. ICLR 2022.

[5] Christianos et al. Shared experience actor-critic for multi-agent reinforcement learning. NeurIPS 2020.

[6] Sun et al. Trust region bounds for decentralized PPO under non-stationarity. AAMAS 2023.

Out-of-Distribution Evaluation - Environment Selection

Hanabi represents a challenging scenario for AHT and requires a large amount of computational resources. Prior papers have used billions of training steps to train agents, in contrast with the tens of millions used in our work [1,2]. Unfortunately, experimenting on Hanabi is not possible for this current paper due to computational constraints, but we agree that this game is very interesting for NAHT purposes.

There are two barriers to the reviewer’s suggested strategy of using explicit cross-play minimization techniques to generate opponents. First, existing cross-play minimization techniques such as BrDIV [3] are designed for the two-player AHT scenario and do not directly apply to the N-player scenario, and such an extension would merit its own paper.

Second, explicitly training agents that are bad for our agent violates a key assumption of AHT — that the uncontrolled teammates are acting in good faith / have some minimal level of competency.

Out-of-Distribution Evaluation - Agent Team Generation

We do not mean to claim that the OOD experiment would represent performance when paired with arbitrary conventions or against specific classes of interest, such as human opponents, and will modify the language in the paper to clarify the limited scope of the OOD experiment. These goals remain open challenges for the AHT, ZSC, and human-AI coordination communities. The goal of our work is to introduce the NAHT problem and a viable approach to solving it.

We also acknowledge that the teammate generation method suggested by the reviewer would accentuate the generalization challenge more than the teammates we have used in the OOD generalization experiments. However, the matched-mismatched experiment (Fig. 11) and cross-play tables (Tables 12-15) clearly demonstrate the generalization challenge. Specifically, the fact that performance decreases when we pair agents with another team trained under a different seed/algorithm implies none of the uncontrolled teams perform optimally against all other teams in evaluation. This emphasizes the need for adaptation and generalization when dealing with previously unseen uncontrolled agents. Note that other papers [2, 4] have also used the teammate generation method we propose to train agents that can robustly deal with a wide range of teammates.

Question - Agent Modeling for Few-Shot Learning

As POAM was not trained under the framework of allowing multiple episodes to learn about a new set of teammates, we do not expect that it would perform well under this setting. We agree that this would be an interesting contribution for few-shot AHT, but this is out of the scope of our current paper.

Question - NAHT methods for AHT Performance

Theoretically, an optimal NAHT agent should perform optimally for N=1, which is the AHT scenario. We confirm our hypothesis empirically in the new bit matrix results. Please see Section 1 and Table 1 of the rebuttal pdf for a discussion.

Question - Lower Probability for Correct Action as Training Progresses

POAM’s encoder-decoder (ED) models both controlled and uncontrolled agents.

Since the controlled agents are updated during the training process, this creates a moving target for the encoder-decoder, thus making the problem of modeling the controlled agents both more challenging and noisy. The observation that the overall action probabilities are lower at 19m than at 15m is largely due to noise in the modeling of the controlled agents.

Fig. 4 in the rebuttal pdf shows the probability of predicting the correct action for the uncontrolled agents and controlled agents separately.Note that the action probabilities shown in Fig. 4 of the original paper would be the average of the two plots shown in the aforementioned Fig. 4.

For uncontrolled agents (Fig. 4, left) we observe that the accuracy of action predictions for the uncontrolled agents increases much more consistently as training goes on, and is higher than that for the controlled agents, indicating that the ED is able to model the uncontrolled agents more easily. From these plots, we can also see that the observed sudden decrease in action probabilities from 15m to 19m occurs for the controlled agents only.

References:

[1] Hu et al. Off-belief learning. ICML 2021.

[2] Hu & Foerster. Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning. ICLR 2020.

[3] Rahman et al. Generating teammates for training robust ad hoc teamwork agents via best-response diversity. TMLR 2023.

[4] Strouse et al. Collaborating with humans without human data. NeurIPS 2021.

We apologize for the misunderstandings, and thank the reviewer for clarifying their questions. We would like to address the two issues raised in the follow-up.

Issue 1:

We reiterate that we believe that the drop in task score from matched to mismatched conditions means that the current OOD teammates does still pose a generalization challenge, even if that challenge is not major. We will clarify the limited scope of our experiment in the paper.

However, we agree that the reviewer’s proposed modification to the OOD experiment would be an interesting direction to deepen the analysis of the OOD properties of POAM. We currently follow the reviewer's advice by splitting the MARL training algorithms for generating teammates into two sets: agents generated with the algorithms, QMIX, MAPPO and IQL, will be used for training, whereas agents generated with the algorithms, VDN and IPPO, will be used for evaluation. We hope to present results on this experiment before the rebuttal period ends. However, if time doesn’t permit and the paper is accepted, we will add the results to the Appendix.

Issue 2:

We apologize for our mistake in referencing the common rebuttal. R 9xa8 raised the same question as this reviewer, inquiring why it is valid to use off-policy data to update the value function. Our reply was actually in the specific response to R 9xa8 (copied below). We are glad that R 9xa8 found our argument convincing.

While the data from the uncontrolled agents is not “on-policy” with respect to controlled agents, here are some reasons why using off-policy data to update POAM’s critic network is useful.

Useful cooperative behavior can be learned more quickly by bootstrapping based on transitions from the initially more competent, uncontrolled teammate policies (Section 5.6 of [1]). Early in training, value function learning based on the competent, uncontrolled agents’ data leads to controlled agents’ high appraisal of all uncontrolled agents’ decisions leading towards high returns. Controlled agents then learn to adopt similar decisions after using the learned value function for policy updates. Also, [2] demonstrated the improved data efficiency and stability in single-agent RL from using off-policy data to update the critic in an otherwise on-policy algorithm."

References:

[1] Rahman et al. A general learning framework for open ad hoc teamwork using graph-based policy learning. JMLR 2023.

[2] O'Donoghue et al. Combining policy gradient and Q-learning. ICLR 2017.

Points addressed in common rebuttal

• Additional agent modeling baselines and suggested references
• Number of test domains

On Off-policy data for training PPO critic

(R 9xa8) The decision to use data from uncontrolled agents is not well justified in the paper. The value function is also "on policy" in PPO, so it is unclear why this additional data is helpful in NAHT training.

While the data from the uncontrolled agents is not “on-policy” with respect to controlled agents, here are some reasons why using this particular off-policy data to update POAM’s critic network is useful.

Useful cooperative behavior can be learned more quickly by bootstrapping based on transitions from the initially more competent, uncontrolled teammate policies (Section 5.6 of [1]). Early in training, value function learning based on the competent, uncontrolled agents’ data leads to controlled agents’ high appraisal of all uncontrolled agents’ decisions leading towards high returns. Controlled agents then learn to adopt similar decisions after using the learned value function for policy updates. Also, [2] demonstrated the improved data efficiency and stability in single-agent RL from using off-policy data to update the critic in an otherwise on-policy algorithm.

Insights on Out-of-Distribution Performance

It is difficult to pinpoint causal factors that lead to the difference in performance in these domains. While they are broadly used to test coordination and cooperation, there are multiple factors that could influence the difficulty of NAHT in these domains. Nevertheless, we can look at some of the results in these domains and make educated guesses.

For example, Fig. 11 in the Appendix shows the return of teams trained together (matched seeds) versus those of teams that were not trained together (mismatched seeds), across all naive MARL algorithms. We observed that for the three domains where there is a larger difference in performance between matched and mismatched seed teams (5v6, 3s5z, MPE-PP), there is also a larger difference in performance between POAM and IPPO-NAHT in the OOD experiments. This perhaps indicates that coordination is highly sensitive in those domains, so agent modeling makes a larger difference.

Clarity - Improved Figures

Thank you for the suggestion. We will add smoothing to all figures

Questions - Addressing Cooperation With >2 Agent Teams

Addressing the question about the observation space first: POAM does not require the observation space to include any team ID information. Instead, the goal of POAM’s agent modeling module is to infer this information.

In theory, applying POAM to the scenario of Team A + Team B + Team C would be as simple as extending the training/evaluation framework to sample from two uncontrolled teams, rather than one. In practice, we anticipate credit assignment problems. For example, if the uncontrolled teams B and C do not interact well together, the overall team performance might drop greatly, but this would not be the fault of the controlled team A.

Addressing the scenario of blending multiple teams would be an interesting direction for future work.

References

[1] Rahman et al. A general learning framework for open ad hoc teamwork using graph-based policy learning. JMLR 2023.

[2] O'Donoghue et al. Combining policy gradient and Q-learning. ICLR 2017.

The maximum theoretical value for the action probabilities is not 1.0 when the modeled teammates are stochastic

To illustrate this, we construct a toy example based off of the bit matrix game that is described in Section 4 (The Need for Dedicated NAHT Algorithms). For each agent, the action space of the bit matrix game is {$0, 1$}. Let $p(A)$ denote the true action distribution of the uncontrolled agent, and let $q(A; \theta)$ denote the modeled action distribution, parametrized by $\theta$.

The quantity displayed in Fig. 4 is the average probability of the observed, ground-truth actions under the current agent model: $E_{a \sim p(A)} [ q(A = a; \theta)])]$.

Suppose the uncontrolled agent selects 1 with probability ⅓, and 0 otherwise, i.e. $p(A) = 1$ w.p. $⅓$. We call this the Bernoulli(⅓) agent. Assume that the modeled action distribution $q(A, \theta)$ exactly equals the ground truth agent policy, $p = q$, to compute the maximum possible value of the action probabilities.

Then $E_{a \sim p(A)} [ q(A = a; \theta)] = E_{a \sim p(A)} [ p(A = a)] = p(A = 1) q(A=1; \theta) + p(A = 0) q (A = 0; \theta) = ⅓ * ⅓ + ⅔ * ⅔ = 5/9 \approx 0.555$.

While this is a toy example, it serves to illustrate the point that if the ground-truth agent policies are stochastic, the maximum action probability displayed in Fig. 4 (right) will not be 1.0.

The action probability discussed previously is very closely related to the action modeling loss that the encoder-decoder is trained with. We chose to display the action probability rather than the action modeling loss because we thought it would be more interpretable for readers, but if the reviewer thinks it would be clearer to display the action modeling loss, we are happy to make that modification.

POAM’s encoder-decoder achieves the minimum action loss on the bit matrix game.

The action loss is given by the formula $L(\theta) = -E_{a \sim p(A)}[\log q(A=a; \theta)]$. Using the same notation and set up as above, for the Bernoulli(⅓) uncontrolled agent, the minimal action loss can be computed by setting $p=q$:

$L(\theta) = - p(A=1) \log q(A=1; \theta) - p(A=0) \log q(A=0) = -⅓ \log ⅓ - ⅔ \log ⅔ = 0.6365$ (using $\ln$).

Scale of Observation MSE in Fig. 4

Fig. 4 shows the observation MSE over an entire episode across multiple training checkpoints, from 0 to 19 million timesteps. Later in training, the ED requires a few time-steps of history in order to predict observations with low error. The initial MSE tends to be large early in training and early in the episode, as pointed out by the reviewer, but by the end of training and the end of the episode, we verify that it has reduced to the following values for predator prey and 5v6:

• MPE-PP: 0.041
• 5v6: 9.011e-07

For MPE-PP, we verified that we can further reduce the observation MSE by increasing the epochs of encoder-decoder updates at the beginning of training, although this does not improve performance on primary metrics.

To verify that POAM’s ED can indeed achieve the theoretical minimum observation and action loss in practice, we train the encoder-decoder on data generated by the Bernoulli(⅓) agent only (Fig. 2 in the rebuttal pdf).

Points Addressed in Common Rebuttal

• Only one evaluation domain
• Question about on/off-policy algorithms for POAM
• Requested baselines: in-distribution performance for Fig. 6; performance of uncontrolled teams

Accuracy of Inferred Actions and Observations in the Dec-POMDP setting

We agree that the agent model should experience a drop in accuracy from operating in partially observable environments, due to limited information about teammates. However, the agent model does not need to be perfect to be useful for NAHT; it only needs to be able to characterize the uncontrolled agents sufficiently s.t. POAM agents may best respond to the uncontrolled agents.

Experiments demonstrate that compared to IPPO-NAHT (equivalent to POAM without agent modeling), POAM has improved sample efficiency, asymptotic performance (Fig. 3), and out-of-distribution generalization to unseen teammates (Fig. 10) in the partially observable SMAC tasks.

Agent Modelling Network Performance in Fig. 4

The reviewer thinks that this performance [0.6] does not indicate that the [agent] model has been successfully trained.

We provide three counter-arguments:

(1) The maximum theoretical value for the action probabilities is not 1.0, as the maximum depends on the stochasticity of the modeled policies. For example, if an agent turns left 80% of the time and right 20%, even if we perfectly model the policy, then the expected modeled action probability is $0.8^2 + 0.2^2 = 0.68$. A follow-up comment will make a formal argument. Further, partial observability might preclude achieving the maximum theoretical value.

(2) We have included loss plots of the encoder-decoder in Fig. 1 of the rebuttal PDF to show that the observation and action losses reduce smoothly during training.

(3) We provide additional empirical results on the bit matrix game showing that in a fully observable setting the encoder-decoder reduces both observation and action losses to the theoretical minimum value (Fig. 2 in rebuttal pdf). Derivation of the minimum action loss in follow-up comment.

Clarifications/Typos

We thank the reviewer for pointing out typos and clarity issues, and will implement all corrections in the paper.

History Length - Line 163

POAM uses a recurrent encoder-decoder, where the hidden state is only reset at the start of the episode. The lowercase $t$ in the expression $h^t=(o_i^k,a_i^{k-1} )_{k=1}^t$ refers to the current timestep of the episode, which varies. Thus, POAM deals naturally with changing history lengths.

Un- vs Non-controlled Agent - Line 316

Non-controlled agents are the same as uncontrolled agents; we will make the terminology consistent.

Common Reward Function

A common reward means that all the agents in a team get the same reward at every time step. This follows the standard Dec-POMDP setup, as specified in Section 2.

…do the controlled agents learn a single [encoder decoder] network and then use it in a distributed manner?

Controlled agents learn a single encoder-decoder and use it in a distributed fashion, a practice called parameter sharing [1]. This was mentioned in Line 394 of the original paper: “POAM, which employs full parameter sharing…”

Questions

Why is observation decoding loss based on mean squared error (MSE) and action decoding loss based on likelihood?

The MSE loss is equivalent to the negative log-likelihood loss (NLL) under a Gaussian distribution, and is the standard loss for modeling continuous features (as we have in our tasks). Thus, the observation decoding loss is a unified methodology with the action decoding NLL loss, which uses a Categorical distribution for the discrete action space.

How to calculate probability p?

We noticed that Eq. 2 is missing a summation over the controlled agent index $i$, for $i \in C$, which may have reduced readability. We will fix this typo.

$p$ is the probability of observing the actions of agents $-i$, where the probability distribution is parameterized by the output of the decoder. In practice, since the experimental tasks have discrete actions, $p$ is the Categorical distribution over the action space (corresponding to the ground-truth Categorical agent policies).

In the pre-training of uncontrolled agents, what cooperative decisions do they learn? For example, if three uncontrolled agents are needed for a task in which five agents form a team, how can we pre-train only the three agents separately?

The uncontrolled agents in our paper are derived from running MARL algorithms. Thus, they are trained within their own team, under whatever assumptions that MARL algorithm makes. The partial uncontrolled team is not trained separately; it is simply a subset of the full uncontrolled team that was trained to work together.

What is the range of observation and action? …the MSE in Fig. 4 appears to have a very large error rate.

The obs. range was normalized to [-1,1] for the experiments, while the discrete action space is one-hot encoded. The observation MSE decreases over training and over the episode, and can be further decreased by increasing agent model training epochs. We will post a further discussion of the observation MSE in Fig. 4.

In Fig. 5, how to train agent modeling network in POAM without UCD?

Training without UCD means that the encoder-decoder’s training dataset is limited to observations corresponding to agents from the controlled set. Importantly, agents from the controlled set may still observe uncontrolled agents, making it possible to predict observations and actions corresponding to uncontrolled agents.

What is the difference between POAM without a modeling network and IPPO?

POAM w/o modeling network is equivalent to IPPO-NAHT. The difference between IPPO-NAHT and IPPO is the use of uncontrolled agent data to train the value function.

References:

[1] Christianos et al. Scaling multi-agent reinforcement learning with selective parameter sharing. ICML 2021.

Thank you for reviewing our rebuttal with great care, and for the thoughtful follow-up. We address the questions point-by-point below:

Q1:

In Figure 2, the author uses theoretical probability, but the correlation with the loss is not clear.

The reviewer mentions Fig. 2 in their question, but we are not sure which Fig. 2 they refer to. Fig. 2 in the submitted paper is a diagram illustrating POAM, while Fig. 2 in the rebuttal PDF shows the encoder-decoder (ED) observation and action loss values, rather than action probabilities. We proceed on the assumption that the reviewer is asking for further clarification of the relationship between the action probability and action loss, but are happy to continue discussing this topic if we have misunderstood.

The action loss is the expected negative log likelihood loss, as shown in Eq. 2 of the submitted paper: $L(\theta) = -E_{a \sim p(A)}[\log q(A=a; \theta)]$ On the other hand, as described in our comment above, the action probability displayed in Fig. 4 of the submitted paper is the average probability of the observed, ground-truth actions under the current agent model: $E_{a \sim p(A)} [ q(A = a; \theta)]$ Thus, the only differences between the two quantities is the $\pm$ sign and the log function within the action loss. Conceptually, we wish to minimize the action loss, and maximize the action probability. The solution set for minimizing the action loss is the same as maximizing the action probability due to the monotonic log function, which does not change the ordering of solutions.

To give a numerical example comparing the action probability and action loss, please consider the bit matrix game example described in our follow-up comment titled, “The maximum theoretical value for the action probabilities is not 1.0 when the modeled teammates are stochastic”. Recall that $A$ is the action random variable that takes values 0 or 1; $p(A)$ denotes the true action distribution of the uncontrolled agent; and $q(A; \theta)$ denotes the modeled action distribution, parametrized by $\theta$.

The uncontrolled Bernoulli(⅓) agent chooses A=1 with a probability of ⅓, and A=0 with a probability of ⅔ . Given this uncontrolled agent policy, a perfect decoder network would generate a $q(A; \theta)$ distribution that also chooses A=1 with a probability of ⅓, and A=0 with a probability of ⅔, i.e. $p = q$.

As computed in our comment titled, “POAM’s encoder-decoder achieves the minimum action loss on the bit matrix game”, the perfect decoder (i.e. when $p=q$) would achieve an expected action loss of $- p(A=1) \log q(A=1; \theta) - p(A=0) \log q(A=0) = -⅓ \log ⅓ - ⅔ \log ⅔ \approx 0.6365$ (using $\ln$) for the Bernoulli(1/3) agent. On the other hand, this perfect decoder would achieve an expected action probability of $p(A=1) * q(A=1) + p(A=0) * q(A=0) = ⅓ * ⅓ + ⅔ * ⅔ = 5/9 \approx 0.5556$.

Q2

In Figure 3, the magnitude of the error bars for all algorithms appears to be the same. Please check if this result is correct.

We thank the reviewer for pointing out this error, which was due to a variable name error in the plotting code. Unfortunately, we are not able to attach a corrected figure due to the discussion rules, but the corrected confidence bound values for Figure 3 in the rebuttal pdf are listed below, and will be corrected in the next draft of the paper. Importantly, we also verified that the significance of our results and corresponding analysis did not change due to this error.

Group names, left to right: IPPO, IQL, MAPPO, QMIX, VDN:

IPPO-NAHT: 0.699, 0.491, 0.532, 1.189, 0.685 POAM: 1.719, 1.696, 1.402, 2.108, 0.709

(continued in part 2 below)

Q3

The reviewer believes the authors should provide more details regarding the MPE-PP setting. Specifically, the reviewer wonders how a pre-trained prey agent can be trained.

We use the pre-trained prey policy provided by the ePymarl MARL framework, as mentioned in the Appendix at line 643. The prey policy was originally trained by Papoudakis et al. [1] by using the MADDPG MARL algorithm to train both predator and prey agents for 25M steps . We visualized the prey policy and confirmed that the prey agent moves to escape approaching predators.

Please see the code here for the exact parameter file, titled prey_params.pt. Since the prey policy is pre-trained and fixed, our predator-prey task is a fully cooperative task from the perspective of the predator (learning) agents. We will improve the existing explanation of the MPE environment and predator-prey task to the Appendix by adding this discussion.

Q4

The author did not provide baseline performance data related to uncontrolled agents. Without providing the performance in a scenario where all agents are uncontrolled, it is difficult to evaluate the effectiveness of the proposed algorithm.

We already addressed this point in the common rebuttal section titled ”Baseline Selection (R 5dYr, R 9xa8)”. We wish to emphasize that we did both (1) evaluate the performance of the uncontrolled agent teams, where all agents are uncontrolled, and (2) assess the performance of the uncontrolled agents in the NAHT evaluation setting, and use it to contextualize POAM, the proposed algorithm.

To summarize, we provided three analyses of the uncontrolled agent performance in the submitted version of the paper:

Matched-mismatched evaluation: Section A.4.1. Fig. 11 shows the performance of each uncontrolled team on all tasks when the uncontrolled team is trained together (matched seed condition), versus when we mix two teams that were trained using the same algorithm but different seeds (mismatched seed condition). Cross-play tables: Tables 12-15 display the full cross-play results for all uncontrolled teams (generated by MARL algorithms) on all tasks. Naive MARL baseline: the naive MARL baseline in Fig. 3 is the performance of the best uncontrolled team, as evaluated through the cross-play scores shown in the cross-play tables mentioned above. The baseline shows how well naive MARL agents that were not trained in the NAHT setting, would perform when evaluated in the NAHT setting.

Again we thank the reviewer for their time and consideration , and are happy to answer follow-up questions.

References

[1] Papoudakis et al. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. Neurips 2021.

Scale of Revisions Required for Paper

It is not completely apparent to us why the revisions discussed here would be substantial enough to merit rejecting the paper. The planned revisions to the main paper are largely minor, and the revisions to the Appendix consist of further experimental details and supplemental results and discussion, drawn from the current rebuttal/discussion on OpenReview. Please see the list of revisions that would be made, based on our discussion with the reviewer (5dYr).

Appendix

• Discussion: relationship between action probabilities and action loss, as explained in our discussion with R 5dYr
• Discussion: theoretical lower bound on action loss / upper bound on action probabilities (as explained in discussion with R 5dYr)
• Discussion: why the observation loss is the MSE loss, while the action decoding loss is the negative log likelihood loss (as explained in the discussion with R 5dYr)
• Supplemental result: plot of encoder-decoder loss (Fig. 1 of rebuttal PDF)
• Experimental detail: how the pre-trained prey agent is trained on MPE-PP (as explained in the discussion with R 5dYr)
• Experimental detail: the observation/action ranges in MPE-PP and StarCraft tasks

Main Paper

• Typos: modelling -> modeling, non-controlled -> uncontrolled, adding a summation over the controlled agent index $i$ for $i \in C$ to Eq. 2
• Correction: correcting the plotting of the confidence intervals for the OOD experiment plots (Fig. 6)
• Additional baseline: in-distribution performance of uncontrolled teams as an additional set of baselines for Fig. 6 (already implemented as Fig. 3 of the rebuttal pdf).

Summary of Successfully Clarified Issues

The following is a list of all questions/concerns that were previously raised by the reviewer, and successfully clarified by our response. Again, given that we have addressed all issues brought up by the reviewer, we respectfully request the reviewer to reconsider their position on the paper.

Misconceptions

• Only one evaluation domain
• Lack of evaluation of uncontrolled agents

Validity Concerns

• Validity of inferring actions/observations of other agents in a POMDP setting
• Whether the agent model has been successfully trained
• Magnitude of error bars is the same in Fig. 3 <- we acknowledge the error. However, follow-up analysis shows that the - significance of results is not affected.

Requested Additional Analysis

• Requested baseline of the in-distribution performance for Fig. 6

Clarification Questions

• Meaning of the term, “common reward function”
• Why the observation decoding loss is based on the mean squared error while the action decoding loss is based on the likelihood
• How to compute probability p
• Whether POAM can deal with changing history lengths
• Difference between POAM without modeling network and IPPO
• Relationship between action probability and action loss

Requested Further Experimental Details*

• How uncontrolled agents are trained
• How pre-trained prey agents were obtained in MPE-PP
• Range of observation and actions
• How the agent modeling network can be trained without uncontrolled agent data

Questions About Extensions to Method

• Question about whether POAM could be implemented with an off-policy algorithm

References

[1] Yu et al., The surprising effectiveness of PPO in cooperative multi-agent games. NeurIPS 2022.

[2] Papoudakis et al., Agent modelling under partial observability for deep reinforcement learning. NeurIPS 2021.