Inverse Attention Agents for Multi-Agent Systems

We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.

The method is currently only tested in relatively simple MPE environments

MPE environments emphasize different interaction types: competitive, cooperative, and mixed, fostering diverse agent dynamics. In future work, we plan to apply our methods to other environments, such as StarCraft. In terms of agent number, we already conduct a task with 8 agents in a 4x4 grassland setting. Additionally, we extended our analysis to a 10x10 grassland test, with the results presented below.

As the number of agents increases, the computational complexity of inferring attention weights for multiple agents may become a bottleneck. The paper does not fully address how the method scales with a larger number of agents or more diverse agent types.

The inverse-att network will scale up in the benefit of the attention structure. We present the results for 20 agents in the Grassland environment with a 10-10 scale. The Inverse-Att agents not only scale effectively but also outperform other ablations and baseline methods.

The method assumes that agents are of the same type, which limits its generalization.

We do not assume that all agents are of the same type. For instance, in a competitive environment like Adversary, there are two distinct types of agents: sheep and wolves. Since inverse attention employs a goal-focused (GF) representation, this mechanism can be applied to all types of teammates, regardless of their roles.

The inverse attention mechanism is trained in an offline manner, which may limit its adaptability in a rapidly changing environment.

Inverse attention is initially trained offline to abstract the relationships between agents' observations and actions. However, an online training phase is introduced later (as described in the second while loop of Algorithm 1), where MARL is applied to adapt more effectively to other agents and dynamic environments. Experimental results in Tables 1–5 demonstrate that, even when interacting with various unseen agents or human players, inverse attention consistently outperforms all other baseline methods.

Does the performance of inverse attention agents degrade as the number of agents increases, particularly in larger environments like the 4-4 Grassland game?

The 4-4 and 10-10 results show that inverse-att agents outperform the other baselines with a large margin which could scale up as the agent number increases.

Why does the cooperative gain between multiple inverse attention agents not increase linearly?

Yes, we believe cognitive loops or mutual inference inaccuracies indeed plays a role in the absence of linear reward increase.

Why do inverse attention agents perform inconsistently when cooperating with humans in some roles (e.g., sheep in Adversary), and how can this cooperation be made more robust?

We did not observe any inconsistencies in the behavior of Inverse-Att agents, including the sheep in the Adversary environment. When comparing them to other methods, Inverse-Att agents consistently achieve much higher rewards with relatively low variance compared to all other baselines. Could you clarify what you mean by "not robust"?

How is the accuracy of the inverse attention network related to task success, and why doesn't high inference accuracy always correlate with better performance in tasks like Adversary?

The higher accuracy of the inverse attention network allows agents to achieve better performance by reducing the noise in their input. This advantage applies across all tasks, including competitive scenarios like Adversary. For instance, if Agent A knows that teammates B and C are moving left to avoid a wolf, it might decide to go right, avoiding the crowd and reducing its risk of being caught.

Performance Difference Between Inverse Attention Agents and Baseline Agents

At the network structure level, Inverse Attention Agents utilize the attention mechanism, while baseline methods (e.g., MAPPO, IPPO, MAA2C, ToM2C) rely on MLPs. MLPs are input-sensitive, learning fixed mappings between specific input configurations and outputs, which makes them less effective in scenarios with dynamic teammates or opponents. In contrast, attention mechanisms are input-invariant, dynamically computing relevance across entities without depending on fixed mappings. At the methodological level, the Inverse Attention approach models agents by inferring their goals. By integrating this inferred information with environmental data, it constructs a more comprehensive state representation, leading to improved action decisions.

Inverse Attention Performance in Competitive Tasks and Cooperative Tasks

We have no evidence showing that the inverse attention mechanism performs better in competitive tasks than in cooperative ones. Our experiment results show that Inverse Attention Agents outperform baselines by a margin in every cross competitions, demonstrating its superior performance in both competitive and cooperative tasks.

We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.

It is not clear to me and was not discussed in the text how the agent observation can be decomposed into a combination of goals.

This is discussed in Section 3.2, and in Supplementary A.3, where we provide a detailed example of how to construct the dataset for goal-focused (GF) training. The GFs are treated as goals, allowing agents to directly act based on these goals.

The authors collect the attention inference dataset but do not provide an analysis of how different realizations of such attention weights collections may influence the resulting inverse-attention agent performance

We collect attention inference dataset from part of the Self_Att agent's training trajectory. All Self_Att agents are trained with the same hyperparameters, and we believe the current training trajectory is sufficient to cover majority cases, so we do not believe that changing random seeds would be necessary. Additionally, the number of agents is fixed, so we do not need to train inverse attention networks with different number of attention weights.

The authors provide no ablation study.

Inverse-Att features two novel modules: the gradient field and the inverse attention network. Self-Att is an ablation of the inverse attention network, included as a baseline in the main paper. As shown in experiment sections, it consistently outperforms all other baseline models.

The ablation study for Inverse-Att without the gradient field is extensively discussed in [1]. Additionally, we conducted experiments in the Spread 3-agents game, Adversary $3$-$3$ game, and Grassland $3$-$3$ game, replacing the gradient field with relative positions for Inverse-Att agents only. This variant is referred to as Inverse-Att-Vanilla.

In our evaluation, Inverse-Att-Vanilla competed against the same set of baseline agents: MAPPO, IPPO, MAA2C, ToM2C, and Self-Att. For Inverse-Att-Vanilla agents, the input consisted solely of raw environmental observations, while all other agents retained gradient field (GF) inputs. The results of these experiments are presented in the updated Appendix, which details the performance of models competing with Inverse-Att-Vanilla. Furthermore, Table 13 in the Appendix compares the performance of Inverse-Att-Vanilla and Inverse-Att against the same set of agents in identical agent pools. The findings demonstrate that while Inverse-Att delivers strong performance compared to other baselines, it outperforms all other agents in most cases. This highlights the critical importance of incorporating GF representations into the observation space.

Ablation for GFs

Comparison between Inverse-Att-Vanila and Inverse-Att agents

The reference of the "Attention is all you need" paper has an incorrect year of publication.

Thank you for pointing out this oversight. We have revised our paper to correctly cite it.

The font sizes on figures 2 and 3 should be increased to improve readability

Thank you for your suggestion, we have increased the font size and updated the figures in the paper.

[1] SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning

The MPE environments are very simple and similar to each other. The scalability test only tries at most 4 agents which is small.

MPE environments emphasize different interaction types: competitive, cooperative, and mixed, fostering diverse agent dynamics. In future work, we plan to apply our methods to other environments, such as StarCraft. In terms of agent number, we already conduct a task with 8 agents in a 4x4 grassland setting. Additionally, we extended our analysis to a 10x10 grassland test.

In many environments, there are no explicit multiple goals, and how to decompose an observation into $N$ goals is unclear.

Goals are not needed. We follow [1] to use gradient fields, and we treat Goal-Focused Features (GFs) as proxies for goals.

[1] SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning

We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.

The proposed inverse attention network requires the observation of agent $j$ when inferring the attention weights of agent $j$

This approach can be extended to partial observations by applying inverse attention only to agents within the observable range. Since the inverse attention network leverages the attention mechanism, it is more robust to a dynamic number of inputs. We demonstrate the impact of partial observation by testing different visibility radii in all tested games (spread, adversary, grassland, navigation, tag): 1.5, 1, and 0.5. Across all three settings, agents using inverse attention consistently achieve the best performance.

Radius=1.5

Radius=1

Radius=0.5

When training the inverse attention network, this work only uses the agent i’s own data.

This setting is more challenging and realistic, as in most real-world environments like card games or hunting, it is impossible to access the states of other agents. Agents can only rely on their own observations, making it equally unfeasible to determine the exact goals of other agents. In this context, inverse attention provides a viable approach to modeling other agents through their inferred goals, offering a more generalized method for representing their behaviors.

This work does not compare with the existing works from the ad hoc teamwork domain.

The Tom2C agent is from the adhoc domain, which belongs to the “agent modeling” category according to https://arxiv.org/abs/2202.10450.

We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.

Scalability Analysis

We present the results for 20 agents in the Grassland environment with a 10-10 scale. The Inverse-Att agents not only scale effectively but also outperform other ablations and baseline methods.

Am I correct in assuming that as part of the IW networks training, each independent agent should have encountered those states for it to make an accurate prediction of the other agent's attention weights.

This is an excellent question, as current methods do not explicitly model other agents. Instead, we rely on inverse attention mechanisms to represent other agents. In this context, the "state" refers to a higher-level abstraction that encompasses both "other agents" and their representations, which are framed as inverted goals. This approach has the potential to generalize to unseen states of previously unobserved agents, much like how multi-agent reinforcement learning (MARL) can, to some extent, generalize to unseen physical states.