Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration

We sincerely appreciate your time and input. We respond to your comments and questions below.

1. The motivation of the proposed method is not clearly stated.

We respectfully disagree with the reviewer's comment. In Lines 28-36 (right), we clearly state our main motivation for this paper: "we are interested in settings where agents lack explicit communication channels during execution. Such settings are of particular interest, because, while communication-based methods leverage inexpensive simulators for training, they may incur substantial computational overhead when executed in real-world environments". Moreover, in the intro (Lines 43 right– 64 left), we provide a detailed discussion of significant drawbacks and problematic assumptions of existing agent modelling (AM) methods. Remarkably, many of these claims (e.g., redundant state information, AM without exploration and policy optimization, single-agent AM) are empirically validated by our results and the extensive ablation study, demonstrating their detrimental impact on MARL performance. These challenges highlight the need for a more principled approach to AM in MARL and further motivate our approach.

2. Real contributions of the paper

Due to space constraints, please see our response to Question 7. in the rebuttal of reviewer CpzR.

3. This paper assumes that states contain information redundant for optimizing agents’ individual policies. Why?

This is an established assumption which has been validated by the NeurIPS paper ([48]-th reference in our paper) and the AAMAS paper [1], supported both by the curse of dimensionality and by intuitive reasoning. Specifically, in large and complex state spaces, each agent naturally prioritizes information that is more relevant or proximal to them while assigning lower importance to remote or less goal-related information.

4. What does the “adversarial targets” mean? Do you mean the target entity in the actual environment?

The term "targets" has been used many times before the adversarial exploration section (e.g., see lines 155 right, 214 left, 175 right, 181 right, 185 right, 187 right, 195 right) in the paper to denote the reconstruction targets of the encoder-decoder.

5. This paper designs an adversarial exploration method to “discover novel, high-value states while improving the discriminative abilities of others”. But the count-based intrinsic reward only can drive agents to visit novel states. Such an intrinsic reward could not serve the purpose of encouraging the exploration of high-value states.

Thanks for this comment. By the means of adversarial exploration, each agent is motivated to reach novel states, while she challenges the state modelling abilities of other agents. By doing so, our exploration schema aims at improving the joint belief about the joint state, and thus the joint policy, as the policy network of each agent uses her belief as extra input. Therefore, our method is more likely to reach novel, high-value states than the original which hashes observations.

6. The citation format is incorrect. In many figures, the legends obscure the main figures. Figure 1 could be presented in a better way.

Thanks for your comments. We will fix these minor typos in the camera-ready version. Regarding Fig. 1, we have thoroughly explain its components in high-level in Lines 147 (right) - 169 (left).

7. What do the adversarial exploration and discriminative abilities mean?

Thanks for this question. By discriminative abilities of an agent under partial observability we mean the ability of the agent to identify meaningful information about the unobserved global state based on her local information. In other words, discriminative abilities represent how well the agent models the global state. Our approach defines the meaningful state information within the state modelling framework (see lines 129 left - 158 left).

Adversarial exploration is extensively explained and motivated in Lines 246 (left) - 267 (left).

8. The overall presentation of the paper lacks clarity

We kindly disagree with the above comment that there are major presentation issues about the paper. Reviewer hakk finds that "the paper was clear and easy to follow, which made understanding the ideas very smooth and engaging". Some of the points you found unclear are thoroughly explained earlier in the paper, so we believe there is a chance that you missed these explanations and this rendered the points, our motivation and contributions unclear. If this is the case, we hope through a more careful reading to appreciate our ideas and results more. Otherwise, if you insist that there are specific points which require better explanation from our end, please let us know to fix them.

9. Scalability with more agents

Thanks for this question. See our response to reviewer hakk on question 1.

[1] Li et al. From explicit communication to tacit cooperation: A novel paradigm for cooperative MARL. 2024

1. Motivation and Discussion about Assumptions of related work

We respectfully disagree with the reviewer's comment. In the introduction (Lines 43–64), we provide a detailed discussion of significant drawbacks and problematic assumptions of existing agent modelling (AM) methods. Remarkably, many of these claims (e.g., redundant state information, AM without exploration and policy optimization, single-agent AM) are empirically validated by our results and the extensive ablation study, demonstrating their detrimental impact on MARL performance. These challenges highlight the need for a more principled approach to AM in MARL. Regarding Lines 59–64, we believe that the assumptions outlined — namely, a priori knowledge of state features, centralized execution, and a focus solely on team games — are quite restrictive. These constraints limit the practicality of such algorithms, making them less applicable to a wide range of real-world scenarios.

2. Assumption: z_i contains meaningful information about the unobserved state, informative for optimizing the policy.

Assuming that there exists a latent belief space which contains unobserved state information meaningful for optimizing the policy has been very standard in AM (e.g., see [40, 42, 44]). We note that this is the main assumption in order for AM to be applicable under partial observability.

3. It rules out information that may be beneficial to decision making, but not acquired due to the defined beliefs.

The reviewer must have misunderstood that according to our definition if a state feature is beneficial to decision making, then the framework considers this feature to be "informative" (see first bullet, line 139). Such a feature could be identified by w_i, and thus z_i, being conditioned on w_i, is informative of this feature during exploration. Also, our definition implies that if a feature cannot be inferred through z_i and it is not relevant to policy optimization, then this feature is "not-informative".

4. Comment on "even when utilizing a compressed embedding"

Thanks for this comment. It is known from [9] that providing full state information as an additional input to the policy can degrade performance due to redundant, non-informative state features. In our work, we demonstrate that this issue can persist even when using a compressed embedding z_i, as shown in Fig. 5 (left), where applying filters leads to improved performance. We will ensure that this detail is clarified in the final version.

5. Even though the weight converges to including partial non-informative, it can still find a local optimum by policy. More importantly, this suboptimality is difficult to be verified.

Due to space constraints, please see our response to Question 8. in the rebuttal of reviewer CpzR.

6. Difference from LIAM, MAVEN and [3]

Due to space constraints, please see our response to Question 10. in the rebuttal of reviewer W3Cu.

7. Multiple components: difficult to understand from the theoretical perspective

We believe that all loss components have been well-explained in Sec. 3 and are well-validated by a plethora of experiments (see Fig. 5, 7, 15 - 26). More specifically, from a theoretical standpoint we give detailed intuition of each component:

1. L_rec: 172 left - 182 right
2. L_wcritic: 213 left - 218 left
3. intrinsic reward: Sec. 3.2
4. L_norm: 196 left - 200 left
5. L_KL: 202 left - 207 left

Our method does not rely on "tricks" but on conceptual components (1–3), motivated by how we think of the ideal MARL behavior against partial observability (see Lines 77-92 (left) and 98-104 (right)). The components 4–5 were introduced to ensure that the proposed method is theoretically sound (also illustrated in Fig. 7, 17) and adheres to our definition of state modeling (lines 152 & 205 left).

8. Not easy to track the main contribution of this paper

Due to space constraints, please see our response to Question 7. in the rebuttal of reviewer CpzR.

9. Comparison to MAVEN

Due to space constraints, please see our response to Question 6. in the rebuttal of reviewer CpzR.

10. Question about "intrinsically motivated...models"

Since each agent updates its policy independently, conditioned only on its own observations and beliefs, it is intrinsically motivated to seek novel observations o_i, thereby discovering novel z_i. However, since o_i serve as part of the target for other agents' decoders, they inadvertently increase the reconstruction loss of those agents.

11. Question about "To do so...z_i."

Since z_i is a function of only o_i, and agent i is motivated to reach novel z_i, then the only way for the agent to achieve this is by discovering novel o_i that lead to novel z_i. In Appendix E.4.7 we show how we effectively handle the dynamic nature of the reward.

[@] Papadopoulos et al. (AAMAS 2025) - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks.

We sincerely appreciate your time and input. Please see our responses below:

1. Scalability with more agents

Thanks for this question. Along with Spread-8 and LBF 7s-20x20-5p-3f (see Fig. 2, 3), we also add results on other large LBF tasks: namely 8s-25x25-8p-5f and 7s-30x30-7p-4f. We note that these tasks have been benchmarked by the AAMAS2025 paper [1]. Below, we show that SMPE outperforms all baselines in these tasks as well.

2. Computational overhead: SM and AM filters a bottleneck?

Thanks for this question. From the table below, we observe that even in larger settings, SM and AM filters only introduce a moderate (and completely justified) extra computational overhead, less than ATM, and far less than EOI, EMC and MASER.

3. Evaluated scenarios

Thanks for this comment. All LBF and RWARE tasks from [41] are not difficult to solve. More specifically, the RWARE tasks from [41] are not the -hard ones. Moreover, in all LBF tasks from [41] (most of them are cooperative-competitive, that is an easier version of the problem), our method converges fast to an optimal policy. This is the main reason we selected more challenging scenarios from these benchmarks.

Remarkably, [1] benchmarks all of our selected tasks and explicitly highlights as open challenges most of our LBF tasks along with MPE tasks with 5 or more agents, in all of which SMPE performs significantly better than all baselines.

4. Why was MAT not considered? SM in transformer-based architectures?

Thanks for this question. Since both ATM and MAT are transformer-based algorithms, we chose to include only one of them to ensure a more diverse set of baselines, rather than over-representing algorithms from the same direction of approaches. We selected ATM over MAT because MAT is not implemented in (E)PyMARL, whereas ATM and all other baselines we consider are. This ensures that all methods are evaluated under the same protocol. Additionally, MAT has already been benchmarked in the same environments (see [1]), where it generally underperforms compared to our baselines. Notably, the authors have not highlighted MAT as a "best" algorithm. Regarding if SMPE would fit a transformer-based architecture, we have not evaluated it yet and we have left this direction as a future work.

5. hyperparameter tuning, such as the number of trials, tuning methodology, whether tuning was done per scenario or per environment, and parameter sharing.

Thanks for this comment. We followed [41]: For hyperparameters, optimization was performed for each algorithm separately in each environment (not scenario). Then from each environment, we selected one task and optimised the hyperparameters of all algorithms in this task. We evaluated algorithms using 6 independent runs (seeds).

Except for MPE, where [41] (and also [1]) noted that parameter sharing was more detrimental, we used parameter sharing for all other tasks across all algorithms. All evaluated algorithms ran with the same configuration in parameter sharing. We selected the configuration of parameter sharing based on [1,41], and we did not run further experiments with independent/sharing policies.

6. Validity of MAPPO results

Thanks for this comment. We evaluated MAPPO using the EPyMARL code and adhered to the same parameters as in [41]. We are confident that all baselines were run correctly, and we can upload all log files to a GitHub repository after the camera-ready version. As the reviewer pointed out, our MAPPO results are on par with those reported in the AAMAS 2025 paper [1], where we verified that the algorithm and environment hyperparameters were the same. In contrast, [2] does not use the (E)PyMARL library, which could lead to significant differences in the evaluation protocol. Moreover, we find it peculiar that MAPPO manages to reach good performance far before 20M steps (e.g. in 2ag-tiny-hard, 4ag-small-hard). We also conducted additional experiments with different random seeds, but we did not observe the behavior described.

7. References Not Discussed and further suggestions

Thanks for this comment. We will make sure to incorporate all suggestions.

[1] Papadopoulos et al. (2025) - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks.

[2] Mahjoub et al. (2025) - Sable: A Performant, Efficient, and Scalable Sequence Model for MARL.

We sincerely appreciate your time and input, along with the positive evaluation. We respond to your comments and questions below.

1. A more detailed analysis in the style of rliable [42] could be more informative than some subfigures in Fig 3 and 4.

Thanks for this comment. We will consider using rliable in the camere-ready version.

2. Scalability with more agents

See our response in question 2. to reviewer W3Cu

3. Comment on missing related works

Thank you for this comment. Does the reviewer some specific reference in mind? We will be happy to add missing related works.

4. Figure 1 is not entirely clear and the arrow convention is not sufficient to distinguish the gradient and the data flow.

Thank you for this comment. Our overview of SMPE^2 and the arrow convention follows a visualization approach similar to that of many well-established MARL papers (see [43], [44], [46]). If the reviewer still has objections about the figure, we are open to discuss possible improvements.

5. Note that the authors mention the same work twice for no reason as two separate entries. Line 262 first column has a typo: “By doing do”. The Guan et al. paper is mentioned twice in the references. Line 1194 should be well-entagled instead

Thanks! We will fix these minor typos.

Due to space constraints, below we include responses to some of the questions of the other reviewers.

6. Comparison to MAVEN

We add experimental comparison with MAVEN on challenging RWARE and LBF scenarios. We use the mean values over 6 seeds. Our method significantly outperforms this method.

7. Real contributions of the paper

Regarding the main contributions, they have been extensively highlighted:

• in the introduction (Lines 94 left - 69 right)
• in a series of research questions (Q1-Q8) we address
• in important results (e.g., see lines 320-326 right: these tasks have been viewed as open challenges by [@])
• in section 3

Based on the above, the main technical contributions of our paper are the following: We propose the state modelling optimization framework, on top of which we propose the SMPE MARL method. Based on state modelling, each agent of SMPE aims at learning meaningful beliefs about the unobserved global state, from each own perspective, by trying to predict what the other agents observe, and use her belief in order to enhance her own individual policy. To ensure that the beliefs of each agent are informative to her decision making, the framework entails learning of the belief with respect to her policy optimization. Moreover, SMPE introduces the AM filters which aim to filter out redundant global state information which may be detrimental to the inference of the beliefs, and thus of the MARL performance. Our approach introduces two novel loss objectives for learning the beliefs and the AM filters of each agent: (a) a reconstruction loss for learning the AM filters via a self-supervised manner, which ensures that the AM filters identify global state features that can be inferred by the formed agents' belief of the agent, and (b) an RL loss that ensures that the inferred beliefs incorporate information relevant to her policy optimization through backpropagation on the AM filters. Additionally, our method further harnesses the rich state information captured by the agents' beliefs. Each agent uses a count-based intrinsic reward on her own agent belief. This simple exploration schema is proved to be of great importance for joint exploration under partial observability, by encouraging each agent to discover novel observations and also by helping other agents form better explored beliefs about the unobserved global state through the means of an interesting adversarial competition. Remarkably, our experiments validate each of the above conceptual component of our method, and also suggest that SMPE outperforms state-of-the-art methods in challenging, well-established benchmarks.

8. Even though the weight converges to including partial non-informative, it can still find a local optimum by policy. More importantly, this suboptimality is difficult to be verified.

Our method controls the contribution of L_rec and L_wcritic to the AM filters through the selection of the hyperparameters: $lr_{wcritic}$ (learning rate in L_wcritic) and $lr_{w}$ (learning rate in L_rec). In practice, we found that a good hyperparameter selection is to set $lr_{wcritic}$ 100 times smaller than $lr_{w}$ (see Table 3, appendix), so that policy optimization does not impede reconstruction. One way to verify if the weight is trained well w.r.t. policy is by looking at the cumulative reward. This is because w affects z and thus both the policy and exploration. In our results (e.g., see Fig. 5) we consider that w is near-optimal w.r.t. policy.

We sincerely appreciate your time and input, along with the positive evaluation. We respond to your comments and questions below.

1. A more detailed analysis in the style of rliable [42] could be more informative than some subfigures in Fig 3 and 4.

Thanks for this comment. We will consider using rliable in the camera-ready version.

2. Scalability with more agents

Regarding the scalability of our algorithm, Figure 2 shows that it clearly outperforms the other baselines in MPE Spread with 8 agents, albeit by a smaller margin compared to MPE Spread with 3, 4, and 5 agents. Furthermore, additional evaluations in LBF 8s-25x25-8p-5f and 7s-30x30-7p-4f (tasks have been benchmarked by the AAMAS2025 paper [1]) demonstrate that our algorithm achieves superior performance compared to the others.

3. Comment on missing related works

Thank you for this comment. Does the reviewer some specific reference in mind? We will be happy to add missing related works.

4. Figure 1 is not entirely clear and the arrow convention is not sufficient to distinguish the gradient and the data flow.

Thank you for this comment. Our overview of SMPE^2 and the arrow convention follows a visualization approach similar to that of many well-established MARL papers (see [43], [44], [46]). If the reviewer still has objections about the figure, we are open to discuss possible improvements.

5. Justification on how the baselines are selected, especially the ones that are fairly simpler than the proposed method.

Our set of baselines includes some of the most well-known and relevant algorithms to ours. The choice of simpler algorithms was intentional, as MAA2C serves as the backbone of our algorithm, making it a natural baseline. Additionally, while MAPPO is a generalization of the single-agent PPO algorithm, it achieves strong results in our benchmarks, as described in [47].

6. The choice of t_SNE seems somewhat arbitrary?

We kindly disagree with this comment. t-SNE is a well established method used widely in the MARL community for embedding visualization (see [43],[44],[45]).

7. Are all the timesteps pooled together for the classification or is the accuracy reported for each timestep separately?

Thanks for this comment. Indeed all the timesteps pooled together for the classification.

8. The timestep 45 makes sense but it can also come across as cherry-picked

Thanks for this comment. The timestep 45 was not cherry-picked but selected on purpose to be at the end of the trajectory to represent the belief of each agent closer to the solution of the task. Furthermore in the Appendix E.4.8 we provide similar embedding visualizations for the timestep 30.

9. Note that the authors mention the same work twice for no reason as two separate entries. Line 262 first column has a typo: “By doing do”. The Guan et al. paper is mentioned twice in the references. Line 1194 should be well-entagled instead

Thanks for your comments. We will fix these minor typos.

Due to space constraints, below we include responses to some of the questions of the other reviewers.

10. Difference from LIAM, MAVEN and [3]

We respectfully disagree with the reviewer's comment.

• Regarding LIAM, please see lines 669-674. Also, note that LIAM does not use AM for exploration which is a major contribution of our paper.

• Regarding MAVEN, some clear differences are the following: -- MAVEN does not account for non-informative state information which can be detrimental to MARL performance. -- Our method is not based on committed exploration but on an adversarial competition among the agents, guided by z_i and a count-based method. -- Our latent variables are not shared but are unique to each agent. -- Our z_i is used as input to each agent's policy, rather than a component of the joint action-value function.

• Regarding [3], first of all, we believe that it is irrelevant as it solely considers single-agent RL, and thus it does not have to deal with MARL challenges, including multi-agent decentralized execution and non-stationarity. Additionally, [3] employs an intrinsic reward based on the prediction error of embeddings of pixel-based states, aiming to identify novel pixels. In stark contrast, we use a count-based hashing method on the belief z_i, which is informative of the unobserved state and policy optimization, ultimately improving joint exploration.

[3] Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction". 2017.