Reconstruction-Guided Policy: Enhancing Decision-Making through Agent-Wise State Consistency

Q1: The paper has multiple grammatical errors and spelling mistakes.

R1: Thank you for your suggestion. We have made corrections to address this issue.

Q2: The illustrative example in Lines 71 - 80 is difficult to follow. I think the authors are trying to say that agent $x_1$ incorrectly assumes that agent $x_0$’s coverage is represented by the 2nd and 3rd elements in the state vector instead of the correct 3rd and 4th elements. Is this what the authors mean here? Since this is an example used to illustrate the necessity of the method, I think it can be explained more clearly.

R2: Thank you for your suggestion. This is not what we mean here. We have made adjustments in the introduction to make it more clearly. For more details, please refer to the Introduction of the manuscript.

Q3: While it is beneficial that the authors benchmark on SMACv2 instead of only on SMAC, it is still quite a limitation in the work that most of the conclusions regarding the method are drawn from results on SMAC. This together with only 3 continuous action space benchmarking tasks is a rather limited evaluation set and not very extensive as the authors mention in the paper.

R3: Thank you for your suggestion and positive feedback. First of all, We would like to clarify that

• (a): our conclusions are not solely drawn from results on SMAC, but rather reached it by comprehensively considering the results from both SMAC and SMACv2. For example, in Section 5.3, our original text is: "we compare the retention ratio of RGP with the PTDE and SIDiff, which also reconstruct the state for decision making, on the environment of SMAC and SMACv2, and illustrate the results in Table 2." And we observed that regardless of whether it was on SMAC or SMACv2 maps, our method consistently maintained higher performance retention ratios compared to PTDE and SIDiff. Therefore, we concluded that our method effectively improves performance retention ratios.

• (b): For the evaluation in continuous action space, existing works, such as [1,2,3], conducted experiments only in discrete spaces. In contrast, we performed experiments in both continuous and discrete spaces. To further address your concern, we conducted additional experiments on the zerg_90_20_vs_20 and zerg_360_epo maps of SMACv2. The zerg_90_20_vs_20 involves 20 agents battling against 20 enemies under the zerg_90 configuration, where agents have a 90-degree field of view. The zerg_360_epo introduces random masking of observations for each agent, under the zerg_360 configuration, where agents have a 360-degree field of view. The experimental results are presented in the table below. The results show that our method (RGP) outperforms the baselines in both environments. This demonstrates the effectiveness of RGP in a broader range of environment settings. The corresponding content has been updated in Appendix C.4 of the manuscript.

[1] Chen, Yiqun, et al. "Ptde: Personalized training with distillated execution for multi-agent reinforcement learning." the Thirty-Third International Joint Conference on Artificial Intelligence.

[2] Xu, Zhiwei, et al. "SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning." Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.

[3] Zhou, Yihe, et al. "Is centralized training with decentralized execution framework centralized enough for MARL?." arXiv preprint arXiv:2305.17352 (2023).

Q4: When there is improvement over a baseline it is slight and oftentimes within one measure of error of other methods. For example, in Table 1 RGP+HPN overlaps with the results of HPN-QMIX on all scenarios and RGP overlaps with QMIX, VDN or QPLEX on all scenarios.

R4: Thank you for your feedback. Our method achieved relatively good performance, for example, in Table 1, the win rate of RGP is 73.1±4.6, while QMIX achieves only 38.1±28.3 in 6h_vs_8z. Besides, as shown in Table 3 , under the Protoss_90 setting, the results show that Qmix achieved 27.6±8.2, while RGP achieved 37.2±6.5, representing a performance improvement of 9.6.

Q5: In Table 3, the error is not taken into account when computing the Performance Improving Ratio of one method over another. This sketches an incomplete picture since it only uses the mean. For a more statistically sound comparison, error propagation should be incorporated into these calculations using standard deviation propagation techniques. Similarly, for the Performance Retention Ratio of one method over another in Table 2.

R5: Thank you for your valuable suggestion. We have updated the corresponding data in the paper.

Thank you for your thoughtful consideration and for raising your score. We're glad that our revisions addressed your concerns. Your support means a lot to us!

Q1: The comparison with the related work can be improved: The paper ignores significant works based on agent (opponent) / state modelling, world models, transformer-based methods which also aim to deal with partial observability. For instance, the authors argue that one main challenge is state inconsistency. In particular, many works (e.g. [1]) leverage state embeddings (which can be much more compressed and informative) instead of using the reconstructed state (which can be less informative for each agent's training, also due to redundant information, see [2]). Therefore, although I believe that the proposed method is interesting, the state inconsistency challenge has partially been addressed by related work, and thus the authors should better highlight their contributions in the context of related work.

R1: Thank you for your valuable suggestion. The other methods, when addressing state inconsistency issues, either overlook environmental information [1,3] or introduce compounded errors [4]. Our approach effectively avoids these problems. We have revised our related work, in which we highlighted the difference between our paper and the previous works [1,2]. Please refer to the manuscript for more details.

[1] Papoudakis, Georgios, Filippos Christianos, and Stefano Albrecht. "Agent modelling under partial observability for deep reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 19210-19222.

[2] Guan, Cong, et al. "Efficient multi-agent communication via self-supervised information aggregation." Advances in Neural Information Processing Systems 35 (2022): 1020-1033.

[3] Jing Sun, Shuo Chen, Cong Zhang, Yining Ma, and Jie Zhang. Decision-making with speculative opponent models. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[4] Qihan Liu, Jianing Ye, Xiaoteng Ma, Jun Yang, Bin Liang, and Chongjie Zhang. Efficient multi-agent reinforcement learning by planning. arXiv preprint arXiv:2405.11778, 2024.

Q2: One concern is that the baselines regarding partial observability are not very established in MARL community. Some of them have not even been published at a top-tier AI/ML venue. To further improve the paper, I would suggest that the authors also include well-known methods (as baselines) based on alternative ways to deal with partial observability.

R2: Thank you for your suggestion. We conducted experiments in SMACv2 with three random seeds and compared our method with ATM[5]. The table below shows the mean and standard deviation. In the table, protoss_360 represents the setting where agents have a 360-degree field of view in the protoss scenario. We can observed that our results consistently outperform the baseline, demonstrating the effectiveness of our approach. The corresponding training curves have been incorporated into the manuscript.

[5] Yang, Yaodong, et al. "Transformer-based working memory for multiagent reinforcement learning with action parsing." Advances in Neural Information Processing Systems 35 (2022): 34874-34886.

Q3: The attached repo does not contain any files.

R3: Thank you for your feedback. We have checked it and updated the anonymous repository.

Q4: Minor weakness: The authors do not include information about hyperparameter details of the baselines.

R4: Thank you for your valuable suggestion. In Section 5.1, Implementation Details, we mentioned that all baseline methods were tuned using the hyperparameters from PyMARL2 [6]. Details about the hyperparameters of the baselines have been added to Appendix A2.

[6] Hu, Jian, et al. "Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning." arXiv preprint arXiv:2102.03479 (2021).

Thank you for reviewing our paper and for sharing your concerns. Below are our responses to address them.

Q10: I still believe that [1] and [2] have already somewhat captured the state-inconsistency problem (motivation). I suppose that in the example of the obstacle that the authors mentioned, each agent should have incorporated such information into their policy network, if some other agent had noticed the obstacle and also if that obstacle was important for the task.

R10: We agree with your opinion that [1] and [2] have captured the issue of state-inconsistency to some extent.

However, compared to the state reconstruction-based method we proposed, they focus on the local observations and actions of each agent, without explicitly considering the importance of environmental information.

As you mentioned, the information about the obstacle is recorded by the agent in the form of observation history $h_t$​. However, in [1], the recurrent network is not utilized, and therefore, there is no observation history. The method relies solely on the current observation $o_i^t$​ of agent $i$ to predict the actions of other agents, thereby overlooking potential environmental information. In [2], the gradients of the decision network are not backpropagated to the recurrent encoder. In other words, the observation history in the recurrent encoder is only used to reconstruct the current observations and actions of other agents. This limitation prevents the effective capture of comprehensive environmental information, potentially causing the agent to make suboptimal decisions.

In contrast, our method explicitly incorporates the reconstruction of global information (agent-wise state) through the observation and observation history. This approach allows us to more effectively capture complete environmental information, thereby facilitating more efficient decision-making.

Therefore, we believe that our improvements to the global state reconstruction method are both valuable and necessary.

[1] Jing Sun, Shuo Chen, Cong Zhang, Yining Ma, and Jie Zhang. Decision-making with speculative opponent models. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[2] Papoudakis, Georgios, Filippos Christianos, and Stefano Albrecht. "Agent modelling under partial observability for deep reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 19210-19222.

Once again, thank you for your thoughtful review, as well as for taking the time to consider our additional results and visualizations. We appreciate your valuable feedback and the effort you have invested in evaluating our work. We sincerely hope that our responses have adequately addressed your concerns. Your support and insights are highly valuable and mean a great deal to us.

Q3: The theoretical foundation for using diffusion models over other generative models could be further clarified, as comparable methods like VAE and MLP produced similar results.

R3: Thank you for your feedback. Our original intent of applying diffusion is to generate the agent-wise state, rather than refine diffusion. Therefore, diffusion just serves as one module in our method. Theoretically speaking, any generative model, including VAE, GAN, or even simple MLP, can replace the diffusion in our method. We choose diffusion here just because it is particularly well-suited for modeling complex data distributions and has demonstrated superior performance in tasks across CV, NLP, and decision-making. We assume the underlying reason for VAE and MLP achieving comparable performance with diffusion may stem from our replacement of the U-Net architecture in the original DDPM paper [9] with an MLP, due to computational resource limitations. Nonetheless, as shown in Figure 7, diffusion still outperforms VAE, MLP, and GAN in most cases.

[9] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.

Q4: Misleading use of letters/symbols, unclear annotations on some images, poor writing and reading experience.

R4: Thank you for your comment. We have corrected the issues you pointed out in the manuscript, clarifying the image representations and standardizing the use of letters and symbols.

Q5: The paper uses $\epsilon$ repeatedly across equations, specifically in Eq (3) and Eq (6). Could you clarify why the same $\epsilon$ notation is applied in both contexts? Given that Eq (3) refers to a noise variable for the diffusion process while Eq (6) applies to the exploration parameter, distinguishing between these variables would improve clarity and reduce confusion.

R5: Thank you for your suggestion. We have revised the manuscript and represent the noise involved in the diffusion process as $\zeta$, while the exploration parameter for the greedy algorithm is denoted as $\epsilon$.

Q6: Figure 2 is challenging to interpret with respect to the network design for RGP. Could you provide a more detailed explanation of how each component in Figure 2 maps to the decision and guidance modules described in Sections 4.1 and 4.2? Specifically, illustrating the data flow within and between these modules and explaining how key operations like the generative model, RNN, and attention mechanisms are integrated would be very helpful.

R6: Thank you for your feedback. We have updated the legend to Figure 2 and provided a more detailed explanation. The integration of key operations is as follows.

The decision module is active during both training and execution. At each timestep, the local observations $o_t^i$ and reconstructed trajectory states $h_{t-1}^i$ (computed from the previous time step t-1) are input into the diffusion model to generate the reconstructed agent-wise state $s_t^i$. Subsequently, the reconstructed agent-wise state $s_t^i$, local observations $o_t^i$, and reconstructed trajectory states $h_{t-1}^i$ are processed by RNN and MLP to estimate the reconstructed action value $Q_i$ and the reconstructed trajectory states $h_{t}^i$. The action value $Q_i$ is used to decide which action the agent takes. Besides, all reconstructed action values are optimized using a mixing network via TD loss.

Considering that the diffusion-based generation process lacks ground truth for training, we design a guidance module, which is active only during training. Attention mechanisms in the guidance module take local observations $o_t^i$ and the decomposed global state $\bar{\boldsymbol{S}}_t$ as inputs, producing agent-wise states $\bar{s}_t^i$, serving as ground truth for generation. Afterward, the agent-wise state $\bar{s}_t^i$, local observations $o_t^i$, and trajectory states $\bar{h}_t^i$ are used by the RNN to estimate action values. Similarly, all action values $\bar{Q}_i$ are optimized using the same mixing network. This approach helps us obtain high-quality agent-wise states, which in turn guide the reconstruction process of diffusion.

Q1: Limited exploration of potential biases introduced by the generative model in state reconstruction, particularly in complex environments.

R1: Thank you for your suggestion. In response, we analyzed the bias between the reconstructed states and the ground truth of the state (hereafter referred to as the true state) on the zerg_90 map of SMACv2, where 90 indicates that the agents have a 90-degree field of view. We calculated the cosine similarity between the reconstructed state and the true state, with the results illustrated in Figure 8 of Appendix C.3. For more details, please refer to the manuscript. From the figure, it can be observed that the state reconstructed by our method is significantly closer to the true state compared to the baseline. Furthermore, the similarity between the state reconstructed by our method and the true state consistently exceeds 0.9, demonstrating that our method introduces minimal bias during reconstruction.

Q2: The approach's scalability to more agents and highly dynamic environments is not thoroughly evaluated.

R2: Thank you for your feedback. Compared to most previous works [1,2,3,4,5,6,7], our work was evaluated in environments with more agents and highly dynamic environments. Most existing studies[1,2,3,4,5,6,7] use the SMAC environment to conduct experiments, involving less than 20 agents. In contrast, as Table 1 illustrates, we evaluated our method on the SMAC map 27m_vs_30m, which features the largest number of agents (involving 27 agents) in SMAC. Moreover, we also conducted experiments in SMACv2, which introduces additional high dynamics compared to SMAC.

To further address your concern, we conducted additional experiments on the zerg_90_20_vs_20 and zerg_360_epo maps of SMACv2. The zerg_90_20_vs_20 involves 20 agents battling against 20 enemies under the zerg_90 configuration, where agents have a 90-degree field of view. The zerg_360_epo introduces random masking of observations for each agent, under the zerg_360 configuration, where agents have a 360-degree field of view. The experimental results are presented in the table below. The results show that our method (RGP) outperforms the baselines in both environments. This demonstrates RGP's scalability to environments with more agents and high dynamics. The corresponding content has been updated in Appendix C.4 of the manuscript. And please refer to SMACv2[8] for more details about zerg_360_epo.

[1] Chen, Yiqun, et al. "Ptde: Personalized training with distillated execution for multi-agent reinforcement learning." the Thirty-Third International Joint Conference on Artificial Intelligence.

[2] Xu, Zhiwei, et al. "SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning." Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.

[3] Yang, Yaodong, et al. "Transformer-based working memory for multiagent reinforcement learning with action parsing." Advances in Neural Information Processing Systems 35 (2022): 34874-34886.

[4] Wang, Jiangxing, Deheng Ye, and Zongqing Lu. "More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization." The Eleventh International Conference on Learning Representations.

[5] Zhou, Yihe, et al. "Is centralized training with decentralized execution framework centralized enough for MARL?." arXiv preprint arXiv:2305.17352 (2023).

[6] Qihan Liu, Jianing Ye, Xiaoteng Ma, Jun Yang, Bin Liang, and Chongjie Zhang. Efficient multi-agent reinforcement learning by planning. arXiv preprint arXiv:2405.11778, 2024.

[7] Jing Sun, Shuo Chen, Cong Zhang, Yining Ma, and Jie Zhang. Decision-making with speculative opponent models. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[8] Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan,Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.

In addressing Q3, the authors attempt to justify their choice of diffusion models by citing their theoretical advantages and superior performance in various tasks. However, this defense is severely undermined by a lack of clarity and consistency. The discrepancy between the reviewers' observations of similar performance across VAE, MLP, and diffusion models versus the authors' claim that diffusion outperforms the others in most cases remains inadequately addressed. The explanation regarding the replacement of the U-Net architecture with an MLP due to computational constraints is superficial and fails to convincingly demonstrate why diffusion models should still be preferred. Furthermore, the theoretical justification specific to generating agent-wise states is vague, leaving readers questioning the true merits of the chosen approach. The authors also neglect to provide a comprehensive comparison of computational efficiencies or scalability between diffusion models and the alternatives, which is crucial for validating their choice. This omission not only weakens their argument but also leaves a significant gap in understanding the practical implications of their methodology.

Additionally, the authors neglect to address the broader concern about the overall poor writing and reading experience, focusing solely on symbols and annotations in their response to Q4. This oversight suggests that more extensive revisions are necessary to enhance the manuscript’s language and structure, which the authors have failed to adequately convey.

The response to Q7 is equally disappointing. Although the authors present additional experiments to demonstrate robustness, they fail to adequately contextualize the significance of the win rates reported. The improvement in win rates for RGP over QMIX is presented with standard deviations, yet there is no discussion on whether these improvements are statistically significant. The authors' response to Q7 raises concerns regarding the robustness evaluation of their learned policies (RGP) in dynamic multi-agent environments. Specifically, the implementation of a 50% probability that other agents will never observe an enemy after one agent has observed it introduces substantial ambiguity and potential issues that undermine the validity of their robustness claims. In Q9, the authors attempt to explain the high variability observed in baseline methods like SIDiff and QPLEX, but their explanations remain speculative and lack supporting data or references. They attribute the high variance of SIDiff to inconsistencies between training and execution states and attribute QPLEX's high variance to its dual dueling mixing network expanding the exploration space. However, these explanations are not backed by empirical evidence or citations, making them appear unfounded. The selection of high-variance baselines is justified superficially, without adequately addressing how this impacts the reliability of their comparative assessments. This approach raises concerns about the validity of the comparative results presented, as the high variability could mask the true performance differences between methods.

Upon examining the authors' response to Q10, it becomes clear that significant deficiencies remain unaddressed, further justifying the paper's marginal standing below the acceptance threshold. The response begins by deflecting the scalability inquiry back to Q2, which is not provided, leaving readers without a complete understanding of how scalability has been previously addressed. This omission demonstrates a lack of thoroughness and fails to engage directly with the reviewer’s concern about the proposed method's applicability to larger multi-agent environments. Furthermore, the authors assert that increasing the number of agents does not result in a rise in the number of model parameters due to their centralized training and distributed execution framework. While parameter sharing is a valid strategy for managing scalability, the response lacks empirical evidence or detailed analysis to substantiate the claim that computational demands remain manageable. The assertion that computational demands grow linearly with the number of agents is stated without supporting data, making it difficult to assess the practicality and efficiency of the proposed approach in real-world, large-scale scenarios.

In summary, the paper suffers from inadequate presentation quality and numerous missed details that significantly detract from its overall impact and clarity. The authors' responses to the reviewers' comments are often vague, lacking the necessary specificity and depth to address the concerns raised convincingly. I firmly think this paper is below the acceptance threshold and will keep the score.

Q15: The response to Q10 begins by deflecting the scalability inquiry back to Q2, which is not provided, leaving readers without a complete understanding of how scalability has been previously addressed.

R15: Thank you for your response. First, we would like to clarify that our RGP focuses on addressing the partial observability issue of agents, rather than the computational complexity faced by a large number of agents. Given the specific target of our method, we believe that not evaluating RGP in scenario involving a large number of agents does not significantly impact the validity of our approach.

Next, for the scalability to a scenario with more agents, as we discussed in R2, most existing studies[2,6,7,8,9,10,11] use the SMAC environment to conduct experiments, involving less than 20 agents. In contrast, as Table 1 illustrates, we evaluated our method on the SMAC map 27m_vs_30m, which features the largest number of agents (involving 27 agents) in SMAC. We also conducted additional experiments on the zerg_90_20_vs_20 maps of SMACv2. The zerg_90_20_vs_20 involves 20 agents battling against 20 enemies under the zerg_90 configuration, where agents have a 90-degree field of view. Compared to other methods, we have conducted experiments in environments with a larger number of agents.

[2] Chen, Yiqun, et al. "Ptde: Personalized training with distillated execution for multi-agent reinforcement learning." the Thirty-Third International Joint Conference on Artificial Intelligence.

[6] Xu, Zhiwei, et al. "SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning." Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.

[7] Yang, Yaodong, et al. "Transformer-based working memory for multiagent reinforcement learning with action parsing." Advances in Neural Information Processing Systems 35 (2022): 34874-34886.

[8] Wang, Jiangxing, Deheng Ye, and Zongqing Lu. "More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization." The Eleventh International Conference on Learning Representations.

[9] Zhou, Yihe, et al. "Is centralized training with decentralized execution framework centralized enough for MARL?." arXiv preprint arXiv:2305.17352 (2023).

[10] Qihan Liu, Jianing Ye, Xiaoteng Ma, Jun Yang, Bin Liang, and Chongjie Zhang. Efficient multi-agent reinforcement learning by planning. arXiv preprint arXiv:2405.11778, 2024.

[11] Jing Sun, Shuo Chen, Cong Zhang, Yining Ma, and Jie Zhang. Decision-making with speculative opponent models. IEEE Transactions on Neural Networks and Learning Systems, 2024.

Q16: While parameter sharing is a valid strategy for managing scalability, the response lacks empirical evidence or detailed analysis to substantiate the claim that computational demands remain manageable. The assertion that computational demands grow linearly with the number of agents is stated without supporting data.

R16: Thank you for your response.Since we use a homogeneous agent architecture where all agents share the same network, the number of agents can increase without increasing the network parameters. On the other hand, we employ a centralized training and distributed execution framework, where each agent's decision is made independently. Additionally, we only added modules in the agents to predict the agent-wise state without changing the parameters of the mixing network. As a result, the computational increase caused by diffusion is linear as the number of agents grows.

We also conducted tests in the Predator-Prey environment, measuring the memory consumption when using 3, 6, and 9 agents. The results are shown in the table below. From the table, we can observe that as the number of agents increases, the GPU memory consumption grows linearly.

Q17: "Additionally, the authors neglect to address the broader concern about the overall poor writing and reading experience, focusing solely on symbols and annotations in their response to Q4. This oversight suggests that more extensive revisions are necessary to enhance the manuscript's language and structure, which the authors have failed to adequately convey. "

R17: We have conducted a thorough review and revision of the manuscript. In addition to the changes to the symbols and annotations you suggested, we have also provided further clarification in several descriptions. For example, to improve readability and understanding, we revised the legend in Figure 2 and adjusted its description to better reflect our main idea. We have also revised the description of the decision module in lines 234 to 240 to make it more fluent. We have made further revisions regarding tenses and spelling.

Q13: Although the authors present additional experiments to demonstrate robustness, they fail to adequately contextualize the significance of the win rates reported. The improvement in win rates for RGP over QMIX is presented with standard deviations, yet there is no discussion on whether these improvements are statistically significant.

R13: To further demonstrate the effectiveness of our method from a statistical perspective, we performed a t-test on the results obtained by RGP and QMIX based on R7. The results are shown in the table below, where "Zerg" indicates that we trained on zerg_360 and executed on zerg_360_epo, and "Protoss" indicates that we trained on protoss_360 and executed on protoss_360_epo. We can observe that the p-values obtained for both settings are below 0.05, which statistically indicates that our method shows a significant improvement over the baseline methods. This demonstrates the robustness of our approach.

Q14: In Q9, the authors attempt to explain the high variability observed in baseline methods like SIDiff and QPLEX, but their explanations remain speculative and lack supporting data or references. This approach raises concerns about the validity of the comparative results presented, as the high variability could mask the true performance differences between methods.

R14: Thank you for your feedback. We would like to further clarify：

(1) The choice of baselines is reasonable. The widely accepted criteria for selecting baselines are popularity, effectiveness, and relevance. QPLEX is commonly used as a baseline in multi-agent reinforcement learning papers [2,3] (popularity) and performs well in some environments (effectiveness), such as protoss_360 and terran_360. On the other hand, SIDiff is highly relevant to our method (relevance) and was published in 2024. Therefore, we chose both of them as baselines.

(2) The implementation of the experiments is correct. The code of QPLEX we used is the publicly available pymarl2 [4], with hyperparameters set according to those provided in the original paper to ensure optimal performance. SIDiff does not have public code, and we replicated it based on the details in the paper, using the hyperparameters specified in the original work. Moreover, QPLEX also exhibits high variance in experimental results in other papers (e.g., [2,3]). SIDiff, a more recent work, has not yet been widely cited, but the experimental results reported in the SIDiff paper [5] also show considerable variance in certain environments (e.g., Figures 3 and 4 in the SIDiff paper). Additionally, it should also be noted that while both methods show high variance in some maps, they do not exhibit high variance across all maps. Therefore, the high variance in both baselines likely arises from the limitations of the models in specific environments.

(3) The analysis is of appropriate depth. We have made some preliminary speculations on the reasons for the high variance of these two methods in certain environments (in R9). However, since our approach is not built upon these two models, nor is the focus of this paper on surveying the performance of various benchmarks, conducting further investigations, such as conducting experiments to explore the deep reasons behind the high variance of QPLEX and SIDiff, falls outside the scope of our work.

(4) The high variance of QPLEX and SIDiff in certain environments does not affect the evaluation of RGP's performance. We can observe that in environments with high variance, neither QPLEX nor SIDiff serves as the best-performing baseline. Therefore, the high variance exhibited by these two models in their respective environments does not impact the evaluation of RGP's experimental performance.

[2] Chen Y, Mao H, Zhang T, et al. Ptde: Personalized training with distillated execution for multi-agent reinforcement learning[J]. IJCAI2024

[3] Huang, Wenhan, et al. "Multiagent q-learning with sub-team coordination." Advances in Neural Information Processing Systems 35 (2022): 29427-29439.

[4] Hu, Jian, et al. "Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning." arXiv preprint arXiv:2102.03479 (2021).

[5] Xu, Zhiwei, et al. "Beyond local views: Global state inference with diffusion models for cooperative multi-agent reinforcement learning." arXiv preprint arXiv:2408.09501 (2024).

Thank you for sharing your concerns with us. We also greatly appreciate the time and effort you have dedicated to reviewing our response. Regarding the points you raised, we believe there may have been a slight misunderstanding of our method, and we would like to clarify:

Q11: "The discrepancy between the reviewers' observations of similar performance across VAE, MLP, and diffusion models versus the authors' claim that diffusion outperforms the others in most cases remains inadequately addressed. The explanation regarding the replacement of the U-Net architecture with an MLP due to computational constraints is superficial and fails to convincingly demonstrate why diffusion models should still be preferred."

R11: Thank you for your comment. We would like to kindly clarify that diffusion is not the preferred, but rather optional. Diffusion is not an indispensable part of RGP; rather, it is the generative model that is truly essential. The core of RGP lies in solving the partial observability problem through agent-wise state reconstruction. Therefore, the key contribution of RGP is the framework for reconstructing agent-wise states, not the specific generative model used within the framework (since generative models are the work of other researchers, and are not proposed by RGP). As we mentioned in R3, any generative model, even an MLP, can be used for agent-wise state reconstruction in RGP. We simply selected a typical and representative generative model as the tool to implement state reconstruction, but this does not mean that diffusion is an indispensable component of RGP. Hence, which specific generative model performs best is not the primary focus of this paper, what is crucial is whether the RGP framework is effective. Therefore, we did not include the results of adapting different generative models to RGP in the main text of the manuscript, but rather placed them in the appendix Figure 7.

The purpose of including Figure 7 in the appendix is to further explore which generative model adapts better to RGP, providing additional insights for readers when selecting a generative model for deploying RGP. The fact that the performance of MLP and VAE in Figure 7 is comparable to that of diffusion does not mean RGP is ineffective. On the contrary, the fact that it performs well even in the MLP setting highlights the effectiveness of the RGP framework, indicating that good performance can still be achieved even with the simplest generator configuration.

Q12: "The implementation of a 50% probability that other agents will never observe an enemy after one agent has observed it introduces substantial ambiguity and potential issues that undermine the validity of their robustness claims."

R12: Thank you for your feedback. The setting of a 50% probability that other agents will never observe an enemy after one agent has observed it is not our own implementation, but rather part of the official SMACv2 [1] setup. Specifically, during an episode, if an enemy is observed for the first time by one agent, that agent can continue observing the enemy as usual. However, other agents have a 50% probability of never being able to observe that enemy during the rest of the episode. We consider this scenario a potential state change you mentioned, as the probability of agents observing the enemy introduces uncertainty and variability in their local states, depending on their prior observations. Due to this limited observation, agents may miss certain enemies and be unable to coordinate focused attacks with teammates, making them more vulnerable to enemy aggression. This significantly increases the challenge of the environment.

The 50% probability can be replaced with any value, and the official documentation suggests that the lower the probability, the harder the task becomes. When the probability is set to 50%, models struggle to achieve modest performance according to the official document. Therefore, to show robustness and increase randomness, we chose to set the probability at 50%. We hope this addresses your concern.

[1] Ellis, Benjamin, et al. "Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

Q1: Figures need a bit more clarity: What do the shaded regions in Fig 5 represent? How many runs were used for Figure 5? Depending on the number of seeds and the claim being made, this has to be specified as context to the reader.

R1: Thank you for your comment. Following the settings of [1, 2, 3], we conducted the experiments shown in Figure 5 using three random seeds. Thick, dark-colored lines denote the mean, and the shaded areas represent standard deviations. We have clarified this explanation in the manuscript.

[1] Liu, Qihan, et al. "Efficient Multi-agent Reinforcement Learning by Planning." The Twelfth International Conference on Learning Representations.

[2] Ellis, Benjamin, et al. "Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

[3] Chen, Yiqun, et al. "Ptde: Personalized training with distillated execution for multi-agent reinforcement learning." the Thirty-Third International Joint Conference on Artificial Intelligence.

Q2: The paper lacks a limitations section. Besides the one limitation regarding smaller fields of view, it would be nice to see what other challenges to do with partial observability in MARL would need to be addressed next, and if there are any other shortcomings of RPG. Certain parts of the work speculated reasons about why results were seen (such as the impressive score of RGP+FACMAC in Predator-Prey). This could be summarized at the end of the conclusions section.

R2: Thank you for your valuable suggestions on our work. We have added the summary of the experiments and limitations in the section of conclusion.

Q3: [Very Minor] Line 483: typo “continuous”.

R3: Thanks for your comment! We have fixed this issue in our paper.

Q4: What is the uncertainty measure in Table 1? Is it a standard error?

R4: Thank you for your comment. Yes, you are absolutely right, the data in Table 1 represents the mean with standard deviation from three random seeds. We have added the description in Table 1 .