Human-like Communication Strategies for Improved Multi-Agent Reinforcement Learning

1. This paper is not well-written, there are lots of typos. For example, centralized -> decentralized in line 128, need -> needs on line 202, is can be -> can be on line 242, would want to -> would like to on line 239, etc.
2. As for line 162 that "For example, switching the observations of two agents, so each one observes the other’s position instead of its own (but controls its own movements)," I have some different viewpoints. Is it still valid when the reward function is a invariant to observations or agents' observations are symmetric? In either case, we can still always find a relation between reward function and observations.
3. About line 193-195 that "In the general vector case, it is natural to view the observation as a collection of feature-sets, each one corresponding with a textual term that represent an object," why can this form of observation representation can enhance identification of the relevancy of an object?
4. As for line 199-200 that "For image observations, it is possible to use existing pretrained object detection models to extract the mask and additional features," even if the proposed text-to-mask is good representation, the inaccuracy brought up by the pretrained feature extractor can still lead to some unexpected errors. In turn, the embedding is still uninterpreted.
5. As for line 208-210 that "As many other works ... implements both agents," why do you choose the decoupled policies for selecting actions and communications, respectively? If there is no justifiable reasons, this setup would only result in more parameters.
6. As for line 237-239 that "For $\pi^{comm}$ ... with the receiving agent," why is this reasonable? Is there any special considerations behind this design?
7. As for line 243-244 that "Then at inference ... messages only," is this a logical reason to explain why encoder network is disregarded? Could you give more clarification about this?
8. As for line 289-292 that "We propose ... demonstrations," how is the rule-based policy or human demonstrations be guaranteed to find for every task? Also, the use of BC contradicts the claim in line 65-67 that "Another appraoch would be ... which is hard to collect."

The proposed framework mainly integrates existing methods such as text-to-mask models and PPO, offering limited original theoretical contributions.

The experiments are conducted in relatively simple environments, limiting the framework’s demonstrated applicability to more complex real-world MARL scenarios.

The analogy to human communication is weak, as the proposed strategies focus on rule-based data exchange rather than capturing the depth of real human interactions.

The paper overlooks a deeper exploration of communication policy instability during training, which could pose challenges in dynamic environments.

While the related work is discussed, the experimental section lacks comprehensive benchmarks against state-of-the-art communication strategies, making it difficult to assess the framework’s comparative advantage.

• The consideration of ‘complex’ problems in the introduction is the same as the cheap talk discovery problem (e.g., the phone booth maze) described in [1]. Please cite it
• The approach could be better motivated in the introduction by being more specific about which aspects of human-like communication are being considered here
• With only 2 agents with 2 relatively simple environments, the method's effectiveness could be better supported. It should also be compared against other MARL communication baselines like AEComm (Lin et al., 2021).

[1] Lo, Y. L., de Witt, C. S., Sokota, S., Foerster, J. N., & Whiteson, S. Cheap Talk Discovery and Utilization in Multi-Agent Reinforcement Learning. In The Eleventh International Conference on Learning Representations.

• The paper is not very well written. There are several instances of grammatical mistakes, informal wording and poor sentence formation. Some examples include:

  • Line 58: "it could be beneficial assimilating to human beings,..." (assimilating could be replaced by a more appropriate word)
  • Line 87: "problems is both interesting and challenging, thus gained some attentions..." (informal language)

  Fixing these errors and sentence formation issues would improve the paper significantly

• The introduction of the paper does not do a very good job at motivating the approach. Elaborating upon human communication strategies such as prior knowledge, contextual understanding and efficient information exchange with specific examples of how they are useful in human context and how they could be useful in the context of multi-agent RL would help motivate the approach better. Section 4.1 does attempt to do the above, partially, however, improving its sentence formation and incorporating it in the Introduction would strengthen the introduction.

• The approach is not compared against standard MARL baselines such as IPPO [1], MAPPO [2], QMix [3],

• The experiments are performed on rather toyish and non-standard environments. Validating the results on standard environments (for eg. environment from OpenAI's Multi Particle Environment suite) and environments with higher dimensional observation spaces (such as Starcraft) would strengthen the paper significantly.

• The authors do not provide any hyperparameter details.

References:

[1] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
[2] Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?, 2020.
[3] Tabish Rashid, Mikayel Samvelyan, C. S. D. Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. ArXiv, abs/1803.11485, 2018