Learning to Communicate Through Implicit Communication Channels

Response to Reviewer #4

Q1: All of the environments studied in this work have the same quirk as Hanabi, namely that agents cannot view the information they need but they can view the information for all other agents and have to communicate that information to other agents. This work would be more convincing if the key communication challenge between settings were more unique.

A1: We understand the concern about the homogeneity of environments. While Hanabi indeed shares characteristics with other tasks, it presents unique challenges for implicit communication due to its rules and constraints, which require effective coordination. Future iterations of this research will explore a broader set of environments to validate the generality of ICP further.

Q2: The task of learning an implicit communication channel in this paper does not seem too different from learning an explicit discrete communication channel, with the only major difference being that the "scouting" actions actually have some information prior whereas discrete channels are typically arbitrary. I would've liked to see how algorithms for explicit discrete communication channels compare to the proposed techniques in this paper as baselines. Furthermore, DIAL should be added as an explicit baseline instead of just comparing with VDN baselines.

A2: We agree that including DIAL as an explicit communication baseline is meaningful. However, due to the complexity of the Hanabi environment and its significantly delayed reward feedback, we anticipate that DIAL may not exhibit substantial improvements in this scenario. Additionally, as an on-policy algorithm, DIAL requires considerable computational resources and time for experiments in Hanabi, making it challenging to complete within the current timeframe. Therefore, we will focus on evaluating DIAL in the Guessing Number and Revealing Goals environments, these results will be included before the rebuttal phase concludes.

Q3: How does the delayed map approach perform in Guessing Number and Revealing Goals?

A3: We are conducting this experiment and will present the results when the experiment is completed (within the rebuttal period).

Q4: Can the proposed technique be used in situations where there is no clear delineation between "scouting" and "regular" actions?

A4: For situations where there is no clear delineation between "scouting" and "regular" actions, we have discussed in Section 8, and we leave it as future work.

Thank you again for your review. We hope that we have addressed your main questions in the rebuttal. As the author-reviewer discussion period is approaching its end, we would like to know if you have any additional questions or concerns. We are happy to provide our response if so.

Response to Reviewer #3

Q1: ICP proposes add actions to the action space that maps information into certain ‘embeddings’ to be broadcasted to other agents. How is this not explicit communication?

A1: We believe that you might have some misunderstanding of our method. ICP does not introduce new actions into the action space. As explained in General Response A2, we defined a subset of original action space as scouting actions, as they have either no or uniform effects on the state $s_t$ or the reward $r_t$ but primarily influence the observation function $O_i(s_t)$, ICP can utilize these actions as carriers for indirectly transmitting information with fewer drawbacks (such as unpredictable environment dynamics' changing).

Q2: Unless I am missing something, the channels are provided so the agents are not learning to construct these channels, they are simply learning to use those channels (in an explicit manner). This makes the paper title confusing.

A2: Thank you for your comment. We agree that our algorithm is learning implicit communication through implicit channels, while the construction part is not particularly through learning. We have updated our title to 'Learning to communicate through implicit communication channels'.

Q3: Results in Hanabi with only 3 random seeds are not enough. SAD reported results with 13 random seeds. Figure 2b is also missing error bar.

A3: We trained the on-policy algorithm in Hanabi using only three random seeds due to the high computational cost, with each seed requiring approximately 150 hours of training. In fact, all three randoms obtain very close results, which is evidence that the method has a solid performance. We are working on other random seeds, though with the computation resource we have (2 Nvidia 3090 GPU), it is unlikely to obtain results for 13 seeds by the end of the rebuttal period.

In contrast, we trained the off-policy algorithm using six different random seeds. Figure 2b presents the average episode length, evaluated over 10,000 episodes across various seeds. Additionally, we updated the error bars in the new version of the PDF to reflect these changes.

Q4: Writing clarity can be improved. For instance, phrases like 'communication gradient method' and 'hat mapping method' are often used and assumed with a lot of context before defining it.

A4: To provide context for these phrases, we have added references in new version of PDF.

Q5: Some statements are not well justified. For instance, in line 251-254, it is unclear to me how this method specifically simplifies the learning process and promotes better coordination and communication among the agents.

A5: We thank the reviewer for pointing this out. We have updated this statement in new version of PDF.

Q6: Isn't broadcasting an assumption rather than a benefit? This is not possible in many practical scenarios.

A6: This is indeed an assumption. By saying "benefits" we meant that this broadcasting feature might benefits us, and we want to leverage this feature (which many previous works did not). It enables the use of 'efficient embedding techniques' such as 'hat mapping method' within our framework, enhancing its overall effectiveness. We have updated the statement to avoid this confusion.

Q7: What do you mean by 'efficient embedding techniques' in line 268?

A7: One example of `efficient embedding techniques' is 'hat mapping method', we also updated this statement in new version of PDF.

Thank you again for reviewing our manuscript. We noticed some factual misunderstandings in the review (i.e. we did not add new actions to the action space), which could have drastically affected the evaluation. We made a rebuttal on this and other points. Notice that the author-review discussion period is approaching an end. Would you please take a look at our rebuttal, and potentially discuss with us on the concerns/questions you raised?

I still find the line between explicit and implicit communication a bit unclear here. Both of the newly designed environments have actions given specifically for communication and I find the definition of scouting actions to be quite narrow for implicit communication. Using the Revealing Goals as an example, agents can communicate with just directional movement actions if they could learn that a certain direction corresponds to certain information. Hence, I don't see why implicit communication action has to be limited to those in which the state does not change.

The motivation of our method is to provide a learning algorithm for a more accurate and efficient communication mechanism compared to existing implicit communication approaches, such as Theory of Mind (ToM). In our framework, any action can be leveraged as part of the implicit communication channel through the mapping mechanism defined by our method to achieve precise and effective information exchange. In real-world scenarios, many non-critical actions (e.g., subtle gestures or tones among team members on a battlefield) exist and they could serve as carriers for our communication strategy. However, such nuanced actions rarely appear in abstracted and simplified reinforcement learning environments, because they have been deliberately removed from the testbeds. This limits the scope of our testing, and results in having two newly designed tasks that are similar to Hanabi (the existence of such actions in real world is one of the motivations for designing the Hanabi game).

Meanwhile, there are actions in real-world environments also influence key dynamics such as rewards, state transitions, and environmental dynamics. These effects are often unpredictable in existing implicit communication channels—while they may successfully transmit information, they could inadvertently lead the environment into suboptimal states. To avoid this issue, we focus on predefined scouting actions, which are already present in environments like Hanabi and have minimal impact on the environment's dynamics. By using such actions, our method achieves significant improvements over many existing algorithms.

We fully agree with you that it will be more elegant if the frame is general enough to "select" which actions are used for communication. For future work, we plan to extend our method to account for the dynamic impacts of other actions and explore how to define carriers of information in broader environments. These carriers may no longer be confined to scouting actions as currently defined.

I sympathize with the issue with compute. But the difference between SAD (green dash line) and the proposed method is simply too close to support the conclusion.

Thank you very much for encouraging us. We are working hard under limited computing power and we believe that the privilege of research shouldn't be limited to only people with resources abundant.

To our best knowledge, the difference between SAD and our method is quite large, and we wanted to explain why. We would like to provide more detailed data to highlight this difference. According to the original SAD paper, it achieved a score of 23.81/25 with a win rate of 41.45%. In contrast, our method achieved a score of 24.91/25 with a win rate of 91%. It is more than double that of SAD in terms of win rate. The apparent discrepancy between the significant difference in win rates. The seemingly smaller difference in average scores can be attributed to Hanabi's total score cap of 25.

In fact, there are certain deals that make the game impossible to win (like getting a bunch of 5s and 4s as the starting hand), which means the theoretical maximum one could achieve is less than 25. Bearing this in mind, winning the game 91% of the time, and achieving 24 points for almost all of the rest 9%, is close to the theoretical maximum. This means not only do we double the winrate of SAD, our improved result also approaches the theoretical maximum. We will include this analysis in the appendix of our paper. We hope this explains why we claimed the empirical results to be a significant improvement.

Thank you very much again for reviewing our manuscript, further discussing it with us, and encouraging us. We would like to follow up on the latest discussion on the distinguishment of implicit/explicit communication, and the distinguishment in the performance of our method/SAD. Especially on the latter one, where the performance of our method and SAD are not really close in Hanabi (despite being visually close on the figure). Could you please take a look at our reply and potentially discuss with us on these?

Thank you very much for the positive feedback! This encourages us a lot. We fully agree that it would be more desirable if the agent could learn to pick actions, that are not scouting actions but close to scouting actions, to implement implicit communication (e.g., there are some directional moves that are suitable for implicit communication while some are not). This will mimic how implicit communication emerged in human history (from early humans engaging in hunting and gathering through silent cooperation, to modern military operations using gestures and codes for covert communication, and even in everyday social interactions where intentions are conveyed through expressions, tone, and body language). We are excited to work on this in our future works.

Response to Reviewer #1

Q1: In experiments, for agents trained with VDN, is the action space the combination of ''regular actions'' and ''scouting actions''?

A1: Yes, the action space of VDN and ICP are same, for VDN only used one action policy to choose both regular actions and scouting actions, while ICP use action policy to choose {regular actions, send_info}, when send_info is chosen, information strategy combined with mapping mechanism will select scouting actions.

Q2: In figure 3b, why are those methods not trained to the same length?

A2: In Figure 3b, we compared several algorithms with significant differences in sampling efficiency and convergence speed. Given the complexity of the Hanabi environment and the substantial training time it requires, we set the training time for on-policy algorithms to 150 hours and for off-policy algorithms to 20 hours, and plotted the results accordingly.

Response to Reviewer #2

Q1: The paper does not compare results for common communication architectures such as CommNet, TarMAC, SARNet etc for their environments but only with value decomposition networks (VDN), that do not perform communication as part of their actions.

A1: We believe you might have some misunderstanding of our work's background. Communication architectures such as CommNet, TarMAC, SARNet etc are mainly used for explicit communication problems. They allow a direct communication channel, which is not available in our problem. For more details of our work's background, you may refer to the General Response A1 and Section 1 of our paper.

Q2: The results for Hanabi are quite remarkable, however, game rules indicate, the hints are only constrained towards revealing either the color or the number. Since ICP implicitly encodes the local observation of each agent (both for DIAL and RGMComm) into message vectors, I believe each agent observes in essence have access to the complete global state, which inherently breaks the rule of Hanabi, unless I am mistaken. I would like more clarification on this.

A2: As described in Section 4, and also explained in General Response A2, hint actions in Hanabi also carry two types of information: (1) information reflected through the environment and (2) information reflected through the choice of the scouting action itself. Our method utilized the second type of information to transmitted information, by using the hint actions to implicit communication, we think each agent can indirectly access a part of others' partial observation (message size are limited by hint action space and message strategy are learned), thus benefit cooperation. This implicit communication does not break the rule of Hanabi.

Thank you again for reviewing our manuscript. We noticed essential factual misunderstandings in the review, which could have drastically affected the evaluation. We made a rebuttal on this and other points. Notice that the author-review discussion period is approaching an end. Would you please take a look at our rebuttal, and potentially discuss with us on the concerns/questions you raised?

We thank the reviewer again for acknowledging our problem setting of implicit communication, and acknowledging the fact we do not violate the Hanabi game rules. In the new summary, we believe there are still some factual discrepancies, as we explained in the previous reply. As the author-reviewer discussion period is approaching its end, we request you to look at the reply and potentially discuss it with us. Meanwhile, because the initial rating was conducted based on multiple factual errors, it seems not very fair to insist on this rating. We request you to reevaluate the work based on the updated view of the work. We are more than happy to respond to any further questions you may have.

We sincerely thank all the reviewers for their constructive feedback and valuable suggestions. In response to your comments, we have conducted additional experiments and incorporated the results into our paper to provide a more comprehensive analysis.

Specifically, we have implemented and evaluated DIAL with a direct communication channel (DIAL-Cheat) and ICP with the delayed map approach (ICN-DIAL-DM) in the Guessing Number and Revealing Goals environments. The updated results can be found in Figures 2 and 3(a).

Summary of Updates:

1. Implementation of DIAL-Cheat:

   • This approach adds a discrete direct communication channel to the environment, with the message size matching the scouting action space size. This channel allows agents to send broadcast messages while acting.
   • DIAL-Cheat demonstrated improved performance over our method in the Guessing Number environment but showed lower average performance and higher variance in the Revealing Goals environment due to the small size of the message sapce.

2. Implementation of ICN-DIAL-DM:

   • This approach first pre-trains the information strategy in a modified environment (with scouting actions removed and a discrete communication channel added) with DIAL and then fine-tunes both the information strategy and action policy in the original environment.
   • ICN-DIAL-DM performed slightly worse than ICN-DIAL-RM in the Guessing Number environment but still outperformed the baseline methods (VDN-on-policy and VDN-off-policy). However, in the Revealing Goals environment, its performance was comparable to the baselines and exhibited significant variance.

We believe these additional experiments and analyses provide deeper insights into the strengths and limitations of our method under various conditions. Please let us know if there are further aspects you would like us to address or clarify. Once again, we deeply appreciate your feedback and the opportunity to improve our work.