Emergent Robust Communication for Multi-Round Interactions in Noisy Environments

Weaknesses

This paper does not explore the type of messages learned by the agents. It would be interesting to visualize in some way some of the messages learned by the agents to communicate.

We thank Review zpnz for the interesting suggestion. Since the agents learn an abstract language using abstract tokens, we do not have a good way to visualize the messages sent, but it could be an interesting future work direction.

---

The entropy term in the loss is not defined in the paper and the motivations for the use of this term do not seem good enough.

The loss functions used in both agents are fully detailed in Appendix E.1. Additionally. The entropy term was fully evaluated in [Chaabouni et al., 2022]. In the text, we point to this paper for further details (see Section 2.2.2).

---

In table 2, I wonder about the significance of these results; of course the agents are trained with noise, they will perform better in noisy testing than agents that were not trained with noise.

We can see a clear difference in the emergent communication protocol learned. Looking at Table 1 and Table 2, we can understand that the agents trained in the NLG (with noise) can create a robust communication protocol to handle noisy and noiseless messages.

---

** The use of noise is framed as a major contribution. However, using random noise in the messages is not new; the authors give the example of the work with the zipfs law to analyse language properties (page 2) but this is not the only place where noise is used and analysed [1, 2, 4].**

We thank Reviewer zpnz for the suggestions. We added these works to the extended related work section in Appendix A. Nevertheless, we maintain our claim for novelty for the following reasons. All works [1, 2, 4] simplify the problem by having a DIAL method, meaning gradient flows between agents. In our case, we implement a RIAL architecture where each agent sees the other as part of the environment, substantially increasing the problem's complexity. As such, the Speaker does not even know that the message is suffering modifications, making the coordination problem extremely complicated. [1] also uses a continuous communication problem, making it distant from the scope of our work.

---

In page 5: "and effective game round, $i\in\{1,\ldots,I\}$" it seems that the number of $I$ rounds in now defined as while before in section 2.2 it was defined as $N$

There is a difference between $I$ and $N$. $I$ is the number of rounds played in a game and $N$ is the maximum number of rounds a game can have. We will make this distinction more clear, see Section 2.2.2.

Questions

In section 2.2, it is unclear to me what is the unk token and how the noise is applied. From equation (1) I understand it results from the noise function if, but I fail to understand where comes from. Is it a fixed token? If so, I cannot agree that noise is being applied to the message, also because, according to the equation, the message is not affecting any of the generated noise. Could the authors elaborate on this?

Yes, unk is a fixed token. We apply noise to the message by substituting a variable number of message tokens with the unk token. The noise applied is externally introduced in the message. The Speaker does not even know if the message contains noise, adding additional complexity.

---

It is unclear to me what $\hat{x}$ means. In page 3, it is both stated "where the goal is to try to identify the image $\hat{x} \in \mathcal{C}$ that the Speaker received, $\hat{x}=x$" and "round when the Listener plays the I don't know (idk) action, $\hat{x} = \hat{x}_{idk}$". It seems to have different meaning in each case. In the first case it seems to denote the guess of the listener and in the second it seems to define an action. Could the authors clarify?

$\hat{x}$ is the action selected by the Listener. This action can be an image or the action ``I don't know'' ($x_{\text{idk}}$). If an image is selected by Listener, $\hat{x}$ is also one of the images of a set (candidates set) given to the Listener.

In page 6: "where we linearly increase the noise level in the communication channel from 0 to $\lambda$.". From equation (1) $\lambda$ represents a probability of wether insert noise or not. How does increasing $\lambda$ will increase the level of noise? While it can happen, it does not seem necessarily true that it will happen.

For each token in the message, we draw a number from the uniform distribution $l\sim\mathcal{U}(0,1)$. If $l>\lambda$ we change the message token by the unk token. In expectation the number of removed token is $\lambda\cdot T$, where $T$ is the message length.

Weaknesses

There are several works introducing the multi-round Lewis Game and its corresponding listener architectures [1,2]. Justification would be better to underscore your contribution by comparing and contrasting these literatures.

We thank Reviewer VCLK for the suggestions. We added these works to the extended related work section in Appendix A. Nevertheless, there are significant differences between [1,2] and our work: In [1], the Listener processes one candidate at a time, and the vocabulary used contains only two tokens. In our case, our vocabulary has 20 tokens, and the Listener tries to discriminate between all candidates at each round, which substantially increases game complexity. In [2], the game design is similar to ours, where the Listener receives a message and decides to play another round or select a candidate as the final answer. Nonetheless, our approach is far more complex and realistic than in [2] since we have a discrete communication channel that can suffer modifications from an external source instead of a continuous communication channel [2]. Complementarily, [2] simplifies the problem by using a DIAL approach, allowing gradients to flow between agents. In contrast, our method uses a RIAL approach, where agents perceive others as part of the environment, meaning no gradient flows between agents. In this case, the Speaker does not know that the message is being modified, complicating the coordination task.

---

From the experimental results (Table 3,4,7,8), the accuracies of game variants with/without multiple rounds do not differ a lot. More elaboration or experiments are expected to ablate the contribution of the multi-round setting.

With these results, we observe no clear advantage to using multiple rounds; the agents have enough predictive power to play the game successfully with only one round.

---

Training and testing in the same noisy environment may not be enough to show that “robust communication” emerges: We consider the noise may come from the observation beside the communication channel, for example from the sender or receiver sides. [3]

We thank Reviewer VCLK for the suggestion. This is an interesting experiment to explore. We already have further experiments in Appendix F; one of them (discrimination task) does this, adding noise to the observation of the Speaker and the Listener's candidates.

---

Instead of also replacing message tokens during testing, other interfering methods to the communication channel can be applied, for example randomly dropping tokens in the messages.

This is what we do, the noise we introduce acts by replacing message token by a pre-defined token called the unk token.

---

What will be the result when the agents are trained with $\lambda=0.75$ and test with 0.5 and vice versa?

We thank Reviewer VCLK for the suggestion. We can easily add further results considering this (experiments are already running).

---

How would compare with previous works on adding noise to the communication channel? [4,5]

We thank Reviewer VCLK for the suggestions. We added these works to the extended related work section in Appendix A. Still, there are some major differences. [4] uses a DIAL setting, and the noise is introduced in the continuous space (before sampling discrete tokens). [5] also implements a DIAL setting and the introduction of noise is adversarial, meaning message tokens are changed by other valid tokens, trying to deceive the Listener. In contrast, our method uses a RIAL approach, where agents perceive others as part of the environment, meaning no gradient flows between agents. In this case, the Speaker does not know that the message is being modified, complicating the coordination task.

Questions

What is the maximum number of round $I$ set to? Throughout the training process, does the average number of communicative rounds vary a lot? For example, from more rounds to fewer rounds? This may help validate your multi-round design.

Yes, the avg. number of rounds increases as the training progresses. We can add a section to the appendix displaying this information. Although the number of rounds increases, the overall performance remains the same.

---

There are several references missing on page 6.

This was a problem when splitting the document into 2 parts (main paper and appendix). We fixed the problem, thank you.

Weaknesses

The authors' reliance on established paradigms like the Lewis Game, even with modifications, raises concerns about true innovation. Is this just a repackaging of old concepts?

The Lewis Game is a well-established task in emergent communication since it presents several crucial challenges to overcome to understand and study how communication can emerge to solve a cooperative task. Most related work presented in the paper and the related work proposed by all reviewers has experiments based on a derivation of the original Lewis Game. In our proposed work, we follow the work proposed by [Chaabouni et al., 2022] that complexifies the original Lewis Game by using huge natural image datasets for the input distribution and increases the number of candidates given to the Listener, increasing the difficulty of the discrimination task. The main focus of our work is to study and explore how noisy conditions can affect the emergence and quality of the communication protocol in such tasks. As described in Section 3, previous methods are not robust to noise in the communication channel. As such, we proposed new agent architectures and training schemes, creating robust communication channels.

Questions

Given the modifications you've introduced to the Lewis Game, how do you justify that these changes bring genuine innovation in the context of multi-agent reinforcement learning, as opposed to simply adding complexity to an existing framework?

We would like to emphasize that the primary purpose of our work is not to create a MARL system. The main trends in MARL focus on studying centralized training with decentralized execution, how partial observability affects performance, and how to exchange information through communication. In contrast, emergent communication studies aim to assemble cooperative tasks to study emergent communication and its properties.

We now clarify the novelty of our work. First, we present a series of experimental evaluations that give new insights into how noise affects the emergence of communication when using more realistic datasets (in our case, ImageNet and CelebA). Secondly, we merge noisy and time-dependent environments, creating highly complex environments that generalize the original Lewis Game. Finally, we derive new agent architectures to play these realistic and more challenging games. For example, most (if not all) methods proposed in previous works could not take advantage of the Multi-Round Indecisive Lewis Game (MRILG) because these methods are not prepared to have time dependencies in their architecture.

---

The concept of noise adaptation in emergent communication is not new. How does your approach fundamentally differ from existing solutions, and what unique challenges does it address in the multi-agent reinforcement learning landscape?

In the introduction, we talk about [Ueda and Washio, 2021]. Even though previous works consider noisy environments in emergent communication, the difference to our work is two-fold. First, we only consider discrete messages as [Ueda and Washio, 2021]. Second, much like [Chaabouni et al., 2022], we take into consideration more realistic datasets (discriminate natural images) applied to the context of having noisy communication channels. We added to appendix a section for related work, see Appendix A.

Weaknesses

The paper could benefit from comparing the proposed architecture to existing approaches in the field of emergent communication. This would provide a better understanding of the novelty and effectiveness of the proposed method.

The main focus of our work is to introduce a new approach to handle external interference in the communication channel. We present a comparative analysis with [Chaabouni et al., 2022], which proposes the original LG (SS) (see section 3), where the Listener implementation is a self-supervised agent. We demonstrate in section 3 that our proposed architecture, where the Listener is an RL model, is crucial to developing robust communication protocols that can handle the noiseless and noisy case.

Additionally, we argue that our work is different enough from previous approaches where a direct comparison could be meaningless or time-consuming, where severe alterations are needed to modify previous architectures. We now elaborate on our argument. We can divide the emergent communication literature into two major clusters. On one set, the works employ discrete communication channels, forcing a RIAL implementation where each agent learns its parameter set and treats other agents as part of the environment. The architectures proposed in [Lazaridou et al., 2016, Choi et al., 2018, Bouchacourt and Baroni, 2018, Lazaridou et al., 2018, Graesser et al., 2019, Li and Bowling, 2019, Ueda and Washio, 2021] do not consider time dependencies. As such, to compare such architectures, we need to make substantial architectural changes to make sure agents can go through multiple rounds.

On the other hand, we have works that simplify the problem of emergent communication where the gradient can flow between agents (DIAL methods) [Havrylov and Titov, 2017, Mordatch and Abbeel, 2018, Tieleman et al., 2019, Guo et al., 2019, Chaabouni et al., 2020, Rita et al., 2022]. This oversimplification of the problem is unnatural when looking from the viewpoint of simulating human communication, which is non-differentiable. From this standpoint, comparing our approach with such works appears inappropriate because the introduction of noise is not straightforward. In the naive case, the noise would have almost no negative impact since gradients flow through agents, where protocols are robust by default. For example, a VAE architecture is robust to noise since the gradient flows freely from the decoder to the encoder, and the latent space is continuous.

---

The evaluation of the proposed architecture is focused on the newly developed referential game. It would be valuable to evaluate the architecture on a wider range of tasks and compare its performance to other architectures to assess its generalizability.

In Appendix F, we introduce and evaluate the agents trained in every Lewis Game variation in other transfer learning tasks with the main purpose of testing their capability to generalize to new tasks.

Additionally, the vast majority of works in emergent communication are based fundamentally only on this task - a variation of the Lewis Game. The LG is a simple but highly customizable task that evaluates several properties of emergent languages, allowing the connection to human languages.

---

While the paper mentions the capability of the proposed architecture to handle noise, there is limited discussion on how the architecture specifically addresses and mitigates the impact of noise in the communication channel. Providing more details on this aspect would enhance the clarity and understanding of the proposed method.

In section 3.2, we motivate that our novel agent architecture perform better than the baseline. In section 3.2.1, we demonstrate that if we generalize the Lewis Game, where the communication channel can modify messages by introducing noise, our proposed architecture can still be robust and have high performance in both games (Lewis Game without noise and Lewis Game with noise). The second architectural change appears in section 2.2, where we define the MRILG (multi-round indecisive Lewis Game) and further extend the Listener architecture to contain an extra action, the I don't know action. When the Listener chooses this action, the pair of agents play a subsequent round in the Lewis Game, which allows the Listener to receive and process more information.

To give some additional information, the original Lewis Game (LG) is played only in one round - the Speaker receives an image and creates a message describing the image. The Listener tries to guess the image received by the Speaker over a set of candidate images taking only the message into account. In MRILG, several rounds of the LG can be played as long as the Listener chooses the idk action. During these several rounds, the Speaker's input image and the candidates' set are always the same.