Emergent Language based Dialog for Collaborative Multi-agent Navigation

We thank the reviewer for the valuable comments, and hope the following point-by-point responses address the reviewer’s concern.

Question 1:

Among the mentioned baselines, RMM [1] is the only work concerning language generation ability. Unfortunately, it’s hard to reproduce RMM to get sufficient text examples for comparison. We take the example provided in the RMM paper, run our model using the same start position and the same target, and compare the emergent texts of our model with the natural language of RMM, as depicted in Appendix Table 5. There is no obvious alignment between the natural language and the emergent language. This observation draws a similar conclusion as that of Appendix A, i.e., the emergent language captures different views about the surrounding and the task ontology with the natural language learned from a large scale of human-annotated data. This will lead to an interesting content of our future work.

Question 2:

Much existing research on navigation task concerns only language understanding ability of one agent. The most similar work is RMM [1], which focuses on training the natural language generation and understanding of two agents, under the supervision of a large scale of labelled dialogues. While our work concerns about the emergent language generation and understanding, without any labelled language data. The intrinsic advantage of our design is to train a multi-agent navigation model all from the scratch, without any human intervention, which is, to the best of our knowledge, the first work that proves the feasibility of multi-pass emergent communication in realistic dynamic environments, and provides an empirical method for the implementation.

Weakness 1:

Thank you for pointing out this concern. There is no explicit underlying rules or heuristics that the emergent language adheres to. Actually, we performed intensive analysis of the emergent language, by detecting the alignment of emergent language with visual surrounding, and of emergent language with task, which we consider as two essential characters of language. In order to make further study on whether there are implicit rules underlying the emergent language, following [2], we analyzed the compositionality of emergent language, by calculating the topographic similarity between observation pairs and corresponding message pairs under two different settings.

In the first setting, we calculate the editing distances of two dialogue histories up to the same turn $t$ as following:

$LevenshteinDistance(H_{utterance, t}^{T, i}, H_{utterance, t}^{T, j})$

$H_{utterance, t}^{T, i} = \langle U_1^{T, i}, U_2^{T, i}, ..., U_t^{T, i}\rangle$

$H_{utterance, t}^{T, j} = \langle U_1^{T, j}, U_2^{T, j}, ..., U_t^{T, j}\rangle$

In the second setting, we calculate the editing distances of two messages in the same turn $t$:

$LevenshteinDistance(U_t^{T, i}, U_t^{T, j})$

The curves of topographic similarity are shown in the Figure 5 in the paper. The first setting observes an uptrend in topographic similarity (colored in orange), while the second setting a downtrend (colored in blue). The uptrend suggests that the longer history conveys the more information. The downtrend suggests that each message progressively enriches information given the previous turns, and the messages at the beginning of the task carry more information than that of the later steps. As a metric evaluating the compositionality of the emergent language, this score suggests that the emergent language could composite task-specific features helpful for task solving.

Weakness 2:

Please refer to the response to Question 2.

[1] Roman H R, Bisk Y, Thomason J, et al., RMM: A recursive mental model for dialogue navigation, EMNLP 2020.

[2] Lazaridou A, Hermann K M, Tuyls K, et al., Emergence of linguistic communication from referential games with symbolic and pixel input, ICLR 2018

Weakness 1:

Thanks very much for the suggestion, inspired by your suggestions we have conducted some analysis about the emergent language, and have added it to the paper. More efforts will be made to have a deeper analysis of the language.

we conducted an experiment to analyze the compositionality of emergent language, by calculating the topographic similarity between observation pairs and corresponding message pairs under two different settings.

In the first setting, we calculate the Levenshtein distances of two dialogue histories generated by the Tourist up to the same turn $t$.

$LevenshteinDistance(H _{utterance, t}^{T, i}, H _{utterance, t}^{T, j})$

$H _{utterance, t}^{T, i} = \langle U _1^{T, i}, U _2^{T, i}, ..., U _t^{T, i}\rangle$

$H _{utterance, t}^{T, j} = \langle U _1^{T, j}, U _2^{T, j}, ..., U _t^{T, j}\rangle$

In the second setting, we calculate the Levenshtein distances of two messages in the same turn $t$.

$LevenshteinDistance(U _t^{T, i}, U _t^{T, j})$

The curves of Topographic similarity are shown in Figure 5 in the paper. The first setting observes an uptrend in topographic similarity (colored in orange), while the second setting is a downtrend (colored in blue). The uptrend suggests that the longer history conveys the more information. The downtrend suggests that each message progressively enriches information given the previous turns, and the messages at the beginning of the task carry more information than those of the later steps. As a metric evaluating the compositionality of the emergent language, this score suggests that the language could composite task-specific features helpful for task solving.

Those results and more details have been added to the paper and we are trying hard to explore more characteristics of the emergent language.

Weakness 2:

It is a very interesting idea to apply our method to fine-tune a language-aware model. Since the proposed method is formulated for navigation tasks, the emergent language could capture the most task-related information directly. By comparison, training both of the agents (the Tourist and the Guide) learning natural language, requires the language to capture two types of information: the distribution of natural language and the task-specific information. In natural language setup, a direct method is to take a supervised method to train the language modules to achieve the task. However, in supervised training, the quality of natural language is learned directly but the task-related information lays behind the language which is learned indirectly. This is a possible reason why the natural language has a poorer performance (proved by the comparison between our method with the RMM). Applying our method to a language-aware model could mitigate this problem. With our method learning the task-related information, the supervised method learning the language structure, a much better performance could be achieved.

Some possible methods to incorporate our method into the existing language-aware model without causing much language drift could be tried:

1. Use our method as a finetuning approach with KL loss making sure that the fine-tuned model is not too far from the initial language-aware model, similar to the RLHF methods applied on LLM.
2. Train the supervised method and our RL methods on the language modules iteratively. The supervised method is used to maintain the quality of natural language, and our RL method to learn task-related information following [1].

We apologize for the misc/typos, and have fixed them and also added the missing reference. We will also move Figure 4 (Emergent language analysis) to the main body of the paper in the revised version.

[1] Lowe R, Gupta A, Foerster J, et al., On the interaction between supervision and self-play in emergent communication, ICLR 2020.

We thank the reviewer for the valuable and constructive comments, and add the following clarifications to address the reviewer’s concerns:

Question 1:

The claim “we also empower the oracle agent to provide instructions progressively “is backed by the experiment results on different settings of dialogue strategy which is shown in Table 4. When comparing the “only in beginning” setting with other settings with more dialogue frequency, it observes that the “only in. beginning” setting has a poorer performance, indicating that providing all information only at the beginning is not enough, and the message in each turn conveys additional helpful information for the following task. To better verify this point, we conducted experiments of “direct communication” (i.e., sending the shortest path repeatedly in each turn) and observed a similar performance with that of the “only in beginning” setting (tl=10.09, ne=6.88, sr=25.74, spl=22.78, vs tl=10.79, ne=6.44, sr=25.37, spl=21.13), proving that the subsequent utterances provide additional helpful information instead of repeating the same information. Our further study also found that especially when the Tourist made wrong movement actions, the multi-turn communication could provide helpful instructions, leading to a much better performance.

Question 2:

At the very beginning of our research, we tried to directly train a multi-agent navigation model in an end-to-end manner using reinforcement learning, and observed a discouraging non-convergence of the training. Under this setting, we jointly trained the language generation module and understanding module for both agents (i.e., four modules in total) by simultaneously maximizing the whole accumulated rewards. However, the model collapsed into a poor status since the generation and understanding are not coordinated at the start of training which can cause the state to quickly become large and chaotic. Therefore, we formulate the navigation task as a two-pass task and train each pass separately in a supervised learning manner: In the first pass (the so-called “Localization” task), the Tourist was encouraged to learn how to generate language accurately describing its visual surrounding, and the Guide to correctly understand the message from the Tourist to predict where the Tourist is, with the goal of making proper guideline for future steps. In the second pass (the so-called “Movement” task), the Guide was encouraged to generate language accurately describing its guideline, and the Tourist to correctly understand the message conveying the guideline, in order to make proper movement. Then we returned back to jointly training for the navigation task as mentioned at the beginning, but with parameters initialized by the two-pass training tasks. Therefore, we call each pass a pre-training task of the targeted navigation task, the principled fundamentals of which are explained in Section 4.5.

Question 3:

Yes, your thought is right. We do choose the vocabulary size of 26 to draw a similarity with the English language. The experiment in Figure 3b is to explore the effect of the vocabulary size on the performance and the 26 may not be the best choice among all possible choices of vocabulary size in the proposed navigation task.

Question 4:

In Figure 4 (a) of section 5.5, we use PCA as a dimension reduction technique to reduce the encoded Tourist’s visual image $\bar{e} _o^t \in \mathbb{R}^{dim}$, and encoded dialog history $\bar{e} _{H _{dialog}}^G \in \mathbb{R}^{dim}$ into 2D space. Similarly, in Figure 4 (b), the encoded dialogue history $\bar{e} _{H _{dialog}}^T \in \mathbb{R}^{dim}$, and the encoded next step action $\bar{e} _p^{pos _i \rightarrow pos _j } \in \mathbb{R}^{dim}$ are reduced via PCA dimension reduction tool. Then, in Figure 4 (a), we cluster the encoded visual images using KMeans and connect the center of the clusters and the center of the corresponding messages, then we find the parallel. Similarly, in Figure 4 (b), we cluster the encoded next step action using KMeans and connect the center of the clusters and the center of the corresponding messages. Thanks for your advice, and we have added those details to the appendix.

We thank the reviewer for the valuable comments, and hope the following clarification addresses the reviewer’s concern.

Questions:

Thank you for your thought-provoking question. We had conducted experiments in a similar setting, referring to “only in beginning” in Table 4. We let the Tourist encode the shortest path instruction as a part of dialog history in each turn. Inspired by your question, we immediately conducted experiments using “direct communication” (i.e., sending the shortest path repeatedly in each turn) and observed a similar performance with that of the “only in beginning” setting (tl= 10.09, ne=6.88, sr=25.74, spl=22.78, vs tl = 10.79, ne=6.44, sr=25.37, spl=21.13). By comparing with other results in Table 4, both performance of “direct communication” and “only in beginning” are poorer than that under other settings (“soft”, “20%”, “50%”, “100%”), suggesting that the dialogue progressively provides additional information. Our further study also found that especially when the Tourist made wrong movement actions, the multi-turn communication could provide helpful instructions, leading to a much better performance.

Weakness 1:

Thank you for your suggestion, and we had noticed this problem before but did not make much attempt at it. We performed language analysis by detecting the alignment of emergent language with visual surroundings, and that of emergent language with task, which we consider as the essential characters of language. Motivated by your suggestion, we conducted an experiment to analyze the compositionality of emergent language, by calculating the topographic similarity between observation pairs and corresponding message pairs under two different settings.

In the first setting, we calculate the Levenshtein distances of two dialogue histories generated by the Tourist up to the same turn $t$.

$LevenshteinDistance(H_{utter, t}^{T, i}, H_{utter, t}^{T, j})$

$H_{utter, t}^{T, i} = \langle U_1^{T, i}, U_2^{T, i}, ..., U_t^{T, i}\rangle$

$H_{utter, t}^{T, j} = \langle U_1^{T, j}, U_2^{T, j}, ..., U_t^{T, j}\rangle$

In the second setting, we calculate the Levenshtein distances of two messages in the same turn $t$.

$LevenshteinDistance(U_t^{T, i}, U_t^{T, j})$

The curves of Topographic similarity are shown in the Figure 5 of the paper. The first setting observes an uptrend in topographic similarity (colored in orange), while the second setting is a downtrend (colored in blue). The uptrend suggests that the longer history conveys the more information. The downtrend suggests that each message progressively enriches information given the previous turns, and the messages at the beginning of the task carry more information than those of the later steps. As a metric evaluating the compositionality of the emergent language, this score suggests that the language could composite task-specific features helpful for task solving.

Those results and more details have been added to the paper and we are trying hard to explore more characteristics of the emergent language.

Weakness 2:

Thanks for your suggestion. We do find the font size is too small when we zoom in, which is unfriendly to readers. We have adjusted (and further improvements will be made) Figures 2 & 3 carefully and hope it could improve the readability of this paper.

Weakness 3:

Thank you for your suggestion and the old version is indeed not clear enough. We have added the pseudocode and the descriptions of the baseline methods to the Appendix (and more improvements will be made). And hope the revised version could be more friendly to readers.

We thank the reviewer for the valuable comments, and hope the following clarification addresses the reviewer’s concern.

Weaknesses: we thank the reviewer for giving us valuable advice. The R2R and CVDN datasets are initially designed for natural language navigation between two agents, therefore their test data are used to evaluate the alignment quality of natural language with the environment, which are demonstrated on the corresponding leaderboards (but with the environment settings unknown to the public). Our work focuses on emergent language communication therefore it is infeasible to draw a comparison with related work on the leaderboard using test data. However, in order to assess the language understanding and generation ability of our method, we train and evaluate it under paralleled environment settings (i.e., with the same visual groundings of train and val dataset) with the related work on R2R and CVDN. We have mentioned this in section 5.1 and will give more explanations in the future revised version.

Question1: Please refer to the above response to Weaknesses.

Question 2: Thanks for your suggestion. We do find the font size is too small when we zoom in, which is unfriendly to readers. We have adjusted (and further improvements will be made) Figures 2 & 3 carefully and hope it could improve the readability of this paper.