LangNav: Language as a Perceptual Representation for Navigation

Thanks to the authors for responding to my comments. After reading all the reviews and responses, I have understood the concerns of other reviewers and the authors' detailed rebuttals.

Here, I would like to succinctly summarize the main strengths and weaknesses of this paper:

Strengths:

• Extensive comparisons across recent VLN baselines. However, during the rebuttal period, one experiment was conducted to address the concerns of five reviewers, although our concerns varied.

• Language as a perceptual representation for VLN. While I agree with the weakness 1 of Reviewer PNyi, which the authors have rebutted and made commitments. The motivation is sound, nonetheless, the latest work in Vision-Language Models (VLM), even the powerful GPT-4V, still suffers from significant hallucination issues. Therefore, I wish to see more analysis and discussion on how to correct language representation failures, rather than always relying on an erroneous, discrete, and hallucinatory representation. I'm not questioning your argument.

  ... language-based features can help the visual perception module for the VLN agent.

• I concur with this point of Reviewer z42V, i.e., "a good paper to motivate future work in this direction". Besides this point, I agree with the authors' rebuttal and strength 1 of Reviewer 2DEF.

  • Given that no new models or methods are developed, the key contributions of this paper are relatively light.
  • Overall, I think this is a good workshop paper to motivate future work in this direction, but in itself, does not meet the bar as a conference paper.

Weaknesses:

• The biggest weakness of this paper, I agree with the weakness 4 of Reviewer 2DEF.

  I believe one of the most important reasons for introducing LLMs to embodied AI tasks is to leverage LLMs' exceptional language understanding, task planning, tool-using, explicit reasoning with commonsense knowledge, communication, etc., abilities...

I still believe that it is necessary to explore and analyze it (as mentioned in strength 2) in the future. I expect it to get better language perceptual representations, rather than directly applying it to navigation task.

Overall, this paper can stimulate discussion and bring new ideas to the VLN community, supported by substantial work. I decided to maintain my rating as "6: marginally above the acceptance threshold".

Thanks for the review!

Gap in converting vision into language

We agree! To see whether (usual) continuous vision features can rectify close this gap, we conducted additional experiments where we augment a traditional vision-based navigation model with language features. More specifically, the original RecBert uses ResNet-152 to extract the visual feature to represent the image, while our extension additionally concatenates the language feature of the caption to be the image representation. The captions are the same as what are used in LangNav and the features are extracted by BERT-base. We pre-train and fine-tune the RecBert and its extension with 100-shot training trajectories and full R2R train set. The results are listed in the tables below.

We can see that the language features improve the performance in both few-shot and full learning cases, which indicates that language-based features can help the visual perception module for the VLN agent. We hope that this addresses the reviewer's concerns the domain gap that may result in converting vision into language.

Q1: Could you please provide a comparison between the latest COT technology on GPT-4 in terms of language representation (for instance, Graph of Thoughts: Solving Elaborate Problems with Large Language Models, not just NavGPT)...

Thanks for the question! Incorporating more sophisticated reasoning techniques such as Graph of thought and tree of thought is an interesting suggestion. However, we note that it is not at all clear how one would apply these more sophisticated prompting techniques to the navigation case (as of now). Note that NavGPT already uses chain-of-thought, and hence we do compare against a strong COT-based baseline. We will discuss this more in the next version of the paper.

We thank the reviewer 2DEF for appreciating our work!

(1) Focus of our paper: We agree that more focus should be placed on the language-based representation itself. We note that our two case studies (i.e., efficient synthetic data generation and ALFRED-to-R2R transfer) actually are explicitly leveraging the fact that language can serve as an efficient and robust representation of one's perception. However, we agree that this aspect of our work should be emphasized, and will clarify this point more in the paper.

Moreover, your suggestion of aligning our focus with the benefit of using language as representation inspired us to perform a third "case study", where we experiment to see whether language-based representation allows us to interpret and edit our policy. Concretely, we randomly pick 10 R2R trajectories where the LangNav fails to choose the correct direction, like the second example in Figure 4 in the paper. For the step where the LangNav acts wrong, we manually correct the incorrect captions using visual analysis and the given instructions. We provide three examples that demonstrate that, by merely correcting the incorrect captions, LangNav is able to change the decision from incorrect to correct. The original trajectory examples are in the link to original trajectories and the modified trajectory examples are in the link to modified trajectories, where we mark the modified caption in bold. We applied this procedure to 10 selected R2R trajectories and discovered that our LangNav was able to revise its decision correctly in 70% of these trajectories by only rectifying incorrect captions. For the other 30% examples, the failure reason doesn't fall into the vision captions in the current step. In Example #3, the agent decides to continue moving forward in Step 2 since it does not recognize from its historical data that it has already ascended the stairs. We thank the reviewer for suggesting these experiments. We will include these results in our next version of the paper.

(2) How far can this method (of using language alone) go? In the response to (1), we show that we can manipulate the vision captions in the prompt and discover that inaccurate visual perceptions contribute significantly to the failure cases, which indicates that although the current VLM system is not yet perfect and the performance of our LangNav is influenced by the inaccurate, not comprehensive, and ambiguous captions, we remain optimistic that future improvements in vision and text models will greatly enhance LangNav's long-term performance, especially with the great potential of VLM we see from the GPT-4V.

(3) Emerging trend of developing visual-LLMs (VLMs): We agree that visual-LLMs could preserve more visual details compared to the language-only representations. However, using language-based representation is not contradictory to the continuous visual features. To demonstrate how the language can help on top of continuous visual features, we performed further experiments where we extend the RecBert by concatenating language features to the visual features to represent the candidate image. More specifically, the original RecBert uses ResNet-152 to extract the visual feature to represent the image, while our extension additionally concatenates the language feature of the caption to be the image representation. The captions are the same as what are used in LangNav and the features are extracted by BERT-base. We pre-train and fine-tune the RecBert and its extension with 100-shot training trajectories and full R2R train set. The results are listed in the tables. We can see that the language features improve the performance in both few-shot and full learning cases, which indicates that language-based features can help the visual perception module for the VLN agent.

These results indicate that language as a perceptual representation can provide additional benefits on top of continuous visual features. We thank the reviewer for suggesting this experiment and will include it in the next version of the paper.

(4) Fair comparison to RecBert and justification for the performance potential of LangNav: To fairly compare the performance of our LangNav with RecBert and justify the performance potential of our LangNav, we present the results of evaluating both LangNav and RecBert which are trained on full R2R train split. We pre-train and fine-tune the RecBert only on R2R train split without any synthetic data (neither PREVALENT data nor GPT-4 synthetic trajectories). The results are listed in the table. It is observed that LangNav still lags behind RecBert in performance. Nonetheless, we attribute this gap primarily to the mismatch between visual and textual data. As highlighted in the response to point (1), the role of unclear and imprecise captions in causing failures is considerable. Therefore, we maintain a positive outlook on LangNav's future capabilities, anticipating that advancements in vision and text processing technologies will significantly boost LangNav's efficacy in the long run.

(5) Commonsense for navigation: We agree that the capabilities of LLMs are the primary reason why we wish to leverage them and their language-based representations. To illustrate the commonsense reasoning ability of the LLM used in LangNav, we have visualized the decision-making process in a specific navigation example from LangNav. For each step, we display an image showing the direction the agent faces. From Step 2 to Step 3, the agent demonstrates commonsense reasoning. Upon receiving the instruction, 'turn left, walk forward, and then turn left,' the agent interprets and integrates these directions, ultimately making a decision equivalent to 'turning left back,' as demonstrated in the example.

I want to thank the authors for the extremely detailed response to my comments. I really appreciate the great efforts in delivering additional experiments. Many of my questions have been nicely addressed. After reading all the reviews and responses, I decided to keep my rating as Accept. Meanwhile, I hope the authors can experiment and discuss more my major concern of

I believe one of the most important reasons for introducing LLMs to embodied AI tasks is to leverage LLMs' exceptional language understanding, task planning, tool-using, explicit reasoning with commonsense knowledge, communication, etc., abilities.

However, it is unclear in this paper whether LangNav shows any of the above abilities (e.g., using commonsense to navigate).
It is likely that after tunning LLaMA for VLN, the model lost these abilities, and LLaMA is simply applied as a good initialization for learning the downstream R2R task.

We are glad that our rebuttal has addressed most of the reviewer's concerns! To show more about the ability of LangNav, we show three more examples attached in LangNav's Navigation.

In Example #1, we have observed that, although the instruction of where to stop is vague and ambiguous ("Wait at the entrance."), our LangNav agent manages to stop in front of the double wooden doors which is the most likely to be the entrance.

In Example #2, from Step 2 to Step 4, our LangNav agent erroneously enters a bathroom on the right in Step 2, but then corrects its course by leaving the bathroom and proceeding to the living room. In contrast, we have also visualized the trajectory (Example #2) of RecBert under the same instructions in RecBert's Navigation, where we can see that, RecBert misinterprets the instruction "walk up the stairs" too literally and ascends any staircase it encounters, including the one in Step 4, showing a fundamental misunderstanding of the intended navigation path.

In Example #3, we can see that our LangNav can generalize to the outdoor scenario, given that all of the training trajectories are sampled from the indoor scenarios. On the other hand, in the same example (as seen in RecBert's Navigation), RecBert, following identical instructions, becomes trapped in an alley, pacing back and forth. This comparison highlights LangNav's superior innate understanding that the most effective strategy in an alley is to continue straight ahead to find a way out.

We hope the above examples could address the reviewer's concern!

We would like to thank Reviewer z42V for these valuable comments.

(1) Comparison beween synthetic data and realistic data: To address the reviewer's corcern about the quality of the synthetic data, we conduct an experiment where we prepare 93 training trajectories for both real-world data and synthetic data, and finetune the LLaMA2-7B model. In order to control the scene constraint, the 93 real-world trajectories all originate from the same scan used to sample the 10 trajectories for prompting LLMs. The evaluation results are shown in the table. The evaluation shows that, under the same constraints of the training scene diversity, synthetic trajectories have comparable or even better quality with the real-world trajectories. We infer the gain of the synthetic data is from the more accurate language description of the scene and the higher abundance of the object concepts due to GPT-4's hallucination.

(2) Contributions of the paper: We respectfully disagree with the reviewer's assessment that our paper makes a relatively light contribution due to the lack of development of new models or methods. We would like to remind the reviewer that our core contribution is not a model or method (since, as the reviewer notes, we make use of existing methods/models); rather, our core contribution is to suggest that language itself can serve as a perceptual representation for navigation especially in low-data regimes, which has been underexplored compared to uses of language in other embodied tasks. We conduct two experiments that exploit the use of language via (1) synthetic data generation and (2) sim-to-real transfer to improve VLN.

Moreover, we conducted additional experiments where we augment a traditional vision-based navigation model with language features. This is a "new model" insofar as existing VLN systems only rely on visual perception features. More specifically, the original RecBert uses ResNet-152 to extract the visual feature to represent the image, while our extension additionally concatenates the language feature of the caption to be the image representation. The captions are the same as what are used in LangNav and the features are extracted by BERT-base. We pre-train and fine-tune the RecBert and its extension with 100-shot training trajectories and full R2R train set. The results are listed in the tables below.

We can see that the language features improve the performance in both few-shot and full learning cases, which indicates that language-based features can help the visual perception module for the VLN agent. We hope that this addresses the reviewer's concerns about the modeling contribution of the work.

(3) Reiterating the differences between RecBERT and LangNav models: We thank the reviewer for the suggestion. We will reiterate the detailed differences between RecBERT and LangNav models in the revised version of the paper, ensuring that the distinctions are clearly outlined.

(4) End-to-end learning setup with vision-to-text module: We think that end-to-end learning for the vision-to-text module would be an interesting direction to explore! With these settings, the vision-to-text module can benefit from LM policy and learn to focus more on navigation-relevant details. To investigate how a better vision-to-text module can help improve the VLN performance, and interpret the failure reason, we randomly pick 10 R2R trajectories where the LangNav fails to choose the correct direction, as illustrated by the second example in Figure 4 of the paper. For the step where the LangNav acts wrong, we manually correct the incorrect captions using visual analysis and the given instructions. We provide three examples which demonstrate that, by merely correcting the incorrect captions, LangNav is able to change the decision from incorrect to correct. The original trajectory examples are in the link to original trajectories and the modified trajectory examples are in the link to modified trajectories, where we mark the modified caption in bold. We applied this procedure to 10 selected R2R trajectories and discovered that our LangNav was able to revise its decision correctly in 70% of these trajectories by only rectifying incorrect captions. The results indicate that a navigation-oriented vision-to-text module is essential for VLN tasks. End-to-end learning is a promising way to learn such a model.

(5) Methods like RLHF to make the model robust to deviation: Alongside the strategy outlined in Section 3.3, we conducted an external experiment using the DAgger algorithm for imitation learning. This process involves utilizing a trained language model (LM) policy to infer trajectories within the training environment. These inferred trajectories are then combined with the existing training dataset to form a new training set. The foundational concepts of the DAgger algorithm and the reviewer's proposed strategy bear a strong resemblance. In our comparison table, we illustrate the performance differences among the baseline LM policy, the LM policy enhanced through DAgger, and our method designed to address deviations. The findings demonstrate that although DAgger substantially improves VLN performance, the approach we developed, as described in Section 3.3, ultimately yields superior results over the DAgger algorithm.

Dear Reviewer z42V,

Thank you for your constructive comments and suggestions. As the discussion phase is nearing its end, we wondered if you might still have any concerns that we could address. We believe our response addressed all your questions/concerns, and hope that the work's impact and results are better highlighted with our responses. Thank you!

Best wishes,

Authors

The detailed responses provided by the authors to all the reviews is greatly appreciated. After going through the author responses as well as other reviewers feedback, there are several outstanding concerns with the main hypotheses of the paper. While the additional experiments conducted by the authors are appreciated, they do not change the position of the paper which has been correctly called into question in other reviews. In response to the first weakness, authors conducted additional experiment using 93 training trajectories. While the author efforts are commendable, it is not clear how much conclusions we can draw from such a small scale experiment. Overall, I am hesitant to change my rating from "3: reject, not good enough".

(5) Exploration of visual perception: We strongly agree with the reviewer that more exploration in more accurate vision captions is needed to understand the system more deeply. As shown in third case study we conducted in (1), we show that we can manipulate the vision captions in the prompt and discover that inaccurate visual perceptions contribute significantly to the failure cases, which indicates that although the current VLM system is not yet perfect, we remain optimistic that future improvements in vision and text models will greatly enhance LangNav's long-term performance.

To further explore how the language can help on top of continuous visual features, we performed further experiments where we extend the RecBert [9] by concatenating language features to the visual features to represent the candidate image. More specifically, the original RecBert uses ResNet-152 to extract the visual feature to represent the image, while our extension additionally concatenates the language feature of the caption to be the image representation. The captions are the same as what are used in LangNav and the features are extracted by BERT-base [10]. We pre-train and fine-tune the RecBert and its extension with 100-shot training trajectories and full R2R train set. The results are listed in the tables. We can see that the language features improve the performance in both few-shot and full training set cases, which indicates that language-based features can help the visual perception module for the VLN agent.

These results indicate that language as a perceptual representation can provide additional benefits on top of continuous visual features. We thank the reviewer for suggesting this experiment and will include it in the next version of the paper.

Q1. What are "gold" trajectories? The gold trajectories represent the optimal or expert-defined shortest path from the start point to the goal, as provided by the environment.

Q2. How is the quality of the synthetic trajectory generated by LLMs? The quality of the synthetic trajectory generated by LLMs is quite good. Quantitatively, we can see from the results in the response to (3) that within the same constraints, the synthetic trajectory is comparable to the real R2R trajectory. Qualitatively, as we demonstrated in the examples in Figure 3 and Section H, the synthetic trajectories are informative. The generated trajectories follow the desired instruction and exhibit consistent spatial layouts in each step. As we illustrated in Section 4.1.1, the synthetic trajectories have a strong prior knowledge base, spatial consistency, and are descriptive.

Q3. How to apply the random approach to the synthetic trajectory? Great question! The random approach is only applied to the R2R trajectories, and not the synthetic trajectories.

[8] Côté M A, Kádár A, Yuan X, et al. Textworld: A learning environment for text-based games.

[9] Hong, Yicong, et al. Vln bert: A recurrent vision-and-language bert for navigation.

[10] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding.

(3) Analysis on quality of synthetic data: Thanks for pointing this out. We infer that the fast convergence of synthetic data, compared to the full realistic dataset, is due to the use of only 10 real trajectories from a single scene to prompt LLMs.

For example, here are four generated instructions from the LLM:

1. Start from the main entrance door, pass the living room, and enter the kitchen on your right. Locate the refrigerator, then turn left and stop just before the dining table.

2. Navigate from the couch in the living room, move towards the mantel, and then stop next to the fireplace. Avoid any furniture and obstacles on your path.

3. Begin at the foot of the bed in the master bedroom. Walk forward and enter the attached bathroom. Once you're inside, stop next to the bathtub.

4. Start in the family room, walk towards the TV, then turn right and pass the bookshelf. Stop when you reach the large bay window overlooking the garden.

We can see from the above synthetic instructions that (1) patterns of the synthetic instructions are similar, which are like "Start from place A, go pass place B, stop at place C", (2) scenes are limited to the living area and a single floor, however, the R2R tasks always require the agent navigating across floors and in some non-living area.

In order to fairly compare the quality of the synthetic data and the real-world data, we conduct an experiment where we prepare 93 training trajectories for both real-world and synthetic data, and finetune LLaMA2-7B. The real-world trajectories all originate from the same scene used to sample the 10 trajectories for prompting LLMs. The evaluation results are shown in the table. The evaluation shows that, under the same constraints of the training scene diversity, synthetic trajectories have better quality than real-world trajectories. We infer the gain is from the more accurate and diverse language description of the scene.

This indicates that the underperformance of the generated trajectories compared to real trajectories is due to scene diversity. To further investigate the influence of the scene diversity, we conduct an experiment where we use 1000 navigation instructions sampled from various R2R scenes to prompt GPT-4-turbo to generate 2000 synthetic trajectories. The scaling results are listed in the table. We can see that although the 2000 trajectories generated by GPT-4-turbo are not of the same quality as those generated by GPT-4, scaling up using these trajectories outperforms the results from the 10000-trajectory set.

(4) Novelty in sim2real transfer: We understand that LangNav is not the first work in learning high-level embodied knowledge through language. However, the differences between our sim2real experiments and [6,7] are fundamental.

For [6], the agent learns the text policy from TextWorld [8] (which is extended to analog the ALFRED scenes) and then evaluates it in the ALFRED environment [9]. The learning and testing are in the two views (text & vision) of the same underlying world. While our sim2real experiments exhibit the ability of transferring knowledge between two very different underlying worlds: R2R and ALFRED. The significant differences are demonstrated in Figure 5.

For [7], the agent learns to transfer the knowledge between two different game environments (Boulderchase and Bomberman). There are two main differences between their work and our sim2real transferring. First of all, the Boulderchase and the Bomberman are not designed for navigation tasks, and their environments are more simplistic and gamified compared to our testbeds (ALFRED and R2R). Thus, the difficulty level of transferring policies is different. Secondly, the transferring settings are different. In Section 3.4 of [7], it's noted that the transferred policy can further interact with and learn from the full new environment. While in our settings, we either do not allow or restrict the policy learned in ALFRED to only a few scenes in R2R.

We want to thank Reviewer PNyi for these valuable comments.

(1) Focus of our paper: We agree that more focus should be placed on the inherent merits of the language-based representation. We note that our two case studies (i.e., efficient synthetic data generation and ALFRED-to-R2R transfer) actually are explicitly leveraging the fact that language can serve as an efficient and robust representation of one's perception. However we agree that this aspect of our work should be emphasized, and will clarify this point more in the paper.

Moreover, your suggestion of focusing on the inherit merits of language inspired us to perform a third "case study", where we experiment to see whether language-based representation allows us to interpret and edit our policy. Concretely, we randomly pick 10 R2R trajectories where the LangNav fails to choose the correct direction, like the second example in Figure 4 in the paper. For the step where the LangNav was incorrect, we inspected the captions, and when the captions were incorrect, we manually corrected them . We provide three examples which demonstrate that, by merely correcting the incorrect captions, LangNav is able to change the decision from incorrect to correct. The original trajectory examples are in the link to original trajectories and the modified trajectory examples are in the link to modified trajectories, where we mark the modified caption in bold. We applied this procedure to 10 selected R2R trajectories and discovered that our LangNav was able to revise its decision correctly in 70% of these trajectories. For the other 30% examples, the failure reason doesn't fall into the vision captions in the current step. For example in Example #3, the agent decides to continue moving forward in Step 2 since it does not recognize from its historical data that it has already ascended the stairs.

This case study highlights another benefit of working in language space---i.e., improved interpretability and editability. And we expect that LangNav will be able to leverage independent advances in vision-only and text-only models. We thank the reviewer for inspiring this experiment and will include it in the paper.

(2) Testing in exotic environments: We thank the reviewer for pointing this out! We agree that the potential application in some exotic environments would be a strong motivation for LangNav. To validate that LLMs can synthesize useful data in more exotic environments, we conduct an experiment where we handcraft a trajectory in a real office environment and then prompt GPT-4 to generate synthetic trajectories within the scope of the office environment. Here are the sampled real trajectory and two generated synthetic trajectories: link to real/synthetic trajectories in office environments.

We can see that the synthetic trajectories (1) contain abundant common object-scene correlations in office environments, (2) exhibit great spatial consistency (example), and (3) incorporate a significant amount of captions and objects that do not directly relate to the given instruction. In summary, the LLM is capable of generating synthetic trajectories in the office environment with same quality to the household environment in Sec. 4.1.1. However, due to the lack of actual benchmarks for these types of exotic environments, we can't provide quantitative results beyond the household environment during this phase. We nonetheless think this is an interesting experiment and will update the paper with these qualitative results on exotic environments.

Dear Reviewer PNyi,

Thank you for your constructive comments and suggestions. As the discussion phase is nearing its end, we wondered if you might still have any concerns that we could address. We believe our response addressed all your questions/concerns, and hope that the work's impact and results are better highlighted with our responses. Thank you!

Best wishes,

Authors

We would like to extend our sincere thanks to Reviewer PNyi for their timely reply before the deadline.

In response to the reviewer's feedback, we acknowledge the need to enhance the clarity of our paper's focus and hypothesis. We aim to adjust our emphasis in the updated manuscript to highlight that:

1. Utilizing language as a perceptual representation is advantageous in the low-data setting due to its efficiency in synthesizing data with LLMs and its adaptability across various environments.

2. In contexts where training data is abundant, employing language as a perceptual representation not only maintains its efficacy but also amplifies visual perception capabilities, as demonstrated in our response to weakness (5).

3. The use of language as a perceptual representation offers superior interpretability, allowing for the editing and understanding of language to deduce reasons for failures.

We plan to more clearly articulate these points in the revised version of our paper, hoping this refined emphasis will adequately address the concerns raised by the reviewer.

We would like to thank Reviewer Sthf for these valuable comments.

(1) Incomplete visual/spatial information: This is an interesting point, and we agree with the reviewer's concern regarding the influence of incomplete visual/spatial information in the real world. However, the fact that we obtain nontrivial performance when we transfer a policy trained on ALFRED to R2R (even with zero R2R examples) indicates that the LangNav agents do not overfit to the captions/objects from the domain, and are able to generalize the navigation knowledge across domains even in the presence of incomplete visual/spatial information. Moreover, we show below (in our response to (2)) that even in the presence of continuous visual features, language captions can still improve performance.

(2) Applicability to real-world robots: We acknowledge that real-world scenarios are much more complex than the R2R environment. Beyond the more complex and varied scene configurations, objects in the real world are also dynamic, rather than static. Because of this, we strengthen our belief that language is an important perceptual representation to bridge the gap (a view that both Reviewer 2DEF and Reviewer PNYi agree with). Our Case Study 2 also demonstrates that combining the high-level abstraction characteristic of language with the common-sense reasoning ability of LLMs is a more promising way to transfer knowledge between different environments.

On the other hand, we also find that language-based features can also help the traditional visual perception module of the navigation agent. To further explore how the language can help on top of continuous visual features, we performed further experiments where we extend the RecBert [1] by concatenating language features to the visual features to represent the candidate image. More specifically, the original RecBert uses ResNet-152 to extract the visual feature to represent the image, while our extension additionally concatenates the language feature of the caption to be the image representation. The captions are the same as what are used in LangNav and the features are extracted by BERT-base [10]. We pre-train and fine-tune the RecBert and its extension with 100-shot training trajectories and full R2R train set. The results are listed in the tables. We can see that the language features improve the performance in both few-shot and full training set cases, which indicates that language-based features can help the visual perception module for the VLN agent.

These results indicate that language as a perceptual representation can provide additional benefits on top of continuous visual features. We will include these results in the paper.

(3) Sim2sim transfer: We thank the reviewer for pointing this out! We agree that our experiments are still conducted within a simulated environment. We will change the name to 'Sim-to-Sim' (or 'domain transfer') in our subsequent version.

Q1 Does the model's performance vary with longer or shorter episodes, and if so, how?

Yes, the model's performance varies with the length of the episode. We evaluated the LangNav model, which was fine-tuned on 2,000 GPT-4 synthetic trajectories and 100 real-world trajectories, to calculate the success rate (SR) and navigation error (NE). The results are listed in the table. As can be seen from the table, as the episode length increases, the task becomes more complex, and the navigation performance decreases, which aligns with our intuition.

We would like to thank the reviewer for expressing their concerns!

For (1), we think the reliance on off-the-shelf object detectors/image captioning models is actually core feature of our approach. Since we disentangle the the image module and the language module, our approach can make use of independent advances in vision and language systems. As language models continue to improve (and as vision models continue to improve), LangNav can automatically (and immediately) make use of these advances. Also as noted by Reviewer 2DEF, language-guided visual navigation enjoys LLMs' generalization power and bridges the visual domain gap. Furthermore, it largely reduces the cost of generating synthetic navigation trajectories.

For (2), we agree that the application of our LangNav system in realistic scenarios is essential! We actually conducted additional qualitative analysis on whether hallucinated text in non-standard environments can produce spatially grounded text. We indeed find that this is the case. See here for the examples, where we sampled a seed trajectory in a realistic office environment and then prompted GPT-4 to generate synthetic trajectories. We find that the synthetic trajectories are spatially consistent when attempting to draw the layout of the described environments (as also noted by the reviewer 2DEF). On the other hand, the R2R environment, despite being a simulation, is derived from actual household settings, complete with realistic image panoramas. [1] has shown that the policy learned in R2R could be adopted to the realistic environment.

Do the above points change your view of the contribution at all?

[1] Anderson et al. Sim-to-Real Transfer for Vision-and-Language Navigation.

Dear Reviewer Sthf,

Thank you for your constructive comments and suggestions. As the discussion phase is nearing its end, we wondered if you might still have any concerns that we could address. We believe our response addressed all your questions/concerns, and hope that the work's impact and results are better highlighted with our responses. Thank you!

Best wishes,

Authors