General Scene Adaptation for Vision-and-Language Navigation

3.2 Comparison with TTA

Our work is more closely related to TTA, as we both aim to adapt the agent during inference without additional supervision. Several TTA methods, such as TENT and SAR, are included as baselines in our experiments to evaluate their applicability to the GSA-VLN task. While TTA provides a viable solution to the scene adaptation problem, our approach goes beyond TTA by incorporating memory-based methods, where model parameters remain fixed during adaptation but the input varies based on the dynamically updating memory bank. This integration of memory-based methods demonstrates the broader applicability and versatility of our setting compared to TTA.

To further clarify this, we have added Section A.9 in the revised manuscript to discuss the relationship between our work and these two related areas. Thank you for allowing us to improve our manuscript.

 

[1] Liu, Bing. "Lifelong machine learning: a paradigm for continuous learning." Frontiers of Computer Science 11.3 (2017): 359-361.

---

Weakness 4. Benchmark Consistency

Thank you for your valuable suggestion. We appreciate the opportunity to clarify the consistency of model performance across datasets and the rationale behind the presentation in Table 3.

4.1 Purpose of Table 3

Table 3 is primarily designed to illustrate the unique characteristics of our GSA-R2R dataset, including:

1. Scene Diversity: The comparison between residential and non-residential scenes.
2. Instruction Types: The impact of different instruction styles (e.g., Basic, Scene, and User).

Given that other VLN benchmark datasets, such as R2R, focus only on residential scenes with basic instructions, their evaluation conditions align with our Test-R-Basic split in GSA-R2R. For this reason, we initially chose not to include performance on other datasets in Table 3.

4.2 Revisions to Enhance Completeness

We agree that adding the R2R performance of each baseline can enhance the completeness of the comparison. Since we have already referenced these results in Line 428, we have now added a column to Table 3 in the revised manuscript to present the R2R performance alongside GSA-R2R results for comparison.

4.3 Performance Across Datasets

Comparing the performance of R2R and our GSA-R2R, we provide the following observations:

• Ranking Consistency: The ranking of baseline performance is consistent across datasets, with ScaleVLN achieving the highest scores and HAMT the lowest.

• Large Performance Gap: Specific performance numbers are significantly lower on GSA-R2R compared to R2R. This performance drop reflects the increased complexity of GSA-R2R, which includes more diverse scenes, complex paths, and challenging instructions. These results highlight the additional challenges posed by GSA-R2R and its value in evaluating methods under more realistic and diverse conditions.

We hope these revisions and clarifications address your concerns and provide a more complete understanding of the benchmarks used in our study. Thank you again for your constructive feedback.

We greatly appreciate Reviewer J7gj for the thorough review of our paper and for providing highly constructive comments. Our responses are detailed below.

---

Weakness 1. Adaptation Novelty

We acknowledge that the concept of adaptation has been explored in embodied environments, and we have discussed related works in the second and third paragraphs of the Related Work section. However, scene adaptation remains an underexplored area within VLN, and our work seeks to address this gap.

1.1 How Our Work Differs from Related Efforts

The only directly related work in VLN is Iterative VLN (IVLN), which we have discussed in Lines 51–53 and 73–78 of the manuscript. Our approach differs from IVLN in two key ways:

1. Enabling Both Memory-Based and Adaptation-Based Methods: We include a significantly larger number of instructions per scene, allowing agents to leverage both memory-based and adaptation-based methods for scene-specific tasks.
2. Introduction of OOD Data: While IVLN only utilizes the original R2R dataset, which is limited in scene and instruction diversity, our work incorporates both ID and OOD data which can better evaluate agent adaptability and align more closely with real-world scenarios.

1.2 Novel Contributions of Our Work

Although the concept of adaptation itself is not entirely new, our work introduces several novel contributions that address previously overlooked aspects of VLN research:

1. General Scene Adaptation Task: We are the first to propose the general scene adaptation task for VLN, addressing both ID and OOD scenes and instructions.

2. Largest Indoor VLN Dataset: Our GSA-R2R dataset is the largest indoor VLN dataset to date, comprising 150 scenes and 90K instructions (Table 1 of the paper).

3. Diverse Environments and Instructions: We expand the range of environments by incorporating 20 scene types, including both residential and non-residential buildings. We also introduce diverse speaking styles, opening a new research direction in VLN that investigates the impact of instruction diversity on navigation performance.

We believe these contributions are critical for advancing VLN research and provide a foundation for future work in this area.

---

Weakness 2. Related Work Scope

Thank you for your valuable suggestion. We appreciate your feedback regarding the structure and scope of the Related Work section.

2.1 Reason for Original Structure

Since our primary focus is on proposing the GSA-VLN task and the GSA-R2R dataset, we initially deferred detailed descriptions of related adaptation methods to Section 4 alongside the introduction of our GR-DUET method. Similarly, key differences and contributions were outlined in Lines 363–367 within the method section to maintain consistency with the experimental context.

2.2 Revisions Made

However, we agree that discussing related methods and their differences from our approach should be included in the Related Work section to enhance clarity and accessibility. Therefore, we have reorganized the structure of the paper with two modifications:

1. We have integrated a detailed discussion of adaptation methods and memory-based approaches in VLN, including how they address environment-specific challenges and their limitations in our task.
2. We explicitly highlighted the key differences between our proposed method and existing memory-based works, including the global topological map for enhancing spatial understanding of history and the specialized training strategy for addressing the input distribution shift problem.

We hope these changes address your concern and enhance the paper’s accessibility. Thank you again for your constructive feedback.

---

Weakness 3. Lifelong Learning and TTA

Thank you for your insightful comment. Our work indeed shares certain similarities with Lifelong Learning and Test-Time Adaptation (TTA), as all these settings involve continuous learning. However, there are distinct differences that set our approach apart.

3.1 Comparison with Lifelong Learning

Lifelong Learning focuses on acquiring new skills or tasks over time while preserving the ability to perform previously learned tasks [1]. In contrast, our setting involves an agent adapting to a single scene over its lifetime. Rather than acquiring and retaining multiple skills or tasks, our work emphasizes repeated mastery of the same navigation skill within a scene-specific context. This makes our approach distinct from the broader objective of lifelong learning.

We thank Reviewer J6Xq for taking the time to review our paper and for the constructive feedback provided. Please refer to the following responses to address your comments.

---

Weakness 1. LLM-based Baselines

Thank you for the suggestion to include LLM-based methods as baselines.

1.1 Existing LLM-based Baselines in Our Experiments

Our experiments already include NavGPT2, a recent LLM-based method from ECCV 2024. As shown in Table 3 of the paper, NavGPT2's performance reveals that LLM-based methods, when applied in a zero-shot manner without adaptation techniques, do not exhibit significant advantages for the scene adaptation problem.

1.2 Adding More LLM-based Baselines

We are happy to include additional LLM-based methods in our evaluation. While InstructNav is a relevant VLN method, it operates in continuous VLN settings with low-level actions (e.g., "move forward" or "turn around"). Our GSA-VLN task, on the other hand, focuses on discrete environments. This fundamental difference makes a direct comparison challenging. Therefore, we incorporate another recent LLM-based method, MapGPT [1], which has demonstrated strong zero-shot capabilities, along with NavCoT, for evaluation. Due to the scale of our GSA-R2R dataset, evaluating proprietary LLM-based methods on the entire dataset is computationally expensive. To address this, we sampled one environment for each type of instruction to conduct meaningful comparisons. The results are summarized in Table 1 below:

Table 1: Comparison of LLM-based methods and GR-DUET on GSA-R2R.

The results reveal that LLM-based methods perform poorly on the GSA-R2R task compared to GR-DUET. While LLMs can handle instructions with different styles, they struggle with the environmental adaptation required for GSA-VLN, particularly in processing visual information and interacting with persistent environments.

We believe one promising direction for future research would be adapting LLM-based methods to specific environments using the memory bank provided by GSA-VLN. Techniques such as Retrieval-Augmented Generation (RAG) or Parameter-Efficient Fine-Tuning (PEFT) may enhance the ability of LLM-based methods to handle scene-specific challenges in tasks like GSA-VLN.

Thank you again for the suggestion, and we have included these results and discussions in Section A.6 of the revised manuscript.

 

[1] Chen, Jiaqi, et al. "Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation." ACL. 2024.

---

Weakness 2. More Diverse Styles

Thank you for your suggestion regarding the diversity of character styles. We would like to address a potential misunderstanding and clarify the scalability of our approach.

2.1 Scalability of Our Method

We do propose a method for generating instructions for different character styles, which can be highly scalable. As detailed in Lines 885–894 of the paper, our approach can be extended to generate instructions for thousands of characters. Specifically, by utilizing the SummScreen dataset, our method can scale up to approximately 3,000 unique characters, which we believe provides sufficient diversity for research purposes.

2.2 Why We Selected Five Characters

For this study, we selected five characters with highly distinct speaking styles, ensuring diversity in age, gender, and language patterns. The reasons for this choice are as follows:

1. Representativeness: The five selected characters represent a wide range of user styles, enabling us to effectively evaluate whether adaptation methods can handle significant variations in user instructions.
2. Practicality: Adding more characters would not necessarily provide additional insights into the core problem, as the existing selection already covers distinct and challenging styles. We believe that the results on these five characters are sufficiently representative to showcase the instruction adaptation abilities of different methods.

2.3 Generating Infinite Styles

We appreciate the suggestion of generating instructions in an infinite number of ways, which aligns with our initial considerations. To this end, we have explored persona-based methods, such as the one proposed in [1], which can generate up to 1 billion personas. However, as described in Lines 856–863, these methods typically provide brief background descriptions of personas without detailed dialog histories. This often leads to generated instructions containing irrelevant or unrealistic content. In contrast, our approach uses detailed dialog profiles as prompts, ensuring that the generated instructions are both realistic and contextually appropriate. Our method prioritizes instruction quality and usability over quantity.

2.4 Code Release

We will release all the code for our instruction generation pipeline. This includes the character-based method as presented in our work, and the persona-based method we explored, enabling further exploration of diverse instruction styles. By making this code available, researchers can generate a broader variety of character styles and customize instructions to suit their specific needs and applications.

We agree that expanding our work to include even more diverse instruction styles would be a valuable direction for future research. We are excited to explore such possibilities and thank you for the constructive suggestion.

 

[1] Ge, Tao, et al. "Scaling synthetic data creation with 1,000,000,000 personas." arXiv preprint arXiv:2406.20094 (2024).

---

Weakness 3. GR-DUET Details

We apologize for the lack of sufficient details regarding the baseline DUET model, which may have caused difficulties for readers unfamiliar with this method. While Section 4.1 provides a brief overview of DUET’s key ideas, we acknowledge that this explanation may not be detailed enough to fully convey DUET’s architecture and its integration into our GR-DUET method. To address this, we have added a comprehensive description of DUET in Section A.1.1 of the revised manuscript, including:

• Model architecture: Key components and design principles.
• Functionality: How DUET processes textual and visual information individually and then combine them for cross-modal reasoning.
• Relevance to GR-DUET: How DUET forms the foundation of our approach and is adapted within GR-DUET for enhanced performance.

We believe these additions will make the manuscript more accessible to readers unfamiliar with DUET, ensuring they have the necessary background to fully understand our contributions. Thank you for pointing out this gap and giving us the opportunity to improve the clarity of our paper.

---

Question 1. Human Study Details

Thank you for raising this point. We provide the following clarifications regarding the participants and methodology of our human study:

Participant Demographics

• The study included 15 participants, comprising university students and staff aged between 20 and 35 years old.
• The participants represented a diverse range of genders and backgrounds, ensuring a variety of perspectives while maintaining a degree of homogeneity necessary for unbiased evaluations.

Competence of Participants

The task given to participants was straightforward and did not require specialized knowledge:

• Task 1: Determine whether an instruction aligns with the corresponding trajectory.

• Task 2: Assess whether the instruction exhibits a distinct speaking style.

Given the simplicity of these tasks, we are confident that our participants were competent to perform the evaluation accurately and reliably.

Visualizations

In Lines 1073–1079 of the manuscript, we include visualizations of the GSA-R2R dataset, which were also shown to the participants during the evaluation. Since OpenReview does not support images in the response, to further ensure transparency, examples of the user interface used in the human study are now included in Section A.8 of the revised manuscript.

Question 2: Why LLM Fails

Thank you for raising this question. Regarding the statement, "since the dataset is generated by LLM, why doesn't LLM perform well on it?", we would like to clarify and address this potential misunderstanding.

2.1 Our Dataset is Not Entirely Generated by LLM

The GSA-R2R dataset includes novel environments and diverse instructions. While part of the instructions are generated with the help of LLMs, the generation process is rooted in speaker-generated basic instructions. The LLMs used in our pipeline are not reasoning from scratch but transforming existing instructions into different styles based on user-specific prompts and context. Thus, the dataset is not purely LLM-generated and represents a broader mix of styles and real-world challenges.

2.2 Instruction Following vs. Instruction Generation

The observed underperformance of LLMs on this dataset stems from the fundamental difference between instruction transformation and instruction-following navigation. Transforming instructions into different styles relies primarily on understanding textual characteristics, such as user preferences and professional speaking habits—a task well-suited to LLMs. In contrast, instruction-following navigation involves multi-modal understanding and reasoning, such as interpreting spatial contexts, visual inputs, and sequential tasks, which are significantly more complex and have not been solved by current LLM-based VLN methods, including InstructNav. This distinction highlights the unique challenges posed by the GSA-VLN task, which cannot be addressed by instruction transformation alone. Moreover, our dataset includes not only the instruction adaptation problem but also the environment adaptation, which is not addressed by LLM-based methods.

2.3 Can LLM solve the instruction adaptation problem?

If we only consider the instruction adaptation problem, we would like to provide an additional experiment which has been included in the revised manuscript here. We tested an intuitive idea of whether an LLM could translate styled instructions back into a basic style to facilitate understanding for navigation models. This was evaluated on the Val-R-Scene split using three sets of instructions:

1. Basic: Instructions after Stage 2.
2. Scene: Instructions transformed from Basic after Stage 3.
3. Translated: Scene instructions translated back into Basic style by an LLM.

The performance of these instruction types is summarized in Table 1.

Table 1: Performance comparison of instruction styles on the Val-R-Scene split.

LLM-based translation improved performance over Scene instructions but did not fully close the gap with Basic instructions. This limitation arises from the open-vocabulary nature of LLMs, which introduces noise and leads to information loss, thereby reducing the effectiveness of the approach. Since this solution is not environment-specific, it falls outside the scope of scene adaptation and is not included in our work.

As noted in our response to Question 1, we have explained the reasons for not including InstructNav in our evaluations and instead adopted MapGPT as a comparable substitute. We believe this addresses any concerns regarding the lack of comparison. We hope this clarification resolves any misunderstandings regarding our dataset and the performance of LLM-based methods. Thank you for the opportunity to elaborate further.

Thank the authors for their response. After a careful reading of their data based rebuttals, my concerns are addressed. I kindly suggest that the authors add these analyses to the appendix.

I'll raise my score to 6.

We appreciate Reviewer WxGN for the time and effort in reviewing our paper and offering constructive feedback. Please find our responses to the comments below.

---

Weakness 1. Novelty of GSA-VLN

We are sorry that reviewer WxGN misunderstood and overlooked the novelty of our task. We would like to clarify that the novelty of GSA-VLN is not derived from "fine-tuning on historical information or using trained models". Instead, our work introduces fundamental differences that distinguish GSA-VLN from the standard VLN task, as outlined below:

Key Differences from Standard VLN Tasks

1. Lifelong, Cumulative Memory: In standard VLN tasks, agents rely solely on isolated, episode-level history. In contrast, GSA-VLN leverages a lifelong, cumulative memory through the memory bank, enabling agents to continuously adapt to their environment over time. This feature is crucial for handling persistent environments where repeated interactions are required.
2. Dynamic Model Updates: Unlike standard VLN tasks, where models are fixed during inference, GSA-VLN allows for dynamic model updates at inference time. This enables agents to refine their performance based on scene-specific contexts, facilitating adaptation that is not possible in current VLN paradigms.

These distinctions are clearly outlined in the Introduction section and elaborated upon in Section 3.2 (Lines 189–205), where we provide detailed equations and descriptions to demonstrate how GSA-VLN supports dynamic adaptation to persistent environments, a concept not explored in traditional VLN tasks.

Additionally, our novelty extends beyond proposing the GSA-VLN task. We introduce the GSA-R2R dataset, which significantly expands scene diversity and includes various instruction styles, providing a robust benchmark for evaluating scene-specific adaptation. We also propose the novel GR-DUET method, which achieves state-of-the-art performance. We believe the combination of the GSA-VLN task, the GSA-R2R dataset, and the GR-DUET method constitutes a significant and novel contribution to advancing VLN research. We have clarified these points in the revised manuscript and hope this addresses your concerns regarding the novelty of our work.

---

Weakness 2. Dataset Comparisons

Thank you for your feedback. Here we would like to clarify that we do not overlook prior works that incorporate additional scenes, and we have discussed the differences between these works and ours in the paper.

As stated in Line 85, we reference HM3D-AutoVLN and ScaleVLN you mentioned, which use the HM3D dataset as augmented training data but do not modify the evaluation splits. The Youtube-VLN also belongs to this line of work. However, this contrasts with our GSA-R2R dataset, which preserves the original R2R training data while introducing expanded evaluation splits to better assess adaptability in diverse scenarios.

Because of this motivation, the comparison in Table 1 focuses specifically on the evaluation splits of embodied navigation tasks. Since the aforementioned works share the same evaluation splits as R2R, including them in Table 1 would not provide meaningful insights or distinctions. Instead, our dataset comparison highlights the unique features of GSA-R2R, such as its broader range of environments and diverse instruction types, which are absent in prior datasets.

Nevertheless, we acknowledge the need for more detailed discussions to emphasize the differences between GSA-R2R and prior works. In the revised manuscript, we have expanded the Related Work section to provide a clearer and more thorough comparison with HM3D-AutoVLN, ScaleVLN, and YouTube-VLN.

---

Weakness 3. Use of Metrics

We appreciate your feedback and would like to address the concerns regarding the metrics used in our evaluation and the datasets chosen for experimentation.

3.1 Clarification on Evaluation Metrics

We believe there may be a misunderstanding regarding the metrics used in our evaluation. The metrics we use in our evaluation, including Trajectory Length (TL), Navigation Error (NE), Success Rate (SR), Success weighted by Path Length (SPL), and normalized Dynamic Time Warping (nDTW), are standard and widely recognized in VLN research. These metrics have been consistently used as benchmarks in prior works to comprehensively evaluate navigation success, efficiency, and instruction fidelity [1, 2]. None of these metrics are newly proposed by us. These metrics provide a comprehensive evaluation of agent performance and are sufficient to demonstrate the effectiveness of our method. We believe that adhering to these standard metrics ensures the comparability and validity of our results within the VLN research community.

3.2 Comparison to Other Datasets

Our primary contributions focus on introducing the GSA-VLN task and the GSA-R2R dataset, which are specifically designed to enable and evaluate scene-specific adaptation. Therefore, our experiments are centered around these contributions to provide meaningful insights into the proposed task and dataset. The datasets you mentioned, such as REVERIE, RxR, CVDN, and SOON, belong to traditional VLN settings and do not encompass the scene and instruction diversity emphasized by GSA-VLN. While these datasets provide valuable benchmarks, they lack the specific characteristics necessary for evaluating scene-specific adaptation, which is the core focus of our work.

We acknowledge that adapting these datasets into GSA-VLN (e.g., GSA-REVERIE or GSA-CVDN) would be an exciting direction for future research. However, such extensions are beyond the scope of this paper. We appreciate your suggestion and will consider these opportunities in our future work.

 

[1] Wang, Zun, et al. "Scaling data generation in vision-and-language navigation." ICCV. 2023.

[2] Liu, Rui, et al. "Volumetric Environment Representation for Vision-Language Navigation." CVPR. 2024.

---

Question 1. Instruction Quality

1.1 How is the navigation model used here implemented?

The implementation details of our model-based selection process are provided in Section A.2 (Lines 832–838) of the paper. In summary, we utilize a DUET model trained on unselected paths from the same 150 environments chosen for the GSA-R2R dataset. This setup ensures that:

• The GSA-R2R instructions serve as a validation seen split for the model
• The model is familiar with the environment's visual and spatial characteristics but has not encountered the specific instructions.

This design isolates the evaluation from the impact of unfamiliar environments, allowing the model to focus solely on instruction quality. Using this approach, the model achieves a high Success Rate (SR) of 73.6%, indicating robust performance.

1.2 How can we ensure that it is a good instruction discriminator?

We designed this process to identify the most robust model for evaluating instruction quality. The underlying rationale is:

1. If a well-trained model can successfully navigate to the correct target using a specific instruction, other sufficiently trained models should also have the potential to succeed.
2. Conversely, it is highly unlikely for a model to consistently reach the correct target when following incorrect instructions.

This approach validates the model’s role as a reliable instruction discriminator.

Furthermore, the baseline adaptation models, including GR-DUET, share the same architecture as DUET. This ensures that they only need to bridge the gap from unseen to seen environments to perform well with the provided instructions. This aligns directly with the goal of scene adaptation, as outlined in Line 48 of the paper.

1.3 Is it enough to explain the quality of the instruction?

To ensure comprehensive validation, we supplement the model-based evaluation with human evaluation, as detailed in Section 3.3.4. Table 2 reports that GSA-R2R instructions achieve approximately 80% match accuracy, closely aligning with the 86% human success rate on the original R2R dataset. This similarity highlights the high quality of the GSA-R2R instructions, demonstrating their alignment with human expectations. These results, combined with the model-based evaluation, provide strong evidence of the robustness and quality of the instructions in GSA-R2R.

We hope this clarifies the robustness of our approach and demonstrates that our model effectively discriminates between instruction quality.

We thank Reviewer WxGN for the valuable feedback and insightful suggestions, which have helped us refine and clarify our work. We have carefully addressed all the raised concerns in our response.

We would greatly appreciate it if Reviewer WxGN could provide further feedback. Your input is invaluable to ensuring the quality and clarity of our work. Or, if our responses have satisfactorily resolved the concerns, we respectfully request reconsideration of the score based on the clarifications and improvements provided.

Weakness 3. Instruction Adaptation

Thank you for your insightful suggestions. We are happy to provide more details about our findings and insights on the instruction adaptation problem.

3.1 Ablation Study on Each Instruction Type

We have already included an ablation study comparing Basic, Scene, and User instructions in Table 11 of the paper, keeping environments and paths constant. The results show that stylized instructions (User and Scene) lead to a similar SR drop for the DUET model. This demonstrates that these styles introduce additional challenges due to their increased complexity and variability in language.

3.2 Further Analysis

Our primary focus has been on establishing a high-quality benchmark for studying instruction adaptation. While GR-DUET primarily addresses environment-side adaptation without optimizing for instruction styles, we conducted extensive experiments to evaluate how existing methods handle this challenge, including those with potential for instruction adaptation, such as Masked Language Modeling (MLM) and Back-Translation (BT).

Table 5 in the paper presents results for different adaptation methods across varying instruction styles. TTA methods, such as TENT, perform better with Scene instructions due to the distinct language patterns introduced by conversational fillers, which are more recognizable than the subtle word variations in User instructions. However, the advantage brought by Back-Translation (BT) is significantly reduced in instructions with diverse styles compared to Basic instructions. This is because BT struggles with larger gaps between the training corpus and human instructions, highlighting its difficulty in adapting to speaker-specific variations effectively.

We further present the mean performance and standard deviation (std) of various methods across different speakers in User instructions in Table 2.

Table 2: Mean SR and Standard Deviation (std) of baseline methods across different speakers.

MLM achieves the lowest std, demonstrating improved adaptation to instruction styles by learning speaker-specific vocabulary. BT achieves the highest overall performance among adaptation-based methods but also shows the highest std, reflecting its sensitivity to the training corpus of the speaker model it uses. Specifically, BT overfits to its training speaker's style, leading to inconsistencies when applied to diverse styles, as it amplifies performance variations.

These results underscore the challenges of instruction adaptation and provide a foundation for future research.

3.3 Different Dialects or Detail Levels

We appreciate your suggestion to explore the impact of dialects and detail levels. our approach models speaking styles as a general term including differences in vocabulary, catchphrases, dialects, and detail levels. For example, the characters have backgrounds from different places, naturally introducing dialects into their instructions. This diversity mirrors real-world scenarios and provides a more comprehensive test of instruction adaptation by including linguistic variability. Rather than isolating specific variables like dialects, our general setup provides a robust framework for evaluating model performance across a wide range of language styles. Models capable of adapting to these combined variables are better suited to real-world applications. Future work could delve into isolating individual factors to further analyze their specific impact and propose tailored methods for improving adaptability.

We hope our findings inspire further exploration of model-side adaptations for specific language patterns in VLN tasks. Instruction style adaptation remains a promising direction for future research, and we look forward to seeing the community build upon our benchmark and methods.

We are grateful to Reviewer t9oJ for dedicating time and effort to reviewing our paper and providing thoughtful and constructive feedback. Our detailed responses to the comments are outlined below.

---

Weakness 1. Scalability and Resource Use

Although the memory bank in GR-DUET updates continuously during navigation, it stabilizes once the agent has explored most of the environment. Updates after this point are minimal, involving only new instructions and actions, which require relatively little memory. Moreover, GR-DUET employs coarse-grained embeddings for nodes that are not neighbors of the current node, ensuring that GPU memory usage does not grow significantly even when processing the entire graph. To illustrate this, we present key computational metrics across episodes for one of the largest environments in GSA-R2R in Table 1.

Table 1: Variations in computational costs of GR-DUET across different episodes.

As agents execute more instructions, we observe gradual increases in CPU memory usage, GPU memory usage, and inference time. However, these metrics stabilize as the graph coverage approaches 1, indicating that the agent has explored most of the environment.

From the model perspective, GR-DUET contains 180 million parameters, occupying only 2.1 GB of disk space. The maximum inference time of 67 ms translates to a throughput of 15 frames per second, making it suitable for real-time navigation tasks. Its computational complexity is 1.63 GFLOPs (excluding visual feature extraction), enabling deployment on robots without excessive computational demands. All these statistics demonstrate that GR-DUET is computationally scalable and practical for deployment on resource-constrained systems. . In the revised manuscript (Section A.5), we include additional visualizations and analyses for two more environments of varying sizes to further validate these findings.

---

Weakness 2. Dataset Diversity

Thank you for your insightful suggestion. To minimize the sim-to-real gap, current VLN research relies on photorealistic environments, which limits the datasets available for studying VLN compared to navigation tasks that can utilize synthetic or procedurally generated datasets. Within these constraints, we have made significant efforts to expand the diversity of GSA-R2R to include 20 distinct scene types, compared to just six in R2R. This diversity covers a wide range of daily scenarios and exceeds that of existing embodied navigation datasets, as highlighted in Table 1 of our paper.

Regarding the three types of scenes recommended:

• Commercial Spaces: We already include multiple commercial spaces such as cinemas, shops, and restaurants, as illustrated in Figure 2 of our paper.
• Outdoor Spaces: While the dataset focuses on indoor environments, some scenes include outdoor elements such as gardens, yards, and swimming pools adjacent to houses. However, outdoor navigation tasks [1-2] require fundamentally different capabilities and face challenges distinct from indoor navigation. Consequently, outdoor VLN is generally studied separately within the research community. Since GSA-R2R focuses on indoor, scene-specific adaptation, incorporating full outdoor spaces falls outside the scope of this work.
• Dynamically Environments: Dynamically changing environments are a valuable and realistic direction for embodied navigation research [3-4]. However, expanding GSA-R2R to include dynamic environments introduces significant challenges, such as environment manipulation and maintaining consistency in dynamic layouts. These challenges extend beyond the current scope of our work. We believe that future research can build on the foundation provided by GSA-R2R to address dynamic adaptation.

We agree that extending GSA-R2R to represent even broader real-world scenarios, including additional scene types or dynamic environments, would further strengthen its applicability. We appreciate the reviewer’s valuable suggestions and will consider these directions as future works.

 

[1] Liu, Shubo, et al. "Aerialvln: Vision-and-language navigation for UAVs." ICCV. 2023.

[2] Li, Jialu, et al. "Vln-video: Utilizing driving videos for outdoor vision-and-language navigation." AAAI. 2024.

[3] Li, Heng, et al. "Human-aware vision-and-language navigation: Bridging simulation to reality with dynamic human interactions." NeurIPS. 2024.

[4] Zhou, Qinhong, et al. "HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments." ICLR. 2024.

2.2 Is the memory bank pre-set or updated dynamically?

The dynamic update procedure is detailed in Equation 1 of the paper. As described in Lines 180-188, "This memory bank dynamically expands as the agent executes instructions, capturing four key components: visual observations ($\mathbf{O}$), the instructions ($X$), selected actions ($\mathbf{A}$), and trajectory paths ($\mathbf{P}$). For example, after executing $k$ instructions in environment $E$, the memory bank is updated as follows: $\mathcal{M}_E = \{X^{1:k}, \mathbf{O}^{1:k}, \mathbf{A}^{1:k}, \mathbf{P}^{1:k}\}$ ."

After each episode, the data collected during that episode is stored in the memory bank, enabling the agent to build a comprehensive history of its navigation.

2.3 How is the correctness of the stored memories ensured?

The stored memories represent the agent’s execution history, including user-provided instructions, visual observation from sensors, actions taken, and trajectories followed. Since this data directly reflects the agent’s experience and is ``factual'', the concept of "correctness" does not strictly apply. We understand the concerns that there may exist misalignment between instructions and paths due to navigation errors. However, all the memories are treated as unlabeled data and are primarily used for unsupervised learning techniques, such as Back-Translation or Masked Language Modeling.

For example, in GR-DUET, trajectory and observation data are employed to construct a global, consistent graph for decision-making. These components are directly derived from the agent’s observations and are inherently accurate, ensuring reliable support for the decision-making process.

2.4 How are the initial model parameters (L194, L198) initialized?

The initial model parameters are determined by the specific method but must ensure sufficient generalization across diverse environments. While the goal is to adapt to environment-specific models at inference, the trained model must retain universality to handle a wide range of environments. Overfitting the model to a specific target environment during training would compromise its generalization ability, making the method unrealistic and impractical. To address this, we train GR-DUET on the full training split of R2R and augment the instructions with data from PREVALENT. This approach ensures that GR-DUET learns generalized navigation capabilities while remaining adaptable during inference.

We have incorporated these details into Section 3.2 of the revised manuscript. Thank you for highlighting this opportunity to enhance the clarity of our work.

---

Question 3. Comparison with SG-Nav

Thank you for the suggestion. We have carefully read SG-Nav and found its approach to be highly impressive. The idea of constructing consistent 3D scene graphs aligns closely with the goals of our GSA-VLN task and represents a potential solution for enhancing scene adaptation. This concept is also reflected in the OVER-NAV baseline used in our work, which constructs an omnigraph similar to scene graphs to consistently record encountered objects. However, both GR-DUET and OVER-NAV primarily store historical information within a topological map, limiting the stored data to visual observations or detected objects. In contrast, the 3D scene graphs used in SG-Nav offer a more powerful representation of the environment, which could potentially help agents retrieve and leverage relevant historical information more effectively for navigation tasks.

Unfortunately, SG-Nav could not be directly applied to our VLN task due to fundamental differences in task design:

1. Task Objective: SG-Nav is tailored for object-goal navigation, where the input is a target object rather than fine-grained natural language instructions.
2. Action Space: SG-Nav employs low-level actions (e.g., move forward, turn left, turn right, and stop), whereas VLN requires the agent to select a waypoint from a discrete set of candidate locations based on complex instructions.
3. Adapting SG-Nav to VLN: Substantial modifications would be required to adapt SG-Nav’s approach to hierarchical VLN tasks, particularly for handling fine-grained instructions and multi-step navigation scenarios.

While we could not include SG-Nav as a direct baseline due to these technical challenges, we plan to explore its ideas, particularly its 3D scene graph representation, in future work. This could lead to improved memory mechanisms and more robust scene adaptation in VLN tasks. We have also included a discussion of SG-Nav in the Related Work section of the revised manuscript.

We believe we have made a comprehensive effort to include all relevant VLN works with long-term memory mechanisms as baselines. However, if you have additional baseline suggestions, we would be happy to consider them and incorporate them into our analysis.

Question 4. Instruction Style Adaptation

Thanks for raising the point about instruction adaptation. We would like to clarify the scope of our work and present our findings and insights on this emerging problem.

4.1 Novelty of Instruction Style Adaptation

Previous VLN works mainly utilize instructions with plain and concise language, overlooking personal speaking habits. In this work, we are the first to propose the problem of instruction style adaptation, addressing this critical gap. While our primary focus is on establishing a benchmark with sufficient high-quality data to study this problem, we acknowledge that solving instruction adaptation requires future efforts. Our GR-DUET model focuses more on environment-side adaptation without specific optimization for instruction styles. This limitation is explicitly discussed in Section A.5 of our paper. We view instruction adaptation as an open problem and a promising research direction enabled by the benchmarks and data provided by our work.

4.2 Analysis of Performance

Although our method does not include instruction-specific optimizations, we conducted extensive experiments to evaluate how existing methods handle this challenge, including those with potential for instruction adaptation, such as Masked Language Modeling (MLM) and Back-Translation (BT).

Table 5 in the paper presents results for different adaptation methods across varying instruction styles. TTA methods, such as TENT, perform better with Scene instructions due to the distinct language patterns introduced by conversational fillers, which are more recognizable than the subtle word variations in User instructions. However, the advantage brought by Back-Translation (BT) is significantly reduced in instructions with diverse styles compared to Basic instructions. This is because BT struggles with larger gaps between the training corpus and human instructions, highlighting its difficulty in adapting to speaker-specific variations effectively. We further present the mean performance and standard deviation (std) of various methods across different speakers in User instructions in Table 3.

Table 3: Mean SR and Standard Deviation (std) of baseline methods across different speakers.

MLM achieves the lowest std, demonstrating improved adaptation to instruction styles by learning speaker-specific vocabulary. BT achieves the highest overall performance among adaptation-based methods but also shows the highest std, reflecting its sensitivity to the training corpus of the speaker model it uses. Specifically, BT overfits to its training speaker's style, leading to inconsistencies when applied to diverse styles, as it amplifies performance variations.

These results underscore the challenges of instruction adaptation and provide a foundation for future research.

4.3 Potential Solutions

We have also explored additional strategies for addressing instruction adaptation:

• Weighted Masked Language Modeling: Vanilla MLM treats all words equally and randomly masks a proportion of them, which is inefficient for focusing on key or unseen words. We modified the MLM approach to prioritize masking unseen or rare words specific to a speaker's style. This achieved a 1% improvement in SR, but required extensive hyperparameter tuning for each environment, making it impractical for real-world deployment.

• LLM-based Instruction Translation: We tested an intuitive idea of whether an LLM could translate styled instructions back into a basic style to facilitate understanding for navigation models. This was evaluated on the Val-R-Scene split using three sets of instructions:

  1. Basic: Instructions after Stage 2.
  2. Scene: Instructions transformed from Basic after Stage 3.
  3. Translated: Scene instructions translated back into Basic style by an LLM.

  The performance of these instruction types is summarized in Table 4.

Table 4: Performance comparison of instruction styles on the Val-R-Scene split.

LLM-based translation improved performance over Scene instructions but did not fully close the gap with Basic instructions. This limitation arises from the open-vocabulary nature of LLMs, which introduces noise and leads to information loss, thereby reducing the effectiveness of the approach. Since this solution is not environment-specific, it falls outside the scope of scene adaptation and is not included in our work.

While we have not fully solved the instruction adaptation problem, our work lays the groundwork by:

1. Establishing a benchmark dataset with diverse instruction styles.
2. Providing a comprehensive evaluation of how existing adaptation methods handle this challenge.

We hope our findings will inspire further exploration into model-side adaptations for specific language patterns in VLN tasks. Instruction style adaptation remains an exciting area for future research, and we look forward to seeing the community build upon our dataset and methods.

---

Question 5. Adaptation Efficiency

Our work mainly focuses on scenarios where the agent operates in a persistent environment over an extended period, potentially its entire lifetime. In such cases, the overall performance after adaptation is more critical than the adaptation speed. This explains why we include 600 paths for each environment in GSA-R2R, significantly more than previous VLN datasets, ensuring sufficient instructions for effective adaptation. Nevertheless, we conducted further analyses on GR-DUET to better understand its adaptation efficiency.

5.1 How quickly do the agents adapt to different environments?

GR-DUET builds a global graph for adaptation, which stabilizes once the agent has explored most parts of the environment. Based on the graph coverage data (Table 1 of Question 1), we find that 90% coverage is achieved with at most 400 instruction-trajectory pairs in large environments and as few as 100 pairs in small environments. To measure adaptation speed, we treat the first X instructions (the number required to reach 90% graph coverage) as the adaptation phase and divide them into groups of 50 instructions. Performance within each group is measured, and linear regression is applied to calculate the slope of performance improvement, serving as a proxy for adaptation speed. Results show that among the 150 scans, 94 achieved a positive slope, with a mean slope of 0.26, indicating rapid adaptation in most cases.

5.2 Are there any cases where adaptation is less effective or slower?

To understand slower or less effective adaptation, we analyzed adaptation speed across various environmental characteristics.

First, Table 5 shows the mean adaptation slopes in environments with different numbers of floors.

Table 5: The adaptation speeds of GR-DUET in environments with different number of floors.

Adaptation becomes less effective as the number of floors increases, except for a few cases with four floors. This is intuitive, as distinct floor layouts and styles make prior memory from other floors less relevant to the current navigation.

Conversely, Table 6 shows that adaptation efficiency improves as the number of rooms increases, particularly in buildings with more than 15 rooms.

Table 6: The adaptation speeds of GR-DUET in environments with different number of rooms.

After viewing specific buildings, we find that environments with many rooms (e.g., hotels or student accommodations) often have repetitive layouts and identical rooms, allowing the agent to leverage memory from similar spaces effectively. These findings suggest that GR-DUET performs well in environments with repetitive structures but struggles in environments with dissimilar memory (e.g., multi-floor buildings with distinct layouts).

5.3 Are there specific environments where the memory mechanism struggles to adapt?

We calculated mean adaptation slopes for different scene and instruction types, as shown in Table 7.

Table 7: The adaptation speeds of GR-DUET in different types of scenes and instructions.

From the results, we can see that GR-DUET adapts faster in residential environments than in non-residential environments. This is likely due to the training environments being predominantly residential (from R2R), introducing a bias that favors residential scenarios. For different instructions, agents adapt fastest to Scene instructions and least effectively to User instructions. Scene instructions often include conversational fillers, which provide more distinct language patterns than the word variations in User instructions, making them easier to adapt to.

These analyses highlight both the strengths and limitations of GR-DUET’s adaptation mechanism. While it excels in environments with repetitive layouts, it struggles in multi-floor or irregular environments. Future improvements could focus on leveraging dissimilar memories (e.g., between floors) and reducing training biases to enhance adaptation in more diverse scenarios.

We hope these findings and insights guide future research and improvements in the GSA-VLN task.

---

Question 6. Practical Deployment

While our work is motivated by real-world scenarios, we follow the standard practice in VLN research of addressing scene adaptation problems within simulators as a foundation for future practical deployment. Nonetheless, we believe our proposed method can be effectively deployed in real-world systems, such as robotics or autonomous agents.

To address this concern, we provide a detailed analysis of the computational and memory overhead associated with our GR-DUET. We use the largest environment in GSA-R2R as an example to demonstrate the resource requirements of GR-DUET during inference.

1. Memory Requirements

• GPU Memory: GR-DUET requires a peak of 4.3 GB of GPU memory during inference, which is well within the capacity of modern GPUs. For instance, it can be deployed on terminal servers equipped with hardware like the NVIDIA Jetson AGX Orin or similar devices.

• CPU Memory: The method requires at most 5.3 GB of RAM, which is easily manageable by most modern robotics platforms.

2. Computational Overhead

• Inference Latency: GR-DUET achieves an average inference latency of 67 milliseconds per frame, allowing efficient navigation in most real-world environments.
• Throughput: The system processes 15 frames per second, which is sufficient for environments where navigation speed is moderate and does not require high-frequency updates, such as indoor environments with static obstacles.

3. Model Characteristics

• Model Size: The model includes 180 million parameters, occupying approximately 2.1 GB of disk space.
• Computational Complexity: The model has a computational cost of 1.63 GFLOPs (excluding visual feature extraction), making it feasible for implementation on robots without imposing excessive computational demands.

These metrics demonstrate that GR-DUET is highly practical for deployment in real-time systems. Its resource requirements align with the capabilities of robotics platforms such as TurtleBot2 and LoCoBot, which have been commonly used in previous works $1-2$ . We have included this discussion in the revised manuscript to highlight the practical applicability of our approach.

 

[1] Anderson, Peter, et al. "Sim-to-real transfer for vision-and-language navigation." CoRL, 2021.

[2] Xu, Chengguang, et al. "Vision and Language Navigation in the Real World via Online Visual Language Mapping." CoRL Workshop, 2023

We appreciate Reviewer dHDF for the time and effort in reviewing our paper and offering constructive feedback. Please find our responses to the comments below.

---

Question 1. Memory Mechanism Scalability

We agree that the scalability of GR-DUET is an important consideration for addressing the scene adaptation problem. Below, we provide further analysis to demonstrate the scalability of our method in larger and more complex environments from both computational and performance perspectives.

1.1 Computational Costs and Memory Usage

Although the memory bank in GR-DUET updates continuously during navigation, it stabilizes once the agent has explored most of the environment. Updates after this point are minimal, involving only new instructions and actions, which require relatively little memory. Moreover, GR-DUET employs coarse-grained embeddings for nodes that are not neighbors of the current node, limiting GPU memory growth despite inputting the entire graph into the model. To illustrate this, we analyze key computational metrics across episodes for one of the largest environments in GSA-R2R (Table 1).

Table 1: Variations in computational costs of GR-DUET across different episodes.

As agents execute more instructions, we observe gradual increases in CPU memory usage, GPU memory usage, and inference time. However, when the graph coverage approaches 100%, indicating that the agent has explored most places, these metrics stabilize with minimal additional overhead. This demonstrates that GR-DUET is computationally scalable and does not sacrifice much memory efficiency for improved performance. In the revised manuscript (Section A.5), we include additional visualizations and analyses for two more environments of varying sizes.

1.2 Navigation Performance

In large environments, the memory bank grows, introducing more inputs to the model. However, GR-DUET’s pre-training stage on the entire ground-truth graph ensures that it effectively handles these inputs. By focusing on surrounding viewpoints rather than the full graph, GR-DUET mitigates the impact of environment size on computational efficiency and performance. To validate this, we compare the performance gap (Success Rate, SR) between the original DUET and GR-DUET across all 150 environments in GSA-R2R with varying numbers of viewpoints (Table 2).

Table 2: Mean SR gap between GR-DUET and original DUET by environment size.

The results show that GR-DUET consistently outperforms the original DUET, even in environments with a large number of viewpoints. This indicates that GR-DUET effectively learns to focus on relevant parts of the graph, ensuring scalability in large environments. Additionally, we include a scatter plot of SR gaps versus navigable areas in Section A.5 of the revised manuscript, which further confirms that GR-DUET consistently performs better in all large environments.

---

Question 2. Memory Mechanism Details

The focus of our paper is on introducing a new task and dataset, which is why most of the content is dedicated to detailing these aspects. The method presented serves as a baseline, and due to space constraints, certain details were not included in the original manuscript. We thank the reviewer for highlighting this, and we provide the requested clarifications below, which have also been added to the revised manuscript.

2.1 How is the memory utilized?

In GSA-VLN, our framework does not impose a specific way to utilize the information in the memory bank. Instead, it provides the entire history of information, enabling different methods to selectively leverage this data to improve performance for improved performance. The memory bank can be used in various ways, such as additional input for decision-making or for unsupervised learning purposes, as outlined in Section 3.2.

We thank Reviewer dHDF for the valuable feedback and insightful suggestions, which have helped us refine and clarify our work. We have carefully addressed all the raised concerns in our response and uploaded a revised manuscript.

We would greatly appreciate it if Reviewer dHDF could provide further feedback. Your input is invaluable to ensuring the quality and clarity of our work. Or, if our responses have satisfactorily resolved the concerns, we respectfully request reconsideration of the score based on the clarifications and improvements provided.

Question 2: Language vs. Environment

Thank you for raising this important point. Below, we address each aspect of your question in detail.

2.1 Is the adaptation of language style in scene adaptation the main one?

We believe both language adaptation and environment adaptation are equally important in scene adaptation, and they are inherently intertwined. While instructions can sometimes be generated or refined independently, their content often reflects the specific characteristics of the environment. In our work, the instructions are not purely "generated" by LLM but rather "refined" based on speaker-generated inputs. This refinement preserves the connection between language and the environment, making absolute isolation of these influences impractical. For example, instructions in a shop often involve specific product categories, such as "head to the produce section.", while instructions in a home commonly reference rooms like the "bedroom" or "kitchen." These contextual differences illustrate how environment and language styles are inherently linked, introducing unique challenges for navigation models. Our results further indicate that the adaptation process must address both linguistic and visual factors. For instance, solely focusing on language or visual adaptation (e.g., using MLM or MRC) does not lead to significantly higher performance gains, highlighting the importance of considering both aspects together.

2.2 Is it a possible solution to directly use LLM to convert these different styles into the same one?

Yes, this is a possible solution and we have provided some preliminary experiments to explore its potential in our previous response. We emphasize them again here. We tested an intuitive idea of whether an LLM could translate styled instructions back into a basic style to facilitate the understanding of these styles for navigation models. This was evaluated on the Val-R-Scene split using three sets of instructions:

1. Basic: Instructions after Stage 2.
2. Scene: Instructions transformed from Basic after Stage 3.
3. Translated: Scene instructions translated back into Basic style by an LLM.

The performance of these instruction types is summarized in Table 1.

Table 1: Performance comparison of instruction styles on the Val-R-Scene split.

The results reveal that LLM-based translation improved performance over Scene instructions but did not fully close the gap with Basic instructions. This limitation arises from the open-vocabulary nature of LLMs, which introduces noise and leads to information loss, thereby reducing the effectiveness of the approach. Since this solution is not environment-specific, it falls outside the scope of scene adaptation and is not included in our work. However, it represents a promising direction for future work to solve the instruction style problem, like fine-tuning a LLM-based translator or adding the translated instructions into the training process of the navigation model.

2.3 Will the distribution of visual and spatial objects have a greater impact than language?

Our results suggest that the relative impact of visual/spatial distributions and language styles depends on their variation from the training data:

• Environment Adaptation: As shown in Table 4-6 of the paper, our GR-DUET method addresses the environment changes and achieves a performance increase of approximately 8% in Success Rate (SR). This demonstrates the significant impact of visual and spatial variations.
• Language Adaptation: Language style changes lead to smaller performance drops (around 3%–5%), as shown in Table 11 of the paper. However, the impact can increase significantly when language styles involve more distinct variations (e.g., highly divergent characters like Moira), as shown in Table 6. In these cases, our GR-DUET achieves smaller performance gains, reflecting the difficulty of adapting to large linguistic distribution shifts.

We think both language and environment styles play critical roles in scene adaptation, and their relative importance depends on specific factors such as the degree of variation from the training data.