CONSTRAINT-AWARE ZERO-SHOT VISION-LANGUAGE NAVIGATION IN CONTINUOUS ENVIRONMENTS

Thank you very much for your review and comments.

Q1: There are no explicit schemes to guarantee the navigation process will fulfill the constraints. Does the BLIP-2 score enough to finish the sub-tasks and walk a perfect trajectory to solve all the constraints?

Thank you for raising this concern. BLIP-2 is a suitable choice for our task, as demonstrated by its success in VLFM[1], where it effectively handles open-vocabulary object queries for object navigation. While it is difficult to guarantee that all constraints are fully satisfied due to the inherent imperfections of current perception models, we have designed two complementary mechanisms to enhance constraint fulfillment:

1. Trajectory Mask: This mechanism encourages the agent to explore new areas, improving the chances of satisfying constraints.
2. Historical Decay: This mitigates abrupt changes in the value map when switching sub-instructions, providing smoother navigation.

[1] Yokoyama, Naoki, et al. "Vlfm: Vision-language frontier maps for zero-shot semantic navigation." 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

Q2: Will the agent always inflexibly go back and try to complete the first step when mistakenly arrives at a proper next waypoint without satisfying the first sub-instructions? Is there a more flexible framework for such scenarios in your method?

We didn't design a specific error correction module. However, we did experiments in both simulation and real-world and found that it rarely goes back. On one hand, we have the trajectory mask to discourage the agent from going back to the explored area. On the other hand, as long as the sub-instruction switches, historical decay would give the past value map a low weight, which helps the agent flexibly skip the previous hard constraints and concentrate on current constraints.

Q3: How long does it take for the whole round of the planning process, including semantic mapping, generating sub-instructions, value maps, and path planning?

We studied the model efficiency in section 4.4 and we reported that the proposed CA-Nav took about 0.12 seconds for each action decision. One action decision procedure (i.e. the planning process) includes: semantic mapping, value mapping, and FMM path planning. Processing one instruction took about 13s but we didn't take this part into account since it would be done before the episode started.

###　Q4: Why does the success rate vary for the same instruction and same starting positions shown in the supplementary Table 7?

The variability in success rate primarily stems from the robot's vibration during navigation, which can cause fluctuations in Lidar data. These variations affect camera pose estimation, leading to slight differences in the dynamically constructed map and subsequent waypoint selection.

Thank you very much for your review and comments.

Q1: Is LLM really needed here?

1. The paper CLIP-Nav[1] highlights the advantages of LLMs, particularly GPT-3, in processing complex instructions. LLMs excel at capturing the sequential structure of instructions and handling nuanced language, which would be challenging for rule-based approaches. Similarly, CA-Nav leverages LLMs to identify and interpret constraints effectively, including distinguishing between object and location constraints within the same sub-instruction. It's a task that is non-trivial for simpler methods.
2. Our ablation study in Table 6 demonstrates that stronger LLMs (e.g., GPT-4) significantly improve CA-Nav's performance compared to smaller models, indicating the benefits of using LLMs.

Q2: What are the common failure cases? Is the task naturally ambiguous or ill-defined? Can humans do this task with the same observation space?

1. The common failure cases in our approach can be categorized into two main types:

   • Constraints Detection Errors: The agent may fail to correctly identify constraints, such as landmarks or directional cues, leading to incorrect switching decisions.
   • Suboptimal Path Planning: Errors in waypoint selection often stem from inaccuracies in the value map generated from BLIP2. When the value map does not accurately represent the environment or constraints, the agent may select inefficient or incorrect paths. In summary, the main cause of failure lies in perception modules. We use several perception modules to detect constraints and build value maps. However, they can hardly be completely correct all the time. Moreover, the 3D reconstruction in Habitat Simulator has low visual fidelity and low physical fidelity which would futhrer affect the percetion mouldes. Similar discoveries and opinions are proposed in [2]. With the development of foundation models, these problems could be mitigated.

2. The task is neither ill-defined nor ambiguous. Wang et al.[3] shows that humans can achieve a success rate of approximately 0.86, and agents trained on large-scale data can approach human-level performance, indicating the task's well-defined structure and learnability. The primary challenge lies in the zero-shot setup, where the agent lacks prior knowledge of long-horizon navigation strategies and environmental understanding.

[1] Vishnu Sashank Dorbala and Gunnar Sigurdsson and Robinson Piramuthu and Jesse Thomason and Gaurav S. Sukhatme. CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation. arXiv preprint arXiv: 2211.16649

[2] Theophile Gervet and Soumith Chintala and Dhruv Batra and Jitendra Malik and Devendra Singh Chaplot. Navigating to Objects in the Real World. arXiv preprint arXiv: 2212.00922

[3] Zun Wang and Jialu Li and Yicong Hong and Yi Wang and Qi Wu and Mohit Bansal and Stephen Gould and Hao Tan and Yu Qiao. Scaling Data Generation in Vision-and-Language Navigation. arXiv preprint arXiv: 2307.15644

Thank you very much for your review and comments.

Q1: Difference from VLFM?

The difference between CA-Nav and VLFM is two-fold:

1. From the task perspective: VLFM addresses Object Navigation, where the agent explores the environment to locate a specific object. CA-Nav, however, focuses on the more complex VLN-CE task, where the agent must follow natural language instructions to navigate. This requires tracking navigation progress and managing sequential sub-tasks.
2. From the method perspective: VLFM focuses on frontier-based exploration (FBE), which limits decision options to the boundary of the explored area. In contrast, CA-Nav utilizes the entire value map, offering a larger and more flexible decision space. To efficiently leverage this expanded space, CA-Nav employs a superpixel clustering method for waypoint selection, grouping semantically consistent regions and reducing noise. Additionally, CA-Nav introduces a Constraint-aware Sub-instruction Manager (CSM) to automatically track navigation progress and switch sub-instructions based on predefined constraints, such as landmarks, locations, and directions. To further enhance navigation, CA-Nav applies value map refinements, including trajectory masking and historical decay, ensuring smoother and more robust execution of sequential sub-tasks. To the best of our knowledge, this is the first attempt at a constraint-based instruction-switching mechanism in the VLN community.

Q2: How does LLM handle complex or ambiguous instructions? Is there a mechanism to handle possible decomposition errors? Can the decomposition process be quantitatively assessed? For example, testing instructions of different complexity for accuracy or robustness.

1. As shown in the table below, instructions with more than 5 sub-instructions can be perceived as complex. We show detailed decomposition results of one complex instruction "Go straight. Pass the stairs on the right and continue straight. When you get to the stairs, go up past those as well. Go into the room with the coaches and then turn right. wait near the glass table with white chairs." The decomposition results are:

"1313": {"destination": "glass table with white chairs in the room", "sub-instructions": ["Go straight","Pass the stairs on the right and continue straight","When you get to the stairs go up pass those as well","Go into the room with the couches","then turn right","wait near the glass table with white chairs"], "state-constraints": { "0": [["direction constraint","forward"]], "1": [["object constraint","stairs"],["direction constraint","right"]], "2": [["object constraint","stairs"],["direction constraint","up"]], "3": [["location constraint","room"],["object constraint","couches"]], "4": [["direction constraint","right"]], "5": [["object constraint","glass table"],["object constraint","white chairs"]] }}.

It has 6 sub-instructions and some of them have more than one constraint. For example, sub-instruction 1 contains two different constraints and the execution details are depicted in pseudo-code Algorithm1(Line 26) and Algorithm2. Specifically, during sub-instruction 1, at each step, CSM will check both of the two constraints, if one of them has been satisfied it will be removed from the list. CSM switches to the next sub-instruction until all constraints are satisfied.

2. We didn't design a specific module to handle decomposition errors. However, we established a maximum step threshold to prompt the agent to switch constraints when progress stalls. Similarly, we also set minimum switch steps in order to ensure adequate focus on each constraint before switching.

3. We add a table to show the quantitative analysis of instruction decomposition. We tested instructions of different complexity for accuracy, and the results are: | Number of sub-instructions | 1 | 2 | 3 | 4 | 5 | 6 | 7 | |----------------------------|------|------|------|------|------|------|------| | success/fail trajectories | 4/11 | 50/180 | 151/503 | 145/635 | 78/324 | 27/137 | 3/34 | | SR | 0.36 | 0.28 | 0.30 | 0.23 | 0.24 | 0.20 | 0.09 |

Most instructions are of moderate difficulty with 3 or 4 sub-instructions. In general, the success rate of navigation tends to decrease as the complexity of the instructions increases.

Thank you very much for your review and comments.

Q1: From Table 1, why the results are only slightly better than other zero-shot VLN approaches, especially for the SPL metric?

1. The baseline A$^2$Nav is not entirely training-free, as it leverages HM3D's room boundaries to train five navigators to handle different actions. In contrast, our approach is completely training-free, making it more challenging. However, compared to the baseline NavGPT-CE (sr=16.3), CA-Nav (sr=25.3) outperformed it by a large margin.

2. It should be noted that we focus on the zero-shot setting where the agent has not learned prior knowledge of efficient long-horizon planning. We admit that the improvement in SPL is limited because long-horizon planning is a challenging problem, especially when the robot needs to explore unfamiliar environments without prior knowledge. However, this is a common limitation of current zero-shot approaches. We will continue to work on this aspect in the future.

Q2: Is there one major component that affects the results of the approach? Why CA-Nav impact SR but largely remain the same for SPL and NE?

1. Yes, CSM is the major component, as it not only affects the results but also enables the robot to navigate by following instructions, distinguishing it from simple object navigation. In the VLN community, we are the first to propose a constraint-based instruction-switching mechanism. As shown in Table 3, if we ablate all constraints all metric's performance gets worse.
2. Since CA-Nav is a zero-shot method, it can not learn prior knowledge of long-horizon navigation planning like learning-based methods, which often results in longer paths. When the paths are generally longer, SPL becomes less sensitive to changes in path length. The SPL metric is calculated as follows:$$ $

SPL = \frac{1}{N}\sum_{i=1}^{N}\mathbb{I}(d(p, p^)<\epsilon)\frac{L^}{max(L, L^{*})}

$