ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination

Thank you for your kind suggestions and questions. From our perspectives, there are mainly three reasons for using points as intermediate subgoals instead of directly building end-to-end policies. (1) In real-world navigation scenarios, it is common for robots to interact with dynamic surroundings such as our humans. The action model must be able to run at high frequency, and it is difficult for an end-to-end VLA model to perform at high control speed.

(2) The robustness of the point-goal navigation policy is confirmed in some recent works, such as iPlanner[1] and ViPlanner[2]. They propose efficient and lightweight models for collision avoidance and point-goal-reaching functions. It is much easier to build a general collision avoidance model than a navigation model to understand language task instructions. Therefore, the end-to-end model often suffers from poor generalization ability with limited data sources. During this rebuttal period, we evaluate the pre-trained SPOC [3] model, which is an end-to-end policy trained in ProcThor. The SPOC performs poorly in the HSSD and HM3D object navigation benchmarks.

(3) The object navigation problem emphasizes the scene understanding for efficient exploration behaviors in novel environments. Such high-level knowledge for navigation task planning is already inherited in LLMs or VLMs trained with Internet datasets. It would be expensive and unnecessary to re-collect the embodied data with action labels to build the scene understanding abilities. But as the pre-trained VLMs tend to understand 2D images instead of 3D spaces, our proposed method provides an easy way to build a bridge between the VLMs to the point-goal navigation policy, which is imagination.

We think one major disadvantage of using the VLMs for task planning is about agent memory. How to effectively encode the history information and let pre-trained VLMs to understand is one of the possible directions in the future works. And using in-context learning or RAG can help the VLMs to learn from additional navigation data.

Reference: [1] Yang, Fan, et al. "Iplanner: Imperative path planning." Robotics, Science and Systems (RSS), 2023. [2] Roth, Pascal, et al. "Viplanner: Visual semantic imperative learning for local navigation." 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024. [3] Ehsani, Kiana, et al. "SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Thank you for your valuable comments and suggestions. During this period, we quantitively analyze the failure modes in the HM3D benchmarks and append an additional experiment with SpatialRGPT. We find that the failures are mainly caused by the following reasons: 1) Stuck in the local optima. 2) Object detection errors. 3) GPT output errors. 4) Low Imagination Quality. The stuck-in-the-local optima error happens when the GPT 4-mini selects the explored directions. Object detection errors occur because of the rendered image quality in HM3D and HSSD. Sometimes, it tends to happen when similar objects appear simultaneously (sofa and chair). The GPT output errors are sometimes the hallucination happens and the GPT mistakenly thinks it has found the target objects and decides to stop another round of the decision process. The low imagination quality can sometimes lead to the wrong exploration direction and cause episode failure. But it is not the main reason.

We also follow your suggestions and conduct an ablation study using open-source SpatialRGPT to infer the direction. We use both perfect imagination and generated imagination to evaluate the planning ability of SpatialRGPT. The performance is shown in the table and we find that the SpatialRGPT can get similar performance on the HSSD dataset but gets confused using the HM3D-rendered images as inputs.

Response to the concerns about computationally intensive novel view synthesis: Our method only needs to imagine the future observation at sparse timesteps, and on the contrary, the mapping and localization-based approaches require estimating the self-location and constructing the map in real-time. Although the current version of the novel-view-synthesis process still requires tens of seconds, but in the future, a lightweight model can further enhance the efficiency of our framework.

Thank you for taking your precious time to review our paper. We are wondering whether the evaluation of different VLMs and the quantitive failure modes analysis help address your concerns. We are looking forward to your reply.

Thank you for proposing your constructive concerns and questions. We try to deal with your concerns in the following content.

Q1: There are some concerns about the imagination module. The ImagineNav framework relies on the diffusion models for novel view synthesis. However, these could lead to some mistakes e.g. non-existing objects in the scene. This could lead to incorrect navigation decisions by the VLM and reduce the overall success rate.

Although the pre-trained novel view synthesis model cannot guarantee consistency in the test scenes, without additional in-domain fine-tuning, the imagined non-existent objects will not significantly influence the navigation performance. This is because imagination is only used to guide the VLMs selection process, and the object detection is still based on the real observed images. According to our experiment results, although the imagined images can get around 1/3 percentage (1331/4926) with non-existent objects, the failure episodes ratio caused by this is only 3/100. And we list all the failure reasons and their proportion in the table below. The most common failure reasons are still the detection error instead of the imagination error.

Q2: A small concern about this approach is its performance in multi-room or occluded scenarios. The use of human habits without interactive perception could cause the robot to become trapped in local optima, preventing it from locating the target.

Thank you for raising these important concerns. We agree with the idea and how to gather the massive interactive exploration data to enhance the where2imagine module will be one of our future works.

Q3: The framework seems to only rely on immediate views for navigation decisions, without fully utilizing historical information from the navigation process. This lack of memory may limit the robot's ability to explore effectively and plan global paths over long distances or in intricate environments.

During the rebuttal period, we try to incorporate the history information in the planning process by concatenating a fixed number of history frames as additional inputs for VLM. Thanks for your suggestions and we find all the original methods can gain some improvements in both success rate and SPL. The comparison between the original approach with or without history frames is shown in the table below. We will calibrate all the metrics with history inputs in the paper tables after the rebuttal.

Q4: But the quality of the generated novel view (especially the emergence of non-existing objects) is concerned.

More visualizations are appended in the supplementary materials and as the statement in the answer of Q1, the non-existent objects from imagination will not significantly influence the performance as the object detection module still based on the real observation input to ensure whether the agent has found the final targets.

Thank you for raising the suggestions and concerns. And we try to answer your questions in the following content:

Q1: How often does it hallucinate non-existing object or leave out objects that should be there? How does the rest of the pipeline deal with this?

An efficient object exploration process should best utilize the prior knowledge about indoor scenes. As the novel-view synthesis modules are often trained with indoor scene datasets (such as RealEsate10k), the imagination module tends to imagine images that obey the training data. Although generated novel-view images may not ensure the object’s permanence, as our pipeline only imagines the future views at sparse timesteps, not every step, to provide additional information for VLMs planning, the imagined non-existent objects will not significantly influence the agent performance. We quantitively analyze 100 episodes process and visualize all the generated imagined images. We find there are 1331 images (4926 in total) containing non-existent objects compared with the label-rendered images in the scene. But the failure caused by this is only 3 episodes. This is because imagination is used as prior knowledge for exploration guidance, but an additional object detection module running on the real images is utilized to determine when and where to stop.

Q2: There is no way to ensure the waypoints generated by Where2Imagine is reachable, especially in out-of-distribution scenarios. The method still needs some kind of local map for low-level path planning. Some tests on real robot or new environments will erase this concern.

We quantify the success rate of the generated waypoint coordinates from the Where2Imagine module. Success is defined as the minimum distance between the waypoints to the scene obstacles is greater than 0.25m. We evaluate two versions of the Where2Imagine module, which differs in input modalities (RGB, Depth), and the performance is shown in Table below. All the evaluations are performed in the novel scenes, as the training data follows the Habitat-Web data, which uses MP3D as the data collection scenes. The evaluation scenes are HSSD and HM3D. We randomized the agent spawn pose 1000 times and let the Where2Imagine module predict the waypoints. Although the raw predicted waypoint can be unreachable, the low-level action predictor can deal with such scenarios. And we will implement the real-world robot experiment after the rebuttal period.

Q3: The method does not deal with the ambiguity of object reference. For example, navigate to the chair (example used in Fig. 2) is very ambiguous as there might be many chairs in the environment.

Thank you for raising the important question. In this paper, we mainly discuss the object navigation problem, where a target object category is proposed as the goal. And find any of the chairs can satisfy the success criteria. But as the VLMs are flexible and we also use open-vocabulary object detections, our framework can be used for instance-level object searching.

Q4: Some details are very unclear, especially on the novel view synthesis (i.e. future view imagination). What data is used to train the diffusion model? Is it in or out of distribution for indoor navigation? How about resolution and the speed?

We use the pre-trained novel-view synthesis model proposed in PolyOculus: https://yorkucvil.github.io/PolyOculus-NVS/. No additional in-domain data are used to fine-tune the novel-view synthesis model. And the generated image resolution is 256x256. The generation speed of 6 images with a single RTX 4090 card is around 40 seconds.

Q5 & Q6: More qualitative visualizations of the environments, imagined views, and waypoint distribution would be nice. The structure of the VLM analysis output in Fig. 3 and Fig. 4 are inconsistent. Which one is used in evaluation?

Thank you for your kind suggestions. We append more visualization in the supplementary materials Figure 10. The Figure 10 contains both the visualization of the Where2Imagine output and the corresponding imagined images. In the experiments, we follow the VLMs output process shown in Figure 3, the Figure 4 is a summarized output from the output in Figure for illustration.

Q7: Why is the success rate lower with NVS model added in Table. 2 (row 3 vs 5)? More explanation is needed. The row3 in Table 2 means that the agent uses the oracle imagination (the test scene renderer) and the row5 in Table 2 means that the agent uses a pre-trained novel-view-synthesis model. This gap can help evaluate how the imagination quality influences navigation performance. And we modify Table 2 to avoid possible ambiguity.