Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation

Thanks for your valuable comments.

1. Model Parameter Size Analysis

As shown in Section 4.1, our Dynam3D has an advantage in terms of parameter size compared to major large model-based VLN methods: NaVid 7B, Uni-NaVid 7B, and Dynam3D 3.8B. As shown in Tables 1 and 2 of the original submission, our Dynam3D achieves stronger performance with the smaller parameter scale. For more ablation study analysis to more clearly demonstrate the performance contribution of each component in Dynam3D, please refer to our response to Reviewer QXMr.

2. Robustness Analysis to Noise

Thanks for your insightful suggestions. As demonstrated by the real-world experiments in Section 4.3 of the original submission, our Dynam3D exhibits excellent performance in real-world navigation based on the Stretch 3 robot. This reflects the robustness of our Dynam3D in real-world environments. A ROS Nav2-based SLAM system is developed for precise localization and pose estimation on the Stretch 3 robot by combining sensory data from LiDAR, IMU, and a depth camera. Although no official data is available, user reports and our rough estimates suggest a positioning accuracy of approximately ±5 cm and an orientation accuracy of about ±3 degrees. For the Intel RealSense D435i depth camera used in Stretch 3 robot, its official specifications indicate a depth error of approximately ±4 cm within a 2-meter range and ±6 cm within a 3-meter range. To further validate the practical impact of this noise on navigation performance, we add different types of noise to the simulator and evaluate the performance of our Dynam3D on R2R-CE Val Unseen in noisy environments. In order to simulate SLAM noise, we introduce localization noise, uniformly sampled from -5 cm to +5 cm, and orientation noise, uniformly sampled from -3 degrees to +3 degrees, to the agent's position at each simulation step. To simulate depth camera noise, we add random noise to the depth maps acquired from the simulator. For depths within 2 meters, noise is sampled between -4 cm and +4 cm; for depths at 2~3 meters, noise is sampled between -6 cm and +6 cm; for depths over 3 meters, noise is sampled between -10cm and +10 cm.

As shown in the table above,even with the simultaneous addition of simulated SLAM noise and depth noise, the navigation success rate (SR) only decreased by approximately 2% (comparing the last row to the first row), demonstrating Dynam3D's robustness to noise. A significant reason is, unlike point cloud-based 3D-VLM models require near-perfect point clouds for their point cloud encoder to extract semantics based on texture and fine geometry, our Dynam3D leverages 2D foundation models (e.g., CLIP, FastSAM) to extract visual semantic features (patch-level), which are unaffected by SLAM and depth noise. Our Dynam3D acquires 3D-aware representations through a hierarchical encoding of patch-instance-zones, even with minor spatial position deviations due to noise, this strategy remains sufficient for perceiving spatial geometry and scene layout at the object and zone levels.

3. Comparing VLN in Discrete and Continuous Environments

In the discrete environment VLN, a predefined navigable connectivity graph with nodes is used. The agent can only move between these nodes in the graph and is typically equipped with a panoramic RGB-D camera. Due to the limitations of the computational resources and model capacity in few years ago, discrete environments significantly simplify the navigation process for easier research. For example, it eliminates the need for low-level action prediction, ignores collision avoidance, and disregards potential environmental changes.

However, a significant drawback is that navigation models trained in discrete environments are almost impossible to deploy on real robots due to two main reasons: 1) Real-world environments rarely have perfectly predefined navigation connectivity graph, and 2) The vast majority of robots do not possess panoramic RGB-D cameras due to their large size and high cost. In contrast, continuous environments (VLN-CE) , especially with a monocular camera setting, show greater practical value. Although VLN model performance on continuous environment benchmarks (R2R-CE, REVERIE-CE) has not yet surpassed that of discrete environments (R2R, REVERIE), the gap for real-world deployment is significantly reduced. The continuous settings require models to predict low-level actions, e.g. rotation and forward movement, and handle collision avoidance. What's more, continuous VLN largely mitigates a key limitation, i.e. discrete VLN models tend to overfit to perfect connectivity graphs, which severely restricts their generalization in real-world navigation.

4. Discussion for the application scenarios of Dynam3D and the requirements for data quality.

Dynam3D is designed for language-guided indoor navigation tasks carried out by robots equipped with precise SLAM and monocular RGB-D cameras, and is capable of adapting to real-time human and object movements.

For data quality, posed RGB-D images are required as model input, the specific accuracy requirements for depth images and camera poses can be found in the Robustness Analysis to Noise section.

Thanks for your insightful comments and suggestions!

1. Analysis of Dynam3D's Generalization Capability

1) 3D Representation Generalization

As detailed in the Supplementary Material (Line 503 of the original submission), we follow SceneVerse and g3D-LF to collect over 5K scenes with 2M language annotations from ScanNet, HM3D, Matterport3D, 3RScan, ARKitScenes, and Structured3D. This data is used to train our Dynam3D representation model and retrain the g3D-LF feature field model. HM3D, Matterport3D, and Structured3D are from simulators or renderers. The remaining training data ScanNet, 3RScan, and ARKitScenes provide over 2.5K scenes captured using real RGB-D cameras in real-world environments. This helps eliminate the sim-to-real gap and ensures strong generalization of the learned representations to real-world scenarios.

2) VLN Model Generalization

As detailed in the supplementary materials (Line 503 of the original submission), we conducted mixed training across 861 environments (HM3D, MP3D) to enhance the navigational generalization of our Dynam3D. This involved over 4 million navigation samples, comprising both model-generated data (ScaleVLN, NavRAG; 4M+) and human-annotated data (R2R-CE, REVERIE-CE; 20K+). This diverse dataset included various navigation instructions, such as step-by-step following, high-level description, and user-demand instructions. To further validate the generalization performance of our Dynam3D, we conduct zero-shot evaluations on the HM3D-OVON Val Unseen dataset for open-vocabulary object navigation to show the cross-dataset performance and zero-shot generation:

2. Optimize inference latency with better action prediction policies

The dynamic 3D-VLM contexts arising from real-time updates of rendered panoramic patches, instance and zone tokens preclude the application of static-context inference acceleration techniques such as KV caching that is commonly employed in traditional video-based VLMs. However, as suggested by Reviewer kSWU, our Dynam3D has the potential to accelerate the navigation process using multi-step forecasting or diffusion-based policies by using high-quality 3D-aware representations. For example, a large 3D-VLM can execute high-level encoding and reasoning at 2 FPS. A smaller action expert model can use latent features from the 3D-VLM (similar to the VLA model π₀) as input to predict multiple consecutive navigation trajectory points at over 20 FPS. During robot movement, the 3D representation is continuously updated in real time at over 10 FPS. Currently, our Dynam3D still uses text output for action prediction primarily to ensure a fair comparison with previous video-based models that also used text output. However, this navigation action output mode should only be a temporary solution. Once sufficient training data is available, smoother and faster navigation can be achieved with the better choice of predicting multi-step continuous trajectory points.

3. Better demos to showcase spatial reasoning ability

Thank you for the suggestion. As NeurIPS does not support media uploads this year, we are unable to include additional demos. Nevertheless, our Dynam3D exhibits clear advantages over video-based models in 3D perception, localization, and reasoning with better 3D-aware representations. If paper is accepted, we will provide more demonstrations in the camera-ready version to highlight its spatial reasoning capabilities and potential in object-centric tasks such as mobile manipulation, which require precise object localization and geometric understanding.

4. Dynamic Representation and Lifelong Memory in Section 4.2

Thank you for your suggestion. In our Dynam3D, the robot discards outdated representations within its current field-of-view and adds or updates new ones to ensure real-time understanding of dynamic environments with each new observation. However, this does not conflict with its advantage of maintaining lifelong memory. Features corresponding to regions outside the robot’s current view of the robot are still preserved and not updated, and serve as long-term memory to support navigation. This is due to the limited field of the monocular camera, where the majority of the scene is preserved since it lies outside the current view of the robot. These features are retained until new observations covering those regions are available to update them. This highlights the ability of our Dynam3D to balance real-time adaptability in dynamic environments with the effective use of long-term memory.

Thank you very much for your insightful comments! In response, we include some previously omitted experiments and added new results. This provides a clearer presentation of the contributions of our method.

Table 1. More experiments for ablation study.

1. The contribution of rendered panoramic 3D tokens, instance tokens, and zone tokens.

As shown in Table 1, the removal of rendered panoramic patch tokens (Row 2), instance tokens (Row 3), zone tokens (Row 4), or both instance and zone tokens (Row 5) from the full model (Row 1) lead to performance drops of varying degrees. The impact on performance follows the order: patch > instance > zone tokens.

2. The significance of complex representation designs, such as rendering panoramic tokens using g3D-LF and obtaining instance-level representations via FastSAM. Their effectiveness for pre-exploration and lifelong memory.

As analyzed in VLN-3DFF[1], rendered panoramic patch tokens substantially mitigate the limited field of view caused by monocular cameras on most robots. Comparing Rows 1 and 6 in Table 1, replacing rendered panoramic tokens with CLIP patch features extracted from the current monocular view results in about a 10% drop in navigation success rates on both R2R-CE and REVERIE-CE. Moreover, lifelong memory experiments in Table 3 of the original submission show that tokens rendered from feature fields have greater potential to leverage pre-exploration (although the comparison is not entirely fair) and lifelong memory for performance improvement than the methods that directly extract patch-level features from current observations. For obtaining instance-level tokens via FastSAM, rows 1 and 3 in Table 1 demonstrate the performance gains brought by instance tokens. Moreover, instance-level representations are highly valuable for object-centric tasks in large-scale scenes when extending 3D-VLMs, such as instance grounding and mobile manipulation, although Dynam3D has not yet attempted such applications.

3. The heterogeneity of different types of tokens, such as patch, instance, and zone tokens, does not overly confuse the model learning.

Although we do not explicitly define learnable token-type embeddings, each token type has its own positional encoding network. This allows the 3D-VLM to distinguish between different token types. Additionally, during representation pretraining, patch, instance and zone tokens are aligned or distilled into the CLIP semantic space using corresponding language and 2D visual features at varying granularities. Despite sharing the same semantic space, each token type exhibits distinct semantic granularities.

4. Merging Discriminator Accuracy.

Evaluated on MP3D scenes from the R2R-CE Val Unseen split, the merging discriminator achieves 78.6% accuracy in determining the merger of 2D and 3D instances. Although not perfect, the frustum culling strategy removes most near-field old instances, leaving only instances near the view boundaries and at a distance to be merged. Thus, the current accuracy is sufficient to effectively support instance token updates.

5. More details on the processing of VLN data.

As detailed in the Supplementary Material (Line 503 of the original submission), we use a pre-trained waypoint predictor[2] to generate candidate waypoints and select the GT waypoint closest to the ground-truth (GT) path and destination via the simulator API. The agent can achieve 96 SPL and nearly 100% success rate (SR) on R2R-CE Val Unseen when it strictly follows these GT waypoints. This provides reliable supervision for training. Each next GT waypoint yields a heading angle (in 15° increments) and a forward distance (in 25 cm increments), which are used as GT textual action supervision for the 3D-VLM model.

[1] Wang et al. Sim-to-real transfer via 3d feature fields for vision-and-language navigation. In 8th Annual Conference on Robot Learning, 2024.

[2] Hong et al. Bridging the gap between learning discrete and continuous environments for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.

Thanks for your sharing. Good work.

1. Handling Large-Scale Scenes, e.g., Outdoors

Thanks for your valuable comments! Although currently primarily designed for indoor navigation, our Dynam3D model architecture is in fact also suitable for outdoor navigation.

1) Representation and Computational Overhead

Unlike fine-grained voxel-based methods, our instance and zone-level tokens effectively handle large-scale scene representation without introducing an excessive number of tokens and huge computational overhead. The number of 3D patch tokens rendered via feature fields is fixed at 12x48. Furthermore, the KD-Tree feature retrieval and sparse sampling strategies employed in HNR and g3D-LF ensure highly efficient feature field rendering. Even in outdoor urban scenes, a single RTX 4090 GPU can render 12x48 panoramic patch tokens at over 15 FPS.

2) Generalization and Training Data

Although the model architecture of our Dynam3D can be adapted to large scenes and outdoor navigation, its current generalization capabilities for outdoor navigation are limited because the training data of both the 3D representation and the 3D-VLM model originate from indoor environments. However, it’s possible to fine-tune and adapt the model using outdoor navigation datasets such as Touchdown, map2seq, and VLN-VIDEO.

3) Outdoor Robot Deployment

For indoor navigation, the Stretch 3 robot used in the Dynam3D experiments is equipped with a precise SLAM system, accurate camera pose estimation, and sufficient depth camera precision. However, the localization system and depth camera do not perform well in outdoor environments due to the narrow field-of-view of the LiDAR on the robot. Nonetheless, this issue can be easily mitigated by equipping the robot with a wide-angle 4D-LiDAR to increase the field-of-view. Our Dynam3D model supports sensory inputs from the wide-angle LIDAR with no need for drastic modifications.

2. Handling Large Inter-Frame Changes

The frustum culling of Dynam3D and its token addition, deletion, and update strategies are well-suited to handle significant changes between observation frames.

1) Large robot and camera movements leading to significant inter-frame changes.

In such scenarios, the overlap between the current frame and historical frames is minimal. Our Dynam3D handles this situation effectively by primarily adding new patch, instance, and zone tokens. For example, it directly encodes 2D instances as new 3D instances as described in lines 152–156 of the original paper. This eliminates the need for extensive 2D-to-3D instance merging.

2) Rapid movement of people or objects within the scene causing huge inter-frame changes.

Frustum culling can effectively delete old features and update new ones when a moving object moves away from or traverses out of the camera field-of-view. However, frustum culling may fail to remove outdated features from earlier frames when objects move closer to the camera. This is due to occlusions caused by objects in the new frame. A potential future improvement is to address this issue using 3D object or point tracking methods.