NextBestPath: Efficient 3D Mapping of Unseen Environments

Thank you for your comments and for recognizing that our model is well-designed and the results are good.

W1: The networks for obstacle and value maps prediction are simple, and using networks to predict value maps for guided exploration is not new. The technical contribution of this work is weak.

Our main technical contributions lie beyond the network design.

First, we propose a new paradigm by shifting from state-of-the-art next-best-view prediction to next-best-path prediction for active mapping.

Second, compared to prior works that also predict long-term goals, we propose a new criterion for selecting long-term goals based on coverage gains, which is more closely aligned with the final objective of active mapping. We also propose an efficient data collection method and training strategy for training the coverage gain decoder.

Finally, we unify the models for obstacle prediction and value map prediction, while prior works typically use separate models for navigation and exploration goal prediction. Our unified model is more efficient and the multi-task learning further enhances the performance.

We will make these contributions more clear in the revised version.

W2: Some of the paths computed by the proposed method are too close to the walls, see Fig 4(b) left, making them collision-prone. The visualization of paths does not have to show shadows.

The visualization may make the trajectory look like it is close to the wall, but this is only due to the scaling and perspective of the cameras, not actual proximity.

After testing, the minimum distance to the closest obstacle along our predicted trajectories is 0.6 meters for the trajectory in Fig 4(b), which ensures no risk of collision. Additionally, collision checking is implemented in our simulator to verify the validity of the path.

We include the shadowing in the images to highlight that this is a 3D reconstruction task, not a 2D task. For better visualization, we included a video in the supplementary material, which more clearly demonstrates the active mapping process of our method.

W3: It would be more useful if more test can be conducted to find out how to choose the size of range for a given scene.

Thank you for your suggestion. This is an interesting future direction to explore. Our experiments in Figure 5 show that the range of 40m achieves the best performance on average for all the scenes. To further investigate, we analyzed whether this range performs best for each individual scene, and found that 76.67% of the evaluated scenes achieve their best performance with this range. This result suggests that this hyper-parameter can generalize effectively across different scenes.

W4: It is unclear whether the other alternative methods being compared were trained on AiMDoom or pretrained on other datasets. If so, the comparison would be unfair.

We did retrain all the methods we compared in Table 2 on the AiMDoom dataset for a fair comparison. We will make this clear in the paper. We will also release these model weights upon acceptance.

Thank you for your comments. We address each point in the following.

W1: The scenes in AiMDoom contain minimal furniture or objects, resulting in mostly open space. This does not align with real-world environments, making these scenes suboptimal for training and evaluation purposes.

Existing methods face challenges in actively mapping large and complex 3D scenes. The proposed AiMDoom dataset mainly aims to provide a systematic benchmark for evaluating models’ capabilities across various difficulty levels of scenes for active mapping. As shown in Table 1, AiMDoom has greater navigation complexity than existing real-world datasets. It is also easier to scale up in dataset size with the automatic generation code.

W2: A discussion is needed to explain why this paper considers only a 3-DoF setting, while baseline methods, such as MACARONS and SCONE, support 6-DoF camera pose planning.

W3: For 3-DoF trajectory planning, several well-known works exist. The authors should discuss why this paper’s approach offers advantages over previous works.

Though baseline NBV methods for single objects or outdoor scenes use a 6-DoF setting, existing methods for indoor 3D mapping such as those evaluated in the MP3D dataset [1,2,3], commonly utilize a 3-DoF setting with actions limited to turning left, turning right, and moving forward. In alignment with these prior works and the focus on indoor 3D scenes, we keep the 3-DoF setting as in previous work.

In Table 3 we already do compare against several recent works for 3-DoF trajectory planning. Thank you for mentioning the TARE paper on 3-DoF trajectory planning. TARE employs a hierarchical strategy for exploration based on non-learning control and planning optimization; its viewpoint sampling process in subspace is confined to the sensor's range, where they use additional lidar sensors. In contrast, our proposed learning-based method can predict optimal poses and potential obstacles over a broader range even using only a single depth sensor. Due to differences in focus areas of the tasks and settings in simulators, we did not compare against TARE methods in our experiments. We will include discussions of these studies in the related work section.

Q1: Do the obstacle map and value map encompass the entire scene? This could result in significant computational costs in large-scale environments.

We crop the scene centered around the current camera position for obstacle map and value map prediction. We convert the 3D point cloud into a stack of 2D images as inputs, which makes it more scalable to large environments. You could refer to Section 2 of the supplementary material for the details.

Q2: If the long-term goal is updated at each step, does this strategy enhance performance?

Thank you for the suggestion. It is interesting to explore. We will conduct further experiments on this issue in the coming days.

Q3: In Table 2, why is the comp. metric considered valid?

This metric evaluates the completeness of the reconstruction. Since all the methods use the same maximum budget for exploration, the comp(%) metric will be low if a model fails to explore some areas within the budget. Therefore, existing methods on the MP3D dataset all use this metric.

[1] Active neural mapping, ICCV 2023

[2] Occupancy anticipation for efficient exploration and navigation, ECCV 2022

[3] Uncertainty-driven planner for exploration and navigation, ICRA 2022

We conducted this experiment on the AiMDoom Normal level, extending our previous ablation studies. The table below shows the results, the Original Strategy adheres to the original approach of updating long-term goals upon completing a path, while the New strategy updates goals at each step.

The results indicate that the New Strategy, which frequently updates long-term goals, performs worse than the Original Strategy. This inferior performance is mainly due to the lack of decision continuity in the New Strategy, where the agent frequently changes its long-term goals. Such frequent shifts can cause the agent to oscillate between paths, wasting movement steps, particularly as our experiments were conducted with a limited number of steps. Additionally, the predictive accuracy of the value map is not perfect, and forecasting over long distances naturally entails uncertainty. New Strategy accumulates more predictive errors by recalculating predictions at every step, and frequent updates in decision-making can exacerbate these errors.

Despite these challenges, our results still surpassed the performance of previous state-of-the-art next-best-view (NBV) methods, as detailed in Table 2 of the main paper. This suggests that predicting coverage gains over long distances can indeed benefit efficient active mapping, even when the goal is updated at each step.

We thank the reviewer for the detailed comments. We address the raised points in the following.

W1: The active mapping task addressed in the paper focuses solely on maximizing coverage and relies on the assumption of accurate poses.

W2: The paper is set in a very narrow context and lacks discussion with active SLAM.

W5: We would like to see comparisons with active SLAM methods (Chaplot et al., ICLR 2020) and (Bircher et al., ICRA 2016).

It is true that our work focuses on maximizing coverage and assumes accurate poses, however, this is also true for numerous earlier work which we reference in [1-8]. This focus allows the development of algorithms capable of reconstructing detailed 3D models of complex environments while minimizing exploration time, which is still a very challenging task.

Please also note that we outperform UPEN [9] by a large margin, while UPEN already achieved higher map coverage than (Chaplot et al., ICLR 2020) on the MP3D dataset (67.9 % vs. 52.1%). In the case of Bircher et al. (ICRA 2016), they consider sensors and a task setup that differ significantly from ours, making a direct comparison infeasible.

This being said, we agree we should have emphasized the difference between active mapping as in [1-8] and active SLAM, and we will clarify the active mapping problem we addressed in the paper.

Q1. L068: You write "scene uncertainty does not directly align with the ultimate objective of 3D mapping". Is the uncertainty of predicted occupancy not the uncertainty of the 3D map? What do you mean here?

We aim to maximize the coverage of the scene by the camera, because it was shown to be a good criterion for active mapping in [1-8]. Maximizing coverage is related to minimizing uncertainty, but maximizing coverage formalizes better the ultimate goal of active mapping (Please also see our answer above about the emphasis on active mapping).

Q2: The computed information gain during training is using the ground-truth (eq 2) while inference uses the ground-truth instead. It is not clear whether the use of the ground-truth.

During training, we use the ground-truth point cloud to calculate the coverage gain when moving the camera from pose A to pose B along a trajectory. This coverage gain is used to train the coverage gain decoder at camera pose A.

In the inference phase, our model automatically predicts the coverage gain in the value map, and uses it to select the next best path. The ground-truth is only used in evaluation to calculate the AUCs metrics.

Q3: The authors should clarify the first term in eq. 3. What does it mean ground-truth coverage gain?

As replied in Q2, we use the ground-truth point cloud to calculate the coverage gain between camera poses. This is treated as ground-truth coverage gain in Eq (3).

W3: The approach is very similar to (Georgakis, 2022).

1. Georgakis et al. model uncertainty, here the authors predict occupancy and a value map that should have the interpretation of information gain/uncertainty;

2. While Georgakis' objective is point-goal navigation, one can use its exploration policy as a pure mapper.

We respectfully disagree with the statement that our approach is very similar to (Georgakis, 2022):

• (Georgakis, 2022) considers paths as well, but relies on the average of the uncertainties at each point on paths sampled with Rapidly exploring Random Trees (RRTs). This uncertainty average is not really representative of the value of the path, as it is possible that seeing the scene from one point on the path will remove the uncertainties for the other points on the path evaluated on the path. By contrast, we learn to predict the coverage gain obtained by summing all the coverage gains obtained by moving along the path.

In fact, we compared our approach to (Georgakis, 2022) on the MP3D dataset. The results, presented in Table 3 of the main paper and Table 1 of the supplementary material show that our approach outperforms (Georgakis, 2022) by a large margin, with over 10% absolute improvement in the Comp(%) metric. Note that we could not evaluate (Georgakis, 2022) on our AiMDoom dataset due to the substantial resources required to train their entire system (64 GPUs over three days of training for the pretrained navigation model they used). To ensure a fair comparison, all methods we compared in Table 2 were trained on our AiMDoom dataset.

3. Georgakis' value map is based on explicit computation of covariance from ensembles without the use of any ground-truth.

Note that we use the ground truth coverage gain only during training. This ground truth is computed automatically and does not require any human intervention. We thus do not see this as a drawback.

4. Georgakis chose a long-term goal and then estimated paths based on occupancy maps, similar to the approach here.

Please see our answer for Point 1) for the fundamental difference between how we and (Georgakis, 2022) measure the value of a path.

Moreover, their approach incorporates a pretrained navigation model which is limited to only two datasets and requires large GPU resources to train. In contrast, we directly predict an occupancy map and use Dijkstra's algorithm to generate a trajectory for navigation.

W4: The expression in (2) defines the gain as minimal error to the ground-truth. This will not encourage the agent to go to new unvisited directions but rather to directions where the prediction error will be very small.

It seems there is a misunderstanding here: Equation 2 measures the difference in coverage between a new viewpoint and the current viewpoint: This encourages the agent to select viewpoints from which new parts of the scene can be seen.

Additionally, to avoid falling into local optima during training, we use Equation 1 to effectively balance exploration and exploitation.

[1] A reinforcement learning approach to the view planning problem, CVPR 2017

[2] Next-best view policy for 3d reconstruction, ECCV-W 2020

[3] GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction, CVPR 2024

[4] Macarons: Mapping and coverage anticipation with rgb online self-supervision, CVPR 2023

[5] Scone: Surface coverage optimization in unknown environments by volumetric integration, NIPS 2022

[6] Active neural mapping, ICCV 2023

[7] NARUTO: Neural Active Reconstruction from Uncertain Target Observations, CVPR 2024

[8] Active Neural Mapping at Scale, IROS 2024

[9] Uncertainty-driven planner for exploration and navigation, ICRA 2022

We thank the reviewer for providing constructive comments and the recognition of our work.

Q1: In L243, the point clouds are cropped at the current location of the agent. How does it work and what kind of parameters are used? My understanding is that the crop size may influence how much history information is used for the next path prediction.

The point cloud is sliced vertically based on the camera's position and transformed into density images through a projection function that normalizes point coordinates into image space. Key parameters in this transformation include the camera's current position, which centers the projection, and the radius (40m$\times$40m), which defines the observational area around the camera. The image dimensions, set at 256$\times$256, determine the resolution of the output images. This projection function adjusts the 3D coordinates to align within these dimensions, ensuring that the value of each pixel in the resulting images accurately represents the density of the point cloud data.

You could refer to Section 2 of the supplementary material for a clearer understanding of these details and the formulas.

The larger the crop size, the more history information is used for the next path prediction. We will provide an ablation on the crop size for your interest in the next few days.

Q2: In L245, the 3D point clouds are projected onto 2D image to simplify the processing. This strategy works for scenes with a single layer but may lose generalization ability in scenes with multiple layers as part of the depth information is discarded.

We use a stack of 2D images rather than a single image to represent the scene, which has the potential to capture essential depth information even for scenes with multiple layers. Since prior works focus on single-layer scenes and it is already challenging to reconstruct large single-layer scenes, we do not prioritize results for multi-layer scenes in this work.

Q3: In the ablation study, the efficacy to the final reconstruction results of both the obstacle map and multi-task training are tested, however, it would be good to see the accuracy of the obstacle map itself and the value map itself instead of final reconstruction accuracy.

Thank you for the suggestion. For the prediction of the value map, it is prohibitively expensive to obtain ground-truth coverage gains for all pixels in the map. This is why we use per-pixel data for training value maps, and thus we cannot report the performance of value map prediction in inference.

For the obstacle map, we can evaluate its performance given the ground-truth obstacle map. The table below presents our experimental results. The results from the ablation study continue to demonstrate that multi-task learning outperforms training tasks independently, indicating that the tasks are mutually beneficial and collectively enhance learning.

Q4: The new dataset is synthetic, which may create a domain gap. A more recent version of the Scannet dataset could be used in the future.

Thank you for your feedback. Considering ScanNet++ is certainly interesting, however our dataset is designed to address the challenge of actively reconstructing large-scale complex 3D environments, which is one of the key difficulties for existing methods. As indicated in Table 1, real-world datasets including the Scannet are limited by smaller navigation spaces and lower complexity. The ScanNet++ you mentioned does not improve upon these aspects. We also plan to release our codes and models, enabling future research to evaluate our methods on other datasets like Scannet++.

We conducted this ablation study on crop size, training four different models on the AiMDoom Normal level training split. These models processed input crop sizes ranging from 20m × 20m to 50m × 50m, with each model tasked with predicting a value map and an obstacle map within a 40m × 40m area. The table below shows the results.

The results indicate that the best results are achieved when the input crop size matches the crop size of the area being predicted. This is because when the input crop size is either smaller or larger than the crop size of the output maps, it leads to predictive errors. If the input crop size is too small, it restricts the model’s ability to formulate effective long-term goals. Conversely, if the input crop size is too large, the predictions for obstacles near the camera become inaccurate, adversely affecting both exploration and reconstruction efficiency.