AutoNeRF: Training Implicit Scene Representations with Autonomous Agents

The input of the system is unclear. For exploration, RGB frames along with the depth maps are required (Sec. 4 Task Specification), while for NeRF training, the depth information is not leveraged. It is highlighted in Sec. 4.2 that 'no depth information is required for reconstruction' is an important property. From the reviewer's perspective, this is merely the characteristic of NeRF itself. For the proposed AutoNeRF, the depth information is essential for the proposed exploration module to update the 2D top-down map.

The exploration policy indeed requires depth input. The lack of required depth information at reconstruction time has different implications:

1. This is a quality of NeRF models, motivating the interest in such 3D scene representation and thus to focus on how to autonomously collect NeRF training data.
2. Previous work (Gervet et al., Navigating to Objects in the Real World, Science Robotics, 2023) has shown that RL-trained modular policies can be deployed on real robots with low-resolution noisy depth sensors. However, in such a scenario, while the navigation policy will be able to properly explore the environment, a low-resolution noisy depth input might lead to poor 3D reconstruction with methods relying on depth, showing the interest of our framework. We could have a policy to navigate properly from low-quality depth and, in a second stage, obtain satisfying reconstruction from NeRF that does not depend on depth sensing quality.
3. Finally, the fact that such NeRF models do not require depth input is highly dependent on the quality of the underlying collected data. Without a proper diversity of camera frames and viewpoints, NeRF training is likely to fail. By following standard practices of training NeRF models without depth input, we increase the importance of the active data collection part in the final NeRF model performance, allowing to better evaluate considered baselines.

How the automatic process helps the downstream tasks is unclear. It is known that a trained NeRF can be applied for these four tasks (as also mentioned in the Related Work section). The paper directly adopts the methods without further modification, and the NeRF training does provide any feedback to the exploration policy training. Moreover, the exploration phase already builds a 2D map online that can be used for Task 2&3. The benefit of involving an extra NeRF training phase for these two tasks should be further elaborated.

Downstream tasks are used in this work as an evaluation protocol of the underlying quality of a given NeRF model. While many recent works focus on evaluating NeRFs through novel view synthesis, we consider a suite of tasks that allow a more thorough evaluation. Among these tasks is occupancy map estimation from the NeRF representation which allows to verify the geometric understanding of the 3D scene by the NeRF model.

Though the paper conducts thorough experiments on multiple tasks, merely 5 Gibson scenes are evaluated. It should be noted that the methods of [Chaplot et al., 2020a] and [Ramakrishnan et al. (2021)] are evaluated on Matterport3D besides the Gibson datasets. As the paper targets an AutoNeRF problem, the method is expected not to be limited by the data-driven exploration policy. The generalization ability of the proposed modular learning strategy should be discussed compared to the relevant one-phase works of A-C.

All baselines in this work are evaluated on 5 validation scenes that are different from considered scenes at training time.

The difference with previous work can be easily explained when considering the time involved in evaluating one agent episode on a new scene: in previous work ([Chaplot et al., 2020a] and [Ramakrishnan et al. (2021)]), such an evaluation simply involves neural network forward passes and simulation renderings. All of this can be done very rapidly, leading to an evaluation time of a few seconds for a given episode (with the ability to parallelize to have even faster evaluation). In our case, evaluating one episode involves navigation and simulation like previous work, but also training a NeRF model for this specific episode in order to evaluate its quality. Semantic Nerfacto takes up to 30 minutes to train in large house-scale scenes, which means that one agent rollout takes 30 minutes to evaluate against seconds for previous work. We have already put a lot of computation into our evaluation protocol in order to assess the generalization abilities of all the proposed agents.

As the authors attempt to cover a wide range of tasks, the presentation of results lacks organization, making it challenging for readers to interpret and examine the outcomes.

The different considered downstream tasks allow to more thoroughly evaluate the performance of the policy in its ability to collect training data that will lead to high-quality NeRF representations, which is one of the contributions of this work. This is in stark contrast to a large body of recent work which solely focuses on novel view synthesis, while we attempt to go one step further and focus on concrete applications in robotics. Please refer to our global answer to all reviewers for more details about the contributions of this work and the conclusions drawn from experiments.

Could you please share the results of the vanilla Semantic NeRF model mentioned in Figures 9 and 10? While it is claimed to deliver superior results compared to Semantic Nerfacto, there is a lack of supporting evidence.

The vanilla Semantic NeRF model leads to a higher-quality 3D mesh after applying our mesh extraction protocol. The difference was assessed qualitatively by witnessing holes and irregularities in the Semantic Nerfacto final mesh. The latter still provides satisfying results as can be seen quantitatively and with qualitative BEV maps, and is much faster to train, but our aim here was to show the best practical results in terms of final 3D scans one could get when applying AutoNeRF.

It appears that the four types of rewards exhibit different strengths. Are there any criteria or trends that can assist researchers in selecting the most appropriate reward type for specific tasks? For instance, is there a reason why the policy with the obstacle coverage reward performs better than others in the rendering task?

Different reward functions can lead to diverse policy behaviors, which is why we report the performance of the different modular policies on all the downstream tasks. The chosen reward might depend on the specific considered task, but in this work we were rather interested in finding the reward function that leads to the best overall performance on the different downstream tasks. From collected results, obstacle coverage and viewpoints coverage lead to the best overall performance.

In this work, the rendering ability of a NeRF model is evaluated by sampling camera poses in the scene and comparing NeRF rendering with rendering from the simulator. The difficulty of the task is thus linked to rendering different objects that might not have been observed thoroughly, while redundant parts of the scene such as floors, walls, ceilings have likely been observed from different viewpoints by any policy with ease. This explains the high performance of the obstacle coverage reward, as it guides the policy to focus on covering specific objects or obstacles in the scene.

Could you provide a more detailed explanation of the experiments featured in Table 1 and elaborate on their implications?

Table 1 presents an application of AutoNeRF to a practical scenario where a user would have bought a brand new robot that navigates using a neural policy (different from AutoNeRF) that was pre-trained in simulation. They now want to use the robot in their house, but even if the robot policy was pre-trained on a large and diverse set of scenes, it is quite likely it will struggle with some specificities of their house. A solution to this would be scene-specific fine-tuning: fine-tuning the robot policy on their house. However, such fine-tuning must be performed safely so a good thing would be to do it in simulation. A question is thus: can we use AutoNeRF to autonomously explore a new scene, collect data and train a NeRF? We can then extract a mesh representation from the trained NeRF, load this mesh into a simulator and fine-tune the robot policy of interest. Table 1 shows results of similar experiments performed in simulation: we show that such a procedure allows to improve the performance of a PointGoal agent, pre-trained on diverse scenes, by fine-tuning it on previously unseen scenes scanned by AutoNeRF.

The novelty of the paper is limited. The idea of using Nerf for scene construction in SLAM is not new. The authors also adopted an off-the-shelf NERF module. The main contribution of this paper lies in validating the effect of different exploration policies during image collection on downstream robotics tasks.

Please refer to our global answer to all reviewers for a detailed description of the contributions of this paper.

How many images are used to train the NERF model? In task specification it says "The agent can navigate for a limited number of 1500 discrete steps", does it mean to capture 1500 images for NERF training?

The agent can indeed navigate for 1500 steps, leading to a training set that is made of up to 1500 training samples. However, we perform a filtering to remove images associated with close camera poses. In practice, when deploying the modular policy trained with the obstacle coverage reward on previously unseen scenes, the mean number of collected images is 754, with a minimum of 264 in the smallest scene and a maximum of 1257 in the largest scene.

The scientific contribution of this paper is unclear. Although the authors conduct a comprehensive evaluation of the designed autonomous approaches, it does not point out their core contributions to this field. The proposed autonomous exploration strategy is straight-forward, and could not be considered as main contribution of this paper. Additionally, the absence of a comparison between their methods and existing techniques in autonomous visual exploration—which could have been utilized for collecting NeRF training data—is missing. The experimental evidence alone does not constitute significant scientific contribution.

Please refer to our global answer to all reviewers (“Contributions of this work” section) for a detailed description of the contributions of this paper. As mentioned, our contributions are on the experimental side, and we do compare the introduced modular policies with state-of-the art end-to-end visual exploration methods (Ramakrishnan et al. An exploration of embodied visual exploration, IJCV 2021). We believe that autonomous NeRF training data collection is an important topic and, to the best of our knowledge, this is the first work to target this direction, providing diverse experimental results.

The claims are backed by empirical results, but the lack of theoretical analysis or concrete mathematical justifications for the algorithm.

We follow practices in the Embodied AI domain, where contributions are evaluated experimentally by assessing the ability of introduced methods to properly navigate in previously unseen environments at test time. Indeed, all 3D scenes used at test time were not seen during training by any policy. We thus respectfully believe that such a theoretical analysis is out of the scope of this work, while being an interesting future direction to keep in mind.

Did you evaluate the end-to-end based and modularized based method on completed unseen (or largely different) scenario? I am curious about the actual performance in such scenarios. While the rule-based method, i.e., the frontier-based exploration, did not outperform learning-based methods (such as end-to-end based and modularized methods) in your experiments, it demonstrated fairly consistent results across the majority of scenes, regardless of whether they were familiar or unfamiliar scenarios.

Please refer to our global answer to all reviewers (“Generalization to new scenarios” section) for a description of our experimental setup which applies to end-to-end and modular methods.