OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views

We thank the reviewer for the constructive and overall positive feedback.

We are pleased that you appreciate two fundamental aspects of our work: The importance of open-set uncertainty estimation as a strong signal for guiding NeRF optimization, and the demonstrated advantage of our implicit method compared to explicit representations as used in OpenScene [Peng et al. 2023]. Furthermore, we appreciate your recognition that the core idea is straightforward to integrate into existing NeRF representations and that the paper is presented in a clear manner.

Next, we address the remaining questions:

• The reviewer asks if we first optimize a NeRF model to generate novel views. This is correct: the novel views stem from our optimized NeRF model, which is then further optimized with the new views. However, this strategy does not further complicate the training procedure, nor does it add computational overhead beyond the novel-view rendering. This is because the novel-views (and the corresponding pixel-level CLIP features) can be seamlessly integrated into the same optimization by simply adding them to the existing set of training images. Thank you for bringing this to our attention, we will enhance and clarify this aspect in the paper.

• It is correct that details of this work have been addressed in previous research, for example [Kerr at al. 2023, Fan et al. 2022, Kobayashi et al. 2022, Zhi et al. 2021] etc. Given this, we appreciate your acknowledgment that the devil lies in the details, and that our novelty includes the combination of these distributed efforts into a single method for open-set 3D segmentation. On top of that, we are the first to directly lift pixel-level CLIP features to 3D within the scope of neural radiance fields. Further, as stressed earlier, a major contribution is how to leverage these CLIP features within open set 3D segmentation. To this end, we presented a novel confidence driven inference strategy and established a rigorous evaluation setup for the semantic segmentation task, demonstrating that our simpler yet more intuitive and faster setup leads to superior results (e.g., when compared to the more complex baseline of LeRF which requires multi-scale feature computation and additional DINO regularization terms).

We thank the reviewer for the constructive and insightful feedback. Below, we address the remaining questions:

Experiments on Larger-Scale Datasets
We provide additional results on the ARKitScenes [Baruch et al. NeurIPS’21] dataset in the attached supplementary material. ARKitScenes is a large-scale, real-world dataset providing posed RGB-D video sequences from an iPhone/iPad device along with 3D reconstructions. We focus on long-tail, in the wild objects that are small and typically hard to detect and segment. As explained in LERF, we found ScanNet to be challenging to optimize high-quality NeRFs due to limited views and poor poses. However, more recent datasets, such as ARKitScenes, do not exhibit these problems. In fact, as shown in Fig.7, the posed RGB-D recordings from an iPhone can directly be used to optimize our NeRF representation.

Confidence Estimation and Novel Camera View Selection
The reviewer comments on the uncertainty visualized in Fig.3 and the selected camera poses shown in Fig.2, and raises the question why most of the selected camera poses (yellow) are around the table in the center of the scene, instead of the door or cabinet which exhibit larger uncertainty. We agree with you that the illustrations is a bit misleading; instead of the camera pose, we should visualize the camera frustum to show more clearly what is visible from the selected camera. Indeed, the current illustration might give the impression that the novel camera poses around the table are also looking at the table. However, this is not the case: what the cameras are looking at is (i.e., the camera target) is selected based on the estimated uncertainty. The actual camera position is placed at a distance from the target (to make sure the object and its context is visible) while making sure that the camera itself is not colliding with the scene geometry (see Sec.3.3 for details).

Object Boundaries vs. Entire Objects
The reviewer is curious why entire objects instead of object boundaries are emphasized in the uncertainty visualization (Fig.3). Uncertainty at object boundaries is an effect that is typically observed when visualizing the entropy of semantic-class probability distributions from a trained model. This means that the model is confident about the object label, but less sure about the object mask boundaries. In such a setup, the model acts as a regularization over the training data.

Instead, Fig.3 shows the variance over open-set features computed independently for each view. These features originate from OpenSeg, an approach that provides pixel-aligned CLIP features based on a segmentation of the input image. When computing the uncertainty, there is no regularization across views, such that these features can disagree significantly. This explains why entire objects are highlighted in the heatmap instead of only object boundaries.

We hope that you find our answers convincing, and that you might find additional insights in the discussions with the other reviewers.

We thank the reviewer for the constructive and positive feedback.

In particular, we are pleased that you recognize the performance of our model for open-set 3D semantic segmentation compared to the baselines, and specifically for long-tail classes, including our novel-view rendering based on confidence estimation that brings new insights for improving open-set 3D scene understanding. We are also happy, that you are convinced by the showcased results on arbitrary concept queries, even on in-the-wild-scene recorded with a commodity phone lidar scanner.

Next, we address the remaining questions and concerns:

Runtime and Memory
The reviewer is asking about computational requirements of our method compared to other models. We measure the runtime and memory requirements for our model and compare it to LERF. For a fair comparison, we report all measurements on the same hardware and settings (image resolution, feature backbone, etc.). The table below shows the results:

We look at the number of model parameters and the average runtime to perform a query. Our model relies on approx. 3 times fewer parameters than LERF while being 6-7 times faster and performing 2 times better on segmentation. These significant differences are largely due to our less complex pipeline: our model does not require the additional head for DINO regularization, our CLIP branch is significantly smaller than LERFs which implements a multi-scale feature field. In terms of runtime, for each prompt, LERF needs to compute features at multiple scales to select the best one, and additionally performs negative-prompt to suppress background noise. We also integrated negative prompts into our model, but did not find them to be helpful in terms of performance. We are grateful for your input and will add this table to the updated version of the paper.

Details and Observations during Evaluation Next, the reviewer is curious about details of the evaluation protocol and observations made during the evaluation. Specifically, we follow the exact same evaluation protocol as OpenScene (Sec.4), and compute the 3D semantic segmentation performance on the provided 3D scene point clouds. For NeRF based methods (ours and LERF), we aggregated projected features on a per-point basis to obtain the final semantic class (as described in Sec. 4.1 and 4.2). The interesting observation here is that directly sampling the NeRF representations at 3D point positions does not perform as well as the rendering and aggregating strategy. This can be explained by the fact that NeRFs are optimized in the projected image space. We compared these two alternatives in Sec. 4.3 (“Sampling or rendering?”).

Baseline Experiment Finally, the reviewer mentions the performance drop in one of the baseline experiments, when rendering novel views from random positions (Sec.4.3, “Impact of Novel Views”). In the above review, this aspect is listed as a weakness of our method. However, only to clarify, this is not our method, but merely a baseline experiment. In fact, this analysis shows the benefit of our proposed uncertainty-guided camera pose selection over randomly selecting poses. We will make this clearer in the paper.

We hope that we were able to answer all your questions and are open to further clarifications if needed.

Additional “In-the-wild” Examples
The reviewer would like to see additional results from real-world scenes recorded in-the-wild. Unlike Replica, real-world recordings typically do not have perfect viewpoint annotations which can make training of NeRFs harder. In the attached supplementary we show additional “in-the-wild” real-world results on the ARKitScenes dataset which consists of iPhone RGB-D recordings with imperfect poses in natural environments. Specifically, we tried to pick “long-tail” objects, such as “door handle”, “remote control” etc. that are hard to localize and segment.

Comparison with SemanticNERF’s Uncertainty Formulation
The reviewer asks for a comparison of our confidence estimate (Sec.3.3) and a related uncertainty formulation in SemanticNeRF. Thanks a lot for this question, this aspect is less highlighted in the paper but nevertheless important.
In our work, we compute the generalized variance over high-dimensional OpenSeg/CLIP features projected to a single 3D point. SemanticNeRF computes the entropy of the per-semantic-class probability distribution for each 2D pixel. Both metrics are used to quantify uncertainty, however what they measure is different:

• The entropy in SemanticNeRF measures the confidence of the semantic class prediction. Ideally, there would only be one peak for a single class in the predicted distribution. However, if the semantic prediction is uncertain, the probabilities will be spread out over multiple classes, resulting in larger entropy. SemanticNeRF uses this measure to analyze the effect of noisy and sparse supervision (Fig.5 and Fig.6 in SemanticNeRF).
• Since our OpenNeRF (and also LERF) operates in the open-vocabulary setting, we cannot assume to know all existing semantic classes in advance, i.e., we cannot compute the entropy over class-probabilities. However, we can measure the variance of the open-set features in the high-dimensional space. Ideally, for each point in 3D space the projected open-set features would be equal. The more they disagree, the more they are distributed around their mean. As a measure for uncertainty, we therefore compute the multi-dimensional variance, which is summarized in a single scalar by the generalized variance used in this work. We use this single scalar to select novel camera views.

Unlike SemanticNeRF which computes the entropy only for analyses, in our work, the generalized variance guides the novel-view selection mechanism used to improve the open-set features.

Dependence on NeRF quality
The reviewer wonders if our mechanism based on novel-views still works if the rendering quality of the NeRF deteriorates. As correctly pointed out, our method strongly depends on the quality of the rendered novel views. In case of low-quality novel-views, the extracted open-set features will naturally not be helpful. This is a clear limitation of our method that we will further discuss in the paper. A similar observation is made in LERF, which equally depends on high-quality views to train a NeRF.

We hope that the additional experiments, results and explanations are helpful for your final decision.

We thank the reviewer for the constructive and detailed feedback. In particular, we appreciate the recognition of the improved 3D segmentation results of our method over the baselines LERF and OpenScene on all sets (head, common, tail) as well as the proposed uncertainty to measure to improve open-set 3D scene understanding.

Below, we address the remaining questions and concerns:

Comparison with LERF
The reviewer asked for a detailed comparison to the LERF baseline as shown in the paper. Below we highlight the most important aspects:

1. First, OpenNeRF proposes an overall simpler model (with fewer components that are easier to tune) which results in notably faster and improved scene understanding performance (see Table 1 below). In particular, the key differences in the model are:

• The usage of pixel-aligned open-set features (from OpenSeg) as opposed to per-image features (from CLIP). This results in better localized open-set features as evidenced by our improved segmentation masks.
• A direct consequence is that we do not need complex multi-scale mechanisms for feature representation as in LERF. Therefore our CLIP head is significantly smaller, since it does not need to represent a multi-scale feature field. Importantly, during inference, LERF requires a rendering of each scale, which linearly slows down the querying process by the number of scales. This effect is further emphasized by the negative prompts proposed in LERF to suppress background noise, which did not help seem to bring improvements in our setup.
• Finally, our model does not depend on additional regularization terms; LERF requires an additional head for DINO regularization to keep the features consistent. In our setting, DINO regularization did not make a significant difference due to the pixel-aligned features. Overall, our simplified design avoids the representation of multi-scale CLIP and DINO features, which results in an overall slimmer and faster model.
  | Method | Model Parameters | Avg.Runtime per Frame | mIoU | | --- | --- | --- | --- | | LERF | 234 million | 22.6 | 10.5 | | OpenNeRF | 84 million | 3.4 | 20.4 |

Table1: Our model relies on approx. 3 times fewer parameters than LERF while being 6-7 times faster and performing 2 times better on segmentation.

2. Second, beyond the actual NeRF model as scene representation, a key aspect of the work is the demonstrated use of rendered novel views, based on the proposed uncertainty estimation, as a promising approach to extract refined open-set features. In fact, the reviewer pointed out an additional interesting experiment, to analyze the performance of baselines models using the proposed novel-view rendering technique. We integrated it into the LERF pipeline as shown in Table 2 below. Interestingly, the relative performance boost due to the novel views appears even more significant for LERF.

Table 2: Effect of adding uncertainty-based novel-views to LERF and OpenNeRF (metric is IoU).

Since this mechanism depends on the novel-view rendering capabilities of NeRF, it can be easily integrated into NeRF-based methods such as LERF. However, it is a bit harder for OpenScene which does not support rendering of novel views. Nevertheless, one could imagine an extension of OpenScene where the provided textured meshes are rendered from novel camera poses based on our uncertainty measurement. This would then require the regeneration of the image-based training data as well as training/fine-tuning the 3D sparse convolutional Minkowski networks to obtain the improved 3D open-set features.