Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation

We thank the reviewer for the encouraging and constructive comments; please find below our response to the mentioned questions.

Q1: Is it possible to extend the proposed method on panoptic segmentation of 3D scenarios? Please present your design briefly for this.

To perform Open Vocabulary panoptic segmentation, our pipeline can be used as is with background object class names in the input class list (walls, floor, ceiling), since YoloWorld can also generate bounding boxes for these categories. A panoptic segmentation network can be used for the proposal generation network, such as OneFormer3D [1].

Q2: As shown in Table 1, the inference time of the proposed method is 21.8, which is slower than OpenScene (3D Distill). Please add the explanation for this phenomenon in the corresponding text (first paragraph of Section 5.1).

In our revised version, we will provide a detailed explanation of this result. This is primarily because OpenScene relies on a U-Net model to predict CLIP features for each 3D point. This U-Net is trained by minimizing the cosine similarity loss between the predicted 3D CLIP features and the CLIP features aggregated from 2D; the 2D features are extracted through projections of the 3D point cloud on the predictions from 2D open-vocabulary semantic segmentation networks and then aggregated into per-3D-point CLIP features.

• OpenScene 3D Distill: This approach is much faster since it relies on per-point CLIP features predicted using the trained 3D U-Net only, which takes less than a second to predict.
• OpenScene 2D: Bypasses the 3D backbone predictions, directly constructing per-3D-point CLIP features from 2D open-vocabulary semantic segmentation. However, this method is significantly slower, requiring inference from many multi-view frames to generate the 3D features.

Q3: In Line 405, it should be 4.29 but not 04.29 for OpenScene (3D Distill).

We thank the reviewer for highlighting the typo; we will correct it and update it in the revised version.

[1] Kolodiazhnyi, Maxim, et al. "Oneformer3d: One transformer for unified point cloud segmentation." CVPR 2024.

We sincerely thank the reviewer for their thoughtful feedback, which will greatly contribute to improving the clarity and quality of our work. Please find our detailed responses below.

Q1: On the Improvements of the Segmentation Performance Observed in Open-YOLO3D

Even though Open-YOLO3D's enhanced segmentation performance can also be attributed to the high performance of the YoloWorld model, we argue that a naive 3D mask classification method cannot achieve the best out of 2D object detectors. To further demonstrate this, we conducted several experiments, with the results summarized in the table below.

In row 0, we match 2D bounding boxes with the highest confidence scores from YoloWorld to 3D masks based on the IoU overlap of the 3D mask projections. The class prediction of the best-matching bounding box was then assigned to the corresponding 3D mask.

In row 1, YoloWorld was used to generate bounding boxes, which were then leveraged to create multi-view crops. These crops were processed to predict clip features, which were subsequently aggregated into a single feature representing a 3D mask.

Both approaches yielded significantly poorer results compared to our proposed MVDist method. This suggests that relying solely on the prior knowledge of pre-trained models is insufficient for effectively utilizing multi-view predictions from object detectors. In contrast, our approach leverages MVPDist, which encodes point frequency across multiview frames and integrates class predictions projected onto LG maps. This method achieves consistently superior performance with minimal computational overhead, relying only on selection operations.

Q2: On the inherent challenges in Open-YOLO3D and their resemblance to those faced by Open3DIS

Our 3D mask classification method can be used with any class-agnostic mask generation pipeline. Table 7 in our paper presents the results of using the Open3DIS approach to generate 3D instance masks by clustering 2D masks obtained from SAM, which were prompted with bounding boxes provided by an object detector. Additionally, we append the 3D masks generated by a 3D proposal network similar to Open3DIS; we use ISBNET similar to Open3DIS instead of Mask3D for a fair comparison with Open3DIS.

While SAM's slower processing adds an extra 4.5 minutes to generate 3D proposals from 2D masks, this method achieves improved performance in terms of mAP compared to Open3DIS, while still being 1 minute faster overall.

Q3: On the effectiveness in identifying sparse-diverse and less common categories in outdoor environments

We conducted an experiment to evaluate our method on the NuScenes dataset using Mask3D as the proposal network, which was trained on the ScanNet200 dataset. The mean average precision (mAP) of all models is reported in the table below, with comparisons made against OpenMask3D and SAM3D. We report the results of OpenMask3D and SAM3D as in the SAM3D paper.

For evaluation, since NuScenes provides 3D bounding boxes, we generate instance ground truth masks by masking the points within each box. For the input point cloud to Mask3D, we use the LiDAR point cloud represented in the global coordinate system. Each point is assigned an RGB color by projecting it onto the camera images using the intrinsic and extrinsic parameters of each camera.

The results highlight challenges in generalizing to outdoor environments from models pre-trained on indoor datasets like ScanNet200, primarily due to LiDAR data sparsity and limited multi-view frames. Both Mask3D and OpenMask3D struggle to adapt to outdoor scenes due to low-quality 3D proposals from Mask3D, pre-trained on ScanNet200. However, they outperform SAM3D in indoor scenes, achieving superior results on ScanNet200.

On NuScenes, the method performs comparably to OpenMask3D, with performance constrained by the dataset’s limited multi-view frames, as most instances are associated with only a single 2D frame.

We thank the reviewer for their constructive comments, which will significantly enhance the clarity of our work. Kindly find below our detailed responses.

Q1: Complexity Analysis of VAcc

The visibility computation method in OpenMask3D projects 3D points onto frames to identify visible points, using depth maps to filter out occluded points and ensure points fall within the frame's dimensions. Visibility is determined by checking if a point's projected depth matches the depth map value. This process iterates through N 3D masks, M frames, and P points, resulting in both computational and time complexities of O(N × M × P). However, it uses sequential loops, limiting parallelization efficiency.

In our proposed method, VAcc, we reformulate the visibility computation using tensor operations, enabling parallelization across multiple cores. While maintaining the same computational complexity O(N × M × P), it reduces time complexity to O(N × M × P / c), where c is the number of cores. This makes VAcc a faster alternative to the method used in OpenMask3D.

Q2: Effect of Occlusion and Light Conditions

Occlusion Analysis

This analysis evaluates model robustness to occlusion using the testing protocol from [1]. Images are systematically occluded by dropping patches (1% of image size) centered on pixels from 2D projections of 3D ground truth instance masks. The experiments include:

1. Salient Patch Drop

• Simulates occlusion of semantically important foreground objects by removing patches from 2D masks derived from instance masks, excluding walls, floors, and ceilings.
• Larger masks have more patches removed to uniformly test the impact of occlusion on detector performance.

2. Non-Salient Patch Drop

• Focuses on background occlusion by removing patches from areas corresponding to walls, floors, and ceilings.
• Tests the detector's robustness to non-critical occlusion.

These experiments assess how meaningful and arbitrary occlusions affect object detection under extreme conditions.

Table: Occlusion analysis

We show that our method is robust to moderate levels of salient occlusion where the important objects are occluded while being extremely robust when the non-salient objects are occluded.

Analysis of Extreme Lighting Conditions:

We test the model across different levels of lighting conditions, starting with 0.05 intensity (almost black) up to 2.5 intensity (extremely bright). A brightness level of 1 corresponds to the original lighting conditions. The robustness results demonstrate that the pipeline performs well under moderate fluctuations in light intensity but encounters challenges in extremely low-light conditions.

Table: Analysis under different Lighting Conditions

Q3: How MVPDist Mitigates Misclassifications from Object Detectors

We observe that object detectors sometimes predict bounding boxes for incorrect classes with high confidence. Consequently, a naive 3D mask labeling approach that associates a 3D mask with the bounding box of highest confidence across views can lead to incorrect predictions. To further demonstrate this, we conduct the experiment in row in the table below:

Our MVPDist method significantly outperforms confidence- and IoU-based 3D-to-2D instance matching by encoding 3D instance information into the distribution rather than relying solely on confidence scores or IoU from YoloWorld.

Q4: How Does Integrating 2D Segmentation Models Affect Speed and Performance?

Table 7 in our paper presents the results of using the Open3DIS approach to generate 3D instance masks by clustering 2D masks obtained from SAM, which were prompted with bounding boxes provided by an object detector. While SAM's slower processing adds an extra 4.8 minutes to generate 3D proposals from 2D masks, this method achieves improved performance in terms of mAP compared to Open3DIS, while still being ~1 minute faster overall.

[1]Naseer, et al. "Intriguing properties of vision transformers." NeurIPS 2021

We thank the reviewer for the constructive feedback. A well-documented code along with pre-trained models will be publicly released. Our detailed responses are provided below.

W1: On the Contributions

Efficiently labeling 3D masks using predictions from object detectors is a challenging task. In our work, we propose LG+MVPDist, a method that enables accurate class predictions from multiview images, effectively addressing the limitations of 2D object detectors. Unlike CLIP features, which can be aggregated into a single strong feature that encodes multiview information about the object, object detectors generate a set of bounding boxes for all input classes, making it difficult to efficiently match the best 2D box to its corresponding 3D mask.

In the experiment (row 0) detailed in Table I below, the class label for the 3D mask is determined by selecting the YOLOWorld-predicted bounding box with the highest IoU overlap with the 2D bounding box constructed by projecting the 3D mask onto the corresponding view. The class prediction is then taken from the bounding box with the highest confidence score across the views with top-k visibility.

In row 1, the matched bounding box is further used to generate crops for constructing visual CLIP features for the 3D masks, following a similar approach to OpenMask3D. These visual features are then used to predict the class label.

Our results highlight that LG Maps and MVPDist significantly outperform both of these techniques. This improvement is primarily due to the limitations of object detectors, which can sometimes assign incorrect class labels with high confidence.

MVPDist mitigates this issue by filtering out incorrect labels from certain views. It achieves this by constructing class distributions from multiple views and assigning the mask the most frequent class. Since MVPDist encodes the frequency of points projected onto different views, the distribution leans toward views with higher point densities. This ensures that the classification process prioritizes frames where the object is most clearly represented, reducing the impact of less informative views and improving overall accuracy.

We hope our simple and effective approach will serve as a solid baseline and help ease future research in fast and accurate open-vocabulary 3D instance-level recognition.

Table I: Contributions Analysis

W2: On Additional Ablation

We appreciate the reviewer's feedback and will restructure the ablation table for better clarity in the revised version. Below, we present our updated ablation study, introducing an experiment in row 0 where we don’t use our proposed MVPDist+LG maps. In this setup, we assign the 3D mask a class label based on the YoloWorld prediction with the highest IoU overlap and confidence across the views. This experiment in row 0 relies solely on YoloWorld predictions.

Our results demonstrate that incorporating MVPDist+LG maps significantly enhances performance by leveraging point frequency across views to predict class labels, rather than depending entirely on the 2D object detector, which can occasionally assign incorrect class labels. Our observations show that YoloWorld occasionally predicts the wrong class with the highest confidence across views for the same 3D instance.

Updated Ablation Table

W3: Title

We note that the motivation behind the title is to highlight our contributions towards effectively adapting the popular YOLO-based architecture for real-time open-vocabulary 3D instance segmentation. While YOLO-based design has been recently explored for 2D open-vocabulary detection literature, we are the first to investigate and adapt it for 3D open-vocabulary instance segmentation.

We sincerely appreciate the reviewer’s thoughtful and valuable feedback. A well-documented code along with pre-trained models will be publicly released. Our detailed responses are provided below.

W1: Regarding the effectiveness and generalizability of baseline Mask3D for open vocabulary instance proposals

We conducted experiments to evaluate Mask3D’s generalizability to unseen geometries and assess its ability in terms of mask proposal generation. The results are presented in Table 2 and Table 6 (main manuscript). Furthermore, we report in Table below the results of Table 2 on Novel/Base split of replica classes; the split will be publicly released. The Base classes (39 classes) consist of those that are semantically similar to at least one class from the ScanNet200 dataset, while the Novel classes (9 classes) include all remaining ones. Furthermore, we highlight that our method can be used with any class-agnostic proposal generation method. We report in Table 7 of our manuscript that our approach (MVPDist + LG maps), which utilizes 3D proposals clustered from 2D masks following the Open3DIS methodology, achieves superior results compared to Open3DIS (point-wise CLIP features).

• Table 2: Mask3D was trained on ScanNet200 and tested on Replica. This demonstrates Mask3D’s ability to generate proposals for out-of-distribution (OOD) datasets where objects exhibit distinct geometries and characteristics. Furthermore, we show in the table below that our method performs much better compared to Open3DIS on the Replica Base/Novel split.

• Table 6: Mask3D was trained on ScanNet’s 20 classes and tested on ScanNet200, which includes 200 classes. Out of these, 53 classes are considered base classes (which are semantically similar to the 20 classes in ScanNet), while the remaining 147 classes are labeled as novel. This indicates that Mask3D encountered objects with entirely different geometries during testing, showcasing its robustness in handling novel objects in indoor environments. We adopted the split proposed by OpenMask3D authors.

Table: results on Base/Novel replica splits with Mask3D as proposal network

W2: On the experimental evaluation of the proposed method for open-vocabulary 3D instance segmentation

We use the same evaluation setting that OpenMas3D and Open3DIS adopt. For proposal generation, OpenMask3D uses Mask3D trained on the ScanNet200 training set, whereas Open3DIS uses ISBNET, also trained on the ScanNet200 training set.

1. In-Distribution Test (Table 1) : Evaluation on ScanNet200 validation set with a proposal network trained on ScanNet200 training set. We compare against Open3DIS and OpenMask3D under the same setting.

2. Indoor Out-of-Distribution Test (Table 2): With proposal network trained on ScanNet200 and tested on the Replica dataset to assess generalizability to unseen indoor distributions.

3. Generalizability to Novel Geometries (Table 6): Trained on ScanNet with 20 classes and tested on ScanNet200, to evaluate adaptation to new geometries and categories.

Q1: In Table 1, does the class-agnostic Mask3D model have access to mask annotations for the same classes as those in the validation set? Do the other methods use the same class-agnostic segmentation model?

Yes, in this evaluation setting, the classes used for training are similar to those used for validation. OpenMask3D utilizes Mask3D, trained on the ScanNet200 training set (the same pre-trained network in ours), while Open3DIS uses ISBNET, also trained on the ScanNet200 training set. According to the Open3DIS paper (page 8, Table 9), both ISBNET and Mask3D deliver comparable results.

Q2: Since the proposed approach relies on a class-agnostic 3D instance generation model, what are the advantages of using only mask annotations, rather than both instance and label annotations, for training? Are there practical scenarios where only mask annotations are available?

In all previous methods (OpenMask3D, Open3DIS), and ours, both class and mask annotations are used during training. However, during inference, class predictions are disregarded and replaced with an open-vocabulary classification approach.

Q3: What does the tag "(Closed Vocab)" mean in Table 1? Does it indicate that the Mask3D method uses both mask annotations and object class annotations for training?

A closed-vocabulary tag indicates that the model was fully supervised on classes similar to those present in the validation set.

Q4: What is the performance of Mask3D (Closed Vocab.) on the Replica dataset?

Since the Replica dataset does not include a training set, it was not feasible to evaluate Mask3D in a fully supervised setting on this dataset.

Q5: Typo on line 092

We thank the reviewer and will fix the typo in the final version.