SA3DIP: Segment Any 3D Instance with Potential 3D Priors

We thank the reviewer for the constructive and detailed feedback.

W: Innovation of the proposed approach
In Fig. 1 of our paper, we demonstrate the existing problem of previous methods, which use under-segmented 3D primitives and inherit the over-segmented 2D masks to the final 3D instance segmentation. We believe our method is problem-oriented, trying to alleviate these two major defects.
We therefore design two modules accordingly:

1. Complementary primitives generation module to generate more accurate and finer-grained 3D primitives to avoid any error accumulation.
2. Introducing the 3D space prior for providing instance-aware constraint, which was implemented by a 3D detector.

The combination of both modules works great at solving the aforementioned problems.

W: Overall performance gain analysis
It is true that from ablation study the performance gain seems to heavily depend on 3D detector. However, we believe that the performance gain of the complementary primitives module is not as minor as it looks. We think it is related to the definition of the metric AP (ratio of correctly identified instances to the total number of identified instance), which is more in favor of under-segmentation rather than over-segmentation, since the former introduces relatively fewer false instance. We randomly choose two scenes (scene0011_00 & scene0644_00) and conduct random 10%, 20%, 30% over/under segmentation tests based on their GT instances. Results are averaged from three experiments and shown in the table. APs with subscript O stand for over-segmentation results, and U for under-segmentation.

It can be observed that under whatever percentage the under-segmentation seems to give higher APs, due to its high precision and fewer false positive cases. While for over-segmentation, it gives fewer precision and higher recall. This is consistent with our results, which give finer-grained (20% more in average, shown in the table below) primitives but slightly lower APs. We show the number of instances in comparison with SAI3D in the table.

Upon closer look in the ablation study in our paper, adding only our complementary 3D superpoint primitives module even slight deters the performance, due to the AP metric. However, our final results after experiencing whole pipeline (after adding 3D space prior) reverse the minor drop and produce an extra gain at around 1% compared to adding only 3D detection. The counts for instances decrease as well. The superior performance and decreased instance number indicate that our results achieve both high recall and high precision. It proves that he combining of finer-grained (and slightly over-segmented) primitives and instance-aware refinement (merge those over-segmented primitives) provides a thorough solution.

Q1: Histogram vectors clarification
Histogram vector represents a distribution of 2D mask ids which are covered by the projection of a 3D primitive. It reflects which 2D mask has the correspondence with the current 3D primitive. The histogram vectors are then used to calculate the affinity (similarity) matrix for each pairs of 3D primitives, indicating the likelihood that two primitives belong to the same object (the same 2D mask).
In the implement of codes, assume that for a scene there are M 2D images and N 3D points. For the m-th image, we project the N points to the image using the corresponding pose and camera intrinsic. We record the 2D mask label for every 3D point according to which pixel it is projected onto (0 for invisible points in the image). After this we get a (N, M) matrix, indicating the 2D mask label for every point in every 2D view.
Next, we load all K 3D primitives in the scene. Again for the m-th image, we record the 2D mask labels each primitive covers according to the (N, M) matrix we get earlier, since multiple labels may be covered by one single primitive due to the ambiguity or inaccuracy in 2D masks. We maintain a normalized matrix of size (K, V) for each view, where V refers to the 2D mask counts in the view. This matrix of size (K, V) is actually the histogram vector for all primitives in the m-th view.

In the end, we use the (K, V) matrix for cosine similarity calculation to obtain the affinity matrix of size (K, K) which represents the likelihood of each pair of primitives.

Q2: Visual comparison of SAMPro3D
We showcase the comparison in Fig. 3 of global rebuttal PDF. Our method clearly alleviates the over-segmented instances appear in SAMPro3D, which is consistent with our initial purpose of introducing the 3D instance-aware space prior.

We thank the reviewer for the constructive and detailed feedback.

W1: Novelty of the proposed approach
It is true that other methods, including ours, rely on SAM, and this heavy reliance of 2D segmentation results is exactly what we were trying to avoid by introducing the 3D prior. In Fig. 1 of our paper, we demonstrate the existing problem of the previous methods (like SAI3D and SAMPro3D), which inherit the over-segmented 2D masks to the final 3D instance segmentation.
Thus, our method is problem-oriented, trying to alleviate two major defects existing in the pipelines of the current works through 3D priors.

1. The under-segmented 3D primitives and subsequent error accumulation
2. The part-level over-segmented tendency of the 2D foundation segmentator.

We therefore design two modules accordingly. First, we use the complementary primitives generation module to generate more accurate and finer-grained 3D primitives to avoid any error accumulation. Second, we introduce the 3D space prior for providing instance-aware constraint, which we choose to implement using a 3D detection. The combination of both modules works great at solving the aforementioned problems.

W2/Limitation: Documentation or analysis of the new annotations for ScanNetV2
We need to claim that all methods of this type (such as SAMPro3D, SAI3D, ours and so on) are zero-shot ones, which means they utilize the generalization capability of 2D foundation model and do not depend on training. Thus, the ScanNetV2-INS dataset we propose is simply of a evaluation use, since their output instance segmentation results do not change when switching the ground truth file. We will elaborate the details about ScanNetV2-INS dataset as follows:

1. Meaning: There are two major deficiencies in the original ScanNetV2 dataset: missing instances and incomplete instance masks. To solve these, we assigned new class-agnostic labels to the missing ones and re-labelled the incomplete instance masks.

2. Tasks: Following the recent methods of utilizing 2D foundation model to perform zero-shot 3D class-agnostic instance segmentation, our ScanNetV2-INS dataset also focus on this specified task and for evaluation only.

3. Format: We generate a txt file (e.g., sceneid.txt) for each scene in the original ScanNetV2 validation set. The txt file contains N separate lines corresponding to N points in the scene, where the number of each line is calculated as: Number = instance id (refers to n-th object in the scene) + 1000 * semantic label id (refers to scannetv2-labels.combined.tsv). The newly labelled instances were assigned a semantic label id of 41 since we focus on class-agnostic segmentation.

4. Limitation: Our dataset focus on the evaluation use of 3D class-agnostic instance segmentation, due to the recent trend of utilizing 2D foundation segmentator to perform zero-shot 3D class-agnostic instance segmentation and the high expense of annotation on 3D data.

5. More statistics: In Fig. 3 of our paper, we demonstrate how many scenes with more than (10, 20, ..., 100) instances in ScanNetV2 and ScanNetV2-INS. Our proposed dataset clearly provides larger number of scenes with more instances. We will give more statistics below. The first table below illustrates the instance count of the original ScanNetV2 and ScanNetV2-INS. It can be seen that our ScanNetV2-INS dataset incorporates more instances. In the second table, we show the number of instances with varying point counts within specified ranges for two datasets, ScanNetV2 and ScanNetV2-INS. ScanNetV2-INS dataset features more smaller objects, which requires the model to have finer-grained instance perception capabilities. The statics from these two tables could, to some extent, explain why the ScanNetV2-INS dataset is more challenging than ScanNetV2.

   Instance Count	Min	Max	Avg	Total
   ScanNetV2	2	47	14	4364
   ScanNetV2-INS	2	54	17	5596

   Point # Per Instance	<500	500-1000	1000-2000	2000-5000	5000-10000	>10000
   ScanNetV2	252	452	1119	1690	567	284
   ScanNetV2-INS	692	748	1366	1873	626	291

Q1: Requirement of the posed images
Yes, posed images are also required by other competitors. As we clarified in W2/Limitation above, this type of methods (such as SAMPro3D, SAI3D, ours and so on) are zero-shot ones, which means they utilize the generalization capability of 2D foundation model, since to date there is no 3D foundation model due to the limited 3D labelled data. The correspondence between 2D and 3D space is the focus of these methods. Thus, posed images are essential for acting as a bridge between 2D foundation models and 3D space. We will include this discussion in the final version of our paper.

Q2: Further figure clarification
In general, the whole pipeline shown in Fig. 2 of our paper can be separated in three parts. Step A: 3D primitives generation exploiting both geometric and textural priors. Step B: scene graph construction where the primitives serve as nodes and affinity matrix of 3D primitives guided by 2D masks generated using 2D segmentators as edge weights. Step C: Region growing and instance-aware refinement on the constructed scene graph. We provide a more straightforward version with larger images in Fig. 1 of global rebuttal PDF document.

We thank the reviewer for the constructive and detailed feedback.

W1/Q: RGB values as texture prior are not robust enough / Motivations for using color values / More experiments
In the first place, our motivation is that we found distinct instances with similar normal often exhibit different color. Meanwhile, the commonly used 3D indoor datasets have only xyz and rgb attributes for their 3D points. Thus, we opt to explore RGB values as texture prior.
It is true, as concerned also by other reviewers, that texture prior such as RGB values are not robust enough when being used solely. We have conducted further experiments about the priors and their weights, shown in Tab. 1 of the global rebuttal PDF document. In our approach we assign less weight to the textural prior, thus to exploit it while minimizing its negative impact. For ScanNetV2, we set the geometric weights ($W_n$) as 0.96 and texture weights ($W_c$) as 0.04 (Line 214). Under this setting complementary primitives generation module yields a good initial state for the subsequent merging and refinement. We have conducted more experiments on Matterport3D (table below) and Replica dataset (Tab. 2 of the global rebuttal PDF document).

It can be seen in the last row of three tables that, our performances on all three datasets exceed that using only geometry prior. We will include this discussion in the final version of our paper.

W2: Non-significant gain of complementary 3D superpoint primitives module
In our approach, we designed the complementary 3D superpoint primitives module for alleviating the existing problem about under-segmented 3D primitives based only on geometry and subsequent error accumulation.
For the non-significant gain of this module, it is related to the definition of the metric AP (ratio of correctly identified instances to the total number of identified instance). This AP metric is more in favor of under-segmentation rather than over-segmentation, since the former introduces relatively fewer false instance. To further illustrate this, we randomly choose two scenes (scene0011_00 & scene0644_00) and conduct random 10%, 20%, 30% over/under segmentation tests based on their GT instances. Results are averaged from three experiments and shown below. APs with subscript O stand for over-segmentation results, and U for under-segmentation.

It can be observed that under whatever percentage the under-segmentation seems to give higher APs, due to its high precision and fewer false positive cases. While for over-segmentation, it gives fewer precision and higher recall. This is consistent with our results, which give finer-grained (20% more in average, shown in the table below) primitives when exploiting both geometry and texture but slightly lower APs. We show the number of instances in comparison with SAI3D in the table.

However, our final results when experiencing whole pipeline (after adding 3D space prior) reverse the minor drop and produce an extra gain at around 1% compared to adding only 3D detection. The counts for instances decrease as well. The superior performance and decreased instance number indicate that our results achieve both high recall and high precision. It proves that he combining of finer-grained (and slightly over-segmented) primitives and instance-aware refinement during merging process (merge those over-segmented primitives) provides a thorough solution.

W3: Semantic instance segmentation
We have conducted the semantic instance segmentation on ScanNet200 dataset, better aligning our capability with previous benchmark method (such as SAI3D). The results are provided in the following table. Our method clearly results in better performance than other methods on AP, head and tail.

We thank the reviewer for the constructive and detailed feedback.

W1: Ablation studies on each feature individually
We have conducted ablation studies on each feature we intend to exploit (e.g., geometry, texture and 3D space prior). The results are provided in the following tables. APs with subscript 1 stand for ScanNetV2 dataset, and 2 for ScanNetV2-INS. We assigned several weights for geometry and texture to test their contribution. Specifically, we conduct the config with $W_n$=0.4 and $W_c$=0.6 which yields the similar number of 3D primitives as the primitives used in SAM3D, SAI3D and others, for a fair comparison. The experiments show that the config with $W_n$=0.96 and $W_c$=0.04 suits best for our approach. More experiment on Matterport3D is shown below as well, and experiments on Replica is in Tab.2 of the global rebuttal PDF.

It can be observed that texture prior are not robust enough when being used solely due to influence on shadows, reflection and so on. Thus, we adopt the complementary primitives module, exploiting both texture and geometry. A greater weight for geometry and less for texture yield a good initial state for the subsequent merging and refinement. It can be seen in the last row that with 3D space prior, the performance exceeds that using only geometry prior.

W2: Super Primitives Definition
In the context of our paper, 3D superpoints/primitives refer to clusters of 3D points. The points within the same group exhibit homogeneity on certain attributes. The purpose of generating primitives rather than using raw points is to introduce prior knowledge (such as geometry and texture attributes we use) and reduce computational complexity to the subsequent process.

W3: Progressive Region Refinement Examples
In Fig. 2.C of the paper we have provided one visual example that proves the effectiveness of our refinement process. We showcase the detailed visualization at each stages in the region growing and refinement process in Fig. 1 bottom of the global rebuttal PDF document.

W4: 2D to 3D Space Integration
Due to inaccuracies in camera poses, the point clouds generated from posed RGB-D images are not perfectly aligned in 3D space. This misalignment can introduce noise when lifting 2D results into 3D space, as seen in methods like SAM3D. In contrast, projecting 3D mesh-sampled point clouds into 2D space experiences less influence from such inaccuracies. We showcase a example of SAM3D in Fig. 2 of global rebuttal PDF document., which uses only 2D to 3D projection. It is clear that the 2D to 3D projection introduces a lot of noise, and this explains why recent methods, including ours, tend to explore 3D to 2D projection.

Q: Main assumptions & reasons for key design choices
In general, we have made two major assumptions in our pipeline:

1. At the Complementary primitives generation stage, it is assumed that points belong to the same semantic or instance labels share similar inherent attributes (geometry, texture and so on).
2. At the instance-aware refinement stage, it is assumed that the points inside the 3D bounding boxes have high confidence to belong to the same instance.

The reasons for the key design choices are elaborated as follows:

1. Attributes and weights for primitive generation: In the first place it is based on the observation that distinct instances with similar normal often exhibit different color. We believe it is a promising solution to the under-segmented primitives. While afterward we chose to assign greater weight to the geometric prior and less weight to the texture prior, as explained also in W1, due to the variations introduced by lighting conditions, shadows and so on.
2. SAM variant for 2D segmentation: We use instance-level segmentation function of Semantic-SAM for ScanNetV2 dataset since not so many scenes contain a large amount of small objects, and we opt to SAM-HQ for ScanNet++ dataset due to its high-resolution scenes along with detailed objects.