Dual-level Adaptive Self-Labeling for Novel Class Discovery in Point Cloud Segmentation

The major concerns of reviewer Zvms are that motivation is not clear and straightforward, and the paper is quite scattered. We respectfully disagree with reviewer Zvms and detail the response for each point in below. We hope our response can address reviewer Zvms's concern.

Motivation is not straightforward.

In the second paragraph of the introduction, we have highlighted the shortcomings of point-wise clustering based on equal-size constraints. To reiterate, these include 1) overlooking the inherently imbalanced data distribution and 2) neglecting the rich spatial context information of objects. Those two weaknesses are closely related to novel class discovery for the point cloud segmentation problem and are both important issues. Therefore, we believe our motivation is clear, and we respectfully disagree that solving a problem from two important perspectives is a weakness.

The point-wise clustering means we segment novel classes in the point cloud relying on each individual point, ignoring structured information, such as superpoint or region[3,4]. The existing point-wise clustering methods [3,4] usually employ equal-size constraints on cluster size to avoid degenerate solutions, where all classes are assigned to a single cluster.

degenerate solution and semi-relaxed optimal transport problem

In unsupervised clustering, the degenerate solution means we learn a single cluster [1,2] for all classes. Specifically in point cloud segmentation, it means we segment all unlabeled points to a single class. To avoid the degenerate solution, they [1, 3] usually introduce a uniform distribution constraint and formulate the pseudo-label generation problem as an optimal transport problem (Eq 3 in manuscript). However, such a uniform and equality constraint contradicts with point cloud segmentation problem, where the class distribution is inherently imbalanced, thus generating a noisy pseudo label (Figure 2 in manuscript). To mitigate this issue, we propose a semi-relaxed optimal transport problem, which relaxes the equality constraint into a KL constraint (Eq 4) and enables us to generate high-quality pseudo-labels (Figure 2 in manuscript). Note that our manuscript has clearly motivated the semi-relaxed optimal transport in the second paragraph, and we have updated the illustration of degenerate solutions for better understanding.

The data-dependent annealing strategy is specifically designed for the self-labeling process and orthogonal with dual-level representation. It adaptively adjusts the regularization strength of KL constraint in semi-relaxed optimal transport problems, enhancing the flexibility of our formulation (see the last sentence of Paragraph 4 in the Introduction).

Figure 1

We appreciate your suggestion and apologize for any misunderstanding. Please refer to the General Response Part 1 for clarification.

I appreciate the answers from the authors. Some of my concerns have been addressed, and I can increase my rate. However, I still think this paper requires a focus on a specific problem and a major revision before publication.

We appreciate your valuable feedback and hope the following response could address your concern.

Augmentation

Thanks to your suggestion regarding our two augmented views, we indeed adopt augmentation to generate two views for learning transformation-invariant representation. Using augmentation to create two views is a well-established technique for learning a transformation-invariant representation, widely employed in novel class discovery literature [2,3], and more recently applied in point clouds[1,4,5]. In our comparison, the previous sota method [1] also adopts the same augmentation as ours to generate two views. Therefore, our comparison ensures a fair assessment.

In our method, we first generate two views of the same point cloud by different augmentation, including naive rotation and scale, which are adopted from previous work [1] and have been illustrated in Implementation Details. Then, we generate point-level and region-level imbalanced pseudo-labels through adaptive self-labeling on two views in parallel. Subsequently, we swap the pseudo-labels generated from the two views to learn transformation invariant representation.

To show the effect of augmentation, we conduct ablation on the two views and augmentation.

From the results above, we draw the following conclusion:

• Compare column 1 and 2, augmentation on one view has improved by 4.4% in novel classes compared with no augmentation.
• Compare column 2 and 4, employing two views has improved by 23.1% in novel classes.
• Compare column 3 and 4, our proposed techniques result in a significant improvement of 16.6%.

[1] Luigi Riz, Cristiano Saltori, Elisa Ricci, and Fabio Poiesi. Novel class discovery for 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9393–9402, 2023.

[2] Enrico Fini, Enver Sangineto, Stephane Lathuiliere, Zhun Zhong, Moin Nabi, and Elisa Ricci. ` A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9284–9292, 2021.

[3] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.

[4] Fuchen Long, Zhaofan Yao, Ting abd Qiu, Lusong Li, and Tao Mei. Pointclustering: Unsupervised point cloud pre-training using transformation invariance in clustering. In CVPR, 2023.

[5] Zihui Zhang, Bo Yang, Bing Wang, and Bo Li. Growsp: Unsupervised semantic segmentation of 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17619–17629, 2023c.

Heavy notation

We appreciate your suggestion. We have simplified most notation in Section 3.1 for better readability but kept the sub/superscripts in Section 3.2 for clarity. The superscripts are usually used to distinguish the point and region, the subscripts are usually the index.

Thank you for your valuable feedback on our submission. We appreciate your detailed feedback and appraisal of our work, which we have taken into careful consideration in our rebuttal response. As the rebuttal process is coming to an end, we would be grateful if you could acknowledge receipt of our responses and let us know if they address your concerns. We remain eager to engage in any further discussions.

We appreciate your valuable feedback and hope the following response could address your concern.

Content lacks a smooth organization .. and figure 1 is incomplete ..

We appreciate your suggestion and have carefully considered your suggestions. However, we believe that our organizational structure is reasonable, as we employ a standard format. We present an overview of our method in Section 3.1, followed by a description of our network architecture, encompassing dual-level representation and a prototype-based classifier. Subsequently, we outline the overall self-labeling framework and provide details on imbalanced pseudo-label generation and the adaptive regularization strategy. To improve clarity, we have reorganized Section 3.1 to ensure coherence. It's notable that other reviewers (cd9p, 74h9) have also praised that "it is well-written and easy to follow". We would appreciate it if you could specify which part lacks a smooth organization.

We appreciate your advice on Figure 1, and please refer to the General Response Part 1 for details.

Analysis of the experimental results is not sufficient

Hpyer-parameters of DBSCAN

Please refer to General Response.

Decrease in the IoU value for "Building"

Explanation of the decrease of "Building"

We appreciate your feedback. As depicted in Table 4, the inclusion of region-level features results in only a relatively small decrease in the "Building" class (53.1% -> 51.5%), while the average mIoU increases by 4.2%. We note that the performance of "Building" is improved on the training data after incorporating region-level learning, as shown in Figure 3. Therefore, we speculate that the marginal decrease in the building class may be due to the model overfitting to the training set.

Distinct from other classes, such as plants, cars, and grounds, "Building" is positioned farther away from the sensor, leading to a sparse spatial distribution and relatively small regions generated from DBSCAN. Therefore, we suspect that this phenomenon may arise from the prototypes capturing the bias of the relatively small region associated with the "Building" in the training data, which is inconsistent with the test data. This issue can be addressed by introducing additional regularization. For instance, as demonstrated in the table in General Response 2, configuring DBSCAN to generate relatively larger regions (e.g., with epsilon=0.7) significantly increases the results for the building class.

Is this method more beneficial for smaller objects?

Our method is more beneficial to dense objects instead of smaller objects. Specifically, we utilize DBSCAN, which is a density-based clustering algorithm, to generate regions for point clouds with relatively high density. Therefore, our method is beneficial for objects relative to density, rather than size.

Thank you so much for taking the time to review our submission. We appreciate your detailed feedback and appraisal of our work, which we have taken into careful consideration in our rebuttal response. As the rebuttal process is coming to an end, we would be grateful if you could acknowledge receipt of our responses and let us know if they address your concerns. We remain eager to engage in any further discussions.

Ablation study on SemanticKITTI

We appreciate your suggestion and conduct additional ablation on split 0 of SemanticKITTI. The results are shown below:

Similar to SemanticPOSS, each component enhances performance. Specifically, adaptive regularization and region-level learning individually contribute to a 6.6% and 5.0% improvement in mIoU for the model.

Minor Point

We appreciate your advice. But the table is too wide to insert another column. To make it clear, we add an explanation of the split index to the caption.

We appreciate Reviewer 74h9 valuable feedback and respond to the concerns in the following.

Unknown the number of novel classes.

We agree that the utility of the novel class discovery setting is a valid point for consideration. In our manuscript, we divide novel class discovery in point clouds into two sub-problems. The first is to estimate the number of novel classes. The second problem is to discover novel classes after determining the number of novel classes, which is challenging due to imbalanced class distribution. Additionally, to ensure a fair comparison with other methods [2], we use the number of novel classes as a prior and focus on the second problem.

And it is more realistic to consider unknown $|C_u|$. To estimate $|C_u|$, we extend the classic estimation method [1] in NCD to point cloud semantic segmentation. Specifically, we set the candidate range of the total number of categories (seen classes number $\textless |C_{all}|\textless$ max classes), and apply Kmeans to cluster the labeled and unlabeled point clouds in the training data, using the representation extracted from a known-class pre-trained model. Then, we evaluate the clustering performance of known classes under different $|C_{all}|$, and select $|C_{all}|$ with the highest clustering performance as the estimated $|C_{all}|$. In practice, for computational simplicity, we randomly sample 800,000 points from all scenes to cluster. We conduct experiments on splits 0 of SemanticPOSS, setting max classes to 50, which is a large class number in point cloud semantic segmentation. Consequently, the estimated $|C_u|$ is 3, which is close to the ground truth value (GT is 4).

In addition, we conduct experiments in SemanticPOSS split 0 to compare the results of our method and previous sota[2] when the estimated value deviates from the true value. As shown in the table below, our method can still significantly outperform NOPS with estimated $|C_u|$.

In conclusion, our sub-problem division is reasonable, and our method shows significant improvement compared to the previous sota, even when the estimated novel class deviates slightly from the real value.

In addition, we find that under the estimated $|C_u|$, our method achieves better results than the ground truth $|C_u|=4$. As shown in Figure 2 in the paper, when $C_u=4$, many plants and buildings are misclassified as the car class (a minor class), resulting in poorer learning for buildings and plants. In contrast, at $C_u=3$, the model ignores the car class, which is relatively small compared to plants and buildings, allowing for better learning of buildings and plants, thereby achieving superior results.

[1] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7492–7501, 2022.

[2] Luigi Riz, Cristiano Saltori, Elisa Ricci, and Fabio Poiesi. Novel class discovery for 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9393–9402, 2023.

Parameters of DBSCAN

Please refer to the General Response Part 1.

Initialization and updating process for class prototypes

Please refer to the General Response Part 2.

Discussing our framework in indoor scenarios

Our framework is general and can be applied to indoor scenarios with little modification. The key difference between indoor scenarios and outdoor scenarios is that Indoor scenarios 1) have more classes; 2) are more denser; 3) include additional RGB information.

As for more novel classes, our results in Table 2 show we can achieve satisfactory results compared to NOPS on harder splits, where half (6 or 7) classes are novel. As for more dense points, we have to adjust the DBSCAN to generate appropriate regions. For the additional RGB information, we can utilize it to enhance the quality of regions and adjust the backbone to process the input RGB information.

In conclusion, our method can be relatively easily extended to indoor scenarios.

The learning of representation and prototypes

We appreciate the advice from the reviewers and further elaborate on the learning of representation and prototypes. In the beginning, the representation and prototypes are randomly initialized, which is very noisy. However, there are three key factors that guarantee us to gradually improve the representation and prototype. The first one is the learning of seen classes, which improves the representation ability of our model, thus improving the representation of novel classes implicitly. In order to prove that known classes can help the representation of novel classes, we cluster the representation of novel classes obtained from the known-class supervised pre-trained model and a randomly initialized model on SemanticPOSS split 0.

The results indicate that features extracted from the known-class pre-trained model exhibit better clustering performance compared to features extracted from a randomly initialized model. The former outperforms the latter by nearly 7% in mIoU for novel classes, demonstrating that known classes can indeed enhance the representation of novel classes.

The second one is the view-invariant training, which learns invariant representation for different transformations and promotes the representation directly. Some studies [3, 4] have advanced unsupervised representation learning for point clouds by incorporating transformation invariance.

The third one is the utilization of spatial prior, which enforces the point in the same region to be coherent, which may be validated by Figures 2 and 3, and unsupervised clustering [1, 2].

Those factors gradually improve the representation and prototypes, leading to an informative prediction $P$. Then, our adaptive self-labeling algorithm utilizes $P$ and several marginal distribution constraints to generate pseudo-label $Q$. Finally, the $Q$ guides the learning of representation and prototype. In conclusion, the above three factors and our self-labeling learning process ensure our method learns meaningful representation and prototype gradually. Furthermore, we visualize the representation of novel classes during training in Appendix G.1, showing that as the training time increases, the learned representation gradually becomes better, validating our analysis.

[1] Fuchen Long, Zhaofan Yao, Ting abd Qiu, Lusong Li, and Tao Mei. Pointclustering: Unsupervised point cloud pre-training using transformation invariance in clustering. In CVPR, 2023.

[2] Zihui Zhang, Bo Yang, Bing Wang, and Bo Li. Growsp: Unsupervised semantic segmentation of 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17619–17629, 2023c.

[3] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pretraining for 3d point cloud understanding. In ECCV, 2020. 3

[4] Zhang Z., Girdhar R., Joulin A., Misra I. Self-supervised pretraining of 3d features on any point-cloud. In: ICCV, 2021