NGTTA: Non-parametric Geometry-driven Test Time Adaption for 3D Point Cloud Segmentation

Q1: One drawback of the paper is the lack of clarity in writing. The paper could significantly benefit from polishing the text. A non-exhaustive list of issues is given in the questions.

R1: Thank you for your suggestions. We have corrected these errors and carefully review the writing in the revision of our paper. The following are responses to some writing inquiries:

1.1 "non-parametric geometric" refers to non-parametric geometric features, specifically the point-wise features extracted using PointNN in our method.

1.2 “data of geometric.” This is a writing error, and we will make corrections in the next version.

1.3 "source data in L263." This statement is correct; it means that the previous method requires obtaining source domain data and target domain data in order to customize the "intermediate domain".

Q2: The prior work PointNN by Zhang et al. (2023) appears to play the central part of the proposed method, since it is the non-parametric geometric model; yet, it needs to be recapped sufficiently in the main part of the paper or in the appendix.

R2: Thank you for your suggestions. In the uploaded revised version of our paper, we have included the introduction of PointNN in Appendix A.7.

Q3: Which 12 combinations of $\alpha$ are presented in the tables?

R3: Sorry for the confusion. In our experiments, $\alpha_1$=10, $\alpha_2$=0.5, and $\alpha_3$=1 yield the best performance. Therefore, when conducting the ablation study for $\alpha_1$, we fixed $\alpha_2$ at 0.5 and $\alpha_3$ at 1. The same approach applies for the ablation studies of $\alpha_2$ and $\alpha_3$ as well.

Q4: The hyperparameter selection is biased to the target domain...

R4: Thank you for your question. In our setup, apart from $\sigma$ and $\gamma$, the other hyperparameters are universally applicable across all datasets, as their adjustments need to be selected based on final performance. To ensure the applicability of our method, we conducted ablation experiments on 3DFRONT$\rightarrow$S3DIS to select the other hyperparameters, keeping them constant across all datasets.

As for $\sigma$ and $\gamma$, they are primarily chosen based on the average entropy of the datasets, since the entropy distributions vary across different datasets. For instance, the average entropy for ScanNet is 0.6, while for S3DIS it is 0.3. We believe this approach is acceptable because the average entropy can be obtained with a single forward pass of the model on the target domain without requiring additional training.

Certainly, in future work, we will explore more universal hyperparameter settings to enhance the generalizability of our method.

Q5: Does the $j$-th sample in (5) mean a $j$-th point in the $i$-th point cloud (scene)?

R5: Yes, in Eq (5), $E_{ij}$ represents the entropy of the $j$-th point in the $i$-th scene.

Q6: How is in (11) chosen in practice?

R6: Thank you for your question. The $\beta$ is set to 0.9 to emphasize non-parametric geometric features more in the early stages of training, while focusing more on well-adapted model features in the later stages of training.

Q7: Did the authors perform the comparison only for a single initialization? I.e., only one test run?

R7: In our method comparison, the TTA series of methods, including NGTTA, are all trained on pre-trained source models, ensuring that the initialization weights are consistent. Additionally, we do not employ data augmentation techniques, which results in minimal performance variability, making the comparison fair.

Q8: Does the non-parametric model have a decoder as visualized in Figure 2

R8: Yes, PointNN extracts point-wise non-parametric geometric features through multi-layer downsampling (encoder) and multi-layer upsampling (decoder).

Q9: The arrow in Figure 2 should be fixed.

R9: Thank you for your careful review. We have made modifications to Figure (2) in the revised version.

Q10: The paper would benefit from having additional qualitative results, especially for the 3DFRONT→ScanNet benchmark, for which they are currently missing.

R10: Thank you for your suggestions. In the uploaded revised version of our paper, we have included the visualization comparison of NGTTA and Source Model on 3DFRONT$\rightarrow$S3DIS and 3DFRONT$\rightarrow$ScanNet in Appendix A.6.1 and the visualization of TSNE during the adaptation process in Appendix A.6.2. Through these visualizations, we can gain a clear understanding of the NGTTA adaptation process and the superiority of the results.

Dear Reviewer 86r6:

We sincerely thank you for taking the time to provide detailed comments. We have made corresponding revisions based on your questions and added a substantial amount of experiments.

Given that the end of the discussion period is approaching, we would like to ask if there are any unclear aspects regarding our work. We are more than happy to provide further explanations as needed.

Our work primarily explores Test Time Adaptation in point cloud scenes, conducted under the challenging SIm2Real setting, which is still an underexplored area. Our approach involves innovative attempts and achieves state-of-the-art performance. We hope that this work will be recognized and valued.

We would like to express our gratitude once again for your feedback and suggestions, which have greatly helped us improve our work.

Best and sincere wishes,

The authors

Q1: It seems that the explanation regarding the motivation for using the non-parametric geometric model for domain generalizability is insufficient. Is there a way to demonstrate that the non-parametric geometric model is generalizable?

R1: hank you for your suggestion. The inspiration for this idea mainly stems from the results of several papers. Firstly, regarding PointNN [1], the results in its original paper demonstrate that PointNN approaches the performance of fully supervised models like PointNet and PointNet++ even without parameters. Additionally, PointNN outperforms methods such as PointCLIP, PointCLIPV2, and ULIP in the absence of any supervisory signals, indicating that PointNN is an effective feature extractor for unlabeled new data. Furthermore, explicit geometric information has been shown to possess strong generalization capabilities, as evidenced by studies like X-3D [2] and RepSurf [4], which validate this conclusion. Moreover, PointDGMamba [3] has demonstrated in experiments that incorporating explicit geometric information like X-3D can significantly enhance domain generalization capabilities. Therefore, these results suggest that non-parametric geometric models should be effective in domain generalization tasks.

[1] Zhang R, Wang L, Guo Z, et al. Parameter is not all you need: Starting from non-parametric networks for 3d point cloud analysis[J]. arXiv preprint arXiv:2303.08134, 2023.

[2] Sun S, Rao Y, Lu J, et al. X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5074-5083.

[3] Yang H, Zhou Q, Sun H, et al. PointDGMamba: Domain Generalization of Point Cloud Classification via Generalized State Space Model[J]. arXiv preprint arXiv:2408.13574, 2024.

[4] Ran H, Liu J, Wang C. Surface representation for point clouds[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 18942-18952.

Q2: If I understand correctly, the non-parametric geometric model ultimately allows for good clustering without the need for learning, but I’m not sure if applying it at the feature level is the best approach. If the non-parametric model is truly domain generalizable, wouldn’t it be a better approach to use it simply as a refinement method in the target domain as a form of post-processing, or to utilize it when generating pseudo-GT in a pseudo-labeling approach?

R2: Yes, your understanding is correct. We do apply the non-parametric geometric features to the generation of pseudo-labels, and you can refer to Eq. (10) in the paper for details. However, we believe that due to the challenging task of scene segmentation, the pseudo-labels of many difficult samples are inaccurate, and even label correction by non-parametric geometric features is difficult to solve. Therefore, it is not enough to only use non-parametric geometric features to generate pseudo-labels. To this end, we additionally introduce the distillation training of geometric features for difficult samples to improve the performance of scene segmentation tasks.

Q3: The paper lacks sufficient qualitative results, which makes it difficult to fully understand or visualize the practical effectiveness of the proposed method.

R3: Thank you for your suggestions. In the uploaded revised version of our paper, we have included the visualization comparison of NGTTA and Source Model on 3DFRONT$\rightarrow$S3DIS and 3DFRONT$\rightarrow$ScanNet in Appendix A.6.1 and the visualization of TSNE during the adaptation process in Appendix A.6.2. Through these visualizations, we can gain a clear understanding of the NGTTA adaptation process and the superiority of the results.

Q4: There are writing errors, such as the sudden introduction of "S2T" in the caption of Figure 1 without prior explanation.

R4: Sorry for the confusion. This was our oversight. "S2T" refers to the feature distribution of the source model in the target domain, and we have updated Figure (1) in the revised version of the paper.

Dear Reviewer Reviewer dGga:

We sincerely thank you for taking the time to provide detailed comments. We have made corresponding revisions based on your questions and added a substantial amount of experiments.

Given that the end of the discussion period is approaching, we would like to ask if there are any unclear aspects regarding our work. We are more than happy to provide further explanations as needed.

Our work primarily explores Test Time Adaptation in point cloud scenes, conducted under the challenging SIm2Real setting, which is still an underexplored area. Our approach involves innovative attempts and achieves state-of-the-art performance. We hope that this work will be recognized and valued.

We would like to express our gratitude once again for your feedback and suggestions, which have greatly helped us improve our work.

Best and sincere wishes,

The authors

I have some doubts regarding the motivation for using a non-parameterized model and whether the proposed method for its usage is optimal. Additionally, I find it difficult to fully accept the academic contribution claimed in the paper.

Q1: The statement "We can see that as the sample count decreases, the threshold will gradually increase" is reasonable. However, it’s stated that tail categories with fewer samples will have a larger entropy threshold and "We can see that as the sample count decreases, the threshold will gradually increase", but the reasoning behind this isn't obvious from the equation itself.

R1: Sorry for the confusion. First, the statement "We can see that as the sample count decreases, the threshold will gradually increase" mainly refers to Eq. (4), which indicates that in our sampling mechanism, the threshold is inversely proportional to the number of classes. This demonstrates that our sampling mechanism allows for a higher threshold for minority classes to sample more instances.

As for the assertion that minority classes typically have higher entropy, this conclusion is drawn from observations of experimental results rather than derived from the formula. As shown in Table R1, we report the proportions and average entropy for each class in ScanNet and S3DIS. We can see that the entropy of minority classes is typically much higher than that of majority classes, which supports the conclusion above. In the uploaded revised version of our paper, we have included this additional result in Appendix A.4.1

Table R1: Category-Wise Entropy Result on ScanNet and S3DIS

(a) ScanNet

(b) S3DIS

Q2: Whether Eq.~(4) is applied individually to each point cloud or collectively across all point clouds. Does each point cloud have its own unique $\sigma_c$ , or do all point clouds share the same $\sigma_c$ ?

R2: Sorry for the confusion. $\sigma_c$ is shared among all point clouds in the dataset.

Q3: For Eq.~(13), the notation could be confusing for readers. Since $\mathbf{f}i \mathbf{f}{ij}$ also appears to be a vector, the mathematical symbols could be refined to avoid potential misinterpretation

R3: Sorry for the confusion. $f_i\in R^{M\times C}$ represents the features of the $i-th$ scene, which is defined in Eq. (1). $f_{ij}$ represents the features of the $j-th$ point in the $i-th$ scene, as defined in line 284. We will emphasize these definitions again before Eq. (13) in the next version.

Q4: Lastly, it would be insightful to discuss the model’s performance when transitioning from indoor datasets to outdoor datasets. Are there any notable differences in performance, or specific challenges encountered when adapting to outdoor data?

R4: Thank you for your advice. Firstly, Test Time Adaption and domain transfer usually difficult to transfer models from indoor datasets to outdoor datasets, because their categories are not shared, which makes techniques such as pseudo-label, entropy minimization and subsequent evaluation stages impossible. To prove that NGTTA is also applicable to outdoor data, we do a simple experiment where we take the source model PointMetaBase-L trained on 3DFRONT, distill it using a non-parametric geometric model, and update the BatchNorm parameter of the model on the SemanticKITTI dataset. Then we extract the point-wise features of the source model and the updated model respectively, and judge the discriminability of the features by SVM classifier. We hope to use this simple experiment to judge whether NGTTA can improve the feature representation ability of the source model on the outdoor data set, and the results are shown in Table R4. It can be seen that by introducing a non-parametric geometric model, the feature discriminability of the model on outdoor data can be improved, which can prove the effectiveness of NGTTA to some extent. In the uploaded revised version of our paper, we have included this additional experiment in Appendix A.5.1 .

Table R4: SVM Accuracy of NGTTA on 3DFRONT$\rightarrow$SemanticKITTI

Q5: Although the method’s applicability to multiple architectures is partially demonstrated, more rigorous evaluation on additional 3D segmentation backbones (e.g., MinkowskiNet, RandLA-Net) would help generalize the findings. Providing results for diverse architectures could highlight the robustness of the NGTTA framework, particularly if model performance varies significantly with architecture.

R5: Thank you for your suggestion. We have added experiments with MinkowskiNet (ResNet-UNet) and RandLA-Net on 3DFRONT$\rightarrow$S3DIS and 3DFRONT$\rightarrow$ScanNet, and the results are shown in Table R5. As can be seen, NGTTA can be applied to various point cloud architectures, demonstrating the generalizability of our method. In the uploaded revised version of our paper, we have included this additional experiment in Appendix A.5.3 .

Table R5: Results of MinkowskiNet and RandLA-Net

Q6: Visualization of Intermediate Domain Effects: While the paper presents the concept of an “intermediate domain,” it would benefit from visualizations that illustrate this intermediate representation’s impact on the distribution alignment between source and target domains. For example, t-SNE or UMAP plots showing the evolution of feature space alignment as the model adapts would provide more concrete evidence of the intermediate domain’s role in reducing domain gaps.

R6: Thank you for your suggestion. In fact, we attempted to visualize this concept in Figure (3) of the paper, but due to space limitations, we did not provide a detailed explanation. Here, we would like to add some clarification. Firstly, the orange section represents the feature distribution of the target domain, and we can see that the height of the feature distribution is quite consistent, indicating that sufficient supervisory signals have led to well-normalized features. Figure (3) (a) represents the feature distribution of the source model, which shows a highly inconsistent feature distribution, highlighting the differences in distribution. Figure (3) (c) displays the feature distribution of the non-parametric geometric model; although there are still some differences compared to the target domain distribution, the height of the feature distribution is also quite consistent. Compared to the source model, it is closer to the target domain distribution, suggesting that the non-parametric geometric model can be considered an intermediate domain.

However, we believe your suggestion is better, as using t-SNE can provide a clearer visualization of feature alignment during the adaptation process. Therefore, in the uploaded revised version of our paper, we have included the visualization of TSNE during the adaptation process in Appendix A.6.2 .. Furthermore, from the t-SNE visualization results, it can be seen that NGTTA effectively facilitates the gradual migration of the source model to the target domain distribution.

Q7: Comparing the performance of the non-parametric geometric approach against more established geometric features such as FPFH would provide stronger evidence of its efficacy and clarify the specific benefits of the proposed method.

R7: Thank you for your suggestion. We report the performance of FPFH and our chosen non-parametric geometric model, PointNN, on 3DFRONT$\rightarrow$S3DIS and 3DFRONT$\rightarrow$ScanNet, with the results shown in Table R7. It can be observed that PointNN outperforms FPFH. We believe this is primarily because FPFH calculates geometric features based solely on the local neighborhood of each point, resulting in a very limited receptive field. In contrast, PointNN expands the receptive field by aggregating geometric features through multiple layers of down-sampling and up-sampling, which is crucial for scene segmentation. In the uploaded revised version of our paper, we have included this additional result in Appendix A.5.2 .

Table R7: Results of FPFH

Dear Reviewer 71ZT:

We sincerely thank you for taking the time to provide detailed comments. We have made corresponding revisions based on your questions and added a substantial amount of experiments.

Given that the end of the discussion period is approaching, we would like to ask if there are any unclear aspects regarding our work. We are more than happy to provide further explanations as needed.

Our work primarily explores Test Time Adaptation in point cloud scenes, conducted under the challenging SIm2Real setting, which is still an underexplored area. Our approach involves innovative attempts and achieves state-of-the-art performance. We hope that this work will be recognized and valued.

We would like to express our gratitude once again for your feedback and suggestions, which have greatly helped us improve our work.

Best and sincere wishes,

The authors

Q1: The paper lack of reports of runtime related results (time/memory consumption) to demonstrate the cost of introducing NGTTA.

R1: Thank you for your suggestion. Regarding memory consumption and time, we reported the GFLOPs and TP (throughput (ins./sec.)) of different models in Table 3 of the paper. Please refer to the paper for more details. And from the results, we can see that the non-parametric geometric model we adjusted only increases memory consumption by approximately 0.9 GFLOPs.

Q2: The performance gains on real2real setting are not significant.

R2: Thank you for your question. Although the performance improvements brought by our method are not particularly outstanding compared to Sim2Real, we believe that the main comparison should be made with previous TTA and UDA methods. As shown, our approach also surpasses earlier TTA and UDA methods on real2real datasets, demonstrating the effectiveness of NGTTA. We will focus on improving the performance on real2real tasks in our future work.

Q3: It lacks insightful analysis for some results, e.g., why most TTA methods drop greatly on 3DFRONT$\rightarrow$ScanNet setting compared with source only, while not on 3DFRONT$\rightarrow$S3DIS setting ?

R3: Sorry for not explaining this phenomenon in the paper. We believe it is due to the characteristics of the dataset. First, previous TTA methods typically minimize the entropy loss in a straightforward manner, making them very sensitive to class imbalance and sample difficulty. In the case of ScanNet, the input scale is significantly larger than that of S3DIS, which exacerbates the class imbalance issue. Furthermore, the point cloud density and resolution in ScanNet are lower than those in S3DIS, making it inherently more challenging. Specifically, the average entropy for ScanNet is 0.62, while for S3DIS, it is 0.31, indicating that the model is less confident when dealing with ScanNet. These factors contribute to the difficulties in optimizing previous TTA methods on ScanNet, leading to a substantial decrease in performance.

Q4: The paper lacks of category-wise IoU results which could better demonstrates how the proposed method improve the overall performance compared with other methods.

R4: Thank you for your suggestion. We have added the category IoU performance for 3DFRONT$\rightarrow$ScanNet, as shown in the following Table R4. We can see that compared to TENT and Baseline, a significant improvement of NGTTA is observed in the performance of minority and difficult classes. For example, categories such as door, window, bookshelf, and desk have a quantity ratio of only 2% to 4%, and their original performance was poor. As discussed in the third point, previous TTA methods are quite sensitive to class imbalance and the difficulty level of samples. Consequently, after TENT optimization, training may collapse, resulting in an IoU of 0 for these categories. In contrast, NGTTA employs a Category-Balance Sampler to address the class-imbalance problem and distills the underlying manifold features of difficult samples using a non-parametric geometric model, leading to significantly improved performance in these categories. In the uploaded revised version of our paper, we have included this additional result in Appendix A.4.2 .

Table R4: Category-Wise IoU Result on 3DFRONT$\rightarrow$ScanNet.

Q5: How is the visualization made with features in Figure3? It’s better to also mention what x-axis and y-axis means.

R5: The meaning of this figure is the visualization of sample feature distribution in the source domain and target domain, and each curve represents the distribution of each feature channel. So the X-axis represents the values of the feature distribution, while the Y-axis represents the probability densities of different values of the feature distribution.

Dear Reviewer HLQ7:

We sincerely thank you for taking the time to provide detailed comments. We have made corresponding revisions based on your questions and added a substantial amount of experiments.

Given that the end of the discussion period is approaching, we would like to ask if there are any unclear aspects regarding our work. We are more than happy to provide further explanations as needed.

Our work primarily explores Test Time Adaptation in point cloud scenes, conducted under the challenging SIm2Real setting, which is still an underexplored area. Our approach involves innovative attempts and achieves state-of-the-art performance. We hope that this work will be recognized and valued.

We would like to express our gratitude once again for your feedback and suggestions, which have greatly helped us improve our work.

Best and sincere wishes,

The authors