PCoTTA: Continual Test-Time Adaptation for Multi-Task Point Cloud Understanding

We greatly appreciate your detailed review. We're glad to see your kind recognition of the innovation of our pioneering framework for Continual Test-Time Adaptation in multi-task point cloud understanding.

In the following, we will address each of your concerns:

Q1: Typo. Thanks. We will carefully proofread the paper and revise all typos in the revision.

Q2: Learnable prototype in Fig 3(a). Sorry for the confusion. Each learnable prototype is designed to capture distinct features and characteristics of the target domain, paving the way for handling subsequent unknown testing data. These learnable prototypes are dynamic and adjusted during the adaptation process to better align with the source prototypes, which represent the source domains’ features.

Our APM pairs and mixes based on the similarity between the source-learnable prototypes and the current target, effectively incorporating all source domain information. Catastrophic forgetting is effectively mitigated by explicitly fusing inherent source representations (prototypes) and applying graph attention mechanisms to target features, thus achieving good stability of the models.

Q3: CTTA setting. Strictly following typical CTTA methods $9, 37$ , we perform continual test-time adaptation on one additional new dataset, consisting of two target domains to ensure fair comparisons. Our testing samples’ sequence changes randomly and continuously between these two target domains, ensuring a fair comparison in line with the CTTA setting. We will clarify it in the revision.

Q4: Specific challenges of PCoTTA. Simply combining CTTA with point cloud learning faces great challenges. Firstly, compared to grid-structured 2D images, 3D point clouds are unordered and more challenging in CTTA. To address this, we specifically designed a graph attention mechanism to fully learn token sequences within a complete graph structure. This effectively captures contextual relationships and semantic features between unordered and irregular point cloud patches. Secondly, the catastrophic forgetting issue is underexplored for point cloud multi-task learning in continually varying domains, where the model would inevitably forget the knowledge of previously learned tasks while adapting to new ones. Particularly, in 3D point cloud understanding, different tasks are more challenging due to variations in data density and distribution. It’s harder to obtain a unified model that generalizes over raw 3D points compared to grid-like 2D images. To address this, we build a new CoTTA benchmark for multi-task point cloud understanding, and devise task-specific prototype banks, including both source and learnable prototypes, and design APMs by explicitly fusing inherent source prototypes with the current target and applying graph attention mechanisms to target features, thus achieving good stability of the models. Note that our PCoTTA is specifically designed for 3D point clouds, but it has potential to be adjusted to 2D tasks and we will explore it in future work.

Q5: More analysis for the experiment results. We reproduced the CTTA method using PIC as the backbone network. In our benchmark, all sources are treated as one expanded dataset for PIC pre-training, enabling the integration of multi-domain information. Notably, our PCoTTA forms prompt-query pairs from two different sources.

PIC shows significant results and excels in multi-task scenarios, but it lacks specific designs for multi-domain learning. In contrast, our PCoTTA effectively addresses this by aligning the testing data features with the familiar source prototypes and dynamically updating learnable prototypes through the GSFS module.

To the best of our knowledge, no existing CTTA methods support multi-task learning. We propose to integrate CTTA and PIC to facilitate multi-task and multi-domain learning, and our novel design achieves significant improvements. In particular, our task-specific prototype bank enhances multi-task learning capabilities, and the Gaussian Splatted-based Graph Attention efficiently refines target data representation to align with source domains.

Q6: Learnable prototypes. Learnable prototypes are initialized as trainable parameters and designed to capture semantic features across target domains. In CPR, these prototypes become more distinct from each other through the repulsion loss L_pr, while the most similar prototype to the current target is drawn closer to the target features.

In general, this process can be viewed as end-to-end unsupervised clustering, with separation based on the number of potential target domains. Consequently, the number of learnable prototypes ideally approximates the number of target domains, though this is not strictly required.

We conducted an additional ablation study on the number of learnable prototypes, as shown in Table C in rebuttal PDF, and the results indicate minimal changes. Additionally, we show the case with no learnable prototypes (i.e., quantity 0), where our method degrades to aligning the target feature with solely considering source prototypes’ similarities. While this case achieves some degree of test-time adaptation, its performance is less decent than our PCoTTA.

Q7: More experiments to verify catastrophic forgetting and error accumulation. Thanks for your valuable suggestions. Please refer to Q3@Reviewer SkYG for more details.

Q1: Thanks for the comment. We’d like to point out that many recent papers of TTA/CTTA [I, II, III] reveal that the use of source prototypes like features, tokens, and statistics from the source domain do not pose privacy issues and can be utilized to further enhance the adaptability on target data. Firstly, TTAC [I] calculates category-wise and global statistics of source domains as anchors and pre-saves them for streaming online test-time adaptation. Similarly, the published work [II] generates the class-wise source prototypes before model deployment and presents an auxiliary task based on nearest source prototypes to align the source and target features. Besides, TPS [III] computes each class prototypes of source domains, enabling the prototypes to be cached and reused for all subsequent test-time prediction.

Since our work is inspired by them, we strictly follow the same setting. In line with [I, II, III], our source prototypes are pre-cached before deployment and stored as constant parameters (token-like vectors) alongside the pre-trained source model. After deployment, our method does not access the source data to ensure no interaction with the source data is involved in the test-time adaptation stage. Different from them, our method focuses on domain-level prototypes instead of class-level prototypes, avoiding pseudo labeling of categories and instead highlighting the inherent features of the entire domain. Please note [I, II, III] are three examples of this common practice, and there are many emerging in recent years. We just wish to convey that this is indeed a well-recognized practice in the community.

Similar to [I, II, III], source prototypes are the key and indispensable information that we exploit to devise our methodology. Without it, our method is incomplete in addressing the test-time feature shifting during the continual adaptation and would lead to less decent results. We have also analyzed the cases without source prototypes in our ablation studies, as shown by models B and C in Table B of the rebuttal PDF. In this case, our method relies solely on learnable prototypes (i.e., incomplete framework), achieving a certain degree of adaptation. Although this reduces our method's effectiveness, it still outperforms CoTTA [37]. Specifically, CoTTA achieves 58.3, 56.7, and 55.2 during the 3 different rounds, while our PCoTTA achieves 36.8, 36.2, and 35.7, demonstrating superiority and the state-of-the-art performance in continual test-time adaptation for 3D point cloud.

In revision, we will improve the clarity regarding it in the paper.

[I] Su et al. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. In NeurIPS 2022.

[II] Choi et al. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In ECCV 2022.

[III] Sui et al. Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models. Preprint in ArXiv 2024.

Q2: Thanks for the suggestion. Firstly, we would like to stress that our main contribution is that we introduce a new multi-task continual test-time adaptation task for point cloud understanding, and present the first, pioneering framework PCoTTA for this new task, rather than merely a graph attention module. The proposed PCoTTA improves the point cloud model's adaptability and robustness in continuously changing domains by aligning the target domains with all source domains. Besides, we also present a new multi-task benchmark for this new CoTTA setting.

Secondly, although graph attention has been previously studied in the domain adaptation field, it has not been exploited in continual test-time adaptation or the CTTA scenarios in 3D data. Graph attention is effective at capturing contextual relationships between nodes on 2D images as pixels are regularly distributed, but it may be less effective on unordered 3D data. Considering this, we formulate a Gaussian kernel function to calculate attention coefficients based on token similarity, which does not require ordered input and facilitates unordered point cloud patches, thus characterizing 3D point cloud data. We then devise Gaussian Splatted-based Graph Attention that takes the calculated attention coefficients as input and fuses source-learnable prototype pairs with the target data features, aligning them with the sources. This method enables comprehensive, patch similarity-based adaptation for effective CTTA in 3D point cloud understanding.

Lastly, the key difference between a 3D point cloud and a 2D image is that 3D data is disordered, unstructured, and sparsely distributed, making traditional 2D image methods less effective or even cannot be directly applied. As aforementioned, our method involves specific designs for 3D point cloud data, which may need extra adjustments when applying to 2D images, and may lead to less decent performance on 2D images.

Thank you again, and we’d like to provide a bit further clarification here.

Below shows our setting belongs to the CTTA and justifies the exploitation of source prototypes. (Our method is incomplete but still works without exploiting source prototypes information, still outperforming CoTTA [37]. Please see our last response above).

This can be evidenced by the recent CTTA works [IV, V, VI] that involves source prototypes information during the continual test time adaptation. For example, RMT [IV] extracted source prototypes of each class and pre-cached them before the adaptation, and then used the source prototypes to calculate the contrastive loss during the continual test-time. Similarly, SANTA [V] also pre-computed the source prototypes before the adaptation and used them for the target alignment during the continual test time adaptation. Besides, OBAO [VI] also conducted in a similar manner and acknowledged that this is fair in the CTTA setting. As pointed out in Page 8 of this ECCV paper [VI], “Some previous methods [IV, V] directly penalize the movement of target domain samples in the feature space relative to the source prototypes. This can be broadly interpreted as penalizing the movement of corresponding elements between ˆV and Vt in our defined CRG.”

Since we are inspired by and following these works, our setting belongs to the CTTA category. We would revise the paper to re-clarify this issue to eliminate misunderstandings.

Besides, we reproduce two CTTA methods, RMT [IV] and SANTA [V], which use source prototypes, and present the comparison results in the Table below, demonstrating that our method still outperforms these CTTA methods with source prototypes. The reasons for the superiority to these methods lie in three aspects, shown below.

1. Usually, these methods heavily rely on the student-teacher architecture to realize the consistency regularization. As a result, they would inevitably introduce the pseudo label noise in their approaches, leading to error accumulation. Although they use symmetric cross-entropy or other techniques to alleviate the pseudo label noise, such problems still exist and cannot be fundamentally addressed. In contrast, our PCoTTA framework does not use any online or offline pseudo labeling techniques [9, 37], which inherently avoids the risk of error accumulation. In Table 1 of the manuscript, the results across 3 continuous rounds also illustrate the effect in avoiding error accumulation.

2. These methods are specifically designed for CTTA in 2D images and perform well on 2D images. However, compared to 2D images, 3D point cloud data is disordered, unstructured, and sparsely distributed, making these 2D image-based CTTA methods less effective or even cannot be directly applied. Our method involves specific designs for 3D point cloud data, e.g., Gaussian Splatted-based Graph Attention for comprehensive, patch similarity-based adaptation, well-suited for 3D data, and achieves better performances than these methods.

3. These methods often focus on single tasks and all lack specialized design in multi-task learning, which may lead to gradient conflicts in the optimization process of continual test-time adaptation. Instead, Our PCoTTA devises task-specific prototype banks where individual source-learnable prototype pairs are used for different adaptations in each task, thus favoring the multi-task learning in our setting.

[IV]. D ̈obler et al. Robust Mean Teacher for Continual and Gradual Test-Time Adaptation. In CVPR 2023.

[V]. Chakrabarty et al. SANTA: Source Anchoring Network and Target Alignment for Continual Test Time Adaptation. In TMLR 2023.

[VI]. Zhu et al. Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation. In ECCV 2024.

We deeply appreciate your thorough review. We are pleased to see your kind recognition of the innovation of our framework for Continual Test-Time Adaptation in multi-task point cloud understanding and our three new modules. Additionally, we appreciate your acknowledgment of the effectiveness demonstrated through extensive experimental results.

In the following, we will address each of your concerns:

Q1: More experiments to verify catastrophic forgetting and error accumulation. As for the issue of catastrophic forgetting, we have verified it with both quantitative and qualitative experiments. As shown in Table 1 of the manuscript, our continual test-time adaptation setting is similar to the one in CoTTA $37$ where we track the continuous performance across 3 independent evaluation rounds, with samples shuffled randomly in each round. The results in 3 rounds demonstrate the stability, i.e., robust continuous online learning abilities, and resilience against catastrophic forgetting in continually varying target domains. Moreover, we provide T-SNE visualizations for three independent validation rounds in Figure A and task-specific visualizations for each round in Figure B in the rebuttal PDF. Our method remains stable across continuous rounds, demonstrating that our proposed APM and GSFS effectively mitigate catastrophic forgetting by explicitly leveraging constant source prototypes and source domain representations, thereby avoiding over-reliance on adaptively learned information. Finally, our APM pairs and mixes based on the similarity between the source-learnable prototypes and the current target, effectively incorporating all source domain information. As a result, catastrophic forgetting is effectively mitigated by explicitly fusing inherent source representations (prototypes) and applying graph attention mechanisms to target features, thus achieving good stability of our models.

As for the issue of error accumulation, we have verified it both theoretically and empirically. Firstly, as evidenced by $38$ , pseudo labels tend to introduce pseudo label noise and could lead to error accumulation in long-term adaptation. In contrast, in our PCoTTA framework, we do not use any online or offline pseudo labeling techniques $9, 37$ , which inherently avoids the risk of error accumulation. In Table 1 of the manuscript, the results across 3 continuous rounds also illustrate the effect in avoiding error accumulation.

Q2: Source prototype estimation. Sorry for the confusion. Following Point-MAE $26$ , our method reshuffles the patch-wise tokens after the Transformer decoder to maintain their order. Additionally, similar to PIC $9$ , during the testing, our mask is applied to the query target (i.e., the test output), and therefore, patch-wise token shuffling is not needed. Averaging all tokens to form a source prototype ensures a general and comprehensive representation of source domains, avoiding bias from individual samples. This permutation-invariant method effectively summarizes the domain’s information. We will improve the clarity in the revision.

Thank you for your valuable comment.

As per your comment, we conducted the experiment with and without APM using T-SNE visualization just now, obviously indicating undesirable alignment between targets and sources when APM is not used. This is also evidenced by the quantitative results of models B and C in Table B of the rebuttal PDF.

Given the T-SNE visualization figure is not able to be attached here, we will include it in the revision.

We greatly appreciate your thorough review and valuable feedback. We are pleased that you recognized the novelty and practical application of our PCoTTA framework for continual test-time adaptation in multi-task point cloud understanding. We appreciate your acknowledgment of our new benchmark aimed at advancing future research. Additionally, we thank you for your confirmation that our experimental validation and clear writing effectively demonstrated the superiority of our method.

In the following, we will address each of your concerns:

Q1: Other tasks. Thanks for your valuable suggestions. Following PIC $11$ , our PCoTTA is fundamentally designed for regression tasks, making them ‘unified’ with position output (x, y, z) and a single loss. This focus on regression tasks inherently affects its ability to discrimination tasks such as classification. Honestly, multi-task learning in point cloud is still in its early stage, and our focus does not lie in unifying as many tasks as possible. In future work, we would like to specifically enhance the diversity of tasks by developing models applicable to other tasks like classification.

Q2: Real-world application. Thanks. Our method shows potential for many real-world applications, e.g., autonomous driving and virtual reality, as indicated by the analysis of its computational efficiency, e.g., number of parameters and running time, in Table D. Since our PCoTTA is an end-to-end test-time adaptation method that does not employ a teacher-student model or pseudo labeling technique, it is more efficient and suitable for real-time deployment.

However, other tasks like classification are not considered provided we are following PIC’s multi-task setting. As future work, we would like to investigate on how to enhance the diversity of point cloud understanding tasks within a single framework. In addition, though our model is efficient (0.06 seconds at inference), we still need to consider the computing power of devices, and may need to further enhance its efficiency through reducing the size of the backbone model.

Q3: Efficiency metrics. We present an analysis of model parameters and running time in Table D in the rebuttal PDF. The results show that our method can infer target data in a fast speed, and our model has the fewest parameters compared to other CTTA methods.

Q4: More Comparisons. As suggested, we compare our PCoTTA with PointNext. With the same setting, we evaluate it on our new benchmark, where it is trained on all source domains and tested on unseen targets with 3 independent evaluation rounds. Moreover, we reproduced ViDA $1$ , a specialized method for CTTA. Table A shows our method’s superiority.

$1$ Liu et al. Vida: Homeostatic visual domain adapter for continual test time adaptation. In ICLR 2024.

We would like to express our sincere gratitude for your review. We are pleased that you recognized the significance of our compiled point cloud CTTA dataset and the novelty of our three modules to address catastrophic forgetting and error accumulation. Additionally, we are glad that the clarity of our writing ensured a clear understanding of our paper's contributions.

In the following, we will address each of your concerns:

Q1: CTTA setting. Thanks. We acknowledge that the CTTA task prohibits access to source domain data to ensure real-world applicability and data privacy. Strictly following this protocol, our PCoTTA, which utilizes source prototypes, is designed carefully to ensure a fair comparison. The source prototypes are derived exclusively from the source model, which is pre-trained before any adaptation. Source prototypes leverage the structured knowledge embedded within the source model, aiming to improve the robustness and efficiency of the adaptation without compromising data privacy or violating the setting of CTTA. This is analogous to utilizing the parameters (prompts) of a pre-trained source model $9$ in CTTA, and is a common practice in CTTA.

Q2: Missing related works. Following your advice, we will include these papers in the revised version. Since $a$ , $c$ , and $d$ do not have official open-source code or models available, we were unable to compare our results with them. We reproduced $b$ in our setting and evaluated it on our proposed benchmark. The results are shown in Table A, indicating that $b$ just performs similarly to CoTTA $37$ , and our method outperforms it significantly. We attribute this to its focus on domain adaptation without specific designs for multi-task learning, which prevents it from outperforming our method.

Q3: More experiments to verify catastrophic forgetting and error accumulation. As for the issue of catastrophic forgetting, we have verified it with both quantitative and qualitative experiments. As shown in Table 1 of the manuscript, our continual test-time adaptation setting is similar to the one in CoTTA $37$ where we track the continuous performance across 3 independent evaluation rounds, with samples shuffled randomly in each round. The results in 3 rounds demonstrate the stability, i.e., robust continuous online learning abilities, and resilience against catastrophic forgetting in continually varying target domains. Moreover, we provide T-SNE visualizations for three independent validation rounds in Figure A and task-specific visualizations for each round in Figure B in the rebuttal PDF. Our method remains stable across continuous rounds, demonstrating that our proposed APM and GSFS effectively mitigate catastrophic forgetting by explicitly leveraging constant source prototypes and source domain representations, thereby avoiding over-reliance on adaptively learned information. Finally, our APM pairs and mixes based on the similarity between the source-learnable prototypes and the current target, effectively incorporating all source domain information. As a result, catastrophic forgetting is effectively mitigated by explicitly fusing inherent source representations (prototypes) and applying graph attention mechanisms to target features, thus achieving good stability of our models.

As for the issue of error accumulation, we have verified it both theoretically and empirically. Firstly, as evidenced by $38$ , pseudo labels tend to introduce pseudo label noise and could lead to error accumulation in long-term adaptation. In contrast, in our PCoTTA framework, we do not use any online or offline pseudo labeling techniques $9, 37$ , which inherently avoids the risk of error accumulation. In Table 1 of the manuscript, the results across 3 continuous rounds also illustrate the effect in avoiding error accumulation.

Q4: More ablations and clarification of CPR. Thanks. As per your constructive advice, we present additional ablation studies in Table B, evaluating the use of CPR and GSFS individually. These results clearly demonstrate the incremental benefits of each component and their combined effect on improving performance.

While contrastive learning is a well-established technique, our CPR leverages domain prototypes (the new knowledge), introducing a novel aspect to the method. Traditional contrastive learning focuses on instance-level representations, whereas our approach innovates on the use of domain-level learnable prototype interactions. Equipped with the proposed CPR and learnable prototypes, our method provides structured and informative representations of various target domains to guide the adaptation process. We will revise this in the new version.

Q5: Visualizations comparisons with other CTTA methods. Thanks. We have provided the visual results of other CTTA methods in Appendix A.3. Our PCoTTA excels in producing high-quality predictions across multiple tasks, even as the target domain changes. This is due to our three innovative modules, which minimize discrepancies between source and target domains, enhancing overall prediction quality. We will include these visualizations in the revised paper.

I appreciate the author’s response and the additional experiments, but I still have some unresolved concerns.

1）CTTA setting problem: First, I am very clear about the description of the CTTA setting in papers [9] and [37]. In CTTA, it is permissible to use source domain pre-trained model parameters. So, I would like to ask if the source prototypes you used are the features/tokens extracted by the model or the model's parameters/token-wise visual prompt. As you described in Line 158, “we save all source prototypes Zs derived from the model at the last epoch,” so you not only used the model parameters but also the source prototypes. In the CTTA setting, only the pre-trained source model is allowed to be accessed; you cannot access any part of the source model's training process, as this would still involve interacting with the source data. How did you obtain the source prototypes? For example, if the source model is already deployed on an end-device, how do you obtain the source prototypes? Existing source-free TTA and source-free CTTA methods [35, 37, a, b] do not access any source domain features/tokens. Therefore, I suggest that the method proposed in this paper should not be compared with source-free methods.

2）Paper [a] has official open-source code available at https://github.com/Lily-Le/EcoTTA.

3）What are the details of the reproduction of paper [b] in table A? Why is it performing worse than the source model?

If my remaining concerns can be addressed, I am willing to improve my rating.

Thanks for your comments. We would like to further address your concerns as follows:

Q1: Sorry for causing confusion. We agree that the source prototypes we used are the tokens extracted by the model during the source pretraining stage and pre-computed and pre-saved in the device for test-time adaptation. We’d like to stress that this does not violate the fairness of the CTTA setting.

In fact, many recent papers of TTA/CTTA [I, II, III] reveal that the use of source prototypes like features, tokens, and statistics from the source domain do not pose privacy issues and can be utilized to further enhance the adaptability on target data. Firstly, TTAC [I] calculates category-wise and global statistics of source domains as anchors and pre-saves them for streaming online test-time adaptation. Similarly, the published work [II] generates the class-wise source prototypes before model deployment and presents an auxiliary task based on nearest source prototypes to align the source and target features. Besides, TPS [III] computes each class prototypes of source domains, enabling the prototypes to be cached and reused for all subsequent test-time prediction.

Since our work is following and inspired by them, we strictly follow the same setting. In line with [I, II, III], our source prototypes are pre-cached before deployment and stored as constant parameters (token-like vectors) alongside the pre-trained source model. After deployment, our method does not access the source data to ensure no interaction with the source data is involved in the test-time adaptation stage. Different from them, our method focuses on domain-level prototypes instead of class-level prototypes, avoiding pseudo labeling of categories and instead highlighting the inherent features of the entire domain.

As per your constructive advice, we will carefully re-clarify this issue in the revised manuscript to eliminate misunderstandings.

[I] Su et al. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. In NeurIPS 2022.

[II] Choi et al. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In ECCV 2022.

[III] Sui et al. Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models. Preprint in ArXiv 2024.

Q2: Thank you for the information, we have also noticed this link, however, this is a community implementation rather than the official code. Additionally, according to the issue mentioned here https://github.com/Lily-Le/EcoTTA/issues/1#issuecomment-1667177037, the repository's author noted that the replicated results were not satisfactory and unsuccessful. Nonetheless, we still tested it on our new benchmark and obtained sub-optimal results. Therefore, we chose not to include these results in the rebuttal process to avoid any biased comparisons. We have cited this inspiring work in the revision and would endeavor to reproduce EcoTTA for comparisons in the future.

Q3: Apologies for the confusion. Firstly, PointNext in Table A is another point cloud understanding method that we reproduced under the multi-task setting, as suggested by Reviewer ynhY for additional comparisons. It is not the source model for ViDA [b].

Since we are following the same settings as CoTTA [37] for reproduction, we also use PIC [9] as the source model for ViDA and thus equip ViDA with multi-tasking and multi-domain learning capabilities. As such, all these CTTA methods will start from the same source pre-trained model and ensure fair comparisons.

From the table, we can observe that ViDA indeed achieves better results than PIC, but it does not outperform our method. The reasons for the superiority of our PCoTAA to ViDA lie in several aspects. Firstly, ViDA employs a teacher-student framework and uses a consistency loss where the teacher models’ predictions serve as the pseudo labels of the student model. However, as evidenced by [38], pseudo labels tend to introduce pseudo label noise and could lead to error accumulation in long-term adaptation. In contrast, in our PCoTTA framework, we do not use any online or offline pseudo labeling techniques [9, 37], which inherently avoids the risk of error accumulation. Secondly, ViDA is specifically designed for CTTA in 2D images and cannot well tackle the CTTA in 3D data. This is because compared to grid-structured 2D images, 3D point clouds are unordered and high dimensional, which is more challenging in CTTA. To address this, we specifically designed a graph attention mechanism to fully learn token sequences within a complete graph structure. This effectively captures contextual relationships and semantic features between unordered and irregular point cloud patches.

There are serious problems with the reproduction of SANTA. Your baseline model is PIC, which is a transformer-based model, and each transformer block includes layer normalization. However, in the reproduction details, you claim, "we only update the BatchNorm layer parameters in the source model during adaptation." This reproduction and the associated experiments have significant flaws, and I also have concerns about the reproduction results for the comparison method TENT [35].

Thanks for your comment. We think it may cause you a misunderstanding here. We ensured to use PIC [8] as the source model for all reproduction methods. In PIC, the model comprises the token embedding module (Encoder) and Transformer blocks, where the Encoder contains BatchNorm layers (shown below). Following the original works SANTA [V] and TENT [35], their BatchNorm layers are updated. This ensured fairness in reproduction. Please see the network details below. We hope this clarifies your concerns.

…
(MAE_encoder): MaskTransformer(
    (encoder): Encoder(
      (first_conv): Sequential(
        (0): Conv1d(3, 128, kernel_size=(1,), stride=(1,))
        (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
        (3): Conv1d(128, 256, kernel_size=(1,), stride=(1,))
      )
      (second_conv): Sequential(
        (0): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
        (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
        (3): Conv1d(512, 384, kernel_size=(1,), stride=(1,))
      )
    )
    (blocks): TransformerEncoder(
      (blocks): ModuleList(
        (0): Block(
          (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (drop_path): Identity()
          (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=384, out_features=1536, bias=True)
            (act): GELU(approximate=none)
            (fc2): Linear(in_features=1536, out_features=384, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
          (attn): Attention(
            (qkv): Linear(in_features=384, out_features=1152, bias=False)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=384, out_features=384, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
        )
…

Thanks for further comment.

As per your comment, we further update the LayerNorm parameters for SANTA [V], TENT [35], and AdaBN [18]. AdaBN [18] was compared in our paper and involves BatchNorm update. The results are shown in the below table, indicating that the update of LayerNorm only improves a bit and our method still outperforms them obviously. Our method does not update the source model (including Transformer Blocks).

We will discuss the above in the revision.

[V]. Chakrabarty et al. SANTA: Source Anchoring Network and Target Alignment for Continual Test Time Adaptation. In TMLR 2023.

[18] Li et al. Revisiting batch normalization for practical domain adaptation[J]. In Preprint ArXiv 2016.

[35] Wang et al. Tent: Fully test-time adaptation by entropy minimization. In ICLR 2021.