SMART-PC: Skeletal Model Adaptation for Robust Test-Time Training in Point Clouds

We thank the reviewer for recognizing the novelty of our skeleton-based framework and the effectiveness of skeletal representations, as well as our thorough experimental validation.

Weakness

1 and 2. Our method supports two modes of adaptation: one with backpropagation and one that is backpropagation-free. The goal of the backpropagation-free mode is to demonstrate that, during pretraining, the use of a skeletal prediction branch encourages the model to learn robust and meaningful geometric features. As a result, during test-time adaptation, simply updating the statistical parameters of the BatchNorm layers (i.e., running mean and variance) is sufficient to enhance performance on corrupted datasets. This effect is validated by our results in Table 1 of the main paper.

To further support this claim, we conducted additional experiments using the BFTT3D[1] method, employing three different pretrained models:

 1. Org-SO: A baseline model pretrained with only an encoder and classification head.

 2. MATE-SO: A model pretrained using the MATE framework, which includes an encoder, a reconstruction decoder, and a classification head.

 3. SMART-PC-SO: Our model, pretrained with an encoder, a skeleton-based decoder, and a classification head.

Each of these models was evaluated using the same BFTT3D adaptation strategy. The results in Table 1 show that our pretrained model significantly outperforms both the baseline and MATE-pretrained models in the backpropagation-free setting across all three datasets. This provides strong evidence that the skeletal prediction branch enables the model to extract more structure-aware and corruption-resilient features, which allow effective test-time adaptation even without updating model weights via backpropagation.

Table 1: Mean Accuracy (%) of BFTT3D with Different Pretrained Models

[1] Backpropagation-free network for 3d test-time adaptation

---

3. We confirm that our method does not use any supervised ground truth for skeleton prediction. Instead, the model learns to predict the skeleton structure (position and radius) in an unsupervised manner using the loss functions defined in Equations 11, 12, and 13. These losses serve as supervisory signals and include: (1) a point-to-sphere distance loss ensuring coverage of the input shape, (2) a skeleton-to-point loss encouraging compactness, and (3) a radius regularization term. Together, they provide strong geometric guidance, even without explicit supervision.

Our goal is not to recover the point cloud itself (as in standard reconstruction branches), but to extract a more structured and abstract representation that captures the underlying geometry. The effectiveness of this approach is validated by the results in Table 1 of the main paper, which show that the features learned through the skeleton prediction branch lead to significantly better performance on corrupted datasets, especially in the backpropagation-free adaptation mode, compared to prior reconstruction-based methods like MATE. Additionally, we tested our pretrained model (SMART-PC-SO) with the BFTT3D method in the backpropagation-free setting, and compared it to Org-SO and MATE-SO. As shown in Table 1, our skeleton-pretrained model achieves higher robustness across corruptions, further supporting the generalizability and strength of the learned skeletal features.

---

4. Since our implementation is largely based on the official MATE codebase, it is crucial to ensure the reproducibility of their reported results for a fair comparison. To this end, we faithfully reproduced their results using the exact code, hyperparameters, and pretrained models provided in their public GitHub repository (we witnessed a discrepancy in their results on ModelNet40-C and ScanObjectNN-C). For transparency and verification, we have also included the corresponding log files in our anonymous repository (https://anonymous.4open.science/r/SMART-PC-ICML-737C/).

---

5. We thank the reviewer for pointing this out. Please refer to our response to Reviewer k4Xv for the updated Table 1. We regret that, due to space constraints, we are unable to include the full content here.

---

6. We conducted this experiment, and the results are shown in Table 2 in our response to reviewer k4Xv. For DDA and CloudFixer, we report the FPS values directly from their respective papers. For all other methods, we measured the FPS ourselves, including the time required for adaptation. Importantly, the reported FPS for SMART-PC corresponds to the backpropagation-free mode, highlighting its efficiency under test-time adaptation without gradient updates.

---

7. We will revise and improve the writing of our paper in the final version.

We thank the reviewer for recognizing the novelty, clarity, and effectiveness of our method in improving robustness and efficiency for point cloud test-time training.

1. METHODS AND EVALUATION CRITERIA - inconsistent Notation for Mask Predictors

We'll correct the notations in the final version for clarity and consistency.

---

2. METHODS AND EVALUATION CRITERIA - Clarification on Decoder Structure in Figure 2

Our intention was to illustrate that each skeleton point has an associated radius, predicted separately using MLP_s and MLP_r. Both outputs are used to construct the final figure. For visualization purposes, they were shown connected. However, we acknowledge that this may cause confusion. In the final version, we will revise the figure to clearly separate these components and better reflect the equations in the main paper.

---

Theoretical Claims

By incorporating skeletal representation learning, we believe our model introduces a structural inductive bias that prioritizes shape and topology, which aligns with deep learning theory emphasizing that such biases improve generalization and robustness under distribution shifts [1].

[1] Shortcut learning in deep neural networks. Nature Machine Intelligence

---

EXPERIMENTAL DESIGNS OR ANALYSES

1. We have included a visualization [Figure 1] (https://anonymous.4open.science/r/SMART-PC-ICML-737C/skeleton.pdf) that illustrates how our method abstracts both clean and corrupted point clouds into compact and meaningful skeleton representations. The top row shows a point cloud under different conditions (clean, Uniform noise, Background noise), and the middle row relates to the corresponding skeletons. From this visualization, the model can estimate the skeleton under a uniform noise, preserving the overall shape of the clean data. This shows that the skeleton's abstract representation is less sensitive to the noise compared to the original points (Success Case). But under a harsh noise condition (right), the skeletons try to cover all the background outlier points, making the radius excessively large, hindering the effective representation of the main inlier points. Accuracy gain (bottom row) shows the trend; more visualizations will be provided in the final version.

---

2. We provided an ablation study in Table 1 to quantify the impact of each skeletal loss component. The best performance is achieved with coefficients (0.3, 1.0, 0.4)—as suggested in the Point2Skeleton paper—showing a good improvement in mean corrupted accuracy (72.95%) compared to other settings. This confirms that each loss term contributes to learning more robust features.

Table 1: Ablation study of skeleton loss coefficients (ModelNet40 / ModelNet40-C, online adaptation)

---

3. As described in the main paper, the Radius Regularization Loss (Equation 6) is designed to avoid instability caused by overly small radii, especially under noisy conditions. This loss encourages the model to learn larger and more stable radii, which improves the robustness of the skeletal abstraction. Although we do not observe excessively large radii, the Point-to-Sphere and Sampling losses (Equations 11 and 12) implicitly constrain radius size by preserving geometric consistency. As shown in Table 1, removing the regularization term leads to a drop in performance, confirming its importance.

---

4. In [Figure 1] (https://anonymous.4open.science/r/SMART-PC-ICML-737C/skeleton.pdf), we have included a failure case (right) to showcase a potential limitation of the skeleton estimation. From this figure, when the noise expands far beyond the main object's dimensions, the skeleton gets distracted by this out-of-distribution dimensionality change, and while it remains focused on the original points, its radii get excessively large in a way that it no longer represents the original point cloud's shape.

---

5. In standard mode, the batch size is 1, and the model is reset for each sample, so updating BN statistics (backpropagation-free) cannot help the model. To further investigate, we conducted an ablation study on the ScanObjectNN-C dataset in standard mode using the backpropagation-free setting. We tested batch sizes of 8, 16, and 32 with a single iteration for efficiency, as shown in Table 2. It shows that in standard mode, SMART-PC can improve performance on corrupted datasets in the backpropagation-free setting when the batch size is increased.

Table 2: Ablation study of SMART-PC (standard mode and backpropagation-free) on ScanObjectNN-C.

We appreciate the reviewer’s thoughtful feedback and their recognition of our contribution to pretext task design and its connection to prior work in test-time adaptation for point cloud recognition.

1. Theoretical Claims

To clarify, our method does not use labeled source data during the test-time training phase. The classification loss discussed in Section 3.4 is only applied during the pre-training phase on the source dataset to learn the initial model parameters. During test-time training, only the target domain data without labels is used, and no source labels are required. Only the encoder and decoder are adapted at test time; the classifier is frozen (Figure 2). Furthermore, we emphasize that our method strictly follows the standard definition of Test-Time Training (TTT), as established in prior works such as the MATE paper. We acknowledge that our wording in Section 3.4 may have caused confusion, and we will clarify this distinction in the final version of the paper.

---

2. Comparison with Recent Test-Time Adaptation Methods

Thank you for the suggestion. As shown in Table 1, we repreduced BFTT3D[2] and SVWA[3] using our settings. Since SVWA produced very low results in our standard setting (MoodelNet40-C: 5.54% and ScanObjectNN-C: 21.79%), we used BS=4, It=1, and N_v=2 for adaptation with the pretrained model available on their GitHub repository. Diffusion-based methods (CloudFixer[1], DDA[4]) are reported separately due to slow Frame/Second (see Table 2). Due to time constraints, we were unable to reproduce them during the rebuttal period.

[1] CloudFixer: Test-Time Adaptation for 3D Point Clouds via Diffusion-Guided Geometric Transformation

[2] Backpropagation-free network for 3d test-time adaptation

[3] Time Adaptation in Point Clouds: Leveraging Sampling Variation with Weight Averaging.

[4] Diffusion-driven adaptation to test-time corruption.

Table 1: Top-1 Accuracy (%) Comparation With More Recent Methods (🆕)

Table 2: Frame per Second Comparison (FPS), N_v is the number of sampling variations.

---

3. Clarification on Batch Normalization Updates

During pretraining, our method learns more abstract and robust features through the skeleton prediction branch. These features are resilient to corruption, such that during test-time adaptation, simply updating the statistical parameters of the BatchNorm layers (i.e., running mean and variance) can effectively suppress noise without requiring backpropagation. As shown in [Figure 1] (https://anonymous.4open.science/r/SMART-PC-ICML-737C/skeleton.pdf), the predicted skeleton offers a noise-resistant representation that aligns corrupted inputs with clean data. Further details will be provided in the supplementary material.

---

4. Clarification on Feature Fusion in the Classification Branch

The encoder and decoder outputs share the same shape (B, N, D), where B is the batch size, N the number of tokens, and D the feature dimension. These features are combined through element-wise addition and passed to the classification head. This design enriches the encoder’s representation with structural information from the decoder, which captures the skeletal geometry of the object. As shown in Table 2 in the main paper, this improves classification accuracy. We will clarify this mechanism and revise Figure 2 in the final version.

---

5. Comparison and Advantages of the Proposed Backpropagation-Free Approach over BFTT3D

Unlike BFTT3D, which uses class-based prototypes from the source dataset during adaptation, our method relies solely on the target dataset without accessing any source information. Another key distinction is efficiency: as shown in Table 2, our method in the backpropagation-free mode achieves significantly higher FPS compared to MATE, SVWA, BFTT3D, CloudFixer, and DDA. In terms of performance, our method also outperforms BFTT3D across three corrupted datasets, as reported in Table 1. Together, Tables 1 and 2 demonstrate that our skeleton-based pretraining enables the model to learn robust geometric features, allowing effective adaptation with only statistical parameter updates, achieving both high accuracy and fast inference.