Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learning

We thank the reviewer for their thoughtful feedback and appreciation of our paper’s clarity, motivations, results, and for recognizing the value of large-scale training and strong performance on ScanObjectNN. We address the concerns and questions point-by-point below.

W1: Adaptation of ideas

While our work is inspired by progress in 2D vision, we emphasize that it is not a direct adaptation, but rather a non-trivial and novel composition of ideas tailored to the 3D point cloud domain, supported through intuitive and empirical findings. We believe that certain findings may also be valuable to other SSRL methods for point clouds.

We agree that components such as latent prediction or self-distillation are conceptually present in numerous works on 2D SSRL, with various having a counterpart implementation in the 3D domain (we thank the reviewer for pointing out Cardace et al. [a]). However, these 3D SSRL methods often propose a rather direct adaptation from the 2D domain, thereby not, or partially, addressing the challenges of 3D data.

In contrast, we developed our method from the ground up, explicitly accounting for these challenges. As such, it cannot be directly reduced to prior methods. For example, previous SSRL methods with dual latent objectives such as iBOT or DINOv2 (for images) are not asymmetric, neither between the student and teacher networks nor between their global and local objective branches, whereas our design introduces asymmetry in both aspects.

We are also not directly aware of any prior work that considers disabling attention between mask queries at the decoder/predictor level. It is simple and effective, yet works such as Point-MAE or point2vec do not explore this design choice, while they could also benefit from it.

W2 and Q1: Additional linear evaluations

Why did we not run more Linear evals on other SSRL methods with a standard transformer backbone? We ran into challenges with their codebases, including environment setup (e.g., compiling custom CUDA kernels), errors during execution, and difficulties when porting models into our framework.

However, we agree with the reviewer that these are valuable for a direct comparison, especially as a linear probe can be a better indication of the semantic quality of learned representations compared to full fine-tuning. With some extra work, we now have some additional results:

As shown above, AsymDSD-S consistently outperforms prior methods. Notably, it improves over the next-best method by +5.8% on OBJ_BG and +9.1% on PB_T50_RS.

We did not include results of Point-GPT as the linear performance was rather poor, even after some modifications (<70% on ScanObjectNN).

Note that Recon MAE is an improved implementation of Point-MAE with two standard transformers (two-tower network). Point-RAE also uses a standard transformer, but has an additional attention-based module. We will update Table 1 to reflect these details more accurately: models using a standard transformer backbone will be marked with ST, and an additional column AM will indicate the presence of extra attention layers/modules.

Q2 Ablation on removing self-attention

Table 5c shows in in the first two rows that the default design with full self-attention (over the visible context c and masks m) decrease performance -2.6% on full fine-tuning compared to the design with cross-attention (from each masked query) onto the visible context c. Figure 13a and Figure 13c in the appendix visualize these two designs.

Q3 Figure 2: encoder shape difference

The teacher and student encoder have the same layers and thus the same number of parameters, which we represented through identical lengths of the two encoders in the figure. The width of the encoder rectangle represents the number of patches it process (sequence length). We will clarify this by updating the caption.

We thank the reviewer for their detailed feedback and constructive suggestions. Below we address the main concerns point-by-point.

W1 and Q1: Large scale pre-training

We would like to clarify that our main reported results (e.g., 90.53% on ScanObjectNN) are obtained from pretraining on the standard ShapeNet dataset (Sec. 4.1). This setup is fully reproducible on standard consumer-grade hardware (e.g. single rtx 4090) and does not require the large-scale 930K mixed dataset, which was used only for the scaling study.

The submitted code includes scripts and configuration files for this ShapeNet pretraining setup, and Table 6 in the appendix lists all hyperparameters in detail. We will also release all main pretraining checkpoints (AsymSD‑MPM‑S, AsymSD‑CLS‑S, AsymDSD‑S, AsymDSD‑S*, and AsymDSD‑B*) on HuggingFace to further support reproducibility.

W2 and Q2: Compute efficiency concerns

• EMA update cost is negligible: The EMA update itself accounts for only ~12 ms out of 500 ms per iteration, so it does not significantly impact runtime.
• Multi-mask reduces, not increases, cost of the EMA update/teacher: Multi-mask sampling amortizes the teacher computation over multiple masks, which both reduces memory usage and increases throughput by ~70% (Table 5d), with no loss in accuracy. Multi-mask has no influence on the frequency of EMA updates.
• Overall training time is competitive: Our full pretraining takes 18 hours, which is similar to several other SSRL methods such as point2vec and significantly faster than ReCon (54 hours). Moreover, the method converges quickly, reaching 88% ScanObjectNN accuracy in just 60 epochs (<4 hours) on ShapeNet, already outperforming all other SSRL methods with a standard transformer.
• Deployment efficiency: The encoder that is kept after pretraining is identical in architecture and size to many methods in Table 1, meaning there is no additional inference cost compared to existing approaches.
• INT8 deployment is supported, yielding 87.13% accuracy on ScanObjectNN, which can likely be improved further with quantization-aware training.

We would like to highlight that our SSRL method is not intended for real-time on-device pretraining. In addition, we view scaling-up as a forward-looking research direction, inspired by developments in NLP and 2D CV.

W4: Local crops and point cloud density differences

While a minimum 5% crop may seem small, it is intentionally chosen to challenge the model to infer global semantics from fine-grained details rather than relying solely on larger contexts. This can increase robustness to point clouds with large occluded regions or lower point density. This threshold was empirically determined to deliver strong results.

Robustness to point cloud density: Our multi-crop strategy keeps the number of point patches consistent while varying the spatial scale of the object. This introduces scale and density invariance that helps the model generalize across diverse geometric configurations and reduces overfitting to features at a fixed scale/density. Figure 12b in the appendix illustrates this effect: in a zero-shot visualization of patch embeddings for lamp objects, the embeddings of the conical lamp covers are homogenous (light green color) across different scales

W5 and Q3: Prioritizing semantic abstraction over reconstruction

Our method is indeed designed to prioritize semantic abstraction, as this is critical for robust representation learning. However, we note that semantic understanding is also essential for point cloud reconstruction, especially with larger occluded regions [a]. Purely reconstruction-based objectives (e.g., Chamfer distance) such as in Point-MAE optimize for minimal point-wise error. While this can lead to reconstructions that are geometrically close to a sampled point set in Euclidean distance, this does not necessarily lead to the most semantically coherent reconstructions.

We also did not include results on reconstruction benchmarks as other works such as Point-MAE do not quantitatively report on this. However, in Table 3 we show strong part segmentation performance, which often correlates with geometric fidelity. Our results are comparable to Point‑MAE, suggesting that low-level geometric understanding is preserved despite the focus on semantic abstraction.

Q4 Evaluation on real sparse point clouds

We agree that evaluating on real scene-level LiDAR data is important. However, our method, like most prior 3D SSRL works, uses a standard transformer backbone, chosen deliberately to ensure fair comparison. While this architecture is widely used, it is not well-suited for very large, scene-wise datasets such as ScanNet or KITTI, which tend to work better with hierarchical backbones. For this reason, we focused our evaluation on the established object-centric benchmarks (ModelNet40 and ScanObjectNN) reported on by most other 3D SSRL methods.

For a more challenging dataset, we included results on Objaverse‑LVIS in the appendix. This benchmark covers roughly 1,000 diverse object categories and is substantially more challenging than ModelNet40 or ScanObjectNN. We will move these results to the main text and release the full benchmark setup publicly. This might incentivize the adoption of this more challenging benchmark by future works

We acknowledge that scaling to large, sparse scene‑level point clouds remains an open challenge (also noted as a limitation in the paper), and we are actively exploring ways to adapt AsymDSD for this scenario by using different backbone designs.

[a] deep point cloud reconstruction

We thank the reviewer for their constructive feedback and are glad they found the paper clear and recognize its competitive performance. Below we address the main concerns point-by-point.

W1 Technical Novelty and Synergy Between Objectives

We appreciate the reviewer’s concern regarding novelty and would like to clarify our position. While the fusion of masked token prediction with an EMA teacher has indeed been explored in prior 2D SSRL methods and in 3D works such as Point2Vec, we do not claim novelty on this general idea alone.

Importantly, AsymDSD is not a direct adaptation, but rather a non-trivial and novel composition of several ideas tailored to the 3D point cloud domain, supported through intuitive and empirical findings. We address unique challenges of 3D data such as global shape leakage and efficiency concerns that existing methods do not consider. We particularly want to emphasize the following contributions:

• Asymmetry in design: We introduce asymmetry both between the teacher and student networks and between the global invariance and local masked modeling branches, which prior dual-objective methods including those on images (e.g., iBOT, DINOv2) do not. In the main text, we list numerous benefits to this design choice.
• Disabling attention between mask-queris: We propose a simple yet effective design at the predictor level by disabling attention between mask tokens, which to our knowledge has not been explored in prior 3D SSRL methods. This prevents masked tokens from “cheating” by leaking global shape information through attention.

Synergy between objectives: It is unclear whether the improvement stems from their integration or just from one dominant branch (e.g., invariance).

The gains are not driven by one dominant branch. As shown in Section 4.3 (Table 1):

• The global invariance objective alone (AsymSD‑CLS‑S) achieves 88.72%, and the masked point modeling objective alone (AsymSD‑MPM‑S) achieves 88.58% on ScanObjectNN.
• Combining both objectives with AsymDSD achieves 90.53% (+1.81%), exceeding both branches individually. This indicates a synergystic effect between the two branches. (See also our response to W5)

In Figure 7, we also show that the attention patterns of the two branches are relatively aligned. When replacing the latent-space MPM with raw point reconstruction (MAE), we observe an inverted attention pattern, suggesting a mismatch between the objectives. Following our additional experiments in W5, we will include a direct visualization of the MAE attention pattern in the final version to further clarify this difference.

W2: Mixup augmentation

We are aware of orthogonal approaches such as mixup (also for point clouds), but note that these methods tradionally still rely on labeled data. We believe that scaling SSRL methods to larger unlabeled datasets ultimately leads to stronger representational quality and broader applicability (generalization).

W3: Marginal gains over SOTA in some settings

We would like to clarify that ModelNet40 performance under full fine-tuning is close to saturated, and improvements there are naturally small. We included these results for completeness to show that AsymDSD does not regress on this benchmark.

As shown above (gathered from Table 1), on the more challenging ScanObjectNN splits, AsymDSD achieves much larger improvements (+1.9% to +3.6%) under full-finetuning compared to prior methods using the same standard transformer backbone.

We also note that when all other factors (backbone, dataset, pretraining protocol) are fixed, i.e. the experiment as explained in Sec 4.1, then the performance gains are inherently limited. This makes the improvements on ScanObjectNN particularly meaningful.

Finally, we view linear probing as a better indicator of representation quality than full fine-tuning. We will move additional linear probe results (see the table in our response to Reviewer 3Aup) into the main paper, which further highlight the strength of the learned representations.

W4: Method Complexity vs Practicality Trade-Off

We recognize that AsymDSD combines several components, and there is a natural trade-off between complexity and practicality. However, the components are kept simple (not over-engineered) and have clear motivations to be included:

• Ablation evidence:  Table 5 shows consistent performance drops when removing elements such as multi-crop, predictor (asymmetry), or using a cross-attention–only predictor. While the masking strategy is less fundamental, inverse block-wise masking is required to achieve the full performance of AsymDSD.
• Efficiency-focused design: Multi-mask is a prime example of this philosophy: it is not intended to increase accuracy, but rather to reduce memory usage and improve throughput by nearly 70% at no performance cost (Table 5d).

Reproducibility: We have ensured the design remains accessible and reproducible:

• Many components (e.g., multi-mask, cross-attention–only predictor) are straightforward to implement in a few lines of code.
• Other parts have widely shared implementations (e.g. EMA module)
• Most importantly, we have included the codebase and configuration files for pretraining in the submission and will release them publicly, so other researchers can reproduce the results easily.

W5: Unclear Advantage of Latent-Space Prediction

We agree with the reviewer that it is valuable to isolate the effect of latent-space prediction over point coordinate reconstruction. To this end, we ran some additional experiments:

Key observations: We see that our implementation of MAE (raw point reconstruction) greatly improves over Point-MAE. We expected this, as MAE also benefits from addressing the global shape leakage through our decoder and from the scale invariance introduced by variable‑sized global crops. This reveals that some of our contributions extend beyond our framework, and are thus valuable to other SSRL methods.

However, when we combine the global invariance learning (CLS) with MAE (CLS + MAE), the performance slightly drops compared to just using CLS. Only when we combine it with our latent MPM (CLS + MPM) we see a synergistic effect emerging, where the model exceeds both objectives in isolation.

We thank the reviewer for pointing out this particular ablation. We will include the results in the main text.

W6: Limited evaluation on scene-level tasks

Similar to most prior 3D SSRL works, our method currently uses a standard transformer backbone to ensure fair comparison. While widely adopted, this backbone is not ideal for very large scene‑level datasets, which typically benefit from hierarchical designs tailored for sparse and large‑scale inputs. For this reason, we focused our evaluation on the established object‑centric benchmarks (ModelNet40 and ScanObjectNN) that are standard in the 3D SSRL literature.

It is worth noting that ScanObjectNN is already a challenging real‑world dataset, especially the OBJ_BG and PB_T50_RS splits, which include background clutter, occlusions, and partial scans. We further ran a experiments on Objaverse‑LVIS (see appendix), which spans nearly 1,000 diverse object categories and is substantially more challenging than ModelNet40 or ScanObjectNN. We plan to move these results into the main text and will release the full benchmark setup publicly, with the aim of encouraging wider adoption by future work.

Still, we acknowledge that scaling to large scene‑level point clouds remains an open challenge (also noted as a limitation in the paper), and we are actively exploring ways to adapt AsymDSD for this scenario by using different backbone designs.

The author has addressed my concerns. I will raise the rate. By the way, some mixup works are unsupervised via reconstruction like Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes.

We thank the reviewer for their thoughtful feedback and for highlighting the strengths of our work, including its clear positioning within the field, scalability, and the use of creative design choices. Below, we address the identified weaknesses and questions.

W1: Clarification: Centering/sharpening

We appreciate the reviewer highlighting this point. While centering and sharpening are standard components in self-distillation frameworks (e.g., DINO), and thus not novel in our setting, we recognize that our current explanation could benefit from more clarity. We will revise this by including the following concepts:

• Explanation of what the collapse modes intuitively mean:
  1. $H\left(p_\theta(\mathrm{z})\right)= 0$ - the model collapses to a single latent representation $z$ that dominates, always assigned probability 1.
  2. $H(p_\theta(\mathrm{z}\mid \mathrm{x}))=\log |\mathcal{Z}|$ - The model always outputs a uniform distribution over the latent space.
• How centering and sharpening countract these failure modes:
  1. Centering subtracts a running mean of the teacher logits, reducing the logits of latents that are frequently assigned high probability thus helping avoid low‑entropy collapse.
  2. Sharpening makes the distribution more peaked, pushing it away from a uniform distribution and thus avoiding high‑entropy collapse.

W2 and Q2: Generalizability to other architectures

We appreciate the reviewer’s recognition that AsymDSD is model‑agnostic and can be adapted to different backbone architectures. Our current results use a standard transformer backbone to ensure fair comparison with the majority of prior 3D SSRL methods, which follow the same evaluation protocol. This choice isolates the effect of the self‑supervised framework itself rather than conflating it with gains from more advanced backbones.

Still, we agree it is valuable to demonstrate results on other architectures and are actively working on this. In particular, we are currently porting AsymDSD to PointMamba, a state‑space–based backbone. Due to time constraints, we are unable to include results in this rebuttal, but we plan on adding them to this work.

Q1: Explanation of the attention–distance visualization

The visualization in Figure 7 shows the average attention distances for the three different pre-training methods. For each transformer layer/block (x‑axis), we plot the mean spatial attention distance of patches for each head (y‑axis), averaged over patches and point cloud samples.

How we compute it: For a query patch $i$ at layer $l$ and head $h$, let $\alpha_{ij}^{(l,h)}$ be its attention weight to patch $j$. Let $c_i, c_j$ be the 3D centroids of the corresponding point‑patches. The expected attention distance for that query is:

$d_i^{(l,h)} = \sum_{j} \alpha_{ij}^{(l,h)} \|c_i - c_j\|_2.$

We then average $d_i^{(l,h)}$ over all queries $i$, and samples. Since we have 6 different heads with the standard model (ViT-S), there are 6 averages for each layer.

How to read it: Lower values correspond to a head specializing to local contexts, whereas high values correspond to a head specializing to global contexts. When all heads have roughly the same value, there is no such specialization between heads.

Key takeaway: The attention‑specialization patterns for our method are relatively consistent across objectives: early layers show more head specialization, while later layers focus more on global context with less head‑to‑head variation. This contrasts sharply with the attention pattern from reconstruction‑based masked modeling (MAE), where the last layers exhibit strong specialization across different spatial contexts. To make this difference clearer, we additionally trained our model with MAE. Its inverted pattern confirms this distinction (we will include this visualization in the final version).

Minor corrections

We thank the reviewer for pointing out the minor typos. These have been corrected.