Thank you for the insightful feedback, we would like to answer your questions as follows:

Why tokens in point cloud transformers are highly redundant?

Thank you for the question, it touches on a key motivation behind our work and points to an important direction for future research. Our intuition regarding the high token redundancy in point cloud transformers stems from the inherent properties of 3D data similar to how single pixels in a 2D image are not very informative on their own. Unlike in Vision Transformers, where semantic information is often captured in structured patches (e.g., 16×16 pixels), 3D point clouds are unordered, spatially noisy, and frequently incomplete. Individual points in 3D space typically carry very limited information, and many of them correspond to repetitive structures within larger objects, such as flat surfaces or repeated geometric patterns, which further amplifies redundancy.

This perspective is further supported by the design of state-of-the-art methods like Point Transformer v3 and Sonata, which rely on fine-grained attention mechanisms to model local context. In both the first and last layers of these transformer backbones, point tokens tend to exhibit high redundancy, a phenomenon we visualize using PCA projections. At these stages, individual tokens contribute minimal unique information, making self-attention computationally expensive and inefficient. Nevertheless, it's important to retain a subset of these tokens, as some may belong to semantically rich or structurally complex objects that become meaningful only when captured by the broader receptive fields of deeper layers. Identifying which tokens to preserve is non-trivial, and this challenge motivates our proposed globally-informed graph representation, which guides token selection based on global context.

In future work, we plan to explore this redundancy more formally by applying the transfer entropy framework to analyze and justify the globally-informed graph representation introduced in our method. We believe this will offer both theoretical insights and practical improvements for designing more efficient transformer architectures in the 3D domain.

Aggressive Merging Method versus ToMe

• Findings: ToMe and other prior works on Vision Transformers primarily focus on 2D images, where each token typically covers a relatively large and semantically meaningful region of the image. This makes individual tokens highly informative, which in turn limits the degree of safe token merging, usually only a small fraction of tokens are merged progressively at each layer to avoid significant information loss. In contrast, our study highlights a key difference in the 3D domain: transformer backbones such as PTv2, PTv3, and Sonata frequently encounter substantial token redundancy. This might stems from the structural repetition inherent in 3D scenes. As a result, we observe that the attention mechanism can become overused on redundant or uninformative tokens, particularly in the early and late layers where token dimensionality is shallow. We believe these findings will open up new directions for designing more efficient transformer architectures tailored to the unique characteristics of 3D data.

• Method: Leverage our finding, we propose a token compression strategy that aggressively reduces redundancy by identifying and merging tokens in low-energy regions of the 3D scene. Our method employs a globally-informed energy scoring mechanism to quantify and detect redundant regions. By merging up to 99% of the tokens, while preserving the most informative ones, we demonstrate that a well-design token compression method can significantly reduces computational cost with negligible impact on accuracy.

Ablation study for the number of bins ( $K$ )

The central idea of spatial-preserving token merging is to ensure that destination tokens are evenly distributed throughout the input space, which helps maintain both semantic and positional information after merging. In our approach, we aggressively merge all source tokens into destination tokens when the merge rate $r$ exceeds 50%, making it crucial for these destination tokens to be well-dispersed. Without this binning strategy, multiple tokens could collapse into the same destination token, leading to significant loss of spatial information. By requiring all source tokens within a bin to merge into its corresponding destination token, we promote a more uniform spatial distribution of merged tokens. To validate this point, we conducted an ablation study on the ScanNet dataset.

These $K$ parameters however is not yet optimal, in our paper we define $k$ as: $\lfloor T * (1 - r)\rfloor$ , where $T$ is the number of token in each patch.

Comparision with RandLA-Net

• A key difference between RandLA-Net and our method lies in their design philosophy and integration. Our method is an off-the-shelf technique aimed at reducing redundant information and computational overhead inherent in large-scale point transformer architectures such as PTv3 and the foundation model Sonata. In contrast, RandLA-Net is an independent architecture that must be trained from scratch.
• Our framework also includes an analytical tool that identifies redundant regions in the point cloud using a proposed globally-informed graph structure, enabling more effective compression. Before comparing our method directly with RandLA-Net, it is instructive to contrast the underlying architectures of PTv3/Sonata and RandLA-Net. While both PTv3 and RandLA-Net are designed for large-scale point clouds (on the order of millions of points), they adopt fundamentally different strategies. RandLA-Net employs random sampling and local feature aggregation based on KNN, which can become computationally expensive and less scalable. PTv3, however, demonstrated that the KNN-based aggregation used in RandLA-Net imposes a computational bottleneck and limited receptive field. To address this, PTv3 introduced a serialization scheme that orders and groups points sequentially, replacing the costly KNN approach.
• Our method builds upon PTv3's serialization strategy by dividing space into smaller scales and selecting a representative set of DST tokens, rather than sampling points randomly as RandLA-Net does. This preserves spatial structure more effectively and avoids the clustering issues inherent to random sampling. Moreover, our method performs aggregation using DST tokens at a more global scale (e.g., 1000 points), as opposed to RandLA-Net's local KNN-based selection, further enhancing efficiency and representation power.

Beyond PTv3 and Sonata

PTv3 and Sonata are the first large-scale, scene-level point cloud transformers capable of performing self-attention across thousands of points simultaneously. This design allows them to capture fine-grained and important details during training. However, it also leads to significant redundancy in the data, an issue our method aims to address.

In contrast, other techniques mitigate computational cost by partitioning the point cloud before processing, either through spatial grouping (e.g., RandLA-Net, Yogo [1], SuperPoint [2]) or via 1D serialization with localized self-attention (e.g., OctFormer [3], which attends to up to 128 points). The grouping-based methods typically exhibit lower redundancy due to their coarse representation and small number of groups, meaning that pruning one or two groups has limited impact on efficiency. Meanwhile, in serialization-based approaches, the self-attention layer is not the main bottleneck, the point cloud itself is. Thus, a different strategy is needed, such as completely or temporarily pruning redundant points.

Given the demonstrated strength of PTv3 and Sonata as state-of-the-art 3D architectures and foundational models for 3D pretraining, we anticipate that future architectures will build upon them. We believe integrating our redundancy optimization method into the training process of such models is a promising direction, particularly as our work so far has focused primarily on improving inference efficiency.

[1] You Only Group Once: Efficient Point-Cloud Processing with Token Representation and Relation Inference Module

[2] Efficient 3D Semantic Segmentation with Superpoint Transformer

[3] Octformer: Octree-based transformers for 3d point clouds

6. Comparing against merging as ToMe designed for 3D point cloud

We conduct an extensive empirical study involving a range of existing token reduction and merging strategies, including adaptations of principal mechanisms used in ToMe (Token Merging) and related approaches. While ToMe has shown success in 2D vision tasks, we found no existing implementation or variant tailored for 3D point cloud transformers. This is likely because, to the best of our knowledge, our work is the first to reveal and systematically study token redundancy in large-scale 3D point cloud architectures.

To enable a fair and meaningful comparison, we adapted ToMe’s core idea of merging based on feature similarity to the 3D domain and evaluated it alongside our own method, which is specifically designed to preserve spatial structures unique to point clouds. All methods were rigorously tested across multiple strong transformer architectures (PTv3 and Sonata) and diverse downstream tasks, including indoor and outdoor semantic segmentation, and point cloud reconstruction.

This broad and thorough evaluation not only highlights the robustness and generality of our findings, but also emphasizes the distinct advantage of our approach, which accounts for the spatial and structural complexities of 3D data—something generic 2D-based techniques like ToMe are not equipped to handle out of the box.

In summary, we see this work as a first step toward a larger research direction. By identifying and demonstrating token redundancy in 3D transformers, we hope to inspire the community to further explore and improve token merging mechanisms tailored for 3D data. Our method provides an initial, effective solutions, yet we believe more advanced and higher-performing approaches will naturally follow as this research direction gains traction.

In which we applied a default merge rate of 0.6 and an aggressive merge rate of 0.9 for the experiments. We did not have enough time to complete our analysis for the GFLOPs in these experiments. However, we reported a significant drop of memory consumption from 3.44GB to 1.45GB while only having 1.2% drop in total AP score. This suggests that we would likely observe a huge drop rate with GFLOPs. Fine-tuning experiment on this dataset is also promising.

[1] Yang, Yunhan, et al. "Sampart3d: Segment any part in 3d objects." arXiv preprint arXiv:2411.07184 (2024).

4. Clarify the essential differences between the proposed method vs (Ptv-3 and RandLA-Net).

We thank the Reviewer for pointing out a potential confusion with RandLA-Net, and we appreciate the opportunity to clarify this.

While RandLA-Net is a standalone network architecture designed for point cloud processing (notably following a U-Net style structure), our method is a token merging strategy designed to be integrated into existing point transformer architectures (e.g., PTv3, Sonata) to reduce redundancy in a principled and efficient manner.

A. Key Differences: RandLA-Net uses:

• Random sampling for point cloud downsampling, which may result in loss of spatial structure and does not consider feature-level redundancy.

• Local feature aggregation (LFA) modules using KNN (K=16) with small receptive fields, limiting global context awareness.

• Nearest neighbor interpolation during decoding, which can introduce approximation errors in dense reconstructions.

In contrast, our method:

• Is not a downsampling or point reduction technique, but rather a token merging mechanism guided by spatial and feature similarity.

• Performs bin-wise selection of representative tokens and computes token redundancy across multiple self-attention heads, preserving spatial fidelity and feature diversity.

• Utilizes a large receptive field (≥1000 points) for redundancy detection and performs recovery via head-wise recomposition, not interpolation.

B. Additional Comparison with RandLA-Net-Style Pipeline

To further validate the differences, we conducted an adapted RandLA-Net implementation within PTv3 for a direct comparison. In this baseline:

• We apply random sampling, followed by KNN feature pooling (K=16),

• Run self-attention on the pooled set,

• And use nearest neighbor interpolation to recover the original token positions, mimicking the RandLA decoding.

As shown in the table below (see attached figure), our method outperforms this baseline consistently across multiple reduction ratios (r). This highlights the advantages of our approach in terms of spatial structure preservation, head-wise processing, and feature-guided compression.

In summary, we emphasize that:

• RandLA-Net is a specific architecture with a fixed sampling and aggregation mechanism;

• Our method is a general, plug-and-play token merging strategy applicable across architectures;

• The superiority of our method is empirically demonstrated through a carefully controlled comparative experiment.

We hope this clears up any confusion and reinforces the novelty and generality of our proposed approach.

5. Applying in another framework outside PTV3/Sonata

We thank the Reviewer for the insightful suggestion. As highlighted in our contributions, our primary objective was to uncover and address token redundancy in large-scale point cloud transformer architectures, specifically in PTv3 and Sonata—the most recent and relevant state-of-the-art models designed for dense point cloud representations with large token counts (≥1000 tokens) and wide receptive fields.

Our method is particularly well-suited to such settings, where redundancy becomes computationally costly. In contrast, earlier architectures often operate with significantly fewer tokens, where redundancy is minimal and the gains from merging would be negligible. This explains our focus on PTv3 and Sonata, where the impact of our method is measurable and meaningful.

We agree that integrating our method into other architectures is a valuable direction. For instance, RandLA-Net could theoretically adopt our merging approach after each Local Spatial Encoding layer. However, since it relies on local aggregation with small receptive fields and relative positional encodings, merging may disrupt its core operation, requiring non-trivial adaptations.

In summary, while our current focus is on dense models, our method remains architecture-agnostic and well-suited for broader adoption as future architectures move toward larger receptive fields and denser inputs.

2. Theoretical Directions to Analyze Token Redundancy in 3D Point Cloud Transformers

While our paper only provides some intuition as a previous response, we also present here a few more concrete directions toward a feasible theoretical foundation. In particular, one promising idea is to construct a token interaction graph where each node represents a token (e.g., a voxel or point embedding), with edges weighted by either self-attention affinities or spatial proximity. This graph can serve as a structured representation for deeper theoretical study.

a. Spectral Graph Coarsening as a Framework for Token Merging Evaluation

Inspired by PiToMe [1], which employs spectral graph theory to preserve intrinsic spectral properties when merging tokens, one can adapt this framework to 3D point cloud transformers. Specifically, consider using the token connectivity graph and perform graph coarsening by merging nodes with similar local neighborhoods while ensuring that the spectrum (i.e., eigenvalues of the normalized Laplacian) between the original and the coarsened graph remains close.

b. Topological Redundancy via Persistent Homology

We can then further analyze which tokens are topologically essential by using persistent homology to extract features like connected components, loops, and cavities from the point cloud or the token graph. Then, test whether removing certain tokens alters the topological signatures significantly. Tokens whose removal leaves those signatures mostly unchanged can be considered topologically redundant, a theoretical grounding for merging them without harming downstream task performance.

c. Unified Spectral–Topological Framework

By merging the above ideas, a joint framework can be:

• Graph Spectral Stability: Use spectral distance metrics to quantitatively gauge how similar a coarsened (merged) token graph is to the original in terms of eigenvalue distributions. A small spectral distance indicates that the global relational structure is maintained.

• Topological Invariance: At the same time, one can measure how topological invariants, such as Betti numbers [2,3], which capture essential geometric features like connected components and holes, change when tokens are removed or merged. If these invariants remain stable, it suggests that the overall structure of the point cloud is preserved despite the reduction in tokens.

Such an approach has the potential to deliver a rigorous theoretical basis showing that merging redundant tokens can substantially reduce computational cost, while safeguarding both structural and topological integrity of the data representation.

[1] Accelerating Transformers with Spectrum-Preserving Token Merging, NeurIPS 2024

[2] Wang, Hao. "Betti Number for Point Sets." Journal of Physics: Conference Series. Vol. 2555. No. 1. IOP Publishing, 2023.

[3] Paul, Rahul, and Stephan Chalup. "Estimating Betti numbers using deep learning." 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.

3. Unclear if the proposed method remains robust to outdoor scenes

We thank the Reviewer for the thoughtful and detailed feedback. We would like to clarify that results for outdoor scenes were already included in the supplementary material; however, we apologize if this was not sufficiently highlighted or easy to locate. To address this, we provide a concise summary of our findings for this setting below:

• We ran experiments with LiDAR point clouds on off-the-shelf results for the nuScenes dataset [2], presented in Table 7 of the supplementary material. Specifically, we report results using a standard merge rate of 80% and an aggressive merge rate of 95% (K=32). A summary is provided below:

The results suggest that our binning strategy generalizes well to outdoor point clouds with highly variable local point densities.

Additionally, our method also works with object-centric datasets, including

• For the 3D reconstruction task with ObjectVerse and Google Scanned Object datasets, please see Sec. 5.3, Fig. 6, and Fig. 7 in our main manuscript.
• For the 3D part segmentation with SAMPart3D[1] model (conducted during rebuttal).

We thank the Reviewer for the follow-up concerns and the opportunity to clarify our contributions. We would like to emphasize the following key points. Below we address your remaining concerns.

1. Requiring Formal Theoretical Results

We would like to emphasize the following key points:

a. Novel Empirical Discovery for Efficient 3D Point Cloud Transformer:

Our paper is the first to systematically uncover and quantify token redundancy in large-scale 3D point cloud transformer architectures, including PTv3 and the foundation model Sonata. This insight is crucial for the community, as it exposes a significant inefficiency in current state-of-the-art models, models that are increasingly used despite their heavy computational and energy requirements. We believe that findings will draw the community’s attention to real-world deployment of these architectures, which incur substantial and avoidable costs, particularly in resource-constrained environments.

b. Comprehensive Experimental Validation:

We conduct an extensive empirical study involving a range of existing token reduction and merging techniques, alongside our own novel method that captures spatial characteristics specific to 3D point clouds. These methods are evaluated across multiple architectures (PTv3, Sonata) and diverse tasks, including indoor and outdoor segmentation, and point cloud reconstruction. This broad validation underscores the robustness and generalizability of our findings.

c. Impactful Contributions and Future Directions:

While we appreciate the Reviewer’s interest in theoretical analysis, we respectfully argue that the strength of our paper lies in its empirical rigor and practical insight. Our contributions lay a foundation for a number of impactful future research directions, including:

• Investigating whether token redundancy is prevalent in other non-transformer 3D point cloud architectures.

• Designing new transformer variants specifically optimized for token efficiency in 3D vision tasks.

• Developing task-specific token merging or compression strategies for further performance gains.

• Theoretical studies to understand the underlying reasons behind token redundancy in 3D models.

We believe that requiring a formal theoretical proof - while valuable - should not be viewed as a prerequisite for validating the novelty or significance of our contributions. Rather, we see theoretical understanding as a natural and important future step, which our work motivates and opens the door for.

Transfer Entropy Analysis for Token Merging

Following [1, 2], entropy can be used to measure the information quantity of a network:

Since directly measuring the probability distribution of tokens is non-trivial, we approximate it with a Gaussian distribution [2]:

The entropy of a feature set is then:

Transfer Entropy (TE) Definition

We define Transfer Entropy as the change in information after applying a token merging function:

This quantifies the amount of information lost or altered due to merging.

The Transfer Entropy Rate is:

Experimental Setup

We analyze the ScanNet validation set with a default merging rate of 70%, reporting transfer entropy rates for:

• TE A: Original → Moderate merging (w/o Global Informed Graph)
• TE B: Original → Adaptive-aggressive merging (w/ Global Informed Graph)
• TE C: Moderate → Aggressive merging

Layer-wise Transfer Energy Rate Results

Observation: Transfer rates remain consistently small (< 0.1) across layers, indicating minimal information loss from compression via merging functions. This aligns with our segmentation performance results and supports the hypothesis—based on a transfer entropy framework—that token representations in point transformer models contain significant redundancy.

While this does not constitute a complete theoretical proof, the transfer entropy framework provides a principled analytical assessment that offers clear evidence supporting our finding. We believe this is a good starting point for a more rigourous theoretical reasoning.

[1] Guan, Chaoyu, et al. "Towards a deep and unified understanding of deep neural models in nlp." International conference on machine learning. PMLR, 2019. [2] Lin, Sihao, et al. "Mlp can be a good transformer learner." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Thank you for the insightful feedback. We will incorporate both your concerns and the discussion into our manuscript.

Other Model

Theoretically, our technique can be applied to other point transformer models such as OctFormer[2] or FlatFormer[1]. However, PTv3 and Sonata are the first transformer-based models to operate on point clouds at a fine-grained level, processing self-attention across more than 1000 points simultaneously. This scalability is particularly beneficial for large-scale scenes, where millions of points may exist with uneven density and varied semantic importance (e.g., a few points may hold crucial information while many do not). This makes point-level self-attention crucial.

In contrast, other models group points first (e.g., PointNet, PointNet++), then apply attention with a limited number of points (up to 128 for OctFormer and FlatFormer), limiting their capability. This also reduces attention overhead, but at the cost of representation power.

Notably, Sonata is the first pretrained foundation model at the scene level, with a large receptive field for self-attention. It has also been used to power 3D-language models such as SpatialLM[3].

Simplicity

A key insight in our work is that point transformer models overuse tokens, leading to excessive computation and memory usage. Our globally-informed graph provides additional context to the local transformer, helping determine which areas contain more informative tokens.

The simplicity of our method ensures robustness and can be used to for future design decisions in efficient 3D point cloud processing.

Adaptation of Other Methods

To extend the evaluation scope for other baselines, such as ALGM, we tested their performance across different threshold settings. For PiToMe, applying merging rates above 50% is theoretically infeasible due to its formulation, which requires maintaining a set of preserved tokens. When the merging ratio exceeds 50%, all tokens become candidates for merging, resulting in the concept of a "preserved set" invalid. We explored adapting their energy-based formulation to determine a set of protected tokens, but found that this approach yielded worse results compared to simply allowing all tokens to be considered for merging when the merging ratio surpasses 50%. Here, we present our analysis of adapting PiToMe to our patchified merging technique, evaluated on the Scannet validation set. By fixing a percentage of tokens to remain unmerged (selected using PiToMe's energy score), we merged the remaining tokens, ensuring that the total number of retained tokens (merged and protected) was 20% of the original token count.

Based on the experimental results, our method, by distributing DST sets more evenly across the point cloud to cover a broader range of areas, is more effective than the sparse distribution of DST sets used in PiToMe to maintain a fixed protected set.

S3DIS Score Clarification

We mistakenly swapped the fine-tuned and off-the-shelf evaluation scores. The corrected results are:

• Fine-tuned: 74.3
• Off-the-shelf: 72.3

[1]Liu, Zhijian, et al. "Flatformer: Flattened window attention for efficient point cloud transformer." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. [2]Wang, Peng-Shuai. "Octformer: Octree-based transformers for 3d point clouds." ACM Transactions on Graphics (TOG) 42.4 (2023): 1-11. [3]Mao, Yongsen, et al. "SpatialLM: Training Large Language Models for Structured Indoor Modeling." arXiv preprint arXiv:2506.07491 (2025).

Lacking Learnability

We agree that while our proposed globally-informed graph provides a way to determine which areas contain more redundant information (and can thus be processed with more aggressive merging), automatically determining the merging rate for each region is indeed a promising next step in our research direction. We view this as a natural extension to make our method more adaptive and fully learnable.

Missing Comparison

We appreciate the suggestion to compare with other recent token pruning and downsampling strategies. Below, we discuss relevant methods:

• YOGO
  YOGO performs cross-attention between regions and original points to enable a wider receptive field. However, it relies on furthest point sampling, which is computationally expensive for large scenes with many points and requires many point groups. Additionally, it requires training from scratch. As an attempt to compare, we adopt their grouping strategy in our ablations.

• TokenHalting
  This method determines which token to prune at each layer using a learnable criterion. At the final detection head, pruned features are restored for full processing. However, this approach is primarily suitable for architectures without explicit spatial downsampling. It is less applicable to our architecture, which benefits from spatial relationships.

• SuperPoint
  This method proposes a new architecture that gradually groups points into super points. Self-attention is then performed on aggregated features within each group. While efficient, it lacks fine-grained point-level processing and depends on hand-crafted features. It also requires training from scratch.

Our method instead targets redundancy within general-purpose architectures such as UNet and Point Transformer (PTv3, Sonata). These architectures already demonstrate strong performance across both indoor (ScanNet) and outdoor (LiDAR) point cloud benchmarks. Our token merging strategy can be applied off-the-shelf, improving efficiency without retraining, which contrasts with the above methods that require full retraining or architectural changes.

We also tried to adopt YoGo by directly integrating their downsampling methods into the PTv3 model and performing off-the-shelf inference on the ScanNet dataset. we simulated the cross-attention mechanism in two steps: (1) pooling $n$ Key and Value ($K, V$) tokens into $k$ centroids using a kernel size of 64 and a stride of 32 to mimic self-attention across regions, and (2) performing self-attention between the $n$ Query ($Q$) tokens and the $k$ pooled $K, V$ tokens, thus simulating cross-attention between the original points and the regions.

Experiment with LiDAR Point Cloud and Object Point Cloud

For experiments with LiDAR point clouds, we include results on nuScenes [2] in Tab. 7 of the supplementary material. We report the relevant summary here:

We also evaluate our method with a recent object point cloud segmentation model, SAMPart3D [1], on their provided dataset:

We applied a default merge rate of 0.6 and an aggressive merge rate of 0.9 for the experiments. We did not have enough time to complete our analysis for the GFLOPs in this experiments. However, we reported a significant drop of memory consumption from 3.44GB to 1.45GB while only have 2% drop in total AP score. This suggest that we would likely observe a huge drop rate with GFLOPs. Fine-tunning experiment on this dataset is also promising.

Additionally, we assess our method on the SpatialLM dataset for object detection using the SpatialLM 3D language model:

The metrics represent the F1 score of object detection at IoU thresholds of 25 and 50. In the language and 3D-grounded domains, our results show that the tokens in the point cloud transformer remain highly redundant. Our method therefore can significantly boost the computational performance in term of inference speed and peak GPU memory consumption.

We hope these comparisons and extensions clarify the generality and applicability of our method across different domains and architectures.

[1] Mao, Yongsen, et al. "SpatialLM: Training Large Language Models for Structured Indoor Modeling." arXiv preprint arXiv:2506.07491 (2025).

I appreciate the authors' rebuttal, which was convincing. I will raise my score.

Thank you for your insightful feedback. We will address both of your concerns, including providing additional justification for the choice of token merging and clarifying any confusion regarding the merging algorithm, and incorporate these changes into our manuscript.## Method Justification: Token Merging Placement

Before finalizing our approach, we analyzed the bottleneck in the point cloud transformer model and found that while the attention layer is the primary bottleneck, it is also redundant. In this study, we replicate the analysis of different token merging strategies at a 75% merge rate. For this experiment, we adopt a fixed merging rate rather than a weighted scheme. When merging an MLP layer, we use the output features of the self-attention layer before it as the merging criterion. We evaluated three configurations: merging only the MLP layers (MLP), merging only the attention layers (Attention), and merging both the MLP and attention layers (Attention + MLP). We use Scannet validation set for this experiment.

The bottleneck is clearly located in the Attention layer, suggesting that merging at this stage significantly impacts performance. In contrast, the MLP layers, which preserve more detailed information, are critical for maintaining accuracy. Interestingly, increasing computational effort for token merging within the MLP layers led to higher overall computational costs. Based on this analysis, we chose to focus on merging within the Attention layers.

Conflicting Reasoning

We acknowledge a mistake in the original writing (Line 196). We propose the revised clarification for Lines 196–197:

"Tokens with lower energy (i.e., more aligned with global structure) are considered less informative and can be merged more aggressively, whereas high-energy tokens (i.e., less aligned with global structure) are preserved to retain critical information."

Writing Improvements

We appreciate the feedback and will revise the manuscript to enhance clarity. Specific improvements include:

• Reorganizing the order and flow of supplementary material.
• Correcting figure captions and resolving typographical issues (e.g., repeated "the").

Merge Rate Selection

We empirically determined a fixed merging rate of 32 using the validation set of ScanNet and applied it across all datasets. Although this fixed rate yields strong performance, we acknowledge the potential benefits of automatically determining adaptive merge rates per patch—a promising future direction.

Qualitative Comparison: Global vs. Local Energy Score

Thank you for your suggestion. We will include a qualitative comparison between global and local energy scores in our revised version. Below are the comparison results on the ScanNet dataset (without augmentation):

Thank you for your valuable feedback and suggestions. We provide detailed clarifications below.

ScanNet Results

The evaluations for the validation sets of both ScanNet and ScanNet200 are included in Tab. 1 of the manuscript. The results are illustrated in the first and second columns, respectively.

Latency Comparison

We analyze latency on the validation scenes of ScanNet and compare various grouping strategies. For each method, we adjust its parameters so that the number of points retained is around 20% of the original point cloud. The results are summarized below:

We introduce several additional grouping techniques for token reduction (for more information please see Sec. B and Tab. 5 in the supplementary material):

• Progressive-Tome
  We design an $O(N)$ token merging method by constructing a local graph among tokens, where edges are formed between adjacent tokens in a 1D serialized order. In our experiments, we merge the top 80% of edges with the highest similarity by averaging their corresponding tokens. The unmerging process follows the same approach as our main method.

• PoolTome
  To enhance spatial communication, we first apply average pooling with a large kernel size and stride, both set to $\log_2(N)$, where $N$ is the number of tokens. Attention is then computed over the pooled features. Finally, unpooling is performed by duplicating the attention outputs back to their corresponding original token positions. The overall attention computation cost is $O(\log_2(N))$. This method can be considered a token merging strategy that relies purely on locality.

While point cloud downsampling techniques such as Random Drop and Voxel Downsampling offer computational efficiency, they are often challenging to apply in off-the-shelf scenarios, as they typically overlook both point-wise feature representations and the latent space of segmentation models. In contrast, our method identifies redundancy using a globally informed graph structure, enabling effective merging and unmerging of points while preserving the features essential for accurate model performance.

The author's rebuttal addressed my concerns regarding accuracy and efficiency. The additional discussion on PTv3 and Sonata is also appreciated. Therefore, I maintain my recommendation as weakly accept. Overall, I support acceptance primarily because I believe the proposed direction has its value and merits further attention from the point cloud community. While the manuscript may not present substantial technical novelty, it introduces a simple and effective baseline that could stimulate progress in this area.

PS: Regarding the discussion of Sonata (self-supervised pre-training) from the author’s response to other reviewers: I would like to share my perspective. The purpose of large-scale pre-training is not necessarily to improve the efficiency of pre-training itself, but rather to enhance the efficiency of downstream fine-tuning—regardless of data or computational resources used during pre-training (as seen in methods like DINO or CLIP). Thus, the heavy use of resources in pre-training should not be viewed as an inherent drawback. That said, efficiency in pre-training and efficiency in fine-tuning are not mutually exclusive. In fact, it would strengthen the manuscript if the authors could demonstrate that fine-tuning from well-pretrained weights (e.g., Sonata) combined with their proposed method leads to improved efficiency—perhaps using the ScanNet data efficiency benchmark. I hope the authors consider this as an optional suggestion for revision.

Thank you for your interesting suggestion. Combining our technique with well-pretrained weights (e.g., Sonata) for downstream task training is a promising and impactful direction. Despite the limited time available for the rebuttal, we have attempted to conduct a preliminary experiment to explore this.

In this experiment, we fine-tuned Sonata on the ScanNet dataset using our method with a 70% merging rate. This setup differs from the experiments in the main paper in the following ways:

• Main paper: Fine-tuned from a checkpoint where the downstream task had already been trained from pretrained Sonata (Self-pretrained Sonata → Downstream task training → Fine-tuning with our method).

• Rebuttal experiment: Trained the downstream task directly from pretrained Sonata (Self-pretrained Sonata → Downstream task training combined with our method).

Results Summary

1. Linear Probing

In this setting, the encoder was frozen, and we trained a smaller decoder (a linear probing).

• Setup: batch size = 12, 1 GPU: H100
• Row 1: Results from the original paper.
• Row 2: Our attempt at training the downstream task with token merging (merge 70% tokens).

2. Decoder Fine-tuning

In this setting, both Encoder and Decoder are updated, we trained this setting using 75% number of epochs due to time constraints.

• Setup: Fine-tune Sonata with a lightweight decoder (batch size = 16, H100 GPUs).
• Row 1: Results from the original paper.
• Row 2: Our results using token merging (merge 70% tokens).

Conclusion

In both setting the performance remains nearly identical, while GPU memory usage and training time drops significantly. This demonstrates that our technique can enable efficient downstream training, not just fine-tuning pre-trained downstream task models.

While this experiment is limited in scope due to time constraion (one dataset and one merge rate ), it provides initial evidence that Sonata with 70% token merging preserves features sufficiently for downstream training. This also suggests promising results for efficient downstream task training of large models (e.g., Sonata with a large decoder and full fine-tuning). We plan to extend this experiment to larger-scale settings and will report results in the revision.