PointMamba: A Simple State Space Model for Point Cloud Analysis

• To question 4: “…several Mamba-based methods after this paper…”

Reply: Good suggestion! Indeed, there are several papers after this paper, we list the comprehensive comparisons as follows:

PCM [1] combines Vision Mamba with PointMLP and incorporates consistent traverse serialization at each stage. To enhance Mamba’s capability in managing point sequences with varying orders, PCM introduces point prompts that convey the sequence’s arrangement rules.

Point Mamba [2] uses an octree-based ordering scheme and combines Mamba with PCT and OctFormer as their baseline. Mamba blocks with bi-directional scanning extract hierarchical point features, and a Feature Pyramid Network (FPN) is utilized for classification or segmentation tasks.

Mamba3D [3] introduces an enhanced Vision Mamba block, which includes both a token forward SSM and a backward SSM that operates on the feature channel. It proposes a Local Norm Pooling block to extract local geometric features.

PointTramba [4] introduces a hybrid approach that integrates Transformers and Mamba. It segments point clouds into groups and utilizes Transformers to capture intra-group dependencies, while Mamba models inter-group relationships using a bi-directional, importance-aware ordering strategy.

Note that some methods (e.g., [4]) are clearly based on our code or baseline, which proves the value of our approach to the community. We will add these discussions in the revised version.

[1] Point could mamba: Point cloud learning via state space model

[2] Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy

[3] Mamba3d: Enhancing local features for 3d point cloud analysis via state space model

[4] PoinTramba: A Hybrid Transformer-Mamba Framework for Point Cloud Analysis

• To weakness 1: “(No need for experiments)…encourage the inclusion of indoor segmentation and detection tasks…”

Reply: Thanks for this very constructive suggestion, and we will explore a unified Mamba-based foundation model for various 3D vision tasks (including indoor segmentation, detection tasks, and classification tasks on Objaverse-LVIS ) in the future. We also appreciate the reviewer's understanding that finishing such experiments in a limited time is difficult.

• To weakness 2: “…Point-BERT…genuinely interested in exploring novel discoveries in self-supervised learning…”

Reply: We totally agree with this insightful point. Actually, this paper also explores a new self-supervised pre-training paradigm named serialization-based mask modeling (Sec. 4.2). Specifically, different previous methods (e.g., Point-BERT) using random masking, we consider the unidirectional modeling of Mamba and propose to randomly choose one space-filling curve to generate the serialized point tokens for mask modeling. In addition, the proposed extremely simple order indicator effectively maintains the distinct characteristics of these different scanning strategies. As a starting point for this area, we hope this paper can provide useful insight for the community and encourage researchers to focus on the potential of Mamba in 3D vision tasks.

• To question 1: “… consistent stability across all of these tasks…providing mean and standard deviation…”

Reply: Thanks! We promise that all code will be released and the results can be easily reproduced. Following your suggestion, we report the mean and std in all datasets under three runs, as shown below. We can see that the proposed method achieves consistent stability across different tasks/datasets.

• To question 2. “… a larger number of points…training time…possible solutions…real-world applications…”

Reply: Thanks! Following your suggestion, we evaluate the training time (second/epoch) with different input lengths (from 128 to 2048) with a batch size of 16, as shown below. We find that with point tokens increasing, the advantages of our approach over Point-MAE will be even more pronounced, further indicating the efficiency of our PointMamba. In addition, to apply the PointMamba to Auto-Driving, a possible way is to design an efficient voxel-based tokenizer that can effectively transform the larger number of point clouds into a series of long point/voxel tokens, which is a promising future work.

• To question 3: “…improved performance compared to…the initial version? …borrow from Point Transformer…the main contribution lies in integrating Mamba into Point-MAE”

Reply: Thanks! In fact, the improved performance compared to the initial version is not simply replacing the reordering strategy but rather the comprehensive new proposed technology. Specifically, compared with the initial version, we make the following new contribution: 1) The proposed point scanning strategy transforms the unstructured 3D point clouds into a regular sequence, which can provide diverse perspectives on spatial locality via scans from different clues; 2) The order indicator maintains the distinct spatial characteristics of different scanning when model training, preserving the integrity of the spatial information, which is crucial for the unstructured point clouds ; 3) Serialization-based mask modeling randomly choose a space scanning curves for masking, allowing the model exact the general local relationships from different scanning clue, better matching the requirement of unidirectional modeling of Mamba.

Furthermore, we would like to clarify that our method are totally different from Point-MAE: 1) For processing steps, Point-MAE directly generates the random tokens via a lightweight PointNet. In contrast, we propose a point scanning strategy, which leverages the space-filling curves to transform the unstructured point clouds into a regular sequence. Although Point Transformer also adopts space-filling curves, the motivation and objective are different. Point Transformer utilizes space-filling curves to partition the point cloud,  capturing spatial contexts. In contrast, our work mainly focuses on transferring the point clouds to serialization-based sequences and combines it with Mamba to implement global modeling. 2) For feature extraction, Point-MAE employs a vanilla Transformer. We merge the vanilla Mamba with the proposed order indicator, making the Mamba preserve the integrity of the spatial information from different scanning curves;  3) For pre-training, the PointMAE is the traditional random mask modeling paradigm, while our PointMamba proposes serialization-based mask modeling, which employ serialization masking modeling, combining with the proposed order indicator; 4) We also provide detailed theoretical analyses of why PointMamba work well in point cloud tasks, while it seems lack on PointMAE.

Overall, the current version makes substantive improvements compared with Point-MAE and our initial version. Extensive experiments also validate the effectiveness of our method (See Fig.1 of the manuscript).

For question 4, please see the reply presented in the “comment” part.

• To weakness 1&2: “… may not take full advantage of the unique characteristics of point cloud data…/…may not be applicable to point cloud…are largely similar to…Point-MAE.”

Reply: Thanks. We respect the reviewer's opinion. However, we believe this paper considers the characteristics of point clouds. Specifically, through a pilot of experiments, we first demonstrate that simply replacing the Transformer with a Mamba can not achieve ideal performance due to the unidirectional modeling. Based on this, we customize the key innovative designs to handle the point cloud data: 1) The proposed point scanning strategy transforms the unstructured 3D point clouds into a regular sequence, which can provide diverse perspectives on spatial locality via scans from different clues; 2) The order indicator maintains the distinct spatial characteristics of different scanning when training, preserving the integrity of the spatial information, which is crucial for the unstructured point cloud; 3) Serialization-based mask modeling randomly choose a space scanning curves for masking, allowing the model exact the general local relationships from different scanning perspectives, better matching the requirement of indirection modeling of mamba. Thanks to these proposed simple yet innovative key designs, our PointMamba works well and performs better than its Transformer-based counterparts in point cloud tasks. We will make these technical contributions more clearly in the revised version.

Besides, compared with the Point-MAE, the differences are also distinct: 1) For processing steps, Point-MAE directly generates the random tokens via a lightweight PointNet. In contrast, we propose a point scanning strategy, which leverages the space-filling curves to transform the unstructured point clouds into a regular sequence; 2) For feature extraction, Point-MAE employs a vanilla Transformer. In contrast, we merge the vanilla Mamba with the proposed order indicator, making the mamba preserving the integrity of the spatial information from different scanning curves; 3) For pre-training, the Point-MAE is the traditional random mask modeling paradigm, while our PointMamba proposes serialization-based mask modeling, allowing the model exact the general local relationships from different scanning curves via the proposed order indicator; 4）We also provide detailed theoretical analyses of why PointMamba work well in point cloud tasks, while it seems lack on Point-MAE.

Overall, although our method presents a simple pipeline, the processing steps, feature extraction, pre-training, etc, are significantly different from the previous methods. As R2 said, “…the proposed method is simple, elegant, and effective, establishing a solid Mamba-based baseline for point cloud analysis tasks…”, we believe our method would be interesting and valuable for people in this area. We will deeply discuss the differences in the revised version.

• To weakness 3: “…trade-offs between model complexity and performance…the effect of partially using PointMamba becomes worse instead.”

Reply: Thanks. Please refer to Tab.1, Tab.2, and Tab.3 of the manuscript, where our method consistently performs better on all conducted datasets compared with the single-modal SOTA Transformer-based methods. Note that some methods (e.g., ACT) adopt multi-modal information (e.g., text description or 2D), which is unfair for comparison, but we still surpass it in most datasets.

• To weakness 4: “…have the most obvious impact on the experimental results…no additional description or setting in the paper.”

Reply: Thanks. We prove that directly replacing the transformer with mamba in Point-MAE can not achieve ideal performance due to the unidirectional modeling of mamba in Point Cloud. Therefore, we propose serialization operations, including point scanning strategy, order indicator, and serialization-based mask modeling. Therefore, serialization operations naturally reflect the performance gains, which verify the effectiveness of our proposed key designs.

Besides, we would like to clarify that Fig.4 is not the only proposed core. It just means the proposed serialization-based mask modeling paradigm and the comprehensive overall core designs are presented in Fig.2. Compared with the Transformer-based method (e.g., Point-MAE, PointGPT), we present substantive differences: point scanning strategy, order indicator, serialization-based mask modeling (please see the reply to weakness 1&2). Furthermore, the Hilber is a representative space-filling curve, which we adopt into our proposed key designs. The preliminary of Hilbert is listed in Sec. 3 of the manuscript, and we provide further descriptions in reply to the minor problem of Reviewer 2 (Please refer to it).

• To question 1: “…only works well in Hiber conditions?…”

Reply: Thanks. We have conducted experiments to analyze the effect of different scanning curves in our key designs, as shown in Tab.6 of the original manuscript. Specifically, we argue that using the space scanning curve sequences along a specific pattern of spatial locations offers a more logical sequence modeling order for SSM. The experiments also prove that it can achieve notable performance gains compared with the random paradigm. Among them, Hilbert achieves the best results due to their superior locality-preserving properties.

• To question 2: “…obvious innovative contribution…specialized Mamba technique for 3D point clouds?”

Reply: Good question! This paper is not just a replacement of the Mamba block (Equation 4) for the previous Transformer block, please see the reply to weakness 1&2.

• To Weakness 1: “Some experiments are missing…masking 90% point patches…the performance of max pooling?”

Reply: Good question! We conduct the mentioned missing experiments:

(1) As shown in the table below, masking 90% point patches may harm performance, as a higher masking ratio makes reconstruction tasks too difficult during pre-training.

(2) Average pooling takes into account the entire sequence with all the information, while max pooling selects the maximum value of each token in each dimension through a nonlinear process. We empirically find that only use max-pooling may destroy the global information of the extraction and cause performance drop, as shown in the table below.

• To Weakness 2: “… benefits of integrating PointMamba with Transformer architectures … a transformer layer can be used as the final prediction head …”

Reply: Good suggestion! Based on your advice, we kept the total number of blocks the same and replaced the last layer with a Transformer. As shown in the table below, we observed a confirmed performance drop. We argue that optimizing two modules simultaneously can be challenging, and developing an effective hybrid architecture is a promising topic for future research.

• To Weakness 3: “… introducing more orders …”

Reply: Thanks! Based on your suggestion, we conducted an experiment by adding an additional z-order sequence to triple the input length. As shown in Fig. 3 and Appendix A of our paper, using two orders enables global modeling for PointMamba; however, adding redundant information may lead to performance degradation.

• To Weakness 4: “…Figure 4 is somewhat ambiguous.”

Reply: We apologize for the ambiguity. During pre-training, different serialized point tokens have different order indicators. We will clarify Fig.4 in the next version.

• To Minor & Typos:

Reply: Thanks for the comments! Here is a brief introduction to Hilbert space-filling curves. Denoting the coordinates of voxelized points as $(x,y,z)$ and convert them into binary format using $log_2 n$ bits ( e.g., $x\to(x_{m}x_{m-1}...x_{0}),m=\lfloor log_2 n \rfloor$). $x_m,y_m,z_m$ is then iterated to $x_1,y_1,z_1$, making exchanges when the current bit is 0, and inversions otherwise. By concatenate the bits into $(x_{m}y_{m}z_{m}x_{m-1}y_{m-1}z_{m-1}\ldots x_{0}y_{0}z_{0})$ and apply a 3$m$-fold Gray decoding, the traversal position $T$ can be obtained. All voxels are then sorted into a single sequence based on their traversal position $T$. By recording the traversal position for all potential voxel coordinates, the points can be serialized. We will add this introduction and carefully revise our paper.

• To Weakness 1:

Reply: Thanks. We would like to emphasize that simply replacing the Transformer with the Mamba cannot achieve ideal performance due to the unidirectional modeling (as shown in below table). Thus, we customize the key designs to handle the point cloud data: 1) The proposed point scanning strategy transforms the unstructured 3D point clouds into a regular sequence, which can provide diverse perspectives on spatial locality via scans from different curves; 2) The order indicator maintains the distinct spatial characteristics of different scanning when training, preserving the integrity of the spatial information, which is crucial for the unstructured point cloud; 3) Serialization-based mask modeling randomly choose a space scanning curves for masking, allowing the model exact the general local relationships from different scanning perspectives, better matching the requirement of unidirectional modeling of Mamba.

Considering that this paper is the first work that discusses the potential of Mamba in point cloud tasks, it would bring new insights to the community and be interesting for many people. A similar toy example is the paper you mentioned, PointNeXt, an empirical study, that mainly studies the data augmentation in point cloud without specific new designs (90% of performance gains from data augmentation and hyper-parameter adjustment, while the remaining 10% comes from micro-designs such as residual connections), but many people still like such a simple yet effective paper. We hope the reviewer could reconsider the motivation and insights of our method (also see reply to weaknesses 2 & 3).

• To Weaknesses 2 & 3:

Reply: Thanks! It seems that the reviewer might be confusing different performance metrics on ShapeNetPart (Inst. mIoU, 86.2%, and 87.0% for PointMamba and PointNeXt, respectively, are correct). Note that, the performance of PointNeXt is somewhat misleading and shouldn’t be directly compared due to the following reasons: 1) PointNeXt adopts extensive data augmentation, while our method and other Transformer-based methods only adopt random rotation for ScanObjectNN and random scaling for ShapeNetPart; 2) PointNeXt adopts the voting strategy and manual refinement on this dataset (i.e., ShapeNetPart), which is an unfair post-processing that is usually resisted by the Transformer-based methods.

As such, no vanilla Transformer-based method can outperform PointNeXt when these unfair practices are applied. Therefore, we believe the reviewer should not deny a series of good Transformer-based works for these reasons. For fair comparisons, we remove all tricks and pre-training, as shown below, where we perform better than PointNeXt on most datasets.

Overall, PointNeXt is the MLP paradigm instead of Transformer-based, and it mainly discusses the effect of various data augmentation in vanilla PointNet. In this paper, we aim to unlock the potential of Mamba in point cloud tasks, discussing whether it can be a viable alternative to Transformers. Thus, the direct comparison of our PointMamba is the vanilla Transformer-based methods. Through extensive experiments, we clearly outperform the SOTA Transformer-based counterparts and even include the cross-modal method (ACT, ICLR 23).

• To weakness 4:

Reply: Thanks! 1) We will include the suggested methods in Figure 1 in the revised version. In addition, the clarification of PointNeXt is presented in reply to weaknesses 2&3; 2) Although Mamba performs inference advantages, it can not work well in the point cloud by simply replacing Transformer with Mamba. PointMamba not only keeps the efficiency of Mamba but also makes it achieve promising performance in the point cloud. 3) Besides, as shown below table, PointMamba presents superior training efficiency (second/epoch) compared with the convincing Transformer-based method, especially when the length of point tokens increases.

• To weakness 5:

Reply: Good suggestion! We now provide the analysis of our point-scanning strategy on ShapeNet-Part and ModelNet40. These results clearly prove the effectiveness of our proposed key designs when adopting Mamba to point clouds. We will release the code to ensure the results are convinced.

Dear Reviewer Daqu,

We sincerely appreciate your time and effort in reviewing our paper. We hope our explanations have addressed your concerns. As we are in the discussion phase, we welcome any additional comments or questions regarding our response or the main paper. If further clarification is needed, please do not hesitate to mention it, and we will promptly address your inquiries. We look forward to receiving your feedback.

Best regard,

Paper 940 Authors

Dear Reviewer Daqu,

Thank you for your time and valuable feedback. As the discussion phase is nearing its end, we remain open to addressing any remaining questions or concerns. We would greatly appreciate it if you could consider improving the evaluation after reviewing our responses. Thank you very much for your consideration.

Sincerely, Paper 940 Authors