Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection

[Q1, Q2] Manuscript refinement.

We apologize for the typos and presentation issues. We promise to thoroughly proofread the entire manuscript and correct the grammatical and typographical errors as much as possible. Considering that some concepts may be difficult for newcomers to this field, we will include more intuitive explanations and visualizations to aid their understanding. For sure we will release our code and pre-trained models to provide detailed implementation.

[Q3, Q4] Computational complexity.

Firstly, it's important to clarify that Voxel Mamba's computational complexity significantly differs from Mamba in language models. In our implementation, its layers and parameters are carefully aligned with standard 3D object detectors.

Besides, in Table 5(e) and Figure 3 of the main paper, we compared our method with previous state-of-the-art (SOTA) methods in terms of latency, performance, and GPU memory. Here, we provide a more detailed comparison between Voxel Mamba and previous SOTA serialization-based methods. Actually, Voxel Mamba has comparable to or even lower computational demands than other SOTA serialization-based methods. As shown in the following table, Voxel Mamba differs from DSVT only in the 3D backbone but outperforms DSVT by +1.5 L2 mAPH with less GPU memory and inference latency. All the results are tested on the same device.

[Q5] Robustness to weather and light.

Thanks for the constructive suggestion. Adverse weather can cause scattered points due to attenuation and backscattering. To evaluate Voxel Mamba's robustness to adverse weather, we simulated various fog densities using the fog augmentation method in [1] on the point clouds during inference. The table below shows the mAP results on the nuScene dataset with fog severity from 1 to 5 (heavy). One can see that Voxel Mamba consistently outperforms DSVT across all fog densities. Moreover, with the increase of severity, the performance advantage of Voxel Mamba increases, indicating our method's superior robustness to adverse weather.

[1] Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather. ICCV21

[Q6] Indoor semantic segmentation.

Thanks for the question. Please kindly refer to shared response Q3 for details.

[Q7] Real-time Voxel Mamba.

It is a common practice to optimize 3D detection algorithms for deployment in autonomous driving systems. Frameworks such as TensorRT are widely used to enhance the memory efficiency and inference speed. Considering that the frequency of outdoor LiDAR sensors ranges from 5 to 20 Hz, the optimized Voxel Mamba can indeed meet the real-time requirements.

[Q1] Clarifying our motivation.

Thanks for the question. Please kindly refer to our responses to Q1 of Reviewer kB9g for details.

[Q2] Efficiency and performance for different model sizes and resolutions.

We appreciate the reviewer's feedback and conducted additional experiments. The results regarding the model size (i.e. feature dimension) and BEV resolution (voxel size) are shown in the following two tables.

The above table demonstrates a consistent performance improvement as the dimension increases, which correlates with the model scale.

The above table demonstrates that Voxel Mamba remains effective across different voxel sizes. Increasing voxel size reduces input BEV resolution, enhancing efficiency. Voxel Mamba performed best with a voxel size of 0.32m.

[Q4] Generalization of Voxel Mamba.

Thanks for your question. Please kindly refer to the shared response Q3.

[Q5] More comparison with other state-of-the-art methods.

We would like to draw attention to Figure 3 in our manuscript, which provides a comprehensive comparison of our method with state-of-the-art approaches commonly benchmarked in the field of 3D object detection.

[Q6] Robustness to density and sparsity.

We re-benchmarked the performance in different range intervals on Waymo to assess Voxel Mamba's robustness to density and sparsity. The following table indicates that Voxel Mamba is more robust than DSVT.

[Q7] Motivation of downsampling in ASSMs.

Though space-filling curves can preserve the 3D structure to a certain degree, proximity loss is inevitable due to the dimension collapse from 3D to 1D. As a result, a local snippet of the curve can only cover a partial region in 3D space. Placing all voxels in a single group cannot ensure that the effective receptive field (ERF) could cover all voxels. Therefore, we introduce downsampling (hierarchy of state space structures) and consequently improve the ERF of the model.

[Q8] Clarification on pipeline.

Please kindly refer to our responses to Q4 of Reviewer kB9g for details.

[Q9] Clarification on title.

We respectfully disagree with your opinion. To clarify, while 3D object detection is often associated with point clouds, it is not exclusively a point cloud-based task.

[Q10] Computational complexity with point densities.

Refer to shared responses for details. With increased point densities (4-frame), Voxel Mamba is 20ms faster than DSVT, which is more pronounced in 1-frame and indicates our method's efficiency.

[Q11] Advantage over Transformers and Spconv.

For details on Transformers, please kindly see Q1 of Reviewer kB9g. It is important to note that traditional CNNs are typically avoided in 3D detection due to slow inference on sparse data. Compared to SparseConv, Voxel Mamba has significantly larger ERFs, enabling better capture of long-range dependencies in point clouds. Additionally, our method is more deployment-friendly on autonomous driving than previous methods since no padding tokens are needed.

[Q12] Motivation of IWPE. The window-based position embedding has been proven effective in previous window-based transformers, as it retains the proximity of voxels within a local window. As a group-free model, we do not explicitly partition the voxels into windows. Instead, we implicitly encode their positions within the window as additional information in the voxel sequence, which significantly enhances the SSM model's ability to extract useful information from the context.

[Q13] Dynamic object.

As indicated in Table 1 of our manuscript, all our experiments were conducted in a single-frame setting. Consequently, the inputs do not include dynamic objects, as each frame is processed independently. To address the concern about non-static elements, we report the multi-frame results in the shared response.

[Q14] Degrade with truncation. To evaluate Voxel Mamba's robustness to truncation, we compared it with DSVT under various truncation levels using the cutout methods in [1]. The table below shows the mAP of Voxel Mamba and DSVT on the nuScene dataset with truncation severities ranging from 1 to 5. One can see that Voxel Mamba consistently outperforms DSVT across all truncation densities, indicating our method's superior robustness.

[1] Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving. CVPR23

[Q15] Adversarial attacks, dataset overfitting, and multi-task learning

Thanks for your suggestion. In this paper, we aim to establish a strong backbone in 3D object detection, and those issues will be discussed in future work.

Dear reviewer:

Thank you so much for reviewing our paper. We hope our explanation could resolve your concern. During this discussion phase, we welcome any further comments or questions regarding our response and main paper. If there requires further clarification, please do not hesitate to bring them up. We will promptly address and resolve your inquiries. We are looking forward to your feedback.

The author's rebuttal addressed many of the issues I raised. However, I still have some doubts about this article. Although the author has further elaborated on the motivation, there are still some areas of doubt. For visual perception tasks such as object detection, is contextual information really important, and what is the interpretability of using RNN type network frameworks. Overall, the paper is quite innovative, so I'm inclined to rate it as boardline acceptable.

We really appreciate for this reviewer's recognition on the innovation of our work. Yes, the contextual information is important in point cloud object detection. Since 3D point clouds lack distinctive textures and appearances, the detectors will more heavily rely on contextual information for learning semantic information. We have developed several designs to enhance Voxel Mamba's ability to capture contextual information in the paper. To increase contextual information for voxel tokens, we proposed a group-free strategy to serialize the whole space voxels into a single sequence. Besides, we also integrated a multi-scale design to broaden the effective receptive field and capture context information at various scales.

We used RNN to address the inefficiency of Transformers when expanding input context. In the large context setting, we adopted Mamba (RNN) design to enhance the efficiency, as latency is crucial in outdoor 3D object detection for autonomous driving. As shown in the table following, RNNs have a significant speed advantage over Transformers with the longer input voxel sequences, which often reach the order of $10^5$ in 3D detection scenarios.

If there is any misinterpretation of your question, we are open to further discussion

[Q1] Comparison with other variants.

Thanks for your insightful comments. To demonstrate the improvements of our method, we have conducted additional experiments comparing Voxel Mamba with group-based and group-free bidirectional Mamba. The group-based bidirectional Mamba uses the DSVT Input Layer to partition the voxel groups. The group-free bidirectional Mamba is based on our Hilbert Input Layer. Each variant retains the same number of SSM layers for consistency in comparison.

The above table shows that enhanced spatial proximity can significantly improve the detector's performance. Additionally, our innovative multi-scale design yields additional performance gains, which can be attributed to the expansion of ERFs.

[Q2] Advantage over group-based operations.

Thanks for your suggestions. Please kindly refer to shared response Q1 for the advantage of group-free operations.

[Q3] Comparison with group-based Mamba.

To address the reviewer's concern, we compared the performances between group-free and group-based mamba (refer to the table in our response to Q1). The results demonstrate the effectiveness of our group-free strategy, which can enhance the 3D proximity in the 1D sequences. Besides, we provide the ERFs of group-based (DSVT partition) Mamba in Figure 1 in the rebuttal PDF.

[Q4] Analysis on ASSMs.

We appreciate the reviewer’s insightful comment. The transitioning from {1,1,1} to {1,2,2} and to {1,2,4} enhances performance due to an enlarged effective receptive field and improved proximity by using larger downsampling rates at late stages. ASSMs with {2,2,2} or {4,4,4} compromise performance compared to {1,1,1}, indicating that using larger downsampling rates at early stages will lose some fine details. Thus, we set the stride as {1,2,4} to strike a balance between effective receptive fields and detail preservation. Additionally, your inference about the similar performance of different space-filling curves is astute. Voxel Mamba is designed with significantly large ERFs, which indeed diminishes the impact of the type of space-filling curves on performance. We will expand our analysis in revision.

[Q5] Curve template memory.

Yes, we record the traversal position in the space-filling curves of all potential voxels offline. The curve templates are adaptively generated based on the input BEV resolution. To address the reviewer's concern, the table below lists GPU memory usage for templates. The input BEV resolution is (468, 468), with three scale curve templates for each ASSM scale.

[Q1] Motivation.

Thanks for the question and we are sorry for not making the motivation clear enough. The motivation of our work is to leverage the advantages of SSM in linear attention for more effective feature learning in 3D point cloud based object detection. The advantage of SSM over Transformer lies in its ability to more efficiently explore large context (as shown in Fig. 4 of the main paper), i.e., an extremely long voxel sequence, for effective feature learning. From the table below, one can see that for more than 4K voxel contexts, a single transformer layer consumes more than 5ms, which is impractical for perception tasks. However, even for a sequence of 16K voxels, SSM only consumes 0.94ms.

Unlike previous window-based voxel transformers, we are the first to model the voxel sequence from the entire scene. By utilizing Mamba's selective scan mechanism, we can effectively retain useful information from the context without the need for complex voxel grouping. This group-free approach is insightful for handling large-scale sparse data.

[Q2] Effectiveness of multi-scale design.

We appreciate the reviewer's insightful observation. In fact, as a sequence-based model, the receptive fields of voxel features only rely on the sequence length, and they do not necessarily need to be aligned. Mamba can selectively retain useful information after scanning the sequence back and forth. Here we further compare ASSM with FPN on the Waymo dataset with 20% training data. Following Voxel Mamba, we utilize Spconv and its counterpart SpInverseConv as downsampling and upsampling operations. We keep the same downstrides in both structures. The table below shows that our ASSM achieves superior performance in the 3D object detection tasks.

[Q3] Comparison with state-of-the-art pillar-based methods.

Thanks for the suggestion. The following table compares our pillar-based Voxel Mamba with previous state-of-the-art pillar-based methods. We can see that Voxel Mamba outperforms previous methods on both L2 mAP and mAPH while achieving comparable inference speed.

[Q4] Detailed experiments and ablation studies.

We apologize for any confusion regarding the details of our experiments. For fair comparison, our experiment settings follow the schemes in DSVT, using the Adam optimizer with weight decay of 0.05, a one-cycle learning rate policy, and a maximum learning rate of 2.5e-3. Models were trained with a batch size of 24 for 24 epochs on 8 NVIDIA A6000 GPUs. During inference, we use class-specific NMS with IoU thresholds of 0.7, 0.6, and 0.55 for vehicles, pedestrians, and cyclists, respectively. Ground-truth copy-paste data augmentation was used during training and disabled in the last epoch. Our setting aligns closely with the previous state-of-the-art DSVT, and the primary difference is on the backbone. In addition, Voxel Mamba does not adopt the IoU-rectification scheme.

In our ablation study, we assessed the effectiveness of various components and drew the following conclusions:
(a) Space-filling curves enhance spatial proximity and locality, significantly improving performance (Table 5d).
(b) Position embedding with IWPE boosts detection performance by providing rich 3D positional and proximate information (Table 5a).
(c) Downsampling rates of ASSM show that larger effective receptive fields improve performance, but very large downsampling rates at early stages can degrade performance. We found a balance with a stride of {1,2,4} (Table 5).
(d) Bidirectional SSMs with a Hilbert-based group-free sequence significantly improves accuracy, validating our group-free strategy. ASSM enhances effective receptive fields and proximity, while IWPE further boosts Voxel Mamba’s performance by capturing 3D positional information (Table 6c).

[Q5] Multi-frame results.

Please refer to the shared responses Q2 for details.

[Q5] Description of window sweep.

The window sweep adopts the same regional grouping method as SST, extended to 3D voxels. We use a fixed BEV window size of (12, 12) and implement Region Shift between each two ASSM blocks.

[Q6] Detail of Figure 2.

We apologize for this problem. We will revise Figure 2 to accurately reflect the complete architecture.

Dear reviewer:

Thank you so much for reviewing our paper. We hope our explanation could resolve your concern. During this discussion phase, we welcome any further comments or questions regarding our response and main paper. If there requires further clarification, please do not hesitate to bring them up. We will promptly address and resolve your inquiries. We are looking forward to your feedback.

[Q1] Latency of generating curves.

In our implementation, we record the traversal position in the space-filling curves of all potential voxels offline. The voxels are serialized by simply looking up these traversal positions. Thus, the latency for serializing voxels based on Hilbert and Z-order curves is identical. The detailed generation times of the Hilbert Input Layer are shown in the shared responses Q1.

[Q2] Random Curves setting.

In the Hilbert Input Layer (HIL) implementation, we use fixed templates for each resolution in the ASSMs. With the downsampling rate {1,2,4} in ASSMs, Voxel Mamba requires three different scale curve templates.

However, the "random" in Table 5(a) means that we do not perform any sorting on the input sequence for SSMs. We observed that the Voxel Feature Extractor (VFE) and the downsampling and upsampling operations tend to preserve the spatial proximity of neighboring voxels in the 3D space in the 1D sequence. This inherent characteristic leads to a favorable outcome for SSMs. To ensure a real random order, we generate three random templates and use them for all frames following the HIL setting. The following shows the results on the Waymo open dataset with 20% training data. These results indicate that disrupting voxel proximity significantly degrades performance, highlighting the effectiveness of our Hilbert Input Layer.

[Q3] Hilbert Input Layer + attention.

We truly appreciate this valuable comment. We conducted an experiment by replacing the window grouping in DSVT with the serialization approach in Voxel Mamba. We use the same group size of 48 as DSVT, and only pad the last voxel groups for each sample to enable the parallel computation of Transformers. We employ curve shift as an alternative to the X/Y-axis partition and perform experiments on 20% Waymo training data. The following table compares this variation with Voxel Mamba and DSVT.

One can see that this variant with the Hilbert Input Layer shows slightly lower performance than DSVT. We hypothesize that this drop is due to the limited group size in the window attention mechanism. This constraint might limit the full potential of the HIL voxel flattening. While it does not outperform Voxel Mamba in accuracy, it does offer a favorable trade-off between performance and latency.