State Space Model Meets Transformer: A New Paradigm for 3D Object Detection

Thank you for your constructive review. We are glad that you find our idea well-motivated and experiments sufficient and extensive. Below are our responses to the weaknesses and questions:

W1. Numerical comparisons to support the choice for SSMs.

Thank you for the suggestion. We provide numerical comparisons with methods that update scene point features using attention layers. As shown in the following table, we use standard self-attention, FlashAttention [R1], and selective SSM to update the scene feature in the decoder layers. While attention-based methods improve the detection accuracy of baseline models, they still fall short compared to methods employing SSMs. This limitation arises because global attention is less effective at handling local relationships in point clouds and may introduce noise that disrupts scene point feature updates. In contrast, selective SSM uses 1D convolution to model local relationships and incorporates a hidden state selection mechanism to focus on extracting relevant features. Therefore, we adopt SSM to address the issue of fixed scene point features in the decoder. In our proposed DEST method, query points representing foreground objects serve as state points, providing richer and more task-specific foreground information to enhance scene point feature updates.

W2. The relation of the serialization strategy in our method and PTv3 [R2].

We clarify the relationship between our serialization strategy and PTv3 [R2] and will include it in the revised manuscript.

PTv3 employs z-order and Hilbert curves for point cloud serialization and introduces the shuffle order strategy to enhance the generalization of each attention layer. In our method, we generate six different serialized results for the point cloud by reordering the x, y, and z axes of the Hilbert curve, enabling more comprehensive observations. Unlike Serialized Attention in PTv3, random serialization disrupts sequential feature modeling in SSM and degrades performance. Thus, we apply distinct serialization orders across decoder layers without using the shuffle order strategy.

Q1. Replace the inter-state attention module with a bi-directional Mamba.

The inter-state attention module can be replaced with the bi-directional Mamba module, which models relationships among state points to capture inter-object dependencies within a scene. As shown in the following table, the bi-directional Mamba achieves performance comparable to the inter-state attention module.

[R1] Dao, Tri, et al. "Flashattention: Fast and memory-efficient exact attention with io-awareness." Advances in Neural Information Processing Systems 35 (2022): 16344-16359.

[R2] Wu, Xiaoyang, et al. "Point Transformer V3: Simpler Faster Stronger." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Thank you for your detailed review. We hope our following responses can address your concerns.

Q1. The computational cost of Extension of $\Delta$.

The extension of $\Delta$ does not introduce significant computational overhead. The dimensions of the discrete SSM parameters $(\overline{A}，\overline{B})$ remain unchanged before and after extending $\Delta$. Therefore, the extension of $\Delta$ only affects the parameter generation without altering SSM processing. This is clarified in line 287 of the manuscript and Algorithm 1 in the Appendix.

To further clarify the computational overhead of the proposed ISSM, we provide the FLOPs for both parameter generation and SSM processing. As shown in the following table, under the same model structure, the computational overhead of ISSM is only slightly higher than that of Selective SSM. Both Selective SSM and ISSM adopt the multi-head structure of Mamba2 with 32 heads, where the number of scene points is 1024, the number of state points is 256, and the feature dimension is 256.

Q2. The motivation and design of the Spatial Correlation module.

The proposed ISSM needs to address a complex state update challenge: how to select appropriate scene points for updating state points. To tackle this, we develop the Spatial Correlation module, which leverages the relative positional relationships between scene and state points to select suitable scene points for state updates. Similar to VDETR, we employ vertex relative position encoding to determine the relationships between scene and state points. Unlike VDETR, which injects positional information into the attention map, we use it for generating SSM parameters $(B,C,\Delta)$. Additionally, we design a delay kernel for $\Delta$ to dynamically adjust the update magnitude of state points based on their distance from scene points.

Q3. Explain the importance of each component in the method section.

Thank you for your suggestion. We provide more illustrations about the inter-state attention module, the Hilbert serialization and the bidirectional scanning mechanism.

For the inter-state attention module, we focus on modeling feature interactions among state points. In 3D object detection, objects in a scene often exhibit strong correlations. For example, tables and chairs commonly appear together, while beds and toilets rarely coexist in the same room. To capture such relationships, we employ a standard self-attention mechanism for state points, enabling them to leverage scene semantics and enhance detection accuracy.

For the serialization strategy, we generate six different serialized results for the point cloud by reordering the x, y, and z axes of the Hilbert curve, aiming for comprehensive observation of the point cloud. Compared to PTv3 [R1], which only swaps the x and y axes, our serialization results are more diverse. Additionally, unlike Serialized Attention in PTv3, the sequential feature modeling in SSM is more sensitive to changes in the serialization method. Thus, we apply distinct serialization orders across decoder layers without the Shuffle Order strategy used in PTv3.

For the bidirectional scanning mechanism, we aim to model bidirectional point-to-point relationships in a single pass. We follow Vision Mamba [R2] and introduce the bidirectional scanning mechanism into our ISSM. Unlike the original Mamba, our ISSM uses scene points as system inputs and state points as system states. In the backward ISSM, we reverse the order of scene points while keeping the state points unchanged.

Q4. The meaning of "Param." in Table 5.

In Table 5, the results in "Param." represent the total parameters of our DEST-base model. We clarify this in the revised manuscript to avoid any ambiguity.

Q5. Typos.

We thoroughly revised the manuscript to correct typos and improve any unclear expressions.

[R1] Wu, Xiaoyang, et al. "Point Transformer V3: Simpler Faster Stronger." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[R2] Zhu, Lianghui, et al. "Vision mamba: Efficient visual representation learning with bidirectional state space model." arXiv preprint arXiv:2401.09417 (2024).

Thank you for your insightful review. We are glad that you find our topic meaningful, the experiments convincing, and the method design creative! Below are our responses to the questions:

Q1. Other methods to solve "fixed scene point" from previous works.

To the best of our knowledge, this study is the first to identify and address the performance limitations in 3D detection caused by fixed scene point features. However, some techniques from previous works could be adapted to address this issue. A straightforward approach involves introducing a module in the decoder to update scene point features, such as using standard self-attention, FlashAttention, or selective SSM.

The following table shows that incorporating a feature-updating module in the decoder improves detection performance. However, the performance gains of these methods remain limited compared to our proposed method. We attribute this to the unique design of our method, where query points representing foreground objects serve as state points, providing richer and more task-specific foreground information to enhance scene point feature updates effectively.

Q2. Simultaneously updating scene and query point features is helpful for most DETR-based models.

Simultaneously updating is beneficial for most DETR-based models for the following two reasons:

1. DETR-based methods commonly use the multi-layer transformer decoder to refine query points iteratively. The issue of fixed scene point features limits the performance improvement from later decoder layers.
2. Updating scene point features in decoder provides more adaptive and context-rich information to the query points, enhancing their refinement process and ultimately improving detection accuracy.

The experimental results in Q1 also validate that simultaneously updating scene and query point features benefits the DETR-based models, even when using a simple global attention mechanism.

Q3. Combine SSM and Transformers in 3D object detection?

Combining SSM and Transformer in 3D object detection is indeed feasible. As shown in the table of Q1, we incorporate selective SSM into the transformer decoder to update scene point features. This simple combination method improves the detection accuracy, and it holds potential for further exploration in the future.

Thank you for your detailed review. We are glad that you find our paper well-written and our method innovative! Below are our responses to the weaknesses and questions:

Q1. The motivation for introducing SSM in this paper.

We introduce SSM with the following two motivations:

1. To integrate modules within the decoder for updating scene point features, thereby overcoming the performance bottleneck caused by fixed scene point representations in DETR-based methods.

2. To avoid the substantial computational and memory overhead associated with self-attention mechanisms when applied to updating dense scene point features.

Q2. Why is SSM suitable for 3D detection tasks?

Applying SSM to 3D detection tasks offers the following three advantages:

1. Simultaneous Update of Scene and Query Point Features: By treating query points as system states and scene points as system inputs, SSM enables simultaneous feature updates of query points and scene points.
2. Effective Feature Modeling: As detailed in Appendix A.1, state points in SSM effectively extract information from scene points to achieve accurate detection. Additionally, scene points leverage the rich foreground information encoded in state points to update their features.
3. Linear Complexity: Compared to transformers, SSM significantly reduces computational and memory overhead, making it well-suited for handling dense scene points.

W1. The proposed method results in a noticeable increase in latency.

The need for multiple updates to scene point features in the decoder inevitably introduces some latency. Although we employ an SSM with linear complexity to minimize computational overhead, certain delays are unavoidable. Considering the substantial performance gains, this trade-off is acceptable.