3DET-Mamba: Causal Sequence Modelling for End-to-End 3D Object Detection

Q1: Computational costs and scaling up performance.

A1:

• To further verify the effectiveness of 3DET-Mamba, we report latency and memory costs in the following table. It can be seen that our model achieves better results with less computational cost and lower latency compared to 3DETR. The results further verify the effectiveness of 3DET-Mamba.
• Besides, we conduct more experiments in which we scale up the model size (more mamba block). As shown in Figure 2 and Figure 3 in the PDF file, both scaling the encoder and decoder can further improve the performance on ScanNet which further shows the scaling ability of 3DET-Mamba.

Performance Comparison of Computational Efficiency

Q2: Discussion on the proposed method.

A2:

• In this paper, we first explore the potential of Mamba in 3D object detection tasks which have not been studied. However, directly using mamba to deal with 3D object detection tasks results in poor results. This is mainly caused by 1) It is difficult for Mamba to model unordered and non-causal 3-D point clouds, 2) the original Mamba block lacks the ability to extract local features, and 3) previous works only explore Mamba as an encoder for classification tasks.
• To handle the above challenges, we propose a local-to-global scanning technique which is composed of Inner Mamba blocks and dual Mamba blocks and can aggregate local and global information. Besides, in dual Mamba blocks, we use furthest point sampling and nearest point sampling to construct the ordered point cloud sequence to handle the non-casual problem. Finally, the query-aware Mamba block is designed to decode scene context information into object sets with the guidance of box queries.
• We hope our work can Inspire further exploration of mamba in 3D detection tasks and construct Mamba-base 3D foundation models in the future.

Q3: Some typos.

A3: Thanks for carefully reviewing our article. We will modify all these typos in the next version.

Dear reviewer,

Thank you for taking the time to review our work and for your feedback! We will add experimental results of computational efficiency in the revision of our article.

Sincerely, Authors of paper112

Q1: Difference between Algorithm 2 and Figure 3.

A1: Thanks for reviewing our article carefully. The algorithm 2 shows the detailed operation steps of our model. The figure 3 is a schematic figure and may omit some details. According to the suggestion of the reviewer, we will modify Figure 3 in the revised version.

Q2: Question about Table 4.

A2:

• Thanks for pointing out our mistakes. We will modify the headers of Table 4 in the revised version.
• We want to clarify that in Table 1, we report the results of 3DETR in the original paper which is trained for 1080 epochs. However, in Table 4, for a fair comparison, we only change the decoder to 3DETR decoder and use our mamba-based encoder. We train for 540 epochs which is consistent with 3DET-Mamaba. We will clarify it in the next version.

Q3: Some typos.

A3: Thanks for your comments. We will revise all the typos in the next version.

Q4: Question about randomly choosing the initial point.

A4: The aim of this paper is to explore the potential of Mamba in 3D object detection. We follow 3DETR[A] to random sample an initial point and then sample a set of points using FPS which are evenly distributed in 3D space. Besides, based on our experiments, the experimental results are stable and we repeat the experiments. The results are shown in the following table. Since the difference between the random seed and the GPU device, there is a slight deviation from the submitted article. However, it can be observed that the experimental results are still stable.

Repeated experimental results of 3DET-Mamba

[A] Misra I, Girdhar R, Joulin A. An end-to-end transformer model for 3d object detection[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 2906-2917.

Thanks for your valuable comments.

Q1: Comparison of time and space complexity between 3DET-Mamba and other methods and the effects of changing resolution.

A1: To demonstrate the superior performance of 3DET-Mamba, we compare the FLOPs and latency of our model with the pervious transformer-based architecture, 3DETR [A], as presented in the Table r1 below. It can be seen that 3DET-Mamba achieves higher accuracy (as detailed in Table 1 of our manuscript), reduced FLOPs and lower latency, as shown by Table r1. In Table 1 of our manuscript, we show that 3DET-Mamba effectively models high-resolution point clouds (i.e., 4096 point clouds), achieving a significant +5.7 mAP@0.5 increase. Furthermore, to explore the impact of changing point resolution, we conduct experiments on changing the density of point clouds. As shown in Figure 1 in the attached pdf file, by increasing the density of point clouds, the performance can be continuously improved which shows the effectiveness of 3DET-Mamba again.

Table r1: Performance Comparison of Computational Efficiency

Q2: Additional ablation studies.

A2: Thanks for the valuable comment.

• We first want to clarify that the intention of our work is to explore the feasibility of Mamba in 3D object detection which has not been studied. Previous works mainly explored the usage of Mamba in classification and segmentation tasks. However, this task is challenging due to 1) the non-casual and irregular 3D point clouds hinder the modeling of causal sequences. 2) The original Mamba block is good at extracting global information from long sequences but ignores detailed information to a certain extent which is important in 3D detection tasks.

• We address the first challenge using ranked queries. To further verify the effectiveness of this approach, we conduct ablation studies shown in Table r2 below, which indicate that discarding ranked queries leads to a decrease in performance.

  Table r2: Ablation Study on Ranked Queries

  Rank Query	mAP25	mAP50
  ✗	65.6	48.1
  ✓	66.9	48.7

• To clarify the relevance of NPS and FPS in modeling patch sequences, our results in Table r3 demonstrate that combining FPS and NPS can further improve performance by enabling Mamba to models the point cloud in terms of spatial distribution and continuity.

  Table r3: Impact of NPS and FPS

  NPS	FPS	mAP25	mAP50
  ✓	✗	65.4	47.7
  ✗	✓	65.0	46.3
  ✓	✓	66.9	48.7

• Existing Mamba-based works mainly use Mamba as the encoder. However, there is still no exploration of using Mamba as a decoder. To explore the potential of Mamba as a 3D decoder, we propose query-based Mamba which can better model the relationship between each query by taking advantage of Mamba. As shown in Table 5 in our article, our query-aware Mamba block can outperform the transformer-based decoder.

• As shown in Table r1, compared to the transformer-based models, our method (which includes the query-aware Mamba block) exhibits lower FLOPs and latency. This demonstrates that the improvements are indeed due to the design of the novel Mamba block, rather than increased computational power.

Q3: Comparisons of FCAF3D and speed.

A3: Thank you for your valuable comments. In this paper, we focus on indoor 3D object detection using point clouds, a crucial 3D data representation form. Compared to voxels (used in FACF3D), point clouds offer benefits such as high-efficiency storage. We benchmark our 3DET-Mamba against other open-source point-based methods, with results in Table 1 of our manuscript demonstrating our approach's effectiveness. However, it is not trivial to directly apply Mamba to point cloud scenes due to their unordered and non-causal nature. 3DET-Mamba addresses these challenges by introducing a novel local-to-global scanning mechanism and developing the Inner Mamba and Dual Mamba to capture fine-grained point cloud features. Additionally, we designed a Query-aware Mamba Block to decode point cloud information. We hope our work will inspire further use of Mamba as a foundational component for point cloud understanding.

Q4: Clarification of FPS and NPS.

A4: The FPS and NPS in our article represent the furthest point sampling and the nearest point sampling. The furthest and nearest point order can be obtained after sampling. We will modify the words in dual Mamba to avoid confusion.

Q5: Existing methods of query-based mamba works.

A5: Thanks for your valuable comments. We carefully investigate query-based mamba works. Currently, there is still no work to explore query-based mamba for object detection. However, some concurrent works introduce query-based mamba for other tasks [B][C]. TM-Mamba[B] modifies the Mamba parameterizes $A$, $B$, $C$, $∆$ as the function of the input and text query to ground the human motion. However, it is not applicable when the number of queries is greater than 1. Besides, QueryMamba[C] combines a query-based transformer decoder and the Mamba encoder. However, it still does not explore query-based Mamba decoder. We will add these related works in the revised version.

[A] Misra I, Girdhar R, Joulin A. An end-to-end transformer model for 3d object detection[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021. [B] Wang X, Kang Z, Mu Y. Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motion[J]. [C] Zhong Z, Martin M, Diederichs F, et al. QueryMamba: A Mamba-Based Encoder-Decoder Architecture with a Statistical Verb-Noun Interaction Module for Video Action Forecasting@ Ego4D Long-Term Action Anticipation Challenge 2024[J].