LION: Linear Group RNN for 3D Object Detection in Point Clouds

W1: Some claims are obscure and not well supported by evidence ...

Before feeding voxel features into linear RNN, we need to flatten 3D voxel features into 1D sequence features. Unlike the common 3D sparse convolution operation that directly deals with 3D voxel features in 3D space, the linear RNN only processes 3D voxel in 1D space, bringing limitations in handling spatial modeling. Here, the limitation is that the linear RNN is a sequence model, which is less effective in perceiving 3D spatial information (e.g. two adjacent voxels). Spatial modeling means capturing the local 3D geometric information for each voxel.

W2: The proposed techniques are not novel.

• Although the 3D spatial descriptor consists of a common spconv-based module, it is crucial to help our linear RNN-based network to better capture 3D local spatial information. Therefore, the combination of 3D spatial descriptor and linear RNN operators could make each voxel feature perceive the local spatial information and long-range relation, which is important to improve detection performance.
• Voxel merging and expanding operations are applied to reduce the computation cost without harming the detection performance. But we do not claim these operations are our contributions (Refer to L63-71 of the main paper).
• For voxel generation, we make the first attempt to leverage the auto-regressive properties of linear RNNs (refer to Table 5 in the main paper) to generate new voxel features for 3D detection. This is different from the mentioned method FSDv2. Specifically, FSDv2 first votes center points and converts these points to virtual voxels. For virtual voxel features, FSDv2 aggregates these point features by MLP. Therefore, we could categorize it as a KNN manner for voxel diffusion (Refer to the experiments in Table 5 of the main paper). Finally, we will add this discussion in the revised paper.

W3: Ablation for larger group size.

Thanks. We provide the ablation studies of different group sizes in the following table. Here, we set a minimum group size of 256 for all our four LION blocks (Baseline: [256, 256, 256, 256]). We could observe that the manners with larger group size (i.e, II, III, IV, V) bring consistent performance improvement over the baseline (I). However, we carefully find that there is a drop in performance from IV to V by further enlarging the group size. This might lead to less effective retention of important information in excessively long sequences due to the limited memory capacity of linear RNNs.

W4: More detailed runtime evaluation.

We provide the detailed latency of each part of our LION-Mamba. Considering that some linear RNN-based operators (Mamba, Retnet, or RWKV) are usually applied to replace Transformer in modeling long sequences thanks to their efficiency. Therefore, we sufficiently compare the latency with transformer-based backbone (DSVT) to illustrate the efficiency of our LION. Here, we evaluate the latency on one NVIDIA GeForce RTX 3090 with a batch size of 1. Due to the quadratic complexity of the transformer, DSVT adopts a small group size of 48 in their paper. When we increase the group size of DSVT to 256, the latency obviously increases and the corresponding memory is even unacceptable (produces about 20G GPU memory for inference and fails to train this model due to OOM). In contrast, benefiting from the high efficiency of the linear RNN in modeling long sequences, our LION could adopt a larger group size (4096) for feature interaction while maintaining acceptable latency (146.2 ms) and low GPU memory (about 3G during the inference).

Furthermore, we provide a detailed ablation study on the Waymo validation set with 20% training data to analyze the effect of each component on the efficiency and performance.

Q1: It seems like the multiple diffused voxels can occupy a single position.

We apologize for the unclear presentation. We merge the diffused voxels and raw voxels by summing these voxels at the same position in the voxel merging operation to deduplicate voxels. We will make it clearer in the revised version.

Q2: How many new voxels will be generated? Does the voxel generation reduce the efficiency?

Actually, the number of new generated voxels depends on the number of input voxels and the diffusion ratio. Moreover, the voxel generation does reduce the efficiency of the whole network. To better illustrate the effect of voxel generation, we provide the ablation study on the important hyper-parameter of diffusion ratio in the following table. We find that a larger diffusion ratio brings more latency but better performance. In this paper, we set the diffusion ratio as 0.2 to trade off the performance and latency.

W1-1: In this paper, the author has transformed the irregular point cloud into regular voxel representation. However, L172-175, the author claim max or average-pooling operations is not suitable to achieve downsampling or upsampling operations. This is conflicted.

Sorry for this misleading! Since the distribution of voxels in 3D space is sparse, the regular max or average pooling operations on a dense 2D image are not suitable for performing voxel downsampling or upsampling.

W1-2: Furthmore, I think the motivation provided for the voxel generation approach is not sufficient, making it difficult to accept the motivation behind the proposed voxel generation method.

Our motivation of voxel generation has two folds: 1) Voxel generation can densify some key voxel features to enhance the feature representation in highly sparse point clouds; 2) Voxel generation can mitigate the information loss from voxel merging operation, which is an effective operation to reduce the computation cost in our LION. We will make them clearer in the revised version.

W2: The author should report the test set results on the waymo dataset of the experimental results. In addition, the authors should also report the results on the multi-frame setting on the waymo dataset to prove the effectiveness of the method in the multi-frame setting. I want to the know the test set results on waymo dataset and multi-frame setting results.

Thanks for your nice suggestion. We provide the results with 3 frames on the val set in the following table. It could be found that our LION-Mamba-L even outperforms DSVT with 2.2 mAPH/L2, which effectively illustrates the superiority of our LION.

Furthermore, for submitting Waymo test benchmark, it is a common trick to unify the training and validation datasets to train the model for better performance. Therefore, we need to reorganize our dataset and re-train our model for the test set. Considering the limited time, we train our LION-Mamba-L with 3 frames for only 12 epochs to save the training time. Our LION-Mamba-L achieves state-of-the-art performance on Waymo test dataset, whose corresponding results are as follows (We also provide the screenshot for results on the Waymo official website with 3 frames in the uploaded global PDF file):

W3: The authors should report the inference time of LION so that we have a clearer understanding of the latency and resource consumption of the proposed method.

Thanks for your suggestion. We provide the detailed latency of our LION-Mamba and LION-Mamba-L inference time in the following table. We evaluate the latency on one NVIDIA GeForce RTX 3090 with a batch size of 1. For a more detailed discussion of the Transformer-based method DSVT in terms of latency and performance, please refer to W4 of Reviewer 9fJS.

W4: When will release the code for the community?

We will release all codes and models by September 30, 2024.

Q1: L124, I’m not clear about the Window Partition. As far as I understand, non-empty voxels are extracted line by line into the window along the X or Y axes. Every time a sequence of 4096 lengths is filled, the remaining non-empty voxels are continued to be placed in the next window. So, how do you deal with it if the final window is not enough for 4096 voxels. Regarding the division and description of windows, it should be clearer. The current description is difficult for me to understand.

We apologize for the unclear presentation. For the final window that is not enough for 4096 voxels, we repeat these remained voxels in this window up to 4096 voxels. We will make it clearer in the revised version.

Thanks for the author's rebuttal. My concerns have been solved and I raise up my score. Release code will be great for the community.

Thanks for your careful review and valuable suggestions. Here, we further address our contributions in this paper:

1. Linear RNN-based 3D detection framework: support kinds of linear RNNs (e.g., Mamba, RWKV, RetNet) to allow long-range feature interaction.

2. 3D spatial feature descriptor: compensate for the lack of linear RNNs in capturing local 3D spatial information.

3. Voxel generation: utilize auto-regressive property of the linear RNN for voxel diffusion and obtain a more discriminative feature representation in highly sparse point clouds.

[Novelty]

1. Model Structure.

   • The encoder-decoder structure is not our contribution and we adopt it to reduce the computational costs while maintaining superior performance. We provide the experiments in the following table.

   Method	Latency (ms)	Vehicle	Pedestrian	Cyclist	mAP/mAPH (L2)
   LION-Mamba(w/o encoder-decoder structure)	180.1	66.8/66.4	75.1/70.2	72.1/71.1	71.3/69.2
   LION-Mamba	146.2	67.0/66.6	75.4/70.2	71.9/71.0	71.4/69.3

   • Although the 3D spatial descriptor consists of a common and simple spconv-based module, it is crucial to help our linear RNN-based network to better capture 3D local spatial information. Therefore, the combination of 3D spatial descriptor and linear RNN operators could make each voxel feature perceive the local spatial information and long-range relation, which is important to improve detection performance ((Refer to the Table 3 and 4 in the main paper)).

   • The structure of LION layer is a widely used structure (e.g., Swin Transformer, FlatFormer, and DSVT). We keep consistent with DSVT for convenience.

2. Window Partition.

   We follow the previous work FlatFormer (Refer to L130-133 in the main paper). Besides, we do not claim that the equal-size window partition is not our contribution.

3. Voxel Generation.

   Thanks for your valuable feedback. We would like to emphasize that our primary contribution in voxel generation lies in the first attempt of the auto-regressive capacity of the linear RNN. The part of “distinguishing foreground voxels without supervision" only serves it. We appreciate your point regarding the distinction of foreground voxels without supervision, and we will revise this part in the final version.

[Latency]

Thanks. We provide the comparison of different methods. We evaluate the latency on single 3090 with a batch size of 1. LION is an early attempt to adopt linear RNN to 3D object detection, and the running time needs to be optimized compared with the highly-optimized operator (e.g., sparse conv). We will improve the speed in future work for the real-time application. Please refer to the limitation (L323-325) in the main paper. For Figure 1 in the main paper, we will revise it in the final version. For a more detailed discussion of the Transformer-based method DSVT, please refer to W4 of Reviewer 9fJS.

[Motivation]

1. Motivation to use X/Y-axis window partition.

   For linear RNN models, different sequence arrangements lead to varying feature interactions. More arrangements can improve feature richness, so we use X/Y-axis window partitioning to generate diverse sequences.

2. Why did the author choose only these four offsets?

   Setting more different offsets usually results in producing more voxels, which brings more computation costs for the following LION layers. To reduce computation cost, we only consider the offset of each voxel for diffusion in BEV space and ignore the offset along the Z axis. Here, we simply choose these four offsets along the two diagonals in BEV space.

[Weakness]

1. A sequence of thousands based on window partitions must include the voxel and all its neighbors ...

   Although a sequence with thousands of voxels usually contains the voxel features of their neighbors, two adjacent voxels might be far apart in this sequence (please refer to Figure 4 in the main paper) since the linear RNN extract features along the sequence. Therefore, we adopt an additional local sparse convolution (or 3D spatial feature descriptor) to address this problem.

2. Bidirectional modeling.

   Sorry for missing this detail! In this paper, all linear RNNs (Mamba, RWKV and RetNet) adopt the bidirectional manner for better feature interaction. We will add the details in the revised version.

[Question]

1. Comparison with HEDNet

   Thanks. We provide the comparison with HEDNet in [Latency].

2. Visualization of long-range dependencies.

   Thanks. We provide the visualization for long-range dependencies in Figure 1 in our uploaded PDF file.

3. Performance of replacing the LION Layer with DSVT block.

   We replace the LION layer with DSVT block to validate the effectiveness of LION, as shown in the following table. * represents the results without our 3D spatial feature descriptor and voxel generation. All models are trained with 20% data and 12 epochs. We report the official results of DSVT (I). We can observe that the performance of integrating DSVT into LION (II and IV) is lower than LION (III and V). Finally, the manners with our 3D spatial feature descriptor and voxel generation (IV and V) produce much better performance than the methods (II and III), which effectively demonstrates the importance of our proposed components.

   #	Method	Latency (ms)	Vehicle	Pedestrian	Cyclist	mAP/mAPH (L2)
   I	DSVT	136.7	67.2/66.8	72.5/66.4	70.1/69.1	69.9/67.4
   II	LION-DSVT*	122.7	64.8/64.3	71.0/63.6	67.4/66.2	67.7/64.7
   III	LION-Mamba*	123.2	66.2/65.7	73.7/67.2	68.7/67.6	69.5/66.9
   IV	LION-DSVT	157.7	66.1/65.7	74.4/68.8	70.7/69.8	70.4/68.1
   V	LION-Mamba	146.2	67.0/66.6	75.4/70.2	71.9/71.0	71.4/69.3

W1: Figure 3 (a) is confusing. Since the 3D Spatial Feature Descriptors are neural layers with learnable parameters, it would be better to represent these layers with blocks like “LION Layer”.

Thanks for your nice suggestion! We will revise "3D Spatial Feature Descriptors" as "3D Spatial Description Layer" to make it clearer in the final version.

W2: It would be beneficial to discuss the rationale for positioning the 3D Spatial Feature Descriptor (3D SFD) after the Linear Group RNN, as placing it beforehand could potentially better preserve spatial information.

Thanks! We provide the corresponding results of different placements as following table. For Placement 1, we place the 3D SFD after voxel merging. For Placement 2, we place 3D SFD before the voxel merging. We agree with your explanation that placing 3D Spatial Feature Descriptor beforehand could potentially better preserve spatial information. Finally, we will add this discussion of the rationale for positioning the 3D Spatial Feature Descriptor in the revised version.

W3: An ablation study on Voxel Merging and Expanding would help quantify their contributions to the framework's performance.

Thanks! Note that removing the voxel merging and voxel expanding operations means that the input voxels of the linear group RNN are not processed by any downsampling or upsampling operations, which results in some additional computational cost. In the following table, we provide the experiment of our method with/without the voxel merging and voxel expanding operations. It can be observed that our method with adopting the voxel merging and voxel expanding operations could effectively reduce the computational costs while maintaining superior performance.

Q1: Since objects-of-interest in autonomous driving is relatively small comparing to the whole scene, why long-range relation (K=4096, 2048, ...) is important (line 133 - 135) in this scenario?

Good question! In fact, capturing the long-range relationship is helpful to obtain richer context information for better understanding the whole scene. Therefore, it is not only important for detecting large objects, but also for detecting small objects.
In this paper, we build the long-range relationship by the linear group RNN operations. We provide the ablation studies of different group sizes in the following table. Here, we set a minimum group size of 256 for all our four LION blocks (Baseline: [256, 256, 256, 256]). We could observe that the manners with larger group size (i.e, II, III, IV, V) bring consistent performance improvement over the baseline (I). However, we carefully find that there is a drop in performance from IV to V by further enlarging the group size. This might lead to less effective retention of important information in excessively long sequences due to the limited memory capacity of linear RNNs. Finally, to better understand the long-range relationship, we provide the visualization in Figure 1 (Please refer to our uploaded PDF file). We will add these discussion and experiments in the final version.

Dear AC and Reviewers,

Thank you for taking the time to review our manuscript. We are pleased to see that the reviewers have recognized the strengths of our work. Specifically, all reviewers (Reviewers ehGN, wvps, 5dKF and 9fJS) appreciate the performance of our LION in realizing superior performance on Waymo and nuScenes. In addition, Reviewer wvps considers the problem we have studied in this paper as important since the long distance relation is critical in point cloud detection. Reviewer 5dKF finds the SOTA performance of LION achieved based on Linear group RNN as very interesting. Reviewer 9fJS thinks that LION is an early attempt of linear group RNN for 3D detection. Finally, both Reviewer 5dKF and Reviewer 9fJS think that our method proves the generality of the LION by supporting different linear operators (RetNet, RWKV, Mamba).

However, some reviewers also have some concerns:

1. Reviewer ehGN has concerns why long-range relation is important.
2. Reviewer wvps has concerns about novelty.
3. Reviewer 5dKF hopes that the authors should report the results of our LION with multi-frames on the Waymo validation and test set.
4. Reviewer 9fJS thinks that the ablation for larger group size should be provided since it is a main claim

We have provided detailed responses to these concerns. Thank you again for your time and consideration!

Best regards,

All authors