FusionFormer: A Multi-sensory Fusion in Bird's-Eye-View and Temporal Consistent Transformer for 3D Object Detection

Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1 More experiments

Only one dataset is used in this paper. The generality of the framework should be analyized.

Author response:
It is worth noting that previous studies on multimodal fusion have primarily focused on the nuScenes dataset, and there is little research conducted on other dataset, such as Waymo. This may be attributed to the differences between the datasets since the Waymo dataset primarily consists of LiDAR data, and the available images predominantly capture a forward-facing view while lacking rearward images.

We are trying to evaluate the generalizability of our method on the Waymo dataset and other tasks such as segmentation. The experiments are still ongoing. We will include relevant experimental results in the final version of our paper.

Q2 Compared with CMT

In introduction, Fig.1(b) is not compared with their method in details.

Author response:
Thank you for your feedback. We have added a more detailed comparison in Figure 1(b).

We also list the key differences with CMT here:

• CMT tokenizes image and LiDAR features directly and uses a transformer decoder to generate object detection predictions, while our method generates fused BEV features that are then connected to the head for prediction.
• CMT uses compressed BEV lidar features, while we preserve the Z-axis information by using voxel features.
• CMT have challenges in achieving long term temporal fusion, while our method supports long term temporal fusion.

Q3 Single frame BEV feature result

The most baselines only consider LiDAR and camera, which are not fair to compare. Can you compare with them without temporal information? And this can also further verify the superiority of Multi-modal fusion encoder.

Author response:
We present the results as requested on nuScenes test set. Note that we do not use any test-time augmentation. Our single-frame FusionFormer (FusionFormer-S) achieves competitive performance. Nonetheless, we emphasize we aim to propose a uniform architecture for multi-modal, temporal consistent architecture.

Performance comparison on the nuScenes test set. "L" is LiDAR. "C" is camera. "T" is temporal. The results are evaluated using a single model without any test-time-augmentation or ensembling techniques.

Q4 Multi-modal and temporal fusion

The two topics, i.e., multi-modal fusion and temporal consistence, are different, causing that the focus of this paper is unclear.

Author response:
Thank you for your comments. Both multimodal fusion and temporal fusion are currently hot topics in the field of autonomous driving research, with significant implications for improving perception accuracy and stability. This study aims to present a unified Transformer-based multimodal temporal fusion framework. Our proposed Transformer-based multimodal fusion encoding module adapts to various multimodal feature inputs, resulting in fused BEV features. Our approach outperforms fusion methods such as BEVFusion in terms of multimodal fusion performance. Furthermore, compared to previous transformer-based methods that do not support temporal fusion such as CMT and DeepInteraction, our approach, based on the proposed temporal fusion module, is capable of supporting temporal fusion and achieves state-of-the-art performance on the nuScenes dataset.
In conclusion, we introduces a unified Transformer-based multimodal temporal fusion framework that achieves state-of-the-art results on the nuScenes dataset.

Q2 Clarification of the novelty

Lack of novelty. The proposed framework has a large overlay with both DeepInteraction and Bevformer. The BEV grid representation, temporal BEV feature fusion, and deformable attention have already been used in Bevformer and the cross-modality interaction of 2D features and 3D features have also been proposed in DeepInteractions. I would encourage the author to highlight the novelty and differences.

Author response:

We have clarified our novelty and the difference with SOTA methods in the general response. And we detail the response to your question here. We also take this opportunity to emphasize the main contributions of our paper here:

• We advocate using point cloud features in voxel format as opposed to using compressed LiDAR BEV features along the Z-axis. The recent approaches, BEVFusion, DeepInteraction and CMT, rely on LiDAR BEV features which is compressed along the Z-axis. Compared to fusing LiDAR features in the form of BEV features, our voxel feature fusion approach exhibits remarkable improvements in both object center position prediction and orientation prediction accuracy as shown in Table 5 in our paper (we provide the table here for your convenience).

Study for the representation of the LiDAR feature on the nuScenes val-set, as in Table.

• Based on our voxel feature fusion approach, we have discovered that it is possible to enhance visual-based 3D object detection by replacing LiDAR features with voxel features generated from monocular depth predictions. This novel idea provides a new direction for improving the performance of visual-based 3D object detection.

Results of camera based 3D detection on the nuScenes val set.

• We have proposed a unified multimodal temporal fusion method based on Transformers, which has achieved a new state-of-the-art performance on the nuScenes dataset.

We apologize for any misunderstanding, however, we again list the key differences between our method with BEVFormer & DeepInteraction here:
The difference between our method with BEVFormer

• BEVFormer only supports single-modality visual input. In contrast, our method focuses on providing a novel multimodal fusion solution.
• BEVFormer adopts a recurrent-based temporal fusion approach that does not support long-term temporal fusion. In contrast, our method efficiently achieves long-term temporal fusion.

The difference between our method DeepInteraction

• DeepInteraction focus on the interaction between multimodal features, while our method focuses on modal fusion.
• DeepInteraction preserves the original form of each modality feature after interaction, while our method generates fused features.
• DeepInteraction uses compressed BEV lidar features, while we preserve the Z-axis information by using voxel features.
• DeepInteraction relies on both modal inputs simultaneously, while our method exhibits robustness in the presence of missing modalities.
• DeepInteraction is not easily applicable for temporal fusion, while our method supports temporal fusion.

Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1 Differences with DeepInteraction

The motivation is not clear or sufficient. The author claims that " state-of-the-art multi-modality frameworks need explicitly compressing the voxel features into BEV space" and the proposed approach is aimed to address this issue. However, existing works like DeepInteraction maintain per-modality representations (2D perspective for camera and voxel space for LiDAR) and learn the interactions between these cross-modality interactions without the need for BEV intermediate representations. What's the difference between the proposed approach and DeepInteraction? Please fully discuss the differences and highlight the novelty of the proposed methods with SOTA methods.

Author response:

We have clarified our novelty and the difference with SOTA methods in the general response. And we detail the response to your question here.

We apologize for any misunderstanding, however, we again list the key differences with DeepInteraction here:

• DeepInteraction focus on the interaction between multimodal features, while our method focuses on modal fusion.
• DeepInteraction preserves the original form of each modality feature after interaction, while our method generates fused features.
• DeepInteraction uses compressed BEV lidar features, while we preserve the Z-axis information by using voxel features.
• DeepInteraction relies on both modal inputs simultaneously, while our method exhibits robustness in the presence of missing modalities.
• DeepInteraction is not easily applicable for temporal fusion, while our method supports temporal fusion.

Thanks for the detailed feedback. I am still not convinced about the novelty part. From the current response, the voxel representation is novel and distinct with deepinteraction and bevformer. However, all the other argued novelty in comparisons are basically the differences between deepinteraction and bevformer which the proposed method builds upon. I would encourage the author to highlight the differences between the proposed method and deepinteraction&bevformer.

Thanks for the response. Previously you are comparing proposed method vs Bevformer and then compare the proposed method vs Deepinteraction. However this is not sufficient for highlighting the novelties. I would encourage the author to list the differences between the proposed method vs (Bevformer+DeepInteraction). In other words, Bevformer and DeepInteraction is treated as a group. You should highlight the differences and novelties that neither BEVFormer nor DeepInteraction have. Otherwise, still it is unclear if the proposed method is just a combination of these two papers. For example, temporal fusion is not included in deepinteraction but is included in bevformer and if the temporal fusion is the same as the bevformer, then I think the technical contribution and novelty would be limited.

Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1 The technical contribution

The core idea of utilizing BEV queries for temporal and cross-modality fusion is widely used in previous methods like BEVFormer. Although the proposed method is different in detailed design, the core application of BEV queries is unchanged. This harms the technical contribution of the proposed method.

Author response: We first clarify our novelty in the general response. And we detail the response to your question here.

We apologize for any misunderstanding, however, we again list the key differences with BEVFormer here:

• BEVFormer is a visual only method. As such, it is not possible for them to use BEV queries to fuse cross-modality information.
• BEVFormer adopts a recurrent-based temporal fusion approach that does not support long-term temporal fusion. In contrast, our method efficiently achieves long-term temporal fusion.

We also take this opportunity to emphasize the main contributions of our paper here:

• We advocate using point cloud features in voxel format as opposed to using compressed LiDAR BEV features along the Z-axis. The recent approaches, BEVFusion, DeepInteraction and CMT, rely on LiDAR BEV features which is compressed along the Z-axis. Compared to fusing LiDAR features in the form of BEV features, our voxel feature fusion approach exhibits remarkable improvements in both object center position prediction and orientation prediction accuracy as shown in Table 5 in our paper (we provide the table here for your convenience).

Study for the representation of the LiDAR feature on the nuScenes val-set, as in Table.

• Based on our voxel feature fusion approach, we have discovered that it is possible to enhance visual-based 3D object detection by replacing LiDAR features with voxel features generated from monocular depth predictions. This novel idea provides a new direction for improving the performance of visual-based 3D object detection.

Results of camera based 3D detection on the nuScenes val set.

• We have proposed a unified multimodal temporal fusion method based on Transformers, which has achieved a new state-of-the-art performance on the nuScenes dataset.

Q2: Runtime comparison of module.

The runtime comparisons are missing. Because this work incorporated several attention modules in different encoders, it's essential to report the latency of each module.

Author response:
We have added a time analysis for different modules, and the results are presented in the table below on a single A100 GPU. The time for the multimodal fusion encoding module represents the total time for 6 layers of encoding, while the time for the temporal fusion encoding module represents the total time for 3 layers of encoding. The time for each attention module represents the time for a single module in a single layer of encoding.

Nonetheless, we already provide the runtime analysis against other baselines in Table 7 of Appendix. We replicate this table here for your convenience. Our method has a competitive runtime compared to those method.

Appendix Table 7 Efficiency comparison on the nuScenes val set. ”L” is LiDAR. ”C” is camera. ”T” is temporal. The ”-S” indicates that the model only utilizes single-frame BEV features without incorporating temporal fusion techniques.

Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1 More experiments.

The proposed method is only tested in one dataset (nuScenes) and one task (3D object detection), which may not be enough to show the generalizability of the proposed sensor fusion scheme. It would be better to include more datasets (Waymo) and tasks (e.g., segmentation)

Author response:
It is worth noting that previous studies on multimodal fusion have primarily focused on the nuScenes dataset, and there is little research conducted on the Waymo dataset. This may be attributed to the differences between the datasets since the Waymo dataset primarily consists of LiDAR data, and the available images predominantly capture a forward-facing view while lacking rearward images.

We are trying to evaluate the generalizability of our method on the Waymo dataset and other tasks such as segmentation. The experiments are still ongoing. We will include relevant experimental results in the final version of our paper.

Q2 Complexity comparison with other baselines

Though the proposed method makes use of the z-axis information, it seems to greatly increase the model complexity. To make fair comparisons with other baselines, the authors may better include complexity analysis such as FLOPS, #of parameters, and FPS

Author response:
Due to the constraints on the page limit of the main text, we present the complexity efficiency comparison in Appendix A.2.

As shown in Table 7, we compare the efficiency of FusionFormer and existing methods. The efficiency and performance are tested on a single Tesla A100 GPU with the best model setting of official repositories. In comparison to BEVFusion, FusionFormer demonstrates superior performance with notable improvements of 3.1% in mAP and 2.7% in NDS, while maintaining a similar processing speed.

Appendix Table 7 Efficiency comparison on the nuScenes val set. ”L” is LiDAR. ”C” is camera. ”T” is temporal. The ”-S” indicates that the model only utilizes single-frame BEV features without incorporating temporal fusion techniques.

In addition, we conducted a comparative experiment on computation cost between our method and the previous state-of-the-art method, CMT. As shown in Table 8, our method has similar FLOPS and parameters as CMT.

Appendix Table 8 Computation cost comparison on the nuScenes val set. ”L” is LiDAR. ”C” is camera. ”T” is temporal. The ”-S” indicates that the model only utilizes single-frame BEV features without incorporating temporal fusion techniques.

Q3 Marginal improvement

From Tables 1 and 2, the proposed method only provides a marginal improvement.

Author response:
The nuScenes 3D object detection leaderboard has currently reached a high level. In this scenario, our method still achieves overall performance improvement and significant improvement in certain aspects.

• We achieves significant improvements in terms of reducing the orientation error（mAOE）and classification error (mAAE). Our method reduces the mAOE and mAAE compared to the previous state-of-the-art methods by 7.1% and 6.3%, respectively. These two errors are often considered challenging in point cloud-based detection. This indicates that our method is capable of better fusing image features to compensate for the shortcomings of point cloud features.
• Our method exhibits better detection performance in scenarios where point clouds are sparse, particularly over long distances. This superior performance can be observed in Figure 6(b) of our paper.

In summary, our method successfully achieves unified multimodal temporal fusion and delivers superior detection performance.