DH-Fusion: Depth-Aware Hybrid Feature Fusion for Multimodal 3D Object Detection

We would like to express our sincere gratitude to you for the valuable comments and constructive feedback. Below, we address each each of question or comment in detail.

Comment 1: "Lake of Novelty: The Depth Encoder in DH-Fusion is similar to the 3D Position Encoders in PETR (PETR: Position embedding transformation for multi-view 3d object detection). The Depth-Aware Global Feature Fusion (DGF) module and Depth-Aware Local Feature Fusion (DLF) module in DH-Fusion are analog to the Hierarchical Scene Fusion (HSF) module and Instance-Guided Fusion (IGF) module in IS-Fusion (IS-Fusion: Instance-scene collaborative fusion for multimodal 3d object detection). In conclusion, the contribution of this work seems like "A+B," which is limited."

Response 1: We disagree. In this work, we for the first time point out that depth is an important factor to consider during feature fusion by providing insightful analysis. This new finding makes our method differ from previous fusion methods fundamentally. On one hand, our proposed depth encoder is used to adaptively adjust the weights of features from different modalities at different depths; it is conceptually different from 3D Position Encoders in PETR, which are used to relate 2D image features with reference point positions for obtaining 3D features. On the other hand, our proposed DGF and DLF modules perform feature fusion in an adaptive way, i.e., the RGB/point cloud features are assigned with different weights as depth varies; such a dynamic mechanism is novel and has not been used by IS-Fusion or previous fusion methods. Therefore, we believe our method is novel, and the provided insights are inspiring for the community.

Comment 2: "For the nuScenes test leaderboard, DH-Fusion achieved a Top 10 ranking only with 384x1056 image size and SwinTiny backbone. Please provide results when using larger 900x1600 image size and ConvNeXtS backbone."

Response 2: Thank you for your feedback. We provide a more comprehensive evaluation of the performance of DH-Fusion with different image encoders, including using larger 900x1600 image sizes and the ConvNeXtS backbone. We observe a further improvement while using a larger backbone and image size, indicating the scalability of our method.

Table 1: Performance with different image encoders

Comment 3: "Please analyze the theoretical reasons for DH-Fusion's robustness advantage against various corruptions, as in nuScenes-C."

Response 3: DH-Fusion’s robustness against various data corruptions in the nuScenes-C dataset is attributed to the combined effects of depth encoding and cross-attention. On one hand, depth encoding is helpful in the way of allowing the model to dynamically weigh features as depth varies. For example, on foggy days, those objects at a far distance are more invisible on RGB images than those at a near distance, and thus the RGB features should be down-weighted at a far distance. On the other hand, the cross-attention mechanism allows DH-Fusion to dynamically focus on the most relevant features and suppress those ineffective features caused by corruptions across modalities at global and local levels. In this way, it reduces the negative impact of corruptions to some extent.

Comment 4: "In Table 1, for experiments on the nuScenes dataset, It's necessary to include metrics like mATE, mASE, mAOE, mAVE, and mAAE as done in other papers."

Response 4: Thank you for your valuable suggestion. We will provide the results of our method on the metrics mATE, mASE, mAOE, mAVE, and mAAE.

Table 2: Results on nuScenes test set

Table 3: Results on nuScenes validation set

Comment 5: "Do the authors plan to release the code and provide pre-trained, especially nuScenes test leaderboard models for further research?"

Response 5: We appreciate the reviewer's interest in our work and the potential for further research using our models. We plan to release our code and pre-trained models, including those used for the nuScenes test leaderboard, upon the acceptance of our paper.

Thank you for your continued feedback. We would like to highlight that our method balances both accuracy and efficiency. Specifically, our DH-Fusion-light is the first to achieve over 10 FPS on a RTX 3090 GPU, while maintaining comparable accuracy, thereby meeting the requirements for real-time applications. Additionally, while the latest methods we compared are evaluated solely on the nuScenes dataset, we have further evaluated our methods on nuScenes-C, providing a more comprehensive evaluation. Furthermore, as reviewer qiTv also noted, the datasets we employed are sufficient to demonstrate the effectiveness of our approach.

We would like to express our sincere gratitude to you for the valuable comments and constructive feedback. Below, we address each of question or comment in detail.

Comment 1: "How about the algorithm's performance on small object detection? small object could be normal-sized object at far distance or small-sized object at near distance, is it possible that the proposed depth-aware module hurts the detection performance of small-sized object at near distance? since according to Figure 5, LiDAR modality will have relatively larger weights at near distance, but it is in low resolution, so not good for small object detection."

Response 1: Thank you for raising this important point. To address your concerns, we consider cars as normal-sized objects, and pedestrians, motorcycles, and bicycles as small-sized objects. We conduct experiments to evaluate our method on normal-sized objects at far distance and small-sized objects at near distance. For these above small objects, our method outperforms the state-of-the-art method IS-Fusion, as well as our baseline BEVFusion, demonstrating our robustness for small object detection. It is true that LiDAR modality has larger weights at near distance, as it is in high resolution as shown in Fig. 1(b) of the original submission, so it does not hurt but in fact helps small-sized object detection at near distance.

Table 1: Performance on small objects, including normal-sized objects at far distance (>30m) and small-sized objects at near distance (0-20m). The numbers are AP.

Comment 2: "Compare with SOTA, the achieved performance improvement is not that significant. As shown in Table 1, the performance gap between the proposed method and IS-Fusion is small and IS-Fusion even achieves slightly better mAP, it is not clear whether the proposed method can achieve similar performance improvement as indicated in ablation study when using IS-Fusion as baseline."

Response 2: Thank you for your feedback. To address the concern, we conduct an experiment using IS-Fusion as the baseline, and integrate our depth encoder into its IGF module, which allows to adjust the weights of image features with depth during instance feature fusion. We note that our method still achieves improvements when applied to a stronger baseline. This demonstrates the generalization ability of our approach across different baselines. We plan to dedicate more time to refining our experiments in future work to achieve more significant performance improvements.

Table 2: Ablation studies using IS-Fusion as the baseline.

Comment 3: "In Figure 5(b), it would be good to add a color bar to indicate the magnitude corresponding to each color."

Response 3: Thank you for your valuable suggestion. We will add a color bar to Figure 5(b) in the final version of the paper to indicate the magnitude corresponding to each color.

Comment 4: "There is no paragraph explaining the weakness of the proposed method."

Response 4: We apologize for the oversight. We condense the discussion of our method's limitations into the conclusion due to limited space, and do not dedicate a separate section to it. We will address this in the final version of the paper.

Dear Reviewer kR8p

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best wishes

AC

We would like to express our sincere gratitude to you for the valuable comments and constructive feedback. Below, we address each question in detail.

Comment 1: "I noticed this paper encodes depth information using cosine functions, but I haven't seen experiments validating the impact of cosine functions. Would there be a significant performance drop if distances were used directly instead of cosine functions?"

Response 1: We appreciate the reviewer's question regarding the use of cosine functions for encoding depth information. We conduct the experiments using normalized depth directly as the depth encoding in our feature fusion module, without applying cosine functions. Our experimental results in the table below show a performance drop when using normalized depth directly. We believe that depth encoding benefits from the use of cosine functions to capture the periodicity and symmetry of the depth information relative to the ego vehicle. The cosine function helps in better representing the variations in depth, leading to model performance improvement.

Table 1: Ablation studies of cosine functions.

Comment 2: "There is no discussion of limitation in main text, but a justification is given in Checklist: using an attention-based approach to interact with the two modalities makes the detection results sensitive to modality loss."

Response 2: We apologize for the oversight. We condense the discussion of our method's limitations into the conclusion due to limited space, and do not dedicate a separate section to it. We will address this in the final version of the paper.

We would like to express our sincere gratitude to you for the valuable comments and constructive feedback. Below, we address each of question or comment in detail.

Comment 1: "What is the computational complexity of the DH-Fusion model, and how does it compare with other state-of-the-art methods in terms of runtime and resource usage?"

Response 1: Thank you for your question. The parameters of our DH-Fusion-light, DH-Fusion-base and DH-Fusion-large are 40.38M, 53.05M, and 56.94M, respectively. The flops of these models are 271.6G, 822.8G, 1508.2G, respectively. The runtime of these models are 72.46ms, 114.94ms, and 175.44ms on a 3090 GPU, respectively. In Table 1 of the original submission, we compare our method with other SOTA methods in terms of FPS. Specifically, under the same configuration, our DH-Fusion-light runs faster than BEVFusion, and achieves a real-time inference speed; our DH-Fusion-base maintains comparable inference speed, compared to FocalFormer3D; our DH-Fusion-large runs 2x faster than IS-Fusion.

Comment 2: "The authors only present results on nuScenes dataset. The algorithms should be also evaluated on other prevailing public dataset like KITTI; While the method shows strong performance on the nuScenes dataset, its generalizability to other datasets or varied real-world conditions might require further investigation."

Response 2: In our original submission, we have actually presented results on two datasets: the nuScenes dataset and the nuScenes-C dataset. The detailed results can be found in Table 1 and Table 2. In particular, our experiments on the nuScenes-C dataset, which includes various realistic noise conditions, show that our method exhibits high robustness under diverse real-world corruption conditions and very good generalization ability. Since the KITTI dataset provides only stereo images instead of multi-view images, our method cannot be directly applied on it, and we believe that the nuScenes dataset, providing 700 different scene sequences for training, 300 scene sequences for validation and testing, is sufficiently large and diverse, serving as a standard benchmark in the field of 3D object detection. The methods we compared [1, 2] against typically conduct their experiments solely on this dataset as well.

[1] Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D.L., Han, S.: Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation. In: ICRA (2023)

[2] Yin, J., Shen, J., Chen, R., Li, W., Yang, R., Frossard, P., Wang, W.: Is-fusion: Instance-scene collaborative fusion for multimodal 3d object detection. In: CVPR (2024)

Comment 3: "The depth-aware fusion might be tailored to the specific characteristics of the training dataset, potentially leading to overfitting and reduced performance on diverse or unseen data."

Response 3: No, our method is not tailored to any dataset. Specifically, our depth-aware fusion method adaptively adjusts the weights of different modalities based on depth, which is inherently independent of the specific dataset used for training. We acknowledge that the performance of our method on the nuScenes-C dataset, where we achieve 68.67 NDS and 63.07 mAP, shows a decrease compared to the nuScenes dataset. The reduced performance on nuScenes-C can be attributed to the increased difficulty of this dataset, which includes various corruptions. Despite this, our method still outperforms other approaches, demonstrating its robustness.

Comment 4: "While the paper includes ablation studies, a more extensive set of experiments that isolate the impact of different components of the system could provide deeper insights."

Response 4: We appreciate the reviewer's suggestion for a more extensive set of experiments to isolate the impact of different components of the system. In fact, we have already provided a detailed discussion on the impact of various components on the model in the paper. Specifically, section 4.4 thoroughly examines the influence of different components, including the effect of DGF and DLF, and the effect of the depth encoding.

Dear Reviewer tCBR

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best wishes

AC