TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning

The reviewer expresses gratitude to the authors for their feedback; nevertheless, there is a lingering issue as some of his concerns remain unaddressed.

Q1. The progress in points is somewhat limited compared to previous approaches such as DistillBEV. He maintains that this paper presents only a narrow scope of new insights, as emphasized by Reviewer oss3. Regarding the feature distillation component, its characteristics did not appear markedly distinct from those of DistillBEV. Instead, it seemed predominantly foregrounded, albeit in a divergent manner. Besides, through ablation experiments, it has been ascertained that the depth distillation module yields only marginal improvements.

Q2. The reviewer acknowledges the inclusion of new references to established works. However, there are still some works not included as follows.

[1]. BEVSimDet: Simulated Multi-modal Distillation in Bird's-Eye View for Multi-view 3D Object Detection.

[2]. Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection.

[3]. Distilling Focal Knowledge From Imperfect Expert for 3D Object Detection.

[4]. BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object Detection.

Thank you for your time and effort.

Q1. Compare with other related works in the R50 framework.

Thank you for your suggestion. We have shown the comparison in General Response to All Reviewers. We are sure that our comparison is fair. According to the Reviewer Policy, we have reported the results of several methods using the LiDAR->Camera mode on ResNet50 and published in top-tier conferences. As there is a lack of consistency in the baselines reproduced by everyone, we have also included the baseline in the table for comparison. It is evident that our method performs competitively. We hope it can provide a clearer understanding.

Q2. The comparison is biased?

We apologize for any confusion and misunderstanding that we may have caused you. We need to state that we place great emphasis on comparing fairly and we have enough confidence to ensure that our comparison is fair.

• Firstly, BEVDistill also used two distillation modules (BEV Feature Level and Instance Response Level) proposed in the paper for the BEVDepth experiment, which is referred to in Appendix B.3 in BEVDistill. These two modules are silimar to distillation methods in 2D object detection and are proved to be generalizable. The author adapted instance-level distillation on two different frameworks (bevdepth and bevformer), while bev feature level distillation is generalizable, like our inner-feature distillation. Moreover, as shown in Table 7 in our paper, we also list the baseline results and show the improvement respectively to make a fair comparison.

• Secondly, both BEVDistill and our method used the same and common data augmentation for training, such as image flipping, scaling, etc., and neither of us used other augmentation methods such as TTA for test set. Therefore, there is no unfair comparison here.

• Thirdly, we are a little confused about your comment. We are not quite sure which paper's 'Table 2' you are referring to (BEVDepth/BEVDistill/TiG-BEV?). The Table 2 in our paper shows the NuScenes Test Leaderboard result, all participants do not need to, and are not allowed to provide checkpoints for it. While the results we provide for the baseline are consistent with the results reported by BEVDistill, because our implementation of the baseline method is also completely consistent. We all use the BEVDet_v1.0 codebase to reproduce the result of the BEVDepth Baseline. What's more, the config of the baseline we used was given by the author of BEVDistill, we had ever asked for it. So our baselines for test leaderboard are the same and we also found that our results reproduced by the same config were also the same. Therefore, from Table 2, we can see the superiority of our method clearly and it's also completely fair.

Q3. Report results based on long-term temporal methods.

Thanks for your suggestion. We did not specifically design for long-term setting before. In order to show results based on long-term setting, we trained a LiDAR Teacher with BEVDet_v2.0 codebase [1]. Due to resource limitations, our teacher model appears to be not powerful enough (mAP:49.7/ NDS:52.3), but it still can be effective. Without any tuning, we were still able to further improve the BEVDet4D-Depth on the long-term setting (8 + 1 frames) as shown in the table.

Note: For the table, we use ResNet50 as the backbone, the resolution is 256 $\times$ 704.

       

Overall, thank you for bringing your concerns to our attention. We hope that our response has effectively addressed them and cleared up any misunderstandings.

Reference:

[1] https://github.com/HuangJunJie2017/BEVDet/tree/dev2.0

Hi Reviewer,

This paper obtained the mixed opinion and I see your comments carefully. I wonder whether there are some misunderstandings in this paper. So could you give the updated comments given the author's rebuttal?

It might be helpful for us to score this paper.

Many thanks,

AC

Thanks for your recognition and valuable feedback.

Q1. The proposed 'inner-depth supervision' is a object-normalized version of the absolute depth from BEVDepth?

We are very sorry for the confusion. We propose the inner-depth supervision module, which is mainly motivated by focusing on the internal relationships of foreground objects. It's not a trick. The introduction of relative depth is different from normalization which amplifies the differences between similar depth clusters in a certain area. It does not involve any normalization operations. Insteadly, it establishes a connection between the depths of the internal points of foreground objects, rather than predicting the depth of each point in isolation.

Absolute depth supervision ignores the implicit relationships between points, while our inner-depth supervision aggregates the depths of points belonging to the same object, allowing the model to learn the relationships between internal points of objects and project object features more accurately into the BEV space.

A more intuitive visualization after inner-depth supervision can be seen in Appendix B.2 (Fig 8), and even the depth prediction of the edge has become clearer.

Q2. The channel-wise and pixel-wise relationship supervision has been investigated by previous works?

Thanks for the additional references, we have added an extra paragraph of these related works in Appendix A according to your kind advice.

We agree that some related work has explored the relationship supervision between features in the fields of knowledge distillation and style transfer. These involve knowledge distillation between teachers and students of different scales or knowledge transfer between different domains, but few works have been done on different modalities of teachers and students, where one feature is learned from 2D data and the other from 3D data. Models under these two different modalities not only differ in model parameters and feature domains, but also in data format and source, which poses some difficulties that are not present in normal relationship learning.

Besides, in normal knowledge distillation, direct feature alignment can be a good auxiliary method, but in our setting, differences between modalities can be harmful. Therefore, our work has conducted in-depth exploration on this issue. From the experimental results, it is evident that our method is feasible and can significantly outperform direct feature alignment, which not only meets expectations but also makes sense.

In terms of specific design, we have made detailed designs through extensive trials. We did not learn the relationship of the entire feature map, because a large amount of background noise can cause interference and heavy computational burden. Through exploration, we applied relationship supervision to foreground objects which is more efficient and can bring better improvement. Moreover, we used uniform sampling points to learn the relationship within foreground objects to save computational costs. In addition, we also found that learning both spatial correlation and channel-level correlation can complement each other. Finally, we found that learning feature correlation implicitly depends on previous depth prediction. The inner-depth supervision we used better constrained the feature projection at the foreground level, which further assisted in the learning of inner-feature. The combination of these two modules can achieve better results. Thus, our paper proposes a method that integrates and complements each other.

Thanks for your constructive and highly detailed comments that help us a lot.

Q1. Compare with other related works and discuss differences.

Thanks you very much for this suggestion, we make the comparison in General Response to All Reviewers, please refer to it for a more clear understanding. In the table, we list three relevant related works published in top-tier conferences and discuss our differences among these works.

Q2: How generalizable the method is?

It's an interesting topic. Your inquiry has provided valuable insights, motivating us to reevaluate the contributions of our method and explore potential synergies with other approaches.

The framework of the DETR-like series leverages implicit depth information and position encoding. In this paradigm, learnable queries, preset on the BEV grid, are employed to seek relevant image features based on their spatial location in the BEV space. This approach contrasts with the LSS-like methodology, where explicit depth information supervises the process, projecting image features onto the BEV. It's noteworthy that our supervision includes the relative depth within foreground objects, constituting a form of explicit guidance for depth prediction.

However, the DETR-like approach encounters challenges when querying image features due to occlusion and other phenomena in the image. This often leads to inconsistencies between the detected image features and their counterparts in the BEV space, resulting in occasional false detections. Addressing these issues with implicit depth information remains a challenge.

In recent developments, we've observed innovative approaches, such as FB-BEV [1]. specifically, FB-BEV ultilizes BEVFormer with depth-aware module, like BEVDepth, integrates LSS-like module for explicit depth supervision to enhance DETR-like module. This method aims to harness the complementary strengths of both methods, yielding a more accurate BEV representation.

Similarly, we recognize the potential for enhancing the DETR-like approaches by incorporating our method and our experimental validation of this concept has yielded promising results. Essentially, our method acts as a plug-and-play module that seamlessly integrates with and improves the DETR-like approach by utilizing auxiliary inner-depth information, akin to the approach outlined in FB-BEV.

Note: For the table, we use ResNet50 as the backbone, the resolution is 256 $\times$ 704. We don't use temporal information due to limited resources.

Q3: Minor comments and typos.

We sincerely apologize for any writing issues in the previous version. We have diligently addressed all typos and continued fine-tuning the paper's writing to enhance its overall quality. Our team is actively engaged in preparing for the final submission, and we are committed to delivering a polished and refined manuscript. Thank you for your patience and guidance.

(1) The upper index A in the depth loss?

The upper index A in the depth loss means "Absolute", we use absolute depth supervision like BEVDepth. The loss was named with the first letter A of Absolute to correspond and distinguish it from the Relative inner-depth supervision loss we proposed.

(2~5) A bit confusing to talk about 3d and 2d features; Include the math notation also in the Fig 4; Increase the font size of Fig 5,6 a bit; Report resolution and backbone in Tab 2.

Sorry for your confusion and thank you for your suggestion. We have made modifications according to it and updated in the latest manuscript.

(6) Include the latest works on multi-camera 3DOD in Tab 1,2.

We have updated Table 2 in the latest manuscript. For the sake of fairness in comparison, we have included the detailed table in Appendix B.2 (Table 8), as we did in the General Response to All Reviewers.

(7) Verify the method on more than one dataset.

As for other datasets, we have already shown the performance of our method on KITTI in Appendix B.2 (Table 10).

If you have any other suggestions, we welcome and appreciate your continued feedback.

We sincerely appreciate your recognition of our motivation, and your valuable feedback serves as a tremendous source of encouragement to us. We apologize for any concerns that may have arisen, and we are committed to addressing them.

Q1. Experimental analysis on other 3D dataset.

We are very sorry for the misunderstanding. We have conducted relevant experiments on the KITTI dataset before which can be found in Appendix B.2 (Table 10). With our proposed method, we can achieve a higher 3D AP compared to CMKD [1].

Q2. Illustrate the performance of small and far object detection.

Thank you for your suggestions. We can illustrate the performance of small and far object detection from two perspectives:

• For qualitative analysis, there are more visualization results shown on Figure 9 in Appendix B.2. These supplementary visualizations also serve to demonstrate that our proposed method is capable of mitigating a range of issues in detection, including the reduction of false positives and ghosting objects, as well as the refinement of certain 3D locations and orientations of bounding boxes. These benefits also apply to small and far objects.

• For quantitative analysis, we conduct some additional studies and take BEVDet4D-R101 for an example.

  • We consider small objects such as pedestrians, motorcycles, bicycles, traffic cones, and barriers, and evaluate the performance of small object detection by comparing the mAP across these categories:

    Baseline	$\mathcal{L}^A_{\rm{depth}}$	$\mathcal{L}^R_{\rm{depth}}$	$\mathcal{L}_{\rm{bev}}$	mAP$_{pedestrian} \uparrow$	mAP$_{motorcycle} \uparrow$	mAP$_{bicycle} \uparrow$	mAP$_{traffic\ cone} \uparrow$	mAP$_{barrier} \uparrow$
    BEVDet4D-R101				43.3	34.5	32.4	54.8	55.2
    BEVDet4D-R101	$\checkmark$			44.6	35.6	32.8	56.3	55.9
    BEVDet4D-R101	$\checkmark$	$\checkmark$		45.4	37.3	33.7	57.8	58.8
    BEVDet4D-R101	$\checkmark$	$\checkmark$	$\checkmark$	45.7	38.8	37.5	58.9	57.2

    As demonstrated in the table, our proposed method enables more accurate detection of small objects.

  • We use the distance between the object and the ego in the ego coordinate system to filter out far objects, and evaluate the performance of far object detection by comparing the mAP across different distance ranges:

    Distance (m)	Baseline	$\mathcal{L}^A_{\rm{depth}}$	$\mathcal{L}^R_{\rm{depth}}$	$\mathcal{L}_{\rm{bev}}$	mAP $\uparrow$	NDS $\uparrow$
    [0,30)	BEVDet4D-R101				44.2	53.7
    [0,30)	BEVDet4D-R101	$\checkmark$			46.6	55.6
    [0,30)	BEVDet4D-R101	$\checkmark$	$\checkmark$		47.4	55.8
    [0,30)	BEVDet4D-R101	$\checkmark$	$\checkmark$	$\checkmark$	48.6	56.8
    [30,60)	BEVDet4D-R101				10.2	28.7
    [30,60)	BEVDet4D-R101	$\checkmark$			10.3	28.5
    [30,60)	BEVDet4D-R101	$\checkmark$	$\checkmark$		11.7	30.1
    [30,60)	BEVDet4D-R101	$\checkmark$	$\checkmark$	$\checkmark$	12.6	30.2

    As shown in the table, our method can improve detection performance even for distant objects.

Q3. Compare with other LiDAR-to-Camera Methods.

Please refer to General Response to All Reviewers, we list some latest works in the table. We also compare with CMKD as said in Q1. We're sorry for any concern this may have caused you.

Reference:

[1] Hong Y, Dai H, Ding Y. Cross-modality knowledge distillation network for monocular 3d object detection[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 87-104.

Thank you for your positive and thorough review of our paper. We greatly appreciate you taking the time to provide such thoughtful feedback.

Q1: Comparisons with state-of-the-art techniques.

Thank you for your advice. We have made a comparison with other state-of-the-art techniques in General Response to All Reviewers. Besides, we also found that our method is not only highly competitive in 3D detection tasks, but camera-based models pre-trained with our method can also serve as better backbones for other 3D camera-only tasks, such as 3D occupancy prediction task.

Q2: An in-depth discussion.

Thank you for your valuable insight and we have added it in Appendix B.4.

• One limitation that will inevitably be involved in depth-related explicit supervision is the ground truth from LiDAR. Due to the sparsity of LiDAR, both absolute depth supervision and relative depth supervision will be affected to some extent, though the performance of object detection will be better with inner-depth supervision under the same conditions. For sparse point clouds, a good way to alleviate it is through depth completion.

• Another limitation is the occlusion problem of the target. Due to visual occlusion, the ground truth of the occluded object that we can obtain is limited, which will affect the learning of depth prediction and inner-depth learning. A good way to alleviate it is to consider multi-frame images for inner-depth supervision of the same object's interior, but a potential risk here is that if the intrinsic and extrinsic parameters are inaccurate, it will directly affect the coordinate system transformation between multiple frames, thereby affecting the ground truth of depth values.

Q3: Evaluated method on other datasets.

Thanks for your suggestion. We have evaluated our method on KITTI before and results can be seen in Appendix B.2 (Table 10).

       

We also have further added details, refined our approach and conducted new experiments to address concerns raised by other reviewers. The revisions have improved the quality and clarity of our work. We hope that you will find our changes satisfactory and look forward to hearing your assessment of the revised manuscript.

Differences among recent works (1)

Compared with methods of recent works, we would like to claim that:

(1) Our motivation mainly focuses on a better LiDAR-to-Camera learning scheme to effectively leverage LiDAR modality from both the low level (inner-depth supervision) and the high level (inner-feature supervision), which well relieves the modality gap.

(2) Our inner-depth provides fine-grained depth cues for object-level geometry understanding, which can implicitly benefits BEV representation learning.

(3) We pay attention to the relationships inside the foreground objects by inner-feature supervision, while most of the existing work focuses on BEV feature alignment and response distillation, similar to BEVDistill.

• For BEV feature distillation, most existing methods choose to use strict feature alignment. In our paper, we illustrate that forcing alignment of features from two different modalities is not ideal, as it will be affected by modality differences.

• Some techniques like response distillation are not the main focus of our motivation, so we did not specifically use them although the improvement is promising.

• We completely abandon strict feature imitation and instead learn the knowledge contained in the LiDAR-based teacher through modeling the internal relationships of foreground targets, allowing the camera-based student model to learn a better BEV representation.

(4) Our extensive experiments have shown that the improvement brought by feature alignment is limited, while distillation through learning inner-feature relationships is superior.