VeXKD: The Versatile Integration of Cross-Modal Fusion and Knowledge Distillation for 3D Perception

1. Q1: Experiment on More Baselines

Thank you for your suggestion. We have conducted 3D object detection KD experiments on both BEVDepth and the temporally fused BEVFormer as student models. As shown in Table 1 attached to the global response, our KD framework was applicable to both models and improved their performance.

2. Q2: Modified Related Work

Thank you for your constructive suggestion and we have modified our related work. Due to space limitations and in order to avoid abusing the official comments, we abstract the modifications instead of posting the entire related work section.

• Adding different operations on utilization of feature distillation

DistillBEV[5] decomposes the region of feature maps and enhances attention to false positive regions. Recently, inspired by pretext tasks in large language models, masked generative distillation has been proposed [1]. Unlike attentive distillation, generative distillation masks part of the student’s feature map using a random mask. This masked map is then processed with a generator network to reconstruct feature maps that closely approximate the teacher’s, thus enhancing knowledge transfer. However, random masks can destabilize algorithms, especially in 3D object detection with pronounced foreground-background imbalance. Zeng et al. [2] address this by using a learned distillation head to predict coarse foreground boxes where random mask distillation is applied, focusing more on foreground regions to enhance stability. Our method, while inspired by masked generative distillation, aligns more closely with attentive distillation and eliminates the need for additional generator networks.

• The efficacy of KD

To boost KD efficacy by minimizing the modality gap, Huang et al [3] developed a "vision-centric" multi-modal teacher, reducing reliance on LiDAR model operations to align more closely with camera-based students. Conversely, SimDistill[7] adds a branch to the student model to simulate multi-modal processing, narrowing the teacher-student gap but increasing the student model's size and inference time, deviating from traditional KD goals. Our work focuses on developing a modality-general fusion model without altering the existing pipelines of either teacher or student networks.

• More cross-modal research on camera-based students

Recent works have advanced short-term [8, 9] and long-term [4] temporal fusion in multi-camera 3D perception. Zheng et al.[6] use a long-term temporal fusion teacher to impart temporal cues to a short-term memory camera student, while work [3] warps time series ground truth into the current timestamp for long-term temporal supervision.

3. W3: Novelty in Masked feature distillation

Thank you for your thought-provoking comment. Indeed, our work shares some similarities with FD3D, as both are inspired by the work Masked Generative Distillation [1].

Both [1] and FD3D [2] are generative distillation in nature, where certain feature positions are masked and reconstructed from other feature locations to achieve KD. FD3D further generates bounding boxes through a learned distillation head, ensuring that only features within the bounding box are involved. In contrast, our method utilizes masks to selectively choose features to align pixel-wise akin to attentive distillation. This requiring only an auxiliary network for generating spatial masks, thereby eliminating the need for additional complexities of reconstruction.

Our approach also differs in mask generation: FD3D is instance-based, using BEV queries to guide distillation within coarse bounding boxes. We have adapted our method to address the semantic complexities of the BEV space, initializing and learning dense BEV queries for fine-grained, pixel-wise mask generation that interacts directly with both student and teacher feature maps, independent of bounding boxes. This allows for greater generalization across various downstream tasks besides object detection.

While both our method and FD3D utilize learned masks to identify useful feature locations and filter out noisy ones, FD3D focuses primarily on model compression in single-modality scenarios. Our study, however, considers cross-modal KD scenarios, including how to incorporate more modality-general information within the teacher and performing attentive distillation of this fine-grained modality-general information through a collaborative mask learning process involving student and teacher.

In summary, compared to the mentioned work, the novelties of our BEV guided masked learning and distillation include:

1. Focusing on attentive over generative distillation.

2. Our method generates masks based on interactions between the learned dense BEV queries and feature maps of both student and teacher, enhancing finegrained feature extraction and adaptability to various tasks.

3. Our method explores cross-modal KD scenarios, aiming to enhance the teacher model with broader modality-general information and in return utilize the information contained in the learned BEV queries to perform mask guided selective attentive distillation.

References:

[1] Masked generative distillation

[2] Distilling Focal Knowledge From Imperfect Expert for 3D Object Detection

[3] Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection

[4] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection

[5] Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation

[6] Distilling temporal knowledge with masked feature reconstruction for 3d object detection

[7] Simdistill: Simulated multi-modal distillation for bev 3d object detection.

[8] Bevdet4d: Exploit temporal cues in multi-camera 3d object detection

[9] Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.

The authors have addressed most of my concerns. However, some issues remain. Notably, as mentioned in the weaknesses section, the state-of-the-art work VCD [3], which focuses on L+C->C, is not compared in Table 1.

Q1: CenterPoint+VeXKD (L+C->L) Compared to TransFusion-L (L)

Thank you for your question. To ensure fair comparisons with existing KD methods like S2M2-SSD and UniDistill, we chose CenterPoint as the LiDAR student model. CenterPoint inherently underperforms compared to TransFusion-L by 5.2 mAP and 7.1 NDS, largely because TransFusion-L utilizes a more advanced DETR head. However, as shown in the modified Table 1 attached to our global response, TransFusion-L operates nearly twice as slow as CenterPoint in terms of FPS, due to the time-consuming attention operations. By applying cross-modal KD to CenterPoint, while preserving its original fast inference speed, we brought its performance closer to that of TransFusion-L, thus achieving a more favorable balance between precision and real-time performance. Therefore, cross-modal KD does make sense for LiDAR-based student models.

Q2: The inference speed comparison

Thank you for your constructive feedback. As revised in the global response, we have added a comparison of the number of inference floating point operations(FLOPs) and required time for different models to Table 1. This adjustment allows for a clearer view of the performance-realtimeness tradeoff offered by knowledge distillation. Once again, we appreciate your suggestions. :)

Q3: Incorporation of Temporal Information

Thank you for your constructive questions. As clarified in the global response, our teacher model and LiDAR student models inherently incorporate multi-sweep LiDAR inputs, thus implicitly integrating temporal information. Inspired by your feedback, we conducted additional experiments with BEVFormer, a camera-based student model that explicitly incoporate temporal fusion operation, and observed performance gains, as detailed in the global response. As technologies for multi-modal explicit temporal fusion continue to mature and become open-sourced, we foresee the potential for integrating explicit temporal KD operations as a pluggable module into the VeXKD framework in future developments.

1. Q1: Adaptation on other feature representation

In our manuscript, the experiments were conducted on the BEV feature map. The BEV feature space has become a focal point of research in recent years due to its favorable compatibility with multiple modalities and its similar processing pipeline.

However, as long as the student and teacher models can achieve spatial alignment, the specific feature space in which KD is conducted would not impact its effectiveness that much empirically. Our approach of masked distillation is inspired by previous KD work conducted in the RGB image feature space [1] and has been adapted to address the BEV space's semantic complexity by initializing and learning dense BEV queries. Similarly, the methodology we have developed for masked feature distillation and the construction of a modality-general fusion model can be adapted to other feature spaces, such as RGB or depth image features.

2. Q2 & Q4: Add teacher model type column to KD methods & The Comparison of different teacher model

Supplementary Table: Overview of Teacher Models Used in Knowledge Distillation Methods

Thank you for your constructive suggestions, which prompted me to add a table representing the teacher models used in various KD methods. As indicated in the table, most KD methodologies consider the architectural similarity between the teacher and student models when choosing the teacher model. For instance, S2M2-SSD employs a multi-modal teacher model similar to PointPainting[2], which fuses image segmentation results with LiDAR features before voxelization to supervise LiDAR student right from voxelization process. Unidistill adopts the BEVFusion model as a teacher, which aligns structurally with single-modal students. BEVDistill ensures structural similarity with the BEVFormer student by using Object DGCNN[3] as LiDAR teacher, which is based on the DETR encoder and decoder architecture.

In our VeXKD study, we adopted a BEVFusion pipeline similar to Unidistill but modified the fusion module to enhance the teacher's efficacy in KD. This modification was inspired by observing performance gaps between Unidistill and those KD methods exclusive to camera students like BEVDistill. Our ablation study also reveals that a significant portion of the performance gain is attributable to these modifications in the teacher models.

Additionally, when implementing our research, we explored building a multi-modal teacher model using the global attention method described in [4]. However, replicating the fusion teacher with this method resulted in significant GPU memory consumption, challenging the training process. This experience inspired the replacement of global attention with more efficient deformable attention for modality-general fusion.

As mentioned in the limitations of our paper, the lack of research quantifying the modality-general information contained in teacher models makes the process of experimenting with different teacher models time-consuming, effort-intensive, and fraught with uncertainty. Our paper aims to provide an example schema for extracting modality-general information from teacher models without adapting the teacher's model architecture. We hope that future research will include more theoretical analyses on the different teachers in cross-modal KD.

3. Q3: Incorporation of temporal information

Thank you for your suggestions. As clarified in the global response, both our teacher model and LiDAR students inherently incorporate multi-sweep LiDAR inputs, thereby implicitly integrating temporal information.

Inspired by your feedback, we conducted additional experiments with BEVFormer, a student model that combines temporal information, and observed performance gains, as detailed in the global response. As technologies for multi-modal explicit temporal fusion continue to mature and become open-sourced, we foresee the potential for integrating explicit temporal KD operations as a pluggable module into the VeXKD framework in future developments.

References

[1] Huang, Tao, et al. "Masked distillation with receptive tokens." arXiv preprint arXiv:2205.14589 (2022).

[2] Vora, Sourabh, et al. "Pointpainting: Sequential fusion for 3d object detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[3] Wang, Y., & Solomon, J. M. (2021). Object dgcnn: 3d object detection using dynamic graphs. Advances in Neural Information Processing Systems, 34, 20745-20758

[4] Man, Yunze, Liang-Yan Gui, and Yu-Xiong Wang. "BEV-guided multi-modality fusion for driving perception." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Q1: Adaptation on other feature representation

In response to your valuable insights, we would like to offer some clarifications. When conducting KD, it is crucial for the feature maps of both the student and the teacher to reside within the same feature space to ensure spatial and semantic compatibility. In this regard, the BEV feature space has gained significant attention in recent years due to its favorable compatibility with multiple modalities and the similar perception pipeline across modalities in 3D perception. Indeed, the volume of research focused on the BEV feature space far surpasses that on other feature spaces in recent years.

Furthermore, if student and teacher feature maps are in different feature spaces, projection can be used to align them. Once the spatial alignment is done, the specific feature space used for KD theoretically and empirically would not impact its effectiveness that much. For example, our masked distillation is inspired by previous work in the RGB image feature space[1] and is adapted to better address the semantic complexity of the BEV space through strategies like the initalizing and learning of dense BEV queries. Similarly, the methodologies we have developed for masked feature distillation and the construction of a modality-general fusion model can be adapted to other feature spaces, such as RGB or depth image features.

Q2: Adaptation on other modalities

We appreciate the opportunity to further clarify the adaptability of our methods with the inclusion of additional modalities. Our methodology is designed to handle each modality symmetrically, ensuring that integrating new modalities does not necessitate alterations to the existing codebase, but rather requires configuration updates to accommodate them. Additionally, these new modalities should be integrated into the training of the new fusion teacher to extract the modality-general information from new modalities.

Once the teacher model is trained, the masked feature distillation operation can be applied to facilitate specific feature mask learning and selective feature distillation for the newly integrated student modality model. For the student model, augmenting the existing task loss with the KD loss is sufficient to complete its training.

It is worth noting that our framework primarily conducts KD within the BEV feature space for LiDAR and camera students. However, BEV feature space is compatible with various modalities, including raw mmWave radar points[2, 3].

Incorporating additional modalities necessitates training the new fusion teacher and new student models, adding the data processing operation for new modalities. Completing this retraining process within the rebuttal period is challenging. We hope this clarification is helpful and addresses your concerns effectively.

References

[1] Huang, Tao, et al. "Masked distillation with receptive tokens." arXiv preprint arXiv:2205.14589 (2022).

[2] Stäcker, Lukas, et al. "RC-BEVFusion: A plug-in module for radar-camera bird’s eye view feature fusion." DAGM German Conference on Pattern Recognition. Cham: Springer Nature Switzerland, 2023

[3] Harley, Adam W., et al. "Simple-bev: What really matters for multi-sensor bev perception?." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.