DiffuBox: Refining 3D Object Detection with Point Diffusion

We thank the reviewer for finding our method novel and of high quality, pointing out that we have extensive experimentation across datasets. We address detailed concerns below.

Q1: Comparison to more recent 3D detectors

Thank you for this suggestion, and we are actively working towards including more recent baselines to strengthen our work. While we are currently in the middle of performing these experiments and cannot provide the results before the rebuttal deadline, we will include the updated results during the discussion period.

Q2: Computation and runtime vs performance trade-off analysis

We perform an analysis on the latency and runtime of our method in the common questions section. Additionally, we plot an ablation figure showing the number of diffusion steps to performance trade-off in Fig. 1 of the uploaded Rebuttal PDF, and further discuss their implications in the common questions section. We will include these in the final version.

Q3: More diverse visualizations

We have included additional, more diverse visualizations in Fig. 3 of the uploaded Rebuttal PDF. To summarize, we provide additional qualitative results that showcases DiffuBox on car, cyclist, and pedestrian classes. IoU values of the boxes before and after DiffuBox are included for visualization purposes, showing gains in alignment and better shape estimation after applying our method. Our method is effective across different actor class types. We thank you for this valuable suggestion, and will include these in the final version.

Q4: Robustness of method across object sizes and sensor quality

We have performed additional analysis on objects of different sizes in Fig. 2(b) of the uploaded Rebuttal PDF. Observe that performance gain is clear across all object sizes. This result arises naturally, since we conduct our diffusion modeling in the normalized box view (NBV), which effectively corrects for boxes of all sizes. We believe that this is a benefit of our work, and we will include these results in the final version as well. Regarding sensor set-up robustness, we experimented with have significant differences in the sensor setup, and we further report results on nuScenes in the common questions. Regarding different sensor modalities, please refer to our response to reviewer YzBS Q3.

We thank the reviewer for their valuable feedback! Below, we address additional concerns:

Q1: Request for results on additional dataset

We are happy to provide additional experimental results on the nuScenes dataset, and report the results in the common questions section. Our results confirms DiffuBox’s effectiveness, even on this new dataset.

Q2: Computational expense vs performance trade-off analysis

We investigate the tradeoff between performance and computational cost by varying the number of diffusion steps, and plot the mAP against per box refinement time (corresponding to different number of diffusion steps). The figure is included in Fig. 1 of the uploaded Rebuttal PDF, and additional discussion is provided in the common questions section.

Q3: Additional discussion about varying sensor set-ups

The three datasets we experimented with have significant differences in the sensor setup, and we further report results on nuScenes in the common questions section. We detail the sensor set-up for different datasets in the table below:

We believe that the diversity of sensor set-ups within these datasets help justify our method’s robustness to sensor distributions. We admit that we do not evaluate on other modalities of LiDAR such as solid state LiDAR sensors, and will leave that to future work. We will include this discussion in the final version.

We thank the reviewer for finding our method novel and effective, and address their concerns below:

Q1: Additional discussion on false negatives

Thank you for this suggestion! We include an analysis plot of Recall vs IoU threshold in Fig. 2(a) of the uploaded Rebuttal PDF. DiffuBox is able to improve IoU for mislocalized detections and reduce false negatives that arise from match IoU being lower than the threshold, which forms a major cause of reduced cross-domain performance [1]. However, one limitation of our method is that we are inherently a refinement method, and we cannot recover from false negatives due to completely missed detections. We will include these results and discussion in the final version as well.

Q2: Intuition behind the quality of our refinement results

Our method can be thought of as learning a relevant shape as a prior, as well as correcting for the localization. Diffusion models excel at object-level shape refinement [2, 3] due to their iterative nature and ability to approximate the scoring function. Viewing the points as part of a distribution within the bounding box, the diffusion model is able to guide the box towards better localization and shape. We will further point out reason why the diffusion model can refine the output bounding boxes in the final version.

Q3: More recent methods as baselines

Thank you for your valuable suggestion, and we believe an additional baseline will strengthen our results. We are actively performing experiments with more recent detectors. Unfortunately, we could not obtain the results before the rebuttal deadline, and will provide updates during the reviewer-author discussion period.

[1] Wang, Yan, et al. "Train in germany, test in the usa: Making 3d object detectors generalize." CVPR 2020.

[2] Zhou, Linqi, Yilun Du, and Jiajun Wu. "3d shape generation and completion through point-voxel diffusion." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[3] Vahdat, Arash, et al. "Lion: Latent point diffusion models for 3d shape generation." Advances in Neural Information Processing Systems 35 (2022): 10021-10039.

We thank the reviewer for finding our method interesting, robust, and pointing our the benefits of our domain agnostic method. We further address their concerns below:

Q1: Cumulative image description of overall method

We thank the reviewer for pointing out this possible point of confusion, and included an algorithmic description of the overall training and inference workflow in Alg. 1 of the Rebuttal PDF. We hope this clarifies the overall DiffuBox method, and will include it in the final write up.

Q2: Evaluation on bigger dataset

Thank you for the suggestion. We are happy to report DiffuBox’s strong performance on a larger dataset. We conducted additional experimental results on NuScenes and reported the performance in the common questions section.

Q3: Latency vs. mAP

We perform ablation on the number of denoising steps and measure their corresponding performance and latency. The results are reported in Fig. 1 in the Rebuttal PDF and further discussed in the common questions section.

Thank you so much for the insightful comments. Your valuable suggestions are very helpful for further strengthening our paper.

Q1: The idea is simple. Refining box is not innovate although the authors use recently popular diffusion model to implement this goal.

Thank you for your comments regarding the novelty of our work. Domain adaptation for 3D object detection is indeed a challenging task. Existing methods [1, 2] typically require finetuning the model based on target dataset statistics or pseudo labels. In contrast, our model achieves zero-shot adaptation without any finetuning, outperforming these baselines. We deliberately kept our algorithm simple to highlight our key insight: that the diffusion model is domain-agnostic and can effectively refine box size and location.

Q2: The proposed diffusion-guided box refine module can be seem as a new stage for detection. However, only old PointPillar. SECOND and PV-RCNN are used as baselines, without new works.

Thank you for the valuable suggestions. We believe adding more advanced baselines will strengthen our paper. We are currently working on incorporating more recent baselines with our proposed diffusion model. Unfortunately, we could not obtain the results before the rebuttal deadline, but we will provide updates during the reviewer-author discussion period.

Q3: The authors may have written it in this way to avoid comparisons with SOTA detection models.

Thank you for the suggestion to apply our model directly to the detection task. Existing works [1] demonstrate significant challenges in domain adaptation for 3D object detection, particularly in terms of shape normalization and localization accuracy. We focus on domain adaptation because the diffusion model naturally addresses these issues by learning normalized box shapes that are disentangled from size, and by correcting mislocalized boxes. Nevertheless, we tried our method in the in-domain evaluation and only observed a minor improvement (mAP from 77.77 to 79.53 for KITTI in-domain). Such a result is expected: a model trained on an in-domain dataset inherits biases from that dataset, giving it an advantage when evaluated in-domain. Conversely, our denoising model aims to remove these biases, enabling better performance in out-domain evaluations. Without the learned bias, in-domain performance does not necessarily improve.

[1] Wang, Yan, et al. "Train in germany, test in the usa: Making 3d object detectors generalize." CVPR 2020.

[2] Yang, Jihan, et al. "St3d: Self-training for unsupervised domain adaptation on 3d object detection." CVPR 2021.

To complement the results obtained with CenterPoint, we conducted additional KITTI$\rightarrow$Lyft experiments using DSVT as the base object detector. The results, presented in the table below, further validate the generalizability of DiffuBox and its capacity to enhance domain adaptation performance across various object detection frameworks and object classes.

Encouraged by the insightful feedback from reviewers spFc, DshE, and 7LbH, we have supplemented Tables 1, 3, and 4 with additional KITTI$\rightarrow$Lyft experiments which employ CenterPoint as the base object detector. Our experiment results demonstrate consistent and significant localization improvements achieved by DiffuBox across all object classes.