3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features

We thank you for your comments and appreciation to our paper. We respond to your concern below.

To respond the baseline method has not been trained on posed images, while we train the model on posed images: We refer the reviewer to see our table. 1. We compare our method with NeRF-Det and ImvoxelNet, which are both trained on the posed images. Actually, these methods use stronger priors with more posed images while we only use two posed images during training, our 3DiffTection can still outperform them on single-image 3D object detection.

To respond to whether geometric ControlNet truly introduces 3D awareness: Please refer to our correspondence visualization results, and we provide more visualization in our appendix as well as our quantitative correspondence evaluation. Note that the correspondence evaluation all come from geometric controlnet with input identical pose, and it never accesses correspondences ground truth, which means we have a totally fair comparison with the original DIFT model. With this context, our method can find correspondences between multiview images without input pose, while the DIFT model fails. If our model does not learn 3D awareness, this means it can not find the accurate correspondences, and during novel-view synthesis, it can not accurately warp the features to the correct places. The current correspondence experiments and novel-view synthesis experiments demonstrate the effectiveness of our model learning 3D awareness.

To respond the efficiency The efficiency is currently out of scope but it is an important topic. Currently, there are some efforts on this track, for example, Q-Diffusion [1], especially being efficient during inference. We will also put some efforts on this track in the future.

[1]Q-Diffusion: Quantizing Diffusion Models. ICCV2023

We sincerely appreciate your time and efforts form reviewing our paper.

To respond to how semantic controlnet and novel view synthesis assist in enhancing the models:

Why does novel view synthesis help enhance the 3D object detection model? Novel-view synthesis is a task that “warps” the pixels of the condition image into the target one, which inherently enforces the model to learn the 3D structures of the scenes, otherwise the shifting would be wrong. Learning 3D information is important for 3D object detection. The similar spirit is also proposed in NeRF-Det. To demonstrate that our geometric ControlNet instills 3D awareness in the model, we compare the SD features with and without our geometric ControlNet. For the comparison, we use ScanNet, which allows us to measure performance against ground truth and report accuracy. Moreover, it is important to note that ScanNet was not used for training at any point, ensuring a fair comparison. The correspondence analysis can demonstrate the hypothesis above. If the model learns the pixel shifting, it should have been expressed by correspondences. To be more specific, we conduct quantitative correspondence experiments, as shown in the following table. Experiments also demonstrate that training a geometric controlnet with novel-view synthesis task can improve the correspondence results.

Correspondence experiment on ScanNet dataset.

|Methods | AUC@5 | AUC@10 | AUC@20 | Precision | MS |

|DIFT-SD | 5.61 | 6.15 | 14.73 | 16.95 | 4.57|

|3DiffTection| 7.49 | 8.92 | 18.66 | 20.11 | 6.93|

Why does Semantic-ControlNet help enhance the 3D object detection model? The motivation for utilising another semantic controlnet is that the features from stable diffusion + geometric controlnet are designed for image generation and aren’t specifically fine-tuned for 3D detection. Therefore, there is further potential for optimization towards detection. We want to emphasize that tuning the controlnet is also a way of parameter-efficient finetuning that does not break the original representations from pretrained Stable Diffusion model parameters. Maintaining parameter efficiency is crucial for achieving label efficiency, a key objective in our research, as demonstrated in our label-efficiency experiment.

We thank the reviewer for the time in reading our paper.

We respectfully disagree with the stated main weakness, i.e, that the proposed method is merely an application of ControlNet. On an intellectual level, a successful utilization of a technique, unless trivial, can still be interesting to the audience for advancing science. For the case of our specific work, our approach introduces ideas from ControlNet in an advanced way and in a completely different manner than originally proposed (3D detection rather than conditional generation). We cannot think of a way to trivially use Controlnet to achieve the same task as indicated by the reviewer, and would appreciate a more concrete proposal that we can discuss.

We provide further explanation below and invite the reviewer to take another careful read of our paper:

1. Our work aims at 3D object detection. Our method highlights that (1) diffusion features are semantically meaningful for use in perception tasks. However, they are not good enough for 3D tasks out of the box. Therefore, (2) we leverage ControlNet to instill 3D awareness into the learned representation. With these improved features we proceed to (3) tune them along with a 3D detection head.

2. As discussed in Sec 3.2, we find that ControlNet’s design is inherently biased towards condition signals that are pixel-aligned with the generated image. Indeed, our early experiments indicate that directly conditioning on the source image fails at generating useful novel views. To overcome this, we propose to modify ControlNet with an epipolar warp operator. This simple operator makes a big difference. Experimentally, please refer to our Tab. 2 (Row 5), where we remove the warp operator and reduce geometric ControlNet to the original ControlNet. The results demonstrate that a simple use of ControlNet does not work for 3D object detection.

3. Finetuning Semantic ControlNet for specific tasks and data itself is also novel. As far as we know, previous works only use ControlNet for generation tasks, we tune it for perception tasks, which is also a new use.

4. Finally, our method leverages both geometric and semantic tuning stages to enable virtual view augmentation, for enhanced detection prediction.

We hope this explanation makes it clear that equating our method to a mere application of ControlNet is a drastic oversimplification. Our approach involves a plethora of insights into the premise and limitations of diffusion models for 3D perception tasks, an understanding of the strengths and weaknesses of ControlNet, geometric reasoning, and drastic improvements in single-image 3D detection over past work. Thus, we believe that beyond the compelling results, readers will also acquire new insights and ideas for future work.