GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation

Thank you for your valuable and constructive reviews. Here we address your questions one by one.

W1. Additional baseline methods on NuImages and COCO-Stuff datasets.

Thank you for your suggestions. In the revision, we have 1) reported the fidelity and trainability performance of ControlNet and GLIGEN on the NuImages dataset in Table 2 and 3 respectively, 2) reported the trainability performance of ControlNet on the COCO dataset in Table 5, 3) reported the inpainting performance of ControlNet and GLIGEN on the COCO dataset in Table 6, and 4) refine the claimed improvement to be more specific in the introduction section. For detailed experimental results, please refer to the common response to Reviewers WbtJ and VRFf.

W2. Additional baseline methods for the inpainting comparison in Figure 5.

We provide a qualitative comparison between GeoDiffusion and Stable Diffusion in Figure 5 primarily to demonstrate how exactly the geometric-awareness enhances the inpainting ability. As shown in Figure 5, GeoDiffusion can better understand the requested inpainting locations especially under multi-object circumstances, and therefore, better deal with the inpainting requests.

Following your suggestions, we further provide comparison with ControlNet and GLIGEN in Figure 5, where GeoDiffusion still demonstrates more superior inpainting results.

W3. More discussion on the camera view control generation.

(1) Importance of camera views has been verified by the ablation study conducted in Table 10 and Appendix B. Specifically, we train a GeoDiffusion without camera view tokens with all the other settings unchanged. As reported in Table 10 (rows 1 and 2), the variant without camera view tokens is surpassed by the default GeoDiffusion with respect to both the image quality (i.e., FID) and layout consistency (i.e., mAP), revealing the importance of camera views.

(2) More qualitative comparison of camera views is provided in Figure 10 following your suggestions. As demonstrated, GeoDiffusion can effectively convert the camera views simply by changing the camera view tokens in text prompts, while maintaining consistent with the given semantic layouts.

Q1. Discussion on the high AP50 but low mAP and AP75 performance of ControlNet in Table 4.

This is primarily because ControlNet can only take image-level condition inputs instead of instance-level ones as GeoDiffusion.

(1) Implementation of ControlNet. To facilitate the layout-to-image generation, ControlNet is implemented by converting the semantic layouts to $C$ binary semantic masks, with each mask for a single semantic class, following the ControlNet paper. Therefore, the model can no longer distinguish different instances of the same class. When two instances of the same class are close with each other, the model would tend to generate a single but large instance to cover the areas of both instances, which is a typical reason to obtain a high AP50 but low mAP and AP75.

(2) Qualitative verification. A qualitative example is provided in Figure 5. As demonstrated, ControlNet completes the inpaiting requests by considering the left two people as a single instance, and therefore, only complements the right part of the human body.

Thank you for your valuable and constructive reviews. Here we address your questions one by one.

W1. Additional baseline methods on NuImages in Tables 2 and 3.

As discussed in Section 4.3, we compared GeoDiffusion with recent diffusion-based methods, including ControlNet, ReCo, and GLIGEN, on the COCO-Stuff dataset using publicly available checkpoints. In these comparisons, GeoDiffusion has demonstrated significant superiority.

Following your suggestions, we have also reported the performance of ControlNet, ReCo, and GLIGEN on the NuImages dataset in Tables 2 and 3, where GeoDiffusion continues to show consistent improvement. For detailed experimental results, please refer to the common response to Reviewers WbtJ and VRFf.

W2. Comparison between the camera-dependent generation in GeoDiffusion and the view-dependent prompting in DreamFusion.

(1) Controllability. In GeoDiffusion, camera view tokens are explicitly utilized as input conditions to generate images from different camera views. This approach differs fundamentally from DreamFusion, where view-dependent prompting serves primarily as prior knowledge to enhance the supervision signal in training the NeRF model. DreamFusion still relies on explicit camera parameters to generate images from various views. In contrast, our method leverages direct input conditions, offering more intuitive controllability over the image generation process.

(2) Purpose. Our primary goal with camera views in GeoDiffusion is not to introduce a novel view encoding method but to demonstrate that text prompts can serve as a unified representation for various geometric conditions, facilitating independent manipulation without interdependencies. As illustrated in Figure 10, the simple conversion of camera view tokens in GeoDiffusion can directly generate images from different perspectives while maintaining semantic integrity. This showcases our method's ability to handle geometric conditions more flexibly than DreamFusion.

W3. Ablation study on the amount of real data required to effectively train GeoDiffusion.

In Section 4.2.2, under the "necessity of real data" paragraph, we detail an experiment designed to ascertain the minimum quantity of images needed to train GeoDiffusion efficiently. Our approach involves the "subset" mode, where we randomly select subsets of the real image dataset (e.g., 10%, 25%, 50%, and 75%) for training individual GeoDiffusion models. Each of these models then generates synthetic images to augment their respective subsets. Subsequently, these augmented datasets, comprising both real and synthetic images, are used to train object detectors.

As illustrated in Figure 1(b) and reported in Table 8, our findings reveal that GeoDiffusion can achieve consistent performance improvements even with as little as 10% of the real images. Notably, the benefits of GeoDiffusion are more pronounced under data-scarce conditions. This demonstrates that the amount of real data required to train GeoDiffusion is considerably less than what would typically be needed to train an object detector directly, thereby enhancing the practicality and applicability of our approach.

W4. Transferability across datasets and comparison with copy-and-paste synthesis.

(1) Different number of bins across datasets affecting transferability. To address your concern, we would like to clarify that in our experiments, all optimization hyperparameters, including the number of pixels in each location bin, learning rates, and foreground loss weights, have been consistently maintained across both datasets (NuImages and COCO-Stuff). We acknowledge that we have not explicitly tested the model's transferability from one dataset to another. However, the diversity of classes and scenarios covered in NuImages and COCO-Stuff provide a robust assessment of our method's transferability. The distinct scenarios presented in the training and test sets of NuImages, in particular, attest to the effectiveness of our approach across varied contexts.

(2) Comparison with copy-and-paste synthesis. In the comparison with copy-and-paste methods, it's important to highlight that our paper's primary objective is to introduce a unified framework capable of not only generating high-quality detection data but also supporting geometric-aware image generation and inpainting requests. While copy-and-paste methods may offer greater flexibility in data augmentation, they typically fall short in generating realistic images and providing additional geometric controls. For a more comprehensive comparison with related works, we invite you to review the detailed analyses presented in Table 1 and Section 2 of our main paper.

Thank you so much for your acknowledgement of this work. Here we address your concerns one by one.

Q5. Transfer between datasets with different image sizes and aspect ratios.

Usually, it is difficult to require large-scale image datasets like COCO-Stuff to have a unified image size and aspect ratio for all the images.Therefore, in order to perform mini-batch-based optimization, we follow common practices [2, 3, 4] to first resize all images to a unified resolution $H\times W$ (e.g., $H\times W=512\times512$ for the default Stable Diffusion) as the model input shape during training, while during testing, the model will first generate images of $H\times W$, which are then resized to the original image resolution through (bilinear) interpolation. Thus, transfer between different datasets with different image sizes and asepct ratios would not be a problem here.

Q6. Further discussion on the application for long-tailed and domain adaptation.

(1) Further discussion on long-tailed generation. Following your suggestions, we conduct a preliminary experiment on the LVIS dataset with the exact same optimization recipe with the COCO-Stuff dataset as discussed in Section 4.3, and provide a qualitative evaluation in Figure 12, where the annotations of "rare classes" are highlighted in the images. As shown in Figure 12, GeoDiffusion demonstrates superior generation capabilities even for the long-tailed rare classes. We will keep improving the performance in the final revision.

(2) Further discussion on domain adaptation. As shown in Figure 9, GeoDiffusion has already demonstrated flexible domain adaptation capabilities between daytime, rainy and night circumstances, while the generation of foggy images are currently not support primarily due to the absence of foggy images in the NuImages dataset. We will further explore this specific application in the future work.

Besides, we notice that the score of the paper in the OpenReview System has not been changed. To complete the score-changing procedure, you might also need to edit the "Rating" option in your official review. Thank you.

[2] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models. In CVPR. 2022.

[3] https://github.com/CompVis/latent-diffusion

[4] https://github.com/huggingface/diffusers/

Thanks the authors for their response. I found the response just partly addressed my concerns. The concern regarding training in one dataset and testing in another dataset (different image sizes or aspect ratios in transfer testing) is not addressed, so basically, cannot be handled right? Also, the long-tail and domain adaptation is not addressed either. Therefore, I can only marginally increase my score to 6.

Thank you for your acknowledgement and valuable suggestions. Here we address your questions one by one.

W1: Extension to 3D geometric controls.

Thank you for your insightful comments. It is nature for GeoDiffusion to intergrate various geometric controls, as long as they can be discretized and integrated into the text prompts. As highlighted, our current work primarily demonstrates the integration of camera views, a vital aspect in 2D object detection for autonomous driving, as evidenced in our ablation study (Table 10, rows 1 and 2).

To address the extension into 3D domains, one needs to incorporate 3D conditions, such as depth, exact 3D locations, and angles, into the generation process. This can be achieved by projecting 3D LiDAR coordinates onto the 2D image plane, followed by text-prompted 2D conditional generation, akin to the process used in GeoDiffusion. Such methodologies have been preliminarily explored in recent studies [1, 2]. Limited by the scope of this paper, our primary focus is on data generation for 2D object detection. The implementation of support for 3D geometric controls is left as a subject for future work.

W2. Advantages of GeoDiffusion over methods such as GLIGEN, particularly in terms of trainability for object detection.

(1) Utilization of foundational pre-trained models. Unlike GAN-based methods, a significant advantage of GeoDiffusion lies in leveraging large-scale pre-trained text-to-image diffusion models, such as Stable Diffusion. These models enable the generation of highly realistic and diverse detection data, which is crucial for effective data augmentation.

(2) Transferability of the text encoder. In contrast to existing methods like GLIGEN and ControlNet, which require designing specific bounding box encoding modules and training parameters from scratch, GeoDiffusion capitalizes on the transferability of the text encoder. This approach allows for more efficient adaptation and reduces the need for annotated training data. This is particularly beneficial for long-tailed classes with scarce data annotations. For instance, GeoDiffusion shows remarkable improvement in rare classes such as trailers and construction vehicles, achieving +11.9 and +15.0 mAP, respectively, compared to GLIGEN in the Top-2 rare classes on NuImages, as demonstrated in Table 2. The efficacy of the text encoder's transferability for geometric conditions is further supported by our ablation studies in Table 9 (rows 1 and 2).

(3) Usage of the foreground prior re-weighting. This strategy significantly enhances generation performance by focusing on the modeling of foreground objects. The positive impact of this approach is evident in the comparison between the results in Table 9 (rows 2 and 5).

W3-W4. Paper formatting and typos.

Sorry for the careless mistakes. We have fixed the formatting and typo problems in the revision.

[1] Gao R, Chen K, Xie E, et al. Magicdrive: Street view generation with diverse 3d geometry control[J]. arXiv preprint arXiv:2310.02601, 2023.

[2] Yang K, Ma E, Peng J, et al. BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout[J]. arXiv preprint arXiv:2308.01661, 2023.

Thank you for your acknowledgement and valuable suggestions. Here we address your questions one by one.

W1: Extension to 3D Geometric Controls.

Thank you for your insightful comments. It is nature for GeoDiffusion to integrate various geometric controls, as long as they can be discretized and integrated into the text prompts. As highlighted, our current work primarily demonstrates the integration of camera views, a vital aspect in 2D object detection for autonomous driving, as evidenced in our ablation study (Table 10, rows 1 and 2).

To address the extension into 3D domains, one needs to incorporate 3D conditions, such as depth, exact 3D locations, and angles, into the generation process. This can be achieved by projecting 3D LiDAR coordinates onto the 2D image plane, followed by text-prompted 2D conditional generation, akin to the process used in GeoDiffusion. Such methodologies have been preliminarily explored in recent studies [1, 2]. Limited by the scope of this paper, our primary focus is on data generation for 2D object detection. The implementation of support for 3D geometric controls is left as a subject for future work.

Q1. Justification of inconsistent performance metrics.

(1) FID score for the 512 $\times$ 512 model is better than that of the 800 $\times$ 456 variant in Table 2. This can be attributed to the alignment between the pre-trained resolution of the Stable Diffusion model and the fine-tuning resolution of the 512 $\times$ 512 model. The Stable Diffusion model, in our study, is originally pre-trained at a resolution of 512 $\times$ 512. When we fine-tune this model at the same resolution (512 $\times$ 512), it results in a more effective transfer of learned features and parameters. This alignment of resolutions minimizes the discrepancies that might arise from adapting to different resolutions, and thereby enhancing the model's performance in terms of the FID score. However, when fine-tuning at a different resolution (as in the case of the 800 $\times$ 456 variant), the model faces the challenge of adapting to a new resolution, which can lead to a slight decrease in FID score.

(2) Lower mAP for pedestrian and traffic cone than the real-image-only baseline in Table 3. In Section 4.2.1, we discuss the challenges that GeoDiffusion faces, particularly with the high variance and small objects, such as pedestrians and traffic cones, where the performance of our model is slightly lower compared to the Oracle real-image-only baseline. This indicates that the quality of the generated data for these two categories may not be sufficient to significantly enhance the performance of object detectors. Similar difficulties are encountered by other generative models as well. As shown in Table 3, except for the real-image baseline, our method outperforms other generative competitors, including LostGAN, LAMA, Taming, ReCo, GLIGEN, and ControNet. We are the first to demonstrate that generated data can serve as effective data augmentation to improve the overall performance of object detectors. We will continue to focus on improving the unsatisfactory performance in generating high variance and small objects in our future work.

Q2. Clarification on the 4X training acceleration in Section 4.2.1.

Sorry for the confusion caused. The measure of training acceleration is calculated based on the number of training epochs required to achieve a comparable level of performance. In Table 2, we demonstrate that our GeoDiffusion model, trained for only 64 epochs, significantly outperforms models like LostGAN, LAMA, and Taming, which were trained for 256 epochs. This indicates that GeoDiffusion achieves the same or better performance four times faster in terms of epochs, hence the 4X acceleration in training. To ensure clarity, we have revised the manuscript to explain this comparison more explicitly.

Thanks for the clarification! I have read the rebuttal and most of my concerns have been addressed. I would raise my score to clear accept.