ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models

Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.

$**1. Ablations for fine-tuning on both cropped foreground objects and entire images:**$

The fine-tuning process is the basis of the training of the following object-wise conditioning module. The fine-tuning on both cropped foreground objects and entire images makes the fine-tuned models capable of synthesizing the images of foreground objects, which are needed by building image lists for the next step. We provide some visualized samples in Fig. 1 of the PDF file attached to the global response. It shows that the fine-tuned model is capable of generating diverse foreground objects and complete scenes. The training of the object-wise conditioning module is designed to realize layout control based on the fine-tuned model and uses the image lists containing synthesized images of foreground objects as conditions. Therefore, it's hard to conduct an ablation study that fine-tunes the diffusion model without cropped foregrounds. Because the models only fine-tuned on entire images have difficulties in generating foreground objects required by image lists since they cannot bind the prompts to corresponding objects in entire images that may contain multiple categories of objects.

$**2. Post-processing step with the corrupted label filtering:**$

See the analysis and experiments provided in part 1 of the global response. We provide results of all methods without post-processing to compare the generation capability directly and fairly.

$**3. Experiments with a larger number of real images:**$

See the experiments provided in the parts 2 and 3 of the global response. We provide experiments with different methods trained on the COCO training set (80k images) and add experiments of our ODGEN trained on 1000 images from the Apex Game and Underwater Object datasets.

$**4. Quantitative evaluation of concept bleeding:**$

This work mainly focuses on the concept bleeding problem of different categories. The mAP of YOLO models trained on synthetic images only in Tab. 3 of our paper can be a proxy task to prove that concept bleeding is alleviated since our approach achieves state-of-the-art performance in the layout-image consistency of complex scene generation conditioned on bounding boxes. In addition, we add the quantitative evaluation with BLIP-VQA ($\uparrow$) [a] which employs the BLIP model to identify whether the contents in synthesized samples are consistent with the text prompts. The results are shown in the table below. Our ODGEN outperforms other methods and gets results close to ground truth (real images from the COCO dataset sharing the same labels with synthetic images). The results are averaged over 41k synthetic images following the labels of the COCO validation set.

[a] K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu, T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation, Advances in Neural Information Processing Systems, vol. 36, 2024.

Dear Reviewer FcFR,

Thanks for your positive feedback. We are glad that most of your concerns have been well addressed. We add a detailed explanation for your remaining concern.

1. $**Specific Domains**$:

Taking the Robomaster dataset in RF7 as an example, most images in the dataset contain multiple categories of objects like "armor", "base", "watcher" and "car", which are either strange for Stable Diffusion or different from what Stable Diffusion tends to generate with the same text prompts. For the model fine-tuned on entire images only, it cannot obtain guidance on which parts in entire images correspond to these objects. As a result, the fine-tuned models cannot synthesize correct images for foreground objects given text prompts like "a base in a screen shot of the robomaster game". We agree with you that "adding experimental results with the models fine-tuned on entire images only" can make our statement more convincing. We provide FID results of foreground objects synthesized by models fine-tuned on both entire images and cropped foreground objects (text prompts are composed of the names of objects in the image and the scene name, e.g., "a car and a base in a screen shot of the robomaster game"), and models fine-tuned on entire images only, in the table below for comparison:

Table: FID ($\downarrow$) results of foreground objects in the Robomaster dataset synthesized by models fine-tuned on different data.

It shows that our approach helps fine-tuned models generate images of foreground objects better. Similarly, we also provide results on the Road Traffic dataset, which is relatively more familiar for Stable Diffusion than the Robomaster dataset.

Table: FID ($\downarrow$) results of foreground objects in the Road Traffic dataset synthesized by models fine-tuned on different data.

Results also show improvement of our approach compared with models fine-tuned on entire images only. Since links are not allowed in the responses, we cannot provide supplemental visualized results here but will add them to the revised manuscript.

2. $**General Domains**$:

The fine-tuning step is designed for specific datasets. As for datasets of general domains like COCO, Stable Diffusion can generate the same categories of objects as the objects in the COCO dataset without fine-tuning. Therefore, we skip the fine-tuning step and directly use Stable Diffusion to synthesize images of foreground objects and use them to build image lists. The results in our paper show superior improvement achieved by our ODGEN compared with prior works.  Thanks for your valuable suggestions and efforts in reviewing this paper again!  Authors

Dear Reviewer FcFR,

Thank you for your positive feedback and efforts in reviewing this paper again!

Authors

Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.

$**1. Generation cost:**$

Before training and inference, we can generate an offline library of foreground objects to accelerate the process of building image lists. With the offline library, we can randomly pick images from it to build image lists instead of synthesizing new images of foreground objects every time. In the generation stage (Fig. 2c in our paper), our approach pads both the image and text lists to a fixed length. Therefore, the computational cost for inference with offline libraries doesn't increase with more foreground objects. Other methods like InstanceDiffusion [a] and MIGC [b] need more time for training and inferencing with more objects. Taking the models trained on COCO as an example, to generate an image with 6 bounding boxes on a V100 GPU, ODGEN takes 10 seconds, ControlNet takes 8 seconds, and Instance Diffusion takes 30 seconds.

During the inference, we compare using the same offline image library as training against using a totally different image library. We get very close results as shown in the table below (mAP metrics are provided in the YOLOv5s/YOLOv7 format):

Both outperform other methods significantly as shown in Tab. 3 of our paper. It indicates that ODGEN is capable of extracting category information from synthesized samples of foreground objects in image lists instead of depending on certain images of foreground objects. ODGEN is not constrained by the image library of foreground objects used in training and can be generalized to other newly generated offline image libraries consisting of novel synthesized samples of foreground objects, which ensures that the use of offline libraries won't reduce the practicality of ODGEN.

$**2. Detector performance improvement with synthetic data:**$

We provide several visualized detection results of YOLO models trained with and without synthetic data in Fig. 3 of the PDF file enclosed in the global response. It shows that the synthetic data helps detectors detect some occluded objects.

$**3. Object layout statistics:**$

As illustrated in Sec. 3.3 (Line 167-173) in our paper, the statistics of the object layout are taken from the training dataset. For example, ODGEN trained on 200 images takes the statistics from the 200 images.

$**4. The number of synthetic images:**$

We have added the ablations of the number of training samples (1k, 3k, 5k, and 10k) in Tab. 1 in the PDF file attached to the global response. It shows that using 5k synthetic images gets results close to using 10k synthetic images and outperforms using 1k or 3k synthetic images under our experimental setups.

$**5. Post-generation filtering step:**$

See the analysis and experiments provided in part 1 of the global response.

$**6. Expression:**$

We will change the expression to "We propose to fine-tune the diffusion model with domain-specific images and cropped foreground objects" in the revised manuscript to clarify the difference between our approach and the fine-tuning method on entire images only.

[a] InstanceDiffusion: Instance-level Control for Image Generation, CVPR 2024

[b] Migc: Multi-instance generation controller for text-to-image synthesis, CVPR 2024

Dear Reviewer PYXX,

We are writing to kindly remind you that the time left for discussion is only about one day. Could you please confirm that you have read the rebuttal and check if you still have any concern about our work? Thanks for your efforts in reviewing this paper and proposing valuable comments.

Authors.

Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.

$**1. Comparison with InstanceDiffusion:**$

We didn't include InstanceDiffusion since it consists of an UniFusion module which at least requires bounding box labels and segmentation masks as training data. However, our approach and other methods included for comparison only need bounding box labels. We add an additional comparison by employing the open-source InstanceDiffusion model to generate images following the same setups as Tab. 3 in our paper (mAP results are provided in the YOLOv5s/YOLOv7 format):

We also include InstanceDiffusion in the experiments mentioned in the second part of the global response (YOLO models trained on 80k real images v.s. YOLO models trained on 80k real images + 20k synthetic images). Our approach outperforms InstanceDiffusion as shown by the results in Tab. 2 in the PDF file attached to the global response.

Other works in the second group of data synthesis approaches: Earlier work like LayoutDiffusion [64] is not implemented with latent diffusion models and thus is not included for comparison. Works [21, 37] are designed for the semantic segmentation task while our ODGEN is designed for the object detection task.

$**2. Comparison with copy-paste:**$

The copy-paste method requires segmentation masks to get the cropped foreground objects, which are not provided by the RF7 datasets used in this paper. Besides, our approach only requires bounding box labels, which are easier to annotate than masks.

$**3. Datasets synthesis pipeline design:**$

We agree that the current method cannot exactly reproduce the foreground object distributions in the training datasets. The number of objects in an image is sampled from a joint distribution across different categories while the positions and sizes of bounding boxes are independent. The current dataset synthesis pipeline is designed for simplicity and serves as a method to compare the fidelity and trainability of different methods. As illustrated in the limitations part (Suppl. A), the dataset synthesis pipeline has not been fully optimized yet, which would be interesting to explore in future work.

$**4. Details of the training data:**$

The whole training process on domain-specific datasets, including the fine-tuning on both cropped objects and entire images and the training of the object-wise conditioning module, only depends on the 200 images used for the baseline YOLO detectors training. Besides, the foreground/background discriminator used for corrupted label filtering is also trained and evaluated on the 200 images. We will highlight this point in the revised manuscript to make it clearer. Thanks for your valuable advice.

$**5. Experiments with more training data:**$

See the experiments added in the parts 2 and 3 of the global response.

$**6. Experiments for the trainability evaluation on COCO:**$

See the experiments of training YOLO with "80k real v.s. 80k real + 20k synthetic" images added in part 2 of the global response. The current experiments of trainability on COCO (Tab. 3 in our paper) are designed to evaluate the trainability of using synthetic data only. We train YOLO models on synthetic datasets and evaluate them on real COCO data (sampled from the COCO validation set). We find that results of different methods have high degrees of discrimination and our approach achieves improvements by a large margin compared with existing methods.

$**7. Generalization to novel categories:**$

It is an interesting topic to evaluate the generalization to novel categories not included in the training process. We use ODGEN trained on COCO to synthesize samples containing objects not included in COCO like moon, ambulance, and tiger et al. We show visualized samples in Fig. 4 in the PDF file enclosed in the global response. It shows that ODGEN trained on COCO is capable of controlling the layout of novel categories. However, we find that the generation quality is not stable enough, which may be impacted by the fine-tuning process on the COCO dataset. We are also trying to train a generic ODGEN on about 3000k images covering more than 3000 categories. We hope to get a powerful and generic model covering most common categories in future work.

$**8. Corrupted label filtering:**$

The label filtering step is not applied to experiments on COCO. The filtering rate of the "corrupted label filtering" step is fixed for experiments on RF7. The proposed discriminator is only designed to justify foreground and background and doesn't discriminate specific categories. It is designed to filter the boxes in which no objects are synthesized. Even if object A is partially occluded by object B, the discriminator is accessible to the information of object B and still judges it as foreground. Therefore, we don't find apparent performance drops on occluded objects. The accuracy of discriminators on the test sets sampled from the RF7 datasets is over 99% (details are provided in Suppl. D.2, Line 505-509).

$**9. Benefiting other works with our ODGEN**$

It's good to know that work [A] may be benefited from this paper. We also hope that ODGEN can help and inspire the community in the future.

[21] Y. Jia, et al. Dginstyle: Domain generalizable semantic segmentation with image diffusion models and stylized semantic control. CVPR 2024 workshop.

[37] D. Peng, et al. Diffusion-based image translation with label guidance for domain adaptive semantic segmentation. ICCV 2023.

[64] G. Zheng, et al. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. CVPR 2023.

Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.

$**1. Fine-tuning on foreground objects:**$

Both the model fine-tuning and the object-wise conditioning module are designed to enhance foreground object generation but not limited to small objects. These two parts are conducted in sequence and contribute to synthesizing both large and small objects together. Given an object detection dataset, we fine-tune Stable Diffusion with the proposed approach to make it capable of synthesizing all kinds of foreground objects and provide visual prompts for the training of the object-wise conditional module. The image lists in the object-wise conditioning module provide the information of category and position for foreground objects. As shown by the visualized samples and quantitative results, our approach obtains improvement in generating dense and small objects compared with other methods and achieves better layout-image consistency.

$**2. Visual relationship between image lists and synthesized images:**$

The image list is designed to provide the information of category and localization for the control of foreground object layouts. The synthesized images share the same category with objects in generated images and are pasted to the same position as the objects in generated images to build image lists. The objects in image lists and objects in synthesized images may all be apples or cakes but have different shapes and colors. More visualized samples are provided in Fig. 2 of the PDF file enclosed in the global response.

$**3. Attribute binding:**$

Our approach is designed to synthesize object detection datasets that focus on the categories of foreground objects. As a result, the attribute binding is not in the scope now. Our work focuses on the concept bleeding problem of categories while MIGC pays additional attention to attributes. Taking the right part of Fig. 6 in our paper as an example, our ODGEN fixes the concept bleeding problem of categories and generates the concepts of motorcycle and vehicle correctly. We consider to address the challenge of attribute binding in future work to obtain ODGEN models with powerful attribute binding capability by training on datasets containing annotations of attributes (e.g., Visual Genome [a]). We will also highlight the difference between ODGEN and MIGC in the revised manuscript.

$**4. Overlapping objects:**$

The image list in the object-wise conditioning module contributes to the improvements in object occlusion. Our ODGEN introduces guidance for objects with synthesized visual prompts. We employ an image list to paste the visual prompt for each object separately on different empty canvases. As a result, ODGEN has access to the localization of overlapping objects with visual knowledge in the image list. We ablate this design in the ablations part (the middle part of Fig. 6 in our paper) to show that it contributes to the synthesis of overlapping objects. Experiments also show that our method contributes to the improvement in complex scene synthesis compared with prior works. We will emphasize this part in the method section of the revised manuscript.

$**5. Open-source:**$

We have plans to open-source the code and model weights.

[a] Krishna R, Zhu Y, Groth O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 2017, 123: 32-73.

Dear Reviewer N8et,

We are writing to kindly remind you that the time left for discussion is only about one day. Could you please confirm that you have read the rebuttal and check if you still have any concern about our work? Thanks for your efforts in reviewing this paper and proposing valuable comments.

Authors.

Thanks for your valuable comments. We provide detailed responses below to resolve your concerns.

$**1. Encoder architectures:**$

In this paper, we change the channel number according to the maximum object numbers that can be found in a single image. For datasets like MRI in which most images contain only one object, we can use fewer channels to make the model more lightweight. For a generic model trained on large-scale datasets, we can set the channel number with a higher enough value that can cover most circumstances. For example, we are working on a model trained on about 3000k images and set the input channels as 150 to make it applicable to images containing at most 50 foreground objects.

$**2. More extensive evaluation with larger-scale datasets:**$

See the experiments provided in parts 2 and 3 in the global response.

$**3. ``Synthetic only" not performing well:**$

To the best of our knowledge, the current synthetic data are mostly used to serve as augmentation data instead of training detectors on them only. A similar conclusion was drawn by prior work like GeoDiffusion [a]. Although our approach has achieved improvement in complex scene synthesis, there still exists noticeable gaps between real data and synthetic data. For example, we train ODGEN on 80k images from COCO and synthesize 10k images to train YOLO models from scratch. However, compared with the YOLO trained on 10k real images from scratch, the YOLO model trained on synthetic data only still falls behind (YOLOv7 mAP@.50: trained on 10k synthetic images only 24.40 v.s. trained on 10k real images only 53.20, same labels applied). We look forward to further improvement in future work with more powerful generative models and control methods to narrow the gaps. Besides, as illustrated in the limitations part (Suppl. A), the current dataset synthesis pipeline (Fig. 3 in our paper) is not fully optimized and cannot reproduce the distributions of foreground objects completely, which may also lead to worse performance of training on synthetic data only on RF7 datasets (labels used for COCO experiments are directly obtained from real images).

$**4. FID and mAP:**$

FID is used to evaluate the distance of distributions between generated samples and real samples by extracting features from images and computing the mean and covariance values. However, it's hard for FID to reflect the condition(layout)-image consistency, which is required by the object detection task and can be reflected by mAP results. Taking Fig. 5 in our paper as an example, ReCo failed to generate the bed within the orange box or generate cups within the purple boxes. Similar misalignment phenomena can be found in Fig. 10, 11, and 12 as well. It will hurt the detector's training and make ReCo perform much worse on trainability (in terms of mAP) than our ODGEN.

$**5. Comparison with the fully supervised approach:**$

We presented fully supervised experiments with training on real data only as the baseline on RF7 datasets in Tab. 2 of our paper. Similar experiments on the COCO dataset are added to part 2 of the global response. The copy-paste method requires segmentation masks to get the cropped foreground objects, which are not provided by the RF7 datasets employed in this paper. Besides, our ODGEN can be implemented without segmentation masks.

$**6. Computational complexity and training time:**$

Taking the model for the COCO dataset as an example, our ODGEN shares a very close scale of parameters with ControlNet (Trainable parameters: ODGEN 1231M v.s. ControlNet 1229M. parameters in the UNet of Stable Diffusion are included). We provide the training time for 1 epoch on COCO with 8 V100 GPUs of different methods in the Table below (the training code of MIGC is not open-source and thus not included here):

[a] GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation, ICLR 2024.