Generating Images with 3D Annotations Using Diffusion Models

We answer the questions raised by reviewer B6MH below.

 

Q1 How to define camera extrinsic matrix and whether to use class-level canonical-space as the extrinsic matrix of identity. How to define objects with multiple symmetry axes such as round tables?

Our 3D-DST generation pipeline saves pose annotations in terms of the 3D viewpoints (azimuth, elevation, and in-plane rotation), as well as object distance, object center, and Blender camera parameters. This follows the convention in standard pose estimation datasets, such as PASCAL3D+ and ObjectNet3D. Similar to other 3D datasets such as Omni3D, the intrinsic matrix can be obtained from the focal length and principal point of the camera, and the extrinsic matrix is simply an identity matrix as we use a canonical camera. In 3D-DST, the object rotation matrix can be computed from the viewpoint parameters.

Moreover, as all CAD models share the same annotation convention, we further ensure the T-pose of all CAD models are aligned both in-category and across-category. For instance, the “front” of a vehicle and the “front” of a dog would face the same direction as they share similar semantic parts. This allows 3D models trained on our synthetic 3D-DST data to exploit semantic similarities across categories.

Regarding how to define objects with multiple symmetry axes such as round tables, we simply follow the convention in PASCAL3D+ and ObjectNet3D and set the azimuth rotation to zero. This applies to both rigid objects commonly used for pose estimation, such as round tables, bowls, and bottles, as well as other objects less used, such as apples and golf balls. As mentioned by the reviewer, these symmetry issues have been well-studied by previous works in pose estimation. In this work, we simply follow the convention in downstream tasks and we find models pretrained on our 3D-DST data working fairly well for both 3D pose estimation (Table 4) and 3D object detection (General Response 3, Section E).

 

Q2 It is counterintuitive to claim that edge maps are superior to depth maps because depth maps provide more 3D information, such as occlusion relationships, which goes beyond the 2D representation of edge maps.

We acknowledge that further study is necessary regarding the choice of edge maps or depth maps as the 3D visual prompts. We make the following observations:

1. The current inferior results when using depth maps as 3D visual prompts are attributed to the domain gap between the MiDaS depths used for training and the depths used for inference. Running MiDaS on synthetic images or running diffusion models conditioned on synthetic depths both result in large domain gaps with the MiDaS depth maps obtained from real images during training. We believe certain finetuning is necessary to close the domain gap and to fully demonstrate the advantage of depth maps over edge maps. We leave this as an interesting direction for future work.
2. Although edge maps provide less 3D information than depth maps, it is still a compelling choice for 2D image generation despite its simplicity. With the scale-distance ambiguity, it abstracts away the need to sample a wide range of scales and distances during rendering, and directly provides necessary shape information that has been “projected” onto the 2D plane. Moreover, results on scene image generation with 3D-DST (in Section E of supplementary materials) also show that edge maps work fairly well for multiple objects with mutual occlusion.

We thank Reviewer B6MH for the valuable feedback and constructive comments, and we address the weaknesses below.

 

W1 Limited technical contributions:

Thanks to the reviewer for the comments. Please refer to General Response 2 where we summarize our results and highlight our contributions.

 

W2&3 The idea of 3D Visual Prompt via CG rendering and LLM Prompt is more like a combination of multiple previous effective techniques. The author's excessive focus on introducing background knowledge of known technologies like diffusion or cross-attention is unnecessary if the method utilized in this article relies on off-the-shelf approaches:

We respectfully disagree with the reviewer’s biased opinions against works built on existing methods. A successful work builds on existing technologies but solves new and challenging problems [A]. In this work, we show that with the help of diverse LLM prompts and controllable visual prompts, we can obtain high-quality images with 3D annotations. We demonstrate that using 3D-DST data for pretraining can achieve improved performance for image classification, 3D pose estimation, and 3D object detection. By combining graphics-based rendering with strong generative power of diffusion models and our diverse LLM prompts, we achieve promising results on both in-distribution data and out-of-distribution data.

In Section 3.1 we discussed the background of existing generative methods (diffusion models and ControlNet) for readers lack of related context, which is a common practice. Moreover, we discussed the limitations of existing methods and presented the motivation of our work, as a way to highlight our contributions. Given the reviewer’s advice, we will re-organize Section 3.1 to make the contents more succinct and highlight our key ideas.

[A] T Gupta et al. Visual programming: Compositional visual reasoning without training. In CVPR, 2023.

 

W4 Simple text prompts seem to be less directly relevant to the paper's introduction on adding 3D geometry control to diffusion models:

We acknowledge that the current introduction in the manuscript was not very clear about the challenges with “simple text prompts” and the motivation of our “diverse text generation”. We have revised this in the next revision. Here we would like to make two arguments motivating the necessity of LLM prompts.

1. Unlike typical 2D diffusion models such as Stable Diffusion, introducing 3D visual prompts as a condition to diffusion models largely limits the diversity of background and appearances, as well as the richness of details. This motivates us to encourage the diversity of output images from textual inputs, leading our proposed approach of generating diverse text prompts with LLMs. We qualitatively compare the generated images with and without LLM prompts in Figure 5.

2. In terms of the purpose of our work, we aim to develop a synthetic data generation method with 3D annotations that demonstrates better realism and diversity compared to graphics-based rendered data. Thus diverse LLM prompt serves as a core module of our approach and is key to achieving improved performance on OOD data in downstream tasks as presented in Table 3.

 

W5 Training details of networks.:

Given the limited space in the main text, we move some of the model training specifics from Section 4.1 “Implementation details'' to Section B in the supplementary materials. In Section 4.1 we report the network architectures, model sizes, and synthetic data size. In Section B of supplementary materials, we go through the choice of hyperparameters and training settings for both image classification and 3D pose estimation. Hyperparameters not discussed in our manuscript follow the choices from the publicly released code of DeiT, ConvNeXt, etc. We are happy to improve the reproducibility of our experiments so please let us know if additional details should be covered in our revision.

We thank Reviewer eSpe for the valuable feedback and constructive comments. Here are our responses to each concern.

 

W1 Purpose of 2D image classification task:

In this work, we present a synthetic data generation approach, 3D-DST, that produces synthetic images with controllable viewpoints and automatic 3D annotations at no extra cost. To demonstrate the advantage of our approach, we provide the following experimental results:

1. Improved 2D image classification on both in-distribution and out-of-distribution datasets (Table 1) by using our 3D-DST data for pretraining. These results demonstrate that our synthetic images possess a high level of realism (for ID classification) and diversity (for OOD classification) compared to other synthetic data, such as the 2D images in Text2Img.
2. Improved 3D pose estimation (Table 4) and improved 3D object detection (General Response 3, Section E). These results not only show the great potential of our approach on multiple 3D tasks but also verify the quality of the automatically obtained 3D annotations in 3D-DST.

We agree with the reviewer that improved 2D image classification can be achieved with standard 2D diffusion models without 3D-DST. In Table 3 we compared our approach with Text2Img that utilizes a 2D diffusion model for synthetic image generation, along with other complex modules such as language enhancement and CLIP filtering. Results show that pretraining on our 3D-DST images outperforms pretraining on Text2Img by a wide margin for both ID and OOD settings.

Despite the fact that 2D image classification doesn't rely on the automatic 3D annotations generated by 3D-DST, we argue that our approach is still a compelling choice for 2D tasks. The 3D visual prompts effectively ensure the class of the salient object, and with our 3D control module, viewpoints of objects in our 3D-DST images follow a uniform distribution, as opposed to the strong biases observed from a standard image diffusion model (Figure 6).

 

W2 Some necessary ablations are missing:

We thank the reviewer for pointing out the missing ablation experiments. Please refer to General Response 4 where we present the new ablation study results.

 

W3 The title is a bit misleading:

Thanks for the suggestion. We are considering using “Generating Images with 3D Annotations using Diffusion Models” as the new title. If you have other suggestions for the title, we are very glad to modify it accordingly.

 

Q What is the prompt to LLM for enriching the description?

In contrast to previous works (e.g., VisProg), which directly acquire desirable outputs from LLM-powered chatbots by feeding curated prompts describing the tasks and several example outputs for in-context learning, our approach leverages the generative power of LLMs to produce a series of diverse prompts describing the target objects. More specifically, we start from “A photo of a [class]”, and run LLM in an auto-regressive manner. We find that LLMs are capable of giving detailed descriptions regarding the color, texture, and shape information of the object, as well as diverse scene environments and actions. As demonstrated in Figure 5, prompts like “A photo of a taxicab covered in a thick layer of snow” and “A photo of a park bench covered in fallen leaves” give diversity to the objects’ appearance. And prompts like “A photo of a park bench nestled under the shade of a towering oak tree” enrich the background of the generated images. We will release our codebase with the implementation of our LLM generation module to reproduce our results.

Dear Reviewer eSpe,

Thank you very much for the valuable feedback and constructive comments!

 

Regarding the qualitative examples in Figure 7:

The horizontal frames are due to the ImageNet image preprocessing. Images are first upsampled and then padded to the same shape. Images with higher resolutions improve the performance of the Stable Diffusion and running diffusion models in batch mode significantly improves the data generation speed given limited time for the rebuttal experiments. In general, we find the quality of the produced images is not impacted by the image boundaries. We will re-run the experiments again without the boundaries.

 

Again, we appreciate the reviewer’s continuous efforts to provide helpful suggestions and valuable input, which make the paper better. If there is any further information or clarification you need from us, please feel free to let us know!

We thank the reviewer for the valuable feedback and constructive comments. Here are our responses to each question.

 

[Q1] Limited technical significance.

Thanks to the reviewer for the comments. Please refer to General Response 2 where we summarize our results and highlight our contributions.

 

[Q2] Though with promising application potential, it is hard to say what general principles that can guide the research in other domains can be distilled from the paper.

In general, our 3D-DST presents the following conclusions and guidelines that may benefit other 3D-from-2D tasks:

1. By leveraging the strong generative power of diffusion models, our 3D-DST data generation approach is an effective way to produce images with 3D annotations that demonstrate a higher level of realism and diversity. We demonstrate improved performance on both indoor and outdoor scenes with single-object or multiple-object images. These results show that 3D-DST is a generic approach with great potential in a wide range of 3D-from-2D tasks.
2. Despite the availability of 3D annotations from synthetic data, previous synthetic data generation methods were often hindered by the domain gap between synthetic and real, and the huge efforts to produce complex textures, realistic materials, and complex lighting conditions in graphics-based rendering. 3D-DST addresses the above problems with the powerful generative capabilities of diffusion models and allows us to quickly generate synthetic images for various 3D-from-2D tasks. In this work, we only consider existing 3D annotations from real images, but a lot of more useful 3D annotations as by-products can be exploited in future works.
3. Using graphics-rendered images is widely exploited in previous works as an effective approach to improve models’ performance on standard benchmarks. However, with 3D-DST, we argue that generating diverse synthetic images for pretraining is also an important approach to improve models’ robustness on OOD data, as demonstrated by 3D pose estimation on OOD data in Table 4. This is accomplished by our proposed 3D-DST pipeline and is not possible from previous synthetic data methods.

 

[Q3] There is no good guarantee that the objects in the generated images are consistent with the geometry described via the edge maps. Evaluations on whether the generated images are faithful to images and text conditions.

Thanks to the reviewer for raising this important question. We develop two metrics to measure the quality of the 3D annotations, namely real-to-synthetic performance (R2S) and pose consistency score (PCS). Moreover, we develop a simple yet effective approach, K-fold consistency filter (KCF) to remove images likely with wrong 3D annotations. We present some quantitative results and qualitative examples to demonstrate the quality of 3D annotations in our 3D-DST data. For future revision, we will conduct a thorough analysis and involve human comparison to quantitatively study the quality of 3D annotations in our 3D-DST data, as well as the efficacy of KCF. Please refer to General Response 1 and Section F for detailed results and discussions.

 

[Q4] It would be better if more potential applications could be discussed.

We agree with the reviewer that more 3D applications should be discussed. Therefore in General Response 3 and Section E in supplementary materials, we present the qualitative and quantitative results of 3D object detection with 3D-DST. Compared to other results shown in our paper, generating synthetic data for 3D object detection (i) involves scene image generation with multiple objects from multiple categories, and (ii) requires both 2D bounding box and 3D bounding box annotations to be accurate enough. Results show that our 3D-DST can also produce good scene images with multiple objects and diverse viewpoints, and state-of-the-art models pretrained on our 3D-DST scene images achieve improved performance on downstream tasks.

We thank the reviewer for the valuable feedback and constructive comments. Here are our responses to each question.

 

[Q1] The baseline result NeMo w/ AugMix is missing.

The baseline result of NeMo w/ AugMix is missing due to limited time and computing resources when we prepare the manuscript. We have added this baseline experiment and updated the results in our revision (Table 4).

 

[Q2] Could you discuss or ablate using other rendering types other than canny edges? Does canny edges work the best and why?

In Section 4.3 and Figure 4, we visualized some examples and discussed the choice of different rendering types. In general, we find using edges as visual controls works the best. We acknowledge the fact that using depth maps could potentially provide useful 3D information for 2D image generation. However, this approach is mainly hindered by the domain gap between the synthetic depths (during inference) and MiDaS depths from real images (during training). Meanwhile, canny edges as visual prompts are a compelling choice despite its simplicity. With the scale-distance ambiguity, it abstracts away the need to sample a wide range of scales and distances during rendering, and directly provides necessary shape information that has been “projected” onto the 2D plane. Moreover, results on scene image generation with 3D-DST (in Section E of supplementary materials) also show that edge maps work fairly well for multiple objects with mutual occlusion.

 

[Q3] Missing discussion for limitations.

Thank the reviewer for pointing out this issue. We will involve a thorough analysis of limitations in our next revision. Here we briefly summarize our limitations:

• Limitations of 3D models: Our data generation pipeline starts with the rendering of 3D object models. Despite the availability of large 3D model datasets, the abundance and diversity of 3D models can be limited for specific categories, such as gyromitra (a type of fungi) or plastic bags.
• Privacy and copyright issues: As our model is built on large pretrained generative models, e.g., Stable Diffusion, the generated dataset may reflect data presented in the training set, violating privacy or copyright agreements.
• Biases in training data of generative model: Although our model can generate diversified viewpoints and distances by adding 3D control to the data generation process, our pipeline may suffer from other biases inherent in the training data. As the Stable Diffusion model is trained on LAION-2B, known biases of the dataset may be observed in the 3D-DST dataset.