DSPart: A Large-scale Diffusion-generated Synthetic Dataset with Annotations from 3D Parts

Q1: In Table 5, it shows that the PRF-filtered result under the UDA setting is close to the human-filtered result, except background. Is there any insights why background is special?

A: In Table 5, “torso” and “legs” of human-filtered data are 3.7 and 4.03 higher respectively in terms of IoU. These two parts occupy the most area of the foreground. Although the head IoU is close, and the tail seems to be much better, the performance of the background is still severely affected by the performance of the torso and legs.

Q2: It is interesting that, in Table 4 (Syn-only), the IoU of back legs and the tails are significantly worse than the previous synthesized dataset, CC-SSL. Is there any insights on that?

A: It is caused by the limitation of ControlNet and Stable Diffusion in generating small parts and accurate 3D poses from unusual viewpoints (refer to Limitation in the supplementary of current submission). The tail shapes of DSPart-Animal images are not always accurate and sometimes have distortions. However, CC-SSL was simply rendered by 3D assets which always have well-shaped tails. For back legs, back-views of animals are often the unusual viewpoints that 3D-DST fails. So DSPart-Animal has a very limited number of back-view animal images, and it will decrease the pixel frequency of the back legs. It also influences the performance of the tail.

Answer to weakness 1:

There is still a domain gap between the diffusion-generated data and real data. However, we demonstrate that DSPart is closer to real data distribution through experiments on PartImageNet and PascalPart than other synthetic datasets. Currently, the problem in terms of scalability is more caused by the bias of ControlNet and Stable Diffusion which leads to a high failure rate in generating images of objects with more complex structures from unusual viewpoints.

Although it will introduce a more severe viewpoint bias, a more scalable way to create animal images (where there is a high failure rate) is to follow the viewpoints from the human-filtered data. The generation pipeline will be more comfortable with those viewpoints and they can be applied to most 3D objects to generate more reliable samples. However, for DSPart, we want to create a viewpoint space that is as large as possible for our synthetic data. Although most unusual viewpoints are filtered by KCF or PRF, we can still obtain some successful examples for certain 3d objects under unusual viewpoints.

Answer to weakness 2:

The results shown in Table 3 are obtained using the 13k human-filtered data to achieve the highest performance. The PRF results in Table 5 are the results of 13k filtered images from other 150k unfiltered samples. For the left 26k images in DSPart-Animal, we are using more strict thresholds and we sample these 26k images from more than 300k unfiltered samples. We report the results of these 26k images which we use to evaluate the filtering quality here:

We will make these clearer in the revision and add the results of the unused DSPart-Animal images in the supplementary.

We thank the reviewer for their valuable feedback.

Q1. Is it able to adjust poses of 3D shapes to increase the diversity of 3D data?

A: Generally, we can adjust the 3D rotation, i.e. SO(3), and also some translation that won’t make the object truncated. For articulated objects like animals, we can create more animal poses by adjusting certain pose parameters of SMAL to change the joint angles. We can start with the SMAL poses and viewpoints combinations from the human-filtered data to guarantee the accuracy of generated samples. However, the difficulties lie in not making impossible and unrealistic animal poses when manipulating.

Answer to Weakness 1:

We agree that it is difficult to define part for large-scale categories. We think DSPart can consider the part definition approach of SAM. Our current 3d part definitions are adding spatial information to surface-based parts (closer to Pascal-Part style) since we don’t want annotating our defined parts in real images to be nearly impossible. These surface-based parts also enable us to leverage SAM. We can render the 3D assets from Objaverse into images from several viewpoints so that all parts can be visible, and then use SAM to give a subregion-based part segmentation. Then we can add our extra spatial information to these SAM-generate parts. Note that if two parts are both in one continuous and smooth surface, SAM often cannot segment the boundary (e.g. airplane head and body). Therefore, we still extra efforts but SAM can make the scaling up of part definitions much easier.

Answer to Weakness 2:

(1) Annotating 3D data is challenging but our annotation protocol only requires basic navigation operations of Blender that can be taught quickly in a one-hour session to annotators.

(2) It indeed limits the scalability but the scalability is still much better than annotating 2D parts in more fine-grained part definitions. A lot of self-occlusion, low resolution for some images, and natural shading will make annotating fine-grained parts in real images nearly impossible. SAM will not work well even with accurate bounding boxes when dealing with fine-grained parts.

Answer to Weakness 3:

We can generate the aforementioned images by utilizing multi-control, adding translations, and introducing occluders. However, the current design of DSPart aims to address challenging object-level part problems (e.g., 2D-3D part correspondences). Therefore, we are creating datasets in simpler scenarios to focus exclusively on our targeted challenges.

Q1: Given the categories included in DSPart-Rigid and DSPart-Animal, to what extent is this dataset likely to be generalizable to other object categories or part segmentation tasks?

A: In 2d part segmentation, the dataset can be generalizable to other object categories that are able to share the same part definitions as the available ones.

For potential 3d tasks that can leverage DSPart, we are confident that the 3d parts in DSPart can construct reasonably good part shape space that can generalize to novel shapes. We select our 3d assets that have enough shape variance within each category from large-scale 3d datasets (ShapeNet [1] and Objaverse [2]), making it possible to construct a promising shape pace of that category by interpolation of our selected shapes. Since parts have much simpler shapes (i.e. cuboid-like, cylindrical, spherical, etc), they can be represented by the interpolation of limited representative samples.

Answer to weakness 1:

We use Stable Diffusion 1.5 and control_v11p_sd15_canny for controlNet.

For the controllable data pairs, we believe the 3D-DST pipeline we employed exactly uses ControlNet to generate such pairs. Specifically, for each 3D asset, we render it from various viewpoints and utilize the edge maps of the rendered images to generate synthetic images. The 3D control is achieved by using the edge maps of different rendering configurations.

Answer to weakness 2:

The pipeline can be applied to any existing 3D shape part dataset to generate image and annotation pairs to scale up. However, the problem is the lack of annotated real datasets for evaluation.

Reference

[1] Chang et al. - ShapeNet: An Information-Rich 3D Model Repository.

[2] Deitke et al. - Objaverse: A Universe of Annotated 3D Objects.

Q2: Please provide additional details about PRF in response to Weaknesses-2 and Weaknesses-4. This may include explaining why PRF outperforming KCF is still meaningful, even if PRF has access to more reliable data, quantifying the contribution of each component within PRF, and providing visual examples.

A: Given weakness 2, we train the KCF on the same human-filtered data as PRF in two settings: 1) For each fold, we train the 3D pose estimation model on the human-filtered data for 50 epochs and then train on the train split of this fold for the left 50 epochs 2) We train the 3D pose estimation model on the human-filtered data for 100 epochs and directly evaluate it on the unfiltered data (the same process as PRF). The 2nd setting can prevent the model from training on less reliable data although it could not be called K-fold anymore since we do not perform the train/val splits on unfiltered data. The experiment results are shown below:

Surprisingly, training only on reliable data (KCF 2nd) will not obtain better-filtering results based on 3D rigid rotation. That the UDA performance of KCF 1st is worse than the original KCF in Table 5 further demonstrates it. We provide a visual comparison of examples that are filtered by PRF but not by KCF with their corresponding metrics and thresholds (please refer to Figure 4 in the supplementary of revision).

For weakness 4, we agree that current ways of comparing efficiency are misleading, which will be revised and clarified in the revision. The KCF’s inefficiency lies in its K-fold design which requires several train-test splits for every bunch of data. A fair comparison is to compare PRF with the 2nd setting above, which directly demonstrates the meaningfulness of PRF in changing metrics and algorithms to 3D animal pose estimation. The human-filtered data is also an important contribution of the PRF pipeline. It can be severed as more reliable data for fine-tuning or testing purposes.

For the doubt in scalability, the pose space indeed contains bias for future filtering. It is mainly caused by the bias of ControlNet and Stable Diffusion, which leads to a high failure rate in generating images of some animal poses from unusual viewpoints. However, at the performance level, our filtered images by PRF can achieve close performance as the human-filtered data, which is enough for scaling up the data volume following our image generation pipeline.

Q3: Please fully justify why the DSPart dataset is superior. As suggested in Weaknesses-3, exploring different versions of DSPart-Rigid could be insightful.

A: We perform the following experiments on the car category for further illustration. | data | # CAD models | # viewpoints | # images | syn-only | UDA | |:-----------------------:|:-----------:|:-----:|:-----:|:------:|:-----:| | UDAPart | 5 | 500 | 2500 | 58.40 | 64.71 |
| DS-UDAPart | 5 | 500 | 2500 | 58.89 | 69.27 |
| DS-UDAPart | 5 | 500 | 5000 | 59.14 | 70.41 |
| DS-UDAPart | 5 | 500 | 10000 | 59.01 | 70.53 |
| DS-UDAPart | 5 | 800* | 4000 | 59.34 | 71.64 |

Note that the viewpoints are uniformly sampled from the same interval of azimuth and elevation angles. The 5 CAD models of car categories are not included in DSPart-Rigid and we generate another set of images called DS-UDAPart. To control the viewpoints to be the same, we choose not to filter the images, and thus there will be some synthetic images with inaccurate annotations under unusual viewpoints.

The comparison between row 1 and row 2 implies the benefit of the realism of generated images. The comparison between row 2 and row 3 implies the benefit of increased data volume. However, when the amount of data grows beyond a certain point, performance improvements become increasingly marginal because the improvement brought by diverse textures has already saturated (row 3 & 4).

Q1. Please clarify the technical novelty of the proposed framework in response to Weaknesses-1. Highlighting the key advancements distinguishing 3D-DST from the proposed framework would be beneficial.

A: We thank the reviewer for their feedback and would like to clarify the main contributions of our work, which lie in three key aspects:

• Proposed PRF Filtering Strategy:

Our PRF strategy introduces a novel filtering approach that goes beyond 3D rigid body poses to incorporate articulation accuracy. By utilizing 2D and 3D part annotations in the DSPart dataset, we replace the 3D rigid rotation (SO(3)) metrics from KCF in 3D-DST with a combination of PA-MPJEP, S-MPJPE, and 2D PCK. This approach achieves more accurate filtering and avoids the vulnerability to unreliable associated with K-fold splits used in 3D-DST. Importantly, we are the first to apply 3D animal pose estimation for filtering diffusion-generated synthetic animal images. While this requires approximately 100 hours of human annotation, it also ensures a reliable and high-quality training subset for fine-tuning scenarios.

• Dataset Features and Annotation:

For a dataset paper, we argue that the form of data and annotation is also a crucial novelty. Compared to previous synthetic datasets, our DSPart dataset provides several key advantages:

(1) Realistic textures and plausible backgrounds for the generated images.

(2) Accurate camera parameters, 2D part masks, and 3D part geometry for each image.

As a comparison, among these attributes, 3D-DST satisfies only the first point and 3DCoMPaT++ satisfies only the second point. To the best of our knowledge, no existing synthetic datasets combine both features.

• Dataset Design:

The ways of shape selection and part definitions in DSPart are novel and meaningful for potential research. The assets of DSPart-Rigid are selected to have enough shape variance within each category since we want the shape space of that category to be represented by interpolation of our selected shapes. That shows a promising future in using our 3d parts to construct part-shape spaces. Unlike other fine-grained 3d part datasets that include many small parts and parts in the object interior which are nearly impossible to annotate in real images and too difficult for the current model, our 3D part annotations are fine-grained with adding spatial information to surface-based parts (more close to Pascal-Part style). Thus annotating real images for evaluation will be relatively more approachable. Adding spatial information can also motivate research to solve existing 3D spatial awareness problems (i.e. spatial ambiguity) of current algorithms.

In conclusion, we argue that DSPart is novel as a synthetic dataset. The filtering strategy has limited technical novelty, but as a dataset paper, we think it is not the most important component of this paper. The unique features of DSPart are designed to address current challenges and inspire new 2D-3D tasks in computer vision. We hope the reviewer will reconsider our contributions in light of these points.