Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer

Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.

[Q1] Training on Objaverse dataset and quantitative comparison with state-of-the-art methods.

[A1] Thank you for your valuable comment. To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach.

[Q2] Lack of more quantitative results.

[A2] We add more quantitative results in Table 2-3 of the uploaded PDF.

Reference:

[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion.

Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.

[Q1]: Similar architectures to image generation and video generation.

[A1]: We appreciate the insightful question raised by Reviewer yUww. Our primary contribution lies in the innovative unification between the latent DiT framework and 3D content creation task, where previous methods, such as Shape-E and Michelangelo, have struggled to scale effectively for generalized 3D generation. As noted by Reviewer 3ERi, the 'origin, generative-based 3D AIGC pipeline is absolutely missing', and we address this gap by introducing the native 3D generation pipeline leveraging the specially designed 3D VAE (D3D-VAE) and the image-conditioned 3D latent DiT (D3D-DiT). This pipeline not only outperforms existing multi-view diffusion-based methods but also shows promise for scalability with increased 3D data availability.

The innovative nature of our work has been recognized by Reviewer 3ERi, who described it as 'a firm step towards a converged 3D AIGC framework,' and by Reviewer U1iF, who highlighted 'The DiT model is innovatively applied for the first time in conditional 3D generation'. We believe these points underscore the novelty and significance of our contribution to the field of 3D content creation.

[Q2]: It seems that a private dataset is used for training.

[A2]: To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach. As replied above, we will release the model trained on the Objaverse dataset to further promote openness and reproducibility in our research.

[Q3]: Results on in-the-wild input images.

[A3]: We present the results using in-the-wild images as conditional inputs in Figure 2 of the uploaded PDF, demonstrating that our method is capable of generating high-quality meshes. We will include more visualization results in the final version of the paper.

[Q4]: Another paper named ’DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data’.

[A4]: Thanks for the reminder. We also noted the naming problem. Actually, the paper mentioned was made public on 6 June, 2024, after the NeurIPS submission deadline, and we had conducted a thorough search to ensure that the name ’Direct3D’ was not already in use before submitting our paper to NeurIPS. We will cite and discuss the paper to help readers to distinguish the two works.

Reference:

[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion.

Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.

[Q1]: Quantitative comparison with state-of-the-art methods.

[A1]: To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach. As we replied in the global rebuttal part, we will release the model trained on the Objaverse dataset upon paper acceptance to further promote openness and reproducibility in our research.

[Q2]: Quantitative metrics of the explicit triplane latent representation.

[A2]: To validate the effectiveness of the explicit latent representation we employed, we established a validation set from the Objaverse dataset that does not overlap with the training set. We compared the reconstruction metrics of the VAE using explicit latent representation against methods such as 3DShape2VecSet [2] that utilize implicit 1D latent space. As shown in the table below, the explicit latent representation outperforms the implicit 1D latent space across all metrics.

[Q3]: Quantitative metrics of the semi-continuous surface sampling strategy.

[A3]: We also conducted ablation experiments of the semi-continuous surface sampling strategy on the Objaverse evaluation set. As shown in the table below, our proposed method demonstrates a notable improvement in the reconstruction quality of VAE.

[Q4]: Implementation details of near-surface sampling strategy.

[A4]: Actually in L258 of the original manuscript, we have mentioned that during the training of the VAE, we sampled 20,480 uniform points and 20,480 near-surface points for supervision. For the near-surface sampling strategy, we first randomly sample points on the surface of the mesh, then add a Gaussian perturbation with a std of 0.02 to each point to obtain the near-surface points. During training, the weights of BCE loss for both the near-surface points and the uniformly distributed points were set to 1.

[Q5]: A detailed discussion of how pixel-level and semantic-level conditions are implemented.

[A5]: We integrated the CLIP condition using cross-attention, a common approach in conditional diffusion models. However, we observed that meshes generated solely based on these semantic-level conditions lacked detailed alignment with the input image. To address this, we introduced additional pixel-level conditions using DINO, which previous research [2] has shown to outperform other pre-trained models in extracting structural information for 3D tasks. Specifically, we concatenated image tokens extracted by DINO with the noisy latent tokens and fed them into the self-attention layer, differing from the CLIP conditioning. This approach significantly enhances generation quality, allowing for a more detailed recovery of the conditioned input. We have conducted a qualitative comparison in Section A.3 and Figure 9 of the appendix.

[Q6]: A detailed discussion regarding the differences between the vector set representation used in 3DShape2VecSet and the triplane representation employed as the latent representation.

[A6]: Thanks for your valuable comment. In [A2], we conducted a quantitative comparison of the VAE reconstruction accuracy between the explicit latent representation and the vector set representation employed by 3DShape2VecSet. The results indicate that our explicit latent representation outperforms the vector set representation across various metrics, including Chamfer Distance, Volume IoU, and F-Score. In addition to geometric quality, another advantage of the explicit latent representation is its significant improvement in training efficiency for diffusion models. A recent work CLAY [3] also utilizes the vector set representation, and the training time of their diffusion model is approximately ten times longer than ours. Given the extensive computational requirements, with training necessitating thousands of GPU hours, it is challenging for us to complete the experiments within such a short timeframe. We will include a detailed discussion and comparative results in the final version of the paper.

Reference:

[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion.

[2]. Banani et al., Probing the 3d awareness of visual foundation models.

[3]. Zhang et al., CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets.

Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.

[Q1]: Training on Objaverse dataset and quantitative evaluation.

[A1]: Thanks for your comment. To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach. Finally, we will release the model trained on the Objaverse dataset upon paper acceptance to further promote openness and reproducibility in our research.

[Q2]: Eq (2) lacks input of $L_{BCE}$ and $L_{KL}$, which cause trouble in understanding the pipeline.

[A2]: Thanks for pointing this out. $L_{BCE}=BCE(o_{pr}(**x**), o_{gt}(**x**))$, where $o_{pr}(**x**)$ denotes the predicted occupancy of the given points $**x**$, which is the output of the decoder. And $o_{gt}(**x**)$ is the ground truth semi-continuous occupancy. $L_{KL}=KL(**z**)$, where $**z**$ denotes the latent representation obtained by the encoder.

[Q3]: Better to denote the shape of noisy latent and the output of DiT in Figure 2 to help readers better understand it.

[A3]: Thank you for your valuable comment. We will reorganize Figure 2 in the final version of the paper to enhance its clarity.

[Q4]: What is the reconstruction error of the proposed 3D-VAE?

[A4]: Here we evaluate the reconstruction loss of our D3D-VAE. For a given sampled point, we compute the Binary Cross-Entropy (BCE) between the VAE-predicted occupancy and the ground truth occupancy as the reconstruction error; Our VAE employs an explicit triplane latent representation, and we conducted a quantitative comparison in our split Objaverse evaluation set with methods such as Michelangelo [2] and 3DShape2VecSet [3], which utilize an implicit 1D latent space. As shown in the table below, our VAE demonstrates superior reconstruction quality compared to the implicit 1D latent space.

Reference:

[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion.

[2]. Zhao et al., Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation.

[3]. Zhang et al., 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models.

Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.

[Q1]: The distinctions between our Direct3D and LN3Diff.

[A1]: We thank reviewer 3ERi for bringing up the concurrent work. While LN3Diff was accepted to ECCV after the NeurIPS submission deadline, we are happy to discuss the work and will add this part to our final version.

Although both LN3Diff and Direct3D utilize the triplane latent, the VAE of these two methods are fundamentally different. As LN3Diff uses rendered RGB images as input, it is challenging for its VAE to encode high-fidelity geometric information compared to Direct3D, which directly applies dense point clouds with normal information as input. Furthermore, while our D3D-VAE employs direct supervision on the ground truth mesh, LN3Diff relies on supervision from rendered depth maps, which strictly speaking cannot be considered as direct 3D supervision. While this approach may lead to smoother generated mesh surfaces, it still struggles to capture the intrinsic structural information of the underlying 3D data.

[Q2]: Training on Objaverse dataset and quantitative evaluation.

[A2]: We re-train our model on the Objaverse dataset and conduct quantitative comparison with other methods, please refer to the global rebuttal or Table 1 of the uploaded PDF.

[Q3]: The requirement of high-quality 3D point clouds, and can only model geometry information.

[A3]: Direct3D is specifically focused on geometry generation, and we deliberately decouple geometry from texture to streamline the learning process for better geometry generation. In fact, we regard the exclusion of texture generation not as a drawback but as an advantage of the proposed pipeline: the separation introduces more flexibilities to the texture generation part as we can immediately apply advanced texture generation methods such as SyncMVD to our pipeline. It's note worthy that CLAY also employs a similar strategy by separating geometry and texture, by using a post-processing technique of normal-conditioned MVDream [1] for texture generation. Similar texture generation methods can be integrated into our approach easily if necessary.

[Q4]: The proposed DiT shares very similar design with DiT-3D / PixArt.

[A4]: We apologize for any confusion raised in the manuscript and we will provide more details in the final version. The proposed DiT is fundamentally different from DiT-3D and PixArt. DiT-3D is specifically designed for 3D voxel data and utilizes 3D attention to capture spatial relationships. In contrast, our DiT is designed for triplane latent space, which we treat as 2D images, as illustrated in Section 3.2 of the main manuscript, and therefore does not require 3D attention. Regarding PixArt, it is designed for 2D images and combines cross-attention with DiT to handle conditional information. However, we observe that the mesh generated solely based on these semantic-level conditions lacks detail alignment with the input image. To address this, we introduced the additional pixel-level conditions using DINO, which has been revealed in previous work [2] that outperform other pre-trained vision models in extracting structural information beneficial for 3D tasks. Specifically, we concatenate the image tokens extracted by DINO with the noisy latent tokens, and feed them into the self-attention layer. This approach significantly enhances generation quality, allowing for a more detailed recovery of the condition input. We believe this finding is crucial for advancing 3D generation research, as it explores the integration of image conditions with the current state-of-the-art DiT architecture.

[Q5]: Whether you can include comparison with other 3D origin generative-based methods, e.g., LN3Diff and CraftsMan?

[A5]: As LN3Diff only released the text-to-3D model, it is challenging for us to train and re-implement it from scratch within such a short timeframe. Therefore, we apologize for not being able to present a comparative analysis with LN3Diff and we will make every effort to train LN3Diff and include a comparison in our final version. Additionally, CraftsMan was publicly released after the NeurIPS submission deadline (on 23 May, 2024), yet we still conducted a comparison, with the results presented in Figure 1 of the uploaded PDF. It can be observed that our Direct3D demonstrates superior generalization compared to CraftsMan, and the generated meshes exhibit greater detail. On one hand, the VAE employed by CraftsMan still utilizes the implicit 1D latent space proposed by 3DShape2VecSet [3], resulting in lower reconstruction quality compared to the explicit latent representation that we have adopted. We conduct a quantitative comparison of the VAE with different latent representations in the Objaverse evaluation set, and the results are presented in the table below. On the other hand, CraftsMan relies on the multi-view diffusion model for conditional generation, leading to significant instability in their results.

[Q6]: Whether the trained models will be released?

[A6]: Yes, we will release the models trained on the Objaverse dataset upon paper acceptance to promote openness and reproducibility in our research.

Reference:

[1]. Shi et al., MVDream: Multi-view Diffusion for 3D Generation.

[2]. Banani et al., Probing the 3d awareness of visual foundation models.

[3]. Zhang et al., 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models.

Thanks the Direct3D author for the helpful discussion. Though the proposed method and writing was not sound enough at the submission, after adding the promised experiments and clarifications here, the proposed method can be considered as a NIPS paper. However, since these stuffs are missing in the submission and the Objaverse-version model was trained after the paper submission, I still wonder whether this paper is sound enough to be accepted in this round. I will keep my score as now and leave the decision to the AC, and personally I prefer to reject this paper now see a polished submission in the next round with all the promised experiments included.