3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion

1. Discussion and comparison to sparse-voxel based generation methods are omitted, including but not limited to:

   1. Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process, CVPR 2023
   2. SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation, CVPR 2023
   3. Locally Attentional SDF Diffusion for Controllable 3D Shape Generation, SIGGRAPH 2023
   4. One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion, CVPR 2024
   5. Make-A-Shape: a Ten-Million-scale 3D Shape Model, ICML 2024
   6. XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies, CVPR 2024
   7. MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model, NeurIPS 2024

   These methods are also parameter-efficient thanks to sparsity.

2. The shape and texture generation quality is lower than sota methods like CLAY, MeshLRM, and MeshFormer. The PBR also lacks high-frequency details.

3. Factual error in A.1.1: vecset can be extended to store color or PBR properties, by adding these properties along with PE in the encoding stage and decoding them by querying with positions. They are also differentiable renderable when combined with a differentiable marching cube algorithm and a differentiable rasterizer.

4. 4.2 Fig. 5 comparison with other methods: in the qualitative result part, all the meshes are not aligned, placed at different locations, and with different scales. Also, quantitative results on the image-to-3D task besides user studies are not reported.

5. Although claimed in the abstract, no real-world test cases are shown.

1. The primary concern is the $\textcolor{blue}{\text{apparent similarity}}$ between PrimX (the core design of this work) and PrimDiffusion (Chen et al., 2023b), which is scarcely mentioned, even in the "difference with related work" ( Sec. A.1.1). The authors should clarify the fundamental differences between two methods and prove (or clarify) why extending 3D Primitives to PrimX which including the PBR attribute (Material $\in \mathbb{R}^{a^{3}\times 2}$) is non-trivial.

2. Although the authors provide geometric and texture reconstruction results (e.g., Fig. 1, Fig. 5, and Fig. 7), they only show the predicted RGB images (texture) from the front view which could potentially be directly derived from the input image. The authors should showcase texture rendering from diverse viewpoints, including side and $\textcolor{blue}{**back views**}$, to better validate the model's capabilities.

3. Compared to cutting-edge 3D generation works (e.g., Real3D, LGM, CRM, InstantMesh), 3D Topia-XL requires substantial training resources and time, taking approximately 128 NVIDIA A100 GPUs and 14 days to converge. Apart from the mentioned "v-prediction", could pre-trained 2D models or other training strategies help accelerate convergence? How can the authors reduce training costs to democratize the proposed 3D generative model?

4. In the Image-to-3D setting (Sec. 4.2), 3DTopia-XL achieves overwhelming superiority in PBR rendering due to baseline methods' incapability to support spatially varied materials. It may be beneficial for the authors to render objects without reflections in a fixed and simplified lighting environment, then quantitatively compare with other methods on a common GSO dataset and report PSNR, SSIM, LPIPS, and CD metrics.

If the authors could address my concerns by providing corresponding quantitative or qualitative results based on the weaknesses and review feedback, I will consider improving my score.

1. In Eq. 3, v is defined as a set of N volumetric primitives; however, it is not clear what each primitive denotes.

2. It’s not immediately clear what N and D represent. Are these dimensions related to the number of primitives and data channels (e.g., geometry and texture details), or is there another interpretation? Further defining these terms would add clarity.

3. The paper mentions using initialized positions and scales, where I is a unit local voxel grid. However, it’s not entirely clear how these initialized positions and scales contribute to the overall volumetric representation in the PrimX model. More context on what this setup accomplishes geometrically would help clarify this step.

4. Unclear about the role of permutation equivariance in PrimX and its impact on the Transformer model's design. Clarifying how permutation equivariance inherently maintains structure among primitives or why it aligns well with Transformer processing would be helpful.

Weakness： 1）My concern is regarding the accuracy of the PBR materials. The fourth row in Figure 7 shows noticeably incorrect PBR materials. The paper claims to generate 3D assets with PBR textures, so it should include experiments on a 3D synthetic dataset where the ground truth metallic and roughness data are available. For example, quantitative evaluations can be handily conducted on a synthetic dataset with ground truth PBR data using PSNR or SSIM for albedo, metallic, and roughness maps. 2）As mentioned in line 292, the paper introduces patch-based compression aimed at incorporating inter-channel correlations between geometry, color, and materials. However, there is no experimental evidence to support this. As shown in Table 3, comparing line 1 and line 2, when the grid compression rate remains the same, reducing the feature dimensions representing different geometry, color, and materials from 6 to 1 actually results in a decrease in PSNR. Authors could provide additional experiments or analyses that directly demonstrate the benefits of incorporating these correlations, such as comparing the proposed approach to a baseline that processes geometry, color, and materials independently. 3）It seems that the experimental results are missing an ablation study where patch-based compression is not applied. It would be even better if qualitative comparison results could be added. The author could include an ablation study that compares their patch-based compression approach to a baseline without compression, showing both quantitative metrics and qualitative visual comparisons. 4）The current generated results lack diversity. It is better to include quantitative measures of diverse objects with complex topologies, such as challenging chairs, plants, and buildings,

In summary, I have concerns about the novelty of the paper. The 3D representation primarily extends existing work on M-SDF, and the performance of the extended features, such as PBR material, is not satisfactory. Additionally, despite the use of patch-based compression, the reported computational cost is substantial, requiring 16 nodes of 8 A100 GPUs around 14 days to converge. In comparison, InstantMesh only needs 8 NVIDIA H800 GPUs. Moreover, compared with instant mesh, the generated geometric results in this paper tend to be overly smooth, losing geometric details, as shown in Figure 5.