High-quality Text-to-3D Character Generation with SparseCubes and Sparse Transformers.

Thank you for your valuable suggestions and comments! We have tried to address your concerns in the rebuttal text:

Contribution.

Previous methods like Instant3D [1] and InstantMesh [2] relied on triplane representations to train large reconstruction models. However, the triplane approach limits the ability to implement sparse methods. Directly using high-resolution voxels is extremely challenging due to computational resource and memory overhead, as shown in Table 3 of our paper. Our method effectively addresses this issue by significantly reducing both computational cost and memory usage.

Differences Between Our Method and XCUBE.

Although both XCUBE and our method utilize sparse voxels, there are significant differences. As you mentioned, XCUBE is supervised by 3D losses, meaning that the sparse voxel hierarchies are pre-defined and can be directly used for training. In contrast, our method is only supervised by 2D losses, with ground truth limited to 2D images and no known 3D structure. During training, we derive sparse information solely from 2D images, whereas XCUBE requires 3D priors to obtain sparse information. This is the essential distinction between the two methods.

Relationship Between 3D Character Generation and Sparse Cubes/Transformers.

We have demonstrated that our method also achieves state-of-the-art performance on a general object dataset (LVIS subset). However, the primary focus of our method is 3D character generation, as characters often feature very thin structures like hair and fingers, as you noted. These structures provide a challenging testbed to showcase the effectiveness of our approach.

Implementation Details.

The rendering process is the same as in FlexiCubes. Learnable positional embeddings are employed in both stages. The shapes of the positional embeddings for the first and second stages are $32^3 \times 1024$ and $64^3 \times 1024$, respectively. During training, the second-stage positional embeddings are initialized by upsampling those used in the first stage. In the cross-attention mechanism, the positional embeddings act as the query sequence, while image tokens from DINO serve as the key-value sequence.

Inference Time

Since the final sparse cube resolution we achieve is 256, the resulting mesh contains significantly more faces than InstantMesh [2]. Consequently, mesh extraction and UV unwrapping require more time, making our method slower than InstantMesh. Specifically, inference for the coarse proposal network and the sparse transformer together takes approximately 5 seconds, mesh extraction requires 5 seconds, and UV unwrapping takes 10 seconds.

Comparison Between the Coarse Proposal Network and the Sparse Transformer.

We evaluated our method using the same test set as in the experiments of Table 1 in our paper, with the results shown in Table 1 below. Additionally, we compared the quality of the finger meshes generated by the coarse proposal network and the sparse transformer. The results, presented in Figure 11 of the appendix n our paper, clearly show that the sparse transformer significantly improves the quality of the fingers in the generated meshes.

Table 1: Comparison between the coarse proposal network and the sparse transformer.

[1] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint, 2023. [2] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint, 2024.

Thanks for the rebuttal and explanations.

There might be some misunderstanding in this question, Differences Between Our Method and XCUBE. I totally understand the difference between your settings and XCUBE. My point is: I am curious how the proposed methods compared to other sparse-based representations. You can do the exactly the same thing as you do while using XCUBE's backbone. Would it be feasible? And will there be advantages? There may not be enough time to do it now.

Thank you for your valuable suggestions and comments! We have tried to address your concerns in the rebuttal text :

Novelty of Our Work

Previous works like CharacterGen [1] and InstantMesh [2], did not utilize high-resolution voxels. Instead, they relied on triplane representations. Directly using high-resolution voxels is extremely challenging due to the computational and memory overhead, as shown in Table 3 of our paper. Our method effectively addresses this issue, significantly reducing both computational cost and memory requirements.

Results Using FlexiCubes as a Post-Processing Step.

The results presented in our paper are the outputs obtained using FlexiCubes.

Quantitative Improvements in PSNR and SSIM.

Using of mask loss, depth loss, and normal loss focuses on generating higher-quality meshes, but slightly reduce PSNR and SSIM. To address this, we employed a multi-view back-projection approach to further enhance texture quality, as discussed in the last subsection of Section 3.3.

Comparison with CharacterGen.

We conducted a detailed comparison between CharacterGen and our method, as shown in Table 1 of our paper. Specifically, we implemented Instant3D with DMTet, the approach adopted by CharacterGen [1]. For further evaluation, we also utilized the weights released by CharacterGen. The results, presented in Figure 10 in the appendix, clearly demonstrate that our method significantly outperforms CharacterGen, producing meshes with finer details and textures that are noticeably sharper and more realistic.

Training Strategies.

The entire network can be trained in an end-to-end manner. However, to expedite training, we first optimize the coarse proposal network, then use it to warm up the second stage, and finally optimize the entire network, as described in the second subsection of Section 3.3.

More Qualitative Results.

Due to the file size limitations of the PDF, we were unable to include more qualitative results. However, additional results can be viewed in the videos provided in our supplementary material.

[1] Hao-Yang Peng, Jia-Peng Zhang, Meng-Hao Guo, Yan-Pei Cao, and Shi-Min Hu. Charactergen: Efficient 3d character generation from single images with multi-view pose canonicalization. TOG, 2024. [2] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint, 2024.

Thank you for your valuable suggestions and comments! We have tried to address your concerns in the rebuttal text :

Paper Structures and Writings.

Thank you for your advice. We have addressed and corrected the issues you mentioned.

Geometry Quality.

Generating meshes with extremely high geometric quality is a challenging task. However, compared to other methods, our results already demonstrate significantly higher quality and are the closest to the ground truth. We plan to further enhance the geometry quality in future work.

Comparison with Human-Specified Methods.

We have compared our method with Tech [1], a human-specified method. The specific results are shown in Figure 9 in the appendix. From these results, we can observe that SMPL-based methods are not well-suited for anime characters.

Design of The Features Within SparseCube

We use learnable positional embeddings in both stages of our method. The positional embedding shapes for the first and second stages are $32^3 \times 1024$ and $64^3 \times 1024$, respectively. During training, the second-stage positional embeddings are initialized by upsampling the first-stage positional embeddings. Additionally, in the second stage, the positional embeddings are sparsified based on the results from the first stage. In the cross-attention mechanism, the positional embeddings act as the query sequence, while the image tokens from DINO serve as the key-value sequence.

[1] Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. In 3DV, 2024.