GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement

1. The approach involves multiple stages.

During inference, the model directly generates a mesh in a feed-forward manner, followed by texture refinement. Although the inference process consists of the two stages, our ablation study in Fig.10 demonstrates the effectiveness of texture refinement. Additionally, we modified the manuscript by adding more comparisons with MeshLRM in Fig.11. It shows how our method facilitates the generation of higher texture details. For future work, we aim to enhance detail generation while simplifying the pipeline and reducing computation time.

2. Missing evaluation (FID, CLIP) of generation quality.

As the model performs a 3D reconstruction task with ground truth novel views of 3D assets available, we use 3D reconstruction metrics to evaluate the quality of the generated outputs. These metrics provide a more accurate reflection of reconstruction accuracy compared to generative metrics.

3. Time breakdown for each step.

We provide a detailed breakdown of the time required for generating a single shape on a single 80G A100 GPU. Specifically, generating triplane features from multi-view inputs takes 0.703s, while mesh generation using DiffMC takes 0.126s. At this stage, we obtain a mesh with faces and vertices. We mention that the Marching Cube algorithm is time efficient. Following this, we apply our texture refinement process, which requires an additional 3.518s for 20 iterations of optimization. In total, a shape can be generated within 5s.

4. Source of depth and normal loss supervision.

We would like to clarify that depth and normal losses are designed for training. The training dataset consists of Obajverse-like 3D assets, where depth and normal information are rendered from the underlying 3D shapes. During inference, the model generates a 3D mesh with the input of multi-view images, where depth and normal information are not required.

1. Ablation on model architecture.

We conduct an ablation study to compare upsampling approaches (PixelShuffle vs. Deconv) and different resolutions of the triplanes (32, 44, 128, and 256), as detailed below. All experiments were performed using a single A100 node with 8 GPUs, training the NeRF stage for one day.

Additionally, we conducted ablation of image encoder (DiNO VS Conv.) in the supplementary.

2. Time cost.

We provide a detailed breakdown of the time required for generating a single shape on a single 80G A100 GPU. Specifically, generating triplane features from multi-view inputs takes 0.703s, while mesh generation using DiffMC takes 0.126s. At this stage, we obtain a mesh with faces and vertices. Following this, we apply our texture refinement process, which requires an additional 3.518s for 20 iterations of optimization. In total, a shape can be generated within 5s.

3. Open source.

To support the academic community, we commit to open-sourcing our work, including the code and pre-trained checkpoints, to facilitate further research and development in 3D generation and downstream applications. 4. common failure case of this method. We thank the reviewer for the valuable question.We added limitations into the modified manuscript. In our evaluation, when input multi-views are synthetic with pixels that are not perfectly aligned across views, our model has to hallucinate a reasonable shape to handle the inconsistency. As shape reconstruction from synthetic multiviews could be an important application, the generalizability of our shape reconstruction model will be considered for future study.

5. Table 1.

We thank the reviewers for the useful feedback. We fixed the typos and updated Table 1.

1. Typos and word consistency

We thank the reviewer for the useful feedback. We fixed typos in the modified version.

2. Comparison with baselines.

LRM and SV3D take a single-view image as input, whereas the other baselines that came out more recently, including CRM, LGM, and InstantMesh, use multiple views as input, making them more comparable to our setup.

3. Illustrate limitations of the work.

We added limitations to the modified manuscript:

"Current triplane features at a resolution of 256 are sometimes insufficient to capture fine geometric details. Meanwhile, alternative representations— explicit (e.g., sparse voxels) or implicit (e.g., vector sets)—offer promising avenues for exploration. Additionally, our approach currently requires camera-conditioned input. It would be more advantageous to develop methods capable of handling unposed input. Moreover, our work focuses on object-centric assets, leaving the exploration of complex scenes with composite objects as a valuable direction for future research."

(Below are responses to the questions)

4. Full-resolution supervision.

By employing mesh representation, the surface of an instance can be efficiently detected using the Marching Cubes (MC) algorithm. Subsequently, by leveraging rasterization instead of neural rendering, we eliminate the need for sampling along camera rays. Additionally, our SDF and color MLPs are lightweight, consisting of only 3 and 4 layers, respectively. In contrast, MLPs used in NeRF renderings typically have around 8 layers, further highlighting the efficiency of our approach.

5. Plucker embedding.

We represent Plücker coordinates as r=(o x d,d), where o and d denote the origin and direction of a camera ray for each pixel, respectively, and x represents the cross-product. In this formulation, both the origin and direction of the camera ray are taken into account, while the embedding remains independent of any specific point along the ray.

6. 3D asset dataset.

The internal 3D asset dataset was taken outside the company and used during the NeRF training stage, while only the Objaverse dataset was utilized during the geometry refinement stage.

7. CD and IOU.

We thank the reviewer for their valuable feedback. In the modified version, we have clarified the text by introducing the full name "Chamfer Distance" (CD) before using its abbreviation. Additionally, we specified that IOU refers to mIOU. Both CD and mIOU are computed in 3D space as mentioned in Sec. 4.1 (Evaluation), where we sample 16k points on the ground truth and generated meshes for evaluation.

1. Comparison with Mesh-LRM.

For non-open-source works like MeshLRM, which lack publicly available implementations or official demos, direct comparison is challenging. Inferring implementation differences solely from manuscripts is also challenging. In such cases, we rely on qualitative comparisons based on reported results, as shown in Figure 11 of the modified manuscript.

We argue that the texture refinement is useful and it provides advanced results than the baselines. As shown in Figure 11, while some concurrent works demonstrate improved results in certain aspects, they fall short in achieving challenging 3D reconstruction tasks such as reconstructing detailed textures. In contrast, our proposed texture refinement enables us to achieve superior visual quality. Additional results of our method are shown in Figure 10, where our method can generate good texture details on challenging cases such as human faces and text.

2. Bumpy geometry.

We acknowledge that our geometry is not flawless, as bumpy surfaces may occasionally occur. We speculate that this issue could be mitigated with larger-scale training and advancements in 3D representations. Improving the quality of geometry will be a focus of future research.

3. Limited novelty.

We understand that this field is rapidly evolving, and newly published arXiv papers may influence the reviewers' decisions. However, we have made every effort to compare our work with concurrent studies included in the manuscript, to the best of our knowledge. We sincerely hope that the reviewers will evaluate our contributions in comparison to works that are accessible.

“1. MeshLRM has released an online demo from its first author."

We understand the reviewer is referring to the demo available on the project website. We believe there is some misunderstanding regarding the term "unofficial." As stated on the authors' project page, the demo is described as an unofficial demo reproduced in Hillbot. In this context, "unofficial" implies that aspects such as the training data, model implementation, training resources and training process may not have been faithfully reimplemented as described in the original paper. Consequently, the results produced by the demo may differ from those reported in the original work. Therefore, we chose to base our comparisons on the results presented in the original paper, specifically as shown in Fig. 11 of our paper.

2. The grid-like artifacts in Fig.6.

We thank the reviewer for the valuable feedback. Upon investigation, originally we found that the grid artifact issue is likely caused by the deconvolution operation on triplanes as used in previous works. To address this, we chose to use pixel shuffle layers to mitigate grid artifacts in textures and geometry. An ablation study was conducted to evaluate this change, as presented in the next. While it improving the quality, further experiments on upsampling techniques can be explored to further eliminate the grid artifacts. We will include this analysis in the limitations section and, as part of future work, conduct more extensive experiments on model design to better address these artifacts.

3. Ablation of model design.

We thank the reviewer for the valuable feedback. We conduct an ablation study to compare

(1) upsampling approaches (PixelShuffle vs. Deconv) and different resolutions of the triplanes (32, 44, 128, and 256), as detailed below. All experiments were performed using a single A100 node with 8 GPUs, training the NeRF stage for one day.

(2) DiNO encoder vs Conv.

In the appendix, we present an ablation study on DiNO encoder and Convolution layers. The DiNO encoder is pre-trained, and in our pipeline, it is fined-tuned with our model. In this ablation, we notice that the Convolutional layers shows faster convergence that DiNO. We explain this to the inductive bias learned by the DiNO encoder, which makes the model challenging and slow to converge to the new task. While careful design of hyperparameters for DiNO fine-tuning and initialization could improve its performance, in practice, we replace it with convolutional layers, which have fewer parameters and converge more quickly.

We appreciate Reviewer GvuK's dedication to the review process and their valuable feedback.

1. Comparison Using the Public Demo

We thank Reviewer GvuK for their comment, and we believe we have reached a consensus that the demo is reproduced, which introduces potential ambiguity regarding the data, implementation, training strategy, etc. Consequently, it is unclear whether the demo outperforms, matches, or underperforms the model described in the paper.

Following Reviewer GvuK’s suggestion, we evaluated the public MeshLRM demo. We observed that our results appear qualitatively superior to those provided by the demo. Specifically, when testing the examples shown in Figure 6 of MeshLRM, we noted several issues with their demo outputs: the fox and tiger examples exhibit blurry colors, while the monkey example displays visibly distorted legs.

Furthermore, we also noticed that the demo results seem to differ as well from those presented in their paper, raising questions about the consistency between the demo and the model described in their submission. Similar concerns have also been noted in the ICLR reviews of MeshLRM, where discrepancies between the demo and paper results were highlighted.

We noticed that the 3rd follow-up comment by the reviewer regarding the demo was not available to us while the manuscript was still editable. However, we would be happy to include visual comparisons to the demo results in the final version to further clarify our evaluation. Nonetheless, we have made every effort to provide fair and thorough comparisons based on the information accessible to us.

2. Convolution vs Linear & Unpatchify

• “Linear & Unpatchify”?

In MeshLRM, “Linear & Unpatchify” is used for upsampling triplane features. As stated in the MeshLRM paper: “Lastly, each triplane token is unprojected with a linear layer and further unpatchified to 8 × 8 triplane features via reshaping. This operator is logically the same as a shared 2D-deconvolution with stride 8 and kernel 8 on each plane.”

In our case, we did not use deconvolution layers for triplane upsampling, as highlighted in the ablation comparisons provided in our last response.

• “Patchify & Linear”

We assume the reviewer might be referring to the “Patchify & Linear” operation illustrated in Figure 2 of the MeshLRM paper for extracting image features. However, MeshLRM does not provide detailed implementation specifics for this operation, leaving room for interpretation. It is possible that these two operations could be equivalent theoretically, but without additional details, such equivalence remains speculative.

We fully understand and agree reviewer’s concern and want to clarify that we do not present replacing the encoder as a novel or distinct contribution. Rather, we consider our systematic model modifications as a cohesive contribution, supported by detailed ablation studies that demonstrate the effectiveness of each individual component.

1. Pipeline design.

We argue that the "similar overall pipeline" used in previous works did not achieve better results compared to ours, highlighting the effectiveness and contributions of our design improvements. We summarize our contributions in two key aspects:

Model Improvements: We identify and implement several effective model modifications based on the "similar overall pipeline." As detailed in Section 3.1 and Section 4.2-4.3, these modifications are both useful and necessary to enhance visual quality. We conducted additional ablation studies for model designs which are shown below.

Texture Refinement: We propose a novel texture refinement approach that significantly improves the quality of results, particularly in challenging 3D reconstruction tasks such as reconstructing text and human faces in 3D.

For open-source works, such as LGM, we perform both visual and quantitative comparisons. However, for non-open-source works like MeshLRM, which lack publicly available implementations or official demos, direct comparison is challenging. In such cases, we infer comparisons qualitatively based on reported results.

In Figure 11 of the revised manuscript, we provide additional visual comparisons with MeshLRM. The results on the GSO dataset demonstrate that our proposed texture refinement method achieves detailed and high-quality textures, showcasing its effectiveness.

2. Ablation on model architecture.

We conduct an ablation study to compare upsampling approaches (PixelShuffle vs. Deconv) and different resolutions of the triplanes (32, 44, 128, and 256), as detailed below. All experiments were performed using a single A100 node with 8 GPUs, training the NeRF stage for one day.

3. DiNO encoder vs Conv.

In our ablation study, the DiNO encoder is trained with the pipeline. We explain this to the inductive bias learned by the DiNO encoder, which makes the model challenging and slow to converge to the new task. While careful design of hyperparameters for DiNO fine-tuning and initialization could improve its performance, in practice, we replace it with convolutional layers, which have fewer parameters and converge more quickly.

Dear Reviewer dBU7,

Thanks again for your constructive and insightful feedback to strengthen our work. As the rebuttal period is ending tomorrow, we wonder if our response answers your questions and addresses your concerns. Any feedback is welcome and much appreciated!

Best,

Authors