LRM-Zero: Training Large Reconstruction Models with Synthesized Data

We thank the reviewer for appreciating our writing clarity, the novelty and the value of our method, the generalization of Zeroverse for training both NeRF and 3DGS-based LRM models, and the effectiveness of our ablation study. We will respond to the reviewer’s comments below:

1. Results on OmniObject3D: As shown in Tab. 6, LRM-Zero has a 3.51 PSNR gap to GS-LRM on OmniObject3D. We believe that this is because OmniObject3D objects are more realistic (real captures) and have more diverse semantics than other testing datasets, e.g. GSO and ABO. Since LRM-Zero does not learn real-world object semantics from its training data Zeroverse, it thus performs worse on OmniObject3D than GS-LRM, which learns semantics from its training data Objaverse.
2. Training instability: We have identified that having too much boolean difference augmented objects in the training data is the main cause of our training instability issues. As shown in experiment 13 in Tab. 3 in the rebuttal pdf, when reducing the ratio of boolean difference augmentation, we do not need to change any training hyperparameters from GS-LRM, including the 0.5 perceptual loss weight. Thus, when extending LRM-Zero/Zeroverse to other tasks, if there is no substantial change in the Zeroverse data synthesis pipeline, there should not be training instability issues as long as the ratio of boolean difference augmentation is kept low. If not, our training stability experiences still provide valuable guidance: avoid making the synthetic object too complex, e.g. with boolean difference augmentation.
3. Related works on multi-view stereo: Thanks for curating the list of relevant works on MVS. We will include them in the revision.

Thanks for the authors' reply. Some of my questions are addressed. However, I think my following questions are not answered:

(1) L132: In public release version, seems the texture dataset is different from that is used in the paper. Will the performance become worse with the released texture dataset?

(2) Would it be interesting to finetune LRM-Zero on a small set of high-quality captured scenes to learn semantics and improve the performance?

For question (1), authors can answer it if they have any results on the public texture dataset. For question (2), though authors do not answer explicitly, I think the third row in Tab.1 of the rebuttal pdf shows that such finetuning on the Objaverse dataset with semantics may not help to improve performance on object reconstruction.

Thanks for the response. It would be nice to add the results on public texture dataset in the final version to help the readers understand the performance difference.

We thank the reviewer for appreciating the novelty and potential value for future works, the clarify of our writing, the comprehensiveness of our experiments and related works. We respond to the questions of the reviewer below:

1. Technical contributions: Building upon the prior work Xu et al. $1$ , we have added boolean difference and wireframe augmentations, which are crucial to the performance of LRM-Zero, and scaled up the data synthesis pipeline to synthesize and render up to 8M objects. Using the GS-LRM $2$ model architecture, we also iterated on data and model design together to resolve training instability issues and achieve competitive performance with GS-LRM. We believe that such experiences are major contributions in the large scale training era. For this reason, we also share all the details about our experiences in the paper, which is valuable to the community, as recognized by reviewer NfXg.
2. Sparse-view instead of dense-view setting: This is a good question. LRM-Zero indeed has a larger performance gap to GS-LRM under the sparse-view reconstruction setting, as shown in Tab. 1 and 7. This is because LRM-Zero does not learn real-world object semantics from Zeroverse. As a result, LRM-Zero cannot hallucinate the appearance and shape of an object where there is occlusion, which is common in the sparse-view setting. The sparse-view reconstruction task requires generation ability and knowledge of object semantics from the reconstruction model, which in turn requires training on a large dataset of realistic objects, e.g. Objaverse. LRM-Zero is more suitable for the dense-view reconstruction task. See the discussion of this limitation in Sec. B. Semantics in the Appendix.
3. Application of a dense-view reconstructor: As discussed above in 2., Zeroverse and LRM-Zero are not suitable for tasks that require generation ability and object semantics. To compensate for this, one can combine LRM-Zero with a generative model that possesses rich object semantics, e.g. the text-to-multi-view diffusion model from Instant3D $1$ . Also, when working with extremely sparse input views, we can use a video diffusion model to obtain dense input views, and then do the reconstruction with LRM-Zero.
4. Larger data size: We show LRM-Zero trained on more Zeroverse data (2x, 4x, 10x, and 20x) in Tab. 8 in the Appendix. As discussed in Sec. E Data size in the Appendix, we do not observe visible gains in increasing the data size, even up to 20x. Furthermore, when increasing the data size without also increasing the training steps, the model cannot converge well and performance worse than when using less data (experiments 2, 9, and 10 in Tab. 8). We also perform an experiment on GS-LRM, training it on a randomly sampled subset of 200K Objaverse objects. As shown in Tab. 2 in the rebuttal pdf, GS-LRM’s performance only drops by 0.1 PSNR on GSO. This is likely because for the single-object reconstruction task, the results start to saturate with about 200K realistic data. However, we believe that the advantages of Zeroverse, i.e. the data size, texture quality, controllability are more valuable when extended to other tasks, such as scene reconstruction and relighting, where data is scarse (see Sec. B Beyond object-level reconstructions in Appendix).

References

$1$ Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.06214, 2023

$2$ Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-lrm: Large reconstruction model for 3d gaussian splatting, 2024.

We thank the reviewers for appreciating our writing quality, the value of the contribution of our work, and the comprehensiveness of our experiment results. We will address the questions of the reviewer below:

1. Further improving LRM-Zero: This is a good question. We have made many attempts to reduce the gap between LRM-Zero and GS-LRM, including adjusting the ratios of augmentation, training hyperparameters, and scaling up the model size, data size, and training steps. However, our best result reported in the paper still has a gap to GS-LRM, as shown in Tab. 1. Among our attempts, increasing the data size was the most promising one, since we can synthesize unlimited amount of Zeroverse data, but it did not provide visible benefits, even at 2x or 3x model sizes, as shown in Tab. 8. This leads to a conclusion that LRM models trained on Zeroverse has a gap to Objaverse on the single object reconstruction task, where Objaverse is sufficient. However, we believe that the advantages of Zeroverse, i.e. the data size, texture quality, controllability are more valuable when extended to other tasks, such as scene reconstruction and relighting, where data is scarce (see Sec. B Beyond object-level reconstructions in Appendix).

2. Novelty of our method: Building upon the prior work Xu et al. $1$ , we have added boolean difference and wireframe augmentations, which are crucial to the performance of LRM-Zero, and scaled up the data synthesis pipeline to synthesize and render up to 8M objects. We also iterated on data and model design together to resolve training instability issues and achieve competitive performance with GS-LRM. We believe that such experiences are major contributions in the large scale training era. For this reason, we also share all the details about our experiences in the paper, which is valuable to the community, as recognized by reviewer NfXg.

3. Training instability: There are two reasons behind LRM-Zero’s training instability:

   1. In order to make our synthesized Zeroverse dataset generalize to real 3D data, it needs to be a superset of real-world objects. To achieve this, we need to make Zeroverse harder than Objaverse. GS-LRM, which does not have structured outputs like NeRF-based LRM, is itself prone to training instability issues. The benefits of making Zeroverse harder than Objaverse might not be obvious for the single object reconstruction task, but can open opportunities for other tasks where data is scarce.
   2. Having too much boolean difference augmented objects in the training data is the main cause of our training instability issues. As shown in experiment 13 in Tab. 3 in the rebuttal pdf, when reducing the ratio of data with boolean difference augmentation, we don’t have training instability issues with the default training hyperparameters from GS-LRM, including the 0.5 perceptual loss weight.

   With the above reasons, we think that the instability is explainable within the current presented framework.

4. Qualitative comparison: In Fig. 8 in our supplementary material, we have included 6 qualitative comparison results from LRM-Zero and GS-LRM on GSO and ABO. As suggested by the reviewer, we have included additional qualitative comparison results on our anonymous website (please check https://lrmzero2024.github.io/page_lrm_zero_vs_gs_lrm.html). The 4th to last row is where LRM-Zero outperforms GS-LRM on objects with detailed texture, since Zeroverse objects use a high-quality texture dataset. The last three rows are from Fig. 8 in our supplementary material showing where LRM-Zero performs worse than GS-LRM when there is invisible region in the input views (3rd to last row) and where they performs similarly when the input views have good coverage (last two rows). The remaining results are challenging samples that LRM-Zero and GS-LRM perform similarly or GS-LRM performs slightly better.

5. Training on both Objaverse and Zeroverse: As suggested by the reviewer, we conduct experiment 3 in Tab. 1 in the rebuttal pdf. It shows that training on both Objaverse and Zeroverse performs better than Zeroverse only, but worse than Objaverse only. This is likely due to the reasons discussed in 1. above: the size of Objaverse is adequate for the single object reconstruction task, and the advantages of Zeroverse are not exploited in this task but has high potential in other scenarios.

References

$1$ Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image-based relighting from optimal sparse samples. ACM Trans. Graph., 37(4), 2018.

We thank the reviewer for appreciating the value of our work to the community, the comprehensiveness of our ablation experiments, and our writing clarity. We reply to the questions from the reviewer as follows:

1. Qualitative comparison results: In Fig. 8 in our supplementary material, we have included 6 qualitative comparison results from LRM-Zero and GS-LRM on GSO and ABO. As suggested by the reviewer, we have included additional qualitative comparison results on our anonymous wepbage (please check https://lrmzero2024.github.io/page_lrm_zero_vs_gs_lrm.html). The 4th to last row is where LRM-Zero outperforms GS-LRM on objects with detailed texture, since Zeroverse objects use a high-quality texture dataset. The last three rows are from Fig. 8 in our supplementary material showing where LRM-Zero performs worse than GS-LRM when there is invisible region in the input views (3rd to last row) and where they performs similarly when the input views have good coverage (last two rows). The remaining results are challenging samples that LRM-Zero and GS-LRM perform similarly or GS-LRM performs slightly better.
2. LRM-Zero vs GS-LRM at longer training steps: it is true that the performance gap between LRM-Zero and GS-LRM is enlarged as they are trained for a longer time, as shown in table 8 and 9. As discussed in Appendix Sec. E, we believe that many of our early exploration on scaling experiments, including the larger performance gap at 2x training steps, suggest that the model converges slower on Zeroverse than on Objaverse. However, we believe that LRM-Zero's competitive performance is still impressive and that this does not undermine the potential of extending this work to other 3D tasks where data is more scarce (see Sec. B Beyond object-level reconstructions).
3. Generalization of our augmentation configuration: we did not optimize the Zeroverse augmentation configuration for the testing set metric numbers. Instead, Zeroverse aims to reduce the structural gap by introducing augmentations that allows for better coverage of common shape of real-world objects, e.g. boolean augmentation for concave shapes and wireframe augmentation for thin structure shapes, as detailed in Sec. 3.2. As shown in Table 4/5, some augmentations do not improve numerical metrics (e.g., PSNR) but we found that they are essential for human’s visual alignment. Given these philosophies behind, we think that our method should be generalizable and not just overfit the testing set (also evident in Table 9/Fig 5 with diverse testing data; and Table 5 with different model architectures).
4. Extending the primitive set: Thanks for the suggestion. This is an interesting idea that would ideally reduce the training cost on carefully ablating the shape augmentation design. However, since this requires lots of workload on both data synthesis and model training, we could not test this idea during the rebuttal period. This is a great idea for future work that we will consider.
5. Full computation budget: Experiments reported in Tab. 1, 2, 3, and 4 take 96 A100 days (we will correct Sec 4). The scaling experiments in Tab. 8 range from 96 to 384 A100 days. The total exploration, including 33 default-scale experiments and 31 scaling-up experiments, is approximately 9120 A100 days. The cost of Zeroverse data generation is approximately 3200 CPU node days. Note that the final model does not cost much, but the data exploration did. We recognized these hidden budgets. Thus, we decided to reveal all technical details regarding stability and release the data creation code. We hope that this can largely reduce the cost in reproducing or following our works.