Real3D: Towards Scaling Up Large Reconstruction Models with Real-World Images

• Novelty

(1) Actually, the self-supervised approach and the cycle-consistency method on single-image collections were already introduced in InfiniteNature-Zero [1], which addresses unposed scene generation, a more challenging task than this canonical-space-limited object generation setting. This related work is not even cited in the paper.

(2) The idea of using more in-the-wild data to improve generalization has been demonstrated effective in lots of previous semi-supervised approaches, such as Depth Anything [2] for depth estimation and GTA-Seg [3] for semantic segmentation. It’s not surprising that model performance improves with additional data.

(3) The ability to learn on single-image collection depends on the 3D prior of TripoSR (or other LRMs), thus this is, in essence, a two-stage training work and is more tedious to train from scratch.

• Experiment (Please also refer to "Questions" section)

(1) This comparison in this work is potentially unfair since Real3D is further fine-tuned on the baseline, TripoSR, on more collected data.

(2) The training data: Objaverse covers a wider data distribution range compared to OmniObject3D and CO3D datasets; MVImgNet and claimed collected real data may also cover the data distribution of the tested set.

[1] InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images. ECCV 2022 (Oral)

[2] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. CVPR 2024

[3] Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant. NeurIPS 2022

1. The paper seems to lack discussion on the role of synthetic data loss. From my understanding, the paper finetunes the TripoSR model using an Objaverse subset and a filtered monocular dataset, but the performance gains over TripoSR may stem from selected Objaverse subset and the monocular dataset. Specifically, Table 5 analyzes the improvements brought by the monocular dataset (using Equ. 10). For instance, introducing clean data loss improves PSNR from 18.44 to 18.60, adding semantic guidance further increases PSNR to 18.81, and introducing curriculum pixel-level constraints raises it to 19.18. However, the analysis only focuses on $L_{SELF}$ as defined in Equ. 10. The question remains: if we finetune TripoSR using only the Objaverse subset (with $L_{RECON}$ only), will there still be improvements compared to TripoSR? If so, how do these gains compare to those from $L_{SELF}$? Another example is the comparison between Real3D and TripoSR in Table 2, which shows limited improvements over TripoSR. How much of these gains are due to $L_{RECON}$, and how much is due to $L_{SELF}$ ?

1. The evaluation metrics are suboptimal. The paper employs PSNR, LPIPS, and SSIM, which are mainly designed for image reconstruction tasks. For single-image-to-3D generation, the unseen parts of an object inherently involve uncertainty and diversity, meaning there should not be a single, fixed ground truth for 3D shape or novel view images. Although CLIP scores are provided in some tables, this metric is also vague, and cannot reveal the image quality. The paper would benefit from including more generative-based metrics, such as FID, or conducting a human study or GPT-based evaluation.

2. The performance improvements shown by the current metrics are minimal, which may also relate to the issue noted above. An average PSNR below 20 suggests that the generated and target images are very dissimilar, so the small gap between methods may not reveal meaningful differences.

3. The ablation study in Table 5 is not fully convincing; it does not clearly demonstrate that each component has a consistently positive impact on performance. For example, in some cases, the inclusion of multiple modules results in lower scores than the baseline.

4. The semantic-level supervision relies on multi-view image CLIP embedding similarity; however, CLIP embeddings do not guarantee multi-view invariance. Although the authors state they avoid rendering novel views far from the input viewpoint, this approach represents a trade-off: either sampling views close to the input, which minimizes the intended effect, or introducing erroneous rotational symmetry that artificially increases multi-view CLIP similarity. Additionally, the ablation studies in Table 5 do not clearly demonstrate the benefits of this loss (full model vs. full model without segmentation guidance).

5. The data curation process lacks qualitative or quantitative evaluation. More examples or results would be helpful to illustrate the advantages of this process.

1. The LRM model has already been trained on MVImgNet. This raises questions for me about its novelty.

2. The performance improvements appear marginal, as demonstrated by the examples shown in Figure 1. Additionally, Tables 3 and 4 shows that Real3D performs similarly to TripoSR. I have yet to see impressive performance gains from incorporating real-world data into the training process, and this casts doubt on the Real3D scalability potential.

3. The results of Real3D seem sensitive to the elevation angles of input images, an issue that also affects the base LRM. Real-world datasets often contain images with significant elevation variations, unlike synthetic datasets where images are usually rendered with lower elevation angles for training. It would be valuable to explore strategies to address this challenge in future research.