Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly

We deeply appreciate the reviewer veoH for the thoughtful feedback and time invested in evaluating our work. We respond to each question below.

Q1:Applying DeepPriorAssembly on mobile edge devices.

We agree that DeepPriorAssembly is now not capable of directly applying on mobile edge devices. Actually, most of the large models (e.g. ChatGPT, StableDiffusion, LLaVA) face the same challenge of inference on mobile edge devices. A more practical approach for users would be to submit the captured images to remote servers for processing, where the reconstructed 3D scenes can then be sent back to the users.

Q2:The instance scale for Shap·E.

We got the same experimental conclusion as yours regarding Shape·E's reconstructions. Shape·E is indeed sensitive to the scale of input images. Actually, we conducted ablation studies to explore the effect of instance scales on the generation quality of Shape·E, as shown in Sec.C.2 and Fig.10 in the Appendix. The visual comparison of the generations with different instance scales in Fig.10 in the Appendix show that Shap·E is quite sensitive to the scale of instances in the images, where a too small or too large scale will lead to inaccurate generations with unreliable geometries and appearances. For quantitatively evaluate the effect of instance scale, we further report the numerical results of the scene reconstruction performance under different instance scales, as shown in Table B in the rebuttal PDF. Thought both quantitative and qualitative ablation studies, we observe that the Shap·E performs the best in shape generation from instance images at a scale of 6.

For all the experiments conducted in the paper, we set the scale to 6, following the conclusions of ablation studies. We believe that the scale of 6 is a universal appropriate scale that fits to most of the instances, as demonstrated by the comprehensive quantitative and qualitative evaluations on diverse datasets in Sec.4 and Sec.B of the Appendix.

Q3:Quality of estimated depth.

We ensure the quality of estimated depth by adopting the large model Omnidata which is trained under large scale dataset for depth estimation. In practical experiments, the quality of estimated depth is quite convincing due to the powerful large model with abundant knowledge learned by large-scale training on large-scale datasets.

We also admit that the depth point cloud may suffer from inevitable occlusions of unseen areas and distortion, especially in some difficult scenes. However, we justify that DeepPriorAssembly does not require very high-quality depths since they are only used as a prior for recovering the layout of the scene containing the reconstructed high quality 3D shapes with 2D/3D matching. The estimated depths are not directly used as the scene geometries, where the high quality of reconstructed scenes is ensured by the high quality of the reconstructed 3D shapes.

Q4:Applying DeepPriorAssembly to outdoor scenes.

We further conduct experiments to evaluate DeepPriorAssembly on complex outdoor scenes and scene containing animals, as shown Fig.A in the rebuttal PDF. The first image comes from KITTI dataset, others are collected from the Internet. With the help of powerful large foundation models, DeepPriorAssembly demonstrates superior zero-shot scene reconstruction performance in these real-world outdoor scenes.

We sincerely appreciate Reviewer DEJS for the acknowledgment of our work and constructive feedback. We respond to each question below.

Q1:Technical contribution.

We are the first to explore the cooperation among large foundation models for another extremely difficult task where none of them can accomplish alone. The key motivation of our method stems from the recent success in large foundation models, which lead a revolution in language/vision computing. The large models show brilliant capabilities and remarkable performance, but are limited in a specific task with a specific modality. Driven from this observation, we propose to explore an effective solution that leverages existing expert large models, designed and trained for specific tasks, to address the extremely challenging task of 3D scene reconstruction from single images. We aim at a zero-shot framework where no part of it necessitates extra data collection, preparation, or time-consuming data-driven training.

To this end, we propose DeepPriorAssembly, a novel framework which assembles diverse deep priors from large models for scene reconstruction from single images in a zero-shot manner. We rethink this task from a new perspective, and decompose it into a set of sub-tasks instead of seeking to a data-driven solution. We narrow down the responsibility of each deep prior on a sub-task that it is good at, and introduce novel methods related to poses, scales, and occlusion parsing to enable deep priors to work together in a robust way. We believe DeepPriorAssembly introduces a new direction for the society to flexibly exploit the potential of existing powerful large models.

Q2:Running time of DeepPriorAssembly.

We have reported the running time of each stage of DeepPriorAssembly in Sec.F and Table 5 in the Appendix. Reconstructing a scene from a single image takes 171.2 seconds in total. The inference of Grounded-SAM, Open-CLIP and Omnidata takes only about 1 second. The most time-consuming parts include the StableDiffusion, Shap·E and the RANSAC-like optimization. For a balance between the efficiency and quality, we can optionally reduce the sample number M of StableDiffuion and the iterations r of RANSAC-like solution, which can reduce the total time to less than 60 seconds. For the baseline method PanoRecon, the inference time is 30.6 seconds. We further justify that we do not require any extra time for data collection, data preparing and data-driven training. In contrast, PanoRecon requires 5 days of data-driven training, as reported in its paper, not including the time for data preparation.

Q3:Potential error accumulation.

We demonstrate that the effective integration of additional large models does not compromise the robustness of our method. In contrast, this design enhances both its robustness and accuracy. For instance, we incorporate StableDiffusion to enhance and inpaint images, resulting in an improvement in CDL1 from 0.125 to 0.110, as shown in Table 2 of the ablation study. The CLIP model is introduced to filter out poor samples, leading to more robust results, as shown in Table 3. The other proposed strategies and constraints (e.g., RANSAC-like solution, 2D/3D-Matching) are also designed to improve the robustness. Please refer to Sec.4.4 for comprehensive ablation studies on the effectiveness of integrating each module.

Q4:More comparisons with recent methods.

We have additionally compared our method with other SOTA data-driven scene reconstruction works BUOL [CVPR 2023] and Uni-3D [ICCV 2023] in Sec.D and Fig.12 of the Appendix. The results demonstrate that our method achieves better and more visual-appealing results under both 3D-Front and ScanNet datasets. Specifically, our method significantly outperforms other methods using real-world images in ScanNet.

Following your suggestions, we further compare DeepPriorAssembly with ScenePrior [CVPR 2023] under ScanNet dataset, as shown in Fig.B in the rebuttal PDF. The reconstruction results of ScenePrior is provided by its authors. As shown, DeepPriorAssembly clearly outperforms ScenePrior in terms of the quality of scene geometries. Moreover, ScenePrior can only reconstruct the geometry, whereas DeepPriorAssembly is capable of recovering high-fidelity scene appearances as well.

Q5:Relation between running time and the object numbers.

The relation is that scenes containing more objects will often lead to longer running time. However, as we analyze in Sec.F of the Appendix, the inference of Grounded-SAM, Open-CLIP, and Omnidata takes only about 1 second. The most time-consuming parts include StableDiffusion, Shape·E, and the RANSAC-like optimization. For these three components, we can process all instances of the scene in parallel (e.g., with multiple GPUs), which can significantly reduce the required running time. Specifically, by processing each instance of the scene with a separate GPU in parallel, the running time for a scene containing multiple instances may not be much longer than the time for scenes containing only one instance.

Q6:Leveraging 3D object detection for layout estimation.

The input for DeepPriorAssembly is only one single scene image, where no 3D data is available for 3D object detection. An alternative approach is to leverage the back-projected depth point clouds for 3D object detection. However, the depth point cloud is often of low quality and corrupted by occlusions of unseen areas. Therefore, it is extremely difficult for existing 3D object detection methods to accurately detect 3D objects from the corrupted depth point cloud. We also emphasize that DeepPriorAssembly does not require very high-quality depths since they are only used for recovering the layout of the scene containing the reconstructed high quality shapes. The estimated depths are not directly used as the scene geometries, where the high quality of reconstructed scenes is ensured by the high quality of the reconstructed 3D shapes.

We sincerely appreciate the reviewer 67ke for the invaluable feedback and time invested in evaluating our work. We respond to each question below.

Q1:The applications of single-view scene reconstruction.

The task of single-view scene reconstruction greatly contributes to the domain of AIGC, AR/VR, robotics, games, 3D design, etc.

1. AI content generation for 3D is a hot topic recently which generates diverse 3D models from user prompts (e.g. images.). Most previous works focus on generating 3D objects, which is much easier than complex scenes. DeepPriorAssembly flexibly generates complete scenes from single images, advancing the development of this field.
2. Recovering scenes from single views plays a curtail role in the field of augmented/virtual reality, which allows spatial interaction between human and environments.
3. This task also contributes to the field of robotics. DeepPriorAssembly can recover the environment around a robot using its camera, facilitating comprehensive scene understanding for semantic recognition, collision detection, human-robot interaction, and more.
4. Single-view scene reconstruction can greatly improve the efficiency of game production. Given a game scene image drawn by a game illustrator, DeepPriorAssembly can directly reconstruct the 3D game scene for interaction, eliminating the need for a 3D modeler to manually recreate each 3D object from the scene image and fit them to the correct position, rotation, and scale.

We will add more discussion and analysis on the applications of DeepPriorAssembly in our revision.

Q2:The motivation and originality of DeepPriorAssembly.

The key motivation of our method stems from the recent success in large foundation models (e.g. ChatGPT, VLM, StableDiffusion, CLIP, etc.), which lead a revolution in language/vision computing. By greatly scaling up sizes of training samples and model parameters, large models show brilliant capabilities and remarkable performance. However, they are limited in a specific task with a specific modality, which limits their capability in high level perception tasks.

Driven from this observation, we propose to explore an effective and robust solution that leverages existing expert large models, designed and trained for specific tasks, to address the extremely challenging task of 3D scene reconstruction from single images. Note that most large models are public available where everyone can access these deep priors without additional efforts. We are committed to provide new insights for the society on assembling existing powerful large models at different domains and tasks for tackling another more challenging task without extra knowledge. That is, we aim at a zero-shot framework where no part of it necessitates extra data collection, preparation, or time-consuming data-driven training.

To this end, we propose DeepPriorAssembly, a novel framework which assembles diverse deep priors from large models for scene reconstruction from single images in a zero-shot manner. We rethink this task from a new perspective, and decompose it into a set of sub-tasks instead of seeking to a data-driven solution. We narrow down the responsibility of each deep prior on a sub-task that it is good at, and introduce novel methods related to poses, scales, and occlusion parsing to enable deep priors to work together in a robust way. We are the first to explore the cooperation among large foundation models for another extremely difficult task where none of them can accomplish alone. We believe DeepPriorAssembly introduces a new direction for flexibly exploiting the potential of existing powerful large models.

Q3:The evaluation fairness with other methods.

Actually, all the previous methods on single-view scene reconstruction requires additional task specific data collection and time-consuming training. In contrast, our method merely assembles existing available large models without requiring any extra knowledge. We believe that other methods require more strict data settings than DeepPriorAssembly, but are also limited to the known data distribution. For example, PanoRecon requires a large collection of image-scene pairs from 3DFront dataset and task specific training, which performs well on the test set of 3DFront but fails to generalize to out-of-distribution images in the real-world. We believe that the evaluation is fair for other methods since none of them can handle this task in our experiment conditions, i.e., without data collecting and even without data-driven training.

Q4:Applying DeepPriorAssembly to outdoor scenes.

We further conduct experiments to evaluate DeepPriorAssembly on complex outdoor scenes and scene containing animals, as shown in Fig.A in the rebuttal PDF. The first image comes from KITTI dataset, others are collected from the Internet. With the help of powerful large foundation models, DeepPriorAssembly demonstrates superior zero-shot scene reconstruction performance in these real-world outdoor scenes.

Q5:Complexity of Figure 2.

We will simplify Figure 2 to provide a clearer illustration of the main idea by moving some framework details to separate figures.

We deeply appreciate the reviewer osDy for the thoughtful feedback and time invested in evaluating our work. We respond to each question below.

Q1:The robustness of DeepPriorAssembly.

We demonstrate that the effective integration of additional large models does not compromise the robustness of our method. In contrast, this design enhances both its robustness and accuracy. For instance, we incorporate StableDiffusion to enhance and inpaint images, resulting in an improvement in CDL1 from 0.125 to 0.110, as shown in Table 2 of the ablation study. Additionally, we introduce the CLIP model to filter out poor samples, leading to more robust scene reconstruction results (0.118 to 0.110), as demonstrated in Table 3 of the ablation study. The other proposed strategies and constraints (e.g., RANSAC-like solution, 2D-Matching, 3D-Matching) are also designed to improve the robustness of DeepPriorAssembly. Please refer to Table 2 and Table 3 for comprehensive ablation studies on the effectiveness of each module in improving the robustness of our method.

Q2:The heuristic selection of depth shift and RealmDreamer solution.

We greatly appreciate the insightful advice from reviewer osDy on depth shift estimation. We fully agree that using multiple image-depth pairs to compute the appropriate depth shift will lead to more accurate and robust depth scale and shift. We used only one image-depth pair to reduce the requirement for ground truth depths, and found the results to be convincing. Additionally, we provide ablation studies on the number of image-depth pairs in Table A in the rebuttal PDF.

RealmDreamer metrically grounds the depth scale and shift by aligning relative depths with the metric depths predicted by DepthAnything. We agree that this solution could potentially be a better approach for directly solving depth shift. However, RealmDreamer was released on arXiv in April 2024, only one month before the NeurIPS deadline, by which time we had already completed the core development of DeepPriorAssembly. To evaluate the effectiveness of a RealmDreamer-like solution in our framework, we replace our depth solution with it and conduct experiments in complex outdoor scenes, as shown in Fig.A in the rebuttal PDF. The results indicate that the RealmDreamer-like solution integrates well into our pipeline. Nevertheless, we justify that our depth solution also performs well across different datasets, as demonstrated by the comprehensive experiments in Sec.4.

Q3:The effectiveness of proposed pipeline for shape reconstruction from SAM instances.

As far as we know, none of the previous approaches can reconstruct complete and high-fidelity shapes from occluded 2D instances. We propose the first solution for this difficult task from a new perspective by assembling deep priors from large models, without requiring task-specific data preparing, model designing or training. Comprehensive evaluations and ablation studies demonstrate the effectiveness of the proposed pipeline for recovering complete and high-fidelity shape reconstructions from occluded and low-resolution SAM instances.

Q4:Why not train a LRM-like for shape reconstruction and pose estimation.

The key motivation of our method is to explore an effective and robust solution that leverages existing expert large models, designed and trained for specific tasks, to address the extremely challenging task of 3D scene reconstruction from single images. We are committed to provide new insights for the society on assembling existing powerful large models at different domains and tasks for tackling another more challenging task without extra knowledge. That is, we aim at a zero-shot framework where no part of it necessitates extra data collection, preparation, or time-consuming data-driven training. Training a LRM-like model for solving the part of shape reconstruction from SAM instances is not our way, as it requires extensive task-specific efforts in model design, data preparation, and training.

Moreover, we justify that none of the previous approaches can reconstruct complete and high-fidelity shapes from occluded 2D instances produced by SAM. The difficulties that prevent LRM-like techniques from successfully solving this task lie in:

1. The manner, location, and ratio of occlusions in SAM instances are unpredictable and vary significantly from scene to scene and instance to instance, making it extremely difficult to collect corrupted instance-complete shape pairs and to stably train an LRM-like model.
2. The 2D instances often suffer from low resolution. For small instances or instances far from the camera, the resolution can be low, leading to difficulties for a LRM-like model to accurately capture the semantics in the instance image and reconstruct shapes with details.
3. Real-world images contain diverse categories of instances, requiring an extremely large annotated dataset for this task. The scale of available 3D data (10M) is still much smaller than publicly available 2D/language data.

Our proposed pipeline effectively utilizes large models pretrained on billions of 2D/language data, demonstrating superior performance in reconstructing high-fidelity shapes given only corrupted SAM instances containing occlusions and low resolution as inputs.

Q5:Applying DeepPriorAssembly to real images, outdoor scenes or scenes containing animals or people.

We justify that the ScanNet dataset is a real-captured dataset. We have shown the scene reconstruction results under real images of ScanNet dataset in Fig.14 of the Appendix. We additionally conduct experiments to evaluate DeepPriorAssembly on complex outdoor scenes and scene containing animals, as shown in Fig.A in the rebuttal PDF. The first image comes from KITTI dataset, others are collected from the Internet. With the help of powerful large foundation models, DeepPriorAssembly demonstrates superior zero-shot scene reconstruction performance in these real-world outdoor scenes.

Dear reviewer,

Please read the author rebuttal and the other reviews and post a comment as to how your opinion has or has not changed and why.

Dear Reviewer osDy,

Many thanks for all the helpful comments and positive assessment. We will include the new experiment results and more reconstruction sampls in the revised version of the manuscript. Following your suggestions, we consider training a reconstruction model similar to ZeroShape for predicting complete shapes as a future work of DeepPriorAssembly. We will include the insights to Sec.5 of the revised paper and will conduct experiments to explore the effectiveness of this alternative.

We really appreciate you for upgrading the score.

Best regards,

Authors

Dear reviewers,

As the reviewer-author discussion period is about to end, we are looking forward to your feedback on our rebuttal. Please let us know if our responses address your concerns. We would be glad to make any further explanation and clarification.

Thanks,

The authors