Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model

We thank the reviewer for appreciation of the visual quality and efficiency of our method. We make replies to the questions from the reviewer as follows:

• Janus problems: We include more than 150 results from a diverse set of prompts in the paper/website (https://instant-3d.github.io/) and additional 300 uncurated comparison results in the openreview Supplementary Material (see the instant3d_uncurated_comparison.pdf file). As we can see from the results, a vast majority of them are free from the Janus problem. We are happy to tone the claim down and submit a revised version before the rebuttal ends.

• Domain gap: While our first-stage sparse view generation model is fine-tuned on objaverse dataset, we would like to point out that we have conducted a light-weight fine-tuning for only 10K steps, which enables the model to mostly preserve the original capability of the SDXL model to handle a wide range of prompts including complex compositional concepts that are not available in the objaverse dataset, as demonstrated in our paper, website (https://instant-3d.github.io/, e.g., “a squirrel dressed like Henry VIII King of England”), and openreview Supplementary Materials (i.e., instant3d_uncurated_comparison.pdf). In the meanwhile, our second-stage sparse-view reconstruction is a deterministic model that takes multi-view images from the first-stage as input and lift them to 3D, which is trained on renderings from Objaverse dataset that cover a wide range of different shapes and allows the model to learn generic 3D priors for reconstruction. We observe that the model can work well on generated images and didn’t notice obvious failures in this stage. We update the anonymous website to include the generated four-view images and our reconstructed NeRFs https://instant-3d.github.io/rebuttal_image2nerf.html. From the results, we can see that our reconstructor can faithfully reconstruct the appearance of the objects that closely match the generated four views, which demonstrates the robustness and generalization of our reconstructors.

  That being said, we do observe that our model fails to handle some over-complicated prompts, for example, those related to complex spatial arrangements of multiple subjects and complex scenes. In addition, the generated assets also are not as photorealistic as the 2D images generated by SDXL, which may be attributed to the information loss in the fine-tuning stage due to domain gaps. One possible solution to this problem is to include single-view natural images such as those from LAION dataset in the fine-tuning, and we believe this will be an interesting future work. We will include more examples of failure cases and discussions in the revised version.

• Diversity: Our text-to-multiview generation network is fine-tuned from a 2D text-to-image diffusion model SDXL, and we apply regular diffusion sampling and denoising methods during inference. Therefore, it naturally inherits the capability of the 2D diffusion models in generating diverse images from random samples, which are then lifted to 3D to generate diverse 3D assets using our sparse-view reconstructor. In contrast, previous methods such as DreamFusion and ProlificDreamer apply Score Distillation-based optimization, which are prone to generate similar results even with different initializations, as discussed in Section A.5 of the DreamFusion paper [1]. We also updated our anonymous website to include some comparisons on diversity with DreamFusion and ProlificDreamer, please check it via https://instant-3d.github.io/rebuttal_diversity.html where we generate 4 different shapes for each prompt with different seeds. From the results we can observe that our method generates more diverse results than baseline methods. For example, for the prompt “a chimpanzee dressed like a football player”, our method generates diverse players while DreamFusion and ProlificDreamer failed to do so.

[1] Dreamfusion: Text-to-3D using 2D Diffusion, Poole et. al, https://arxiv.org/abs/2209.14988 (2022)

We thank the reviewer for the acknowledgement of the technical novelty and state-of-the-art performance of our methods. We make replies to the questions from the reviewer as follows:

1. Gaussian blob: Following the reviewer’s suggestions, we try replacing the Gaussian blobs with square bounding boxes at the center, and update the anonymous website with some example results (https://instant-3d.github.io/rebuttal_blobs.html). As we can see from the results, the square bounding boxes work reasonably well when they are relatively small, but suffer from obvious degradation in image quality and introduce undesired black background when the size increases.

2. Image features: We provide our thoughts from three perspectives and would like to share some empirical observations.

   (a) Pre-trained vs from-scratch. We have trained our model without initializing the visual encoder from a pre-trained weight (i.e., trained it from scratch). We found that a pre-trained model gives faster convergence in the beginning, but the gap keeps decreasing as the training budget increases. We have not seen the complete elimination of the gap under our limited compute budget but we guess that the gap can possibly disappear with more data and longer training time.

   (b) Different pre-trained methods. For the same ViT architecture, there are multiple pre-trained methods, e.g., MAE, DINO, CLIP, and ImageNet-CLS. Among them, we have tested with CLIP-init visual encoder but the results are worse than DINO. We guess that it is because CLIP is only trained for its [CLS] token with img-text contrastive loss thus it is hard to preserve all raw spatial information. For the other two (i.e., MAE and IN-CLS), we have not given a test. But we guess that MAE might perform similar to DINO, and IN-CLS would perform similar to CLIP based on how their pre-training method is built: MAE supervises spatial tokens while IN-CLS only supervises the [CLS] token. DINO supervises [CLS] token as well but its spatial token is good according to its pre-training strategy and also shown in previous papers [1].

   (c) Different model architectures (e.g., ViT vs CNN). We have not experimented with CNN-based visual encoder, but we do think that CNN should give similar results. The only consideration in our mind is that CNN usually needs a different training schedule than transformer thus makes the model tuning harder. Given this, we pick ViT in our exploration since easy tuning is crucial for large-scale training.

3. Comparison: We haven’t been able to compare to ATT3D because there is no public implementation. As for mesh synthesis methods, we compare to ProlificDreamer (please see Figure 4 in the paper and the uncurated comparison in the supplementary materials) that uses a three-stage optimization method (NeRF -> texture refinement -> mesh refinement) to generate meshes with textures, similar to Magic3D.

4. Novel view consistency: We would like to point out that our novel views are rendered from a NeRF in the format of a triplane (predicted by our transformer-based reconstructor). Therefore, it’s view-consistent by the nature of volume rendering. We provide turntable videos of our generated objects in the anonymous website https://instant-3d.github.io/ where we can see that the rendered views are consistent with each other.

5. Tiled generation vs Multi-Channel: One main advantage of our tile-based generation is it does not need to change the architecture of the original diffusion model. Multi-channel methods rely on specific attention modules in the original diffusion architecture, and need to add extra parameters such as view conditioning to help the model distinguish between different views. In contrast, tile-based generation can be applied to a wide range of network architectures without modifying the network architectures or any extra inputs. Therefore, our tile-based method has several benefits like:

   (a) The method is more transferable among different generative methods and model architectures without specific adaptation.

   (b) The light-weight fine-tuning can be easily applied to the new model.

   (c) Any acceleration techniques developed for the base model can be directly applied, thus our method can easily enjoy the fastest deployment developed by the community.

   (d) Similar to (c), any on-device/platform-specific code (e.g., cpp code; swift code) can be easily applied with our model as well, which enables more potential applications.

   In the meanwhile, we agree with the reviewer that multi-channel generation is an interesting future direction to explore.

(1/2)

We thank the reviewer for appreciation of our technical contributions and high-quality results. We make replies to the questions from the reviewer as follows:

• Color saturation: To demonstrate our method, we have included more than 150 results in the paper/website (https://instant-3d.github.io/) and additional 300 results in the supplementary materials (see the instant3d_uncurated_comparison.pdf file). As far as we can tell, we do not observe very prominent color saturation problems when compared to the baseline methods such as DreamFusion and ProlificDreamer that consistently exhibit over-saturated colors. In Figure 7 of the paper, we also show results of our model fine-tuned on non-curated data randomly sampled from Objaverse. While the quality is worse than our model fine-tuned on curated data, it does not have over-saturation problems as those in the SDS-based methods. Therefore, we believe the main reason is that our method uses normal feed-forward diffusion inference (DDIM) while optimization-based methods use score distillation losses.

• Resolution: In the limitation section, we mentioned that the degradation of rendering quality is mainly due to limited resolution of the NeRF tri-plane predicted by our transformer-based reconstructor instead of the four view generation. It’s not fully clear to us what the reviewer means by “measuring the PSNR/SSIM/LPIPS of single image and multi-view images”, and we would be happy to make further replies if the reviewer can make further clarification.

• Diversity: Thanks for the suggestion. We update the anonymous website to include some comparisons on diversity against optimization-based methods DreamFusion and ProlificDreamer in https://instant-3d.github.io/rebuttal_diversity.html. As we can see, our method can generate more diverse results compared to the other two methods. For example, for the prompt “a chimpanzee dressed like a football player”, our method generates diverse players while DreamFusion and ProlificDreamer failed to do so.

• Data filtering: We update the anonymous website to include renderings of some of the positive and negative examples during data curation in https://instant-3d.github.io/rebuttal_curation.html. We also had included the IDs of the shapes in the objaverse dataset that we use to fine-tune the diffusion model in the openreview Supplementary Material (see the curated_data directory), which can be used to inspecting the objects on https://objaverse.allenai.org/ and also ensures the reproducibility.

• Missing citations: Thanks for pointing out! We will update the citations in the revised version.

• Object-centric COCO: We update the anonymous website to include some examples on object-centric COCO prompts: https://instant-3d.github.io/rebuttal_object_centric_coco.html. On the other hand, all the prompts we use are from the DreamFusion gallery (https://dreamfusion3d.github.io/gallery.html). These prompts are similar to Object centric COCO (e.g., it contains “a kitten looking at a goldfish in a bowl” whose style is like the COCO captions), and have complex compositional concepts (such as “a hippo biting through a watermelon”, “a pug wearing a bee costume”, “a squirrel dressed like Henry VIII King of England”) that does not exist in the Objaverse dataset. We believe that our results on these prompts show that our model can generalize well beyond the Objaverse dataset. The main text of the paper only shows a small amount of the results, please see the anonymous website https://instant-3d.github.io/ and the uncurated results in the openreview Supplementary Material (See the instant3d_uncurated_comparison.pdf file) for more results and their prompts. Note that we only perform a light-weight fine-tuning for only 10K steps, and therefore our fine-tuned model can still mostly preserve the capability and knowledge of the original SDXL model, which enables our model to generate shapes that are not in Objaverse from complex prompts.

• View condition: We do not canonicalize the shapes in Objaverse except that we scale all the objects to a cube within [-1, 1]. The input to the transformer includes both the images and their camera poses in the world coordinate system. Therefore, if we keep the images unchanged and only transform the camera poses, the output object will be transformed correspondingly.

• Minor mistakes: Thanks for pointing them out! We will correct them in our revised version.