ZeroNVS: Zero-shot 360-degree View Synthesis from a Single Real Image

Thanks for your comments. We will address the weaknesses and questions below.

• Qualitative results. We agree that more qualitative comparisons are needed, and we include them in new Figures (Figures 8 and 9) and also in the revised supplementary website. Object-centric methods which can mask out the background achieve good results in object centric scenes if the background is not considered. However, the focus of ZeroNVS is to also handle complex scenes with background, such as the many RealEstate10K, Mip-NeRF-360 and DTU scenes in the supplementary material.
• SDS anchoring. We added a new figure (Figure 9) in the main paper which illustrates results with and without SDS anchoring. We also include a gallery of video results with and without SDS anchoring in the supplementary website.
• On technical contribution. We hope that the additional support for SDS anchoring which we have discussed above is helpful. Regarding the handling of the scale, Table 2 shows that Zero-1-to-3 cannot appropriately model scenes without our proposed camera conditioning representations when trained on the same data. Table 5 demonstrates the value of each of our successive improvements to the camera conditioning representation.
• L2 norm question. Only to the translation part.
• Degeneration during SDS anchoring. In practice we observe that this does not happen. It is possible that attempting to do this sort of procedure autoregressively would eventually lead to degeneration, but the novel view DDIM samples are sampled once from the input image at the start of training and are held fixed during training, so there is no possibility for them to degenerate. For examples of how SDS anchoring can improve background quality, please consult Figure 9 and also the revised supplementary website.

Thanks for your comments. We address the questions below.

• Metrics and qualitative comparisons. We agree that more qualitative comparisons are needed, and we have included far more extensive qualitative comparisons in the revised version. They are in the new Figures (Figures 8 and 9) and also in the revised supplementary website.
• Runtime of FID. We regret the miscommunication – we did not mean that the runtime of FID was too high. Rather, FID requires thousands of samples, and the runtime of SDS distillation-based methods such as DreamFusion and our method for generating one sample is on the order of hours. Thus, FID is impractical in this case, and we rely on view synthesis metrics on small datasets like DTU and Mip-NeRF 360 and user studies for our analysis.
• Fair comparison to baselines. We first wish to clarify an apparent misunderstanding. For Table 2, we do not compare against the released weights for Zero-1-to-3 and PixelNeRF, but rather compare against retrained versions which are trained with identical data to ZeroNVS for a fair comparison, as we indicated in the table caption and text in the experiment section. Thus, for Table 2 the only difference between Zero-1-to-3 and ZeroNVS is the novelties we proposed for conditioning in our approach section, as the training data is identical. These novelties are what lead to the significant performance gains.
• Comparison to SOTA. Please note that in our Table 1 and Table 2, we compare to a diverse set of recent methods, including DS-NeRF, PixelNeRF, SinNeRF, DietNeRF, NeRDi, Zero-1-to-3, and PixelNeRF. Since Zero-1-to-3 and NeRDi are SOTA approaches for view synthesis, we consider the combination of Table 1 and Table 2 a fair evaluation against existing SOTA.
• Can the model be finetuned on DTU? The model can be finetuned on DTU and this would likely further improve performance. We chose not to do this as our performance was already SOTA for LPIPS and we preferred to emphasize the zero-shot nature of the model. We will finetune ZeroNVS and DTU and report the performance in a revised version. Due to the limited time, this may not be ready before the end of the rebuttal period.
• ACID. Although ACID contains many examples of forward trajectories, this is still a useful signal for SDS distillation. Since ZeroNVS is trained on lots of examples of small perturbations of a forward-facing camera, this helps the scene geometry to quickly converge early in SDS distillation even when the sampled cameras are still mostly forward-facing and near the input camera, due to the diffusion model’s prior for parallax. Further support for this argument can be found in Table 4, where training the model without ACID data results in worse performance on DTU.
• Ablations. Our diffusion models require an 8xA100 machine to train, which is the same as Zero-1-to-3. Training one diffusion model for 60,000 steps takes approximately a week, and since our ablations necessitated training 8 unique models (four in Table 4, and four additional unique models in Table 5), we trained these models for fewer steps. We provide more training details in Appendix A of the supplementary material.
• Typo. We have fixed this and various typos in the revised version.

Thanks for your comments. We address the questions below.

• Clarification on Contributions. We have substantially revised our introduction and approach to more clearly delineate the contributions, namely
  • A model, ZeroNVS, which solves the challenging problem of zero-shot single-image 360-degree view synthesis, described in the Approach section and with extensive results provided;
  • New camera conditioning representations, which enable training on large-scale mixed datasets of real scenes, as supported in Table 5;
  • SDS anchoring to address the limitations of SDS as applied to scenes, as supported in Table 3 and Figure 9;
  • Extensive quantitative analysis and SOTA performance, as supported in Tables 1 and 2.
• Comparison with GenVS. We have now discussed GenVS as concurrent work in the related work section. Please note that GenVS has not released the code yet. Please also note that GenVS was presented at ICCV (Oct. 2-6), whereas the ICLR deadline was at the end of September.
• Differences from Zero123: The key differences between our method and Zero-1-to-3 are the camera conditioning representations we develop in the approach section, and our proposed system for SDS anchoring. We show in Table 2 that Zero-1-to-3, even when trained on the same exact dataset as ZeroNVS, achieves far worse performance. This is due to the more expressive camera conditioning representations which enable training on real-world scene data, a claim we support in Table 5.
• Result of ZeroNVS on GSO dataset. GSO does not consist of real-world scenes, but rather is a synthetic dataset of object-centric scenes on white backgrounds, thus out of distribution of ZeroNVS which targets real images with potentially complex backgrounds. Instead, we compare against Zero-1-to-3 by training both Zero-1-to-3 and ZeroNVS on the same data, and measuring the performance on DTU and Mip-NeRF 360 datasets in Table 2. ZeroNVS significantly outperforms Zero-1-to-3.

Thanks for your comments. We would like to clarify that multi-plane models are not a part of our architecture. ZeroNVS predicts NeRFs from a single image. The confusion may have arisen from our use of the same camera normalization scheme as that in the baseline Stereo Magnification (Zhou et al, 2018).

We address the questions below:

• Technical contributions. The problem (scene-level 360-degree novel view synthesis from a single image) is a significant problem in computer vision, and demonstrating that it is possible with novel adaptations to existing techniques is a valuable contribution.

• On DDIM sampling. We have added a citation and description in the draft. DDIM (Ho et al, 2020) is the sampling procedure for the 2D diffusion model. Different from SDS, which is a technique that allows one to use the diffusion model as a critic of novel views rendered by the NeRF, DDIM sampling generates 2D samples from the distribution of the diffusion model.

• Typo. We have attached a revised manuscript in which various typos have been fixed.

• PSNR and SSIM. Although we argue in Figure 7 and in the text that PSNR and SSIM should be weighted less compared with LPIPS, we still include PSNR and SSIM as this is standard in the view synthesis literature. As part of the rebuttal, we also include significantly more qualitative comparisons with baselines in our revised supplementary material, and added qualitative comparisons in Figures 8 and 9 of the main text. We hope that they can be consulted for a more comprehensive picture of the performance.