How to Use Diffusion Priors under Sparse Views?

Thanks for the careful review. We appreciate for concerns and valuable suggestions and questions. Here are our corresponding responses.

• Analysis of the motivation. With detailed discussion about mode-seeking in previous works [A, B], here we provide a theoretical analysis of our motivation. The optimization objective of $\min _ {\theta} \mathbb{E} _ {t, \mathbf{v} _ j}[ \omega(t) D _ {KL}({q _ t^{\theta}(\mathbf{x} _ t^j)} \Vert{p _ t^*(\mathbf{x} _ t^j)})]$ derives ${q _ t^{\theta}(\mathbf{x} _ t^j)} \sim \mathcal{N}(\mathbf{x} _ t^j; \sqrt{\bar{\alpha} _ t}\mathbf{x} _ 0^j, (1 - \bar{\alpha} _ t)\mathbf{I})$ to the high-density region of $p _ t^*(\mathbf{x} _ t^j)$. Considering two mode $\mathbf{m} _ 1, \mathbf{m} _ 2 \in \mathcal{M}(\mathbf{x} _ t^j, y)$ of $p _ t^*(\mathbf{x} _ t^j)$, where $\mathbf{m} _ 1$ is the target mode and $\mathbf{m} _ 2$ is the failure mode. $\mathcal{M}$ is the mode range of $p _ t^*(\mathbf{x} _ t^j)$ decided by $\mathbf{x} _ t^j$ and $y$, which is the text prompt, and we do not elaborate on the conditions of the diffusion prior distribution for brevity. We denote the distance of two modes as $D _ M = \Vert \mathbf{m} _ 1 - \mathbf{m} _ 2 \Vert _ 2$. We want $\sqrt{\bar{\alpha} _ t}\mathbf{x} _ 0^j \approx \sqrt{\bar{\alpha} _ t}\mathbf{m} _ 1$ for any $t$. When $t$ is small, i.e. $t \rightarrow 0$, we have $\sqrt{\bar{\alpha} _ t} \rightarrow 1$, thus it is not hard for deriving $\sqrt{\bar{\alpha} _ t}\mathbf{x} _ 0^j$ to $\sqrt{\bar{\alpha} _ t}\mathbf{m} _ 1$. However, when $t$ is large, i.e. $t \rightarrow 1$, we have $\sqrt{\bar{\alpha} _ t} \rightarrow 0$, thus we have $\sqrt{\bar{\alpha} _ t}\mathbf{x} _ 0^j \rightarrow 0, \sqrt{\bar{\alpha} _ t}\mathbf{m} _ 1 \rightarrow 0, \sqrt{\bar{\alpha} _ t}\mathbf{m} _ 2 \rightarrow 0$. So, the optimization direction will still be affected, resulting in mode deviation. Back to the proposed IPSM, since IPSM introduces the rectified distribution $\tilde{q} _ t^{\theta, \phi}(\mathbf{x} _ t^j)$ as the intermediate state for narrowing the mode range $\mathcal{M}^{'}(\mathbf{x} _ t^j, y, \mathbf{M}^{i \rightarrow j} \odot \mathbf{I} _ 0^{i \rightarrow j}, \mathbf{M}^{i \rightarrow j})$, optimization directions are constrained and guided, thus suppressing mode deviation and promoting reconstruction quality.

• Comparison with iNVS [C]. TLDR of core difference: iNVS uses re-projection and Inpainting Stable Diffusion (ISD) for constructing a better way of conditioning for the diffusion model, while ours is for building an intermediate distribution to guide SDS optimization of 3D representations. Difference: iNVS aims to improve the conditional encoding of diffusion priors for image-to-3D. Unlike Zero-1-to-3 [D], which directly encodes the input image and the pose of novel views, iNVS uses the characteristics of ISD and re-projection techniques to encode the geometric prior of the input image on novel views into the conditional diffusion model, and fine-tunes ISD on large-scale external 3D dataset, thereby further improving object-consistency in generated images of novel views. This work aims to lift 3D information from inline prior for boosting SDS in sparse-view reconstruction. Unlike iNVS, this work starts from the analysis of the optimizing objective of SDS, and builds an intermediate distribution between the diffusion prior and the rendered images distribution by guiding the sampling trajectory of the pre-trained ISD with inline priors, thereby suppressing mode deviation and promoting improvements in reconstruction.

• SDS using view-conditioned diffusion prior.

  • An intuitive evaluation of Zero-1-to-3 on objects and scenes. As shown in Fig. 2 of the attachment, we provide the novel-view results of Zero-1-to-3 when given an image including an object and a scene. With the change of azimuth, Zero-1-to-3 can generate satisfactory results of objects in different new perspectives, but not for scenes, i.e. always fixed at the input perspective. This is because Zero-1-to-3 is fine-tuned on the large-scale 3D object dataset Objaverse [E]. During the optimization process of Zero-1-to-3, inherent inductive biases about objects are introduced into the model, making it hard to produce satisfactory results from new perspectives on scenes.
  • Quantitative experimental results of SDS using Zero-1-to-3. We product experiments of SDS using Zero-1-to-3 on LLFF with 3 input views shown below. Please note that Zero-1-to-3 controls the camera position by controlling polar, azimuth, and radius, but does not provide a way to control the camera posture. So we do not add noise to the camera posture for generating pseudo views when using Zero-1-to-3. We can notice that Zero-1-to-3 added as a prior cannot improve the quality of scene reconstruction. This is consistent with the intuitive visualization results we mentioned above. 3D diffusion priors lack a large amount of 3D data for learning the data distribution of the real 3D world, making it difficult to produce simple and direct effects on sparse-view reconstruction.

Table 1. Quantitative experimental results of SDS.

References:

[A] DreamFusion: Text-to-3D using 2D Diffusion. ICLR 2022.

[B] Stable score distillation for high-quality 3d generation. arXiv:2312.09305, 2023.

[C] iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis. SIGGRAPH Asia, 2023.

[D] Zero-1-to-3: Zero-shot one image to 3d object. CVPR 2023.

[E] Objaverse: A universe of annotated 3d objects. CVPR 2023.

Thanks for the efforts and patience of the careful reviewing. We appreciate the suggestions and questions for this paper. Here we provide detailed responses.

• Qualitative results for SDS. As shown in Fig.1 (a) of the attachment, the guidance of SDS will produce the imaginary reconstruction caused by mode deviation when using the diffusion prior directly as we demonstrated in the main manuscript. This property is reasonable and acceptable in text-to-3D generation tasks, but it fails in specific scene reconstructions limited by sparse views. As shown in Fig.1 (b) of the attachment, we can observe that SDS will also produce large floaters during optimization, which indicates the characteristic of training instability of SDS, since SDS overlooks the inline prior of sparse views and is hard to provide stable guidance towards target mode. We provide more qualitative comparison with SDS in the video of the supplementary material.
• Detailed experimental settings. Here we provide detailed experimental settings below. It is supposed to be noted that all baselines optimized per scene share the identical protocol of the experimental settings.
  • LLFF Dataset includes 8 scenes in total. For each scene of the LLFF dataset, following RegNeRF [A], we choose every 8th image as the test set and use 3 views uniformly sampled from the remaining images. The original resolution of the LLFF dataset is $4032 \times 3024$. Following DNGaussian [B], we downsample the resolutions of images to $8\times$ for both training and testing. In Tab. 1 of the attachment, we report the level of sparsity.
  • DTU Dataset includes 124 scenes in total. The prevailing pre-training method like PixelNeRF [C] utilizes 88 scenes for training and 15 test scenes, i.e. IDs: [8, 21, 30, 31, 34, 38, 40, 41, 45, 55, 63, 82, 103, 110, 114], for per-scene fine-tuning and testing. Following the prevailing works of optimizing per scene, i.e. RegNeRF [A], and DNGaussian [B], we directly optimize our model per scene on the 15 test scenes. For each scene of the DTU dataset, following RegNeRF [A], the IDs of 3 input training views are: [25, 22, 28], and the IDs of test views are: [1, 2, 9, 10, 11, 12, 14, 15, 23, 24, 26, 27, 29, 30, 31, 32, 33, 34, 35, 41, 42, 43, 45, 46, 47]. The original resolution of the DTU dataset is $1600 \times 1200$. Following RegNeRF [A], we downsample the resolutions of images to $4\times$ for both training and testing.
• Extreme circumstances. For the sparse-view scene reconstruction task, researchers focus on using inline priors and external priors to suppress the overfitting issue. Therefore both methods on the mentioned extreme circumstances have poor effects. We construct corresponding data and conduct experiments with the state-of-the-art method DNGaussian [B].
  • Two opposite input views. We select 2 opposite views of each scene on the MipNeRF-360 dataset, i.e. the IDs of training views of each scene: [2, 26] of bicycle; [22, 151] of bonsai; [57, 185] of counter; [1, 57] of garden; [14, 171] of kitchen; [2, 79] of room; [26, 34] of stump. The test views are selected every 8th image following Mip-NeRF. The quantitative comparisons with state-of-the-art methods DNGaussian and FSGS are shown in Tab. 2 of the attachment. We report the PSNR, SSIM, LPIPS, and AVGE for each scene and the average of all scenes. It can be seen that our method outperforms the state-of-the-art method DNGaussian on every scene and our model achieves improvements of $22.03$%, $19.21$% on average PSNR and AVGE scores respectively.
  • Extrapolation scenarios. We select 2 views on 0 and 90 degrees of each scene on the MipNeRF-360 dataset, i.e. IDs: [2, 14] of bicycle; [22, 248] of bonsai; [57, 145] of counter; [1, 15] of garden; [14, 37] of kitchen; [2, 291] of room; [26, 28] of stump. The test views are selected on the 180 degrees, i.e. IDs: [26] of bicycle; [151] of bonsai; [185] of counter; [57] of garden; [171] of kitchen; [79] of room; [34] of stump. The quantitative results similar to Tab.2 are shown in Tab. 3 of the attachment. It can be seen that our method outperforms the state-of-the-art method DNGaussian on every scene and our model achieves the improvements of $25.78$%, $21.34$% on average PSNR and AVGE scores respectively.
• 3DGS as the backbone.
  • Reason for choosing 3DGS. NeRF, as an implicit 3D representation, has the advantages of photographic-realistic rendering quality, but it takes the issues of slow training and rendering speeds. 3DGS, as a new explicit 3D representation, has the advantages of fast training and rendering speeds, which provides researchers with a convenient 3D representation framework, which is why we chose 3DGS as the backbone.
  • How about NeRF? In text-to-3D and image-to-3D tasks, score distillation methods are widely used on different backbones. For example, the representative methods using NeRF as the backbone include Dreamfusion [D]; the representative methods using 3DGS as the backbone include DreamGaussian [E]. Since this paper proposes a score distillation method, it can theoretically be migrated to NeRF, just like VSD [H] can be applied to both NeRF and 3DGS.

[A] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan, “Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs,” CVPR 2022.

[B] J. Li, J. Zhang, X. Bai, J. Zheng, X. Ning, J. Zhou, and L. Gu, “Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization,” CVPR 2024.

[C] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” CVPR 2021.

[D] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text-to-3d using 2d diffusion,” ICLR 2022.

[E] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” ICLR, 2023.

Thanks for your efforts and patience in reviewing this paper. We appreciate the positive comments, valuable concerns, and suggestions on our work. Here are our responses to the mentioned weaknesses and questions.

• Additional ablation study. To supplement more complete experimental results, we provide additional ablation study using 3 views on the LLFF and DTU dataset in Tab. 1 and 2 respectively. We can see that $\mathcal{L} _ {\rm{depth}}$ presents a strong prior for optimization since it directly provides the 3D geometric guidance on 3D representations. Notably, although both $\mathcal{L} _ {\rm{geo}}$ and $\mathcal{L} _ {\rm{IPSM}}$ use re-projection techniques to introduce the 2D visual prior information of the sparse views to promote optimization, $\mathcal{L} _ {\rm{IPSM}}$ achieves satisfactory performance comparable to direct 3D guidance of $\mathcal{L} _ {\rm{depth}}$ as shown in Tab. 2. At the same time, it is difficult for $\mathcal{L} _ {\rm{geo}}$ to promote optimization independently without the assistance of other regularizations.

Table 1: Additional Ablation Study on the LLFF dataset with 3-view setting

Table 2: Additional Ablation Study on the DTU dataset with 3-views setting.

• Additional experimental results with more input views. Thanks for the concern on the performance of our method with more input views. Experimental results using more input views can further explore the robustness of our method when working with sparse views. We provide additional experimental results under 6 and 9 input views on the LLFF dataset in Tab. 3 and 4 respectively. Denote that * represents results reported in ReconFusion and # represents results reported in DNGaussian. Notably, our method uses exactly the same parameters as the LLFF dataset with 3 views for training.
  • 6 input views. As shown in the first Table below, we achieve an improvement of $11.18$% on LPIPS compared to ReconFusion. It is supposed to be noted that ReconFusion requires additional computational resources for pre-training an encoder with external data as we demonstrated in the main manuscript. Excluding methods that require additional resources for pre-training, our method achieves improvements of $8.12$%, $8.34$%, $31.82$%, $30.68$% on PSNR, SSIM, LPIPS, and AVGE respectively, compared to DNGaussian, which is the state-of-the-art method based on the 3DGS.
  • 9 input views. Similar to the experimental results of 6 input views, our method still outperforms all state-of-the-art methods on SSIM, LPIPS, and AVGE scores and achieves comparable results on PSNR. As shown in Tab.2 of the attachment, compared to 3DGS-based DNGaussian, we achieve improvements of $7.94$%, $8.38$%, $26.11$%, $33.77$% on PSNR, SSIM, LPIPS, and AVGE respectively.

Table 3: Quantitative comparisons with 6 input views on the LLFF dataset

Table 4: Quantitative comparisons with 9 input views on the LLFF dataset

• Broken figure of the pipeline. We are sorry that Fig. 3 in the main manuscript may be broken on different PDF viewers. On Google Chrome, Fig. 3 can be viewed normally. We will replace it with an image file.