Deceptive-NeRF: Enhancing NeRF Reconstruction using Pseudo-Observations from Diffusion Models

We would like to thank the reviewer for regarding our idea as a "promising concept" and providing constructive criticism that helped us identify areas that need improvement. Below is our response to your concerns.

Generalization Capacity (Question 1). Please refer to the discussion in the general response. We believe that our method has the potential to achieve good few-shot novel view synthesis results on any 3D scene when large-scale general 3D scene datasets are available for training. We will soon report the results of generalizing our model to other indoor datasets beyond Hypersim to demonstrate its generalization ability.

Quality of Pseudo-Observations. (Question 2) Thank you for your thoughtful suggestion to include a comparative analysis between our generated pseudo-observations and the ground-truth images. We have been working on it and will provide these visual comparisons soon to further strengthen our arguments.

Runtime analysis (Question 3). Please refer to the discussion in the general response and we will soon provide runtime analysis and comparison.

Stronger Baselines (Question 4). Thanks for pointing out our missing baselines and we'll report their results soon.

We would like to express our gratitude once again for your precise feedback in improving our work. Please review our revised manuscript. We have further addressed your concerns as follows:

Generalization Capacity. In Section E, we report the performance of our model, trained on the HyperSim dataset, on two additional datasets. This demonstrates that our method not only surpasses the baseline but also confirms that models trained on the HyperSim dataset are effective on other indoor datasets. As you rightly pointed out, we have emphasized in our limitations section the performance decline when models trained on HyperSim are generalized to arbitrary 3D scenes beyond indoor environments.

Runtime Analysis. We have included a discussion about the runtime of our method, with a time breakdown. Please refer to Section C or the general response provided above for more information.

Quality of Pseudo-Observations. We have included a comparative analysis between our generated pseudo-observations and the ground-truth images in Section D of our revised manuscript. Additionally, as suggested by Reviewer QhhZ, we provide a comparison with image restoration models, along with quantitative results. We hope they will aid in understanding the behavior of the deceptive diffusion model.

MonoSDF. Thank you for suggesting MonoSDF [1] as an additional baseline. We experimented with MonoSDF integrated into SDFStudio [2], but in our few-shot setting (5-20 input views), MonoSDF failed to provide reasonable reconstruction. This is likely due to the fact that MonoSDF, as a neural implicit surface reconstruction approach, prioritizes accurate reconstruction of the object surface and therefore requires dense observations to achieve reasonable results. The comparison in Table 1 covers most of the state-of-the-art methods in the field of few-shot novel view synthesis at the time of submission. While approaches for the task of neural implicit surface reconstruction [1,3,4] are highly relevant to our task, they have a different focus. Specifically, these methods prioritize the reconstruction of object surfaces, while our focus is on photo-realistic novel view rendering directly without recovering the surface. Additionally, neural implicit surface reconstruction often requires additional inputs such as normals, depths, or masks, in addition to RGB inputs, to ensure the accuracy of the surface reconstruction, while our method does not. We have revised the paper to include a discussion on MonoSDF in Section 4.1.

References
[1] Yu et al., “MonoSDF Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction”. (NeurIPS 2022)
[2] Yu et al., “SDFStudio: A Unified Framework for Surface Reconstruction”. (https://github.com/autonomousvision/sdfstudio)
[3] Wang et al., “NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction”. (NeurIPS 2021)
[4] Oechsle et al., “UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction”. (ICCV 2021)

Thank you for your constructive feedback and for acknowledging our work as 'reasonable, effective, and well-elaborated'. Below is our response to your concerns:

Filtering Strategy (Weakness 1). Based on our experience, our filtering strategy, together with our other designs, has to some extent alleviated the occurrence of view inconsistency. We appreciate the reviewer's suggestion to adopt the confidence score as in NeRF-W and will implement it as soon as possible and report the results.

Computational Cost (Weakness 2). Please refer to our discussion in the general response. Please kindly note that the 10-day fine-tuning time was achieved using personal computer-level computing resources (a single NVIDIA GeForce RTX 3090 Ti GPU) and can be greatly improved.

Other benchmarks (Weakness 2 and Question). Please refer to our discussion in the general response. We have included the experimental results on LLFF in the appendix, and our method outperforms most of the baselines. We will report our method's results on other indoor scene datasets to demonstrate its generalization ability.

We would like to express our gratitude once again for your constructive feedback on improving our paper. Please review our updated manuscript, which includes the following revisions based on your suggestions:

Confidence Score as in NeRF-W. We have updated Table 2 to report the performance of a filtering strategy using the confidence score used in NeRF-W [1]. This approach yielded results comparable to our original design. Specifically, following [1]’s pre-filtering step, we discarded low-quality pseudo-observations that scored below the 50% threshold in the NIMA [2] assessment.

Computational Cost. We have discussed the computational cost of our method. Please see Section C or the general response above.

Other Benchmarks. In response to your inquiries about “Can this finetuned CLDM be applied to any in-the-wild NeRF reconstruction dataset” and “Whether you can apply your method to other popular NeRF benchmarks”, we have evaluated our method on two additional datasets. As outlined in Section E, these evaluations demonstrate that our method not only surpasses the baseline but also confirms that models trained on the HyperSim dataset can also work effectively on other indoor datasets. In Section F, we report our method's performance on the popular NeRF benchmark LLFF. While our approach exceeds most baselines, it does exhibit a performance decline due to the domain gap from our indoor training data. Nevertheless, our method shows promise for broader generalization to diverse in-the-wild scenes, a potential that should be further realized as large-scale datasets of general 3D scenes become increasingly available.

References
[1] Martin-Brualla et al., “Neural Radiance Fields for Unconstrained Photo Collections”. (CVPR 2021)
[2] Milanfar et al., “NIMA: Neural Image Assessment”. (IEEE TIP, 2018)

We appreciate your thoughtful comments and the attention to detail you gave to our paper. Below is our response to your concerns.

Writing issue (Weakness 1). Thanks for pointing that out. We will improve writing to ensure logical flow and minimize grammatical and vocabulary errors.

Image Restoration Networks (Weakness 2). Comparing our model with the fine-tuning process with existing image restoration networks is a reasonable approach to demonstrate the effectiveness of our proposed method. We will conduct the corresponding experiments and report the results as soon as possible.

Computational Burden (Weakness 3). Please refer to the discussion in the general response. I will soon provide more experimental analysis on the runtime.

Frequency Mask (Question 1). When we refer to "we use a linearly increasing frequency mask," we are describing the utilization of a technique that resembles the frequency regularization in [1]. To be specific we initiate training without any positional encoding and progressively enhance the visibility of certain frequency components as the training process advances. This strategy is employed to prevent overfitting to high-frequency elements within the input data. We will clarify this in the modified version to assist readers in understanding.

Depth from NeRFs (Question 2). We compute the depth estimation of a pixel as $\hat{z}(\mathbf{r}) = \sum_{k=1}^{K} w_k t_k,$ where $t_k$ are the sampling locations, and the rendering weights $w_k = T_k (1 - \exp(-\sigma_k \delta_k)).$ $\sigma_k$ is the volume density value inferred from the NeRF MLP. $\delta_k = t_{k+1} - t_k$ is the distance between adjacent sampled points, and $T_k = \exp\left(-\sum_{i=1}^{k-1} \sigma_i \delta_i\right).$

View consistency (Question 3). In our pipeline, the diffusion model does not generate a new view image from scratch, which can easily lead to inconsistent views (as seen in many 3D diffusion generation efforts). Instead, it takes as input an image rendered from a coarse NeRF and refines it to remove artifacts. The coarse image processed by the diffusion model is consistent with other views in nature and the view consistency can still be maintained after diffusion’s refinement. Moreover, our progressive strategy involves gradually moving away from the input view to synthesize a new view. As a result, the artifacts encountered by the diffusion model at any given time are limited, reducing the difficulty of the refinement task it faces.

Shared text embedding (Question 4). As described in [2], we optimize for a new embedding vector within the embedding space of a frozen text-to-image model. This vector represents elements that appear in each input image but cannot be directly described using natural language, such as the stylistic information of a room. This embedding can be treated like any other word and is utilized to compose novel textual queries for our generative models.

References
[1] Yang et al., “FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization”.
[2] ​​Gal et al., “An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion”.