UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections

Explicitly tracing the reflective ray

In this work, similar to many recent methods that are tailored to handle reflections [12,14,22,50], we follow Ref-NeRF and parameterize part of view-dependent appearance as a function of the reflected view direction $\mathbf{\omega}\_r$. Explicitly tracing the reflective ray is an interesting direction for us and it is shown effective in the concurrent work NeRF-Casting.

Ref-NeRF's model v.s. our model

Our custom baseline “RefV” uses the same physics based model as Ref-NeRF (L215), where we separate the color into diffuse and specular components. As shown in Table.3, our method outperforms RefV in both surface reconstruction and rendering on 3 datasets. Fig.5 shows that RefV reconstructs artifacts on the surfaces, while our method performs better. We also observe that when using the hash grid backbone, RefV may have optimization issues with separate diffuse and specular components as visualized in Fig.13. We discuss this in L533-543.

Q1: Unifying neural representations

What we mean is that we combine the two neural representations for the radiance field, i.e., camera view and reflected view radiance field. We will rephrase the term “scene representation” in the abstract.

Q2: Missing citations

Thank you for pointing this out. We will add them in our paper. MS-NeRF focuses on view synthesis and combines $K \geq 2$ camera view radiance fields, where each camera view radiance field has a volume density field. It is unclear how to extract the surface since there are $K$ different volume density fields and iso-surfaces can be extracted from each of them. In contrast, we use a single SDF field to represent geometry and directly extract iso-surface from it.

Q3: Ref-NeRF as a single radiance field

Thank you for pointing this out. We will rephrase.

Q4: Ablation study on loss weights

We perform an ablation study on the eikonal loss weight $\lambda\_1$ on Ref-NeRF real dataset and summarize the results as follows. Setting $\lambda\_1=10^{-4}$ produces the best performance. If $\lambda\_1$ is too large (e.g., $\lambda\_1=10^{-2}$), we observe that the training becomes unstable and may fail. This is similar to our findings on BakedSDF (Sec. C).

As shown in [46,49], the normals, estimated as the gradients, of the SDF field are well-defined and more accurate than those computed from the volume density field. Recent BakedSDF [50] and Ref-NeuS [14] combined SDF field with reflected view radiance field to reconstruct reflective surfaces. They found that using eikonal loss $\mathcal{L}\_{\text{eik}}$ for regularization is enough to get accurate normals and thus it is not necessary to use the normal smoothness loss $\mathcal{L}\_p$ or the orientational loss $\mathcal{L}\_o$ like Ref-NeRF. In our method, we still use $\mathcal{L}\_p$ and $\mathcal{L}\_o$ for additional regularization. However, since we use eikonal loss already, we simply set $\lambda\_2, \lambda\_3$ to relatively small values and do not tune them a lot. We only slightly increase the weight $\lambda\_2$ for normal smoothness loss $\mathcal{L}\_p$ on real data because of the potential noise (L481), similar to Ref-NeRF.

Q5: Input redundancy

It is correct that $\mathbf{b}$ is dependent on $\mathbf{x}$. The reason that we use both $\mathbf{b}$ and $\mathbf{x}$ is to align with the state-of-the-art implicit reconstruction pipelines such as VolSDF [49] and NeuS [46], which use both $\mathbf{b}$ and $\mathbf{x}$ to compute radiance.

Q6: Normals and orientation loss

Since the normal estimated from the volume density gradient are often extremely noisy, Ref-NeRF uses the predicted normal $n’$ throughout its pipeline, e.g., computing reflected view direction $\mathbf{\omega}\_r$, including for orientation loss. As shown in [46,49], the normals, estimated as the gradients, of the SDF field are well-defined and much smoother than those computed from the volume density field. Compared with the noisy normal of the volume density field of Ref-NeRF, we use a SDF field as our geometry representation and thus the normal is already smoother and more accurate. So we use $n$ in our pipeline (also for orientation loss) and only use the predicted normal $n’$ for regularization in Eq.10. Note that BakedSDF [50] and Ref-NeuS [14] also use the SDF normal $n$ for their reflected view radiance fields.

Q7: Volumetric rendering the SDF

Sorry, we missed the reference here. We use VolSDF [49] as our representation (L199).

Why learned weights separate specular/diffuse regions without explicit supervision

In Ref-NeRF, the ablation study shows that when rendering reflective surfaces, using the reflected view direction as the MLP’s input is explicitly better than using the camera view direction of NeRF. In our method, the main difference between two radiance fields is the view directional input. The reflected view radiance field uses reflected view direction as Ref-NeRF, while the camera view radiance field uses camera view direction. Similar to the findings in Ref-NeRF, our reflected view radiance field can model the appearance of reflective regions better than the camera view radiance field. Thus the weight $\mathbf{W}$, which corresponds to the rendered color $\mathbf{C}\_{ref}$ of reflected view radiance field (Eq.8), for these reflective regions increases during optimization to reduce the composed color loss $\mathcal{L}\_{\text{color}}$. Conversely, when the color computed from the camera view radiance field is better, the weight $\mathbf{W}$ decreases.

Q1: Stopgrad on any components of the normal smoothness loss

As in Ref-NeRF, we do not use stopgrad on any components.

Q2: Weight field to mix the outputs of the two radiance branches

Thank you for the suggestion. For this rebuttal, we evaluate our method while removing the weight field. The final color is composed as $\mathbf{C} = \mathbf{C}\_{ref} + \mathbf{C}\_{cam}$. We evaluate on the Ref-NeRF real dataset and summarize the results as follows. We observe that our method performs better than this ablation.

As shown in Fig.3, the learned weight field has the advantage of leading to better interpretability as it can be used to detect highly reflective regions from different viewpoints with volume rendering. This is a challenging task because reflection is an attribute of surfaces that objects or parts of objects have. Therefore, it is difficult to detect such regions using existing semantic models. For example, we tried state-of-the-art open vocabulary semantic segmentation methods with “reflective objects” or “reflective surfaces” as prompt, and they were unable to successfully segment the reflective objects in the image.

Q3: Results of ENVIDR

The metrics and rendered images of ENVIDR are the official results provided by the authors of ENVIDR. On the Ref-NeRF real dataset, ENVIDR is evaluated on “gardenspheres” only. In the limitation section of the ENVIDR paper, it is said that ENVIDR is unable to handle unbounded scenes, while the “sedan” scene of Ref-NeRF real dataset is unbounded.

Q4: Use $n’$ instead of $n$

As shown in [46,49], the normals, estimated as the gradients, of the SDF field are well-defined and much smoother than those computed from the volume density field of NeRF. Compared with the noisy normal of the volume density field of Ref-NeRF, we use a SDF field as our geometry representation and thus the normal is already smoother and more accurate. So we only use $n’$ for regularization in Eq.10. Note that BakedSDF [50] and Ref-NeuS [14] also use the SDF normal $n$ for their reflected view radiance fields.

Thank you very much for the response to my questions. I especially appreciate the response in the global rebuttal.

I'm not fully satisfied with the answer to the question about specular/diffuse separation--I feel like the authors are just reiterating what's happening, as opposed to providing any reasoning.

I also think since reviewer 5Tqb and I both had questions about this, it would be useful to run a quick (even on a smaller subset of scenes) the effect of using different normals in the reflected view direction and/or loss formulations for the camera ready version.

Nonetheless, I think the experimental results, alongside some of the arguments laid out in the global rebuttal, such as the comment of Ref-NeRF not being a complete physical model (not explicitly handling interreflections, etc), make a stronger case for the paper. I am happy to increase my score.

Thank you for the comments. We will add an ablation study about the normal that we use, i.e. SDF normal $n$ or predicted normal $n’$, for the reflected view direction and loss formulations in the paper.

W1: Time consuming

Our method has three fields and is based on volume rendering. Thus our rendering efficiency is not high. Recently, 3D Gaussian Splatting has become popular in view synthesis and there are some concurrent works focusing on reconstructing the surface [1*] or rendering reflections [2*]. Therefore, we think extending our pipeline with 3D Gaussian Splatting is a potential future direction to improve efficiency. Thank you for your suggestion to explore eliminating the multiple fields, this is an interesting direction for future works.

W2: Explanation for why some reflected view radiance field methods are not robust to real-world scene

Disambiguating the influence of geometry, color and reflection is an ill-posed problem in image-based 3D reconstruction. Methods such as Ref-NeRF assume that there are no inter-reflections or non-distant illumination, which is not often the case in real world data. Please see the global rebuttal for more details.

For the reflected view radiance field, the view-dependent appearance mainly depends on $\mathbf{\omega}_r$, the reflected view direction around the normal $\mathbf{n}$. However, the geometry and surface normal $\mathbf{n}$ are unknown in the beginning and need to be optimized during training. Therefore, the directional input $\mathbf{\omega}_r$ of view-dependent appearance keeps changing during training. We conjecture that this makes the optimization process more complex and ill-posed.

W3: Results of Factored-NeuS

The metrics of Factored-NeuS on DTU are the official results from their paper (the public ICLR 2024 submission). We just found that Factored-NeuS was open-sourced recently. Thus we evaluate Factored-NeuS on ShinyBlender dataset and summarize the results as follows. Our method performs better than Factored-NeuS in both rendering and surface accuracy. As shown in Fig.2 of the rebuttal PDF, Factored-NeuS reconstructs explicit artifacts on the “helmet” scene, while our method reconstructs the surface more accurately.

W4: Results with inaccurate camera pose

Thank you for the suggestion. In this rebuttal, we evaluate some scenes of the Ref-NeRF real dataset on camera poses with uniform noise. Specifically, we first add random translation noise (with a range as [-0.01,0.01]) along each axis for each camera pose. Note that before adding noise, the scene is already normalized to a unit cube following Mip-NeRF360. Second, we add random rotation noise to the camera poses. For each camera pose, we randomly sample a 3d unit vector and rotate the camera pose around it (with a range as [-1,1] degrees). As shown in Fig.3 of the rebuttal PDF, both the rendering and surface become worse. Integrating existing methods for pose optimization during training could be an interesting future direction.

[1*] Huang et al. 2D Gaussian Splatting for Geometrically Accurate Radiance Fields. SIGGRAPH, 2024

[2*] Jiang et al. GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces. CVPR 2024

Applying the idea in 3D Gaussian Splatting

Thank you for your interesting suggestion. Recently, 3DGS techniques are advancing rapidly and there are some concurrent works trying to reconstruct the surface [1*] or render reflections [2*]. GaussianShader [2*] introduces shading attributes for each 3D Gaussian, such as diffuse, tint, roughness and normal, to model reflections. Its rendering is similar to Ref-NeRF. Therefore, we think it is possible to apply our idea to 3DGS. We can combine the original 3DGS and GaussianShader representation with an optimizable “weight” attribute, similar to the weight field in our method, that is assigned to each 3D Gaussian point. We consider this as a potential future work.

Reduced editability

We discuss this limitation in the paper (Sec. G). Though our method does not support editability, with the high-quality mesh extracted from our method, it is possible to perform editing tasks such as relighting. For example, recent single image relighting methods [3*] use diffusion models for relighting and adopt “radiance cues” -- renderings of the object’s geometry with various roughness levels under the target environment illumination -- as conditioning. The mesh extracted from our method can be used to render such radiance cues.

[1*] Huang et al. 2D Gaussian Splatting for Geometrically Accurate Radiance Fields. SIGGRAPH, 2024

[2*] Jiang et al. GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces. CVPR 2024

[3*] Zeng et al. DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation. SIGGRAPH, 2024