Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects

We sincerely appreciate the thoughtful comments provided by the reviewers and we respond to each question in the following.

[W1] This paper appears to be a combination of numerous prior works.

Our first stage is fundamentally different from other work in both the implementation and the design concept behind it. The other two stages build on previous work, but they are modifications, to improve the results and therefore also contain novelty. Also, the way our system is constructed is novel. We do use and build upon components from previous work, but creating a complete and well working system and adapting existing components and combining them with our novel Stage 1 is still a significant challenge. We provide more details below.

[W1 / 1] The first stage borrows from the papers NeuS and PhySG.

In the first stage, we proposed a novel surface reconstruction method. The volume rendering module is better suited to radiance reconstruction and the surface rendering module ensures higher quality specular and diffuse separation close to the surface. Both working together enable accurate and efficient reconstruction of glossy object surfaces. Previous methods consider only using volume rendering (NeuS, VolSDF) or surface rendering IDR(PhySG, IndiSG) to reconstruct geometric information, and cannot handle the reconstruction of glossy objects, as shown in Tab. 1, Fig. 3, and Appx Fig. 7.

[W1 / 2] The second stage seems inspired by NeRFactor and IndiSG

We have indeed incorporated the concept of continuous light visibility from NeRFactor to enhance the binary light visibility of IndiSG, which predicts light visibility as either 0 or 1. We predict light visibility as 1-Σw. Existing ablation studies demonstrate that employing this approach yields better results in Tab 3 and Fig 10. Indirect lighting is currently a widely used method for lighting modeling, and we do not claim novelty in this aspect.

[W1 / 3] The final stage also bears similarities to IndiSG

In the third stages, we present a specular albedo estimation method based on spherical Gaussians. Although the component is simple, the exceptional effectiveness of our way of utilizing them is evident in Tab. 2, Fig. 4, and Appx Fig. 10.

[W2] The proposed Specular Albedo Index (SAI) lacks solid theoretical grounding. The authors don't clearly explain why the specular albedo is applied only to the indirect component (as seen in eq-13) and not to the direct component (eq-10).

Both direct and indirect lighting incorporate the specular albedo, as described in Eq. 11.

[W3] Furthermore, the experiments indicate that the introduction of SAI causes the method to underperform in its illumination estimation. It would be beneficial if the authors could elaborate on this performance decline and offer deeper analysis and insights

When our model complexity is more inclined towards focusing on materials, it restricts the complexity of the lighting model, leading to a decline in the quality of lighting reconstruction. We have added this analysis in the section E of appendix.

We sincerely appreciate the thoughtful comments provided by the reviewers and we respond to each question in the following.

[W1] The lack of consistency with vocabulary and definitions for diffuse radiance and specular radiance.

Thank you for your suggestion. Currently, the terminology is mainly aligned with the 'neural radiance field' used in NeRF for representing colors. Due to your suggestion, we realized that our definition in the main text should be improved. We change the description of diffuse radiance in the main text using “We use the term diffuse radiance to refer to the component of the reflected radiance that stems from a diffuse surface reflection”. We would like to break down the radiance measured by a pixel in the image into two components, the diffuse radiance and the specular radiance. The diffuse radiance is the component due to a diffuse surface reflection and the specular radiance includes all other non-diffuse components. Regarding the confusion of the usage of the term “surface diffuse radiance”: we simply meant diffuse radiance (as used in other parts of the paper) that stems from the surface rendering part of stage one. We have modified this terminology in the revised manuscript. “We propose to simultaneously optimize the radiance from the volumetric rendering and the surface rendering. For surface rendering, we further split the reflected radiance into a diffuse and a specular component.”

These terms are also used in physically-based rendering and atmospheric science. For example “Advanced Global Illumination”, chapter 2 “The physics of light transport”.

We also checked the reference “A bi-illuminant dichromatic reflection model for understanding images”, but we think the used terminology is not common in the NerF literature. To better align our work with the most relevant previous work, we therefore prefer not to change the terminology. There are many different communities working on reconstruction, lighting, and appearance models and we understand that terminology is not always consistent between communities.

[W2] The term "diffuse albedo" is also somewhat confusing, though albedo is commonly used (for greyscale?). Using something like body reflection color is less confusing.

Many literatures use diffuse albedo and albedo interchangeably. To distinguish our specular albedo, we opt for diffuse albedo. This terminology is also used by multiple other papers cited by our work. For example, see PhySG, NeRD, and IndiSG.

[W3] My only other complaint is the reference to a number of appendices, which aren't present in the review copy. Is it possible to squeeze just the critical points into the actual paper?

Thank you for your suggestion. Due to space constraints, it's challenging to incorporate all content into the main text. We have revised the document to place the appendix after the references, ensuring hyperlinks for images and tables to facilitate viewing and review. We also have uploaded a joint revised version to make the review easier.

[Q1] Are the authors using linear or sRGB data to train the networks? If linear, then is the original data linear, or are they taking the sRGB inverse of sRGB JPG data?

The training data consists of sRGB JPG or PNG images. However, we use linear data to train the MLP network, which means the output from the MLP, representing diffuse and specular colors, is in linear color space. Subsequently, we apply the linear-to-sRGB equation (Eq 6) to transform the results before utilizing them for training.

[Q2] Just to clarify, can the stage 3 models of direct illumination learn multiple direct illuminants? Are there any examples of multiple direct illuminants in the data sets evaluated?

In the third stage, we use environment maps to represent direct lighting, and environment maps have the capability to represent multiple direct light illuminants. We have conducted an experiment to show the capability to learn multiple direct illuminants in the appendix. We include images showcasing the lighting reconstruction for DTU scans 97 (tin) in Appx Fig.14, observing that the lighting consists of multiple direct light illuminants from the reconstructed environment maps.

[Q3] How is the illumination related to the visibility calculations in stage 2? Or is there an implicit assumption of a single direct illuminant?

For the visibility calculations, we demonstrate the method of combining light visibility with spherical Gaussians for illumination in Section E in the revised appendix.

[Q4] Following Bonneel et. al, 2017, a useful evaluation of intrinsic decomposition is to demonstrate that the decompositions are good enough to enable editing of illumination or reflectance separately. Is there a similar task whereby the authors could show a practical use of the decomposition and use it for comparison with other methods?

In Tab. 2, we present the comparison of lighting reconstruction using different methods, and in the supplemental material video, we showcase the results of relighting. In the revised appendix, we demonstrate the relighting comparison results for other synthetic scenes in Appx Fig 20.

We sincerely appreciate the thoughtful comments provided by the reviewers and we respond to each question in the following.

[W1] It would have been more convincing to see the quantitative results on the whole dataset. Including these results would be informative to confirm that the more complex image formation model does not lower performance on scenes with Lambertian objects.

In the appendix, we validated our method on the whole DTU dataset in Appx Tab 7 and observed improvements in the reconstruction Chamfer distance compared to NeuS.

[W2] The approach is quite complex and involves training several different neural networks and setting hyperparameters in different stages. How are they all determined? Which ones need to be tuned from scene to scene, or when dealing with different datasets? While Section 3.1 is well written and provides more details, I found Sections 3.2 and 3.3 to be quite short and difficult to follow.

In the original submission, the hyperparameters and details of our 2nd and 3rd stages are set in the appendix section B and D respectively in the supplementary material. We have revised and incorporated the appendix at the end of the references now and further added section E for stage 3.

[W3] The images in the paper are very small (Figures 3, 4 and 5) making it difficult to compare the results from various methods.

Thank you for your suggestion. We have placed the large figures on an anonymous website for viewing and comparison at https://authors-hub.github.io/Factored-NeuS.

[Q1] I would appreciate if the authors could clarify the technical novelty especially around the specular albedo estimation, and how this part of their approach differs from the work of Zhang et al 2022b (IndiSG).

In the first stage, we proposed a novel surface reconstruction method. The volume rendering module is better suited to radiance reconstruction and the surface rendering module ensures higher quality specular and diffuse separation close to the surface. Both working together enable accurate and efficient reconstruction of glossy object surfaces. Previous methods consider only using volume rendering (NeuS, VolSDF) or surface rendering IDR(PhySG, IndiSG) to reconstruct geometric information, and cannot handle the reconstruction of glossy objects, as shown in Tab. 1, Fig. 3, and Appx Fig. 7. In the second and third stages, we present continuous light visibility supervision based on SDF and a specular albedo estimation method based on spherical Gaussians. Although these components are simple, the great effectiveness of our way of utilizing them is evident in Tab. 2, Fig. 4, and Appx Fig. 10. Therefore, our work contains new components and also builds on existing ideas. In addition, originality also comes from how the ideas are combined and what problem it solves. In our case, we built a method to do inverse rendering for glossy objects in an unsupervised manner. It is something that previous methods fail on this task.

[Q2] Why is the approach evaluated only on masked objects?

We did not use masks when conducting experiments on real dataset DTU and SK3D. Methods like IndiSG use masks when running synthetic datasets. To maintain consistency, we also use masks when running synthetic datasets.

[Q3] I would appreciate it if the authors computed the metrics for the whole DTU dataset and showed the accuracy gap with NeuS and Geo-NeuS on all the scenes.

Regarding Geo-NeuS, the method utilizes additional point cloud supervision to improve surface reconstruction results, and we were unable to reproduce the original paper's average result of 0.508 using their code as the github issue, as mentioned in the GitHub issue. We have compared our method with NeuS on the entire DTU dataset in Appx Tab 7 and found that our approach exhibits overall improvement in performance as shown in the appendix.

We sincerely appreciate the thoughtful comments provided by the reviewers and we respond to each question in the following.

[W1] Lack of reference to NeRO [a], DIP [b]. The authors should compare the proposed method with them.

We appreciate the reviewer's feedback regarding the lack of reference to NeRO [a] and DIP [b]. We duly incorporate references to NeRO [a] and DIP [b] in our revised manuscript. For comparison, we compared our method with the already published work NeRO. Our method is compared with NeRO on the established real datasets DTU and SK3D for evaluating glossy objects. Our findings demonstrate that NeRO performs less effectively than our approach on real glossy datasets DTU in Appx Fig. 17. NeRO not only struggles to accurately restore detailed information but also fails to address the negative impact of partial highlights on the geometry. Moreover, the presence of shadows causes NeRO to mistakenly reconstruct shadowed areas as real objects and fill them in (bricks and skull models). In addition, our quantitative evaluation of Chamfer distance for the DTU dataset performs better as presented in Appx Tab. 7. When running the SK3D dataset, similar issues were encountered as in the DTU. Even in scenarios with a simple background and straightforward geometry, NeRO still tends to lose certain details and fills in shadowed areas (flower pot model) as shown in Appx Fig. 18. In addition, our quantitative evaluation of Chamfer distance for the SK3D dataset performs better as presented in Appx Tab. 8. Furthermore, we extended our comparison to include new glossy datasets, Glossy-Blender dataset and Glossy-Real dataset in Appx Fig 19 and Appx Tab 9, where although NeRO performs better, but our method is also capable of mitigating the impact of highlights on geometry. Our method demonstrated comparable results to NeRO. The experimental results have been presented in the Appendix. DIP is a concurrent work that will be published in 3DV 2024. We added a citation, but we think the paper only establishes that DIP looks good on synthetic data and does not claim that the method can handle glossy objects.

[W2] Lack of novelty. [W2 / 1] For the first stage, they propose to perform joint volume and surface rendering, which is similar to S^3-NeRF [c]

We respectfully disagree with this observation. To address the degradation issue in the model training caused by the reduced sampling range during the transition from volume rendering to surface rendering in UNISURF, S^3 NeRF defines volume rendering as the integration of surface rendering colors on the ray to increase the sampled region. This approach is fundamentally different in implementation and motivation from ours. When we applied the construction method of S^3 NeRF, which involves integrating surface rendering colors into volume rendering, to modify our model structure, we found that this modeling approach did not eliminate the impact of highlights on the surface reconstruction results. The experimental quantitative and qualitative results have been shown in Appx Fig 11 in the appendix. In addition, we exhibit differences in many other aspects, such as our use of Signed Distance Function (SDF) instead of an occupancy field to represent the implicit function and determine the position of surface points.

[W2 / 2] For the first stage, joint appearance and BRDF modeling which follows TensoIR [d]

At the first stage, the decomposition of diffuse and specular is not a true BRDF model. This is because the MLP in the first stage is used solely for predicting the components of diffuse and specular reflection, rather than predicting material properties such as albedo and roughness. The decision to directly predict colors instead of material properties in the first stage serves two purposes: reducing model complexity by focusing on the direct prediction of specular reflection color, and optimizing geometry for better reconstruction. By decomposing highlights through the network in the first stage, surfaces with specular reflections can be reconstructed more effectively, demonstrated by the presence of flower pot ablation in Fig 5, and without encountering the concavity issues observed in other methods. No prior work has utilized this method for geometry (SDF) optimization. After the completion of the first stage, where we believe the model's geometry has been optimized, the third stage focuses on predicting materials (BRDF model) based on the optimized geometry. During this process, the geometry model remains fixed and unchanged.

[W5] Symbols used are a bit messy.

We appreciate the reviewer's feedback on the symbols used in our manuscript. We respectfully disagree with the notion that our symbols are messy. We use the IndiSG paper as the starting point for our notation, but we also introduced several changes. We had been making multiple revisions to our symbol usage while drafting the manuscript, with a dedicated focus on improving clarity and coherence. We believe that the current set of symbols accurately represents the technical aspects of our work and facilitates a comprehensive understanding of the methodologies employed. At the same time, we are fully open to further enhancing our manuscript. If the reviewer could provide specific instances or aspects of our symbol usage that they find unclear or confusing, we would be more than willing to address these concerns in a targeted manner. Such specific feedback would be invaluable in helping us refine our manuscript to better meet the needs of our audience. We are committed to making our research as clear and understandable as possible and welcome any constructive feedback to achieve this goal.

[Q1] The datasets for evaluation should be more diverse. The lighting of SK3D dataset is not complex and the backgrounds are all black. The proposed method should also be evaluated on the Glossy-Blender dataset and Glossy-Real dataset proposed by NeRO [a] whose surfaces are more reflective.

We presented results on the Glossy-Blender dataset and Glossy-Real dataset in Appx Fig 19 and Appx Tab 9, demonstrating comparable outcomes to NeRO. It is worth clarifying that our method performs better than NeRO on the real-world DTU and SK3D datasets.

[Q2] What’s the performance on objects that are diffuse or not very glossy?

In the original submission, we presented material decomposition results for diffuse objects such as hotdog in Fig 4 and Tab 2. In the revised paper, we further showcase results for the entire DTU dataset in Appx Tab 7 and Fig 17, which includes many objects that are diffuse or not very glossy. Our method is also well-suited to handle this type of data.

[Q3] Better visualize the pipeline and models of all stages for easier understanding.

Thank you for your suggestion. In the revised version, we have illustrated all stages of the model in Fig.1 and Appx Fig.6.

[Q4] The proposed method chooses simplified modeling for diffuse and specular component in the first stage, where diffuse does not consider shading and specular only consider reflection direction and normal as input of MLP. Since the geometry will be fixed in the following two stages, how would the design affect the geometry reconstruction?

This is one of the main insights from our work. If the lighting and material model is too complex, the surface reconstruction will be unstable and overfit. If the lighting and material model is too simple, the surface reconstruction will interpret glossy reflections as bumps and grooves as shown in Fig.5 in the paper. We design a surface reconstruction method that is robust in real scenes which is a trade-off between being too complex and overly simplistic. We experimented with many different versions of simpler and more complex material and lighting models for surface reconstruction and this is the best one we found.

[W2 / 3] For the first stage, decompose color into explicit diffuse and specular components for glossy surfaces which follows Ref-NeRF[e].

A distinction between our method and Ref-NeRF is that we use surface point colors for both diffuse and specular components, whereas Ref-NeRF utilizes volume rendering colors for diffuse and specular components. In our ablation study, we demonstrate that using volume rendering colors does not completely eliminate the impact of highlights on the geometric reconstruction of the flower pot. Additionally, Ref-NeRF relies on a density field, while our approach is based on an SDF field, allowing us to obtain smoother 3D shape models. (When using an SDF field, the incorporation of additional feature vectors from the SDF network output into the color prediction model is necessary.)

[W2 / 4] The visibility and indirect light modeling in the second stage mostly follow NeRFactor [f] and IndiSG [g].

We have indeed incorporated the concept of continuous light visibility from NeRFactor to enhance the binary light visibility of IndiSG, which predicts light visibility as either 0 or 1. We predict light visibility as 1-Σw. Existing ablation studies in Tab 3 and Appx Fig 10 demonstrate that employing this approach yields better results. Indirect lighting is currently a widely used method for lighting modeling, and we do not claim novelty in this aspect. However, there are many possible ways to construct a complete system and it is not trivial to arrange all parts into a well working system, even if some of these parts have been used before and are mainly adapted.

[W2 / 5] The BRDF modeling in last stage follows PhySG [h].

In the third stage, our contribution is the introduction of specular albedo, a feature absent in PhySG. Through analysis, we discovered that incorporating specular albedo enhances the reconstruction of material properties and rendering colors.

[W3] Evaluations/analysis are not thorough. [W3 / 1] the diffuse/specular component of the first stage is not visualized/analyzed. There is also no ablation study on the design of the modeling of the two components.

We actually think our experiments are quite extensive, especially compared to other papers in the field. In the original submission, we presented an ablation study in Fig. 5 to demonstrate the impact of designing the two components on shape reconstruction for the first stage. Additionally, our goal in the first stage is the reconstruction of well-defined geometric shapes for glossy objects rather than generating diffuse/specular components. Nevertheless, we showcase the decomposed results of diffuse and specular components in Appx Fig 12 in the appendix, demonstrating a reasonable outcome of the decomposition process.

[W3 / 2] It would be better to visualize the geometry / albedo / specular / roughness / visibility / indirect light / environment map at the same time for all the datasets. Environment maps are only visualized in one ablation study in supplementary.

Thank you for your suggestion. We have added these components for all indiSG, DTU and SK3D datasets in Appx Fig 13, Fig 14 and Fig 15 in the appendix.

[W3 / 3] Visibility and indirect lights are only included in the videos of three objects without any analysis.

In the original submission, we have included the ablation study for visibility in Appx Fig 10 and analyse the necessity of including the visibility components. In the appendix, we additionally showcase ablation experiments on indirect light in Appx Fig 16. Through analysis, it is evident that the reconstruction quality of materials and rendering deteriorates in the absence of indirect light.

[W4] The progressive decomposition is not end-to-end, the geometry is fixed after the first stage.

In the first stage, the decomposition of diffuse and specular components contributes to the improvement in geometric quality, as evidenced by flower pot ablation. After the training in the first stage, if the model parameters are not fixed and the subsequent 2nd and 3rd stages are executed, the geometric quality experiences a decline due to the increased complexity of model training. Not being end-to-end is a disadvantage in some sense, but we obtain better quality with our current method which is an advantage.