Fast Inverse Rendering by Unified Voxelization of Scene Representation

Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.

[Q1]: The explicit scene representation of UniVoxel is very similar to Voxurf [1], except that it additionally predicts various properties of object materials in space.
[A1.1]: Thanks for your valuable comment. We agree that some previous works like Voxurf have explored explicit scene representation, but this is not the main contribution of our method. We emphasize that our contributions are twofold. First, in contrast to previous methods that only use voxel grid to model geometry or appearance, we design a unified framework of scene representation, which allows for efficient learning of geometry, materials and illumination in a unified manner.
[A1.2]: Second and more importantly, as also mentioned by reviewer Ft2v, we propose to leverage Spherical Gaussians (SG) to model the local incident light fields, which is capable of modeling the joint effects of direct lighting, indirect lighting and light visibility without the need of expensive multi-bounce ray tracing. The SG parameters of surface points can be predicted directly through voxel grid and lightweight MLP networks, which enables seamless integration of illumination modeling into the unified voxelization framework, thus significantly improving training efficiency.
[A1.3]: We conduct experiments on Shiny Blender dataset [2] and the results are shown in Table 4 and Figure 9 of the appendix. We also highlight the comparison results with Voxurf in the following table, demonstrating that our method achieves better geometry quality than Voxurf.

[Q2]: UniVoxel cannot recover Envmap compared to other related works, which may limit its application scenarios.
[A2.1]: Thank you for your valuable comment. In theory, it is possible to filter the incident light map of all surface points to recover the environment map, but more sophisticated design is required to eliminate the influence of indirect lighting.
[A2.2]: Although our UniVoxel, by utilizing SG to model the local incident light field, cannot directly acquire the environment map, it achieves comparable accuracy to the state-of-the-art methods while having higher training efficiency (about 40 $\times$ faster than MII [3] as shown in Table 1 of the paper), which is the main advantage of our approach.
[A2.3]: Environment map and incident light field are two different technical approaches for representing illumination. The former can recover the environment lighting of the scene, but the obtained lighting is not sufficiently good for direct application. On the other hand, the latter has higher training efficiency (18 minutes vs 58 minutes as shown in Table 2 of the paper), although it cannot directly obtain the environment light. In the future, we will explore how to combine the advantages of these two methods.

[Q3]: There is no obvious advantage in the experimental results.
[A3.1]: Thank you for your meticulous comment. We acknowledge that our UniVoxel does not have significant advantages in terms of performance compared to other methods. However, as shown in Figure 3 of the paper, our UniVoxel is able to recover more high-frequency details on the albedo maps. Additionally, our UniVoxel achieves comparable reconstruction quality to the state-of-the-art methods, with significantly higher training efficiency, which is the main advantage of our method.
[A3.2]: Actually, we can control the shading component on the albedo maps by balancing the weight of the SG smoothness loss $L_{sg}$. In Figure 11 of the appendix, we have demonstrated that the light and dark shadows on the albedo maps can be eliminated by reducing the weight of $L_{sg}$. Due to the complexity of lighting condition in outdoor scenes, the constraints on illumination should be relaxed to achieve more realistic materials.

[Q4]: Can the explicit voxelization of scene representation model reflective surfaces such as metal or mirror?
[A4]: Yes, we have conducted experiments on the challenging Shiny Blender dataset [2]. The quantitative results in Table 4 of the appendix show that our UniVoxel achieves better geometry quality compared to other NeRF-like methods. The visualization in Figure 10 of the appendix also demonstrates that our UniVoxel is capable of reconstructing more realistic materials such as metal.

[Q5]: Can you provide a video comparing the training speed with other methods?
[A5]: Yes, we have included the video in the supplementary materials. It can be observed that when the training of our UniVoxel is almost completed, MII [3] is still in the process of recovering geometry. While the normal and albedo estimated by TensoIR [4] have rough outlines, their detailed parts still need several hours to optimize.

[Q6]: More related works.
[A6]: Thanks for pointing this out. We have added the reference of these works in the related work section.

Reference:
[1]. Wu et al., Voxurf: Voxel-based efficient and accurate neural surface reconstruction.
[2]. Verbin et al., Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields.
[3]. Zhang et al., Modeling Indirect Illumination for Inverse Rendering.
[4]. Jin et al., TensoIR: Tensorial Inverse Rendering.

Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.

[Q1]: The novelty of the proposed method.
[A1.1]: Thank you for your meticulous comment. We agree that volumetric representation has been studied in recent works, but this is not the main contribution of our method. We emphasize that our contributions are twofold. First, in contrast to previous methods that only use voxel grid to model geometry or appearance, we design a unified framework of scene representation, which allows for efficient learning of geometry, materials and illumination in a unified manner. Second and more importantly, as also mentioned by Reviewer Ft2v, we propose to leverage Spherical Gaussians (SG) to model the local incident light fields.
[A1.2]: Although some works like PhySG [1] has explored the application of SG in the context of lighting, they directly represent the entire scene’s environment map with SG, requiring expensive multi-bounce ray tracing. In contrast, we utilize SG to model the local incident light radiance, which is capable of modeling the joint effects of direct lighting, indirect lighting and light visibility. And the SG parameters of surface points can be predicted directly through voxel grid and lightweight MLP networks, which enables seamless integration of illumination modeling into the unified voxelization framework, thus significantly improving training efficiency.
[A1.3] We have reorganized the introduction section of the paper to clarify these major contributions.

[Q2]: While the paper acknowledges PhySG, it does not sufficiently recognize the prior exploration of SGs for modeling incident light.
[A2]: Thanks for your suggestion. We have added more descriptions about PhySG in section 2.1 and section 3.4 of the main paper, and explained the differences between our approach and it.

[Q3]: The unconventional modeling of the environment map using 128 Spherical Gaussians in this paper departs from traditional methods and calls into question the asserted superiority of SG over environment maps.
[A3.1]: Thanks for your valuable comment. Following your suggestion, we have adopted the conventional modeling of environment map, representing it as a learnable parameter of size 256x512x3. As shown in the following table, it obtains better reconstruction quality than using mixture of SG to represent environment map, and achieves comparable results to our method, but it also leads to much longer training time.
[A3.2]: Environment map and incident light field are two different techniques for representing illumination. Methods based on environment map can usually achieve better reconstruction quality, but they require more training time due to the need of multi-bounce ray tracing. On the other hand, methods based on incident light field usually can achieve higher training efficiency. In the future, we will explore how to combine the advantages of both methods and obtain high-quality reconstruction effects with shorter training time.

[Q4]: Explanation of the observed floating noise in TensoIR's [2] results.
[A4]: Thank you for your insightful suggestion. TensoIR has achieved impressive results on the MII synthetic dataset [3], but its performance in real-world scenes is poor and may have floating noise. The main reason is that the environment in each view of the real-world scenes is not completely static, making it difficult to handle complex lighting with a single environment map. One solution is to use a separate environment map for each view, but this would significantly increase training time. Our approach only requires introducing the view embedding as an additional input to the illumination model, allowing us to effectively handle complex real-world lighting without affecting training efficiency.

[Q5]: Experiments on the challenging shiny NeRF-synthetic dataset.
[A5.1]: Following your suggestion, we conducted experiments on the Shiny Blender dataset [4]. The quantitative and qualitative results of the estimated normal maps are shown in Table 4 and Figure 9 of the appendix. It can be observed that our approach achieves better geometry quality compared to our methods, especially in the specular surfaces.
[A5.2]: We also present the qualitative comparison of geometry, materials and illumination on the Shiny Blender dataset in Figure 10 of the appendix. Although our method does not recover the details of illumination, it can be able to predict realistic albedo and roughness, while TensoIR fails to reconstruct materials in the specular surfaces and bakes the lighting into the albedo maps.

Reference:
[1]. Zhang et al., PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting.
[2]. Jin et al., TensoIR: Tensorial Inverse Rendering.
[3]. Zhang et al., Modeling Indirect Illumination for Inverse Rendering.
[4]. Verbin et al., Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields.

Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.

[Q1]: The novelty of the utilization of feature grids for more efficient training.
[A1.1] Thank you for your valuable comment. We agree that some prior works have explored the use of feature grids to improve training efficiency, but this is not the main contribution of our method. We emphasize that our contributions are twofold. First, in contrast to previous methods that only use voxel grid to model geometry or appearance, we design a unified framework of scene representation, which allows for efficient learning of geometry, materials and illumination jointly.
[A1.2] Second and more importantly, as also mentioned by Reviewer Ft2v, we propose to leverage Spherical Gaussians (SG) to model the local incident light fields, which is capable of modeling the joint effects of direct lighting, indirect lighting and light visibility without the need of expensive multi-bounce ray tracing. The SG parameters of surface points can be predicted directly through voxel grid and lightweight MLP networks, which enables seamless integration of illumination modeling into the unified voxelization framework, thus significantly improving training efficiency.
[A1.3] We have reorganized the introduction section of the paper to clarify these major contributions.

[Q2] The memory concerns.
[A2] Thanks for your construstive suggestion. The volumetric representation of our method is not in conflict with the hashable grids proposed in InstantNGP [1], and it is easy to apply it to our framework. We plan to implement it in the future.

[Q3]: A notable limitation of the proposed method is that it often results in albedo estimates that include shading components.
[A3]: Thanks for your meticulous comment. Actually, we can control the shading component on the albedo maps by balancing the weight of the SG smoothness loss $L_{sg}$. In Figure 11 of the appendix, we have demonstrated that the shading components can be eliminated by reducing the weight of $L_{sg}$. Due to the complexity of lighting condition in outdoor scenes, the constraints on illumination should be relaxed to achieve more realistic materials.

[Q4]: Equal numbers of parameters when comparing SG and SH representations.
[A4]: Thanks for your valuable commet. Following your suggestion, we use a higher order SH representation of illumination (order-4 with 75 parameters). For a fair comparison, we also adopt the SG with 12 Gaussian lobes (72 parameters) for lighting modeling. The results on the MII synthetic dataset are presented in the following table. It can be observed that using higher order SH leads to some improvement in reconstruction performance, but using fewer parameters, SG performs better than SH, which demonstrates the effectiveness of SG in lighting modeling.

[Q5]: How are global illumination and self-occlusion handled during relighting?
[A5]: Thanks for your meticulous comment. We have added the relighting procedure in the Section B of the appendix. As the incident light field obtained from previous training is not applicable to the new illumination, we adopt a similar procedure as previous works like TensoIR [2] and NerFactor [3], where we compute light visibility using Eq.7 in the main paper and consider only direct lighting.

Reference:
[1]. MÜLLER et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.
[2]. Jin et al., TensoIR: Tensorial Inverse Rendering.
[3]. Zhang et al., NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination.

Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.

[Q1]: Emphasis on the illumination model.
[A1]: Thanks for insightful suggestion. Following the suggestion, we have rephrased the introduction section and emphasize more on the novelty regarding the proposed illumination model.

[Q2]: Comparison of geometry quality.
[A2.1]: Thanks for your suggestion. Due to the lack of ground truth normal in MII dataset, we conducted additional experiments on the Shiny Blender dataset [1], and the quantitative results are shown in Table 4 of the appendix. We also highlight the comparison results with TensoIR in the following table, showing that our method achieves more accurate geometry quality.
[A2.2]: In Figure 9 of the appendix, we demonstrate qualitative comparisons with other methods and different illumination models. It can be seen that our SG-based local illumination model produces higher geometry quality on specular surfaces compared to using environment maps. Additionally, using SH and NeILF as illumination models can achieve comparable geometry quality to ours. However, the quality of material recovered using our illumination model is superior to others, and the training efficiency is also higher, as demonstrated in Table 2 of the paper.

[Q3]: Reference of Neural-PBIR.
[A3]: Thanks for pointing us to the interesting work. We have added the reference of Neural-PBIR in the related work section.

[Q4]: How will you render the appearance? Are you going to sample rays uniformly, with importance sampling or just compute the integral analytically?
[A4]: We discuss how to render the appearance in the implementation details (Section B) of the appendix: we utilize Fibonacci sphere sampling method over the half sphere to sample incident lights for each surface point.

[Q5]: Using per-point SG parameters to compute the radiance at every point through rendering equation instead of computing per-ray SG parameters.
[A5]: Thank you for your constructive suggestion. One challenge of using per-point SG parameters to calculate radiance is that the rendering equation needs to be calculated for each sampling point on a ray during training. Compared to rendering with per-ray SG parameters, which only require one calculation of the rendering equation per ray, this approach requires several times more GPU memory and significantly prolongs training time. But I believe this is a very insightful idea, and we will explore ways to reduce the computational cost of this method to reconstruct fury objects.

[Q6]: How much more GPU memory does it require?
[A6]: We present the batch size and required GPU memory for training each method on the MII synthetic dataset in the following table. It can be seen that our method does not significantly exceed the GPU memory of other methods, thanks to our efficient implementation. However, for larger-scale scenes, we would need to increase the resolution of voxelization and also require more GPU memory. Nevertheless, this can be addressed by the hashable grids proposed in InstantNGP [2]. And we plan to implement it in the future.

[Q7]: Why are the color of the normal maps from different methods very different?
[A7]: Thank you for pointing this out. The way Nvdiffrec and Nvdiffrec-mc convert normal maps into RGB images is very different from other methods. We are working on it and will address this issue in the final version.

Reference:
[1]. Verbin et al., Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields.
[2]. MÜLLER et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.
[3]. Jin et al., TensoIR: Tensorial Inverse Rendering.
[4]. Zhang et al., Modeling Indirect Illumination for Inverse Rendering.

Thanks for the answers! I think my questions to the paper have been resolved.

Dear Reviewer Ft2v,

Thank you for taking the time and effort to review our response! If you have any further questions, please let us know and we will respond promptly!

Thank you once again for your time and attention.

Best,

Paper 2483 Authors