RobIR: Robust Inverse Rendering for High-Illumination Scenes

We are glad and appreciate that the reviewer recognizes the novel ideas of RobIR and the proposed method successfully estimates BRDF and illumination. Since many of the questions have already been answered in the common response, our additional response to the reviewer’s comments is below:

Q1: Why ACES is more suitable than other tone mapping like sRGB and sigmoid.

As previously mentioned in the common response, high-illumination scenes often require HDR tone mapping to produce the final image, mapping values from 0 to A into the range of $[0, +\infty]$. However, sRGB is not designed for processing HDR input. ACES allows PBR output colors to be in a larger value range, which better distinguishes very dark shadow regions and aids in the decoupling of object materials.

Additionally, ACES has significant advantages over sigmoid curves. Although both map an infinite range to $[0,1]$, ACES is a more commonly used HDR tone mapping curve in graphics, consistent with many rendering engines, and provides better contrast control while preserving more detail in highlights and shadows.

Q2: Lost details in albedo.

As previously mentioned in the common response, although the smooth loss does cause some loss of detail, the rendering precision of NeuS is a bigger factor in this loss. In the submitted rebuttal PDF, we included the results of NeuS rendering, which clearly show some blurring. This blurring affects other BRDF components supervised by NeuS, exacerbating the over-smoothing issue. Therefore, we are very eager to use methods with higher rendering and geometric quality, such as 2DGS, in future work.

Q1: Why ACES is more suitable than other tone mapping like sRGB and sigmoid.

As previously mentioned, high-illumination scenes often require HDR tone mapping to produce the final image, mapping values from $[0, +\infty]$ into the range of $[0, 1]$. However, sRGB is not designed for processing HDR input. ACES allows PBR output colors to be in a larger value range, which better distinguishes very dark shadow regions and aids in the decoupling of object materials.

Additionally, ACES has significant advantages over sigmoid curves. Although both map an infinite range to $[0,1]$, ACES is a more commonly used HDR tone mapping curve in graphics, consistent with many rendering engines, and provides better contrast control while preserving more detail in highlights and shadows.

Thank you very much for the constructive feedback. We hope that our response below will address your concerns.

Q1: Unknown tone mapping assumption.

Our proposed method only takes a collection of images with camera poses as input, and the ground truth tone mapping of these images is unknown. All the methods we compare in our paper use datasets with unknown ground truth tone mapping. We introduce scene-specific ACES tone mapping to approximate various tone mapping scenarios; for example, HDR scenes are typically optimized to vanilla ACES tone mapping. Therefore, if the tone mapping method were known, theoretically, we wouldn’t need to use our proposed scene-specific ACES. However, this is nearly impossible because different physics engines employ different tone mappings, and tone mapping itself involves numerous parameters. Additionally, variations camera parameters in real-world scenes make it even more difficult for us to determine the ground truth tone mapping.

Q2: The problem of the existing inverse rendering method.

The existing inverse rendering methods, such as NeRO, use sRGB tone mapping. However, in high-illumination scenes, we typically need tone mapping that can handle HDR inputs to produce the final image. We are the first to apply ACES tone mapping in inverse rendering, which fundamentally enables our method to handle inverse rendering in high-illumination scenes. Additionally, the introduction of scene-specific ACES tone mapping allows us to approximate other tone mappings (e.g., sRGB), making our method compatible with non-high-illumination scenes as well.

Building on more accurate tone mapping, we achieve more precise visibility modeling through Regularized Visibility Estimation (RVE) and NeuS Octree, which effectively removes shadows and disentangles other parts of the BRDF estimation more accurately.

Therefore, the failure (especially shadow baking) of existing inverse rendering methods in high-illumination scenes stems from 1) the mismatch of tone mapping between image processing and optimization, as well as 2) inaccurate visibility estimation.

We thank the reviewer for the positive review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Q1: How does the relighting work.

We imported the reconstructed albedo and roughness maps into Blender and performed relighting using Blender's scripting. Fig. 10 aims to demonstrate that we can extract high-quality PBR materials that can be directly applied in creative tools like Blender.

Q2: Why use an MLP instead of directly using the results from the NeuS Octree.

Good question. This is because querying Octree tracing is extremely time-consuming. By using an MLP, we can cache the results of the NeuS Octree and speed up inference.

Q3: Over-smooth details in Fig. 2.

This is a finding we want to highlight: smoothing helps correct geometric errors. Although the smooth loss results in some loss of detail, it effectively fixes geometric breakages caused by reflections or shadows in the normal map. Addressing these issues will aid in the decoupling of shadows and indirect illumination.

Q4: Environment map consists of the object color.

The estimation of environment map is still not robust in current IR frameworks. However, we want to emphasize that, as shown in Figure 5, our method has significantly improved the accuracy of environment map estimation compared to previous approaches. Additionally, we are one of the few methods that quantitatively compare environment map metrics (see Table 1). We will focus on addressing the estimation of environment maps in cases like those shown in Figure 11 in our future research.

We thank the reviewer for the positive review as well as the suggestions for improvement. We will revise the typo errors in the paper based on these insightful suggestions. Our response to the reviewer’s comments is below:

Q1: Comparison with inverse rendering methods with differentiable path tracing.

Great suggestion. We will add citations to these papers in our paper. We have shown a comparison with NvDiffRecMC in the rebuttal PDF. As can be seen, NvDiffRecMC still cannot remove baked shadows and indirect illumination from PBR materials, demonstrating the robustness of our method.

Q2: Show some video examples of moving shadows while rotating envmaps.

Of course, no problem. Unfortunately, we are unable to submit a video during the rebuttal period. However, we can provide other examples to demonstrate that our results are indeed strong. In Fig. 9, we offer several examples that illustrate the robustness of our method in shadow removal, which can partially demonstrate our capability to achieve the task you mentioned.

Q3: This method is limited to dielectric materials.

We are very grateful to the reviewer for pointing out this issue and noticing that we discussed it in the Limitations section. We want to emphasize that our approach is fundamentally a general method that can be integrated with other NeuS-based inverse rendering methods for glossy objects, such as NeRO. In the NeRO paper, there is also an issue with shadow baking in PBR materials for glossy objects. We believe that our approach can help NeRO achieve higher quality PBR material decomposition for metallic and glossy objects.

Q4: Novel view synthesis on NeRF synthetic scenes.

From our perspective, we believe that novel view synthesis (NVS) is not the primary focus of inverse rendering. Instead, we are more concerned with relighting and the decoupled BRDF components and PBR materials. Using the rendering equation to constrain the process can negatively impact NVS performance, resulting in rendering metrics that are less competitive than methods that directly output color. Additionally, NVS performance is highly sensitive to the base method used; for example, approaches based on NeuS are likely to perform worse in NVS compared to methods based on TensoRF or 3D-GS, which can introduce a degree of unfairness. Despite this, we still provide NVS results (on hotdog, lego, ficus, and mic) to demonstrate that the fidelity of our reconstruction is acceptable.

We want to emphasize once again that, although our NVS quality may not be the highest, the image supervision is sufficient to decouple high-quality and smooth PBR materials. These can be effectively used for tasks that are of greater interest in inverse rendering, such as relighting and shadow removal.

Q5: Why does the learnable parameter $\gamma$ have an exponent 0.2.

This is a very detailed question. It's an engineering trick to allow ACES to stretch across a wider range.

Q6: The use of $\eta (x)$ in Eq. 12 is inaccurate.

Thank you very much for pointing out this issue. Your understanding is completely correct. In Eq. 12, the supervision value for $Q(x, \tau)$ should be an N-dimensional vector, representing the visibility of N direct SGs at coordinate x. We will replace $\eta (x)$ with a different mathematical symbol in the paper.

Finally, we would like to thank the reviewer once again. Many of the reviewer's comments were very accurate, which truly made us very happy.

We are glad and appreciate that the reviewer recognize that the results of RobIR is thorough. Our response to the reviewer’s comments is below:

Q1: Visualization of the tone-mapping curve.

Great suggestion. We highly appreciate the suggestion, which can improve the readability of the article. We have already submitted visualizations with different tone mappings and different gamma settings in the submitted rebuttal PDF.

Q2: Missing evaluation of the tone-mapping curve.

This is also a very good suggestion. We would like to thank the reviewer for the in-depth understanding and for attempting to provide a method to evaluate real-world scenarios without GT tone mapping. Here, we provide tone mapping evaluations for two our rendered scenes: truck and chessboard.

• Truck: We rendered the scene in Blender and used Blender's built-in Filmic tone mapping to handle the high dynamic range. The final optimized $\gamma$ for the truck scene is 1.0, meaning the optimized tone mapping is the vanilla ACES. Considering that the Filmic curve is modified from ACES and their distributions are generally consistent with only some differences in detail, we believe that the optimization of tone mapping for this scene is accurate.
• Chessboard: We also rendered this scene in Blender, but used sRGB to process the BRDF output. The final optimized $\gamma$ value is 0.42. In the submitted PDF, we compared the differences between the ACES curve with $\gamma=0.42$ and the sRGB curve. It can be seen that $\gamma^{-0.2}$ stretches the ACES curve very well, making it closer to the distribution of the sRGB curve. Combined with more accurate visibility modeling (NeuS Octree and Regularized Visibility Estimation), we can remove the baked shadows and reflections from the PBR material in the chessboard scene (See Fig. 6 and Fig. 8).

Q3: Albedo quantitative comparison.

We greatly appreciate the reviewer's attention to the hue differences in albedo in inverse rendering, as directly using PSNR might lead to unfair comparisons. We carefully studied the "Ground Truth Dataset and Baseline Evaluations for Intrinsic Image Algorithms" for evaluating and decomposing albedo and found that their decomposition formula is $I(x) = S(x)R(x)+C(x), \text{where}\ S(x), R(x), C(x)\ \text{denotes illumination, albedo, specular respectively}$, which is not compatible with our method. Nonetheless, we are grateful to the reviewer for suggesting a more reasonable evaluation of albedo.

Regarding the evaluation of albedo, we believe that qualitative assessment is more important. Therefore, in Fig. 4, we provide extensive comparisons to demonstrate that our method does not bake shadows into the albedo. The removal of shadows cannot be solved merely by adjusting the hue (scale). Quantitative experiments only serve as a supplement to albedo evaluation. Although every method has issues with hue affecting PSNR to some extent, our method remains the closest and does not bake shadows. Furthermore, we also provide SSIM and LPIPS metrics, which are less sensitive to hue, and these metrics consistently show that our method achieves higher quality albedo, in agreement with Fig. 4.

Q4: Albedo estimation in real-world scenes looks over-smoothed.

In Fig. 7, the Man scene is actually successful in removing complex lighting and shadows, resulting in a smooth albedo, as the material of the sculpture is uniform. However, in the Bear scene, there is indeed an issue of over-smoothing. As mentioned in the common response, the smooth loss does have some impact, but it is more attributable to the rendering precision of NeuS. We are very much looking forward to considering structures that balance both rendering quality and geometric quality, such as 2DGS, in our future work.

Q5: About the radiance values and indirect illumination.

Your understanding is very accurate. The radiance values (before tone mapping) and indirect illumination are in HDR, which are then mapped to the $[0,1]$ using scene-specific ACES tone mapping. The activation functions for albedo and roughness are sigmoid, while SG-based direct light is modeled as nn.Parameter. The indirect light is supervised by NeuS (with NeuS supervision values $[0, 1]$ are mapped back to the same range in BRDF estimation stage using the inverse ACES mapping in Eq. 7).

Q6: The statement on the role of scene-specific ACES tone mapping.

In Figure 8, we conducted a rigorous ablation study on the effect of ACES. Introducing scene-specific tone mapping indeed plays a crucial role in shadow removal.

Finally, we would like to thank the reviewer once again. Many of these points were things we had not noticed before, and these suggestions will significantly improve the readability of the paper.