Inverse Rendering using Multi-Bounce Path Tracing and Reservoir Sampling

We thank the reviewer for appreciating our contributions. Below, we address your concerns and questions in detail.

Compare to using implicit field as geometry, the use of triangle mesh may harm the rendering quality. This is common as the mesh extraction can lead to an degradation in geometry quality.

We recognize the potential challenges associated with extracting meshes from implicit fields, which may compromise geometric quality. Therefore, we adopted a relatively high marching cubes resolution (256 or 512). Furthermore, the extracted mesh serves only as the initial geometry, which undergoes further optimization during the training process. This optimizing, driven by rendering loss, effectively resolves initial degradation and produces high-quality results. For example, in the Hotdog scene (Fig. 6), our method recovers more texture details compared to implicit methods such as TensoIR.

In addition, explicit geometry offers advantages in ray intersection calculations. Implicit representations, such as SDFs, typically rely on sphere tracing for ray intersection, which is both less accurate and less efficient. Consequently, our mesh-based representation does not compromise rendering or geometry quality.

The paper proposes to refine mesh with fixed face topology.

We acknowledge that our method does not optimize topology, and we will explicitly include this as a limitation in the revised paper. To the best of our knowledge, there are currently no effective methods for topology optimization during training. While approaches such as DMTet[*] can optimize topology, they often generate geometry with significant issues, such as holes and self-intersections. We provide a comparison of our method with DMTet and TensoIR (① and ② in https://iamnotaproject.github.io/reb/mesh.html). It is worth noting that TensoIR's normal results directly queried from the implicit surface representation, rather than derived from an extracted mesh.

A derivation of Equation 9 can improve the clarity.

We apologize for any lack of clarity in our explanation of Equation 9. Specifically, $p_{\text{dir}}(\omega_i)$ represents the distribution of the sampled ray (as defined after Eq. 8), while $\gamma_i$ denotes the sampling weight assigned to each sample during the reservoir sampling process (L312). For a more detailed description of the reservoir sampling process, please refer to the shared comment provieded earlier.

[*] Shen T, Gao J, Yin K, et al. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis[J]. Advances in Neural Information Processing Systems, 2021, 34: 6087-6101.

We thank the reviewer for appreciating our contributions and results. Below, we address your concerns and questions in details.

There is no code available to evaluate the method independently.

We will make our source code publicly available upon acceptance of the paper.

... how does the algorithm determine when the geometry is sufficiently accurate...

We use the default configuration of NeuS2, as it allows the model to converge effectively and produce satisfactory results. To investigate the impact of training steps in their configuration, we conducted experiments by reducing the steps to 10% and 50% of the default configuration. The results showed a gradual improvement in performance as the number of steps increased. However, since the default configuration already delivers sufficient accuracy and quality, we determined that additional adjustments to the steps are unnecessary.

... lack a comparison of the results of the specular reflection. Is this estimate much worse than for the other methods? ...

Our method is not restricted to diffuse materials and is capable of handling mildly rough specular reflections. However, we acknowledge as a limitation that it cannot handle extremely sharp mirror reflections (e.g. the ShinyBlender dataset or the NeRO dataset).

We already provide comparisons with baselines in scenes featuring specular reflections: (1) In the relighting result of Hotdog (Fig. 6), our method produces better specular highlights on the edge of the disk compared to baseline methods; (2) In the “Relighting 2” result of Gamepad (Fig. 14 in the Appendix), we specifically selected a view angle with strong specular reflections on the gamepad. Our method faithfully recovers these reflections, while baseline methods fail.

In this context, it would be interesting to know how the inaccurate initial estimate of the geometry affects the estimate of the specular reflections.

Here, we illustrate how an inaccurate initial geometry estimate impacts the estimation of specular reflections in this example (https://iamnotaproject.github.io/reb/specular.html)

... the texture is modeled implicitly using Instant NGP in the method. How is it transferred to the triangle mesh?

Although material properties are implicitly modeled by Instant-NGP during the training stage, texture maps can be extracted during inference using the well-defined tool: xatlas: https://github.com/jpcy/xatlas . Thank you for highlighting this point and we will include a detailed explanation in the Appendix.

Dear reviewer,

We would like to express our sincere gratitude for the time and effort you have dedicated to reviewing our work and providing valuable feedback. As the deadline for discussion is approaching on November 26, we kindly ask whether our response has sufficiently addressed your concerns.

Why run instant-NGP first?

We use Instant-NGP to initialize the radiance field, facilitating the subsequent radiance field rendering (Eq. 3). The choice of Instant-NGP over NeuS2 is due to its superior novel view synthesis performance, which enhances the accuracy of radience field rendering. Please refer to Fig. 9 in the Appendix, which demonstrates the effectiveness of our radiance field rendering design.

Why skip metallic?

We apologize for the misleading representation in Fig. 2, which does not include metallic. In fact, our method does support metallic optimization, and we optimize metallic for scenes in the OWL dataset (L494-495, Figs. 13 and 14 in the Appendix).

For the TensoIR synthetic dataset, we skip metallic optimization, consistent with the experimental settings of prior works, including TensoIR and GS-IR. This synthetic dataset is rendered using dielectric Disney BRDF materials without metallic properties, so optimizing metallic for this dataset is unnecessary.

For real-world datasets with more complex materials (e.g., OWL), we include metallic in our optimization process to achieve better quality results.

2. Parameter-space ReSTIR: We’ll describe the differences between our method and Parameter-space ReSTIR (Chang et al. 2023, abbreviated as “P-RIS” in the following) below:

   1. Different settings: P-RIS assumes known geometry (mesh) and focuses on optimizing texture map parameters on the mesh, while our method tackes a general inverse rendering task that jointly optimizes geometry, material, and lighting.
   2. Different motivation: P-RIS aims to simplify gradients in inverse rendering by transitioning from memory-intensive pixel space to parameter (texel) space and leverages reservoir sampling to efficiently estimate gradients. In contrast, our method uses reservoir sampling to mitigate Monte Carlo noise in path tracing, enabling more robust rendering results to facilitate training convergence.
   3. Different solution spaces of reservoir sampling: P-RIS performs reservoir sampling in texture space to estimate gradients, whereas our method applies reservoir sampling in primary space to estimate rendering results (first bounce of path tracing).

   Although both methods leverage reservoir sampling, they apply it to fundamentally different scenarios.

… the motivations described in the Eq. (9) emphasizes that there is no analytical integral of $L(\omega_i)f_r(\omega_i,\omega_o)$ so that we need reservoir sampling to approximate the distribution of $L(\omega_i)$. This makes much less sense as there is a clear solution to sample the environment map lighting and we can even consider MIS to sample both BRDF and lighting…

We believe there has been a misunderstanding. Our reservoir sampling is used to approximate the distribution of $L(\omega_i)f_r(\omega_i,\omega_o)$, not the distribution of $L(\omega_i)$. Therefore, our method is different from MIS sampling, which only combines the two samples obtained from BRDF sampling and light sampling using MIS weight but does not directly sample from the distribution of their product.

We acknowledge that the description of reservoir sampling in our manuscript could be clearer. Please refer to our common comment provided above for a revised description of the reservoir sampling process. We will also revise the explanation in Sec. 4.1. accordingly. Again, we apologize for the lack of clarity and the confusion it caused.

It will be beneficial to have a deeper discussion to why we need reservoir sampling to sample environment map and also more analysis on the impact of temporal reuse and spatial reuse.

To address this, we have included qualitative comparisons between our reservoir sampling and MIS, as well as an ablation study on the usage of spatiotemporal reuse (https://iamnotaproject.github.io/reb/MIS.html, ② and ③). These results demonstrate that our reservoir sampling strategy and spatiotemporal reuse significantly reduce noise levels, enhancing overall performance.

The total optimization time is 4.5 hours... This paper may consider using more advanced PBDR framework, such as Mitsuba 3...

In addition to the vertex-based material representation and precomputed indirect illumination, Neural-PBIR directly leverages a highly optimized off-the-shelf renderer(PSDR-jit) as their PBDR framework, and similarly, Mitsuba 3 leverages Dr.jit. We agree with your point of view that these JIT (Just-In-Time-Compiler) frameworks use heavy engineering optimization techniques to improve the performance. However, a significant restriction of using these JIT frameworks is that they only support (a very limited) pre-defined operations inside the loop. Our method uses neural networks (using PyTorch) and Instant-NGP, which are unfortunately not able to be traced by the JIT. In Mitsuba 3, if unsupported operations are included within the loop, the JIT optimization must be disabled to ensure proper updates to the PyTorch network. As a result, the performance will drop significantly, reverting to the behavior of a standard loop.

Despite this, we believe that the 4.5-hour optimization time is acceptable, especially since it is comparable to TensoIR (reported as 5 hours in their paper). The slightly longer runtime is a worthwhile trade-off for the significant improvements in result quality delivered by our method.

We thank the reviewer for your appreciation of our contribution and results, and we'll address your concerns and questions below.

...explicitly discuss how the proposed method differs from or improves upon Neural-PBIR and Parameter-space ReSTIR...

Our work differs from the two mentioned methods in the following ways:

1. Neural-PBIR: The Neural-PBIR framework consists of three components: (1) geometry extraction, (2) material and lighting distillation, and (3) further optimization of geometry and material using an off-the-shelf PBIR renderer [1]. The core of this inverse rendering process is initializing high-quality geometry and materials using the optimized NeuS network and learnable per-vertex material parameters. The final optimization using an existing PBIR renderer can be considered as an independent post-processing step. In fact, all inverse rendering methods that utilize explicit geometry and material texture representations, including ours, can integrate this PBIR renderer for refinement. Thus, our approach corresponds to the first two components of Neural-PBIR. While the geometry extraction methods in Neural-PBIR and our approach are similar (both train an SDF network and extract explicit geomeetry via the marching cubes algorithm), there are substantial differences in material estimation:

In terms of material representation, Neural-PBIR assigns per-vertex albedo and roughness values for explicit geometry and uses linear interpolation for triangle faces. In contrast, our method employs Instant-NGP MLPs to represent materials at any point in space, and additionally accounting for metallic. Although MLP-based representations have a slower optimization speed, they offer resolution independence, and are not constrained by the number of vertices or fixed mesh topology, and allow flexible optimization with richer material details.

For indirect illumination, Neural-PBIR adopts a strategy similar to TensoIR [2], which queries the neural radiance field for indirect ray radiance. This approach heavily depends on the radiance field quality, making it prone to error. In contrast, our method computes indirect illumination directly through multi-bounce path tracing, achieving more accurate results by avoiding reliance on potentially suboptimal radiance fields.

During the rendering process, Neural-PBIR employs stratified sampling of 256 fixed light directions per vertex to compute color. While this precomputed sampling accelerates rendering, it can introduce errors in visibility and indirect illumination, especially if the geometry is inaccurate. In contrast, our method leverages an advanced ReSTIR sampling strategy, which effectively controls color variance and reduces the required sample count.

Furthermore, Neural-PBIR's precomputed light directions remain fixed since the geometry does not update during the second stage. While this approach speeds up rendering, inaccuracies in geometry exacerbate errors in visibility and indirect illumination precomputations, which may affect rendering quality.

As Neural-PBIR has not released its code, we compared the albedo of the Hotdog scene as presented in their paper. We verified that their training data matches ours (① in https://iamnotaproject.github.io/reb/MIS.html). Neural-PBIR also demonstrates high-quality environment map results in their paper by distilling the environment map from background observations in RGB image (see Eq. 12, and Appendix Sec. B.2 and Fig. 11 in their paper). This approach can be integrated as a plug-and-play module. However, our method, along with other baselines (TensoIR, NVdiffrec-MC, GS-IR), assumes only foreground observations, without leveraging this setting.

[1] Zhang C, Miller B, Yan K, et al. Path-space differentiable rendering[J]. ACM Transactions on Graphics (TOG), 2020, 39(4): 143: 1-143: 19.

[2] Jin H, Liu I, Xu P, et al. Tensoir: Tensorial inverse rendering[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 165-174.

Dear reviewer,

We would like to express our sincere gratitude for the time and effort you have dedicated to reviewing our work and providing valuable feedback. As the deadline for discussion is approaching on November 26, we kindly ask whether our response has sufficiently addressed your concerns.

Thank authors for the response and it partially solves my questions! I think the following two experiments may improve the paper.

1. Experiments on Stanford-ORB dataset. This will answer questions about comparisons with Neural-PBIR and if the the proposed framework improves prior published works.

2. More experiments demonstrates the benefit of connecting neural representation and physically-based differentiable rendering. It will be great if that can lead to better geometry reconstruction or less color bleeding artifacts. This will justify the design choice of not using the most efficient optimization framework and also helps distinguish from prior works.

Also, I may still suggest that directly adopting ReSTIR in inverse rendering may not be considered as a major contribution and the major contribution should still be an inverse rendering system that achieves better reconstruction quality. The reason is because any sampling methods that reduce variance can be used in inverse rendering task to accelerate the optimization process. Unless it is specifically designed for inverse rendering, it does not bring too much novel insights. Also the evaluation needs to be more comprehensive as different sampling methods may work very differently for different lighting lighting and scene configurations.

Thanks authors for updating the text and including the extra description of RIS and ReSTIR! It helps clarify some questions of the proposed method. I will list my remaining questions below.

Technical novelty: RIS is a well-known sampling technique in rendering. Reservoir sampling as well as spatiotemporal reuse were proposed in Bitterli et al. to reduce rendering cost, especially for the case with many light sources. Adopting these sampling technologies may reduce the optimization time and noise, but the technical novelty may not be very strong unless the quality or speed are significantly better than prior works. I would suggest more experimental results on more public available benchmark to enhance the paper, such as Stanford-ORB dataset.

Difference from parameter-space ReSTIR: I agree that (1) is a major difference but then the following up question is we need to verify if the joint optimization can help us improve the reconstruction quality, especially geometry reconstruction quality. That will also further differ the proposed method with Neural-PBIR as in that paper differentiable rendering does not significantly improve the geometry reconstruction quality. (2) and (3) are less convincing to me because I feel P-RIS is solving a more fundamental issue of PBDR to reduce the variance of gradients while the proposed method is using reservoir sampling to reduce the noise of the rendering results, which was analyzed in prior ReSTIR paper already.

Comparisons with MIS: Thanks a lot for preparing the results and more details of the comparisons will be really appreciated. My first question is the MIS rendering seems to have a global color shift compared to the ground-truth, which is unexpected as MIS should be unbiased. Second, how many samples are used to create the MIS and reservoir sampling results? One benefit of reservoir sampling is that we can significantly increase the value of m and reduce the value of N but this can also be unfair since creating samples can cost time. In this case, maybe total rendering time is a fairer comparison?

Optimization time: Thanks for the analysis. If incorporating pytorch network will cause the optimization time to be significantly longer, one experiment to prove that it is worth the cost is to show that optimizing these network parameters will lead to higher reconstruction quality. Maybe it will improve the geometry quality. Maybe it will solve lighting baking issue. Such an experiment will significantly help the paper, in my opinion

Descriptions of Denoiser

We adopt the Edge-Avoiding À-Trous Wavelet Transform (EAWT)[1] as an efficient and stable denoiser. This method uses a wavelet decomposition, where the input $c_0(p)$ is iteratively smoothed using a $B_3$-spline kernel $h$ [2]. At each level $i$, the signal is decomposed into a residual $c_{i+1}$ as follows:

where $h_i$ expands its support by inserting $2^{i-1}$ zeros between coefficients at each step, ensuring computational efficiency.

To preserve edges during smoothing, a data-dependent weighting function $w(p, q)$ is introduced. This function incorporates information from the ray-traced input image (rt), normal buffer ($n$), and position buffer ($x$):

where

is based on color differences between pixels $p$ and $q$. The $\sigma$-parameters control the sensitivity to variations. These weights ensure that the filter adapts to the scene structure, preventing excessive edge blurring.

The computation of $c_{i+1}(p)$ involves a normalization factor $k$, defined as:

Using $k$, $c_{i+1}(p)$ is then computed as:

We perform three iterations of this process, balancing accuracy and computational efficiency.

[1] Dammertz, Holger, Sewtz, Daniel, Hanika, Johannes, and Lensch, Hendrik PA. "Edge-avoiding a-trous wavelet transform for fast global illumination filtering." Proceedings of the Conference on High Performance Graphics, 2010, pp. 67–75.

[2] MURTAGH F.: Multiscale transform methods in data analysis.

We sincerely thank the reviewer for appreciating our contributions and results. We agree that computer graphics and computer vision can complement and enhance each other. Below, we address your concerns and questions in details.

comparisons with DMTet

DMTet optimizes geometry starting from a randomized initial guess and allows for topology changes during the optimization process. However, it often generates geometry with issues such as holes and self-intersections. Our goal in using this optimization method is to achieve more stable geometry. We provide comparisons between our geometry optimization results and those of DMTet, along with a comparison of our mesh before and after optimization (① and ② at the annoymous GitHub page https://iamnotaproject.github.io/reb/mesh_denosier.html).

it cannot handle objects with transparent materials

We acknowledge that our method currently cannot handle transparent objects. Following your suggestion, we will include a discussion of this limitation in our revised paper and propose it a direction for future work.

Could you elaborate on the design of this denoiser?

The denoiser used in our pipeline is an off-the-shelf tool that we directly incorporate into method. We apologize for not providing a detailed explanation of its design in the original paper. Please refer to another comment for the details of our denoiser. We'll include a detailed elaboration on the denoiser as a new paragraph in the Appendix of our revised paper. To demonstrate its effectiveness, we also present rendering results with and without the denoiser in the annonymous GitHub page (https://iamnotaproject.github.io/reb/mesh_denosier.html, ③).

visualizations of the optimized environment maps

Existing inverse rendering methods generally utilize two types of representations for environment lighting: spherical Gaussians (SG) (e.g., in TensoIR) or trainable environment maps (as used in our method). Our decision to use trainable environment maps is based on the observation that SG representations often struggle to capture high-frequency details. Besides, TensoIR adopts a resolution of 16×32 for computational efficiency, whereas we use a higher resolution of 128×256, enabling us to represent high-frequency details more effectively. This is consistent with the results achieved by our method.

We acknowledge that recovering environment maps is a highly ill-posed problem, since input images primarily provide low-frequency directional intensity information (from highlights and shadows) but lack high-frequency details. Consequently, the optimization gradient is sparse and noisy. As shown in (https://iamnotaproject.github.io/reb/mesh_denosier.html, ④), our optimized environment map accurately captures directional lighting distribution but is inevitably noisy due to the inherent nature of Monte Carlo methods. Despite this, we note that environment map reconstruction is not the primary focus of our approach. The main objective of our inverse rendering method is to utilize the reconstructed geometry and material for downstream applications such as relighting and rendering, where the optimized environment map is ultimately discarded. Despite the noise in the environment map, our method achieves significantly higher-quality geometry and material reconstructions compared to baseline methods.

In Lines 401 -- 403, you mentioned some adjustments made to the albedo...

For albedo visualization, we follow the convention established by TensoIR (which, in turn, references NeRFactor). This approach is based on the assumption that recovering the absolute brightness of the albedo and illumination from an image is inherently ambiguous. For instance, multiplying the albedo by 2 while simultaneously halving the environment lighting produces the same rendering result. To address this ambiguity, we perform a least-square alignment between the brightness of our reconstructed albedo and the GT albedo. Specifically, each RGB channel of the albedo is scaled by a global scalar factor that minimizes the mean squared error w.r.t. the GT albedo. For real images, where GT albedo is not available, we apply a similar least-square brightness scaling step to the relighted images. It is important to note that this is a fair comparison, since we apply this adjustment across all baselines.

Dear authors,

Thanks for your reply! I tentatively have no further questions. By the way, if I understand the policy correctly, ICLR allows authors to upload a revised PDF during the discussion period. So maybe some visuals and detailed explanations could be updated to the PDF directly.

Dear Area Chairs and Program Chairs,

Thank you for overseeing the review process of our paper.

Our current title, "Inverse Rendering for Shape, Light, and Material Decomposition using Multi-Bounce Path Tracing and Reservoir Sampling," is quite lengthy. Since inverse rendering inherently refers to shape, light, and material decomposition, we propose a more concise title: "Inverse Rendering with Multi-Bounce Path Tracing and Reservoir Sampling."

We kindly seek your approval for this change. Thank you for your time and consideration.

Best regards,

The authors