MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference

Dear reviewer, thanks for your valuable comments and recognizing our well-founded motivation and well-organized structures. We are glad to provide further clarifications and experiments to address your concerns.

1. About albedo term.

The primary goal of our work is modeling reflection effects in complex scenes for novel view synthesis. Therefore, following Ref-Gaussian, we directly rasterize the diffuse color $c_{diffuse}$ using per-Gaussian spherical harmonics and use other material properties such as albedo, metallic, and roughness to evaluate the specular color $c_{specular}$.
We observe that incorporating albedo in modeling the diffuse component, as done in traditional graphics, will significantly increase the optimization difficulty, especially in complex real-world scenes. On the other hand, since the diffuse component is not sensitive to viewing direction, delegating the prediction of diffuse color to the Gaussians allows us to better disentangle reflection effects from illumination. The full rendering equations we used can be written as follows,

$$\begin{aligned} c_{\text{diffuse}} &= (1 - m)c_d, \\\\ c_{\text{specular}} &= \left(((1 - m) \cdot 0.04 + m \cdot a) \cdot F_1 + F_2 \right) \cdot \text{env}(\omega_i, r), \\\\ F_1,\ F_2 &= \text{lookuptable}(\omega_i, r), \\\\ c_{\text{final}} &= c_{\text{diffuse}} + c_{\text{specular}} \end{aligned}$$

where $\text{env}(\omega_i, r)$ denotes querying from the mip-mapped environment cube maps using the incident direction and roughness. For indirect illumination, this term is replaced by a combination of the residual color and the color obtained via 2D Gaussian ray tracing. $F_1, F_2$ are obtained through table lookup based on the incident direction and roughness. From the above equation, we can see that the albedo component contributes little to the specular color when the metallic value is small. That is why we claim that the contribution of albedo to non-reflective surfaces is minimal. Through this way, we do not need to devote much attention to learning albedo in non-reflective regions, which is why we argue that our modeling strategy significantly reduces the optimization difficulty of material decomposition.

To further demonstrate this, we compare our method with inverse rendering methods which use albedo to compute both diffuse and specular components on the real-world Mip-NeRF 360 dataset. As reported in the table below, our superior results highlight the advantages of our decomposition approach in complex real-world scenes.

2. The choice of $M_0$ and $\Gamma$.

In our implementation, $\Gamma$ is set to 1 or 0 according to whether the average metallic value in a region is larger than 0.5 or not, practically. And $M_0$ is set to be the top 30% of metallic values in a region, practically. We conduct additional ablation studies on these two thresholds, as reported below.

For $M_0$, too small threshold makes the reflection strength prior ineffective, while too large threshold results in surfaces with overly strong reflection strength, both of which harm the modeling of accurate reflection effects. For $\Gamma$, too small value causes most regions to be wrongly identified as highly reflective surfaces, while too large value fails to identify true surfaces with high reflectiveness, both of which hinders the learning of material decomposition. Overall, however, our method remains robust to the choice of these thresholds.

3. Visualization of occlusion and indirect illuminance.

Due to the policy restriction, we are unable to show visualization of the occlusion maps or indirect illumination. To quantitatively evaluate the results, we compute ground truth binary indirect illumination masks and occlusion masks for each view on the ground truth meshes. We then evaluate the rendering quality in indirect regions and the accuracy of binarized occlusion maps on ShinyBlender and GlossySynthetic datasets. We compare our method with both reflection modeling method Ref-Gaussian and inverse rendering methods including TensoIR, Relightable3DGS, GS-IR. For fair comparison, we add monocular normal priors for all baseline methods.

The results reported in the table below significantly show our advantages in approximating indirect illumination and localizing occlusions. The reason is that our multi-view material inference and environment modeling are end-to-end differentiable with respect to geometry, which allows for joint optimization of geometry, appearance, and material decomposition. In contrast, baseline methods typically first optimize geometry and then use the fixed geometry to predict occlusion and perform inverse rendering, which makes them less flexible in handling complex regions.Additionally, the material modeling strategy adopted in inverse rendering methods is not well-suited for highly reflective scenes, as we analyzed in our response to the first question "About the albedo term". We will add more visualization results in terms of indirect illumination and occlusion maps in the revised version.

4. About numerical results in Tab. 2.

Sorry for the mistake. We wrongly reported the third row (w/o $\mathcal{L}_n$) of ShinyBlender dataset in Tab. 2. It should be 35.37 / 0.975 / 0.050. We will correct this in the revised version.

5. About normal prior.

The normal prior is helpful for better convergence of geometry in the early training stage, which indeed benefits subsequent multi-view material inference and environment modeling. However, our method does not heavily rely on the normal prior. As shown in the third row of Tab. 2 of the main paper, even without the normal prior, our method still achieves state-of-the-art results on the ShinyBlender dataset. In fact, even without perfect geometry, our multi-view material inference and environment modeling continue to improve the geometry in later training stages. This is because our material constraints and environment modeling are differentiable to the depth and normal, enabling end-to-end joint optimization of geometry, appearance, and material properties for improved overall performance.

We further change different monocular normal estimation models and report our rendering results with different normal priors, as reported in the table below. We achieve similar outstanding results based on various normal priors, which demonstrates our robustness to imperfect normal priors.

[1]. Xiao Fu, Wei Yin, et al. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In European Conference on Computer Vision, pages 241–258. Springer, 2025.

[2]. Chongjie Ye, Lingteng Qiu, et al. StableNormal: Reducing diffusion variance for stable andsharp normal. ACM Transactions on Graphics (TOG), 43(6):1–18, 2024.

[3]. Mu Hu, Wei Yin, Chi Zhang, et al. Metric3d v2: A versatile monocular geometric foundationmodel for zero-shot metric depth and surface normal estimation. IEEE Transactions on PatternAnalysis and Machine Intelligence, 2024.

Dear reviewer, thanks for your valuable comments and suggestions. We are glad to provide further clarifications and experiments to address your concerns.

1. Visualization of reflection strength prior.

Due to the policy restriction, we are unable to show visualization of the reflection strength priors. To quantitatively evaluate the effectiveness, we binarize our reflection strength prior and uses the ground truth materials to identify highly reflective surface regions that are either predicted or the ground truth. We then compute the Accuracy and F1-score of the classification results on GT meshes between the predicted and ground truth regions, as reported below. The experiments are conducted on ShinyBlender dataset where there are ground truth meshes and materials. The results demonstrate that our reflection strength prior accurately localizes the highly reflective surfaces. We will include more visualization of the reflection strength prior in the revised version.

2. Visualization of indirect illumination.

We have included several examples of indirect illumination in the main paper, such as:

• the reflection of the flowerpot in the tray at the bottom in the first row of Fig.1;
• the reflection of the teapot lid’s handle on the lid surface in the second row of Fig.4;
• the reflection of ground seams on the metallic ball in the fourth row of Fig.5;
• inter-reflections in the first and third rows of Fig.2 in the supplementary material.

Unfortunately, due to the policy restriction, we are unable to provide more visualization here. As an alternative, we evaluate the rendering performance in indirect illumination regions labeled by ground truth masks calculated on the ground truth meshes from each view. The experiments are conducted on ShinyBlender and GlossySynthetic datasets, where there are ground truth meshes. The results are reported in the table below. We do not include LPIPS metric here, as it is designed for full image evaluation. Our method shows significant improvements over the baselines in indirect illumination regions, demonstrating the effectiveness of our environment modeling module. We will include more visual examples in the revised version.

3. About environment modeling module.

Ref-Gaussian separates surfaces into direct and indirect illumination regions and uses rendered residual colors from Gaussians to approximate indirect radiance. However, this approach has significant limitations in modeling complex inter-reflections between objects. IRGS introduces 2D Gaussian ray tracing, but it requires Monte Carlo sampling at every ray-surface intersection to evaluate the rendering equation, which significantly slows down the training and inference, and makes it less scalable to complex real-world scenes.

Our method combines the strengths of both approaches. We apply the split-sum approximation in indirect illumination regions and conduct 2D Gaussian ray tracing along the reflected rays to model higher-order lighting effects. As a result, we achieves significantly better rendering quality. Qualitative comparisons can be found in the second row of Fig. 4, the fourth row of Fig. 5 in the main paper, and the first, third, and fourth rows of Fig. 2 in the supplementary material. The numerical comparisons in the indirect illumination regions, as shown in the table above, further highlight the strengths of our approach.

4. Why not supervise roughness & The use of albedo in diffuse and specular color.

The primary goal of our work is modeling reflection effects in complex scenes for novel view synthesis. Therefore, our approach of material modeling and its usage differs from inverse rendering methods such as IRGS and GS-IR. Following Ref-Gaussian, we directly rasterize the diffuse color $c_{diffuse}$ using per-Gaussian spherical harmonics and use other material properties such as albedo, metallic, and roughness to evaluate the specular color $c_{specular}$, rather than using albedo and metallic to compute both diffuse and specular components as done in inverse rendering methods. This design avoids the challenging task of evaluating albedo and metallic values for all locations in complex real-world scenes. Moreover, since the diffuse component is not sensitive to viewing direction, delegating the prediction of diffuse color to the Gaussians allows us to better disentangle reflection effects from illumination. In such case, our rendering equation can be specifically written as follows,

$$\begin{aligned} c_{\text{diffuse}}&=(1-m)c_d,\\\\ c_{\text{specular}}&=\left(((1-m)\cdot 0.04+m\cdot a)\cdot F_1+F_2\right)\cdot\text{env}(\omega_i, r),\\\\ F_1,\ F_2&=\text{lookuptable}(\omega_i, r),\\\\ c_{\text{final}}&=c_{\text{diffuse}}+c_{\text{specular}} \end{aligned}$$

Where $\text{env}(\omega_i,r)$ denotes querying from the mip-mapped environment cube maps using the incident direction and roughness. For indirect illumination, this term is replaced by a combination of the residual color and the color obtained via 2D Gaussian ray tracing. $F_1,F_2$ are obtained through table lookup based on the incident direction and roughness.

From the above equation, we can see that roughness only influences our color modeling indirectly, whereas the metallic value directly determines the proportion of light attributed to specular and diffuse components. Therefore, we apply the reflection strength prior as a regularization term for the metallic prediction.

5. Ablation on normal prior.

We conduct additional ablation studies on ShinyBlender dataset under three experimental settings: A. Full model; B. Full model without normal prior; C. Full model without normal prior and without other components (including material regularization and environment modeling). We report the normal accuracy using MAE, as well as the geometric reconstruction accuracy using Chamfer Distance (CD) below.

The results indicate that, beyond the normal prior, our material regularization and environment modeling also contribute significantly to geometry learning. This is because both our material constraints and environment modeling are differentiable to the depth and normal, enabling end-to-end joint optimization of geometry, appearance, and material properties for improved overall performance.

6. About the segmentation of SAM2.

Sorry for the misunderstanding in Fig.3 and description. We don't enforce SAM2 to perform semantic segmentation of the scene. Instead, we encourage SAM2 to distinguish patterns that exhibit different appearances due to varying material properties, which is not limited to semantic objects. It can divide a single object into multiple patches to identify regions with different material attributes. Furthermore, the segmentation granularity of SAM2 is controllable by configuring the number of anchor points per side and specifying whether to prioritize traversal based on minimum or maximum patch areas. By setting an appropriate granularity level, we can prevent regions with different materials from being merged into a single part. We provide an additional ablation study on the impact of segmentation granularity, and report the granularity level, the average number of patches predicted by SAM2, and the rendering metrics, as shown in the table below.

In the table, G1 represents the finest segmentation granularity, while G4 corresponds to the coarsest. The experimental results show that finer granularity leads to better performance, however, the difference is not significant. Our method exhibits robustness to the choice of segmentation granularity. We use the same setting as G1 in our implementation. We will add more segmentation visualizations in the revised version.

7. About self-occlusion shadows.

Our method is able to effectively handle inter-reflection effects caused by self-occlusion between reflective objects. As for the shadow cast by one object onto another behind it, it may indeed be learned in the diffuse component. However, our method mainly focuses on modeling reflection effects in the scene for novel view synthesis. Since the light sources in the scene are fixed, shadows can be naturally interpreted as an intrinsic component of the diffuse color. Therefore, the shadows in the diffuse map do not harm the rendering quality from different views. We will add visualizations of multi-view rendered shadows in the revised version to demonstrate this. To explicitly disentangle shadows from illumination, future works may include incorporating image relighting priors to remove shadows from input images, or leveraging additional input images captured under varying lighting conditions to provide richer illumination priors.

I appreciate the authors' thorough and thoughtful rebuttal. Most of my concerns regarding the weaknesses and questions have been adequately addressed. The additional experiments are detailed and clearly demonstrate the effectiveness of the proposed method.

However, the contribution related to environment map modelling still feels limited, as it primarily combines the strengths of existing methods without introducing substantial novelty.

Based on the overall improvements, I have decided to raise my score to Borderline Reject.

I hope that more comprehensive visualizations will be included in the final version of the paper.

Dear reviewer, thanks for your valuable comments and recognizing our impressive results, novel motivation, and well-organized structures. We are glad to provide further clarifications and experiments to address your concerns.

1. Explanation of Eq. (1).

Eq. (1) describes how Gaussian ellipsoids in 3D space are rasterized into 2D images, following the classic volume rendering formulation[1][2],

$$\hat{\Psi} = \sum_{i=1}^N \psi_i * o_i * p_i * \prod_{j=1}^{i-1}(1-o_j).$$

Here, $\psi_i$ denotes a selected attribute of the Gaussian $G_i$, such as diffuse color, albedo, and metallic. $o_i$ represents the opacity of $G_i$. The term $p_i$ indicates the probability that a given pixel lies within the projection of the 3D-space Gaussian onto the 2D screen space. The value of $p_i$ reaches 1 at the center of the projected Gaussian and decays outward. It is equivalent to evaluating a 2D Gaussian distribution function at the given pixel location. The term $\prod_{j=1}^{i-1}(1-o_j)$ denotes the accumulated transmittance, indicating how much light passes through the preceding Gaussians before reaching $G_i$. We will add a more clear clarification in the revised version.

2. Ablation studies on patch size.

The patch size refers to the side length of the image patch warped from the reference view to the source view. We conduct additional ablation studies on the effect of patch size in Multi-view material consistency and Reflection strength prior on Ref-Real dataset, as reported below. As shown in the table, different patch sizes exhibit similar results, except when the patch size is too small, which makes the patch error more sensitive to outliers. These results demonstrate our robustness to the selection of patch size. In our implementation, we use path size as 7 for both multi-view material consistency and reflection strength prior. We also notice that there is a typo in line 182 of the main paper, where we mistakenly wrote the patch size as 3×3 instead of 7×7. We will correct it in the revised version.

3. 3DGS or 2DGS.

We exactly use 2DGS as our representation. We will carefully review the paper and ensure consistent description.

4. About Figure 3.

We provide a more clear clarification of the pipeline of Figure 3 in the following. We will enlarge Figure 3 and add symbols alongside their corresponding textual descriptions in the revised version to improve interpretability.

  i) Select a reference view and M source views.

  ii) Warp pixel patches from reference view to source views. (Eq. 4)

  iii) Compute reflection scores between patches on GT RGB images. (Eq. 6, Fig. 3a)

  iv) Back-project the reflection scores into a 3D point cloud and compute the reflection strength prior. (Fig. 3b)

  v) Use rendered metallic maps to compute $\Gamma$ and $M_0$. (Fig. 3c)

  vi) Use the reflection strength priors $w_{ref}$, $\Gamma$, $M_0$ to compute $L_{ref}$. (Eq. 7)

[1]. Robert A Brebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. In Seminal graphics: pioneering efforts that shaped the field, pages 363–372. 1998.

[2]. Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph., 42(4):139–1, 2023.

Dear reviewer, thanks for your valuable comments and recognizing the impressive results and the novelty of our work. We are pleased to provide further clarifications and additional experiments to address your concerns.

1. About video visualization.

In the supplementary video, we provide novel view synthesis results on both synthetic and real scenes, material decomposition results, and material editing results in terms of diffuse color and roughness. We also provide novel view synthesis of changing environment maps for relighting at the end of the video. Unfortunately, due to the restrictions on rebuttals, we are unable to show more environment relighting results. As for additional relighting applications such as inserting extra light sources (point lighting or parallel lighting), we can render these results based on our inferred material decomposition. We will include more visualization results in the revised version of our video.

2. About complexity of our method.

We provide a flowchart for the two regularization strategies introduced in Section 3.2 to make it more clear.

    (I) Multi-view Material Consistency.

      i) Select a reference view and M source views.

      ii) Warp pixel patches from reference view to source views. (Eq. 4)

      iii) Compute $L_{mv}$ between patches on rendered material maps. (Eq. 5)

    (II) Reflection Strength Prior.

      i) Select a reference view and M source views. (The same as above)

      ii) Warp pixel patches from reference view to source views. (Eq. 4) (The same as above)

      iii) Compute reflection scores between patches on GT RGB images. (Eq. 6, Fig. 3a)

      iv) Back-project the reflection scores into a 3D point cloud and compute the reflection strength prior. (Fig. 3b)

      v) Use rendered metallic maps to compute $\Gamma$ and $M_0$. (Fig. 3c)

      vi) Use the reflection strength priors $w_{ref}$, $\Gamma$, $M_0$ to compute $L_{ref}$. (Eq. 7)

As the reviewer observed, the i) - iii) steps of both processes share a very similar logic, which allows us to use the same implementation in practice. The only difference is which maps are applied to compute the patch errors.

3. Evaluation of environments and materials.

It is not easy to directly evaluate the predicted diffuse and specular colors in these benchmarks, as there are no ground truths available for reference. As an alternative, we evaluate the performance of the material decomposition on Synthetic4Relight dataset which provides ground truth materials, as reported in Tab. 1 in supplementary materials. We further evaluate the accuracy of the learned environment maps on ShinyBlender dataset, as reported below. The results demonstrate that our multi-view consistent material decomposition and environment modeling contribute to learning more accurate environmental illumination.