Normal-GS: 3D Gaussian Splatting with Normal-Involved Rendering

We thank the reviewer for helpful comments and suggestions. We are glad to address the issues raised in the review.

Q1: Comparison to NeRF-based methods

We would like to include the comparisons to NeRF-based methods here and in the final version. It is important to highlight that 3DGS-based methods, including ours, are relatively fast and could support real-time rendering compared with NeRF-based methods. As illustrated in Table C, on the Mip-NeRF 360 dataset, our method performs the second best in terms of the PSNR and achieves comparable SSIM and LPIPS scores. The involvement of the proposed IDIV could provide better novel view synthesis quality compared with Ref-NeRF, which further assures the novelty of our design. Please also note that the training time for Zip-NeRF is > 10 hours on high-performance GPUs, which is much longer than 3DGS-based methods, including ours.

Table C. Rendering quality comparisons on the Mip-NeRF 360 dataset.

Q2: Definition of normals in the context of 3D Gaussians.

We acknowledge that we did not define normals for true surfaces but rather established a normal direction for each Gaussian. As discussed on L202-203, we define the normal for a 3D Gaussian based on its geometric properties. Specifically, during optimization, 3D Gaussians often exhibit flatness around the surface areas, as observed in [5, 7, 45]. Therefore, for a near-flat Gaussian ellipsoid, its shortest axis thus functions as the normal vector. Consequently, we use the shortest axis of each Gaussian as its normal attribute. For areas overlapped by multiple Gaussians, the normal is the weighted average of overlapped Gaussians. As shown in Fig.1, we found with our IDIV design, the model can also render a reasonable surface normal for the semi-transparent cover in this definition. Moreover, defining physically correct normals for semi-transparent areas remains an interesting and open problem and we leave it for future work.

Q3: Metrics on the reconstructed geometry other than the normal accuracy.

We agree that the overall geometry quality is also important, in addition to the normal accuracy. We follow your suggestions and add experiments to evaluate the reconstruction quality of our method on the DTU dataset in terms of the mean Chamfer distance. We also report the rendering quality indicated by PSNR to demonstrate the advantages of our method. The results of 3DGS [2], SuGaR [8], and 2DGS [44] are adopted from Table 3 of 2DGS [44]. For a fair comparison, we conduct experiments under the same setting as 2DGS [44]. As Table 9 illustrates, our method achieves outstanding rendering quality while maintaining better geometric quality than 3DGS.

Table A. Geometric and rendering quality comparisons on the DTU dataset.

We appreciate your recommendations regarding the comparisons to NeRF-based methods, clarification of defining normals, and measurements of the overall geometry quality. We will add them in the revised version.

We thank the reviewer for appreciating our work and making constructive comments.

Q1: Clearly explain the network output, and how components like opacity and diffuse albedo are calculated.

We would like to clarify it here and in the final version. The whole procedure about how to calculate IDIVs and the specular color is the same as your comments in the second paragraph of the weakness section and as our explanation on L194-197. As for the opacity, because the updating strategy of the Scaffold-GS depends on the value of opacity, we followed the Scaffold-GS and predicted it using a global MLP. As for the diffuse albedo, we reused the color MLP of the Scaffold-GS to predict it.

Q2: How is “3DGS w/ IDIV” in the ablation studies (section 4.2) (b) implemented?

For “3DGS w/ IDIV” in Fig. 5 (b), we attached the IDIV as an additional attribute to each 3D Gaussian without using global MLPs. As described on L299, in this setting we applied the Laplacian and Total-Variation loss to IDIVs. This was done by first rendering IDIVs following the 3DGS rendering pipeline and then applying image-space regularizers. Please note that this setting was “sensitive to the tuning of loss weights” as discussed on L302 and motivated us to use the locally shared structure, such as Scaffold-GS, to regularize IDIVs.

Q3: Quantitative Ablations.

Thanks for the constructive suggestions. To better verify the effectiveness of the components, we add the quantitative results on DTU by comparing the mean chamfer distance and PSNR for geometry and rendering quality evaluation. Please refer to Table B. Our base model is the Scaffold-GS. 1) We add depth-normal into the Scaffold-GS, the results of (b) in Table B shows that the depth-normal loss can significantly improve the geometry quality but ruin the rendering with more than 2dB PSNR drop. 2) Then we introduce the specular component into (b), which can improve the rendering quality without compromising the geometry, which further assures the importance of involving normal into the rendering. 3) We add the proposed IDIV into the model to get our final model (d). Our full model further improves the PSNR by more than 0.2 dB with similar geometry quality. Compared with the base model (a) and the results of 3DGS (CD: 1.96, PSNR: 35.76), our full model achieves a better balance between the geometry quality and the rendering fidelity.

Table B. Quantitative ablation studies on the DTU dataset.

Q4: Some typos.

Thank you for pointing out these typos. We will fix them in the final version.

We will also refine our writing as suggested and include the mentioned experiments in the revised version. We appreciate the reviewer's valuable help and comments.

I somewhat agree with the comment from reviewer 5i8r that the motivation for directly optimizing the surface normals, which are byproducts of geometry, without targeting the geometry itself, is weak.

From the perspective of geometry estimation, I believe that comparisons with methods like SuGaR and 2DGS, which were conducted in the rebuttal, should be included in the paper.

However, since the calculation of normals is essential for using geometry as a rendering component, I still believe this research holds value.

Assuming that the implementation details and comparisons with existing surface reconstruction method are supplemented, I will maintain my score.

We thank the reviewer for appreciating our work and providing constructive comments.

Q1: The illumination of diffuse and specular components is independent of each other. This is physically implausible.

We agree that diffuse and specular components cannot be independent physically. However, our main purpose is to strive for a better balance between rendering and geometry quality. Our insight is to keep the model simple without solving the highly-unconstrained inverse rendering problem, which needs too many extra regularizers. We do believe our framework could be further improved by introducing additional parameters like metalliciness [16, 17, 18, 45] for downstream applications.

Q2: Visualization of Diffuse, Specular and IDIVs.

Thanks for the helpful suggestions. We are glad to provide more results to show the plausibility of our method. We show the visualization of our specular and diffuse components in Figure A of the one-page PDF. We choose scenes containing specular materials to better demonstrate the effectiveness of our method. From the visualization, we can observe that our method successfully disentangles the diffuse and specular components. We also visualize IDIVs in Figure B of the one-page PDF. We choose an outdoor scene for better visualization. We can observe that IDIVs approximately align with the sunlight in bright regions (on the table) and roughly diverge in the shadow region (under the table), which indicates that our IDIVs capture local incident lighting information.

Q3: The anchor-based regularization would prevent the reconstruction of detailed geometry like bicycle spokes in Fig. 6.

We acknowledge the detailed geometry like bicycle spokes is not fully modeled by our method. The thin structure is a challenging problem in discrete Gaussian representation which also related to the aliasing issue. We leave it to the future work.

Q4: Furthermore, since IDIV depends on the surface normal through $\Omega^{+}$, it also requires the smooth surface normal.

We agree with your comments. A smooth and correct normal is important for many applications, including estimating lighting and supporting capturing specular effects. We applied the depth-normal loss to regularize estimated normals.

Q5: Scaffold-GS features.

We agree that view-independent Gaussian attributes are desirable. However, during our experiments, we found that view-dependent features would produce slightly better rendering quality (~0.1-0.2 dB). We believe that the viewing direction could help determine the inside-outside direction of the surface, thus benefiting the rendering quality.

We are glad to include the suggested results in the revised version. Thank you again for your valuable feedback.

Thank you very much for your review and valuable feedback. Below we address the concerns raised in the review.

Q1: L102-103, the authors claim that PBR-Based 3DGS methods show lower rendering quality than original 3DGS, which is not correct, please refer to the tables in Gaussian Shader [7].

We appreciate the reference to Gaussian Shader [7], which indeed demonstrates improvements in rendering quality on synthetic datasets. We agree that [7] demonstrates better performance under controlled, synthetic conditions. However, our primary concern is the Gaussian Shader’s reliance on a global environment map, which does not align with real-world scenarios with analysis on L165-174 of our paper. In our experiments, as illustrated in Figure 4 and Table 1, [7] faced difficulties in modeling complicated real-world scenes. These experiments are re-run directly using their released codes.

Q2: 2DGS [44] is not mentioned or included in the experiment section.

We would like to clarify that the omission of 2DGS [44] from our experiments was not intentional. We discussed and cited 2DGS on L94-95 of our paper. However, it is important to note that 2DGS was recently released (with codes available on May 3rd). This timing is the primary reason for its exclusion from our experiments. Additionally, as noted on L94-95, 2DGS sacrifices some rendering fidelity in novel view synthesis due to the lack of a clear relationship between geometry and appearance, as shown in Table 3 and Table 4 of [44]. We also want to emphasize that 2DGS and our method are conceptually orthogonal and could potentially be combined. Unlike 2DGS, our method explicitly addresses the interactions between lighting and normals. We have also added comparisons with 2DGS, focusing on geometric reconstruction quality and rendering quality. Please refer to our response to Q3 and Table A for these comparisons.

Table A. Geometric and rendering quality comparisons on the DTU dataset.

Q3: The quality of normals is not always the primary factor when evaluating geometry. Instead, the overall geometry quality holds greater importance and should be the focus of evaluation.

We agree that overall geometry quality is important, but we believe that accurate normal is also very useful for many applications and has nice properties such as strong generalization capability [A, B]. We appreciate the reviewers' recognition of the accuracy of our normal predictions. To address the importance of overall geometry quality, we followed your suggestion and further evaluated our method's performance on the DTU dataset. We focused on both the reconstruction and rendering quality. We assessed the mean Chamfer distance and PSNR, adopting results for 3DGS [2], SuGaR [8], and 2DGS [44] directly from Table 3 of 2DGS [44]. For a fair comparison, we followed the process from 2DGS. As shown in Table A, our method achieves outstanding rendering quality while with a better mean Chamfer distance than 3DGS.

[A] Bae, Gwangbin, and Andrew J. Davison. "Rethinking inductive biases for surface normal estimation." CVPR. 2024.

[B] Guangyao Zhai et.al Monograspnet: 6-dof grasping with a single rgb image. ICRA. 2023.

Q4: The integration of PBR … with 3DGS is not a novel concept. ...existing methods mostly enable “relighting” while the proposed method is not.

We acknowledge the significant contributions of previous PBR-based methods [7, 16, 17, 45], especially as they enable “relighting” editability. However, as we discussed on L44-51 and L98-101, prior attempts often require approximations and meticulous regularization terms, which compromise either rendering quality or geometric accuracy. This motivates us to design a straightforward method to integrate normal information explicitly and simply into 3DGS. We also directly compared with the PBR-based method, Gaussian Shader [7], in our experiment section and our method achieves better rendering results and normals.

Q5: The motivation and novelty of our work.

Thank you for your feedback. We respectfully disagree with the characterization of our method as merely a combination of existing approaches. Our primary motivation is to address the balance issue between rendering and geometry quality, rather than focusing on relighting or intrinsic decomposition. From a physically-based rendering perspective, we identify that the core issue arises from the lack of interaction between appearance and normal estimation. Solving this problem is challenging, and our approach aims to simplify the model while avoiding the complexities of highly unconstrained inverse rendering problems, which often require numerous additional regularizers. Our method introduces only a few additional IDIV and reflection parameters, yet achieves remarkable improvements in normal estimation and maintains rendering quality. We believe our approach provides valuable insights into the issue and establishes a new baseline that could benefit the community.

Q6: ... under the assumption of Lambertian reflectance. ... incorporate specular components to account for specular effects.

We would like to clarify that our approach does not assume objects are Lambertian. As stated on L151-153, our initial consideration is for a simple ideal case with Lambertian objects. For more complex materials, including those with specular effects, we address this in Section 3.4 of our paper. As illustrated in Figure 2 of our paper, our method is designed to handle both diffuse and specular components. This allows us to account for a range of material properties beyond the Lambertian assumption.

We thank you again for your valuable feedback and we will include suggested experiments in the final version.