GNeRP: Gaussian-guided Neural Reconstruction of Reflective Objects with Noisy Polarization Priors

Thank you for your comments. We appreciate your acknowledgement of the novelty and effectiveness of our method. We have carefully considered your comments and listed the responses below: (Weakness - W):

W1. We add an illustration to the Appendix B Figure 6. (a) to show the pipeline of training data preparation. Please see the training data box for intuition. The inputs include polarization images (AoP and DoP), RGB images, and object masks. A polarization camera is needed for polarization image acquisition, we add the claim to the limitation discussion in Appendix E.

W2. Thanks for pointing out self-occlusion as a potential challenge. We tested additional scenes and found the shading caused by self-occlusion is a limitation of our method, and we added a failure case in the limitation discussion shown in Figure 14.

W3. Thanks for the suggestion of testing scenes with mixed materials. During the discussion time, we processed the raw capture of the Camera scene in PMVIR. It’s a Sony camera with various materials. The reconstruction is shown in Figure 9. As is shown in the figure, our method reconstructs more details of small structures with different materials concurrently. However, the lack of GT mesh in the dataset makes quantitative evaluation unavailable.

W4. Thanks for suggesting testing complicated objects. In the initial submission, the Ironman scene shown in Figure 5 has detailed geometry such as the edges of armours. During the discussion time, we rendered two additional synthetic scenes (Bunny and Dragon) using Mistuba render. The results are shown in Figure 8 and Table 3. As shown in GT normal in Figure 8, The geometry of the Dragon is complicated such as claws, the head with feelers, and scales on the body. It shows our method reconstructs the geometry preserving more details while the geomrty of the other methods are over-smoothed.

W5. Thank you for the suggestion on illustrations. We have illustrated the principle of how Gaussians of normals supervised by polarization improve NeuS in Figure 1. (b). We include the polarization prior in the Figure and modify the caption for clarification.

Thank you for providing feedback. We appreciate your agreement with the performance and weak assumptions of our method. We have carefully considered your comments and listed the responses below (Weakness - W, Question - Q):

W1. (a) We estimated the polarized 2D Gaussians based on unitary projected normals vectors derived from the Angle of Polarization ( the right term of Eq. 8). However, the projected normal vectors are unnecessarily unitary in the left term. The scale factor is multiplied for the equivalence of Eq. 8. The final loss gets rid of the dependent of it, so it isn’t explicitly calculated. We added a detailed description of Gaussian estimation in the Appendix A.3.

(b) In the paper, M and M’ are set to 6 and 4, respectively. To estimate 3D Gaussians, we sample 2 points along the ray (before and after the center) and 4 points along the rays passing adjacent pixels. Hence, when estimating 2D Gaussians in polarization images, we only query the corresponding pixels, and M’ is set to 4.

(c).1 The equivalence is ensured by the relation between the azimuth angle and the Angle of Polarization (AoP), shown in Figure 2 and the equation in the last paragraph of Sec. 3.1 (azimuth angle + pi/2 = AoP mod pi). So we can recover the orientation of 2D projected normal vectors from AoP, which should align with the orientation of splatted 3D normal vectors. Therefore, covariance should be the same, too. A visualization of 2D Gaussians is Figure 4. (e). Low saturation white pixels indicate that the 2D Gaussian at that location is close to isotropic, resembling a circle. Conversely, high saturation areas indicate strong anisotropy, resembling an ellipse, with the color representing the orientation of the ellipse. This is shown in the left column of the figure.

(c).2 The eigenvalues of covariance will be irregular in diffuse regions due to noisy AoP. Hence we introduce DoP reweighting to reduce their impact since DoP will be small in these regions, as mentioned in Sec. 3.2.3.

(d)Thanks for the error pointed out. We have fixed it.

W2. Thanks for the suggestion on the setting of an additional ablation study. In our understanding, the required experiment is exactly w/ReW. L_{mean} in Table 2.

W3. Thanks for the advice on the evaluation of diffuse scenes. We processed the raw capture of the Camera scene in PMVIR, which is a more diffuse-dominant scene. A visual comparison is shown in Appendix C.3. Since the ground truth mesh isn't provided, quantitative evaluation is unavailable. However, Figure. 9 shows that our method reconstructs more details of small structures such as buttons, knobs, and slots of the camera. This proves that our method can handle diffuse objects effectively.

W4. Thanks for the additional related paper mentioned. We have added it to the Related Work Section.

Minor. We have changed the Figure 2 as the suggestion. The relation between AoP and the normal vector is more clear. Hope it can help you solve W1. (c).

Q1. A detailed explanation of Gaussian calculation is added to the Appendix A.3. 2D Gaussians of polarization images are introduced in the second paragraph.

Q2. We think the question is the same as W2.

We would like to express our gratitude to the reviewer for his comments and acknowledgments regarding the design of our method and the improved results achieved compared to existing methods. We respond to the weaknesses and questions below.

Weaknesses:

1. In the initial submission, we presented the results of our PolRef Dataset (Ironman, Snorlax, Cow, Duck, Cat, and Vase) and PANDORA dataset (Owl, Black Vase, and Gnome). During the discussion time, we additionally rendered synthetic scenes of Bunny and Dragon. The supplementary results are now provided in Table 3 and Figure 8. Furthermore, we evaluate our method on a diffuse object (Camera) captured in PMVIR, although ground truth meshes are not available for the PANDORA and PMVIR datasets. Visual comparisons are provided in lieu of quantitative evaluation. They show that our method consistently outperforms existing methods on these datasets.
2. We have added the limitations discussion of our method in Appendix E. The primary limitation lies in the requirement of polarimetric imaging, which incurs higher costs compared to RGB cameras. Moreover, the training and inference speeds of our method are lower than those of existing NGP-based reconstruction methods. Moreover, shading caused by self-occlusion may deteriorate the performance. We add a failure case shown in Figure 14.
3. We have included a description of our PolRef dataset, along with details of data collection and the evaluation protocol, in Appendix B and Figure 6.

Questions:

1. We have provided an explanation of the formula to obtain Σ and Λ in Appendix A.3.
2. As mentioned in Weakness 3, we have added a description of the dataset collection process in Appendix B.

We thank the reviewer for their insightful comments. We are encouraged that the reviewer acknowledges that our method achieves good results on the typically challenging task of reconstructing reflective objects and outperforms existing methods quantitatively. We respond to the weaknesses and questions below (Weakness 1 - W1, Question 1 - Q1):

W1. To validate the effectiveness of our method on diffuse objects, we processed the raw capture of the Camera scene in PMVIR. A visual comparison is shown in Appendix C.3. Since the ground truth mesh isn't provided, quantitative evaluation is unavailable. However, Fig. 9 shows that our method reconstructs more details of small structures such as buttons, knobs, and slots of the camera. This proves that our method can handle diffuse objects effectively.

W2. The setting of hyper-parameters is relatively fixed in our method. For clarification, we rearrange the notation of the weights of loss functions in Eq. 9 and add a description of the settings in Appendix D.3. The only hyper-parameter that needs to be tuned is alpha. We tested different choices of alpha on the Bunny scene, as shown in Appendix D.3.2, to facilitate the practical implementation of the method.

W3. In the initial submission, we showed reconstructed meshes of the Owl, Black Vase, Cat, and Vase scenes in the Appendix. Moreover, we rendered two additional synthetic scenes (Bunny and Dragon) with ground truth normals during the discussion time, and qualitative and quantitative comparisons are shown in Appendix C.2, along with an additional evaluation metric, Mean Angular Error (MAE) of normals.

W4. Thank you for pointing out the errors. We have fixed them.

Q1. As mentioned in W2, we list them in Appendix D.3.

Q2. Yes, as parameterized by the covariance matrix, the learned Gaussians are anisotropic. Normal vector can be seen as the mean of 3D Gaussians, and changes of normal vectors within the neighborhood are captured by covariance of Gaussians. Therefore, we claim that the 3D Gaussians capture more details. We have modified the description in Sec. 2.4.

Q3. As mentioned in W2, the required experiments are listed in Appendix D.3.2. Due to the limitation of computational resources, ablation studies were only done on two of the objects in Table 1. We plan to complete them when the computational resources become available. We’ll post them if the results are available during the discussion.

Q4. The first term supervises the eigenvalues of the covariance matrix, and the second term supervises the eigenvectors of the covariance matrix. Similar to PCA techniques, the covariance matrix determines the magnitude and direction of normal changes through eigenvalues and eigenvectors. If the local shape is like a plane, normals will change smoothly in all directions, and the difference between eigenvalues will be small, resulting in the Anisotropy (defined as the ratio of eigenvalues in the paper) approaching 1. If there are some details like edges, normals tend to change abruptly and exhibit directionality, which is represented by eigenvectors. We have modified the description in Sec. 3.2.3.