Geo-3DGS: Multi-view Geometry Consistency for 3D Gaussian Splatting and Surface Reconstruction

We would like to sincerely thank the reviewer for the detailed and insightful comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

W1. Difference from GSDF.
GSDF directly use the NeuS framework to learn the SDF, which requires learning an implicit radiance field. Different from GSDF, our SDF learning does not rely on implicit radiance field learning and further leverages multi-view photometric consistency regularization. To balance new Gaussian primitives derived from depth maps and the existing Gaussians, we empirically employ downsampling to control the number of new Gaussians.

For densification, GSDF first trains 3DGS branch for 15k iterations until convergence. Then, after it warm-ups NeuS with 3DGS for 5k iterations, it then uses the depth map from SDF to guide 3DGS. This process is complex and grealy depends on the learning of NeuS. To better leverage the geometry information of 3DGS to facilitate 3DGS itself, we impose the multi-view photometric consistency to boost the geometry consistency of the depth maps rendered by 3DGS, and then we can use the more accurate depth information to help 3DGS densification.

For our 3DGS-driven SDF initialization, it is common to use detph prior guided near-surface sampling to help SDF learning. In this part, our focus is how to leverage the geometry information of 3DGS to directly initialize SDF. Therefore, we leverage rendered depths (Eq. (10)), zero-level set constraint (Eq. (10)), and weight distribution constraint (Eq. (11)) to achieve this, which are different from GSDF.

W2. Typography.
We are very sorry for these typos and will modify them in our revised version.

W3. Poor performance.
Although NeuralAngelo achieves the best performance on Tanks and Temples dataset, it requires a very long training time. Compared to the concurrent GSDF, GDSR that also leverages SDF, our performance is much better than these methods. Compared to 3DGS and Scaffold-GS, our achieved LPIPS is much better than theirs. Moreover, as shown in Figures 5, 8, 9, compared to 3DGS and Scaffold-GS, our method can significantly reduce blurring artifacts thanks to our geometry-assisted Gaussian densification.

W4. Lack of significant improvement for the multi-view photometric consistency regularization.
How to combine 3DGS and SDF to improve both rendering and surface reconstruction is still an open problem. To achieve this goal, our proposed method is a systematic design. For the improvement of the multi-view photometric consistency regularization, although it improves surface reconstruction by 0.02, it improves the rendering metric, LPIPS by 0.009, which significantly reduces artifacts as shown in Figure 10.

W5. Demo.
We have submitted reconstructed surfaces demos in our supplementary.

Q1. SDF architecture.
We use the hash-encoding-based SDF. Note that, we do not introduce implicit color fields like NeuS, as mentioned in L351. We follow 2DGS to downsample images by a factor of 2 (978x546) for Tanks and Temples dataset, and use full-resolution images (1600x1200) for DTU dataset. The downing operation accelerates the convergence on Tanks and Temples dataset.

We would like to sincerely thank the reviewer for the detailed and insightful comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

W1. Small-overlap or textureless issues.
Traditional photometric consistency cannot tackle these issues well based on the multi-view projection. However, as demonstrated in [1] [2], computing photometric consistency using multi-view deep features can greatly overcome these issues. Therefore, we think that the introduced multi-view photometric consistency is still an important cue for 3DGS.

[1] Learning signed distance field for multi-view surface reconstruction, CVPR 2021.

[2] PSDF: prior-driven neural implicit surface learning for multi-view reconstruction, TVCG 2024.

W2. Difference from NeuSG, GSDF and 3DGSR.
Both NeuSG and GSDF introduce an extra implicit radiance field to help SDF learning. Unlike them, our method directly learns SDF from the geometry of 3DGS. 3DGSR learns the normal information of 3DGS by introducing a new attribute for each Gaussian by following R3DG [3]. This cannot effectively guarantee the intrinsic geometry-normal consistency of 3DGS. Different from these methods, we reveal that the multi-view geometry consistency is also important for 3DGS and leverage this consistency to further facilitate 3DGS and more efficient SDF learning.

[3] Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing, ECCV 2024.

W3. Some typos.
We are very sorry for these typos and will further modify them in our revised version.

Q1. How to choose base pixels?
We randomly choose base pixels. We do not directly use the entire rendered depth to compute the multi-view photometric consistency regularization because it will introduce expensive computation cost for 3DGS. Therefore, we still employ pixel/patch based projection.

We would like to sincerely thank the reviewer for the detailed and insightful comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

W. Difference from MVS and NeRF, and our contributions.

1. Efficient and accurate surface reconstruction with 3DGS. MVS usually adopts depth map estimation and fusion to obtain point clouds and then obtain surfaces by Poisson surface reconstruction. NeRF uses SDF to learn neural implicit surfaces, which usually requires simultaneous implicit radiance field learning. Different from these works, our introduced SDF directly learns neural surfaces from the geometry of 3DGS without the need to learn the implicit radiance field. To effectively achieve this, we find that the multi-view geometry consistency is ignored by 3DGS, which is never revealed by previous works. Therefore, we leverage the photometric consistency to encourage multi-view geometry consistency.

2. Photometric regularization and densification help both reconstruction and rendering. Photometric regularization facilitates the geometry consistency of 3DGS. This improves the SDF initialization efficiently, which is hard for previous NeRF-based methods. On the other hand, photometric regularization helps populate empty areas for 3DGS, as shown in Figure 1(a). Moreover, since 3DGS gradually densifies from sparse point clouds computed by SfM, it is usually challenging for 3DGS to reconstruction the entire 3D scene completely. Our densification leverages more complete geometry information to encourage 3D Gaussians densification. By comparing Model-B and Model-C, Model-C and Geo-3DGS in terms of F1 and LPIPS metrics, our designs improve both reconstruction and rendering. The ablation results in Figures 10 and 12 further support this.

Q1. SDF architecture.
In L731, we show that our SDF architecture is based on hash grids.

Q2. SDF initialization.
Following the practices in NeRF-based surface reconstruction methods, we use the geometric initialization (Atzmon&Lipman(2020)) to initialize our SDF for fair comparisons with these methods. Solving Poisson Equation using Gaussian Splats to initialize SDF is interesting. On the other hand, as the Gaussian splats of 3DGS are inconsistent geometry information, we believe that photometric consistency still can help the SDF initialization.

We would like to sincerely thank the reviewer for the detailed and insightful comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.

W1. Distinct advantages of adapting photometric regularization to Gaussian Splatting and the unique challenges.
3DGS can quickly converge and render images and depths in a real-time manner. In contrast, NeRF-based methods cannot render images and depths in a real-time manner. With the efficiently rendered depths by GS, our method can efficiently impose photometric consistency. However, only using this photometric regularization cannot guarantee satisfactory surface reconstruction and rendering because 1) Although photometric regularization improves the geometry consistency for 3DGS, how to reconstruct surfaces for 3DGS remains open. 2DGS fuse depth maps by TSDF to obtain surface models, which usually needs to tune hyper-parameters for TSDF and requires a large amount of memory. Some methods, such as Neusg and GSDF, use SDF to learn surfaces for 3DGS, but hey require learning implicit radiance field at the same time. Therefore, adapting photometric consistency to GS and learn surfaces with lightweight SDF is challenging . To circumvent memory-expensive TSDF fusion and time-consuming implicit radiance field learning, we leverages SDF and further introduce 3DGS-driven SDF initialization. With our designs, our method only uses the depth being optimized by 3DGS to efficiently guide SDF learning.

W2. Comparisons with GeoNeus and Neuralangelo.
We will add the evaluation results of GeoNeus on DTU. For Neuralangelo, it requires 8 GPUs and a very long training time to sample a large number of rays to get the results. In contrast, our method is efficient by training around 48 minutes on a single GPU. We think that an efficient reconstruction method is more practical in real-world applications.

Q1. Convergence time comparison with Neuralangelo.
Our method is more efficient because: 1) Neuralangelo requires learning implicit radiance filed, which introduces a huge number of extra hash grid evaluations. 2) Our method leverages explicit geometry cues from 3DGS. This can efficiently make SDF focus on near-surface learning.

Q2. Purpose of Eq. (11).
This equation aims to constrain the weight distribution of SDF, regularizing the noise far from the depths of 3DGS.