Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set

1. Reconstructing details. Since we use the marching cubes algorithm to extract the zero-level set as a mesh surface, our results is limited by the resolution used by the marching cubes. While 2DGS uses multi-view depth images to reconstruct meshes with Poisson surface reconstruction, which achieves higher resolution and recover more geometry details. We additionally use the same way of reconstruction as 2DGS, and show visual comparison in Fig. F in our rebuttal PDF, where we can see that our reconstruction show more geometry details than 2DGS.

2. Results between Tab.2 and Tab.4. Thanks for pointing this out. Before the deadline, we updated our results frequently, but we forgot to update the latest results everywhere in the paper. We wrongly reported an older version of ablation study, which was conducted on the first four scenes in TNT dataset. We will update the ablation study with the final version, which was conducted on all of the TNT scenes, as reported in Tab. B in the rebuttal PDF.

3. About ablation study. The TNT scenes are large and complex, therefore we believe the quantitative results on TNT dataset are more convincing than that of DTU dataset. However, the qualitative differences are more distinct on small object scenes such as DTU dataset, therefore we chose DTU dataset for visualization. Due to the time limit, we select the first four scenes in DTU dataset to conduct additional ablation study to demonstrate the effectiveness of our method, as shown in Tab. C in our rebuttal PDF. We will update the results with all scenes of DTU in our revision.

4. Rendering evaluation setting. For fair comparison, we directly borrowed the results of all baseline methods from Table 4 in 2DGS paper. Due to the limit of memory, current methods, including NeRF-based and 3DGS-based, coherently train outdoor scenes with a down-sampling factor of 4 and train indoor scenes with a factor of 2. Our experiments keep the same settings with baseline methods. We will clarify this in our revision.

5. About far away points. Indeed, most of the Gaussian points are distributed around the ground truth surfaces, which can be seen as a noisy point cloud. The Gaussians that are significantly far away from the surfaces usually have small opacity values. In our implementation, we practically utilize a filtering strategy before pulling operation to filter out the Gaussians that have too small opacities, similar as what SuGaR does at 9000 epoch. Through this strategy, almost all of the Gaussian points are distributed around the surfaces. If there are several points that are still far away, the loss gradient of these points will be averaged by other points and they will be ignored by the learned SDF.

6. why not make $f(\mu_j’)=0$. Our preliminary experiments show that directly constraining the Gaussians to be the zero-level set will hurt the learning of SDF, as shown in Fig. G in our rebuttal PDF. The reason is that the signed distance field has large uncertainty during the optimization, and the pulled queries produced by inaccurate pulling will lead to inaccurate zero-level set. Directly using a hard constraint will make the optimization inefficient. Our results show that implicitly aligning Gaussian disks to the zero-level set resolves this problem.

7. About w/o Pull GS. We train SDF by pulling randomly sampled queries to their nearest Gaussians on the zero-level set. The setting of “w/o Pull GS” means that we do not pull Gaussians onto the zero-level set and just pull randomly sampled queries onto their nearest Gaussians. The full model with pulling GS means that we align the Gaussians with zero-level set of SDF, which means that the positions of Gaussians are encouraged to move towards the zero-level set.

8. Comparison with GOF. Our method shows comparable rendering results with GOF, as reported in Tab. D in our rebuttal PDF.

9. Number of Gaussian points. We sample 100,000 points to train SDF in each epoch. Because we need to calculate the nearest Gaussian for each sampling point, the number of sampled Gaussian points per epoch is also 100,000.

Thank you for the response. I have also read the other comments and responses. I have some following questions.
1. Reconstruction details. 2DGS uses TSDF fusion to reconstruct meshes. However, the method combines Poisson surface reconstruction to reconstruct meshes from Gaussian points to show the details. It is unfair. Moreover, it is easy to extract higher resolution meshes from SDF, as shown in MonoSDF [1]. Can you show higher-resolution meshes on DTU or TNT dataset? In fact, DTU is a small-scale object-centric dataset. However, the proposed method cannot reconstruct details well on DTU dataset.
Yu, Zehao, et al. "Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction." Advances in neural information processing systems 35 (2022): 25018-25032.
3. About ablation study. Since TNT dataset is more challenging, it is better to show ablation qualitative results on this dataset. Since the the proposed method cannot handle details well, I would like to know how the proposed strategies improve surface reconstruction on the TNT dataset.
6. Why not make $f(\mu'_j)\approx0$? In fact, I wonder if the $f(\mu'_j)\approx0$ operation can help reconstruction instead of $f(\mu'_j)=0$. Moreover, can you try this using the filtered Gaussian points as you mentioned in 5. About far away points.
8. Comparison with GOF. The results shown in Table D are not consistent with those in the main paper. Can you check them carefully?
In addition, since the proposed method is efficient, why only using 4 scenes to conduct ablation studies on DTU dataset? I am curious whether the results in Table C are convincing.

Dear reviewer ZhFP,

Thanks for your questions.

1. Reconstruction Details

If compared methods can recover comparable surface positions, then more geometry details definitely produce better numerical results. However, if one method fails to recover accurate surface positions, then more geometry details could not improve the numerical results a lot. If a reconstructed surface does not come from an accurate zero-level set, it would have inaccurate surface position which drifts away a lot from the GT surface, which hurts the numerical results. To show this, we draw a figure below to illustrate the comparison. This figure shows that our surface (▲) is nearer to the ground truth (*), although the geometry details recovered by the marching cubes algorithm are not many. In contrast, the other surface (■) has much more geometry details which also look quite similar to the ground truth, but fails to recover accurate surface position, drifting far away from the ground truth, which hurts the numerical results a lot. This is also indicated quite clearly in Fig.C in our rebuttal PDF, where the error map across the whole surface obtained by 2DGS looks much darker (bigger errors) than ours.

*: GT surface, ▲: Our surface, ■: 2DGS surface
---------------------------------------------------
Our surface vs. GT
   *        ***
  *▲*      *▲  *▲      
▲▲▲ ▲▲  ▲▲▲▲ ▲▲▲ ▲▲ ▲▲▲▲
      ▲▲ *        *▲
       **
---------------------------------------------------
2DGS surface vs. GT
   *        ***
  * *      *   **
**   *    *      *  ****
      *  *        **
       **
             ■
            ■ ■
  ■■■      ■   ■■    ■
■■   ■    ■      ■  ■ ■
      ■  ■        ■■   ■
       ■■
---------------------------------------------------

Moreover, we seriously checked our code, especially the evaluation part, we did not see bugs. For fair evaluations on TNT, we use the official github code released with the dataset of the original paper “Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction”.

8. Comparison with GOF

We achieved comparable rendering quality to GOF in indoor scenes. We also noticed that our rendering quality is worse than GOF in outdoor scenes. The main reason is that outdoor scenes include lots of huge Gaussians, each of which covers a large area. This may be caused by the lack of view coverage. These huge Gaussians usually struggle to meet all the constraints that we imposed for SDF inference. For example, they can not be aligned well with the gradient at the zero-level set, and their large size amplifies little normal adjustment to a large impact on the rendering, since it can not take all area it covers into account. This also raises up a future work direction, that is how to control the size of Gaussians during our inference, so that we can maintain good rendering quality of GS.

Dear reviewer ZhFP,

Thanks for your questions.

1. Reconstruction details

Since our method infers signed distances from 3D Gaussians, we believe the 3D Gaussians are the key to recover geometry details. Although 3D Gaussians are dense (for most time but we still have some sparse cases), they are not dense enough like point cloud scans to recover fine geometry. This is because 3D Gaussian is not a point but a sphere or a plane which covers some space. This is fine for rendering, since the pixel color is usually produced by several overlapped Gaussians, but obviously using one Gaussian is hard to infer fine geometry in the area it covers, especially when the Gaussian is very large, which makes them much different from points back projected from the rendered depth. Just like the extreme case raised up by reviewer SW8k, the large holes in the middle of Figure 15 in the outdoor scene, we observed huge Gaussians in these areas. Although these huge Gaussians may work well in rendering, but they cannot recover any geometry covered by them. Thus, how to control the size of Gaussians for SDF inference could be an interesting future work direction.

Another reason is like what we mentioned in the limitation. The MLP we use to estimate an SDF prefers low-frequency information, which also affects the performance of recovering fine geometry. This can also be resolved in our future work by capturing high-frequency geometry either using high frequency positional encoding or without using MLP.

We will add this discussion in our revision.

2. MipNeRF360 ablation

We conduct additional ablation studies on MipNeRF360 dataset, as shown in the following table. Due to the time limit, we select two scenes from indoor and outdoor scenes, respectively. The terms that are irrelevent to rendering are omiited. We are trying our best to report the results on all scenes along with the ablation results on DTU before the discussion ends. We would like to clarify in advance that ablations related to rendering metrics do not show as clear distinctions as those related to reconstruction metrics. This is because the rendering metrics are usually more stable than reconstruction metrics, which is evident in previous works.

The term "w/o $L_{Tan}$" has the most significant impact on rendering quality because we align the Gaussian normals with the surface normals to obtain consistent orientated Gaussians, which enhances the rendering quality. The term "w/o Pull GS" has the second most significant impact on rendering metrics, because we optimize the Gaussian positions by pulling them towards the zero-level set to achieve better rendering quality. The other terms "w/o $L_{Thin}$", and "w/o $L_{Oth}$" are mainly designed for learning SDF and have a minor improvement on the rendering metrics.

Please feel free to let us know if you have any more questions.

Best,

The authors

We tried two implementations to add high-frequency information. The first one is appending positional encoding to the input XYZ coordinates, like the original NeRF. The other is replacing the input 3D points with multi-resolution hash encoding, like Instant-NGP. All parameter settings are consistent with those of the original implementations. The experiment is conducted on the Ignatius scene of the TNT dataset as requested. Using the same experimental setting, as reported in the following table, both adding positional encoding and hash feature grid from instant-ngp can slightly improve the reconstruction visually and numerically. We do not have time to try more parameter options like frequencies in pe and grid resolution in hash feature grid, but we will report that in our revision.

Ours: 0.71

+positional encoding: 0.73

+hash encoding: 0.75

1. Explanation of the method. When training 3DGS, 3D Gaussians are progressively approaching to the zero-level set. At the same time, we pull Gaussians to align with the zero-level set of SDF. Under the joint optimization of these two process, along with our novel pulling operation and constraints, the Gaussian point clouds that get pulled on the zero-level set gradually turn out to be clean, which can be used as the pulling target to infer signed distance in the field, leading to surfaces fitted by the SDF that gradually recovers detailed and accurate geometry.

We agree that the learned SDF field is coarse at the very beginning, which is not a good guidance for Gaussians. Therefore, we firstly train 3DGS for 7000 epochs, leading to relatively stable Gaussians, and then start to pull queries onto Gaussians to estimate a rough SDF, and finally we pull Gaussians on to the zero-level set, and queries are also following to get pulled onto the pulled Gaussians since then.

2. Visual results. The surfaces learned by 2DGS are usually fat and a little bit drift away from ground truth surfaces, although their meshes seem to show more details. Our method is able to capture more accurate surfaces by using 3D Gaussians pulled onto the zero-level set and pulling query points onto Gaussian disks at the same time, leading to much more accurate zero-level set. We additionally report comparisons of error maps on meshes obtained by 2DGS and ours in Fig. C in rebuttal PDF, which highlights our superiority in terms of the accuracy of extracted surfaces. Additionally, the holes in the middle of Figure 15 are due to the ill-conditioned distribution of Gaussians, making it difficult to distinguish geometric structures.

3. 2DGS numerical results. 2DGS wrongly reported the mean result on TNT in its original paper. The calculated average value is indeed 0.26.

4. Results between Tab.2 and Tab.4. Thanks for pointing this out. Before the deadline, we updated our results frequently, but we forgot to update the latest results everywhere in the paper. We wrongly reported an older version of ablation study, which was conducted on the first four scenes in TNT dataset. We will update the ablation study with the final version, which was conducted on all of the TNT scenes, as reported in Tab. B in the rebuttal PDF.

5. $L_{Thin}$. We did mention NeuSG in Line 175. We'll make it more clear that the thin loss is inspired by NeuSG in our revision.

6. Closed Surfaces. SDF can also represent open structures. As a result, it will reconstruct double-layer surfaces. You can refer to the second and third row of “NeuS” in Figure 4 in the paper of NeuralUDF (CVPR2023), where the SDFs successfully distinguish the collar and cuffs, but the reconstructed cloths have tight double-layer surfaces. To avoid the influence of double-layer surfaces on evaluation accuracy, we practically delete the back faces according to the visibility of each face under each camera view. Through this way, we can accurately reconstruct open structures with single-layer surfaces. Additionally, although UDF can reconstruct open surfaces, extracting the zero-level set from UDF as a mesh surface is still a challenge, resulting in artifacts and outliers on the reconstructed meshes. Our method can also learn UDF, and we additionally conduct an experiment using the same setting to learn UDF and compare surfaces extracted from the SDF and the UDF, as shown in Fig. D in the rebuttal PDF, which is a corner of “Barn”, showing the shortcomings of UDF learning.

7. Normal loss. We also tried cosine similarity as the normal loss. We did not see any difference on the performance for these two constraints on normal.

8. D-SSIM loss. We also tried the D-SSIM loss. But we did not see any difference on the performance. For simplicity, we do not include it in the loss function.

9. Performance of NeuralPull. In this paper, we resolve the problem of learning SDF from multi-view images through sparse and noisy Gaussians by innovatively pulling queries onto Gaussian disks on the zero-level set of the SDF. Although we are using pulling, our pulling is much different from NeuralPull. The difference lies in that NeuralPull pulls a query to a point while we pull a query onto a Gaussian disk. 3D Gaussians are constrained to be a disk-like shape which covers an area rather than a point during optimization, which inspires us to introduce this pulling variant. Therefore, the performance of our method is not limited to the bottleneck of NeuralPull. Figure 2 in our paper shows that merely pulling queries to the point cloud, like NeuralPull, does not work well with 3D Gaussians, due to the sparsity. Additionally, we conduct an experiment that using a sparse ground truth point cloud of “Ignatius'” scene to train both NeuralPull and our pulling operation. The point clouds are initialized as spheres to simulate the Gaussians used by our method. The results in Fig. E in our rebuttal PDF highlights the superiority of our proposed method over NeuralPull.

10. Ground of surface. The ground truths of TNT dataset have provided bounding box for culling the reconstructed meshes. For example, the ground truth points of “Truck” and “Barn” contain the ground while the points of “Caterpillar” do not contain the ground. We first crop the meshes using the provided bounding box and then visualize them.

11. Reconstructing details. Since we use the marching cubes algorithm to extract the zero-level set as a mesh surface, our results is limited by the resolution used by the marching cubes. While 2DGS uses multi-view depth images to reconstruct meshes with Poisson surface reconstruction, which achieves higher resolution and recover more geometry details. We additionally use the same way of reconstruction as 2DGS, and show visual comparison in Fig. F in our rebuttal PDF, where we can see that our reconstruction show more geometry details than 2DGS.

1. Eikonal loss. We believe pulling based methods may not need an Eikonal loss to guarantee the learned distance field is an SDF. This is because we use a normalized gradient and a predicted signed distance when pulling a query. It has been proved by NeuralPull[1] that adding the Eikonal loss will significantly degenerate the learning of SDF, as reported by ``Gradient constraint'' in Table 8 in the original paper of NeuralPull. The reason is that NeuralPull depends on both predicted SDF values and gradient directions to optimize the SDF field. It makes the optimization even more complex when adding additional constraint on the gradient length. We also additionally report a result with an Eikonal loss in Fig. B in our rebuttal PDF. The comparison indicates that the results degenerate significantly with an Eikonal loss.

2. SDF property. It has been proved that an MLP which is trained using a pulling loss can converge to a signed distance function. The reason is that the sign of distances needs to turn over when across the surface points so that the pulling loss can be minimized on both sides of the surface points. Please see Theorem 1 in the original paper of NeuralPull for more details.

3. Weight of splatting loss. We set the weight of splatting loss to 1.0, following all 3DGS baseline methods. And we set the weights of thin loss, tangent loss, pull loss, orthogonal loss as 100, 0.1, 1, 0.1, respectively. The weight of thin loss is larger than others because the scaling factor is usually small and we want the smallest scaling factor to be as small as possible, which is the same as NeuSG[2]. A smaller weight of thin loss will cause Gaussians to be not flat enough, leading to an inaccurate alignment of Gaussians to the zero-level set.

4. Densification operation and pull loss. There are three reasons why we separate the densification operation and pulling operation. (1) The 3D Gaussians are sparse and unstable at the early stage of training. Current works usually start to add additional losses after certain epochs of training 3DGS, such as the distortion loss (Eq.13) in 2DGS[3] and the regularization loss (Eq.8) in SuGaR[4]. (2) The densification operation causes the number of Gaussians to grow rapidly, leading to a collapse of memory overflow. Therefore, current works generally stop densification after certain epochs of training. For example, original 3DGS stops densification at 15000 epoch while SuGaR stops it at 7000 epoch. Here we follow the setting of SuGaR. (3) The pulling operation needs to sample query points for each Gaussian, which is time-consuming. Therefore, we only sample query points for all Gaussians once before adding pulling loss for efficiency. If the number of Gaussians are changing during densification, we need to re-sample query points, which will bring additional time consumption. We conduct an ablation study on different stages of stopping densification or start pulling, on DTU scan37 scene, as reported in Tab. A in our rebuttal PDF. The first one is to start pulling loss from beginning, which significantly degenerates the performance and slows down the convergence. Another one is to add densification operation until end of training, which takes much longer training time and degenerates the performance.

[1]. Ma B, Han Z, Liu Y S, et al. Neural-Pull: Learning Signed Distance Function from Point clouds by Learning to Pull Space onto Surface. International Conference on Machine Learning. PMLR, 2021: 7246-7257.

[2]. Chen H, Li C, Lee G H. NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance. arXiv preprint arXiv:2312.00846, 2023.

[3]. Huang B, Yu Z, Chen A, et al. 2D Gaussian Splatting for Geometrically Accurate Radiance Fields. ACM SIGGRAPH 2024 Conference Papers. 2024: 1-11.

[4]. Guédon A, Lepetit V. SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5354-5363.

1. Comparison between flat loss and surfels. Our method requires calculating the inverse of a three-dimensional covariance matrix (Eq.(5) in our paper) to deterimine the distribution probability of a query point within its nearest Gaussian ellipsoid. This allows us to maximize the probability for pulling query points towards nearest Gaussians. The surfel setting, however, merely provides two dimensional of scaling factors, which does not meet our requirement. In general, pushing Gaussians to be flat or setting Gaussians as surfels serves a similar purpose in mitigating the bias of rendering depth and determining Gaussian normals, thus the flatting loss is a reasonable choice. As an evidence, we replace our flat loss (Eq.(3) in our paper) with the surfel setting in GaussianSurfels[1], as shown in Fig. A in the rebuttal PDF. The optimization procedure fails at the start with surfel setting, which demonstrates that it is not a good choice in the differentiable pulling operation.

2. About L_tangent. We consider the direction of the smallest scaling factor as Gaussian normals, following previous works[2,3]. The loss pushes the normal to get aligned with the gradient on the zero-leve set.

3. About $n_j$. It should be $\bar{n_j}$. We will correct it in our version.

[1]. Dai P, Xu J, Xie W, et al. High-quality Surface Reconstruction using Gaussian Surfels. ACM SIGGRAPH 2024 Conference Papers. 2024: 1-11.

[2]. Guédon A, Lepetit V. SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5354-5363.

[3]. Chen H, Li C, Lee G H. NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance. arXiv preprint arXiv:2312.00846, 2023.

Dear AC,

Thanks for initiating the discussion. We are also looking forward to having a discussion with reviewers. It would be great if reviewers could let us know if their concerns are addressed after reading our rebuttal materials. We appreciate that a lot.

Best,

The authors