NeuroSURF: Neural Uncertainty-aware Robust Surface Reconstruction

• Weakness (results):

  • We thank the reviewer for pointing out all related work. We'll add it to our citations. But the main reason we did not compare against all relevant methods is we focused on improving sampling, enabling curvature information, and using depth as input. That is why we only choose the most representative methods to show our ability to improve it. Moreover, as we mentioned, the final goal is beyond just reconstructing a mesh from the point cloud. We would like to have neural implicits representation which might potentially benefit for more downstream tasks. A deeper reason is that we think the trained neural implicit representation can enable more applications such as the deformation of the shape, which we plan to do next. For example, the dynamic object would be much easier recorded by a RGB-D camera.

  • marching cube resolution: all results are rendered using 128 voxel resolutions.

  • We can certainly add them.

  • on the contrary. The reason we give comparison methods the point cloud extract from the voxel grid is because these are more clear and oriented point clouds. Considering the original input, which are depth maps, we could give them the point cloud generated by re-projecting depth into 3D world. Then honestly we will overperformance all comparison methods a lot since the point cloud is noisy (especially in real-world dataset) and doesn't have normals.

  • We can add it of course. We will add it in the appendix.

• Questions:

  • $l_e$ is an Eikonal loss, which ensures the optimized neural network indeed is an SDF field. (we will update the PDF).
  • it is a hyperparameter, we set to $\tau=1e-3$ during mesh extraction.
  • yes. the point has zero uncertainty due to missing data on depth, which could be because of open areas or occlusions. For occlusion is the limitation of our method which we listed in the limitation section.
  • Theoretically, yes. As the neural implicits should represent $f(x, \theta)$ the continuous SDF, which shouldn't have a problem for representing intersections. In practice, depending on sampling and training steps, even the marching cube step could have failure cases on heavily self-intersected areas. The interpolation is designed to complement the inefficient resolution of the initialized voxel grid.
  • Please refer to the biased sampling part in related work and also the answers for the method weakness part.
  • main curvature $H = k_1 + k_2$, Gaussian curvature is $K = k_1k_2$, It is certainly true that main curvature and Gaussian curvature capture different geometry information. Since we are not using curvature values directly, rather than using them as an index to indicate the geometry feature of the surface, and if the reviewer see Fig 2 and Fig 11, the distribution of main and Gaussian curvature are similar. That means choosing a slightly different threshold to divide the low, median, and high curvature categories when using two different curvatures should have relatively similar sampling results. We will add an ablation example to demonstrate and also add an explanation in the appendix. A more mathematical way of think about it is $G = K_1k_2$, $H^2 \geq 4G$, $H$ is kind of a upper bound of $G$ since $|H| \geq 2\sqrt{|G|}$. So, there is no specific reason that we chose the main curvature rather than the Gaussian curvature.

Thank you for the insightful comments and suggestions provided by the reviewers, which we believe have significantly contributed to improving the quality and clarity of our manuscript. In the following sections, we systematically address each reviewer's questions and concerns, providing clarifications and additional information where necessary.

• Weakness 1 (Method):
  • It is true that noisy depth and missing data on depth would damage the quality of computing gradients on depth. Thus we first applied a median filter on depth before computing. For normal computing, the FASL method considers patches with window size $k$ ($k=5$ in our case), not only four neighbors. For curvatures and normals per voxel, the weighted average in equations (19) and (20) can also help reduce the noise since if depth is missing, the uncertainty value $w_i^v$ will also be small. These will cause empty areas when we generate mesh, considering uncertainty (since the uncertainty will be small/zero), which we list in the limitations that our work can not do shape completions.
  • We assume the reviewer means that if we can build voxel ready, which can extract mesh from it, why we still need to use a neural network to learn it. The answer is that the neural representation of SDF and voxel representation fundamentally differ in both theoretical and application aspects. Theoretically, the neural network is a continuous representation. Advantages, for instance, when it comes to differentiable analysis, such as taking derivatives, the discontinuity is no longer an obstacle. For applications, when you can encode the shape into a neural network rather than in a discrete manner, it would be suitable for some downstream tasks such as dynamic object modeling. Imaging each time step you need a different voxel stored in discrete cases, for a neural network, you just need to give input $(x,y,z,t)$ to the network. That would be our future work. Let lone neural implicits representation allows easy change of the reconstructed mesh resolution as we pointed out in the paper.
  • We can certainly add a pipeline figure. We will update our PDF. For section 3.4, since we proposed a sampling method, our work can be used to improve the neural-pull/SIREN without changing the fundamental input, architectures, and loss function. It works as follows: given a set of depths, initialize the voxel grid, and extract the point cloud directly from the voxel grid (this part is to solve non-available point cloud problem in real applications). If the point cloud is too sparse, a method like neural-pull will fail. Since neural-pull works as given point cloud $\{x_i\}_i$, it samples around the input points $\{\hat{x}_i\}_i$, then it runs the nearest neighbor search to pair $\hat{x}_i$ with a point in the point cloud $x_i$, then the network learns to pull $\hat{x}_i$ to $x_i$. But if the input point cloud is too sparse, the nearest neighbor search fails. That is why neural-pull results in Fig 7 are bad. Using equation (4), one can skip the nearest neighbor search by directly localizing the surface point $x_i$ using equations (1) and (4) for $v_i$ and $v_p$. And learn to pull $v_i$ and $v_p$ to $x_i$.
  • Yes. We will modify our manuscript for a clearer expression.
  • We have 3 core part: 1) sec 3.1 integrate curvature into voxel which enables curvature-guided sampling. 2) sec 3.2 interpolation sampling deals with sparse extracted point clouds. 3) 3.3 Uncertainty estimation which enables open surface reconstruction. The overall motivation: 1) chose voxels: as appreciated by reviewer PKLg, depth images can be cheaply available from the depth sensor. The following question will be, why not use depth directly? One can get the point cloud by re-projecting depth using camera poses. The answer is noise and camera poses. Since we will need the camera poses anyway, we can do it jointly when integrating depth into a voxel structure. In the experiment shown in Figure 10, real-world dataset (TUM_rgbd) and Figure 13 (appendix) dataset red_wood, we all estimated camera poses when we computed normals and curvatures. The second reason is noise. As explained in the first point. 2) curvature-guided sampling: The first reason, as demonstrated in Figure 8 and Figure 9 and explained in section 2 (Related works, biased sampling), is that curvature-guided sampling enables faster convergence. Second reason: We extract surface points using equation (1), which is already studied in Yang et al. 2021 (Geometry processing with neural fields), leading to a concentration on high curvature areas. Curvature-guided sampling solves these biases. 3) interpolation sampling: deals with sparse situations and bypasses the nearest neighbor search.

Thank you for the insightful comments and suggestions provided by the reviewers, which we believe have significantly contributed to improving the quality and clarity of our manuscript. In the following sections, we systematically address each reviewer's questions and concerns, providing clarifications and additional information where necessary.

• weakness 1 (Theoretical limitations):
  • Curvature sampling limitation: we assume reviewer means for main curvature $H = k_1 + k_2$, and saddle region $k_1 > 0$, $k_2$ negative. This will cause $H$ to be relatively small. Since we divide $H$ into low, median, and high categories, this situation will be classified into the low $H$ category. It is certainly true that main curvature and Gaussian curvature capture different geometry information. Since we are not using curvature values directly, rather than using them as an index to indicate the geometry feature of the surface, and if the reviewer see Fig 2 and Fig 11, the distribution of main and Gaussian curvature are similar. That means choosing a slightly different threshold to divide the low, median, and high curvature categories when using two different curvatures should have relatively similar sampling results. We will add an ablation example to demonstrate and also add an explanation in the appendix. A more mathematical way of think about it is $G = K_1k_2$, $H^2 \geq 4G$, $H$ is kind of a upper bound of $G$ since $|H| \geq 2\sqrt{|G|}$.
  • noise sensitive: that is one reason we chose voxel as the base structure. Indeed, the curvature is computed per pixel using its neighborhood pixel depth. However, we did not choose to directly use point cloud generated by re-projecting depth to 3D, but rather fuse them into a voxel grid, other than enable easier localization when sampling. Another reason is it reduces the noise by taking a weighted average of both gradient, curvature, and SDF. See equation (18-21). Another step we did to reduce the noise is a median filter is applied for each depth before computing normals and curvatures. We will add it to the paper.
  • Discontinuities in voxel: that is certainly a disadvantage of classical surface representation and motivates us to continue to train a neural network after initializing the voxel. The key point is, that even though it is a discrete representation, we could have a continuous approximation on each point near the voxel center. But the reviewer is totally correct that if the voxel grid is too coarse, there will be more discontinuity on the original grid. That is why we did 64 and 256 voxel resolution experiments to show that under the coarse, discrete case, our sampling can reduce the discontinuity by Taylor expansion (equation 4) to improve the results.
• Inadequate Experiment Results:
  • The reason why we compare to baselines such as IGR and neural-pull is they are the most fundamental methods and our contribution is focused on improving sampling, bringing a bridge between more real applications (use depth, instead of already existing point cloud), and dealing with unbiased sampling situations. And easily swap to other methods as reviewer MUY9 appreciated. But we thank the reviewer for pointing out the related works, we will certainly cite them and for the new paper Neural Kernel Surface Reconstruction we can try to add a comparison against it.
  • instant NGP: We also thought about after instantNGP came out, should we all go for neural rendering? But we still believe incorporating depth information would benefit in different ways. As we focus on improving the sampling and show more possibility of using more information from depth images. A deeper reason is that we think the trained neural implicit representation can enable more applications such as the deformation of the shape, which we plan to do next. For example, the dynamic object would be much easier recorded by a RGB-D cameras.
  • TSDF: indeed we used TSDF, as TSDF means truncated SDF, which is a way to truncate SDF value when updating the voxel distance, but the way we compute distance $d_s$ is different as the pre-computed gradient allows us to project voxel to surface plane rather than compute point-to-point distance. The gradient voxel structure is based one paper gradient-SDF (sommer et al). on 2022 with modified curvature attributes stored as well. According to the paper, the voxel integration performance is comparable with other state-of-the-art methods. More importantly, the new voxel data structure allows us to do Taylor expansion (equation 4) without the nearest neighbor search.

We would like to extend our gratitude for your valuable feedback and expert insights on our paper. We are especially grateful for your positive feedback on the technical aspects of our work. We have carefully reviewed each of your points and are pleased to answer your comments and questions.

• weakness 1 (motivation): Here we briefly explain our motivation for choosing voxel as a base structure and curvature-guided sampling and we will explain technical details in Question section.
  • Voxel structure: as appreciated by reviewer PKLg, depth images can be cheaply available from the depth sensor. The following question will be, why not use depth directly? One can get the point cloud by re-projecting depth using camera poses. The answer is noise and camera poses. Since we will need the camera poses anyway, we can do it jointly when integrating depth into a voxel structure. In the experiment shown in Figure 10, real-world dataset (TUM_rgbd) and Figure 13 (appendix) dataset red_wood, we all estimated camera poses along when we computed normals and curvatures. The second reason is noise. In real applications, acquired depth can be very noisy. Directly re-project them to 3D space will end up in a very noisy point cloud. Even though each point still has computed normal and curvature, these attributes will be less accurate as we fuse it to a voxel and then sample from it. The computation of normal and curvature depends on the quality of depth. Unless one extends the pipeline to do a depth pre-processing/denoise/complication first, normals and curvature only computed from a single depth won't be robust. However, please see the equations (18-21) in the appendix. One can tell that the normal and curvature integrated into voxels are weighted averages. The weight is computed from equation (16), which can potentially average out normals from large error SDF points. The third reason is uncertainty. Without voxel integration, we do not have weights as prior to train the neural network to learn the uncertainty. Please see section equation (5) on how uncertainty values are obtained.
  • curvature sampling: The first reason, as demonstrated in Figure 8 and Figure 9 and explained in section 2 (Related works, biased sampling), curvature guided sampling enables faster convergence. Second reason: We extract surface points using equation (1), which is already studied in Yang et al. 2021 (Geometry processing with neural fields), leading to a concentration on high curvature areas. The third important fact about curvature guided sampling: sampling using (principle) curvature has been done in Nevello et al. 2022 (Exploring differential geometry in neural implicits), but please note that they need ground truth mesh to compute the curvature then use it to do curvature-guided sampling to reconstruct the mesh back again. To the best of our knowledge, we are the first method that computes and incorporates main curvature from depth to facilitate the reconstruction.
• weakness 2 (computation of uncertainty): uncertainty computation in voxel initialization (section 3.1) used equations (14-16) in the appendix (we put them in the appendix because it is a standard procedure for classical voxel-based method). Initiatively, given an already existing point cloud (let's say we already integrated the first $k-1$ depth, now we deal with depth $k$), after transforming points from depth $k$ to world coordinates (equation 12), we now compare current voxel to the incoming points from depth $k$, as we assume points from depth $k$ are on the surface of the object (should have $0$ SDF value), then voxel center $v_i$ are projected to surface plane to compute the distance (equation 14) with the help of normal $g_i$, the uncertainty (weight) $w_i$ is determined by the value of SDF $d_s(v_i)$. Then, the positive $d_s$ means this voxel is seen by this depth without occlusion or out of the visible area. The linear part in equation 16 is a tolerance of certain errors (up to $v_sT$), the 0 $w_i$ means this voxel is not visible/does not have a corresponding point to update the distance value $d_s$. Thus, after integrating all depths, a voxel with high $w_i$ means it has been seen (updated) by more depth images with smaller errors, thus the distance value is more reliable. $0$ uncertainty parts indicated empty area or totally occluded part. That is why we use interpolated uncertainty to train our network to indicate the uncertainty and enable open surface representation by modified Marching cubes. Regarding the occlusion or empty area caused by missing data, we listed in limitations that our work does not cover shape completions.

We thank the reviewer for the insightful feedback and appreciate their recognition of our technical contributions. In response to the weaknesses and queries raised, we provide the following clarifications and commitments:

• weakness 1 (ablation study for four loss parts): We can certainly add the ablation study for the four loss parts. And expected results will be
  • 1. without normal loss $l_\mathcal{N}$ there might be challenges in accurately reconstructing complex geometries due to uncertainties in surface orientation.
  • 2. without uncertainty loss $l_\mathcal{W}$ will cause some artifacts on open areas (similar to Figure 10 IGR/siren results).
  • 3. without Eikonal loss $l_\mathcal{E}$ may result in trivial solutions for the optimized function $f(x,\theta)$ (e.g. solution equation 10), potentially equating to zero across most areas. This loss ensures that $f(x,\theta)$ adheres to an SDF function field.

Another reason that we did not exclude the normal term $l_\mathcal{N}$ is that our pipeline allows us to compute the normals (sec 3.1) with curvature. In our situation, the normal information is always available, unlike IGR or SIREN which need to deal with no normal situation. But we can certainly add the ablations. Please give us two days. We will include it in the appendix.

• Weakness 2 (network architecture): In Section 4 (Evaluation), we mentioned our use of an 8-layer MLP with 256 nodes and ReLU activations. We want to emphasize that our focus was on sampling and interpolation, and our method adopted the IGR network architecture. This choice was made to demonstrate that our method could be seamlessly integrated with other approaches, as appreciated by reviewer MUY9. When combined with other methods, such as Neural-Pull (illustrated in Figure 7), we only modified the sampling part, keeping the Neural-Pull's network architecture unchanged.
• Question 1: we will update our PDF and add the ablation study. Please give us ~2 days.
• Question 2: the architecture of $f(x, \theta)$ aspects as explained before. The mathematical insights of $f(x, \theta)$ are explained in section 3 (Method). Essentially, the function $f(x, \theta)$, represented by a neural network (MLP), takes points location $x$ as input, $\theta$ is the trainable parameter. The output generated by this function is the Signed Distance Field (SDF) value of the point, along with an uncertainty value. This uncertainty metric reflects the reliability of the output SDF value. It is crucial for $f(x, \theta)$ to adhere to the mathematical constraints of an SDF, which is the primary reason for incorporating the Eikonal loss in our framework.