Zero-Level-Set Encoder for Neural Distance Fields

This paper introduces an encoder/decoder neural network designed to predict Signed Distance Functions (SDFs). This network is trained using the Eikonal equation, eliminating the need for ground truth SDF supervision. The pipeline takes a 3D meshe as input and subsequently outputs the SDF value at any specified query point.

This paper proposes to represent 3D geometry using neural networks, using an encoder-decoder architecture. Using this design, it primarily addresses the task of 3D shape reconstruction from meshes. The major technical contribution is the "hybrid" encoder architecture. Given a mesh, the method uses (1) a graph convolutional encoder to extract per-vertex features; (2) a multi-resolution grid structure to accumulate features from the vertices on grid nodes. The authors also leverages the Eikonal loss to learn the neural signed distance field, obviating the need for pre-computing the SDF values in the training data. Although this is also claimed as a major contribution, it has been widely used in the neural shape modeling literature as of 2023. Experiments have validated some of the design choices (such as the interpolation scheme when aggregating the grid features), and have compared to a few recent baselines. In general, the proposed method does show superior performance in terms of local geometric details. However, quantitative results do not consistently surpass certain baselines and some more recent work should have been considered as baselines.

This paper proposes an efficient encoder-decoder architecture to encode 3D shapes as implicit neural signed distance fields. The core idea is to combine graph and voxel-based encoders coupled with an implicit decoder that can be trained using the Eikonal equation enforced on the shape boundary using surface samples without the need for computing signed distance values on the ground truth data. A modified loss function is also presented for handling meshes that are not watertight or have unoriented normals.

First of all we would like to thank all the reviewers for taking the time to read and understand our manuscript, as well as to provide us with valuable feedback. We will use this feedback to make improvements to the paper in the following days. We will make another post once we have updated the paper.

In the following we want to address concerns raised by multiple reviewers, while the responses to individual reviewers will be posted as direct comments.

Motivation

The biggest weakness raised by both Reviewer ysCV and Reviewer Msao, is the lackluster motivation of using a mesh as input to a neural network and computing the signed distance as output.

Reviewer ysCV:

My central and huge concern is about the utility of such a pipeline: it inputs a mesh, outputs its SDF. This function (computing an SDF) can quickly be performed without any learning based technique, using standard geometric computing librairies like IGL or trimesh. Overall, using a neural network for this [...] has no clear motivation.

Mostly: Why use a neural network to replace a traditional pipeline?

Reviewer Msao:

Motivation: First of all, I'm wondering what's the practical application of the proposed method. [...] On the other hand, given a mesh, one can directly compute the (signed) distance function by computing the distance from the query point to the surface. What's the benefit of introducing a neural network?

We believe that this is the main point of miscommunication in our paper. Our original motivation for researching in this direction was to use neural networks for capturing deforming rigid objects (see deforming dragon and armadillo datasets). We wanted to create a utility to assist in fast (but approximate) collision detection and collision handling for solid-object simulations, something for which SDFs have been found to be useful in previous research on the topic:

• Xu and Barbič. “Signed Distance Fields for Polygon Soup Meshes.” 2014
• Koschier et al. “An Hp-Adaptive Discretization Algorithm for Signed Distance Field Generation.” 2017
• Macklin et al. “Local Optimization for Robust Signed Distance Field Collision.” 2020

SDFs are very attractive for determining collisions of solid objects, as an explicit map can be computed to a desired resolution ahead of time and queried much faster, than computing explicit triangle-point distances. If the object is deforming however, SDFs are not typically used, as this map needs to be updated very frequently, which can become quite costly for interactive applications, where a single time-step is only allowed to take ~40ms including physical solvers. This is where the idea for this work originated, i.e. would it be possible to use neural networks to accelerate updating approximate SDF maps and to enable fast SDF queries for deformable objects? While computing distances from a mesh is a solved problem, it needs to scale appropriately to be usable in the context of solid-object simulations. We believe that a neural network with an efficient encoder and an "as-small-as-possible" decoder could take good strides in that direction. This is also why we discarded auto-decoder approaches as untenable, since even a single optimization loop during inference already breaks computational constraints.

The previous paragraph should simply provide context to the original motivation of our work. In the process, we found that our architecture also worked incredibly well on other datasets and the general task of encoding 3D geometry into neural fields. In the process of shifting focus, we have simply not re-evaluated the idea of using meshes. As such, we truly appreciated the following suggestion of Reviewer ysCV to train our network on point-clouds instead, as this makes for an even more generally usable architecture.

How does the proposed method perform in scenarios with noisy or incomplete data (pointclouds instead of meshes)?

We acknowledge that for the way our paper was written and considering the goals of compared related works, that it may have seemed a bit nonsensical to compute the SDF from a mesh, which can be obtained reasonably efficiently from the mesh data. Therefore, we have taken the suggestion into account and are rerunning all results using only point-data for our method as well. In other words, we completely discard the mesh connectivity and only use it to sample additional points for training. As mentioned in the method section of the paper, we simply construct a k-nearest neighbor graph and use this for graph convolutions instead of the mesh connectivity. Similar to related works, we now reconstruct SDFs directly from point data. We are seeing promising initial results using this pipeline and will update the paper once we obtain a better picture.

By using point-clouds instead of surface meshes, we are able to present a more general framework. One can interpret using surface meshes, as was done originally, as a specialization of this framework. In this way, we show both a fairer comparison to other baselines, i.e. all methods now use only point data, while also expanding the usability of our method.

Ablation

Another point that Reviewer ruBo and Reviewer Msao seemed to agree on, is that the contribution of the novel hybrid encoder architecture was impacted by not providing more thorough ablation, or rather more relevant cases in the ablation study.

Reviewer Msao

While the hybrid encoder is interesting, the whole pipeline is more of a straightforward combination of standard modules (as of 2023) [...]. While admittedly this shouldn't be a major weakness per se, it is crucial to have more thorough ablation experiments to validate the intuitive combination. [...] To me, such experiments are critical in validating the technical contributions, but are missing.

Reviewer ruBo

How important are the individual graph and voxel components of the proposed encoder network, and the encoder network itself as a whole? An ablation study would be helpful to understand this contribution better.

The motivation for doing the ablation study in the way it was originally presented, was to keep the network complexity, i.e. number of parameters, identical across all runs. This would highlight the importance of specific architectural design decisions, such as interpolating grid features back to the input vertices. Otherwise, it would be difficult to tell if an improvement came from having an increased number of network parameters available to train, or from the actual way of interconnecting nodes.

Nevertheless, as there seems to be genuine interest in the investigation of architectural variations, we are currently running an updated ablation study. We will update the paper with the results once the runs finish. We may opt to move the previous ablation into the appendix to preserve it for reference. To the reviewers further credit, this seems to indeed yield some very interesting insights into the effect when using only the voxel and graph components, with or without interpolation.

Related Work

Finally, all Reviewers ruBo, Msao and ysCV pointed out relevant, but missing, related works. Since these works had a lot of overlap between reviewers, we would like to address them jointly here.

Reviewer ysCV

Finally, if the point is about demonstrating that an SDF can be learned without explicit supervision, only by solving the Eikonal equation: this has already been demonstrated in SAL and SAL++ (Atzmon et al., these references are missing)

Reviewer Msao

The Eikonal loss is considered as a major technical novelty, but it has been proposed for learning 3D shapes [...] This hurts a major technical contribution of this paper. At least the IGR paper by Gropp et al. should be cited and discussed.

Reviewer ruBo

How does the proposed method compare against the missing related work listed above in the Weaknesses section? While not all of these works are encoder-decoder models, a comparison is necessary since they too share the advantage of the proposed method of not requiring ground truth SDF values at samples.

• SAL: Sign Agnostic Learning of Shapes from Raw Data (CVPR 2020)
• Implicit Geometric Regularization for Learning Shapes (ICML 2020)
• SALD: Sign Agnostic Learning With Derivatives (ICLR 2021)

Architectures

We thank the reviewers for making us aware of these works, we will update the paper in the following days, such that we discuss these papers appropriately, as they also solve a very similar problem. Indeed, the formulations of the loss in IGR and SALD have similarities with our formulation and we were very curios to read about the solution to flipping signs in the SAL paper. It would also be interesting for future work, whether the specialized weight initialization scheme could be combined with the one presented for SIREN networks.

Nevertheless, none of the presented architectures are suitable for direct comparison with our method, as all of them rely on a similar auto-decoder structure as was initially presented in DeepSDF [Park et al. 2019]. An exception is the variational encoder-decoder presented in SAL and SALD, which is however also (optionally) augmented by an optimization loop to improve the results. Since the encoder of SAL is claimed to be PointNet (Qi et al. 2017), the results will likely be very similar to Occupacy Networks (which we do compare with). In fact, the authors of SAL and SALD explicitly state that their encoder architecture is similar to the one used by Mescheder et al. 2019 for Occupancy Networks. That is as long as the optimization loop is not employed.

As one of our main goals was to have an efficient architecture for reconstructing and querying SDFs, we ruled out comparing with auto-decoder methods, as they rely on optimizing latent codes during inference. As such, they are typically able to reconstruct high fidelity SDFs at the upfront cost of computation time as well as computationally expensive SDF queries. To put things into perspective, the named papers all need to evaluate an 8-layer MLP decoder with 256 to 512 hidden units to compute a single signed distance. In contrast, we need to perform one tri-linear interpolation and evaluate 3 layers with 64 hidden units per SDF query, once the latent grid has been computed. While the computation of our latent grid is not free, it is typically orders of magnitude faster than querying all SDF values for surface reconstruction.

In practice, auto-decoders often also have reduced test sets, as evaluation on new inputs is relatively expensive compared to encoder-decoder approaches. This is also corroborated by e.g. the suggested related work of Reviewer Msao, POCO [Boulch and Marlet 2022].

Eikonal loss

Regarding the Eikonal loss for learning SDFs in an unsupervised manner we would like to clarify the following. While previous works demonstrated the ability to learn an SDF without supervision by using the Eikonal equation, it has yet to be demonstrated in a pure forward-only encoder-decoder setting to high fidelity. Other works, including IGR by Gropp et al. to the best of our knowledge, make use of latent code optimization at test time, while our network directly computes valid latent codes in a single forward pass. This means our method is inherently more efficient in adapting to new shapes, as we forego the costly optimization loop during inference. If nothing else, our method takes a step in the direction of making the Eikonal loss more usable as a training target, and neural distance fields as a whole more usable for practical and performance critical applications.

We have now uploaded a rebuttal revision of our paper as well as an updated video as supplemental material. We have made the following changes, to address all major concerns about the paper:

• re-ran all experiments of our model using only point-cloud inputs, and updated all figures and metrics. This addresses major concerns about the motivation and usefulness of our work and the fairness of comparisons to other baselines. We also adjusted the text to be consistent with the new experiments and pipeline.
• cited the requested related work and discussed it in an additional brief paragraph.
• re-ran the ablation study to cover more interesting variations of our architecture. We have also rendered the results of different variations in the appendix for visual comparison. If interest remains in the original ablation study, we would be willing to add this to the appendix for the camera ready version.
• improved the selection of comparison metrics for evaluation on point clouds.

We hope that these changes find the approval of the reviewers and we stand by for any further questions or comments.