Implicit Neural Surface Deformation with Explicit Velocity Fields

Thank you very much for the positive feedback!!! We sincerely thank the reviewer for the thoughtful and encouraging feedback on our work. We are delighted to hear that the reviewer appreciated our methodology and the significance of our contributions. Regarding the weaknesses and questions raised by the reviewer, we address them as follows:

Correspondences: It is indeed the primary limitation of our work. However, as demonstrated in section 4.3, Fig.11, our pipeline can integrate with a mesh-matching approach to obtain the required correspondences. Compared to using mesh shape matching combined with interpolation, such as the state-of-the-art method in [1], our approach (as shown in Appendix Section A10) offers the ability to interpolate incomplete shapes and handle topologically inconsistent shapes. Additionally, we have updated the appendix with experiments where correspondences are obtained from [2] (as [1] cannot handle partial shapes). These updates include error maps of the correspondences, showcasing how our method can integrate with existing approaches to address the correspondence challenge effectively.

[1] SMS: Spectral Meets Spatial: Harmonising 3D Shape Matching and Interpolation (CVPR 2024). Cao, Dongliang and Eisenberger, Marvin and El Amrani, Nafie and Cremers, Daniel and Bernard, Florian

[2] Unsupervised learning of robust spectral shape matching (SIGGRAPH 2023) Cao, Dongliang and Roetzer, Paul and Bernard, Florian

Readability of Fig. 2 and Fig. 7: We thank the reviewer for the suggestion, we will include a larger version in the Appendix. Additionally, we will re-render Fig. 7 to improve its clarity.

Correspondence distribution/selection: The correspondences in our method are selected uniformly, which means they more or less cover the entire shape under the current settings. To further explore different scenarios, we added two new examples:

1. Partial Shape Interpolation: In this case, only a partial shape of the target and partial correspondences are available. In the updated manuscript Fig. 27, we present examples of partial shapes along with the correspondence error. The correspondences were obtained using [2], and our method successfully handles these cases.

2. No Correspondences: We also examined a scenario where no correspondences are provided. Even in the absence of correspondences, our method recovers reasonable deformations. These results are shown in the updated manuscript Fig. 26.

That said, we acknowledge the limitations of these cases. For partial shapes, correspondences are available in regions where the shape deforms, while for the no-correspondence case, we only tested small deformations. Based on our observations, our method remains robust under relatively small deformations when correspondences are nested. Moreover, it appears that correspondences are only necessary in regions where deformation occurs.

We sincerely appreciate the reviewer's positive feedback.

• Regarding the "Line segment effect", we have added a new figure (Fig.30) in our supplementary material (Please download supplementary material zip file and check 4738_SuppMat.pdf). We visualized the trajectory of a linear interpolated movement (first row) and the point trajectory from our velocity field (second row). It is evident that our velocity field generates a curved movement that preserves the length of the leg, whereas linear interpolation shortens the leg length to achieve linear motion.

• Regarding the Large deformation: from the results of Fig.30, we do not think that our method produces a velocity net that prefers linear movements. Regarding the very large deformation, we believe that under our current setting, the problem is hard to fully constrain. We would like to point out that our method is different from a template or skeleton-based method that can deal with very large and different movements, such as [1, 2]. These methods focus on the articulation of the human movement and have much stronger assumptions than our method. We proposed a more general framework to deform the implicit field which makes the deformation physically plausible. However, as shown in Fig. 22, our approach successfully handles relatively large deformations, even in cases with topological inconsistencies.

[1] Human Motion Diffusion as a Generative Prior (ICLR 2024) Yonatan Shafir and Guy Tevet and Roy Kapon and Amit H. Bermano

[2] Learning dynamic relationships for 3d human motion prediction (CVPR 2020) Cui, Qiongjie, Huaijiang Sun, and Fei Yang.

We sincerely thank the reviewer once again for their insightful feedback. We believe our work has the potential to inspire further research in this direction. Your positive review is both encouraging and greatly appreciated.

We would like to express our sincere gratitude to the reviewer for the insightful feedback on our manuscript. We greatly appreciate the positive remark about our method and the further experiment suggestions, which will enhance the robustness and comprehensiveness of our study. We included the additional results and updated our manuscript. In response to the weaknesses and questions raised by the reviewers, we address them as follows:

Unsupervised: We agree with the reviewer and we apologize for the misleading word “unsupervised”, which means only no supervision on intermediate shapes. We updated the PDF to clean up the misunderstanding.

Correspondences: It is indeed the primary limitation of our work. However, as demonstrated in section 4.3, Fig.11, our pipeline can integrate with a mesh-matching approach to obtain the required correspondences. Compared to using mesh shape matching combined with interpolation, such as the state-of-the-art method in [1], our approach (as shown in appendix Section A10) offers the ability to interpolate incomplete shapes and handle topologically inconsistent shapes. Additionally, we have updated the appendix with experiments where correspondences are obtained from [2] (as [1] cannot handle partial shapes). These updates include error maps of the correspondences, showcasing how our method can integrate with existing approaches to address the correspondence challenge effectively.

[1] SMS: Spectral Meets Spatial: Harmonising 3D Shape Matching and Interpolation (CVPR 2024). Cao, Dongliang and Eisenberger, Marvin and El Amrani, Nafie and Cremers, Daniel and Bernard, Florian

[2] Unsupervised learning of robust spectral shape matching (SIGGRAPH 2023) Cao, Dongliang and Roetzer, Paul and Bernard, Florian

Larger deformation: It is true that our method occasionally struggles with very large deformations. However, as shown in Fig. 22, our approach successfully handles relatively large deformations, even in cases with topological inconsistencies. We would like to point out that our method is different from a template or skeleton-based method that can deal with very large and different movements, such as [1, 2]. These methods focus on the articulation of the human movement. We proposed a more general framework to deform the implicit field which makes the deformation physically plausible. Furthermore, we have included additional experiments in the appendix to demonstrate the capability of our method in addressing extreme partial shape deformation scenarios.

Despite its current limitations, we believe our method represents a strong starting point for direct implicit surface deformation using point cloud inputs and hope it will inspire more follow-up works.

[1] Human Motion Diffusion as a Generative Prior (ICLR 2024) Yonatan Shafir and Guy Tevet and Roy Kapon and Amit H. Bermano

[2] Learning dynamic relationships for 3d human motion prediction (CVPR 2020) Cui, Qiongjie, Huaijiang Sun, and Fei Yang.

Volume Preserving: This is an insightful observation by the reviewer. Indeed, we set the weight for the divergence-free loss to zero when performing cross-category interpolation, as mentioned in Lines 352-353 of the manuscript. We emphasize that we enforce volume preserving as a loss, not by constructing the velocity field with a divergence-free structure, such as [3], which makes it less flexible to handle non-volume preserving cases.

[3] Divergence-Free Shape Interpolation and Correspondence (CGF 2019). Marvin Eisenberger and Zorah Lähner and Daniel Cremers

Broken input: Our implicit neural network is initialized as a sphere, which naturally tends to close gaps or holes on surfaces. As a result, small broken features, such as incomplete legs, are likely to be reconstructed as whole, continuous structures. However, if the broken parts are too far away from each other, our method might struggle with it. We will include this in our paper’s discussion section. Instead of this situation, we demonstrate a different form of shape completion: our method’s ability to handle cases where the target shape is only partial. For details, please refer to the updated manuscript Fig. 27 in the appendix.

Visualization velocity field: We thank the reviewer for the thoughtful suggestion. In response, we have added visualizations of the new experiments, including the velocity fields and the deformed point clouds. Please refer to the updated manuscript Appendix Fig. 24-27 for details.

Large Boundaries: We thank the reviewer for pointing this out. SDF indeed has its limitations, and we will include this discussion in the limitations section of our work.

We would like to thank the reviewers for the thorough and insightful feedback on our manuscript. We are especially appreciative of the positive comments. Additionally, we are grateful for the suggestions. Regarding the weaknesses and questions raised by the reviewer, we address them as follows:

Line segment in Fig. 4: We apologize for any confusion. To clarify, our method does not rely on any pre-defined (or linear) path when deforming shapes. Instead, we work directly with a source point cloud and a target point cloud. As shown in Fig. 4, our meshes at $t = 0$ and $t = 1$ align well with the respective point clouds, illustrating the effectiveness of our approach. In case we misunderstood the reviewer’s concern, we kindly ask to clarify what is meant by “line segment”, so that we can address the concern more thoroughly.

No complex deformation: We believe our experiments encompass a diverse range of scenarios, including non-isometric deformations and incomplete shape deformations. Non-isometric deformations in our context involve not only gesture changes but also category changes. If the reviewer is referring to examples with larger movements, we kindly direct their attention to Fig. 22 in the appendix. Additionally, we have included more examples of larger deformations to further demonstrate the robustness and versatility of our method in newly added experiments Fig. 25-27.

Efficiency Analysis: We apologize for not emphasizing the efficiency aspect more clearly. However, we emphasize that mention the running time in Section 4.1 (Line 339-342 in the original manuscript and Line 342-347 in the updated version). As stated in the paper, NISE requires approximately 1.5 hours per deformation step, excluding the pre-training time for the SDF networks of the source and target meshes. while NFGP takes around 15 hours per step (over 75 hours for five steps). In contrast, our method is significantly more efficient, requiring only 1/5 of the time compared to NISE, which takes an additional 20 minutes. To provide further clarity, we have added a section in the appendix detailing the training processes for both the comparison methods and our approach.

Ablation of Network Size: We have not done any network size ablation because our main contribution is not focusing on network architectures and we just use two simple MLP with 8 layers, as mentioned in section 4. Line 286-287. However, we will add network size ablation with a small node size and fewer layers. We will update the results in the following days.

Use UDF instead of SDF: The short answer is: yes, it can be used for non-watertight surfaces. The level set equation operates on level-sets, meaning it can be generalized to UDF. However, from a mathematical perspective, SDF captures more intrinsic properties of the surface. If the goal is to represent an open surface, we sincerely recommend considering an uncertainty-aware SDF approach [1]. In this method, the open surface is treated as a pseudo-closed surface, where the open regions are assigned a weight of 0. During the marching cubes process, surfaces are generated only for regions with weights greater than 0. Alternatively, a hybrid representation [3] could also be considered. From a practical standpoint and based on our limited experience with UDF, it often struggles with identifying zero-crossings. While some works [2] propose methods to address this issue, they require learning to fit the UDF locally to an SDF, which can add complexity.

[1] Enhancing Surface Neural Implicits with Curvature-Guided Sampling and Uncertainty-Augmented Representations, (ECCVW 2024) L. Sang and A. Saroha and M. Gao and D. Cremers

[2] Neural Surface Detection for Unsigned Distance Fields (ECCV 2024) Federico Stella and Nicolas Talabot and Hieu Le and Pascal Fua

[3] HSDF: Hybrid Sign and Distance Field for Modeling Surfaces with Arbitrary Topologies (Neurips 2022) Wang, Li and Yang, Jie and Chen, Weikai and Meng, Xiaoxu and Yang, Bo and Li, Jintao and Gao, Lin

Equation.4: Yes, the reviewer is correct. In our setup, we not only fix the starting point but also fix the ending point. However, since we were introducing the velocity ODE equation and aimed to maintain the form of the Cauchy initial problem for the ODE [1], we specified only the initial condition in Eq. (4). Based on the reviewer’s feedback, we have revised the corresponding text and added the condition \phi(x,1) = x^1 to clarify this point.

[1] The Cauchy Problem for Ordinary Differential Equations. In: Parametric Continuation and Optimal Parametrization in Applied Mathematics and Mechanics. (Springer 2003). Shalashilin, V.I., Kuznetsov, E.B.

We now understand what the reviewer means by “line segment.” The reviewer is correct that we estimate the trajectory of each point on the shape through our velocity field. However, we do not impose any prior assumption, such as requiring points to move along straight lines. Instead, we constrain the movement using spatial smoothness and volume-preserving terms to ensure the motion remains physically plausible.

We believe the “line segment effect” observed may be due to the limitations of static visualizations. To address this, we have included two videos (associated with Fig. 24, and Fig.3) to illustrate the movement better and highlight our method's results. Please download the Supplementary Material zip file. Additionally, Fig. 27 was produced using the same loss function as Fig. 3, and it demonstrates that our method does not produce linear movement but instead achieves physically plausible trajectories (see the stretching of the legs and tails of Fig.27)

We hope the added video clarifies the behavior of our method.

Update: We also uploaded the network size ablation in the supplementary material zip file, please download the zip file to see the new added experiment results.

We thank the reviewer for their insightful questions and positive feedback. We particularly thank the reviewer for suggesting better motivation reseasoning, an experiment with no correspondences, and a better explanation about OLSE vs. MLSE. Regarding the weaknesses and questions raised by the reviewer, we address them as follows:
Motivation:

Why are physically plausible Intermediate shapes necessary:

We answer this question in two aspects:

1. why do we need to interpolate intermediate shapes? The real world is continuous. However, in most cases, we only have discrete observations. To recover the continuous movements and deformation that happen in the real world, it is useful and important to be able to interpolate between discrete observations. For example, some human movements are recorded by 1 FPS (frames per second) sequence and we would like to have at least 30 FPS, which means we need to interpolate 30 meshes in between.
2. why do we emphasize physically plausible shapes? The interpolation should reflect the physical properties of the discrete observations. That is, it should carry out the information that was “hidden” between two given inputs. Thus, we emphasize the concept of “physically plausible” during our task to better infer real-world movement or deformation. We show that our work indeed recovers movement close to what happens in our physical world by conducting additional experiments on a real-world dataset of human movement recordings, and we compare our results to the ground truth intermediate shapes. Please see the newly added Fig. 24 with the error table Tab.1.

Why it is reasonable to assume that correspondences are available: It is true that finding correspondences is still an open problem, however, we believe it is reasonable to assume accessible correspondences for two main reasons: First, mesh-based shape-matching methods are well-studied, as illustrated in Fig. 11 of the main paper, and our method seamlessly integrates correspondences derived from these established approaches. Second, fitting a template to represent underlying geometry is standard practice when handling complex real-world data, as seen in datasets like 4D-Dress [2] and BeHave [3]. These datasets employ the SMPL human model, which is fitted to the data and provides natural correspondences. In such cases, our method can be directly applied.

To further support our claim, we present additional challenging results in the appendix using correspondences obtained from [1], highlighting the robustness of our method (See Fig.27). Additionally, we include experiments on [2] to demonstrate the applicability of our approach to these datasets (See Fig.24 and Fig 25).

[1] Unsupervised learning of robust spectral shape matching (SIGGRAPH 2023) Cao, Dongliang and Roetzer, Paul and Florian, Bernard

[2] 4d-dress: A 4d dataset of real-world human clothing with semantic annotations (CVPR 2024) Wenbo Wang and Hsuan-I Ho and Chen Guo and Boxiang Rong and Artur Grigorev and Jie Song and Juan Jose Zarate and Otmar Hilliges.

[3] Behave: Dataset and method for tracking human object interactions. (CVPR 2022) Bharat Lal Bhatnagar and Xianghui Xie and Ilya Petrov and Cristian Sminchisescu and Christian Theobalt and and Gerard Pons-Moll.

No correspondences and correspondences error:

We thank the reviewer for the thoughtful suggestions for additional experiments to clarify our method. In response, we have added two experiments: (1) no correspondences are given (see the updated manuscript Fig. 26), and (2) only partial correspondences with partial target shape are given (see the updated manuscript Fig. 27).

In the absence of correspondences, our method still recovers reasonable deformations. We attribute this performance to our joint-training design, which incorporates physical constraints such as volume preservation and smoothness, as well as geometric properties like normals in the implicit field. Although the examples focus on cases with small deformations. We appreciate the reviewer for highlighting this avenue for further exploration.

Correspondences on comparison method:

1. NISE: While it is possible to incorporate correspondences in NISE, this approach requires dense (one-to-one) correspondences. Moreover, NISE performs only linear interpolation between the Euclidean positions of the correspondences. As a result, it is likely to fail in scenarios involving bending, articulation, or non-isometric deformations.
2. NFGP: Training NFGP also relies on correspondences but in a different format. The process begins by training an SDF network to fit the implicit field of the input shapes. Then, a set of handle points is defined, along with their corresponding rotation and translation parameters, to transform these points into target positions. This requires the user to manually divide the shape into several parts and assign separate rotation and translation parameters for each part.

We agree with the reviewer that additional experiments can better showcase the strengths of our method. In response, we have conducted and updated the following experiments, which, while slightly different from the specific request, still address the core concerns and highlight our method’s robustness. Please download the supplementary material zip file to check them.

1. GT correspondences.
2. No correspondences.
3. Quantitative (Tab.2) and qualitative (Fig.28) noisy correspondences ablation. The error of correspondences is introduced by swapping vertex indices.

We have not used correspondences obtained from mesh shape-matching methods for the following reasons:

1. Time Constraints and Proper Training Challenges: The discussion period is coming to an end, there are no pre-trained models available for datasets where intermediate shapes serve as ground truth for quantitative evaluation. Datasets which is suitable for this task, such as 4D-Dress, is not a standard shape-matching dataset, and no training protocols are provided by existing mesh-matching methods. To ensure fair and accurate evaluations, we chose to avoid potentially improper training within such a limited time.
2. Comparable Error Simulation: Our error simulation is comparable to results obtained from shape-matching methods. Typically, in shape-matching algorithms, when aligning shape $\mathbf{P}_1$ to shape $\mathbf{P}_0$, the algorithm provides a re-ordered set of vertices for $\mathbf{P}_1$, where vertices with the same index are treated as corresponding pairs. Inaccurate correspondences occur when this order is misaligned, resulting in swapped vertex indices. By manually swapping vertices and controlling the percentage of swaps, we effectively simulate noisy correspondences that are typical in real-world shape-matching scenarios.

We present visual results in Fig. 28, along with error maps to analyze the noisy correspondences. Additionally, we include the correspondences used in Fig. 11, which were obtained through a shape-matching method [1]. In Fig. 11, the error map’s error bars are based on geodesic distances. In contrast, for our simulated noisy correspondences, the error bars are based on Euclidean distances. This distinction arises because geodesic distances cannot be computed without a mesh structure. Typically, smaller Euclidean distances correspond to larger geodesic distances, as points must travel along the surface of the mesh.

By comparing the error maps, we demonstrate that our noisy correspondences not only effectively simulate errors inherent in shape-matching methods but also encompass a broad range of possible error scenarios.

Due to size limitations, we have included the table and figure in the supplementary materials. Please download the provided zip file to review the results, and refer to Section S10 for detailed explanations.

Additionally, we would like to emphasize the following points regarding our method and its assumptions about known correspondences:

1. Robustness to Correspondence Errors: Our method demonstrates resilience to inaccuracies in correspondences. (See Section S10, S4.2)
2. Robustness to Partial Correspondences: Our method remains effective when using partial correspondences or correspondences generated by previous methods. (See Fig. 11 and Fig 27 in supplementary material)
3. Practical Relevance: In real-world applications, correspondences are often available (e.g., datasets such as 4D-Dress and BeHave). (See Fig. 24 Fig. 25 in supplementary material)

Therefore, from both a theoretical and practical perspective, we believe that our assumption of known correspondences does not diminish the significance of our contributions.

[1] SMS: Spectral Meets Spatial: Harmonising 3D Shape Matching and Interpolation (CVPR 2024). Cao, Dongliang and Eisenberger, Marvin and El Amrani, Nafie and Cremers, Daniel and Bernard, Florian