GenCorres: Consistent Shape Matching via Coupled Implicit-Explicit Shape Generative Models

Thank you for your valuable comments and suggestions. Below are our clarifications for your concerns. We value your input and would love to hear any follow-up you have to the response.

Q: Confusing notation.

In the original submission, the implicit generator uses lowercase letters $g^{\phi}$. The explicit generator uses bold letters $\mathbf{g}^{\theta}$. In the revised manuscript, we have made changes to the notations to make them clearer:

• $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters.

• $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0).

• $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$.

• $h^{\psi}$ denotes the encoder in the VAE.

• $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage.

We have also provided the explanations in the revised main paper.

Q: Data is generated synthetically and does not deal with the real challenges (e.g., partiality, noise, clutter, …)

We have included additional experiments in the supplementary material, submitted during the rebuttal period, to demonstrate that our method is also applicable to real scans. We trained the implicit network on a real human dataset featuring various clothes and hairstyles. The experiments indicate that our method can still provide smooth interpolation even when the learned space represents a more diverse range of shapes. Please refer to the newly submitted supplementary material for more details. Additionally, we want to emphasize that the Human dataset we evaluated, which comes from SMPL, is sufficiently challenge for inter-shape matching.

Q: The topological constraint given by the template limits the generality of the method.

The template is introduced for the explicit mesh generator. In fact, any mesh generator in the literature requires a template. The template is not used in the implicit generator, which is a key contribution of this paper. The implicit generator can already provide pair-wise correspondences between pairs of implicit surfaces, which are not constrained by topology. Specifically, given any two shapes, we can perform latent space interpolation between them and propagate the correspondences along the interpolation path using equation (7) in the main paper. This framework is applicable to any shape collection, regardless of its topology. These correspondences can be evaluated through the interpolation results shown in the paper; smoother and more rigid interpolations indicate better correspondences. In this paper, we utilize the template solely for the task of joint shape matching of deformable shapes, as they typically share similar topology and structure.

Q: How would linear interpolation behave in the presence of significant diverse geometrical topology data? Is template triangulation the main limitation in this case?

In this paper, we focus on deformable shapes (humans and animals), which usually do not have significant topological changes. For other types of shape collections with significantly diverse geometrical topologies, we leave this for future research. Note that we perform linear interpolation for the implicit surfaces, where template triangulation is not involved.

Q: Would it be possible to test the method on a class for which a dense correspondence is not provided; for example, some classes of ShapeNet (e.g., cars, airplanes)?

We propose a general framework for computing correspondences between implicit surfaces. We are encouraged by your recognition of our framework's potential applicability to a wide range of shape collections, including man-made shapes. For man-made shapes, we can adopt alternative energies, such as as-affine-as-possible (AAAP) energy, which enforces local region transformations to be affine rather than rigid, in place of ACAP energy. However, we want to underscore that deformations among man-made shapes are more complex than deformable shape collections. Also there are structural variations, where AAAP should be enforced among shapes or parts that are structurally similar. Those issues should be addressed by other papers (we are currently looking into this).

Q: Computational cost of the method compared with NeuroMorph.

Similar to SALD, we train our model with 8 NVIDIA TITAN V GPUs (12G). For the Human collections (about 1000 shapes), training the implicit generator takes about 3 days, while the explicit generator requires approximately 12 hours. For the FAUST dataset (80 shapes), NeuroMorph is trained with 1 GPU for roughly 3 days. Note that NeuroMorph requires a pair of shapes as input, so the training complexity is $O(n^2)$ for $n$ training shapes. In contrast, our method directly learns the shape space, meaning the complexity scales linearly with the number of training shapes.

Thank you for your valuable comments and suggestions. Below are our clarifications for your concerns. We value your input and are keen to discuss any further questions or comments you may have following our response.

Q: Math notations and what are the different fonts for g?

In the original submission, the implicit generator uses lowercase letters because the signed distance is a scalar. The explicit generator uses bold letters because the output vertices are vectors. They can also be distinguished by the network parameters: $\phi$ for the implicit generator and $\theta$ for the explicit generator. We used g because both of them are generative models. In the revised manuscript, we have made changes to the notations to make them clearer:

• $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters. Yes, the reason for $\mathbb{R}^3 \times \mathcal{Z}$ is that it computes the signed distance value at the specific 3D coordinate for the shape defined by a latent code.

• $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0). $\mathbf{x}$ is the 3D query location and $\mathbf{z}$ is the latent code.

• $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$.

• $h^{\psi}$ denotes the encoder in the VAE, where $\psi$ are the network parameters.

• $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage, where $\theta$ are the network parameters.

We have also provided these explanations in the revised main paper.

Q: What does d stand for in equation 3, some displacement between a vertex and an infinitesimally close corresponding vertex?

Yes, $\mathbf{d}_i^{\mathbf{v}}(\mathbf{z})$ in equation (3) denotes the displacement between a vertex and an infinitesimally close corresponding vertex. In the original submission, we further multiplied it by $\epsilon$ because this simplified equation (6), (7) and (8) by avoiding the constant $\epsilon$. However, you are correct that this is not necessary. In the revised submission, we have followed your suggestions and did not scale it by $\epsilon$. The final expressions of the regularization terms remain the same.

Q: Explanation of capital letter symbols (C, F, G, E).

We provide detailed expressions of $C$ and $F$ in the appendices of the main paper. $C$ and $F$ are derived from the matrix representation of Equation (3).

The expression of $G$ is given in equation (7). It is just a component of $\mathbf{d}_i^{\mathbf{v}}(\mathbf{z})$ that is independent of the perturbation direction $\mathbf{v}$.

$E$ is derived in equation (8) and serves as a metric to measure the local rigidity and local conformality distortions between an implicit surface and its perturbation. The expression of $E$ can also be found in equation (8) and, like $G$, is independent of the perturbation direction $\mathbf{v}$.

Q: Results using 3D body scans with various clothes, hairstyles, etc.

We have included additional qualitative results in the supplementary material submitted during the rebuttal period. We trained the implicit network on a new human dataset featuring various clothes and hairstyles. Our proposed regularizations again consistently improve the baseline network, SALD. Please refer to the newly submitted supplementary material for more details.

Q: Interpolation between a lion and a horse.

We have included more qualitative results in the supplementary material submitted during the rebuttal period. We trained the implicit network on a new quadruped shape dataset. Visualization of the interpolation results show that our method transitions smoothly between a lion and a horse. Please refer to the newly submitted supplementary material for more details.

Q: Citation, acronyms and typos.

We have added the new ARAP reference and corrected the acronyms and typos in the title.

Thank you for your insightful and valuable feedback. Below are our clarifications for your concerns.

Q: The need of a relatively large set of input shapes.

Our approach follows the stream of approaches that leverage big data (e.g. nearest neighbors and graph-based learning) to solve traditional problems. The goal is to open a new direction that has many problems to work on. For example, the approach we developed to train the implicit shape generators that preserve inter-shape deformations can be extended in another setting. As another example, training an explicit shape generator to solve the inter-shape correspondence problem can stimulate future research in this area.

Q: How well does GenCorres perform with man-made shapes or other shapes that do not correspond to articulated objects?

We propose a general framework for computing correspondences between implicit surfaces. Specifically, we analyze the changes of the generated shape under the infinitesimal perturbation of the latent code and formulate the correspondence computation as a linearly constrained quadratic program problem (refer to Equation (6) in the main paper). We are encouraged by your recognition of our framework's potential applicability to a wide range of shape collections, including man-made shapes. For man-made shapes, the key question pertains to identifying a suitable deformation model, as inter shape deformations among man-made shapes are notably complex. One possibility is that instead of using ACAP energy, we can adopt alternative energies, such as as-affine-as-possible (AAAP) energy, which enforces local region transformation to be affine rather than rigid. Also, using L1 norm rather than the L2 norm in this setting maybe more appropriate. Another challenge is to address structural variations and only enforce AAAP among structurally similar shapes. All these questions deserve other papers to study (we are looking into this as well).

Q: What is the influence of the hyperparameters (lambda and epsilon) in the final result? Are the values specified in the paper valid for other sets of input shapes?

We did not perform an extensive grid search for the optimal hyperparameters. However, we found our methods to be quite robust against variations in $\lambda_{geo}$, $\lambda_{cyc}$ and $\epsilon$. From our experiments, when $\lambda_{geo}$ and $\lambda_{cyc}$ are set very large, the data term increases, leading to an over-smoothing of the learned shape due to the regularization term enforcing greater similarity between shapes. We set $\epsilon$ to a small value in finite differences to achieve better gradient approximation. These hyperparameters are valid for both Human and Animal categories.

Q: The limit number for which the method significantly degrades.

For the pair-wise shape matching experiments on FAUST, we conducted ablation studies where we gradually decreased the number of additional DFAUST shapes. The results of these studies are presented in the table below.

Evaluations of pair-wise matching on FAUST (in cm).

Thank you for your insightful and valuable feedback. Below are our clarifications for your concerns.

Q: This can be done both differentiably. (is this where finite differences come in as mentioned in 4.4?)

Yes, both autograd and finite differences are differentiable.

Q: How long does the training take?

Similar to SALD, we train our model with 8 NVIDIA TITAN V GPUs (12G). For the Human collections, the training of the implicit generator takes about 3 days, while the explicit generator requires approximately 12 hours.

Q: Ablation studies to validate the necessity of the implicit approach.

We conducted ablation studies for the joint shape matching experiments on DFAUST. The results are shown in the table below. GenCorres-NoImp does not utilize the implicit approach (1st stage) and directly registers the template to each target shape as initialization. GenCorres-NoInit omits both the 1st and 2nd stages, using only the 3rd stage. The results show that GenCorres is superior to both GenCorres-NoImp and GenCorres-NoInit, thanks to better initialization. Without latent space interpolation to generate intermediate shapes, GenCorres-NoImp often fails to register the template directly to a vastly different target shape, leading to poor initialization. GenCorres-NoInit, which does not employ any correspondence initialization, tends to get stuck in undesired local minima. Note that the success of training 3D-CODED relies on a very large dataset (23000 human meshes with a large variety of realistic poses and body shapes).

Evaluations of JSM on DFAUST using geodesic errors of the predicted correspondences (in cm).

Q: Will codes and preprocessed data be released?

In the newly submitted supplementary material, we provide an additional experiment that applies the implicit generator to a new human dataset featuring various clothes and hairstyles. We have also included the corresponding code for this experiment along with a readme file. More code and data from the main paper will be released later.

Q: Relevant papers and typos.

We have added the relevant papers and corrected the typos.