Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses

Dear reviewer RKb2,

Thank you for your positive comments. We are especially grateful for your recognition of the use of Jacobian fields as "innovative" and for acknowledging that our approach is well "justified". We provide our responses to your queries below.

Impact of the Sampling Method and Number of Keypoints. We appreciate your suggestion for an in-depth analysis. We used FPS for keypoint extraction due to its simplicity and efficacy, which have made it a popular choice in geometry analysis (e.g., PointNet++ [Qi et al., NeurIPS 2017], KeypointDeformer [Jakab et al., CVPR 2021], DeepMetaHandles [Liu et al., CVPR 2021]). Here, we investigate the impact of the number of keypoints and their sparsity. We trained our model using SMPL and DeformingThings-Animals meshes, adjusting the pose extractor to extract 50, 25, and 10 keypoints, respectively. We evaluated the Point-wise Mesh Euclidean Distance (PMD) for SMPL meshes and Fréchet inception distance (FID) for DeformingThings4D-Animals meshes, following our paper. The results are summarized in Tables 1 and 2. Notably, the model trained with just 10 keypoints still outperforms the pretrained SPT model in the pose transfer using SMPL meshes. Please refer to Figures 3 and 4 in the PDF file. We will include this analysis in the future revision.

Table 1: PMD measured on SMPL pose transfer experiments with varying number of keypoints. "Ours-$N$" denotes a variant of our network trained to extract $N$ keypoints.

Table 2: FID measured on meshes whose poses are transferred from class bear3EP of the DeformThings dataset. "Ours-$N$" denotes a variant of our network trained to extract $N$ keypoints.

Potential Drawback of Representing Poses as Keypoints. To prevent the leakage of source shape details into target shapes, we use Jacobian fields instead of vertex coordinates when extracting our pose representation. This way, our network predicts per-triangle local transformations to deform the given template to match the pose example. Furthermore, training the refinement module by optimizing intrinsic-preservation losses helps preserve intricate local details. Please refer to Figure 5 in the attached PDF file for qualitative results of the ablation study. We will include more results in the revised version.

Assumption on T-Poses. Thank you for pointing this out. It is true that our method requires a canonical pose for an identity (e.g., T-pose for humanoids). However, we believe our method remains practical for its intended purpose since default or canonical poses of deformable objects have been widely used not only for humanoids but also for animal shapes, including in works like SMAL [Zuffi et al., CVPR 2017], A-CSM [Kulkarni et al., CVPR 2020], BARC [Ruegg et al., CVPR 2022], BITE [Ruegg et al., CVPR 2023], VAREN [Zuffi et al., CVPR 2024], and 3D Fauna [Li et al., CVPR 2024]. These studies define a canonical pose as a shape standing still with its legs straight for quadrupeds. Nonetheless, there may be cases where obtaining, or even defining a canonical pose of an object is challenging. As such, lifting the necessity of template shapes is one of the directions that we are heading to.

Pose Transfer between Mixamo Meshes. We appreciate your suggestion. We downloaded 28 motions in the test split of SPT [Liao et al., ECCV 2022] from the Mixamo repository, comprising a total of 3,025 frames. We followed the preprocessing steps of SPT to extract meshes from skeleton configurations. Since the pretrained SPT model was trained on a larger dataset consisting of shapes from AMASS, Mixamo, and RigNet, we re-trained the model to analyze its performance in the same problem setup as ours. For training, we used 3,025 different poses of a single character. After training, we used 300 poses transferred to 8 different characters to compute Point-wise Mesh Euclidean Distance (PMD) by leveraging the ground-truth correspondences. We report the results in Table 3. Please refer to Figure 6 in the attached PDF file for qualitative results. As illustrated, our method better retains the smoothness and details of the surface even when transferring poses involving articulations of limbs. This is also reflected in lower PMD in Table 3. We will add these results in the revision.

Table 3: PMD measured on Mixamo pose transfer experiments.

Training & Inference Time Comparisons. We summarize the training and inference time in Table 4. The inference time includes the time to transfer a pose of one identity to another via network forward passes. We used Mixamo meshes with approximately 10K vertices and 20K faces for evaluation. Note that for SPT, we measured the inference time using the official pretrained model. Our method demonstrates better runtime performance than SPT while outperforming NJF and ZPT in terms of pose transfer accuracy. We will include this analysis in the revised version of our paper.

Table 4: Training and inference time comparison

Given the additional clarification and experimental results, I am more inclined to recommend acceptance of this paper. However, I am also want to see the potential drawbacks of using jacobian fields for shape representation and failure cases caused by simply using FPS as basis(for extreme poses and loose clothes). Also, the performance seems drop a lot for SPT after your retraining, is it possible to train on the same dataset to compare with their reported results?

Dear reviewer xBSu,

We highly appreciate your positive comments on our work, particularly regarding the writing and experiment procedures. Our response to your question can be found below.

Extension to Images and Videos. We strongly agree that our work can be extended to other modalities, such as images and videos. For instance, the proposed pose representation and our cascaded diffusion model can be utilized as a prior for 3D-aware image editing tasks where estimating motion trajectories and handling occlusions are crucial. Specifically, we believe our method can be adapted to recent drag-based image editing techniques, such as DragGAN [Pan et al., SIGGRAPH 2023] and DragDiffusion [Shi et al., CVPR 2024]. When a user inputs a set of drag-based instructions describing complex articulations of animals that cannot be resolved in 2D, we may consider lifting the object to 3D by using a single-view 3D reconstruction model (e.g., Zero-1-to-3 [Liu et al., ICCV 2023]). The coarse 3D delegate of the image can then be given to our model as an input, which extracts a pose representation from the shape. As the shape is edited following the instructions, our diffusion model, trained on a collection of pose representations, can provide guidance signals via Score Distillation Sampling [Poole et al., ICLR 2023] to ensure the resulting deformation remains realistic while faithfully reflecting the user's input. Nonetheless, this is one possible scenario where our method can be applied, and we hope to extend it to a variety of applications in the future.

Dear reviewer skND,

Thank you for recognizing our work as a study of a "novel" and "important" task, and for appreciating our "sound" and "well-motivated" approach. We have carefully reviewed your queries and provide our responses below.

Temporal Consistency. Thank you for providing the constructive comment. In our work, we mainly focus on learning a transferable pose representation from pose variations of a single identity, which is more challenging than our prior works. While our framework has not been explicitly designed for motion transfer--such as by using a temporal attention mechanism or training with motion supervision--our framework can transfer motions by transferring each frame individually as demonstrated in Figure 1 in the PDF file in our global response. We applied the One Euro Filter, a simple and easy-to-implement filter, to refine the smoothness of motions. We plan to extend our work for motion transfer in the future. We will include more results in the revised version.

Spatial Consistency. While we only showed single-view images in our paper, our method produces meshes consistent in 3D as illustrated in Figure 2 in the accompanied PDF file. In particular, we rendered one of our results shown in Figure 5 of the main paper from 4 different viewpoints. We will add more images rendered from various viewpoints in the revision.

Qualitative Results for Ablation Study. The qualitative result can be found in Figure 5 in the PDF file. As shown, directly using vertices results in low-quality meshes with noticeable distortions and artifacts. Our base model employing Jacobian fields produces high-quality shapes, which can be further improved by using the proposed refinement module. Note the preservation of intricate details near limbs. We will add the analysis and more results in the upcoming revision.

Impact of the Sampling Method and Number of Keypoints. We appreciate your suggestion for an in-depth analysis. We used FPS for keypoint extraction due to its simplicity and efficacy, which have made it a popular choice in geometry analysis (e.g., PointNet++ [Qi et al., NeurIPS 2017], KeypointDeformer [Jakab et al., CVPR 2021], DeepMetaHandles [Liu et al., CVPR 2021]). Here, we investigate the impact of the number of keypoints and their sparsity. We trained our model using SMPL and DeformingThings-Animals meshes, adjusting the pose extractor to extract 50, 25, and 10 keypoints, respectively. We evaluated the Point-wise Mesh Euclidean Distance (PMD) for SMPL meshes and Fréchet inception distance (FID) for DeformingThings4D-Animals meshes, following our paper. The results are summarized in Tables 1 and 2. Notably, the model trained with just 10 keypoints still outperforms the pretrained SPT model in the pose transfer using SMPL meshes. Please refer to Figures 3 and 4 in the PDF file. We will include this analysis in the future revision.

Table 1: PMD measured on SMPL pose transfer experiments with varying number of keypoints. "Ours-$N$" denotes a variant of our network trained to extract $N$ keypoints.

Table 2: FID measured on meshes whose poses are transferred from class bear3EP of the DeformThings dataset. "Ours-$N$" denotes a variant of our network trained to extract $N$ keypoints.

Clarification on NJF. In our experiments, we followed the setup of the morphing humans experiment in [Aigerman et al., ACM ToG 2022] where an MLP takes PointNet latents encoded from both source and target shapes and predicts a Jacobian field of the source shape that morphs it to the target shape. We did not use the setup of the re-posing humans experiment as our work does not rely on parameterizations, such as skeletons. We will clarify this in the revision.

Clarification on FID Computation. Unlike 2D images and videos with abundant reference data collected over the Internet, assessing the realism of 3D shapes remains challenging as there is no standardized way. We followed previous work including MeshDiffusion [Liu et al., ICLR 2023], 3DShape2VecSet [Zhang et al., ACM ToG 2023], MeshGPT [Siddiqui et al., CVPR 2024], and Make-A-Shape [Hui et al., ICML 2024] that compute FID using images rendered from multiple viewpoints without texture. Specifically, we rendered images from 4 viewpoints (front, back, left, and right) at zero elevation. Following MeshDiffusion [Liu et al., ICLR 2023], we applied a grayscale diffuse material for reference and output shapes and applied Phong shading to obtain images with depth cues. We will include these details of the evaluation settings in the revised version.

Template Meshes for Animals. For quadrupeds, used in our experiments and comprising a large portion of animal species, we consider shapes standing with straight legs as the animal equivalent of the T-pose for humanoids. This convention has been used in several works, including SMAL [Zuffi et al., CVPR 2017], A-CSM [Kulkarni et al., CVPR 2020], BARC [Ruegg et al., CVPR 2022], BITE [Ruegg et al., CVPR 2023], VAREN [Zuffi et al., CVPR 2024], and 3D Fauna [Li et al., CVPR 2024]. Still, there may be cases where obtaining, or even defining a canonical pose of an object is challenging. Therefore, lifting the necessity of template shapes is one of the directions that we are heading to.

Limitations Following NJF [Aigerman et al., ACM ToG 2022], our method uses differential operators to compute Jacobian fields. Therefore, it may not be directly applicable to meshes with defects (e.g., duplicate vertices) or multiple connected components.