TapMo: Shape-aware Motion Generation of Skeleton-free Characters

(1) Data. Thank you for your advice. The experiment on HumanML3D dataset focus on evaluating the motion quality of our text-based motion generation, which ignores the mesh quality of the animation. In addition, we conduct extensive experiments on Mixamo and ModelsResource-RigNet datasets, featuring a diverse set of non-human characters, to evaluate the mesh animation quality as Table 2 and Figure 4 shows.

More generated results for non-humanoid characters and unseen characters can be seen here: https://1drv.ms/v/s!AiJmM4J25MWVzvwreLZLlS_PWkDeHQ?e=rS32C3

(2) Driving signal. Thanks for this insightful comment. Whole-body motion generation, such as body gestures and facial expressions, is indeed a valuable research direction that warrants further study.

(3) Representation. Yes, the handle-based deformation strategy currently faces challenges in achieving fine-grained motion control, leaving skeleton-based animation methods dominant in this field. However, the handle-based deformation strategy employed in TapMo offers advantages in efficiently controlling a diverse range of characters in a unified and flexible manner. Our qualitative results illustrate that a limited number of handles (K=30) for generating body animations for these characters is sufficient. Additionally, we conducted tests using 60 handles, the same as SfTP. However, the animation results were inferior to using 30 handles, mainly due to an excess of redundant handles. This abundance adversely affects the Handle Predictor's ability to learn reasonable skinning weights, impacting the overall animation quality.

(4) Quadruped animals. TapMo proves its versatility by successfully animating non-humanoid upright characters like the slime in Figure 1, the potato elf in Figure 4, and the jellyfish, the ghost in Figure 7, enabling them to move in a human-like manner. Additionally, TapMo can drive animals with multiple feet to perform simple actions like walking or running, treating their bodies as human bodies without arms. These quadruped animals and motions haven’t seen by the model in training. These results convincingly demonstrate the remarkable adaptability of our TapMo to characters with diverse shapes and forms. When it comes to more difficult motion of quadrupeds, such as catching prey, their distinct movement patterns significantly differ from those of humans, making it impossible to learn them solely from human motion data. Nevertheless, leveraging the versatility of our Handle-based Mesh Deformation approach, our TapMo system can be extended to animate quadrupeds by training it on quadruped mocap data.

(5) GAN loss. As indicated by the results presented in Table 2, the inclusion of the GAN loss proves beneficial for our shape-aware motion diffusion model in generalizing to unseen characters. This is evident in the lower "A-L(U)" metric achieved when incorporating the GAN loss. However, the GAN loss will also cause a slight increase in the "A-L(S)" and Handle-FID metrics. Since generalization ability is our focus in dealing with a wide variety of characters, we use the GAN loss in our TapMo.

(6) Metrics in Table 1. The value of Diversity metric depends heavily on the pre-trained motion feature extractor. Since the motion representation in our TapMo is differ from that in standard HumanML3D, we adopt the same configurations as theirs and retrain the feature extractor to align with our motion representation. When compared with our real distribution, the Diversity metric of TapMo decreases by 0.452 (16.724 vs. 17.176). This aligns with the trends observed in existing methods, such as T2M and TEMOS compared with their real distribution. The Diversity metric of MDM is most closely aligned with its real distribution (9.559 vs. 9.503).

The analysis method for the Multimodal Dist metric is similar to Diversity. For Multimodal Dist, our TapMo is superior to MDM with 0.687 higher than our real distribution, while MDM is 2.592 higher than its real distribution. T2M demonstrates the best performance on Multimodal Dist with 0.366 higher than its real distribution. As for Multimodality, TapMo performs at a comparable level with MDM and significantly outshines other methods.

(1) The proposed Mesh Handle Predictor is implemented through a GCN-based network capable of handling graph structures with an arbitrary number of vertices and topologies. This enables the processing of different characters with varying numbers of mesh vertices and distinct mesh topologies in a unified manner.

We predefine K=30 handles for all characters and utilize the Handle Predictor to learn and predict a V*K-dimensional skinning weight, assigning the V mesh vertices to K handles. The vertex number V is specialized for different characters. With the exception of the first handle, designated as the root handle, the network autonomously learns the semantics and order of the mesh handles. For instance, in the case of a character lacking legs, as illustrated by the mushroom in Figure 5, vertices will not be allocated to handles associated with leg semantics. In other words, all vertices will have approximately zero skinning weight for the leg handles.

During training, our Mesh Handle Predictor is trained without handle annotations. It learns the semantics of mesh topologies in a weakly-supervised manner by optimizing the loss in Eq. 5.

(2) Shape-aware Motion Diffusion. The central aim of our Shape-aware Motion Diffusion is to generate motion for a diverse range of skeleton-free characters. The primary challenges lie in the inherent variation in shapes and mesh topologies among different characters, and employing a uniform mesh deformation method can readily result in mesh distortion. Therefore, the consideration of character shapes stands out as the key factor in enhancing the quality of the generated animations.

While the latest diffusion-based motion generation methods have proposed significant enhancements to improve the quality and diversity of generated motion, the focus of our work differs significantly. Our primary objective is not aligned with those methods. Instead, we concentrate on animating a diverse array of non-rigged characters uniformly, placing emphasis on ensuring the quality of animation across this variety.

(3) Experiments. Thanks for pointing out this problem. In the updated version, we have added citations and provided comparisons with these two diffusion-based methods that are relevant to our work.

(4) Thanks for your advice. We will revise our manuscript carefully to highlight our contribution and introduce our method more clearly. More generated results for non-humanoid characters and unseen characters can be seen here: https://1drv.ms/v/s!AiJmM4J25MWVzvwreLZLlS_PWkDeHQ?e=rS32C3

Thanks for the response, and I have a few more questions to discuss with the authors.

1. I am not sure whether those two weak supervisions (i.e.,$L_h$ and $L_a$) really work. It seems this manuscript does not mention where these loss terms were added. Assuming they were added on the predicted x0, the noise should also be quite significant. Maybe the authors could explain more.

2. Moreover, the consistency loss for skeletal lengths itself is not strong; in fact, achieving consistent skeletal lengths could be accomplished through forward kinematics. The reported results from baseline methods using forward kinematics are very bad, which however are expected to be better. Hence, I'm not sure if there might be an issue with the implementation.

(1) Mesh deformation feature. We thank you for bringing this issue to our attention. The mesh deformation feature is formed by concatenating three-level features within the Mesh Handle Predictor, each with dimensions of 64, 128, and 256 respectively. Subsequently, a linear layer is employed to map the mesh deformation feature to a dimension of 512. This transformed feature is then concatenated with the text token to construct the condition token in diffusion model. The mesh deformation feature encapsulates both the shape and skinning information of the non-rigged character in latent space, providing a comprehensive representation for further shape-aware motion generation.

The mesh-specific adaptation refers to a local translation vector associated with the predicted mesh handles. The final local translation of the handles is the cumulative result of the generated local translation and the adaptation. Consequently, the mesh vertices can be deformed through Eq. 3.

The implementation of the discriminator is inspired by VIBE (CVPR 2020, Kocabas Muhammed et. al.), incorporating a GRU module to extract motion sequence features and an Attention module to fuse these features. The final classification of the motion sequence is achieved through a linear mapping with a sigmoid function. Notably, before feature fusion, we concatenate the features with the mesh deformation feature to enable the discriminator to perceive the mesh shape as well. The discriminator and the diffusion model are trained jointly.

We have included a comprehensive description of the mesh deformation feature in the revised Appendix (Page 13, Implementation Details).

(2) Handle-FID. The two SMPL human models and their predicted handles utilized in our experiments can be visualized through the following link (https://1drv.ms/f/s!AiJmM4J25MWVzsMKfQ9Fx2L-RXFg3w?e=0aeCBy). It's best to view the OBJ files in MeshLab so that you can observe the skinning weights. We manipulate the beta parameters in the SMPL model to introduce a significant shape difference between the two models. The color of the mesh surface indicates the predicted skinning weights.

The purpose of introducing the Handle-FID is to evaluate the global geometry quality of the animation results. Thus, for ease of implementation without losing universality, we use beta parameters to adjust the shape of the SMPL model instead of using the MGN.

(3) Skeleton-free papers. Thanks for your insightful advice. The ZPT introduces an implicit pose deformation module, which contrasts with our explicit mesh deformation approach. On the other hand, the HMC represents an enhanced version of SfPT, employing a coarse-to-fine transfer of mesh motion to mitigate mesh distortion. However, it's noteworthy that the computational costs associated with HMC are considerably higher than those of SfPT. In our study, we propose the learning of mesh-specific adaptation to address the mesh distortion issue, akin to the fine-grained deformation in HMC but with improved efficiency.

We have incorporated citations and provided a detailed analysis of these two relevant papers in the final version of our paper (Section 2. Related Work. Skeleton-free mesh deformation).

(4) PMD metric. Given that TapMo is a generative model producing diverse motion outputs, calculating the PMD metric using the ground-truth mesh motion of the SMPL model becomes impractical.

(5) The Motion Adaptation Fine-tuning strategy is helpful for preserving the shape geometry of the characters. The qualitative results can be seen here: https://1drv.ms/v/s!AiJmM4J25MWVzvwqiMJ3f-TJaiyzLg?e=yAm9x8

(6) Eq1 in SfPT. The practical implementation of Eq. 1 in the SfPT code aligns with our approach. Although our formula expression differs from Eq. 1 in SfPT, it is because we choose to separately represent the translation term instead of combining it into the rotation matrix. This decision is made to enhance the clarity of expressing the mesh deformation process.

(7) Local translations and local rotations. The local translations and local rotations have K-1 dimensions in the first dimension, excluding the root handle. We appreciate you for pointing out this problem. We have thoroughly revised the content to rectify the error and enhance the overall clarity of the writing.

(8) Adjacent handles. We predefine the handle adjacency based on three principles: 1) clear semantics of handles, 2) shared by most characters, and 3) significant impact on mesh deformation. As illustrated in Figure 3 ``Spring Loss '', we establish the handles' adjacency on the arms and legs of the character. This approach eliminates the need for separate processing for each character, as all characters share the same handle adjacency. During inference, the handle adjacency is not needed.

(9) Skinning weights of the vertices. Yes. Thank you for helping us correct this writing error.

(1) Foot-ground penetration. Thank you for the positive feedback and for bringing attention to the issue of foot-ground penetration in animations—a challenge that is indeed widespread in most text-motion generation models. We appreciate the opportunity to discuss this further. In current kinematic-based motion generation models, addressing this problem typically involves post-processing steps. In contrast, PhysDiff [Ye Yuan et.al. ICCV 2023] employs advanced physical simulation techniques to dynamically adjust foot-ground interactions, resulting in a significant reduction in penetrations. This method represents a novel approach in the realm of text-motion generation models. We are committed to exploring and addressing this challenge in our future research endeavors, striving to find a more effective resolution for skeleton-free animations.

(2) The adversarial loss is one of our special design for the skeleton-free animation pipeline, which helps the network generate reasonable motion adaption to reduce mesh distortion.

Specifically, a mesh motion discriminator is designed to process the mesh vertices of a motion sequence along with the character's shape feature as inputs. It then produces a classification result indicating whether the mesh motion is driven by the predicted handles or the pre-rigged skeleton. The input mesh vertices are automatically sampled on the character's arms using the predicted handles and skinning weights. Recognizing that skeleton-driven character arms can maintain a reasonable articulated structure and shape in a motion sequence, we employ the adversarial loss to penalize abnormal deformations resulting from the handle-based driving approach.

The adversarial loss is formulated as Eq.8 in the paper. The optimization process involves simultaneously optimizing the diffusion model and the discriminator using adversarial loss and diffusion-based reconstruction loss. In addition, a margin parameter of 0.3 is introduced, dictating that the discriminator is optimized only when the score of the fake sample exceeds the margin parameter, and vice versa.

We have added the detailed training strategy in the revised Appendix (Page 13, Implementation Details).