BRIDGING THE GAP BETWEEN HUMAN MOTION AND ACTION SEMANTICS VIA KINEMATIC PHRASES

Dear Reviewer DPfm:

we extend our deepest gratitude for your time, efforts, thorough review, and constructive advice in improving the paper. We sincerely appreciate for recognizing KP designation as well-principled and well-thought-out. We sincerely for recognizing our work as well-presented. Herein, we respond to each of your concerns.

1. Analysis of failure modes. In Appendix F.3.3 and Figure 10, we include a typical user-studied KPG failure case of current models. where redundant motion is generated. To demonstrate more, please refer to the general response part 2.

2. More qualitative results. Please refer to the general responses.

3. Training data. In Tab. 3, all methods with * are trained on HumanML3D only, including the KP-guided latent space. The rest is trained with KPB.

4. Details of the adopted method. Our model is mostly designed for a fast and lightweight effectiveness evaluation of KP. The results show that even by simply incorporating KP with a simple VAE structure, considerable improvements are achieved.

5. Alignment. For alignment, the losses in Eq. 4 and Eq. 6-7 are designed to align the motion latent space with the KP latent space.

6. Decoder inputs. During training, as the reconstruction loss in Eq. 4, an arbitrary combination of $z_m$ and $z_p$ are sent into the decoders, resulting in the flexible input of the decoders.

7. Interpolation details. Sorry for the potential ambiguity. The interpolation is not conducted directly via manipulation of the latent codes. Instead, we extract latent code from the corrupted motion and KP sequences. With self-supervised learning, our model is capable of reconstructing the complete KP sequence from the latent code extracted from the corrupted KP sequence. With the reconstructed KP sequence, we further feed it into the encoder for a latent with complete information, then reconstruct the whole motion with the motion decoder.

Thank you again for your constructive feedback. We will incorporate the corresponding modifications and expansions in the revision. In addition, the corresponding code and model weights will be open-source to ensure replication. If there are any further questions, please inform us. We would be very happy to do anything we can that would be helpful in the time remaining. Thanks!

Thanks for your feedback!

With the current setting of KPG, most methods barely fail for these phrases, given the simplicity of these Phrases (e.g., "left-hand moves forwards" for position phrases, "left-hand moves closer to the right hand" for position distance phrases, "left arm bends" for limb angle phrases). The descriptions corresponding to these phrases are also covered in datasets like HumanML3D, facilitating the success of these.

However, the discrepancy between the successful KPs and the others is noticeable. For example, Position Phrases and Pairwise Relative Position Phrases could be executed with similar motions, like "left-hand moving forwards" and "left-hand is first behind the pelvis, then in front of the pelvis". However, the discrepancy shows the current models are unable to learn such reasoning ability.

Also, please notice the setting in Appendix F.2. For each prompt in KPG, the models would generate 120 frames of motion, while the target KP is only required to exist for 5 consecutive frames. This temporal constraint is rather loose considering that the phrases do not need such a long time to execute for a human. Initially, we tried a more strict setting, where only 10-frame motions are generated. All methods fail to provide feasible results. Considering our current focus on basic semantic consistency, we switch to the loose setting and leave the exploration of temporal constraint for future work.

Please let us know if there is anything helpful we can do in the remaining time.

Dear Reviewer HZyv:

First and foremost, we extend our gratitude for your time and efforts on the review, as well as the helpful advice in improving the paper. We sincerely for recognizing our work as well-presented. Herein, we respond to each of your concerns.

1. The claim of "bridging the gap". First, we believe that we should not let the perfect be the enemy of the good. Despite the limited granularity in the current version, it could be easily extended with the proposed framework. Second, we argue that the target of KP is not fully replacing language descriptions for action semantics representation. In essence, KP is designed as an automatic symbolization system for motion. We expect KP to become an interface for motion-semantics mapping, where the major burden of understanding KP is left to the machines, while humans could interfere with the process as assistance, given the easily interpretable atomic Phrases. Thus, humans are not required to be able to generate KP sequences from scratch.

2. The number of Phrases. Here we list the details for KP filtering. These will be updated in the appendix.

   • Position Phrase (PP). It is composed of 14 joints (except the pelvis and both eyes) and 3 reference vectors with originally 42 Phrases. Since there are joints that define the reference vectors (e.g. hips for the leftward vector). Specifically, shoulders and hips are excluded for the leftward vector, and hips are excluded for the upward vector, resulting in 42-6=36 PPs.
   • Pairwise Relative Position Phrase (PRPP). It is composed of 136 joint pairs and 3 reference vectors with originally 408 Phrases. 24 joint pairs that are linked by limbs are filtered out. 30 Eye-related pairs are filtered out except (for the left eye, and right eye) due to the others are covered by head-related pairs. 4 triplets that the relationship barely changes along the reference vector (e.g., right knee, left hip, and leftward vector) are filtered out. These result in 408-243-303-4=242 PRPPs.
   • Pairwise Distance Phrase (PDP). It is composed of 105 joint pairs (except for both eyes). 24 pairs that are linked by the human body are filtered out, resulting in 81 PDPs.
   • Limb Angle Phrase (LAP). The arms, legs, and their link with the upper body are included as 8 LAPs.
   • Limb Orientation Phrase (LOP). The 8 arm and leg limbs, head, collarbones, hips, torsos, and upper body, paired with the 3 reference vectors, result in 45 LOPs in total. 12 pairs that barely change along the reference vector (e.g., left-right hip, leftward vector) are filtered out, resulting in 45-12=33 LOPs.

3. KPG evaluation of naturalness. We agree that naturalness is necessary, and also provide user-study results on naturalness in the supplementary materials. However, KP-based accuracy is not an all-in-one metric. With its main target as semantic consistency evaluation, it is more an alternative to R-Precision instead of FID. We believe pursuing an interpretable naturalness metric like KP-based accuracy for semantic consistency is worth exploring.

4. Equations. Thanks for your advice. We will incorporate these in the revision.

5. Typos. Thanks for pointing out. We will fix it.

6. Phrase. Sorry for the ambiguity. By Phrase, we mean the signs as $[-1, 0, 1]$. We will clarify this in the revised version.

7. Human cognitive view. It means that the reference vectors are derived from human cognitive references for egocentric descriptions of actions. For example, we are more interested in which direction vectors humans adopt when saying "My hand is moving leftwards", and find the vector from right hip to left hip.

8. KP for action recognition. Thanks for your constructive feedback. We conduct a simple experiment for action recognition on the BABEL validation set, where promising results are shown with Top-1 accuracy improving from 41.14 to over 45. The details are updated in the supplementary materials.

Thank you again for your helpful feedback. We will incorporate the corresponding modifications and expansions in the revision. In addition, the corresponding code and model weights will be open-source to ensure replication. If there are any further questions, please inform us. We would be very happy to do anything we can that would be helpful in the time remaining. Thanks!

Dear Reviewer 537a:

First and foremost, we extend our deepest gratitude for your thorough review and helpful advice in improving the paper. We sincerely appreciate for recognizing our work as well thought out and motivated. Herein, we respond to each of your concerns.

1. The marginal advantage over SOTA. Notably, our text-to-motion model could be identified as improving the latent-based diffusion proposed in MLD with the newly proposed KP. This choice is made to conduct a fast and light-weight effectiveness evaluation of KP. Therefore, a proper comparison target should be MLD, upon which considerable advantages are shown in the user-study results and KPG. For T2M-GPT and MotionGPT, which are the current SOTAs, we would like to emphasize the different paradigms they adopted, as well as the model scale. Both of them tokenized motions into a codebook, coupled the codebook with text tokens, and then adopted a different pipeline as an auto-regressive decoder-only generative model, which resembles GPT. Also, they possess a considerably bigger model scale. For example, the 18-layer transformer is adopted by T2M-GPT for the text-to-motion task, however, in comparison, only a 4-layer denoiser and a 4-layer with a frozen CLIP text encoder are adopted in our method. Therefore, we would claim that despite the diverse paradigm and model scale, by incorporating KP into an MLD-like pipeline, we manage to achieve comparable performance with SOTAs, proving the efficacy of KP. Furthermore, given the motion quantization ability of KP as mentioned by Reviewer DPfm, also its effortless conversion to natural language, we are optimistic about incorporating it with T2M-GPT for future work.

2. Qualitative results with the standard metric values. Please refer to the general responses, part 1.

3. Details of motion modification. We are sorry for the oversimplified illustration in Fig. 3. Here we provide a more detailed figure at this link to illustrate the motion modification process. The image is also updated in the supplementary materials as modify.png, which will also be incorporated in the PDF.

4. Details of KPG evaluation. Given the generated motion X with t frames and the target KP $c_i \in [-1, 0, 1]$, we first extract the KP sequence corresponding to $c_i$ as $C_i \in [-1,0,1]^t$. Then, $c_i \in C_i$ is recognized when there is the sequence $c_i, c_i, c_i, c_i, c_i$ is a consecutive subsequence of $C_i$. For PRPP and LOP evaluation, the sequence $c_i^0, c_i^0, c_i^0, c_i^0, c_i^0, c_i^1, c_i^1, c_i^1, c_i^1, c_i^1$ is the expected subsequence. We will update the details in the PDF.

5. Temporal ambiguity of KP. Notably, the conversion from motion to KP could involve no temporally down-sampling. In this sense, KP would not introduce new temporal ambiguity upon the captured motion. Given the example "the left eye is in front of the right eye and then behind the right eye", if it is included in the raw motion, KP would faithfully reflect the corresponding information.

Thank you again for your constructive feedback. We will incorporate the corresponding modifications and expansions in the revision. In addition, the corresponding code and model weights will be open-source to ensure replication. If there are any further questions, please inform us. We would be very happy to do anything we can that would be helpful in the time remaining. Thanks!

Dear Reviewer 87dC:

First and foremost, we extend our deepest gratitude for your thorough review and helpful advice in improving the paper. We sincerely appreciate for recognizing our work as fairly motivated. Herein, we respond to each of your concerns.

1. Presentation. Thanks for your kind advice on making the paper more understandable. For the metrics in Tab. 3, we explain the details in Appendix F.2 due to the page limit.

2. Comparison to SOTA. Notably, our text-to-motion model could be identified as improving the latent-based diffusion proposed in MLD with the newly proposed KP. This choice is made to conduct a fast and light-weight effectiveness evaluation of KP. Therefore, a proper comparison target should be MLD, upon which considerable advantages are shown in the user-study results and KPG. For T2M-GPT and MotionGPT, which are the current SOTAs, we would like to emphasize the different paradigms it adopted, as well as the model scale. Both of them tokenized motions into a codebook, coupled the codebook with text tokens, and then adopted a different pipeline as an auto-regressive decoder-only generative model, which resembles GPT. Also, they possess a considerably bigger model scale. For example, the 18-layer transformer is adopted by T2M-GPT for the text-to-motion task, however, in comparison, only a 4-layer denoiser and a 4-layer with a frozen CLIP text encoder are adopted in our method. Therefore, we would claim that despite the diverse paradigm and model scale, by incorporating KP into an MLD-like pipeline, we manage to achieve comparable performance with SOTAs, proving the efficacy of KP. Furthermore, given the motion quantization ability of KP as mentioned by Reviewer DPfm, also its effortless conversion to natural language, we are optimistic about incorporating it with T2M-GPT for future work.

3. Clarifying the main target. Thanks for your kind advice, and we will update our presentation in the revision.

4. Visualization of more diverse motions beyond walking. Please refer to the general response.

5. Figures that are not explanatory enough. Thanks for your kind advice, and we will polish the figure captions in the revision.

6. "For objectivity and actuality, KP captures sign changes with minimal pre-defined standards." By objectivity and actuality, we mean KPs are defined solely on the sign changes of the indicators calculated raw motion only, with little further human-defined thresholds, which could introduce unexpected subjectivity.

7. Proper abstraction of KP. In the first place, the proper abstraction is a design motivation of KP. Compared to raw motion, it is designed as more abstract with interpretable atomic Phrases. Compared to natural language, it tends to preserve more detailed information though not as concise as text descriptions, which could be shown by the higher diversity while keeping competitive consistency as shown in the user studies.

8. The derivation of the number of KPs. Here we list the details for KP filtering. These will be updated in the appendix.

   • Position Phrase (PP). It is composed of 14 joints (except the pelvis and both eyes) and 3 reference vectors with originally 42 Phrases. Since there are joints that define the reference vectors (e.g. hips for the leftward vector). Specifically, shoulders and hips are excluded for the leftward vector, and hips are excluded for the upward vector, resulting in 42-6=36 PPs.
   • Pairwise Relative Position Phrase (PRPP). It is composed of 136 joint pairs and 3 reference vectors with originally 408 Phrases. 24 joint pairs that are linked by limbs are filtered out. 30 Eye-related pairs are filtered out except (the left eye, and right eye) because the others are covered by head-related pairs. 4 triplets that the relationship barely changes along the reference vector (e.g., right knee, left hip, and leftward vector) are filtered out. These result in 408-243-303-4=242 PRPPs.
   • Pairwise Distance Phrase (PDP). It is composed of 105 joint pairs (except for both eyes). 24 pairs that are linked by the human body are filtered out, resulting in 81 PDPs.
   • Limb Angle Phrase (LAP). The arms, legs, and their link with the upper body are included as 8 LAPs.
   • Limb Orientation Phrase (LOP). The 8 arm and leg limbs, head, collarbones, hips, torsos, and upper body, paired with the 3 reference vectors, result in 45 LOPs in total. 12 pairs that barely change along the reference vector (e.g., left-right hip, leftward vector) are filtered out, resulting in 45-12=33 LOPs.