M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation

W1: The proposed framework is complex and may require significant computational resources to implement.

To ensure reproducibility, we will open-source the training and inference code of proposed $M^3$GPT framework.

For computational resources, please refer to reply to Q2 in Global Response.

W2-1: The paper could benefit from a more detailed discussion of evaluation metrics and benchmarks.

Thanks for your suggestions. We will add these discussion in final version.

(1) Evaluation Metrics. The table below shows the evaluation metrics for each task.

• FID: Measures distance between the distributions of generated and real motion.
• R-Precision: Measures retrieval accuracy of generated motion/text relative to input, reflecting semantic similarity.
• Div: Measures diversity of generated motions.
• Bleu, Rouge, CIDEr: Measures accuracy, matching degree and semantic consistency between generated and real text.
• FID$_k$, Div$_k$: FID and Div calculated using kinetic features.
• MPJPE: Measures the mean per joint position error between generated and real motion.
• BAS: Computes beat alignment degree between generated dance and input music.
• BCS/BHS: Measures convergence/alignment between generated and real music.

(2) More Benchmarks. The tables below adds two benchmarks. As shown in below table, and Table3/4 of main paper, we observe that:

• Text-to-Motion: $M^3$GPT achieves comparable FID and best R-Top1 and Div score, showing high-quality motion generation with strong semantic alignment and diversity.
• Motion-to-Text: $M^3$GPT achieves best R-precision and CIDEr score, indicating fluent and semantically consistent motion descriptions generation.
• Motion Prediction and Inbetween: $M^3$GPT achieves best FID, Div and MPJPE score, reflecting superior motion prediction quality, diversity and precision.
• Music-to-Dance (Table 4): $M^3$GPT achieves comparable Div/BAS and best FID$_k$ score, indicating high-quality dance generation with good beat.
• Dance-to-Music (Table 4), $M^3$GPT achieves comparable BCS and best BHS score, indicating improved semantic alignment of music with input dance.

W2-2: The paper could benefit from a more in-depth discussion on potential real-world applications and limitations of $M^3$GPT.

(1) Potential applications: $M^3$GPT has may real-world applications, such as animation creation, visual digital human control, music-based dance choreography creation. Taking animation creation as a example, traditional manual methods adjusts character joints frame-by-frame, which is extremely time-consuming and labor-intensive. With $M^3$GPT, one can simply provide a textural description of desired motion, and $M^3$GPT will automatically generate the corresponding animation, largely reducing labor and time cost.

(2) Limitations: $M^3$GPT focus on modeling human body movements, but currently lacks the capability to handle hands. As a result, $M^3$GPT cannot address speech-driven motion generation, which involves gestural rather than body movement.

Q1: Could you provide more details on the training process, including the computational resources required and the time taken to train $M^3$GPT?

Yes, please refer to our reply to Q2 in Global Response.

Q2: Have you conducted any ablation studies to understand the contribution of each component of $M^3$GPT to the overall performance

Yes, in Table 2 of main paper, we have conducted ablation studies to show the effectiveness of each component of $M^3$GPT. Specifically, $M^3$GPT includes 3 major components designed to improve performance: (1) Introduction of two auxiliary tasks, music-to-dance (A2D) and text-to-music (T2D). (2) Joint optimization of LLM and motion de-tokenizer. (3) Instruction-tuning. As shown in the table below, each component contributes to the overall performance of $M^3$GPT, specifically for the complex music-to-dance task.

Q3: Can you provide more examples or case studies where $M^3$ has been applied to real-world scenarios? What challenges were encountered?

We have applied $M^3$GPT to several real-world scenarios, including animation creation, dance choreography creation, music creation from dance.

The main challenge is generating long sequence (longer than 5 minutes), where the quality of generated content degrades over time. We leave the generation of high-quality long-sequence (longer than 5 minutes) as a future research direction.

Dear Reviewer KMVC:

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.

W1: In Table3, some comparation results are missed for text-to-motion task.

Thanks for your suggestions. We will add the related works mentioned, MoMask [1], PRO-Motion [2] and MotionLCM [3], in revised version.

(1) To ensure fair comparison, we reproduce MoMask and MotionLCM on Motion-X dataset using their official code. As shown in the table below, our method achieves superior performance than MotionLCM, and comparable performance with MoMask.

(2) Since PRO-Motion is not open-sourced, we were unable to reproduce it on Motion-X dataset, so its result is not included in Table3.

W2: The parameterizations (dimensions) of the variable in Section 3 are all missing.

Thanks for your suggestions. We will add the dimensions of variable in section 3 as follows in revised version.

• $N_m$ (line 128), the size of motion codebook $\mathcal{B}_m$, is set to 512.
• $d_m$ (line 131), the dimension of raw motion features, is 263.
• $d$ (line 133), the dimension of embeddings in motion codebook, is set to 512.
• $d_a$ (line 149), the dimension of raw music, is 128.
• $T_m$ (line 131) and $T_a$ (line 149) denotes the sequence length of motion and music, respectively. So their values depend on the input sequence.
• $L_m$ (line 133), the sequence length of diverse motion tokens, is $T_m/4$. Here, 4 is the downsampling rate of motion tokenizer.
• $L_a$ (line 150), the sequence length of diverse music tokens, is $T_a/128$. Here, 128 is the downsampling rate of music tokenizer.

W3: T5 is utilized as the language model backbone. While most previous model utilize CLIP or BERT, you should include ablation study about the text encoder.

Following your suggestions, We add ablation study on CLIP and BERT as text encoder. Specifically, we use T5, CLIP and BERT as text encoders respectively, each combined with T5 decoder. We then conducted experiments on the text-to-motion task using these configurations. As shown in below table, T5 encoder can achieve the best performance, validating its superiority over CLIP and BERT in our framework.

Q1: Is it just a matter of migrating multimodal model to the motion domain?

$M^3$GPT is not simply migrating multimodal model to motion domain. Unlike most multimodal models such as BLIP2, LLaVA, Ferret, which primarily focus on understanding tasks, $M^3$GPT takes a step further by incorporating multimodal tokenizers and de-tokenizers, achieving both motion comprehension and generation across text, motion/dance and music modalities. Additionally, $M^3$GPT jointly optimizes LLM and motion de-tokenizer, further enhancing its capabilities in motion generation.

Q2: What new issues have been encountered in the process?

We have encountered two new issues:

1. Within the multi-task framework, music-to-dance task poses greater challenges than text-to-motion. This difficulty arises because both input and output modalities of music-to-dance are unfamiliar to LLMs, and there is limited training data available for this task. Therefore, effectively leveraging text-to-motion task to enhance more complex music-to-dance task is a critical issue that needs to be addressed.
2. The multi-stage training leads to error accumulation in motion generation, due to the combined effects of LLM's prediction and de-tokenizer. Therefore, mitigating error accumulation from multi-stage training is another critical issue that needs to be addressed.

Q3: What is your innovation in solving these problems?

1. To solving the first issue outlined in Q2, we design two strategies. (1) a shared tokenizer for motion and dance data, and (2) two auxiliary tasks: music-to text and text-to-dance. Through these auxiliary tasks, $M^3$GPT implicitly learns to decompose complex music-to-dance into simpler music-to-text and text-to-dance tasks. Also, the shared tokenizer ensures that text-to-dance and text-to-motion tasks occupy the same matching space, enabling mutual reinforcement. In this way, $M^3$GPT builds the connection between text-to-motion and music-to-dance, leveraging text-to-motion to enhance its music-to-dance capabilities.
2. To solving the second issue outlined in Q2, we jointly optimize LLM and motion de-tokenizer during training. This strategy can help mitigate error accumulation caused by multi-stage training, thereby enhancing the model's motion generation capabilities.

Dear Reviewer 2Gpw:

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.

W1: For supplementary videos, it's hard to judge the quality of music-motion generations without combining them in one video; Long-term dance generation on aist++ is only a little longer than 5s.

(1) Due to the rebuttal period restrictions on video uploads, we will present music and dance in one video for clearer evaluation in revised version.

(2) Regarding long-term dance generation, AIST++ comprises musics that are slightly longer than 5s. So the generated dances on AIST++ are relatively short. To showcase our model’s capability for long-term dance generation, we have included a dance generation of over 90s on FineDance in supplementary video.

W2: How the gradient from de-tokenizer backpropagate to LLM?

In implementation, we do not directly backpropagate de-tokenizer gradient to LLM. Instead, with the goal of minimizing L1 loss between the predicted and real motion, we search for the motion's token sequence that could minimize this L1 loss in original motion space. As the motion de-tokenizer continuously optimizes, the target motion's token sequence, which supervises LLM training, dynamically changes. This dynamic adjustment reduces L1 loss progressively, achieving joint optimization.

W3: For motion prediction and in-between tasks, pose-level metrics like MPJPE are needed for comparison.

In the tables below, we add MPJPE as a metric. Our method still outperforms existing works on MPJPE metric in both tasks.

|Methods on Motion-X| Motion Prediction (MPJPE$\downarrow$)| Motion In-between ( MPJPE) |   | ---- | ----- | ----- | |T2M-GPT | 80.2| 63.7 | |MoMask |67.9 | 55.2| |MotionGPT | 71.3| 59.9| | M3GPT (Instruction-tuned)| 54.2|51.0 |

Q1: Did authors try different LLM architecture or different size of T5, what is conclusion and training time for T5?

Yes, Please refer to our reply to Q1 in Global Response.

Q2: Did you evalute two auxiliary tasks music2text (A2T) or text2dance (T2D) quantitatively or qualitatively?

(1) For quantitative evaluation, we report the Bleu@4 and CIDEr for A2T, and R Top1 and FID for T2D. As shown in the tables below, our method consistently outperforms single task training.

(2) For qualitative evaluation, Figure 4 of uploaded PDF visualizes the generation of text2dance. Compared to the singe-task model, $M^3$GPT generates more realistic dance aligned with input music.

Q3: For zero-shot tasks, can they combined together for more application? For instance, generating a long dance while we specify text prompt for one segment in the middle of sequence. How about zero-shot text2music?

(1) Our model can perform additional zero-shot tasks. It can generate a long dance while specifying text prompt for one segment, as visualized in Figure 3 of uploaded PDF. This application integrates music-to-dance, zero-shot music+dance-to-dance, and zero-shot music+dance+text-to-dance tasks. As shown in that Figure 3, the generated dance is coherent and consistently aligning the input text prompt.

(2) Our model can perform zero-shot text-to-music. The table below shows a quantitative evaluation on text-to-music task. Even without training on text-to-music task, $M^3$GPT can obtain comparable even superior performance over existing methods.

Q4: For tab.4, what is the result of Instruction-tuned only on AIST++ or FineDance?

Please refer to our reply to Q3 in Global Response.

Q5: How did you implement motion in-between in pretrained stage use the equation(4)?

Motion in-between can be implemented using Equation 4. Formally, given a motion sequence $\boldsymbol{m} = \left(m_1, \dots, m_N\right)$, motion in-between task uses the first $n_1$ and last $n_2$ frames, $\boldsymbol{m_1}=\left(m_1, \dots, m_{n1}, m_{N-n2+1}, \dots, m_{N}\right)$, to predict intermediate frames $\boldsymbol{m_2}=\left(m_{n1+1}, \dots, m_{N-n2}\right)$. So the future frames in $\boldsymbol{m_1}$ are only used as input, and the target frames are the middle frames in $\boldsymbol{m_2}$.

In implementation, $\boldsymbol{m_1}$ is fed into the motion tokenizer to produce a token sequence, serving as source input of LLM ($\boldsymbol{q_s}$). Also, $\boldsymbol{m_2}$ is fed into the motion tokenizer to produce a token sequence, serving as target output of LLM ($\boldsymbol{q_t}$). Thus motion in-between can be formulated as $p_{\theta}\left(\boldsymbol{q}_t^i |\boldsymbol{q}_t^{<i}, \boldsymbol{q}_s \right)$ (Equation 4).

Q6: Is there any special token for motion and music in the unified token vocabulary, like start/end of motion/music token? If not, how to ensure the pretrained model generate desired modality?

Yes. There are special tokens for motion and music, such as <start_of_motion> and <start_of_music>. These special start and end tokens control the beginning and end of the model's decoding process.

Additionally, we use task-specific instruction to control model to generate desired modality. For example, the instruction Generate a motion for the caption is used for text-to-motion task; Instruction Generate a music based on the dance is used for dance-to-music task.

Q7: The statistics of music2text (A2T) and text2dance (T2D) paired dataset are missing.

Please refer to our reply to Q4 in Global Response.

Dear Reviewer Wkeh:

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.

W1: There is still doubt how can text-to-motion or music-to-dance tasks benefits from multimodal framework.

We argue that text-to-motion and music-to-dance tasks benefits from multimodal framework in two main aspects:

1. A shared tokenizer for motion and dance data: The shared tokenizer inherently broadens the scale and diversity of the training data, allowing this tokenizer to learn more generalizable representations for both motion and dance data.
2. Constructing two auxiliary tasks, music-to-text (A2T) and text-to-dance (T2D): Through auxiliary tasks, $M^3$GPT implicitly learns to decompose the complex music-to-dance task into simpler A2T and T2D tasks. Also, with a shared motion tokenizer, text-to-dance and text-to-motion tasks reinforce each other within the same matching space. In this way, $M^3$GPT builds the synergies between music-to-dance and text-to-motion, facilitating mutual reinforcement.

As shown in the table below, the shared tokenizers and auxiliary tasks generally lead to gains in both tasks, validating their effectiveness.

W2-a: Comparison with MotionGPT on Motion-X is missing.

The table below presents the results of MotionGPT on motion-X. Our method outperforms MotionGPT across all 4 tasks on motion-X.

W2-b: Lacking comparison with T2M-GPT on motion prediction task.

The table below presents the results of T2M-GPT on motion prediction task. Our method largely outperforms T2M-GPT.

W2-c: Lacking comparison with MoMask on motion In-between task.

The table below presents the results of MoMask on motion In-between task. Our method outperforms MoMask in motion In-between task.

W3: The qualitative comparison of generated motions by the proposed $M^3$GPT framework from text or music is absent.

(1) Figure 1 of uploaded PDF file visualizes generated motions from text. We compare our generations with MDM and MoMask. As seen, our model generates motions of higher quality, with unrealistic motions from MDM and MoMask highlighted in red.

(2) Figure 2 of uploaded PDF file visualizes generated dances from music. We compare our generations with Bailando. As seen, the dances generated by our $M^3$GPT are more danceable.

Q1: Curious about the long-duration dance generation.

We maintain coherence and consistency in generating long dance sequences during LLM prediction phase. For long-duration dance generation, we sequentially generate 5-second dance segments. Each dance segment is generated using corresponding music and previously generated dance segments as control conditions. This strategy ensures that each newly generated dance segment aligns seamlessly with the preceding ones, maintaining overall coherence.

Q2: Could the authors provide more insights into the reasons behind the performance gap between AIST++ and FineDance datasets?

The main reason is that the training and testing settings on FineDance are inconsistent in our work, whereas they are consistent in other works.

Existing works typically train and test separately on AIST++ and FineDance—using 5-second clips for AIST++ and 30-second clips for FineDance. In contrast, we develop a unified multitask model across datasets. To maintain consistency, we slice AIST++ and FineDance data into 5-second clips for unified training, aligning with the 5 to 10-second range of motion-X data. During testing, we generate longer 30-second clips on FineDance to compare with existing methods. This inconsistency on FineDance could lead to performance drop compared to existing models.

To further validate our method, we finetune $M^3$GPT on FineDance using 30-second clips. As shown in the table below, $M^3$GPT can achieve competitive, even superior performance compared to existing methods, demonstrating its potential on FineDance.

Q3: Should evaluate different sizes of $M^3$GPT.

Please refer to our reply to Q1 in Global Response.

Dear Reviewer ZwnC:

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.