Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs

We thank you for your thoughtful reviews and suggestions.

We enthusiastically encourage the reviewer to watch our demos which present in our opinion some of the most impressive results to date. We tested on different machines and Windows PCs should be most stable and have no problem loading the videos. Please be reminded that the zip file has to be expanded and unzipped before clicking "index.html"; Common browsers such as the latest Chrome should work. On the other hand, the videos can be directly available in the folders.

Thank you for your suggestions and we are committed to making a more stable page with more diverse demos.

More ablation studies are provided, please refer to the general response.

Render Engine

After Motion-Agent generates the joint representations, they are converted into motion capture files. We then map the generated motion data to a randomly selected 3D avatar in BLENDER to form an animated character. Finally, we render the video in BLENDER for visual demonstration.

Can this framework benefit downstream tasks such as robotics and policy learning?

We believe our framework has the potential to contribute to the fields of robotics and policy learning, which we also plan to explore in future work. The current motion representation is based on human joint-level data, which may be adapted for humanoid robot control. Leveraging LLM offers an attractive alternative to robotic control, navigation, and manipulation.

What are the benefits of using VQVAE over other models like diffusion/VAE?

We believe this question is similar to a question asking why diffusion/VAE-based models are not used in the next-token prediction in language models (LLMs). While diffusion and VAE models have shown impressive results in text-to-motion generation, we argue that if a robust and accurate next-token prediction method exists for motion generation, these models may not be necessary. While diffusion and VAE models can indeed be used as post-processing tools to further refine the generated motion, our experiments suggest that they are not essential for achieving high-quality motion generation in our framework.

On the other hand, a key benefit of using VQVAE over models such as diffusion or VAE is its support for bidirectional translation, which allows for both generation and captioning tasks. In contrast, diffusion models are primarily designed for generation tasks and do not natively support the same level of flexibility for captioning or other reverse tasks. This bidirectional capability of VQVAE-based methods makes it particularly suitable for tasks that require both understanding and generating motion, such as the motion-language framework we propose, where both motion generation and motion-to-text conversion are critical.

Which body joint representations were used for training?

We use the HumanML representation.

Reference: Guo, Chuan, et al. "Generating diverse and natural 3d human motions from text." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

How is m1 preserved—does the new generation only respond to the new prompt?

In our approach, both motions $m_1$ and $m_2$ are represented as token sequences before they are decoded into motion representations. Specifically, let us assume that motion $m_1$ corresponds to token sequence $s_1$, and motion $m_2$ corresponds to $s_2$, such that $m_1 = D(s_1), m_2 = D(s_2)$, where $D$ is the detokenizer.

In the first turn, when the prompt $p_1$ is provided, the token sequence $s_1$ is generated. This sequence can then be preserved and temporarily stored. In the second turn, when prompt $p_2$ is added, the new token sequence $s_2$ is generated. To combine the two motions, we concatenate the sequences into $s = (s_1, s_2)$, which is then decoded into the final motion sequence.

While the motions themselves are not directly preserved, the token sequences are preserved. Since $s_1$ is preserved and used in the concatenation, the corresponding motion $m_1$ will remain almost identical to its original form. Thus, while the motion sequence is extended with new content, the original motion is effectively preserved in the final generated sequence.

On the other hand, if such preservation is not desired, a new $s_1$ can be generated in the second turn, which might result in a different motion and cost additional computation.

Are the motions generated by MotionLLM concatenated and sent to the motion detokenizer after each prompt?

If the user asks for generation, GPT-4 may call MotionLLM multiple times, and MotionLLM will generate corresponding motion tokens. After generating all tokens, they are concatenated and decoded together by the decoder.

We thank you for your thoughtful reviews and suggestions.

How does MotionLLM.generate or MotionLLM.caption work—are they called every time during inference, or just when needed?

GPT-4 will decide whether to call MotionLLM in each turn. So they will only be called when needed.

Interpenetration

We would like to clarify that the apparent issues of body interpenetration and lack of physical constraints arise due to the graphical rendering process, which is a separate step after motion generation. Specifically, after Motion-Agent generates the joint representations, they are converted into motion capture files. We then map the generated motion data to a randomly selected 3D avatar in BLENDER (graphics package) for visualization purposes.

However, 3D avatars come in various shapes and sizes, and while the generated joint movements follow physical constraints, the mapping of these joints to the avatar’s mesh may cause interpenetration, especially when the avatars have differing proportions. We acknowledge that this graphical issue is not ideal, but it stems from the complexities of avatar's discrete mesh geometry for skinned animation, rather than the motion generation process itself.

These graphical or visualization inconsistencies may be resolved by relevant future graphics research, by incorporating better avatar normalization and collision handling techniques. We believe this will reduce the likelihood of interpenetration and improve the overall visual quality of the generated motions.

Different genders/body types

They are arbitrarily chosen just for the purpose of visualization. The raw output consists of just the coordinates of the pertinent joints.

Motion In-between

We believe Figure 5 does not accurately represent the other "in-between" task that predicts motions (that best interpolate) between keyframes. Instead, we illustrate motion composition, where direct concatenation may result in unnatural transitions. Motion-Agent addresses this problem by generating a semantically coherent motion (specifically, by using GPT-4 to create intermediate motion descriptions) to smoothly connect adjacent segments.

Notwithstanding, we heeded the suggestion and finetuned MotionLLM to perform the "in-between" task, and the results are provided below. However, we would like to emphasize that this is not the primary focus of our paper. Additionally, we believe it is unnecessary to empower MotionLLM with these capabilities, as GPT-4 is more effective at generating new motion captions for both prediction and in-between motion generation.

Comparison of motion prediction and motion in-between on part of AMASS dataset.

Constrain length for comparison

We appreciate the reviewer’s suggestion to constrain the motion length for a more consistent comparison with other methods. However, we respectfully disagree with imposing a fixed motion length as a constraint for evaluating generated motion. The length of motion inherently varies based on the type of action being generated—some motions are brief, lasting less than a second, while others require longer trajectories for accurate representation.

Imposing a fixed temporal window, similar to those used in motion recognition tasks, would not be ideal. Although some datasets use fixed-length videos for motion recognition, they often introduce problems such as irrelevant short motions or forced cropping of longer motions, which can compromise the evaluation process. Thus, we believe that allowing variable motion lengths provides a more realistic and meaningful comparison across different methods. Moreover, the ability to generate variable-length motions should be viewed as an advantage rather than a disadvantage, as it demonstrates greater flexibility and adaptability in motion generation.

More modalities

Indeed, GPT-4o is renowned for its multimodal capabilities, and we believe it is feasible to integrate visual and auditory inputs into our framework, potentially enhancing both performance and application scope.

In our initial experiments, we have observed promising results:

Visual Input: When provided with an image of a volleyball player along with a prompt to generate a sequence of motions corresponding to the picture, Motion-Agent successfully produces a series of actions such as receiving and spiking. This demonstrates the framework's ability to interpret visual cues and translate them into meaningful motion plans.

Audio Input: Similarly, when given an audio description of the desired motion, GPT-4o accurately transcribes the voice input into textual prompts. Motion-Agent then generates detailed plans by invoking MotionLLM based on these prompts. This showcases the potential for auditory information to guide motion generation.

Integrating vision and audio modalities can significantly enhance the model's performance by providing richer contextual information, leading to more accurate and diverse motion outputs. It also opens up new avenues for applications in areas like virtual reality, human-computer interaction, and animation, where multimodal inputs are valuable.

We recognize the great potential in incorporating additional modalities and are committed to exploring this direction in our future work.

We thank you for your thoughtful reviews and suggestions.

Efficiency of Motion-Agent

The Motion-Agent framework focuses on conversational motion generation by combining MotionLLM for motion-language translation and GPT-4 for conversational control. Instead of building other task-specific datasets that restrict to a finite set of problems using LLMs, we use LLMs for such tasks without dataset collection and training, and hence more flexible and efficient.

For example, MotionChain (Jiang et al., 2024) depends on manually constructed datasets. A single-turn conversation dataset (95k training entries and 18k testing entries) is created by sampling HumanML3D with GPT-4 to form prompt-response pairs. However, this dataset is limited to a narrow range of tasks and queries, restricting its adaptability to out-of-domain scenarios. For multi-turn dialogues, MotionChain samples a large multi-turn dataset from the single-turn dataset resulting in inefficiencies. Instead, Motion-Agent leverages existing motion-language datasets such as HumanML3D for translation tasks, while employing GPT-4’s generalization capabilities for conversational control. We believe our approach can better leverage existing data, and thus significantly enhance efficiency and scalability.

Moreover, Motion-Agent is training-efficient for generating longer motions. For example, instead of training on all permutations of short motions, MotionLLM is trained to generate each motion independently, allowing them to be combined as needed and appropriate. This design avoids unnecessary computational overhead and enhances the framework’s adaptability and scalability.

Generalization capabilities of Motion-Agent

MotionLLM is trained on a dataset containing approximately 20K motions, covering the majority of simple and atomic human motions. When faced with complex, abstract out-of-domain prompts, GPT-4 can decompose them into simpler, in-domain motions, so that Motion-Agent can generate the motion by producing each component individually and then smoothly concatenating them. In contrast, other models remain constrained to the domain of their training data.

Generation length

In practice, the length of motion generation is constrained by several factors, including the context length of the conversational LLM (e.g., GPT-4) and hardware limitations, such as memory and processing power. However, theoretically, the decoder itself has no restrictions on the motion length and can ensure smooth transitions between motions. Since MotionLLM can be invoked repeatedly, it is possible to generate motion sequences of virtually unlimited length. This means that, in theory, Motion-Agent can achieve infinite motion generation by repeatedly invoking MotionLLM to generate motion tokens and concatenating them into longer sequences.

Comparisons with state-of-the-art RVQ-VAE models

We provide an additional ablation study, please refer to the general response.

Additional MotionGPT benchmarks

MotionLLM does not explicitly address motion composition tasks, as these are primarily handled by the planner agent, GPT-4. The planner generates intermediate descriptions of motion, which are then interpreted by MotionLLM to produce the final composition. While MotionLLM can indeed be fine-tuned on additional tasks, such as motion prediction and in-between generation (as demonstrated below), and has shown competitive performance in these areas, our work does not focus on such tasks. We believe that motion composition is better suited to the planner agent, as this approach provides greater flexibility and supports more intuitive conversational editing. MotionLLM’s core design is focused on translational tasks, where it has already demonstrated strong performance in both motion generation and captioning.

Comparison of motion prediction and motion in-between on part of AMASS dataset.

LoRA vs Full fine-tune

Please refer to the general response.

Multi-human and human-environment interaction

Though we leave this as future work which is not yet adequately explored, we can provide some insight into how to extend Motion-Agent to handle such scenarios. The core idea is to introduce additional agents (to work with the current MotionLLM agent), to assist these tasks.

For multi-human motion generation, we have presented a simple example in Appendix A.3. The idea is to decompose the interaction into two individual motions, each generated separately. More specifically, another coordinating agent can be introduced to synchronize these individual motions or enforce more physical constraints.

For human-environment interactions, the approach is similar. Additional agents can be introduced to enable the system to understand the scene context for track objects, ensuring that the motions align with environmental changes or object movements.

Though not fully explored, this agent-based approach allows for scalability and adaptability to more complex scenarios involving multiple humans or environmental factors.

Are there limitations regarding the diversity or realism of the generated motions when handling highly complex or extended sequences?

When given highly complex or long prompts, such as movie scripts describing complex action sequences and responses, Motion-Agent may not always produce satisfactory results to the users in a single turn. However, this is exactly where multi-turn capabilities may help. As the generation process can be iterative, one can refine and edit the output through multiple turns, enabling the generated motion to satisfy the user’s request for fast and easy convergence to final outcome, as demonstrated in Figure 1.

We thank you for your thoughtful reviews and suggestions.

For the ablation study on different tokenizers, please refer to the general response.

Standard benchmarks

The success of Motion-Agent relies on the robust bi-directional translation capabilities of MotionLLM and the well-known conversational and generalization strengths of LLMs. As presented in the paper, we have compared MotionLLM with SOTA models, demonstrating that MotionLLM performs competitively in bi-directional generation tasks. For conversational tasks involving Motion-Agent, the generated motions are derived from concatenating MotionLLM's outputs without additional training. To our knowledge, no standard benchmarks currently exist for such complex, flexible, multi-turn conversational motion generation tasks.

Different LLMs

We have conducted experiments with different LLMs in simple two-turn conversational prompts, detailed in Appendix A.2. In these experiments, we substituted GPT-4 with open-source models (Llama, Gemma, and Mixtral) to examine any impact on performance. Our observations indicate that the general performance remains consistent, suggesting that Motion-Agent’s core functionality is robust across different LLM choices. We attribute this consistency to the simplicity of the required understanding and planning tasks, which are not overly demanding for modern LLMs. Additionally, models having been trained for JSON output may better generate the correct format. We welcome suggestions for additional evaluation approaches to further explore the influence of different conversational LLMs on Motion-Agent.

GPT-4 and other conversational LLMs are used without modification, and we plan to open-source our prompts and implementation in the future. Both the API and the web-based versions of these models are sufficient for reproducing our results.

The following provides more comparisons between GPT-4 and Llama 3.1 8B. Our findings indicate that a lightweight, open-source LLM can achieve results comparable to GPT-4. GPT-4 may tend to produce more accurate and reasonable motion descriptions and slightly better generation results with MotionLLM, but the observed differences are relatively minor:

A football player is doing some warm-ups on the pitch.

GPT-4:

1. MotionLLM.generate(`A person stretches their legs.');
2. MotionLLM.generate(`A person performs jumping jacks.');
3. MotionLLM.generate(`A person jogs in place.');
4. MotionLLM.generate(`A person kicks the ball lightly as a warm-up.')

Llama 3.1 (8B):

1. MotionLLM.generate(`A person runs in place.');
2. MotionLLM.generate(`A person jumps up and lands.');
3. MotionLLM.generate(`A person stretches their arms overhead.');
4. MotionLLM.generate(`A person bends to touch their toes.')

A person seems to be shaking something like a spray can, then starts punching something, and raises both arms above the head while stepping to the left.

GPT-4:

1. MotionLLM.generate(`A person shakes a can.');
2. MotionLLM.generate(`A person punches forward repeatedly.');
3. MotionLLM.generate(`A person raises both arms above their head.');
4. MotionLLM.generate(`A person steps to the left while raising both arms above their head.')

Llama 3.1 (8B):

1. MotionLLM.generate(`A person shakes something like a spray can.');
2. MotionLLM.generate(`A person punches something.');
3. MotionLLM.generate(`A person raises both arms above the head.');
4. MotionLLM.generate(`A person steps to the left.')

Computational requirements

MotionLLM uses Gemma2-2B as its backbone, one of the smallest publicly available LLMs, which enables it to run efficiently on a single consumer-level GPU without requiring additional optimizations such as quantization.

For the conversational LLMs, if APIs are used, the computational overhead is handled remotely. We have also experimented with other models such as Llama-8B and Gemma2-27B, both of which can be deployed locally on servers with reasonable computational resources.

The inference time of Motion-Agent aligns with that of the underlying LLMs, and can benefit from typical LLMs optimizations. Given the generally acceptable speed of GPT-4 and other LLMs, our system's inference time is consistent with user expectations for real-time interaction.

We thank you for your thoughtful reviews and suggestions.

GPT-4 for decomposition

When the prompt involves a request for a sequence of actions—such as "do A, then B, then C..."—Motion-Agent generates each individual short motion (e.g., A, B, C), which has been trained on the HumanML3D dataset. This dataset contains motion sequences of at most 10 seconds, covering a wide range of simple, "atomic" motions suitable for our purposes. During training, GPT-4 is not involved, as decomposition only occurs during inference.

In long motion generation, GPT-4's role is analyzing and decomposing complex prompts into manageable segments. For example, in Figure 3, second turn, given the prompt, "Generate another motion where a person is kicked down and then stands up to fight back by slapping and kicking", the corresponding plan generated by GPT-4 is:

1. MotionLLM.generate('A person gets kicked down to the ground.')
2. MotionLLM.generate('A person pushes themselves up from the ground to stand up.')
3. MotionLLM.generate('A person slaps with their hand.')
4. MotionLLM.generate('A person delivers a kick.')

Similarly, for some abstract prompts such as "floor exercise in artistic gymnastics" in Figure 3, GPT-4 interprets this prompt as a sequence of actions consisting of cartwheels, backflips, handstands, etc. By generating and concatenating these short motions, Motion-Agent effectively produces coherent, longer motion sequences.

Add citations

We appreciate the suggestion and we will include these papers in the revision.

Thank you for your comments. We understand the concern about the potential gap between GPT-4 and MotionLLM. Since MotionLLM was originally pre-trained as an LLM model (i.e., GEMMA-2), it possesses strong language understanding capabilities. In our experiments, we observed that GPT-4 and MotionLLM communicate smoothly. As demonstrated, our Motion-Agent achieves state-of-the-art accuracy both quantitatively and qualitatively. We believe the issue you mentioned would be significant if MotionLLM had been trained from scratch.

Additionally, we include few-shot examples in the instructional prompt given to GPT-4 to provide an overview of “simple and atomic” motion descriptions. As shown in the examples, rather than merely parsing user prompts like a small, lightweight LLM might, GPT-4 can generate plans with deeper reasoning. These outputs are more akin to expressions found in the HumanML3D dataset. To our surprise, GPT-4 can understand complex and abstract motions by explaining them in simple terms.

Nevertheless, we agree that the HumanML3D dataset might not contain all types of atomic actions. Consequently, there is a possibility that an instruction from GPT-4 cannot be executed accurately by MotionLLM. We believe this issue can be resolved with the expansion of human motion datasets.