Versatile Motion-Language Models for Multi-turn Interactive Agents

• One of the key concerns is the dataset quality. 1) Are all inputs of GPT-4o in Fig. 2a are text-only? If so, how do authors generate the "user input" shown on the left of Fig. 2a? 2) As shown in Fig. 2b, how can intergen generate two motions with the single prompt (see the generated result of caption 1)? 3) The GPT-4o always exists some hallucinations, such as "sorry/ I don't know", which result in the failure cases in your dataset. Has the data been checked by researchers one by one? Besides, the motion data here is also generated, which reduces the data quality for training.

• The technical soundness.

  • The authors use the RVQ to compress motions. As stated in the main text, the motion tokens are contacted with texts into the LLM jointly. I have some concerns about this. If $\mathbf{z}$ is one compressed token by the encoder, and it is quantized by $\{{\bf z}_1, {\bf z}_2\}$, aka. $\mathbf{z} = {\bf z}_1+ {\bf z}_2 + \delta$, where $\delta$ is the quantization error. According to the statement in the paper, the method feeds ${\bf z}_1, {\bf z}_2$ directly into the LLM. However, the method can feed ${\bf z}_1+ {\bf z}_2$ into LLM directly, which is more token efficient and equal to the method proposed by authors.
  • A lot of research verifies that the adaption between vision tokens and language tokens is needed. For example, MotionLLM and LLaVA series. Therefore, the technical design here is surprising. An ablation is needed.

• No video demo provided for animation is not acceptable.

• The multi-round interaction is one of the main claims of the paper, which is not thoroughly verified by experiments. The reasoning cases in Figure are limited. The results of more interaction turns should be provided. Besides, As shown in the Fig. 4, how does your method insert the motion into the text sequence? How do users determine the position to insert? The paper lacks a clear statement for user interaction.

• It is not clear about the detailed evaluation protocol in Table 1. What is the test set (size)? As shown in the appendix, the score ranges from 0 to 10. However, the scores shown in Table 1 are quite poor. What is the key contribution of the evaluation protocol claimed in L101-102?

• The authors have fundamental errors in discussing related works. In L125-126, why does Chen et al. (2024a) include speech and music? Besides, this is the most related work about motion-LLM that authors should discuss the difference.

• The writing issue. In L142-143, "Xu et al. (2024b)" should be "(Xu et al. 2024b)".

This work lacks unique contribution and a lot of details are not clear or well-discussed. Therefore, I vote to reject clearly.

1. Lack of qualitative results. The biggest concern is the lack of demo video in the submission. I didn't see either video or the link to videos. I could see many numbers in tables, visualization in figures. Nevertheless, without video demonstration, it's hard to evaluate the real performance, especially for interaction synthesis.
2. The quality of dataset is unknown. Since the inter-MT^2 is based the generation of interGEN, it is hard to guarantee the overall quality of the interaction dataset. Whether it's faithful to the text, or if the interaction is natural. Some examples and stats about this dataset will be appreciated. 3. The effectiveness of modeling motion token using LLM is unknown. It is questionable if it's really effective to merge the motion tokens and text tokens together, given that motion token and text token are substantially different modalities.
4. Some typos. In the paper, "TM2T" is written as "T2MT" in several places, e.g., Figure 6.
5. Baseline implementations are missing. TM2T and MotionGPT are both designed for single person motion generation. It is not clear how they are adopted in current context. For example, have you changed the model architecture? Some details regarding the implementation are appreciated.

The architecture of VIM is pretty similar to LLaVA, so nothing new here. I’m more interested in the multi-turn conversation dataset. Could you show more examples of the videos generated by the model? Also, does the text generated by GPT-4o not need any data cleaning? Most of the motions in this dataset are generated by the text2motion diffusion model, so the quality obviously can’t be guaranteed. As for InterGEN in Figure 1b, can it generalize well enough to the text generated by GPT-4o? I have my doubts.

1. While the unified motion-language generative model is a notable concept, the method is overly conventional within the multimodal large language model (MLLM) field, lacking technical contributions specifically designed for interactive human motion.

2. The experiments on the proposed dataset in motion-related tasks, as shown in Table 3, are insufficient. The paper falls short of demonstrating the impact of synthetic conversational data on motion-related tasks, to validate the effect of the proposed dataset.

3. In Table 4, the ablation studies on the motion tokenizer require comparisons with state-of-the-art RVQ-VAE models, such as MoMask [1]. Given that the proposed approach also utilizes RVQ-VAE, such comparisons would provide a more comprehensive understanding of the model’s relative effectiveness.

4. The paper provides only static visualizations of the model’s results, which limits the clarity of the motion generation process. Including dynamic visualizations would offer a more intuitive and detailed depiction of the generated motions.

[1] MoMask: Generative Masked Modeling of 3D Human Motions