MMG-VL: A Vision-Language Driven Approach for Multi-Person Motion Generation

1. The analysis and discussion of the dataset in the paper are insufficient. Maybe you can provide more details such as data distribution, annotation process, or specific dataset statistics.

1. There is a lack of discussion and comparison with prior works on language-guided human motion generation in 3D scenes[1,2]. This paper only incorporates 2D scene information into the generation process, which merely informs the model of where to place the motion. The motivation behind the current setting needs further clarification.

2. The proposed model lacks novelty in design, as it merely combines existing components. It uses a VL module to interpret input descriptions into instructions, which then guide motion generation through an MDM model. However, the model does not incorporate detailed scene information, relying solely on interpreted instructions, which would fail to generate results that align accurately with the scene.

3. The proposed model appears to perform significantly worse than existing methods on R-Precision, FID, and MM-Dist metrics, so it is unclear why the authors claim "it still demonstrates competitive performance."

4. The paper lacks essential details, such as training details and the construction details of the HumanVL dataset. Please refer to the following questions.

[1] Wang et al., HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes. NeurIPS 2022.

[2] Yi et al., Generating Human Interaction Motions in Scenes with Text Control. ECCV 2024.

W1.Insufficient related work review

1. "FreeMotion: A Unified Framework for Number-free Text-to-Motion Synthesis" and "CrowdMoGen: Zero-Shot Text-Driven Collective Motion Generation" both perform text-to-nPerson motion synthesis and utilize large models for text-level design. These two paper should be cited and the differences between them and this paper should be clarified.

W2.Dataset construction has some problems.

1. Lack of human-human interaction. Although this paper have placed actions of 2 to 7 people in the room, from the content and materials provided, I don't seem to find any interaction actions between people. If there is no interaction between people, then what is the difference between this and single-person actions?
2. MDM is a model for generating single-person actions and does not consider how to interact with the scene. If human body behaviors are constructed through MDM, how does it interact with the scene?
3. Does the dataset only contain images of the scene? If there is only images and no 3D object representation, how to put motion into the scene?

W3.The model implementation lacks many details.

1. Why leverage a LoRA to fine-tune a VLM? Since the data scale is limited, will the fine-tuning process cause the VLM to overfit to limited action descriptions, resulting in the loss of generalization ability of the VLM?
2. I am worried that if a full-image are used to extract the viusal feature, how can we ensure that the generated actions can interact with the correct object in the scene?
3. What is the difference between the input representation and the representation provided by HumanML3D?
4. How to ensure that there will be no collisions or interpenetrations between the actions of multiple people generated by only using VLM to extract image features?
5. What is the specific fusion process of $v_{feat}$ and $l_{feat}$ at line 265?
6. Line 279 mentioned that the motion generation process for each individual is based on their respective instructions. Then what is the difference from single-person action generation? Why can this paper achieve the generation of actions of more people?
7. From Line 368, this paper conducts full fine-tuning of the MDM. But how to deal with the difference between the fused feature $c$ in this paper and the text feature in MDM?

W4.Lack of experimental details.

1. It is best to provide an ablation experiment to verify the role of LoRA, such as removing the fine-tuning process and comparing the results.
2. It is better to describe the details of evaluation metrics (SQ, SD, CC, EI, MPC, MRC).
3. There are few qualitative results, and the generation effect is not persuasive.

Although the problem this paper focuses on is valuable, this paper only proposes a simple pipeline based on existing methods, the novelty of the proposed method is limited. Here I highlight my major concerns.

1. The novelty of the proposed method is limited. For the two key modules in the proposed method, the VLAI uses existing ViT encoder and LLM, and the SID uses existing human-motion methods. Thus, the proposed method appears to be a combination of existing methods, and lacks substantial innovation.
2. About the modeling of human-scene interaction and human-human interaction. This paper relies only on LLM for scene and text understanding, and does not explicitly model these interactions. Then the upper limit of the method depends on LLM, and the error output of LLM will also affect the performance of the method.
3. The user study is unscientific. In the user study in this paper, only five PhD candidates took part in the experiment. The small number of people in the user study, the lack of details of the user study, etc., will make people question the user study.
4. The performance on the main metrics for human motion generation is bad, like FID, R Precision. These metrics reflect the quality of the generation.

1. Layout and Formatting: The layout is too dense, with minimal spacing before section captions. The authors should consider removing some content or relocating parts to the appendix to restore a more standard layout.
2. Lack of Motivation for Multi-Person Dataset: The rationale for introducing a multi-person dataset is unclear. The proposed method and sample data do not explicitly model interactions between individuals, which could be addressed with iterative single-person motion generation. In fact, the HumanVL multi-person data generation proposed here is performed by generating each person individually with MDM, without accounting for other individuals, reducing the significance of a multi-person setting. The current multi-person data appears to be merely a spatial combination of multiple single-person motion sequences. It would be more interesting if the authors consider modeling inter-person interactions.
3. Unclear SID Module Description: The SID module is not clearly described. In the formulas in Section 3.3.2, it’s unclear how the SID module interacts with the environment; the single motion sequence seems to rely solely on text instruction $c_i$, which doesn’t make sense in this context. The authors should explain how environmental information is incorporated into the SID module.
4. Lack of Experimental Metric Descriptions: There is insufficient explanation of how Diversity, Multimodality, and Multi-modal Distance metrics are calculated.
5. Typos:
   • Line 230: "Dulti-person" should be "Multi-person"
   • Line 278: "f_{split}" should be "f_{MGC}"