Scaling Large Motion Models with Million-Level Human Motions

Dear Reviewer,

Thank you for your thoughtful review and positive feedback. We have carefully considered your questions and suggestions and provide our responses below. Please let us know if you require further clarification.

Response to: Insufficient visualizations in supplementary material, video blurring, motion quality, and deformation issues.

• Regarding visualization quantity and video blurring:

  Thank you for your feedback. Due to conference supplementary material size constraints (typically 100MB), we were limited in the number of visual examples we could include and had to compress video resolution. To better demonstrate MotionLib’s richness, we plan to release a dedicated website with additional high-resolution examples upon final publication.

• Regarding motion quality and deformation:

  We acknowledge that motion data extracted and refined from web videos may still exhibit imperfections (e.g., deformations), even after RL optimization. However, we also feel more positive since our motion model is significantly enhanced using such motion data, demonstrating the scaling law in this task (Table 2, 4). This suggests the larger potential of our MotionLib dataset. As we keep refining this dataset using additional strategies, the gain of performance and generalization could be more significant. Considering this, we consider the building of MotionLib as a long-term iterative process, and our future work can incorporate stricter filtering or advanced algorithms, in addition to current steps we adopted (e.g., 3D keypoint optimization, physics constraints, and RL strategies; see Appendix B.2).

  In addition, we suggest the issue of data quality and quantity is a trade-off for large-scale pretraining. Motivated by LLM's pretraining, we believe combining large-scale pre-training with motion fine-tuning on high-quality subsets can further mitigate quality issues while preserving scalability benefits.

Response to: Questioning the effectiveness of the claimed motion representation improvement (foot sliding, pose jittering).

It is important to emphasize that MotionLib’s primary contribution lies in advancing the semantic understanding and generalization capabilities in motion generation, rather than eliminating physical issues (e.g., foot sliding or jittering) entirely. Quantitative improvements in R-Precision, MMDist (Table 3), and OOD generalization (Table 4) demonstrate that our Puppet model achieves superior alignment between text instructions and nuanced human motion, validating the semantic gains we expect. In addition, although MotionLib may include more motion noises compared to datasets like HumanML3D, it excels in the quality and quantity of motion descriptions. After investgation, we notice the texts of HumanML3D, MotionX are hightly duplicate and contain a large amount of noisy texts (e.g.,"moving the hands speaking some thing", "the person's eyeglasses pass"). Instead, our MotionLib avoids these issues by incorporating more clean and diverse texts.

Our methodology follows a "first scale up, then refine down" philosophy.

That means, while physical issues (e.g., foot sliding and jittering) persist in some generated motions —— an expected byproduct of large-scale automated processing — they do not negate the foundational progress of motion semantics. Such trade-offs are inherent to data-driven research at scale. Crucially, MotionLib establishes a robust semantic foundation for general-purpose motion models, enabling future work to address physical fidelity through: (1) further data quality improvements on our MotionLib (e.g., using stricter filtering), (2) post-training refinement (e.g., fine-tuning on high-quality subsets).

We posit that prioritizing scalable semantic understanding is pivotal; physical issues can then be iteratively resolved without compromising the model’s generality.

Thank you again for your valuable feedback. We hope our response clarifies your concerns.

Dear Reviewer,

Thank you for your thoughtful review and positive feedback. Below are our responses to your questions and suggestions. Please let us know if you require further clarification.

Response to: the discussion of LMM [1] and Dataset Comparison

We appreciate you highlighting the relevant work of LMM [1]. Below, we clarify the scale and unique contributions of MotionLib in comparison:

• Dataset Scale: The term "million-scale" in our paper refers to the number of motion sequences. MotionLib contains 1.2 million (1.2M) motion sequences (as in Table 1), totaling 137 million (137M) frames. In contrast, LMM’s dataset comprises 320K sequences and 100M frames. Thus, MotionLib is currently the largest motion generation dataset in both sequence count and total frames. We will explicitly add the frame count to Table 1 for clarity.
• Key Distributions:
  • Task Focus: MotionLib is optimized for text-to-motion generation, featuring 2.48M fine-grained text-motion pairs (including fine-grained part-level descriptions). LMM, meanwhile, targets multi-modal inputs (text/image/audio).
  • Performance: On HumanML3D, Puppet outperforms LMM in R@1, R@3, and MMDist, showing better text-motion alignment. Notice that our FID performs worse than LMM. We attribute this to the difference of used motion tokenizer. As shown in Table.7, our 2D-LFQ simply performs competitive to vanilla VQ, but showing large improvement as the data become larger and diverse. This means that our lookup-free tokenizer has more advantage when facing large-scale scenarios and thus requires more training data. It's also important to note that Puppet exhibits a strong agent capability to follow the user's instructions empowered by the LLM. Among existing LLM-based generalist models, Puppet excels these models in performance. ||R@1|R@3|MMDist|FID| |-|-|-|-|-| |LMM|0.525|0.811|2.943|0.04| |Puppet-LFQ|0.528|0.82|2.875|0.141|

We will expand the discussion of LMM in the revision, emphasizing MotionLib’s scale, annotation richness, and task-specific advantages.

[1] Large Motion Model for Unified Multi-Modal Motion Generation. ECCV 2024

Response to: Qualitative Comparisons with MotionGPT & T2M-GPT (Experimental Designs)

Thank you for this useful feedback! We agree qualitative examples are important. We will add qualitative comparisons of motion results generated by our Puppet versus T2M-GPT and MotionGPT for the same text prompts in the subsequent revision.

Response to: The Foot Sliding Issue in Dataset (Supplementary Material)

Thank you for raising the issue of foot sliding. Indeed, automatically estimating motion data from large-scale web videos inevitably introduces noises, including physical implausibilities like foot sliding.

The most important reason we choose to down-weight these low-quality samples rather than completely discarding them is: we adopt two-stage training to address the impact of noisy data:

• Motion pretraining — Information Preservation: In a million-scale dataset, even noisy samples might contain valuable motion patterns or contextual information. Discarding them completely could lead to the loss of potentially useful information.
• Motion instruction Tuning — Future Optimization: More importantly, after obtaining a good base model through large-scale motion pre-training. High-quality subsets can later refine the model via instruction tuning, further improving generation quality and physical realism.

Such approach is motivated by the great success of LLMs. In the pretraining stage of LLM, a large amount of corpus is quite dirty. However LLM can still gain massive knowledge from it and develop good responsive ability through instruction-tuning.

In addition, we have other considerations to incorporate these part of data:

• Maximizing the Scale Utilization: Our goal is to explore the potential of large-scale data training. Retaining all samples (even if down-weighted) maximizes the utilization of diversity brought by the data scale.
• Down-weight is a Common Strategy: When dealing with large noisy datasets, reducing the weight of noisy samples via the loss function is a mature and effective strategy that allows leveraging data scale while mitigating the negative impact of noise.

As a summary, we believe that pursuing an increase in data scale first, followed by continuous improvement of data quality via further refinement and filtering, to enhance model performance through down-weighting and subsequent fine-tuning, is an effective pathway for developing large motion models at this stage.

Response to Other Comments (e.g., adding total frame count and correcting typo)

Thank you for pointing them out.

• We will explicitly add the total frame count of MotionLib (approx. 137 million frames) in the revised Table 1 or relevant text.
• We will correct "an compact" to "a compact" in line 30 in the revised version.

Hoping our response clarifies your concerns.

Dear Reviewer,

We appreciate your valuable feedback. Due to space constraints, we only provide concise responses below but would present more during discussion. Please let us know if any clarification is needed.

W1: WHAM accuracy & dataset validity

While no motion estimation algorithm is perfect, our data pipeline (3D keypoint optimization, physics constraints, RL tuning — App B.2) refines initial estimates. Despite noises, Table 2 & 4 show large-scale motion pretraining improves generalization by exposing models to diverse motions. MotionLib provides a foundational resource especially high-quality texts. Its quality can be improved via future algorithm updates and filtering. Scale is critical given current data scarcity.

W2: Data refinement details (RL, simulation, slipping handling)

We refine motion sequences using the PHC policy (Luo et al., 2023) in IsaacGym, which achieves high tracking accuracy (97.1% on AMASS, 95.7% on Human3.6M). The process involves:

• Inputting video-extracted motion to PHC for physics-compliant tracking (balancing, reducing jitter/sliding).
• Using PHC to generate physically plausible motions in IsaacGym, with termination conditions (e.g., early termination for balance loss) flagging low-quality sequences.
• Downweighting flagged sequences during Puppet model training—a common practice for handling noisy data—to prioritize high-quality samples.

W3: 2D motion quantization is already used

While ParCo and MoGenTS explored 2D motion quantization, the core innovation of our MotionBook (including 2D-LFQ) lies in its lookup-free (LFQ) mechanism and its application in the context of large-scale training. Unlike traditional VQ (limited to small codebooks, e.g., 512/1024 codes), LFQ avoids codebook collapse and supports 16K+ codes, enabling scalable learning on our million-scale MotionLib dataset. This addresses a key bottleneck in prior works, which were not designed for such diversity and scale.

W4: T2M model uses a standard AR architecture, lacking novel design

While Puppet uses a standard LLM architecture, our key contribution is the first Scaling Law study for motion generation, systematically analyzing data/model size impacts—similar to foundational VLM work like LLaVA, which also relied on standard architectures.

In text-to-motion, we argue that constructing large-scale motion data with rich, hierarchical text descriptions (MotionLib) and designing effective motion encodings (MotionBook) to bridge text-motion alignment are themselves significant contributions. Our endeavour in data construction and labeling aims to establish a foundation for future large motion models.

W5: Missing comparisons with some recent methods

Thank you for highlighting these recent works. For a more comprehensive comparison, we include their publicly reported results on HumanML3D. Some methods (e.g., LaMP, MoGenTS) were concurrent or very recent (within 4 months) and were not initially compared.

Notably, current T2M methods include specialist models optimized on specific datasets and LLM-based generalist models (aiming for broader instruction/task generalization via LLMs), like our Puppet. Puppet excels among generalist models (Table 3) and remains highly competitive on R@1, R@3, and MMDist compared to specialist models.

W6: High FID in OOD experiments; suggestion to evaluate on HumanML3D (HM3D) using MotionLib

Higher OOD FID Explanation: The UNSEEN-90K test set comprises 11 subsets with substantial distribution shifts from training data, including synthetic data, activity-specific datasets, and varied capture environments. The elevated FID in this challenging OOD setup is expected, similar to observations in other motion tasks (e.g., music-to-motion in LMM). More importantly, Table 4 validates that large-scale training with diverse, large-scale MotionLib data (vs. HM3D or MotionX alone) significantly boosts OOD performance.

HM3D Evaluation: Thanks for the suggestion. We remove all HM3D data from MotionLib and trained models on varying scales of remaining data (0.6M–1.2M samples), then evaluate on the HM3D test set. Results are shown below:

W7: Lack of clarity for train/eval settings in Table 2 ("retrain", "pretrain").

These terms are distinct in our context:

• Retrain: Refers to training the evaluation model (motion autoencoder) separately for each benchmark (MotionX and MotionLib), following HM3D paper’s architecture. This ensures metric fairness (FID, R-Precision) across datasets.
• Pretrain: Describes Puppet’s training:
  • Initialize with public LLM weights (GPT-2, LLaMA).
  • Extend vocabulary with motion tokens.
  • Continue pretraining on target datasets (HumanML3D/MotionX/MotionLib subsets) via autoregressive loss.

Dear Reviewer,

Thank you for your thoughtful review and positive feedback. We have carefully considered your questions and suggestions and provide our responses below. Please let us know if you require further clarification.

Response to: Questions about UNSEEN-90K Dataset in OOD Experiments

In our OOD experiments (Table 4), the UNSEEN-90K testing set is constructed by excluding 11 subsets (~90K samples) from the full MotionLib dataset. The remaining data (primarily Motion-X and web-derived data) serves as the training set.

The excluded subsets are intentionally diverse to ensure significant distributional shifts from the training data, enabling a robust evaluation of OOD generalization. These subsets include:

• Synthetic data: gta_human [1], bedlam [2] (distinct from web-derived training data captured in the real world).
• Domain-specific activities: FIT3d [4] (fitness), RICH [5], ARCTIC [11] (human-object interactions), EgoBody [7] (social interactions).
• Diverse environments: PoseTrack [3], MPI-INF-3DHP [6] (in-the-wild settings), Human36M [8], CHI3d [9], KIT [10] (daily activities captured in the lab).

This selection strategy demonstrates that the UNSEEN-90K testing set exhibits substantial distributional shifts compared to our training data, allowing us to rigorously assess the model’s OOD generalization. As shown in Table 4 in our main paper, models trained on MotionLib outperform those trained solely on HumanML3D or Motion-X, highlighting the benefit of large-scale, web-sourced motion data.

Regarding the examples. Thank you for this constructive suggestion. We will incorporate additional visualization examples in the appendix in our revised manuscript.

[1] Playing for 3D Human Recovery. TPAMI 2024

[2] BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion. CVPR 2023

[3] PoseTrack: A Benchmark for Human Pose Estimation and Tracking. CVPR 2018

[4] AIFit: Automatic 3D Human-Interpretable Feedback Models for Fitness Training. CVPR 2021

[5] Capturing and Inferring Dense Full-Body Human-Scene Contact. CVPR 2022

[6] Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision. 3DV 2017

[7] EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices. ECCV 2022

[8] Human3.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments. TPAMI 2013

[9] Reconstructing Three-Dimensional Models of Interacting Humans. CVPR 2020

[10] The KIT Motion-Language Dataset. Big data 2016

[11] ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation. CVPR 2023

Response to: The Limitation of MotionBook’s Global Feature Extraction

Thank you for your question. You raised a valid concern regarding MotionBook’s use of 2D convolutions for motion encoding. We clarify our design choices below:

1. Division of Roles

• Motion Tokenizer, which focuses on quantizing continuous, high-dimensional motion features into discrete, low-dimensional token sequences, effectively capturing local spatio-temporal features and achieving a compact representation. 2D convolutions are well-suited for this due to their computational efficiency and inductive bias for local patterns.
• LLM backbones, which handles global feature modeling and long-range dependencies within the input token sequence via Transformer self-attention, addressing the limitation of CNNs for global context reasoning.

This hybrid design combines the strengths of CNNs (local feature extraction) and Transformers (global modeling), ensuring efficient and effective motion encoding.

2. Why CNN Over Alternatives (e.g., RNN or Transformers)?

• Fair Comparison: Most prior motion tokenizers (e.g., VQ, RVQ) use CNNs, therefore we maintain consistency for fair benchmarking.
• Empirical Findings: We experimented with Transformers but observed no performance gain, likely due to the limited data amount and lower resolution compared to vision tasks where ViTs excel.

We appreciate your insightful feedback and hope our responses address your concerns. Thank you again for your time and constructive comments!