HGM³: Hierarchical Generative Masked Motion Modeling with Hard Token Mining

W1) Use of HGM³ as the name of both a component and a whole model (HGM³+HTM).

A) Thank you for pointing this out. We agree that using HGM³ for both the framework and a component could lead to confusion, and we are still working on the name of the overall framework. We will post it during the rebuttal period, meanwhile, we would appreciate it if the reviewer can suggest a few candidates.

W2-1 / Q2-1) Demonstrate differences in applying GAT to a masked model from GraphMotion.

A) Thank you for your question. While the application of GAT itself remains consistent with GraphMotion, our method differs in how the representations obtained through GAT are utilized. Specifically, we condition three weight-sharing masked models on these representations, which distinguishes our approach from GraphMotion. Additionally, we believe our method offers other novel contributions, which are detailed in the General Response.

W2-2 / 2-3) Clarify how motions, actions, and specific elements work within the GAT. / No details on the "twelve types of edges representing various relationships between nodes" in Section 3.3.

A) Thank you for your question regarding the hierarchical levels and edge types in our semantic graph. For further details, please refer to Appendix B, where we elaborate on how the hierarchical graph represents sentences through motion, action, and specific levels. Each level provides unique semantic information for text conditioning, guiding the motion generation process. Moreover, as mentioned in line 267, we also describe the twelve types of edges used in the graph in Appendix B, which represent various semantic relationships between nodes. These edges include role-based connections (e.g., agent, patient) and temporal or locational links, which enhance the contextual understanding of the input text. The Graph Attention Network (GAT) uses these levels and relationships to refine the model’s understanding of nuanced text-to-motion alignment, contributing to coherent and contextually accurate generation.

W3) Figure 4 visualization with more samples or a video supplement.

A) Thank you for your suggestion. To improve clarity, we have visualized additional examples beyond those shown in Figure 4 of the main paper and included the results in Section C.

For better demonstration, we have included the videos that correspond to the motions in Figure 4 as well as heatmaps of the predicted reconstruction loss distribution (normalized to [0,1] where darker color denotes more challenging motion) in the supplementary.

W4) Necessity of the student-teacher architecture in the model (see questions section).

A) We have addressed this question in Q1-2 below. Please refer to that part.

Q1-1) Output of HTM and difference between G(student) and F(student) models.

A) Thank you for your question regarding the HTM and the roles of G(student) and F(student).

Output of HTM:

• The HTM (Hard Token Mining) transformer is designed to identify challenging tokens in the motion sequence by predicting the reconstruction difficulty for each token, rather than directly outputting a mask. The ranking of tokens is then determined by applying an argsort(⋅) operation to these predictions in descending order. This ranking process identifies which tokens are most challenging to reconstruct and guides the masking strategy. We have revised lines 240-241 in the manuscript to clarify this distinction (see the text in red).

Differences Between G(student) and F(student):

• G(student) serves as the loss predictor, responsible for predicting the reconstruction difficulty for each token in the motion sequence. These predictions are used to compute the loss for the student model, and the model parameters are updated accordingly. G(teacher)​​ is then updated via an Exponential Moving Average (EMA) of the G(student)’s parameters, ensuring that the teacher model gradually incorporates the improvements learned by G(student).
• F(student), the masked transformer, is trained to reconstruct the challenging tokens identified through the masking process. It receives masked sequences as input and focuses on restoring the original motion tokens by leveraging contextual text embeddings.

By separating these roles, HTM ensures that G(student) focuses on predicting token difficulty, while F(student) specializes in reconstructing difficult tokens. This combination helps the model concentrate on the most challenging parts of the sequence, enhancing its performance in motion generation.

Q1-2) Motivation for student-teacher architecture and need for an ablation study.

A) The student-teacher architecture is designed to address the instability that arises when relying solely on the predictions of G(student) during training. While it is possible to directly use the reconstruction loss predicted by G(student), this approach can lead to unstable training due to fluctuating parameters early in training. To mitigate this issue, we introduced a G(teacher) that provides consistent guidance.

G(teacher) is updated using EMA (Exponential Moving Average) of the G(student)'s parameters, ensuring that it evolves gradually and remains stable during training. This stability allows the teacher to offer reliable guidance, enabling F(student) to progressively improve its ability to reconstruct challenging tokens.

To validate the necessity of the EMA mechanism, we conducted an ablation study on the HumanML3D dataset by removing EMA updates and directly updating the teacher model. As shown in the attached table, performance declines without EMA, as the teacher model’s parameters change abruptly, leading to unstable and less effective guidance for F(student).

These findings demonstrate that while the model can operate without EMA updates, the EMA mechanism significantly enhances training stability and overall performance, making it a critical component of our architecture.

Q2-1) Differences between integrating Graph Reasoning into a masked model and GraphMotion.

A): We have addressed this question in W2-1. Please refer to that section.

Q2-2) Is Motion-Action-Specific embeddings applied to Residual layers?

A) Thank you for your question. Our design is guided by the distinct roles of each transformer in the model. The masked transformer uses hierarchical graph embeddings for fine-grained, multi-level alignment between text and motion, capturing specific nuances in the input. In contrast, the residual transformer refines the motion sequence globally, focusing on coherence and fluidity. For this, we chose generalized text conditioning with CLIP embedding, which was sufficient for global refinement without hierarchical detail.

W1) Limited Innovation; pipeline resembles Hard Patches Mining.

A) We humbly disagree with the claim that our approach is a replica of HPM (Wang et al., 2023a), and we appreciate the reviewer for a chance to clarify the contribution of our work further. While HTM was inspired by HPM’s targeted masking concept, it has been fundamentally re-designed to address the unique challenges of motion generation:

• Architectural Differences: HTM replaces HPM’s encoder-decoder structure with a streamlined design comprising two transformers—a masked transformer (reconstructor) and a loss predictor (HTM transformer)—tailored for motion token reconstruction.
• Purpose: Unlike HPM, which focuses on representation learning for invariant feature extraction, HTM directly prioritizes reconstruction quality by selectively masking challenging motion tokens.
• Input Consistency: HTM resolves the input mismatch issue in HPM by using consistent unmasked token sequences for both teacher and student models, ensuring reliable and robust masking strategies.

We believe these distinctions demonstrate HTM’s novel contributions to the field. For additional details, we kindly refer you to the General Response, where we discuss HTM’s innovations in depth.

W2 Comparison of Hierarchical Text Graph and CLIP token-level features.

A) We appreciate the reviewer’s insights, which is a setting that we totally agree with. We conducted an experiment on HumanML3D dataset using the entire set of CLIP output tokens in a straightforward manner as conditioning features for text-to-motion alignment, and the results showed a noticeable decline in performance compared to the Hierarchical Text Graph approach. (See the table below.) We believe this is due to several important distinctions between the two methods:

• Hierarchical Structure with Multi-Level Filtering: Our Hierarchical Text Graph organizes text into meaningful levels—motion, action, and specifics—enabling the model to prioritize and leverage motion-relevant attributes. This global-to-local structure allows the model to better understand the text by capturing semantic information at varying levels of granularity to guide motion generation. In contrast, using CLIP tokens in a straightforward manner often introduces noise, diluting the focus on motion-specific features. Our focused hierarchical approach played a crucial role in achieving precise alignment between text and motion, resulting in high-quality motion generation.

• Relational and Contextual Encoding with GAT: Our model uses a Graph Attention Network (GAT) to facilitate information exchange between nodes, allowing each node to refine its understanding based on relationships across semantic levels. This relational encoding was essential for capturing dependencies in motion data, as it enables the model to understand temporal and contextual relationships in the text. CLIP tokens alone, in contrast, do not support this dynamic interaction, which leads to poorer alignment and coherence in the generated motions.

Given these observations, we believe that the Hierarchical Text Graph with GAT provides essential structure and contextual interaction that significantly enhances text-to-motion alignment. These factors are not captured by the straightforward use of CLIP token-level features, as reflected in the performance disparity observed in our experiment.

W3) Well-executed A+B technical report, but limited practical value for real-world applications.

A) We believe that our paper offers more than technical contributions, as it demonstrates significant advancements with practical applicability. Our proposed method not only improves the alignment between text and motion but also generates high-quality, realistic motions, as evidenced by our qualitative results. These results showcase motions that are well-suited for real-world applications such as animation, virtual reality, and gaming. To further illustrate the practicality of our approach, we have added supplementary video materials, which include examples of generated motions. We kindly invite you to explore these materials for further insights into the applicability of our work.

Q1) Include more examples and heatmaps in Figure 4.

A) In addition to the motions shown in Figure 4 of the main paper, we have visualized heatmaps of the predicted reconstruction loss distribution for three additional motions and reported the results in Section C.

The predicted reconstruction loss values are normalized to a range of 0 to 1, with lightness values varying from 0.0 (white) to 1.0 (black) to represent the heatmaps. Darker colors indicate regions predicted to be more challenging to reconstruct. We would greatly appreciate it if you could review the revised paper for the detailed results. As expected, the motions with high variations were highlighted in dark.

Q2) Missing cites

A) We have incorporated discussions of the suggested papers into the related work section. Please refer to the revised text in lines 94 and 97.

One last question: StableMoFusion uses token-level CLIP features, and the R-Precision score is very high (i.e., 0.553), which is quite different from the results in your experiments. Could you explain the possible reasons for this discrepancy?

W1) Not much new knowledge from this paper.

A) We appreciate your feedback, which has allowed us to clarify how our work builds on but significantly extends existing methods. While our work draws inspiration from these approaches, it introduces significant innovations to address the unique challenges of masked modeling-based text-to-motion generation:

Hard Token Mining (HTM):

• HTM reimagines the HPM (Wang et al., 2023a) framework by focusing on motion generation, replacing the encoder-decoder structure with a simplified architecture that directly enhances motion token reconstruction.
• Unlike HPM, which emphasizes invariant representation learning, HTM prioritizes reconstruction quality, introducing input consistency in the teacher-student framework to ensure robust and reliable masking strategies.

Hierarchical Generative Masked Motion Model (HGM³):

• While inspired by GraphMotion(Jin et al., 2024)’s hierarchical graph reasoning, HGM³ introduces unified weight sharing and a single latent space for hierarchical levels. These innovations reduce parameter complexity, improve efficiency, and ensure semantic coherence, addressing limitations in GraphMotion’s separate-level design.

For a detailed explanation of these distinctions, we kindly refer you to the General Response, where we discuss the novelty and contributions of HTM and HGM³ in depth.

W2) Video for motion results.

A) To address the concern, we have added videos in the supplementary. These include motions from Fig. 3 and Fig. 4 in the main paper, as well as Fig. 4 in Section C, illustrating the diverse and accurate motions generated by our model. The videos clearly demonstrate our model's ability to align motions with the input text while achieving higher precision over baseline methods.

W3-1) Clarify text conditioning in line 205: CLIP embedding vs. hierarchical semantic embedding.

A) Thank you for pointing this out. We have clarified this in lines 205 to 208 to prevent any further misunderstanding. As you mentioned, in our model, the masked transformer is conditioned on contextual text embeddings $C$, which represent hierarchical semantic information derived from the input text. This revision has been reflected in the manuscript (in red text) to address this issue.

W3-2) Reason for the residual transformer using CLIP embedding instead of hierarchical embedding.

A) Our design choice was based on the distinct roles that each transformer serves in our model. The masked transformer relies on hierarchical graph embeddings to ensure fine-grained, multi-level alignment between the text and the motion, capturing specific nuances in the input text. In contrast, the residual transformer serves as a broad refinement layer, enhancing the coherence and fluidity of the overall motion sequence. For this purpose, we used generalized text conditioning with CLIP embeddings, which was sufficient for global refinement without requiring hierarchical detail.

Q1) Citations in the pdf do not contain clickable links.

A) Thank you for bringing this to our attention. We identified one non-clickable link in the main paper and have corrected it to be clickable. Furthermore, we merged the supplementary with the main paper and ensured that all citations are now clickable. If this issue persists, we would greatly appreciate it if you could let us know.

W1) Discuss any new theoretical insights or algorithmic innovations beyond the integration of existing techniques.

A) Thank you for your thoughtful feedback regarding the novelty of our contributions. While our work draws inspiration from existing techniques, it extends these methods with innovations tailored specifically for motion generation tasks.

• Hard Token Mining (HTM): Our HTM approach diverges from Hard Patch Mining (HPM) (Wang et al., 2023a) by focusing on motion token sequence reconstruction and eliminates the encoder-decoder structure, and HTM ensures input consistency in the student-teacher framework.
• Hierarchical Generative Masked Motion Model (HGM³): Beyond hierarchical semantic graphs, our model introduces unified weight sharing across hierarchical levels and operates within a single latent space, significantly improving efficiency, scalability, and coherence compared to prior methods.

For further details, please refer to the General Response where we provide an in-depth explanation of how HTM and HGM³ introduce theoretical and architectural innovations that extend beyond existing methods.

W2-1) How is $L_{res}$ incorporated into the overall network loss?

A) As shown in Eq. 1 in Section A.2, the residual transformer is a model that is trained to predict $Y^{i+1}$ given token sequences $Y^0$ to $Y^i$ obtained from the residual VQ-VAE. As it is trained separately from the masked transformer, therefore, $L_{res}$​ is not included in the overall network loss $L$ (which is composed of $L_{rec}$​ and $L_{pred}​$). During the inference process, this separately trained residual transformer sequentially predicts $Y^1$ to $Y^V$ from the masked transformer output $\tilde{Y}^0$ , as described in Section A.3. $\tilde{Y}^{0:V}$ is then decoded into the motion sequence via the VQ-VAE decoder.

W2-2) Clarification of the summation range in Eq. 1 of Sec. A.2 ($i=1$ to $V$).

A) Regarding Eq. 1 in Section A.2, since we need to calculate the loss starting from predicting $Y^1$ given $Y^0$ up to predicting $Y^V$ given $Y^{0:V-1}$, it is correct to define the summation range as $i = 0$ to $V-1$.

W2-3) Diagram of flow between masked and residual transformers.

A) We originally had one in the Supplementary in Section A.3, which may not have been clear. We have revised the figure and its description in Section A.3 for better understanding of our model. Additionally, we have updated Figure 2 in the main paper to better illustrate the flow between these two components.

W3) Including supplementary video for qualitative results.

A) We have included and uploaded video materials in the supplementary to show the motions presented in Fig. 3 and Fig. 4 in main paper, and Fig. 4 in Section C. These videos demonstrate how our model generates diverse motions accurately aligned with the input text. Moreover, the videos highlight that our model produces motions with higher precision compared to those generated by other baseline models.

W4) Comparable quantitative results; notable improvement in HumanML3D FID.

A) We appreciate the reviewer for recognizing the improved performance of our model in FID on the HumanML3D dataset. FID is a primary metric for evaluating the quality of generated motion, and this result demonstrates that our model captures a high level of realism and detail. Notably, given that the HumanML3D dataset is significantly larger than the KIT-ML dataset, we believe this suggests a potential that our method tends to perform better with more data.

Moreover, as mentioned in Section 4.2, our analysis is focused more on the FID and R-precision and the minor drops observed in diversity and MModality are considered secondary. Placing too much emphasis on multimodality and diversity could lead to outputs that lose relevance or coherence with the input text. Thus, we believed that ensuring output quality and maintaining alignment with the text was more important than simply generating diverse outputs. Our study focuses on maintaining this balance while improving the primary metrics of FID, R-Precision, and Multimodal Distance.

Regarding the marginal improvement in other metrics, incremental improvements between SOTA models are common in existing research. Therefore, we believe the performance gains achieved by the HGM³ over prior SOTA models are by no means small and represent meaningful evidence of the effectiveness of our new method.

W5) Clarify or correct some technical descriptions (see "Questions" below).

A) We have addressed this in the questions section below. Please refer to that section.

W6) Concatenating supplementary to the main paper

A) Thank you for the helpful suggestion. We have connected the main paper with the supplementary material and re-uploaded it.

Q1) Token prediction in masked and residual transformers.

A) As mentioned in our answer to W2-1, the masked transformer and residual transformer are trained separately. During inference, an empty motion sequence is given with all $n$ tokens masked, and the masked transformer predicts all tokens over $L$ iterations. This prediction results in $\tilde{Y}^0$, which, as described in Section A.3, is then fed as an input to the residual transformer. The residual transformer then sequentially predicts $\tilde{Y}^1$ to $\tilde{Y}^V$.

Q2) Clarify Eq. 4: description of $k$ and $\mathcal{M}$.

A) Thank you for pointing this out. We have revised the text to clarify both $k$ and $\mathcal{M}$. Specifically, the definition of $\mathcal{M}$, which denotes the set of indices for all masked tokens, has been added in line 204. Additionally, we have included an explanation of $k$ in line 210, which represents the index of each masked token.

We also realized that the $k$ used here could potentially cause confusion with the $k$ later used to denote epochs in the manuscript. To address this, we have updated the notation for epochs to $t$, ensuring a clear distinction between the index $k$ of masked tokens.

Q3) L220 (Eq. 5): $\hat{\ell}$ (and $G$) as predicting topological order of token losses.

A) Thank you for pointing this out. To prevent any misunderstanding, we have revised the manuscript to clarify that $\hat{\ell}$ and $G$ are intended to predict the relative ordering of token losses rather than their exact reconstruction values. Specifically, we have modified the text in line 201 from 'predicting the reconstruction loss' to 'predicting the relative ordering of reconstruction losses'. Similar adjustments are made in lines 223 and 240 to ensure that the objective of $L_{pred}$​ as a ranking mechanism is clearly conveyed.

Q4) L249: Clarify if "at each step" refers to epochs ($k$).

A) Thank you for your observation. 'At each step' in line 249 does indeed refer to 'epoch.' We have revised the manuscript to say 'at each epoch' to prevent any confusion.

Q5) L 249: Clarify if $\mathcal{M}=\gamma(\tau_k)\cdot n$ and define $\mathcal{M}$.

A) We had previously defined $\mathcal{M}$ in line 209.5, but we apologize if there was any confusion. To improve clarity, we have now revised the manuscript to specify this definition at line 204. Additionally, regarding your question about $\mathcal{M} = \gamma(\tau_k) \cdot n$ (which has now been updated to $\gamma(\tau_t) \cdot n$ in the revised text), this expression does not define $\mathcal{M}$ directly. Instead, $\gamma(\tau_t) \cdot n$ refers to the total number of tokens selected for masking, whereas $\mathcal{M}$ denotes the specific indices of these selected tokens in the motion sequence. We would like to clarify that $\mathcal{M}$ is recalculated at each epoch $t$, as the masking strategy is updated dynamically based on the model's predictions. This has been further clarified in line 204.

Q6) L339: where is the probability distribution taken from?

A) The probability distribution mentioned here is generated by the masked transformer, which, at each iteration $l$, uses a softmax function to produce the likelihood of each possible token index for the masked positions. This probability distribution reflects the model's confidence in its predictions for each token, with the tokens having the lowest confidence subsequently masked again in the following iteration. Please refer to the revised manuscript for the updated explanation in line 338.

Q7) Sec. A.2, A.3: Differences and similarities between residual transformer blocks in Fig. 2 and Fig. 3.

A) In Sections A.2 and A.3, the blue block in both Fig. 2 and Fig. 3 represents a single residual transformer. During training, the residual transformer takes $Y^{0:i}$ as input, where the tokens are embedded and summed to predict $Y^{i+1}$. Fig. 2 is intended to illustrate this process. In both training and inference, the same residual transformer consistently predicts the next token sequence layer, regardless of the specific input index $i$. This means that, during inference, a single residual transformer is applied iteratively across different inputs. Fig. 3 reflects this by depicting multiple applications of the same residual transformer to illustrate its repeated use, not multiple sub-blocks. Please refer to the revised manuscript for the updated explanation.

I appreciate the thorough, impressively written, rebuttal.

While most of my concerns have been addressed, one key issue remains unresolved.

The newly added video supplementary, particularly the files under the "Additional videos from ours" folder, exhibit low-quality motions. Most of them have severe floating artifacts, some depict unacceptably jittery motions, and some show unnatural motions. The quality of these motions is much lower than in many prior works.

Here are three (out of many) examples:

• The file "a person grabbed something with right hand and put it somewhere" shows severe floating artifacts.
• The file "a person is performing a dance and spinning around" shows an unnatural motion.
• The file "a quarterback will throw a football to another team member in hopes to score a touchdown" shows jittery head motions.

The most important outcome in a work is the quality of synthesized motions. With low-quality results, this work cannot be used, no matter how well it is written and what advanced building blocks it uses.

Given the aforementioned considerations, I believe a rating of 5 remains appropriate.

I urge the authors to find the cause of the exhibited artifacts, fix their model, and re-submit their work to another venue.

We sincerely thank you for your detailed observations regarding the supplementary videos. While we appreciate your thorough review, we humbly disagree with the assertion that the quality of our generated motions is "much lower than in many prior works." Below, we address your concerns.

Comparative Motion Quality

While some floating and jittery artifacts are present, they are not unique to our model. Such issues arise naturally from the SMPL rendering process when no post-rendering corrections are applied. For fair comparisons, we intentionally refrained from applying corrective techniques. These corrections include:

• Foot Contact Stabilization: Ensures feet remain grounded during contact phases, eliminating floating effects.

• Temporal Smoothing: Reduces jittering in movements by filtering sudden transitions between frames.

• Head Alignment Regularization: Minimizes head jitter by constraining rapid orientation changes.

As examples:

• The MMM project page (Pinyoanuntapong et al., 2024b) shows similar jittering or floating artifacts in uncorrected outputs.

• The MoMask project page (Guo et al., 2024) displays outputs where such artifacts appear corrected, but our experiments with uncorrected MoMask outputs reveal similar issues, such as floating and jittering.

These examples demonstrate that such artifacts are common in raw SMPL-rendered outputs across leading methods.

Instruction Alignment and Comparative Performance

To address your concerns, we have created comparison videos with outputs from other models. These videos are uploaded in the supplementary material, showing the three motions you highlighted. The videos clearly illustrate the following:

• While our model exhibits artifacts, they are observed in all other models without post-rendering corrections.

• Despite the artifacts, our model adheres to all instructions for motions containing detailed descriptions. In contrast, other models not only exhibit artifacts but also fail to align their motions with the given instructions, producing outputs that deviate from the semantics of the textual input.

Based on these observations, we respectfully disagree with the notion that our model's performance is inferior. Our method demonstrates superior instruction adherence, which we consider a critical factor for evaluating motion generation models.

Commitment to Future Improvements

The primary goal of our paper is to generate motion that accurately reflects the details and nuances of the input text, rather than focusing on artifact correction methods. To ensure fair comparisons with other models, we deliberately refrained from applying post-processing techniques, such as temporal smoothing or foot contact stabilization, which could artificially enhance visual quality. However, we understand the importance of producing high-quality rendered outputs for broader applications. To address this, we plan to apply the post-rendering correction process as part of our project page. This will reduce artifacts such as jittering and floating motions, while maintaining the generated motion's alignment with the input text.

We appreciate you bringing this issue to our attention and sincerely apologize for the inconvenience caused by the inaccessible video files. We have now re-uploaded the supplementary videos and verified that they can be opened successfully. Please let us know if you encounter any further issues.

Thank you for your understanding and for taking the time to review our work.

(1) I read the HPM (Wang et al., 2023a) paper, and I don't understand why you say HPM is designed to "learn novel representation" and "mask-invariant features." As far as I know, the goal of HPM is consistent with that of your HTM, which is to learn challenging patches (tokens) to improve the model's reconstruction ability. (2) When you say, "HTM eliminates the need to train an encoder for representation extraction," may I ask, what does the VQVAE in the first stage count as?