ControlMM: Controllable Masked Motion Generation

Q6 Whether the optimization in the latent space has some issues about the over-fitting or not is not well discussed. 

A6: Our model is trained and tested on HumanML3D, the motion covering a broad range of human actions such as daily activities (e.g., 'walking', 'jumping'), sports (e.g., 'swimming', 'playing golf'), acrobatics (e.g., 'cartwheel') and artistry. From the evaluation on the test set, in Table 1, our model outperforming other approaches on such a dataset indicates its superior generalization capabilities.

In addition, all visualization demos on our project website are controlled by the synthetic spatial control signals that are not present in the training dataset or testing dataset. The high quality human motions based on the synthetic spatial control signals also showcase that our model is not overfit to the training dataset. Moreover, our model also enables Obstacle Avoidance and Body Part Timeline Control, tasks that the model has never encountered during the training phase.

Dear Reviewer KEsH, If our responses address your concerns, we kindly ask you to consider reevaluating this work and improving your rating. If there are remaining issues or additional guidance you can provide, we would be more than happy to address them.

Thank you for your time and valuable feedback. We fixed the writing and notation in the updated manuscript. Below, we address your questions and comments:

Q1 "a bag of tricks". Controlnet of controllable motion generation similar to MotionLCM, not very novel.

A1: The fundamental problem of current controllable motion generation models is that they cannot simultaneously achieve high-quality motion generation and high-precision spatial control. Compared to MotionLCM, ControlMM has significantly higher quality (FID 0.061 vs 0.531) and more precise (Avg Error 0.0098 vs 0.1897). The key difference is that current controllable models are based on diffusion models, which cannot fully capture the internet spatial and temporal dependencies in the complex human motion, thus leading to low generation quality (high FID score). On the other hand, the masked motion model, which relies on a BERT-like generative masked modeling, can lead to much improved generation quality by explicitly modeling and learning the joint distribution of motion tokens. However, current masked motion models do not have spatial controllability. To address this challenge, we propose ControlMM that consists of two complementary components: motion control model and logits-codebook editing.

Motion Control Model, inspired by ControlNet, aims to add spatial control signals into the pretrained masked motion model. However, the motion control model is very unique from following perspectives. 1) Overall working principle: motion control model achieves spatial control by modifying the underlying motion token distribution. ControlNet realizes spatial control by alternating diffusion process. (2) Architecture: motion control model is a partial copy of the original masked motion model, where it does not possess the motion token classifier in the original model. ControlNet has the identical architecture as the original diffusion model. Moreover, motion control model operates on the discrete motion tokens extracted from motion tokenizer. (3) Training paradigm: motion control model is trained using the combination of (i) generative masking training, (ii) consistency feedback (iii) and training-time differential sampling. ControlNet is trained via the optimal denoising process. (4) Inference process: because the generative masking training learned motion token distribution conditioned on text and spatial control prompts, during inference, motion control model follows the parallel token decoding that iteratively sample from the learned distribution to obtain the high-confident tokens, while remasking and repredicting low-confidence token in the following iterations.

Logits Editing + Codebook Editing To the best of our knowledge, we are the first to successfully apply inference-time guidance to a masked model. This is challenging because Masked Transformers operates on their own learnable tokens, which differ from the codebook tokens utilized by the decoder. To address this, we propose Logits Editing for intermediate generation steps, allowing iterative adjustments to the generation process. In the final generation step, we directly update the embeddings in the codebook space through Codebook Editing. Further details on these approaches can be found in Section 3.3 (Inference-time Logits and Codebook Editing) and Supplementary Section A.10 (Dual-Space Categorical Straight-Through Estimator). This component also offers the additional benefit of being able to apply any arbitrary objective function, such as obstacle avoidance, which other methods (e.g., OmniControl, TLControl, MotionLCM) are not capable of.

Note that all controllable motion models (e.g., GMD, OmniControl, TLControl) require some form of inference-time guidance to achieve accurate control. This is because motion data represents relative positions that depend on previous frame positions and rotations, whereas the control signal specifies in absolute positions. Unlike controllable models in the image domain, where pixel-to-pixel mapping is applicable, motion control faces these unique challenges. Further, working with masked tokens in latent space is more challenging compared to the motion space, as OmniControl. We also discuss these challenges in Section A.9: The Challenges of Motion Control Model.

Q2. Table 3 (L1 vs. L5) shows the control module performs not very well.

A2: Codebook Editing alone generates low quality (high FID), as shown in Table 3 (# 3): Codebook Editing directly perturbs the embedding based on the difference between the control signal's absolute position and the generated motion. This mechanism enables precise control over the trajectory by directly addressing discrepancies. However, because Codebook Editing modifies the motion after generation is complete, this post-hoc adjustment may impact the overall motion quality, especially if the generated motion deviates significantly from the control signal. This demonstrates that Codebook Editing alone is insufficient for achieving high-quality results.

Motion Control Model helps high quality generation (lower FID): The Motion Control Model improves motion quality by guiding the Masked Transformer generation process with the control signal at each layer. As a result, the generated tokens are already close to the desired outcome, requiring only small perturbation from Codebook Editing to achieve precise control with high quality.

Motion Control Model + Codebook Editing: these two components complement each other—leading to improvements in both motion quality and trajectory control precision.

From Table 3:

Q3. The authors miss two latest baselines, TLControl and MotionLCM.

A3: The comparison between ControlMM and TLControl, is shown in Table 1, in terms of generation quality and control precision. However, since TLControl is not open-sourced, we could not directly compare inference speeds in Figure 2. We also added MotionLCM to the main comparison Table 1 and Figure 2. Also, we visualize comparison in the anonymous website. Since TLControl focuses on precise spatial control, this prioritization leads to lower motion quality, as reflected in its higher FID scores. MotionLCM emphasizes speed, enabling faster generation. However, this comes at the cost of even lower-quality motion (higher FID) and imprecise control. ControlMM overcomes these trade-offs by generating high-quality motion with precise control, while achieving real-time performance. The ControlMM-Fast variant generates 9.8 seconds of motion in just 4.94 seconds (in Table 5), without compromising quality. Moreover, the Accurate version of ControlMM achieves even more precise control while maintaining the same high-quality motion generation, setting it apart from existing methods.

Q4 motions jitter

A4: This is a common issue for rendering mesh which does not come from the model itself. Specifically, it arises during the retargeting process for the HumanML3D dataset. The issue is not unique to ours and affects the visualization of all models, including:

• MotionLCM: latent diffusion
  • https://dai-wenxun.github.io/MotionLCM-page/static/videos_mc_sparse/2.mp4 “a man crawls forward on his stomach”
  • https://dai-wenxun.github.io/MotionLCM-page/static/videos_mc_sparse/3.mp4 “a person walks quickly and intentionally in zig-zag pattern forward.”
• MDM: motion space diffusion
  • https://guytevet.github.io/mdm-page/static/figures/kick1.mp4
  • https://guytevet.github.io/mdm-page/static/figures/rope1.mp4
• T2M-GPT: quantized token
  • https://mael-zys.github.io/T2M-GPT/Ours/000066_pred.gif A man rises from the ground, walks in a circle ...
  • https://mael-zys.github.io/T2M-GPT/Ours/004742_pred.gif a man starts off in an up right …

Q5 generation quality

A5: For quantitative analysis, the FID score measures generation quality, which is a key strength of our ControlMM model. Our model achieves FID score of 0.061, significantly lower than the 0.271, 0.218, and 0.531 achieved by OmniControl, TLControl, and MotionLCM, respectively. Additionally, we provide a qualitative comparison in Figure 5 and video demos comparison to SOTA on the website to further illustrate the advantages of our approach.

Dear Reviewer KEsH,

Thank you for your review. As the discussion deadline is approaching in two days, could you please check our response and let us know if anything remains unclear? If your concerns are resolved, we would appreciate it if you could consider reevaluating the work. Let us know if further clarification is needed.

We want to further clarify our response to your question Q2. Table 3 (L1 vs. L5) shows the control module performs not very well ... it seems that Codebook Editing is the main component for editing precision

We’ve added visualizations to the Component Analysis section of our anonymous demo website (https://anonymous-ai-agent.github.io/CAM/#component-analysis). These visualizations compare motions generated using only the Codebook Embedding versus those generated by the full model. When using only the Codebook Embedding, the motion quality is noticeably worse—it lacks realism and does not align well with the provided textual descriptions.

Our Motion Control Model is specifically designed to enhance motion quality. As highlighted in A2, incorporating the Motion Control Model significantly improves motion quality, with the FID score improving from 0.190 to 0.069.

Thank you reviewer KEsH for valuable feedback.

Q1.1 I notice that ControlNet has been claimed in MotionLCM. This difference is marginal for me currently.

A1.1 We would like to clarify that in the related work section, we acknowledge OmniControl (Realism Guidance) and MotionLCM (Motion ControlNet) as the first to adopt a ControlNet-like structure. However, both OmniControl and MotionLCM are built on diffusion models, which is the framework that the original ControlNet was specifically designed for. In contrast, we are the first to apply ControlNet-like to Generative Masked Model. The detailed and unique implementation of applying ControlNet-like structure to Generative Masked Model is summarized as below. Moreover, we are also the first to introduce inference time guidance/optimization for the Generative Masked Model which provides the additional advantage of enabling the application of arbitrary objective functions, such as obstacle avoidance—a task that both MotionLCM and OmniControl are not capable of. In addition, even through our work is designed for enabling ControlNet for Masked Motion Model. The techniques proposed in our work could be easily extended to generative masked image/video models, such as MUSE, MAGVIT, and MEISSONIC.

To further clarify the novelty and challenge of applying these two components to the Masked Model, we describe each component with additional details as follows:

1. ControlNet-like (Motion Control Model): (1) Underlying Working Principle: Motion Control Model achieves spatial control by modifying the underlying motion token distribution. ControlNet realizes spatial control by alternating diffusion process. (2) Network Architecture: motion control model is a partial copy of the original masked motion model, where it does not possess the motion token classifier in the original model. ControlNet in OmniControl (Realism Guidance) and MotionLCM (Motion ControlNet) has the identical architecture as the original diffusion model. Moreover, motion control model operates on the discrete motion tokens extracted from motion tokenizer. In contrast, OmniControl (Realism Guidance) operates in raw motion space and MotionLCM (Motion ControlNet) operates in continuous latent space. (3) Training Paradigm: motion control model is trained using the combination of (i) generative masking training with consistency feedback, which has different training loss functions, compared with OmniControl and MotionLCM (ii) and training-time differential sampling, which are not required by diffusion models. (4) Inference process: because the generative masking training learned motion token distribution conditioned on text and spatial control prompts, during inference, motion control model follows the parallel token decoding that iteratively sample from the learned distribution to obtain the high-confident tokens, while remasking and repredicting low-confidence token in the following iterations. The challenge of applying ControlNet-like to Masked Model is that the model needs to decode [Mask] tokens for an initial motion token generation. However, [Mask] token only existing in Masked Transformer not the decoder from VQVAE. We overcome this by approximating the [Mask] token from codebook, as described in Sec A.9.
2. Logits Editing + Codebook Editing: We are the first to successfully apply inference-time guidance to a masked model. This is challenging because Masked Transformers operates on their own learnable tokens, which differ from the codebook tokens utilized by the decoder. To address this, we propose Logits Editing for intermediate generation steps, allowing iterative adjustments to the generation process. However, applying guidance directly to embeddings is impractical for Masked Models, as their Masked Transformers use learnable tokens that differ from the codebook space which requires for decoder of the Motion Tokenizer to reconstruct motion tokens to raw motion space as mentioned in Section A.10. In the final generation step, we directly update the embeddings in the codebook space through Codebook Editing. These components also offer the additional benefit of being able to apply any arbitrary objective function, such as obstacle avoidance, which other methods (e.g., OmniControl, TLControl, MotionLCM) are not capable of.

Q1.2 More reasonable choice is to compare ControlNet-like of ControlMM (L5 in Table 3) to MotionLCM

A1.2 We respectfully disagree. Limiting the comparison to only certain component to MotionLCM may be not very reasonable. Current SOTA methods commonly utilize ControlNet-like modules and/or inference-time guidance. However, by design, MotionLCM is fundamentally different as it prioritizes speed by distilling the diffusion process into just 1–4 steps. This contrasts with methods like GMD and OmniControl, which require 1,000 steps for guided diffusion. Utilizing only 1-4 step guided diffusion is impractical for MotionLCM. Additionally, TLControl does not employ a ControlNet-like architecture either. Therefore, restricting comparisons solely to architectures with ControlNet-like components would make direct comparisons across baseline methods infeasible.

For clarity, the breakdown of components, along with FID (quality) and trajectory error (precision control), is provided in the table below.

Q.2 Effectiveness of Motion Control Model, As pointed out by reviewer d9b4 Motion Control Model alone is not good enough

A2 We addressed this concern for reviewer d9b4, who is satisfied with our response. We hope this clarification also addresses your concern. In sum, Motion Control Model and Codebook/Logits Editing are complementary components and combining them leads to the optimal motion quality and trajectory control precision. Motion Control Model is training-time optimization that improve motion quality/fidelity by learning joint categorical distribution of motion tokens, while guiding the generated motion to closely align with the input spatial control signals and text prompts. As a result, the generated motion is required only small perturbation from Codebook/Logits Editing during inference to realize more precise control.

Reviewer d9b4 concerned that "OmniControl does not seem to use any inference-time optimization (only ControlNet-like part)". However, we addressed that OmniControl also utilizes comparable architectures ControlNet-like (Realism Guidance in OmniControl and Motion Control Model in ControlMM) and Inference Time Guidance (Spatial Guidance in OmniControl and Logits + Codebook Editing in ControlMM). Moreover, the table below compares only the ControlNet-like modules of OmniControl (Realism Guidance) and ControlMM (Motion Control Model). Our Motion Control Model demonstrates superior performance, showing better quality (FID and R-Precision) and more precise control (Trajectory, Location, and Average Error).

Q3. Components are not highly related to each other, the contribution of the proposed method is limited by this.

A3. Our model consists of Motion Control Model and Logits+Codebook Editing, each with its own benefit. Combining these components is not a limitation, but rather a complementary approach that helps improve both quality and precision control, as shown in the table below. We explain each component as follows:

1. ControlNet-like (Motion Control Model) The purpose of this module is to ensure high-quality motion generation by connecting the control signal module to each layer of the Masked Motion Model. This allows the model to predict probability distributions of the codebook that are close to the control signal. While Logits + Codebook Editing can perturb the logits far away from the true distribution, the Motion Control Model helps guide the model to output logits that are close to the true distribution, as it is trained to output logits w.r.t. to the true distribution of ground truth tokens obtained from VQVAE.
2. Inference Time Guidance (Logits + Codebook Editing): Logits Editing and Codebook Editing can be considered the same component, as they utilize the exact same optimization function. The optimize function updates logits during earlier unmasking steps (Logits Editing), while the codebook embeddings is updated during the final unmasking step (Codebook Editing). Optimizing logits in the early stage can help to maintain high quality while optimizing codebook embedding in the last stage helps improve precise control. Moreover, because it can incorporate any optimization function during inference, this component is versatile and can adapt to arbitrary tasks such as obstacle avoidance.

The advantage of Logits Editing and the Motion Control Model is their ability to maintain motion quality (R-Precision, FID, and Foot Skating), while the advantage of Codebook Editing lies in its ability to achieve precise control (Trajectory Error, Location Error, and Average Error). By combining these components, ControlMM achieves state-of-the-art performance in both quality and precision control. From Table 3:

Q4 comparison between L1-L5

A4. We assume that reviewer KEsH's question is about the effectiveness of the Motion Control Model (L5). We have addressed this in Q2 by comparing it to OmniControl. Moreover, a comparison between the baseline control (L1) with the Motion Control Model (L5) may not provide a clear picture. Instead, we answer by comparing the full model with and without the Motion Control Model. The purpose of the Motion Control Model is to maintain the quality of generation. As shown in the table below, without the Motion Control Model, the FID score worsens significantly from 0.061 to 0.142.

Q5. Speed of the algorithm. How can this method be used by users. MotionLCM treats this as an important motivation.

ControlMM-Fast is ~20x faster and more accurate compared to other SOTA

ControlMM-Accurate is relatively faster than SOTA methods (GMD and OmniControl) while delivering significantly better precision control (measured by Trajectory Error, Location Error, and Average Error) and higher quality (measured by R-Precision and FID).

• ControlMM-Fast achieves even greater speed, completing motion generation in 4.94 seconds, compared to 87.33 seconds for OmniControl and 132.49 seconds for GMD. With this speed ControlMM already enables real-time generation, producing 9.8 seconds of motion in just 4.94 seconds. All models are tested on the same NVIDIA A100 GPUs.
• With such high generation speed, ControlMM-fast also maintains better quality, with an FID of 0.059, compared to 0.218 (OmniControl) and 0.576 (GMD). It also outperforms in precision control, achieving a 2% Trajectory Error, compared to 3.87% (OmniControl) and 9.31% (GMD). Note that the number of iterations in Logits + Codebook Editing can be adjusted during inference to trade off between speed and precision control without affecting motion quality (FID and R-Precision). As the table shown below, ControlMM-Accurate can achieve Average Error of 0.98 cm and 0.00% of Trajectory Error, while still faster than OmniControl and GMD.

MotionLCM focuses on speed, which comes at the cost of lower quality and precision control.

• Motion Quality: our ControlMM achieves a FID score of 0.059 (ControlMM-fast), outperforming MotionLCM which has a FID score of 0.531.

• ​​Control Precision: MotionLCM has an average joint error of 18.97 cm, while our ControlMM has an average joint error of 0.0098. Additionally, according to the Trajectory Error metric, MotionLCM exhibits joint errors exceeding 50 cm in 18.87% of its generated motions, compared to 2.00% and 0.00% for our Fast and Accurate versions, respectively. As shown in demos on the https://anonymous-ai-agent.github.io/CAM/, ControlMM also shows higher quality and more precise control in generation motion, compared with MotionLCM. For more visualization compared to MotionLCM, we also uploaded the visualization comparing MotionLCM to our ControlMM in the "Supplementary Material" zip file, located in the 'ControlMMvsMotionLCM' folder.

Real-World Application ControlMM is designed to generate high-quality, precise motion generation with real-time performance, making it practical for real-world applications. MotionLCM prioritizes speed at the expense of precision and quality.

For more details on ControlMM-Fast and ControlMM-Accurate, refer to Table 5 in the manuscript.

Q6.  hyper-parameters, Can the iteration be infinite?

A6. We detail the hyperparameters in the manuscript (Sec. A3). The default settings use 100 iterations for Logits Editing and 600 iterations for Codebook Editing with learning rate 6e-2. We conducted a parameter search on the evaluation set to determine the minimum number of iterations required to reduce the Location Error to zero. With the limit of iteration, the optimization process does not run indefinitely. However, these iterations can be fewer if the loss reaches zero before the maximum iteration limit. Moreover, the table below demonstrates that increasing the number of iterations for Codebook Editing continues to improve the Average Error without any signs of degradation in generation quality (FID).

Q7. Stickman motion for motion jittering

Thank you for your suggestion. We have also uploaded the skeleton motion in the "Supplementary Material" zip file under the 'jittering' folder. The skeleton visualization shows no jittering issue, which confirms that the issue arises during the retargeting process for the HumanML3D dataset and not from our model.

Q8. revised Table 2 does not report MoMask

Note that Table 2 has not been revised and remains the same as the first version. MoMask cannot control joint positions because it encodes and quantizes all body joints into a single token. As a result, modifying only the upper body is not possible with MoMask. In contrast, MMM (Masked Motion Model) addresses this limitation by training separate codebooks for the upper and lower body, allowing more control over upper body.

Dear Reviewer KEsH,

Thank you for your review. Could you please check our response and let us know if anything remains unclear? If your concerns are resolved, we would appreciate it if you could consider reevaluating our work.

Note that we already added MotionLCM to Table1, Figure 2, Figure 6, and demo website.

Since we did not receive final response from reviewer KEsH, we summarize the "My remaining concerns" of reviewer KEsH as follows:

Note that the questions from reviewer KEsH only compare with MotionLCM, which is a 1-step diffusion method without guided diffusion, making it not directly comparable to our ControlMM. It is also helpful to see our answers to similar questions raised by other reviewers (summarized in the "Similar Questions Summary" at the very top post), as this should provide a clearer picture.

Q1.1 ControlNet-like has been claimed in MotionLCM

• Our ControlMM is the first to introduce [1] Motion Control Module (ControlNet-like) and [2] Logits/Codebook Editing (Inference Time Guidance) to the Masked Motion Model. In contrast, MotionLCM is similar to OmniControl which both utilize ControlNet-like on diffusion-based models.

Q1.2 Should only compare Motion Control Module (ControlNet-like) to MotionLCM.  
Q2 Motion Control Module (ControlNet-like) alone does not perform well (​​compared to MotionLCM)
Q3 Motion Control Module (ControlNet-like) and ​​Logits/Codebook Editing (Inference Time Guidance) are not highly related to each other
Q4 Comparison of Motion Control Module (ControlNet-like) alone

• All SOTA methods, i.e., GMD, OmniControl, and TLControl, utilize some form of Inference Time Guidance to achieve high-precision control. Moreover, GMD and TLControl do not use a ControlNet-like architecture. If the comparison is restricted to only ControlNet-like components, none of the baseline methods can be directly compared.
• Motion Control Module is designed for maintaining quality, combined with Logits/Codebook Editing (Inference Time Guidance) which is designed for high precision control which leads to the SOTA results in both quality and precision control. Additionally, Motion Control Module (ControlNet-like) alone performs better than OmniControl with ControlNet-like part.
• Please refer to the table in answer “A1.2” which includes a comparison of all components of SOTA methods to provide a complete overview.

Q5 MotionLCM is faster so is more practical

• Our ControlMM achieves SOTA performance across all metrics, while maintaining real-time performance, and is still faster than other methods.
• MotionLCM focuses only on speed, which causes it to perform the worst in all other metrics.
• Please see our demo https://anonymous-ai-agent.github.io/CAM and "Supplementary Material" zip file, located in the 'ControlMMvsMotionLCM' folder. MotionLCM quality and control precision are significantly worse which make ControlMM more practical.

Q6. Can the iteration be infinite?

• We use a fixed number of iterations so it will not run infinitely.

Q7. jittering issues

• The skeleton motion visualization in the "Supplementary Material" zip file, located in the 'jittering' folder, shows no jittering issues, confirming that the problem does not originate from our model.

Q8 Not report MoMask

• MoMask does not have controllability.

Thank you for your time and valuable feedback. Below, we address your questions and comments:

Q1. comparisons with MoMask

A1. MoMask itself lacks the ability to control joint positions, which makes it not directly comparable in Table 1. Note that the original MDM also does not provide controllability. However, OmniControl reports results for MDM by applying a simple imputation strategy: it directly fills in the control joints information and allows MDM to diffuse the motion following control joins. This approach works for MDM because it operates directly in the motion space, where such imputation is straightforward. In contrast, this method is not applicable for quantized tokens in latent like MoMask.

Q2. Could you clarify whether this joint control data is also used during test set validation? If so, would this constitute a data leakage issue? 

A2. Yes, we use the ground truth joint information during both training and testing, following the settings of GMD and OmniControl. However, this is not data leakage. Simply replacing control joints with the motion directly will result in poor motion quality (high FID) since the motion will not be consistent with generated joints. Our experiments show that our ControlMM can balance between both precision control and high quality. Note that the dense signal control demonstrated on the website does not come from ground truth data in the dataset. These signals are synthesized, such as controlling the right hand to draw a heart in “the person draws a heart with hand” or defining a trajectory using a sinusoidal wave forming a circle in “a person walks in a circle clockwise.” These examples illustrate the model's ability to handle arbitrary control signals beyond the training data.

Q3. How would the system determine which joint information to use for control in practical applications, where such data may not be known in advance?

A3. The trajectory control can be synthesized to demonstrate various patterns, such as walking in a circular or zigzag manner, or guiding hand movements to draw specific shapes. Additionally, it can be adapted for sparse signal control, such as in a goal-reaching task where only the final frame is constrained by the control signal. These examples highlight the model's flexibility in handling both dense and sparse control signals.

Q4. KIT Dataset

A4 We also tested ControlMM on the KIT dataset and compared it to state-of-the-art (SOTA) methods. Despite the KIT dataset being significantly smaller than HumanML3D, ControlMM consistently outperformed other SOTA methods in both quality and precise control, demonstrating its robustness in resource-limited settings. We added KIT dataset result in Section "A.12 KIT DATASET".

Q5 Qualitative examples that showcase the diversity

A5 We added diversity generation under the same prompt for both signal control joints and object avoidance. These visualizations have been added to the Diversity section of our anonymous website demo https://anonymous-ai-agent.github.io/CAM/.

Q5. User study

A5 We conducted pairwise comparisons in the user study by asking participants to answer two questions: “Which motion is more realistic (better quality)?” and “Which motion is generated with more precise control?” The motions were generated from 30 randomly selected text descriptions from the test set of the HumanML3D dataset. We engaged 6 participants and presented them with three pairwise comparisons: our method versus OmniControl, our method versus GMD, and our method versus MotionLCM. Our ControlMM was rated significantly more realistic and demonstrated better precision control compared to other state-of-the-art methods.

Dear Reviewer gWgL,

Thank you for your review. As the discussion deadline is approaching in two days, could you please check our response and let us know if anything remains unclear? If your concerns are resolved, we would appreciate it if you could consider reevaluating the work. Let us know if further clarification is needed.

Thank you for your time and valuable feedback. Below, we address your questions and comments:

Q1. Motion Control Module alone does not seem to perform well.

A1: Codebook Editing alone generates low quality (high FID): Codebook Editing directly perturbs the embedding based on the difference between the control signal's absolute position and the generated motion. This mechanism enables precise control over the trajectory by directly addressing discrepancies. However, because Codebook Editing modifies the motion after generation is complete, this post-hoc adjustment may impact the overall motion quality, especially if the generated motion deviates significantly from the control signal.

Motion Control Model helps high quality generation (lower FID): The Motion Control Model improves motion quality by guiding the Masked Transformer generation process with the control signal at each layer. As a result, the generated tokens are already close to the desired outcome, requiring only small perturbation from Codebook Editing to achieve precise control with high quality.

Motion Control Model + Codebook Editing: these two components complement each other—leading to improvements in both motion quality and trajectory control precision.

From Table 3:

Q2: OmniControl does not seem to use any inference-time optimization

A2: Spatial Guidance in OmniControl is guided diffusion, which is an inference-time optimization approach that optimizes the diffusion noise. Different from OmniControl，our method relies on the masked motion model that does not leverage the diffusion process. Therefore, we cannot apply the noise optimization scheme adopted by OmniControl. Instead, our method optimizes the logits (i.e., unnormalized probabilities of motion tokens) and codebook embeddings to enhance spatial controllability. Therefore, I believe the comparison with OmniControl is fair. Even Though Spatial Guidance in OmniControl updates noise only once in each diffusion step, it requires 1000 diffusion steps. Comparing to our “Fast” version which requires only 100 iteration for Codebook Editing already achieves better Trajectory Error while significantly faster and higher quality (lower FID) From Table 5:

Note that all controllable motion models (e.g., GMD, OmniControl, TLControl) require some form of inference-time guidance to achieve accurate control. This is because motion data represents relative positions that depend on previous frame positions and rotations, whereas the control signal specifies in absolute positions. Unlike controllable models in the image domain, where pixel-to-pixel mapping is applicable, motion control faces these unique challenges. Furthermore, working with masked tokens in quantized latent space as ControlMM is more challenging compared to the motion space as OmniControl. We also discuss these challenges in Section A.9: The Challenges of Motion Control Model.

Dear Reviewer d9b4, If our responses address your concerns, we kindly ask you to consider reevaluating this work and improving your rating. If there are remaining issues or additional guidance you can provide, we would be more than happy to address them.

Dear Reviewer d9b4,

Thank you for your review. As the discussion deadline is approaching in two days, could you please check our response and let us know if anything remains unclear? If your concerns are resolved, we would appreciate it if you could consider reevaluating the work. Let us know if further clarification is needed.

Thank you for your time and valuable feedback. Below, we address your questions and comments:

Q1: A more detailed and intuitive explanation of the model's inner workings could be beneficial for reviewers.

A1: We updated figure 3 for better visualization of the motion control model, the key element of our approach. In addition, we added more implementation details in section 3 and supplementary materials. Intuitively speaking ,ControlMM is based on generative masked motion model, which is very similar to BERT-like language model that leverages token masking and reconstruction to learn the probabilistic distribution of human motion sequence. The existing generative masked motion models do not have spatial controllability, while exhibiting high-quality motion generation capacity. This paper aims to integrate spatial control into the generative masked motion model. Our approach has three key components. Motion Control Model is a training time optimization approach that aims to learn the accurate mapping from the text prompt and the spatial control prompt to the motion sequence by fine-tuning the pretrained generative masked motion model via a ControlNet-inspired approach. Logits Editing + Codebook Editing These are analogous to Guided Diffusion, which applies inference-time guidance. However, adapting this concept to Masked Models introduces a unique challenge: the embeddings of the Masked Transformer and the decoder from VQVAE operate in different spaces. To address this, we update the logits (not embeddings) during the initial unmasking steps—a process we call Logits Editing. At the final step, we directly update the embeddings in the codebook space, referred as “Codebook Editing”

Q2: How well the model would perform in more extreme or unforeseen out-of-distribution cases

A2: Our demo on the website demonstrates that ControlMM effectively handles out-of-distribution (OOD) scenarios across various tasks. Note that all dense signal controls presented in the demo are synthesized, and are not in the dataset. For instance, it can generate creative motions such as "a person draws a heart with their hand" with heart-shape trajectory or "a person walks in a circle clockwise" following syntactic trajectories like a sinusoidal circle wave. These examples illustrate the model's ability to handle arbitrary control signals beyond the training data. Additionally, the model successfully tackles tasks like Body Part Timeline Control and Obstacle Avoidance, even though it was never explicitly trained on these tasks, demonstrating its ability to adapt to arbitrary objective functions.

Q3: It would be interesting to see if the model can be trained from scratch or with different pretrained models and still achieve similar performance.

A3: This is interesting question. Note that MoMask is the current state-of-the-art Masked Motion Model. Experimenting with other models may result in a performance drop. So we investigate three different training strategies: (1) Frozen MoMask: MoMask base model is frozen and motion control model (MCM) is trained with the initial weight copied from the base model, (2) Frozen MoMask + MCM scratch: MoMask base model is frozen, and motion control model is trained from scratch. (3 ) MoMask scratch + MCM scratch: MoMaks and Motion Control Model are jointly trained from scratch. As shown in the table below, jointly training MoMask and MCM from scratch leads to significantly degraded quality (i.e., much higher FID). By keeping the MoMask base model frozen, training the MCM from scratch or from the initial weights copied from yields the best and similar momotion generation quality (i.e., much lower FID)

Dear Reviewer 9WzV, If our responses address your concerns, we kindly ask you to consider reevaluating this work and improving your rating. If there are remaining issues or additional guidance you can provide, we would be more than happy to address them.

Dear Reviewer 9WzV,

Thank you for your review. As the discussion deadline is approaching in two days, could you please check our response and let us know if anything remains unclear? If your concerns are resolved, we would appreciate it if you could consider reevaluating the work. Let us know if further clarification is needed.

Thank you for your time and valuable feedback. Below, we address your questions and comments:

Q1. Writing

A1. We increased the size of figures, fixed inconsistent notation in Section 3.1, added an algorithm in the supplementary material (Section A.2), provided details about root positions and rotations in Section A.3, and included CondMDI in the related work section.

Q2. The importance and influence of previous works which are the building blocks of the current proposed paper

A2. In addition to Section 3.2, where we discussed the ControlNet-like mechanism, we added more details in A.10 regarding the inspirations from the existing work and the difference between our work and existing solutions. We also added the relevant references in section 3, when we explain our ControlMM solution. In addition, we add more implementation details in the supplementary materials. To clarify our contribution highlighted in the abstract, ControlMM is the first work to enable spatial control over masked motion models so that we can simultaneously achieve high-quality motion generation with high-precision spatial control. Towards the end, we propose two key components: motion control model and inference-time logits and codebook editing. Both schemes are fundamentally different from the existing solutions highlighted in section 2.

Q3. Can you please elaborate on if and how the explained realism guidance is different from your suggested approach?

A3. OmniControl is a controllable diffusion motion model, while ControlMM is a controllable masked motion model. The realism guidance leveraged by OmniControl follows the ControlNet scheme originally proposed for text-to-image diffusion mode. The motion control model exploited by ControlMM is inspired by ControlNet, which has the unique features as follows. 1) Overall working principle: Motion Control Model achieves spatial control by modifying the underlying motion token distribution. Realism Guidance realizes spatial control by alternating diffusion process. (2) Architecture: motion control model is a partial copy of the original masked motion model, where it does not possess the motion token classifier in the original model. Moreover, motion control model operates on the discrete motion tokens via effective motion tokenizer. (3) Training paradigm: motion control model is trained using the combination of (i) generative masking training, (ii) consistency feedback (iii) and training-time differential sampling. Realism Guidance is trained via the denoising process. (4) Inference process: motion control model follows the parallel token decoding and ControlNet leverages the iterative denoising.

Applying a ControlNet-like approach to masked models, which operates directly in motion space is more challenging. ControlMM functions in the quantized latent space, requiring a decoder from the Motion Tokenizer. Moreover, the decoder and Masked Transformer operate in different spaces. The decoder does not support mask tokens. We address this limitation by averaging all embeddings in the codebook to approximate mask token. Moreover, we also concatenate relative motion control to solve “Ambiguity of Motion Control Signal”. We described more in Section A.9: The challenges of Motion Control Model.

To directly compare Realism Guidance and our Motion Control Model performance, the table below presents results for OmniControl using only Realism Guidance without inference time guidance (Spatial Guidance) and ControlMM using only the Motion Control Model without inference time guidance (Logit and Codebook Editing). Our Motion Control Model shows better quality (FID) and more precise control.

Q4. Provide additional insight. 

A4. We added more insights about how challenge and solution to integrate the novel ControlNet-like and Inference-time guidance in Masked Motion Model in supplementary Section A.9: The challenges of Motion Control Model Section A.10: Dual-Space Categorical Straight-Through Estimator

Dear Reviewer D6gn, If our responses address your concerns, we kindly ask you to consider reevaluating this work and improving your rating. If there are remaining issues or additional guidance you can provide, we would be more than happy to address them.

Dear Reviewer D6gn,

Thank you for your review. As the discussion deadline is approaching in two days, could you please check our response and let us know if anything remains unclear? If your concerns are resolved, we would appreciate it if you could consider reevaluating the work. Let us know if further clarification is needed.

• Added visualizations to the "Component Analysis" section in https://anonymous-ai-agent.github.io/CAM/#component-analysis