MotionBind: Multi-Modal Human Motion Alignment for Retrieval, Recognition, and Generation

We thank Reviewer YpBN for their thoughtful and constructive feedback. We greatly appreciate the recognition of our paper’s novelty and unique contributions in integrating human motion into a unified multimodal framework, the effectiveness and expressive power of the MuTMoT and REALM architectures, and the comprehensive evaluation and robust performance across tasks and datasets.

Below, we address all the raised concerns and clarify key aspects of our design. We believe the issues noted are either due to missing clarifications or minor omissions and do not outweigh the demonstrated novelty, breadth, and rigor of our proposed framework. We respectfully hope that the reviewer will reconsider their rating, taking into account our clarifications and the demonstrated strengths, novelty, and impact of our unified multi-modal motion framework.

Response to Major Concerns:

On text-to-motion generation on HumanML3D, the method does not deliver the best results in any of the metrics.

• Our method consistently delivers highly competitive performance on text-to-motion generation for HumanML3D, frequently ranking second-best, and achieves state-of-the-art results on KIT-ML as well as six additional benchmarks spanning reconstruction, cross-modal retrieval, and zero-shot action recognition. Our primary objective is to demonstrate the effectiveness and generalizability of our multi-modal framework across a diverse set of tasks, rather than focusing exclusively on optimizing for a single dataset or metric.

While the model supports any-to-motion generation, the authors only quantitatively evaluate text-conditioned motion generation.

• As acknowledged in the limitations section, our quantitative evaluation for motion generation focuses on the widely studied text-to-motion generation task, and this decision reflects the current state of the field, where standardized and robust benchmarks exist primarily for text-conditioned motion generation. While we recognize the importance of evaluating other modalities, there is currently a lack of well-established benchmarks for image- or video-conditioned motion generation beyond pose estimation. That said, our framework is "an original step," as noted by Reviewer cqJT, to support any-to-motion generation at inference time without additional training, thanks to its multi-modal alignment.
• Thus, we argue that our work should not be viewed as limited in scope. In addition to achieving state-of-the-art or highly competitive results on text-to-motion generation, we demonstrate strong performance on six additional benchmarks across motion reconstruction, cross-modal retrieval, and action recognition. Our primary goal is to showcase the versatility and generalizability of our multi-modal framework across diverse tasks. We believe the breadth and rigor of our evaluation present a compelling case for the effectiveness of our approach. We agree that expanding quantitative evaluation to additional conditioning modalities is a valuable direction for future work, and we plan to explore this in subsequent research.

The approach assumes encoders from LanguageBind are sufficiently expressive for all modalities, including human motion. If LanguageBind's representations do not fully capture the nuances of motion, it might hinder the effectiveness of motion retrieval and generation.

• We would like to clarify that LanguageBind provides a strong foundation for visual, text, and audio modalities. However, we do not assume LanguageBind is sufficiently expressive for human motion modalities. In fact, central components of our method are MuTMoT’s dedicated motion encoder, which is explicitly designed to capture the nuances of human motion from pose sequences, and lightweight adapters that make LanguageBind’s implicit understanding of human movement usable by aligning it with the representations from MuTMoT’s encoder. Our empirical results, including strong performance across cross-modal retrieval, generation, and zero-shot action recognition, demonstrate the effectiveness of this design.

The retrieval-augmented component of REALM depends on retrieving relevant motion. This could potentially be a bottleneck if the database is insufficient or out-of-distribution.

• While REALM utilizes reference motions to enhance generation fidelity, it continues to perform strongly with less relevant reference motions and even without retrieval, as shown in the ablation below varying the retrieval set size (top-$k$).

Table 1: Impact of Reference Motion Quality. We randomly sample two references from the top-$k$ most semantically similar motions in a reference database.

These results show that REALM performs consistently, with only gradual performance degradation as less relevant reference motions are included or even when no reference is provided, confirming the robustness of our generation pipeline.

A qualitative comparison with other methods should be provided.

• We agree that qualitative comparisons are valuable. Accordingly, we have conducted qualitative comparisons against four baseline methods (T2M-GPT, MotionDiffuse, MDM, and ReMoDiffuse), where our approach demonstrates superior or comparable motion fidelity and semantic alignment. Due to format constraints, we are unable to include the visual results here, but we will include a comprehensive version in the supplementary material.

While the paper is generally clear, certain sections, particularly the mathematical formulations and model descriptions, could be simplified for better accessibility.

• We appreciate your suggestion and will revise the mathematical formulations and model descriptions to improve clarity in the final manuscript.

Response to Additional Questions:

I think the use of dataset-specific embedding modules and heads seems to limit generalizability instead of improving it... Could the authors clarify the generalizability of this method?

• The dataset-specific embedding modules and heads do not limit the model to particular datasets. Instead, they address a practical and common challenge in motion modeling: different datasets often use different representation formats, such as varying numbers of joints or feature dimensionalities (e.g., KIT-ML uses 21 joints with 251-dimensional vectors, while HumanML3D uses 22 joints with 263-dimensional vectors). Our dataset-specific embedding modules map these diverse raw motion formats into a unified latent space, and the corresponding heads project the shared representation back to the original format as needed.
• This modular design is a key enabler of generalizability and robustness. First, it allows us to train a single model across multiple heterogeneous datasets, exposing it to a wide range of motion patterns, formats, and distributions. Second, by learning from various representations during training, the model is better positioned to handle unseen datasets that share a format with any of the training sets, without requiring any additional steps. This stands in contrast to models trained on a single dataset (format), which typically need additional adaptation steps and/or retraining to generalize to new datasets. In summary, the dataset-specific modules do not limit the model’s generalization; rather, they are what make it possible to unify and scale across diverse motion data sources within a single framework.

Is MuTMoT an encoder or an encoder-decoder?

• MuTMoT is an encoder-decoder architecture, and we will update the manuscript to consistently describe it as such. For clarity, only the encoder is used for retrieval and action recognition during inference, whereas both the encoder and decoder are employed for reconstruction and generation tasks. We will make this distinction explicit in the revised manuscript to avoid any confusion.

What does the \mathbf{w} in Eq. (3) represent? How is it obtained?

• \mathbf{w} represents learnable contribution weights used to combine motion embeddings from different temporal scales produced by the MuTMoT encoder’s hierarchical blocks.
• As described in LL156-162, each block in the MuTMoT encoder captures motion information at a different temporal resolution, producing a global embedding at that scale. The weights \mathbf{w}, as noted in LL163-165, are learned during training and determine the relative importance of each temporal scale when computing the final fused motion representation. This allows the model to adaptively emphasize the most relevant temporal features.

Are the MuTMoT part and the REALM part trained at the same time or separately?

• We follow a two-stage training process. In the first stage, we train MuTMoT, including both the encoder and decoder, using a combination of motion reconstruction and contrastive alignment objectives, enabling it to learn robust motion representations and align them with other modalities. In the second stage, we train REALM on top of the pretrained MuTMoT encoder and decoder. This two-stage approach ensures that the motion representations are well-formed and semantically aligned before being used for conditional motion generation.

We thank Reviewer S428 for the thoughtful and constructive comments. We appreciate that they recognize our proposed method as effective, acknowledge the state-of-the-art performance on multi-modal retrieval, and highlight the benefits of the proposed REALM module for multi-modal motion synthesis. Below, we address each concern and will incorporate these valuable suggestions in the final manuscript.

Response to Major Concerns:

Failure case analysis is currently missing from the paper. Could the authors provide a systematic analysis or categorization of common failure modes?

• Thank you for the valuable suggestion. We fully agree that analyzing failure cases is essential for understanding the limitations of generative motion models. Due to format constraints in the rebuttal, we are unable to provide an extensive qualitative breakdown here. However, based on the video examples included in the supplementary material, we have systematically categorized the most common failure modes as follows:

  1. Floating Feet: Instances where the feet hover above the ground plane, breaking physical realism. For example, in stands_up_from_a_laying.mp4, the character’s feet appear to hover during the standing motion, indicating insufficient ground-contact constraints, as also observed by Reviewer Nskc.
  2. Abrupt Transitions: Sudden changes in pose or velocity without appropriate anticipation or interpolation. An example, although subtle, can be found in a_person_steps_forward_while.mp4 (00:00:23 - 00:00:25), where the transition of the arms is unnaturally sharp.
  3. Velocity Jitter: High-frequency trembling or shaking in joint trajectories, particularly at the extremities. This is visible in a_man_swings_with_his_right_arm.mp4, where the arm visibly wobbles near the peak of the swing.
  4. Foot Skating: Feet slide or drift across the floor during periods when they should remain stationary. This is observed in a_person_walks_backwards_then.mp4 (00:01:25 - 00:02:06), where the supporting foot does not stay planted.
  5. Anatomical Violations: Body parts rotate or contort beyond biomechanical limits, resulting in implausible postures. For instance, in walks_back-and-forth.mp4 (00:03:03 - 00:04:12), the legs and torso exhibit rotations that are not physically plausible for a human.
  6. Lack of Object Interaction: The generated motion fails to convincingly reflect interaction with an implied object. In a_man_picks_up_object_from_the_ground.mp4, the hand does not make realistic contact with the imagined object or perform a grasping motion, in part due to limited modeling of fine-grained hand articulation in SMPL.

• We would like to clarify that these issues are well-known limitations in current motion generation systems and reflect broader challenges in modeling fine-grained physical realism. Such artifacts are not unique to our method but represent active areas of research and ongoing improvement across the field. We are committed to advancing these aspects in future work, and our analysis aims to transparently contextualize both the strengths and current limitations of our approach.

• In the final manuscript, we will provide a structured taxonomy of these failure cases, supported by qualitative examples and visual overlays. We will also present a comprehensive analysis to contextualize the model’s behavior, along with side-by-side qualitative comparisons against relevant baselines across diverse scenarios. This will further highlight both the strengths and limitations of our approach.

How does REALM respond to input conditions that are unrelated to human motion, for example, an image of a blue sky or other motion-irrelevant modality data? What kind of motion sequences would it generate in such scenarios?

• Thank you for the interesting question. When presented with entirely motion-irrelevant input conditions, such as an image of a blue sky, REALM tends to produce default or low-activity motions, such as standing still or casual walking. This behavior likely reflects the model’s tendency to fall back on generic motion priors in the absence of semantically meaningful cues.
• In contrast, for inputs that are not fully descriptive but still weakly associated with human activity (e.g., an image of a basketball), the model often generates plausible actions such as dribbling or shooting, indicating that it leverages learned semantic associations from pretraining.
• We agree that analyzing such cases provides important insight into the model's behavior under ambiguous or irrelevant conditions. We will include these examples and a discussion of their implications in the failure case analysis in the final supplementary material.

The paper primarily uses the AIST++ dataset, which contains music-driven dance data. How does the model generalize to other audio domains, such as speech? Have the authors tested REALM on inputs from datasets like BEAT or EMAGE, which focus on co-speech gestures? If not, could the authors comment on potential domain gaps, and whether the model would need fine-tuning or architectural adaptation to handle such input?

• As shown in the supplementary visualizations, REALM is able to generate plausible motion sequences even when conditioned on audio inputs outside the music domain, such as audio of someone boxing, despite being aligned primarily with the music-driven AIST++ dataset. This generalization is largely enabled by the pre-trained LanguageBind space, which captures broad audio-video and audio-text relationships from other domains as well. However, while this demonstrates some cross-domain robustness, we have not explicitly evaluated REALM on co-speech gesture datasets like BEAT or EMAGE.
• While REALM produces reasonable results for speech audio with strong emotional or rhythmic cues (e.g., angry speech), its outputs are currently limited to SMPL body motion, without fine-grained hand or facial keypoints, as in SMPL-X, required by datasets like BEAT v2. Consequently, there would be domain gaps, and further fine-tuning would be necessary to fully handle fine-grained co-speech gestures. We consider this a promising direction for future work.

Closing Remarks

We would like to reiterate that our work introduces an "effective alignment model" and a generative model that "effectively leverages information from multiple modalities," as you acknowledged. Furthermore, we respectfully highlight that:

• We propose "a broadly generalizable, modular and extensible architecture for multi-modal alignment" along with "notable design of learnable frame-level tokens and time-aware modulation" for motion generation, as noted by Reviewer Nskc.
• Our framework is a "unique contribution to the field" with "effective and powerful proposed architectures," as noted by Reviewer YpBN.
• Our work "addresses a clear gap by incorporating human motion into unified multi-modal embedding spaces, which until now have largely omitted motion data." "This is a notable innovation beyond prior works that aligned only motion and text," as noted by Reviewer cqJT.
• "Moreover, the introduction of REALM is an original step that allows any-to-motion generation -- going beyond conventional text-to-motion synthesis by conditioning on arbitrary modality inputs, which is a novel capability in this domain," as noted by Reviewer cqJT.

Taken together, we argue that these reviews affirm that "the idea of binding human motion into a general multi-modal embedding space and leveraging it for cross-modal retrieval/generation is fresh and significant in scope," as summarized by Reviewer cqJT.

Finally, as noted across all reviews, our framework achieves state-of-the-art or highly competitive results across multiple multi-modal motion tasks, including cross-modal retrieval, (zero-shot) action recognition, and motion synthesis, evaluated on several benchmarks (HumanML3D, KIT-ML, AMASS, AIST++, BABEL-60, and BABEL-120).

We hope these clarifications and acknowledgments help convey the strength and significance of our contributions. We respectfully hope Reviewer S428 will reconsider their rating in light of our clarifications and commitments to address their suggestions in the final manuscript.

We sincerely thank Reviewer cqJT for their highly positive feedback and recognition of our paper’s contributions. We greatly appreciate Reviewer cqJT highlighting our work’s novelty and significance in incorporating human motion into unified multi-modal embedding spaces, and the originality and technical soundness of both the proposed MuTMoT and REALM architectures.

We are also encouraged by the reviewer’s acknowledgment of our comprehensive and rigorous experimental evaluation, covering motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-driven motion generation, evaluated on several benchmarks (HumanML3D, KIT-ML, AMASS, AIST++, BABEL-60, and BABEL-120).

This reaffirms our belief in the potential impact our work offers toward general-purpose multi-modal human motion understanding and generation. We respectfully hope that the reviewer will continue to advocate for our paper during the discussion phase.

We thank Reviewer Nskc for their thoughtful feedback and recognition of the novelty, strength of our quantitative results, and generalizability of our proposed framework. We appreciate the acknowledgment of our modular design choices and thorough ablation studies. The concerns raised primarily stem from misunderstandings or points needing clarification, which we address individually below, along with revisions planned for the final manuscript. We hope that the reviewer will reconsider their rating.

Response to Major Concerns:

The method relies on frozen LanguageBind encoders for all non-motion modalities. The framework lacks novel contributions toward modality-specific understanding.

• We respectfully clarify that our method does not aim to advance modality-specific encoding (e.g., of text, video, or audio). Instead, we address the challenge that existing large-scale, multi-modal pretrained models, such as LanguageBind, contain rich implicit knowledge about human activity due to their training on large-scale datasets with content rich in human activity, but are not directly compatible with human motion representations.
• Currently, no sufficiently large-scale datasets exist with detailed human-motion annotations paired explicitly with other modalities (text, audio, images, video) at the scale required to retrain or substantially enhance encoders like LanguageBind. Given this limitation, retraining these encoders from scratch or fine-tuning them for motion tasks is impractical and data-limited.
• Therefore, our proposed method, introducing MuTMoT with lightweight adapters, is strategically designed as the optimal approach to exploit the latent knowledge within pretrained models like LanguageBind. Specifically, MuTMoT encodes detailed human-motion sequences and aligns them effectively to LanguageBind’s latent representations in a shared embedding space via these adapters.
• This design choice--recognized as "novel and unique contribution to the field", "technically sound and well-grounded," and "effective" by other reviewers--enables us to explicitly leverage and make the implicit multi-modal understanding already embedded in LanguageBind usable for downstream motion tasks, achieving robust multi-modal alignment and strong downstream performance across a diverse range of motion-related tasks without requiring large-scale paired motion data.

Only text-conditioned generation is quantitatively evaluated.

• As acknowledged in the limitations section, our quantitative evaluation for motion generation focuses on the widely studied text-to-motion generation task, and this decision reflects the current state of the field, where standardized and robust benchmarks exist primarily for text-conditioned motion generation. While we recognize the importance of evaluating other modalities, there is currently a lack of well-established benchmarks for image- or video-conditioned motion generation beyond pose estimation. That said, our framework is "an original step," as noted by Reviewer cqJT, to support any-to-motion generation at inference time without additional training, thanks to its multi-modal alignment.
• Thus, we argue that our work should not be viewed as limited in scope. In addition to achieving state-of-the-art or highly competitive results on text-to-motion generation, we demonstrate strong performance on six additional benchmarks across motion reconstruction, cross-modal retrieval, and action recognition. Our primary goal is to showcase the versatility and generalizability of our multi-modal framework across diverse tasks. We believe the breadth and rigor of our evaluation present a compelling case for the effectiveness of our approach. We agree that expanding quantitative evaluation to additional conditioning modalities is a valuable direction for future work, and we plan to explore this in subsequent research.

The supplementary generation videos exhibit noticeable artifacts.

• We acknowledge that some of the generated motions exhibit artifacts. Such issues are well-known limitations in current motion generation systems and reflect broader challenges in modeling fine-grained physical realism. These artifacts represent an active area of research and ongoing improvement across the field. In the final manuscript, we will include a systematic analysis of common failure modes and provide a comparative assessment against existing methods to contextualize these challenges more clearly.

Although the model achieves strong retrieval performance, removing text paraphrasing causes R@1 to drop from 69.56 to 62.74, as shown in Sec. B.3, Table 3.

• We agree that text paraphrasing augmentation contributes to improved retrieval performance by enhancing the model’s robustness to linguistic variation. However, we emphasize that MuTMoT’s architectural design also contributes meaningfully to the overall gains. Specifically, the removal of core architectural components, such as adapters, leads to even larger performance drops (R@1 drops to 62.03). This indicates that while data augmentation via paraphrasing improves the model’s robustness, our framework’s architectural choices are equally crucial for achieving strong retrieval performance. We believe that both the data and the model design play complementary roles, and we have included these ablations to transparently present their respective impacts.

While training and inference are briefly described in the supplement, the paper lacks a step-by-step explanation or diagram of the overall motion generation pipeline.

• While Figure 1 provides an overview of the REALM architecture and the motion generation pipeline, we agree that a more detailed, step-by-step depiction would improve both clarity and reproducibility. In the final manuscript, we will revise the figure to explicitly illustrate the full end-to-end motion generation process and include a corresponding textual walkthrough to clearly explain each stage.

During contrastive training, all non-corresponding modality-motion pairs appear to be treated as negative samples, without consideration for potential semantic similarity. This could penalize semantically related but unmatched pairs (false negatives), potentially degrading the embedding granularity and generalization ability.

• We respectfully disagree with the assumption that all non-corresponding modality-motion pairs are treated as negative samples. As described in LL201-208, we introduce a dynamic semantic margin component into the InfoNCE loss, which modulates the contribution of negative pairs based on the semantic similarity between their associated non-motion inputs. Specifically, if two non-motion inputs are semantically dissimilar (below a threshold in the LanguageBind space), the corresponding pair is penalized more heavily. This mechanism helps mitigate the impact of false negatives and preserves embedding granularity and generalization as shown in the ablation experiment (Table 4).

Response to Additional Questions:

Could you clarify the impact of frame-wise conditioning versus simpler global token conditioning through ablation?

• Thank you for the insightful question. We conducted an ablation to directly assess the impact of Frame-Wise Conditioning (FWC) versus Global Token Conditioning (GTC), and the results are summarized in the table below:

Table 1: Impact of Frame-Wise Conditioning (FWC) vs. Global Token Conditioning (GTC). Evaluation on the KIT-ML test set.

These results demonstrate that FWC provides consistent gains across all key metrics. Notably, FID improves by 18.8%, and R@1 increases by 3.0 points with FWC. This can be attributed to FWC’s ability to provide fine-grained, temporally aligned conditioning signals at every timestep, as opposed to a single global token that may lose important temporal cues.

How sensitive is the model to the quality of the reference motions?

• To assess this, we conducted an ablation varying the retrieval set size (top-$k$). We randomly sample two references from the top-$k$ most semantically similar motions in a reference database (training set of the benchmarks). Similarity is computed using cosine distance between text-conditioning embeddings and MuTMoT motion embeddings.

Table 2: Impact of Reference Motion Quality

These results show that REALM performs consistently well across a range of top-$k$ values, with only gradual performance degradation as less relevant reference motions are included. Notably, even in the absence of reference motion (No Ref), the model still achieves competitive results, confirming the robustness of our generation pipeline.

In Closing

We believe our responses, together with the new results and additional clarifications, help address the raised concerns and reaffirm the significance of our contributions. We kindly hope the reviewer will consider these in their final assessment.

Thank you again for your kind response and for taking the time to review our rebuttal. We truly appreciate your constructive feedback in helping to improve our work.