MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms

We authors team sincerely acknowledge your insightful suggestions and positive feedback on this work. We would include reviewer hAdU in our acknowledgment if accepted due to their constructive suggestions. We would like to resolve your concerns as follows.

---

Q1: Discussing with Hertz et al. (2023).

A1: Thanks for pointing out this! We acknowledge the technical details of motion (de-)emphasizing and replacement is similar to the method in P2P. However, P2P does not show how to introduce self-attention into the editing process, which is what our method can do. Besides, introducing related techniques into the animation community and fine-grained explainability on attentions has also not been observed. We have revised accordingly in the PDF.

Q2: Other motion editing baselines.

A2: We are happy to discuss related baselines. We noticed that there are limited methods that can support interactive motion editing. We find the most related work is MotionFix (SIGGRAPH Asia 2024, to appear in Dec. 2024). MotionFix was uploaded to Arxiv in Aug. 2024 and the ICLR submission deadline is Oct.-01 2024. We treat this as a concurrent work. We also cite it in related work.

MotionFix needs human annotation from Amazon Mechanical Turk (AMT). As the interaction fashion is different from ours, we test some cases of MotionFix and provide the feedback to reviewer. MotionFix cannot achieve style transfer, replacement, and example-based motion generation, due to limited/no annotation about these tasks. Although their method can achieve some editing results like “jump higher”, MoitonFix cannot control the jumping times of the result. This is mainly due to its limited data annotations.

In contrast, we would like to highlight that our MotionCLR is promising to build a motion editing dataset in a very scalable way, which is what we are now working on.

Q3: Attention manipulation layers.

A3: This is an insightful question! We manipulate all attention layers and find it is the best setting. We infer the reason is that all attention matters. We provide the ablation in Appendix A.3 (Tab. 6) in our original submission.

Q4: Weight adjustment ways.

A4: Thanks for the detailed question! MotionCLR supports both adding and multiplication editing modes, which were specified in Appendix E.1 of the original submission. For convenient presentation, we provide only adding mode for presentation.

Q5: FID of manipulating results.

A5: The observation is right. We specified this question in Sec. 5.2 of the original paper: “When adjusting weights are too large, the attention maps are corrupted, resulting in artifacts.”. We hope this will resolve your question.

---

We would like to acknowledge reviewer hAdU again for their such positive recommendation, which encourages us a lot!

We appreciate the contribution to handling this submission and the constructive feedback. As this is an early attempt at motion editing, we do agree that this work will play an important role in the community. We would like to resolve your concerns as follows.

---

Q1: About the evaluation protocol.

A1: Your suggestion on the evaluation is quite insightful! We would like to resolve this concern from two aspects.

• As this work is an early exploration of motion editing and the early work exploring the explainability of cross-/self-attention in motion generation, the community still lacks a thorough evaluation protocol to evaluate the task.

• Although we treat this might be acceptable and tolerable for early exploration in the community, we try to additionally evaluate the alignment between the attention map with the root velocity via the IoU metric (Intersection over Union). For example, if the attention value of the attention is larger than 65% of the maximum attention values, we treat the word of action as activated. Similarly, we also applied this activation assessment on the velocity and found the execution area of actions. We calculate the IoU value of two areas as follows.

  adjusting weight	-0.1	0	+0.1
  IoU (%)	74.3%	75.5%	76.2%

We treat the results as complementary to the action counting result in Sec. 5.4. The IoU values show a good alignment between attention values and the action execution.

We have revised accordingly in the PDF (Appendix G). If reviewers have more suggestions on evaluation metrics, we are happy to include them, which will be useful to our community. We will also include reviewer b8bV in our acknowledgment if accepted, due to this constructive suggestion.

Q2: About scales of TMR-sim..

A2: Thanks for so detailed checking. You are right, we re-clarified them in the PDF.

---

Dear reviewer b8bV, if your concerns are resolved, could you please consider reevaluating this work and improve your rating? If not or you have further guidance, we are happy to resolve them.

Dear reviewer b8bV,

Thanks for your in-depth and insightful discussion! We are happy to know your high evaluations of our efforts!

We do understand your views on the quantitative metrics. For more clarification, we would like to share some evaluations in attention-based image editing areas, like MasaCtrl (ICCV-23), PnP (CVPR-23), and MoA (SIGGRAPH-24). Most of these methods rely on metrics like CLIP-score, PSNR, and DINO-features, which motivate us to use TMR-sim and FID metrics. Despite these, we author team quite respect your attitude on this and recognize this is a limitation in the visual editing community.

To make our results more convincing, we provided almost all the experiments we could think of before the submission. More importantly, we also package our method as a web application via gradio-app, whose video record is provided in the supplementary. All these efforts are to provide a holistic understanding of our method for readers.

We always value the suggestions from reviewers significantly and are looking forward to further updates.

Best,

MotionCLR author(s) team

Dear reviewer b8bV,

Thanks for your prompt reply and engaging to improve the quality of this submission with us.

---

We would like to answer your confusion on "sum-to-one" at first. This is because CLIP tokenizes a sentence into 77 tokens in all cases, including some placeholder tokens. In Fig.3-c, we only visualize the weight of "a", "person", and "jumps". As a result, the summation of three tokens is less than 1. If we visualize all 77 tokens, the figure will be not clear enough for presentation.

We do understand that, as a reader, you may be confused that the manipulation will be on the attention map or similarity matrix. We have thought about this before the submission. Therefore, due to page limits, we provide PyTorch-like codes in Code-1 (Appendix F.1, page 35). Actually, our manipulation is on the attention map (after Softmax).

P.S.: We guess that the reader will be curious about why we do not operate the similarity matrix directly. Answer: The value scales of the similarity matrix are without any guarantee (like "sum-to-one"), which is the motivation for using Softmax. Therefore, the adjusting weight in motion (de-)emphasizing is relatively hard to determine. With the sum-to-one property, it is much easier to determine the adjusting scale (like 0-1) for users. In this fashion, we can control the weight more precisely. You can refer to Fig.-14 (Appendix A.1, page 18). It is interesting that manipulation of the cross-attention maps will affect the number of "jump"s, additionally reflected by the self-attention maps in Fig.-14.

---

We thank you again for joining such a detailed discussion. We are still open to answering your question before the deadline.

Best,

MotionCLR author(s) team

The largest cross-attention weight for "jump" is approximately 0.05, as shown in Figure 3-(c). While this is about 5 times higher than the average weight (1/77 ≈ 0.013), this relatively small scale raises some concerns about the robustness of the word-frame correspondence. In this example, the combined attention weights for the three relevant tokens account for only 8% of the total attention at the peak frame, with the remaining 92% distributed across placeholder tokens. If the word-frame correspondence is indeed well-represented in the proposed architecture, should the attention weights for the sentence be larger?

Let's suppose Figure 3-(c) represents a typical result of well-aligned word-frame correspondence. In contrast to the relatively small scale of the observed cross-attention weights in Figure 3, the magnitude of the weight adjustment in Table 2 (+0.30) appears disproportionately large. Adjusting the weight of a single token by +0.30 out of 77 candidates would allocate 40% of the total attention to that token, making it 23 times larger than the average attention weight. This significant discrepancy raises questions about whether such adjustments align with the model’s learned dynamics.

Moreover, directly adjusting the attention weight risks violating the contraint of the sum-to-one. Especially, subtracting attention weights introduces a critical issue, as it can result in negative values. Considering that the attention map is a linear combination of token values constrained to a 77-dimensional simplex (where values range between 0 and 1 and the sum equals one), subtracting weights risks violating these constraints more severely than adding weights. Negative attention weights would push the model into an out-of-distribution state, as such values were never encountered during training, potentially leading to ill-posed outputs.

Table 2 reports acceptable FID scores for weight adjustments in the range of -0.4 to +0.5. However, considering the scale of the cross attention map, I believe the anlaysis on weight adjustment between -0.3 ~ 0.3 is more important, but the table 2 lacks the results. Also, the evaluation is based on only 100 examples that were manually selected by the authors. This raises questions about the representativeness of the results. Given that the HumanML3D test dataset contains many single-verb examples, it would be helpful to clarify why the authors chose to manually construct a test set instead of conducting experiments on all single-verb examples in the dataset. This manual selection process reduces the generalizability of the experimental results.

Additionally, although the authors mentioned that they considered the design choice for manipulating the similarity matrix, it would be beneficial if the rationale for this choice were more clearly explained in the paper or supplementary material, supported by quantitative evidence. Mentioning that this work represents early exploration in the text-to-motion domain does not fully address the lack of quantitative results. Providing a more thorough explanation and stronger supporting evidence would significantly strengthen the paper’s claims.

In conclusion, I respectfully disagree with the authors' assertions, as the paper generally lacks sufficient quantitative results on its main contribution—word-frame correspondence—and the design choices for the subtasks. As reviewer cnao pointed out, the authors' assertion regarding word-frame correspondence appears to require further validation. I believe that word-frame correspondence should first be rigorously proven before being applied to various subtasks. However, the authors conducted experiments on subtasks and used the results as evidence for word-frame correspondence. I find this approach problematic, as all the experimental results are indirect evidence of word-frame correspondence and heavily rely on qualitative results. This critically weakens the reliability of the assertion and leads me to lean more toward clear rejection. (In the image domain, Liu et al.*[1] analyzed self-attention and cross-attention, using their quantitative analysis as evidence to support and justify their method.)

Furthermore, the use of the term explicit modeling for the word-frame correspondence seems difficult to support, as no explicit guidance is provided at the word-frame level. For instance, in the image domain, Stable Diffusion employs cross-attention across modalities, but it would not typically be considered explicit modeling of region-word correspondence. To achieve fine-grained correspondence, models like GLIP have been proposed in the image domain. Similarly, I kindly suggest that the authors consider utilizing datasets with fine-grained labels, such as BABEL (although it is not a sentence-annotated dataset and requires preprocessing to generate sentence-level annotations), to establish truly explicit, fine-grained word-frame level modeling.

Thank you for providing additional experimental results. While I appreciate the authors’ efforts to address concerns, the justification for the manual construction of the test set is not particularly persuasive, and I do not believe that experimental results based on such a small, manually constructed test set are valid. The experiments in the reply are still based on a test set of only 100 samples, raising concerns about potential bias, as these samples may have been specifically constructed to support the authors' hypothesis. Additionally, the small size of the test set is insufficient to convincingly demonstrate the generalizability of their approach.

It would be helpful to know if the test set includes long text descriptions and how it performs in such cases. What are the effects of longer texts? How many different verbs are present in the test set? Conducting experiments on the HumanML3D test dataset could provide a more robust and straightforward resolution to these concerns.

Moreover, the design for adjusting the attention map lacks elaboration and does not account for the simplex property, further casting doubt on the validity of the attention map manipulation. Specifically:

• The authors' explanation in L363 states, "Therefore, the suggested value of the weights ranges from −0.5 to +0.5."
• There is an absence of experimental results in the range of -0.3 to 0.3 in the main paper.
• There is a lack of deeper analysis on cross-attention mechanisms (e.g., what happens when attention weights have negative values? How is the violation of the simplex property handled?).

These points suggest that the authors may not have fully accounted for the scale and behavior of the cross-attention map, as well as the implications of violating the simplex property. This raises concerns about whether the architectural design and the attention weight manipulation process were rigorously developed. Allowing users to adjust the weights does not fully address these issues.

The table provided in the reply shows that manipulating attention scores performs better than manipulating the similarity matrix, but this conclusion is drawn solely within the context of the small test set. As mentioned in my previous comment and supported with a reference, the authors should evaluate the effects of manipulation not only on the targeted words but also on other words. A clear comparison demonstrating the effectiveness of word-frame correspondence is necessary to substantiate their claims.

Furthermore, I find it difficult to agree with the architectural design being described as enabling "fine-grained and explicit word-frame correspondence". The authors neither utilize fine-grained annotations, such as those in the BABEL dataset, nor explicitly guide the training process to ensure fine-grained word-frame correspondence. Simply incorporating cross-attention does not guarantee fine-grained or explicit word-frame correspondence, and this does not qualify as a novel contribution of the paper.

Conclusion

While the authors have made efforts to address some of the concerns and I appreciate the messages the authors want to deliever to the community, the issues I raised remain significant. Addressing these concerns would require major revisions to the manuscript, including a re-evaluation of the paper's main contributions and the inclusion of more robust quantitative experiments to substantiate their claims. These changes are essential to strengthening the paper’s core arguments and logical structure. Given that ICLR is a top-tier conference on representation learning, the evidence provided does not yet meet the level of rigor required for acceptance. I believe the paper has strong messages if it's validated properly, so even though I can't agree with how to validate the assertion on this paper, I believe the paper will take important role once it clearly proves the correspondence.

Final Evaluation: I would like to lower my evaluation of this paper from 5: marginally below the acceptance threshold to 3: reject, not good enough

Reviewer DjgQ provides us with constructive suggestions to discuss the limitations and the emphasis on improvement. We authors appreciate this sincerely and will add DjgQ to our acknowledgment if accepted.

---

Q1: About OOD cases.

A1: We acknowledge our work will related to some OOD motions. Therefore, we author team is now collecting the motion data to enlarge our database and train our V2 model on the larger dataset. We hope our efforts will resolve this issue.

Q2.1: More complex motions.

A2.1: As shown in Fig. 7, our method can accurately handle a motion sequence with actions more than one. We also provide the attention map visualization of a complex motion in Appendix D (Fig. 26). The results show the good localization ability of attention map in a complex motion.

Q2.2: Test case issues and limitation discussion.

A2.2: Thanks for helping us enhance the exploration of the extreme test of our method. We test these cases of limited training samples. We trained the model as you suggested and found the generated results almost similar to the original unedited motion. The reason mainly comes from the limited cases the model has seen during training. As a result, the model cannot distinguish “cartwheel” and “turns”, because the model learns to memorize limited examples in training. That is to say, the model does not know what motion is not a “cartwheel”, thereby restricting the controllability of attention conditions. We included this in our limitation part of PDF.

Q3: Improved baseline comparison.

A3: Your suggestion is quite insightful! This work is based on MDM and MotionDiffuse. The direct comparison is as follows.

As our work introduces explicit modeling on word-level correspondence, it significantly improves the generation quality than baselines. We thank you for this constructive question, and will include you in our acknowledgement if accepted.

---

If you have no further concerns, could you please consider reevaluating this work and improving your rating? If not or if you have further guidance, we are happy to resolve them.

Dear Reviewer DjgQ:

Thanks again for your efforts in reviewing. We are looking forward to your reply. Would you mind checking our response and confirming if there are unclear explanations?

We are highly encouraged if your concerns have been addressed. If your concerns are resolved, could you please consider reevaluating this work and improving your rating? If you need any more clarification, we can provide it as soon as possible before the discussion deadline.

Best,

MotionCLR author(s) team

Dear reviewer DjgQ,

Thanks for your response and emphasizing our method is novel and creative! We quite acknowledge your effort to discuss with us and so detailed check.

1. Thanks for such detailed checking. For the DDIM/DPM-solver design choice, we treat both choices are acceptable/comparable, and both choices can be used in the inference stage. Besides, the R-Precision metric between DDIM and DOM-solver is comparable. As a result, we do not specify the default choice deliberately.
2. Thanks for this discussion. Because we need to annotate the words that should be edited in the sentence, we manually construct the evaluation set for this, following PnP [1] in the image editing domain. Because of the difference in evaluation sets, the results in Table 1 and 2 are not the same.
3. For the typo, we will revise it accordingly in the manuscript.

Your insightful comments help us to polish the paper a lot. For the fine-grained correspondence between cross-attention and the the final motion, we provide direct evidence to respond b8bV via the IoU metrics between cross-attention and the the final motion execution. This shows the attention activation is well aligned with the final motion.

[1]: Tumanyan, N., Geyer, M., Bagon, S., & Dekel, T. (2023). Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1921-1930). Citation:540

---

We paste the evidence to respond to b8bV via the IoU metrics as follows. If you have any suggestion to help us improve the evaluation, we are very very grateful.

Question: Is the cross-attention activation of the verb aligned with the action execution in the motion?

Answer:

We treat the following result as the direct verification of the “word-motion correspondence”.

We introduce the moment retrieval function in TMR, which can identify the action execution area in the motion. For example, for a motion generated by “a man walks forward”, we can calculate the similarity between the word “walks” and all 20-frame subintervals along the sequence. As a result, we can obtain the word-action similarity along the sequence. If the similarity value of the action is larger than 65% of the maximum attention values, we treat the action of the word is executed. Besides, if the attention value of the attention map is larger than 65% of the maximum attention value, we treat the word of action as activated in the attention map. Similarly, we also applied this activation assessment on the velocity and found the execution area of actions.

Note that the velocity-based and moment-retrieval-based approaches introduced here are for a "double-check", both of which are approaches to determining the action execution in the motion domain.

To verify whether the activation of word in attention is aligned with action execution, we calculate the IoU between (i) attention-activation and root-velocity-execution (E1), (ii) attention-activation and moment-retrieval-execution (E2).

As can be seen, compared with E3 (both explicit action indicators: speed, moment retrieval of TMR), the values in E1/E2 are similar to those in E3. Therefore, the cross-attention activation is well aligned with motion execution.

Moreover, we additionally provide two comparison groups (E4, E5). The setting is that, when calculating IoU, the cross-attention map for each example is replaced by an attention map of a negative example from the test set. As can be seen (E1 vs E4/E5), the cross attention is aligned with the action execution in the motion in the generation process of each motion. We treat results can prove the word-motion correspondence in the method.

Thanks.

MotionCLR author(s) team

We sincerely acknowledge your contribution to handling this submission and provide constructive suggestions for this work. We will add you to our acknowledgment if accepted.

---

Q1: About CLR module design.

A1: Thanks for helping us enhance the clarity of this paper! We would like to answer this question from two aspects.

• CLR module design. The main difference is that the CLR module models the correlation between the motion and each word and separates the timestep injection with the text condition. There is no previous work models text-motion so fine-grained and observe the explainability between each word and the motion sequence. We additionally provide an ablation on w/o down/up-sampling below. The results are similar to that of HumanML3D. The relatively smaller computation in down/up-sampling encourages us to set up this design choice.

  method	Top 1	Top 2	Top 3	FID	MM-Dist	Multi-Modality
  w/o down/up-sampling	0.540	0.731	0.818	0.101	2.984	2.152
  MotionCLR	0.542	0.733	0.827	0.099	2.981	2.145

• About replacing input condition style to cross attention style. The key insight of introducing cross-attention controlling is modeling word-level correspondence with motion, which supports our motion editing applications. The comparison with previous fashions is detailed in Appendix C.3 of the original submission. Besides, the ablation is also provided in Sec. 5.3.

Q2: Results on KIT dataset.

A2: Thanks for discussing the dataset generalizability! We provide our experiments on the largest text2motion dataset HumanML3D with 44970 texts. The KIT-ML is much smaller than HumanML3D, with 6278 texts. We think the results on a larger database are more convincing. Despite these, we also provide additional experiments on the KIT dataset as follows. The conclusion is similar to the result of HumanML3D.

Q3: About Section 3.3 relevance.

A3: We acknowledge you for helping enhance the readability of the paper! Sec. 3.3 is critical to our understanding of cross-/self-attention in motion generation, which supports our versatile downstream tasks, especially Remark 1 and Remark 2. Considering your concern, we agree to merge them with Sec. 3.2 to enhance the readability.

Q4: Positional embedding in cross-attention.

A4: Thanks for discussing this! We treat the positional embedding of CLIP embeddings as a common setting. Therefore, we did not specify this in the original text. The positional embeddings are added to the text directly as most methods do. Thanks for pointing it out. Here is the PyTorch codes.

def encode_text(self, raw_text, device):
    texts = clip.tokenize(raw_text, truncate=True) # [bs, context_length]
    x = self.clip_model.token_embedding(texts).type(self.clip_model.dtype)  # [bs, n_ctx, d_model]
    x = x + self.clip_model.positional_embedding.type(self.clip_model.dtype)# add position emb.

Q5: Long text test.

A5: Good question! We do not find the text with more than 5 words in the HumanML3D dataset, due to the limited number of such samples. Besides, the limited duration of less than 10 seconds in the dataset cannot contain such complex motions. We provide several visualizations of longer text inputs with multiple action verbs in Appendix A.5 of the original submission. Besides, results in Appendix D also verify the explainability of the case with multiple actions.

---

Dear reviewer cnao, if your concerns are resolved, could you please consider reevaluating this work and improving your rating? If not or if you have further guidance, we are happy to resolve them.

Dear reviewer cnao,

Thanks for your reply. We do understand your point that the mentioned RPE/CTC methods may enhance the result. We also agree the relative task will benefit from a larger-scale of motion database. However, the HumanML3D is the largest text2motion dataset that can accessed. We do believe our method will be more scalable when obtaining a large database, which is also our future work. Besides, if you have more replies, could they be posted under your review? We are worried that the current replies will be confused with DjgQ's replies. Thanks a lot!

Best,

MotionCLR author(s) team

Thank you for moving our discussion to the correct location. I mistakenly added my comment in the wrong place earlier. After interacting with the author and considering the feedback from reviewer b8bV, I also lean towards recommending rejection of this work.

While I acknowledge that this paper presents an earlier attempt at motion editing focusing on cross-attention between motion and words, and the overall quality is commendable with diverse experiments and analyses, it falls short in a critical aspect: the lack of quantitative evaluation for the proposed method. This omission makes it difficult to assess the true impact of the work. As the author mentioned, this work might perform well for motion editing without a specific design for motion-word correspondence. However, it is difficult to substantiate the author’s claim based solely on qualitative results. Additionally, the absence of such evaluations leaves the claims of the paper unsupported and raises concerns about reproducibility and practical relevance. I am confident that with proper quantitative evaluations, this paper would likely reach the level needed for acceptance. However, in its current form, I cannot consider it acceptable. I recommend that the authors incorporate quantitative evaluations and resubmit the work. Accordingly, I am changing my rating from borderline reject to reject.

Dear reviewers,

Thanks again for your constructive suggestions! As the discussion period is quite tight, we would like to send you a kind reminder. We were wondering whether you had the chance to look at our response and whether there is anything else you would like us to clarify.

We sincerely hope that our response regarding your concerns will be taken into consideration. If you have no further concerns, could you please reevaluate this work and consider improving your rating? If not, please let us know and we remain open and would be more than glad to actively discuss them with you.

Best,

MotionCLR author(s) team