MotionClone: Training-Free Motion Cloning for Controllable Video Generation

We are very encouraged to see that you found our work " more accessible and efficient". We sincerely thank you for your valuable suggestions, which certainly help improve our work. We have accordingly refined our paper as follows:

Q1: Achieving motion cloning by constraining the temporal attention map has been explored in UniEdit and MotionMaster.

A1: Thanks for your constructive comment. Both UniEdit[1] and MotionMaster[2] are indeed pioneers in employing temporal attention maps for motion control. However, the main contribution of the proposed MotionClone is that it reveals the dominant components in temporal attention maps drive motion synthesis, while the rest mainly capture scene-specific noise or very subtle motions. Based on this observation, a primary motion control strategy is designed to perform substantial motion guidance over sparse temporal attention maps, allowing for efficient motion transfer across scenarios. Furthermore, MotionClone demonstrates the effectiveness of employing motion representation extracted from a certain time step, which only performs a single denoising step without time-costing DDIM inversion. Meanwhile, MotionClone demonstrates its capability to execute motion transfer across scenarios with substantial structure differences, which is particularly crucial for camera motion transfer, as illustrated in Fig. 20 of the updated manuscript. In comparison, both UniEdit and MotionMaster directly copy vanilla temporal attention maps for motion injection and pose significant reliance on DDIM inversion, which produces limited motion transfer and demands higher computational consumption in practical deployment. We will include these two methods in the introduction and make a detailed description of the difference between them.

Moreover, as suggested by Reviewer 3PSB, experiments have demonstrated the effectiveness of MotionClone in Diffusion DiT by using CogVideoX[3] (results are presented in Fig. 15 of the updated manuscript), which further validates its potential and generalizability.

Q2: The motivation or insight of the proposed MotionClone.

A2: Thanks for your valuable comment. As shown in Fig. 2 in the manuscript, it is observed that primary control over sparse temporal attention maps produces enhanced motion alignment than plain control over complete temporal attention maps. Motivated by this observation, we postulate that this is because not all temporal attention weights are essential for motion synthesis, with some reflecting scene-specific noise or extremely small motions. Indiscriminate alignment with the entire temporal attention maps dilutes critical motion guidance, resulting in suboptimal motion cloning in novel scenarios. To validate our assumption, we have visualized the mean intensity of the sparse attention map in Fig. 3 in the manuscript, in which the sparse attention map effectively indicates the area and magnitude of motion and suppresses the interference from motion-independent structures, demonstrating its potential for motion guidance. Therefore, we propose MotionClone, a training-free motion cloning solution for controllable video generation, which demonstrates efficiency by avoiding cumbersome inversion processes and offers versatility across various video generation tasks, establishing itself as a highly adaptable and efficient tool for motion customization.

We are very encouraged to see that you found our work simple but effective and identified it as a practical solution for deployment. We sincerely thank you for your valuable suggestions, which certainly help improve our work. We have accordingly refined our paper as follows:

Q1: Ablation results. The experiments are exhaustive, but they are only the qualitative results. It is necessary to show the quantitative results for each component.

A1: Thanks for your valuable suggestion. We have supplemented the quantitative results of the ablation study, i.e., the CLIP similarity with textual embedding for prompt-following evaluation and the optical-flow based consistency between the generated videos and the reference videos for motion alignment assessment (based on the suggestion of Reviewer gxZm). The detailed results are listed below.

It is observed that plain control over the full temporal attention map generates poor motion alignment. Meanwhile, MotionClone ($t_\alpha = 400$) is able to produce better motion customization compared to other time steps. Inversion.1 (time-dependence representation from DDIM inversion) fails behind Inversion.2 (representation in $t_\alpha = 400$ from DDIM inversion), which validates that the latter can provide more substantial and consistent motion guidance. Further, it is observed that precise prompt has little effect on video quality.

Q2: It is necessary to provide the ablation results of temporal attention in down blocks.

A2: Thanks for your constructive suggestion. We have evaluated the performance of applying motion guidance in down blocks, the qualitative results are represented in Fig. 19 of the updated manuscript, and the quantitative results are listed below.

It can be observed that "up_ block.1" showcases superiority in motion customization, which performs better motion alignment while enabling effective semantic preservation. We conjecture this is because the video motion is determined by the structure of each frame, which is mainly modeled in the "up_ block.1", as validated by previous image structure customization work (e.g., FreeControl [1]).

However, it is worth noting that "up_ block.1" is not necessary for MotionClone to apply motion guidance. As suggested by Reviewer 3PSB, we have validated the effectiveness of MotionClone in Diffusion DiT architecture like CogVideoX[2] (results are presented in Fig. 15 of the updated manuscript), in which we employ an auxiliary attention map in the self-attention module for motion representation. Therefore, MotionClone is expected to work in modules containing rich motion features.

We are highly encouraged to see that you found our work interesting and novel. We sincerely thank you for your valuable suggestions, which certainly help improve our work. We have accordingly refined our paper as follows:

Q1: Can the proposed method be applied to the latest video methods, such as CogVideoX, which employ the full attention layers.

A1: Thanks for your constructive suggestion. We have validated the potential of the proposed MotionClone in the latest DiT-based T2V model, i.e., CogVideoX-2B [1], in which MotionClone demonstrates inspiring effectiveness in training-free motion customization by only using motion representation from a single denoising step, offering both generalizability and flexibility. The results are represented in Fig. 15 of the updated manuscript (the corresponding video files are provided in the updated supplementary materials), and the detailed implementations are listed below.

The video synthesis stage in Diffusion DiT involves the collaboration between textual token $s_{text} \in \mathbb{R}^{(1 \times n_1 \times c)}$ and image token $s_{img} \in \mathbb{R}^{(1 \times n_2 \times c)}$, in which $n_1$ and $n_2$ are token number, and $c$ is channel number. During video synthesis, CogVideoX leverages a full attention module for complete relation modeling, in which the resolution of the vanilla attention map thus reaches $1 \times (n_1+n_2) \times (n_1 + n_2)$. Since we focus on the dependence between image tokens for motion transfer across scenarios, we introduce an auxiliary attention map to better describe inter-frame dependencies. Specifically, within the self-attention module, the image token is reshaped into $s_{img} \in \mathbb{R}^{(1 \times n_{h} \times h \times w \times f \times c)}$, in which $n_{h}$ is the head number of multi-head attention, $h$ and $w$ are spatial resolution, and $f$ is frame number, the corresponding attention map $A_{aux} \in \mathbb{R}^{1 \times n_{h} \times h \times w \times f \times f}$ can be obtained by using self-attention in the last frame axis. Following the operation in Diffusion U-Net, MotionClone applies sparse constraints $A_{aux}$ according to the rank of their values. For given real reference videos, we use null-text as the default prompt for preparing motion representation to facilitate user-friendly deployment.

Q2: Some existing methods have explored motion guidance in T2V models, such as [1]. The authors should cite and discuss these works.

A2: Thanks for your valuable suggestion. We have supplemented the comparison with more methods, including the suggested MOFT[2] and three extra methods recommended by other reviewers, i.e., Space-Time Diffusion Features[3], UniEdit[4], and MotionMaster[5]. We will cite all of these compared methods and provide a corresponding introduction to them. The qualitative results are represented in Figs. 16-17 in the updated manuscript and the quantitative results are listed below. Note that MotionMaster can only handle camera motion, thus it is not included in the quantitative comparison with both camera and object motion.

As presented in Figs. 16-17 of the updated manuscript, the proposed MotionClone achieves better motion alignment while performing effective appearance preservation, thus offering superior motion customization. Such advantages are also validated in the above tab, in which MotionClone gains the best score in both textual alignment and temporal consistency. Moreover, it is worth mentioning that all of the compared methods require time-consuming DDIM inversion, which implies high computation costs in practical application. In comparison, the proposed MotionClone can extract motion representation from a single denoising step with a unified null-text prompt, offering both efficiency and flexibility.

Q3: The generated videos exhibit inconsistent details across different frames, such as the robot's eyes and legs in Fig. 1. A better T2V model might help to resolve these inconsistencies.

A3: Thanks for your valuable comment. MotionClone aims to provide a training-free motion customization solution for existing video diffusion models, however, its capability is still limited by the adopted base model. Given that MotionClone has demonstrated its effectiveness in Diffusion DiT architecture (i.e., CogVideoX[5]), it is expected that its performance can be further enhanced with more advanced models. For instance, MotionClone can handle up to 49 frames in CogVideoX, unlocking its potential for long video motion customization.

Reference

[1] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

[12] Xiao, Zeqi, et al. "Video Diffusion Models are Training-free Motion Interpreter and Controller." arXiv preprint arXiv:2405.14864 (2024).

[3] Yatim, Danah, et al. "Space-time diffusion features for zero-shot text-driven motion transfer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Bai, Jianhong, et al. "Uniedit: A unified tuning-free framework for video motion and appearance editing." arXiv preprint arXiv:2402.13185 (2024).

[5] Hu, Teng, et al. "MotionMaster: Training-free Camera Motion Transfer For Video Generation." arXiv preprint arXiv:2404.15789 (2024).

We are very encouraged to see that you found our motion control strategy simple yet effective. We sincerely thank you for your valuable suggestions, which certainly help improve our work. We have accordingly refined our paper as follows:

Q1: This paper only employs CLIP scores to evaluate the temporal consistency of generated videos, which are not convincing. Please provide appropriate metrics like warping error to improve the soundness of MotionClone.

A1: Thanks for your valuable comment. Given motion of real videos is both diverse and temporally consistent, we use the optical-flow based consistency between generated videos and real reference videos for evaluating motion alignment. Meanwhile, considering that the generated videos are expected to adjust structure according to textual prompt (e.g., turns a black swan into a duck) and focus on primary motion, we introduce smooth operation on the estimated optical-flow to eliminate structural differences, which can be expressed as:

where $\mathcal{L} _ {ref}^{k\rightarrow k+1}$ is the estimated optical flow from the $k$-th frame to the $(k+1)$-th frame in reference video, and $\mathcal{L} _ {gen}$ refers to the optical flow of generated videos, $f$ is the frame number, and $\varphi(\cdot)$ is the smooth operation to eliminate structural differences (spatial mean filter is adopted in current implementation). We adopt RAFT[1] for optical flow estimation, and the corresponding scores of different methods under different mean filter sizes are listed below.

As presented in the above table, the proposed MotionClone achieves the best score in motion alignment under different mean filter sizes, validating its superiority in achieving effective motion transfer.

Q2: Add the comparison with MOFT [1] and Space-Time Diffusion Features [2].

A2: We have supplemented the comparison with more methods, including the suggested MOFT[2] and Space-Time Diffusion Features[3], and two extra methods recommended by other reviewers, i.e., UniEdit[4], and MotionMaster[5]. The qualitative results are represented in Figs. 16-17 of the updated manuscript and the quantitative results are listed below. Note that MotionMaster can only handle camera motion, thus it is not included in the quantitative comparison with both camera and object motion.

As presented in Figs. 16-17 of the updated manuscript, the proposed MotionClone achieves better motion alignment while performing effective appearance preservation, thus offering superior motion customization. Such advantages are also validated in the above tab, in which MotionClone gains the first score in both textual alignment and temporal consistency.

Moreover, it is worth mentioning that all of the compared methods require time-consuming DDIM inversion, which implies high computation costs in practical application. In comparison, the proposed MotionClone can extract motion representation from a single denoising step with a unified null-text prompt, offering both efficiency and flexibility.

Q3: The supplementary materials only show the generated videos of the proposed method. Could the authors provide the comparisons with other methods ?

A3: Thanks for your constructive suggestion. The videos of the compared methods are provided in the updated supplementary materials.