Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance

Dear Reviewer TUep,

Thank you for the constructive comments. We are glad that you found our implementation details sufficient for reproducibility. We will also open-source our code and model.

Q1: A thorough discussion of related works [1,2] would help contextualize the contribution.

A1: Thanks for your constructive feedback. We are happy to clarify their differences and incorporate this discussion into the revised Related Work section.

Works [1,2] and Mover are simlar in that they condition the generation on latent motion trajectories. However, they differ in the methodology and task. (i) Methodology difference: Works [1,2] rely on pretrained DINOv2 features to transfer object representations across frames, yet both papers highlight DINOv2’s limitations in video generation. First, DINOv2 excels at high-level semantic encoding but lacks fine-grained object details. Thus, GEM [1] employs additional identity embeddings to distinguish objects and train an ObjectNet to bridge the domain gap between DINOv2 and the UNet’s feature space. Second, CAGE [2] observes that DINOv2 features inherently retain spatial positioning. This makes it hard to disentangle object identity from the original location, finally reducing controllability. Moreover, the 14 $\times$ 14 patch size in DINOv2 may restrict the granularity of the proposed control. In contrast, our approach mitigates these limitations without relying on any auxiliary modules (e.g., identity embeddings). (ii) Task difference: As you mentioned, we focus on general-purpose video generation, supporting a significantly broader range of motion control tasks compared to [1-2].

Q2: The qualitative results contain a limited number of large-motion examples, raising questions about the model performance under larger and more out-of-distribution motion conditions.

A2: Our visualization results include a number of large-motion cases, such as boxing (supplementary video: 10–15s) and aerobics (47–52s). The supplementary video also shows that Mover performs noticeably better than compared methods in high-amplitude motion scenes (47s-1:11). This capability is supported by our data and benchmark construction process, where we specifically retained a wide range of large-motion cases using expert models and motion-based filtering (Lines 165–170 and 204–207).

To further address your concern, we curate subsets from MoverBench containing high-amplitude and out-of-distribution motion control cases. Evaluation results on these challenging examples are shown in Tables R5 and R6. Notably, Mover consistently outperforms two leading baselines [3,4], with performance gaps further widening under these difficult condition. In addition, Mover’s performance only marginally drops compared to its results on the full benchmark, demonstrating its robustness. Due to rebuttal guidelines, we cannot include additional visual examples here, but we will provide more such cases in our upcoming project page and include these two tables for clarity.

Table R5: Comparisons on MoverBench large-motion subset. For each video, its motion amplitude score is computed as the average of the top 2% largest optical flow values extracted by RAFT [5]. The top 20% highest-score videos are selected.

Table R6: Comparisons on MoverBench out-of-distribution motion subset. We manually curated 50 uncommon motion cases, including complex foreground–background interactions, objects moving out of frame, and rare camera motions.

Q3: One limitation of the proposed control format is its low resolution. Could the authors provide insight into how this impacts fine-grained control?

A3: We use a VAE encoder to compress motion trajectories with a spatial downsampling ratio of 8, which is a relatively low compression. It means that when the trajectory displacement exceeds approximately 8 pixels, the model can theoretically distinguish them. Even when the motion amplitude is very small (a particularly demanding case), the model may still infer the intended motion by leveraging the text prompt (e.g., "slightly move to the left") and the contextual motion patterns from nearby regions.

To further address your concern, we conduct a targeted evaluation. We sample 100 videos from MoverBench and, for each video, select one point trajectory with total displacement falling into one of four ranges: <8 pixels, 8–16 pixels, 16–32 pixels, and >32 pixels. We perform motion control using the selected trajectories and evaluate the control precision by end-point error (EPE). As shown in Table R7, Mover maintains reasonable control accuracy even for very small motions, demonstrating fine-grained motion control capability.

Table R7: Comparisons of motion control performance at varying motion amplitudes.

Moreover, there are two potential solutions to further alleviate the resolution limitation. (i) We have already trained Mover based on the I2V-14B-720P backbone, increasing the video resolution from 480P to 720P and enabling finer motion granularity. We are happy to release this model. (ii) Another direction is to incorporate explicit pixel-level position embedding into our feature replication mechanism. We leave this as a promising future work.

Q4: Have the authors tried using external visual encoders instead of the native encoder or random track embedding?

A4: Thanks for the suggestion. During the rebuttal phase, we evaluate two external visual encoder designs. (i) Mover-Motion_Encoder: we add learnable convolutional layers (like Tora’s 'Trajectory Extractor') after the feature replication step to further encode the motion condition. (ii) Mover-DINOv2: We extract features from a pretrained DINOv2 model and resize them to match the latent feature map for conditioning. As shown in Table R8, both variants underperform compared to our approach. Mover-Motion_Encoder increases optimization complexity and degrades performance. Mover-DINOv2 yields the weakest results. It aligns with prior findings (see Q1 & A1) that DINOv2 features, while semantically rich, introduce limitations for motion-controllable video generation. We are happy to incorporate this experiment for a more complete ablation in the revised paper.

Table R8: Comparisons of visual encoders on MoverBench.

References

[1] Hassan, et al. Gem: A Generalizable Ego-vision Multimodal World Model for Fine-grained Ego-motion, Object Dynamics, and Scene Composition Control. In CVPR 2025.

[2] Davtyan, et al. CAGE: Unsupervised Visual Composition and Animation for Controllable Video Generation. In AAAI 2025.

[3] Zhang, et al. Tora: Trajectory-oriented Diffusion Transformer for Video Generation. In CVPR 2025.

[4] Li, et al. MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance. In ICCV 2025.

[5] Teed, et al. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In ECCV 2020.

Dear Reviewer wpp4,

Thanks for the constructive comments. Please find our response to the specific points below.

Q1: Latent feature replication is conceptually similar to the pixel replication methods.

A1: We would like to clarify that latent feature replication is non-trivial. Compared to pixel-level replication, it offers two advantages crucial for overall performance. (i) Pixel-replicated videos produces out-of-distribution visual inputs for pretrained VAEs, typically requiring auxiliary encoder networks for enhanced performance (e.g., Tora [1], discussed in Lines 93-97). This complicates the fine-tuning of video-generation backbones, which contrasts with our core insight. (ii) Individual pixels lack semantic and contextual information (referred in Lines 34-35). Directly copying them hinders motion pattern alignment across surrounding regions. In contrast, latent features carry richer contextual cues, enabling more coherent motion conditions.

These advantages were rigorously validated in Table 4 and Figure 8 of the main paper, both quantitative and qualitatively. Specifically, our 'latent feature replication' approach demonstrates significant performance gains over the 'pixel replication', lowering FVD by 7.5.

Q2: Concatenating noisy video latents with condition inputs has been widely adopted such as in Wan.

A2: Yes, as clearly stated in Lines 135–138, this operation is used in prior I2V backbones such as Wan [2] and Lumiere [3] for fusing image conditions. Because this strategy is already integrated into the backbone's design, we follow the same approach to maintain compatibility and avoid modifying the model architecture. We also provide an ablation in Table 5, which confirms that the concatenation strategy works well for our task.

Q3: Clarify the distinctions between Mover and ATI.

A3: We note that ATI [4] (arXiv: May 28) was submitted after our NeurIPS 2025 submission deadline (May 15). Our work was developed independently.

Q4: Scalability claim needs more sufficient support.

A4: Thanks for your suggestion. Mover’s scalability mainly stems from preserving the base I2V model architecture. To this end, we (i) simplify motion control extraction by eliminating trajectory encoders like Tora’s [1], and (ii) avoid ControlNet-style fusion module that adds a new branch like MagicMotion’s [5]. To address your concern, we perform ablation studies to evaluate how these two designs facilitate scalability. The first compares Mover with a variant that adds a trainable motion encoder (denoted as Mover-Motion_Encoder), the second compares Mover with its ControlNet-style variant (denoted as Mover-ControlNet). Experiments are extensively conducted on two backbones, using the same data for fairness.

As shown in Table R3, our method achieves both better performance and faster convergence compared to alternatives. Mover-Motion_Encoder increases optimization complexity and degrades performance, Mover-ControlNet converges more slowly, possibly due to using zero convolutions, as also observed in ControlNext [6]. We are happy to incorporate the tables to better clarify our scalability.

Table R3: Scalability comparison of different approaches on two backbones.

Setting illustration: In Table R3, Mover-Motion_Encoder refers to that we add convolutional layers (following 'Trajectory Extractor' in Tora) after the feature replication step to further encode the motion condition. We define convergence as the point where evaluation metrics stabilize within a small range.

Q5: Mover and baseline methods use different backbones.

A5: We choose Wan as the backbone to pursue performance upper bound among open-source models. Notably, we can compare our performance with commercial model (Kling 1.5 Pro). To address your concern, we adjust Mover's backbone and training data scale to match those of two leading methods, MagicMotion and Tora. The comparisons under aligned settings are shown in Table R4. Under the same backbone and training data scale, Mover still outperforms these powerful methods. We are happy to include this table in our revised paper for clarity.

Table R4: Comparison of model performance under different training setups. Mover-Cog-23K and Mover-Cog-630K denote Mover variants under different backbones and training data scales.

Q6: How does Mover handle ambiguous control signals?

A6: Mover naturally addresses the ambiguity issue through two key mechanisms. (i) During inference, users can easily clarify intended motion by specifying more detailed point trajectories. For example, placing multiple points on an object with similar motion suggests global movement, while adding stationary points nearby helps indicate local movement. These adjustments can be made directly through the user interface.

(ii) Text prompt provides complementary motion guidance, further resolving potential ambiguities in control signals. During training, text prompts clearly describe the moving foreground subject and background context, paired with randomly sampled point trajectories. With large-scale training on diverse aligned samples, even given a single point trajectory, the model can rely on text prompts to infer the desired motion.

References

[1] Zhang, et al. Tora: Trajectory-oriented Diffusion Transformer for Video Generation. In CVPR 2025.

[2] Wan Team, et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv preprint arXiv:2503.20314 (2025).

[3] Bar-Tal, et al. Lumiere: A Space-Time Diffusion Model for Video Generation. In SIGGRAPH Asia 2024 Conference.

[4] Wang, et al. ATI: Any Trajectory Instruction for Controllable Video Generation. arXiv preprint arXiv:2505.22944 (2025).

[5] Li, et al. MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance. In ICCV 2025.

[6] Peng, et al. ControlNeXt: Powerful and Efficient Control for Image and Video Generation. arXiv preprint arXiv:2408.06070 (2024).

Dear Reviewer rUr2,

Thanks for your valuable feedback. We are happy to address your comments below.

Q1: Novelty of using trajectory control for video generation.

A1: We would like to clarify that using trajectories for motion control is not our novelty. Point trajectories, optical flow, and segmentation masks are all widely-used motion control forms in prior works, as explicitly discussed in the Introduction (Lines 28–36). Our method simply adopts point trajectories as the motion signal.

Our novelty lies in the feature replication mechanism, which edits the image condition features directly to enable motion control. This design removes the need for training extra motion encoders or injection modules like ControlNet, leaving the base I2V model architecture unchanged. Based on this insight, we validate the method at scale, construct a well-curated benchmark for evaluation, and will release an open-source model that even rivals Kling 1.5 Pro’s commercial Motion Brush. Together, our method, scaled training, evaluation benchmark, and model performance form an integrated contribution to the community.

We are also glad to see your recognition of the overall value of our work. We hope our insights and the released model will be helpful to both the research community and video creators.

After the rebuttal, most of my concerns are resolved, thus I will keep my positive score.

Dear Reviewer b1sz,

Thank you for the valuable comments and for recognizing the value of our method, the benchmark design, and the experimental results. We are happy to respond to each question below.

Q1: Compared to ControlNet that freezes the backbone and tunes half of the parameters, Mover changes the whole parameters of the original DiT backbone.

A1: We fully update the DiT backbone due to a better trade-off between video quality and inference cost. As shown in Table 5 of the main paper, compared to the ControlNet-based Mover variant, our full fine-tuning approach performs favorably ($\downarrow$ 0.2 FID and $\downarrow$ 1.1 FVD), while reducing test time by 22.4% and overall parameter size by over 30%. Moreover, despite updating the entire backbone, Mover effectively preserves the original image-to-video (I2V) capability of the Wan2.1-I2V backbone, as verified both quantitatively (Table 7) and qualitatively (Figure 9). This is because motion condition is occasionally dropped during training.

We also agree that plug-and-play modules have their advantages. In addition to the ControlNet variant, we now train a Mover-LoRA version, which retains Mover’s design while replacing full tuning with LoRA. This version is also a plug-and-play solution and adds only 1.8% parameters. As compared in Table R1 below, Mover (full fine-tuning) achieves the best overall performance over its two variants, i.e., Mover-ControlNet and Mover-LoRA. Notably, Mover-LoRA also consistently outperforms all academic baselines in the main paper. We are happy to release all variants to support diverse usage scenarios.

Table R1: Comparisons of training strategies on MoverBench.

Q2: Among existing methods, Motion Prompting is the most comparable in terms of training data scale, but it is not open-sourced. While a fully fair comparison is difficult, the work needs to clarify the differences in backbones and training data scale with baselines.

A2: We train Mover with large-scale data and a stronger backbone so as to pursue the best possible generation quality. As a result, Mover performs competitively with the commercial model Kling 1.5 Pro Motion Brush in human evaluations, which has not been achieved by prior academic works.

We fully acknowledge your suggestion and will clarify more setting difference. Although Motion Prompting is closed-source, we have compared one of its core design with Mover fairly. Specifically, Motion Prompting formulates motion guidance using random track embeddings, while Mover adopts latent feature replication. Based on the same dataset and backbone, we directly compare the two designs in Table 4 (quantitative results) and Figure 8 (qualitative comparisons). Our approach consistently outperforms the Motion Prompting's design in video quality.

Additionally, during the rebuttal phase, we adjust Mover's settings to align with two leading methods MagicMotion [1] and Tora [2], denoted as Mover-Cog-23K and Mover-Cog-630K, respectively. The detailed comparison on MoverBench is shown in Table R2 below. Under the same backbone and training data scale, Mover still outperforms these two powerful methods. We will include this table in our revised paper for clarity.

Table R2: Comparisons of model performance under different training setups.

References

[1] Li, et al. MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance. In ICCV 2025.

[2] Zhang, et al. Tora: Trajectory-oriented Diffusion Transformer for Video Generation. In CVPR 2025.