Trajectory attention for fine-grained video motion control

We sincerely appreciate your thorough review and feedback! Please find our detailed responses to your observations and suggestions below.

---

W1: Handling complex object dynamics in Image-to-Video cases

The paper raises concerns about object dynamics in the image-to-video case presented in the supplementary material. The examples, such as the dog and the cat, lack additional motion, which could be a limitation. It would be beneficial to see how objects with more complex dynamics are handled by the method.

The outcome dynamics depend on the region where trajectory control is applied and the sparsity of the control signal. By default, dense control is applied to all pixels, resulting in pixel-wise alignment. In contrast, when using sparser trajectories, the results exhibit greater variability, as illustrated in Fig. 8 (c) in the supplementary material (please also see the attached videos for better visualization). However, this approach involves a trade-off between control precision and the generated dynamics.

---

W2: Generalization of camera pose in Image-to-Video scenarios

There is a concern regarding the generalization of camera pose. In the Image-to-Video (first-frame) scenario, the trajectory module is trained with optical-flow data from only 10K video clips. It's unclear how the method would perform under challenging motions, such as clockwise rotation, high-speed zooming in and out, or 360-degree rotations like those seen in NVS-Solver GitHub. In these extreme trajectories, points visible in the first frame may become invisible, potentially leading to anti-aliasing issues. Additional results or a discussion of the necessary limitations would aid in a more comprehensive assessment of the proposed method.

Since our method does not rely on training with camera-annotated datasets, it can naturally generalize to various camera poses. As demonstrated in Fig. 6 of the supplementary materials, our approach effectively handles challenging scenarios such as high-speed zooming and clockwise rotation. However, achieving 360-degree rotations with 3D cycle consistency poses challenges for our method. Implementing 360-degree rotations would require additional design considerations, such as using both the starting and ending frames to perform interpolation tasks, similar to those in NVS-Solver [5]. We have also introduced necessary constraints to address these challenges (please see the limitation discussion in the supplementary material for details).

---

Q1: Customization of intrinsic and extrinsic parameters

How can one obtain or customize the appropriate intrinsic and extrinsic parameters when performing trajectory extraction for a single image or video? Does the camera always need to be directed at the center of the image?

Since we cannot precisely estimate the intrinsic and extrinsic parameters from a single image, we use predefined intrinsic parameters and some hyperparameters for extrinsic parameters. From our observations, these predefined parameters with statistics can effectively generate reasonable results. We can also adjust them accordingly.

As our approach is independent of specific camera settings and relies solely on generated trajectories, the camera's direction can be adjusted freely. For instance, in Fig. 8 (c) in the supplementary material, the camera is oriented towards the right side of the scene.

---

Q2: Dependency on depth information for parameter adjustments

Is it necessary to adjust the camera's intrinsic and extrinsic parameters based on the depth information available?

From our observations, these predefined parameters with statistics can effectively generate reasonable results. We can also adjust them accordingly.

Dear Reviewer,

Thank you for your valuable feedback and insightful comments throughout the discussion phase.

As the discussion period concludes on November 26th, we kindly ask if our responses have effectively addressed your questions. Please don’t hesitate to reach out with any additional questions, concerns, or requests for clarification.

Warm regards,

The Authors

Thank you for your detailed comments and suggestions! We have reviewed them carefully and provided explanations below to clarify the points of concern.

---

W1&Q1: Clarification of trajectory extraction processes

The paper does discuss trajectory extraction for different tasks such as camera motion control on images and videos, and video editing. However, the description of the extraction process could be more detailed and clear. For example, in Algorithm 3 for trajectory extraction from a single image, some steps might require further clarification for a reader who is not familiar with the underlying concepts. The estimation of the depth map and the rendering of views are steps that could be explained in more detail, including the methods and algorithms used. Similarly, in Algorithm 4 for video trajectory extraction, the point trajectory estimation and the combination with camera motion could be more clearly described.

Thank you for your feedback. We have included additional details in the supplementary material for clarification. For more comprehensive explanations regarding the estimation methods we use, clarification of certain concepts, and visualizations, please refer to Sec. A.3 and A.4 in the supplementary material. We will continue to address any remaining points if further clarification is required.

---

W2&Q2: Handling complex real-world scenarios

While the proposed trajectory attention method shows promising results in the presented experiments, there is a lack of exploration of more complex scenarios. For example, in real-world video data, there may be occlusions, rapid camera movements, or multiple moving objects, and it is not clear how the method would perform in such situations.

Thank you for your suggestions. We have included more examples of complex scenarios. As illustrated in Fig. 6 of the supplementary materials, our method effectively handles challenging cases such as occlusions, rapid camera movements (e.g., zooming in and out), and multiple moving objects. Additionally, we have expanded the discussion in the limitations section to provide a more comprehensive understanding of our approach.

---

W3&Q3: Comprehensive comparison with existing methods

The comparison with existing methods, although extensive to some extent, could be more comprehensive. There are many other techniques in the field of video motion control that were not included in the comparison, and it is possible that some of these methods may have unique features or advantages that could have provided a more nuanced understanding of the proposed method's superiority.

Thank you for your suggestions. We have conducted comparisons with the most relevant open-source methods (MotionCtrl [3], CameraCtrl [4], NVS Solver [5], Motion-I2V [6], anyV2V [7], I2VEdit [8]) in our experiments. For other related methods, we have revised the paper to include discussions. For example, while MotionBooth[12] offers camera motion control, its effectiveness is demonstrated only for simple pan motions. CamTrol [13] enables camera control by rendering incomplete warped views followed by re-denoising, which may become less effective when handling large incomplete regions. We were unable to include direct comparisons with it because we can not reach their code currently. For further detailed discussions, please refer to the revised paper, particularly the related work section.

Dear Reviewer,

Thank you for your valuable feedback and insightful comments throughout the discussion phase.

As the discussion period concludes on November 26th, we kindly ask if our responses have effectively addressed your questions. Please don’t hesitate to reach out with any additional questions, concerns, or requests for clarification.

Warm regards,

The Authors

We’re grateful for your constructive suggestions! Below, we’ve outlined our responses to address each of your concerns.

---

W1: Figure-1 clarity improvements

Figure-1 is confusing. It takes some time to understand the input and output of each task. It would be better to reorganize this figure to make it clearer.

Thank you for your suggestions. We have revised Figure 1 accordingly. To enhance clarity, we have used distinct colors to differentiate the reference contents, inputs, and outputs.

---

W2: Input scenarios in Figure-3

In Figure-3, it’s unclear what the model’s input is in two scenarios.

The primary purpose of Figure 3 is to illustrate the trajectory-conditioned generation process, which is general to the tasks discussed in the paper. The main distinction between these tasks lies in the trajectory extraction process, detailed in Sec. 4. However, we acknowledge that it is better to illustrate all these scenarios. Due to page limitations, these additional demonstrations have been included in the supplementary material. Please see Section A.3 for more explanations, as well as Fig. 4 for the visualization of the input of these two scenarios.

---

W3: Motion orthogonal to the image plane

The trajectory attention mechanism operates only in 2D space, making it challenging to distinguish motion orthogonal to the image plane—for example, whether a centered object is moving towards or away from the camera. The coordinates remain the same across frames. Could this be a limitation of this method?

Motion can be modeled whenever there are pixel shifts within the image space. For example, when an object moves toward the camera, it occupies more pixels due to perspective projection. To further support this concept, we have included additional examples in the supplementary material for verification. Please refer to Fig. 6 for examples of zooming-in and zooming-out scenarios.

---

Q1: Trajectory adherence in generated pixels

How do you ensure that when attention is applied along the trajectory, the generated pixel also follows the trajectory? Have you observed any cases where control fails?

The generated pixel can follow the trajectory because 1) the trajectory attention is trained to generate consistent results, and 2) the design of trajectory attention has a strong inductive bias, i.e., the attention has a specific goal with little ambiguity, making it easy to train and generalize. Because of this, we rarely see any failure cases. The control performance would degrade only when the motion trajectories are extremely sparse, e.g., below 1/32 of the original trajectories (Fig. 8 (a) in the supplementary material).

Q2: Explanation of {I_r} and {V_r} usage in Algorithms 3 and 4

In Algorithm 3, are you feeding {I_r} to the model in any particular format? The same question applies for Algorithm 4 with {V_r}.

Our input consists solely of the first frame and the extracted trajectory. The {I_r} and {V_r} are used for trajectory extraction. For more details, please refer to Fig. 4 in the supplementary material.

---

Q3: Fairness in evaluation comparisons

Is the comparison to other work (motion control/camera control) fair? They are trained on different datasets, and they may have some issue generalizing to the evaluation dataset used here. How did you select the evaluation set? Were you able to evaluate on the test set of other papers?

We have ensured a reasonably fair comparison. For the evaluation dataset, since most related works do not provide their datasets, we selected data from publicly available sources and datasets (e.g., DAVIS [11]) that are distinct from our training dataset. For the training dataset, MotionCtrl [3] and CameraCtrl [4] require specially annotated camera parameters, whereas our method only requires video datasets without such annotations. (Note that our method is also not sensitive to the dataset, as shown in Table 1 in the supplementary material.)

---

Q4: Training and inference trajectory consistency

In training, optical flow is used as a trajectory, but in inference, the model takes the camera trajectory as input. Could this cause a mismatch between training and inference? Why not use the camera trajectory as guidance during training as well?

The core idea of our work is to use trajectories for motion control. Camera trajectory is handled by first converting to pixel trajectories, and then seamlessly processed with our framework. As our method is still working with pixel trajectory, there is no mismatch between training and inference.

While MotionCtrl [3] and CameraCtrl [4] are specifically designed for camera control and use camera trajectory as a direct condition, we demonstrate in the paper that such high-level conditioning does not achieve precise control. Our trajectory attention, due to its strong inductive bias, is easier to train and to learn more precise control.

Dear Reviewer,

Thank you for your valuable feedback and insightful comments throughout the discussion phase.

As the discussion period concludes on November 26th, we kindly ask if our responses have effectively addressed your questions. Please don’t hesitate to reach out with any additional questions, concerns, or requests for clarification.

Warm regards,

The Authors

Thank you for your insightful feedback! We have addressed your concerns and provided detailed responses to each point below.

---

W1: Expanding to other architectures

The method is primarily designed for video diffusion models that use decomposed spatial-temporal attention. It is less clear how well the approach generalizes to models with integrated spatial-temporal attention (e.g., 3D DiTs) or other architectures. Expanding the evaluation to include such models would strengthen the contribution.

Our method can be seamlessly extended to DiT [10], as the key insight of modifying attention across frames remains applicable. We present qualitative results in Fig. 9 and provide a detailed explanation of the 3D DiT approach in the supplementary material (Section A.2) due to space constraints.

---

W2: Comparisons with recent or state-of-the-art methods

The paper compares the proposed method with a limited set of existing approaches. Including discussions with more recent or state-of-the-art methods, especially those that have emerged concurrently, would provide a more comprehensive evaluation of the method's relative performance. For example, Collaborative Video Diffusion uses epipolar attention to align contents of different camera trajectories, and Camco also uses epipolar, but to enhance the 3D consistency of generated contents.

Thank you for your suggestions. We have conducted comparisons with the most relevant open-source methods in our experiments (i.e., MotionCtrl [3], CameraCtrl [4], NVS Solver[5], Motion-I2V [6], anyV2V [7], I2VEdit [8]). Additionally, we have revised the paper to include discussions on other concurrent methods. For instance, Collaborative Video Diffusion [1] introduces a collaborative structure with epipolar attention for consistent camera-controlled generation, while Camco [2] also leverages the epipolar constraint for generation. However, the epipolar constraint is, in fact, a weaker variant of trajectory attention. Moreover, due to the current unavailability of their code, we could not include direct comparisons. For more in-depth discussions, please refer to the revised paper, particularly the related work section.

---

W3: Dataset in the experimental evaluations

The experimental evaluations are primarily conducted on the MiraData [9] dataset. While this dataset may offer certain advantages, relying on a single dataset limits the ability to generalize the findings. Evaluating the method on additional, diverse datasets would strengthen the claims about its general applicability.

Thanks to the strong inductive bias of our trajectory attention design, our method is data efficient and can generalize well even with a single dataset. To validate this claim, we have included a new table (Table 1) in the supplementary material, which shows that our approach is not sensitive to the dataset size or the training domains.

---

W4: Robustness to sparse, incomplete, or noisy trajectory information

While the method supports sparse trajectories, the paper does not extensively explore how it performs when the trajectory information is highly sparse, incomplete, or noisy. Real-world applications often involve imperfect data, so robustness to such conditions is important. Going back to my point 2, this is especially concerning since the model is trained on MiraData, which mostly consists of synthetic videos.

MiraData [9] actually incorporates lots of real-world data, with the training set primarily consisting of such samples. As illustrated in Fig. 10 of the supplementary material, the estimated optical flow used as training input is notably sparse and incomplete. This characteristic contributes to the robustness of our methods. Nonetheless, we acknowledge that our methods have limitations when handling extremely sparse trajectories (see Fig. 8 in the supplementary material), suggesting an intriguing direction for future research.

Dear Reviewer,

Thank you for your valuable feedback and insightful comments throughout the discussion phase.

As the discussion period concludes on November 26th, we kindly ask if our responses have effectively addressed your questions. Please don’t hesitate to reach out with any additional questions, concerns, or requests for clarification.

Warm regards,

The Authors

Thank you for your thoughtful feedback! We have provided clarifications and responses to your concerns below.

---

W1: Explanation of dense optical flow dependency

The method heavily relies on dense optical flow information, as shown in Figure 3 of the supplementary material. This dependency can significantly increase inference time due to the computational cost of processing dense optical flow, especially in real-time applications.

Processing optical flow does not significantly increase the overall time cost. For instance, generating or predicting dense optical flow for a video with a resolution of 1024×576 and 25 frames takes approximately 20 seconds, accounting for around 20% of the total inference time. This overhead is reasonable for video generation tasks. Also, our methods support relatively sparse trajectories, as shown in Fig. 8 in the supplementary material.

---

W2: Explanation of challenges in adapting to sparse trajectories

The reliance on dense optical flow makes it challenging to adapt the method to user inputs of sparse trajectories. As noted in DragNUWA, it's difficult for users to input precise trajectories at key points in practical applications, leading to a gap between training and inference. This limitation reduces the method's practicality in scenarios where only sparse motion cues are available.

Although our experiments primarily leverage dense optical flow, this approach also shows promise for sparse scenarios (as detailed in Section A.8 of the supplementary material). However, we acknowledge that our current methods are less effective at handling highly sparse trajectories. Our techniques are designed to provide a general and robust framework for utilizing available trajectories in motion control, as demonstrated in applications such as camera motion control and video editing. Developing user-friendly sparse trajectory designs, however, remains an exciting avenue for exploration.

---

W3: Clarification of H and W usage in the text and Algorithm 3

In line 158, H and W represent the dimensions of the latent features, but in Algorithm 3, H and W are used for image dimensions, which is confusing.

Thank you for your feedback. We have addressed this in the revised rebuttal version.

---

W4: Significance of examples in Fig. 6 and Fig. 9

Some examples in Fig. 6 and Fig. 9 are not significant, like the second example in Fig. 6.

Pictures may not always effectively highlight the differences. We recommend viewing the videos included in the supplementary material, where we have also added a video corresponding to Fig. 9.

Dear Reviewer,

Thank you for your valuable feedback and insightful comments throughout the discussion phase.

As the discussion period concludes on November 26th, we kindly ask if our responses have effectively addressed your questions. Please don’t hesitate to reach out with any additional questions, concerns, or requests for clarification.

Warm regards,

The Authors