SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation

We thank the reviewer for the insightful feedback. In response, we have revised the paper and supplementary website, incorporating the reviewer’s suggestions. All modifications in the revision based on the reviewer’s feedback are highlighted in orange. In the following, we address each of the comments and questions.

W1 & W4: Complex nonlinear & longer trajectories with multiple objects.

We appreciate your constructive feedback. We have updated the website (https://sgi2v-paper.github.io/gallery_VIPSeg.html) to include results with non-linear trajectories with multiple (>2) constraints. We are working to add more examples and hope these results address your concerns.

W2: Parameter robustness.

Thank you for your insightful feedback. While we conduct hyper-parameters analysis on VIP-Seg, we directly apply the same set of hyper-parameters to generate qualitative results using web images. This demonstrates the generalizability of the hyper-parameters. One of the strengths of our approach is that hyper parameters can be easily adjusted without an expensive re-training procedure.

W3: Experiments on other image-to-video diffusion models.

Thank you for your suggestion. We are currently extending our experiments to include other types of video diffusion models, and hope to demonstrate this in future work. We kindly note the following:

• SVD was chosen due to its wide application in previous image-to-video generation papers. Importantly, state-of-the-art open-sourced trajectory-controlled methods (at the time of submission) such as DragAnything and DragNUWA were all built on SVD.
• SVD is unique in that it requires only images as inputs. Other image-to-video diffusion models usually require additional text prompts as inputs, making evaluation challenging because text prompts strongly influence motions.

Given these considerations, we believe SVD is the natural choice for our experiments. We are exploring extensions of the method to emerging architectures, such as video diffusion transformers and hope to tackle this problem in future work.

W5: TraDiffusion and DragEntity

Thanks for bringing up these works. We have revised Section 2 to discuss TraDiffusion and DragEntity (highlighted in orange).

Briefly:

• TraDiffusion is a method for controlling spatial layouts in text-to-image generation. Its underlying techniques are similar to Self-Guidance[1], which is already discussed in the paper.
• DragEntity is a supervised method for trajectory-controlled image-to-video generation. Its techniques are similar to DragAnything, which is discussed in the paper. As the authors did not make the code publicly available, we could not perform experiments with it.

We hope these address your concerns, and we are happy to discuss further if needed!

References:

[1] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynskio. Diffusion Self-Guidance for Controllable Image Generation, 2023.

We thank the reviewer for their insightful and positive feedback. In response, we have revised the paper. Below, we address each of the weaknesses and questions in your review.

W1. MotionCtrl with SVD backbone.

We appreciate the suggestion. After thoroughly reviewing the official MotionCtrl codebase, we could only find a camera-controlled model using the SVD backbone. Nevertheless, we have compared our method against other representative trajectory control models with SVD backbones that are designed specifically for dragging-based input, such as DragAnything and DragNUWA. We believe these comparisons sufficiently demonstrate the efficacy of our approach.

W2: Inference time.

Thank you for your question. Please refer to our general response for the result.

Q1: Experiments on other image-to-video diffusion models.

Thank you for your suggestion. We are currently extending our experiments to include other types of video diffusion models, and hope to demonstrate this in future work. We kindly note the following:

• At the time of development, open-sourced, high-quality DiT-based image-to-video diffusion models (e.g., CogVideoX-I2V) were not publicly available.
• SVD was chosen due to its wide application in previous image-to-video generation papers. Importantly, state-of-the-art open-sourced trajectory-controlled methods (at the time of submission) such as DragAnything and DragNUWA were all built on SVD.
• SVD is unique in that it requires only images as inputs. Other image-to-video diffusion models usually require additional text prompts as inputs, making evaluation challenging because text prompts strongly influence motions.

Given these considerations, we believe SVD is the natural choice for our experiments. We are exploring extensions of the method to emerging architectures, such as video diffusion transformers and hope to tackle this problem in future work.

We hope these address your concerns, and we are happy to discuss further if needed!

We thank the reviewer for the insightful feedback. In response, we have revised the paper and supplementary website, incorporating the reviewer’s suggestions. All modifications in the revision based on the reviewer’s feedback are highlighted in purple. In the following, we address each of the comments and questions.

W1: Novelty.

Please refer to our general response for a detailed discussion. We believe our paper offers several novel insights that are not obvious from previous works like DragDiffusion and FreeTraj. These contributions have the potential to significantly impact both the zero-shot and video-generation communities.

W2: Motivation of Gaussian heatmap.

We appreciate the reviewer’s feedback and revised the paper to clarify the motivation for using the Gaussian heatmap in Section 3.3, Appendix C, and Table 2 (highlighted in purple). Since we use a bounding box to identify the region to move, it may include background pixels not intended to be moved. The Gaussian weighting alleviates this issue by focusing on pixels at the center while reducing focus near the edges of the bounding box. As shown in Table 2, the use of Gaussian weighting slightly but consistently improves the performance in all metrics on VIPSeg. In addition, it only adds negligible overhead for memory and inference speed. Therefore, we adopt it in our method.

W3: ObjMC is flickering in Figure 10.

Thank you for pointing this out. We have added explanations to Section 4.4 (highlighted in purple) and the website (https://sgi2v-paper.github.io/gallery_timestep2.html). We note that ObjMC only measures trajectory accuracy without considering visual quality. As demonstrated on the website, when optimizing latents in early timesteps (i.e., t=[10, 1]) the generated video contains severe artifacts along the target trajectory (as indicated by the bad FID and FVD scores). Yet, CoTracker is robust enough to also track these artifacts along the trajectory, leading to a decreased ObjMC score.

W4. Inference time and memory.

Thank you for your feedback. Please refer to our general response for the result.

W5: Visualized results on the VIPSeg dataset.

Thank you for the great suggestions. Results on the VIPSeg dataset have been added to our website (https://sgi2v-paper.github.io/gallery_VIPSeg.html) and Appendix B. These results demonstrate that our method effectively handles multiple bounding boxes with nonlinear trajectories.

If you have any further concerns, we are happy to discuss them.

We thank the reviewer for their insightful feedback. We have revised the paper to address the reviewer’s comments. All modifications in the revision are highlighted in blue. In the following, we address each of the comments and questions in the review.

W1: experiments on other image-to-video diffusion models.

Thank you for your constructive suggestion. We are currently extending our experiments to include other types of video diffusion models, and hope to demonstrate this in future work. We kindly note the following:

• At the time of development, open-sourced, high-quality DiT-based image-to-video diffusion models (e.g., CogVideoX-I2V) were not publicly available.
• SVD was chosen due to its wide application in previous image-to-video generation papers. Importantly, state-of-the-art open-sourced trajectory-controlled methods (at the time of submission) such as DragAnything and DragNUWA were all built on SVD.
• SVD is unique in that it requires only images as inputs. Other image-to-video diffusion models usually require additional text prompts as inputs, making evaluation challenging because text prompts strongly influence motions.

Given these considerations, we believe SVD is the natural choice for our experiments. We are exploring extensions of the method to emerging architectures, such as video diffusion transformers and hope to tackle this problem in future work.

W2: 3D full attention-based video diffusion models

We are actively investigating application of our method to emerging video diffusion models and hope to tackle this in future work. Please kindly note that, at the time of development, high-quality 3D attention-based video DiT models were not publicly available.

W3: Error in formula

Thank you for identifying this issue. We have corrected the error in Equation 2 and updated Section 3.3 to improve the clarity of each term in Equation 1 based on your feedback.

Specifically:

• $\tilde{F}_n(z_t) \in \mathbb{R}^{h \times w \times d}$ represents feature maps extracted from the modified self-attention layer, given the noisy latent $z_t$ as input to the model.

• $\\tilde{F}\_n(z\_t)[\\mathcal{B}\_{b, n}] \\in \\mathbb{R}^{h\_b \\times w\_b \\times d}$ denotes feature maps cropped by the bounding box $\mathcal{B}_{b, n}$.

These updates are highlighted in blue in the revised manuscript.

W4: Comparison with MOFT

We have included results and discussions regarding MOFT in Section 4.1, Table 1, and Appendix A (highlighted in blue). Key differences between our method and MOFT are as follows:

• MOFT primarily focuses on reference-based motion control, requiring videos with target motions as references. In contrast, our method does not rely on reference videos, offering more flexibility and precise controllability.
• MOFT optimizes on their proposed feature maps, claimed to be motion consistent (i.e., pixels belonging to the same motion exhibit similar vectors). However, semantic consistency is not guaranteed and it is not clear whether the same objects having different motions across frames have consistent feature vectors. As demonstrated in Table 1, we find optimizing on their proposed feature maps is less effective compared to our approach.

Q1. Inference time.

Thank you for your question. Please refer to our general response for the result.

We hope these address your concerns, and we are happy to discuss further if needed!

We thank the reviewer for their insightful feedback. We have revised the paper, website, and supplementary material, incorporating the reviewer’s suggestions. All modifications in the revision are highlighted in red. In the following, we address each of the comments and questions in the review.

W1: Performance gap with DragAnything.

We thank the reviewer for pointing this out. We corresponded with the authors of DragAnything and verified that there is no error in our evaluation; the metrics differ due to implementation details, such as how image resolution is handled. As the DragAnything authors did not release their complete evaluation pipeline, we could not perfectly reproduce their implementation. We describe the differences as follows.

For ObjMC,

• Resolution of video: Our evaluation uses a uniform video resolution of $320\times 576$, while DragAnything resizes the generated videos to the original resolution, which varies between $480×800$ and $1440×256$. Since ObjMC computes error distance, the resolution difference affects the scale of the reported value.
• Invalid Ground Truth Trajectory points: Some ground truth trajectory points in the dataset contain invalid coordinates (e.g., objects moving out of frame). As all the baselines, including ours, can not be conditioned on these points, we exclude them from evaluation.

For FID and FVD,

• DragAnything uses 25 inference steps in their paper. To ensure a fair comparison with baselines, we use longer steps (50) when running their code. This contributes to their improved FID and FVD in our table.

To ensure reproducibility, we have added specific evaluation details and model versions (if applicable) in Table 1 and Appendix A (marked in red). Furthermore, we have uploaded our evaluation scripts in the supplementary material and will publicly release the evaluation scripts upon acceptance.

W2: Performance gap in FVD and ObjMC with supervised baselines.

FID focuses on per-frame visual quality, while FVD is more sensitive to the motion smoothness of the video. As a tuning-free method, we generate videos at the original resolution of SVD (576x1024) and then downsample them to match the output resolution of the baselines for FVD computation. In contrast, the supervised baselines fine-tune SVD to generate lower-resolution videos directly (320x576). Generating smooth motion at a higher resolution is generally harder, and this discrepancy in evaluation settings contributes to the different trends between FID and FVD.

We also toned down claims of “competitive performance” with supervised baselines in the abstract, introduction, and Table 1.

W3: Discrepancy between Table 1 and Figure 4.

The results in Table1 show a different trend than the qualitative results in Figure 4 because Figure 4 aims to highlight the limitations of supervised baselines and show how our tuning-free method generalizes to these failure cases. For instance, DragAnything is fine-tuned on entity-level trajectories and struggles with part-level control, whereas our tuning-free method can cover these cases by benefiting from the generalizability of the underlying image-to-video diffusion model. To avoid confusion, we have clarified the purpose of Figure 4 in its caption and Section 4.3 (marked in red). The general qualitative results can be found on our supplementary website: https://sgi2v-paper.github.io/gallery_baseline.html and https://sgi2v-paper.github.io/gallery_baseline2.html.

Additionally, we have added qualitative comparisons with zero-shot baselines in Appendix B and on our supplementary website https://sgi2v-paper.github.io/gallery_zero.html. These results highlight our advantage over zero-shot baselines

W4: Implementation details of zero-shot baselines.

We appreciate the reviewer’s suggestions and have added implementation details for the DragDiffusion and Freetraj baselines in Appendix A (marked in red). We are also conducting additional experiments with the original DragDiffusion, which we hope to be presented during the rebuttal. For a fair comparison, we treat the edit region of DragDiffusion as the entire image space since our model does not take the edit region into account. We kindly note that the original DragDiffusion, which was designed for image editing, may produce temporally inconsistent frames (e.g., modifying scene contents rather than moving objects or the camera, as shown in an example on their webpage https://yujun-shi.github.io/projects/drag_diffusion_resources/oilpaint_mountain.mp4). This will limit performance in our video generation setting.

Thanks for your reply.

1. For FVD, assume we have four datasets, A, B, C, and D, with dynamic degrees 50%, 55%, 85%, and 90%. When their quality is similar, there is no doubt that A (50%) and B (55%) have the best FVD. In your assumption, C (85%) and D (90%) should have better FVD than B (55%) and D (90%). However, in Table 1, SVD (No Control) achieves a competitive FVD rather than the worst FVD, which is not aligned with your assumption. I also have observed some similar phenomena in my previous motion-control-related work. In summary, sometimes average results will get higher scores. Another example is that in super-resolution tasks, sometimes the results with details close to the ground truth have worse PSNR indicators than slightly blurred results. Sorry for the naive examples here. I just wanted to share some of my observations about FVD and I never ask you to ignore FVD.

2. The upper limit of tuning-free methods is definitely lower than that of training-based methods. However, training-based methods need more data, calculation resources, and most importantly, re-training for each base model. Nowadays, the iteration of video generation models is very fast, and the video quality generated by these methods depends to a large extent on the quality of the basic model. Tuning-free methods can quickly be adapted to a new base model. If authors can apply their results to CogVideoX, their results should be more competitive unless DragAnything trains a new version based on CogVideoX. Based on this consideration, I think the author's method is still competitive among tuning-free methods.

3. I never question your rating. All reviewers give scores around borderline and I don't think they are obviously diverging. The questions you raise are all very valuable, and addressing them will undoubtedly make the article better. If authors want to change your rating, they should put more effort into addressing your concerns.