UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing

Dear reviewer eXku,

We thank the reviewer for the valuable feedback on our paper, we provide clarifications to the concerns below:

Novelty. Using ReferenceNet is not a novel approach in video generation. What is the difference between yours and previous work.

As you mentioned, using a ReferenceNet has been explored previously. However, our approach differs in several key ways:

1. We address the challenging problem of text-guided motion editing in the temporal dimension, which cannot be achieved by simply adapting existing ReferenceNet-based methods.
2. We provide insight that “the temporal attention layers of the generator encode the inter-frame dependency”, enabling training-free motion editing. Building upon this, we explore and investigate feature injection in temporal layers, an area that has not been thoroughly explored.
3. Furthermore, simply performing feature injection on temporal layers results in severe content inconsistency with the source video (Tab. B). In response, we design UniEdit with content preservation and structure control on spatial layers and motion injection on temporal layers.
4. Previous works in video editing are typically tailored to particular tasks. For instance, Rerender-A-Video [1] excelled in style transfer, while Video-P2P [2] focused on local object editing. In contrast, our proposed method can effectively handle motion editing and various appearance editing tasks, showcasing promising performance both visually (https://uni-edit.github.io/UniEdit/) and quantitatively (Tab. 1).

Thus we believe that UniEdit contributes to the advancement of video editing.

Overclaim. "UniEdit represents a pioneering leap in text-guided, tuning-free video motion editing." Why this work is a pioneering leap?

We make the statement because there is an absence of the tuning-free text-guided motion editing method. We have made the statements more precisely in the revised version in Line 94.

Lack of comparison. Many new works, such as Revideo, COVE , accepted by NeurIPS 2024, are all not included in this paper.

The decision time for NeurIPS 2024 overlaps with the submission period for ICLR 2025. Therefore, we did not compare the two methods you mentioned in our initial submission. In the revised version, we have included a comparison with ReVideo [1]. The qualitative results are shown in Fig. B (https://uni-edit.github.io/UniEdit/#Fig_B) on the project page. The main difference between the proposed UniEdit and ReVideo is that our method is text-guided appearance and motion editing, whereas ReVideo is based on editing the first frame and the trajectory. We have included the discussion about ReVideo in the related work section in the revised PDF file. Since COVE [2] has not released the code, we were unable to include it in our comparison.

Note that the editing results we obtained using ReVideo are based on their open-source code (https://github.com/MC-E/ReVideo). The specific process is as follows: First, we use the first frame edited by UniEdit as the image input for ReVideo. Second, for appearance editing, the trajectory input is obtained by tracking feature points in several initial frames. For motion editing, we draw the trajectory based on the editing results of UniEdit as the input. Finally, we generate the "editing mask" for each case as the input mask for ReVideo. Due to limited time, we were unable to obtain quantitative results for ReVideo. This is because it requires manually drawing trajectories and generating editing masks for hundreds of editing cases to input into ReVideo.

[1] Mou, Chong, et al. "ReVideo: Remake a Video with Motion and Content Control." arXiv preprint arXiv:2405.13865 (2024).

[2] Wang, Jiangshan, et al. "COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing." arXiv preprint arXiv:2406.08850 (2024).

Long video editing, All results are only 16 frames, which is too short.

This limitation is due to the base T2V model, not a flaw in the proposed method. Specifically, the base text-to-video generation model we used can only generate 16-frame videos and performs poorly when synthesizing longer or shorter videos. The proposed method can seamlessly support long video editing by simply replacing the base model with a long video generation model. To demonstrate this, we validated the effectiveness of our method on different base models by deploying it on LaVie [1] and VideoCrafter2 [2], as described in Section 5.2. We believe long video editing is a promising research direction, but it is not the focus of this work.

[1] Wang, Yaohui, et al. "Lavie: High-quality video generation with cascaded latent diffusion models." arXiv preprint arXiv:2309.15103 (2023).

[2] Chen, Haoxin, et al. "Videocrafter2: Overcoming data limitations for high-quality video diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

after rebuttal， the authors do not address my question. I still care about the novelty and contribution. So I don't think this paper is enough to publish. Sorry for this decision.

Dear reviewer iKZP,

We thank the reviewer for the valuable feedback! We address the questions below:

Check if it is reasonable to have conducted the experiment with 10 subjects.

Thank you for your suggestion. In addition to the original 10 participants, we have invited an additional 20 participants to conduct the evaluation. We have updated the results in Table 1. We acknowledge that this sample size may not be sufficiently large due to the limited time and resources. Therefore, we have noted in the table that "the results may be subjective due to the limited sample size".

Dear reviewer iKZP,

Thanks for the reply. We would like to make the following clarification:

The sample size may still be insufficient to ensure the reliability and generalizability of the experimental results.

1. A similar participant size is widely used: 30 participants used in Rerender-A-Video [1] (SIGGRAGH 2023), 5 participants in Tune-A-Video [2] (ICCV 2023), and 20 participants used in FateZero [3] (ICCV 2023 Oral).

2. In addition to the user study, we quantitatively demonstrated the effectiveness of our method using CLIP scores, MVFD, and VBench scores. We also provided editing results on three models—LaVie, VideoCrafter2, and CogVideoX—to prove the generalizability of our approach. We believe this sufficiently demonstrates the effectiveness of our method.

16 frames are too short to demonstrate practical utility, long video editing is crucial.

1. The proposed method can edit longer videos (e.g., 49 frames or 81 frames) by implementing our method on other T2V models, like CogVideoX-2b [4], please check our results in Fig. E (https://uni-edit.github.io/UniEdit/#Fig_E) on the project page.

2. The number of frames in the edited video is entirely dependent on the number of frames that the base T2V model can generate, rather than being limited by our method. This is a universal limitation for all video editing approaches based on T2V models, not a unique drawback of our method. For example, as you mentioned with ReVideo [5], it can only edit 14 frames because it is trained based on SVD [6], which only supports generating 14 frames.

3. While long video editing is indeed significant, we believe it is not aligned with the focus of this paper, ReVideo, and many other related works. This topic is out of the scope of our study.

While the emphasis on being tuning-free is appreciated, the limitations imposed by the baseline, particularly in extending the method to editability and long video generation, suggest that incorporating a fine-tuned model might be necessary.

"A fine-tuned method" also cannot solve the long video editing problem you mentioned, as it is similarly limited by the number of frames the base model can edit, such as in the case of ReVideo.

Additionally, the spatial and temporal attention mechanisms included in the proposed method have become standard in many video editing works. While such approaches were impactful in the early stages of video editing research, the field has progressed significantly.

While the feature injection technique has been adapted to different tasks, how to perform text-guided motion editing and feature injection on temporal attention layers is under-explored. Additionally, we believe the value of UniEdit is further demonstrated by designing a unified framework that supports various editing tasks, which is more flexible and easier to use in practice.

Thus, adopting more advanced, fine-tuning-based methods, such as those exemplified in recent works like ReVideo: Remake a Video with Motion and Content Control (NeurIPS 2024), seems necessary to achieve broader applicability.

We gently disagree that fine-tuning-based methods are more advanced than zero-shot methods. Unlike image editing, video editing lacks large-scale paired video data (i.e., [source video, editing prompt, target video]) for extensive training. As a result, zero-shot editing is commonly used in the field. Compared to methods that are trained on a single or a limited set of videos, zero-shot editing in a feed-forward pass does not suffer from overfitting and offers advantages in both speed and flexibility. We believe that both tuning-free editing and fine-tuning-based methods are valuable research topics worth exploring.

We also include a qualitative comparison of the training-based method ReVideo and UniEdit in Fig. B on the project page (https://uni-edit.github.io/UniEdit/#Fig_B). It can be observed that ReVideo sometimes fails to generate edited videos with natural motion.

[1] Yang, Shuai, et al. "Rerender a video: Zero-shot text-guided video-to-video translation." SIGGRAPH Asia 2023.

[2] Wu, Jay Zhangjie, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." ICCV 2023.

[3] Qi, Chenyang, et al. "Fatezero: Fusing attentions for zero-shot text-based video editing." ICCV 2023.

[4] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." *arXiv (2024).

[5] Mou, Chong, et al. "ReVideo: Remake a Video with Motion and Content Control." NeurIPS 2024.

[6] Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets." arXiv (2023).

Dear Reviewer iKZP,

Does the further response from the authors address your concerns? It would be helpful if you could provide your feedback.

Thank you.

AC

Dear reviewer vcxP,

We thank the reviewer for the elaborate review! We address your concerns below:

The authors claim in L217 that "better tackle source video with large dynamics". However, most visual examples are with small motion magnitudes. Does the model struggle with appearance and motion editing in videos with larger dynamics? Temporal inconsistencies are noticeable in videos with large motion (see Fig G.1).

In the "cat" to "dog" case mentioned in Fig G.1, the temporal inconsistency observed during the dog's head-turning process is due to the base model struggling with synthesizing videos with large dynamics. Similar cases are shown in LaVie's Gallary (https://github.com/Vchitect/LaVie/tree/main?tab=readme-ov-file#download-pre-trained-models), where the model can only generate videos with similar or smaller dynamic ranges. Therefore, when the original video has significant motion, the base model sometimes fails to handle it well. This issue can be resolved by using a more powerful base model.

To demonstrate this, we conducted experiments with a more powerful T2V checkpoint with the assistance of the authors of LaVie [1]. The video results, shown in Fig. C (https://uni-edit.github.io/UniEdit/#Fig_C) of the Rebuttal section on the project page, indicate that the inconsistency issue has been resolved, which suggests that the occasional inconsistency is due to the limitations of the base model rather than a flaw in UniEdit. This is further evidenced by 1) the quantitative metrics in Tables 1&2 and 2) numerous visually consistent editing results.

[1] Wang, Yaohui, et al. "Lavie: High-quality video generation with cascaded latent diffusion models." arXiv preprint arXiv:2309.15103 (2023).

The authors claim in L312 that "UniEdit is not limited to specific video diffusion models", which is inaccurate. The proposed framework depends on the separate spatial and temporal attention design of video diffusion models, which is incompatible with full 3D attention DiT-based video generators. And 3D full attention is increasingly being proven to be more effective [1].

Thank you for your reminder! We have revised the claim in our paper. The inability to effectively adapt to models that use only 3D full attention [1] is indeed a limitation of our method. However, we also note that many Unet-based [2, 3, 4] and transformer-based [5] video generation models have separate attention layers, demonstrating that UniEdit has a relatively broad range of applications.

[1] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

[2] Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets." arXiv preprint arXiv:2311.15127 (2023).

[3] Wang, Yaohui, et al. "Lavie: High-quality video generation with cascaded latent diffusion models." arXiv preprint arXiv:2309.15103 (2023).

[4] Chen, Haoxin, et al. "Videocrafter2: Overcoming data limitations for high-quality video diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Ma, Xin, et al. "Latte: Latent diffusion transformer for video generation." arXiv preprint arXiv:2401.03048 (2024).

This work still follows the common "inversion-and-genertation" paradigm, potentially performing better on generated videos but struggling with out-of-domain real videos.

Yes, the "inversion-and-generation" paradigm can exhibit inferior performance on some real-world videos. This is primarily because the pipeline sometimes struggles to faithfully reconstruct the input video, leading to a content mismatch. However, we found that this issue can be effectively addressed by equipping UniEdit with null-text inversion [1]. Specifically, for real-world videos, we first optimize an unconditional embedding at each denoising step $t$ to ensure the model can accurately reconstruct the original video. Then, during the editing process, we replace the unconditional embedding in all branches with the optimized embedding. This approach ensures that the content of the original video is well-preserved, resulting in better performance.

[1] Mokady, Ron, et al. "Null-text inversion for editing real images using guided diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Thanks for the rebuttal with the new experiments. Some of my concerns have been addressed. However, I still lean toward a negative assessment considering its weak generality, inferior paradigm, and less satisfactory results, so I will maintain my original score.

Dear reviewer u6DS,

We would like to thank you for the very detailed review and the constructive feedback you provided! Below, we address your concerns in detail:

Defining Boundaries and Constraints of the proposed method.

Thank you for your valuable suggestion. We elaborate on the boundaries and constraints of UniEdit below:

Supported Edit Tasks

UniEdit supports several commonly used editing tasks, i.e., style transfer, rigid/non-rigid object replacement, background replacement, and motion editing.

Limitations

The limitations primarily lie in the following areas:

1. Lack of Support for Non-Text Inputs: The designed UniEdit is a text-guided editing method, and does not accommodate user inputs in other forms, such as motion trajectories or skeletons. This limits UniEdit to perform precise editing, such as specifying the trajectory of an object's movement in pixel space.
2. Text-to-Video Model's Text Following Capability: Given a target prompt to be edited, the base T2V model is required to effectively respond to the prompt and generate a video that aligns with the described content. Specifically, we can use the target prompt $P_t$ as a condition to denoise without using UniEdit (i.e., using the vanilla T2V generation model) to obtain the generated result $V_t$. Though $V_t$ may have a severe content mismatch or inconsistency with the source model, the generated result should at least conform to the description of $P_t$. In such cases, we can effectively inject content features and motion features into the 'main editing path' to achieve an edited result that matches both the original video content and the target prompt.

Analysis of Failure Cases

We include failure case visualizations and the corresponding analyses below in Appendix B.5:

We exhibit failure cases in Fig. 13. Fig. 13(a) showcases when editing multiple elements simultaneously, and we observe a relatively large inconsistency with the source video. A naive solution is to perform editing with UniEdit multiple times. Fig. 13(b) visualizes the results when editing a video with complex scenes, and the base T2V model sometimes could not precisely follow the semantics in the target prompt, resulting in incorrect editing. This may be caused by the base model’s limited text understanding power, as discussed in [1]. It could be alleviated by leveraging the reasoning power of MLLM [1] or adapting approaches in complex scenario editing [2].

[1] Huang, Yuzhou, et al. "Smartedit: Exploring complex instruction-based image editing with multimodal large language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Mao, Qi, et al. "Mag-edit: Localized image editing in complex scenarios via mask-based attention-adjusted guidance." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

UniEdit's core techniques are indeed inspired by existing approaches. While UniEdit's core techniques are not entirely novel, the integration and adaptation of these mechanisms for tuning-free video motion editing demonstrate a degree of innovation.

We agree with your comments that the idea of using auxiliary branches and feature injection has been adapted to various scenarios. Additionally, we believe the value of UniEdit is further demonstrated in the following aspects:

1. Exploring Text-Guided Motion Editing as a Valuable Task: Existing methods primarily control motion through additional supervision signals (e.g., optical flow [1], trajectories [2, 3]). Using text alone is more flexible and worth exploring.
2. Designing a Unified Tuning-Free Framework that Supports Various Editing Tasks: Many works in the editing field focus on one or a few specific editing tasks. For example, [4, 5] can handle object replacement and stylization tasks but cannot perform motion editing; [6] achieves motion editing with a reference video but does not support other types of tasks. We believe that designing a framework for various editing tasks is more flexible and easier to use in practice.

[1] Yan, Wilson, et al. "Motion-conditioned image animation for video editing." arXiv preprint arXiv:2311.18827.

[2] Wang, Zhouxia, et al. "Motionctrl: A unified and flexible motion controller for video generation." ACM SIGGRAPH 2024 Conference Papers. 2024.

[3] Mou, Chong, et al. "ReVideo: Remake a Video with Motion and Content Control." arXiv preprint arXiv:2405.13865 (2024).

[4] Hertz, Amir, et al. "Prompt-to-Prompt Image Editing with Cross-Attention Control." The Eleventh International Conference on Learning Representations.

[5] Liu, Shaoteng, et al. "Video-p2p: Video editing with cross-attention control." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[6] Tu, Shuyuan, et al. "Motioneditor: Editing video motion via content-aware diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.