ReVideo: Remake a Video with Motion and Content Control

Q1 The Editing of the First Frame

Thanks for this suggestion. The editing method for the first frame is arbitrary, like the setting in AnyV2V. The results presented in the paper utilize text-guided image inpainting tools and InstructPix2Pix. Note that in our framework, editing the first frame is not mandatory. Users can choose to keep the content and modify the motion of the video. We will add a detailed explanation of the editing workflow in the paper.

Q2 Some Motion is Not Kept

We need to clarify that our editing framework uses sparse motion control (trajectory lines) in the editing region, which is challenging to handle slight motions as you mentioned. To support such motion control, our framework may need to incorporate dense motion representations, such as optical flow. A potential solution is to mix sparse trajectories with dense optical flow during the training process, allowing our method to apply sparse and dense motion control selectively. We will explore this design in future work.

Q3 Non-rigid Motion

Thank you for this constructive suggestion. We add a video of editing a side-view driving car in Fig.9 in the attached PDF. We can see that while the car's rigid body is moving, the car's wheels are also rolling (although it might not look entirely realistic). Similar phenomena can be observed in some demo videos in our supplementary, such as adding a tank on the desert. The tank's wheels are moving in a regular pattern, and the shadow behind the tank moves accordingly. We believe that this phenomenon, including the natural motion of dogs and humans that you mentioned, is due to the priors of physical world that the base model (SVD) learned from large-scale video data. The unnatural movement of the shark is also caused by the limitations of SVD's prior. To verify this, we use SVD to perform image-to-video generation based on the shark image. The result in Fig.9 in the attached PDF shows that while the water surface has natural ripples, the shark remains stationary. We believe that by using a more powerful base model (such as the Sora architecture), we can achieve more natural editing results.

Q4 Quantitative Comparison of the Ablation Study

To address this concern, we measure the editing accuracy under different settings in our ablation study, as shown in the table below. We use a point tracking model (CoTracker) to extract trajectory in the edited results and then calculate the Euclidean distance with the target trajectory. We will add this quantitative comparison to our paper.

Q5 Qualitative Video Comparisons

Yes, it is necessary to provide comparison videos of different methods. However, due to the rebuttal policy, we cannot provide external links. Instead, we select two videos to present in Fig.10 in the attached PDF. One can see that AnyV2V has gradual changes in video content in some editing samplings, leading to instability. Pika and AnyV2V both tend to produce static editing results, due to the lack of motion control. While such static results may achieve higher scores in CLIP, it is not natural enough. Additionally, Pika tends to fail in some challenging editing scenarios, such as transforming a part of a lizard into a robot.

Q6 Training Cost

We agree that our implementation involves some training costs compared with training-free methods. However, our approach offers significant performance gains:

(1) Our method allows precise customization of local content and motion in videos, which is not achievable with existing training-free methods.

(2) Our method can produce high-quality and stable outputs. Training-free methods often struggle with video generation quality and stability of the edited content.

(3) Our method has an advantage in inference complexity. Some training-free methods, such as AnyV2V, require a long time of ddim inversion to ensure the editing quality. The table below shows the time costs of different methods during the inference stage. The experiment is conducted on an A100 GPU, with the video resolution being 768x768. Results show that our method has significantly lower time costs compared to other methods. Therefore, we believe that our training cost (4 A100 GPUs) is reasonable and necessary. We will add the complexity analysis to our paper.

Q7 Performance in Editing Multiple Objects Simultaneously

In the demo video in our supplementary, we show examples of editing six individual petals in the same video, as well as editing multiple flowers and balloons in one video. This demonstrates the robustness of our method in multi-object editing. Additionally, R1 raises a concern in Q.5 about the impact of the number of editing regions on performance. Results in R1-Q.5 show that each editing region has a high independence, and the impact of the region numbers is almost negligible.

Q8 Whether works on text-to-video models

Yes, it is possible, but the implementation would be more complex than with an Image-to-Video model. The image condition port in the Image-to-Video model can naturally be used as the input of the edited first frame, whereas the Text-to-Video model lacks this port and would require additional training to incorporate it.

My concerns have been thoroughly addressed in the rebuttal. Overall, this paper is well-structured, offering significant contributions and demonstrating impressive performance. It brings fresh perspectives to the research field, so I have decided to upgrade my final rating to accept.

Q1 About Artifact

We agree that our method still has room for improvement. We want to clarify this concern from two points:

(1) The challenge of this task and our novelty. Our method is the first attempt at local content and motion editing for videos. In Section 3.2 of the main paper, we conduct extensive toy experiments to demonstrate the challenge of this task, as it requires overcoming significant coupling between control conditions. In implementation, we meticulously design both the training strategy and the model architecture to address this challenging task.

(2) Editing Performance. We present several useful applications and high-quality editing results in the paper, which is unachievable for existing methods with a significant gap. Although some artifacts exist in the edges, as you concerned, they are often difficult to detect in the result videos.

We will continue to improve our method in the future.

Q2 Overfitting in Training

We need to clarify that although the three-stage training strategy might be a bit complicated, our extensive experiments in the paper (Section 3.2) demonstrate that the problem we face is indeed challenging. Therefore, we believe that using a progressive learning approach to tackle this difficult problem is reasonable, and the role of each training stage is demonstrated in the paper. We would greatly appreciate any references or suggestions you could provide to help us improve our current training strategy.

In Section 4.1, we show that our training data includes 10 million videos from the WebVid dataset. Therefore, overfitting is not a concern under such a dataset scale. The test results also demonstrate the diversity of the editing scenarios.

Q3 Like a ControlNet

We need to clarify that ControlNet is a commonly used method for condition inputs. In this paper, we also introduce that our method uses conditional injection of ControlNet. Moreover, combining video diffusion with ControlNet cannot solve our problem, and it is not a contribution of this work. The key contribution of our work lies in the proposing of a novel training strategy to solve the unexplored fine-grained video editing task. Additionally, we design a spatiotemporal adaptive fusion module to improve the fusion of motion and content control.

Q4 Detailed Implementation of SAFM

Our SAFM consists of four convolutional layers, each followed by a SiLU activation layer, with a Sigmoid function applied at the end. The SAFM has 2 million parameters, which is negligible compared to the 671 million parameters in the SVD model. We will add this detailed description to the paper.

Q5 Computation Demand

In Section 4.1 of the paper, we describe that our training is conducted on four A100 GPUs. During the inference, it requires 22GB of VRAM to edit a 16x768x1344 video. In the table below, we present a comparison of computational complexity for different methods. The experiment is conducted on an A100 GPU, with the editing target being a 16x768x768 video. The results demonstrate the advantage of our method in inference efficiency. Therefore, we believe our approach is efficient and flexible. We will add a detailed description of complexity demand in the paper.

Q6 Focuses Heavily on Technical Aspects

We want to clarify this concern via the interactive design and capability of our method. To offer an intuitive user interaction, we chose the sparse and easy-to-interact trajectory lines as the motion control. In the editing mode, users can selectively customize content or motion. In scenarios where motion control is difficult to specify, our method can also automatically generate the motion in the editing region. Therefore, our method is user-friendly in controlling inputs and functions. We will include a detailed description of user interaction in the paper.

Q7 Open Source & Limitation

Yes, we will open-source all the code.

We need to clarify that our method is not limited to a specific base model. The reason we use SVD as the base model is that SVD is the best open-source model currently available to us. Using a better base model can further improve our editing results.

We need to clarify that:

1. ControlNet is a commonly used method for condition injection, not a solution for our problem (as verified in Section 3.2). The contribution of this paper is to propose effective solutions for local video editing. Additionally, we modify ControlNet by designing a region-adaptive fusion method to make the condition embedding more suitable for this task.

2. Why is the decoupling training strategy not considered one of our contributions? Dear reviewer, have you ever seen other similar solutions for video editing? The reference you provided is an image classification method—what relevance does it have to our paper? Just because two different images are mixed together?

3. Our method is successfully validated on SVD and WebVid data, but this does not mean that our method can only be implemented using SVD and WebVid. It can be applied to other data and base models. We believe that addressing a challenging problem by a progressive learning approach is reasonable. In many complex generation tasks, such as EMU2[1], the training difficulty is much greater than ours.

[1] Generative Multimodal Models are In-Context Learners

4. Firstly, we are addressing an unexplored and challenging task that requires carefully designed training processes and fusion modules. This is not a simple engineering patchwork. Our core components did not previously exist. Secondly, what is the relevance of the CutMix to our work? Why can it replace our core contribution?

Dear R2, we are eager to address these misunderstandings and look forward to your response.

Dear R2, we would like to know if your concerns are addressed. If any questions, we can discuss them in this phase.

1. ControlNet cannot accomplish our task. This is the fundamental difference. The following works all used ControlNet for conditional injection. We do not claim ControlNet as one of our contributions. It is a commonly used model, just like Stable Diffusion.

[1] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models

[2] ControlVideo: Training-free Controllable Text-to-Video Generation

[3] CameraCtrl: Enabling Camera Control for Text-to-Video Generation

[4] EVA: Zero-shot Accurate Attributes and Multi-Object Video Editing

[5] CCEdit: Creative and Controllable Video Editing via Diffusion Models

[6] Text-Animator: Controllable Visual Text Video Generation

[7] IMAGE CONDUCTOR: PRECISION CONTROL FOR INTERACTIVE VIDEO SYNTHESIS

2. [1] is a paper on watermark removal, where a watermark image refers to the watermark being overlaid on the original image. This overlay is a task, not a solution. We still cannot find any connection between this and our method.

[1] Liu, Yang, Zhen Zhu, and Xiang Bai. "Wdnet: Watermark-decomposition network for visible watermark removal." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.

3. Our method is the first attempt at this task and has already achieved state-of-the-art (SOTA) results. Even if adjustments are needed for other models and datasets, the issues found and solutions proposed in this work are still insightful.

4. CutMix you referenced is entirely different from our approach in both purpose and methodology. We do not think combining two images to enhance image classification performance can replace our contribution. What we propose is a motion-decoupled training strategy, not data augmentation. This decoupling strategy is effective and does not exist in our community.

Dear authors,

Thank you very much for the clarification. However, I still feel the paper is somewhat incremental by combining a ControlNet-like motion injection module with augmented (motion) training data. The approach and task themselves are new and novel. Furthermore, the result videos are impressive, with barely noticeable artifacts. I would like to raise my rating to borderline accept. Thanks again to the authors for the clarification and discussions!

Q1 Practicality of Workflow

Fig.1 and Fig.2 in the attached PDF show that our method can still produce smooth results without specifying trajectory. This is due to our inherent capability to predict the motion in editing area via unedited content, enabling automatic motion generation when trajectory is difficult to specify. However, the content reference is essential in current framework. We will discuss the practicality in our paper.

Q2 Limited Motion Control

We need to clarify that global motion pattern in the editing region is not our editing target but needs to be consistent with unedited areas. The consistency is automatically produced without fixed motion assumption.

Q3 Precise Placement

Thanks for the suggestion. We will discuss related works in progressive learning and multi-condition control.

Q4 Entanglement

• Residual entanglement. Yes, it still exists and can be observed by reducing the weight of motion condition (Fig.3 in the PDF). We believe it is useful and mild. (1)Useful. Inspired by Q.1, the entanglement can automatically produce the motion, when it is difficult to specify the motion of editing region. (2)Mild. Various editing results show that this entanglement is almost unnoticeable in default inference.
• Other solutions. Yes, that's certain. But we think decoupled representations are necessary.
• Decoupling training. The insight of why it works mainly comes from Sec.3.2 in our paper. In Sec.3.2, we use several toy experiments to show the strong coupling issue. The method in Fig.4 ensures no motion correlation between the editing and unediting region, preventing the model from estimating the motion in editing region via unedited content. We will add a more intuitive explanation in the paper.

Q5 Multi-area Editing

In Fig.4 in the PDF, we select a complex wavy line as the editing target and gradually add editing regions. We compute editing accuracy by Euclidean distance between the trajectory in editing result (extracted by CoTracker) and the target. Results below show slight impact of region numbers on accuracy and consistency. We will add this in our paper.

Q6 Clarity

In Sec.3.2 and A.1 of our paper, we describe the trajectory sampling. Threshold $l_{Th}$ is empirically defined as the mean length of sparsed trajectories. Each video has one editing area, the minimum bounding rectangle of sampled trajectories. It is expanded to at least 64x64 pixels. Data augmentation includes sampling frames at varying intervals and randomly reversing the video. We will update these details in our paper.

Q7 Metrics

• Measurement in editing region. Using a tracker helps evaluate our motion control, especially in ablation. Other methods do not have motion control. The content consistency in editing region can be compared among different methods. Below table shows CLIP Score between the editing region and target description. |Methods|InsV2V|AnyV2V|Pika|ReVideo| |-|:-:|:-:|:-:|:-:| |Local CLIP Score|0.2255|0.2378|0.2402| 0.2516|
• Temporal stability. In our paper, we use CLIP-Image score to evaluate the temporal consistency, which is used in several related works (e.g., AnyV2V, Pix2Video). The reference repo is helpful, but most metrics need a target video, which is unachievable in editing task. After reviewing, PSNR Consistency can measure the temporal stability, as shown below. |Methods|InsV2V|AnyV2V|Pika|ReVideo| |-|:-:|:-:|:-:|:-:| |PSNR Consistency|34.94|34.12|40.51|39.26|
• Human evaluation. In Tab.1 in our paper, we conduct a user study for overall video quality and editing accuracy. We agree that more evaluation terms is helpful. We will extend it accordingly.

Q8 Robustness and Ablation

• Choice of motion representation. Yes, trying more motion representations is helpful. But we only find the trajectory that meets our sparse and interactable requirement.
• Impact of trajectory sampling. Training with significantly moved trajectory is crucial. Below table shows its significant impact. In contrast, trajectory numbers have little impact. ||Random sampling|ReVideo| |-|:-:|:-:| |Acc.|13.38|5.21|
• Impact of size and shape. In Sec.A.2 of our paper, we show our support for irregularly shaped regions. Below table shows the performance after randomly expanding the width/height of editing regions on test set. The size and shape have little impact on performance, even when the editing region is entire video. ||Default|+$l\in$[32,64]|+$l\in$[64,128]|Global| |-|:-:|:-:|:-:|:-:| |Acc.|5.21|5.06|5.32|5.48|
• Ablation of SAFM. "w/o SAFM" in Sec.4.3 is a simpler fusion scheme that fuses motion and content embedding by adding and convolution. It fails in complex motion control. This result and weight visualization in Fig.5 can verify the need of region-adaptive fusion.
• Input perturbations. We try two content perturbations in Fig.5 in the PDF. The base model's prior on high-quality videos can mitigate content degradation, and motion control works normally. If the trajectory is perturbed, the motion will follow the perturbation.

We will update the above discussion in our paper.

Q9 Dataset Complexity

In Fig.6 and Fig.7 in the PDF, we try some complex scenarios. Our method can handle dynamic lighting and texture, but scene change affects the content quality, which is a failure case. We will include a discussion in our paper.

Q10 Editing Scenario

Our test has simple semantic interactions, such as wearing a hat or glasses. In Fig.8 in the PDF, we test a more complex interaction where two balloons collide. They do not bounce off each other but overlap. This failure is because our base model (SVD) is weak in physical world modeling. We believe our method can achieve such interactions with a more powerful base model. We will discuss this failure in the paper.

Q11

Yes, we will release all code.