We thank the reviewer for providing encouraging comments on our paper! We provide clarifications to the concerns below:  

The implementation may not be entirely reconstructable. Hopefully, this issue will be fixed when they publish the code as promised.

We will definitely release the code when it is published. We also make detailed descriptions in Section 4 and Appendix A on experimental settings (e.g., model structure, hyper-parameter selection) to make the paper reconstructable.

 

The math is sometimes not entirely correct. For example in line 212/212, M is a matrix but the notation says that it is a scalar from a set −inf,1 or 0,1. The authors should be encouraged to revise the math present in the paper.

Thanks for pointing it out! We will revise to $M_{ij} \in \\{ ... \\}$ and check the math expressions in the paper comprehensively.

 

Minor weaknesses in writing.

We would like to thank the reviewer for a very detailed review! We will fix the typos.

 

Is there a reference that also handles the evaluation process as described in lines 242-245?

Yes. CAMEL [1] uses a subset of LOVEU-TGVE-2023 for evaluating the designed training-based video editing technique, and Customize-A-Video [2] uses the LOVEU-TGVE-2023 dataset for evaluating the motion customization ability.

 

limitation on rating the proposed method.

As you mentioned, the lack of a commonly used benchmark complicates the comparison of the proposed method with existing baselines. In light of this, we evaluate the proposed method in three aspects:

1. Following previous work [3], we present the CLIP scores and user preferences for UniEdit alongside baseline methods (Tables 1 & 2).
2. We additionally calculate VBench [4] scores, a recently proposed benchmark suite for T2V models, for a more comprehensive and precise quantitative comparison (Table 1).
3. We provide plenty of qualitative results in the paper and synthesized videos on our project website to aid readers in making subjective evaluations.

We believe that these are sufficient to demonstrate the superiority of our method.

 

We hope the above responses address your concern. We would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.

 

[1] Zhang, Guiwei, et al. "CAMEL: CAusal Motion Enhancement tailored for Lifting Text-driven Video Editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Ren, Yixuan, et al. "Customize-a-video: One-shot motion customization of text-to-video diffusion models." arXiv preprint arXiv:2402.14780 (2024).

[3] Wu, Jay Zhangjie, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Huang, Ziqi, et al. "Vbench: Comprehensive benchmark suite for video generative models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Thanks for your constructive comments! We address the concerns below:  

Feature injection is a known technique in image editing.

As you mentioned, similar feature injection techniques have been explored previously. However, our approach differs in several key ways:

1. We address the novel challenging problem of motion editing in the temporal dimension, which cannot be achieved by simply adapting existing feature injection methods. We adapt the SOTA non-rigid image-editing technique MasaCtrl [1] for video editing, and it fails to synthesize video that adheres to the text prompt as illustrated in Fig. 5 and Tab. 1.

2. We provide insight that “the temporal attention layers of the generator encode the inter-frame dependency”, enabling training-free motion editing. Building upon this, we explore and investigate feature injection in temporal layers, an area that has not been thoroughly explored.

3. Furthermore, simply performing feature injection on temporal layers results in severe content inconsistency with the source video (Tab. B). In response, we design UniEdit with content preservation and structure control on spatial layers and motion injection on temporal layers.

4. Previous works in video editing are typically tailored to particular tasks. For instance, Rerender-A-Video [2] excelled in style transfer, while Video-P2P [3] focused on local object editing. In contrast, our proposed method can effectively handle motion editing and various appearance editing tasks, showcasing remarkable performance both visually (https://uni-edit.github.io/UniEdit/) and quantitatively (Tab. 1) across these domains.

Thus we believe that UniEdit contributes to the advancement of video editing.

 

Showing a quantitative analysis, and analyzing the features, could support this claim better and show the importance of the insight “the temporal self-attention layers of the generator encode the inter-frame dependency”.

Thanks for the advice! We add additional analysis and results to further support our core insight as follows:

1. Quantitative results. We calculate the difference between the attention map of temporal layers and the optical flow. The intuition behind this is that optical flow captures the movement between two consecutive frames at the pixel level, and we assume temporal layers capture inter-frame dependency in the feature space. Hence, we sample 40 videos and their corresponding 640 frames, and report the average $L_1$ distance and KL divergence:

   Distance Metrics	$L_1$ Distance	KL Divergence
   Random Matrix	0.51	1.33
   Spatial-Attention	0.48	1.14
   Temporal-Attention	0.29	0.81

   It's observed that attention maps from the temporal layers are more similar to the magnitude of optical flow (also visualized in Fig. A in the rebuttal pdf), which supports the hypothesis quantitively.

2. Analyzing features of different layers. Furthermore, we visualize 1) temporal-attention maps at different resolutions and denoising steps and 2) the feature of {spatial/cross/temporal} attention layers in Fig. A and Fig. B respectively.

   In Fig. A, we visualize temporal attention maps between frame $i$ and frame $i+1$ at different layers (resolution) and denoising steps. In the example on the left, it's observed that the attention map values are higher over the walking person and the moving waves, while they are lower over the relatively static sky and beach. The high consistency with the optical flow across layers and timesteps is consistent with the quantitative results and indicates our insights: the temporal attention layers encode the inter-frame dependency.

   In Fig. B, we contrast the characteristics of SA-S, CA-S, and SA-T. SA-S captures the overall spatial structure of the generated frame, CA-S is mainly activated on the area according to the text, while SA-T concentrates on the inter-frame variances. Leveraging these insights, we delicately design UniEdit to achieve content preservation and structure control by feature injection on SA-S layers and motion transferring on SA-T layers.

3. Synthesized frames visualization when scaling the temporal features. To demonstrate temporal layers modeling inter-frame dependency, we multiply the output of the temporal layer in each block by a constant factor $\gamma$ before adding it back to the input features $x_{in}$, i.e., $x_{out}=x_{in} + \gamma * `SA-T`(x_{in})$. The synthesized frames are exhibited in Fig. C. When $\gamma$ is 0, there is no interaction between frames, resulting in no correlation among the generated video frames. The inter-frame consistency strengthens as $\gamma$ increases, which confirms our hypothesis.

Note: The corresponding videos of Fig. A, B, and C are available at https://uni-edit.github.io/UniEdit/#sectionRebuttal

We hope the above responses address your concern. We would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.

 

[1] Cao et al. "Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing." ICCV'23.

[2] Yang et al. "Rerender a video: Zero-shot text-guided video-to-video translation." SIGGRAPH Asia'23.

[3] Liu et al. "Video-p2p: Video editing with cross-attention control." CVPR'24.

Thanks for the elaborate review! We will address your concerns below:  

The number of the participants in the user study might not be representative enough.

Thanks for the advice! We additionally recruited 20 participants to make their evaluations on the synthesized videos of the proposed UniEdit and baselines, the average rating scores of 30 participants (same as Rerender-A-Video [1]) are as follows in Tab. A. It's observed that UniEdit outperforms baselines on temporal consistency and alignment with the target prompt. We will update Tab. 1 in the final version.

Tab. A: User preference comparison with state-of-the-art video editing techniques.

 

The first row of the table 2 is missing. What is that?

The first row refers to not using any of the three components, i.e., performing vanilla text-to-video generation conditioned on the target prompt. As shown in Fig. 6b, vanilla text-to-video generation results in content inconsistencies when compared to the source video. The proposed modules notably enhance the editing results on both quantitative results in Tab. 1&2 and qualitative results on the project website. We will modify Tab. 2 with Tab. B to make it informative and easy to understand in the final version.

 

It will be more informative if you could ablate with motion injection and structure control solely in Table 2 as well.

Thanks for the advice! We complement the results below. It's observed that the use of motion injection significantly enhances ‘Textual Alignment’, indicating successful transfer of the targeted motion to the main editing path, and structure control mainly contributes to the ‘Frame Similarity’ metric. The best results are achieved through the combined use of all components.

Tab. B: Impact of various components.

 

How do you measure texture alignment and frame consistency in Table 2?

We will add the explanation below in the final version:

To evaluate the adherence of the edited video $V$ ($N$ frames) to the target prompt $P_t$, we follow previous work [2] to compute the average similarity of the CLIP embedding of the edited video clip and its corresponding text prompt, a metric we refer to as 'Textual Alignment'. To quantify frame consistency, we calculate the avarage cosine similarity of CLIP embeddings between adjacent frames. Formally, the two metrics are defined as:

$\text{Textual Alignment} = \frac{1}{N} \sum_{i=1}^{N} \frac{\text{CLIP}(V_i) \cdot \text{CLIP}(P_t)}{\|\text{CLIP}(V_i)\| \cdot \|\text{CLIP}(P_t)\|}$

$\text{Frame Consistency} = \frac{1}{N-1} \sum_{i=1}^{N-1} \frac{\text{CLIP}(V_i) \cdot \text{CLIP}(V_{i+1})}{\|\text{CLIP}(V_i)\| \cdot \|\text{CLIP}(V_{i+1})\|}$

 

We hope the above responses address your concern. We would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.

 

[1] Yang, Shuai, et al. "Rerender a video: Zero-shot text-guided video-to-video translation." SIGGRAPH Asia 2023 Conference Papers. 2023.

[2] Wu, Jay Zhangjie, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.