LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation

6. User surveys:

   Good suggestion. We refer to previous works[1,2,3] and adopt a Two-alternative Forced Choice (2AFC) protocol, where participants are shown the original video, the translation result of a compared baseline method and the result of our method. The participants are asked to choose which result they prefer.

   We select 15 video-prompt pairs for evaluation and we collect 420 votes from 28 users. The results are provided in the table below, which also verifies the superiority of our method considering realistic user evaluations. We will include this result in our revised paper.

   Methods	Tune-A-Video	Text2Video-Zero	TokenFlow
   User preference of our method	88.6%	80.0%	72.9%

7. Running costs

   We have measured the inference time and GPU resource consumption of different models for reference. Because the number of generated frames are limited for some methods, for fair comparison, the video length is set to be 8 and the image size is set as 512 x 512 for all the methods. 50-step DDIM sampling strategy is adopted. All the experiments are conducted on one NVIDIA A100 GPU. From the table, it can be seen that our method is efficient concerning both the inference time and GPU memory consumption.

   This could be attributed to reasons of two aspects. The main advantage of our method is that our method skips the DDIM inversion process, which is typically adopted in video editing tasks to preserve the temporal prior while it consumes lots of time. However, the objective could be achieved through latent alignment operation alternatively, which is obviously more efficient. Moreover, since the warping operation is performed on the latent space, a batch of frames could be denoised on parallel. In contrast, Rerender-A-Video perform the warping operation on the pixel level, which makes the frames need to be generated one by one.

Inference time.

GPU memory consumption.

[1] Geyer, Michal, et al. "Tokenflow: Consistent diffusion features for consistent video editing." arXiv preprint arXiv:2307.10373 (2023).

[2] Kolkin, Nicholas, Jason Salavon, and Gregory Shakhnarovich. "Style transfer by relaxed optimal transport and self-similarity." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] Park, Taesung, et al. "Swapping autoencoder for deep image manipulation." Advances in Neural Information Processing Systems 33 (2020): 7198-7211.

Thank you for the supportive feedback.

1. Data-related issues

   In Section 5.1 of our paper, we have claimed that the dataset used for our experiments is DAVIS, which is the same dataset used in "Tune-A-Video." We acknowledge that in some cases, such as in Figure 1, we choose to include partial images which is not from DAVIS. Our intention is to demonstrate the performance of our approach under challenging conditions and highlight its effectiveness on handling complex objects and backgrounds.

2. Base model choice

   Firstly, it is important to note that many of the comparative methods we evaluated in our study also utilized ControlNet, such as Rerender a video. This choice was made to ensure a fair and consistent baseline comparison across different techniques. However, we understand the concern that ControlNet's strong structural control may potentially diminish the observed improvement from LatentWarp. To address this concern, we have conducted experiments comparing the performance of LatentWarp with and without controlnet in the ablation experimentaion in the supplementary material. Such experiments demonstrate the distinct contributions and improvements offered by LatentWarp using LDM directly.

3. Quantitative data details

   Our evaluation dataset comprises of 24 text-video pairs from DAVIS, i.e., twelve videos and each video corresponds to two prompts. We extract the first 24 frames of each video and calculate the metrics.

   The twelve videos are bear, blackswan, boat, bus, camel, car-turn, dog-agility, elephant, flamingo, gold-fish, hike, and train.

   We will supplement these details in our revised paper.

4. Compared methods

   We conducted extensive experiments to demonstrate the effectiveness and superiority of our proposed method in comparison to more recent approaches, including Rerender a video, Tokenflow, Video-P2P, and others. The detailed results of these video-based comparisons can be found in in the supplementary material.

5. Supplementary material

   Please see the corresponding video supplementary materials

Thank you for the supportive feedback.

1. Comparison with Rerender-A-Video.

   Although introducing optical flow into diffusion-based video processing has been explored in prior work, such as Rerender-A-Video, there are several key differences in our approach:

   1. Firstly, we extract both forward optical flow maps and backward optical flow maps between adjacent frames and apply a warping operation on the latent of the last frame to align with the current frame using these flow maps.
   2. The main way we calculate the binary mask for preserving or replacing parts of the warped latent differs from previous optical-flow based diffusion methods, such as Rerender-A-Video. Our methods utilizes a binary mask to determine which regions of the warped latent should be preserved and which should be replaced with latent of the current frame. The details is illustrated in method 4.3.
   3. There are also differences in the fusion stage between our method and Rerender-A-Video. This stage involves combining the warped latents with the generated latents to produce the final output.

2. Presentation of video results.

​ Please see our global response.

3. How to keep the consistency of the unwarped regions.

​ We preserve the consistency of the unwarped regions through cross-frame attention mechanism, i.e., sharing the key and value tokens across attentions of different frames, to encourage the temporal consistency. As illustrated in the ablation study, without cross-frame attention, the left part of the scene exposed to us with the camera turning left would appears significantly inconsistent with the other regions, hampering the video quality, while our method effectively avoids this issue.

4. Comparison with Edit-A-Video and Rerender-A-Video.

​ Since Edit-A-Video has not released the code so far, we provide the comparison with Rerender-A-Video and other methods in the supplementary material. We will cite and discuss these methods in our revised paper.

Thank you for the supportive feedback.

1. Ablation studies on performing latent alignment on the last few denoising steps.

​ We conduct the experiment of applying latent alignment operation on the first 16 steps and on the last 4 steps in the denoising process, and present the video results in the supplementary material. Although the visual consistency can also be maintained through the last few steps, we observe that errors in the warping operation and optical flow estimation would be introduced in the denoising process especially in the last few steps and reflected on the generation result, hampering the generation quality.

2. Ablation studies on hyper-parameter tuning and optical flow network choice.

   Please refer to the global response for abltion on hyper-parameter. We also provide the video results with the optical flow extracted by GMFlow[1] and there are no obvious differences.

3. Editing with structure deviations.

   1. In this paper, we deal with a specific video translation task, preserving temporal consistency of stylized videos with original videos in terms of both object appearance and motion is a primary objective of our method. Thus, our method may handle some specific structure deviations which to an extent can be viewed as a by-product of stylization.
   2. More general video editing, like adding a hat on a running dog, involves significant object structure deviations, and usually requires a motion prior to ensure a consistent editing across frames. The challenging task exceeds the focus of this paper, and we remain it as a direction to extend our work in the future.

[1] Xu H, Zhang J, Cai J, et al. Gmflow: Learning optical flow via global matching[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 8121-8130.

Thank you for the supportive feedback.

1. Presentation of video results

   Please see our global response.

2. Explanation of the equation in section4.2.

   It is not a mistake. $warp(I^{i-1}, m^{i->i-1})$, which is used in our paper, refers to a backward warping operation on the (i-1)-th frame $I^{i-1}$ with the backward optical flow . Differently, $warp(I^{i-1}, m^{i-1->i})$ denotes a forward warping operation with forward optical flow. Both of them can warp the last frame to the current one. However, backward warping has some advantages over forward warping. One of the main advantages is that it can avoid the problem of holes, which is common in forward warping. Holes refer to situations where some pixel points do not have corresponding original pixel points after mapping, resulting in empty spaces or non-existent pixels, which is illustrated in this link. Therefore, we incorporate backward warping in our method rather than forward warping.

3. Clarification of Fig. 5

   Sorry for this misunderstanding. In Figure 5, we aim not to show the improvement of our method, but to demonstrate that latent alignment effectively preserves the fine-grained visual details. We will clarify this in our revised paper.

4. Advantages of zero-shot video translation techniques compared with video diffusion model.

   Compared with video diffusion models, our zero-shot video translation technique has the following advantages:

   1. One significant advantage is the ability to handle longer videos. Image diffusion models typically operate on individual frames, allowing them to handle videos of arbitrary length without the computational and memory limitations that may arise with video diffusion models.

   2. Furthermore, image diffusion models provide flexibility in the editing process. They allow for more fine-grained control over individual frames, enabling targeted modifications and adjustments to specific parts of the video without affecting the entire sequence. This can be particularly useful in scenarios where precise video editing is required, such as in film production or content creation.

5. Hyper-parameter selection.

   Please see our global response.