Image-driven Video Editing with Latent Diffusion Models

Experiments & results:

• Quality. The quality of results is generally not satisfactory, especially in terms of the consistency.
• The ablation study seems not to be sufficient. For example, authors also mention they did not replace the objects as they wanted to keep the accurate object in the edited video, but this does not appear in the ablation study.
• The number of qualitative results. Although LIVE seems to work better on the conventional videos in Video-P2P and Vid2Vid-zero, it is hard to evaluate it because the size of the dataset is too small. The evaluation itself has some limitations and does not show the full performance of the proposed method.
• Video scenarios. The proposed method is mostly tested on the videos that has motion horizontally. How does the method work for the videos that have more complicated motion? For example, a person turning around or an animal jumping?
• Limitations. The paper does not include any failure cases, which is quite common in image/video synthesis papers. Having some limitations can help people better understand the proposed method from different perspectives.

Writing:

• Some typos.
  • In Figure 2 caption, "unet"-> "U-Net", $z1$ -> $z_1$.

1. Didn't see a reference to Figure 1 in the paper. Please introduce Figure 1 in the paper and further explain what the S* in the figure means.
2. The exact meaning of the different parts of Figure 2 is not clear. In a, are the text-video pair and input images given as input together, or are they split into two training sessions? In b, the input for inference should contain three parts: the original video, the input prompt, and the reference image, why is there only an input prompt here? In c, please give the meaning of each part in words.
3. The paper uses a video length of 8 frames for the experiments, but does not provide a detailed analysis of the impact of the video length on the performance of the proposed method.
4. The paper briefly mentions that the proposed method is more efficient than some other methods, but does not provide a detailed analysis of the computational efficiency. This may be an important factor for practical applications.
5. The paper does not provide a detailed discussion of the limitations of the proposed method, such as the types of videos that may not work well with the method or the potential failure cases.
6. What is the source video input in Figure 7 and Figure 8?

The paper claims three contributions: the novel task, the LIVE framework, and extensive experiments. However, these claims lack robust substantiation.

1. Task Definition: The paper fails to formally define the task. It is unclear whether a video caption is essential or merely a requirement for the proposed framework. The scope of video editing covered (e.g., object swapping, addition, removal, style changes) remains ambiguous.

2. Framework Choices: The paper does not explore alternative options for each framework component or their respective advantages and drawbacks. For instance, it does not justify the decision to learn a token embedding without also fine-tuning the text encoder, as done in methods like DreamBooth.

3. Experimental Rigor: Labeling the experiments as "extensive" is misleading given the limited number of test videos. Although the method outperforms baselines, notable artifacts remain (e.g., over-exposure, unintended changes in the background or other objects). A more systematic evaluation is necessary.

The method description could also be confusing or vague in several aspects:

4. How to actually "inflate" a T2I model to a T2V model (compared with the inflation proposed in the Tune-A-Video paper)? In Eq (5), it should be the intermediate feature maps rather than the input latent codes $z_*$, right? Furthermore, if we concatenate the features, the dimension of the matrix $W^K$ and $W^V$ should be expanded as well?

5. In Eq (4), the proposed method optimizes $L_t^{unet}$ first and then optimizes $L_t^{text}$. Why would it be equal to optimizing the sum of them? It should also rigorously define the loss terms (e.g., what are the actual learnable parameters?).

6. The necessity of the prompt-to-prompt technique, especially the replacement of verbs and the background, is not convincingly argued. Why would only replacing the subject not sufficient? What would the attention maps look like?

Moreover, practical concerns are raised:

7. How can we determine when to stop the fine-tuning? I assume that we do not want to "over optimize" the loss terms to zero (which could destroy the original T2I model's capability), right?

8. The lengthy processing time (25 minutes for an 8-frame video?) could significantly hinder the framework's practical impact. A discussion on the trade-off between quality and latency is needed.

1. The writing of the paper is poor. Section 3 Methodology should be restructured so that each training objective is expressed clearly
2. More experiments and ablation studies are needed to justify the effectiveness of each component in the model.
3. In Section 3 equation (4), the loss function combines the loss for training the video diffusion model and the loss for training the text embeddings in textual inversion. It seems like with this joint training objective the two parts are trained together. However, in section 3.3 the author mentioned "However, during the second training objective, we freeze the latent prompt..." which sounds like the training is conducted in two phases. It is unclear to me how the training is conducted with the two training objectives.
4. For the TAV+DDIM baseline model, how did the author enable image conditioning with this method? As far as I know, TAV takes text and video inputs and does not explicitly condition on input images. Did the author use a personalized dreambooth model as the backbone of TAV to condition on the input images?
5. Based on the demo it seems that LIVE works well with a single conditioning image while textual inversion and dreambooth require at least 3-5 input images to train a personalized model. It would be interesting to discuss why LIVE worked well with only 1 image.