Consistent Video-to-Video Transfer Using Synthetic Dataset

Thank you for your review and the rating of our paper. We appreciate your feedback and the opportunity to address your concerns.

1. Need for Fine-Tuning and Dataset Creation Costs:

   • Recognizing the trade-offs between initial training costs and long-term efficiency, we appreciate your observation regarding the fine-tuning requirements and dataset creation expenses. Our method, termed 'one-model-all-vid', does require an initial training phase, a fundamental element of our approach. The primary advantage here lies in its universal applicability and enhanced efficiency during the inference phase. This is in contrast to 'per-vid-per-model' approaches that necessitate repetitive fine-tuning for each new video, as delineated in Table 1.
   • The rationale behind our strategy is that the upfront investment in training a versatile model offers substantial benefits in terms of efficiency in subsequent inference stages. This aspect is particularly advantageous considering the varied applications in real-world scenarios. The initial costs of dataset creation and model training, we argue, are offset by these long-term efficiencies.
   • Please refer to the revised Section 3 of our updated paper, where we explicitly discuss the trade-offs involved in creating a synthetic dataset. This elaboration aims to provide a clearer understanding of our method's rationale and its relative benefits.

2. Incremental Technical Contribution and Long Video Sampling Strategy:

   • Our technical contributions include the creation of a synthetic dataset and the development of an effective method for achieving inter-batch consistency in video editing. We also trained a model capable of conducting video editing without additional control and free of per-video tuning.
   • The Long Video Sampling Correction (LVSC) is not merely a sliding window operation. While a simple sliding window method can process video frames in overlapping segments, it does not ensure that the overlapping frames yield consistent results. The key to our LVSC lies in the algorithmic approach (detailed in Equations 4 and 5 of our paper), which is designed to align predictions across overlapping frames in different batches, ensuring consistency and coherence in the final video output.
   • We have revised our paper to include a new illustrative figure in Appendix C, which delineates the interaction rules between preceding and subsequent batches. This addition aids in better understanding the LVSC's functionality and its distinction from a naive sliding window approach. We kindly request the reviewer to refer to the updated Appendix C for a comprehensive overview of these enhancements.

3. Dataset Release:

   • We're undergoing the release process of dataset. Alongside the dataset, we also plan to make available the code necessary for reproducing the data generation, allowing for broader accessibility and replicability of our work.

We hope this response addresses your concerns and clarifies the contributions and intentions of our research.

Thank you for your insightful review and the positive rating. We appreciate your constructive feedback and will address each point to enhance the clarity and depth of our work.

1. Jitter in Newly Added Content:

   • We acknowledge the issue of jitter, particularly in newly-added content, which stems from the higher complexity in generating these elements without adequate reference data in the conditional video. To address this, a possible solution is the integration of more advanced diffusion models with better generative capacities, which may offer better handling of the complexities involved in newly added content. Additionally, enhancing the diversity of our training data could improve the model's ability to generalize, thus mitigating jitter.

2. Reliance on Synthetic Paired Video Dataset:

   • The quality of our synthetic paired video dataset is indeed pivotal to our model's performance. We utilized real-world video captions to craft this dataset, aiming to mirror real-world scenarios more closely. Nevertheless, we acknowledge potential discrepancies between our dataset and actual scenarios. Future efforts could involve using text-to-video models that align more closely with real-world distributions, further bridging this gap for practical applications.

3. Discussion of Failure Cases and Limitations:

   • We agree that discussing potential failure cases and limitations is crucial. Instances where our model struggles include editing targets that are partially visible or challenging for the model to semantically recognize due to size constraints. We have expanded on these failure cases in the updated appendix (see Appendix G).

Regarding your questions:

1. Quantifying Differences with Reference Datasets:

   • A quantitative comparison, like computing the Fréchet Inception Distance (FID) between our synthetic video dataset and a reference one, provides insightful data on how our dataset mirrors real-world scenarios. Beyond just assessing data realism with FID, future work can involve examining inductive biases by exploring statistical measures like the diversity in video types, their generation success rates, and variety. This will help in accurately measuring disparities with real-world scenarios using various metrics.
   • Regarding our current CLIP-based sampling, which is heuristically driven and biased by human feedback, we note that certain prompts seem more successful in generating synthetic video pairs. Identifying whether this bias originates from the generative model or the sampling criteria is an open question. Future work can involve analyzing successful generation distributions across different prompts and their alignment with the generative model's biases, as well as assessing our sampling criteria's impact on the dataset's diversity and real-world representativeness.

2. Compatibility with Various Content, Instructions, and Methods:

   • Exploring the compatibility of our method with diverse video content, editing instructions, and underlying video generation models is indeed crucial. In our research, the dataset employed for testing encompasses videos from three distinct sources, and the editing instructions cover four types. This diversity leads us to believe that our method is adept at handling a reasonable range of editing scenarios. However, our evaluation predominantly focuses on real-world videos, which are most prevalent in everyday applications. The method's compatibility with more artistic forms, such as cartoons, or semantically ambiguous videos (for instance, abstract art videos), remains less clear. We anticipate potential challenges with the latter, especially in scenarios where semantic objects are not clearly defined. Regarding editing instructions, our approach performs well when the edits do not significantly alter the video's structure, maintaining similarity between the source and target videos. However, edits that induce structural changes (like transforming a sunset scene into sunrise, where the sun's movement is reversed) pose a challenge, as our synthetic dataset does not model such scenarios effectively.
   • Concerning Text-to-Video Generation Methods: The selection of a specific text-to-video model plays a critical role due to the inherent biases each model brings. Compiling a comprehensive synthetic dataset to account for these variances is a task that demands extensive resources. At the time of our submission, our analysis was constrained by the limited availability of text-to-video models for extensive examination. Future research directions include delving deeper into this aspect to validate the versatility of our method across a spectrum of text-to-video generation techniques.

Thank you for your review and rating. We appreciate the opportunity to address your concerns and provide further clarifications regarding our paper.

1. Perceived Lack of Significant Innovation:

   • While we have built upon the high-level design philosophy of Instruct Pix2Pix, our contributions in adapting it to the video domain are significant. The introduction of Prompt-to-Prompt (PTP) for video, especially the incorporation of temporal self-attention replacement, is crucial for the success of video PTP generation, as detailed in Section 3.1. This is a substantial modification necessary for the intricacies of video data.
   • Additionally, our work pioneers inter-batch consistency in video editing with the Long Video Sampling Correction (LVSC) and Motion Compensation (MC). These contributions are not mere extensions but are pivotal in addressing the unique challenges of video editing.

2. Sampling Method and Comparison with Baselines:

   • We respectfully disagree with labeling our sampling method as just an inpainting variation. Our approach is specifically designed for diffusion models, integrating correction steps at each denoising stage's end, distinguishing it from standard inpainting methods in image or video generation.
   • Regarding the inclusion of recent video editing models for comparison, we focused initially on methods that support extensive editing capabilities, such as significant changes in object shapes. However, we agree that incorporating recent advancements in tuning-free video editing methods is crucial for a thorough comparative analysis. Therefore, in our final camera ready version, we would like to include comparisons with the mentioned tuning-free video editing approaches [1,2,3]. We initially did not include Rerender A Video[2] in our comparison due to its primary focus on stylization editing. Nevertheless, we recognize the importance of encompassing a variety of approaches in our analysis and will aim to provide a more comprehensive comparison in the updated version of our paper.

3. Generated Video Results:

   • We recommend referring to the numerical comparisons in the experimental section for a more reliable assessment of our method's performance. The supplementary videos are intended to provide additional context but may not fully represent the method's capabilities.

Regarding your questions:

1. Temporal Replacement and Motion Modification:

   • The temporal attention layer replacement in Section 3.1 is primarily aimed at ensuring structural similarity between the input and edited videos. This might give the appearance of frame-by-frame style transfer, but in reality, the frames are processed collectively in batches, not individually, which aligns well with our video editing framework.
   • Regarding altering the motion in videos, such as changing the direction of a person walking, it presents an intriguing yet complex editing challenge. In scenarios like these, the structure of the input and output videos would significantly differ, which is challenging under our current approach. Our data generation pipeline is designed to produce videos with similar visual structures. While our existing model isn't equipped to handle such extensive structural changes, exploring this capability certainly presents an interesting avenue for future research.

2. Calculation and Improvement of Success Rate:

   • The success rate is calculated as the ratio of successfully generated synthetic videos that pass the CLIP filter criteria (mentioned in Section 3.3) to the total number of generation attempts.
   • To improve this rate, one approach is to source better video descriptions, as discussed in Section 3.2. Other potential methods include employing video generation models with higher capacity or refining the generation algorithm to produce more accurate and relevant synthetic videos.

3. Advantages of Video Diffusion Models Over Optical Flow-Based Methods:

   • Utilizing optical flow in image diffusion models for video editing is a concept worth exploring, yet its application is not straightforward. There are two primary challenges with this approach: Firstly, when frames are processed sequentially, errors can accumulate rapidly, leading to significant inconsistencies. Secondly, occlusions in the video can create regions where warping operations struggle to maintain consistency, further complicating the editing process.
   • In contrast, video diffusion models inherently embed consistency within their architecture, offering a learnable approach to maintaining coherence across frames. A key advantage here is that frames are processed together in a single batch, which helps in preserving the continuity and overall structure of the video more effectively than methods reliant on optical flow in image diffusion models. This integrated processing approach of video diffusion models provides a more robust and cohesive solution for video editing tasks.

Thank you for the review and the positive rating of our paper. We value your input and would like to address your points to further clarify our work's contributions and address your queries.

1. Similarity to Instruct Pix2Pix:

   • Indeed, our work is inspired by the Instruct Pix2Pix framework, particularly in adopting the principle of generating paired data and tuning a model on synthetic data. However, our approach significantly extends this concept into the video domain, which involves unique challenges not present in image editing.
   • A key innovation in our work is the application of the Prompt-to-Prompt (PTP) method to video, where we identified that temporal self-attention replacement is crucial for successful video PTP generation, as detailed in Section 3.1. This adaptation is non-trivial and vital as it ensures the paired synthetic videos retain a similar visual structure.
   • Additionally, while previous works have focused on intra-batch consistency in video editing, our Long Video Sampling Correction (LVSC) and Motion Compensation (MC) mechanisms are pioneering techniques for maintaining inter-batch video consistency in diffusion models. These contributions represent an advancement in the field of text-guided video editing.

2. Temporal Consistency Challenges:

   • Compared to existing works, our approach takes a step towards an ultimate solution for generating highly consistent videos. The implementation of Long Video Sampling Correction (LVSC) in our method represents an advancement in enhancing temporal consistency. We acknowledge that achieving absolute perfection in this domain remains a challenging and ongoing endeavor. Nonetheless, our method's progress in this area is a testament to its potential and the direction for future research to fully realize the goal of flawless temporal consistency in video editing.

3. Large Language Model (LLM) Used:

   • For the in-context learning process, we utilized the MPT-30B model, as described in Section 3.2.