Contextually Harmonious Local Video Editing

[1] Contextually harmonious video editing is not a new concept. Traditional video editing always strives for harmonious integration without altering unintended attributes. Can the authors clarify the distinction between standard video editing and the proposed "contextually harmonious video editing"?

[2] Consistency and preservation of attributes seem lower than recent standards in video editing quality. For instance, the man's face in Figure 1 appears noticeably altered.

[3] Are there video results available? I can not validate the consistency by only two resulting frames in the paper. The sample provided appears to exhibit noticeable flickering, which may disrupt visual continuity.

[4] How do the authors measure harmony? The concept of harmony is highly abstract and subjective. Is there an objective metric used in this study?

Here are some weaknesses in this paper:

1. Although the paper aims to implement a divide-and-conquer approach, it heavily relies on pre-existing models or methods to enhance its pipeline, which raises questions about its novelty as a research contribution. For instance, the pipeline depends significantly on preliminary support from Vision-Language Models (VLMs) for extracting moving objects and gathering additional information, such as size. This dependency introduces potential failure points: what if the VLM model fails to detect the objects accurately or incorrectly estimates the size ratio between the source and target objects? Such reliance could lead to various failure cases stemming from the external model’s limitations.

2. The contribution to temporal consistency is questionable. The authors employ feature consistency based on an inter-frame DIFT similarity matrix, but it remains unclear if this approach can fully address pixel-level flickering issues. Aligning with this concern, Table 1 shows that the proposed method underperforms in terms of temporal consistency, highlighting a critical weakness.

3. The qualitative evaluation is limited without accompanying videos. Providing some qualitative video examples would enhance the presentation of effectiveness.

4. Any reason of choosing 40 videos out of 76 TGVE dataset?

-- Although the paper achieves much better performance, its three-stage pipeline (final stage is a coarse-to-fine stage) is a bit tedious and might not be easy to use.

-- The paper requires pre-estimated masks using pre-trained models. However, the estimated masks could be inaccurate. Therefore, it is necessary to evaluate the method's robustness with respect to the accuracy of masks.

-- No edited videos are provided. It is necessary to evaluate the method's temporal performance.

• The paper claims to be the first to focus on local video editing (line 100), a statement that is questionable given the existence of recent works like "AVID: Any-Length Video Inpainting with Diffusion Model" (CVPR 2024), "Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion" (ECCV 2024), and "VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence" (CVPR 2024). While VideoSwap requires a reference image, this requirement does not diminish its relevance to local video editing. Given these prior works, it would be beneficial for the authors to clarify the distinctions of their method. Specifically, I would like to understand the differences between applying one of these methods with a provided object mask versus using the proposed method (which also extracts the object mask using a segmentation model). A thorough discussion on these distinctions would enhance the clarity and novelty of the paper.

• The paper suffers from significant writing issues that require major revisions. The language is often unclear, and there are numerous grammatical errors, which hinder the readability and impact of the work. Additionally, Figure 2 is particularly difficult to follow in its current form. Splitting this figure into three distinct modules or sections could make each stage more comprehensible. Alternatively, restructuring Figure 2 with improved labels and visual cues could help readers better understand the pipeline, especially when cross-referencing it with the manuscript.

• The paper relies on Vision-Language Models (VLMs) to assess structural similarity and size ratios, yet it lacks a detailed discussion on the reliability of these models in less common scenarios. For instance, it is unclear how the model performs when tasked with identifying rare or unusual objects. Experimental results or evidence demonstrating the VLM's effectiveness and limitations in such cases would greatly strengthen the paper's claims.

• The paper does not provide any information on runtime or GPU requirements, which are critical for assessing the practical feasibility of the proposed method. Including a comparative analysis in terms of computational cost and runtime would offer valuable insight into the method's scalability and reliability, particularly for real-world applications where these factors are crucial.