VEBench: Towards Comprehensive and Automatic Evaluation for Text-guided Video Editing

1. Why davis? and include more different sources and domains to increase diversity

Thank you for your insightful comments on the diversity of our dataset. We chose DAVIS as our primary video source because it is a well-established and widely used testbed in the video editing research community, ensuring that our benchmark aligns with existing standards and facilitates meaningful comparisons with other studies.

However, we understand the importance of incorporating a broader range of video sources to enhance diversity. We are actively working on expanding VEBench to include videos from additional platforms such as YouTube and Videvo, which will allow us to cover a more extensive array of topics and domains, thus improving the dataset’s overall diversity and relevance.

To highlight VEBench's current standing, please refer to the comparison table below, which demonstrates how VEBench surpasses existing datasets in terms of video and text prompt numbers, as well as its unique support for meta-evaluation:

The asterisk (*) denotes data reused from source [1], while the dash (-) indicates that existing benchmarks do not support meta-evaluation, except ours.

We greatly appreciate your feedback and are committed to making these enhancements. If you have additional suggestions for other potential data sources or domains, we would be happy to consider them to further enrich VEBench. Your feedback is invaluable in helping us improve our work.

[1] Wu, et al. "CVPR 2023 Text Guided Video Editing Competition." *arXiv preprint* arXiv:2310.16003, 2023.

2. Providing concrete example in introduction

Thank you! We will definitely improve this part.

3. Edit multiple objects of the same categories

It depends on the edit prompts. For example, "edit the brown bear on the left to polar bear" or "edit all brown bears to polar bears".

Currently, we take each edit target as an exact object in the video instead of objects of the same categories.

We check the dataset and do not meet this problem because the edit prompt is precise and there is no ambiguity in editing an object or objects of the same category.

4. Incorporate various conditional data for video editing

As mentioned in the Section Introduction, the majority of current Video Editing is Text-guided Video Editing. Therefore, from a practical perspective, we have chosen to evaluate this approach exclusively. We agree other types of video editing are also valuable and will provide a discussion in the related work section.

5. Specific challenges for editing long video

This is an insightful suggestion. We did not conduct analyses on these cases because current technologies are largely limited to editing short clips, typically within 25 frames. However, we believe this is an important direction and plan to include longer videos from other sources (e.g., YouTube) to better support the evaluation of this case.

Thank you! It seems there is a misunderstanding in the contribution of our work. We kindly note that VEBench is not only the largest-ever video editing benchmark, but it also serves as the first meta-evaluation benchmark in this field to help develop automatic evaluation tools. By carefully designing the evaluation criteria and providing high-quality human annotations on 1280 edited videos, we believe VEBench paves the way and can accelerate the development of more reliable evaluation tools.

Our responses to your questions are as follows.

1. Without employing fine-tuning or advanced techniques like token probability averaging

We are the first to explore utilizing MLLMs for evaluating video editing. We follow the popular approach [1] to implement our VEScore. We agree that finetuning or other techniques may further enhance the performance. However, finetuning needs to craft corresponding training data, which is non-trivial to obtain and has not been explored in this field. Token probability averaging is also been demonstrated not reliable in expressing uncertainty [2].

[1] Zheng, et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." *arXiv e-prints*, arXiv:2306.05685, June 2023.

[2] Wang, et al. "Self-Consistency Boosts Calibration for Math Reasoning." *arXiv preprint*, arXiv:2403.09849, 2024.

2. Consistency and reproducibility of prompt engineering.

Thank you! We will explore paraphrasing the instruction to test the robustness of VEScore.

We did not specially conduct prompt engineering because this necessitates a development set. While we believe this can be potential with the help of VEBench.

Utilizing GPT, we generated multiple paraphrased versions of the instruction, assessed their performance, and analyzed these variations to verify robustness. Due to computational time considerations, this part of our experiments was based on LLaVA-OneVision-7B. In the future, we will evaluate all MLLMs. The outcomes are summarized in the following table:

TF, FC, and VF denote Textual Faithfulness, Frame Consistency, and Video Fidelity, respectively. And the metrics $r$(↑), $\rho$(↑), and $\tau$ (↑) correspond to Pearson Correlation, Spearman Correlation, and Kendall’s Tau, respectively.

We have included a calculation of the mean and standard deviation for each evaluation metric as follows:

• TF $r$: 0.49 ± 0.02

• TF $\rho$: 0.47 ± 0.02

• TF $\tau$: 0.39 ± 0.01

• FC $r$: 0.20 ± 0.04

• FC $\rho$: 0.21 ± 0.05

• FC $\tau$: 0.17 ± 0.04

• VF $r$: 0.11 ± 0.02

• VF $\rho$: 0.13 ± 0.04

• VF $\tau$: 0.10 ± 0.03

These statistics were derived from multiple repeated experiments. We calculated the mean and standard deviation for each metric to assess consistency and robustness under various rewritten prompt versions. We found that overall, MLLMs are relatively robust to changes in prompts, with relatively small standard deviations and minor differences between the means and the values reported in our paper. Additionally, we have provided the specific prompts used in the article in the Supplementary Material and plan to continue open-sourcing these prompts to ensure the reproducibility of all experimental results.

3. The dependence on large-scale models (32B) or expensive closed-source APIs is impractical

We agree that the VEScore is more expensive than traditional metrics based on CLIP. However, it is still much cheaper than human evaluation.

Besides, we have demonstrated that some small models (e.g., LLaVA-OneVision-7B) also perform well on some criteria. We believe evaluation can be more efficient with the rapid development of small-scale open-source LLMs.

Thank you! It seems there is a misunderstanding in the contribution of our work. We kindly note that VEBench is the largest-ever video editing benchmark. More importantly, it serves as the first meta-evaluation benchmark in this field to help develop automatic evaluation tools. This is an emergent requirement due to the heavy reliance on human evaluation to date. We are also the first to demonstrate the potential of utilizing MLLMs as evaluators for video editing. We believe our work can benefit the community. There are the responses to your questions. Hope these can address your concerns.

1. The amount and diversity

Thank you for highlighting this aspect. We acknowledge that while VEBench is not as large as some general-purpose datasets, it is significantly larger than other specialized video editing datasets, as outlined in the related works section. More importantly, VEBench serves as the first meta-evaluation benchmark for video editing, which we believe will be instrumental in the development of reliable automatic evaluation tools, thus alleviating the current reliance on extensive human evaluation efforts.

The asterisk (*) denotes data reused from source [1], while the dash (-) indicates that existing benchmarks do not support meta-evaluation, except ours.

As illustrated in the comparison table, VEBench surpasses existing datasets in both video and text prompt quantities. Additionally, we have selected the DAVIS dataset due to its prominence in previous studies, ensuring our work aligns with established research. Nevertheless, we are actively working on incorporating videos from other sources like YouTube and Videvo to further enhance the diversity of VEBench. We appreciate your suggestions and are open to any additional feedback you might have on how we can further improve this aspect of our work.

[1] Wu, et al. "CVPR 2023 Text Guided Video Editing Competition." *arXiv preprint* arXiv:2310.16003, 2023.

2. Each video has only 25 frames

Sorry for the misunderstanding. VEBench contains video with frames ranging from 25 to 127. Due to the constraints of popular video editing models, we only adopt 25 frames as the model inputs for evaluation. We will improve the corresponding part in the manuscript.

3. Lacking examples of VEBench and VEScore

Thank you! We will provide more typical cases of our dataset and evaluation results in the revision. We kindly note that statistical results (Table 6, Figure 6) can demonstrate the diversity of VEBench. Besides, an error analysis about VEScore using different MLLMs is also provided in Sec B3.

4. Specific processes and standards when selecting human subjects

Thank you for your feedback. We recruited annotators who hold a master's degree and have a background in multimodal research. Before the official annotation process began, we conducted a one-week trial phase to evaluate accuracy and consistency. This was followed by a two-week formal annotation period using our annotation platform. We ensured that none of the annotators had any conflicts of interest, and we routinely cross-checked their annotations for consistency to maintain high-quality standards.

We would like to thank the reviewer for their valuable feedback and positive comments. We appreciate the recognition of the innovative aspects and potential contributions of our work. Below, we address the concerns raised:

1. The amount and diversity of VEBench.

Though VEBench is smaller than the general-purpose dataset, it is still much larger than other datasets in this field as described in the related work section.

More importantly, please note that VEBench also serves as the first meta-evaluation benchmark for video editing. We believe VEBench can help develop more reliable automatic evaluation tools to address current heavy human efforts.

A comparison with the existing dataset is shown in the table below:

The asterisk (*) denotes data reused from source [1], while the dash (-) indicates that existing benchmarks do not support meta-evaluation, except ours.

We chose DAVIS as a video source because it is the most popular testbed in previous studies. By statistics, VEBench covers a wide range of topics as shown in Table 5 and Figure 6. This demonstrates the diversity of VEBench.

Some works also adopt videos from YouTube and Videvo, but their scales are often limited. We are working on appending these videos of different sources to VEBench.

We will improve the corresponding part in the manuscript to avoid misunderstanding and appreciate it if you could provide any further suggestions.

[1] Wu, et al. "CVPR 2023 Text Guided Video Editing Competition." *arXiv preprint* arXiv:2310.16003, 2023.

2. In-depth analyses of using MLLMs as evaluators

Table 4 presents the performance of MLLMs across various dimensions. Besides, In Table 6, we provide an error analysis about different MLLMs (This part is put in the Appendix due to page limit). We notice that one important issue is the capability of faithfully following our instructions due to the capabilities of weak MLLMs and safety concerns of close-sourced models. Nevertheless, VEScore still performs significantly better than traditional metrics. We believe this approach has potential and can be further improved with the guidance of meta-evaluation annotations in VEBench.

Thank the author for the rebuttals. This work has some existing issues that require further attention, but with some improvements, I believe it has the potential to reach the necessary standard for acceptance.