Evaluation of Text-to-Video Generation Models: A Dynamics Perspective

Q1: Compare to existing metrics, such as Motion Quality in EvalCrafter. Consider to use the average of three motion quality metrics in EvalCrafter (after normalization) and show which one is aligned with human evaluation better?

The differences between our method and the existing metrics, e.g., Motion Quality in EvalCrafter, are as follows:

1. Dynamics Assessment

• Vbench and EvalCrafte rely solely on optical flow, limiting analysis to inter-frame pixel motion dynamics. They fail to capture non-pixel-motion dynamics, such as variations in illumination, style, and color.
• Our method employs seven dynamic indicators across three temporal scales (inter-frame, inter-segment, and full-video), comprehensively assessing video dynamics. This approach significantly improves correlation with human evaluation, when evaluating inter-frame dynamics it achieving an 84% win ratio, surpassing the 73% of optical flow.

2. Evaluation of Models' Capabilities on Dynamics

• Vbench and EvalCrafter created Dynamic Degree and Flow Score metrics based on optical flow, both favoring videos with greater dynamics. However, they overlook the need for videos to match varying dynamics in text prompts, such as low dynamics for low-dynamics descriptions and high dynamics for high-dynamics descriptions.
• Although EvalCrafter's Motion AC-Score assesses a model's ability to generate videos with high or low dynamics, it is a binary metric, which is a coarse measurement and can not reflect the true dynamic controllability.
• Instead, our Dynamics Range, $M_{range}$, measures a model's ability to generate videos with various dynamics, and Dynamics Alignment, $M_{align}$ measures how well models can align video dynamics to text prompts across five grades, enabling a more reasonable and precise assessment of a model's capabilities on dynamics.

3. Quality Metrics

• Vbench and EvalCrafter assess video dynamics and quality separately, neglecting their correlation. It results in that the models tend to generate low-dynamics videos to ensure high quality, limiting to reflect the model quality across different dynamics.
• EvalCrafter's Action Recognition is limited to evaluating human actions using recognition scores and doesn't directly evaluate video quality across different dynamics.
• By incorporating dynamics into existing quality metrics, we quantitatively assess a model's ability to generate high-quality videos across dynamic ranges, achieving a more comprehensive evaluation.

Following your suggestion, we use the average of motion quality metrics in EvalCrafter (after normalization) and calculated the correlation of this average metrics with human evaluation. It obtains 83% Pearson's correlation andb 74% win ratio, which are significantly lower than those in our proposed dynamics metric ( 95% Pearson's correlation and an 84% win ratio).

Q2: Inter-segment dynamics Score

We would like to clarify that the length of each segment is not 8 frames. Instead, segments are divided proportionally according to the video's duration when evaluating global aperiodicity. Experimental results indicate that the global aperiodicity in inter-segment dynamics is robust with respect to the proportion, maintaining a high correlation with human evaluations (>90%) when assessed at proportions of 1/8, 1/4, and 1/2 of the video's length.

Based the on the observation that different patches within the video take different lengths of time to change, we assess the correlation of changes over different time interval (from 1 to the full video length) base on the temporal autocorrelation factor in this paper. This constructs the Inter-segment Dynamics feature combining with the global aperiodicity score, which achieves a 95% correlation with human judgments.

Q3: Why trust Gemini 1.5 Pro for Naturalness Evaluation

1. Gemini 1.5 Pro has been validated to have the best video comprehension capabilities among Multimodal Large Language Models (MLLM) currently available [a].
2. Our validation indicates that Gemini 1.5 Pro achieves a 78% correlation with human ratings in terms of naturalness as shown in Table 3 in the paper.

[a] Fu C, Dai Y, Luo Y, et al. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis[J]. arXiv preprint arXiv:2405.21075, 2024.

Q4: Why is this protocol called DEVIL?

"DEVIL" stands for “Dynamics-based Evaluation of VIdeo generation modeLs”. The name also hints at the tough challenges that dynamics bring to video generation models.

Q5: What are Cross and circles in Figure 2

We apologize for any confusion. The rightmost plot in Figure 2 illustrates the calculation process for Dynamics Alignment and Dynamics Range. A circle represents videos where the dynamics are correctly aligned with the prompt, while a cross indicates videos where the dynamics are misaligned.

Q6: Figure 3 may need to be improved. (b) is only the word cloud for DEVIL instead of for all three benchmarks.

Thanks for pointing this out. It is true that Figure 3 (b) is the DEVIL's word cloud instead of all three benchmarks. We will rectify the caption.  

Q7: Wrong reference in line 136

Thank you for pointing out the error. We have updated the reference to: Teed, Zachary and Jia Deng. "RAFT: Recurrent All-Pairs Field Transforms for Optical Flow." European Conference on Computer Vision (2020).

Q8: What's SIM in equation (2)?

SIM in equation (2) represents the cosine similarity between two feature vectors.

Q9: What does I function mean in equation (14)?

The I in Equation (14) refers to the indicator function.

Q10: Line 223: overall h???

Thank you for pointing out the typo. The correct sentence should read: "The overall naturalness is then determined by averaging the scores of all videos."

Hi Reviewer Kt6U,

The discussion period is ending soon. Please take a moment to review the author's rebuttal and share any remaining concerns or questions.

Thank you, AC

Q1: Demonstrations of Dynamics Score of Real Videos (weakness1) and Fake Videos from Different T2V Methods (weakness2).

Thank you for your valuable suggestion. We have collected some real videos and fake videos from different T2V models with dynamic scores for demonstration. However, due to NeurIPS guidelines, we CAN'T include links in the rebuttal. Therefore, we present them as images in the PDF instead. Please refer to the attached files. We will also create a project page containing the demo videos in the future for a more intuitive understanding, as you suggested.

As demonstrated in the paper, our dynamics scores correlate with human scores at a level exceeding 90%. The improved metrics can better reflect the quality of generation across different levels of dynamics, as measured by root mean square bias. As shown in the table below, the improved metrics achieve a lower root mean square bias in assessing video quality across various dynamic levels for Motion Smoothness, Subject Consistency, Background Consistency, and Naturalness.

Q2: Computational Efficiency

The evaluation of proposed dynamics metrics incur low computational cost, processing approximately 10 FPS on a single NVIDIA A100 GPU with support for multiple GPUs. Evaluating models like VideoCrafter 2 takes about 20 minutes, reducible to minutes with parallel GPU processing—a common practice when developing video generation system.

Hi Reviewer VYfN

The discussion period is ending soon. Please take a moment to review the author's rebuttal and share any remaining concerns or questions.

Thank you, AC

Thank you for your positive feedback and score. We will ensure to include detailed efficiency information in camera-ready version. Your support is greatly important to us, and we look forward to carrying your good wishes forward. Thanks again.

Q1: Concern about segmenting videos based on length relative to total number of frames.

1. Why using relative length. Segmenting videos by relative length enables standardized comparison no matter how long the video is. The following table shows consistently high correlation values with different ratios, demonstrating our method's robustness.

2. Influence of frame rate: We standardized videos to 8 FPS, following Vbench protocol, to eliminate frame rate variability effects.

3. Influence of video length: We group videos based on the video length (max is 8s in tested models) and study the relation between dynamics scores and human scores. DEVIL robustly achieves over 90% correlation whatever the video length is.

Q2: Necessity of Distinguishing Motion Types

Thanks for highlighting this. Our paper focuses on creating a comprehensive dynamic scoring system for various motions, rather than differentiating specific motion types. In video generation, factors such as object categories, contexts, and actions affect efficacy. Therefore, we designed a benchmark with 19 object categories, 4 scenes, and 10 dynamic categories, covering over 40 subtypes. These subtypes include camera movements, actions, complex effects, and environmental dynamics, ensuring our evaluation system accurately measures video generation performance across diverse scenarios.

Q3: Examples of prompts

The following are examples of prompts with different dynamics grades, we will add them to the main paper.

Static: a man is laying on the ground.

Low dynamics: A male fencer adjusts his epee mask and prepares to duel with his sparring partner in slow motion.

Medium dynamics: Tilt up of shirtless sportsman doing pull-ups on bars during cross-training workout at gym.

High dynamics: A runner explodes out of the starting blocks, racing down the track.

Very High dynamics: A medieval siege with catapults launching, walls breaking, soldiers charging, and arrows raining down.

Q4: Report linear regression weight

The linear regression weight of each dynamics score is as follows, we will add them to the main paper.

Q5: Code release

Yes, we will release the code, rompts, and weights.

Q6: Effetiveness of improved metrics.

Improved Metrics enhances model performance evaluation through grouping assessment based on video dynamics. Its effectiveness is demonstrated from two perspectives:

1. Representativeness. Improved Metrics better represent a model's ability to generate quality videos with varying dynamics, as shown by lower root mean squared bias compared to Original Metrics. The table below demonstrates this improvement.

2. Human Correlation. Improved metrics evaluate the model as a whole, not individual videos. To evaluate consistency with human scores, annotators grouped videos by dynamics and assessed their quality(focusing on naturalness due to time limitations). We averaged scores for each group to derive a comprehensive human score per model. The table below compares the human correlation of original and improved metrics.

Our improved metric shows a 0.70 Pearson correlation with human scores, confirming its effectiveness. The original metric, dominated by low-dynamic videos, overlooks dynamics and shows a negative correlation with human scores, despite accurate individual video ratings.

Q7: Effetiveness of Runway CFG

During Runway GEN-2's evaluation, we considered CFG functionality by assigning different CFG weights based on the prompts' dynamics grade. With such strategy, the model achieves a high degree of dynamics control capability, as evidenced by the high Dynamics Alignment metric.

Q8: Some tables are never refered

We will reference the previously uncited Table 1 and Table 2 in Section 3 and Section 4 respectively.

Q9: Details of Human Refinement

We have designed detailed criteria and examples for each dynamics grade, used by both human annotators and GPT-4. See Fig. 7 in the Appendix for more information. This will be clarified in the paper.

Q10: Figure 3 is not very interpretable

1. Fig. 3 is a word cloud of all prompts in the DEVIL prompt benchmark. It shows that users often use similar terms like "rapid" and "extremely fast" for high-dynamics scenarios, making these words more prominent.
2. The number in the preceding paragraph of Eqn 3 should be corrected to 5% to match Equation 3.

Q11: The function SIM is never defined

SIM represents the cosine similarity between two feature vectors.

Q12: Orphaned footnote in line 248

Thank you for pointing out. We will make the necessary corrections.

Q13: Plots in Fig 5 are attractive but very hard to actually compare across the different methods

We'll improve Figure 5 to make it more clear.

Q1: How to differentiate dynamic video from videos with low-quality motions?

We aim to motivate models to produce videos with a wide range of dynamics while maintaining high quality. We recognize that some videos may exhibit high dynamics but poor quality. To address, we have enhanced quality metrics by integrating dynamics scores, enabling a more comprehensive evaluation of models' generative capabilities rather than assessing quality and dynamics independently. When filtering individual videos based on both quality and dynamics, our dynamics scores are complementary to exisiting video quality scores.

Q2: Compare with dynamics degree in Vbench and motion quality in EvalCrafter

Thank you for this insightful comment. DEVIL is superior to existing T2V evaluation protocols in three key aspects.

1. Dynamics Assessment

• Vbench and EvalCrafter are bulit solely on optical flow, limiting analysis to inter-frame pixel motion dynamics. They fail to capture non-pixelmotion dynamics, such as variations in illumination, style, and color.
• DEVIL employs seven dynamics scores across three temporal scales (inter-frame, inter-segment, and video level) to assess video dynamics comprehensively. It significantly improves correlation with human evaluation, achieving an 84% win ratio for inter-frame dynamics, substantially outperforming optical flow's 73%.

2. Evaluation of Models' Capabilities on Dynamics

• Vbench and EvalCrafter created Dynamic Degree and Flow Score metrics based on optical flow, both favoring videos with greater dynamics. However, they neglect the importance of matching video dynamics to prompt requirements, which can range from low to high.
• EvalCrafter's Motion AC-Score assesses a model's ability to generate videos with low/high dynamics according to prompts. However, as a binary metric, it only roughly measures dynamic controllability and doesn't capture the full range of a model's dynamic capabilities.
• We propose the Dynamics Range, $M_{range}$, to drive generating videos with various dynamics, and Dynamics Alignment, $M_{align}$ to evaluate how well models can control dynamics based on text prompts. Our approach enables a more reasonable and precise assessment of a model's dynamic capabilities.

3. Quality Metrics

• Vbench and EvalCrafter assess video dynamics and quality independently, neglecting their correlation. As models tend to generate low-dynamics videos, quality metrics are skewed towards these, failing to accurately represent model performance across the full range of dynamics.
• EvalCrafter's Action Recognition assesses human actions using recognition scores, but it's limited to human activities and fails to evaluate model quality across various dynamics ranges.
• We combine dynamics with existing quality metrics to quantitatively assess models' ability to generate high-quality videos across dynamic ranges, enabling more comprehensive evaluation.

Q3: Influence of Frame Rate

Thanks for the constructive comment. In the paper, we DO standardize the frame rate of each video to 8 FPS, following Vbench. Experiments show our dynamics evaluation maintains high correlation (>0.9) with human ratings across various frame rates. The table below shows Pearson correlation between our dynamics scores and human ratings:

Q4: Details of human alignment

We used a 75/25 split of the annotated data, 75%(3600 videos) for fitting the human alignment dynamic scores, 25%(1200 videos) reserved as a test set. Results in Table 3 are computed exclusively from the 25% test set.

Q5: Effectivenss of the improved metrics

Improved Metrics aims to achieve a more comprehensive and accurate evaluation of model performance by employing grouping assessment based on video dynamics. The effectiveness of Improved Metrics can be demonstrated from two perspectives:

1. Representativeness

Improved Metrics can better represent models' capability to generate high-quality videos with varying levels of dynamics, as measured by the root mean squared bias. As the table below demonstrates, our Improved Metrics achieve a lower root mean squared bias than the Original Metrics.

2. Human Correlation

Our metrics are defined for the entire model rather than individual videos. Annotators grouped videos by their dynamics and assessed their quality, focusing on naturalness due to time limitations. We then averaged the scores for each dynamic group to obtain a comprehensive human score for each model. The table below shows the human correlation evaluation of the original metrics and the improved metrics.

Our improved metric achieved a Pearson correlation of 0.70 with human scores, confirming its effectiveness. The original metric, which averages video quality without considering dynamics, is dominated by low-dynamic videos. This oversight leads to a negative correlation with human scores, despite accurate individual video ratings.

Q6: Naturalness

Naturalness is inspired by the observation that dynamics also accompany unnatural scenarios, e.g., car wheels spinning rapidly while the vehicle remains stationary. We thus propose naturalness to reflect how much the generated videos are like camera-captured ones. Note that it achieved a correlation of 79% with human ratings.

Hi Reviewer 7oNg,

The discussion period is ending soon. Please take a moment to review the author's rebuttal and share any remaining concerns or questions.

Thank you, AC

Before the end of the rebuttal, we look forward to your valuable feedback and further suggestions. We will take full efforts to address your concerns. Looking forward to your reply. Thank you very much.