Beyond FVD: An Enhanced Evaluation Metrics for Video Generation Distribution Quality

Dear Reviewer,

Thank you for your thorough review and insightful feedback.

1. Regarding long-term video evaluation, we are currently conducting experiments in this area. The results of this analysis will be incorporated into the next revised version of our manuscript.

2. Regarding the comment on human evaluation subjects, the participants include a diverse group of individuals, such as extended family members, friends, and fellow researchers. Since most media content is consumed by the average consumer, involving a randomly selected group of individuals is appropriate. Notably, many previous studies evaluating generative models have similarly relied on untrained participants for evaluating the generated content.

   Additionally, we conducted a new study on the T2VQA-DB[1], a database containing recent Text-to-Video model generations along with human mean opinion scores (MOS). We use these MOS to evaluate the performance of FVD and JEDi. Both FVD and JEDi's results align with the MOS scores, with JEDi demonstrating significantly greater sample efficiency compared to FVD on this database (Section 5.5, Pages 9-10).

3. Regarding the comment on the degree of blur distortion introduced in our experiments, we have included the specific blur parameters in our revised manuscript.

   Regarding the suggestion to use metrics like NIQE and Q-Align for assessment, we note that these metrics are designed for image assessment. However, our blur experiment introduces temporal randomness by sampling blur levels along the temporal dimension. Specifically, the sigma parameter controls the degree of blur applied to each frame, and we define a range for sigma from which the blur level for each frame is randomly sampled. The size of this sigma range correlates with the blur intensity: smaller ranges correspond to low temporal blur, while larger ranges correspond to high temporal blur. This approach introduces significant temporal inconsistency in the video, with abrupt transitions between clear and blurred frames. Since image-based metrics cannot capture these temporal inconsistencies, we utilize video-based networks (such as I3D, VideoMAE, and V-JEPA) to extract features and quantify these temporal distortions. Details of these experimental settings are revised in Table 1 on page 7. Additionally, it's important to note that the FVD backbone model, an I3D network trained for video action classification, primarily captures spatial features and may therefore struggle to detect temporal distortions.

Additionally, in Appendix G, we present a new study analyzing samples generated by state-of-the-art video generation models within the I3D and V-JEPA feature spaces, using samples obtained from [2]. The findings from this experiment demonstrate that the V-JEPA feature space employed in JEDi provides a more efficient framework for evaluating video distribution.

[1] Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., & Liu, N. (2024). Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment. ArXiv, abs/2403.11956.

[2] Wang, W., & Yang, Y. (2024). TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generation.

Dear Reviewer,

We have incorporated the long-term video study into our manuscript (Section I, page 34). The study concludes that the JEDi metric is sensitive only to the amount of distortion applied to the clips, maintaining consistent evaluations regardless of clip length. In contrast, FVD is influenced by clip length, with evaluated distributional distances increasing as the clip lengthens under the same level of distortion.

Additionally, we have included demographic information about the raters for our human study (Section F.3, Pg 29).

We hope these responses have addressed all of your concerns. We are also happy to discuss any remaining issues that may still be unclear. If we have clarified any confusion, we kindly ask that you consider revising your scores for our work.

Dear Reviewer,

Thank you for your thorough review and insightful feedback.

JEDI and FVD are generative distribution assessment metrics that rely on reference datasets for evaluations (cannot be used for no-reference quality assessment). We appreciate your recommendation to evaluate our metric using AI-generated conten, and we conducted experiments on the T2VQA database, comparing the JEDi and FVD distances.

Firstly, the T2VQA database has the following characteristics:

• Each video is associated with a single Mean Opinion Score (MOS) aggregating ratings for text-video alignment and video fidelity, obtained through a 0-100 slider (Section 3.2 of [1]).
• Human-evaluated scores exhibit significant variance, clustering around 50 with a standard deviation of 16.6 (Figure 3 and Section 3.2 of [1]).
• The database contains generations from 10 different AI models. Many models yield similar MOS scores, resulting in an ambiguous ranking of the models if used directly (Figure 3 of [1]).
• T2VQA-DB lacks a "ground-truth" video set. To address this, we identified corresponding videos from WebVid10M using the same prompts and utilized them as our ground-truth distribution for evaluation.

We selected generations from three models (#2, 7, and 9) that exhibit a clear gap in video quality, with average rankings of 10 (worst), 5, and 1 (best), respectively, based on MOS scores. Both FVD and JEDi scores ranked these models consistently, with model #2 as the worst and model #9 as the best. Our evaluation results are

Importantly, JEDi achieved convergence with significantly fewer samples, compared to FVD: the number of samples to converge for JEDi are 100, 200 and 300; the number of samples to converge for FVD are 600, 800 and 900.

We initially planned to evaluate JEDi and FVD on the LGVQ and GAIA datasets. However, the LGVQ dataset is not yet available, and GAIA lacks a ground-truth set, with its scores also reflecting action-tag alignment. We are currently exploring alternative AIGC datasets that offer decoupled visual quality scores from human raters alongside a ground-truth set. We will keep you updated on our progress.

Regarding your comment on NIQE evaluator: We acknowledge NIQE's effectiveness in image quality assessment, but highlight its limitations for video data. Specifically:

• Image evaluators, like NIQE, overlook temporal consistency crucial for video evaluation.
• NIQE assesses single-image quality, whereas our work focuses on evaluating generation distribution quality, which examines a model's generalization capability.
• NIQE's reference-free approach differs from our metric, which measures distance from the ground-truth distribution.

Addressing your concern about the monotonicity assumption in Figure 5, we provide visualizations of generation results at various training checkpoints in Figure 22 – the video version of which is uploaded here: link. As shown, the generated videos are visibly higher quality as the number of iterations increases.

[1] Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., & Liu, N. (2024). Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment. ArXiv, abs/2403.11956.

Dear Reviewer,

Thank you for your feedback regarding the references for FVD and JEDi metrics. We would like to clarify that these metrics indeed require reference distributions.

To ensure clarity, we would like to define the following terminologies:

• Generic video distribution (also referred to as a pre-defined or off-the-shelf prior video distribution in your reviews): A video distribution that serves as a broad summary of all types of videos, encompassing samples from the entire real video space.
• Target distribution (also known as the ground-truth distribution): The desired distribution that we aim for our generative model to replicate (e.g. to evaluate a generative model's driving scene generation capability, the target distribution should contain exclusively driving videos and should not be a generic video distribution.)
• Distributional metrics: Metrics that measure the distance between the generated distribution and the target distribution.

We would like to address two key points:

1. FVD is a specific type of metric within the VQA family, designed to measure distributional distances between two sets of features using statistical tools (i.e. Fréchet distance and optimal transport) tailored for distribution-to-distribution comparisons.
2. For metrics like FVD, accurate evaluations rely on precisely defining the target distribution. Using a generic video dataset as a pre-defined target distribution can be inappropriate, particularly when dealing with specialized domains. For instance, autonomous vehicle (AV) videos occupy a unique subspace within the broader video space, with distinct mean and variance characteristics. Evaluating an AV video generation model against a generic video distribution would yield inaccurate results (Figure 2 on Pg 4 visualizes the distribution of 11 video sets within the PCA-reduced feature space, revealing distinct clustering patterns for each video dataset).

To summarize, the proposed metric JEDi, similar to FVD, evaluates the distribution-to-distribution distance in feature space. This distributional property is crucial since non-distributional metrics solely focus on sample quality and overlook distribution bias. For example, let there be a generative model that is trained on a variety of driving videos but is biased towards generating one type of video (e.g., cars always turning left but never right): while non-distributional metrics would give these generations a high overall score if the video qualities are good, a distributional metric can detect the bias in the generated video distribution since it doesn’t match with the target distribution (This concept is also discussed in our manuscript on lines 53–57, page 1-2, introduction section).

We hope our explanations have provided clarity on the concept of distributional metrics. If there are any remaining questions or concerns, we are more than happy to continue the discussion to ensure everything is fully addressed.

Thanks for the authors' feedback. My concerns are all well addressed. I have updated my score to 8.

Dear Reviewer,

Thank you for your valuable feedback. Since our work centers on evaluating the quality of video distributions, we greatly appreciate your and Reviewer goxo's suggestions to discuss our findings' implications for assessing individual videos.

Thus, we have provided a comprehensive discussion in Section J.2 (Pages 34-35), exploring the connections between our research and blind single video quality assessment. Specifically, we delve into the relationships with existing methods and related topics, including NIQE [2], Natural Video Statistics (NVS) [1], and Mahalanobis distance. We propose that the insights derived from the V-JEPA feature space have the potential to advance single-video quality metrics. Furthermore, we outline several preliminary experiments, such as utilizing Mahalanobis Distance to measure sample-distribution distances, which could lay the groundwork for future research.

[1] M. A. Saad and A. C. Bovik, "Blind quality assessment of videos using a model of natural scene statistics and motion coherency," 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), Pacific Grove, CA, USA, 2012, pp. 332-336, doi: 10.1109/ACSSC.2012.6489018.

[2] A. Mittal, R. Soundararajan and A. C. Bovik, "Making a “Completely Blind” Image Quality Analyzer," in IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209-212, March 2013, doi: 10.1109/LSP.2012.2227726.

We welcome any additional feedback you may have and hope that our revisions have adequately addressed your concerns. If we have clarified any confusion, we kindly ask that you consider revising your scores for our work.

Dear Reviewer,

Thank you for your valuable feedback. We will revise our writing based on your suggestions.

For Points 1 & 2: While many studies focus on single-video generation quality, our work centers on a different aspect: assessing generation distribution quality. To our knowledge, the only other work addressing this is FVD. Evaluating video distributions requires mapping the videos into feature spaces. For example, the I3D network used in FVD was trained on a human-action classification task; its features are biased toward spatial content (for recognizing actions) but are less suitable for evaluating temporal consistency. In contrast, VideoMAE and V-JEPA features, pretrained on self-supervised tasks, may be more appropriate for assessing overall video quality. Therefore, we believe defining the feature extractors and their feature space characteristics is important to facilitate understanding of distribution evaluation throughout the paper.

For Point 3: We will adjust the scaling of the figures to improve readability.

For Point 4: In the Table 1 experiment, the testing video dataset is subjected to noise distortions, including low blur (σ ∼ [0.05, 0.75]), medium blur (σ ∼ [0.1, 1.5]), and high blur (σ ∼ [0.01, 3]), where σ represents the per-frame blur intensity. A larger range indicates greater temporal inconsistency. We will update this information in the table caption.

For Points 5 & 6: We will add new experimental results using the T2VQA database, which contains 10K generated videos from 10 AI models along with human evaluation scores. Key details of the database include:

• Each video is associated with a single Mean Opinion Score (MOS) aggregating ratings for text-video alignment and video fidelity, obtained through a 0-100 slider (Section 3.2 of [1]).
• Human-evaluated scores exhibit significant variance, clustering around 50 with a standard deviation of 16.6 (Figure 3 and Section 3.2 of [1]).
• The database contains generations from 10 different AI models. Many models yield similar MOS scores, resulting in an ambiguous ranking of the models if used directly (Figure 3 of [1]).
• T2VQA-DB lacks a "ground-truth" video set. To address this, we identified corresponding videos from WebVid10M using the same prompts and utilized them as our ground-truth distribution for evaluation.

We selected generations from three models (#2, 7, and 9) that exhibit a clear gap in video quality, with average rankings of 10 (worst), 5, and 1 (best), respectively, based on MOS scores. Both FVD and JEDi scores ranked these models consistently, with model #2 as the worst and model #9 as the best. Our evaluation results are

Importantly, JEDi achieved convergence with significantly fewer samples, compared to FVD: the number of samples to converge for JEDi are 100, 200 and 300; the number of samples to converge for FVD are 600, 800 and 900.

Additionally, we are exploring alternative AIGC datasets that provide decoupled visual quality scores from human raters and include a ground-truth set. We will keep you updated on our progress.

Also, thank you for pointing out the formatting issue in Section E.3. Table 4 was intended to be under Section E.3 but was mistakenly shifted. We will correct this in the updated file.

For Point 7: We have included plots of I3D and VideoMAE's feature sensitivities to various distortions in Figures 17 and 18. Figure 17 demonstrates that I3D features are highly sensitive to spatial distortion (salt & pepper noise), while Figure 18 shows that VideoMAE-PT features perceive medium elastic distortion as worse than high elastic distortion on the UCF dataset. Additionally, we included the correlation between evaluated feature distances under different distortions and human opinions in Table 4.

For Point 8: We will revise the equations in Appendix B to improve readability.

Thank you for your feedback. We appreciate your insights and will incorporate these changes in our updated submission.

[1] Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., & Liu, N. (2024). Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment. ArXiv, abs/2403.11956.

Dear Reviewer,

Thank you again for your valuable feedback. We have revised our manuscript accordingly:

1. We have revised Figure 3 (Pg 6) to improve clarity.
2. In Table 1 (Pg 7), red font highlights an inconsistency in FVD evaluation, where the distribution distance betweentest and train exceeds that between test+low-blur and train. We provide detailed experiment settings in the table caption to clarify any potential confusion.
3. We have incorporated new experiment results using the T2VQA database [2] into Section 5.5 (Pg 9-10), as outlined in our prior response addressing feedback Points 5 & 6.
4. We conducted an additional experiment using the TIP-I2V dataset [1], comprising samples generated by current state-of-the-art video generation models. Analyzing these video samples in the I3D and V-JEPA feature spaces provides valuable insights into our metric's performance. The details of this experiment are provided in Appendix G (Pg 32), with a brief discussion included in Section 5.5 (Pg 9-10) of the main text.
5. We have added a new section (Appendix B (Pg 17-18)) to discuss the computational overheads and distribution metric configurations of our work.

[1] Wang, W., & Yang, Y. (2024). TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generation.

[2] Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., & Liu, N. (2024). Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment. ArXiv, abs/2403.11956.

We hope these responses have addressed all of your concerns. We are also happy to discuss any remaining issues that may still be unclear. If we have clarified your confusion, we kindly ask that you consider revising your scores for our work.

Most of the previous concerns are addressed. Some surveys for image quality assessment as well as video quality assessment are suggested to be given in the paper, for better referring of the related topics. I have raised my score.

Thank you for suggesting incorporating a discussion of prior work's human studies. We have added a new survey section (Appendix F.4, pg 30-32), which provides an overview of the human studies conducted for FVD (video distribution-based metric) [1], CMMD (image distributional-based metric)[2], and T2VQA-DB (AIGC video database) [3].

[1] Unterthiner, T., Steenkiste, S.V., Kurach, K., Marinier, R., Michalski, M., & Gelly, S. (2018). Towards Accurate Generative Models of Video: A New Metric & Challenges. ArXiv, abs/1812.01717.

[2] Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., & Kumar, S. (2023). Rethinking FID: Towards a Better Evaluation Metric for Image Generation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9307-9315.

[3] Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., & Liu, N. (2024). Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment. ArXiv, abs/2403.11956.

Dear Reviewer,

Thank you for your insightful feedback. We appreciate your suggestion and acknowledge that our paper focuses specifically on evaluating video distribution generation quality, which is one aspect of broader video qualities. To clarify this scope, we plan to revise our title and abstract accordingly.

Regarding the potential application of our findings to single video evaluation, this work highlights the advantages of evaluating video quality within the V-JEPA feature space over I3D and VideoMAE spaces. Based on this, we believe that the V-JEPA could serve as an effective extractor for single video quality evaluation by comparing the likelihood of a video against ground-truth video sets. Although this direction was not explored in the current paper, it offers a promising direction for future research.

Thank you for suggesting AIGC data for evaluation. We conducted experiments using the T2VQA database [1] and would like to provide context before presenting our results: The T2VQA database has the following characteristics:

• Each video is associated with a single Mean Opinion Score (MOS) aggregating ratings for text-video alignment and video fidelity, obtained through a 0-100 slider (Section 3.2 of [1]).
• Human-evaluated scores exhibit significant variance, clustering around 50 with a standard deviation of 16.6 (Figure 3 and Section 3.2 of [1]).
• The database contains generations from 10 different AI models. Many models yield similar MOS scores, resulting in an ambiguous ranking of the models if used directly (Figure 3 of [1]).
• T2VQA-DB lacks a "ground-truth" video set. To address this, we identified corresponding videos from WebVid10M using the same prompts and utilized them as our ground-truth distribution for evaluation.

We selected generations from three models (#2, 7, and 9) that exhibit a clear gap in video quality, with average rankings of 10 (worst), 5, and 1 (best), respectively, based on MOS scores. Both FVD and JEDi scores ranked these models consistently, with model #2 as the worst and model #9 as the best. Our evaluation results are

Importantly, JEDi achieved convergence with significantly fewer samples, compared to FVD: the number of samples to converge for JEDi are 100, 200 and 300; the number of samples to converge for FVD are 600, 800 and 900.

We initially planned to evaluate JEDi and FVD on the LGVQ and GAIA datasets. However, the LGVQ dataset is not yet available, and GAIA lacks a ground-truth set, with its scores also reflecting action-tag alignment. We are currently exploring alternative AIGC datasets that offer decoupled visual quality scores from human raters alongside a ground-truth set. We will keep you updated on our progress.

[1] Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., & Liu, N. (2024). Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment. ArXiv, abs/2403.11956.

Dear Reviewer,

Thank you for your valuable feedback. We have revised our manuscript accordingly:

1. Changing the paper title in OpenReview at this stage appears to be unfeasible; however, we have updated our title in the PDF to Beyond FVD: Enhanced Metric for Evaluating Video Generation Distribution Quality.
2. We have discovered a new dataset [1] containing additional AIGC samples. Specifically, TIP-I2V is comprised of samples generated by current state-of-the-art video generation models. Our analysis of these samples in the I3D and V-JEPA feature spaces reveals significant insights:
   • Our findings are based on the principle that features extracted from video sets A and B should exhibit the same distribution if they originate from the same source. This is because a consistent extractor should not introduce artificial distinctions between sets that inherently belong to the same distribution.
   • On the other hand, if video sets A and B come from different distributions, a robust feature extractor designed for evaluating distributional differences would highlight these distinctions. A good extractor would encode the unique characteristics of each set, creating clearly separable distributions in the feature space. This separation is critical for understanding the underlying differences and verifying that the extractor is effectively capturing meaningful distributional variations rather than noise or irrelevant features.
   • Our results in Appendix G demonstrate the superiority of V-JEPA feature space in this regard: we show that JEPA-extracted features reveal clear distributional distinctions between 3 AIGC video sets and 1 real video dataset, whereas I3D features do not exhibit such distinct clustering.
3. In the newly revised pdf, we have added a new section (5.5) focused on AIGC studies. This section presents the T2VQA-DB [2] results and discusses our findings on TIP-I2V [1]. A more detailed report on TIP-I2V is available in Appendix G.
4. We added another study on the effect of long-term video on our metric in Section I, Pg 34.
5. We have added a discussion of the implications of our findings for the single-video quality evaluation metrics in Section J.2 (Pg 34-35).

[1] Wang, W., & Yang, Y. (2024). TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generation.

[2] Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., & Liu, N. (2024). Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment. ArXiv, abs/2403.11956.

We hope these responses have addressed all of your concerns. We are also happy to discuss any remaining issues that may still be unclear. If we have clarified any confusion, we kindly ask that you consider revising your scores for our work.

Dear Reviewer,

Thank you for your feedback and review.

However, the number of samples required for convergence should not be considered the primary concern of an evaluation strategy, as not all datasets face the issue of insufficient samples affecting precision

In response to concerns about convergence issues, we clarify that sample deficiency is multifaceted: the sample convergence problem is not solely due to the number of available samples in different datasets; it also arises from the extremely slow generation of high-quality videos using modern generative video models (Section 4.2, lines 334-339). Most existing studies, for example [1], evaluate performance on just 256 videos on KTH, Cityscapes, and UCF-101 datasets. We have conducted UCF-101 analysis in Figure 3, and it needs more than 4000 videos to converge. Thus, 256 samples is extremely far from convergence. This limited sampling significantly hampers the metric’s ability to achieve convergence.

In addition, as AIGC’s quality improves, their distribution increasingly aligns with the ground-truth distribution. Consequently, the margin of error in evaluating the distances between the two distributions decreases. This reduction necessitates both a larger number of samples and the use of a metric with higher sample efficiency to ensure reliable evaluation.

These points demonstrate that our metric’s superior sample efficiency is a significant contribution.

further experimental validation is needed to assess its performance in other aspects as an evaluation metric

Our paper presents an extensive comprehensive analysis, with some experiments included in the Appendix due to space constraints. To our knowledge, few existing works match this level of thoroughness. Our experimental scope encompasses:

1. Non-Gaussianity assessment of Fréchet Distance across 5 architectures and 11 datasets (Figure 2).
2. Convergence evaluations for 5 metrics (with/without autoencoding) on 2-5 architectures and 11 datasets (Figures 14-15-16).
3. Noise sensitivity analysis for 5 metrics, 5 architectures, 2 datasets, and 10 visual variations (Figure 18).
4. Human preference comparisons across 50 combinations of 5 metrics, 5 architectures, and autoencoding configurations on 2 datasets (Table 4).
5. Temporal metric evaluations during video model fine-tuning for 4 metric-architecture pairs on 2 datasets (Figures 4-5).

Building on our already extensive experimental evaluation, we will further supplement this rebuttal with additional analyses on:

• T2VQA-DB, exploring the applicability of our findings to this benchmark dataset
• Long-term video sequences, examining the temporal dynamics and robustness of our approach These supplementary analyses will provide even deeper insights into the effectiveness and the agreement to human subjects of our method.

Although the author said in the experiment that the resolution can be arbitrary, since this work relies on the representation capability of SreamDiffusion[3] based on SD-Turbo or LCM, many "high-resolution" images are not included in the training set of SD-Turbo or LCM. Can Promptus effectively transmit videos with various resolutions and high resolutions that it has not seen during SD training?

We evaluate videos at the resolution required by the backbone network architectures, so we do not have control over the resolution of the videos. We are not familiar with the works you mentioned, nor how they relate to our metric. Could you please clarify your question?

the paper provides limited explanation of the proposed evaluation metric, JEDi. It might be beneficial to reduce the amount of reiteration about other evaluation metrics and instead increase the logical exposition of JEDi to provide a more comprehensive understanding of its strengths and limitations

JEDi is the result of our extensive analysis of various statistical distribution metrics and backbone architectures across multiple datasets. This analysis is fundamental to justifying the development of JEDi. FVD, as a video distribution quality metric, uses Fréchet distance to measure the distance between video distributions in I3D space. While FVD and other related studies are valuable, it is important to acknowledge the limitations of these metrics and the feature space they operate in, and to improve upon them. We prefer to be transparent by:

1. showing all the methods we compared,
2. highlighting that the best combination was V-JEPASSv2+MMDPOLY (JEDi), and
3. proposing this combination as our metric.

Our extensive experiments (see above) already compare JEDi against a broad range of other approaches, which underscores both its strengths and limitations.

[1] Voleti, V.S., Jolicoeur-Martineau, A., & Pal, C.J. (2022). MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation. ArXiv, abs/2205.09853.

Dear Reviewer,

As the manuscript revision period comes to a close, we have not yet received further feedback from you. We would greatly appreciate the opportunity to engage in more discussions with you during the extended discussion period.

During the manuscript revision period, we have added three new experiments: two studies on AIGC datasets and one focusing on long-term video generation. Additionally, we have expanded our discussion to include details on the demographics of our human study and outlined potential avenues for future research. A summary of our rebuttal updates is available in the global post.

For the discussion period, we would like to seek your feedback on the following points:

• With the addition of three new experiments alongside the five existing ones, do you believe our work still lacks sufficient experimental validation?
• Could you clarify what you meant by your comment regarding “high resolution”, “SreamDiffusion”, “SD-Turbo”, “LCM”, or “Promptus”?
• Do you still feel our discussion on the strengths and limitations of our metric is sufficient now?

We look forward to your insights and appreciate your time and effort in providing feedback.