Reviewer 4

R4-W1. Find a good way to combine their measurements into a single number (e.g., F1-score):
Please see our GC2 in general comments.
 

R4-W2 & Q2. Compare with Video Inception Score (VIS) (Table 1, 2, 5, Figure 20 and 21):
We have added VIS for all the experiments based on the UCF-101 dataset. Additionally, we have compared STREAM with VIS under various temporal distortion scenarios in Figure 21 (Appendix A.15). The experimental result shows that VIS cannot discern different levels of temporal distortions such as shuffling video frames within the video or replacement of video with static scenes (which are identical settings to Figure 3 and 4).
 

R4-W3. Clearly state that there are 3 new metrics (STREAM-T, F, and D): STREAM-S stands for STREAM-Spatial quality, composed of STREAM-F and STREAM-D. As suggested by the reviewer, we will make it clearer that we are introducing 3 new metrics (STREAM-T, F, and D).
 

R4-Q1. Spearman’s correlation between STREAM-T and FVD is fairly high, which is completely contrary to the main text:
We appreciate the reviewer’s inquiry about the correlation between FVD and STREAM measures. It is important to clarify that our main argument was not on asserting that FVD is ineffective at detecting temporal details, but rather on FVD has a relative bias towards spatial quality over temporal quality. This distinction is crucial. While Figures 3, 4, and 6 do show a fair correlation between STREAM and FVD, this correlation primarily reflects the fact that both metrics are responsive to changes in video quality, albeit with different emphasis. STREAM is designed to more finely discern both spatial and temporal aspects, whereas FVD, despite its ability to detect temporal changes, tends to be more influenced by spatial quality. Therefore, the observed correlation does not necessarily contradict our main argument but rather highlights the nuanced ways in which these metrics assess video quality.

In fact, it is natural to have a fairly high correlation between STREAM and FVD as both scores have a linearly increasing or decreasing trend as the intensity of temporal degradation increases. However, in this context, as FVD computes a single value for spatio-temporal quality, it remains unclear whether FVD genuinely responds to temporal distortion or exhibits decreasing trend in response to spatial aspects.
 

R4-Q3. How do bin sizes affect the outcome (Figure 17 and 18):
STREAM-T exhibits robustness across various types of visual noise regardless of bin size. We have added an ablation study on histogram bin size in Figure 17 and 18 (in Appendix A.11). Furthermore, in the presence of various types of temporal distortions, STREAM-T demonstrates consistent evaluations, except when the bin size becomes extremely small. We have selected the bin size ($= 50$) that exhibits the most optimal performance.
 

R4-Q4. Provide source code for implementing STREAM:
We have already provided the link of our code at the end of Abstract. Additionally, we are planning to add an example page that demonstrates how the current implementation can be easily and simply executed with just a few lines of code.
 

*Minor suggestions)
(1) “Let real and fake video datasets as X and Y=> be denoted as”:
We have revised this part as ”Let real and fake video datasets be denoted as X and Y, respectively”.

(2) “SwAV is an algorithm, not a network”:
As advised by the reviewer, we have modified this part to “we utilize a proficient image embedding network proposed by Caron et al. (2020)”.

(3) “P&R is first proposed by Sajjadi et al, (NeurIPS 2018)”:
We have clarified this part by explicitly changing it to “improved precision and recall (P&R)”.

(4) “If we define a sphere with centered at sample point => I don’t understand this”:
This part explains the issues encountered when applying the P&R support estimation method straight-forwardly. For better clarity, we explicitly mentioned in the main text as “In the process of P&R support estimation” and provided a more detailed discussion in Appendix A.4.

R3-Q4. For clarity, we will address the reviewer’s question by separately explaining the aspects related to resolution and length.

(1) How is the metric affected by the resolution of videos:
STREAM, along with the majority of existing metrics, relies on comparing generated video samples with corresponding real pairs of the same resolution. Consequently, these metrics are not significantly influenced by variations in resolution. However, metrics like VIS or FVD, which are based on distance or mutual information, do not have an inherent upper bound. While this lack of an upper bound can result in variation in evaluation scale, it is challenging to attribute the observed scale differences as meaningful when comparing evaluation scores among generative models producing images or videos at the same resolution.

(2) How is the metric affected by the length of videos:
STREAM's metric is uniquely designed to be unaffected by the length of videos, unlike VIS and FVD, which rely on video embedders that process a fixed number of frames. STREAM works by aggregating the temporal information from frame-encoded features. It calculates the Fast Fourier Transform (FFT) along the temporal axis using these encoded frame features from an image embedder. This design makes STREAM indifferent to the number of input frames. Furthermore, as the number of processed frames increases, the frequency resolution of STREAM becomes more precise, enhancing the metric's accuracy.

For example, in a 128-frame video with about 5 frames exhibiting spatial or temporal inconsistencies (e.g., visual noise or swapping frames), VIS and FVD (measured for every 16 frames using a sliding window) show significant variations in evaluation values. depending on the sliding window’s stride. Additionally, the average scores calculated in this way is likely to underestimate the actual level of distortions, depending on the stride of the sliding window. In contrast, STREAM, evaluates the entire video signal as a whole, thereby avoiding variations in evaluation values observed in metrics like VIS and FVD. Although we couldn’t report the results in the current version due to insufficient computational resources, we will present these findings in our second version of paper.

Reviewer 3

R3-W1. It is tedious to evaluate a new method with 3 sub-metrics:
Please see our GC2 in general comments.
 

R3-Q1. Experiment when video consists of only one frame repeating throughout the video segment (Figure 4 and 6):
Please see our GC1 in general comments.
 

R3-Q2. Check STREAM metric with gaussian blur (Figure 19):
Thanks for pointing this out. Similar to the usefulness of FVD in practice, We have observed STREAM-S shows significant sensitivity toward Gaussian blur phenomena. For detailed experimental results, please refer to our Figure 19 (in Appendix A.12).
 

R3-Q3. We have conducted the experiments mentioned by the reviewer in Table 5 (in Appendix A.14). Thank you for your comments. Your feedback greatly improved the understanding of our work. We will divide the reviewer’s question into three parts for clarity:

(1) Evaluation on temporally reversed videos:
First, it is indeed a valid point that STREAM-T does not assign low values when the frames of a video are completely reversed. Given the nature of our metric, which focuses on evaluating continuity and consistency through frequency analysis, STREAM-T remains agnostic to the direction of time. However, this unique characteristic allows STREAM-T to be sensitive not only to situations where only a portion of the video sequence is reversed but also to various types of temporal inconsistencies.

However, it is important to note that the generation of a completely reversed video is an uncommon scenario, and in such cases all the metrics (STREAM-T, VIS, and FVD) may not provide optimal responses. Still, other than this rare case, STREAM demonstrates its effectiveness in detecting partially reversed segments within videos that are otherwise temporally natural (Table 6 in Appendix A.14). For example, when the video is entirely reversed on the temporal axis, FVD shows a modest increase from 14.38 to 24.93. This change, while noticeable, is relatively small compared to significant fluctuations (such as those in the hundreds) observed in other cases (Figure 5 and 6 in Section 3.2.1). Thus, in practice, this minor variation would make it difficult to conclusively determine if FVD is significantly impacted by this aspect alone.

Lastly, typical video data, including datasets like CATER, UCF-101, Kinetics, Taichi, and BAIR Robot Pushing, often features repetitive scenes, such as rotations, repetitive exercises, and cooking. In such practical scenarios, STREAM-T is effective in evaluating inconsistencies in continuity and consistency of temporal movements. This aspect, acknowledging the limitations and strengths of STREAM-T in handling various temporal dynamics, will be incorporated into the limitations section.

(2) Evaluation on spatially flipped videos:
Spatially flipped videos, being the mirror image of each video frame, possess highly realistic spatial and temporal quality. Through the experiment, we have observed that STREAM and VIS remains unchanged, while FVD slightly increased from 14.38 to 36.07.

(3) Evaluation on temporally reversed and spatially flipped videos:
This experiment simply applied the above two cases simultaneously, and the metrics should provide scores indicating spatially and temporally realistic results. From the results, VIS and STREAM consistently exhibited scores identical to the reference score, whereas FVD showed an increased value from the reference score (14.38) to 40.01.

Reviewer 2

R2-W1. Performance of STREAM when most of the generated samples are still:
Please see our GC1 in general comments.
 

R2-W2. Reason for using amplitude at frequency 0 when computing STREAM-S:
In STREAM-S, we use the amplitude at frequency 0 (i.e., mean amplitude) calculations for two main reasons. First, the mean amplitude encapsulates the average signal information of a video, offering a comprehensive overview of its spatial characteristics. Second, including amplitudes across all frequencies renders P&R calculation inaccurate.

More specifically, P&R is designed to approximate the support of real and fake distributions, typically using the union of spheres centered on data points with radii equal to the distance to the $k$-nearest neighbor. These spheres are expected to cover at least $k$ data points (or image features) and collectively represent the population data support. However, videos, unlike images, are composed of multiple image frames. For instance, a 16-frame video yields 16 image features obtained by an embedding network. Applying FFT along the temporal axis, we get 8 amplitude features for each video. In this context, using P&R directly on videos might result in inaccurate data support estimation, as a single sphere may not even encompass all data points from one video. Naturally, in such a scenario, the approximated data support may fail to represent the full scope of video data. To address this, we use the mean amplitude, which represents the average feature of a video, as a single data point. This approach ensures a more accurate and representative approximation of real and fake video data support, enhancing the precision of STREAM-S measurements without limiting it to the number of video frames as shown in our experiments.
 

R2-W3. Compare FVD with STREAM in human evaluation (Appendix A.6):
Please see Section 3.2.2 and Appendix A.6. We have already compared FVD with STREAM in our human evaluation experiment. For the reviewer’s convenience, summarizing here, we measured Spearman’s correlation between human-evaluation and metrics (STREAM and FVD) in terms of video realism and temporal naturalness. The results show that STREAM has a higher correlation with both aspects evaluated by humans than FVD (i.e.,corr(realism, STREAM) = 0.9, corr(realism, FVD) = 0.7, corr(temporal naturalness, STREAM) = 0.6, corr(temporal naturalness, FVD) = 0.5).
 

R2-W4. Difference in the style of real and fake video datasets may have a huge effect on the result (Table 5):
Similar to our response in R2-W2, the objective of video generative models is to mimic the spatial and temporal characteristics of real dataset by estimating the real data distribution. Therefore, when there are “differences in style of real and fake video datasets”, an optimal evaluation metric should be able to discern this difference. Therefore, “when the real videos tend to have fast changes while fake videos change slowly”, STREAM-T should give a small score to reflect this. As suggested by the reviewer, we have compared the real videos with fast FPS and generated videos with slow FPS using STREAM-T (Table 4 in Appendix A.13). In the result, STREAM-T has a decreased value when the generated video has a relatively low FPS compared to the real video, which shows the effectiveness of STREAM-T in capturing the temporal difference.

Reviewer 1

R1-W1. Baseline models are limited to a few older models:
We would like to clarify that the unconditional video generative models compared in Section 3.2.2 and 3.2.3 include various recent open-source video models, such as PVDM (2023), MeBT (2023), TATS (2022), and DIGAN (2022) [1,2,3,4]. Given the inherent nature of evaluation, achieving a fair comparison between video generation models relies on the use of open-source pretrained models. We kindly request your understanding that many of the recent models showing state-of-art performance often do not make these models publicly available. Please see our answer in R1-W2 for the new experiments we have conducted.
  - [1] Yu, Sihyun, et al. "Video probabilistic diffusion models in projected latent space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
  - [2] Yoo, Jaehoon, et al. "Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
  - [3] Yu, Sihyun, et al. "Generating videos with dynamics-aware implicit generative adversarial networks." arXiv preprint arXiv:2202.10571 (2022).
  - [4] Ge, Songwei, et al. "Long video generation with time-agnostic vqgan and time-sensitive transformer." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
 

R1-W2. Test a wider range of datasets and tasks:
We appreciate your suggestion for a more comprehensive evaluation of our metric. In response, we conducted two additional experiments: (1) Comparison of unconditional generative models (Table 6) and (2) Evaluation of video prediction models (Table 8, Figure 26 and 27). Given the constraints of time and computational resources, we chose to concentrate on these two experiments. Nonetheless, we believe that similar trends would emerge if STREAM were applied to T2V tasks.

(1) Comparison of unconditional generative models (Table 7): We have compared unconditional video generative models trained on the Taichi dataset, including state-of-the-art models MeBT [2], TATS [3], DIGAN [4], and LVDM [5]. The results in Table 7 (in Appendix A.16) highlight that, unlike FVD, which aims for model-to-model comparison based on one-dimensional score, STREAM not only reveals the strengths and limitations inherent to each video generative model but also enables multi-dimensional comparison between models.
  - [5] He, Yingqing, et al. "Latent video diffusion models for high-fidelity video generation with arbitrary lengths." arXiv preprint arXiv:2211.13221 (2022).

(2) Evaluation of video prediction models (Table 8, Figure 26 and 27): (Updated)
We are pleased to share our added comparison of models across various generative tasks using STREAM. We focused on “video prediction models”, which are currently open-sourced with publicly available pretrained weights: RaMViD [6] and MCVD [7]. For the reviewer’s convenience, we show the results in Table 8 below.

The FVD score indicates MCVD’s superiority in generating better predictions than RaMViD. STREAM offers a detailed analysis of each model’s performance in terms of fidelity, diversity, and temporal naturalness. While MCVD excels in all aspects, RaMViD lags, particularly in fidelity and diversity.

In our detailed assessment (Appendix A.18, Figures 26 and 27), we observe that RaMViD tends to produce videos with blurry scenes containing artifacts, which are the source of its low fidelity score. Moreover, it shows temporally inconsistent transitions; e.g., objects in the video tend to suddenly appear or disappear over time. On the other hand, MCVD’s outputs are of higher fidelity and demonstrate smoother, more consistent transitions with fewer discrepancies in object appearance during scene transitions.

Overall, these experiments show the versatility and depth of STREAM in assessing video generative models, enhancing our understanding of their capabilities and limitations.

  - [6] Höppe, Tobias, et al. "Diffusion models for video prediction and infilling." arXiv preprint arXiv:2206.07696 (2022).
  - [7] Voleti, Vikram, Alexia Jolicoeur-Martineau, and Chris Pal. "MCVD-masked conditional video diffusion for prediction, generation, and interpolation." Advances in Neural Information Processing Systems 35 (2022): 23371-23385.