OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation

Q3: Weaknesses - Captions generated by LLaVA.

1)In Table 8(Paper), we compared Panda’s short captions with our generated long captions on 1, 117 validation samples, evaluated by 10 volunteers. The results demonstrate that our generated captions provide richer descriptions, particularly in element accuracy and temporal events. While some hallucinations are present, accurate descriptions are predominant.

2)We realized the issue of LLaVA on hallucinations and made several attempts to further improve the captions.

Specifically, we incorporated two additional long video captioners: 1) ShareGPT4Video. A recent model trained on captions generated by GPT-4V; and 2) LLaVA-Video-72B-Improved (Ours). Given the high cost of captioning large volumes of videos with commercial products, we developed an improved chain-of-thought pipeline based on the recent powerful LLaVA-Video-72B model. This approach enhances instance-level detail in the videos and reduces hallucinations by segmenting and analyzing objects individually. Notably, LLaVA-Video-72B achieves comparable performance to GPT-4V and Gemini on the video understanding benchmark [1].

We carefully selected 22K videos from OpenVid-1M to recaption using LLaVA-V1.6-34B (Ours-in-submission), ShareGPT4Video, and LLaVA-Video-72B-Improved (Ours-new). We fine-tuned STDiT-256 model using these three subsets, and the experimental results are shown in the table below.

The table shows that training on the dataset with our fine-grained captions leads to improved performance on alignment. The results will be included in the final version. Moving forward, we will continue to explore video captioning methods and collect higher-resolution, longer-duration video data to further enhance OpenVid-1M.

[1] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[J]. arXiv preprint arXiv:2405.21075, 2024.

Q4: Questions - Filtering ratios.

Please refer to Q1 for more details.

Q5: Questions - Optical flow used in Motion Difference.

1. Please refer to Q1 for filtering process details.

2. We retain clips with moderate flow scores and filter out the highest and lowest like temporal consistency. We will make it clearer in the revised version.

Q6. Questions - Fair comparison across different models

1)Since previous methods support varying resolutions, performance comparisons cannot be entirely consistent. However, in our paper, we presented results for our model at 512 and 1024 resolutions, which use significantly less training data compared to most previous methods (with similar resolutions) while achieving superior performance.

2)To address your concerns regarding super-resolution, we selected 1080P videos from Panda-50M to create Panda-50M-HD for a fair comparison (as WebVid-10M does not include high-resolution videos). The table below compares the performance of models trained on OpenVid-1M and Panda-50M-HD.

From the table, we observe that the model trained on our OpenVid-1M demonstrates superior performance, while the model trained on Panda-50M-HD performs poorly. This discrepancy may be attributed to the low-quality videos in Panda-50M-HD (e.g., low aesthetics and clarity, nearly static scenes, and frequent flickering), a problem our data processing pipeline effectively avoids.

Q7. Questions - Is the dataset publicly available?

We will fully open-source our codes, models, the OpenVid-1M dataset and the complete data processing workflow to facilitate the community.

Thank you for recognizing our work and providing valuable and detailed suggestions. We sincerely appreciate your recognition of our dataset's impact, the significance of our efforts, and the clarity of our motivation and writing! We have carefully considered your constructive and insightful comments, and here are our responses to your concerns.

Q1: Weaknesses - Lacking explanatory work.

(1) Filtering models:

Aesthetics and Clarity Assessment: We adopted the LAION Aesthetic Predictor and DOVER to separately assess aesthetic and clarity scores due to their fast inference speeds and alignment with human preferences. These qualities make them efficient and well-suited for integration into our pipeline for processing million-level video data.

Temporal Consistency: Extracting CLIP representations has proven effective for computing cosine similarity between images. We calculate the CLIP similarity between every two adjacent frames in the video and take the average as an indicator of the temporal consistency, measuring the coherence and consistency of the video frames.

Motion Difference: To measure motion amplitude, we utilize UniMatch, a pretrained state-of-the-art optical flow estimation method that is both efficient and accurate. We calculate the flow score between adjacent frames of the video, taking the squared average of the predicted values to represent the motion dynamics, where higher values indicate stronger motion effects.

Clip Extraction: Our observations reveal that fade-ins and fade-outs between consecutive scenes often go undetected when using a single cut detector with a fixed threshold. To address this, we employ a cascade of three cut detectors, each operating at different thresholds. This approach effectively captures sudden changes, fade-ins, and fade-outs in videos.

(2) Filtering ratios

We randomly sampled a subset from the collected raw data and processed it through our data processing pipeline. A panel of evaluators was then tasked with assessing these video subsets, determining whether the videos at each processing step met our requirements. Based on their preferences, we derived score thresholds and filtering ratios for each step after multiple evaluations. Figure 2 in supplementary material provides visualizations of the videos with varying clarity, aesthetic, motion, and temporal consistency scores computed by our pipeline.

Q2: Weaknesses - Models trained on dataset do not learn new knowledge.

Previous datasets contain a significant amount of noisy data, which severely impacts the performance of models trained on them. As demonstrated in Table 7(Paper) and the additional Panda-50M-HD experiments in Q6-2), this issue is evident. Therefore, constructing the high-quality OpenVid-1M dataset is both meaningful and essential to prevent models from learning useless or harmful knowledge. Furthermore, our dataset is also valuable in other research tasks, such as video super-resolution or frame interpolation, which currently lack large-scale, high-quality video datasets like OpenVid-1M.

Thank you for recognizing our work and providing valuable suggestions. We will fully open-source our codes, models, the OpenVid-1M dataset and the complete data processing workflow to support the community. We have carefully considered your constructive and insightful comments and here are our responses to your concerns.

Q1. Differences between MVDiT and MMDiT.

Thank you for your practical suggestion. Below, we outline the differences between our MVDiT and MMDiT:

1) Multi-Modal Self-Attention: We design a Multi-Modal Self-Attention (MMSA) module based on the self-attention module of MMDiT. To handle video data, we repeat the text tokens T times and then concatenate the text tokens with video frame tokens using the same method as MMDiT. This provides a simple yet effective adaptation of MMDiT to video data input.

2) Multi-Modal Temporal-Attention: Since MMDiT lacks the ability to generate videos, we introduce a novel Multi-Modal Temporal-Attention (MMTA) module on top of the MMSA module to efficiently capture temporal information in video data. To retain the advantages of the dual-branch structure in MMSA, we employ a similar approach in MMTA, incorporating a temporal attention layer to facilitate communication along the temporal dimension.

3) Multi-Head Cross-Attention: Since the absence of semantic information may impair video generation performance, explicitly embedding semantic information from text tokens into visual tokens is helpful. To address this, we introduce a novel Cross-Attention Layer to enable direct communication between text and visual tokens.

Additionally, to validate the effectiveness of our proposed MMTA and MMCA, we conducted ablation studies on MVDiT-256. The results are presented in the table below.

From the table, we can find that MMCA boosts video quality and alignment. Please note that after removing MMTA, the model is unable to generate videos and instead produces multiple unrelated images, completely failing to meet the requirements for video generation. We will include these results and analyses in the revised version.

Last but not least, our architecture design of MVDiT received positive feedback from Reviewer zA3J.

We will include these results and analyses in the revised version.

Q2. Demo website.

We are in the process of developing a demo webpage, which will be made publicly accessible upon completion. To address your concerns, we have included additional visualizations of our OpenVid-1M dataset and generated video results in Figure 1 and Figure 3 of the supplementary material. These results will also be incorporated into the appendix of the revised version.

Q3. Video duration distribution comparison.

Thank you for your valuable feedback. Per your request, we compared the video duration distribution using annular charts, with the results presented in Figure 4 of the supplementary material. We will include this comparison in the revised version.

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are our responses to your concerns.

Q1. Novelty of data processing pipeline.

The SD3 pipeline focuses on collecting image training data, whereas our pipeline incorporates steps such as Temporal Consistency, Motion Difference, and Clip Extraction, specifically designed to address challenges unique to video data. Therefore, our approach is fundamentally different from SD3.

The differences between our pipeline and SVD's have already been discussed in Section 4 of the paper. However, we would like to reiterate and clarify them here:

1) Visual quality evaluation: We integrate DOVER for assessing video clarity, which provides better texture and clarity analysis compared to SVD's CLIP aesthetic scoring.

2) Motion evaluation: We adopt UniMatch to replace traditional optical flow methods, such as Farneback and RAFT, enabling more accurate processing of both static and fast-moving videos while maintaining computationally efficient.

3) Time consistency evaluation: Beyond clip extraction to avoid abrupt changes, we additionally incorporate flicker removal to improve temporal consistency.

4) Processing efficiency: Instead of extracting clips from a large video pool as done in SVD, we pre-filter high-quality videos, significantly reducing redundant processing.

5) Open-sourcing: In contrast to SVD's unreleased data processing code and dataset, we will release our codes, pretrained models, the OpenVid-1M dataset and the complete data processing workflow to benefit the community.

Q2. Ablation study of MVDiT.

Per your request, the table below presents ablation studies validating the effectiveness of the scaling parameter α, Multi-Modal Temporal-Attention (MMTA) and Multi-Modal Cross-Attention (MMCA) layers on MVDiT-256.

From the table, we can draw the following conclusions: MMCA boosts video quality and alignment, and parameter α improves video quality and convergence. Notably, we observed that removing α causes the loss to decrease very slowly, indicating that α accelerates training, consistent with the findings reported in [1]. Please note that after removing MMTA, the model is unable to generate videos and instead produces multiple unrelated images, completely failing to meet the requirements for video generation. These results and analyses will be included in the final version.

[1] Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.

Q3. Clearer qualitative evaluation.

Thank you for your valuable suggestion. In the revised version, we will enlarge the video frames in Figures 6 and 8 and provide higher-resolution versions in the appendix to improve clarity.

Q4. The section on Acceleration.

Thank you for your valuable feedback. In the revised version, we will move this section to the appendix and replace it with more qualitative text-to-video results.

Questions:

Q3. SOTA list update.

Following your advice, we compared with the SOTA CogvideoX-5B and Open-sora-plan v1.2 in the table below:

The table shows that our model achieves superior aesthetic performance and comparable clarity to Open-sora-plan v1.2 and CogVideoX-5B while using less training data and parameters, demonstrating the strength of our dataset in ensuring high quality. These results will be included in the final version.

Q4. Fair comparison with Panda-50M

Thank you for the valuable suggestions. In the table below, we select 1080P videos from Panda-50M to create Panda-50M-HD and compare the model performance between OpenVid-1M and Panda-50M-HD.

From the table, we observe that the model trained on our OpenVid-1M demonstrates superior performance, while the model trained on Panda-50M-HD performs poorly. This discrepancy may be attributed to the low-quality videos in Panda-50M-HD (e.g., low aesthetics and clarity, nearly static scenes, and frequent flickering), a problem our data processing pipeline effectively avoids.

Questions:

Q1: The value of short clips.

We employed a Clip Extraction strategy to ensure scene consistency within each clip, resulting in the inclusion of clips shorter than 5 seconds in our dataset.

These short-duration videos hold value in at least two areas:

1. Early-stage training in T2V Generation: Short-duration videos are particularly useful during the early stages of T2V training. Many T2V models, such as Open-sora-plan v1.2 [1] and pyramidal flow matching [2], utilize short clips in the pre-training phase to reduce complexity, gradually increasing video duration and resolution in later stages. Thus, the short clips in our dataset provide a good starting point for dynamic or multi-stage training in video generation.

2. Applications in video restoration: Short-duration videos are also valuable for other video generation tasks, such as video restoration. For example, video super-resolution methods [3, 4] target sequences of 8 and 6 consecutive frames respectively, while video frame interpolation techniques [5, 6] focus on 32 and 14 consecutive frames respectively.

[1] PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, apr 2024. URL https://doi.org/10. 5281/zenodo.10948109.

[2] Jin Y, Sun Z, Li N, et al. Pyramidal flow matching for efficient video generative modeling[J]. arXiv preprint arXiv:2410.05954, 2024.

[3] Zhou S, Yang P, Wang J, et al. Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 2535-2545.

[4] Chen Z, Long F, Qiu Z, et al. Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9232-9241.

[5] Jain S, Watson D, Tabellion E, et al. Video interpolation with diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 7341-7351.

[6] Wang W, Wang Q, Zheng K, et al. Framer: Interactive Frame Interpolation[J]. arXiv preprint arXiv:2410.18978, 2024.

Q2: Self-Attention Module.

As you anticipated, the self-attention operation is conducted along spatial and within the same frame. We will clarify this further in the revised version.

Training a 3D full-attention architecture, as used in Open-sora-plan v1.2, requires immense GPU memory, computational resources, and time (generating a single video on an A100 GPU takes 1.5 hours), as highlighted in its report: "the substantial computational cost made it unsustainable".

In contrast, our OpenVid-1M dataset is designed for high-resolution video generation. Employing 3D full attention in this context would lead to overwhelming GPU memory and computational demands, rendering model training nearly impossible. However, as noted in the Open-sora-plan v1.2 report, 3D full attention excels at capturing joint spatial-temporal features. Exploring methods to extend 3D full attention to high-resolution video generation is an important direction and will be a focus of our future research.

Thank you for recognizing our work and providing valuable and detailed suggestions. We are pleased that you recognized our dataset to be critical, our architecture to be reasonable, and our generation results to be superior! We will fully open-source our codes, models, the OpenVid-1M dataset and the complete data processing workflow to support the community.

Weaknesses:

Q1: The proposed OpenVid-1M is somewhat weak.

Thanks for your suggestions. We realized this issue and made several attempts, especially on improving the captions and enriching the comparison among different captioners in addition to the user study between long captions (Ours) and short ones (Panda) shown in Table 8(Paper).

As you suggested, we incorporated two additional long video captioners: 1) ShareGPT4Video. A recent model trained on captions generated by GPT-4V; and 2) LLaVA-Video-72B-Improved (Ours). Given the high cost of captioning large volumes of videos with commercial products, we developed an improved chain-of-thought pipeline based on the recent powerful LLaVA-Video-72B model. This approach enhances instance-level detail in the videos and reduces hallucinations by segmenting and analyzing objects individually. Notably, LLaVA-Video-72B achieves comparable performance to GPT-4V and Gemini on the video understanding benchmark [1].

Specifically, we carefully selected 22K videos from OpenVid-1M to recaption using LLaVA-V1.6-34B (Ours-in-submission), ShareGPT4Video, and LLaVA-Video-72B-Improved (Ours-new). We fine-tuned STDiT-256 model using these three subsets, and the experimental results are shown in the table below.

The table shows that training on the dataset with our fine-grained captions leads to improved performance on alignment. The results will be included in the final version. Moving forward, we will continue to explore video captioning methods and collect higher-resolution, longer-duration video data to further enhance OpenVid-1M.

[1] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[J]. arXiv preprint arXiv:2405.21075, 2024.

Q2: Ablation study on MVDIT.

The table below presents ablation studies validating the effectiveness of the Multi-Modal Temporal-Attention (MMTA) and Multi-Modal Cross-Attention (MMCA) layers on MVDiT-256.

From the table, we can find that MMCA boosts video quality and alignment. Please note that after removing MMTA, the model is unable to generate videos and instead produces multiple unrelated images, completely failing to meet the requirements for video generation.

These results and analyses will be included in the final version.

Thanks for the detailed response. It is nice that the authors have tried their best to address most of my technique concern. On the other hand, there are some intrinsic drawbacks , including the videos are not first-hand data and the novelty of the model architecture designing, which can not be addressed in this work. Hence, I keep my rating as 6, and look forward to more solid sequel in the future (if this work is accepted).