InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

Q1: Lack of definitions, explanations. Definition of UMT-SIM is absent. In table 2, justification of the design choice of InternVid-10M-FLT is not explained. (At glance, the result of performance improvement after filtering the dataset with a high UMT-SIM score seems trivial.) Figure 5 introduced 3 schemes of interleaving video clips, text, and ASR text. However, comparison between the methods or further analysis is not given. It is doubtful that CLIP is a suitable choice for comparing their method in action recognition tasks. There are likely more appropriate video models to use in the benchmark for a fairer comparison.

1. UMT-SIM: In our work, UMT-SIM refers to the use of Unmasked Teacher (UMT) [R1] to compute the similarity score between a given video clip and the accompanying text. UMT is a video-language model that performs similarly to CLIP in terms of video-text similarity computation, but is trained in different data and task settings. Specifically, we extract the normalized video embedding of the given clip using UMT's video encoder and the normalized text embedding of the text using UMT's text encoder, then we compute their cosine similarity.

2. For InternVid-10M-FLT, we apply filtering strategies to video data in addition to DIV sampling. The employed steps as given as: a) Removing video clips shorter than 1s (approximately 23.15% of the total) or longer than 120s (around 0.84% of the total). b) Computing CLIPScore for each video clip using a randomly sampled frame from the clip with OpenAI's CLIP-ViT-L/14, then selecting clips within the top 30% of CLIPScores. c) Sampling 10M out of the remaining clips using DIV sampling. For DIV (diversity sampling), we aimed to sample video clips from all long videos available to maximize data diversity. This was done by counting the frequencies of long videos in the segmented clip pool and sampling clips with probabilities inverse to these frequencies. Here is a pseudocode example of this process:

    from collections import Counter
    import json
    import random
    import numpy as np

    data = json.load(open("/path/to/to_sample"))
    video_id = set([x["video"].split("/")[-1][:11] for x in data])
    video_id_counter = Counter([x["video"].split("/")[-1][:11] for x in data])
    sampling_weights = [1.0 / video_id_counter[x["video"].split("/")[-1][:11]] for x in data]
    np.random.seed(42)
    sampling_weights = np.array(sampling_weights)
    sampling_weights = sampling_weights / sampling_weights.sum()
    sampled_index = np.random.choice(len(data), 10647458, replace=False, p=sampling_weights)
    data = [data[i] for i in sampled_index]
    json.dump(data, open("/path/to/sampled", "w"))  

3. Interleaving Video Clip Schemes: We introduced three potential schemes without claiming them as our contributions. The intention was to illustrate possible applications of the dataset.

4. Benchmark for Action Recognition Tasks: We acknowledge that CLIP might not be the most suitable model for action recognition. Therefore, we also compared our method with other specialized video models for action recognition (see Table A below). Notably, ViCLIP trained only using contrastive loss and without additional learnable parameters on InternVid outperforms both CLIP-ViP and TVTSv2, where the latter two approaches are also pretrained on large-scale video datasets, demonstrating the effectiveness of our dataset.

Table A. Zero-shot action recognition results on Kinetics 400/600/700.

References:

[R1] Unmasked teacher: Towards training-efficient video foundation models. In ICCV. 2023.

[R2] CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment. In ICLR. 2023.

[R3] TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale. In arXiv. 2023.

Q4: Absence of ablation study and potential impact. In contrast to previous models which are benchmarked in datasets mostly composed of English, InternVID consists of clips with diverse languages. Further analysis about the potential impact caused by this aspect should be considered.

Unlike previous models that are benchmarked mostly on English-based datasets, InternVID encompasses clips from a variety of languages. This necessitates a further analysis to determine the potential impact this diversity might have. Currently, we hypothesize that the language of the video may not significantly impact the generated captions as our deployed image caption models generate English descriptions based purely on input frames. However, in terms of video distributions, there may exist differences (such as in behaviors, activities, and events) between videos stemming from different countries due to varied cultural backgrounds.

To examine this hypothesis, we selected two 2 million subsets of InternVid: one consisting of only English videos (InternVid-2M-EN) and another with only Chinese videos (InternVid-2M-CN). It's important to note that whether the videos are in English or Chinese, we generate captions in English.

Our ViCLIP-B model was pretrained on these subsets, and we conducted zero-shot experiments as described below. Due to resource constraints, we trained the ViCLIP-B with a batchsize of 4096 using 8 A100 GPUs with a mask ratio set to 0.9. The remaining training settings were kept consistent with those outlined in the paper.

We found that the model pretrained with InternVid-2M-EN outperformed that with InternVid-2M-CN notably in both zero-shot action recognition on K400/600/700 and video retrieval. This result can be attributed to the fact that InternVid-2M-EN has a data distribution much closer to downstream task data than InternVid-2M-CN, as all used task videos are sourced from English sources.

Table B. Zero-shot action recognition results of ViCLIP using different pretraining sources on Kinetics 400/600/700.

Table C. Results of zero-shot video retrieval of ViCLIP using different pretraining sources on MSR-VTT, LSMDC, DiDeMo, MSVD, and ActivityNet.

Q5: Minor comments on Figure 2. Similar colors like green and dark green are used, but this may cause confusion. Using more distinct and discrete colors, such as red and blue, could enhance the clarity.

Sure. We will update Figure 2 and check other ones for better clarity.

Q6: Consideration with Table 6. Comment of VideoCrafter, VideoFusion is absent. Does InternVID also improve VideoCrafter, VideoFusion in IS, FID, FVD, and CLIPSIM? Can you provide further analysis about the choice of InternVID-Aes-18M rather than exploiting the full InternVID-200M?

The lack of full training code and details from VideoCrafter and VideoFusion poses a challenge for reproducing their results and further training with additional data.

The decision to use InternVID-Aes-18M instead of the full InternVID-200M is driven by techniques from studies [R4]. These studies suggested that image aesthetic quality significantly impacts text-to-image generation, even more than the sheer quantity of images. We hypothesize that this principle can be extrapolated to video generation as well. Therefore, we decided to curate a subset based on video aesthetics for text-to-video generation.

The process to compute video aesthetics involved calculating the image aesthetic scores of four randomly sampled frames from each video clip. The tool used for this task was the LAION aesthetic predictor. The highest score among the four frames was then chosen to represent the aesthetic score of that particular video.

This approach ensured that our model was trained on aesthetically pleasing videos, potentially improving the overall quality of the generated outputs. However, note that this approach has not been explicitly tested on VideoCrafter or VideoFusion. Therefore, it remains uncertain whether InternVid would result in improved IS, FID, FVD, and CLIPSIM metrics for these models. Further experiments would be needed to confirm this hypothesis.

[R4] Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack. In arxiv. 2023.

Q1: The majority of the comparisons are with CLIP which was pretrained with image-text pairs and not video-text; I would like to see how ViCLIP performs compared to other video or video-language models (e.g., pretrained on HD-VILA, HT100M, YT100M), I think that would be a fair comparison than comparing with image-text models.

We understand your concern about the comparison with CLIP, which was pretrained on image-text pairs. Indeed, to provide a more fair comparison, we included results against CLIP-ViP, a version of CLIP pretrained on HD-VILA with coarse synthetic captions (one frame caption used as the video caption) and ASR text. However, please note that this comparison still has limitations as CLIP-ViP only offers a ViT-base version and some of its variants are trained with different settings.

Turning our attention to Table A, when evaluated on MSR-VTT, our model, ViCLIP-B, trained on InternVid-200M/-10M-FLT outperforms CLIP-ViP-B, trained on HD-VILA-100M with subtitles, achieving a higher R@1 score (50.7%/49.0% vs. 47.7%). CLIP-ViP-B reaches similar performance (49.6%) only when using a combination of captions and subtitles from HD-VILA-100M. It's worth noting that CLIP-ViP-B uses more frames for pretraining than ours (12 vs. 8).

Table A. Fine-tuned video retrieval on MSR-VTT.

Q2: I would encourage authors to dig deeper and find a more concrete argument/explanation: why does pretraining with InternVid-10M-FLT or InternVid-10M-DIV perform better in zero-shot than InternVid-200M, but not when finetuned.

The varying performance of ViCLIP (trained on InternVid-10M-FLT/DIV and InternVid-200M) in zero-shot vs. fine-tuned action recognition can be attributed to the differing demands of these tasks. Zero-shot action recognition primarily depends on the alignment between video and text representations, while fine-tuned action recognition focuses on discriminating video representation as we fine-tune the video encoder using supervised video-label data and discard the text encoder. So it's important to note that there's no theoretical guarantee for correlation between a model's zero-shot and fine-tuned capabilities.

You correctly noted that InternVid-10M-DIV/FLT performs better in zero-shot action recognition (Table 2), but not in the fine-tuned setting (Table 3), when compared to InternVid-200M. Our interpretation is as follows:

1. ViCLIP trained on InternVid-200M likely has a more robust video representation due to a larger quantity and diversity of videos.
2. On the other hand, ViCLIP trained on InternVid-10M-FLT/DIV might excel in video-text alignment—made possible by limiting the number of clips drawn from identical videos. This results in more diverse training data compared to InternVid-200M’s, thereby facilitating contrastive learning, enhancing video-text representation, and driving better outcomes for zero-shot action recognition and tasks like retrieval.

I would like to thank the authors for responding to my comments. I hope the authors would make the said changes in their final version as promised. I am happy to stick to my original rating in favor of acceptance.

Q3: The performance of InterVid-10M-DIV, InterVid-10M-FLT is weird. It shows much better performance in zero-shot action recognition in Table 2, but poor in the fine-tuned setting of Table 3. The authors give reasons for false negatives, but what is the training batch size, given the much larger training data, I think the possibility of the same video clip appearing in the same batch is low.

We reason that the contrasting performance of ViCLIP (trained on InternVid-10M-FLT/DIV and InternVid-200M) in zero-shot vs. fine-tuned action recognition comes from differing task demands. Zero-shot performance largely hinges on aligning video and text representations, while fine-tuned action recognition focuses more on the discriminative abilities of the video representation. As such, there is no guaranteed correlation between a model's capabilities in these two tasks.

Regarding your observation about InternVid-10M-DIV/FLT, we agree it performs better in zero-shot action recognition (Table 2), yet lacks in the fine-tuned setting (Table 3). Our explanation is that ViCLIP trained on InternVid-200M has a more robust video representation due to a higher quantity and diversity of videos. Conversely, the version trained on InternVid-10M-FLT/DIV likely excels in video-text alignment: a result of reducing the number of clips sourced from identical videos. This approach makes their training data more distinct than InternVid-200M’s, which promotes contrastive learning to enhance video-text representation—yielding superior outcomes for zero-shot action recognition and video-text tasks such as retrieval.

In response to the query about false negatives and batch sizes, you're right that the same video clip appearing in the same batch is relatively rare given the extensive size of the training data. However, we consider an alternative explanation for the false negatives: the higher video diversity in InternVid-10M-FLT/DIV and its improved video-text correlation (sorted by CLIP) result in a more effective video-text representation compared to InternVid-10M.

Q1: Some details are not presented clearly. 1) How many descriptions are generated at the coarser level? Are all descriptions used for training the final model at the finer level? 2) Which pre-trained language model is used for processing captions? 3) The details about DIV and FLT are not introduced.

1. We generated both coarse and fine descriptions for all of the collected and segmented video clips, totaling 234 million. However, not all descriptions at the finer level were used in training the final model. For our largest experimental setting conducted on InternVid-200M, we trained ViCILP on a random selection of 200 million clips with synthetic video captions derived from both coarse and fine-level captions.

2. The language model predominantly used for processing captions was T5-summary, accessible at this URL. A small proportion of conversions were handled by Vicuna. We found that, when compared to other options like LLM (7b or 13b), T5 offered significantly faster processing speeds and did not generate hallucinations. On average, T5 completed each summary within 14.96ms while Llama2-7b took longer at 81.51ms, both using A100-80G.

3. For DIV (diversity sampling), we aimed to sample video clips from all long videos available to maximize data diversity. This was done by counting the frequencies of long videos in the segmented clip pool and sampling clips with probabilities inverse to these frequencies. Here is a pseudocode example of this process:

    from collections import Counter
    import json
    import random
    import numpy as np

    data = json.load(open("/path/to/to_sample"))
    video_id = set([x["video"].split("/")[-1][:11] for x in data])
    video_id_counter = Counter([x["video"].split("/")[-1][:11] for x in data])
    sampling_weights = [1.0 / video_id_counter[x["video"].split("/")[-1][:11]] for x in data]
    np.random.seed(42)
    sampling_weights = np.array(sampling_weights)
    sampling_weights = sampling_weights / sampling_weights.sum()
    sampled_index = np.random.choice(len(data), 10647458, replace=False, p=sampling_weights)
    data = [data[i] for i in sampled_index]
    json.dump(data, open("/path/to/sampled", "w"))  

For FLT (filtering), we applied a series of filtering strategies to video data alongside DIV sampling. These included:

a) Removing video clips shorter than 1s (approximately 23.15% of the total) or longer than 120s (around 0.84% of the total).

b) Computing CLIPScore for each video clip using a randomly sampled frame from the clip with OpenAI's CLIP-ViT-L/14, then selecting clips within the top 30% of CLIPScores.

c) Sampling 10M out of the remaining clips using DIV sampling.

Q2: A closely related work, CLIP-ViP[R1], is not included in related works and compared. It also adopts a caption model to generate descriptions for video clips. It is corresponding to the coarser level in this paper.

We acknowledge the omission of CLIP-ViP[R1] in the related works. This will be rectified in the final manuscript where a thorough discussion on CLIP-ViP and its relation to our work will be included.

In response to your request for a comparison with CLIP-ViP, we have provided preliminary results in Table X. Please note, these results are not strictly like-for-like due to differing parameters such as number of sampled frames during training, training epochs, and resource constraints that limited our ability to retrain all models.

As seen in Table A, when evaluated on MSR-VTT, our model (ViCLIP-B used with InternVid-200M/-10M-FLT) outperforms CLIP-ViP-B used with HD-VILA-100M, giving a better R@1 score (50.7%/49.0% vs. 47.7%). However, when captions and subtitles from HD-VILA-100M are combined, CLIP-ViP-B achieves comparable result of 49.6%. An important note is that CLIP-ViP-B uses more frames for pretraining than ours (12 vs. 8).

This suggests that synthetic video captions from our InternVid dataset can compete effectively against HD-VILA-100M's combined subtitles and captions.

• Table A. Fine-tuned video retrieval on MSR-VTT. |Method | Data | #Frames in Train | T2V | V2T | |:---:|:---:|:---:|:---:|:---:| |CLIP-ViP-B | +HD-VILA-100M | 12 | 47.7 | | |CLIP-ViP-B | +HD-VILA-100M(sub+cap) | 12 | 49.6 | | |CLIP-ViP-B | +HD-VILA-100M(sub)+Im-text data | 12 | 49.1 | | |ViCLIP-B | +InternVid-200M | 8 | 50.7 | 49.4 | |ViCLIP-B | +InternVid-10M-FLT | 8 | 49.0 | 49.2 |

[R1]: CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment, ICLR 2023.

Q1: There are not many analyses and design justifications for the proposed ViCLIP. If ViCLIP is claimed as one of the contributions of this paper. The proposed architecture is not novel and the masking idea needs further justification. For example, the efficiency gain vs. the performance drop, and the ablation over the masking ratio.

The primary aim of applying ViCLIP on InternVid is to validate the efficacy of our proposed dataset in video understanding tasks. We concur that the architecture and training methodology for ViCLIP are not novel; it follows the design principles of CLIP, with the addition of mask modeling for training efficiency.

The masking scheme, specifically the choice of a 0.9 masking ratio from VideoMAE, was adopted to balance computational efficiency and performance. Below, we present Table A, which lists the trade-off between efficiency gains and performance drops at different masking ratios for ViCLIP-B. As can be seen, decreasing the masking ratio improves performance but increases GPU memory usage significantly. The masking ratio of 0.9 was chosen because it provides the highest efficiency with a tolerable performance drop on downstream tasks.

• Table A. Fine-tuned video retrieval on MSR-VTT from ViCLIP with different masking ratios. ``Mem" denotes single GPU memory usage with per GPU batch size 128. |Method | Data | Masking ratio | T2V | Mem/G| |:---:|:---:|:---:|:---:|:---:| |ViCLIP-B | +InternVid-10M | 0.7 | 48.0 | 27.9 | |ViCLIP-B | +InternVid-10M | 0.8 | 47.7 | 19.2 | |ViCLIP-B | +InternVid-10M | 0.9 | 47.4 | 12.1 |

Q2: The video caption is generated by language model from frame-level captions. In this case, will this reduce the number of motion-related words that need to be captured from video-based understanding?

From a statistical perspective, generating video captions from frame-level captions using a language model has a negligible effect on the number of motion-related words captured for video-based understanding.

To illustrate this, we counted the unique verbs (using nltk package) in the captions from a 10m subset of InternVid under two settings:

1. In the first setting, the captions are video captions generated by the language model.

2. In the second setting, the captions are frame-wise ones from BLIP2 and tag2text.

We found that the number of unique verbs in the video captions is 109,859, whereas for the frame-wise captions it is slightly higher at 109,895. This small discrepancy suggests that almost no motion-related words are lost during the caption generation process by LM. Therefore, we believe our approach maintains most of the important motion-related information needed for video understanding.

Q3: Given 10M pretrained data, the ViCLIP receives better zero-shot action recognition results from WebVid and better fine-tune action recognition results in Table 2 and 3. And the gain from 50M, 200M INTERNVID pretraining is minor. Does it mean the pretraining data is not the more the better? The performance of vision and language model will be saturated when the pretraining data reach to a certain scale? Could you please provide more insights here?

For the subset of 10M pretraining data, ViCLIP performs better in both zero-shot and fine-tuned action recognition when utilized with InternVid-10M-FLT/DIV compared to WebVid. This performance differences between models using InternVid-10M-FLT/DIV and InternVid-10M underscores the importance of sampling unique clips from videos - a hypothesis supported by our initial explorations.

Our findings from Tables 2 and 5 along with Figures 7 and 8 demonstrate a consistent performance improvement in zero-shot action recognition and fine-tuned video retrieval as the pretraining data size escalates. However, the enhancement in zero-shot video retrieval is only marginal, which we believe stems from the limited increase in video-text diversity during pretraining. In particular, the more clips sampled from the same video, the higher tendency they have to share similar semantics. Furthermore, while a generative captioning method aids in video descriptions, the limited capacity of the captioning model restricts the diversity of produced captions, making them somewhat inferior to human annotations. These findings motivate us to explore scaling InternVid-10M-FLT/DIV and improving video captions through advanced captioning methods, language modeling, and human feedback techniques.

From this standpoint, our current experimental data does not conclusively suggest a saturation point for vision and language model performance as pretraining data scales up. We can only affirm that zero-shot video retrieval performance will reach a plateau at a certain scale of pretraining data when further diversification of the data used becomes unavailable.

Thanks all authors for their effort in the rebuttal.

The rebuttal address most of my concerns, so I keep my rating suggesting acceptance of this work.

Thanks