HowToCaption: Prompting LLMs to Transform Video Annotations at Scale

Also they don't compare the influences of captions length.

Following the suggestion, we assess the length of the generated captions for different prompts from Table 1. We note that we explicitly prompt LLM to generate "short sentences" in sentence <x4>: "Write only short sentences," which contributes to improved performance. We provide here results for the subset of prompts. We will add the full table in the paper.

Other downstream tasks such as VQA and Video Caption should be performed to prove the efficiency of dataset. the paper only test it on text-video retrieval which is far from enough.

We value the suggestion and are initiating further evaluations. While we aim to present results within the discussion period, a thorough analysis may require additional time. In the meanwhile, we provide preliminary zero-shot video clip captioning results on the YouCook2 dataset after only 1 epoch of training on 10% of the HowToCaption dataset, with the captioning model initialized from BLIP (note, we do not do any finetuning on YouCook2):

The setting of MSVD is not describled clearly.

For evaluation on MSVD, we follow the standard split and use 670 videos for testing with 40 captions corresponding to each video. We follow standard practice [6] and count each caption-video pair towards the metrics. We will clarify it in the paper.

[6] Bain, Max, et al. "Frozen in time: A joint video and image encoder for end-to-end retrieval." ICCV, 2021.

the difference between HowTo100M and other large scale caption dataset such as WebVid10M is also not describled.

We discuss the difference between large-scale datasets in Section 2.2. WebVid10M is an extension of the WebVid2M dataset, and similarly to WebVid2M, it contains only short videos that mostly don't have audio. We will add it.

Also, this paper lacks some quantitative analysis of HowToCaption, such as the distribution of caption length and the .

Thanks for the suggestion! We have added more analysis of the captions of the HowToCaptions dataset in the Appendix, Section A.5 (see updated pdf). We included an analysis of caption length and language diversity.

Some settings are not be evaluate such as WebVid2M + CC3M, which is used widely in pre-trained field.

We direct attention to Table 4, where we employ WebVid2M to train the dual-encoder architecture with BLIP initialization. It is important to note that BLIP initialization already encompasses CC3M. Hence, we believe this setup aligns with the reviewer's suggestion.

Writing. [...] there are some grammar errors in paper, such as in third paragraph in 3.3, "first" is wrong.

We thank the reviewer for the valuable feedback. We will proofread the paper and update it in the following revision.

Please let us know if you have any other questions or concerns regarding our paper or response.

We thank the reviewer for providing constructive feedback on our work. In order to adequately address the reviewer's concerns, we would be grateful if the reviewer could elaborate and provide additional details regarding the potential weaknesses of our submission.

Novelty is insufficient. This paper propose a approach, HowToCaption, prompting an LLM to create detailed video captions based on ASR subtitles. However, similar methods have been used widely in dataset industry, including Tencent, OpenAI and so on ...

If the reviewer could provide relevant references or suggestions regarding other methods to compare to that we might have missed, we are happy to provide additional comparisons to and discussions of those methods.

Moreover, we would like to highlight that we pioneer a novel method involving data collection with LLMs for text-video learning, offering significant advancements in the field. This stands in contrast to recent works in prompt engineering [1,2], which focus on prompting LLMs to address pre-existing tasks. We also release a new large-scale dataset, HowToCaption, which contains high-quality video descriptions for instructional videos. Please also note that this is why we chose "Dataset and Benchmark" as the primary subject area, which is in line with the ICRL CfP.

[1] Kojima, Takeshi, et al. "Large language models are zero-shot reasoners." NeurIPS, 2022.

[2] Zhou, Yongchao, et al. "Large language models are human-level prompt engineers." ICLR 2023

Firstly, in ablation study, the authors need to evaluate different speech recognition methods.

We thank the reviewer for the suggestion. In our work, we utilized subtitles and timestamps, officially released with the HowTo100M dataset[3], that were further postprocess into full sentences by [4] (Sentencified HowTo100M). In our preliminary study, we also experimented with the recently released WhisperX model[5] to obtain better quality subtitles and with the original officially released HowTo100M[3] subtitles. We found that all subtitles show similar performance. We report these preliminary results (note that this study is done only on 10k HowTo100M videos):

We observe that the quality of different ASR methods does not significantly alter the quality of the acquired dataset. We assume that the noise introduced in ASR-video alignment is inherent to the nature of speech, often not aligning precisely with the content of the video. Moreover, filtering and alignment postprocessing that filter noisy captions might further mitigate the difference. We will include it in the paper.

[3] Miech, Antoine, et al. "Howto100m: Learning a text-video embedding by watching hundred million narrated video clips." ICCV, 2019.

[4] Han, Tengda, Weidi Xie, and Andrew Zisserman. "Temporal alignment networks for long-term video." CVPR, 2022.

[5] Bain, Max, et al. "WhisperX: Time-accurate speech transcription of long-form audio." INTERSPEECH, 2023.

Thanks for highlighting the strengths of the proposed paper and for providing valuable feedback! In the following, we would like to address the remaining concerns.

This work is more of a prompt engineering than a research paper

We would like to highlight that our main contribution is twofold: we release a new large-scale dataset, HowToCaption, which contains high-quality video descriptions for instructional videos, and we design a human-free framework to transform noisy ASR subtitles into high-quality captions for instructional videos. Essentially, enabling data collection with LLMs for text-video learning offers significant advancements in the field. This stands in contrast to recent works in prompt engineering [1,2], which focus on prompting LLMs to address pre-existing tasks. We hope that the impact of this work might, therefore, be not only a better annotation for one specific dataset but also pave the way towards a general better practice in crawling and processing large amounts of video. Please also note that this is why we chose "Dataset and Benchmark" as the primary subject area, which is in line with the ICRL CfP.

[1] Kojima, Takeshi, et al. "Large language models are zero-shot reasoners." NeurIPS, 2022.

[2] Zhou, Yongchao, et al. "Large language models are human-level prompt engineers." ICLR 2023

It would be better to evaluate the proposed dataset on more challenging tasks, e.g., dense video captioning or moment retrieval

For the main evaluation, we chose the retrieval setting, as it does not depend on downstream training or fine-tuning, which evaluates the influence of our dataset directly. Other downstream tasks, such as captioning, evaluate the pre-training dataset only indirectly, as the final performance depends on many dataset-specific hyper-parameters of training the downstream task. Nonetheless, we value the suggestion and are initiating further evaluations. While we strive to present results within the discussion period timeframe, these tasks are very compute-intensive and might not finish in time. Moreover, we already provide initial results on zero-shot video clip captioning in the Response to Reviewer qBgz.

Please let us know if you have any other questions or concerns regarding our paper or response.

We are thankful to the reviewer for recognizing the novelty and strengths of our paper. The feedback provided is highly valued. In the following, we will try to address the remaining concerns:

Questions:

Can the authors present more qualitative examples from the newly introduced dataset, including both exemplary and flawed examples?

Thanks for the valuable suggestion! We have updated the Appendix (Section A.6) with additional qualitative examples of the HowToCaption dataset, including failure cases (see updated pdf). We aim to update the paper even with more examples in the next days.

Could the authors share some qualitative results for the text-video(+audio) retrieval tasks?

We highly appreciate the suggestion. We will update the Appendix with qualitative retrieval results as well.

Would the authors clarify why evaluations were restricted to retrieval tasks using BLIP? Is it feasible to assess the model's performance in visual captioning or question-answering scenarios?

We value the suggestion and are initiating further evaluations. While we aim to present results within the discussion period, a thorough analysis may require additional time. In the meanwhile, we provide preliminary zero-shot video clip captioning results on the YouCook2 dataset after only 1 epoch of training on 10% of the HowToCaption dataset, with the captioning model initialized from BLIP (note, we do not do any finetuning on YouCook2):

For the main evaluation, we chose the retrieval setting, as it does not depend on downstream training or fine-tuning, which evaluates the influence of our dataset directly. Other downstream tasks, such as captioning, often require fine-tuning on the downstream dataset and, therefore, evaluate the pre-training dataset only indirectly, as the final performance depends on many dataset-specific hyper-parameters of training the downstream task.

The font size in Figure 1 appears too small.

Thank you for pointing out the issue! We will enlarge the font in Figure 1 to make it more readable.

Considering the generality of the proposed framework, which should be applicable to any video-text dataset, have the authors considered employing these techniques across all publicly available benchmarks to train a large-scale video-language model?

We appreciate your suggestion to apply our techniques across all publicly available benchmarks. We fully agree that the HowToCaption method is generalizable to other datasets. Applying it across all publicly available instructional video datasets to transform ASR into captions could significantly contribute to training foundational models at a large scale. We will release the code and dataset for our work, anticipating that it will inspire and accelerate research in this field.

Please let us know if you have any other questions or concerns regarding our paper or response.

Thanks for taking the time to review our paper, highlighting the clarity and significance of our paper, and providing valuable feedback! We would like to address the raised points in the following:

Weaknesses:

Unclear advantages of using this dataset on video retrieval task.

Thanks for asking. LAVENDER uses a multi-modal encoder (i.e., that cross-attends video and text tokens) and only reports results after fine-tuning on each dataset. This differs from the evaluation in Table 5 in 2 ways: first, we focus on the comparison of dual-encoder SOTA methods (i.e., methods that do not cross-attend video and text tokens), second we consider zero-shot text-to-video performance rather than fine-tuned performance (performance of the models fine-tuned on each downstream task). We decided for this setup to evaluate the properties of the proposed dataset as a pretraining dataset for foundational models. To allow for an indirect comparison, we compare the fine-tuned performance reported by the LAVENDER method for MSR-VTT and MSVD with our zero-shot performance. It shows that pretraining with the proposed annotations provides a comparable performance even without cross-modal attention or any supervised fine-tuning.

We, therefore, assume that using our HowToCaption dataset will boost LAVENDER performance as well.

We want to highlight that we intentionally do not report methods that rely on a multi-modal encoder and are trained with image-text-matching loss rather than a contrastive loss. Such methods show an improved performance compared to dual-encoder models but require significantly higher computation costs to perform retrieval. They require a full forward pass for each (video, text) pair to output the similarity of this input pair ($N^2$ forward passes to compute the similarity between $N$ text and $N$ videos), while dual-encoder methods calculate cosine similarity in embedding space ($2N$ forward passes only). For a fair comparison (and as we want to test the data rather than the method), we therefore consider dual-encoder methods.

the term "generate" might be misleading as the LLM merely "rephrases"

While we task the LLM to rephrase the caption, we are using the term generate here as an LLM is fundamentally a generative model with no inherent guarantee that its output is aligned with the task. We will make this more clear in the paper.

In section 3.3, the authors hint at "fine-grained" captions, I am not fully convinced that LLMs always produces better fine-grained captions:

We appreciate the reviewer's feedback. We agree that the term "fine-grained" might be misleading. We will revise the paper to avoid this terminology.