VIDEOPROMPTER: AN ENSEMBLE OF FOUNDATIONAL MODELS FOR ZERO-SHOT VIDEO UNDERSTANDING

Thank you for your detailed review, below we address the key concerns.

Q1: The computational cost of performing multiple inferences far exceeds that of the basic video understanding model.

Kindly note that the VideoPrompter offers a flexible framework, allowing for the substitution of various modules, including the video-2-text model (VGPT), text-2-text model (GPT-3.5), and vision-language model (CLIP). During our initial submission, Video-ChatGPT (VGPT) $1$ was chosen as the video-2-text model due to its competitive performance when compared to Video-Chat and VideoLLaMA. However, the recent introduction of Video-LLaVA $2$ , a model released this week, has set a new state-of-the-art (SOTA) in zero-shot question-answer and video-based generative benchmarks. We performed an experiment where we replaced VGPT with Video-LLaVA and generated video-textual descriptions for HMDB-51 and UCF-101 benchmarks. Conducting a single inference (as opposed to the 10 inferences with VGPT) without any filtering applied (as now we have only one description per video), we observed a 0.21% and 0.75% improvement on previous scores for HMDB-51 and UCF-101 benchmarks, respectively, as shown in the table below. The adoption of Video-LLaVA over VGPT reduced the number of required inferences from 10 to 1, leading to a significant decrease in computational cost. Additionally, the per-video inference time for Video-LLaVA is approximately 6 times less than that of VGPT, with 3.79s per video, while, for reference, the inference time of the CLIP vision encoder is 0.74s per video. We anticipate that ongoing advancements in multi-model methods will further reduce the inference cost of VideoPrompter. We will include the updated results with Video-LLaVA on all benchmarks and models in our final manuscript.

Q2: Except for CLIP, the proposed approach shows relatively limited performance gain in other video-based models.

Please note that the large gain on the vanilla-CLIP can be linked to the fact that its not adopted for videos and when its used with our proposed framework, without any further training, the VideoPrompter brings significant boost in the performance. The video-based models such as ViFi-CLIP are already trained on Kinectics-400 and are adopted to videos to some extent. These models when combined with VideoPrompter increase the performance by 6.12%, 4.27%, and 5.3% for ViFi-CLIP, AIM, and ActionCLIP respectively on HMDB-51 benchmark and similar trend is seen on other benchmarks as shown in Table-2 and Table-6. Kindly, considering that no further training is performed, and the computational cost has been substantially reduced after Video-LLaVA, these score increments can be considered significant.

Q3: The configurations of adopted pre-trained models (AIM, ActionCLIP, ...) remain unclear. For AIM, do the authors directly remove the classification layers?

Kindly note that all of the pre-trained models used in this study (ViFi-CLIP, AIM, and ActionCLIP) are pre-trained on Kinectics-400. Secondly, the original AIM model removed the text classifier from the vanilla CLIP and only trained the visual encoder. We removed the classification layers from AIM and used a text classifier from the vanilla CLIP with the AIM-visual encoder to get zero-shot results. These points have been made more clear in the original paper, as highlighted in red.

$1$ Maaz, et al. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models." arXiv.

$2$ Lin, et al. "Video-LLaVA: Learning United Visual Representation by Alignment Before Projection." arxiv.

Thank you for your detailed review, below we address the key concerns.

Q1: The efficiency of VideoPrompter hasn't been thoroughly examined. It could substantially raise the inference costs.

Kindly note that the VideoPrompter offers a flexible framework, allowing for the substitution of various modules, including the video-2-text model (VGPT), text-2-text model (GPT-3.5), and vision-language model (CLIP). During our initial submission, Video-ChatGPT (VGPT) $1$ was chosen as the video-2-text model due to its competitive performance when compared to Video-Chat and VideoLLaMA. However, the recent introduction of Video-LLaVA $2$ , a model released this week, has set a new state-of-the-art (SOTA) in zero-shot question-answer and video-based generative benchmarks. We performed an experiment where we replaced VGPT with Video-LLaVA and generated video-textual descriptions for HMDB-51 and UCF-101 benchmarks. Conducting a single inference (as opposed to the 10 inferences with VGPT) without any filtering applied (as now we have only one description per video), we observed a 0.21% and 0.75% improvement on previous scores for HMDB-51 and UCF-101 benchmarks, respectively, as shown in the table below. The adoption of Video-LLaVA over VGPT reduced the number of required inferences from 10 to 1, leading to a significant decrease in computational cost. Additionally, the per-video inference time for Video-LLaVA is approximately 6 times less than that of VGPT, with 3.79s per video, while, for reference, the inference time of the CLIP vision encoder is 0.74s per video. We anticipate that ongoing advancements in multi-model methods will further reduce the inference cost of VideoPrompter. We will include the updated results with Video-LLaVA on all benchmarks and models in our final manuscript.

Q2: The selection of Video-ChatGPT? Alternative models should be considered and discussed.

Please note that our choice of video-to-text generative model is defined by the competitive performance $1$ (at the time of submission) of Video-ChatGPT compared to Video-Chat and VideoLLaMA. However, the recent introduction of Video-LLaVA $2$ , a model released this week, has set a new state-of-the-art (SOTA) in zero-shot question-answer and video-based generative benchmarks. We used it as an alternative model and show the results below for HMDB-51 and UCF-101 benchmarks. Note that, for Video-LLaVA single inference is performed (as opposed to the 10 inferences with VGPT) and no filtration is applied (as now we have only one description per video). Thank you for suggesting this analysis. We will include this ablation in our final manuscript.

Q3: An ablation study on the video-specific language descriptors is missing.

As recommended, detailed ablation on language descriptors is provided below. We individually analyze the following components: 1) class-attributes, 2) class-descriptions, and 3) action-context. Moreover, we also analyzed various combinations of these components such as 1) class-attributes + class-descriptions, 2) class-attributes + class-descriptions + action-context. Thank you for suggesting this analysis. We will include this ablation in our final manuscript.

$1$ Maaz, et al. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models." arXiv.

$2$ Lin, et al. "Video-LLaVA: Learning United Visual Representation by Alignment Before Projection." arxiv.

Thank you for your detailed review, below we address the key concerns.

Q1: Leaking examples from downstream datasets to foundation models.

Kindly note that the CLIP is trained on 400M image-text pairs and the VGPT employs video-text pairs and is trained on a subset of ActivityNet-200. While the GPT is only trained on the text data. To investigate any overlapping with the downstream datasets we design the following experiment and present results on HMDB-51 and UCF-101 benchmarks:

1. We show zero-shot results of vanilla-CLIP on HMDB-51 and UCF-101 benchmarks to understand the extent of knowledge the CLIP model contains about these datasets.

2. Second, to understand any data overlap of VGPT, we only employ VGPT directly to classify the input video. We found this setting to degrade the results by a large margin. We relate this to the training of VGPT as it is trained on generating long contexts about the input, rather than specific categories $1$ .

Kindly, from the table below, it can be seen that the individual performance of these modules (vanilla-CLIP and VGPT) is substantially low, which shows minimum data overlap. Thank you for suggesting this analysis. We will include this ablation in our final manuscript.

Q2: The novelty seems moderate if not low.

Kindly note that our work is the first one to study the role of video-to-text models for video understanding. Our work highlights how the recent advances in the video-to-text models can be used in the general video-understanding pipeline, and how video descriptions can further enhance the visual representations. So far, the existing methods

$2,3$ are only limited to text-to-text generative models for enhancing the text classifier representations only. We summarize our contributions as follows:

1. So far, only text-to-text generative models (e.g. GPT) are used to enhance the representations of the text-classifier. Our work discusses the impact of having video-to-text generative models (e.g. VGPT) to enrich the visual representations as well, in addition to only text-based models (Section 2.2).

2. We propose a novel way of querying LLMs called action context (Section 2.3.2). Action context can classify semantically close videos under a single high-level context (category). In the related work, we discuss the limitations of the existing methods $4,5$ and discuss how these methods are only applicable to image benchmarks and are unsuited for video benchmarks where the classes are fine-grained as well as have diverse ranges.

3. We test our framework in three different video settings namely: action recognition, video-to-text, and text-to-video retrieval, and time-aware video tasks. We show results on 7 benchmarks including HMDB-51, UCF-101, SSv2, K400, MSRVTT, Charades, and the time-aware synthetic dataset. Moreover, as our framework offers plug and play module, we show results with various models such as CLIP, ViFi-CLIP, Action-CLIP, and AIM.

Q3 (a): The written presentation could be further improved.

Please note that, sections 2.1, 2.2, and 2.3 have been updated. Now, for more clarity we have re-named section 2.2 from video-specific language descriptors to class-specific language descriptors. The revised text is highlighted in red.

Q3(b): "table 6 is presented in page 7, yet never been referred from the text?

Kindly note that table 6 is referred in video action recognition section on page 6.

$1$ Maaz, et al. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models." arXiv.

$2$ Pratt, et al. "What does a platypus look like? generating customized prompts for zero-shot image classification." ICCV.

$3$ Menon. "Visual classification via description from large language models." arXiv.

$4$ Roth, et al. "Waffling around for Performance: Visual Classification with Random Words and Broad Concepts." arXiv.

$5$ Novack, et al. "Chils: Zero-shot image classification with hierarchical label sets." ICML.

Thank you for your detailed review, below we address the key concerns.

Q1: Can the authors justify the main novelty of this paper?

Kindly note that our work is the first one to study the role of video-to-text models for video understanding and highlights how video descriptions can further enhance visual representations. So far, the existing methods

$1,2$ are only limited to text-to-text generative models for enhancing the text classifier representations only. We summarize our contributions as follows:

1. So far, only text-to-text generative models (e.g. GPT) are used to enhance the representations of the text classifier. Our work discusses the impact of having video-to-text generative models (e.g. VGPT) to enrich the visual representations as well, in addition to only text-based models (Section 2.2).

2. We propose a novel way of querying LLMs called action context (Section 2.3.2). Action context can classify semantically close videos under a single high-level context (category). In the related work, we discuss the limitations of the existing methods $3,4$ and discuss how these methods are only applicable to image benchmarks and are unsuited for video benchmarks where the classes can be fine-grained as well can have a diverse range.

3. We test our framework in three different video settings namely: action recognition, video-to-text, and text-to-video retrieval, and time-aware video tasks. We show results with 4 different models on 7 benchmarks.

Q2: Why not just prompting VGPT for the downstream applications?

As recommended, we designed an experiment where only VGPT is directly used to classify the input video. We found this setting to degrade the results by a large margin. We relate this to the training of VGPT as it is trained on generating long contexts about the input, rather than specific categories $5$ . We also present a more extended setting, where the descriptions from VGPT are given to GPT-3.5 to predict the class name. Here, the GPT-3.5 is also provided by the list of class names. It can be seen that VideoPrompter outperforms both of the aforementioned settings which endorses its design choice. Thank you for suggesting this analysis. We will include this ablation in our final manuscript.

Q3: The comparison with CUPL.

We regret any confusion, it's important to note that a direct comparison with CUPL may not be straightforward, as it can be viewed as a complementary approach to our method. As discussed in $3$ , the performance of such methods benefits from averaging over a large number of descriptors. When we combined CUPL with VideoPrompter i.e. weights of language descriptors are averaged, performance increased by 0.12% for UCF-101. We will further update this discussion in our final manuscript.

Q4: The proposed method optimizes the selection of hyperparameters directly on the target dataset.

Kindly note that Figure 3 is mainly added to show an ablation analysis. The selection of hyper-parameters in our work follows the standard settings in recent studies $1,6$ . For instance, $1$ includes a thorough discussion of how a high value of temperature leads to more diverse descriptions. And $6$ discusses a similar filtering method.

Q5: The Action context is restricted to a tree-type relation.

We thank the reviewer for the pointer, this indeed could be true for some benchmarks as some classes may overlap. We designed an experiment where we prompted the LLM and explicitly mentioned that one child class can be assigned to multiple parent classes. We found such cases to be relatively small, e.g. we didn't find any such case for HMDB-51 and only a few for UCF-101, which increased our performance by 0.06% on top of the previous score. We will include the updated results on all benchmarks and models in our final manuscript.

$1$ Pratt, et al. "What does a platypus look like? generating customized prompts for zero-shot image classification." ICCV.

$2$ Menon. "Visual classification via description from large language models." arXiv.

$3$ Roth, et al. "Waffling around for Performance: Visual Classification with Random Words and Broad Concepts." arXiv.

$4$ Novack, et al. "Chils: Zero-shot image classification with hierarchical label sets." ICML.

$5$ Maaz, et al. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models." arXiv.

$6$ Li, et al."Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." ICML.