ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models

We greatly appreciate your insightful feedback and the opportunity to clarify aspects of our work. Below, we address each of your concerns in detail.

Simplicity of the Benchmark Formulation

We understand your concern regarding the simplicity of our benchmark. However, we argue that this simplicity is intentional and beneficial. The two-choice question format was chosen to precisely evaluate specific capabilities of video-language models (VidLMs) without the confounding effects of more complex task structures. This straightforward approach allows us to directly assess whether a model possesses a particular capability, avoiding confusion that could arise from more intricate tasks. Moreover, our foiling setup offers several advantages: it is task-agnostic, requires no additional fine-tuning or specific heads for evaluation tasks, and allows direct assessment of multimodal pre-training benefits without the risk of shortcut learning. This simplicity also facilitates faster and more straightforward testing, making our benchmark more accessible and easily adoptable by the research community.

Temporal Understanding in Video-Language Models

Your suggestion regarding the assessment of models' temporal understanding, including locating event starts and ends, is insightful. In our benchmark, this is indirectly addressed in the Action Counting test, where models must recognize and sum up the repetitions of an event, requiring an understanding of the event's visual loop and its linguistic representation. Additionally, our Change of State test, particularly in its reverse subtest, challenges models to align specific visual sub-events with their textual descriptions. This requires the model to segment both the video and the text into sub-phases and align them accurately, implicitly assessing the model's ability to determine where sub-events start and end.

Evaluating Multimodal Large Language Models

We acknowledge the growing interest in multimodal large language models and think evaluating them in a generative setting is interesting. That said, our primary aim is to evaluate specific capabilities of VidLMs in a clear and straightforward manner, rather than tackling the broader, unsolved problem of natural language generation evaluation.

Regarding Your Question on Video Analysis

Your query about the model's sensitivity to video length and the impact of noise frames is intriguing. We are currently working on a length-based performance analysis, as preliminary observations suggest that performance deteriorates with longer videos. This is likely due to the sampling methods used by some VidLMs and the increased complexity of modeling longer temporal dependencies. Please, check the appendix of the updated manuscript to see our analysis (D. Further Analysis).

I appreciate the responses given by the authors. I am satisfied with some of the answers provided. However, I also strongly advise the authors to rigorously evaluate existing multimodal large language models on the proposed benchmarks. In the revised version, I see that the proposed benchmark is a challenging benchmark for the existing multimodal large language models, such as Otter. If the authors could extend this work to a new benchmark for multimodal large language models, it would significantly amplify the impact of this study.

Considering the efforts made by the author during the response period, I finally decided to raise the rating from 5 to 6.

Thank you for your insightful comments. We appreciate your feedback and have addressed your concerns as follows:

Missing Recent Models:

We acknowledge the importance of including recent video-language models (VidLMs) in our evaluation. To this end, we have adapted the InternVideo and mPLUG2 models. These models will be featured in the final version of our paper. Regarding UniFormer-v2, we excluded it due to its inherent design as a vision-only model, lacking the necessary capabilities for language processing, which is crucial for our benchmark's objectives. In the following, we present the results of InternVideo and mPLUG2 models (P+T). We will update our manuscript accordingly.

Comparison to NExT-QA:

Our benchmark differs from NExT-QA in several fundamental aspects.

1. Unlike NExT-QA, which primarily focuses on event recognition and localization, our benchmark is designed to challenge pretrained VidLMs with minimally edited counterfactual captions. This novel approach assesses various linguistic and spatio-temporal capabilities, such as understanding relational dynamics, counting repetitive actions, and observing state changes.
2. Our benchmark employs a counterfactual caption generation technique that provides a more nuanced and challenging test of model capabilities. This stands in contrast to NExT-QA’s multiple-choice format, where incorrect choices are often drawn from unrelated questions, leading to irrelevant options. For instance, consider this example from the NextQA paper: “Q: Why did the girl in blue stop and turn around at the start? Options: 0. Pour ingredient in, 1. Scared, 2. Waiting for the lady, 3. Take out vegetable, 4. Dance” (Correct Answer: 2). This format can unintentionally simplify the task for models by including distinctly irrelevant options, thus diminishing the quality of the assessment.
3. Our benchmark is specifically tailored as a zero-shot, test-split-only foiling benchmark, eliminating the need for additional finetuning or task-specific prediction heads. This design allows for a direct assessment of the visio-linguistic competencies acquired by models during their pre-training phase, thus providing a novel and robust measure of temporal understanding and reasoning in video-language models.

Image-Language Model Input:

In our methodology, we sample multiple frames across a video clip. For each frame, using the image-language model, we compute individual scores, which we then average to derive a final score for the whole video clip. Further details of this process are given in Section A.4, Implementation Details, of our paper.

Thank you for your detailed and insightful comments on our work. We highly value your feedback and are pleased to provide detailed clarifications on the points you raised.

Distinctiveness of ViLMA from Existing Foiling Benchmarks:

ViLMA stands out due to its comprehensive approach compared to existing video-language foiling benchmarks. Unlike Contrast Sets (Park et al., 2022), which focuses primarily on recognizing entities and actions, and Test of Time (Bagad et al., 2023), which concentrates on assessing the temporal order of events using synthetic video-text pairs, ViLMA encompasses a broader range of linguistic and temporal phenomena. This includes spatial relations, counting repeated actions, and observing state changes. Additionally, we test a diverse array of models to ensure robustness. Moreover, VALSE (Parcalabescu et al., 2022), while being the most similar to our benchmark, is designed exclusively for static Image-Language Models (ILMs). ViLMA, on the other hand, extends this scope by focusing on Video-Language Models (VidLMs) with an emphasis on temporal grounding. This distinction is critical for advancing the field, as it addresses a wider and more complex set of challenges in video-language understanding.

Explaining the Performance Gap Between T and P+T:

The performance gap between T and P+T is attributable to a plausibility bias, as also noted by Reviewer f3MM. Our methodology to mitigate this bias is outlined in our response to f3MM. In particular, we have taken careful measures to ensure that our human validation process does not reintroduce lexical distributional shifts between captions and foils. This is achieved by aligning the lexical distributions of captions and foils, as indicated in Appendix B.1, Bias Check (pages 22-24 of the revised version).

Incorporation of Recent Video-Language Models:

Acknowledging the importance of including recent video-language models (VidLMs) in our evaluation, we have integrated all the methods you mentioned: We have successfully concluded all experiments related to the Otter model and are currently awaiting the completion of experiments for the other models. Their results, which demonstrate the efficacy and relevance of our approach, will be included in the final version of our manuscript. This inclusion will not only enhance the comprehensiveness of our study but also it will ensure that our findings remain relevant and applicable to current advancements in the field.

Thank you for your comprehensive review of our paper. Your insights have been instrumental in refining our approach, and we address your concerns as follows:

Linguistic Priors in Situation Awareness and Spatial Relations Tests

We recognize the importance of your concern regarding linguistic priors in the Situation Awareness (SA) and Spatial Relations (SR) tests. In our design of ViLMA, we have implemented several strategies to minimize such biases. This included deploying Natural Language Inference (NLI) filters and GRUEN score evaluations, as well as manual validation through Amazon Mechanical Turk (AMT). These measures were aimed at ensuring a balanced representation of lexical items in both factual and counterfactual captions to reduce the models' innate lexical biases. Despite these efforts, completely eliminating plausibility bias, particularly in SA and SR tests, is challenging. For example, creating counterfactual scenarios in SA tests often results in a lower thematic fit (e.g., consider the sentence from Table 1 for SA “A policeman holds a blond man against a wall”), a phenomenon observed in the perplexity patterns of unimodal models. This highlights the inherent challenges in these tests, where subtle biases arise from fine-grained differences in actor roles. Notably, VidLMs consistently scored lower than unimodal LMs and ILMs, indicating their struggle with integrating language with video and utilizing temporal information from videos. Importantly, when combining proficiency and main tests, we observed a significant reduction in the performance of unimodal models, suggesting that our approach effectively mitigates most plausibility biases.

The Rare Actions Test Results

In the Rare Actions (RA) test, we aimed to evaluate the robustness of VidLMs against unusual action-object compositions. Our findings showed a distinct difference in the performance of unimodal LMs and VidLMs. Unimodal LMs typically preferred counterfactual captions, a tendency reflecting a linguistic probability bias. In contrast, VidLMs showed a remarkable ability to correctly identify captions in scenarios deviating from standard linguistic patterns, suggesting their effective integration of visual information. This contrast in the performance is particularly illuminating, highlighting the VidLMs' capability to overcome language-side expectancy biases. The granularity of the RA test, focusing on action or object recognition in potentially out-of-distribution scenarios, serves as an important measure against modality collapse, where multimodal models might overly rely on linguistic cues.

Additional Information on VidLMs

In response to your suggestion, we enhanced our paper with detailed information on the VidLMs, including training data, and architectural designs, included in the appendices (please see Table 3 under Section A.3 Video-Language Models, we will also add parameter sizes soon). Clearly, this addition will provide readers with a comprehensive understanding of the factors influencing the varied performance of these models.

We hope this clarifies the points you asked in your review.