Does Your Video-language Model Actually Understand the Language Input?

About novelty.

Sorry for your misunderstanding on the novelty. 1) About LLM. We do not directly use LLM for data annotation. We only borrow knowledge from LLM. In our paper, we generate multi-level texts to fully understand the given text for coarse-grained language alignment, different from previous single-level text generation methods. Many LLM-based methods only treat each word with the same significance, which leads to that some unimportant words (e.g., article, particle) have a large weight. In realistic language rewriting, some core words (e.g., noun) tend to stay the same. Different from these methods, we can evaluate the significance score of each word to better understand the text. Also, we generate attributes to reason the fine-grained semantics in the given sentence and utilize attribute sampling to purify these semantics-rich attributes, which is a significant difference between our framework and other LLM-based methods. Moreover, these semantics-rich attributes can be used for cross-modal matching. For example, "person" in text always corresponds to a person with a head, two eyes and two arms. 2) Different from previous representation learning-based methods that only construct positives and negatives from a level, we construct positives and negatives from different levels (word-level and structure-level). Besides, we design a brand-new weighted sentence incorporation module for cross-modal fusion. Moreover, our attribute-based text reasoning can effectively mine the core semantics by semantic-based selection and format-based selection, which will reduce the negative impact of noise from generated texts.

Other comments

About the relationship between text representations and model output. We use t-SNE to visualize the feature in Figure 6 in the appendix.

We use t-SNE to visualize the feature in Figure 6 in the appendix. In the figure, without our language alignment modules (coarse-grained language alignment and fine-grained language alignment), the feature distance of two semantics-similar sentences is very far in the latent space. It shows that only utilizing previous text encoders cannot effectively map these semantics-similar sentences into a feature cluster. With our language alignment modules, the features of the two sentences are very close.

Q1: This method leverages multiple off-the-shelf large language models to generate refined text representations. Bring additional training cost and data scale.

About training cost and data scale. We add the time consuming over text repharased by LLM in Table 5. Although we generate some texts, three significant strategies (attribute sampling, semantic-based selection and format-based selection) are utilized to select some texts/attributes for cross-modal fusion, which can effectively reduce the computational cost.

Q2: The dataset contains noisy and incorrect annotations, which may increase challenges by potentially amplifying misalignment when relying on LLMs to hypothesize connections.

In fact, we notice some noisy and incorrect annotations in the text input. Fortunately, we can utilize attribute generation and attribute sampling to mine the ground-truth text semantics. Also, our semantic-based selection and format-based selection can select the correct attributes for fine-grained language alignment.

Q3: A similar concept is utilized in retrieval-augmented methods, which also aim to enhance data quality by incorporating diverse sources.

Our main aim is that most current VLMs implicitly assume that all the texts are predefined by the specific template. These retrieval-augmented methods directly enhance data quality by incorporating diverse sources, which increases the computational cost. Unlike them, we only choose some texts/attributes for cross-modal fusion. Also, these methods cannot remove the noisy and incorrect annotations, which might limit their performance. Different from them, we design three significant strategies (attribute sampling, semantic-based selection and format-based selection) to only mine some significant language knowledge for cross-modal fusion, which can reduce the negative impact of noises.

Q1: The idea of 1) using LLMs for data annotation and augmentation and 2) constrasting positives and negatives for representation learning has been well studied.

About novelty. Sorry for your misunderstanding on the novelty. 1) About LLM. We do not directly use LLM for data annotation. We only borrow knowledge from LLM. In our paper, we generate multi-level texts to fully understand the given text for coarse-grained language alignment, different from previous single-level text generation methods. Many LLM-based methods only treat each word with the same significance, which means that some unimportant words have a large weight. We can evaluate the significance score of each word to better understand the text. Also, we generate attributes to reason the fine-grained semantics in the given sentence and utilize attribute sampling to purify these semantics-rich attributes, which is a significant difference between our framework and other LLM-based methods. 2) Different from previous representation learning-based methods that only construct positives and negatives from a level, we construct positives and negatives from different levels (word-level and structure-level). Besides, we design a brand-new weighted sentence incorporation module for cross-modal fusion. Moreover, our attribute-based text reasoning can effectively mine the core semantics by semantic-based selection and format-based selection, which will reduce the negative impact of noise from generated texts.

Q2: Based on the context of video-text modality alignment, there is a need to have a discussion comparing it to the well-studied image-text setting. How is the misalignment in the image-text domain different from the video-text one? What particular considerations have been taken for it?

Video-text modality alignment vs image-text modality alignment. Image only contains the appearance information (spatial information), but video contains much temporal information. Most video-text datasets focus on the video actions, and they pay more attention to the temporal information. Besides, their sentences are also different, i.e., video-text datasets contain much action information with various verbs. For the image-text datasets, the text does not contain verbs since there is no action information.

Q3: Further clarafication is needed for the methodology. Are both video encoder and textual encoder being updated? There is no gradient backpropagation indicated in Figure 2.

About encoders. Since our framework is plug-and-play, both video encoders and textual encoders follow the original baseline models. Most of these baseline methods use pre-trained encoders without updates.

Q4: The approach to only minimize the constrastive loss on "the most discriminative word" is not properly justified. And the loss is definitely not "Weighted contrastive loss" as indicated in equation (4).

About constrastive loss. Sorry for the misunderstanding. We accidentally deleted some texts and the weighted constrastive loss in our first version. Now, the weighted constrastive loss is available in Eq. (5).

Q5: The ablation study on hyperparameters is missing.

About the ablation study on hyperparameters. We have reported it in Figure 6 in the appendix.

*Q1: The storyline and frame of work seems somehow not that well-aligned. *

About the storyline and frame of work. In the revision version, we have stated the answer in the conclusion section: “Most VLMs cannot fully understand the text input, but we can conduct a simple and plug-and-play module to help them understand the text input.” We observe that overly strict text input is user-unfriendly in some multi-modal tasks, which will limit the model performance. Due to the page limitation, we only choose three representative tasks (VSG, VideoQA, VTR). As for the video captioning task, different labelers have various language usage habits. For example, for a video on playing tennis, a labeler writes a natural caption “two persons play tennis”, while another labeler might give a detailed caption “A lady in a red T-shirt and a man in blue play tennis in the playground.” With various captions for training, we will obtain different results. Fortunately, in Section 3.2, we can generate attributes for mining the latent language semantics of the two captions for fine-grained language alignment.

Q2: Some conventional methods, e.g., synonym/antonym replacement, back-translation, paraphrasing, may work as well.

About different non-LLM data augmentation methods. We compare these methods with our methods on the NExT-QA VideoQA dataset with BLIP2 as our base model as follows:

Q3: The choice of benchmarks could be strengthened.

Since many multi-modal works use traditional datasets for performance evaluation, we follow their setting for fair comparison. Also, our framework can work on other large-scale datasets as follows:

Text-to-video retrieval task on VATEX

Text-to-video retrieval task on YouCook2

Performance (8 frames) on Video-MME without subtitles

Q4: This raises me another questions that I saw most of text cases in these paper are quite short, like in just a few words.

In fact, there are more long sentences than short ones. To make the text in the figures clear, we choose some short sentences in our submission. An example on long sentence is shown in Figure 7 in the appendix.For the augmented texts of the ActyNet-Caps dataset, the distribution on length of their augmented texts is as follows: 1-word to 10-word (18.2%), 11-word to 20-word (26.9%), 21-word to 30-word (52.6%), 31-word to 40-word (1.4%), longer than 50-word (0.9%).

[5] Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning, CVPR 2020 [6] A large cross-modal video retrieval dataset with reading comprehension, PR 2025 [7] Long Context Transfer from Language to Vision, Arxiv 2024

*Q5: Some implementation details are not quite clear to me. *

For instance, for the ActivityNet Captions dataset, we get more augmented texts on the training set, test set and validation set. By adding corresponding video, we can get the augmented video-text pairs. For the case with text augmentation, we combine original pairs and augmented pairs (original training video-text pairs and augmented training video-text pairs for training; original testing video-text pairs and augmented testing video-text pairs for testing). For the case without text augmentation, we only use original pairs, i.e., original training video-text pairs for training and original testing video-text pairs for testing.

• It would be better to have the ablation study on the effectiveness of using negative texts, especially for tasks like captioning/generation.*

We have shown the ablation study on the negative texts in Table 19 in appendix.

Q1: The current review of existing work on video-language model is severely incomplete, which makes the motivation that "video language models don't understand language input" actually invalid.

[a] proposes a framework called VX2TEXT, which is used to generate text from multimodal inputs containing video, text, speech or audio. The main objectives of VX2TEXT are: 1) to extract meaningful information from each input modality; 2) to effectively fuse information from different modalities; 3) to generate human-understandable text output. [b] presents a new method called LaViLa, which uses large language models (LLMs) to learn video-language representations. [b] fine-tunes pre-trained LLMs to be conditioned on visual input and further fine-tuning them to create automatic video narration. [c] proposes a method for fast adaptation of pre-trained contrastive models for multi-channel video-language retrieval. The authors explored two dimensions: video representation (continuous feature vectors or discrete text tokens) and video-text fusion method (multimodal Transformer or pre-trained contrastive text model). [d] shows that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodalinformed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. [e] propose a novel approach to enabling pretrained language models to perform video question answering without any training. The proposed Deep Concept Injection effectively circumvents the necessity of training projection networks, a widely accepted practice in this field, and instead makes insightful use of observed visual concepts as additional input text tokens and as a means for augmenting intermediate features.

Different from them, we propose a novel text-augmented VLM method to improve video-text fusion by text rewriting. Specifically, we first generate various text samples from the original ones based on the pre-trained LLM to target specific text components. A multi-level contrastive learning module is designed to mine the coarse-grained language information. Moreover, we also propose an attribute-based text reasoning strategy to learn fine-grained textual semantics.

In the final version, we will add a comprehensive review about video-language models.

Q2: It is also important to compare with at least some of the existing methods.

In fact, the text augmentation method is not our main contribution. Our main contribution is to propose a simple yet highly effective framework to improve the robustness and performance of VLMs, which can serve as a plug-and-play module for state-of-the-art VLMs. We try to reduce the feature distance between semantics-similar sentences. Due to the page limitation and the short author-reviewer discussion, we will add the comparison with [a-e] in the final version. In the future, we will introduce more state-of-the-art works (e.g., wikiHow and TTC-Loc) into our framework for better performance.

Moreover, our framework can be used in the training-free setting as follows:

VSG task with "R@1, IoU=0.3" as evaluation metric

[3] Training-free Video Temporal Grounding using Large-scale Pre-trained Models, ECCV 2024

Q3: Empirically, the motivation is not really validated in the current empirical results.

In fact, Figure 4 shows the dynamics of the model during training time in terms of the performance, which illustrates the effectiveness of each module.

Cross-data evaluation for zero-shot VideoQA task with WebVid10M as training data

[4] Zero-Shot Video Question Answering via Frozen Bidirectional Language Models, NeurIPS 2022

Also, we will add more analysis and experiments in the final version.

Q1: The observed phenomenon may be more attributable to domain-specific fine-tuning.

About domain-specific small model: In fact, larger pre-trained text encoders, such as BLIP-2 and InternVideo still experience severe performance degradation. Since their tasks, evaluation metrics and datasets are different (BLIP-2 and InternVideo are VideoQA methods on the NExT-QA dataset, while Charades-STA is a VSG dataset), the specific value of performance degradation should not be used for direct comparison. The Charades-STA dataset already contains various text inputs. When augmenting the text inputs, we will get many text queries with various templates with similar semantics, which limits their performance. For the large models, they have some generalization capabilities when handling text input since they have been pre-trained in various text inputs with various templates. Please note that our framework can serve as the plug-and-play module to help some small models achieve performance that rivals that of larger models.

Q2: According to the observation above, the authors' solution is limited to domain-specific small models

About this phenomenon in larger models: Yes, the phenomenon diminishes in larger models. We check the performance of Videochat [1] or Video-LLaVA [2], where "aug" denotes "augmentation".

VideoQA task with "accuracy" as evaluation metric

VSG task with "R@1, IoU=0.3" as evaluation metric

zero-shot VideoQA task with WebVid10M as training data

Also, we will add more analysis and experiments (e.g., few-shot setting) in the final version.

[3] Training-free Video Temporal Grounding using Large-scale Pre-trained Models, ECCV 2024

[4] Zero-Shot Video Question Answering via Frozen Bidirectional Language Models, NeurIPS 2022