Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding

We are thankful for your valuable comments and we provide a detailed response to your questions:

Q1: The representation of spatial concept features by the group tokens is unclear.

A1: To make it more clear, group tokens could be regarded as a number of learnable cluster centers. These group tokens first aggregated semantically similar video tokens through our video encoder, and then were aligned with noun tokens via spatial grounding loss. Note that the ultimate group tokens to be grounded to noun tokens were conditioned on the given video, and there was no 1-1 matching between initial group tokens and noun concepts.

Q2: The compared methods are outdated and do not compare to state-of-the-art methods. The performance of the current methods on several tasks is inferior to many recent works.

A2: In the paper, we compared our method with VIOLETv2 [1], which was published at CVPR 2023, and it was one of the most recent works about video-language pre-training. Here we included one more recent method All-in-One [2] published at CVPR 2023 as well. We can observe in the table below that our S-ViLM still achieved competitive or even better performance in zero-shot text-to-video retrieval on MSR-VTT.

Moreover, for methods such as UCOFIA (ICCV 2023) [3] and Aurora (NeurIPS 2023) [4] they were considered as contemporaneous papers. For methods such as X-CLIP [5], and Ts2-net [6], they took advantanges of more complicated designs including much larger-scale pre-training dataset, better initialization such as CLIP-based video encoder, large language models and the like. Thus, we selected the most appropriate baselines that were reasonable and up-to-date at the same time. We also modified the part of related work to discuss those recent methods comprehensively. For TAL task, since we followed previous protocols to apply G-TAD with our frozen features, we chose the most appropriate baseline methods as well.

Q3: The claimed cut-and-paste operation has been widely used in previous studies.

A3: Although the cut-and-paste operation has been widely used in the image domain [7], it was under-explored in the video domain. To the best of our knowledge, our S-ViLM was one of the earliest attempts to leverage cut-and-paste for videos, especially for video-language pre-training. It was not trivial to adapt this operation to videos in our method, in which the cut-and-paste was not only a form of data augmentation, but also a key component in our temporal grouping loss when constructing background and foreground training examples.

Q4: The text may involve multiple concepts. How can we extract k noun tokens from all sentences?

A4: It should be clarified that we extracted $k$ noun phrases for each caption, instead of $k$ noun phrases for all captions in the dataset. We use spaCy to extract all noun phrases from the caption. For each caption, if the number of noun chunks are greater than $k$, we just keep $k$ of them randomly; otherwise, we choose all the nouns, and the rest of candidates are sampled from the pool with replacement.

Re: "cut-and-paste operation":

Thank you for your prompt reply with more details and references about the cut-and-paste operation. We acknowledge that they are important related works and we would like to include the following discussion in our revised manuscript. It should be clarified that we did not claim the cut-and-paste operation as our innovation, instead our contribution is the proposed intra-clip temporal grouping loss with the cut-and-paste operation, which serves as an essential part to introduce scene changes and promote more temporal-aware feature learning.

Here we would like to discuss the technique proposed in our paper and those methods you listed:

1. "Unsupervised Pre-training for Temporal Action Localization Tasks, CVPR,2022." How to use synthetic videos after the cut-and-paste is different between PAL and our method. Specifically, in PAL, clips from one video are pasted onto the other two different videos, and the goal is to align the features of pasted pseudo action regions from two synthetic videos via contrastive learning. In contrast, in our S-ViLM, we teach the model to distinguish background and foreground features within one synthetic video after cut-and-paste and formulate it as a binary classification problem for each token. Besides, compared with PAL involved in only one modality, the cut-and-paste in our method tightly integrates in video-language constrastive learning as a form of data augmentation. As shown in Eq. (6) and (7), both the grounding loss and the global contrastive loss are weighted based on the cut-and-paste operation, allowing the model to learn alignment between video and text features better.
2. "Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning. NeurIPS 2022." Technically, this paper does not leverage the cut-and-paste operation. The focus of this paper is pre-training on long-form videos. The pre-traning dataset with long-form videos are constructed by consecutive clips from an existing dataset, and all videos are real ones from YouTube and there are no synthetic videos obtained from the cut-and-paste.
3. "Boundary-sensitive Pre-training for Temporal Localization in Videos. ICCV 2021.". Our S-ViLM differs from this work in both the pre-training objective and strategies of the cut-and-paste operation. For objectives, the paper designs a novel boundary-sensitive pretext (BSP) task with synthetic videos after the cut-and-paste. In particular, they construct a synthetic video by merging two videos from four types: same class, diff class, same speed and diff speed, and the training objective is to predict the merging type given the synthetic video. Instead, in our S-ViLM, we look at more fine-grained clip features and predict whether the individual clip is from the background or the foreground. As for the cut-and-paste operation, strategies in BSP depend on datasets with annotation available, while S-ViLM is fully self-supervised. In addition, as mentioned in 1., our method is about video-language pre-training and not only video representation learning.
4. "Temporal perceiving video-language pre-training. Arxiv 2023." The cut-and-paste in this paper (TemPVL) applies in the feature domain instead of the raw video domain, which means that original videos are kept intact without any modifications. They only conduct video token merging before the multi-modal encoding process, and their motivation and goal is different from ours: TemPVL tries to accurately perceive temporal boundaries in videos conditioned on the text description through a multi-modal fusion while our paper is to learn temporal-aware features via temporal grouping independent of text as shown in Figure 2 in the main paper.

Thank you for your positive feedback and below we present our response to your question:

Q1: The novelty may be somewhat weak.

A1: Thank you for allowing us to elaborate on our work's novelty. We will explain it as follows:

a) Impact of our work: As mentioned in Section 1, most of existing video-language pre-training frameworks focused on learning holistic representations to match a <video, caption> pair while fine-grained information such as region-object correspondences, or scene/action changes along the time in a video was under-explored. Nevertheless, such regional or temporal fine-grained information had been demonstrated to play a vital role in localization and reasoning tasks. Incorporting them into the pre-training paradigm was promising to further enhance the model's understanding and reasoning capabilities in both video and language domains. Thus, in our paper, we investigated how to exploit instrinsic spatiotemporally fine-grained structures contained in the pre-training dataset, and demonstrated that making use of structured video-caption interactions indeed contributed to more expressive spatiotemporal features.

b) Our technical contributions: To leverage spatiotemporally fine-grained structures introduced above, we proposed two novel loss designs: intra-clip temporal grouping loss, and inter-clip spatial grounding loss. In particular, we adopted a cut-and-paste operation to introduce scene changes into videos. It was more frequently used in the image domain and we successfully adjusted the operation to our scenario for learning more temporal-aware features, which was a non-trivial effort. As to inter-clip spatial grounding, we proposed a soft grounding loss to capture fine-grained correspondences in a self-supervised learning, without any <region, object> annotations. We presented both quantitative and qualitative results in the paper to showcase the effectiveness of considering fine-grained spatiotemporal structures during the pre-training stage.

Thanks for your positive and valuable comments and below we provide our response to your questions:

Q1: Extra knowledge about determining noun chunks from spaCy is introduced during pretraining.

A1: Noun chunks extraction primarily involves identifying and isolating the basic entities or objects within a text. It does not introduce new information but rather extracts and organizes existing information. This lightweight process is a widely-used preprocessing step in existing NLP or vision-language tasks. Specifically, noun and verb extraction have been leveraged in previous methods such as MCQ [1]. Besides, there are also some methods resorting to heavier pre-trained models in their learning processes. For example, ALPRO [2] made use of frozen CLIP encoders in their prompting entity modeling, and DemoVLP [3] utilized the off-the-shelf object detectors. Thus, we believe it is a comparatively fair comparison when only a lightweight noun extraction tool was involved in S-ViLM.

Q2: The concern regarding benefits from pretraining set are not fully addressed.

A2: Due to the restricted data access policy, the commonly-used WebVid was unavailable to us. Instead, we are running an experiment to pre-train MCQ, one of the best baselines, on VideoCC dataset. We will provide the results in the response and update the paper when the experiment is completed.

Q3: Discussion on Eq.(3)

A3: We want to clarify that the similarity in Eq.(3) did not prevent n_k being similar to many regions. Specifically, it was possible that a noun was grounded to multiple regions, or even as you mentioned, to all regions uniformly. In the latter extreme case, the noun might be some irrelevant object that did not appear in the video, or it was likely to occupy the whole frame, e.g., a scene full of fishes/flowers. Moreover, since there was no annotation for <region, object> correspondence in the pre-training dataset, we applied the soft grounding similarity shown in Eq.(3), and such a design aligned a noun with the visual contents in the video in a self-supervised manner. On the contrary, if a hard grounding was adopted here, i.e., $\max{s(\mathcal{G}, n^k)}$, the noun would be biased towards the specific region and hurt the performance significantly if such a pair was wrong. Instead, our soft version avoided penalizing regions that cannot find any relevant nouns, and always took all the regions into account. The similar idea was also leveraged in [4] about open-vocabulary image segmentation, where <region, object> annotations were absent as well.

Q4: Citing baselines in the Table to save space.

A4: Thanks for your suggestion. We have modified the paper accordingly.

Q5: Action recognition results on Something-Something.

A5: Thank you for your suggestion. We are running experiments on Something-Something and will update the results in the response as well as in the paper when it has finished.

Q6: How are start and end clip indices s, e sampled?

A6: For the clip sampling procedure, we first uniformly sampled the duration of the foreground video $d$ from $[N_t/2, N_t)$ to guarantee it is the majority of the blended video. Then we sampled the start index $s$ from $[0, N_t-d)$, and the end index $e$ was computed naturally as $e = s + d$. We included this detail in our latest version.

[1] Ge, Yuying, et al. "Bridging video-text retrieval with multiple choice questions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Li, Dongxu, et al. "Align and prompt: Video-and-language pre-training with entity prompts." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Cai, Guanyu, et al. "Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval." arXiv preprint arXiv:2203.07720 (2022).

[4] Ghiasi, Golnaz, et al. "Scaling open-vocabulary image segmentation with image-level labels." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

We appreciate your constructive reviews for our paper. Here are our detailed responses to your questions:

Q1: There are works like LAVILA that takes temporal and patch-level information into account.

A1: Thanks for pointing out the relevant method. The focus of LAVILA was to generate more diverse and dense captions for video dataset through automatic annotation from large language models. Compared with LAVILA which incorporated temporal information implicitly, our S-ViLM focused on pretraining method designs, with novel losses, intra-clip temporal grouping and temporal cut-and-paste augmentation, to improve video representation learning. We expect that integrating these two directions can lead to further improvement and we have added a discussion of LAVILA in Related Work in the modified paper.

Q2: Some design choices are not obvious but no ablation study was presented in the paper.

A2: (1) For Eq. (2), we tried BCE loss in our preliminary experiments and it achieved similar performance to that of MSE loss presented in the paper. This was an implementation choice on how to deal with the binary classification of foreground and background clips. Since both MSE and BCE worked well in distingushing foreground and background clips and no obvious difference was observed, we selected a relatively simple MSE loss for temporal grouping. In Eq. (1), $z_i^\text{clip}$, $z_i^b$ and $z_i^f$ were $L_2$ normlized, and it was actually cosine similarity instead of pure dot product following previous protocols. Softmax with a temperature factor $\tau=0.05$ was applied. We highlighted this correction in blue in the paper. (2) The idea to leverage existing region proposal networks has been implemented in DemoVLP [1], and we compared it with our S-ViLM in Table 1. It can be observed that our method significantly outperforms DemoVLP for text-video retrieval on MSR-VTT. On the other hand, using off-the-shelf region proposal networks or segmentation networks may not be a good choice due to following reasons: a) applying the model to a large-scale video dataset was time-consuming; b) the model was trained on the dataset from different domains and might generalize poorly on video datasets; c) we cannot further train the model for adaptation to our data because of lack of region annotations in the pre-training dataset. Instead, adopting grouping blocks to aggregate semantically similar regions in a self-supervised manner contributed to more dynamic and flexible learning, and was a more suitable and efficient choice in video-language pre-training without region annotations.

Q3: The writing could be further improved for enhanced clarity.

A3:

• Sorry for the confusion about "interaction" here. We want to clarify that modality fusion and interaction referred to the same concept that different modalities were further encoded together rather than separately as in the step a). In particular, we can either use cross-attention or feed a long sequence by concatanating different modalities into an additional encoder. In our paper, we skipped this step and simply projected representations from video and language into the same latent space for the subsequent pre-training.
• Region-object groundingness indicates the alignment between a region in the video and an object described in the caption. Specifically, as illustrated in inter-clip spatial grounding in Figure 1, the word “pins” in red corresponds to the red region in the left frame. The sentence should be rephrased as: "Group tokens aggregate semantically similar video tokens via grouping blocks and are then aligned with object concepts by spatial grounding. It promotes region-object groundingness, which indicates the alignment between a region in the video and an object in the caption, e.g., as illustrated in Inter-clip Spatial Grounding in Figure 1, the red region corresponds exactly to the word “pins” in red."
• Thanks for your suggestion. We have corrected the grammar error in our latest version.

[1] Cai, Guanyu, et al. "Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval." arXiv preprint arXiv:2203.07720 (2022).