Slot-VLM: Object-Event Slots for Video-Language Modeling

Thank you very much for your positive feedback regarding the idea, sufficiency of experiments, and the surprisingly good performance. We greatly appreciate your insights and suggestions and will thoroughly incorporate them into our revised manuscript. Below, we provide detailed responses.

Q1: Are the temporal slots really event slots? Since there are m spatial positions after downsampling, what if an object moves around from one spatial patch to another? Then the event cannot be effectively captured by the slots, since the temporal slot attention happens in the fixed spatial position along the temporal dimension.

A1: Thank you for the insightful question. We understand the concern. Our intention with the term “event slots” is to conceptually align these slots with events at a high-level, as mentioned in lines 203-204 of the manuscript: “Temporal slots aggregate the temporal tokens for explaining parts of the input, similar to identifying events. Thus, we name the learned slots as event-centric slots.”

We have considered this question during our design. To alleviate it, instead of learning event slots for each spatial position, we divide the feature map into 4x4=16 non-overlapped large spatial regions (downsampling), with each region corresponding to a large local region of 56x56 pixels when the frame resolution is 224x224. This allows us to observe the temporal dynamics of a large local region (56x56 pixels), which helps infer the evolution of part of an event within that region.

We acknowledge that this method might not perfectly capture the entire object dynamic, but it allows for a partial understanding of object evolution within a large local region. Through leveraging the learned slots from all 16 large local regions, the LLM can achieve a more comprehensive perception of global temporal dynamics.

The temporal slots aggregate semantically consistent temporal segments, akin to how spatial slots aggregate spatially consistent regions (object-centric slots). While the segments within a slot might not encompass a complete event, they are likely part of the same event. We will revise the manuscript to clarify this design consideration and improve the explanation of how event slots work in our model.

Q2: The visualization seems poor. It still looks similar to the visualization of ChatUniV. What is the key difference between Slot-VLM and ChatUniV?

A2: Thank you for your insightful question. Yes, the visualization of slot attention is still not very satisfactory. Actually, the unsupervised object-centric representation learning remains a challenging task in the field.

Our Slot-VLM significantly outperforms ChatUniV (see Table 1) due to two main reasons.

1. Slots can preserve more original information than clustering centers. Slot attention is trained to reconstruct the original features from slots. In contrast, ChatUniV uses the average token within each cluster to represent the corresponding cluster, where the averaging can result in a loss of fine-grained details, whereas the detailed visual information is crucial for accurate video understanding and reasoning.

2. Slots are learnable and can be jointly optimized within the entire framework, whereas clustering (being not learnable) in ChatUniV is task-agnostic. Our ablation study below shows that freezing the slot attention subnetworks at Stage 3 in our framework leads to a noticeable drop in performance when compared with our final scheme with slot attention jointly tuned (see Table F). Note that we will add the results that freezing the slot attention subnetworks at Stage 2 and Stage 3 once we obtain, which we guess they are lower than that freezing slot attention subnetworks only at Stage 3.

Table F: Ablation study on the influence of end-to-end optimization of the slot attention modules. Slot-VLM (slot frozen) denotes the slot attention modules are frozen in Stage 3.

Q3: Add related works regarding object-centric/slot-attention in videos.

A3: Thank you for the helpful suggestion and we will add the following discussion.

Some works have also explored object-centric learning in videos. VideoSAUR [A] extends the method of DINOSOUR [27] to video-based object-centric learning by incorporating temporal feature similarity loss, which introduces a motion bias for each frame. Fan et al. [B] propose an unsupervised method for open-vocabulary object localization in videos, where they run only once slot-attention on the entire spatial-temporal features to obtain video slots for all frames.

In contrast, we leverage slot attention to learn semantically decoupled tokens as the input to LLMs and investigate the effectiveness of aligning these ‘concept tokens’ with LLM input.

(Note [A] and [B] denote the referred two papers, respectively.)

Q4: Is it still necessary to use SlowFast?

A4: Yes, we had demonstrated the necessity of our two-branch design in line 309-320 of our manuscript. Table G shows that directly extending slot learning to spatial and temporal features increases memory and computation requirements, and the optimization difficulty. Table G: Ablation study on the effectiveness of joint spatial-temporal slots learning vs. our two-branch design. The FLOPs and number of parameters for the reduced token generation module are presented.

Thank you very much for your positive feedback and helpful suggestions. We greatly appreciate your recognition of the interestingness and cruciality of the idea, strong performance, meaningful and comprehensive ablation study, and comprehensive reproducibility details. We have carefully considered your valuable suggestions and comments and will incorporate them into our revised manuscript. Please find our detailed responses below.

Q1: My main concern is that the authors claim the temporal branch of Slot-VLM as the event branch. The slot attention focuses on local regions temporally, which makes it only capture low-level local motion instead of actual events with high-level semantics. Moreover, the visualization of temporal attention masks (Figures 8 and 10) can barely provide cues that the model can learn events-centric representation (L53, L55, L65). Thus, I believe the event-centric slot is overclaimed in the entire article. I hope the authors can clarify this.

A1: Thank you for your constructive comments. We will revise the descriptions for clarity. Our intention is to conceptually align event slots with events at a high level, as defined in lines 203-204 of the manuscript: "Temporal slots aggregate the temporal tokens for explaining parts of the input, similar to identifying events. Thus, we name the learned slots as event-centric slots." More precisely, we leverage temporal slots to aggregate semantically consistent temporal segments, akin to how spatial slots aggregate spatially consistent regions, termed object-centric slots. While these temporal segments within a slot might not form a complete event, they are likely part of the same event.

About local region: We understand your concern regarding the focus on local regions. This was a careful consideration in our design process. To alleviate this, instead of learning event slots for each spatial position along the temporal dimension, we divide the feature map into 4x4=16 non-overlapped spatial regions (downsampling), where each region corresponds to a large local region of 56x56 pixels when the frame resolution is 224x224. Observing the temporal dynamics of such a large local region (56x56 pixels) allows for partial inference of event evolution within the region, although it may not be perfect. Note that the temporal aggregation through slots is performed on CLIP features, which possess high-level semantics, thus enhancing the slots’ ability to capture high-level semantics. Moreover, by utilizing the learned slots for all 16 large local regions, the LLM can achieve a global perception of temporal dynamics.

About visualization: We recognize that the current technique using slot attention for forming objects (spatial) and action/event (temporal) segmentation are not yet satisfactory. In Figure 10 (a), the good news is that similar contents tend to be allocated to the same slot and different slots capture different contents. The patterns from slots (Figure 10 (a)) are much better than those obtained from Q-Former (see Figure 10 (b)). We acknowledge that our work is still far from forming idea events, but we have taken a small step towards that goal. The superior performance demonstrates the strong potential of our proposed idea of decomposing the semantics of video tokens to align with LLM. More joint efforts from the community are still needed to push this field forward.

I thank the authors for their response. My main concern about the clarity of "event slots" has not been addressed well, which is the major claim of the paper. As mentioned by reviewer KJVB, the paper only considers "events" that represent the temporal change within a patch. This would limit the proposed method to a simple domain of activities. For example, such a definition of event slots would not be able to handle a scenario where a subject manipulates objects with hands (e.g., Assembly-101 and Ego4D) or kitchen scenarios (e.g., Epic-kitchen) where the subjects and their hands move consistently. I would suggest that the authors revise the claim of the event slot as the term can be misleading. Moreover, from the rebuttal by the authors "slots is performed on CLIP features, which possess high-level semantics, thus enhancing the slots’ ability to capture high-level semantics" is not very convincing. The statement should be substantiated with some qualitative results that can demonstrate the event slots actually capture high-level semantics, as the current ones do not really show meaningful semantics.

Thank you very much for your constructive suggestions, and positive feedback on the novelty and interestingness of our proposed concept, the ability to generate interpretable attention maps, the comprehensive comparisons, and the clarity of presentation. The detailed responses are provided below.

Q1: Include a comparison of the different number of video tokens used among the different video-language models as well as the number of trainable parameters during each of the training stages.

A1: We will include these in our revision. Table B below shows the performance gain is not due to the final number of video tokens, where our Slot-VLM uses only 192 tokens that is much smaller than some other models.

Table B: The number of video tokens used. ‘Varied length’ denotes the number varied with the video length. ‘-’ denotes we did not find information from their papers and will investigate their code in future to fill. * denotes the likely number based on the paper’s description.

Different models have different numbers of training stages and trainable parameters per stage. Most papers lack detailed information. For simplicity, we present the trainable parameters for each stage of Slot-VLM and the likely numbers for VideoChat2 based on the paper's description in Table C. Our Slot-VLM uses far less trainable parameters than VideoChat2 and requires less computing resource in training.

Table C: The numbers trainable parameters (M: million; B: billion) in each stage for VideoChat2 and our Slot-VLM.

Q2: It is also unclear how scalable such an approach will be in handling much longer videos.

A2: Our approach is directly applicable to videos of a few minutes. We have validated our framework on ActivityNet-QA (see Table 1), where the average length of video clips is 111.6 seconds, and the maximum length is 285 seconds, which is comparable to the reviewer referred EgoSchema (180 seconds).

We also tested a 500-question subset of EgoSchema. Table D shows that our Slot-VLM achieves superior performance, which outperforms Video-LLaVA by 7% in accuracy.

Table D: Performance (accuracy) comparison on the subset of Egoschema.

For handing even longer videos, our framework can be extended. For example, for a three-hour video, we could partition the video into chunks, each of $\tau$ seconds, summarizing dense video features into $N_s$ object-centric and $N_f$ event-centric tokens for each chunk. These tokens can then be concatenated sequentially as input to LLM for inference. At the setting of $N_s+N_f=192$ tokens per chunk and $\tau=96$ seconds (1fps) duration per chunk, we equivalently have 2 tokens per frame on average, facilitating hour-long video handling. Note that LLaMA-VID (Li et al., 2023b) also encodes each frame into two tokens but overlooks the exploration of temporal correlation (where ours outperforms LLaMA-VID in three benchmarks in Table 1). This strategy ensures that the total number of video tokens is proportional to the video length in a manageable and tolerable way. More efficient strategies for aggregating spatiotemporal relationships over longer videos will require further research and future efforts.

Q3: Additional ablation over the low frame sampling rate used in the Object-Slots Branch as well as the stride used for pooling in the Event-Slots Branch.

A3: We conducted ablation studies using single branch settings. For the Object-Slots branch, we tested three frame sampling rates: 4 frames (Object-Slot-VLM (4 frames)), 8 frames, and 16 frames per video. For the Event-Slots branch, we tested four pooling strides to achieve spatial resolutions of 2x2 (Event-Slot-VLM (2x2)), 4x4, and 8x8.

Table E shows the results. Increasing the sampled frames or spatial resolutions improves performance but increases the number of video tokens. By default, we use 8 sample frames and a 4x4 spatial resolution to balance complexity (192 tokens in total) and performance.

Table E: Ablation studies on sampling different number of temporal frames for the Object-Slots branch, and on using different spatial resolutions for the Event-Slots branch.

For the schemes in our response A2, the paper for ViperGPT is "ViperGPT: Visual Inference via Python Execution for Reasoning".

The paper for LLoVi is "A Simple LLM Framework for Long-Range Video Question-Answering".

The paper for mPLUG-Owl is "mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality".

We sincerely appreciate your positive feedback on our insights into spatial-temporal modeling in MLLM, the main motivation, and the effectiveness of method, as well as the quality of our writing. We have carefully considered your valuable comments and suggestions and are committed to incorporating them into our revised manuscript. Please find our detailed responses below.

Q1: The proposed Slot-VLM is tested on open-ended MSRVTT-QA, MSVD, and ActivityNetQA with automatic ChatGPT-based metrics. Although the metric is applied by some previous works, I don’t think this is a stable, reliable, and explainable enough evaluation which may vary a lot across GPT versions (check conclusion in [1]) and may generate different responses even for the same input.

A1: We acknowledge your comment and will include the comparisons on multi-choice QA benchmarks in our manuscript (see Response A3 below).

Q2: Also MSRVTT-QA/MSVD/ActivityNetQA is not designed from an object interaction perspective, which may not be effective enough for validating the object-centric design.

A2: Thank you for your comment and suggestion. We explored additional datasets for validation as discussed in Response A3 below.

Note that our framework is expected to generally work well, since in general each image in a video is a capture of the world consisting of objects and backgrounds, where object-centric representations can be considered as the basic units. This can also be intuitively explained by human visual reasoning where object representation acts as a basic and fundamental representation. For humans, visual signals are processed initially through the primary visual cortex (V1) and subsequently integrated into higher-level visual processing areas, resulting in the formulation of complex visual object representations. Such high-level object representations together with brain-stored knowledge are then used for logical reasoning in brain. Similarly, our Slot-VLM generates object-centric and event-centric representations to provide the high-level vision source for effective LLM reasoning, where object-centric design would be generally helpful. The strong performance of our Slot-VLM (Table 1) also demonstrated the effectiveness of our design.

In addition, for the three datasets, we found that there are plenty of video-question pairs that are related to querying the states/actions of objects and relations. For example, what is a dog doing? What’s the shape of the table? What are the animals that appear in the video? What is behind the person sitting in the video? Our object-centric design provides an effective bridge for visual and language modeling.

Q3: Based on the previous two points, I would suggest testing on recent multi-choice QA benchmarks, including STAR [2] / NExT-QA [3] / Egoschema [4], which are annotated with clear answer choices, and designed from a human-object interaction view. And compared with recent model works like Sevila [5], MIST [6], LRR [7], GF [8], which are also based on query-former, or iteratively attention on regions in the video.

A3: Thank you for the helpful suggestions and we will add the comparisons in our revision.

For EgoSchema, we have tested the performance on the 500-question subset and Table A-1 shows the results. Slot-VLM significantly outperforms Sevila by 30% and outperforms the comparable 7B model Video-LLaVA by 19% in accuracy.

Table A-1 Performance (accuracy) comparison on the subset of Egoschema.

[9] ViperGPT: Visual Inference via Python Execution for Reasoning

[10] A Simple LLM Framework for Long-Range Video Question-Answering

For STAR, Table A-2 shows the comparisons. The referred four works [5-8] report the in-domain accuracy where training is performed on STAR. In contrast, without accessing STAR during training, our model is tested in zero shot manner. Our Slot-VLM achieves competitive performance with generalizable models of Flamingo-9B and InternVideo. Note that InternVideo uses a much larger number of videos (12 million) than ours, whereas our model uses only 100K videos for training.

Table A-2 Performance (accuracy) comparison on STAR.

For NExT-QA, Table A-3 shows our Slot-VLM is competitive to VideoChat2. We believe using more training videos like VideoChat2 (1.1 million) would further enhance performance.

Table A-3 Performance (accuracy) comparison on NExT-QA.

[11] InternVideo: General Video Foundation Models via Generative and Discriminative Learning

[12] Mistral 7B

[13] Verbs in Action: Improving verb understanding in video-language models

[14] Understanding Long Videos in One Multimodal Language Model Pass

[15] Language Repository for Long Video Understanding

[16] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark