SlowFast-LLaVA: A strong training-free baseline for video large language models

We thank the reviewer for their insightful comments. We address your comments in the following.

---

Q1: The paper suffers from excessive repetition. Given the simplicity of the core idea, it seems unnecessarily stretched to fill 10 pages and could be more concise. Additionally, table captions, such as in Table 1, lack added value; they simply reiterate the table content without providing further insights. Similarly, the experiment descriptions merely repeat the accuracy values from the tables, without offering additional intuition or interpretation. Furthermore, the prompt aspect could be explored in greater depth, as it represents an additional axis to consider in zero-shot adaptation of vision-based large language models (LLMs) for videos.

A1: Thank you for these suggestions. We will improve our paper writing in the revision. Specifically, we will

• prune the "repeating of accuracy values" from the tables and experiments
• summarize experimental results and give more result interpretations
• report more results and insights about our better zero-shot ability, as suggested by Reviewer Hn2a
• report more results and discussions about our advantage on long-form video understanding, as suggested by Reviewer Hn2a and yZMr
• discuss the possibility to extend our SlowFast design on other Image LLMs and report results, as suggested by Reviewer jACj and yZMr

Please let us know if you have other specific suggestions and we will take care of them together in the revision.

---

Q2: It seems that the prompt aspect in zero-shot adaptation could add substantial value if further explored. Have you considered additional prompt engineering or variations in the prompts to evaluate their impact on the model’s performance?

A2: Thank you for this suggestion. Our prompt exploration is intended to give helpful insights for future work on how to systematically design their own prompts in zero-shot domain adaptation. We evaluated the importance of prompts from task, input, and answer perspectives, and showed their impacts on various VideoQA tasks. We agree that additional prompt engineering possibly further improves the performance of our model and inspires the design of future work. We will discuss more about our prompt experiments in the revision.

We thank the reviewer for their insightful comments. We address your comments in the following.

---

Q1: Experiments were conducted only on LLaVA-NeXT (despite including 7B and 34B models), with no additional results from other open-source LLMs provided.

A1: We focus on LLaVA-style Image LLMs since they are among the most popular model architectures. But we agree that validating our SlowFast idea on other Image LLMs would be beneficial. As suggested, we integrate our SlowFast inputs to Phi-3 Vision (named SlowFast-Phi-3), which is the state-of-the-art mobile-size (4.2B) Image LLM with pixel shuffle structure, and evaluated it on various benchmarks. We directly use the design choices and hyperparameters of SF-LLaVA-7B without ablation studies. To better understand the results of this model, we also implement IG-VLM for Phi-3 Vision as baseline (named IG-Phi-3), which is the state-of-the-art method of adopting Image LLMs for video tasks. The table below shows that SlowFast-Phi-3 achieves convincing results on all benchmarks and outperforms Image Grid by a clear margin.

---

Q2: The quantitative results in Figure 5 lack the results without the SlowFast design.

A2: Thanks for this suggestion. Here we report the answers of LLaVA-7B which simply takes as inputs 12 video frames in a sequence. This baseline has similar number of tokens as SF-LLaVA-7B in paper's Figure 5. By comparing the results of these two models, we observe that i) SF-LLaVA-7B gives more details than LLaVA-7B in Question 1, ii) LLaVA-7B fails to understand the relative order of different events without the SlowFast design in Question 2, and iii) SF-LLaVA-7B and LLaVA-7B both miss the "opening refrigerator" event due to the uniform frame sampling in Question 3. We will include these results in our revision.

• Question 1: What is happening in this video?

• LLaVA-7B: In this video, a person is preparing a coffee using a coffee machine. The scene shows the coffee machine being filled with coffee grounds, the coffee being poured into a cup, and the coffee being stirred. The setting seems to be a kitchen or a similar environment with various kitchen items visible in the background. The overall content of the video is focused on the preparation of a hot beverage, likely coffee, in a domestic setting.

• Question 2: When does the person wash the portafilter?

• LLaVA-7B: The exact time when the portafilter is being washed cannot be determined from the video, but it is clear that the person is engaged in this task during the coffee-making process.

• Question 3: Does this person open the refrigerator?

• LLaVA-7B: This person is not shown opening the refrigerator in this video.

---

Q3: Can the proposed method work for particularly long video understanding, such as LVBench[1]?

A3: As suggested, we evaluated SF-LLaVA on LongVideoBench. We also evaluated IG-VLM as our training-free baseline by using their official codebase. The table below shows that SF-LLaVA outperforms IG-VLM by +3.5% on LongVideoBench and achieves even better performance than PLLaVA (fine-tuned on VideoChat2 with ~900K training samples), using both 7B and 34B LLMs. We will report these numbers in our revision and give more discussions.

Table 1: The results of IG-VLM are reproduced by using their official codebase and the results of PLLaVA are copied from the official LongVideoBench leaderboard.

---

Q4: The paper indicates that it lacks the capability to detect the precise start and end times. Can fine-grained frame-level textual descriptions be introduced to enhance the spatio-temporal understanding of videos?

A4: Thanks for this great suggestions. We believe using frame-level description can improve the spatial-temporal understanding of a video, which is already suggested by many existing methods such as LLoVi and VideoTree. These information will help the Video LLM to better understand what is happening in each frame and precisely detect a target event. On the other hand, we also believe incorporating the timestep of each input video frame is important for detecting an event's start and end times, especially when we want the outputs to be in seconds, not the frame number. We will give more discussion about this topic in our final revision.

We thank the reviewer for their insightful comments. We address your comments in the following.

---

Q1: I suggest authors provide evaluations not only on captioning benchmarks, such as MSVD, MSR-VTT and so on, but also on benchmarks specifically for video-mllm, such as video-mme, mvbench, videovista, MLVU and so on, which could provide more comprehensive video-llm performance comparation, the evaluation resutls could provide important new insights about the model's performance.

A1: As suggested, we evaluate SF-LLaVA on VideoMME and MVBench as shown below. We observe that SF-LLaVA-34B achieves very convincing results compared with other training-based models in the leaderboard, such as InternVL-Chat-V1.5. To compare with training-free baselines, we evaluated IG-VLM by using their official codebase. We find that SlowFast-LLaVA outperforms this strong baseline on these benchmarks by a clear margin.

Due to the limited time, we won't benchmark VideoVISTA and MLVU in this rebuttal. We will cite them in the paper revision and use them in our future work.

---

Q2: the two-stream method is only applied on llava-next image-llm, if authors could give proof that this idea could be generalized to other image-llm models, the two-stream token aggregation design is more persuasive and thought-provoking for futher video-llm research.

A2: We focus on LLaVA-style Image LLMs since they are among the most popular model architectures. But we agree that validating our SlowFast idea on other Image LLMs would be beneficial. As suggested, we integrate our SlowFast inputs to Phi-3 Vision (named SlowFast-Phi-3), which is the state-of-the-art mobile-size (4.2B) Image LLM with pixel shuffle structure, and evaluated it on various benchmarks. We directly use the design choices and hyperparameters of SF-LLaVA-7B without ablation studies. To better understand the results of this model, we also implement IG-VLM for Phi-3 Vision as baseline (named IG-Phi-3), which is the state-of-the-art method of adopting Image LLMs for video tasks. The table below shows that SlowFast-Phi-3 achieves convincing results on all benchmarks and outperforms Image Grid by a clear margin.

Due to the limited time, we will leave the implementation of SlowFast into other suggested Image LLMs in the revision.

---

Q3: Althogh training-free is a good feature for the method, a training-based model could gain a more powerful performance, just as LLaVA-NeXT-Video-DPO is better than SF-LLaVA(Both built on LLaVA-NeXT-Image ).

• I suggest authors discuss specific use cases or scenarios where a training-free approach might be preferable despite potentially lower performance.
• I also suggest authors elaborate on potential trade-offs between training-free and fine-tuned approaches in terms of factors like deployment flexibility, computational requirements, or ability to handle novel domains.

A3: Thanks for this great question. First, training/fine-tuning Video LLMs on large-scale video datasets requires high computation cost. For example, fine-tuning PLLaVA-7B with lightweight parameters using LoRA requires about 35 hours on 8 A100-80GB GPUs. This resource cannot be afforded by many small company and university labs. On the other hand, it also involves video data collection from a legally approved source, human/machine annotation, related training infra support and quality verification. Thus we believe developing training-free Video LLMs can be beneficial for our community. Second, fine-tuning Image LLMs on video will be unavoidable to hurt their results on image tasks. Video-LLaVA tries to solve this by using separate image and video encoders, which increases the model size in the deployment. Another option is to train a unified model, such as LLaVA-OneVision, but this model will possibly require an even larger mixed image and video datasets (e.g., 5M+ samples), multi-round stages, and even more computation resources. Since the training-free methods are never fine-tuned on video data, we can simply deploy a single model on server/device working for different application scenarios. Third, as mentioned by Reviewer Hn2a, since existing Video LLMs are usually fine-tuned on in-domain video datasets, they likely have poor generalization capabilities on other domains. Training-free methods, such as SF-LLaVA and IG-VLM, show great zero-shot video question-answering capabilities on various benchmarks.

We thank the reviewer for their insightful comments. We address your comments in the following.

---

Q1: Limited novelty compared with existing methods such as IG-VLM, or PLLaVA. It has been shown that simply concatenating frames (or frame features) gives a good MLLM without or with only a little training. On top of these existing works, it is straightforward to see or expect that a cleverer selection of feature such as adopting the SlowFast idea can work a little bit better.

A1: First, we understand your concern. However, the significance of a research paper extends beyond the introduction of novel training methods. We believe that the contributions of our work are substantial, and the findings presented in the paper are insightful. Specifically, we present a simple yet effective SlowFast input design tailored for Video LLMs, which achieves surprisingly competitive performance. This approach can serve as a strong baseline for all future research in the field.

Second, compared with IG-VLM and PLLaVA, SF-LLaVA uses a totally different input structure. i) IG-VLM assembles multiple video frames into an image grid and inputs it as a high-resolution image into the MLLM. Limited by the image grid size, IG-VLM is only effective to consume 6 frames (their performance decreases when using more frames). ii) PLLaVA simply concatenates the pooled video features into a sequence without proper temporal modeling designs, thus cannot further improve the performance using more than 16 frames. iii) Our two-stream SlowFast inputs can effectively capture long-range context while preserving detailed spatial semantics. As a result, SF-LLaVA takes over 50 frames as inputs and achieves convincing results on various video tasks.

Third, we respectfully disagree that SF-LLaVA works only a little bit better than simply concatenating frames. i) SF-LLaVA-7B outperforms IG-VLM-7B on all 8 video benchmarks, especially by +11.4% on EgoSchema and +5.7% on TGIF-QA. ii) As a training-free method, SF-LLaVA achieves on-par performance with PLLaVA, which is fine-tuned on VideoChat2 with 15 video datasets and ~900K training samples. Please also note that VideoChat2 contains raw videos from ActivityNet and TGIF, thus these results are not out-of-domain as SF-LLaVA. iii) To strengthen our superiority on long-form video understanding, we evaluate SF-LLaVA on LongVideoBench that contains videos up to an hour long. The table below shows that SF-LLaVA outperforms IG-VLM by +3.2% and +4.1% on LongVideoBench using 7B and 34B LLMs, and achieves even better performance than PLLaVA.

Table 1: The results of IG-VLM are reproduced by using their offical codebase and the results of PLLaVA are copied from the official LongVideoBench leaderboard.

---

Q2: Limited performance gain on top of the baselines such as IG-VLM, as shown in Table 2 (e.g. +1.1% on NextQA), as well as in the ablation, line 413, where adding the fast path improves marginally on top of the 12 frame slow only model.

A2: we respectfully disagree our performance gain on top of IG-VLM is marginal. First, SF-LLaVA consistently outperforms IG-VLM on all 8 video benchmarks, using both 7B and 34B LLMs. Second, NExTQA mostly contains short videos (44s on average), which cannot fully show SF-LLaVA's superior capability on modeling long-range context. We kindly remind that SF-LLaVA-7B outperforms IG-VLM-7B by +11.4% on EgoSchema, which contains long-form videos with 180s duration on average. Besides, as shown in above Q1, SF-LLaVA outperforms IG-VLM on LongVideoBench by over 3.5% on average. Third, Our experiments are completely under the zero-shot evaluation setting. Additionally, considering that adding the Fast pathway only slightly increases the number of input tokens, we softly argue that even achieving +1% improvement on all benchmarks is not trivial. We will report all these results and discussions in the revision.