SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding

Thank you for your review and helpful feedback!

Q1: Have you considered augmenting the uniform frame sampler with a learned importance function or keyframe detector to address the limitation that "a ten‑minute video and a one‑hour video have the same number of input frames"? Incorporating a lightweight attention or reinforcement‑learning module could dynamically prioritize salient segments.

A1: Great question! We plan to develop better frame samplers based on the Slow-Fast idea to further improve the performance. Specifically, the model can first process the entire video using only "Fast frames" with low-resolution (e.g., 1x1 token), which densely captures the long video with high efficiency. Based on the question, the model should output which part of the long video needs to be processed again in detail. Then, the model can "look twice" into the "Slow frames" with high resolutions to generate the final answer. This algorithm can be realized by RL in our future work.

Q2: What are the inference latency and memory footprints of the 1B and 3B models on a representative edge device (e.g., a smartphone GPU)? Detailed profiling would substantiate claims of mobile‑friendliness.

A2: Thanks for this suggestion. We report the runtime of our model on a H100 GPU in the paper's Table 6. Based on existing studies [1], 7B or smaller VLMs can be deployed to mobile devices with further optimization, such as model compression and quantization. However, please understand that the effort to deploy our model to a smartphone device is significant and requires a lot of engineering time. We have to leave this study for future work.

[1] MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning. Zhang et al. ICLR 2025.

Q3: While the Slow-Fast mechanism delivers efficiency, its comparative performance against emerging memory‑augmented or dynamic token‑merging techniques is not explored; this narrows the paper’s experimental scope.

A3: Thanks for this suggestion. In the paper's Table 6, we tried to compare Slow-Fast with widely used video projectors, where Qformer uses dynamic token merging and Perceiver Resampler is considered as an example of memory compression [2]. Please let us know if there is any particular technique that we should compare with. However, considering the limited time of rebuttal and training models require large computational resources, we may have to include the results in our paper revision.

[2] Long Short-Term Transformer for Online Action Detection. Xu et al. NeurIPS 2021.

Q4: The presentation, though generally clear, is dense with acronyms and heavy math in the architecture section, which may impede accessibility for readers less familiar with prior LLaVA and SlowFast literature.

A4: As suggested, we will polish our architecture section. In particular, we will move the heavy math into the appendix and give a more intuitive description of our model design.

Thank you very much for your review and kind words!

Thank you for your feedback!

Q1: The SlowFast Video Projector, which is the main contribution of this paper, is exactly the same as the SlowFast projector proposed in the previous work SlowFast-LLaVA [1]. No architecture modification is applied. The authors experiment with different instantiations of the SlowFast idea termed ISF, but experiments prove that this is not effective, and they used SlowFast-LLaVA's original instantiation. In conclusion, the SlowFast Video Projector should not be the main contribution/innovation of this paper. However, the majority of the paper's content and ablation experiments are about this part.

A1: We understand your concern, but we did not claim the architecture of the SlowFast Video Projector as our main contribution as discussed in the last paragraph of Introduction. Instead, our focus is to incorporate the SlowFast mechanism into an SFT framework and demonstrate its effectiveness on long-form video understanding. We believe this is valuable because, although SlowFast-LLaVA [1] first proposes to use slow-fast inputs in Video LLMs as a training-free model, the optimal way to implement this idea under a SFT framework was not explored. Our paper is the first to conduct such studies and achieve competitive results. Moreover, our contribution is not limited to the SlowFast Projector. Compared to existing Video LLMs, our models achieve state-of-the-art level performance across various benchmarks by using a simple training recipe and only open-source datasets. Considering that both training Video LLMs and collecting large-scale video datasets are very costly, we believe our paper can benefit the community beyond model novelty, especially for many small companies and university labs, to advance research on long-form video understanding.

Q2: The curated data mixture is a main contribution. However, the description of this part is quite brief and does not introduce any innovative techniques for data selection or filtering. A more detailed explanation and supporting experiments are needed for this section.

A2: Thanks for this good suggestion. We tune our data mixture through comprehensive empirical studies. We first build a base mixture that all existing Video LLMs are using. This includes large-scale video datasets with general captioning and QA labels. Then, we keep adding datasets (or their subsets) that are proposed for different video tasks, such as long-form understanding, temporal reasoning, and detailed spatial and temporal perception. We mainly evaluate the new mixture on Video-MME since it tests a model from various perspectives. We decide to add a new dataset if we observe non-marginal improvements. As suggested, our paper revision will give a detailed process of how we build our data mixture with experimental results as support.

Q3: Comparisons between the proposed two-stage training recipe and more complex recipes (e.g., VideoLLaMA 3 [2]) should be included.

A3: This is a good question. First, please understand that we do not have sufficient training resources to conduct complex training recipes, especially during the rebuttal. This limitation likely affects the video research of many other research labs as well. Second, most state-of-the-art Video LLMs, such as VideoLLaMA3, use large-scale in-house datasets to substantially improve their performance. Thus, we cannot even reproduce their recipe by following their papers.

Please kindly consider that the significance of a research paper extends beyond the introduction of novel training methods or architectures. We believe that achieving state-of-the-art performance using a two-stage training recipe and only open-source datasets is substantial, and the findings presented in the paper are insightful for the community.

Q4: The models utilize SlowFast projector for videos, and MLP projector for images. This design seems to be redundant, as mainstream video mllms (InternVL-2.5 [3], Qwen2.5-VL [4], VideoChat-Flash [5]) can process video and image using a unified projector, by treating image as a single-frame video. Discussion and experiments about this dual projector design should be included.

A4: To clarify, we also process images as a single-frame video. They share the same projector parameters, except that video inputs require a slow-fast pooling before the projection. The logic is also very simple from the implementation perspective: if an input has 5 dimensions (i.e., batch size, channel, temporal, height, width), it goes to the slow-fast pooling first, otherwise, it is an image input with 4 dimensions and directly goes to the MLP layers. We will rewrite this part to solve this confusion.

Thank you for your feedback! We tried our best to give detailed answers with solid experimental results. Hopefully, they can solve your concerns and convince you to raise the score.

Q1: The paper lacks novelty, as the effectiveness of the slow-fast mechanism has already been demonstrated in prior work [1] on video understanding.

A1: We claim two main contributions compared to the original SlowFast Network. First, our goal and motivation are fundamentally different. SlowFast Network replaces the optical flow stream with high frame rate inputs, because precomputing the optical flow fields is very costly and slow. However, our slow-fast design aims to model long-range temporal context by taking more input frames while optimizing the token efficiency. Second, our slow-fast design is tailored specifically for VLMs. In the original SlowFast Network, the primary difference between the Slow and Fast pathways lies in their use of different frame rates. In contrast, our approach differentiates the two pathways by resolution. We employ low-resolution frames at a high frame rate in the Fast pathway, capturing fine-grained temporal information within an efficient token budget. Meanwhile, the Slow pathway uses high-resolution frames to enhance spatial details.

More importantly, compared to existing Video LLMs, our models achieve state-of-the-art level performance across various benchmarks by using a simple training recipe and only open-source datasets. Considering that both training Video LLMs and collecting large-scale video datasets are very costly, we believe our paper can benefit the community beyond model novelty, especially for many small companies and university labs, to advance research on long-form video understanding.

Q2: The use of a fixed-frame sampling architecture appears outdated and may be inadequate for addressing the demands of proactive streaming video applications [2].

A2: We agree that a dynamic-frame sampling algorithm could improve the model performance, but disagree that fixed-frame sampling is outdated. First, we clarify that our model mainly uses FPS sampling and only fails back to fixed-frame sampling when the video duration is longer than 128s. Second, our sampling method is adopted by most state-of-the-art Video LLMs, such as LLaVA-Video, Apollo, and VideoLLaMA3, where Apollo particularly studies the impact of using different frame samplers.

To reduce the reviewer's concern on streaming video applications, we evaluate our model on the suggested StreamingBench [1] benchmark. We convert SlowFast-LLaVA-1.5-7B into a proactive streaming assistant by adopting the StreamBridge [3] technique without fine-tuning. The results are shown in the following table. We observe that SlowFast-LLaVA-1.5-7B achieves state-of-the-art performance on Real-Time Visual Understanding and Contextual Understanding, showing our effectiveness for streaming video understanding. We fall behind on Omni-Source Understanding since our model is never trained on interleaved datasets with audio. We will include more detailed results and discussions in our paper revision.

[2] StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding. Lin et al. arXiv 2024. [3] StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant. Wang et al. arXiv 2025.