Improving LLM Video Understanding with 16 Frames Per Second

Thank you very much for your thoughtful and constructive feedback on our paper. Below, we will respond to the questions you have raised.

1: The evaluation on existing general video benchmarks cannot showcase the advantage of high frame-rate training. & Supplement more scenarios beyond sports

In fact, in general video understanding, F-16 also demonstrates significant advantages in short video benchmarks. Note that F-16 uses less data (less video data and image data) compared to most of the models in Table 1, and it only uses LLaVA-Video-178k. Besides, as for more scenarios, we also evaluate F-16 on a very recent benchmark, FAVOR-Bench, which targets evaluating fine-grained video motion understanding. F-16 achieves SOTA results on it among 7B models.

We list the results on these benchmarks here to more clearly present the advantages of F-16:

For long video understanding, although F-16 is not outstanding, it still shows competitive results. This is because only when processing short videos, the frame sampling is done at FPS = 16, matching the training situation. For long videos, the FPS is much lower than 16. For instance, as F-16 samples a maximum of $110\times 16=1760$ Frames, it perceives a 1760-second video at FPS=1. Sparse frame sampling will lead to slightly poorer performance of the high-frame-rate aligner, resulting in average results for long videos.

2: More technical designs and experiments on the aligner.

We studied various high-frame-rate modeling structures. All attempts confirmed that visual-language alignment via linear transformations helps prevent model performance decline, which means linear projections effectively and efficiently align the visual encoder's output semantic space with the LLM's input space. This is consistent with prior work like NVILA.

Appendix B details the studied structures. Using CNN modules to capture frame spatiotemporal differences led to worse results than MLP, suggesting CNN aligners may harm semantic information derived from the visual encoder. Replacing max pooling with a learnable linear layer improved training loss but is worse when testing. Using a self-attention layer to replace the MLP projector's first linear layer to extract frame dynamic changes also affected visual feature semantics, resulting in slightly worse performance when scaling up the training parameters.

Ultimately, we chose a 2-layer MLP structure. Its first linear layer ensures semantic alignment between LLM input and visual encoder output spaces, and the second compresses duplicated information in continuous frames. It is well-known that any continuous mapping function can be represented arbitrarily accurately by a 2-layer MLP with sufficiently large hidden layer dimension, which provides an insight into our motivation. Experimental results also validate this design.

3: More analysis on the aligner is desired, including input temporal range, output number of tokens, inserting positions, etc.

As for the temporal range, we have trained models at different FPS, and set the width $w$ of the processing window equal to the FPS. As the input FPS gradually increases from 1 to 16, the performance of the model shows an upward trend, shown in Fig. 3(b) "Train FPS=Test FPS". Besides training, testing at different FPS is also tried, shown in Fig. 3(b) "Train FPS=16".

Though the input number of tokens to the aligner increases with FPS, the output number of tokens remains the same due to the processing window width of the aligner remains the same with FPS. This setting enables a fair comparison of models with different frame rates.

For inserting positions, since the high-frame-rate aligner is a replacement to the single-frame aligner, only pre- and post-pooling used by other VLLMs are evaluated. Table 3 shows pre-pooling causes larger differences between adjacent frames, and Table 4 shows post-pooling performs much better under high-frame-rate settings. These indicate that the high-frame-rate aligner heavily relies on inter-frame variations in visual features for effective learning. If adjacent sampled frames are no longer similar to each other, the high-frame-rate aligner will struggle to learn well.

4: Present the visualizations of model inputs under different frame rates.

We visualize videos at different frame rates here: https://github.com/F-16-LLM/Rebuttal/blob/main/README.md

Under low-frame-rate cases like FPS = 1, many details will be missed. We will add this part to the updated paper.

Thank you for the positive rating for the paper. We sincerely appreciate your recognition of our work.

Thank you very much for your thoughtful and constructive feedback on our paper. Below, we will respond to the questions you have raised.

1: Can you provide more detailed information about the training procedure, including hyperparameter settings, training time, and the resources (e.g., number of GPUs) used for training?

Regarding the training procedure, F-16 is first initialized from LLaVA-OneVision-7B, as shown in Sec. 4.1. Then, F-16 is trained on general videos. At this stage, the image encoder stays frozen while other parts of the model are updated. To further verify the advantages of high-frame-rate modeling, the model is then fine-tuned on high-speed sports video. LoRA is adapted to the LLM and serves as the only trainable module in this stage.

As for other training settings, F-16 is trained with Adam optimizer and cosine scheduler. The learning rate is set to 2e-5, the warm-up ratio is set to 0.03, and the batch size per device is 1. For general video training, 128 H100 GPUs are used for training for 1 epoch (about 13000 steps of updates), which takes about 35 hours. As for high-speed sports fine-tuning, 64 H100 GPUs are used for training 5 epochs (about 9000 steps of updates), which takes about 20 hours. For comparison, the FPS=1 model takes about 18 hours for general video training and 10 hours for high-speed sports fine-tuning.

Note that the difference in training time primarily comes from two parts: a small portion is due to the increase in encoder duration, while the major part is the catastrophic growth in CPU time when reading more frames in long videos, which leads to less GPU usage. We have not optimized video frame extraction, and just use the Python Library"Decord" for frame extraction. Appropriate optimization or preprocessing can significantly improve training speed, such as extracting frames in advance.

2: What are the computational costs (memory footprint, inference time) of F-16 compared to other video LLMs? This information is important to understand the practical limitations of the proposed approach.

With the high-frame-rate aligner, though 16 times more visual frames are inputted, the tokens into the LLM remain the same as the low-frame-rate model. Therefore, the computational cost for LLM is the same as compared of the low-frame-rate one. The difference mainly comes from the encoder and the aligner.

• Inference time. As compared in Fig. 3(a), the inference time of the proposed high-frame-aligner can be ignored compared with the LLM and the visual encoder, and the comparison to other models can mainly focus on the visual encoder. To our observation, the inference time for the visual encoder grows linearly with the input frame time. Therefore, the inference time of the visual encoder increases by 16x, which makes the total inference time of F-16 1x longer than that of the comparable video LLM with FPS=1.
• Memory footprint. The memory cost can be divided into 4 parts: the final visual tokens, the hidden layer of the aligner, the output of the visual encoder, and the inner computation cost of the visual encoder. The final visual tokens are kept at the same number, which does not increase in memory cost. For the other 3 parts, though it seems that they will grow linearly along with the number of frames, they can be handled sequentially because of the independence between different processing windows. For instance, using a for-loop to process frames between processing windows. Therefore, there is no significant increase in memory cost observed from F-16.