SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM

Q1: I have concerns about its practicality. Firstly, this reasoning method nearly doubles the inference cost for the same question compared to other methods. Secondly, the accuracy of the final response heavily relies on the accuracy of event localization in the first stage. According to the benchmark validation provided by the authors, even when trained on proprietary data, SlowFocus only barely meets the passing mark in temporal grounding mIoU. This significantly impacts the accuracy of temporal reasoning in the second stage.

A:

1. The additional cost is actually minimal, as the low-frequency visual tokens only need to be encoded once. Moreover, the relevant segment grounding is not a time-consuming task, as it only involves processing short sequence lengths. We also report our inference speed compared with baselines in the table below, showing that our method results in only a small increase in inference cost while significantly improving fine-grained video understanding. | Method | LoRA | Inference time (s) | mIoU | B | M | R | C | Acc | Score | |-----------|------|--------------------|-------|------|------|------|------|-------|-------| | VTimeLLM | √ | 1.27 | 27.69 | 0.05 | 0.09 | 0.08 | 0.12 | 9.96 | 0.54 | | LLaMA-VID | × | 1.25 | 0.35 | 0.16 | 0.12 | 0.11 | 0.23 | 15.65 | 0.87 | | Ours | √ | 1.51 | 66.68 | 0.66 | 0.41 | 0.7 | 3.27 | 53.1 | 2.78 |

2. The temporal grounding task on FineActionCGR is challenging, as the ground truth often occurs within a very short time. Therefore, the current mIoU of 66.68 is actually sufficiently practical. We have also conducted experiments demonstrating that the current temporal grounding capability has reached saturation and is not the main limiting factor for further performance improvement, as detailed in the table below. | Stage 2 | Temporal grounding (mIoU) | B | M | R | C | Acc | Score | |---------|---------------------------|------|------|------|------|-------|-------| | 0.1 | 25.17 | 0.24 | 0.21 | 0.19 | 0.86 | 24.88 | 1.33 | | 0.25 | 47.83 | 0.49 | 0.38 | 0.42 | 2.12 | 44.73 | 2.28 | | 0.5 | 61.29 | 0.64 | 0.43 | 0.65 | 3.05 | 52.59 | 2.72 | | 1 | 66.68 | 0.66 | 0.41 | 0.7 | 3.27 | 53.1 | 2.78 | The column stage 2 means the number of epochs for stage 2's fine-tuning.

Q2: The authors chose relatively weak baselines and attempted to claim the superiority of their results. I suggest comparing their method with MovieChat and VideoChat2 to substantiate their claims of advancement. Moreover, despite using more data than LLaMA-VID, their performance on various benchmarks is not significantly better than LLaMA-VID, raising questions about the effectiveness of this approach.

A: Thank you for the feedback. We have added the comparisons with MovieChat and VideoChat2 in the table below.

We have also fine-tuned prior work using the same dataset, as detailed in the table below. The results prove that the improvement achieved by introducing the stage 3 dataset is limited, with the primary gains being attributed to the proposed algorithm. Compared to prior work, our approach requires only a limited amount of additional data to adapt to the MFS algorithm and achieve overwhelming advantages on fine-grained benchmarks.

Q3: Need to validate the effectiveness of their two-stage model on benchmarks involving long videos, which is crucial to substantiate their claims. For instance, they could use benchmarks like EgoSchema and MovieChat-1K.

A: To address the reviewer's concern, we evaluate our method on MovieChat-1K, as detailed in the table below. The results show that although our method is not specifically trained on long video benchmarks (in contrast, MovieChat has undergone targeted training for long videos), it still achieves competitive results.

Thanks for the valuable comments.

First, we would like to clarify that our work is not specifically designed for long videos but rather focusing on fine-grained video understanding, independent of video length.

Even short videos with fine-grained temporal tasks present significant challenges for existing VLMs, as demonstrated by our benchmark evaluation in Table 1 of the main paper. In addition, the ablations in Table 3 of the main paper, that mixed-frequency sampling brings significant performance improvement on fine-grained video understanding, strongly support the effectiveness of our proposed approach (i.e., event localization and video slicing) on short video benchmarks.

Second, we have additionally test our method on MovieChat-1K in our last response, which is a widely used long video benchmark. We now provide the results on EgoSchema, as detailed in the tables below.

The results further demonstrate that, despite not being specifically trained on long video datasets, our method still achieves competitive performance. We will include more evaluation results on other long video benchmarks (such as Video-MME) in the revision.

Hope our response helps the reviewer's final recommendation. Thank you!

Dear Reviewer bGgU

Thanks again for the valuable comments and suggestions. As the discussion phase is nearing its end, we wondered if the reviewer might still have any concerns that we could address. We believe our results on long video benchmarks (including MovieChat-1K and EgoSchema) addressed the questions/concerns.

It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you！

Best wishes,

Authors

Q1: How is the duration of the videos for temporal grounding evaluation? Will the low frequency sampling result in the to lose much information in excessively long videos?

A: The duration of temporal grounding tasks typically ranges from 1 to 10 seconds. To explore this question, we carry out ablations on low frequency sampling (fps) in the table below. Experimental results demonstrate that fps value at 1 is sufficient to perform temporal grounding task.

Q2: In Fig.2, the temporal grounding derives from the low frequency sampled video features. Why the high frequency sampling also has impact on the temporal grounding performance?

A: Actually, when evaluated on temporal grounding tasks, the mixed-frequency sampling algorithm is also applied. This is why high-frequency sampling impacts the temporal grounding performance. We have also conducted an ablation study on the effect of low and high-frequency features when mixed-frequency sampling is turned off. The results presented in the table below demonstrate that under this setting, the high-frequency sampling does not influence the performance.

Q3: The architecture of the temporal encoder is not illustrated.

A: We have described the architecture of the temporal encoder in the main paper (line 171-177), which consists of several learnable queries corresponding to different relative sequential positions. We will illustrate this in Figure 2 in the revised version.

Q4: In Table 5, what does Fps=64 mean? The video dataset usually contains videos with around 30 fps, how does 64 fps work?

A: FineAction is an action-centric dataset that includes some videos recorded at high fps (over 64). For videos with a fps of 64 or lower, we dynamically adjust the sampling frequency to match the original fps, which means all frames are used as input. Additionally, we will clarify the term fps by referring to it as max fps in the revised version for better clarity.

Q5: The proposed method presents overwhelming advantage on the proposed FineAction-CGR dataset while the improvement on other benchmarks are marginal. I understand it is because the rest are comparatively easy and of coarse-grained. However, it is better to provide the evaluation on other challenging video benchmarks like long video datasets, e.g., EgoSchema [1], MovieChat [2] to validate the effectiveness of the temporal grounding in long sequences for more comprehensive comparison.

A: To address the reviewer's concern, we evaluate our method on MovieChat-1K, as detailed in the table below. The results show that although our method is not specifically trained on long video benchmarks (in contrast, MovieChat has undergone targeted training for long videos), it still achieves competitive results.

Q6: How do you condense each frame into Nv tokens?

A: We condense the features of each frame using spatial grid pooling. Specifically, after passing through the visual encoder, each frame is represented by $16 \times 16 = 256$ tokens. These tokens are then further condensed into a $n \times n$ grid, resulting in $n^2$ tokens, depending on the experimental setting.

Thanks for the valuable comments. We address the remaining concerns as follows:

Q1: The duration of temporal grounding tasks typically ranges from 1 to 10 seconds. Is it the temporal length of the target grounding segment? What is the duration of the whole video? And what is the ratio of the target clip of the input video?

A: Yes, it is the length of the temporal grounding task. The duration of the whole video varies from 30s to 15min. According to our statistics, the average ratio of the target clip to the total video length is 4.28%.

Q2: The improvements on MovieChat benchmark seem marginal. I wonder how you deal with the breakpoint mode. Did you use the grounding ability to select the related segments that help answer the breakpoint mode questions?

A: Given that the official repository of MovieChat does not provide a general evaluation code for methods not designed for breakpoint mode (e.g., VideoLLaMA and LLaMA-VID), we adapt by converting the time mentioned in the breakpoint mode questions into discrete values (normalized between 000 and 999). These values are then incorporated into the questions, such as What might happen next at 154?.

While directly feeding the key frame information in the question is less favorable for our method—particularly because MovieChat uses ground-truth keyframe features as input—we still observe that our performance remains competitive, especially in breakpoint mode (lags by only 0.2 in Acc).

We actually leverage target temporal grounding to assist in selecting relevant segments, and observe a significant performance improvement compared to when the target temporal grounding is disabled, as detailed in the table below.

Q1: Motivation not justified: 1) Evaluate the inference speed. 2) The benefit of this approach over uniform sampling at a higher resolution is unclear.

A: The additional cost of inference speed is actually minimal, as the low-frequency visual tokens only need to be encoded once. Moreover, the relevant segment grounding is not a time-consuming task, as it only involves processing short sequence lengths.

We report our inference speed compared to the baselines in the table below, demonstrating that our method results in only a slight increase in inference cost while significantly enhancing fine-grained video understanding.

The phrase at a higher resolution may be confusing. We hypothesize that it refers to more tokens per frame, and we compare our approach with uniform sampling that adopts more token numbers per frame, as shown in the table below.

After increasing the token numbers per frame $N_V$ to 256, the performance gets a minimal improvement, indicating the limitation of simply increasing frame resolutions.

Q2: Table 1 Unfair: SlowFocus is trained on this FineActionCGR dataset.

A: FineActionCGR is employed in stage 3's fine-tuning to align the LLM with our mixed-frequency search method. We carefully split the dataset to ensure no overlap in scenes or activities between the training and testing sets. Detailed ablation studies, as shown in Table 3 of the main paper (lines 280-284), demonstrate that the observed performance gap is primarily due to proposed MFS approach.

Additionally, to address the reviewer's concern, we also fine-tune the prior work (LLaMA-VID) using stage 3. The results presented in the table below prove that the improvement achieved by introducing the stage 3 dataset is limited, with the primary gains being attributed to the proposed algorithm.

Q3: Table 2 Missing Details: 1) Is SlowFocus re-trained on this data? 2) How many frames do the prior works use? What is their inference compute requirement in comparison? 3) Also, can a simple VLM + LLM baseline like LLoVi be added for comparison?

A:

1. No, MSVD-QA, MSRVTT-QA, and video-based generative performance benchmarks are zero-shot benchmarks. Our method is not re-trained on these datasets.

2. To address the reviewer's concern, we have updated Table 2 of the main paper to include more implementation details, such as the number of frames and the computational requirements for inference, as detailed in the table below.

3. For a fair comparison, we also implement LLoVi using LLaVA-1.5 as the video captioner and Vicuna-7B as the LLM, also shown in the table below. | Method | Sampling strategy | Tokens per frame | ActivityNet-QA (Acc) | |---------------|-------------------|------------------|----------------------| | FrozenBiLM | N_L=10 | 256 | 24.7 | | VideoLLaMA | N_L=8 | 256 | 12.4 | | VideoChat | Fps=1 | 256 | 26.5 | | Video-ChatGPT | N_L=100 | 576 | 35.2 | | LLaMA-VID | Fps=1 | 64 | 47.4 | | LLoVi | Fps=1 | 256 | 45.2 | | Ours | Fps=1 | 64 | 48.4 |

Q4: Missing Ablations: Compare against simply feeding the selected high-res frames and the global low-res frames into an existing Vid-LLM (e.g. LLoVi).

A: First, we clarify that LLoVi is not a multi-modal Video-LLM, but rather a language-based combination of a visual captioner and a LLM.

Second, our paper does not introduce the concept of high-res and low-res frames. We hypothesis these refer to high and low frequency. It would be unfair to directly compare our method to one that feeds selected high-frequency frames into LLoVi, as this selection process inherently reveals the ground-truth.

To ensure a fair comparison under this setting, we feed LLoVi (same implementation as Q3) with the video frames sampled at the same frequency, as detailed in the table below. The results demonstrate that although the video captioner provides a substantial amount of language descriptions to the LLM (this process is quite time-consuming and less practical), the video captioner may still miss important temporal details, leading to potential errors in subsequent LLM inference.

Thanks for the valuable comments. Our code and dataset will be publicly released. We address other concerns as follows.

Q1: Motivation not justified. 1) omitting important details on what dataset these results are on, compute used (GPUs) to benchmark inference times, how the baselines were implemented, and why the baselines are not at all competitive (baseline numbers significantly lower). 2) More important, the concern of "benefit of this approach over uniform sampling at a higher resolution" remains unaddressed.

A: 1) Due to character limitations in the rebuttal, we were unable to include all the details in the table. We now provide the requested details as follows:

• what dataset these results are on: FineAction-CGR benchmark.
• compute used (GPUs): single V100.
• how the baselines were implemented: we just follow the official implementation provided by each method.
• why the baselines are not at all competitive: In fact we have provided comprehensive explanations in the main paper (line 261-264) that existing VLMs struggle with accurately predicting temporal segments and capturing fine-grained temporal details due to their lack of sensitivity to precise time boundaries.

2) The term at a higher resolution in the reviewer’s original request is ambiguous. Typically, resolution refers to the size of image/video or spatial tokens per frame. Therefore in our last response we responsed based on such interpretation. We now provide comparisons with uniform sampling at a higher frequency as requested, as detailed in table below.

For LLaMA-VID, increasing the fps to 2 (which also doubles the computational cost) resulted in a performance improvement. However, even with this adjustment, its performance was still significantly lower than ours. This indicates that simply increasing the sampling fps yields only limited benefits while consuming significant computational resources.

Further increasing the fps is impractical, as it surpasses the maximum token length supported by existing open-source VLMs. This limitation underscores the advantage of our proposed method, which enhances effective fps without incurring additional computational cost.

Q2: Unfair results comparisons. 1) The new results in rebuttal further support how baselines used as comparison in the paper were unfair. 2) While the authors newly introduce results where baselines are fine-tuned in the rebuttal, details of baseline fine-tuning is unclear. 3) Also, only a single baseline is provided under these slightly fairer fine-tuning settings for comparison.

A: 1) We have discussed the reasons of subpar performance of baselines in the main paper (lines 261-264) that existing VLMs struggle with accurately predicting temporal segments and capturing fine-grained temporal details due to their lack of sensitivity to precise time boundaries. The data used in stage 3 includes tasks focused on temporal grounding and is specifically fine-grained. This is why the baseline's performance improves after fine-tuning with stage 3's data. We believe this further validates the importance of our benchmark and the rationale behind our stage 3 fine-tuning. Moreover, the remaining performance gap (44.3 in mIoU and 10.29 in Acc) further demonstrates the effectiveness of our proposed method.

2) To ensure fairness, the implementation details of baseline fine-tuning (such as learning rate and LoRA rank) are just kept consistent with our method.

3) Our method is easily plug-and-play and transferable to other baseline models. We have additionally implemented LLaVA-Next as the baseline. The fine-tuning details are kept consistent for fair comparison. The results on FineAction-CGR are shown in the table below, demonstrating that the LLaVA-Next baseline performs better than LLaMA-VID, but still lags behind our method. We will also consider adding more baseline comparisons in the revision.

Q3: Video QnA results. 1) The baselines used are not the best currently. 2) Also, agreeing with the sentiments of reviewer bGgU on need for some long-video QnA benchmarks.

A: 1) Thanks, in fact our method is easily plug-and-play and transferable to other baseline models. We have additionally implemented LLaVA-Next as the baseline. The results on ActivityNet-QA are in the table below. By incorporating LLaVA-Next as the baseline, our method achieves advanced results on ActivityNet-QA. We will also consider adding more baseline comparisons in the revision.

2) Actually we have provided the evaluation results of our method on MovieChat-1K (in the response to reviewer bGgU) which is a widely used long video benchmark. We now provide the results on EgoSchema, as detailed in the tables below:

The results further demonstrate that, despite not being specifically trained on long video datasets, our method still achieves competitive performance.

Q4: Missing ablations. The authors avoid providing the requested ablations.

A: First, we would like to clarify that we made every effort to provide the requested ablation experiments. However, the original request was somewhat unclear.

Specifically, the terms high-res and low-res frames are confusing, as our paper does not introduce these concepts. Could the reviewer be more specific? Consequently, in our last response we conducted the ablations according to the request to simply feed the selected high-res frames and the global low-res frames, which differs significantly from the reviewer's new comment of feeding all frames at that high frequency.

Second, feeding all frames at such a high frequency is impractical. The valid fps for high-frequency frames averages around 10. Maintaining an fps of 10 with 64 tokens per frame is computationally infeasible. For example, a 1-minute video under this setting would result in 38,400 visual tokens, heavily challenging current open-source VLMs. This underscores a key advantage of our proposed method that it enhances valid fps while avoiding increased computational cost.

Hope our response helps the reviewer's final recommendation. Thank you!

Q1: Line 126: Using the same letter "L" to represent both "Low" and "LLM" might appear confusing to readers

A: Thanks, we will make a clearer representation in the revision.

Q2: Table 5: There is some confusion regarding whether this pertains to the low frame rate parts or the high frame rate parts.

A: Thanks, the fps term mentioned in Table 5 of the main paper refers to the low-frequency part. The high-frequency part maintains a fixed sampling number $N_H$. We will provide a clearer representation in the revised version.

Q3: Line 449: 1) What is the data ratio processed by GPT-4V and the Video Recaptioner Model? 2) Do their results show significant distribution differences? 3) Are you using GPT-4V to generate frame-by-frame videos or video captions?

A:

1. GPT-4V and the Video Recaptioner Model are both utilized during the process of generating captions for each video. GPT-4V is utilized to generate captions for the entire video by sampling 10 frames uniformly from each video. The Video Recaptioner Model is utilized to generate captions for the segmented video clips by uniformly sampling 8 frames from each clip.
2. The caption of the entire video generated by GPT-4V tends to summarize coarse-grained and general content. While the Video Recaptioner Model generates captions for segmented video clips, containing more details and action descriptions.
3. The video in FineAction may contain multiple objects and it's better not to generate frame-by-frame captions. Take the InternVID data set as an example, the frame-by-frame captions are not applicable when preceding objects are inconsistent with following objects. Therefore, GPT-4V is utilized to generate a caption by sampling 10 frames uniformly from a video to recognize different objects. This method not only saves cost but also provides global information for the QA generation of downstream tasks.

Q4: The analysis on the effectiveness and rationality of the FineAction-CGR benchmark evaluation is insufficient.

A: The evaluation protocol and metrics for FineAction-CGR align with widely-accepted benchmarks, such as ActivityNet-QA, ensuring the effectiveness of the benchmark evaluation. For other methods, we adhere to the official implementation guidelines. We will add more discussions and case studies on the benchmark evaluation in the revision.

Q5: Why is the visual encoder turned off in all stages? Some recent work also start open the gradient of CLIP encoder, would the results improve in your model if it were turned on?

A: Turning off the visual encoder and fine-tuning the multi-modal projector is a common practice among the baselines we compare, such as VideoLLaMA, Video-ChatGPT, and LLaMA-VID. To ensure a fair comparison, we adhere to this approach.

However, we acknowledge that turning on the visual encoder could be beneficial, and it would be a promising direction to explore methods for fine-tuning the visual encoder to enhance the unified representation of images and videos. Nonetheless, this aspect is not the primary focus of our work.

Q6: Is the use of LoRA for pre-training due to resource constraints? Would full-rank training result in better performance?

A: Yes, we use LoRA to fine-tune the LLM primarily due to resource constraints. We have conducted a comparison with full-rank training, as shown in Table 2 of the main paper, which demonstrates that the performance degradation when using the LoRA approach is minimal.

Q7: It is recommended to include comparisons with closed-source models such as Gemini Pro, GPT-4V, etc., to provide reference scores for FineAction-CGR benchmark in Table 1.

A: Thanks for the suggestion, we employ GPT-4V to evaluate on the benchmark, and the evaluation of Gemini Pro will be added in the revision. The results are presented in the table below. Due to token limitations, each video is sampled at 10 frames, and GPT-4V is tasked with answering questions based on these sampled frames. We observe that GPT-4V performs well on captioning tasks but exhibits suboptimal performance on tasks such as temporal grounding and reasoning, which require fine-grained temporal cues.