Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames

We thank the reviewer for their detailed comments and address each concern below:

Ablation of uniform context

Thank you for this suggestion. We already included an ablation of uniform context in Table 6b of Appendix A. In the revision, we will refer more clearly to this experiment.

For convenience, we have copied Table 6b below. We ablated the effect of the number of uniform context frames, $u$, on both Egoschema and LVBench. In both cases, the context-limit is $k = 120$ frames, meaning that the remaining $m = k - u$ frames are selected by the model.

On LVBench, we achieve moderate improvements (+1.5 points) by introducing uniform context frames (first row vs third row). If we perform no frame selection, and only use uniformly sampled frames, the accuracy is substantially worse (decreases by 9 points) emphasising that good frame selection is key to good performance.

Egoschema, in contrast, does not benefit from uniform context. This may be explained by Figure 7 of Appendix A, which shows that our approach is already selecting an average of 74% of the input frames in the dataset (and we therefore do not need any additional frames). The high proportion of selected frames is in agreement with the dataset’s “temporal certificate” – Egoschema was annotated [40] such that the annotator must look at, at least 100 seconds or 56% of the video to answer the question.

Finally, note that the Accuracy vs Computation analysis in Figure 4 accounts for the uniform context in our method, and it clearly shows the benefit of our method over baseline inference (which is effectively only uniformly sampling frames from the input video).

Motivation for uniform context

Intuitively, uniform context is beneficial for long videos (such as the hour-long videos in LVBench), since during context aggregation, we extract relevant context from each segment of the video independently. Adding a few coarsely sampled, uniform context frames, provides a more holistic view which improves accuracy. This is not required for shorter videos, such as those in Egoschema, which may already fit within the context-limit of the model.

We will clarify this in the revision.

Other methods which use only a single VLM for context selection

Thank you. We will include a discussion of [A] in the revision, and also revise the Related Work section, specifically the “Long-context video understanding” paragraph in light of [A]. Although [A] uses a single VLM, they first compute captions as an intermediate representation of the video, and select frames based on these captions. We considered this design choice too, and ablated this in Table 2, Rows 4 and 5. Our method, based on selecting relevant frames, achieves higher accuracy than selecting frames based on their captions. Moreover, it is not as sensitive to the captioning prompt that was used, as we observed that Egoschema and LVBench require different lengths of captions to achieve high results (detailed captioning prompts are in Appendix D, Lines 1093-1094). For convenience, we have reproduced the relevant parts of Table 2 below here.

These results show that the general principle of curating the input to the VLM is beneficial, and that our instantiation of Partitioned Context Aggregation is the most effective.

[A] D Romero et al. Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering. ACL 2024

Thanks to the authors for answering my questions. I have read the rebuttal and also the responses from the other reviewers , regarding the "Ablation of uniform context" I believe that a demonstration in two datasets where it improves in one of them and not in the other is not really convincing, and the paper would benefit with additional experiments to really show the readers the importance of this part (I understand that maybe the authors did not have enough time to perform extensive demonstrations for now). On the other hand, thanks for the responses regarding my other two points. But despite all of this, as I expressed in the "Questions" section of my original review, my main point about the paper is related to novelty (which has been also raised by buo9), key frame extraction has been very well established and the idea of selecting frames to improve performance has been around for a while, and despite achieving higher performances and some differences with other works, I don't see the additional points of the paper like the inclusion of additional context, as strong enough contributions that other people can benefit from to progress on this task.

We thank the reviewer for their detailed comments and address each concern below:

Difficult to understand how the total number of tokens being processed change

We would like to clarify some confusion about the total number of tokens / frames that are processed.

In Partitioned Context Aggregation (L174 to 198), the reviewer is correct that a video of $N$ frames is partitioned into $l$ segments, each of length $m = N / l$ frames.

However, from each segment, $\\mathbf{x}\_{i}$, we uniformly sample $s$ frames from it (L182 to 185), denoted as $\\mathbf{x}\_{i [s]}$ in Equation 5 (we believe this is the key detail that the reviewer missed).

Therefore, we process a total of $s \cdot l$ frames. Note that both $s$ and $l$ are independent of the number of the video length, $N$. This allows us to control the total amount of computation used to process a video in Figure 4, where we keep $s = 64$ frames constant, and vary $l$ from 2 to 42. Finally, note that for shorter videos where $s \cdot l > N$, or in other words, the video is too short to partition into $l$ segments, we set $l = \lfloor N / s \rfloor$ instead.

We will revise the text to clarify this, and are happy to engage with the reviewer during the discussion period if this is not clear.

How is compute constant regardless of the number of frames?

As described above, we process a total of $s \cdot l$ frames, where both $s$ and $l$ are hyperparameters, and do not depend on the video length, $N$.

Authors compare to LLM-based frame extraction methods, but not traditional ones.

Thank you for this suggestion. As we discussed in Lines 106-111, traditional frame selection models are trained to process only a fixed, small number of input frames (ie 32 for SeViLA [68] and ViLA [59], and 64 for [A]). These models are therefore not suitable for hour-long video datasets such as LVBench which contain an average of 4080 frames.

Moreover, these frame-selection models are trained for specific downstream answerer models, and need to backpropagate gradients through this downstream answerer during training. It is therefore not feasible to train such approaches for large, state-of-the-art answerer models like Gemini 1.5, GPT-4o and Qwen-2.5 VL which we use in our work.

We will clarify the limitations of traditional frame-selection techniques even further in the revision.

[A] S Buch et al. Flexible Frame Selection for Efficient Video Reasoning. CVPR 2025.

We thank the reviewer for their detailed comments and address each concern below:

Evaluation on Cinepile

Thank you for this suggestion. We compare our Partitioned Context Aggregation approach to baseline inference on Cinepile below. We follow the same protocol as the experiments in our paper, using a 32K / 120 frame context window for both approaches.

Table A1: Comparison of DCA on Cinepile.

Our DCA approach therefore still shows improvement over the baseline on this challenging dataset.

Is frame-selection appropriate when there is a lot of relevant context to select?

Thank you. Figure 7 in Appendix A shows that our model selects 74% of the input frames for Egoschema, and 15% of the relevant frames for LVBench. This shows that our method does indeed select the relevant context in a manner that is adaptive to the dataset. Our results on Egoschema (Figure 7 and Table 3) therefore show that when most of the input video is indeed relevant, our method is able to retain it.

Finally, note that Egoschema was annotated with a "temporal certificate” (the minimum duration of the video that an annotator must look at to answer the question) of 100 seconds (or 56% of the video), and our results in Figure 7 agree with this. Similarly, Figure 6 shows that our frame selection corresponds to the "time reference” frames that are annotated for LVBench, with the most frames being selected for “summarisation” questions. Both these results emphasise the adaptivity of our approach in selecting both short and long relative-contexts from a video.

We will refer to Figure 7 (in Appendix A) more clearly in the main text to make this point more clear.

Zero-shot performance of VLM for frame-selection

Thank you for this suggestion. Note that frame-selection for the purpose of question-answering is difficult to evaluate, as it depends on the answering model itself. From our evaluation datasets, only LVBench contains human-annotated “time-reference” frames which we use below to evaluate our frame selection.

In the table below, we extend our analysis from Figure 10 of Appendix B. We evaluate the precision and recall of our selected frames, with respect to the annotated “time-reference” frames, where a “true positive” denotes a selected frame which falls within the annotated time-references for that video, and a “false positive” does not.

We performed this analysis by varying the number of segments, $l$, in our Partitioned Context Aggregation approach. By increasing $l$, our recall increases, whilst precision remains fairly constant. We observe that the “time-reference” frames in LVBench are overcomplete, in that it is not necessary to select all of them to answer the question, which is why we can get accuracy improvements over the baseline with a fairly low recall. In fact, for $l \leq 12$, our recall is lower than the uniform frame sampling baseline, but the accuracy is higher due to the larger precision, showing that a subset of the time-reference frames is sufficient to answer the question. We find, however, that the accuracy correlates well with an additional metric which we define, “At Least One”, which denotes if at least one selected frame falls within the annotated “time-reference” frames.

Table A2: Analysis of frame selection on the LVBench, using the human-annotated “time-reference” frames. “At Least One” denotes if at least one of the selected frames falls within the “time-reference” frames.

Note that the baseline, of uniformly selecting 120 frames from the video, has the lowest precision as expected since most of the frames in the video are irrelevant to the question. Moreover, as the videos in LVBench are an average of 68 minutes long, performing uniform sampling means that we sample at least one frame within the annotated time-references only 20.3% of the time. Using our proposed method increases this to 71.1% which explains our accuracy improvement. Finally, the fact that the “At Least One” metric is less than the overall question-answering accuracy in the first two rows, also shows that the Gemini VLM is also able to correctly answer a question without using any of the annotated time-reference frames in the dataset. This may be due to errors in the annotations of time-reference frames in LVBench, or the fact that the VLM can exploit language biases to answer the question.

We will add this analysis to the appendix, and refer to it clearly from the main text.

Why not choose Hierarchical Context Aggregation?

Thank you for this suggestion. The reviewer is correct that a coarse-to-fine context aggregation approach is effective. However, we found that in terms of accuracy/computation trade-offs, it is not as effective as Partitioned Context Aggregation. As we are not able to upload figures, we include the raw data for Figure 4, Accuracy vs Computation, below, with a line added for Hierarchical Context Aggregation.

In the revision, we will update Figure 4 to include this additional line, and to clarify this point.

Table A3: Accuracy vs Computation Trade-off for different context aggregation methods. The table corresponds to Figure 4 in the main paper.

Note that for our context aggregation methods, the total number of frames is based on the average number of frames selected by our method on the LVBench dataset.

Table 3 implies that longer-context is always better

Thank you. We would like to clarify that as shown in Figure 4, the accuracy of baseline inference saturates around 1024 frames / 264K tokens. We will therefore update the corresponding row for the baseline of Table 3 to 264 tokens, and add a clarifying note that performance saturates after this point.

The authors should refrain from using “most relevant context” as they do not show proof of optimality.

Thank you. We have revised the paper accordingly.

Using the frame-selection justification of the model (j in Eq.4) to iteratively refine the prompt for the VLM can also be explored.

Thank you for this suggestion. We initially performed experiments with this, but did not achieve improvements beyond our current approach, which is why we did not pursue it further.

Ablations of prompts

We ablated different prompting strategies in Tables 4 and 5 of Appendix A, and also detailed all prompts and other implementation details in Appendix D.