$\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation

Thank you for your positive review and suggestions. We are happy that you found our method to be both efficient and effective for long-form video understanding, and our experiments solid. We address your concerns about our paper below.

“Is there any efficiency analysis provided in terms of computational resources, such as FLOPs or GPU memory cost for the proposed method? For example, a visualization like the figure 1 in MovieChat paper would be better.”

We appreciate the suggestion and refer to our response to reviewer iEVz, where we present additional experiments that analyze the computational overhead. These experiments include the time consumption of the LTM module and the impact of sampling more frames. Our results demonstrate that our method sustains a constant memory footprint regardless of the number of frames, with only a slight increase in inference time. Additionally, we observe that increasing the number of basis functions leads to a much smaller rise in memory usage compared to the baseline's growth with additional frames. We will add these experiments to the appendix.

“In Table 3, why do video-llama-based models perform best in sticky settings while VideoChat2 are in no short-term memory sticky?”

This is indeed intriguing and we asked ourselves the same question. We speculate that VideoChat2-based models tend to prioritize global features, whereas Video-LLaMA tends to focus on local features. Both benefit from sticky memories in open-ended generation tasks, as it helps maintain long-term coherence and consistency across outputs, regardless of whether the model emphasizes global or local context.

“Is it possible to set alpha as a dynamic parameter instead of keeping it fixed at 0.9/1.0 statically? A dynamic alpha might allow more flexible adaptation based on varying video contexts.”

Thank you for the thoughtful suggestion. We agree that using a fixed $\alpha$ may not always be ideal, as video content with different temporal dynamics might benefit from varying weightings. While we chose a fixed $\alpha$ to simplify the current experiments and establish a baseline, we recognize the potential advantages of making $\alpha$ dynamic. Adapting $\alpha$ based on video characteristics (as mentioned by reviewer aCqS), such as shot transitions or object persistence, could indeed improve performance and generalization. We plan to explore this dynamic approach in future work, and we will mention it in the discussion section.

“The baseline models used in the paper (Video-LLama and VideoChat2) are quite old (from spring and summer 2023). Given the rapid advancements in video understanding, there are newer and more powerful models available. It would be better to consider including some of these models, such as Llava-OneVision[1], Qwen2-VL[2], or DeepSeek-VL[3], in future experiments?”

While we recognize the rapid advancements in video understanding and the availability of more powerful models, our primary goal in this work was not to develop a SOTA model but to demonstrate the effectiveness of incorporating a biologically inspired LTM in a fair, training-free setting. The baseline models we used (Video-LLama and VideoChat2) were chosen for their relevance to our approach given that their Q-former based architecture enables a training-free approach, but we agree that including newer models like Llava-OneVision, Qwen2-VL, or DeepSeek-VL in future experiments (which would require additional training) may provide valuable comparisons. Our method is highly general and can be integrated into stronger VLMs moving forward. We will highlight this in the discussion section and suggest future work to explore the incorporation of our LTM approach into state-of-the-art models.

Thank you for your positive review and suggestions. We address your concerns about our paper below.

“As the authors mention, one key benefit of using basis functions over processing individual video frames is that fewer basis functions are needed to represent the information in the raw frames. This leads to more compressed representations (Lines 145-147), which is particularly useful for processing long videos. However, in their experiments, the authors use 256 frames in each chunk and 1024 basis functions (Line 238, col 2) for VideoLLaMA and 16-frame chunks and 256 basis functions for VideoChat2 (Line 241, col 2). How do these experimental designs reconcile with the assumption of needing fewer basis functions than video frames? Or are the authors implying that the number of basis functions is fewer than the total number of frames across all the chunks (e.g., 8x256 = 2048 > 1024)? But even with that assumption, for VideoChat2, we get 8x16 = 128 < 256. Could the authors please explain this discrepancy?”

For Video LLaMA, where the number of total frames is large (2048), the basis functions were set to half this total number of frames, ensuring a compressed yet effective representation. In the case of VideoChat2, we used 256 basis functions with $L=128$ frames since the number of frames supported by VideoChat2 is much smaller. Increasing the number of basis functions is computationally lighter than increasing chunk size (as shown in the first two Tables in iEVz response) and it improves the multivariate step regression fit. Larger chunk sizes caused memory issues for VideoChat2, but increasing basis functions did not, allowing for better fitting. This strategy balances memory efficiency and model accuracy without overloading computational resources. This tradeoff will be further discussed in the appendix.

“For the choice of the weighting factor alpha between short- and long-term memories (Eqn. 16), the authors report experiments on the MovieChat dataset and choose the best value of α from those experiments. However, it is unclear how generalizable that value is to other videos or datasets. Also, is it fair to assume that α should be constant for all kinds of videos, or is there scope to make α context- and category-aware, e.g., depending on how often shots are cut in the video, how often background scenes and foreground objects appear and reappear in the video, etc.?”

Thank you for the insightful suggestion. While we intentionally used a fixed $\alpha$ to establish a baseline and simplify experiments, we agree that an adaptive approach—adjusting $\alpha$ based on video characteristics like shot changes or object permanence—might be an interesting idea to improve generalization in future work. We will mention it in the discussion section.

“Have the authors considered the effect of any type of noise when determining relevant locations for the sticky memory? For example, if the signal is slightly perturbed (additive noise), how would the probability function (Eqn. 15) be affected? Understanding the noise characteristics is not essential to determine the utility of the sticky memory procedure, and my rating would not depend on the authors' response to this question. However, the noise characteristics become relevant when considering that video signals can easily be corrupted by transmission, compression, or even adversarial noises.”

This is an interesting question. The impact will depend on the type of noise and on its smoothness in the time dimension. If the signal is perturbed with additive noise, the Gibbs transformation in Eq. 10 will lead to a modified density, which in turn will change the locations for the sticky memory as for Eq. 15. Since the Gibbs transformation is continuous and well-behaved, the effect in the memory representations should be smooth. However, to better understand the practical effect of noise in the memory a careful empirical analysis needs to be conducted.

“In Eqn. 2, $Q \in \mathbb{R}^{R \times d}$, $K \in \mathbb{R}^{L \times d}$, $V \in \mathbb{R}^{L \times d}$ (...) implies that $Z \in \mathbb{R}^{R \times d}$, but the authors have stated $Z \in \mathbb{R}^{L \times d}$ in both Lines 116 and 121. Is this a typo?”

Yes, this is indeed a typographical error. We appreciate your careful review and will correct it accordingly. Thank you for bringing this to our attention.

Thank you for your review and suggestions. We are happy that you found our idea novel, the generalizability from short video to long video understanding well proved in both methods and experiments, and the ablation studies well conducted. We understand your main concerns about our paper and address them below. We hope that our answers clarify and alleviate your concerns.

“The authors claim that the proposed PDF based on Gibbs density is much powerful than the Gaussian model in L96-99, which lacks neither theoretical proof nor experimental results.”

We agree that our claim will benefit from concrete evidence. In fact, at an initial stage of our project, we experimented with the Gaussian model but we abandoned it since its unimodal nature limits its ability to capture complex distributions, leading to poor results. Our method extends Video Q-Former's cross-attention using the same learned projections trained with softmax, making Gibbs density a more natural fit. Nevertheless, we conducted additional experiments comparing both approaches, confirming that Gibbs improves performance. We also observed that most of the responses in open-ended MovieChat-1K degraded significantly under the Gaussian model. We will add this to the appendix.

“Though the method is training-free, there is still computational overhead should be considered.”

We refer to our response to reviewer iEVz, where we present additional experiments analyzing the computational overhead, including the time consumption of the LTM module, the impact of sampling more frames, and the encoding time with the ViT encoder. Our results demonstrate that our method sustains a constant memory footprint regardless of the number of frames, with only a slight increase in inference time. Additionally, we observe that increasing the number of basis functions leads to a much smaller rise in memory usage compared to the baseline's growth with additional frames. We hope that these experiments alleviate your concern.

“The proposed method heavily relies on the video Q-Former. However, recent models seldom use Q-Former (...)”; “The baseline models used for comparison are too outdated (...)” We chose Q-Former because it allows for seamless, training-free integration of our biologically inspired continuous attention component. This is possible because the video Q-Former is designed to process sequences of frame representations using a cross-attention module, which enables our method to augment it with a LTM based on continuous cross-attention. However, our LTM, if we want to go beyond training-free, is not inherently tied to Q-Former and could be used with other architectures. Our primary goal is to provide a proof of concept incorporating a biologically inspired LTM in a fair, training-free setting. While SOTA models continue to evolve rapidly, our focus was on conducting an apples-to-apples comparison to validate the benefits of our approach. Our method is highly general and can be integrated into stronger VLMs in the future. Unlike our training-free approach, this would require additional learning. We will provide insights in the discussion section. Future work can then build on our findings by incorporating the LTM into the strongest VLMs available.

“Tab. 1 and Tab. 2 demonstrate limited improvement for VideoChat2 (...). Recent models have significantly outperformed VideoChat2 (...) give more detailed analysis on the reason of limited improvement (...)”

This is an interesting question. The results in Tab. 1-2 are for multiple-choice datasets, where VideoChat2 models directly incorporate answer options into the prompt, making responses more constrained. In contrast, models like Video-LLaMA generate open-ended responses before selecting the closest option match with LangChain, as explained in the Appendix, creating more room for the LTM to improve reasoning. Nevertheless, in open-ended generation tasks (Tab. 3), our method already provides a significant accuracy boost for VideoChat2 models.

We are fixing the typos, thank you for pointing them out!

Thank you for your positive review and suggestions. We are glad that you found our method novel, our evaluation sound, and the motivation of our work strong and intuitive. We address your main concerns below.

“The paper misses analysis about memory usage and runtime overhead (...) the scaling behavior and trade-offs could be more explicitly benchmarked.”

This is a good suggestion. We agree that it is important to analyze the memory usage and runtime overhead of our method for large videos. Our experiments focus on a fixed-dimensional memory representation that scales independently of video length. To address this concern, we have conducted additional experiments that show the trade-offs in inference time and memory usage as the number of video chunks and basis functions increases, including the impact of the LTM. We will add these results to the appendix.

Inference Time vs L

Memory Usage vs L

Inference Time vs N

Memory Usage vs N

In the first two tables, for $\infty$-Video LLaMA, we use 256 frames per chunk and set $N=256$, while for the baseline (Video LLaMA), we use the total number of frames without chunking. In the last 2 tables for “Ours” we use 4 chunks of 256 frames. We use the Bohemian Rhapsody movie. OOM denotes an Out-of-Memory error on an A6000 GPU. Results show that our method maintains a constant memory footprint regardless of the number of frames, with only a small increase in inference time. We also observe that the increase in memory usage with the number of basis functions is smaller compared to the increase with the number of frames in the baseline. We hope this alleviates your concern.

“The conclusion from Table 4 is not consistent. It is better to test other Q-former video models and give a deeper analysis (...) compare with recent VLM models such as [*1, *2, *3].”

Please note that the main goal of our paper is to validate the benefits of integrating a LTM with continuous attention in a fair, training-free setting, for which the Q-former video models are a suitable choice. We focused on an apples-to-apples comparison to isolate the impact of our approach, paving the way for future integration into stronger VLMs, which keep evolving very rapidly. For more details, see our response to Reviewer SJwR (second question).

“Would segmenting the video by scene change (rather than fixed-length chunks) further boost performance?”

In our early experiments, we tried scene segmentation, but for fast-moving videos, this created small chunks that did not improve performance. Testing different scene-dividing granularities resulted in some chunks containing multiple scenes, which yielded worse results than fixed-length chunks. In any case, we find fixed-length chunks more interesting, since the ability of our method to generate sticky memories without the need for any scene segmentation suggests it is able to identify the most relevant information in the different scenes, bypassing this additional step.

“How are the hyperparameters chosen? For example, is the method sensitive to memory contraction factor? (...) How sensitive are results to the choice of chunk size vs. number of chunks? Is there an optimal chunk length for typical tasks?”

The number of frames per chunk was determined by balancing GPU memory constraints and alignment with the training setup of the base models. For VideoChat2-based models, we selected 16 frames per chunk, as this both fits within memory limits and also matches the number of frames used during the training of VideoChat2. For Video LLaMA, although the model was originally trained with 32 frames, [1] has demonstrated that increasing this to 256 frames does not degrade performance. We therefore used 8 chunks of 256 frames, which aligns with the total number of frames in MovieChat [1]. Finally, the $\tau$ parameter was chosen to be the same as in the $\infty$-former paper [2].

[1] Moviechat+: Question-aware sparse memory for long video question answering (Song et al., 2024)

[2] $\infty$-former: infinite memory transformer (Martins et al., 2022)