VideoLucy: Deep Memory Backtracking for Long Video Understanding

We sincerely thank Reviewer YuVT for the positive and encouraging review. We are grateful for your recognition of our hierarchical memory design, detailed experimental results, and the quality of writing and visualizations. Below, we respond to your thoughtful questions and suggestions in detail.

---

Q1: "Is backtracking sensitive to the initial memory list (Alg. 1 L1) created?"

A: Thank you for your insightful question regarding whether backtracking is sensitive to the initial memory list.

Theoretically, the backtracking process is sensitive to the initial memory list, as the iterative deepening of memory is built upon the time periods and content in the initial list. However, in practical scenarios, this sensitivity rarely affects the final performance, which can be attributed to the following aspects:

• First, our initial memory list is inherently question-aware. As described in Lines 185-194, the initial list $CM_\mathrm{sinit}$ is derived by filtering all the sampled coarse memories through the localization agent, which specifically identifies time periods most relevant to the question. This ensures that the initial memory list is closely associated with the query from the outset, laying a reliable foundation for subsequent backtracking.
• Second, the strong text understanding and processing capabilities of LLMs play a crucial role. In our qualitative experiments, we observed that the localization agent, powered by LLMs (e.g., DeepSeek-R1), rarely fails to identify relevant time periods entirely. Even in cases where the initial memory list may have minor inaccuracies, the iterative backtracking mechanism can gradually correct and refine the memory by mining deeper details, thereby mitigating potential impacts.

In summary, while the backtracking process is theoretically sensitive to the initial memory list, the question-relevant initialization strategy and the robustness of LLMs ensure that, in practice, the initial memory list is constructed accurately enough to avoid significant adverse effects on the backtracking process.

---

Q2: "What is the inference time for this method? How does this compare to other agent-based works?"

A: Thank you for raising this important point. We appreciate the opportunity to clarify the inference time of VideoLucy and how it compares to other agent-based approaches.

By default, our VideoLucy employs Qwen2.5-VL-7B as the MLLM and DeepSeek-R1 as the LLM. The inference time overhead mainly stems from two aspects: the MLLM performing inference on short video clips and the LLM conducting inference on long texts. Practically, both can be accelerated using various inference acceleration frameworks to enhance the inference speed.

In our experiments, we deployed Qwen2.5-VL-7B locally on 8 A100 GPUs using VLLM. For a one-hour video, through batch inference, we can obtain text descriptions of each clip within a dozen seconds. As for the LLM, due to the high cost of local deployment, we utilized the official API. Although the API call time fluctuates, making it challenging to provide accurate inference time under fair and reproducible conditions, our practical observations show that for a one-hour video, VideoLucy can generally generate accurate answers with about five API calls.

For fair comparison with other agent-based systems, a feasible approach is to compare the number of LLM calls. This metric avoids the interference of hardware configurations and deployment environments, providing a more objective reference for comparing the efficiency of different agent-based methods.

We have conducted a quick experiment to briefly compare the number of LLM calls. The experimental setup was as follows: one question-answer pair was randomly selected from each of the 42 videos in EgoMem to form a test set. Then, three agent systems -- VideoAgent, VideoTree, and VideoLucy -- were tested, with the maximum number of LLM calls limited to 20.

The test results showed that the average number of calls required for the three systems was 14.1, 9.6, and 6.3 respectively. This indicates that our VideoLucy has an advantage over other methods in terms of inference speed.

---

Q3: "Consider discussing prior memory-based methods for long video understanding."

A: Thank you for your valuable recommendation of these two seminal works in long video understanding.

LangRepo [R1] pioneers a framework for long video understanding by maintaining a structured language repository tailored for LLMs. By iteratively updating multi-scale video chunks and pruning redundant text through write/read operations, it efficiently captures temporal information while preserving interpretability -- effectively translating video content into a language-aligned format that better fits LLM processing paradigms. Its state-of-the-art zero-shot performance on benchmarks like EgoSchema and NExT-QA highlights its success in balancing long-term context modeling and computational efficiency.

Token Turing Machines (TTM) [R2] presents a seminal contribution to long video understanding by introducing an efficient memory-augmented Transformer framework. Inspired by Neural Turing Machines, TTM integrates an external memory module that dynamically summarizes historical frames into tokens, enabling bounded computational complexity while processing long sequences. This design elegantly addresses the scalability challenge inherent in standard Transformers, as each new observation interacts only with memory tokens rather than the entire history. 

Our VideoLucy, as a memory-based framework, builds on these advancements: we extend their core ideas via hierarchical memory structures and agent-driven backtracking, thereby enhancing recall granularity for complex long videos requiring fine-grained details.

We have incorporated a detailed discussion of these two works in the revised manuscript.

---

References:

• [R1] Kahatapitiya, K., Ranasinghe, K., Park, J., & Ryoo, M. S. (2024). "Language Repository for Long Video Understanding." arXiv preprint arXiv:2403.14622.
• [R2] Ryoo, M. S., Gopalakrishnan, K., Kahatapitiya, K., Xiao, T., Rao, K., Stone, A., ... & Arnab, A. (2023). "Token Turing Machines." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19070-19081).

---

Last but not least, we would like to sincerely thank Reviewer YuVT again for the valuable time and constructive feedback provided during this review.

We sincerely thank Reviewer 4wav for the constructive and thoughtful review. We greatly appreciate your recognition of the proposed agent-based framework, our newly introduced EgoMem benchmark, and the strengths of our empirical results. Below, we respond in detail to your insightful questions and concerns.

---

Q1: "While the integration of agents is effective, the framework builds on existing paradigms without introducing fundamentally new roles beyond refined prompting."

A: Thank you for your valuable feedback regarding the agent architecture. We appreciate the chance to clarify how our work advances beyond refined prompting -- a distinction central to our contributions.

While agent-based systems for long video understanding do share high-level iterative reasoning paradigms, VideoLucy introduces fundamental architectural innovations in memory mechanism design that distinguish it from existing systems. As highlighted in our abstract, the core innovation lies in the hierarchical memory structure with progressive granularity, which explicitly models detail levels and temporal scopes across hierarchical depths. This structure is not merely a prompt engineering technique but a structural redesign of how agents store, retrieve, and integrate video information over time.

Specifically, our deep memory backtracking mechanism represents a departure from conventional agent workflows. Unlike existing systems that either operate on flat memory buffers or fixed temporal windows, VideoLucy's agent employs a systematic backtracking process inspired by human recollection -- starting from coarse temporal contexts and iteratively drilling down into finer-grained details. This requires the agent to dynamically adjust memory access patterns based on accumulated information, a capability that not only stems from the hierarchical memory's explicit encoding of temporal relationships (rather than prompt heuristics alone) but also endows the agent with a novel "memory coordinator" role: unlike existing agents that function solely as "information processors", ours actively plans the timing and granularity of memory retrieval, a role unmodeled in prior paradigms.

Our experimental results (Figure 6) further validate this architectural novelty: VideoLucy's performance gains stem from the synergy between its memory hierarchy and iterative backtracking, not just prompt refinements. For instance, on the Video-MME long split, we observe that ablating the hierarchical memory structure (while keeping prompting strategies identical) leads to a 24.7% drop in accuracy, confirming the structural contribution of our memory mechanism.

We acknowledge that prompting coordinates agent behaviors, but it serves as an interface to execute the underlying architectural innovations rather than being the primary source of novelty. Our innovative design of VideoLucy provides a new and effective approach to long video understanding, making a valuable contribution to the community.

We will strengthen this distinction to better highlight these structural advancements.

---

Q2: "Efficiency concerns: Despite the hierarchical sampling, the iterative backtracking mechanism can be computationally intensive and context-length constrained, especially for ultra-long videos where multiple iterations are required."

A: Thank you for your valuable feedback regarding the computational intensity and context-length constraints of the iterative backtracking mechanism, especially for ultra-long videos.

As for these two aspects, we would like to draw your attention to the Section A.2 Limitation (on Page 15) of our manuscript, where they are already discussed. We elaborate again as follows for your convenient reference:

• Regarding the computational overhead, we acknowledge that agent-based long video understanding systems, including ours, generally require multiple iterations to achieve accurate answers, which indeed results in higher computational costs compared to traditional end-to-end video MLLMs. However, considering that the memories obtained in our framework can be stored, for multiple different questions about the same video, we can fully leverage the video's pre-existing memories to achieve faster responses, instead of repeatedly performing inference on the video's information, as existing video MLLMs do. This can reduce the time overhead of reasoning to a certain extent, which is conducive to deployment in practical applications.

• As for the context-length constraint, the maximum video length that VideoLucy can process is currently bounded by the LLM's maximum context limit, which poses a challenge for processing super-long videos exceeding existing benchmarks (e.g., hundreds of hours). Nevertheless, since VideoLucy is a training-free framework, it can seamlessly integrate with any LLM with enhanced capabilities. As LLM technology advances, we anticipate steady improvements in both the performance of VideoLucy and the maximum video length it can handle. Notably, despite this constraint, VideoLucy has achieved leading experimental results on our EgoMem benchmark, where the average video duration is 6.33 hours -- one of the longest among existing benchmarks to date. This demonstrates that our framework is already capable of effectively handling extremely long videos within the current context-length limits of state-of-the-art LLMs.

---

Q3: "The proposed method performs better than prior agent-based methods across benchmarks. However, there lack of an evaluation on the component. What are the necessary designs for the proposed method? Does the improvement over prior work primarily come from the usage of the latest models?"

A: Thank you for your attention to the systematic evaluation of our method components and the source of improvements. We would like to address this from two aspects:

• On one hand, we have conducted ablation studies on the core components of VideoLucy, namely the hierarchical memory structure and iterative backtracking mechanism, as presented in our paper. In Figure 5, we verify that both the richness of memory information and its relevance to the question gradually increase during the memory backtracking process. Additionally, in Figure 6, we investigate the "Effects of different memory depth and maximum iteration number", which validates the effectiveness of the hierarchical memory structure and helps us select a trade-off number of iterations. Admittedly, similar to previous methods, since VideoLucy is an integrated agent system and long video understanding is a resource-intensive task, we have not performed ablation experiments on each role-specific agent within the system. However, the experiments in Figures 5 and 6 sufficiently demonstrate the effectiveness of the core designs of VideoLucy.
• On the other hand, to ensure fair comparisons, when we reproduced the prior agent-based methods (VideoAgent and VideoTree) on our EgoMem benchmark, we kept the MLLM and LLM consistent with those used in VideoLucy, specifically Qwen2.5-VL-7B and DeepSeek-R1. As shown in Table 4, VideoLucy outperforms these two methods by 25.2% and 23.6% in accuracy, respectively. This fair setting clearly indicates that the improvement stems from our method's design rather than the use of the latest models. We will emphasize such a consistent and fair setup in the experimental explanations.

We believe these experiments adequately validate the necessity and effectiveness of our key designs, including the hierarchical memory structure and iterative backtracking mechanism.

---

Q4: "Hyperparameters optimized for each benchmark: In L241-242, the temporal scopes were tuned for each single benchmark. How were these parameters determined? Similarly, how were Ks determined (L139-140)?"

A: Thank you for your concern regarding the determination of hyperparameters for each benchmark.

The tuning of temporal scopes for different benchmarks is primarily driven by the significant variations in video durations. Using a fixed temporal scope across all benchmarks would be impractical, as illustrated by the following example: a coarse-grained memory scope of 50 seconds would result in only 2 coarse memories for a 100-second video but 72 coarse memories for a 3600-second video.

We conducted preliminary qualitative experiments on several videos of varying lengths and found that too few coarse memories would limit the initial information available during iterative backtracking, thereby increasing the number of iterations required for memory search. Conversely, an excessive number of coarse memories would introduce redundant information, unnecessarily expand the context length, and interfere with effective memory retrieval.

To address this, we determined the temporal scopes based on a principle that ensures the number of coarse-grained memories falls within a reasonable range (typically 10–36) across all benchmarks. Additionally, we maintained a consistent hierarchical relationship where the scope of finer-grained memories is set to approximately one-tenth of the scope of the level above. This design balances the trade-off between initial information richness and context length efficiency, while preserving the hierarchical memory structure's integrity.

Detailed values of the temporal scopes and Ks for each benchmark are provided on Page 1 of the supplementary material. We acknowledge that this benchmark-specific tuning involves empirical considerations, but we emphasize that it is guided by clear principles derived from the structural properties of the datasets (e.g., video duration, event density). We have added a detailed explanation of this part in the supplementary material.

---

Last but not least, we would like to sincerely thank Reviewer 4wav again for the valuable time and constructive feedback provided during this review.

We sincerely thank Reviewer RcwM for the positive and constructive review. We are grateful for your recognition of our idea, strong experimental results, and clear manuscript presentation. Below, we address your insightful comments and suggestions in detail.

---

A.1: "Please consider revising the introduction's first paragraph, especially the section on video understanding. For readers unfamiliar with the movie ‘Lucy,’ it is difficult to grasp the context."

A: Thank you for the suggestion. We have revised the first paragraph to provide clearer context for long video understanding before introducing the analogy to the movie Lucy. The revised paragraph now reads:

"Long video understanding is a highly concerned task, with the core objective of accurately and objectively answering various user questions based on the entire video content. This process demands that a system possess a comprehensive memory and grasp of almost all details within the video; otherwise, information gaps could lead to inaccurate answers.

This need for comprehensive memory brings to mind a scene from the movie Lucy. The protagonist, Lucy, gains full access to her brain's potential due to an accident, acquiring an exceptionally powerful memory. She can recall every detail of her life from birth, even the sensation of her mother stroking her forehead during infancy. This extraordinary ability to trace back and precisely capture all information, whether instantaneous frames or continuous events, is exactly the goal we pursue in the long video understanding task."

We have incorporated this revision into the revised manuscript.

---

A.2: "The concept of the 'human recollection process from coarse to fine' is mentioned but not explained in either the abstract or the introduction."

A: We agree that a clearer explanation of this concept would help readers better understand our method's motivation. Drawing from cognitive science, human recollection typically proceeds from coarse to fine, starting with general impressions and gradually retrieving finer details. For example, recalling a birthday party might begin with the memory of a lively room, and then be refined into specifics like a gift exchange or a candle count.

VideoLucy is inspired by this process. It uses a hierarchical memory structure that separates coarse and fine memories across time scales, coupled with agent-based iterative backtracking to simulate how humans actively retrieve and refine information. We have added this explanation and an illustrative example in the revised introduction.

---

B.1: "Have you considered alternative segmentation methods or analyzed the potential drawbacks of uniform segmentation?"

A: This is a valuable point. We initially explored overlapping segmentation (e.g., the first clip covers 0-100 seconds, the second 90-190 seconds) to reduce event truncation. This aimed to mitigate direct event truncation by allowing event endings in one clip to serve as beginnings in the next.

However, experiments showed this design did not improve understanding accuracy and, conversely, increased the number of memories, negatively impacting system efficiency. Thus, we ultimately abandoned overlapping segmentation.

Our qualitative experiments revealed why the overlapping design failed: LLMs possess the ability to integrate and understand continuous events. For example, if an event spans two clips (e.g., 95-105 seconds across 0-100 and 100-200 second clips), during the memory search process, the LLM can still roughly locate that the event should occur at the end of the first clip and the beginning of the second clip. It can then provide a refined search range that connects these two clips, effectively mitigating the potential consequences of direct truncation to a certain extent.

We sincerely appreciate your suggestion, which helps us improve the explanation of our method.

---

C.1: "Runtime and memory requirements."

A: VideoLucy is a training-free agent. Its resource consumption comes mainly from its MLLM and LLM components.

• Memory and runtime requirements: Local MLLM memory usage (Qwen2.5-VL-7B) scales with video frame count and concurrent clips. An 80GB A100 GPU can handle 10 concurrent 200-frame clips in our tests. More memory allows faster, parallel inference. Limited memory necessitates processing fewer clips simultaneously, extending total inference time. For the LLM (DeepSeek-R1), we used its official API due to substantial local deployment requirements. This meant runtime consumption varied with API response times, preventing precise local performance measurement.
• Runtime performance: Our method's runtime is affected by video length, question difficulty, and API response times. To quantify this, we processed 20 questions for each of four videos (400s, 1600s, 2800s, 4000s) from the Needle-in-A-Video-Haystack dataset. The average times are presented below:

As anticipated, runtime scales with video duration, primarily due to LLM API call time. Using a faster, more stable LLM API will accelerate inference. We have included this analysis in the revised version to clearly illustrate these aspects.

---

C.2: "Why do the baselines differ in Tables 1, 2, 3?"

A: Thanks for raising this concern. The differences in the baseline methods among these tables are mainly due to the fact that many video understanding methods do not have officially published test results on all three benchmarks.

As stated in the captions of each table, the results consist of two parts:

• The top-performing results on each leaderboard as of May 15, 2025 (the deadline for paper submission);
• Results from recent, representative papers that, while not yet officially on the leaderboards, offer valuable insights into the latest advancements.

Because resource investments and testing plans vary, many methods have only published results on one or two datasets. This naturally causes discrepancies in the baseline methods available for comparison across the three tables, not due to any deliberate selection on our part.

We hope the above response addresses your concerns.

---

C.3: "It is unclear which models are used in CapAGT, LocAGT, and other variants. It would be better to specify the base models used and compare with their baseline capabilities."

A: We appreciate your feedback about the clarity of the base models and your suggestion for further comparisons. All agents in our system consistently use Qwen2.5-VL-7B as the MLLM and DeepSeek-R1 as the LLM (please refer to Line 240).

As you suggested, we have now included a direct performance comparison between VideoLucy and the original Qwen2.5-VL-7B, using results from its official technical report:

Additionally, within the limited rebuttal time, we have supplemented comparative experiments using different underlying models. The experimental setup is consistent with Table 9 in the supplementary material. Results are as below:

It can be seen that VideoLucy shows performance advantages with each baseline model. We have added these results to the revised supplementary material as you suggested.

---

C.4: "Can you clarify which aspects of your approach contribute most to the improvement?"

A: Thanks for your insightful comment. VideoLucy's significant improvement in temporal understanding primarily comes from two key features:

• Hierarchical memory structure: This structure uses both coarse-grained memory (for broad event-level dynamics) and fine-grained memory (for detailed temporal changes within events). This multi-scale approach helps the model understand both overall event progression and subtle temporal variations.
• Agent-based iterative backtracking: This mechanism integrates memory across different time segments, allowing the model to establish cross-event temporal connections. This is crucial for understanding how events unfold and interact, especially for long-term temporal dependencies in EgoMem.

Together, these two aspects enable VideoLucy to effectively capture temporal dependencies and integrate information over extended periods, leading to its superior performance on EgoMem's temporal understanding tasks.

---

C.5: "Please analyze the improvements achieved on different tasks."

A: Thank you for your valuable feedback. We evaluated VideoLucy on three benchmarks, focusing on different aspects of long video understanding. For detailed analysis, we use LVBench as an example due to its explicit task categorization.

LVBench includes six task categories: Temporal Grounding (TG), Summarization (Sum), Reasoning (Rea), Entity Recognition (ER), Event Understanding (EU), and Key Information Retrieval (KIR).

VideoLucy significantly improved performance on EU (9.1%), KIR (13.4%), and TG (6.2%). This highlights our framework's strength in capturing continuous event dynamics and preserving critical details through its hierarchical memory and iterative backtracking. VideoLucy also showed smaller yet still significant gains on ER (0.9%), Rea (1.2%), and Sum (0.3%), further demonstrating its overall effectiveness in video understanding.

Task-specific results confirm that performance gains on various benchmarks are mainly due to improved temporal understanding and detail preservation. We hope this addresses your concerns.

---

We would like to sincerely thank Reviewer RcwM again for the valuable time and constructive feedback provided during this review.

Dear Reviewer RcwM,

Thank you once again for your thoughtful and constructive review!

In response to your suggestions, we have revised the introduction for improved clarity, expanded the explanation of our human-inspired recollection motivation, and clarified the segmentation strategy and runtime analysis. We also added comparisons with base models, specified the agent components, and included task-level analyses to highlight contributions to temporal understanding.

Your feedback has greatly helped us improve the clarity and completeness of our work.

We are actively participating in the Author–Reviewer Discussion phase and would be happy to provide further clarification if needed.

Looking forward to your thoughts!

Best regards,

The Authors of Submission 1303

We sincerely thank Reviewer Dmx9 for the thoughtful feedback and recognition of our work’s motivation, contributions, and empirical results. Below, we address each concern in detail.

---

Q1: "Is there a problem with the $y_i = k$ condition in the Line 132? In this way, only the first K video frames can be sampled, and each frame is a clip?"

A: Thank you for raising this question. It is likely that our unclear expression has led to a possible misunderstanding. The process of dividing the clips is as follows:

For a video with $N$ frames, we use $f_i^{y_i}$ to represent each frame, where $i$ is the index of the frame in the video, and $y_i$ is the label indicating which clip the frame belongs to. Then we divide the video into $K$ short clips, and we use $v_k$ to represent each short clip, where $k$ is the index of the short clip. The meaning of $f_i \in v_k$ if $y_i=k$ is that if the clip label $y_i$ of $f_i$ is $k$, then this frame belongs to the $k$-th segment.

In this way, we can control the number of frames in each short clip by simply setting different values of $K$, that is, controlling the temporal scope of memory. When $K=1$, no division is performed, and the memory degrades into a rough understanding of the entire video. When $K=N$, the division is done frame by frame, with each frame being a short clip.

Following your suggestion, we have polished the expression of this part in the revised version to avoid possible misunderstandings.

---

Q2: "The method in this paper needs to design different hyperparameters for different datasets. Can some general design references be given?"

A: Thank you for your concern regarding the hyperparameters for different datasets.

Tuning temporal scopes (i.e., the time window sizes for partitioning coarse/fine-grained memories) across datasets is necessary due to significant variations in video durations; a fixed scope would be impractical.

Preliminary tests on videos of varying lengths showed that too few coarse-grained memories restrict initial information for iterative backtracking, increasing search iterations. Conversely, excessive coarse-grained memories introduce redundancy, unnecessarily lengthen context, and hinder retrieval efficiency.

To address this, we derived general design principles through empirical comparisons across multiple datasets:

• For coarse-grained memories: The temporal scope should be set such that the number of coarse-grained memories falls within a reasonable range (typically 10–36). This range is chosen based on common video durations (from minutes to hours in most datasets) to balance initial information richness and context efficiency.
• For finer-grained memories: The temporal scope is approximately one-tenth of the level above. This serves as a starting point and can be adjusted based on the dataset's sensitivity to detailed information.

These principles balance the trade-off between initial information richness and context length efficiency while preserving the integrity of the hierarchical memory structure. They also provide a general reference for initial hyperparameter tuning on new datasets, reducing trial-and-error efforts.

---

Q3: "Among the 6 types of event understanding tasks in the EgoMem benchmark, Event Order, Event Rectification, and Event Reconstruction essentially test the same ability of the model. What is the basis for this split?"

A: Thank you for your valuable comments. While the three tasks do share a foundation in assessing models' ability to understand temporal event sequences, their core objectives and required capabilities differ.

• The Event Order task centers on accurately selecting events within a target time window from a set containing distractors, then establishing their temporal sequence. This demands not only recognition of events' temporal attributes but also the ability to identify and exclude irrelevant information, ultimately posing a dual challenge of "screening and ordering" event-time associations.
• The Event Rectification task focuses on error diagnosis and correction. The model must first pinpoint temporal logic inconsistencies in a given sequence, then generate a corrected sequence aligned with video content and explain the error's cause. Beyond temporal order comprehension, this task emphasizes mastery of event causality and logical coherence, serving as an evaluation of both "error-correcting ability" and "explanatory capacity".
• The Event Reconstruction task prioritizes event completion and contextual reasoning. When faced with masked events, the model must infer content consistent with both timeline and scenario plausibility, drawing on logical connections between preceding and subsequent events. This task tests the model's grasp of event flow integrity and its capacity for "context-driven filling-in reasoning".

Our intent in designing these three tasks is to comprehensively evaluate how deeply models understand event temporal logic in long videos through multi-dimensional testing. A single task might only reflect performance in one scenario, but combining multiple tasks more effectively verifies whether models have truly mastered the dynamics of event associations, strengthening the benchmark's rigor and discriminative power.

We have incorporated these design rationales into the supplementary material to further clarify the differences among the three tasks.

---

Q4: "Although the problems on the EgoMem benchmark involve the understanding of multiple events, the events are relatively independent of each other and lack interactive relationships. Have the authors considered a type of question that requires a comprehensive review of information from multiple events to get an answer, such as: What do the people in event 1 and event 2 accomplish together?"

A: Thank you for your valuable comments. We fully agree with your view that the interaction between multiple events is an important dimension in ultra-long video understanding, as it can more comprehensively evaluate a model's ability to conduct a comprehensive analysis of complex scenarios. In our subsequent work, we plan to address such issues, and some preliminary design ideas are as follows:

• First, we will sort out the timeline and logical connections of different events in the video, and identify event combinations with interactive relationships such as causality, synergy, and contrast. For example, in a video recording of daily work, there is an obvious synergistic relationship between the "material preparation" event in the morning and the "project report" event in the afternoon.
• Second, for these event combinations with interactive relationships, we will design diversified types of questions. In addition to the one you mentioned, "What did the people in Event 1 and Event 2 accomplish together?", there will also be questions like "What impacts did the occurrence of Event 1 have on Event 2?" and "In what aspects does Event 2 differ from Event 1?", so as to comprehensively examine the model's ability to understand and integrate information about multi-event interactions.

Following your suggestion, we commit to carrying out relevant annotation work in the future and continuously improving the EgoMem benchmark, enabling it to more effectively evaluate the performance of ultra-long video understanding models.

---

Q5: "GPT-4o's ability to understand long videos is not its strength. If the proposed method can be compared with a model such as Gemini2.5-pro that has optimized long video understanding ability, it will better illustrate the effectiveness of this method."

A: Thank you for your valuable feedback. We fully agree that a comparison with models like Gemini 2.5 Pro, which are specifically optimized for long video understanding, would more comprehensively demonstrate the effectiveness of our proposed method.

We would like to clarify that due to Gemini 2.5 Pro's public release date of March 25, 2025, its performance data was not yet available on the leaderboards of mainstream long video understanding benchmarks by the NeurIPS 2025 submission deadline of May 15, 2025. Conducting a full evaluation by calling its API on a large number of videos would have incurred a significant time cost. Given these practical constraints, we chose to compare our method with the latest leaderboards and representative SOTA papers updated before the submission deadline, among which GPT4o is a representative leading method.

To address your concern within the rebuttal period, we conducted an additional, expedited experiment: the Needle-in-A-Video-Haystack test (please see Section 4.2 of our paper for the experimental setup). In this test, we used a 4000-second-long video to evaluate the model's ability to accurately locate and understand key information within extremely long content. The results are as follows:

• Gemini 2.5 Pro: 70% accuracy
• VideoLucy (Ours): 80% accuracy

This result provides preliminary evidence that our VideoLucy framework demonstrates competitive performance, even when compared against a state-of-the-art model optimized for long videos.

Finally, we plan to include the full results of Gemini 2.5 Pro on our newly proposed EgoMem benchmark in a future version of the manuscript. This will provide more comprehensive and convincing experimental support for our conclusions.

---

Q6: "The writing of this paper can be further polished (such as the expression of Lines 128-146)."

A: Thank you for your suggestion. As stated in the first question, we have further polished the expression of this part to make it simpler and understandable.

---

Last but not least, we would like to sincerely thank Reviewer Dmx9 again for the valuable time and constructive feedback provided during this review.