Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video

Thanks for your exceptionally thorough review and your detailed, positive summary of our work's originality and significance. We appreciate the precise suggestions for improvement, which we address in detail below.

W1: Inconsistent notations

1. W1.1: Yes, in the LIVE training dataset, only answer timestamps are provided, not answer intervals. Therefore, $\mathcal{T}\_{timestamp}=\mathcal{T}\_{GT}$.
2. W1.2: No, we generate the ESTP-IT dataset, which includes answer interval annotations. To alleviate the conflicts that arise from timestamp-level supervision between adjacent frames, we incorporate interval-level temporal supervision signals. Concurrently, we employ a modified loss within these intervals to enable the model to discern subtle variations among frames within the given interval.
3. W1.3: Thank you for your suggestion. We provide a detailed definition of $\mathcal{T}\_{uncertain}$ in Appendix D (Stage-2 Input), and we'll revise the main text to include this information.
4. W1.4: Yes, thank you for pointing that out; we'll make the necessary revisions.
5. W1.5: Thank you for your correction. We'll amend this issue, as "scrutinize loss" in Eq. (7) refers to "ask_high loss" in Eq. (5).

W2: Detail of Proactive Dynamic Compression

For a multi-turn conversation, we forward turn by turn, inputting only the system prompt, compression tokens from historical turns, and all tokens from the current turn. This approach enforces compression during training. We provide an example in Appendix D Proactive Dynamic Compression Mechanism Example.

W3: Caption for Figure 2

Thank you for your suggestion. We will revise the caption for Figure 2, replacing "Triangle of Impossibility" with a term like "Multi-objective Trade-off Triangle" to enhance clarity. We also appreciate your recognition of our work's contribution.

W4: COIN training set

In our COIN experiments, we aim to demonstrate the generalizability of our model architecture (VideoLLM-EyeWO). Therefore, we follow the training setup of the baseline (VideoLLM-online), also utilizing the COIN training set. Thank you for your correction; we will subsequently supplement the main text with more detailed settings for COIN training.

W5: Correction on $scale\_k$ Calculation

Yes, thank you for your correction. Values without the maximum also exceed 3.

Q1: Open-source plan

We plan to open-source the entire project, including ESTP-Bench and VideoLLM-EyeWO (including training data and model weights). This effort aims to help researchers reproduce and build upon our work.

L1: How fast does the model run? Is it ready for real-time purpose? If so, on what kind of hardware device?

Our model, as depicted in Fig. 7, operates at an average rate exceeding 2 APS (Answers Per Second) with a 2 FPS (Frames Per Second) input. This experiment runs on an 8-A40 GPU cluster configured with 96 Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz cores. On our 8-H20 cluster, the running speed increases by 1.7 to 2 times. We appreciate your suggestion; further accelerating model execution and identifying suitable hardware setups remain meaningful challenges.

I appreciate the author for providing the rebuttal. My concerns are addressed. I hope the details/corrections will be incorporated in the future draft/revision.

Thank you for your valuable feedback and for recognizing our contributions in establishing the ESTP task. To address your concerns, we provide detailed responses below that clarify the workings of our data generation pipeline and articulate the specific detail of my ESTP-F1 metric and VideoLLM-EyeWO model, distinguishing them from prior art.

W1 and Q1: Data engine detail

W1.1 One-to-Many

In the One-to-Many stage, the core objective is to identify all segments within the captions that can answer the current question-answer (QA) pair. We explored two technical approaches:

1. Inputting all captions into the Large Language Model (LLM): This method involves feeding the entirety of the captions into the LLM, tasking it with identifying all relevant segments and subsequently generating an answer. While this approach yields optimal results, it incurs substantial token consumption.
2. Keyword-based Retrieval and Supplementary Generation: During the one-to-one QA generation phase, we instruct the model to generate keywords pertaining to visual cues. These keywords are then used for retrieval, selecting relevant captions from the Multi-Perspective Caption. These selected captions are then input into the LLM, which is tasked with discerning their ability to answer the question and generating supplementary answers as needed.

W1.2 Integration of Multi-turn QA

Multi-turn QA is constructed by combining multiple single-turn QA pairs that pertain to the same video segment. Specifically, we leverage an LLM to identify logically related QA pairs and subsequently sequence them. Following this established order, the subsequent question is appended after a randomly selected number of preceding answers (one or more). This process is entirely rule-based and does not necessitate any manual annotation. Finally, an LLM is utilized to refine the generated text, ensuring it conforms to a fluent, streaming conversational style.

W1.3 Definition the Response Time

During training, response times are randomly sampled from the latter 50% of the answer interval. The generation of the answer interval itself is derived from the time intervals of relevant captions. The methodology for obtaining these caption intervals is detailed in Appendix C.5: Multi-Perspective Caption Generation. Briefly, for Event Captions, we follow the approach of EgoVLP for calculating interval start and end points. For Scene Captions, we leverage inter-frame similarity to determine potential interval boundaries.

For evaluation, the optimal Response Time definition can be found in Appendix C.3.

W1.4 Manual Annotation

For ESTP-Bench, each data is manually annotated, with specific annotation details presented in Appendix C.2. For Data Engine and ESTP-IT, the data is entirely automatically generated without any manual annotation; more detailed generation processes can be found in Appendix C.4 and C.5.

Thank you for your suggestion. We will subsequently add more details regarding data generation in the Appendix, such as how to prompt the LLM for RAG and the concatenation of multi-turn QA. Concurrently, we plan to open-source the data generation process.

W2: ESTP-F1 metric and the VideoLLM-EyeWO

W2.1 ESTP-F1

Regarding the design of ESTP-F1, our aim was for it to be both simple yet comprehensive. We are the first to introduce F1 as a unified metric in streaming proactive tasks to simultaneously measure both response quality and timeliness. We note that reviewer Cx8y recognized the novelty of ESTP-F1, a point which was also appreciated by reviewer UR4Q.

W2.1 VideoLLM-EyeWO

VideoLLM-online, with its simple training and architecture, is our streaming proactive video LLM baseline. However, it falls short on just-in-time responsiveness, accurate answers, and contextual understanding—all essential for ESTP tasks. Our multi-stage training strategy, proactive dynamic compression, and data engine introduce these critical capabilities. These novel components are unique to our approach and are not found in prior proactive video LLM methods.

We also agree that scalability and generalizability are critical. To validate these aspects, we conduct additional experiments:

(1) Extended Benchmarking: We test our model on other online and offline video benchmarks. This demonstrates its performance across a wider range of video understanding tasks.

Table 2: Performance comparison on the QaEgo4D test set.

QaEgo4D test-online evaluates the model's query performance in an online scenario, where inquiries are made after the answer interval appears. In contrast, QaEgo4D test-offline assesses performance when queries are made after the entire video has been loaded.

(2) Comparison with State-of-the-Art: We also incorporate the latest LiveCC [1] as a baseline. It's one of the most outstanding online video LLMs built on Qwen2-VL, and experimental results effectively showcase the efficacy of our method.

LiveCC*: We re-evaluate LiveCC's performance on our own devices.

[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale

Q2: ESTP-F1 Detail

Specifically, we utilize an LLM to determine the score between the generated text and the ground truth, while also employing the temporal distance of timestamps to quantify response timeliness. Further details are comprehensively explained in Appendix C.3.

Q3: VideoLLM-EyeWO shows strong potential to extend beyond its current tasks

1. VideoLLM-EyeWO is not a model tailored solely for the tasks proposed in our paper; instead, VideoLLM-EyeWO follows a general paradigm that supports adaptation to a wide range of tasks (see our response to W2.1).
2. The proactive task proposed in our paper is not only tailored for specific AI assistant scenarios, but also serves as a unified paradigm for conventional tasks. For example, it reduces to answering a single question given an image in typical image-based QA, and to performing short-range reasoning over a video clip in standard video QA. Therefore, a model like VideoLLM-EyeWO—designed to handle more generalized task formulations—holds strong potential for broader applications beyond those discussed in our paper.

L1: VideoLLM-EyeWO shows strong Scaling Laws

1. The model scales well with size, adapting effectively to various tasks. For the newly proposed proactive task, we successfully trained a model of size 8B, demonstrating its effectiveness in handling the task and achieving promising results.
2. The data scales well with size, with more data leading to significant performance improvements. This is evident from our experiments, where increasing the dataset size consistently improved the model’s accuracy and generalization ability, particularly in complex tasks.
3. Since the data generation is fully automated, it allows for efficient scaling, ensuring a continuous supply of diverse and high-quality data without manual intervention.

I appreciate the authors for providing the detailed responses. Most of my concerns have been addressed. The additional experiments on scalability and generalizability are convincing. I hope these results and details will be incorporated in the future draft or revision. I would like to maintain my concern regarding the novelty of the proposed models and metrics, but overall, it is a good effort to construct such an interesting benchmark. Therefore, I prefer to keep my original score.

Dear Reviewer jDVL,

This is the last call for author-reviewer discussion, which will ends on Aug 8. Could you please read authors' rebuttal and confirm whether your concerns have been addressed asap? Thank you.

Thank you for your valuable feedback and for acknowledging our contributions in defining the ESTP task. Following your suggestions, we have provided detailed clarifications on the "scrutinize loss" and data engine below, along with new experimental tables for OVO-Bench.

W1: Typo of "scrutinize loss".

We apologize for the oversight; the term "scrutinize loss" in Equation (7) indeed refers to the "ask_high loss" introduced in Equation (5). We will rectify this inconsistency in subsequent revisions.

W2: Performance on other online benchmarks.

Thank you for your valuable suggestion. We have conducted experiments on OvO-Bench to demonstrate our generalization capability. Concurrently, we selected VideoLLM-online as our baseline, given that it shares the same base model (LLaMA3 and SigLIP) and data source (Ego4D) as our own model.

Compared to the baseline, our model demonstrates significant improvements. We also incorporate the latest LiveCC [1] as a baseline. It's one of the most outstanding online video LLMs built on Qwen2-VL, and experimental results effectively showcase the efficacy of our method.

LiveCC*: We re-evaluate LiveCC's performance on our own devices.

[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale

W3: Data engine detail.

Due to main text page limitations, relevant details are provided in Appendix C.4 and C.5, with a framework diagram presented in Appendix Fig. 2.

W4: Reference format error.

Thank you for pointing out this oversight. We sincerely apologize for not thoroughly checking this issue, which stems from our reference export process. We will correct all citation formats throughout the paper.

Q1: Stage 3 and stage 2 APS comparison.

The KV Cache Size decreased after training with multi-turn QA (Stage 3) (from 3731.8 to 3242.8 as shown in Appendix Tab. 1). This reduction indicates that the model decreased the frequency of triggering responses. This, in turn, suggests that after multi-turn, more refined data training, the phenomenon of over-responding was reduced, leading to an improvement in APS.

Q2: Detail of proactive dynamic compression.

Q2.1 What are the two levels?

Given the inherent characteristics of streaming proactive tasks—specifically, that information at locations where the model initiates a request is more critical than information at silent locations—we assign a lower compression ratio to "request-triggering" locations (e.g., high-resolution frames, answers).

Q2.2 How many tokens are compressed?

As shown in Appendix Table 1, an average of 90% of tokens are compressed.

Q2.3 What low/high compression rates did you choose?

We do not choose specific, fixed numerical values for the low and high compression rates. Instead, the compression ratio is dynamic and relative, emerging naturally from our compression mechanism, which is designed for the unpredictability of streaming scenarios.

Here is the principle:

• Our method applies a fixed number of compression tokens to all regions, regardless of whether they are "request-triggering" or "silent."
• A relatively low compression rate is inherently applied to "request-triggering" regions (such as high-resolution frames or answers). Because these critical regions contain significantly fewer original tokens, adding the fixed amount of compression tokens has a smaller relative impact, thus preserving their information with higher fidelity.
• A relatively high compression rate is applied to "silent" regions. These regions are often long, containing many consecutive low-resolution frames. The same fixed number of compression tokens represents a more aggressive compression for these larger segments.

This approach is crucial for streaming environments where the length of visual or textual inputs cannot be determined beforehand, as it allows the compression to adapt dynamically without relying on static, predefined ratios.

Thank you for your feedback and for recognizing the contributions of our new benchmark. To address your concerns, we have provided detailed responses below that further clarify the methodological novelty and present additional experimental results to support our claims regarding strong performance and generalizability.

W1: Novelty

Both our complete framework—from data engineering to training and architecture (i.e., compression)—and its components are novel, rather than mere incremental extensions of prior work. The components mentioned by the reviewer cannot be readily adapted to ESTP, and straightforward combinations fail to address its challenges.

We outline the core innovations of our framework, demonstrating how each part meets the unique demands of ESTP and why the mentioned components fundamentally fall short.

W2: Effect of ESTP-IT

In Appendix B, we conducted ablation experiments on ESTP-IT. As shown in Appendix Table 1, lines 1 and 2, ESTP-IT effectively improves the performance of VideoLLM-online. However, constrained by the training methodology, it was unable to achieve optimal performance. This further validates the effectiveness of our multi-stage training strategy.

In the table below, we present the requested experimental results.

W3: Limited discussion of model generalization

Thanks for the suggestion! We've expanded our evaluation to include more benchmarks and additional baselines.

1. More benchmarks:

We select VideoLLM-online as a baseline due to its shared base model (LLaMA3 and SigLIP) and data source (Ego4D) with our VideoLLM-EyeWO. Our model consistently outperforms the baseline in both online and offline tasks, demonstrating strong generalization across diverse scenarios.

• OVO-Bench for online video understanding.

  Table 2	OvOBench Real-Time Visual Perception	OvOBench Backward Tracing
  VideoLLM-online-8B	20.79	17.73
  Ours (VideoLLM-EyeWO-8B)	32.76	28.00

• QaEgo4D for offline video understanding.

  Table 3	QaEgo4D test-online	QaEgo4D test-offline
  VideoLLM-online-8B	29.8	30.20
  Ours (VideoLLM-EyeWO-8B)	36.2	33.00

  QaEgo4D test-online evaluates the model's query performance in an online scenario, where inquiries are made after the answer interval appears. In contrast, QaEgo4D test-offline assesses performance when queries are made after the entire video has been loaded.

2. More baselines: We adopt the most recent LiveCC [1] as our baseline. Table 3 in our response to Reviewer jDVL consistently demonstrates the improvements achieved by our method, showcasing our model's strong generalization capability.

[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale

Thank you for your valuable suggestions for strengthening the evaluation and clarifying the technical details. We have provided detailed answers to each of your points below, supported by corresponding experimental tables.

W1: Limited discussion of model generalization.

Thanks for the suggestion! We've expanded our evaluation to include more benchmarks and additional baselines.

1. More benchmarks:

We select VideoLLM-online as a baseline due to its shared base model (LLaMA3 and SigLIP) and data source (Ego4D) with our VideoLLM-EyeWO. Our model consistently outperforms the baseline in both online and offline tasks, demonstrating strong generalization across diverse scenarios.

• OVO-Bench for online video understanding.

  Table 1	OvOBench Real-Time Visual Perception	OvOBench Backward Tracing
  VideoLLM-online-8B	20.79	17.73
  Ours (VideoLLM-EyeWO-8B)	32.76	28.00

• QaEgo4D for offline video understanding.

  Table 2	QaEgo4D test-online	QaEgo4D test-offline
  VideoLLM-online-8B	29.8	30.20
  Ours (VideoLLM-EyeWO-8B)	36.2	33.00

  QaEgo4D test-online evaluates the model's query performance in an online scenario, where inquiries are made after the answer interval appears. In contrast, QaEgo4D test-offline assesses performance when queries are made after the entire video has been loaded.

• COIN dataset for offline temporal prediction.

  Table 3	Step	Task	Next	Proc	Proc+
  VideoLLM-online-8B	63.1	92.6	49.0	49.7	53.6
  VideoLLM-MOD	63.4	92.7	49.8	49.8	53.3
  Ours	65.9	92.7	50.9	50.8	54.7

2. More baselines: We adopt the most recent LiveCC [1] as our baseline. Table 3 in our response to Reviewer jDVL consistently demonstrates the improvements achieved by our method, showcasing our model's strong generalization capability.

[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale

W2: The ask_high’s ablation study and other key novelty.

• Indeed, ask_high is one of our core novelties, making it highly suitable for streaming and proactive scenarios. It processes key frames at high resolution for frame-by-frame input, providing crucial information for response decision-making and answer generation. Table 4 (lines 4 and 5) further confirms its critical role in these processes.
• Beyond ask_high, our framework introduces several other significant core contributions:
  1. Interval Responsiveness: A new training method and loss function to handle the "silence to response" shift without manual thresholds (in Sec 3.2 Stage-1). Without this, a trade-off between over- and under-responsiveness emerges, requiring manual tuning and leading to suboptimal performance. Unlike the abrupt supervisory signals in LIVE, our method smooths the signal over an interval, fostering a more stable and effective training process.
  2. Proactive Dynamic Compression : A novel compression strategy tailored for streaming, multi-turn scenarios (in Sec 3.3). Traditional compression fails in these contexts because single-turn schemes don't suit interleaved visual/textual streams, and pre-compressing prevents the LLM from perceiving every frame, hindering decision-making.
  3. Data Engine: A new multi-stage pipeline for generating high-quality, proactive training data. Unlike prior video QA, proactive task requires just-in-time and contextual information. Our framework automatically derives captions with time interval for a video , then generating QA pairs for critical segments, finally completing answers and merging questions via captioning.
  4. Holistic Multi-Stage Strategy : The curriculum learning approach itself, which outperforms single-stage training.

W3: Ablation studies.

• Our ablation studies, detailed in the table below and Appendix B, confirm the significant impact of ask_high (Table lines 4-5). These studies also highlight the crucial roles of ESTP-IT, multi-stage training, and the proactive dynamic compression mechanism.
• As suggested, the Table 4 also includes KV Cache Size statistics for dynamic compression (lines 3-4), demonstrating a significant reduction without performance compromise.

Table 4: Ablation Study on ESTP Bench.

W4: Complexity of the pipeline.

Our pipeline comprehensively explores the ESTP task from three distinct yet interconnected perspectives: data, training, and architectural design. While each angle offers a unique focus, they share a common, intuitive design motivation: to efficiently and accurately answer questions at precise moments within a streaming scenario.

We intend to open-source the entire project, including ESTP-Bench, VideoLLM-EyeWO (encompassing training data and model weights), to facilitate reproducibility and further research within the academic community.

W5: Basic settings.

1. W5.1 Input Format : The model processes input frame by frame at 2 FPS. After each frame, the model outputs a decision: continue to the next frame or generate a response. (See Appendix D for a LIVE Input example.)
2. W5.2 historical trajectory storage: Specifically, after the model decides to respond and generate an answer, Proactive Dynamic Compression is executed, storing only compressed, compact tokens in the KV cache. We also keep the System Prompt at the very beginning of the KV cache. (Appendix D, "Proactive Dynamic Compression Mechanism Example," offers more detail.)
3. W5.3 Specific Resolution: Following SigLIP, our input frames use a 384x384 resolution. Low-resolution frames are pooled into 10 tokens (1+3x3), consistent with VideoLLM-online, while high-resolution frames are represented by 50 tokens (1+7x7). (Appendix E provides more detail.)

Q1: Training of proactive dynamic compression.

Q1.1: How the model learns to determine when to compress and which compression level to apply?

Given the nature of streaming proactive tasks—(1) where each response turn naturally segments context and (2) information from "request-triggering" positions is more crucial than from "silent" ones—we employ the following strategy:

1. Compression is performed after a response is completed. This approach prevents interruption of ongoing text generation or the response judgment during frame input.
2. Equal amounts of compression tokens are appended to both "request-triggering" regions (e.g., high-resolution frames, answers) and "silent" regions. This inherently results in a dynamic compression ratio: lower for request-triggering regions and higher for silent ones, as the original token count in request-triggering regions is significantly smaller.

This mechanism, combined with the model's proactive behavior in streaming tasks, imbues the model with the ability to perform proactive, dynamic compression through training on the original task.

Q1.2: How is it trained?

For a multi-turn conversation, we forward turn by turn, inputting only the system prompt, compression tokens from historical turns, and all tokens from the current turn. This approach enforces compression during training. We provide an example in Appendix D Proactive Dynamic Compression Mechanism Example.

Q2: Quantify uncertain interval.

We define the model's erroneously triggered response regions as uncertain intervals. Specifically, we first train for one epoch using the annotated answer intervals as the initial uncertain intervals. Subsequently, we perform an inference pass over the data, identifying positions where high-resolution requests occur outside the annotated intervals and designating these as uncertain intervals. This process, along with input examples, is detailed in Appendix D Stage-2 Input.

Thank you for your timely response! We are pleased to hear that our rebuttal has addressed your concerns well, and we appreciate your consideration of improving the rating. We will carefully revise the paper in accordance with your valuable suggestions, including (but not limited to) adding experiments presented in the rebuttal to further evaluate model generalization on additional online and offline benchmarks, incorporating more baselines, and conducting extended ablation studies. We will also improve the clarity of the implementation and experimental details.

In response to your further suggestion for clarification on the implementation and experimental setups, we provide a concise summary below. More detailed information will be added to the main paper:

• VideoLLM-Eye: Detailed training implementation is provided in Appendix E: Training Implementation Detail of our submission.
• OvObench and QaEgo4D: Evaluations on these benchmarks were conducted under zero-shot testing.
• LiveCC-EyeWO: We fine-tuned the LiveCC model using LoRA. The training data consists of the ESTP-IT 80K dataset and 30K QA pairs randomly sampled (due to time constraints) from the LLaVA-Video-137k dataset, which is part of LiveCC’s original training set.

In addition, we will open-source the entire project, including ESTP-Bench and VideoLLM-EyeWO, covering training data, implement details, and model weights.

Thank you again for your constructive feedback and consideration to raise our paper’s rating!

Dear Reviewer Cx8y,

This is the last call for author-reviewer discussion, which will ends on Aug 8. Could you please read authors' rebuttal and confirm whether your concerns have been addressed asap? Thank you.