LiveStar: Live Streaming Assistant for Real-World Online Video Understanding

Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.

[W2]The calculation methods for the evaluation metrics have not been provided. I want to know the specific computational procedures for these metrics.

First, We would like to address Weaknesses 2 by introducing the evaluation metric we proposed, as this will help clarify the context and facilitate a better understanding of Weaknesses 1.

For each video, we have:

• A set of ground-truth (GT) semantic clips $G = \lbrace g_1, g_2, \dots, g_N \rbrace$, where each clip $g_i$ is defined by: a start time $t_{\text{start},i}$, an end time $t_{\text{end},i}$ and a GT caption.

• A set of model-generated responses $R = \lbrace r_1, r_2, \dots, r_M \rbrace$, where each response $r_j$ has an associated response time $t_{\text{resp},j}$.

For each $g_i \in G$, define the set of matching responses as:

• If $|M_i| = 0$, no response is generated for $g_i$; in this case, we assign an empty string $r_j =$ ' ' as the placeholder response for evaluation.
• If $|M_i| \geq 1$, we select the first matching response $r_j \in M_i$ (i.e., with minimal $t_{\text{resp},j}$) as the representative output.

As detailed in Appendix G, SemCor and SumFluen is calculated using GPT-4o as an evaluator. The process is as follows:

1. SemCor (Semantic Correctness): This metric evaluates the quality of an individual response generated for a specific semantic clip. We prompt GPT-4o to score the pair $(r_j, g_i)$ on three dimensions (each on a 0–10 scale): Semantic Accuracy, Language Quality, and Information Completeness.

2. SumFluen (Summarize Fluency): This metric evaluates the holistic narrative quality of the model’s complete output sequence, treating all responses as a unified story. Let the full model narrative be the concatenation of all responses in temporal order:

   $$\mathcal{N}\_{\text{model}} = \text{Concat}\_{j=1}^M (r_j)$$

   Similarly, construct the ground-truth narrative by concatenating all GT captions in temporal order:

   $$\mathcal{N}\_{\text{GT}} = \text{Concat}\_{i=1}^N (g_i)$$

   We evaluate $\mathcal{N}\_{\text{model}}$ against $\mathcal{N}\_{\text{GT}}$ by prompting GPT-4o to assess them on five storytelling-relevant dimensions(each on a 0–10 scale): Writing Logicality, Language Fluency, Writing Conciseness, Semantic Consistency, and Narrative Completeness.

The metrics TimRedun, TimCover and TimDiff are calculated as:

3. TimRedun (Timing Redundancy) Measures the deviation from the ideal of exactly one response per semantic clip:

4. TimCover (Timing Coverage) Indicates the proportion of semantic clips that were covered by at least one response:

5. TimDiff (Timing Difference) Captures the cumulative response latency for all matched outputs within a semantic clip, while penalizing missed clips by their full duration:

[W1]In the ablation of Impact of Response-Silence Threshold, the divergent trends in the three metrics with varying decoding thresholds need a more rigorous theoretical analysis.

From a theoretical standpoint, an ideal model should trigger a response only once at the first frame of each semantic clip and remain silent otherwise. However, in our proposed LiveStar model, the core of our SVeD mechanism lies in its decoding condition:

Here, the tunable scaling factor α directly influences the model's sensitivity to semantic changes. A higher α value raises the perplexity threshold required to trigger a new response, which means the model will wait for a more significant or prolonged semantic shift before acting.

Specifically, when α is small, the model tends to decode more frequently, potentially generating an output at every frame in the extreme case. This leads to an increased number of unnecessary responses within each semantic clip, with responses occurring earlier in time. As a result, the temporal evaluation metrics reflect a smaller TimDiff , a larger TimRedun, and a higher TimCover . In contrast, as α increases, the model becomes more conservative in decoding, resulting in fewer outputs and delayed response timings. In the extreme case, no output may be generated at all. Under such conditions, both TimRedun and TimCover decrease, while TimDiff increases significantly—particularly when a semantic clip receives no response, leading to a substantial penalty.

The experimental results, computed according to the formulas defined in Answer 1, align perfectly with the above theoretical analysis, thereby validating the role of α in regulating the model's response behavior.

[W3]In your paper, you suggest that the EOS token may lead to pre-training misalignment and vocabulary confusion issues ... Could this issue be empirically investigated through experimentation?

We agree that modern LLMs are remarkably robust and can adapt to various fine-tuning objectives. Our intention was not to suggest that the EOS-based paradigm is fundamentally flawed, but rather to explore an alternative that might better align with the core strengths of vision-language models in a streaming context.

For pre-training misalignment, our core idea is to better align the fine-tuning objective with the model's pre-training. In online video, a majority of frames are silent. The EOS paradigm trains the model to map these rich visual inputs to a single, non-descriptive EOS token. While effective, this differs from the pre-training goal of generating descriptive text. Our SCAM training strategy, by contrast, consistently focuses the model on the descriptive task by only computing loss on meaningful frame-caption pairs. We believe this contributes to LiveStar's strong performance on third-party benchmarks like Ego4D (Table 3), where it surpasses other online assistants.

For vocabulary confusion, we agree that this point requires nuance. Our concern was more practical than theoretical, noting that VideoLLM-online’s implementation used common token ',' (comma) for the EOS role. To avoid overgeneralization, we will remove this claim in our revision and clarify that the issue is implementation-dependent.

Finally, we wish to reiterate our deep respect for the pioneering EOS-based works that inspired this research. Our goal is to propose a complementary approach that contributes to the diversity of methods in this exciting field. We believe that exploring different paradigms is valuable, and we sincerely thank you for helping us improve our work.

[W4]In Table 1 (online split), the other models were not trained on the dataset’s training set. Could this introduce potential fairness issues in the comparative evaluation?

We strictly separated the training and testing datasets to ensure a fair evaluation protocol. The model was first trained using our proposed SCAM to learn the alignment between visual and linguistic representations, without any exposure to test data during training.

To provide a more direct and fair comparison of the models' intrinsic capabilities, we conducted evaluations on a pre-existing, third-party benchmark where all models are on a level playing field. As shown in Table 3 (Ego4D Narration Stream benchmark), LiveStar significantly outperforms all other online assistants.

Furthermore, we evaluated this version of LiveStar (without fine-tuning on OmniStar) on the OmniStar-RNG task. The results are as follows:

If you are interested in LiveStar's performance on additional online (e.g., OVBench) or offline (e.g., MVBench, VideoMME, and LVBench) benchmarks, please refer to our response to Reviewer wXqt and j4Ki, where we provide further evaluation results and analysis across a broader set of tasks.

Thank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work.

Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.

[W1] The authors have only demonstrated the effectiveness of their approach on their proposed benchmark and SVBench... It is recommended to provide ... additional online video benchmarks (e.g., OVBench) and offline video benchmarks (e.g., videomme) ...

We appreciate the reviewer's suggestion for broader benchmarking. Our initial evaluation already spanned three distinct benchmarks designed to cover a variety of online video understanding scenarios: our proposed OmniStar benchmark (Tables 1 & 2), Ego4D Narration (Table 3), and SVBench (Table 6). These benchmarks collectively assess the core capabilities of proactive, real-time response generation that are central to our work.

Nevertheless, we fully agree that validation on additional standard benchmarks is important. Following your suggestion, we have conducted new experiments on both online and offline benchmarks to further demonstrate the generalizability of LiveStar.

Performance on Additional Online Benchmark (OVBench): we conducted additional experiments on OVBench: LiveStar significantly outperforms VideoLLM-online, MovieChat, and Flash-VStream (reported in the original paper), though it trails VideoChat-Online marginally. We attribute this to two factors:

• LiveStar optimizes for text-generation tasks (e.g., narration, QA), while OVBench emphasizes multiple-choice QA.
• OVBench evaluates passive-response models (no output-timing decisions), whereas LiveStar and VideoLLM-online are active-response systems. Task objectives thus differ, and we will discuss this nuance in the revision.

Performance on Additional Offline Benchmark (VideoMME):

These results on both OVBench and VideoMME confirm that LiveStar not only excels at its primary task of online, proactive video understanding but also maintains robust foundational capabilities. For a more detailed discussion of our evaluations on other benchmarks, such as MVBench and LongVideoBench, we invite you to see our response to Reviewer wXqt.

[Q2] Compressing frames into 16 tokens may miss subtle visual cues (e.g., small object interactions, fine-grained actions), as acknowledged in the limitations.

You are absolutely correct, and we acknowledge this trade-off in our limitations section. Compressing each frame into a fixed-size representation of 16 tokens is a deliberate design choice aimed at balancing computational feasibility and semantic comprehension, which is crucial for real-time processing of long video streams.

To enable real-time, long-context video understanding, LiveStar represents video frames with 16 tokens, which can efficiently process more than 10 minutes of video within an 8K context window. This compact representation supports high-level semantic tasks like narration and scene comprehension, while future work will explore adaptive tokenization to allocate more detail to semantically rich frames without sacrificing overall efficiency.

[Q3] A direct comparison with recent models like StreamMind, LION-FS and VideoChat-Online would better position LiveStar.

We appreciate this valuable suggestion. We recognize the importance of benchmarking against the latest models to accurately position LiveStar within the field.

Comparison with StreamMind and LION-FS: Unfortunately, as of now, both StreamMind and LION-FS had not publicly released their model weights or inference code. This prevented us from conducting a direct, head-to-head evaluation on our proposed OmniStar benchmark or other benchmarks like OVBench.

However, to provide a point of reference, we have included the performance metrics reported in their original papers on the Ego4D Narration Stream benchmark, a task that all three models (LiveStar, StreamMind, and LION-FS) have been evaluated on. As shown in the table below:

This comparison, while indirect, shows that LiveStar achieves the lowest PPL and TimeDiff, while maintaining strong competitiveness in Token Accuracy, highlighting the effectiveness of our approach.

New Evaluation on VideoChat-Online: We were able to conduct a new evaluation with VideoChat-Online, which is open-sourced. We tested it on our OmniStar-RNG (Real-time Narration Generation) task. It is important to note that VideoChat-Online is not designed with a proactive response mechanism (i.e., it cannot autonomously decide when to generate a response). Therefore, we could only evaluate it in an offline setting, where we provide it with all video frames up to a pre-specified response timestamp and prompt it for a description.

The results are as follows:

The results show that even in a controlled offline setting, LiveStar significantly outperforms VideoChat-Online across all metrics. This demonstrates LiveStar's superior foundational ability for video-language alignment and coherent text generation.

[Q1] The SVeD decoding threshold parameter α is shown to be sensitive. Is there a way to dynamically learn or adapt this threshold rather than fixing it empirically?

We agree that a dynamic or learnable threshold could offer significant advantages in terms of adaptability and robustness. However, we acknowledge that designing such a dynamic mechanism is a non-trivial research challenge that requires careful consideration of the trade-offs between complexity and performance.

Specifically, α needs to be tailored to different task demands—for instance, being more sensitive for online live streaming and more stable for sports summarization. Additionally, since models of different scales exhibit distinct perplexity distributions, α could be normalized based on baseline perplexity.

To enable more adaptive response strategies for streaming video understanding, we will explore making the threshold parameter α dynamically adjustable in the future. But for the current work, we opted for a fixed, empirically-tuned α to establish a clear and reproducible baseline for our core contributions (SCAM and SVeD).

[Q2] The perplexity is used to determine whether a response should be generated, then a complete response must be produced at each forward pass. Wouldn't this significantly slow down the inference speed ...

Thank you for raising this critical point about inference efficiency. There appears to be a slight misunderstanding of how our Streaming Verification Decoding (SVeD) mechanism operates, which we are happy to clarify. The key is that we do not generate a complete new response at every forward pass.

1. "Generate" Case: When SVeD's verification step determines that a new response is needed (i.e., the perplexity exceeds a threshold), we then perform a full decoding process to generate a new, complete caption. This happens only at moments of significant changes in the semantic clip.

2. "Silence" Case: For the majority of incoming frames where the scene is stable, SVeD's verification check passes. In this "silence" case, the process is lightweight: We perform a single forward pass to compute the logits for the previously generated caption based on the updated context (which now includes the new frame). We use these logits to calculate the perplexity. Crucially, no decoding or token generation occurs, which is the main time-consuming step. We simply verify that the existing caption is still valid.

In summary, generating EOS still requires a full forward pass plus at least one decoding step. Our SVeD in the silent case requires only the forward pass. We also provide direct empirical evidence for this in our paper. Appendix E.2 and Figures 7 & 8 explicitly visualize this efficiency. The green "SVeD" curve (representing the "silence" case) has a lower inference time compared to the "One Step DD" curve (which simulates the generation of a single EOS token).

Thank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work.

Thanks to the author's Rebuttal, it has resolved my doubts. After carefully reading the author's response and discussions with other reviewers, I believe that although LiveStar still has some flaws, the relatively weak evaluation in the initial version has become quite solid after the reviewers' Rebuttal, and the active response design in this paper provides a new idea for such models. Therefore, I am willing to increase my score to 5. Please ensure that all new evaluations and experiments involved in Rebuttal are included in the final version of the paper (I understand that the capacity of the main text is limited, and the author can add relatively minor experiments to the Appendix).

Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.

[W1] The paper focuses on the narrative of video content, but it is only one function... it is not proper to use 'Live Streaming Assistant' to name a model...

We appreciate the feedback on the paper’s scope. Our work extends beyond solely video narration, addressing 15 diverse real-world scenarios and five distinct evaluation tasks: (1) real-time narration, (2) online temporal grounding, (3) storyline QA (multi-temporal answers), (4) streaming QA, and (5) multi-turn interactive QA. Detailed definitions and cases are provided in App. F and Figs. 7–9. LiveStar achieves an average 19.5% improvement in semantic correctness with 18.1% reduced timing difference versus existing online Video-LLMs while improving throughput by 12.0% FPS across 5 tasks. This demonstrates its capability as a comprehensive assistant for real-world online video understanding (including narration, grounding, and QA), and is therefore called 'Live Streaming Assistant'. We acknowledge that the illustrative focus on narration in Figs. 1–2 may have caused confusion; this will be clarified in the revision.

[W2] the paper mainly reports the results on the Omni-Star... the OVBench[4], etc.

We agree broader benchmarking strengthens validation. However, beyond OmniStar (our primary benchmark due to its alignment with real-world streaming tasks), we reported state-of-the-art results on Ego4D Narration (Table 3) and SVBench (Table 6). Per the reviewer’s suggestion, we conducted additional experiments on OVBench: LiveStar significantly outperforms VideoLLM-online, MovieChat, and Flash-VStream (reported in the original paper), though it trails VideoChat-Online marginally. We attribute this to two factors:

• LiveStar optimizes for text-generation tasks (e.g., narration, QA), while OVBench emphasizes multiple-choice QA.
• OVBench evaluates passive-response models (no output-timing decisions), whereas LiveStar and VideoLLM-online are active-response systems. Task objectives thus differ, and we will discuss this nuance in the revision.

[W3] There are 4 challenges proposed, but the solution part only considers the first two challenges...

The Introduction presents two core challenges (not four), with four sub-issues arising from EOS-based decision methods under the first challenge. Since there is no EOS token training, our PPL-based SVeD framework resolves all four issues (including the latter two noted by the reviewer). We will restructure the Introduction to summarize and distinguish challenges and issues, improving coherence.

[W4] I do not think EOS is a big problem that leads to model confusion. It is a simple indicator for salient that any unused token in the LLM vocabulary could serve this role.

We agree that any unused vocabulary token could technically serve as an EOS signal. However, in VideoLLM-online’s implementation, the EOS role was assigned to ',' (comma), a high-frequency token in text that risks conflating linguistic and timing functions. We will remove the "Vocabulary Confusion" claim in revision, as the core concern is implementation-driven rather than theoretical.

Finally, we emphasize our deep respect for VideoLLM-online and EOS-based works: they pioneered this space and inspired our research. Our goal is not to replace EOS-based methods, but to offer a complementary approach; both paradigms may prove synergistic in future work.

Thank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work.

Many thanks to your valuable comments and questions, which help us a lot to improve our work.

[W1 & W3] Limited Benchmarking: The paper does not evaluate LiveStar on video benchmarks like MVBench, VideoMME, and LVBench... primarily on the OmniStar dataset...

We appreciate the reviewer's suggestion regarding broader benchmarking. We would like to clarify that our evaluation already encompasses three benchmarks covering diverse online video understanding scenarios and tasks: OmniStar (Tables 1 & 2), Ego4D Narration (Table 3), and SVBench (Table 6). We are grateful that the reviewer recognizes the comprehensiveness of OmniStar, stating it "covers diverse real-world scenarios and tasks, providing a robust benchmark," which further underscores the breadth of our primary evaluation.

Crucially, the benchmarks highlighted by the reviewer (MVBench, VideoMME, LVBench) are primarily designed for offline video understanding, where models analyze complete videos. In contrast, our work focuses explicitly on online/streaming video understanding, a distinct paradigm requiring capabilities like real-time processing, active response triggering, and continuous comprehension of streaming inputs. These capabilities are central to our contribution but are not the primary targets of the mentioned offline benchmarks.

Furthermore, it is standard practice in online Video-LLMs that models like VideoLLM-online, VideoLLM-MoD, and LION-FS are only evaluated on benchmarks specifically designed for the online setting, as reflected in their papers.

Nevertheless, to further validate the generalizability of LiveStar and address the reviewer's concern, we have conducted additional evaluations on MVBench, LongVideoBench, and VideoMME.

The results demonstrate that LiveStar maintains strong performance and achieves competitive results even on these traditional offline video understanding benchmarks. To further enrich the evaluation, we have also added results on the OVBench benchmark, as detailed in our response to Reviewer j4Ki.

[W2 & Q1] Novelty Clarification: The work may lack groundbreaking innovation, as similar approaches have been explored in existing models like VideoLLM-online.

The novelty of LiveStar lies in its completely shift away from the paradigm of models based on EOS tokens. We introduce a new response-silence framework built on two innovations: Streaming Causal Attention Masks (SCAM) for training and Streaming Verification Decoding (SVeD) for inference.

1. Training Paradigm:

   • EOS based Models: These models are trained to solve a classification-like task for every frame: either generate a response or predict a special EOS token. For the vast majority of frames (silent intervals), the training objective is to map a visually rich frame to a single, static EOS token. It forces the model to learn a trivial, non-descriptive mapping for most of its training data.

   • LiveStar (with SCAM): We completely eliminate the EOS token from our training process. Instead, we use SCAM to train the model on interleaved frame-caption sequences. As described in Section 3.1 and illustrated in Figure 1(d), our model is trained to always generate meaningful, temporally consistent responses. It addresses the issues of pre-training misalignment and vocabulary confusion, encouraging the model to maintain its comprehension capabilities.

2. Inference Mechanism:

   • EOS based Models: During inference, these models must perform a full forward pass and a decoding step for every single frame to decide whether to output a response or the EOS token. Although generating an EOS token may seem sample, in high-frame-rate videos where most frames are silent, this still incurs the cost of generating a token per frame, leading to inefficiencies in overall inference performance.

   • LiveStar (with SVeD): Our SVeD mechanism, detailed in Section 3.2 and Algorithm 1, introduces a more efficient and intelligent decision-making process. For silent frames, we do not decode anything. We perform a single, lightweight forward pass to verify if the previously generated response is still semantically valid by checking its perplexity. This verification step is faster than decoding an EOS token (as empirically shown in Figures 7 & 8). A full decoding process is triggered only when a semantic shift is detected, which leads to faster overall inference.

[Q3] Evaluation Metrics: You introduced new metrics like SemCor and TimDiff. Could you elaborate on how these metrics are calculated...

Calculation Methods For each video, we have:

• A set of ground-truth (GT) semantic clips $G = \lbrace g_1, g_2, \dots, g_N \rbrace$, where each clip $g_i$ is defined by: a start time $t_{\text{start},i}$, an end time $t_{\text{end},i}$ and a GT caption.

• A set of model-generated responses $R = \lbrace r_1, r_2, \dots, r_M \rbrace$, where each response $r_j$ has an associated response time $t_{\text{resp},j}$.

For each $g_i \in G$, define the set of matching responses as:

• If $|M_i| = 0$, no response is generated for $g_i$; in this case, we assign an empty string $r_j =$ ' ' as the placeholder response for evaluation.
• If $|M_i| \geq 1$, we select the first matching response $r_j \in M_i$ (i.e., with minimal $t_{\text{resp},j}$) as the representative output.

As detailed in Appendix G, Semor and SumFluen is calculated using GPT-4o as an evaluator. The process is as follows:

1. SemCor (Semantic Correctness): This metric evaluates the quality of an individual response generated for a specific semantic clip. We prompt GPT-4o to score the pair $(r_j, g_i)$ on three dimensions (each on a 0–10 scale): Semantic Accuracy, Language Quality, and Information Completeness.

2. SumFluen (Summarize Fluency): This metric evaluates the holistic narrative quality of the model’s complete output sequence, treating all responses as a unified story. Let the full model narrative be the concatenation of all responses in temporal order:

   $$\mathcal{N}\_{\text{model}} = \text{Concat}\_{j=1}^M (r_j)$$

   Similarly, construct the ground-truth narrative by concatenating all GT captions in temporal order:

   $$\mathcal{N}\_{\text{GT}} = \text{Concat}\_{i=1}^N (g_i)$$

   We evaluate $\mathcal{N}\_{\text{model}}$ against $\mathcal{N}\_{\text{GT}}$ by prompting GPT-4o to assess them on five storytelling-relevant dimensions(each on a 0–10 scale): Writing Logicality, Language Fluency, Writing Conciseness, Semantic Consistency, and Narrative Completeness.

The metrics TimRedun, TimCover and TimDiff are calculated as:

3. TimRedun (Timing Redundancy) Measures the deviation from the ideal of exactly one response per semantic clip:

4. TimCover (Timing Coverage) Indicates the proportion of semantic clips that were covered by at least one response:

5. TimDiff (Timing Difference) Captures the cumulative response latency for all matched outputs within a semantic clip, while penalizing missed clips by their full duration:

Advantages: The key distinction of our evaluation framework lies in its holistic assessment of both temporal alignment and semantic quality, specifically designed for proactive response models. This stands in stark contrast to traditional paradigms that evaluate models in a passive or offline manner.

• Integrated Temporal and Semantic Evaluation: Traditional offline metrics (e.g., BLEU, CIDEr) or even some online QA benchmarks(e.g., OVBench) focus solely on the semantic correctness of an answer, assuming the response timing is either given or irrelevant. Many streaming video benchmarks also follow a passive paradigm, where the model is prompted at a pre-defined timestamp. While this setup tests understanding at a specific moment, it overlooks a crucial capability for real-time assistants: the ability to autonomously decide when to speak.

• Our framework addresses this gap by introducing integrated temporal and semantic evaluation. Metrics like TimDiff and TimRedun quantify the model's temporal decision-making: Did the model respond at the right time? Did it respond too much? Meanwhile, SemCor and SumFluen evaluate the quality of what was said at that moment. By combining these, our metrics move beyond simple content accuracy to provide a multi-faceted evaluation of a model's comprehensive ability to perceive, reason, and act in a continuous, dynamic environment. This makes them uniquely suited for benchmarking the next generation of proactive online video understanding models.

Thank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work.