Aha! - Predicting What Matters Next: Online Highlight Detection Without Looking Ahead

We thank the reviewer for their positive assessment and for highlighting our framework's real-time capabilities and strong empirical performance. We are grateful for your insightful questions and feedback on the paper's limitations, which we address below.

Q1 (Capturing Long-Horizon Features)

Our framework captures long-horizon features via the SinkCache mechanism. This is a hybrid approach that separates memory into two critical components: 1) a persistent sink that retains the initial, global task-conditioning tokens (i.e., the natural language query), acting as a long-term memory for the mission objective, and 2) a fixed-size sliding window of recent tokens that processes the immediate visual context. Our new ablation experiment (detailed in our response to reviewer 79jx - R1) confirm this design is critical: a sliding window only approach fails because it forgets the task (achieving only a 69.5 mAP in comparison to our reported 92.6 mAP), proving that the sink is what retains the essential long-horizon information.

Q2 (Impact of Window Size)

We thank the reviewer for raising this point and apologize that our notation in Table 3 on page 8 was not sufficiently clear. We will revise the table caption in the revised paper version to explicitly state that $|k_s|$ denotes the number of persistent sink tokens and $n$ represents the size of the sliding window for recent tokens, which directly corresponds to the "window size" in question. Table 3 (right) details our ablation study over these parameters. Our results show that while our optimal configuration using a window size of $n=2048$ achieves our best mAP of 92.6, the framework's performance degrades gracefully as the window size is reduced. For example, halving the window to $n=1024$ maintains a strong mAP of 90.1, and even a minimal configuration with $n=512$ achieves an 84.0 mAP, demonstrating the model's robustness to smaller context windows. This analysis confirms that our chosen configuration provides an excellent balance between performance and memory efficiency. Should the reviewer find a more extensive analysis beneficial, we would be happy to report results on a wider range of window sizes.

Q3 (On Latency and Compute Resources)

We have conducted a detailed performance analysis of our framework on a 1062 second video (~17 minutes) using two NVIDIA A6000 GPUs. The system achieved a sustained throughput of 1 frame per second (FPS), demonstrating high efficiency with 100% peak GPU utilization and 90% peak memory controller utilization. During this process, the framework consumed a peak of 30.49 GB of VRAM across both GPUs and operated well within safe thermal limits at a peak temperature of 65°C, all while maintaining a minimal system RAM footprint of 3.66 GB. While this establishes a strong performance baseline, the 1 FPS rate means that in a live scenario, depending on the GPU, the system would fall behind the incoming video feed (drift), requiring frame-skipping logic for real-time deployment. Given that our framework already leverages the available compute effectively, it is a strong candidate for such optimizations to achieve the higher throughput needed for live applications. We thank the reviewer for bringing up a practical question and will expand Appendix A with this information, and include a summary and pointer to the Appendix in the main paper.

L1 (On Uncertainty Estimation without Ground-Truth Labels)

Justification for Unsupervised Training: This is a critical point, and the reviewer is correct that the uncertainty head is trained without explicit ground-truth labels, which we note as a key limitation in section 5. This is due to the profound difficulty in obtaining such labels; a "highlight" is subjective, and an annotator's confidence is even more so, making it an unsolved challenge to collect these labels at scale.

Regarding the lack of ground-truth labels, which we identify as a key limitation, we adopted a principled, unsupervised approach using a Gaussian Negative Log-Likelihood (NLL) loss[1]. This leads directly to the intuition for the variance diversity loss.

The underlying intuition for encouraging variance diversity in Equations 5a and 5b is to prevent mode collapse. A known issue with this type of unsupervised uncertainty modeling is that the model can learn a degenerate solution by predicting a single, uninformatively high variance for all frames to minimize the NLL loss[2]. The diversity regularizer, $\mathcal{L}_{\text{div}}$, directly counteracts this by penalizing a low standard deviation of the predicted log-variances across a batch. This encourages the model to produce a dynamic and meaningful range of uncertainty values, allowing it to distinguish between predictable and ambiguous moments in a video stream. We will supplement our discussion in the limitations with this additional information.

Future Work: Towards Supervised Uncertainty with MultiVENT-G: We are actively exploring datasets that could provide labels for our supervised uncertainty learning in future work. One promising direction we have identified for supervised uncertainty-aware OHD involves leveraging the MultiVENT-G dataset[3], which focuses on complex, real-world events like disasters and provides a key feature missing in typical HD datasets: human-annotated confidence scores. Annotators rated their confidence on a 1-5 scale that a visual entity pertained to an event role.

This dataset offers three key advantages:

• Map Confidence to Uncertainty: The confidence scores can be directly transformed into ground-truth uncertainty labels (e.g., a 5/5 confidence rating corresponds to low uncertainty).
• Generate a New Training Objective: With these new labels, the uncertainty head could be trained directly with a supervised loss to address the critical issues of interpretability and calibration.
• Validate in High-Stakes Domains: Since MultiVENT-G is focused on disaster events, it allows for the validation of uncertainty-aware models in the exact high-stakes applications our paper targets.

However, there are currently three primary challenges with this approach: (1) Scale, as MultiVENT-G contains only 1,168 videos compared to the ~22.5k in our HIHD dataset; (2) Generalization, as training on MultiVENT-G's specific ontology may constrain the learned uncertainty; and (3) Subjectivity, as the labels are derived from a small team of annotators. We will create a new section in the appendix clearly outlining this promising direction and its associated challenges in Appendix J for our revised paper.

Addressing Safety-Critical Applications and Responsible AI: Finally, in response to the reviewer's important point on safety-sensitive applications, we acknowledge this is currently limited by available data. A core goal in releasing our HIHD dataset is to provide a large-scale resource that can help the responsible AI community develop methods for identifying ground truth and safety critical labels. We are actively exploring collaborations to tackle these issues and, based on the reviewers feedback, will move a stronger summary of this future work to the main paper. Thank you for bringing this important topic to our attention.

L2 (On Lack of Theoretical Support)

We agree that our work is primarily empirical, focusing on introducing a new task (OHD), a novel framework, and a large-scale dataset. While we have not derived new formal guarantees, the architectural and training principles we use are grounded in existing literature. Specifically, our multi-head training approach uses a fixed weighted sum of losses. This is a scalarization strategy in multi-objective optimization that is theoretically guaranteed to find a Pareto-optimal solution[4]. It is also an empirically validated standard in large-scale deep learning.

We have now strengthened the paper's connection to this established literature and explicitly call out the need for future theoretical analysis of OHD systems as an important research direction. We thank the reviewer for encouraging us to make these connections clearer.

[1]: Nix, David A., and Andreas S. Weigend. "Estimating the mean and variance of the target probability distribution."

[2]: Seitzer, Maximilian, et al. "On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks."

[3]: Sanders et al., "Grounding Partially-Defined Events in Multimodal Data."

[4]: Miettinen, Kaisa. "Nonlinear multiobjective optimization."

We sincerely thank the reviewer for their positive assessment and recognition of our work's strengths, and for providing constructive feedback with insightful questions that help us further improve the paper.

Q1, W1 (Justification for Multi-Head Training and Weight Sensitivity)

The reviewer raised an excellent point about providing more justification for our fixed-weight, multi-head training strategy beyond the initial ablation studies. We provide our response here, and will add it to the main paper.

Theoretical and Empirical Justification: Our choice follows established practices in prior work. Using a fixed weighted sum of multiple losses is a common approach in large-scale pre-training (e.g., BERT[1]) and aligns with the scalarization strategy in multi-objective optimization. When each task loss $L_i(\theta)$ is well-behaved, optimizing $L_0(\theta) = \sum_{i=1}^T w_i\,L_i(\theta), \quad w_i>0,\;\sum_{i=1}^T w_i=1$

converges to a point on the convex Pareto front[2], guaranteeing that no objective can be improved without degrading another. While we demonstrated theoretically this would represent trade-offs, we agree an empirical sensitivity analysis is an important analysis. Due to the high computational cost of retraining our model (see Appendix A.1) and the tight rebuttal deadline, we were unable to run this experiment in time. We have added this to our immediate research plan to investigate for future iterations of this work.

W2 (Practical Advantages of Uncertainty Modeling)

We are happy to clarify the practical advantages of the uncertainty head. Its primary advantage is improving the robustness of decision-making in a streaming context.

For an online, causal system like AHA that cannot see the future, expressing uncertainty is a necessity, not an optional feature. A high uncertainty score signals: "Based on what I've seen so far, this frame seems relevant, but I am not confident because crucial context might be missing." This is critical in high-stakes applications like disaster response, where a model providing a deterministic score without expressing confidence could be dangerously misleading.

The practical benefit is empirically demonstrated in our ablation study (Table 3 left, page 8), where removing the uncertainty head results in a concrete 3.5 mAP drop. This shows that the uncertainty score acts as a valuable regularizer, helping the model to appropriately temper its relevance predictions based on its confidence. However, the uncertainty head is trained without explicit ground-truth labels, which we note as a key limitation in Section 5. This is due to the profound difficulty in obtaining such labels. A "highlight" is subjective, and an annotator's confidence is even more so, making it an unsolved challenge to collect these labels at scale.

A promising direction in obtaining these supervised uncertainty scores is to leverage the MultiVENT-G dataset[3], which contains human-annotated confidence scores for disaster-related events. These scores can be directly mapped to uncertainty labels, enabling: (1) supervised training of the uncertainty head to improve interpretability and calibration, and (2) evaluation in safety-critical domains aligned with our target applications. Key challenges include the smaller scale of MultiVENT-G (1.1k videos vs. ~22.5k in HIHD), domain generalization beyond its event ontology, and the subjectivity of human-provided labels. These plans are summarized in our response to reviewer Zkgi (R4) and will be detailed in Appendix J of the revised paper.

W3 (Redundant Predictions in Additional Tasks)

This is an excellent observation. The repetitive predictions on auxiliary generative tasks (like captioning) are an expected outcome, as our framework was not optimized for this. Our primary research goal was to design and optimize a robust framework specifically for OHD. The evaluation on other tasks was a preliminary investigation into the generalization capabilities of our learned representations.

The issue of repetitive text is a challenge observed in streaming VideoLLMs, and state-of-the-art generative models use specialized techniques to mitigate it (e.g., advanced decoding strategies, KV cache manipulation). Our work provides a strong foundation for future research to build upon by integrating these generative-specific optimizations into our OHD framework, and we see this as a promising direction for future work. The code will be made public upon acceptance, and we encourage others to see if they can adapt AHA to these tasks.

Q2 (SinkCache's Robustness Over Extended Durations)

We thank the reviewer for this excellent question that addresses the core challenge of real-world, long-form video analysis. This is an important area, especially for what we have envisioned the works that will come after will be like summarizing long videos such as those in the disaster response or medical domains. We are actively searching for long video highlight datasets. While standard highlight detection benchmarks for hour long videos do not currently exist, we can address this based on the underpinnings of our model and the new ablation studies we have conducted. We believe these are great points that should transfer well to long-term video equally, and which we will explore in future works.

Our choice of SinkCache is theoretically grounded to handle this exact challenge. By design, it separates memory into two crucial components:

• A persistent sink that retains the initial, global task-conditioning tokens, acting as a long-term memory for the mission objective.
• A fixed-size sliding window of recent tokens that processes the immediate context.

Our new ablation studies (provided in our response to Reviewer 79jx - R1) validate this design: we found that simpler strategies like a sliding window only approach (which forgets the task) or a static-window-only approach (which fails to adapt to new events) perform significantly worse, with mAP of 69.5 and 63.2 respectively, compared to our reported 92.6 mAP. Furthermore, we performed a new experiment with a new "Dynamic Sink Cache" that achieves 93.0 mAP and supports making the sink even more task-focused improves performance, reinforcing the importance of this persistent memory.

However, this approach involves an explicit trade off. The fixed size of the sliding window ($n=2048$ tokens in our experiments) means that specific visual context that appears and then falls outside this window will be forgotten. If that same object or scene reappears much later, the model would likely perceive it as "novel" again, which might not be ideal in specific cases.

This trade-off motivates a clear direction for future work, future architectures could incorporate a third, content-based memory bank. This bank would not just hold the initial language instructions but could be trained to dynamically cache compressed representations of task-relevant visual frames from the distant past. This would allow the model to build a true long-term memory of key events, creating a more robust and scalable solution for hour long streams. We are excited to test these ideas as new long-form benchmarks become available. We will include this in section 5 in the main paper, under limitations.

Q3 (Reliability with Ambiguous or Irrelevant Task Objectives)

We thank the reviewer for this excellent question regarding the framework's reliability under imperfect task conditioning. To provide a quantitative answer, we conducted a new set of experiments on the TVSum dataset to measure the impact of ambiguous and irrelevant task objectives on performance.

For this experiment, we define an "ambiguous" prompt as a higher level definition of the original specific task (e.g., using "Vehicle Maintenance" for a video titled "How to change tires for off road vehicles"). We also define an "irrelevant" prompt as a task description taken from a video in a completely different category (e.g., applying the prompt "How to change tires for off road vehicles" to a video having nothing to do with vehicles. We compare the performance in these conditions to our model's optimal top-5 mAP on TVSum, achieved with a standard, specific prompt.

• With an ambiguous prompt, the model achieved a minor performance change by $\Delta = -1.1$ points, indicating that the model successfully captures the broader topic without being overly penalized by the lack of specific details.
• With an irrelevant prompt, performance decreased by $\Delta = -9.7$ points. This is expected, as the model is guided toward incorrect content.

We will include a summary of this analysis in the main paper, a full description in Appendix C, and the specific prompts used in Appendix I of the revised paper.

[1]: JDevlin et al., BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.

[2]: Miettinen, Kaisa. "Nonlinear multiobjective optimization."

[3]: Sanders et al., "Grounding Partially-Defined Events in Multimodal Data."

We are grateful to the reviewer for their exceptionally thorough and positive review. We are thrilled that you recognized the significance of the OHD problem, found our framework to be "elegant, robust, and well-suited," and considered our evaluation "comprehensive and convincing." Your insightful questions have prompted us to conduct further experiments and clarify key aspects of our work, which we believe have made the paper even stronger. We address each of your questions in detail below.

Q1, W1 (Dynamic Inference Weighting)

The reviewer raised an excellent point about our static inference weights and suggested exploring dynamic mechanisms, such as using the uncertainty prediction to adaptively adjust the final score. We followed this suggestion and ran a new set of experiments to investigate this.

We compared our static grid-search approach against two primary dynamic methods on the TVSum dataset using a standard 80/20 train/test split[1]. We investigated:

• A small MLP Gating network trained to learn per-frame weights ($\alpha_t, \beta_t, \epsilon_t$).
• An EMA based Adaptor that adjusts weights based on each head's recent predictive error.

Interestingly, both of these learned approaches proved to be unstable on the small TVSum training set (standard deviation >7), leading to lower average performance (87.9% mAP for the MLP and 87.5% for the EMA adaptor) as compared to our originally reported 92.6% performance. We attribute this instability to overfitting, a known challenge when training complex fusion mechanisms on smaller datasets.

In contrast, static weighting methods proved far more effective and robust. In fact, inspired by the reviewer's feedback, our investigation led us to a new, simpler, parameter-free heuristic that performs on par with our original grid search, reaching a stable mAP of 93.0% when combined with our new Dynamic Sink Cache (detailed in response to reviewer 79jx - R1). This finding suggests that our decoupled prediction heads are already well calibrated and produce robust, directly usable scores, making a complex learned weighting scheme unnecessary and potentially detrimental on existing benchmarks.

Based on this new analysis, we will update the main paper to use this simpler, more generalizable, and higher-performing scoring function. We will also add these new ablations to Appendix D.4. We again thank you for this suggestion, as it has led to a more robust final model.

Q2 (Impact of the Language Modeling (LM) Head)

Thank you for suggesting this valuable ablation. We agree it is an important analysis. Due to the high computational cost of retraining our model (see Appendix A.1) and the tight rebuttal deadline, we were unable to run this experiment in time. We have added this to our immediate research plan to investigate for future iterations of this work.

W2 (Dependence on Engagement Data as Ground Truth)

We appreciate the reviewer’s point that using YouTube "Most Replayed" metric as a ground‑truth relevance signal introduces biases (e.g., clickbait amplification) that may misalign with true importance in safety critical tasks. We deliberately chose this proxy to scale from prior benchmarks of ~50 manually labeled videos to HIHD’s ~24k videos, accepting “popularity” as an imperfect but high‑throughput signal. There are two promising directions of future work that we will add to the discussion in the paper:

• Document transparently We have released a comprehensive report detailing the creation of the HIHD data. By providing this, we aim to empower the community to develop new fairness evaluation frameworks, tooling, and mitigation strategies tailored to engagement based datasets, as well as the responsible AI community improving our current dataset.
• Future work in debiasing For future work, we plan to explore mitigating these biases by first calibrating our model on an expert labeled dataset in high-stakes domains like MultiVENT-G (more below), and then using adversarial training to make the model robust to misleading engagement signals like clickbait, inspired by prior work[2].

Expert Annotated Signals via MultiVENT‑G: To move beyond passive engagement proxies, we are actively exploring the recently released MultiVENT‑G dataset[3] which provides dense, frame level event role annotations by humans, into our training and evaluation pipelines. Additionally, uncertainty labels can be extracted from annotator confidence scores where available. Although MultiVENT‑G’s scale (~1.2k videos) is smaller than HIHD, it offers task‑aligned ground truth that can be used to fine‑tune or validate our heads in high‑stakes domains. Currently our issues are addressing challenges of scale mismatch, annotator subjectivity, and domain specialization, which will be a centerpiece of our future work, ensuring that AHA’s relevance and uncertainty estimates converge toward expert defined importance without sacrificing generalization. We will include more details in Appendix J of our revised paper.

Q3, W3 (Generalization of "Informativeness")

We thank the reviewer for this insightful question. The reviewer has correctly noted that "informativeness" (informational novelty) and "relevance" (task-importance) are not always the same, and ask how a concept trained on procedural videos generalizes. Our framework is explicitly designed to address this distinction using decoupled prediction heads. We report results on additional experiments we have run to illustrate these points.

An extended experiment of the real-world SCOUT robotics video (Section 4.3 in the main paper) reveals three key behaviors that demonstrate this generalization. With the task "what objects are in this room?", the model correctly assigns:

• High Informativeness, Low Relevance to visually novel but task irrelevant events, such as a robot entering a dark room or panning across an empty wall.
• Low Informativeness, High Relevance when focusing on a task-critical object that is no longer new to the scene, for instance, moving closer to a calendar that was already visible in the distance.
• Correlated Informativeness and Relevance: the scores often peak in unison when the robot transitions to a new area and immediately encounters a task-relevant object (e.g., "a shovel"). In these cases, the relevance head typically produces a higher peak, correctly prioritizing the task-specific discovery over the general novelty of the scene.

These new experiments show that our model successfully disentangles these two crucial signals, allowing it to remain focused on the task objective while still recognizing visual novelty. We have expanded Appendix E.1 with this analysis, provided a summary in the main paper, and will include illustrative figures in the revised paper version. We are unable to include the figures in this rebuttal due to NeurIPS' policy against new graphics or links to external sources.

Justification for the Informativeness Labeling Heuristic: The reviewer correctly identifies that our method for generating informativeness labels is a heuristic. We apologize for not making the motivation clearer, and will revise the paper to include this explanation. Our approach is adopted directly from established work in the streaming Video-LLM literature[4]. In that context, the goal is to train a model that knows when to speak during a continuous video stream, generating a response only after acquiring sufficient context but before the moment becomes stale.

The underlying intuition we adapt is that initial frames may lack context, while frames after a "point of sufficient understanding" are redundant for describing a segment's core content. Our informativeness head is trained using a Binary Cross-Entropy (BCE) loss on labels derived from this principle. We hypothesize that this signal, which marks the accumulation of new information, correlates with highlight worthy moments in an OHD setting, which is supported by our strong results in Table 3 (left) on page 8.

Q4 (Generalizability to Different Backbone Models)

This is an important question about disentangling the contributions of our AHA architecture from the strengths of the chosen backbones. We selected Qwen2[5] and SigLIP[6] because they represent the state-of-the-art in open-source vision and language models, providing a strong and reproducible foundation. Our choices were grounded in a thorough literature review; for instance, SigLIP outperforms CLIP at the batch sizes relevant to our framework, and our specific Qwen2-based architecture[7] has a higher performance in distilled versions (7B).

We fully agree that an ablation study using other popular backbones like LLaMA or CLIP would be valuable. However, adapting AHA to entirely different model families requires substantial engineering and re-training that was unfortunately not feasible within the limited rebuttal period. This is a key priority for our future work. This paper represents the first step for AHA, and we have laid a foundation that we hope others in the community can build upon by integrating our architecture with a broader range of backbones. We have clarified our rationale for the current backbone selection in Section 3 and emphasized this limitation and future work direction in Section 5 of our revised manuscript.

[1]: Liu et al., "UMT."

[2]: Zhang et al., "Mitigating Unwanted Biases with Adversarial Learning."

[3]: Sanders et al., "Grounding Partially-Defined Events in Multimodal Data."

[4]: Wang et al., "VideoLLM Knows When to Speak."

[5]: Bai et al., "Qwen Technical Report."

[6]: Zhai et al., "Sigmoid Loss for Language Image Pre-Training."

[7]: Bai et al., "Qwen-VL."

We sincerely thank the reviewer for their positive assessment, their high evaluation of our paper's clarity, and their insightful questions. This feedback has been instrumental in helping us improve the clarity, rigor, and overall quality of our work. We provide a detailed response to each of the weaknesses raised below.

W1 (Fixed Loss Weights and Adaptability)

The reviewer raised an excellent and important point regarding our use of fixed loss weights and its potential impact on adaptability. Our choice is grounded in a combination of well established theoretical principles and strong empirical evidence from the multi-task learning literature.

Theoretical and Empirical Justification: Using a fixed weighted sum of losses is a classic scalarization strategy in multi-objective optimization. It is not only a standard in large scale pre-training (e.g., BERT[1]) but also theoretically guaranteed to find a solution on the convex Pareto front when individual loss functions are well behaved[2]. This means no single objective can be improved without degrading another. Furthermore, recent empirical work has rigorously compared this method against more complex specialized multi-task optimizers (SMTOs) and found that, with appropriate normalization, simple scalarization can match or even surpass them on diverse benchmarks[3]. The main advantages include its scalability and simplicity, as it avoids the significant per step overhead inherent in dynamic re-weighting schemes.

Adaptability to Backbones: Regarding adaptability to different backbone architectures, we agree that this is a crucial aspect of generalization. Our backbone choices were grounded in a thorough literature review of high-performing open-source multimodal models. We selected Qwen2[4] and SigLIP[5] as they represent state-of-the-art open source vision-language models.

SigLIP was chosen as the visual encoder because it has been shown to outperform CLIP at smaller batch sizes (e.g., 4-8k which is reflective of our input in AHA) while offering competitive generalization. For the language backbone, we adopted the LLaVA-OneVision architecture based on Qwen2, which demonstrates strong performance in its distilled 7B variant, which we use for our lightweight model. Both Qwen2 and SigLIP are widely used in concurrent recent streaming vision-language literature[6], further underscoring their competitiveness and community adoption. We will add this additional justification to the paper when describing these components.

We acknowledge that testing on additional backbones would be valuable as these underlying models continue to advance. In future work, we plan to expand AHA to additional backbones and explore meta learning strategies for dynamic head weighting to further improve generality. This paper represents the first step for AHA, laying the foundation upon which future extensions can generalize across a broader range of backbone architectures. We will include a summary of the rationale for our backbone selection in Section 3 of the main paper, as well as the detailed explanation in Appendix D.

W2 (Intuition for Uncertainty Loss (Eqs. 5a, 5b))

We thank you for pointing out the lack of clarity here. We apologize for failing to properly cite our detailed justification (Appendix D.3) in the main paper. We have now added a summary to the main text (included below) and will ensure it correctly references the full analysis in the appendix to make the connection clear for all readers.

Motivation for Uncertainty in an Online Setting: The core motivation is that for online, causal systems like AHA that cannot see the future, uncertainty is not an optional feature but a necessity for robust decision making. Our model's uncertainty score is a proxy for its confidence given an unfolding context. A high score signals: "Based on what I've seen so far, this frame seems relevant, but I am not confident because crucial context might be missing." While the direct performance gain is modest (a 3.5 mAP drop when removed as indicated in Table 3 on page 8), we view this as a crucial first step in modeling the unknown future, which is vital for the high-stakes applications AHA is designed for.

Intuition for Variance Diversity: The uncertainty head is trained without explicit ground truth labels. We instead use a principled, unsupervised approach based on a Gaussian Negative Log-Likelihood (NLL) loss[7]. The specific intuition for encouraging variance diversity is to prevent mode collapse. This is a known pitfall where the model learns a degenerate solution by predicting a single, uninformatively high variance for all frames to trivially minimize the NLL loss[8]. Our diversity regularizer, $\mathcal{L}_{\text{div}}$, directly counteracts this by penalizing a low standard deviation of the predicted log-variances across a batch. This forces the model to learn an expressive and dynamic signal, allowing it to better distinguish between predictable and ambiguous moments in the video stream.

We acknowledge that this unsupervised approach has limitations, and in our roadmap we have some preliminary ideas on how to train this on a supervised signal. A promising direction is to leverage the MultiVENT-G dataset[9], which contains human-annotated confidence scores for disaster-related events. These scores can be directly mapped to uncertainty labels, enabling: (1) supervised training of the uncertainty head to improve interpretability and calibration, and (2) evaluation in safety-critical domains aligned with our target applications. Key challenges include the smaller scale of MultiVENT-G (1.1k videos vs. ~22.5k in HIHD), domain generalization beyond its event ontology, and the subjectivity of human-provided labels. These plans are summarized in our response to reviewer Zkgi (R4) and will be detailed in Appendix J of the revised paper.

W3 (Rationale for SinkCache and Alternatives)

This is an excellent question that prompted a deeper analysis of our memory mechanism and, we are excited to report, led to an improvement in our framework's performance.

Rationale and Alternatives Considered: Our initial rationale for selecting the SinkCache mechanism was its unique hybrid approach to memory. We hypothesized that for task-conditioned Online Highlight Detection, a model must simultaneously maintain long-term context (the task objective) and adapt to recent visual information. SinkCache addresses this by retaining a fixed set of initial "sink" tokens while also keeping a sliding window of recent tokens.

We conducted additional experiments and compared the performance reported in our paper (92.6 mAP) to the performance on TVSum against several alternative memory strategies:

• Unbounded KV Cache ("Full History"): This standard approach achieved a strong 91.7 mAP but is impractical for long videos due to its unbounded memory growth, which leads to out of memory errors.
• Sliding Window Only: This strategy only retains recent context, eventually losing the vital task conditioning information. As a result, its performance was significantly lower at 69.5 mAP.
• Static Window Only: This method only uses the initial tokens as context, failing to incorporate new visual events and performing poorly at 63.2 mAP.

These results confirmed our initial hypothesis that combining long-term and short-term memory is crucial, with our original SinkCache implementation achieving a 92.6 mAP and providing the best trade off between performance and memory efficiency.

A New Approach: Dynamic Sink Cache: The reviewer's question inspired us to refine our hypothesis further. The initial sink tokens capture the system prompt, task objective, and sometimes the first few frames. This prompted us to wonder if we could create a more targeted sink containing only the essential task tokens. This led us to perform a new experiment where we dynamically initialize the sink tokens to contain only the essential task objective tokens (i.e. $\mathcal{Q}$'s tokens), disregarding other initial inputs within the sink token.

We term this new approach Dynamic Sink Cache. This method yielded a new state-of-the-art mAP of 93.0 on TVSum, outperforming all previous configurations. This result provides strong evidence that the most critical component for long-term memory in OHD is the persistent natural language objective. Further results on the natural language objective are provided in our response to reviewer NtaH (R3), where we experimented with unrelated and ambiguous task objectives. We observed a substantial performance drop ($\Delta = -9.7$ mAP) for unrelated objectives and a minor decrease ($\Delta = -1.1$ mAP) for ambiguous objectives.

We are very grateful to the reviewer for prompting this line of thought, as it has tangibly improved our paper. We will update the main paper to include the Dynamic Sink Cache as our primary result and provide this expanded ablation study on memory mechanisms in Appendix G.

W4 (mAP Abbreviation)

Thank you for your careful reading and for catching this oversight. We have corrected the text to define mAP before its first use in the revised manuscript.

[1]: JDevlin et al., "BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding."

[2]: Miettinen, Kaisa., "Nonlinear multiobjective optimization."

[3]: Senushkin et al., "Independent Component Alignment for Multi-Task Learning."

[4]: Bai et al., "Qwen Technical Report."

[5]: Zhai et al., "Sigmoid Loss for Language Image Pre-Training."

[6]: Wang et al., “VideoLLM Knows When to Speak.”

[7]: Nix, David A., and Andreas S. Weigend. "Estimating the mean and variance of the target probability distribution."

[8]: Seitzer, Maximilian, et al. "On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks."

[9]: Sanders et al., "Grounding Partially-Defined Events in Multimodal Data."