HIPPO-VIDEO : Simulating Watch Histories with Large Language Models for History-Driven Video Highlighting

Thank you for your thoughtful and constructive feedback. Your comments helped us clarify and strengthen several key aspects of our work.

The realism of simulated user histories

• We understand the concern regarding the ecological validity of simulated user histories. To address this, we have conducted additional experiments using real user watch histories and included the results in the official comment. (Will be updated soon)

Human verification scale

• We would like to clarify that human verification was conducted on the entire set of simulated watch histories, not just a small sample. Each instance was independently reviewed by three annotators, ensuring comprehensive validation across all 2040 sessions.

Clarification for Results in Table 2

• Consistent with $1$ , we treat both “Agree” and “Neutral” judgments as acceptable, reflecting the inherent subjectivity of personalized highlight evaluation. Following prior work, we report outcomes based on the majority vote per case. Importantly, in 97.44% of cases, the majority of annotators did not select “Disagree,” which supports the plausibility of the simulation. We acknowledge that this table is hard to parse, so we would provide more details in the final version.

$1$ Sul, Jinhwan, Jihoon Han, and Joonseok Lee. "Mr. hisum: A large-scale dataset for video highlight detection and summarization." Advances in Neural Information Processing Systems 36 (2023): 40542-40555.

Thank you for dedicating your time and effort to reviewing our work. Your insightful comments and suggestions have been crucial in enhancing the quality and clarity of our paper. We hope our explanations and newly added results sufficiently address your concerns. If there are any remaining questions or points that need further clarification, we would be happy to address them.

I appreciate the authors’ efforts and find the core idea of the paper interesting. However, I still have some concerns regarding the simulated trajectories. Overall, I find the paper borderline acceptable, but I would not be strongly opposed to its rejection.

Thank you for your thoughtful and constructive feedback. Your comments offered a useful perspective in assessing the robustness of our approach.

The lack of recent baselines

• We conducted additional experiments with two recent approaches: TR-DETR $1$ and UVCOM $2$ . TR-DETR introduces a task-cooperation module that jointly enhances Moment Retrieval (MR) and Highlight Detection (HD) via mutual guidance, while UVCOM proposes a unified architecture that integrates local relation modeling (for MR) and global context understanding (for HD) using a multi-granularity module and contrastive learning. Although both models outperform earlier baselines, HiPHer still achieves superior results across key metrics, underscoring the benefit of incorporating user watch histories for personalized video highlighting. We will include these results and analysis in the final version to provide a more comprehensive benchmark.

$1$ Sun, Hao, et al. "Tr-detr: Task-reciprocal transformer for joint moment retrieval and highlight detection." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.
* $2$ Xiao, Yicheng, et al. "Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Time efficiency and complexity

• To address the reviewer’s concern about scalability and efficiency, we provide a breakdown of HiPHer’s computational complexity
  • n : number of segments for each video (assume all videos have same number of segments for computational convenience) ,
  • m : number of history videos (10 in our setup),
  • C: cost of processing one segment with CLIP (vision + text)
  • d: embedding dimension,
  • L: transformer depth,
  • g: number of GPUs (we use 4× NVIDIA RTX A6000)
• The time complexity can be broken down as follows: target video encoding requires n × C and the transformer-based scoring introduces L × n² × d + n × d² (from self-attention and projection). For history video encoding costs (m × 𝑛̄ × C)/g when parallelized across GPUs. The total cost is therefore: n × C + L × n² × d + n × d² + (m × 𝑛̄ × C)/g.
• In practice, approximately 96% of the inference time is spent on CLIP-based segment encoding, which is common across many video-language pipelines, while HiPHer’s transformer module accounts for only 4% of the total latency. Moreover, if the history embeddings are pre-computed and cached, the history encoding cost is eliminated entirely.
• HiPHer can support near real-time inference with minimal delay, because segment encoding is frame-local and fully parallelizable. With history embeddings pre-computed, it can incrementally process incoming segments, making it suitable for real-world applications.

Ablation on saliency margin γ

• While we initially set γ = 1 following prior works $1,2$ , we conducted an ablation study to assess its impact on HiPHer’s performance. We observed that smaller margins (γ = 0.1–0.2) consistently yield better results, as they enable finer-grained preference modeling. In contrast, larger margins tend to encourage overconfident separation between segments, which reduces generalization. We will include these ablation results in the final version to specify the impact of γ.

$1$ Lei, Jie, Tamara L. Berg, and Mohit Bansal. "Detecting moments and highlights in videos via natural language queries." Advances in Neural Information Processing Systems 34 (2021): 11846-11858.
* $2$ Moon, WonJun, et al. "Query-dependent video representation for moment retrieval and highlight detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

Thank you for dedicating your time and effort to reviewing our work. Your insightful comments and suggestions have been crucial in enhancing the quality and clarity of our paper. We hope our explanations and newly added results sufficiently address your concerns. If there are any remaining questions or points that need further clarification, we would be happy to address them.

Thank you again for your valuable feedback. As the discussion period comes to an end, please feel free to let us know if any further questions or concerns arise. We would greatly appreciate the opportunity to address them and further improve our work.

Thank you for your thoughtful and constructive feedback. We found your comments very helpful in improving and clarifying our work.

More practical details regarding the creation of the dataset

• We used gpt-4o-mini as the backbone model for simulating user behavior, including watch history generation, preference modeling, and saliency annotation.
• Regarding dataset creation cost and effort, we provide below the average token usage and associated costs aggregated across all 2,040 simulation sessions. The total LLM cost amounted to approximately $160. The full simulation pipeline was executed over the course of two weeks, with periodic manual monitoring and quality checks, requiring an estimated 50-60 human hours of operational effort. For every instance, humans manually verified the quality of the simulated results. | Stage | Input Tokens | Output Tokens | Input Cost ($) | Output Cost ($) | Total Cost ($) | | :---- | ----- | ----- | ----- | ----- | ----- | | Video understanding | 21088 | 1234 | 0.01265 | 0.00296 | 0.01561 | | Video selection | 45552 | 1448 | 0.02733 | 0.00348 | 0.03081 | | Preference update | 18683 | 2393 | 0.01121 | 0.00574 | 0.01695 | | Action decision | 23141 | 185 | 0.01388 | 0.00044 | 0.01432 |

Figure 5: Effect of watch history length on preference modeling

• In Figure 5, the sharp increase in R1@0.5 at m = 10 (where m is the number of history videos) suggests that the model begins to capture a more stable representation of user preferences. This aligns with our simulator’s design, where a history length of 10 constitutes the full context for modeling preference.
• HiPHer’s preference modeling module relies on identifying recurring patterns across the watch history. Longer histories help capture consistent user behaviors while shorter histories may include noisy preferences.

Clarifying the intent of Table 4

• Our intent with Table 4 was to compare HiPHer against prior methods that claim to achieve personalization using sentence- or word-level textual inputs (CLIP-It and VSL). These were not intended as ablations, but as external baselines to highlight that such query forms lack the depth of user preference modeling provided by watch history.

Clarifications for Figures and Tables

• Thank you for the valuable suggestions. We fully agree that clearer visualizations and more precise table formatting are essential. We will revise the following elements accordingly in the final version to improve clarity and readability:
  • Figure 1 was designed to illustrate the motivation behind our task that video highlights should adapt to user preferences inferred from watch history.
  • Figure 3 is intended to illustrate diversity and saliency score characteristics of HIPPO-VIDEO, but we agree the caption can be improved to help readers interpret it more easily - such as:
    • (a) exploration ratio within individual watch histories,
    • (b) inter-history preference diversity visualized via t-SNE, and
    • (c) distribution of saliency scores across target videos.
  • Table 2: We agree that Table 2 could benefit from clearer presentation. In the final version, we will revise the table format and accompanying explanation to make the inter-annotator agreement results easier to interpret. Specifically, we will clarify the meaning of each row, the labeling scheme (A: Agree, U: Unclear, D: Disagree), and how majority judgments were used in the summary analysis.
  • Table 3 : “Improv.” refers to the relative improvement of HiPHer over each baseline in Hit@1 (≥ 9), as noted in the table caption.
• We sincerely appreciate the reviewer’s constructive comments and will reflect these improvements in the final version to enhance the clarity of the paper.