Unleashing Hour-Scale Video Training for Long Video-Language Understanding

Thanks for acknowledging our key contributions. We sincerely appreciate your constructive suggestions, and we hope this response letter can help relieve your remaining concerns.

W1: Temporal relationships may be lost.

Ans: Thanks for raising this concern. In MemAug, we do consider the spatial-temporal relationship between the full video tokens in the memory repository and decayed video tokens. In particular, we first directly flatten the full video tokens into 1D embeddings and apply the 1D structure RoPE, which is commonly used by the prior works [1, 2]. Based on their spatial-temporal relationship, we then construct a corresponding 1D RoPE for decayed video tokens. In this way, each decayed token (query) is encoded with positional information that matches the similar spatial-temporal positions of its corresponding full tokens (keys and values) in the cross-attention module.

Ablation studies show a substantial performance drop on the temporal reasoning task when this structural RoPE is removed. It demonstrates that constructing the spatial-temporal relationship between the full video tokens and decayed video tokens is necessary.

Q1: Attention heatmaps on the memory repository for visualizing whether the model learns to retrieve the relevant content.

A2: Thanks for the insightful suggestion. Attention heatmaps over the memory repository indeed offer a way to visualize which frames or regions the model focuses on during retrieval. While we are currently unable to include these visualizations due to policy constraints, we plan to incorporate them in the revised version to better illustrate the model’s behavior.

Q2: Results on temporal reasoning tasks.

Ans: We evaluate the Hour-LLaVA on VideoMME (Temporal Reasoning subset). We also provide the results of LLaVA-OV-SI and Qwen-VL models as references (* indicates our re-implementation). Notably, Hour-LLaVA achieves considerable improvements in the temporal reasoning task over its pretrained model (LLaVA-OV-SI) with gains of 13.5% for the 3B model and 20.9% for the 7B model. Moreover, Hour-LLaVA consistently outperforms the Qwen-VL models on this task, demonstrating its superior temporal reasoning capabilities.

Reference:

[1] RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing 2024.

[2] TC-LLaVA: Rethinking the transfer from image to video understanding with temporal considerations. ArXiv 2024.

We sincerely appreciate your constructive suggestions, and we take these reviews seriously. We hope this response letter can help relieve your concerns.

W1 & Q1: Evaluation on open-ended QA benchmarks is needed.

Ans: Following your suggestion, we evaluate Hour-LLaVA-3B and Hour-LLaVA-7B on MLVU-Dev [1] with free-form generation tasks, including sub-science captioning (SSC) and video summarization (SUM). The generated results are measured by GPT-4-Turbo with carefully designed instructions, where the rating score is on a 2–10 scale. We also report results from other state-of-the-art Video-LMMs for reference. The results demonstrate the strong capability of Hour-LLaVA on open-ended tasks.

W2 & Q2: Human Evaluation is needed.

Ans: We agree that the human evaluation on the VideoMarathon dataset is necessary. To quantify the data quality, we manually check 330 randomly selected VideoMarathon QA pairs, including 80 multiple-choice and 250 open-ended samples.

For the Multiple-Choice (MC) QA samples, we assess the Accuracy of the synthetic samples.

• Accuracy: This evaluation assigns a score of 1 if there is exactly one correct answer. A score of 0 is given if the question is irrelevant to the video, if no correct option exists, or if multiple options are correct. The human evaluation on 80 samples achieves an accuracy of 78.7%, which demonstrates the reliability of the synthetic multiple-choice QA samples.

For the open-ended (OE) QA samples, we validate the data quality from two perspectives:

• Yes/No Bias: Yes/No Bias evaluation [2] validates the extent to which the models tend to respond with ''yes''. In particular, the evaluation consists of ''Yes Percentage Difference'' (Pct. Diff) and ''False Positive Ratio'' (FP Ratio) metrics. Smaller Pct. Diff (closer to 0.0) reveals less bias, which means that the predicted number of ''Yes'' is closer to the ground truth. FP Ratio measures how likely the model responds with “Yes” out of all incorrect responses, where a value closer to 0.5 indicates less bias. We randomly select 200 Yes/No QA samples. As a result, Pct. Diff reaches 0.04 and FP Ratio achieves 0.40, which reflects limited yes/no bias in the VideoMarathon dataset.

• Hallucination: 10 volunteers evaluate the quality of 50 randomly selected OE QA samples. Each sample is rated on a scale from 1 to 4 based on the following criteria: Score 1 - Totally irrelevant; key entities missing; mostly hallucinated; Score 2 - Key entities present; contains major hallucinations; Score 3 - Key entities present; contains minor hallucinations; Score 4 - Almost identical to ground truth; no hallucinations. Scores range from 1 (low quality) to 4 (high quality). Scores above 3.0 are considered largely correct. Human evaluation achieves an average score of 3.29 ± 0.27, indicating that the OE QA samples in the VideoMarathon dataset are consistently reliable.

W3 & Q3: Ablation study on the number of MemAug blocks is missing.

Ans: Thanks for raising this point. We agree that an ablation study on the number of MemAug blocks is required. Using the experimental setup in Section 5.5, we train four Hour‑LLaVA variants with 1, 2, 4, and 8 MemAug blocks. Across three video benchmarks, performance improves up to 2 or 4 blocks and then decreases slightly at 8. More specifically, for short videos (TempCompass), the sweet spot lies in the range from 2 to 4 blocks, while for medium‑ and long‑form videos (LongVideoBench and LVBench), 4 blocks provides the optimal results.

W4 & Q4: Comparisons to advanced Video-LMM are required.

Ans: We have listed all the mentioned Video-LMM below. * indicates our re-implementation.

Comparison among 3B-LMMs: In this setting, we compare three highly relevant 3B LMMs. LLaVA-OV-SI serves as the pretrained model of Hour-LLaVA, while Qwen-VL and Hour-LLaVA both employ Qwen as their LLM decoder. The results demonstrate that Hour-LLaVA-3B significantly improves the long video-language modeling capability of its pretrained model (LLaVA-OV-SI-3B). Moreover, Hour-LLaVA generally outperforms its strong peer (Qwen2.5-VL-3B), especially in medium- and long-video settings.

Comparison among ~7B-LMMs: Following prior works [4, 5], we initialize Hour-LLaVA with LLaVA-OV-SI-7B [3], as Qwen-2.5-VL had not yet been released during our training period. Consequently, Hour-LLaVA-7B inherits the Qwen2-7B as LLM decoder from its pre-training backbone. For a fair comparison, we also add the results of Qwen2-VL-7B below. Similar to the 3B setting, Hour-LLaVA-7B significantly improves its pretrained model (LLaVA-OV-SI-7B) and surpasses its strong counterpart (Qwen2-VL-7B). Moreover, compared with other state-of-the-art LMMs (Qwen2.5-VL and Intern series) built on the stronger Qwen-2.5-7B backbone, Hour-LLaVA-7B performs comparably on most of the following benchmarks. It even outperforms these models on certain benchmarks, such as the medium and long subsets of VideoMME.

Reference:

[1] MLVU: Benchmarking Multi-task Long Video Understanding. In CVPR 2025.

[2] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. CVPR 2024.

[3] Llava-onevision: Easy visual task transfer. ArXiv 2024.

[4] Longvu: Spatiotemporal adaptive compression for long video-language understanding. ICML 2025.

[5] Long context transfer from language to vision. ArXiv 2024.

# Example 1
Question: Where is the woman positioned relative to the man in the scene where they are sitting on the couch?
Synthetic Answer: The woman is sitting next to the man on the couch.
Ground Truth: The woman is sitting to the right of the man on the couch.
Averaged Score: 8.0

# Example 2
Question: After the boy lights the candle in Clip, what does the woman do?
Synthetic Answer: She holds the candle after it is lit.
Ground Truth: She starts to speak and holds the candle.
Averaged Score: 7.3

# Example 3
Question: What is the woman doing when she uses a sewing machine in the video?
Synthetic Answer: She is sewing the cuffed hem onto the pants.
Ground Truth: She is sewing the hem of the pants.
Averaged Score: 7.5

Thanks for your careful review comments, which have significantly improved the quality of our work.

We acknowledge an unintended misreporting: ShareGPT4Video-7B’s SSC is 5.02 (not 3.77). After carefully re-examining each row presented in MLVU [1] (Appendix Table 2) and LongVA [2] (Table 8), we confirm that the "Overall" score is computed as (SSC + SUM)/2. Accordingly, we have recomputed and corrected the "Overall" scores for Hour-LLaVA-3B and Hour-LLaVA-7B, and the updated table now reflects the accurate values.

Reference:

[1] MLVU: Multi-task Long Video Understanding Benchmark. CVPR 2025

[2] Long Context Transfer from Language to Vision. arXiv 2024.

Thank you for the clarification regarding the "Overall" metric definition and the additional SSC/SUM results. While I appreciate the effort to address the initial concern, significant issues remain unresolved.

The calculation of the "Overall" score (SSC×201/418 + SUM×217/418) directly conflicts with the reported results for multiple models. For instance, SharGPT4Video-7B’s claimed "Overall" score of 3.77 cannot be reconciled with its SSC (3.77) and SUM (2.52) values via the formula, as 3.77×201/418 + 2.52×217/418 ≈ 3.12—a clear 0.65-point discrepancy. Similar inconsistencies exist for other models. This inconsistency between the explicitly stated calculation method and the presented results directly undermines the reliability of the comparative data, regardless of whether the results were sourced from LongVA [1] or Video-XL [2].

[1] Zhang P, Zhang K, Li B, et al. Long context transfer from language to vision[J]. arXiv preprint arXiv:2406.16852, 2024.

[2] Shu Y, Liu Z, Zhang P, et al. Video-xl: Extra-long vision language model for hour-scale video understanding[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 26160-26169.

Thanks for the constructive comments. We hope this response letter can help relieve your concerns.

W1: Methodology is mostly adaptations of existing techniques, such as the cross-attention module and standard spatial/temporal token reduction. Hierarchical captioning follows standard LMM practices.

A1: Thanks for raising this concern. As cross-attention and self-attention modules are fundamental building blocks of modern LLMs, their presence is not sufficient to determine whether the overall framework is novel or not. To clarify the novelty of our Hour-LLaVA framework, we analyze the limitations of existing long video large multimodal models (Video-LMMs), explain the necessity of the Hour-LLaVA framework, and provide additional empirical results to support our claims below.

• Limitations of existing long Video-LMMs: Existing long Video-LMMs confront a fundamental dilemma:

  • Compression-based methods prioritize computation affordable and training efficiency, but inevitably suffer from information loss caused by aggressive compression (e.g., uniform sampling 64 frames from a one-hour video);
  • Extension-based methods preserve high-fidelity video context by using sequence parallel (SP) at the cost of significantly increased training time and GPU usage (linear growth with video length), particularly for hour-scale videos.

• Necessity of Hour-LLaVA framework: To break this dilemma, we propose the Hour-LLaVA framework, which intends to keep high-fidelity video context while prioritizing training efficiency. Beyond the extension-based methods, Hour-LLaVA introduces the forgetting mechanism (implemented by spatial-temporal token dropping) that decouples computation from the input sequence length, so training and inference costs no longer rise linearly with video length. MemAug module (implemented by a cross-attention mechanism) is designed to dynamically recover lost video tokens from the memory repository. Without the MemAug module, purely compression‑based approaches inevitably incur irreversible context loss, which leads to sub-optimal performance. As a result, by coupling the forgetting mechanism with MemAug, Hour‑LLaVA delivers compression‑level efficiency while still preserving the full video context sought by extension methods.

• Hour‑LLaVA is more than a cross-attention module: The novelty of Hour-LLaVA lies in the interaction of the two tightly coupled components: (1) the forgetting mechanism guarantees training efficiency, and (2) MemAug adaptively re-injects video tokens from the memory repository to recover information loss caused by the forgetting mechanism. However, MemAug (implemented by cross attention) is necessary but not sufficient. Without the forgetting mechanism, the trade‑off between contextual fidelity and training efficiency collapses.

• Hour‑LLaVA does more than frames/tokens dropping: its learnable MemAug module re‑injects informative cues that token dropping would otherwise ignore. Moreover, the ablation study below shows that the MemAug module consistently improves the performance across different token dropping strategies, which highlights that MemAug effectively recovers information lost during compression and enhances the overall video-language modeling.

  Token Dropping Strategy	MemAug	LongVideoBench	LVBench
  Uniform	w/	54.0	40.6
  	w/o	52.1	38.3
  Random	w/	52.2	38.8
  	w/o	51.7	38.1
  Keyframe	w/	53.5	40.3
  	w/o	52.0	38.9
  Question-Guided	w/	53.4	39.7
  	w/o	51.0	37.5

W2: Heavy reliance on synthetic data generation is unreliable.

A3: We agree that heavily relying on synthetic data is risky. However, human annotation is too expensive to keep pace with the rapidly expanding video data, especially for long videos. Recent advances in LMMs (e.g., GPT-4o and DeepSeek-V3) allow synthetic data to approach or even surpass human quality. In this way, a virtuous cycle is created, which means better LMMs produce higher-quality synthetic data, which in turn further improves subsequent models. Moreover, empirical studies consistently show that incorporating carefully constructed synthetic data improves the performance of video-LMMs, highlighting the practical value of synthetic data.

W3 & Q1: Human evaluation is required.

A4: We agree that the human evaluation on the VideoMarathon dataset is necessary. To quantify the data quality, we manually check 330 randomly selected VideoMarathon QA pairs, including 80 multiple-choice and 250 open-ended samples.

For the Multiple-Choice (MC) QA samples, we assess the accuracy of the synthetic samples by assigning a score of 1 if there is exactly one correct answer. A score of 0 is given if the question is irrelevant to the video, if no correct option exists, or if multiple options are correct. The human evaluation on 80 samples achieves an accuracy of 78.7%, which demonstrates the reliability of the synthetic multiple-choice QA samples.

For the open-ended (OE) QA samples, we validate the data quality from two perspectives:

• Yes/No Bias: Yes/No Bias evaluation [1] validates the extent to which the models tend to respond with ''yes''. In particular, the evaluation consists of ''Yes Percentage Difference'' (Pct. Diff) and ''False Positive Ratio'' (FP Ratio) metrics. Smaller Pct. Diff (closer to 0.0) reveals less bias, which means that the predicted number of ''Yes'' is closer to the ground truth. FP Ratio measures how likely the model responds with “Yes” out of all incorrect responses, where a value closer to 0.5 indicates less bias. We randomly select 200 Yes/No QA samples. As a result, Pct. Diff reaches 0.04 and FP Ratio achieves 0.40, which reflects limited yes/no bias in the VideoMarathon dataset.
• Hallucination: 10 volunteers evaluate the quality of 50 randomly selected OE QA samples. Each sample is rated on a scale from 1 to 4 based on the following criteria. Scores range from 1 (low quality) to 4 (high quality). Scores above 3.0 are considered largely correct. Human evaluation achieves an average score of 3.29 ± 0.25, indicating that the OE QA samples in the VideoMarathon dataset are consistently reliable.

W4: The 22-task taxonomy is comprehensive but somewhat arbitrary.

A5: We need to clarify that the 22-task taxonomy in the VideoMarathon dataset is not arbitrary but carefully designed based on an extensive survey of existing video-language datasets (Line 96). These datasets collectively cover over 100 distinct tasks. The current 22-task taxonomy is a distilled summary of those tasks across a wide range of benchmarks, aiming to provide a comprehensive and structured categorization of video-language understanding tasks.

Q2: Analysis on higher compression ratios. What is the theoretical lower bound?

A6: Thanks for the comment. It is challenging to provide a theoretical lower bound for the video token compression ratio due to the complexity of video modeling. To address this concern, we provide an empirical analysis to identify a suitable compression range. As shown in Figure 4 (left), we evaluate the performance under higher temporal compression ratios. The results indicate that a ratio of 1/2 to 1/4 achieves a great balance between computational efficiency and performance. With a compression ratio beyond 1/4, performance begins to degrade significantly.

Q3.1: How does the memory repository scale? What is the memory footprint compared to baselines?

A7: MemAug with memory repository is a light-weight module. We compare the memory usage of Hour-LLaVA with its baseline when processing different numbers of frames during inference. We report the additional memory usage below, which demonstrates the efficiency of Hour-LLaVA.

Q3.2: A complexity comparison between Hour-LLaVA and its base model is required.

A9: Figure 4 (right) compares the number of visual tokens fed into the LLM decoder across Hour-LLaVA, LLaVA-Video, and vanilla 1-FPS sampling methods. The larger token number indicates the higher computational complexity, highlighting the low complexity of Hour-LLaVA.

Q4: How to explain the generalization of Hour-LLaVA for videos longer than the training period? Are there other contributing factors?

A10: The light-weight MemAug module allows Hour-LLaVA to process the longer videos (more than 2 hours). The proposed VideoMarathon dataset serves as another key contributing factor to the generalization of Hour-LLaVA.

Q5: Explanation of the superiority of uniform temporal forgetting. Is the MemAug module responsible for most of the heavy lifting?

A11: The table in A1 shows that uniform temporal forgetting performs comparably to keyframe-based forgetting. Although we report different token compression strategies, developing a more advanced token compression strategy with clear superiority is not the focus of this work. The table in A1 also presents that when combined with MemAug, most token compression methods achieve a performance gain of approximately 1.5%–2.5%, suggesting that the MemAug module plays a major role in boosting overall performance. This finding may serve as supporting evidence for your question.

We sincerely appreciate your constructive suggestions, and we take these reviews seriously. We hope this response letter can help relieve your concerns.

W1 & Q1: Human Evaluation on the Synthetic VideoMarathon Dataset.

A1: Thank you for highlighting the need to validate the quality of synthetic data. We agree that the human evaluation on the VideoMarathon dataset is necessary. To quantify the data quality, we manually check 330 randomly selected VideoMarathon QA pairs, including 80 multiple-choice and 250 open-ended samples.

For the Multiple-Choice (MC) QA samples, we assess the accuracy of the synthetic samples by assigning a score of 1 if there is exactly one correct answer. A score of 0 is given if the question is irrelevant to the video, if no correct option exists, or if multiple options are correct. The human evaluation on 80 samples achieves an accuracy of 78.7%, which demonstrates the reliability of the synthetic multiple-choice QA samples.

For the open-ended (OE) QA samples, we validate the data quality from two perspectives:

• Yes/No Bias: Yes/No Bias evaluation [1] validates the extent to which the models tend to respond with ''yes''. In particular, the evaluation consists of ''Yes Percentage Difference'' (Pct. Diff) and ''False Positive Ratio'' (FP Ratio) metrics. Smaller Pct. Diff (closer to 0.0) reveals less bias, which means that the predicted number of ''Yes'' is closer to the ground truth. FP Ratio measures how likely the model responds with “Yes” out of all incorrect responses, where a value closer to 0.5 indicates less bias. We randomly select 200 Yes/No QA samples. As a result, Pct. Diff reaches 0.04 and FP Ratio achieves 0.40, which reflects limited yes/no bias in the VideoMarathon dataset.
• Hallucination: 10 volunteers evaluate the quality of 50 randomly selected OE QA samples. Each sample is rated on a scale from 1 to 4 based on the following criteria: **Score 1 **- Totally irrelevant; key entities missing; mostly hallucinated; Score 2 - Key entities present; contains major hallucinations; Score 3 - Key entities present; contains minor hallucinations; Score 4 - Almost identical to ground truth; no hallucinations. Scores range from 1 (low quality) to 4 (high quality). Scores above 3.0 are considered largely correct. Human evaluation achieves an average score of 3.29 ± 0.25, indicating that the OE QA samples in the VideoMarathon dataset are consistently reliable.

W2.1: Limited novelty of the proposed method.

A2.1: Thanks for raising concerns about the novelty of the proposed method. As cross-attention and self-attention modules are fundamental building blocks of modern LLMs, their presence is not sufficient to determine whether the overall framework is novel or not. To clarify the novelty of our Hour-LLaVA framework, we analyze the limitations of existing long video large multimodal models (Video-LMMs), explain the necessity of the Hour-LLaVA framework, and provide additional empirical results to support our claims below.

• Limitations of existing long Video-LMMs: Existing long Video-LMMs confront a fundamental dilemma:

  • Compression-based methods prioritize computation affordable and training efficiency, but inevitably suffer from information loss caused by aggressive compression (e.g., uniform sampling 64 frames from a one-hour video);
  • Extension-based methods preserve high-fidelity video context by using sequence parallel (SP) at the cost of significantly increased training time and GPU usage (linear growth with video length), particularly for hour-scale videos.
  • As video length becomes longer (e.g., introducing VideoMarathon with hour-scale videos), such a dilemma will be intensified. Compression-based methods must incur greater information loss, while extension-based methods should tolerate longer training/inference time. Therefore, a new framework that effectively balances video context with training efficiency is urgently needed.

• Necessity of Hour-LLaVA framework: To break this dilemma, we propose the Hour-LLaVA framework, which intends to keep high-fidelity video context while prioritizing training efficiency. Beyond the extension-based methods, Hour-LLaVA introduces the forgetting mechanism (implemented by spatial-temporal token dropping) that decouples computation from the input sequence length, so training and inference costs no longer rise linearly with video length. MemAug module (implemented by a cross-attention mechanism) is designed to dynamically recover lost video tokens from the memory repository. Without the MemAug module, purely compression‑based approaches inevitably incur irreversible context loss, which leads to sub-optimal performance. As a result, by coupling the forgetting mechanism with MemAug, Hour‑LLaVA delivers compression‑level efficiency while still preserving the full video context sought by extension methods.

• Hour‑LLaVA is more than a cross-attention module: The novelty of Hour-LLaVA lies in the interaction of the two tightly coupled components: (1) the forgetting mechanism guarantees training efficiency, and (2) MemAug adaptively re-injects video tokens from the memory repository to recover information loss caused by the forgetting mechanism. However, MemAug (implemented by cross attention) is necessary but not sufficient. Without the forgetting mechanism, the trade‑off between contextual fidelity and training efficiency collapses.

• Hour‑LLaVA does more than frames/tokens dropping: its learnable MemAug module re‑injects informative cues that token dropping would otherwise ignore. Moreover, the ablation study below shows that the MemAug module consistently improves the performance across different token dropping strategies, which highlights that MemAug effectively recovers information lost during compression and enhances the overall video-language modeling.

  Token Dropping Strategy	MemAug	LongVideoBench	LVBench
  Uniform	w/	54.0 (+1.9)	40.6 (+2.3)
  	w/o	52.1	38.3
  Random	w/	52.2 (+0.5)	38.8 (+0.7)
  	w/o	51.7	38.1
  Keyframe	w/	53.5 (+1.5)	40.3 (+1.4)
  	w/o	52.0	38.9
  Question-Guided	w/	53.4 (+2.4)	39.7 (+2.2)
  	w/o	51.0	37.5

W2.2: Lack of comparison with the latest related work (Vamba).

A2.2: Thanks for pointing out the latest related work, Vamba []. We will cite this work and include a direct comparison in the revision. The comparison with Vamba shows that both Hour‑LLaVA variants (3B and 7B) consistently outperform the considerably larger Vamba‑10B across all three long‑video benchmarks.

Reference:

[1] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. CVPR 2024.

[2] VAMBA: Understanding Hour-Long Videos with Hybrid Mamba-Transformers. ICCV 2025.