Thank the reviewer for the thoughtful and balanced assessment of our work. We are grateful for the recognition of OME10M and OMEBench as valuable large-scale resources for egocentric video QA, as well as for the thorough empirical evaluation across multiple benchmarks and ablation studies. Below, we address each of these points in detail.

Technical novelty: Indeed, some of the ideas in strategy we employed have appeared in previous work, but our design is tailored to the OME10M dataset. Determining which training strategies are more effective for learning on multi-level visual supervision data is a topic worth exploring. On one hand, our outperformance over the baselines to some extent demonstrates the effectiveness of our designs (for instance, VideoChat-Flash also employs various video token compression strategies). Additionally, we introduced further experiments, using VTM as an alternative for video token merging and adopting "from short videos to long videos" as a straightforward curriculum learning baseline. The experimental results on the 1.6M dataset are shown in the table below. These results highlight the value of investigating appropriate training strategies for LMMs and datasets.

Computation Cost of STM and TTP Indeed, during the process of token compression, additional operations introduce new time costs. To better illustrate the extra time cost of token compression, we use the inference time of OpenMMEgo without STM+TTP as the unit, comparing the costs of different operations with the maximum number of frames under the same memory.

Our token compression strategy enables processing two times additional frames with only a 47 % increase in inference time.

Analysis of Failure Modes: We thank the reviewer for the suggestion. Due to the limitations of not being able to include images in the response, we shall address this point in the next version of the paper by incorporating erroneous examples and the difference in results from multiple inferences.

We thank the reviewer for the thoughtful and constructive feedback. We reply to the concerns below and will incorporate the mentioned revisions in the next version of our manuscript.

Line 35: Indeed, a comma is missing at this position and will be inserted in the revised manuscript.

Data Curation Details: The main text outlines only the core design concepts due to space constraints. A comprehensive description of the OME10M generation pipeline appears in Appendix B, covering the end-to-end workflow, all categories of synthetic QA pairs, and the exact prompt templates employed for behavior-based and vision-centric question creation.

The Role of Visual Cues: Intuitively, for general understanding, third-person training data inherently contains descriptions of visual details, which are primarily lacking in egocentric video data. Therefore, introducing annotations of visual details from egocentric video makes the data distribution more uniform and thus easier to learn. On the other hand, the inference of events and actions in egocentric video often relies on these visual details. When egocentric data lacks visual details, general understanding and egocentric video understanding become significantly different tasks and are hard to foster each other. Experimentally, we removed the visual details data in ablation studies, which resulted in a comprehensive performance decline.

These clarifications will be incorporated into the revised version to enhance reproducibility and to shed light on why visual-detail supervision is essential for egocentric video understanding.

We hope these clarifications resolve the reviewer's concerns. Please let us know if any further questions.

We thank the reviewer for the thoughtful and constructive feedback. We reply to the concerns below and will incorporate the mentioned revisions in the next version of our manuscript.

Potential Related Works

1. Token Compression techniques: The reviewer listed many token merging methods developed for efficient visual representation learning, including ToMe, which we have cited in Line 194, and ToSA, a paper posted after the NIPS submission Deadline. Our focus is on adapting token‐compression techniques (merging and pruning) in video understanding LMMs. Accordingly, we cited the classic origins (ToME for merging, LTP for pruning) in single-image and language-model contexts, and we discussed their extensions in video LMMs (e.g., VideoChat-Flash, LongUV, VideoLLaMA 3, etc.). VideoChat-Flash is also used as our baseline.
2. Synthetic Annotations: LLaVA-Video was indeed cited and is one of our baselines, based on which our SFT variant yields substantial gains. LaViLa focuses on visual-representation pretraining and thus differs in scope from LMMs. EVUD, a four-page unpublished manuscript, was inadvertently omitted, and Eagle 2.5 (a concurrent work) also went unnoticed.

We appreciate the reviewer’s suggestions and will include discussions of all these works in the next revision as an additional Related Works section.

---

Additional Baselines & Benchmarks We thank the reviewer for pointing out potential baselines and suggesting the non-Ego4d benchmark, which will indeed help us better present the performance of OpenMMEgo.

• Gemini Model Comparison: Considering the scale difference with Gemini-1.5-Pro and the fact that the Gemini-1.5-Pro API is no longer available, we report the performance of the Gemini-2.0-Flash API on our egocentric benchmark as a comparison. The model size of Gemini-2.0-Flash is closer to our 7B model, and it is reasonable to assume that its training data and process are very similar to Gemini-1.5-Pro.
• AlanVLM: We have included AlanVLM as an extra baseline (the LMM trained on EVUD), serving as a supplementary baseline for synthetic data.
• New benchmarks: We have added results from the HD-EPIC VQA benchmark and the open-ended OpenEQA. We also convert the multi-instance retrieval (V → T) task in Epic Kitchens 100 into a multiple-choice task and compared the mAP performance with EgoVLP and LaViLa-L.

Furthermore, the reviewer notes that certain prior works in token compression, synthetic data, and curriculum learning could serve as alternatives to some of our components, thus forming potential baselines. Among the potential baselines pointed out by the reviewer, due to time and resource constraints, we adapt VTM to OpenMMEgo as the baseline for video token compression and the strategy from short to long videos as the baseline for curriculum learning. In terms of synthetic annotations, LLaVA-Video is already our baseline, and we have additionally included a comparison with AlanVLM (fine-tuned on synthetic data EVDU), so no further comparison is made. We perform SFT on Qwen2.5-VL with 1.6M data.

Aside from being comparable to Gemini 2.0 on certain metrics, OpenMMEgo consistently outperforms all other baselines, further demonstrating the effectiveness of our method.

---

Token Compression Cost vs. Benefit Token compression does incur additional computation. Normalizing the inference time of “w/o STM+TTP” to 1.0, we compare supported maximum frames and relative cost:

Our token compression strategy enables processing two times additional frames with only a 47 % increase in inference time.

Missing Details / Clarifications We appreciate the reviewer’s corrections and provide the following clarifications:

1. Figure 5: All answers are produced by OpenMMEgo; “EgoCatch” was a mislabel.
2. Figure 3 (right): Our radar chart is plotted with the scale set to 0 and 1, where the minimum value is a multiple of 5 below the lower limit and the maximum value is used as the upper limit. However, for aesthetic purposes, we manually labeled the values in the plot with PowerPoint. Unfortunately, due to a saving error, there are two typos (OpenMMEgo w/o STM+TTP's Gen.vis. is 67.5, and OpenMMEgo's Egoplan is 47.7). The proportions of the plot itself are correct. We sincerely apologize for any confusion caused.
3. L171–172 (OMEBench Verification): No, limited by our human resources.
4. L203 (Merging): We merge tokens within each component by summing, following ToME.
5. L215 (“Excessive Semantic Similarity”): Tokens with excessive semantic similarity are pruned only in fast frames. On one hand, in fast frames, we focus on the dynamic changes of these semantic tokens. In our visualization (Fig. 4), the tokens with the highest similarity are often static ones, such as background elements, while the dynamic parts are not pruned. On the other hand, in slow frames, the static temporal information of these semantic tokens is preserved.
6. L245–247 (Curriculum Difficulty): In the Offline Data Curriculum, we are more concerned with the inherent difficulty of the data itself rather than its ease of learning. Using a fine-tuned model to evaluate might be influenced by the extent to which the data can be optimized. In the Online Data Curriculum, our dropout mechanism compensates for this.
7. Figure 3 (left): We will update the left plot per the suggestion; the right-plot typos have been corrected as above.

We hope these clarifications resolve the reviewer's concerns. Please let us know if any further questions.

I thank the authors for their response and additional experiments. In particular, I appreciate the comparisons on new benchmarks, curriculum learning with short -> long videos, and token compression cost analyses. I also appreciate the clarification about the radar plot. These addressed some of the specific concerns I had. However, I'm not convinced about the token compression, synthetic annotations and some baseline comparisons (detailed below). However, since the majority of concerns were addressed, I'm raising my rating.

• Token compression: This is one of the core contributions (L56 - 62). The paper currently lacks a discussion of what prior token compression methods do and how this paper innovates over it. Simply citing three (among a large list) of past work without any discussion or experimental comparison is insufficient. That said, I thank the authors for the new results with VTM. While this is only a single data point (given the rich literature of token merging), this is useful. Is this with the Learnable VTM approach here in Figure 4: https://arxiv.org/pdf/2410.23782?
• Synthetic annotations: LlavaVideo dataset is a baseline, but the approach proposed for generating the LlavaVideo dataset is not compared against. I would like to emphasize this difference. Simply using prior datasets in comparison is not what I'm suggesting. My key concern is that the data curation approach proposed here is not vetted against prior data curation approaches in an apples-to-apples fashion:
  • The current experiments do not clearly demonstrate the importance of the technical contributions in the data generation pipeline. This needs to be done on the same videos with the same frontier models for generation.
  • "core visual fact" extraction (L137), multi-step prompting (L140), and generating QAs for different time scales (L153) are different in comparison to prior work. How much of a role do these play in comparison to techniques used in LlavaVideo?
  • Furthermore, LlavaVideo uses GPT-4o whereas the proposed method uses Gemini-1.5 Pro. For example, GPT-4o (71.9) is worse than Gemini 1.5 Pro on VideoMME (73.2). LlavaVideo also uses a different collection of videos than OpenMMEgo. Purely comparing the datasets (and not the methodology) can be misleading.
• Baseline comparison is not apples-to-apples:
  • For context on Gemini 2.0 Flash vs. Gemini 1.5 Pro: Gemini 1.5 Pro is quite a bit better on video benchmarks as shown in Table 6 here: https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf. So the numbers reported here are likely worse than what 1.5 Pro would have scored.

We sincerely thank the reviewer for raising the score and for acknowledging that many of the key concerns were addressed. We appreciate the positive feedback on our new benchmark comparisons, curriculum learning design, token compression analysis, and the clarification regarding the radar plot. We also thank the reviewer for clearly identifying the remaining concerns, which allows us to provide focused responses. Should any point require further clarification, we would be pleased to provide it.

---

Token Compression:

Thanks for the valuable suggestion regarding the discussion. We recognize that our initial approach—briefly introducing the background in the introduction (L55-62) while focusing on VLM-specific compression in related works (L96)—may have obscured the clarity of our contributions. We appreciate the reviewer for highlighting this issue and providing relevant references on token merging. Since NeurIPS does not permit uploading a revised manuscript, we write a draft below, which will be added in the next version as promised:

Visual token merging has demonstrated significant effectiveness in improving representation efficiency while reducing computational costs, initially in image classification tasks. Key techniques include similarity-based merging (ToMe)[1], learned merge ratio (Token Merger)[2], learned threshold-based merging and pruning (LTMP)[3], decoupled embedding modules (DTEM)[4], and depth-aware spatial token aggregation (ToSA)[5]. These approaches have since been extended to video inputs, with innovations such as learnable spatiotemporal merging (VTM) [6], spatial-temporal token selection (STTS)[7], and temporal interpolation (InTI) [8]. Similarly, in large multimodal models (LMMs), token compression has been adopted to improve inference efficiency and handle long contexts, including methods based on static pixel token reduction (Video-LLaMA3)[9], cross-modal querying reduction (LongVU)[10], and hierarchical compression (VideoChat-Flash)[11]. Building on these advances, we propose a method specifically designed for egocentric videos, which present unique challenges due to dense visual cues and frequent camera motion. We introduce a dual-branch compression strategy that restricts spatial token merging to adjacent tokens for preserving semantic entity boundaries, and performs temporal pruning over entity-aligned tokens in a SlowFast-inspired view to retain motion-salient information.

Regarding the specific experimental point: Yes, we implemented the learnable VTM variant in our framework exactly as depicted in Figure 4 of the paper (https://arxiv.org/pdf/2410.23782). While we acknowledge that including a broader set of baselines would have strengthened our evaluation, we were constrained by the limited time and computational resources available during the rebuttal period. We are particularly grateful that the reviewer recognizes the value of our OpenMMEgo with VTM as a ‘useful' contribution to the experimental analysis.

[1] Token Merging: Your ViT but Faster.

[2] Efficient Vision Transformer via Token Merger.

[3] Learned Thresholds Token Merging and Pruning for Vision Transformers.

[4] Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers.

[5] ToSA: Token Merging with Spatial Awareness.

[6] Video Token Merging for Long-form Video Understanding.

[7] Efficient Video Transformers with Spatial-Temporal Token Selection.

[8] Dynamic and Compressive Adaptation of Transformers From Images to Videos.

[9] VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding.

[10] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding.

[11] VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling.

Data Comparison:

1. We fully appreciate the reviewer's suggestion that apples-to-apples comparisons of data curation pipelines using the same video set and model would be insightful. However, at our current dataset scale, re-annotating the entire corpus with multiple alternative construction methods is unfortunately infeasible due to the high cost of high-quality video-language annotation. This cost consideration also motivated our choice of Gemini 1.5 Pro over GPT-4o, as it provides more economical scaling while maintaining quality.
2. Instead, our current focus has been on two complementary aspects: (a) Demonstrating the value of our dataset relative to existing baselines; (b) Validating that each of the two compositions, particularly the behaviour-based and vision-centric data, contributes positively to the performance of downstream models.
3. The core contribution of our paper is introducing a scalable, practical data construction pipeline for egocentric video understanding. While comparative studies of annotation strategies would indeed be valuable, such analysis is beyond the scope of this work. Our current focus is to clearly document our pipeline's construction and demonstrate its effectiveness through measurable performance gains.

---

Baseline Comparison:

We appreciate the reviewer’s question regarding model comparison fairness. Our selection of Gemini 2.0 Flash over Gemini 1.5 Pro was carefully considered based on the three key factors:

1. Availability: Gemini 1.5 Pro was no longer accessible through official API channels during our evaluation period.
2. Model size and usage context: Gemini 1.5 Pro is a significantly larger model and not aligned in scale with our 7B-size model. Gemini 2.0 Flash provides a more appropriate comparison point, being both closer in size and trained on similar corpora.

More importantly, Gemini 2.0 Flash can outperform Gemini 1.5 Pro on some egocentric video benchmarks, which is contrary to our common expectation based on the Gemini 2.5 technical report. To prove it, we provide two examples:

• On the EgoSchema, Gemini 1.5 Pro scores 63.5 (Table 10 in https://arxiv.org/pdf/2403.05530v2), while Gemini 2.0 Flash scores 71.1.
• On the EgoTempo, Gemini 1.5 Pro scores 36.3, whereas Gemini 2.0 Flash achieves 39.3. (Table 6 in https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf)

Therefore, we believe the use of Gemini 2.0 Flash actually represents a stronger baseline for egocentric video understanding, rather than a weaker one. We will clarify this choice more explicitly in the revised draft.

---

We sincerely thank the reviewer once again for the insightful suggestions. We greatly appreciate the time and effort spent on reviewing our work. We will revise the paper to better reflect the points raised and to improve the overall clarity and completeness.

We thank the reviewer for the thoughtful and constructive feedback. We reply to the concerns below and will incorporate the mentioned revisions in the next version of our manuscript.

Data Curation Details: Due to the limited space in the main text, we have only outlined the core design concepts. In Appendix B, we provide a detailed description of our data generation process, including the workflow, the specific types of data generated, and the prompts used. The visual details annotations are exclusively utilized in the visual details split of OME10M, and questions and answers are generated based on these annotations.

Additional Baselines & Benchmarks We thank the reviewer for pointing out potential baselines and suggesting the non-Ego4d benchmark, which will indeed help us better present the performance of OpenMMEgo.

• AlanVLM: We have included AlanVLM as an extra baseline, serving as a supplementary baseline for synthetic data.
• Video Language Pretrained Models: We convert the multi-instance retrieval (V → T) task in Epic Kitchens 100 into a multiple-choice task to enable a comparison with Video Language Pretrained Models (EgoVLP and LaViLa-L).
• New benchmarks: We have added results from the HD-EPIC VQA benchmark and the open-ended OpenEQA. Due to time constraints, EgoVQA and VidEgoThink have not yet completed testing, but they are in our plan and will be supplemented once finished.

OpenMMEgo consistently outperforms all other baselines, further demonstrating the effectiveness of our method.

Cost of STM and TTP Indeed, during the process of token compression, additional operations introduce new costs. To better illustrate the extra time cost of token compression, we use the inference time of OpenMMEgo without STM+TTP as the unit, comparing the costs of different operations with the maximum number of frames under the same memory.

Our token compression strategy enables processing two times additional frames with only a 47 % increase in inference time.

We hope these clarifications resolve the reviewer's concerns. Please let us know if any further questions.