Unlocking Exocentric Video-Language Data for Egocentric Video Representation Learning

Limited Technical Contribution

We acknowledged existing works that leverage exocentric datasets for egocentric learning in our paper and provided comparisons in both the experiment and the related work sections. Additionally, we highlight our contributions in the general response.

Our proposed techniques are simple yet effective, demonstrating clear improvements by leveraging exocentric datasets compared with our baselines. We believe these contributions should not be dismissed as obvious and naive but recognized for their practicality. Simplicity and intuitiveness should be viewed as strengths, not weaknesses.

Concerns About Using the Exocentric Dataset HTM-AA

We would like to emphasize that our main baseline models are trained using the exact same data sources. For instance, in Table 1, the main LaViLa baselines are trained with Ego4D and the original HTM-AA dataset. As a result, the improvements achieved by our method over the baselines cannot be attributed to domain similarity between the training and evaluation data.

Furthermore, we conducted additional experiments on egocentric QA tasks and embodied tasks, as detailed in our response to Reviewer rhcR, further demonstrating the effectiveness of our method across different domains.

Concerns About Initialization with LaViLa

We would like to stress that our main baseline models are also initialized from LaViLa. Specifically, both our baseline and EMBED are first initialized with LaViLa and then pretrained on Ego4D and either the original HTM-AA dataset or our constructed version. Furthermore, both the original HTM-AA dataset and our constructed dataset are approximately the same size, ensuring the fairness of our comparisons.

In summary, our experimental setting is strictly controlled, with the only difference between our LaViLa-Ego4D+HTM-AA baseline and EMBED being the use of either the original HTM-AA dataset or our constructed version.

Selecting Visually Aligned Sentences

Thank you for your question! Our preliminary findings indicate that non-visually alignable sentences often lack action-related information, which can typically be identified based on text alone. We found that this simple strategy is effective, and we will revise the wording to make this clearer.

Limited Improvement with SSv2

For the SSv2 experiments, the baseline is trained on the original SSv2 dataset, while our model is trained on our constructed version. Since the original SSv2 dataset already shares similarities with egocentric visual content, it is expected that it is less necessary to use our framework and thus we would yield relatively smaller improvements in this case.

Typo

Thank you for catching the typo! We will proofread the draft carefully in the revised version.

It is unclear whether the authors intend to release their models and datasets.

We plan to release them upon paper publication.

There have already been some works on refining annotations using LLMs and leveraging exocentric videos to assist with egocentric videos, making the entire framework not novel.

As in our general response, our framework 1) actively mines egocentric-relevant data from exocentric sources, which extends beyond simply refining existing annotations; 2) leverages exocentric data with flexible language annotations in a relatively large-scale setting, outperforming existing works with a similar motivation such as Ego-Exo.

Considering these points, we view our approach not as a data refining process but as an active video mining and tailored language generation process that has not been explored before. Also, our method is more generally applicable and achieves better performance than existing methods such as Ego-Exo.

How is the quality of annotations on exocentric videos using an HOI detector and how do the authors handle erroneous data?

Thank you for your question! While it is inevitable to introduce some errors when using external tools, we implemented several filtering strategies to minimize these errors. For instance:

• We filter generated narrations with low model confidence (Lines 242–246), which typically correspond to out-of-domain/egocentric-irrelevant videos.
• We average HOI scores across frames (Lines 162–169) to alleviate the errors introduced by the HOI detector.
• On the text side, we use techniques like text classification models to filter out non-visually alignable sentences (Lines 221–229).

That being said, we acknowledge that the constructed dataset is not perfect and there is room for improvement. However, the experimental results demonstrate that the current techniques already achieved significant improvements over the baselines.

The primary concern is the comparison and discussion of the advantages over retrieval-augmented methods.

We agree that retrieving relevant exocentric videos based on their similarities to egocentric ones presents an alternative approach to data construction. However, we would like to highlight the following points:

1. EMBED and retrieval-based methods rely on different criteria for constructing data. Exploring Neural Data Server-style methods for egocentric learning could serve as a complementary direction to our work.

2. Compared with our method of computing HOI scores for clip selection, calculating similarity scores between videos for retrieval can be more complex and time-intensive.

3. We compared our approach with a RAG-based method (Xu et al., 2024) as shown in Table 1, demonstrating improved performance. To ensure a fair comparison, we also implemented an approach inspired by Xu et al. within our framework. Specifically, we retrieved relevant egocentric narrations based on exocentric ones and used Llama-3 to perform style transformations on the exocentric narrations using the retrieved narrations. However, this approach did not yield improvements, likely because retrieving relevant narrations that effectively support style transfer is challenging.

EMBED requires multiple offline models and must iterate through all candidates in a large corpus, such as HT100M in this work, making it time-intensive.

We acknowledge that EMBED requires several models for dataset construction. However, we would like to highlight that: 1) Our baseline models, such as LaViLa and Helping Hands, similarly rely on offline narration generation or HOI models for dataset processing. 2) The dataset construction occurs only during preprocessing and does not require further processing during training, ensuring that the training process remains relatively efficient.

Original noisy HT100M also boosts the performance a lot in Table 4 for exocentric tasks

Our primary motivation is to construct egocentric-relevant data from exocentric sources. Thus, our methods are not specifically designed for cleaning HT100M but for adapting it to egocentric tasks. Consequently, it is expected that the improvements are primarily observed in egocentric tasks. However, the fact that our methods also achieve comparable or improved performance on exocentric tasks highlights their broader applicability.

If possible this work also works in other datasets like K600, Moments In Time etc.?

Thank you for your suggestions! In Table 5, we actually demonstrated that applying our methods to other datasets (Kinetics-700, Something-Something v2, and COIN) can also achieve improvements over baselines.

Statistics of HTM-EMBED

We list the statistics of our dataset in Line 255-260. Overall, there are around 202K videos and 2.4M video-language pairs.

Could you add the time comparison for LaViLa-B with LaViLa-B+EMBED?

Because both LaViLa-B (Ego4D+HTM-AA) and LaViLa-B+EMBED (Ego4D+HTM-AA) are trained with ~2M exocentric video-language pairs, either from the original HTM-AA or our constructed one. Their training time is roughly equal and is ~1.5 days with 32 V100s for a base model.

The technical contribution of this work appears limited, as it primarily proposes a data filtering process for exocentric videos.

We would like to clarify the unique contributions of our work and our framework goes beyond a data filtering process. Specifically, as in our general response, we position our approach as an active video mining and tailored language generation process—a novel direction not previously explored.

Furthermore, we compare our method with several existing models, demonstrating its effectiveness and highlighting its value to the research community.

We will make these points more explicit in the revised version and are happy to incorporate comparisons with any additional references if provided.

Another key concern is the reliance on egocentric retrieval and classification benchmarks, which have been heavily used in recent years. It’s widely acknowledged that improvements on these benchmarks may not necessarily indicate a truly understanding of egocentric content or applicability in real-world scenarios.

Thank you for raising this point!

We would like to mention that egocentric retrieval and classification tasks are themselves representative of real-world applications. Additionally, most existing works on egocentric video learning (e.g., EgoVLP, EgoVLPv2, LaViLa) evaluate models on these benchmarks, and we follow their evaluation settings while also extending the evaluation to exocentric tasks for a more comprehensive analysis. Therefore, the existing evaluation results reflect the real-world applicability of our approach for these areas.

That said, we agree on the importance of evaluating our model's applicability in broader settings. Following your suggestions, we evaluated our baseline model (LaViLa-B, Ego4D+HTM-AA) and our proposed model on egocentric QA tasks (EgoTaskQA [1] and ACQUIRED [2]) as well as an embodied task (Door-Open Human with RoboCLIP [3]) For egocentric QA tasks, we integrated our video encoder with a BART decoder and fine-tuned the model using LoRA. For the embodied task, we conducted a preliminary study by replicating the RoboCLIP setting, substituting its video encoder with our baseline or EMBED. As shown in the table below, our model consistently outperforms the baseline in these settings, demonstrating its generalizability.

[1] Jia et al., EgoTaskQA: Understanding Human Tasks in Egocentric Videos, NeurIPS 2022.

[2] Wu et al., ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos, EMNLP 2023.

[3] Sontakke et al., RoboCLIP: Generating Rewards using a Single Demonstration, NeurIPS 2023.

We hope these results resolve your concerns and we are happy to address additional questions that you may have.

We sincerely thank the reviewer for recognizing the strengths of our paper and engaging in a constructive discussion.

Your suggestion to evaluate our approach in a broader context motivated us to conduct additional experiments on challenging QA and embodied tasks. These results further validate the effectiveness and value of our framework, and we will include them in the revised version.

We will also make our contributions clearer, including systematically identifying the challenges of leveraging exocentric data for egocentric tasks and proposing practical, effective solutions that achieve significant improvements over robust baselines across action recognition, QA, and embodied tasks.

We sincerely thank the reviewers for their thoughtful feedback and constructive suggestions, which have significantly strengthened our work.

During the discussion period, we addressed key concerns raised by the reviewers, including:

• Clarified Novelty: Emphasized that our framework integrates active video data mining and tailored language generation processes, going beyond data filtering or annotation refinement.
• Broadened Evaluation: Added new experiments on challenging QA and embodied tasks, demonstrating the robustness and broad applicability of our method.
• Ensured Fair Comparisons: Highlighted strict control of experimental settings, ensuring identical training strategies and data sources with baselines to isolate the contributions of EMBED.

Our work systematically identifies the challenges of using exocentric data for egocentric representation learning and proposes practical, effective solutions that achieve substantial improvements over baselines across tasks. The reviewers' comments will be fully incorporated into the revised version.