Thank you for your detailed review and constructive feedback on our paper. We have carefully considered your comments and below we provide our responses to address your concerns. Note that we refer to the table/figure numbers in our revised paper.

W1: The Asymmetric Contrastive Objective Lacks the Degree of Methodological Novelty Expected

We provide comparison with previous works: Both the asymmetric design of EgoNCE++ and the LLM-based / vocab-based hard negative mining in V2T loss represent novel contributions to egocentric video understanding.

• In Egocentric Video-Language Modeling: Unlike prior pretraining symmetric objectives, our asymmetric EgoNCE++ considers from both vision-to-text (V2T) negative mining and text-to-vision (T2V) positive sampling perspectives within a unified contrastive loss framework. This design effectively addresses the unique challenges of egocentric video data. As shown in our results on EgoHOIBench, existing hard negative mining techniques struggle to make these fine-grained distinctions (negative mining techniques are listed in Table 5 in the supplementary material, where EgoVLP adopts negative mining based on visual similarities and EgoVLPv2 mines negatives that have similar features), while our method consistently improves robustness and generalization across multiple datasets. Our method also surpasses state-of-the-art works as illustrated in Table 1.
• In Hard Negative Mining: Our use of vocab-based augmentation for efficiently generating tailored negatives offers a new technical insight. This approach significantly boosts performance without relying on large-scale language models, which is a limitation of LaCLIP [3]. Specifically, generating training data for 2.5M samples took approximately 192 GPU hours on A6000 hardware (i.e., 0.05616 ms per text sample to generate 10 negatives). In contrast, the vocab-based negative sampling method requires only 0.039 hours on 16 CPU threads (i.e., 0.05616 ms per text sample to generate 10 negatives), enabling efficient online updates. This efficiency and effectiveness are critical for advancing understanding in egocentric contexts.

W2: The Risk of Overfitting to a Controlled Benchmark and The Generalizability of Our Model.

We understand the concern about the generalizability of our model. While our benchmark is tailored to evaluate fine-grained hand-object interactions, we emphasize that its data sources—the largest egocentric dataset Ego4D—span diverse scenarios and represent real-world variability. This ensures that our pretrained models exhibit robust performance across related tasks.

• Performance on Other Benchmarks: While our ablation studies in Table 3 and Table 4 were primarily conducted on EgoVLP, our primary results, LaViLa++, pretrained on the state-of-the-art model LaViLa shows a great generalization on other public benchmarks in Figure 1. For instance, it is essential to note that our method competes with state-of-the-art models like HelpingHands [1] and HENASY [2] in Table 1 on EK-100-MIR, which use extensive grounding data and more parameters for training. Our LaViLa++ also improves the action recognition of 3805 classes than LaViLa on Epic-Kitchen-100 by +4.53%. Besides, LaViLa++ also achieves a human level of temporal understanding on ActionBench. This underscores the significant impact of EgoNCE++.
• Overfitting Concerns: Overfitting appears unlikely, as we observe minimal descreases across most out-of-domain benchmarks, such as Epic-Kitchens, CharadesEgo, and EGTEA. Also, the diversity of video content in Ego4D and the variability introduced by LLM-based text generation contribute to preventing overfitting. We include vocabulary frequency statistics to support the vocabulary diversity provided by LLM in table below: | Training Data Vocabulary | Verb | noun | |----------------|-------|------| | positives | 7679 | 2171 | | positives+negatives | 15643 | 6654 | | Δ (by LLM) | 7964 | 4483 |

Q1: Comparison of the Proposed Asymmetric Contrastive Objective with Other Contrastive Techniques in Video-Language Modeling

Table 5 records existing hard negative mining approaches in egocentric contexts, but these EgoVLMs fail to address the specific challenges posed by EgoHOIBench. Our method, incorporating asymmetric V2T and T2V losses, effectively addresses these gaps and leads to improved performance on challenging downstream tasks, as shown in Figure 1.

Q2: Can We Elaborate on the Robustness in Generated Negative Captions and How the Bias in LLM Affects the Model's Performance?

We appreciate the suggestion to evaluate the robustness of the generated captions and the potential bias in LLMs.

• Elimination of False Negatives: During pretraining, we remove the false negatives by using spaCy for word extraction and the Ego4D dictionary for validation. In EgoHOIBench, we also rely on the semantic understanding of LLMs to ensure the generated negatives differ significantly in meaning. We explicitly prompt LLMs to "make the action in the text significantly different while keeping the other words unchanged."
• Bias in LLM When Constructing EgoHOIBench by other LLMs: To further evaluate potential biases in LLMs, we generated EgoHOIBench using alternative LLMs such as DeepSeek-200B and GPT-4o. Results in table below show that LaViLa++ and EgoVLP++, pretrained using EgoNCE++ and LLaMA generated negatives, consistently improves HOI-verb and HOI-noun recognition. The results are weaker than those on LLaMA-EgoHOIBench due to the vocabulary shift. The numbers in the table denotes verb/noun/action accuracy respectively:

The GPT-4o and DeepSeek tend to generate different words compared with LLaMA, whose differences are shown in the table below. In the table below, we calculate vocabulary statistics, where the value in position (i, j) represents the proportion of the vocabulary from the i-th set that overlaps with the j-th set:

• Vocab-based Negative Mining is Competitive with LLM-based Method: We mitigat potential biases from LLMs by using a vocab-based approach for caption augmentation. This strategy reduces over-reliance on pre-trained LLMs and maintains diversity of negative samples, which helps avoid bias while still generating effective negatives.

Q3: How Does EgoHOIBench Compare to Existing Datasets, and the Limitations When Applied to Real-World Scenarios?

• EgoHOIBench is designed to capture a wide range of human activities from the diverse Ego4D dataset, making it a valuable complement to existing benchmarks that focus on narrower scenarios (e.g., Epic-Kitchens, which is limited to kitchen scenarios). Our results on EgoHOIBench demonstrate its potential to improve general egocentric hand-object interaction recognition. Additionally, diverse datasets like Epic-Kitchens and EGTEA demonstrate the generalization ability of our pretrained models to domain-specific scenarios, such as kitchen environments.

We hope that our responses have adequately addressed your concerns. Should you have any remaining points or further steps we can take to strengthen our work, we would greatly appreciate your continued feedback. Thank you again for your valuable feedback.

References:

[1] Helping hands: An object-aware ego-centric video recognition model, ICCV2023.

[2] Henasy: Learning to assemble scene-entities for interpretable egocentric video-language mode, NeurIPS2024

[3] Improving CLIP Training with Language Rewrites, NeurIPS2023

Dear reviewer 3po1,

We again thank you for your valuable feedback, which has greatly helped improve the quality of our paper. As the rebuttal deadline approaches (November 26th AoE), we kindly ask if you could review our rebuttal to see whether it addresses your concerns.

We would greatly appreciate any additional feedback or raise further questions you may have during this discussion period.

Best regards,

Authors

Q1: Analysis on Loss and Dataset Design

We haven't analyze much about the relation between our loss and EgoHOIBench. Instead, to evaluate potential biases in LLMs, we generated EgoHOIBench using alternative LLMs such as DeepSeek-200B and GPT-4o. Results in table below show that LaViLa++, pretrained using EgoNCE++ and LLaMA generated negatives, consistently improves HOI-verb and HOI-noun recognition. The results are weaker than those on LLaMA-EgoHOIBench due to the vocabulary shift. The numbers in the table denotes verb/noun/action accuracy respectively:

Besides, the LaViLa++ that pretrained on EgoNCE++ not only performs well on EgoHOIBench, but also generalizes to other benchmarks. For example, as shown in Table 1, our method competes with SoTA models like HelpingHands [1] and HENASY [2]. Considering external datasets and parameters used by [1][2], the improvement is significantly more than marginal. As for the results in Table 4, to ensure fair comparison for all ablations, we employ a fixed number of 10 negatives on EgoVLP. However, our primary results explore more on LaViLa, as shown in Figure 1 and Figure 7, resulting in larger gains than those on EgoVLP.

Q2: Clarification on “Not Made Excessively Difficult” Sentence

The term "difficult" refers to the fine-grained nature of negative examples, which differ by only one word from the positive captions. Our aim is to examine the HOI understanding capabilities of EgoVLMs without requiring a very large number of classes like those in Epic-Kitchens (3805). To avoid false negatives, we prompt LLM to generate negatives with only one word difference, but with drastically different semantics (not made excessively difficult). The LLM is instructed to "make the action in the text significantly different while keeping other words unchanged." Detailed prompt examples can be found in Figure 13 in the supplementary material.

Q3: Noun-Based Clustering Only

Please refer to our response in W2.

Q4: Prevention of False Negatives in Video-to-Text Loss

Please refer to our response in W3.

Q5: Data Selection for HOI-Bench

The video-text data in EgoHOIBench are chosen from EgoMCQ. Please refer to our detailed response in W4.

Q6: Mitigation of Hallucination During Dataset Construction

To minimize hallucinations and ensure a consistent output format, we employ in-context learning and restrict the model's responses to Json format. We randomly sampled 200 data samples for manual evaluation and found no hallucinations.

Q7: Ablation with EgoNCE++ for Loss and InfoNCE for V2T

We conducted ablation studies using EgoNCE++ for T2V loss and InfoNCE for V2T loss. The results in Table 3 show that the model continues to benefit from our T2V positive sampling loss:

Q8: Impact of Number of Negatives on Base Models

We have tested the effect of applying the same number of 10 negatives without EgoNCE++ (rule-based method, defined in line 461-463) and found limitations of rule-based method in generalization, as shown in Table 4. The limited generalization of rule-based methods may result from having seen the sentences during pretraining, while the EgoNCE++ generates new fine-grained negative options that are never seen during previous training processes.

Q9: Histogram for Positive/Negative Similarities

Please refer to our response in W5.

Q10: Feature Space Comparison After EgoNCE++

Thank you for this insightful suggestion. We have visualized feature spaces after applying EgoNCE++ and included these in the supplementary materials. Combining the visualization of feature spaces and the video-text matching similarity scores, the results show that while the overall clustering remains object-centric, video-text matching improves, confirming the effectiveness of our method.

We hope our clarifications address your questions. Should you have any remaining points or further steps we can take to strengthen our work, we would greatly appreciate your continued feedback. Thank you again for your constructive and valuable suggestions.

References:

[1] Helping hands: An object-aware ego-centric video recognition model, ICCV2023.

[2] Henasy: Learning to assemble scene-entities for interpretable egocentric video-language mode, NeurIPS2024

We greatly appreciate the time and effort you invested in providing these detailed observations, questions, and comments. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript. Note that we refer to the table/figure numbers in our revised paper.

W1: Similar Design of Loss and EgoHOIBench

We acknowledge that the similarity between the benchmark design and the loss function may contribute to large improvement on EgoHOIBench. However, our designed loss also performs well on other benchmarks. For example, as shown in Table 1, our method competes with SoTA models like HelpingHands [1] and HENASY [2] which use extensive grounding data and more parameters for training. These results highlight the meaningful impact of EgoNCE++ even without external datasets.

W2: Why There Is a Large Focus on Still Clustering Videos Based on Similar Noun Representations Only?

Since our V2T negative mining on HOI-verbs might damage the strong recognition towards noun, we aim to maintain the noun clustering nature by T2V positive sampling on nouns. Relavent ablation shows its necessity in Table 12 in supplementary material, clustering video representations based solely on nouns yields the largest gain.

W3: Details on Negative Generation (Section 3) and Reduction of False Negatives

• Details on Negative Generation: Thank you for your suggestion of clarifying how negatives are generated in Section 3. In the revised version of our manuscript, we provide more details about how training negatives are constructed using LLMs or dataset vocabulary: (1) LLM-based. Following a process similar to EgoHOIBench, we prompt LLM with in-context learning examples to generate Json-formatted sentences that are semantically distinct from the original text. (2) Vocab-based. Using spaCy, we extract all verbs and nouns from the pretraining datase and replace HOI-nouns or HOI-verbs with randomly selected extracted words.
• Reduction of False Negative: During the negative generation process for both methods, we leverage the established synonym list from Ego4D to ensure minimal overlap and eliminate false negatives, as stated in Line882-885. The LLM-based method would be more capable of identifying synonyms due to the powerful semantic understanding capability, while the vocab-based method is only aware of the synonyms in Ego4D vocabulary.

W4: Details on HOI-Bench Data Collection

We acknowledge the need for clearer information regarding the construction of EgoHOIBench. The main paper will be updated to explain the rationale behind the number of videos, captions, and category choices from Ego4D. Specifically:

• Number of Negatives - 10: Setting the number of negatives to 10 (i.e. 10 noun negatives, 10 verb negatives) will form 100 HOI negatives, which aligns with the typical action recognition setting. Additionally, we aim to examine the HOI recognition capabilities of EgoVLMs without requiring a large number of classes. You can download our data from our anonymous code repository.
• The Chosen Number of Videos - 29K: To filter the videos from EgoMCQ that feature hand-object interactions, we applied two filtters: (1) Captions starting with #C (indicating the wearer, thus excluding data involving others, which starts with #O), and (2) Captions that contain both verbs and nouns, excluding non-interactive daily activities (e.g., '#C walks away'). As a result, the video captions primarily follow the format '#C C #HOI-verb #HOI-noun ...', where '...' represents other phrases or articles, as our statement in Line862-864. Such filtering process resulted in 29K videos.
• Choice of Category: We did not perform special selection of categories during the video filtering process. The vocabulary statistics of EgoHOIBench is shown in Figure 8 in supplementary.

W5: Metric Clarification (Original Equation 6, Now Removed)

Thank you for suggesting drawing a histogram instead. We follow this suggestion in the revised version and present a histogram of positive and negative similarities in Figure 6. This visual representation offers a more intuitive understanding of distribution shift, where our EgoNCE++ effectively makes video-positives better matched than video-negatives.

Comment1: Inconsistent Lower and Upper Cases in Table 3

Thanks for pointing it out, we have revised it in the updated version.

Dear reviewer 7uyU,

We sincerely appreciate your acknowledgment of our efforts in addressing your concerns. Your constructive and insightful suggestions have been critical in enhancing the the quality of our work. We are also grateful for the revised score—thank you for recognizing our improvements.

We welcome any further discussion and are glad to provide additional clarification if needed.

Best regards,

Authors

Thank you for appreciating our work and for providing thoughtful feedback that helps us further improve our work. Note that we refer to the table/figure numbers in our revised paper. Below, we address your concerns in detail:

W1: Can We Make a Feature Space Comparison after Applying EgoNCE++ to See If the Verb-Anchored Changed?

We appreciate the insightful suggestion to include feature space comparisons after applying EgoNCE++. To address this, we have provided additional visualizations in Figure 13 of the Supplementary Material. These visualizations reveal that the feature space preserves its object-centric nature. Notably, the inherent structure of the video or text representation space does not necessarily impact the video-text matching results. Our findings, illustrated in Figure 5 (b) and Figure 6, demonstrate that our approach enhances video-text matching by making the correct video-text pair more distinguishable from negative video-text ones.

W2: Can We Conduct Additional Studies on the Performance Gains versus Computational Expense (FLOPs) in Order to Better Know the Trade-Off for the Approach?

Thank you for highlighting the computational trade-offs of our approach. Our current approach, which uses LLaMA-3-8B to generate HOI candidates, operates offline and requires a higher computational cost. Specifically, generating training data for 2.5M samples required approximately 192 GPU hours on A6000 hardware (equivalent to 276.48 ms per 10 negatives). In comparison, a vocab-based negative sampling method operates far more efficiently, requiring only 0.039 hours on 16 CPU threads (equivalent to 0.05616 ms per 10 negatives) and enabling online updates. A detailed comparison of the trade-offs between the LLM- and Vocab-based methods on EgoVLP are presented in the table here:

W3: It Seems That for Other Datasets the Improvement of EgoNCE++ Is Marginal. Will There Be a Possible Overfitting or Lack ff Variety Since the Primary Performance Measure Is Based on EgoHOIBench?

We recognize that while EgoNCE++ achieves significant gains on EgoHOIBench, its improvements appear relatively modest on datasets such as CharadesEgo. Below, we present a detailed analysis of the factors contributing to these less significant gains:

• Notable Improvement under the Same Setting Comparing LaViLa++ and LaViLa (Figure 5 and Table 1): Notably, our method EgoNCE++ brings large improvement under the same experimental setup. For example, LaViLa++ improves action recognition across 3805 classes on Epic-Kitchen-100 by +4.53%, and achieves a human level of temporal understanding on ActionBench. Our improvements on EK-100-MIR are comparable to HelpingHands [1] and HENASY [2] (see Table 1 on EK-100-MIR), which benefits from extensive grounding data and larger parameter sizes used by our competitors.

• Overfitting Concerns: We find no significant decreases in benchmark metrics, suggesting overfitting is unlikely. The diversity of video content in Ego4D and the variability introduced by LLM-based text generation contribute to preventing overfitting. To illustrate this, we provide vocabulary frequency statistics in table here:

• Large Domain Gaps in Zero-Shot Setting on Datasets like CharadesEgo (Table 7): The marginal generalization improvement on datasets like CharadesEgo can be attributed to the out-of-distribution (OOD) gap between Ego4D and CharadesEgo, which is a common sense in egocentric domain as posed by EgoVLP[3].

• Impact of LoRA Fine-tuning in Fine-tuning Settings (Table 10): The performance on EK100MIR and EGTEA in fine-tuning settings is influenced heavily by the base model's capabilities. Our LoRA fine-tuning approach limits significant changes in feature space distribution, therefore resulting in limited improvements as depicted in Table 10.

• Stability When Pretraining on Larger Model (Table 9): While EgoNCE++ consistently boosts performance, larger models like LaViLa-Large already possess a strong baseline for comprehension. Consequently, the relative gains from negative text supervision alone may become relatively smaller when pretrained on larger models.

We hope these clarifications address your questions. Please let us know if there are additional concerns or suggestions. We greatly appreciate your constructive feedback and support.

References:

[1] Helping hands: An object-aware ego-centric video recognition model, ICCV2023.

[2] Henasy: Learning to assemble scene-entities for interpretable egocentric video-language mode, NeurIPS2024.

[3] Egocentric Video-Language Pretraining, NeurIPS2022.

Dear reviewer DGPT,

We again thank you for your valuable feedback, which has greatly helped improve the quality of our paper. As the rebuttal deadline approaches (November 26th AoE), we kindly ask if you could review our rebuttal to see whether it addresses your concerns.

We would greatly appreciate any additional feedback or raise further questions you may have during this discussion period.

Best regards,

Authors

I think the author properly answered my question during the rebuttal period. • These visualizations in Fig.12 did reveal that the feature space preserves its object-centric nature. • The additional FLOPs experiments make sense and do not significantly increase the computational resource based on LLMs usage. • I agree with authors that after the comparison, the model did helps to other benchmark not including their own EgoHOIBench.

Given the information above, I will raise my point and thanks for the rebuttal and comments made by authors.

We appreciate the reviewer’s thoughtful assessment and acknowledgment of our paper's clarity, motivation, and contributions. Below, we address the concerns in detail:

W1: Response to Concerns about Novelty

We understand the reviewer's interest in better distinguishing the novelty of our approach, particularly in relation to existing works such as negCLIP [1]. Below, we provide a detailed response to emphasize the unique contributions of our work:

• Distinct Problems Addressed: While both negCLIP and our work focus on compositional understanding, the core problems each approach tackles are different. Specifically, negCLIP primarily addresses challenges related to positional and word order sensitivity, such as errors exemplified by statements like "the grass is eating the horse". In contrast, our work focuses on HOI understanding with a particular emphasis on distinguishing precise verbs or nouns in free-form language queries, such as differentiating "C drops the paper" from other similar actions like "C {other verbs} the paper".
• Unique Negative Construction Process: Unlike negCLIP, which relies on hand-crafted text shuffling, our approach leverages advanced LLMs or a more flexible and efficient vocab-based method (as detailed in Section 3) to generate negatives.
• Novel Asymmetric Loss Design for EgoHOI: Our EgoNCE++ introduces an innovative asymmetric loss structure specifically designed for egocentric data， which unifies vision-to-text (V2T) negative mining and text-to-vision (T2V) positive sampling within a contrastive loss framework. This design effectively addresses the unique challenges posed by egocentric video data.
• Improved Generalization: Unlike previous works such as negCLIP, which sometimes exhibits reduced generalization compared to CLIP, our approach maintains robust performance for EgoVLMs without such degradation. Such generalization is likely due to the intrinsic similarity of egocentric videos (featuring hand-object interactions) compared to the broader variance observed in third-person datasets.

We hope these clarifications sufficiently address the novelty and unique contributions of our work. Should you have additional questions or concerns, we would be happy to provide further explanatioins. Thank you again for your thorough review and constructive feedback.

References:

[1] When and why Vision-Language Models behave like Bags-of-Words, and what to do about it? ICLR2023

Dear reviewer Ns3f,

We again thank you for your valuable feedback, which has greatly helped improve the quality of our paper. As the rebuttal deadline approaches (November 26th AoE), we kindly ask if you could review our rebuttal to see whether it addresses your concerns.

We would greatly appreciate any additional feedback or raise further questions you may have during this discussion period.

Best regards,

Authors