HENASY: Learning to Assemble Scene-Entities for Interpretable Egocentric Video-Language Model

We appreciate your acknowledgment of our well-written paper, the importance and novelty of our approach, and the extensive experiments.

1: Comparison with HelpingHands

We recap the key improvements of our work over HelpingHands as below

This discussion is included in Section 2, lines 87-93 of the submission.

We highlight our improvements over HelpingHands across various metrics as below (Tables 1 and 2 in the paper):

Discussion: Our primary objective extends beyond achieving high benchmark scores to advancing interpretability within video-language models. Our model significantly diverges from HelpingHands by emphasizing compositional video understanding. Additionally, it improves upon HelpingHands by focusing on interpretable representations. This shift towards interpretability represents a strategic choice aimed at impacting areas where understanding model decisions is critical.

2: Ablation studies

In Lines 159-161, we identified 3 major limitations in slot-attention (GroupViT) if we merely employ it: (i) direct process of video patch tokens from scratch, which neglects powerful video representations already learned by pre-trained models.

(ii) originally proposed in image domain, making it unable to model dynamic entities with temporal information in videos.

(iii) unable to model object-environment interactions, which is essential to capture spatiotemporal dynamics of the videos.

We address these limitations through: (i) A novel bootstrapping stage: This aims to leverage the powerful pre-trained model for video patches encoding. The corresponding ablation study was in Table 3, row 3.

(ii) Temporal-aware grouping (TAG): This targets preserving temporal dimensions, crucial for our hierarchical dynamic entity assembly approach. Given the foundational role of TAG in our model, we did not conduct a separate ablation study for it, as removing it would undermine the model’s functionality.

(iii) Entity-aware decoder: This decoder plays a crucial role in modeling entities-environment interactions, enhancing our model’s overall performance. It propagates entity-level features from local entity encoder to enrich video-level embeddings from global encoder. It significantly contributes for the effective performance of our model. If we excluded this decoder, the model would rely solely on entity features from our slot-based local entity encoder and would end up with a substantial performance drop (Table 3, rows 1-2).

3: Initialization with other pre-trained models

Due to time constraint of the rebuttal period, we were able to conduct experiments on an additional pre-trained model, EgoVLP, besides the pre-trained model LaViLa reported in the submission. Below is a performance comparison when initializing our model with different pre-trained models, against the baseline results from LaViLa and EgoVLP:

This table demonstrates that our model, when initialized with EgoVLP, not only performs competitively with its initialization using LaViLa, but also consistently outperforms the original EgoVLP across various benchmarks. This highlights the versatility and robustness of our model. We will add this result in our final version with an extra page.

4: Scalability

In our submission, we primarily used TSF-B as the backbone. As suggested by the reviewer, we have trained with a larger version, TSF-L, to further investigate the scalability. Due to time constraints, this model has not finished its training process. Nevertheless, we are pleased to report the current best results and comparison as below:

This table illustrates the performance improvements of TSF-L over TSF-B for both LaViLa and our method. The results demonstrate that our model scales effectively with the larger backbone, showing consistent or greater gains across all benchmarks compared to LaViLa. These evidences underscore the scalability of our approach when equipped with more powerful backbones. We will add our final result to camera-ready version.

We appreciate the reviewer of their acknowledgment that our approach is groundbreaking with strong interpretability, comprehensive experiments. We would like to address concerns and questions raised by the reviewer as follows:

1: Scalability and computation

Scalability: In our submission, we primarily reported results using TSF-B architecture as the backbone. As suggested by the reviewer, we have initiated training with a larger version of the backbone, TSF-L, to further investigate the scalability of our method. However, due to the constrained time frame of the rebuttal period, this model has not finished its training process. Nevertheless, we are pleased to report the current best results and comparison with a baseline (LaViLa) using similar backbones:

This table illustrates the performance improvements of TSF-L over TSF-B for both LaViLa and our method. The results demonstrate that our model scales effectively with the larger backbone, showing consistent or greater gains across all benchmarks compared to LaViLa. These evidences underscore the scalability of our approach when equipped with more powerful backbones. We will add our final result to camera-ready version.

Computation: We have conducted a study to evaluate the computational cost (Table 5, Page 9 of the submission). In this study, we also compared with the SOTA HelpingHands. It is summarized as follows.

2: Resource-constrained devices

In real-world applications, computational resource constraints can vary significantly, leading us to categorize environments into two primary types: cloud computation and standalone systems. For cloud computation, HENASY is particularly well-suited as it can leverage extensive computational resources similar to those used in deploying large language models (LLMs). For standalone systems, we are investigating several optimization techniques to ensure efficient deployment. These techniques include model pruning, quantization, and knowledge distillation, which notably reduce model size and computational load while maintaining competitive performance. The original HENASY model requires 4.8GB of GPU memory, making it compatible with modern devices like the Jetson AGX Xavier (up to 32GB GPU memory), Jetson Xavier NX (8GB GPU memory), and Jetson TX2 (8GB GPU memory). However, deployment in resource-constrained environments is beyond the scope of this paper.

3: Generalize across diverse real-world scenarios and different video domain

As clarified in the session “Benchmarks and Evaluation Protocols” on Pages 7, 8 of the submission, we evaluated the proposed HENASY using different protocols, including zero-shot transfer protocol and visual & textual representation protocol. We conducted evaluation on various datasets, including EgoMCQ, EgoNLQ, EgoMQ, EK100-MIR, EK100-CLS, and EGTEA. It is important to note that those datasets have been acquired by different devices under different real-world scenarios settings and across various video domains as follows:

• EK-100: kitchen activities, collected by GoPro.
• EGTEA: indoor activities, collected by SMI eye tracking glasses.
• EgoMCQ, EgoNLQ, EgoMQ: indoor and outdoor daily activities (cooking, sport, shopping, lawn mowing, etc.), collected by multiple cameras such as GoPro, Vuzix Blade, Pupil Lab, etc.

The experimental results in Table 1, Page 8 on the zero-shot transfer protocol and Table 2, Page 9 on the visual and textual representation protocol demonstrate the effectiveness of our proposed method across various video domains in diverse real-world scenarios. Notably, our model consistently outperforms previous SOTA methods with significant margins.

4: Impact of spatiotemporal token grouping mechanism

Our Temporal-aware grouping (TAG) mechanism has a foundational role in our model, which leverages slot-based methods in images into modeling spatiotemporal token grouping. Concretely, it helps preserve the temporal dimension during our hierarchical dynamic entities assembly process. Given the essential role of TAG in our model, we are unable to conduct a separate ablation study for it, as its removal would undermine the functionality of our model.

5: Comparative analysis of interpretability

We conduct a quantitative experiment on Ego4D for visual grounding task. In the absence of ground truth segmentation masks for Ego4D, we generated these masks ourselves. Due to time constraints, we managed to obtain labels for 200 videos. Below, we present the performance comparison in terms of mIoU scores with HelpingHands, our most related work that supports visual grounding:

Discussion: although being the SOTA in egocentric tasks and visual grounding task, HelpingHands only provides coarse bounding boxes of objects, leading to lower mIoU scores due to inadequate coverage of the target masks. In contrast, our model employs segmentation masks that closely align with the ground truth, resulting in higher mIoU scores, demonstrating superior visual grounding capabilities.

Dear Reviewer cNiH,

As we are nearing the end of the discussion phase, we kindly seek your feedback to ensure that our responses have addressed your initial concerns. If our rebuttal satisfactorily addressed your concerns and questions, we would be very grateful if you would consider to raise your rating.

Thank you very much for your attention.

We thank the reviewer for recognizing that our paper is well-organized, easy to follow, competitive results in real-world applications.

1: Comparison with [1, 2]

We provide the comparison as below:

We note that [2] is published at ICLR 2024 (2 weeks before NeurIPS submission).

2: Comparison with GroupViT

To illustrate the challenges with merely applying GroupViT to video, how our method addresses these issues, and what set us stood out from GroupViT, we provide the following table:

3: Co-training from both egocentric (ego) and exocentric (exo) and transferability from ego to exo tasks

On one hand, it is obvious that incorporating ego and exo is essential for many real applications in robotics and augmented reality. However, the dramatically different viewpoints pose significant challenges. Most ego datasets just recently become available, and very few ego videos are synchronized with corresponding exo videos. Additionally, ego datasets are much smaller in scale compared to exo ones. Therefore, leveraging exo to improve model performance on ego is a viable approach [A]. Early work [B] focused on representation learning using paired ego-exo videos. However, paired videos are much harder to obtain compared to unpaired ones. Recent studies [C] made notable progress with unpaired videos.

On the other hand, transferability from ego to exo tasks presents a significant challenge and opportunity. The primary challenge is the alignment of different viewpoints, particularly between unpaired multi-view [D]. The ongoing research on joint view-invariant learning [E], domain adaptation [F], knowledge distillation [G] offer promising solutions.

Despite co-training with mixed ego-exo data and transferability between ego-exo are exciting directions, our current objectives are solely focused on ego tasks, with an emphasis on providing trustworthiness and transparency to the model at a fine-grained level of alignment. Furthermore, we strongly believe that this capability can help bridge the gap between different viewpoints at fine-grained alignment, and this should be a focus of future research.

[A] Weinland, D., etc "Making action recognition ...", ECCV2010

[B] Yu, H., etc “What i see is what you see …”, ICM2019

[C] Huang, Y., etc “EgoExoLearn …”, CVPR2024

[D] Wang, Q., etc. “Learning from semantic alignment…”, ICCV2023

[E] Xue, Z.S., etc. “Learning fine-grained …”, NeurIPS2023

[F] Choi, J., etc, “Unsupervised and semi-supervised …”, WACV2020

[G] Li, Y., etc. “Ego-exo…”, CVPR2021

4: Visual grounding

We conduct a quantitative experiment on Ego4D, as HelpingHands provides visual grounding results exclusively on this dataset. In the absence of ground truth segmentation masks for Ego4D, we generated these masks ourselves. Due to time constraints, we managed to obtain labels for only 200 videos. Below, we present the performance comparison in terms of mIoU scores:

Discussion: HelpingHands uses coarse bounding boxes for visual grounding, leading to lower mIoU scores due to inadequate coverage of the target masks. In contrast, our model employs segmentation masks that closely align with the ground truth, resulting in higher mIoU scores, demonstrating superior visual grounding capabilities.

Dear Reviewer xaMC,

We sincerely appreciate the time and effort you have dedicated to providing feedback on our work. Your insights are invaluable in helping us improve its clarity and overall quality. We want to follow up to check if our response fully addressed your concerns/questions before ending the discussion period.

Thank you once again, and we look forward to your feedback.

Dear Reviewer xaMC,

As we are nearing the end of the discussion phase, we kindly seek your feedback to ensure that our responses have addressed your concerns. If our rebuttal satisfactorily addressed your concerns and questions, we would be very grateful if you would consider to raise your rating.

Thank you very much for your attention.

We thank the reviewer for their thoughtful evaluation and acknowledgment that our approach is well-motivated, novel, and effective under extensive experiments, providing intuition and experience for future model design. We will fix all noticed typos in our final version. Below, we would like to address the concerns raised:

1: Clarify the improvement over most recent SOTA model (HelpingHands)

We would like to recap the differences between our work and HelpingHands by the following table:

This discussion is included in Section 2, lines 87-93 of the submission.

In addition, we highlight our improvements over HelpingHands across various metrics as below (Tables 1 and 2 in the paper):

Discussions: We emphasize that the primary goal of our proposed method is to advance interpretable video representation by leveraging a compositional video understanding approach. This would create a new vibe to our research community from solely competing on benchmarks to prioritizing trustworthiness, transparency, and reliability. Although the improvements in some metrics may appear incremental, they are substantial within the context of interpretability and compositional analysis, as it pursuits of building transparent and accountable AI systems. Equipping the model with interpretability to ensure trustworthiness while enhancing performance is crucial for practical applications, especially in sensitive areas such as healthcare and autonomous driving.

2: Error Analysis to diminish the influence of randomness

Reproducibility is highly prioritized in our work, and as such, we have implemented fixed seeds in all random-related functions for every experiment to ensure that the community can reliably reproduce the performance reported in our paper.

In response to this concern, we initiated evaluations of our model trained with different random seeds. However, due to the limited time available for the rebuttal and resource constraints, we can only train two additional models. We report the performance in terms of mean and std across several zero-shot benchmarks as below:

The above table with high mean and low std indicates that our method is effectiveness, stable, robust and converges well under randomness.

3: Formulation of masks prediction for Projection loss

In Appendix Section B, we provided a comprehensive description of three steps involving the formulation of the masks prediction. Specifically:

1. Similarity Computation: We calculate a similarity array between the learnable group tokens and the input tokens.
2. Group Assignment: Each input token is assigned to a group token based on the highest similarity score. To retain differentiability essentially for training, we employ the straight-through trick during the assignment step. Obtaining assignment array.
3. Saliency Map Generation: Saliency maps are then generated using the assignment. These maps effectively highlight the spatial locations and dynamic contours of entities across different frames of the video, providing a visual representation of the model’s focus and understanding of the scene.

The complete mathematical formulation and additional details are thoroughly explained in Appendix Section B, ensuring transparency and reproducibility of our methods. We will add reference to corresponding appendix in the projection loss description to make it clearer.

4: Clarification on the use of multiple variables

We use abbreviation to name variables for clarity and succinctness in mathematical formulations.

To recap, there are five types of tokens as follows:

• z: video patch tokens, which capture local features from video frames.
• c: learnable <CLS> token, which is used to aggregate and represent global video features.
• g: learnable group tokens, which are pivotal in organizing video information into meaningful clusters.
• s: segment tokens, which are derived from segmenting video patch tokens after the bootstrapping stage.
• e: entity tokens, which represent distinct entities clustered at the final stage of the process.

To aid readability, we plan to include a notation table in the appendix, summarizing all variables and functions with clear descriptions and references to their respective sections. This will facilitate easier navigation and understanding of the methodology for all readers.

Dear Reviewer EqYJ,

Thank you for taking the time to read our rebuttal and for providing a positive rating. We greatly appreciate the valuable feedback you have given us, and we will incorporate your suggestions in the final revision of our paper.

Thank you once again.