Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning

Slowfast network citation. We appreciate the reviewer for the suggestion. We have added a discussion and cited SlowFast Network[1] in our revised manuscript.

Trainable parameters comparisons. The table below shows the trainable parameters of different methods. Helping hands[2] uses frozen backbone initialized from LaViLa and uses only one additional Transformer Decoder for training, so it has significantly fewer trainable parameters.

Comparison of EgoVideo-G. We compared the performance of models of the same size. The experimental results indicate that our EgoVideo-B and EgoVideo-L models achieve better performance compared to methods of the same size. Both our base and large models were initialized using CLIP. For EgoVideo-G, we aimed to explore the scaling law and test how effectiveness of our data can be on larger models. Therefore, we employed InternVideo2[3], which has a stronger temporal understanding capability, as the backbone. This led to a significant performance improvement in our model. We further examined the impact of our data on EgoVideo-G, as shown in the table below. This result proves that our HOD data can be better leveraged by larger models, showing the possible future directions of incorporating foundation models into egocentric video understanding.

Comparison in Robot Manipulation. In the robot manipulation task, we use EgoVideo-B to test for fairness, and we have clarified this issue in the revised manuscript. The EgoVideo-B model is on the same parameter scale as other baselines.

Input to the Manipulation Model. Our model can dynamically adjust different number of input frames and we use just a single observed frame as input. The capability for single-frame inference underscores the robustness of our EgoVideo. This result also shows that using video-based pretraining can enhance the robot manipulation performance, which we believe reveals the potential for future works on leveraging the strength of video models into the robot manipulation tasks.

[1]Feichtenhofer C, Fan H, Malik J, et al. Slowfast networks for video recognition[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 6202-6211.

[2]Zhang C, Gupta A, Zisserman A. Helping hands: An object-aware ego-centric video recognition model[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 13901-13912.

[3]Wang Y, Li K, Li X, et al. Internvideo2: Scaling video foundation models for multimodal video understanding[J]. arXiv preprint arXiv:2403.15377, 2024.

Analysis of HOD data. We thank the reviewer for the suggestion and provide the following analyses.

1. We analyze the distribution of the words of our HOD against the original EgoClip narrations. We select the top-30 most frequent words, and show their normalized frequency in Figure 3 in our revised manuscript. From this figure, we observe that words in the EgoClip narrations are more long-tailed, while our HOD captions show better distribution. Also, from the words, we can see a lot of "dynamic" words in our HOD data like "up","downwards" and "moves". This from one angle proves our HOD data is dynamically descriptive and of high quality.
2. We compare our HOD data with another data generation technique: LaViLa-Narrator[1]. We evaluate the CIDEr[2] and METEOR metric using the original human narration as the ground truth. From the table below, we can observe that compared with LaViLa-narrator, our CIDEr score is higher, while the METEOR score is slightly lower. This indicates that our narrations are semantically close to the original captions, preserving the original information. The lower METEOR score, we believe, is due to our narrations incorporating more fine-grained dynamic information, which is not present in the original captions.

3. To further verify the quality of our HOD data, we employ GPT-4o[3] for quality assessment. We randomly select 1000 clips(due to time and cost) and let GPT evaluate the score of the caption data for the video clip in a range from 0 to 10. To ensure GPT does not simply assign high scores based on the length of the captions, we also conduct random gerund replacements on our data for comparison. The results, summarized in the table below (also Table 1 in our submisstion), show that our HOD data have a significantly better GPT-Score.

We hope these experiments and explanations can address the reviewers concern.

Inference Time. We believe there is a misunderstanding. All the methods we compared (AVION[4], LaViLa, EMBED[5]) use 16 frames during inference, so our comparisons are fair. We evaluate the straightforward baseline by simply aggregating outputs of AVION-4f and AVION-16f, and we find that the results are similar to AVION-16f, which may be due to the use of the same model parameters. And we evaluate the GFLOPs results below.

In the table we can observe that the increase in inference time for our model compared to ViT at 16 frames, is only similar to ViT's inference time at 4 frames, proving the efficientness of our motion adapter.

[1]Zhao Y, Misra I, Krähenbühl P, et al. Learning video representations from large language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6586-6597.

[2]Maluuba. nlg-eval. https://github.com/Maluuba/nlg-eval. Accessed: 2022-06-01.

[3]Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023.

[4]Zhao Y, Krähenbühl P. Training a large video model on a single machine in a day[J]. arXiv preprint arXiv:2309.16669, 2023.

[5]Dou Z Y, Yang X, Nagarajan T, et al. Unlocking exocentric video-language data for egocentric video representation learning[J]. arXiv preprint arXiv:2408.03567, 2024.

Thanks to the authors for their clarifications. Regarding the experiments on robotic manipulation tasks, I maintain my concerns after careful consideration. The results suggest that the combination of EgoVideo(b) + Ego4D, which may approximate EgoVLP/LaViLa/AVION, competes with robot learning methods like R3M, MVP, and Voltron, achieving surprisingly strong performance. It is weird to me that no prior work has discovered this phenomenon, especially considering that both EgoVLP and R3M were published over two years ago. Given my limited expertise in robotics, I believe the Area Chair could further evaluate this aspect.

Currently, I am inclined to retain my borderline accept (BA) score.

Motion Adapter. We thank the reviewer's suggestion and have added a figure to introduce our motion adapter. The added figure is Figure 4 of the revised manuscript.

Parameter and computational complexity. We compare our EgoVideo with other methods[1-3] in parameter and computational complexity in the following tables. EgoVideo achieves SOTA results at a reasonably low parameter scale.

This table shows that our EgoVideo model maintains a relatively small parameter count, and even with the addition of the motion adapter, the total remains lower than that of LaViLa, highlighting the efficiency of our approach.

This table shows the inference speed of our model and other methods. Thanks to our MotionAdapter, the increase in inference time for our model compared to ViT at 16 frames, is only similar to ViT’s inference time at 4 frames.

Inference process. I believe this is due to a misunderstanding of our approach. We only utilize a hand-object detector and a large language model (LLM) in our data generation pipeline, and our inference process follows the same setting as a standard vision-language pre-training (VLP) model, so it has no impact on the inference process.

Data without annotation. We believe it can be partially solved by using the latest vision language models to provide captions for videos. We list several examples such as GPT4o[5](closed source), or open-source models like llava-next-video[4] and videochat[5]. After video captioning, we can use our HOD data pipeline.

[1] Zhao Y, Misra I, Krähenbühl P, et al. Learning video representations from large language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6586-6597.

[2] Dou Z Y, Yang X, Nagarajan T, et al. Unlocking exocentric video-language data for egocentric video representation learning[J]. arXiv preprint arXiv:2408.03567, 2024.

[3] Zhao Y, Krähenbühl P. Training a large video model on a single machine in a day[J]. arXiv preprint arXiv:2309.16669, 2023.

[4] Li F, Zhang R, Zhang H, et al. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models[J]. arXiv preprint arXiv:2407.07895, 2024.

[5] Li K, Wang Y, He Y, et al. Mvbench: A comprehensive multi-modal video understanding benchmark[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 22195-22206.

Writing error. Thank you very much for your correction. We have updated the pdf to address this error.

Language description rationale & quality. We first explain our data generation process to demonstrate the rationale behind our data pipeline. We use a hand-object detector to obtain the spatial coordinates of the hand and object within the video clip. These coordinates are then normalized to form the hand-object dynamics prompt. Then, we combine this information with the original caption and input it into an LLM to generate new narrations with fine-grained hand-object interaction descriptions. Our rationale is based on two key points:

1. The LLM can understand the hand and object position coordinates we provide, thereby extracting motion and interaction dynamics.
2. The LLM can integrate this information with the high-level original narration to generate semantically richer and more detailed narrations.

To demonstrate the quality of our data, we conduct the following three experiments.

1. We analyze the distribution of the words of our HOD against the original EgoClip narrations. We select the top-30 most frequent words, and show their normalized frequency in Figure 3 in our submission. From this figure, we observe that words in the EgoClip narrations are more long-tailed, while our HOD captions show better distribution. Also, from the words, we can see a lot of "dynamic" words in our HOD data like "up" and "downwards". This from one angle proves our rationale in data generation.

2. We compare our HOD data with another data generation technique: LaViLa-Narrator[1]. We evaluate the CIDEr[2] and METEOR metric using the original human narration as the ground truth. From the table below, we can observe that compared with LaViLa-narrator, our CIDEr score is higher, while the METEOR score is slightly lower. This indicates that our narrations are semantically close to the original captions, preserving the original information. The lower METEOR score, we believe, is due to our narrations incorporating more fine-grained dynamic information, which is not present in the original captions.

3. To further verify the quality of our HOD data, we employ GPT-4o[3] for quality assessment. We randomly select 1000 clips(due to time and cost) and let GPT evaluate the score of the caption data for the video clip in a range from 0 to 10. To ensure GPT does not simply assign high scores based on the length of the captions, we also conduct random gerund replacements on our data for comparison. The results, summarized in the table below (also Table 1 in our submisstion), show that our HOD data have a significantly better GPT-Score.

We hope these experiments and explanations can address the reviewers concern.

Further qualitative analysis. We thank the reviewer for the constructive suggestion. In Figure 8 of our submission, we show several examples for qualitative analysis (indeed in the appendix). Due to the page limitation, we cannot add a comprehensive figure showing many examples. Alternatively, we added in the main text a pointer to the corresponding figure in the appendix. If the reviewer believes a figure in the main text would be more beneficial, we will add the figure in the revised manuscript.

[1]Zhao Y, Misra I, Krähenbühl P, et al. Learning video representations from large language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6586-6597.

[2]Maluuba. nlg-eval. https://github.com/Maluuba/nlg-eval. Accessed: 2022-06-01.

[3]Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023.

Motion adapter has been explored in prior works. As the reviewer pointed out, the idea of Adapters has been used in many previous works[1]. While this is not an entirely new technique, we believe our Motion Adapter distinguishes itself from other works in the following aspects. 1) Our motion adapter is the first adapter for motion sensing in video-based backbones. 2) The architectural design of our motion adapter is specifically optimized to capture fine-grained hand-object dynamics. Thus, we hope the reviewer could re-consider before making up the final decision.

More Ablation Experiments. We thank the reviewer for the suggestion. Without the motion adapter, our EgoVideo-B and EgoVideo-L models are equivalent to ViT-B and ViT-L, respectively. Since our motion adapter can effeciently leverage more frames, we show the performance of one-branch (ViT-B) trained using both 4 frames and 16 frames. Due to the time and computation cost, we do not train a 16 frame ViT-L. Results can be found in the Table below. Our two-branch full EgoVideo model can outperform one-branch model even trained using 16 frames as inputs. As can be seen in Table 7 of the submission, our 2-branch EgoVideo is much more efficient, while outperforming the 16-frame one-branch counterpart.

Experiments on robot manipulation tasks. In our submission, we tested our EgoVideo on the Franka Kitchen dataset designed for the robotic manipulation task. Results show that while not specifically designed for robot manipulation, our model can outperform specialized robotic representation learning models like MVP[2] and Voltron[3], and is comparable to the SOTA method MPI[4]. More details can be found in Table 10 and Figure 7 in our submission. We appreciate the reviewer's suggestion, and will certainly explore applying the model to additional robot manipulation benchmarks, such as Metaworld and Calvin.

[1]Pan J, Lin Z, Zhu X, et al. St-adapter: Parameter-efficient image-to-video transfer learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 26462-26477.

[2]Radosavovic I, Xiao T, James S, et al. Real-world robot learning with masked visual pre-training[C]//Conference on Robot Learning. PMLR, 2023: 416-426.

[3]Karamcheti S, Nair S, Chen A S, et al. Language-driven representation learning for robotics[J]. arXiv preprint arXiv:2302.12766, 2023.

[4]Zeng J, Bu Q, Wang B, et al. Learning Manipulation by Predicting Interaction[J]. arXiv preprint arXiv:2406.00439, 2024.