EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining

We greatly appreciate the time and effort you invested in providing these detailed observations and questions. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.

W1 & Q1: The paper lacks sufficient downstream task evaluations.

Thank you for your valuable feedback. First, we sincerely apologize if the formatting caused you to miss our EgoNLQ experiments in Table 4. We also include robot manipulation experiments in Appendix Table 1. Furthermore, we have added new experiments on EgoMQ, as presented below:

Besides, we add ablation on action recognition task on Epic-Kitchens-100, i.e., EK100CLS below:

In our updated version, we will revise the layout to present EgoNLQ and EgoMQ side by side in two columns.

W2 & Q3：This paper should provide more experiments related to 3D scene understanding.

This work currently focuses on 3D-awareness centered around human actions from an egocentric view instead of indoor scene understanding, as illustrated in Figure 3. Although our model is capable of providing some depth understanding of surrounding objects / humans (as illustrated in the first Ego4D example in Figure3), it is overall constrained within the scope of egocentric hand-object interactions.

We argue that the benefits of 3D-awareness can be reflected through improved performance on general downstream tasks like short video understanding on EK100MIR, EgoMCQ, EK100CLS [added during rebuttal period], long video understanding on EgoNLQ, EgoMQ [added during rebuttal period], and 3D-aware tasks on H2O, FrankaKitchen [in supplementary]. We plan to further explore more general-purpose 3D-aware visual encoders in future work.

Q2: How are the captions generated using LLMs? How do you handle hallucinations?

We carefully prompt an LLM to understand the change of HOI masks to ensure faithfulness compared with original captions. To prove the faithfulness, we evaluate the enriched captions and original captions by calculating the BERTScore[ICLR'20], the results are presented below, which indicates the good quality of our captions.

Regarding the hallucination issue, our perspective is as follows:

• First, through careful prompt engineering and manual verification, we found that the LLM can effectively capture changes in bounding boxes and generate corresponding descriptive modifications, while largely preserving the original semantic content.
• Second, although some hallucinations are present, training with these captions consistently improves performance across most downstream benchmarks, as shown in our ablation studies. This suggests that the benefits outweigh the potential drawbacks.
• Finally, prior work such as LLaVA [NeurIPS’23] has demonstrated that LLMs are capable of understanding bounding box information and producing fine-grained descriptions, which leads to better generalization. Inspired by this, we believe it is feasible to generate high-quality captions by prompting LLMs to understand HOI mask information. Moreover, we intuitively believe that generating captions using video-based MLLMs (e.g., Gemini-2.5-Pro, QwenVL2.5, InternVL3) is more prone to hallucination compared to our proposed pipeline.

Thanks for the authors' detailed explanation. Most of my concerns have been addressed. I decide to raise my rating to borderline accept.

In addtion, I strongly encourage the authors to include these details in the paper, especially the 3D awareness of current benchmarks. In my understanding, the 3D awareness is directly reflected by the spatial-aware tasks like depth estimation on H2O, not by videos understanding tasks. This is because these video benchmarks could be improved due to other factors, e.g., better understanding of objects and contexts. Furthermore, including other details can improve the quality of the paper.

We greatly appreciate the time and effort you invested in providing these detailed observations and questions. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.

W1 & Q1：The role of the 3D-aware decoder is unclear.

$L\_{\rm{depth}}$ refers to the combination of the 3D-aware decoder and the associated depth loss ($L\_{\rm{depth}}$); both components are either jointly added or removed. Therefore, the ablation of $L\_{\rm{depth}}$ illustrates the effectiveness of the 3D-aware video pretraining process.

During rebuttal, we sincerely apologize that we might not be able to conduct more pretraining experiments, but In the revised version, we promise to include additional ablations by replacing the 3D-aware decoder with alternatives such as a lightweight MLP decoder or a ConvNet decoder to further validate the effectiveness of our 3D-aware decoder.

Nonetheless, In this work, our primary contribution is the introduction of 3D-awareness into EgoVLP, demonstrating its general applicability across a wide range of tasks and achieving performance that surpasses previous methods.

W2 & Q2: Potential Unfair Comparison of Using Depth Anything as Teacher Model

We first provide a comparison of different methods' pretraining process as follows:

• First, all methods are evaluated fairly during inference. We ensure fair comparisons by using the same original video-text pairs and maintaining comparable model sizes. However, it is both common and inevitable for different approaches to incorporate distinct foundation models for pseudo-label generation during pretraining. For example, LaViLa utilizes T5-Large for text rewriting, while HelpingHands employs an HOI detector to obtain HOI bounding boxes.
• Moreover, different methods aim to prove the effectiveness of different types of supervision signals, as illustrated in Figure 1 of our paper, often relying on powerful teacher models tailored to their objectives. For instance, LaViLa and AVION leverage a multimodal large language model and LLM to synthesize ten times more text data, i.e., 35M text data for supervision, while we use only 6M text data in total for pretraining. HelpingHands[ICCV'23] and HENASY[NeurIPS'24] not only generates HOI boxes using an HOI detector but also adopts HOI detection as an auxiliary pretraining task.
• Finally, our objective is not to distill knowledge from a specific foundation model, but rather to learn from noisy 3D supervision signals directly. As such, knowledge distillation is not the focus of this work. Depth supervision derived from any depth estimation foundation model, DepthPro [ICLR'25], can serve the same purpose. But we admit that there exists bias from the DepthAnything model. In future work, we plan to include additional experiments using depth maps generated by DepthPro to further validate our approach.

W3 & Q3: Lack of ablation studies for depth estimation on H2O.

Thank you very much for your suggestion. Following your advice, we have conducted ablation studies on almost all downstream tasks including H2O. The results are as follows:

As shown, incorporating our components consistently improves the scale-aware RMSE metric, indicating that our method enhances the understanding of the true depth. Notably, when comparing row 2 and row 4, where enriched text is added, we observe an increase in the scale-invariant RMSE, which indicates impairing the understanding of relative depth structures. Generally, the two components proposed in this work generally lead to improvements on the majority of 2D vision tasks, indirectly demonstrating the effectiveness of 3D-awareness.

We greatly appreciate your thoughtful assessment and comments on our paper. Below, we address the concerns in detail:

W1: The performance improvements across benchmarks seem marginal.

We argue that our improvements are non-trivial. As shown in the ablation study in Table 3(a), EgoDTM achieves a +3.4 mAP gain over the vanilla baseline on the EK-100-MIR benchmark, improving from 29.7 to 33.1. Furthermore, on the depth estimation task, our model achieves a 9.8% improvement in Scale-aware RMSE over the best-performing baseline (LaViLa), as shown in Table 2. We enumerate the additional supervision signals employed by the baselines and present comparisons to demonstrate that the improvements achieved by our method are substantial rather than modest.

Q1: How long does it take to generate the training data? And how do you consider the scalability of the approach?

• At inference time, depth map generation takes approximately 0.45 seconds per 4-frame 720p video, using 4 GB of GPU memory. Generating HOI bounding boxes and masks takes around 4.96 seconds per 4-frame 720p input, with a memory footprint of 3.8 GB. Therefore, generating 2.5 million training samples requires approximately 159 A100 GPU hours for HOI bounding box and mask generation, and 15 A100 GPU hours for depth map generation. For LLM text generation, we spent $418 USD and approximately 8 hours of API usage to generate 2.5 million captions.

• From a scalability perspective, we believe this is a highly cost-effective and easily extensible paradigm. The cost of pseudo-label generation is relatively low, and our 3D-aware decoder is lightweight. We have already demonstrated the feasibility of this approach on the million-sample scale, and scaling it to tens of millions is within reach.

Q2: Common failure cases of EgoDTM.

Based on our observations, EgoDTM achieves notable improvements over the base model in action recognition and is better able to attend to HOI-relevant regions in complex scenes. Nonetheless, it still suffers from a relatively high error rate in challenging scenarios, such as those involving extreme viewpoints or occlusions of human-object interactions. We plan to include the corresponding qualitative examples in the appendix of the revised version.

Thanks for the effort! I will keep my positive rating.

We greatly appreciate the time and effort you invested in providing these detailed observations and questions. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.

W1：The proxy task of depth estimation may not lead the model to truly understand 3D information. And downstream tasks do not include such true 3D tasks.

• On the choice of proxy task: We acknowledge that depth estimation may not fully capture all aspects of true 3D understanding. Richer 3D supervision (such as from multi-view geometry or dynamic 3D correspondences in videos) could provide stronger inductive signals. However, our work represents an initial attempt to incorporate 3D supervision into large-scale egocentric video-language pretraining. Introducing 3D signals and training procedures—whether through data collection or model design—is highly non-trivial at the million-scale pretraining level. Despite these challenges, our results show that depth estimation serves as a practical and effective proxy, enabling the encoder to become more 3D-aware and motivating further research in this direction.
• On downstream task evaluation: We agree that evaluating a visual encoder's physical 3D reasoning capability is essential. However, we argue that there are currently few suitable benchmarks to directly evaluate the encoder's capability. Instead, we choose to demonstrate that 3D-aware encoders have potential for solving 2D perception tasks like short video understanding on EK100MIR, EgoMCQ, EK100CLS [added during rebuttal period, see response to W3 below], long video understanding on EgoNLQ, EgoMQ [action detection task, added during rebuttal period, see table below], and 3D-aware tasks on H2O, FrankaKitchen [in supplementary].

W2：Is the performance gain in the downstream retrieval task mainly attributed to longer captions?.

No.

• First, for downstream vision-language tasks such as retrieval, we only utilize the encoder during inference. As shown in Figure 2, no caption augmentation is performed at inference time. Therefore, the improved retrieval performance reflects the model’s enhanced video-text alignment, which is acquired during training with our enriched captions.
• Furthermore, we highlight that both baseline methods, LaViLa and AVION, leverage LLMs and MLLMs to expand and rewrite the original captions, resulting in 35 million enriched descriptive captions. In contrast, we only have 2.5 million newly generated captions informed by HOI masks. Despite using significantly fewer enriched captions, our method outperforms both baselines.
• To follow your advice, we promise to conduct a pretraining ablation as follows: generate enriched longer and descriptive captions using LLMs without spatial information, to eliminate the impact of spatial-awareness in texts. Due to space and resource constraints, we have not included such pretraining ablation in this version, but we promise to add them to our revised version.

W3: Do you have experimental support for robot manipulation?

Yes, we apologize for the confusion. In Appendix Table 1, we present results from robot manipulation experiments conducted in the Franka environment. EgoDTM outperforms baselines including CLIP, ConvNext and LaViLa, demonstrating the potential value of EgoDTM encoder in robotic applications.Some beneficial results are presented below:

W4: Could you conduct ablation studies on more tasks?

Yes. We have conducted additional ablation studies on almost all tasks to further validate the effectiveness of our model. During rebuttal, we add experiments including action recognition task on Epic-Kitchens-100 (EK-100-CLS) and action detection task on Ego4D (EgoMQ). The results are presented below:

As shown in the table, progressively adding our modules leads to consistent improvements across the majority of tasks, and our model outperforms all baselines in the main experiments. One potential limitation is that depth pretraining may negatively impact the performance on EgoNLQ to some extent. Nevertheless, our proposed AugVTC module is able to mitigate this effect and yields improvements that surpass those achieved by Avion, LaViLa, and the EgoVLP encoder as shown in Table4 of our main paper. Besides, comparing row 2 and row 4 when adding $\mathcal{L}\_{\rm{augvtc}}$, we observe an increase in the scale-invariant RMSE and a decrease in the scale-aware RMSE. We hypothesize that this may result from the proxy task bias introduced by multi-task pretraining.

Q: The paper is not written in the full 9 pages.

Thanks for pointing this out. In the current submission, we aimed to balance content between the main paper and the supplementary material and some detailed results and analyses were moved to the supplementary to avoid exceeding the page limit. However, this led to underutilization of the main paper space. In the revised version, we will reorganize the content to make full use of the 9-page limit, including adding the comprehensive ablation study across downstream tasks as well as results on the EK-100-CLS and EgoMQ benchmark.