EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT

We thank the reviewer for insightful questions and feedback. Due to the word limit, we will simplify the description in the rebuttal.

Details of the QA Generation Process

1.We first use a Dynamic Interaction Filtering Process. Due to the word limit, please refer to Supplementary A for details.

2.After filtering, we construct the QA data. In Section 3.1.2, we provide a detailed description of the formatting and procedures for different data splits. We believe your primary concern is that the selection of clips is not clear, particularly for long-term questions. We add additional details below.

Our source long videos contain hundreds of clips. For generating long-term and CoT QA pairs, we employ the following protocol:

(1) We sequentially scan each video's clips, randomly selecting 30-120s segments (30% for 30-60s, 70% for 60-120s).

(2) We concatenate all clips within each selected window, discarding segments with <5 clips (due to insufficient captions).

After compiling long-term QA clips, we remove duplicates from the short-term subset to create our final clip collection. We will add the complete detail to the revision.

Details of Our Benchmarks

1. EK-Visor: This dataset provides hand-object segmentation annotations, ensuring the quality of bounding boxes. It covers over 200 representative scenes with diverse hand-object interactions, and it is the most widely used dataset for egocentric spatial grounding. It addresses a critical gap in standard benchmarks (e.g., RefCOCO for images, NExT-GQA[1] for videos), which lack egocentric interaction data.

2. EgoExoLearn: There are very few benchmarks for egocentric temporal grounding. Current benchmarks, such as Ego4D-NLQ[2], require locating a temporal window from a natural language query. However, these queries often suffer from poor quality and inconsistent lengths. EgoExoLearn contains the largest number of high-quality temporal boundary annotations. Its annotations are generated by trained human annotators, ensuring the data is high-quality, unbiased, and representative. Furthermore, the dataset provides timestamped annotations at coarse and fine-grained levels, making it ideally suited for building a temporal grounding benchmark to evaluate a model's ability to localize egocentric events.

CoT annotations has no specific design tailored to the characteristics of egocentric videos.

For the selection of clips for our CoTs, we utilized captions from the Ego-HOD[3] dataset. These captions augment high-level semantic information with the hand and object interactions and movements. This ensures that our source information is inherently rich with unique egocentric cues.

Building on this, our CoT generation process is context-aware. It is guided by temporally localized questions grounded in the egocentric clips, ensuring that the reasoning traces are linked to egocentric cues such as hand-object interaction, temporal progression, and egocentric intent. As a result, our CoT annotations contain information for both fine-grained and multi-horizon reasoning, which is crucial for egocentric videos.

Relationship Between Grounding Data and Egocentric Video Understanding

We thank the reviewer for the suggestion and conducted further analysis on these benchmarks. We found that benchmarks such as EgoTaskQA, QAEgo4D, EgoSchema, and EgoPlan focus on high-level understanding, planning, and reasoning. They only require the identification of simple objects and do not involve fine-grained interaction details. Our baseline model, Qwen2-VL, already possesses a certain level of capability, which explains why our grounding data did not yield significant improvements.

Previous works[3,4,5] demonstrated that using hand and object as supervisory signals enhances egocentric video understanding abilities. Therefore, we selected three more fine-grained benchmarks, ERQA, EgoMCQ and EGTEA for evaluation.

As shown in the table, the inclusion of grounding data significantly improved model performance across all three benchmarks. These datasets involve tasks directly related to our hand-object grounding data, such as understanding hand/robotic arm motion, object identification, and object classification.

To provide a more intuitive demonstration of the grounding data's effectiveness, we present inference examples below:

(1) Q:How do you need to rotate the dumbbell for it to fit back in the weight holder?

A:'To determine how to rotate the dumbbell for it to fit back in the weight holder, we first need to understand the orientation of the dumbbell and the weight holder. Grounding of the Key Object: The dumbbell is the key object here. It is shown lying on its side with the handle pointing upwards. The weight holder is positioned above the dumbbell, with its base facing downwards.. Current Position: The dumbbell is currently lying horizontally on the floor, with its handle pointing upwards. Thus, you need to rotate dumbbell counter-clockwise 90 degrees and move it above to fit back in the weight holder.

(2) Q:The right hand holding a blue object is in motion. What is it nearest to?

A:The gripper is nearest to the stove top burner. The key objects: left hand, right hand,the stove top burner, and the blue object. The right hand is described as holding the blue object, which is located near the stove top burner. Since the right hand is moving towards the stove top burner, it must be closest to that area. During the process, the left hand does not move. Therefore, the right hand is nearest to the stove top burner.

These examples demonstrate that accurate grounding of objects and hands and reasoning about their relationships is essential for deriving correct answers. As discussed in our paper, current MLLMs exhibit limited capability in distinguishing hands, underscoring the critical role of precise visual grounding in such tasks.

In summary, grounding data provides a more pronounced benefit in egocentric scenarios that require fine-grained understanding, such as object classification, action recognition, and embodied reasoning in robot manipulation.

The Contribution of Our Training Paradigm

We thank the reviewer for highlighting these recent works in RFT and prompting us to clarify our contributions to the training paradigm.

Our primary goal with RFT was to enhance two crucial skills for egocentric reasoning without degrading the general reasoning abilities. Unlike previous works which mainly optimize for a single task, our framework simultaneously optimizes for spatial and temporal grounding. Our investigation into this multi-task challenge revealed a critical challenge: mixed training caused unstable updates and slow convergence, requiring over 3k steps.

We address this through proposing a simple yet effective single-task batching strategy, where each training batch contains data from only one grounding task, alternated throughout training. This approach stabilized the learning process, accelerating convergence to 1k steps and leading to improved final performance on both grounding benchmarks.

This insight offers practical value for multi-task RFT applications, and we will ensure it is included in the final version to provide a more complete description of our methodology. We agree that further exploration into multi-task RL optimization is a promising direction for future work.

We appreciate the reviewer pointing us to UniVG-R1 and Vad-R1. We would also like to respectfully note that these papers are concurrent work with our submission, as both were released publicly as pre-prints very recently. Therefore, while they represent exciting and parallel advancements in the field, our work was developed independently and addresses the distinct challenge of multi-task egocentric grounding.

[Limitations] Zero-shot evaluation

We thank the reviewer for this important feedback. For EgoTimeQA, we identify that it may share identical information with QAEgo4D, indicating potential data contamination issues. To ensure evaluation fairness, we excluded EgoTimeQA and conducted retraining.

As the table shows, removing EgoTimeQA resulted in a 1.7% drop on QAEgo4D, confirming shared QA patterns. In zero-shot setting, our model still outperforms the baseline by a significant margin of 4.2%, demonstrating its robust reasoning capabilities.

For the evaluation protocol, we conduct zero-shot experiments in MCQ format for QA benchmarks. Regarding spatial-temporal grounding tasks, evaluation is performed after RFT on a small training set. Due to the lack of suitable grounding benchmarks in the egocentric domain, we validate our approach through grounding-relevant benchmarks including ERQA, QAEgo4D, and EgoPlan(Table 2 in the paper). Additionally, we provide results between SFT and RFT in temporal grounding to demonstrate the effectiveness of our RFT paradigm.

[1] Xiao et al, Can i trust your answer? visually grounded video question answering.

[2] Ramakrishnan et al, Naq: Leveraging narrations as queries to supervise episodic memory.

[3] Pei et al, Modeling fine-grained hand-object dynamics for egocentric video representation learning.

[4] Zhang et al, Helping hands: An object-aware ego-centric video recognition model.

[5] Xu et al, Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions.

We thank the reviewer for insightful questions and feedback. We believe that we can address your concerns in the rebuttal.

[Weaknesses] Contributions of Our Dataset.

While our dataset, EgoRe-5M, shares some similarities with existing datasets, our core contributions are distinct and address key limitations in the field.

First, we address the challenge of limited video sources in the egocentric domain. For instance, RoboBrain's data is exclusively sourced from the Open-X-Embodiment[1] dataset, while VidEgoThink's video content is derived entirely from the GoalStep[2] subset of Ego4D. This limited video sources limit the model's generalization capabilities. In contrast, our work introduces a pipeline to curate data from large-scale web data, significantly broadening the scope of training data.

Second, our dataset encompasses a broader spectrum of question types and achieves data scaling. While RoboBrain focuses on action planning and VidEgoThink includes four dimensions of questions, EgoRe-5M further incorporates complex reasoning and temporal grounding tasks. Furthermore, we have achieved a significant scale-up in data volume (5k in VidEgoThink vs 5M in our case), which is crucial for training more powerful egocentric understanding models.

[Questions] Adapting RFT for Other Tasks.

We agree that adapting RFT to other tasks is a feasible direction. The critical element lies in the design of the reward function, which must be tailored to the specific task objectives. For example, for tasks such as long-horizon planning or semantic inference, we can employ a large language model (e.g., GPT-4o) to score the generated outputs with the ground truth, using these scores as the reward signal. For goal-oriented QA, the task could be framed as a classification problem to supervise the model's selection of the correct target. These are relatively intuitive approaches. We believe that the design of more sophisticated reward functions, as well as the exploration of joint multi-task RFT, are valuable and promising avenues for future research.

[Limitations] Data Quality Assurance.

To ensure the quality and reliability of our dataset, we conducted several steps to quantify:

1.CLIP-based Alignment Quality Analysis. We analyze the alignment quality between the caption and the video via the CLIP score. Due to the large scale of the dataset, we sampled 100K instances to compute the score. Specifically, we chose ViCLIP[3] pre-trained on a large amount of video data, and compared it against the original concatenated captions as a baseline. We found that the score of our CoTs is 71.4, while the original caption score is 74.7.

While the score of our data is lower than the descriptive caption baseline, this is an expected outcome. The source captions describe concrete, visible objects and actions, which aligns perfectly with CLIP's training objective. In contrast, our data contains reasoning steps, causal links, and inferred goals (e.g., “to align the dumbbell, a rotation is needed”), which do not always have a direct visual correlate. A score of 71.4 still indicates strong semantic alignment, confirming that our annotations are highly relevant to the video's content.

2.Hallucination detection. To validate whether EgoThinker exhibits hallucinations, we conducted evaluations on two widely-adopted benchmarks: (1) VideoHallucer[4] for assessing large video-language models, which contains object-relation, temporal, semantic detail and extrinsic hallucination detection. (2) POPE (Polling-based Object Probing)[5] for object hallucination detection.

The experimental results demonstrate a marginal performance decrease (0.6%) on VideoHallucer, while achieving a significant improvement of 3.2% on the POPE benchmark. We attribute this differential performance to our model's enhanced object grounding capability, which particularly enhances its robustness against object hallucination.

3.Manual Review. We conducted extensive manual verification on the most complex Chain-of-Thought (CoT) data split to validate our data quality. Based on the length of the generated CoT, we selected 5k samples from our CoT data split for manual review. The goal of this scoring was to assess whether the generated caption contained any non-existent objects or action descriptions when compared to the video. We set up a 4-point scale to categorize hallucination severity: Severe Hallucination, Moderate Hallucination, Partial Hallucination and No Hallucination. We get 4068, 577, 279, 76 for score = 3,2,1,0. We observe that hallucinations primarily manifest as the model outputting non-existent goals or actions based on the provided captions, with hardly any instances of non-existent objects. Furthermore, the majority of captions exhibit no hallucinations or only minor ones, indicating good alignment quality in our CoT generation.

If you have any additional questions or suggestions, please feel free to tell us.

[1] Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration.

[2] Toward hierarchical understanding of procedural activities.

[3] Internvid: A large-scale video-text dataset for multimodal understanding and generation.

[4] Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models.

[5] Evaluating object hallucination in large vision-language models.

Due to the word limit, we will simplify the description in the rebuttal.

[Q1,W1] CoT data justification and egocentric reasoning cases.

We thank the reviewer for this insightful question. For cot data justification, We analyze from two perspectives:

1.Additional Benchmarks: We test EgoThinker on 2 additional benchmarks designed to measure complex reasoning, not just perception: ERQA[1] and VidEgoThink[2].

ERQA, proposed by Gemini Robotics, has various reasoning types such as spatial, trajectory and action reasoning. As shown in the table, our model's performance is comparable to Gemini 1.5, surpassing the baseline by 4.8%. For tasks requiring the model to infer whether an action was completed or understand long-term information (VidEgoThinker and QAEgo4D), EgoThinker's performance on both benchmarks shows our model's superior performance in reasoning tasks.

2.Examples of Egocentric Reasoning: The following examples, derived from our dataset, illustrate why a direct-answer approach is insufficient for these query types. As we are unable to upload images, so we only present QA examples.

(1) Task Planning: Naturally, solving this problem requires breaking down tasks and providing a reasoning path based on logical relationships.

【Q】: How to add mortar mix and cement into the concrete mixer?

【A】: Pick up a sack of mortar mix with left hand while using right hand to hold tools. Then open the sack by tearing the seal with left hand while the right hand steadies the bag. Afterward, pour both the mortar mix and cement into the concrete mixer.

【Why CoT is necessary】: Without reasoning, the outputs is "You should pick up mortar mix and cement, and pour them into the concrete mixer." This is clearly insufficient since it lacks the actionable, sequential details.

(2) Complex Event & State Inference: This task requires combining multiple implicit cues for accurate judgment; relying solely on direct observation without reasoning can lead to incorrect answers.

【Q】: Did the person complete the vegetable washing process?

【A】: The man put the vegetables into the basin and turned on the faucet, but then the camera moved elsewhere. However, when the camera returned to the basin, the man had turned off the faucet and removed his hands from the basin. Therefore, it can be inferred that he completed washing the vegetables.

【Why CoT is necessary】:In the scenario, we can't directly determine if the task is complete. However, by reasoning through the man's actions, camera movements, and the logical sequence, we can arrive at the answer.

3.Embodied reasoning: This requires combining spatial understanding with the context of an action.

【Q】:How do you need to rotate the dumbbell for it to fit back in the weight holder?

【A】:'To determine how to rotate the dumbbell for it to fit back in the weight holder, we first need to understand the orientation of the dumbbell and the weight holder.\n\n1. Grounding of the Key Object: The dumbbell is the key object here. It is shown lying on its side with the handle pointing upwards. The weight holder is positioned above the dumbbell, with its base facing downwards.. Current Position: The dumbbell is currently lying horizontally on the floor, with its handle pointing upwards. The weight holder is positioned above it, ready to receive the dumbbell. Thus you need to rotate dumbbell counter-clockwise 90 degrees and move it above to fit back in the weight holder.

【Why CoT is necessary】The answer depends on a multi-step mental process: (1) Identify the spatial information of the dumbbell and the target holder. (2) Calculate the necessary transformation. A direct perceptual guess is likely to fail.

Furthermore, in scenarios involving camera transitions, spatial understanding, and motion understanding, integrating these pieces of information for reasoning is vital. However, most existing egocentric benchmarks focus on understanding and simple reasoning. Developing more complex egocentric reasoning benchmarks is a promising direction for future exploration.

Regarding whether CoT is always required, the answer is no. As described in the paper, we first assess if a video segment was complex enough to warrant a CoT. For simpler queries, it is not generated or enforced.

[Q3,Q5,W2] Experiments on hallucination detection.

We thank the reviewer for raising this critical point regarding the risk of hallucination. Since the CoT rationales are generated from text captions without direct video access, ensuring factual alignment is paramount. We acknowledge this challenge.

We've researched Hallucination detection and found that most benchmarks to test models for hallucinations rather than verifying if captions contain hallucinations. Methods like VQ2A-Hallucination or Image-CoT-Faithfulness aren't open-source. We also tried using the latest FClip, and ALOHa for verification, but these methods primarily evaluate image hallucination, and weren't suitable for our complex semantic and temporal information. Therefore, we conduct a 4-part audit of our EgoRe-5M CoT data to quantify its factual fidelity.

1.Annotation Process. As detailed in the paper, the CoT data was generated by DeepSeek-R1, a model with strong instruction-following capabilities. Our prompt explicitly constrained the model to generate rationales only from the information present in the source video caption, strictly forbidding the introduction of external or imagined details. This largely guarantees that the generated CoT won't contain hallucination.

2.CLIP-based Alignment Quality Analysis. We analyze the alignment quality between the caption and the video via the CLIP score. Specifically, we chose ViCLIP[3] pre-trained on a large amount of video data, and compared it against the original concatenated captions as a baseline. We found that the score of our CoTs is 62.3, while the original caption score is 70.6.

While the CoT score is lower than the descriptive caption baseline, this is an expected outcome. The source captions describe concrete, visible objects and actions, which aligns perfectly with CLIP's training objective. In contrast, our CoT rationales articulate abstract reasoning steps, causal links, and inferred goals (e.g., “to align the dumbbell, a rotation is needed”), which do not always have a direct visual correlate. A score of 62.3 still indicates strong semantic alignment, confirming that the rationales are highly relevant to the video's content.

3.Manual Review. Based on the length of the generated CoT, we selected 5k samples from our CoT data split for manual review. The goal of this scoring was to assess whether the generated caption contained any non-existent objects or action descriptions when compared to the video. We set up a 4-point scale to categorize hallucination severity: Severe Hallucination, Moderate Hallucination, Partial Hallucination and No Hallucination. We get 4068, 577, 279, 76 for score = 3,2,1,0. We observe that hallucinations primarily manifest as the model outputting non-existent goals or actions based on the provided captions, with hardly any instances of non-existent objects. Furthermore, the majority of captions exhibit no hallucinations or only minor ones, indicating good alignment quality in our CoT generation.

4.To validate whether EgoThinker exhibits hallucinations, we conducted evaluations on two widely-adopted benchmarks: (1) VideoHallucer[4] for assessing large video-language models, which contains object-relation, temporal, semantic detail and extrinsic hallucination detection. (2) POPE (Polling-based Object Probing)[5] for object hallucination detection.

The experimental results demonstrate a marginal performance decrease (0.6%) on VideoHallucer, while achieving a significant improvement of 3.2% on the POPE benchmark. We attribute this differential performance to our model's enhanced object grounding capability, which particularly enhances its robustness against object hallucination.

[Q2] Experiments on another series model

We finetune the more powerful video MLLM InternVideo2.5-8B. The table below shows our fine-tuned results across various datasets. From the table, we can see that our dataset shows minor improvements on EgoTaskQA and QAEgo4D. We believe this is because InternVideo2.5-8B inherently possesses strong video understanding capabilities. However, on EgoSchema and ERQA, which are more dependent on reasoning and are more complex, InternVideo2.5 achieved more significant improvements.

Due to resource and time limitations, we haven't yet completed experiments with larger models, such as the 13B-level model. We will supplement these experiments in the future and further analyze the role of reasoning data.

[Q4] Grounding Data with Egocentric Video Understanding

Due to word limit, please refer to the Relationship Between Grounding Data and Egocentric Video Understanding section in our rebuttal for Reviewer k3DC(last).

[1] Gemini robotics: Bringing ai into the physical world.

[2] Videgothink: Assessing egocentric video understanding capabilities for embodied ai.

[3] Internvid: A large-scale video-text dataset for multimodal understanding and generation.

[4] Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models.

[5] Evaluating object hallucination in large vision-language models.

Dear Reviewer,

We sincerely thank you for your positive feedback to our work.

Our work is differentiated from ST-Think by two fundamental aspects: our data and our methodology.

1.Data field. Our work introduces EgoRe-5M, a large-scale, richly annotated dataset created specifically to train models for egocentric reasoning. In contrast, ST-Think's principal contribution is a benchmark designed to evaluate the reasoning capabilities of existing models. Our efforts to build this training dataset include two novel contributions. First, we developed a novel data sourcing pipeline to automatically filter and curate egocentric video clips from large-scale web data. This directly addresses the critical scarcity of egocentric video sources for training. Second, we have several QA annotation types with a comprehensive annotation scheme to specifically enhance understanding, grounding, and reasoning abilities within egocentric contexts.

2.Method. While both studies use a two-stage training framework, our implementations and objectives at each stage are substantially different. SFT Stage: We strategically integrate our specialized EgoRe-5M data with general-purpose data. This approach is explicitly designed to enhance the model's expert egocentric reasoning skills while simultaneously preserving its broad, general-purpose capabilities. RFT Stage: Our reward functions target different objectives. ST-Think insightfully uses AI-generated scores to refine the model's general reasoning process. In contrast, our RFT stage is tailored to a core challenge in egocentric vision: understanding hand-object interactions. We formulate a reward signal that leverages limited grounding data to specifically sharpen the model's comprehension of these critical interactions, a focus essential for egocentric scenarios.

In summary, while ST-Think provides a valuable benchmark for evaluation, our work introduces the data, resources, and a tailored methodology for training robust egocentric models. We agree that ST-Think offers several insightful ideas, and we will add a detailed discussion and citation in our revised manuscript to properly contextualize our contributions. Thank you again for your constructive feedback. We trust this clarifies the unique aspects of our work and addresses your concerns.

We appreciate the reviewer pointing us to ST-Think, and we would like to respectfully note that this is a concurrent work with our submission, as it was released publicly as pre-prints very recently. Therefore, while it tackles a very similar problem in the field, our work was developed independently and made independent contributions in both data and method for egocentric reasoning. If you have any additional questions or suggestions, please feel free to tell us.