EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views

W1. Contributions, in terms of novelty over related work.

We summarize the types of existing methods that estimate interaction regions. A distinct type involves estimating 2D regions [1]. For methods in 3D, the types include: 1) part-level contact [2]; 2) estimate human or object in isolation [3]; 3) estimate regions for a single exocentric image [4]; 4) estimate from the egocentric view, but only focus on hands [5].

Different from them, we propose estimating 3D interaction regions at dense vertex-level for both humans and objects from egocentric videos. Additionally, we extend egocentric estimation to the body. In this case, interacting subjects and objects may disappear from view, the model should address ambiguity between visual observations and interaction contents, which is more challenging.

[1] Detecting human-object contact in images
[2] CHORE: Contact, Human and Object REconstruction from a single RGB image
[3] DECO: Dense estimation of 3D human-scene contact in the wild
[4] LEMON: Learning 3D Human-Object Interaction Relation from 2D images
[5] ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation

W2. Differences with LEMON.

• Task and interaction clues: LEMON estimates interaction regions for an exocentric image, where humans are observed relatively intact, and human mesh recovery (HMR) methods can be employed to obtain human geometries. It leverages clues such as the object and posed human geometries, as well as the interaction content (semantics) in images. EgoChoir estimates the regions for egocentric videos (temporally dense human contact). From this view, appearances alone may not provide effective interaction contexts, and human geometry is also hard to obtain. Thus, EgoChoir combines head movement, visual appearance, and 3D object as clues to model interaction regions. Note: a significant performance gap remains between HMR methods on egocentric and exocentric views, addressing the concern of "why a geometry-semantics approach is not preferred."
• Technical differences: LEMON regards clues (semantics and geometries) as equally important for modeling. Its core technique involves building semantic correlations between geometries using image content as a shared key-value and constructing geometric correlations through a handcraft feature: curvature. In contrast, Egochoir seeks to differentiate the effectiveness of multiple clues (e.g. appearance, motion) based on interactions (hand or body). It achieves this by introducing a gradient modulation mechanism in the parallel cross-attention that correlates these clue features.

Based on responses to W1/W2, we briefly reiterate contributions:

• We first propose estimating interaction regions of the whole human body and objects from egocentric videos.
• Introducing a heuristic framework EgoChoir. Analogous to human behavior, it harmonizes the visual appearance, head motion, and 3D object to model interaction regions and adapt to distinct egocentric interaction scenarios through a gradient modulation mechanism.
• Annotating contact for collected video clips and affordance for 3D objects, which supports the training and evaluation of the task.

W3. The experiments mentioned and critical discussion.

1. Comparisons are predominantly with LEMON: We report quantitative comparison metrics with five baselines in Tab. 1. LEMON performs better than other baselines, so we just chose it for qualitative comparison.
2. Critical discussion missing: In lines 317-321, we critically clarify that the current model may estimate the region slightly before or after the exact frame due to the lack of spatial relation perception. This problem is currently difficult to solve because recovering human meshes from an egocentric view is challenging. So that the spatial relation is hard to capture.
3. Marginal influence of components, leaving out still better than LEMON: In Tab.2, some metrics, e.g. precision and F1, are shown in percentages, the improvement is actually significant. For instance, head motion $\bar{\mathcal{M}}$ increases the precision from 0.68 to 0.78, a 10-percentage point. Plus, without certain key designs, e.g. object concept $F_{a}$, some metrics (F1:0.66, AUC:75.21) are not higher than LEMON (F1:0.67, AUC:75.97). LEMON relies on image content as guidance, while in egocentric videos, the subject and object disappear in many frames. Its framework is hard to address in this case and leads to lower performance. That's why EgoChoir leaves out some designs still better than it.
4. Questions about Fig. 8: The definition of object affordance is key to understanding the case in Fig. 8. Affordance is first defined as "opportunities of interaction" [1]. In the computer vision field, object affordance refers to the functional regions that support certain interactions [2], it contains implicit functional semantics and is not completely equivalent to the contact region. In Fig. 8, videos display cutting with a knife and pouring from a bottle, so affordance regions are the blade and top of the bottle.
5. Only works for standard objects: The 3D objects we collect also include those from scanning and multi-view reconstruction. We present inference results on these noisy 3D objects in the added PDF.
6. Learned prior? Same interaction across videos: EgoChoir actually learns to reasonably adopt appropriate clue features based on distinct egocentric interactions, to capture effective interaction contexts. These clues are complementary, enabling it to handle situations such as appearance variations and missing subjects or objects. We provide qualitative results about the same interaction across videos in the added PDF, and quantitative results are reported in Appendix Tab. 5.

[1] The ecological approach to visual perception
[2] 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding

Dear reviewer:
As we approach the final day of the discussion phase, we hope to know whether our response has addressed your concerns to merit an increase in the rating, or if there are any issues that you would like us to clarify. Thanks for your time and consideration.

W1. The statistical information about the dataset.

We follow the advice and provide statistical information about the dataset in the newly added PDF, including interaction categories distribution of video clips, the distribution of object affordance annotations, and the distribution of contact on different human body parts. We will add this information in the future version.

W2. Modulation tokens that scale features to adapt to different scenarios.

The model is expected to adaptively capture effective interaction contexts from different interaction clues when inputting distinct scenarios (body or hand interaction). Different input scenarios result in distinct distributions of clue features, such as the appearance $F_\mathcal{V}$ and head motion $F_\mathcal{M}$. To achieve adaptation, the model should differentiate which clue feature is effective for the input scenario based on their feature distributions. However, such specific feature distribution is hard to capture. Therefore, we employ learnable tokens to enhance or weaken certain feature dimensions (e.g. for hand interaction, some dimensions in $F_\mathcal{V}$ represent hand features, while for body interaction, certain dimensions in $F_\mathcal{M}$ express significant head translation and rotation), we provide a schematic figure about this in the newly added PDF. By doing so, clues exhibit significant differences in feature distribution across distinct interaction scenarios. It simultaneously adjusts gradients of mapping layers in the parallel cross-attention, making the layers handle the clue feature discrepancies of different input scenarios. During testing, the key to adapting to different scenarios is the learned mapping layers, the key-value input of the parallel cross-attention depends on the combination of input clue features and modulation tokens, these tokens densely scale clue features (at feature dimension) that are extracted from different scenarios, aligning the feature distribution that the mapping layers handled. Thus, these layers differentiate the utilization of multiple clue features (e.g. whether querying from appearance, object concept, or head motion) under distinct scenarios, achieving the adaption.

Q1-3. The latter part of Eq. 4 and the right part of Figure 3.

• The latter part of Eq.4 may indeed cause confusion, we will remove it and integrate the former part of Eq. 4 into Eq. 3 to make it more concise and clearer. Additionally, we will correct the typo you mentioned, thanks again for pointing it out.

• The right of Fig. 3 actually represents two modules $\Theta_{a}$ and $\Theta_{c}$, which model affordance and contact respectively. They have the same model structure but different parameters and inputs. $\Theta_{a}$ corresponds to object interaction concept and $\Theta_{c}$ corresponds to subject interaction intention (middle of Fig. 3). As described in Sec. 3.3, $\Theta_{a}$ takes 3D object feature $F_\mathcal{O}$ as the query, head motion $F_\mathcal{M}$ and appearance $F_\mathcal{V}$ as two parallel key-value pairs to calculate the affordance feature. For $\Theta_{c}$, the $F_\mathcal{V}$ is the query, while affordance feature $F_{a}$ and $F_\mathcal{M}$ are parallel key-value pairs, it calculates the contact feature. Both $\Theta_{a}$ and $\Theta_{c}$ perform the gradient modulation. We will add notations in Fig.3 and more details in Sec. 3.3 to clarify the procedure.

W1. Leverage the existing annotations in EgoExo4D, such as 3D hand pose and scene annotation.

Thanks for the advice. Combining 3D hand and body poses with the scene could indeed improve the annotation accuracy, which enables the contact to be calculated through the spatial distance at first, with only manual refinement needed. However, while building the dataset, we found that not all data in EgoExo-4D have these pose annotations, which hinders us from making a unified annotation workflow. As a result, we ultimately adopt a semi-automated annotation approach. In the future, as the dataset gradually updates the annotations, we will consider scaling up the dataset in this way.

Plus, integrating 3D hand or body poses with the scene can potentially improve the prediction accuracy of the region in temporal. As we mentioned in the limitations (Sec. 5), due to the lack of spatial relation between the subject and object, the current model may estimate the interaction region slightly before or after the exact frame, while 3D poses and scenes can provide this spatial relation, offering a potential solution.

W2. Explanation of the scenes "non-overlapping."

Here, the "scenes" refer to the background. Both the EgoExo-4D and GIMO datasets include the same interaction in different backgrounds (e.g. cut food in different kitchens). We realize that randomly splitting the training and test sets causes interaction clips with the same background to appear in both sets. In this case, the evaluation metrics cannot reflect whether the model is reasoning based on background information or interaction contents. Therefore, we manually split the training and test sets to ensure that clips with the same interaction have different backgrounds in two sets. This partition tests whether the model is inferring based on interactions and its generalization across different backgrounds.

W3. Elaboration on the application examples.

Thanks for the constructive suggestion. We list some potential application scenarios:

1. Interaction Modeling: some studies [1,2] provide datasets to facilitate interaction reconstruction and generation from egocentric videos. These tasks usually encounter problems such as misalignment, floating, and penetration of the interacting subject and object in spatial. Our method provides a spatial surface constraint, such as the subject contact and object affordance. These representations can be extracted from our method as a condition for those modeling methods to improve the authenticity and spatial rationality of the interaction.
2. Embodied AI: learning skills from human demonstrations is an important part of embodied intelligence [3]. A feasible solution to achieve this is to first parse elements related to interactions from the embodied perspective (egocentric) demonstration, including the contact body part and object functional region. Then, retargeting the perceived elements to the agent's own configuration (e.g. dexterous hand) and facing objects. Taking the middle-level spatial perception to drive low-level control signals to complete the interaction.

We will add a section about applications in the future version.

[1] OAKINK2 : A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion
[2] HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction
[3] DexMV: Imitation Learning for Dexterous Manipulation from Human Videos

W1. How accurate is the semi-automated annotation process?

As described in Appendix B.2, during the semi-automated annotation process, we conduct a manual check and refinement after each round of model prediction to ensure the accuracy of contact annotations. This is because the model estimates approximate regions in the initial round but performs limitedly in fine-grained regions. To further validate the accuracy of contact annotations (semi-automated part: Ego-Exo4D), we conduct a cross-dataset validation. Specifically, we train LEMON to estimate contact in two settings using exocentric images:

1. train on 3DIR dataset [1] and test on annotated Ego-Exo4D part
2. train on Ego-Exo4D part and test on 3DIR

We use overlap object categories in the two datasets for testing, and the evaluation metrics of the two groups are shown in the table:

Note: the contact annotations in 3DIR are done entirely manually and are relatively more accurate. Meanwhile, the two groups are close in terms of metrics, confirming the accuracy of semi-automated annotations. This also suggests a potential way to scale up the dataset.
[1] LEMON: Learning 3D Human-Object Interaction Relation from 2D images

W2. It's not clear where the right-hand side of Figure 3 is in the central system diagram.

Sorry for the confusion. Actually, the right of Fig. 3 represents two modules that have the same model structure but different parameters and inputs. We denote them as $\Theta_{a}$ and $\Theta_{c}$. $\Theta_{a}$ is in the "Object Interaction Concept," while $\Theta_{c}$ is in the "Subject Interaction Intention", modeling affordance and contact, respectively. We will add notations of these modules to the central part of Fig. 3.

W3/Q1. The key contribution to the model fusion network. Quantify the importance of each modality to the process.

The core to fuse modality-wise features lies in what information flow is used to combine them and adapt appropriate clue features (e.g. appearance, head motion, or object concept), to model spatial regions for distinct egocentric interaction scenarios (e.g. body or hand). In our fusion model, the object feature $F_\mathcal{O}$ is taken to capture interaction context from the appearance $F_\mathcal{V}$ and head motion $F_\mathcal{M}$, modeling the affordance region. The appearance feature $F_\mathcal{V}$ then queries complementary interaction contexts from the mined object concept (affordance $F_{a}$) and $F_\mathcal{M}$ to model contact region. Parallel cross-attention is a technical mechanism that completes the above process. In addition, unlike employing vanilla cross-attention directly, we introduce gradient modulation in the parallel cross-attention, enabling the model to adapt to various egocentric interaction scenarios.

Regarding the importance of each modality, Tab. 2 in the main paper shows the quantitative impact of different modalities, we copy the modality-related results to the table below. It includes conditions without head motion ($\bar{\mathcal{M}}$) and affordance $F_{a}$ (source from 3D objects). The right part of Tab. 2 shows different encoding methods for the head motion, e.g. randomly initialize motion encoder (ri. $f_\mathcal{M}$), and use divided space-time attention (d. $f_{st}$) for extracting video features, also reflecting the impact of these modalities.