EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos

Q3.Detail on how the HOI mask prediction model handles the temporal dynamics and variability.

A3. The key module in our cross-view mask prediction model, i.e., ego-exo memory attention, is designed to model such temporal dynamics and variability in hand-object interactions.

Specifically, we use Eq.(2) in our manuscript to illustrate the process:

Thank you for recognising our work and the valuable comments. We hope that the following point-to-point responses address your concerns, and that you could increase the rating accordingly.

Q1.Key factors that influenced the architectural design of the X-Gen model.

A1.1 Key factors of the HOI-aware video diffusion model. The considered task in this paper is cross-view video prediction, which requires the video prediction model to generate egocentric video that

• aligns with the environmental context given by the first frame condition and maintains the temporal consistency of the generated video. This is modeled by temporal attention layers.
• follows the given text instruction, which is modeled by cross attention layers.
• aligns with the hand-object motion presented in exocentric video. The translated ego HOI mask serves as additional input condition to the video diffusion model.

Ablation experiments in Table 3 in our manuscript validate the effectiveness of each modality in generating reliable egocentric videos.

A1.2. Key factors of the cross-view mask prediction model. The cross-view mask prediction model is constructed to translate the hand-object motion from exo view to ego view. The key contribution is the ego-exo memory attention block, which aims to

• model the fine-grained, temporal relationship of ego/exo videos
• leverage exocentric hand-object clues to infer egocentric hand-object mask features

Ablation experiments in Table 4 in our manuscript show the importance of ego-exo memory attention in improving both segmentation and generation.

A1.3. The integration of two models. In the X-Gen model, the core question connecting two models is: "What information from the exocentric perspective is critical in assisting egocentric video prediction?" In this work, we choose to model hand-object dynamics, as hands and interactive objects in ego view reflect the user's intentions and characterize ongoing actions. Hand-object interactions have also received significant attention in previous egocentric research studies, including pose estimation [1], action recognition [2], interaction anticipation [3], hand motion trajectory prediction [4].

Ablation experiments in Tables 2 and 5 in our manuscript reveal that

• both hand and objects clues are critical in assisting egocentric video prediction.
• HOI masks serve as a better option in bridging exocentric and egocentric views.

[1] Kwon, Taein, et al. "H2o: Two hands manipulating objects for first person interaction recognition." CVPR 2021.

[2] Damen, Dima, et al. "Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100." IJCV 2022.

[3] Grauman, Kristen, et al. "Ego4d: Around the world in 3,000 hours of egocentric video." CVPR 2022.

[4] Zhan, Xinyu, et al. "OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion." CVPR 2024.

Q2.Failure cases of X-Gen.

A2.1 Complex hand-object movement. In the third case (last row) of Figure 5 in our manuscript, we provide the visualization results of a failure case where complex hand movement is involved in the action, i.e., C adds the cut tomato to the bowl of salad mixture with his left hand. Complex hand-object motion poses additional challenges for the cross-view mask prediction model to make reliable HOI masks.

A2.2 Camera movement. Another challenge is the potential camera movement in the video. For example, some actions involve head movements, such as retrieving an avocado from a refrigerator behind. These scenarios often result in rapid scene transitions and potential video blurriness, making it challenging for generative models to accurately predict motion. Naive solutions would include:

• filtering videos containing rapid camera movement by estimating the optical flow [1] of the videos.
• controling the video prediction model with camera poses [2].

We will add analyses on such failure cases in our revised manuscript.

[1] Teed, Zachary, and Jia Deng. "Raft: Recurrent all-pairs field transforms for optical flow." ECCV 2020.

[2] He, Hao, et al. "Cameractrl: Enabling camera control for text-to-video generation." Arxiv 2024.

Q6. Details about the pipeline at the inference time? / Iterative processing of two components.

A6.1.Inference pipeline of the cross-view mask prediction model. At inference time, the inputs to the cross-view mask prediction model include

• the exocentric video $\mathcal{V} _ {exo}$ with corresponding exocentric HOI mask $\mathcal{M}_{exo}$.
• the egocentric video $\mathcal{V}_{ego}$ (with $2^{nd}$ frame to $N^{th}$ frame set to zero image)

The model predicts the HOI mask of all egocentric frames $\hat{\mathcal{M}}_{ego}$ all at once. In the memory attention block, Eq.(2) remains the same. Here, the key is defined as:

where $\overline{g}_1$ represents the feature of the visible $1^{st}$ ego frame, and $\overline{g} _ 2,...,\overline{g} _ {n-1}$ refer to the zero image feature. Hence, the model is required to additionally leverage useful exocentric clues to make predictions.

A6.2.Inference pipeline of the video prediction model. The models takes as input:

• The $1^{st}$ egocentric frame $g_1$
• The predicted mask sequence of the ego video $\hat{\mathcal{M}}_{ego}$
• Text instruction $\mathcal{T}$

and outputs the predicted ego RGB frames $\{\hat{g}_1,...,\hat{g}_N\}$ all at once.

A6.3.Iterative processing of mask segmentation and video generation. Thanks for your suggestion. This is a very good choice to improve the quality of both the ego-mask and the ego-video. Currently, we do not take the predicted ego frames back to the segmentation model because the base video diffusion model (i.e. SEINE) is designed to predict all frames (16 frames in our case) all at once by leveraging temporal attention. Iterative processing would involve further investigation on the architecture and perhaps re-training of the video diffusion model, and it is challenging to re-implement this during the rebuttal period. We consider this as a valuable future direction to improve our work.

Q7.Details about the temporal attention blocks.

A7. The temporal attention in the video diffusion model takes as input the hidden video features $F\in\mathbb{R}^{b\times {hw}\times f\times d}$ where $b$, $hw$ and $f$, $d$ refers to the batch size, spatial resolution, number of frames and hidden dimension, respectively. $F$ is then reshaped to $(b\times hw) \times f\times d$. A standard attention mechanism is then operated over the temporal dimension, i.e., modeling the temporal relationship among frames. The output feature has the same dimension as the input feature.

Q8.Explanation on HOI masks extracted from the future video frames.

A8. Yes, we choose GT HOI masks to guarantee the mask quality for merely evaluating the video prediction model.

Q4.Need Justification that using HOI masks is a better option.

A4. In egocentric videos, the primary motion is centered around hand-object interactions, which play a crucial role in tasks such as pose estimation [6], action recognition [7], interaction anticipation [8], hand motion trajectory prediction [9].

In addition, in the Ego-Exo4D dataset, each ego camera is paired with at least four exo cameras, ensuring that hand-object interactions are generally visible in both the ego and at least one exo perspective. Other objects, however, may not be visible in the ego/exo view.

In comparison, we adopt SAM-2 to segment and track every possible object without considering their classes, and train the mask encoder using this all-object masks. Note that we do not train a cross-view mask prediction with this data as the correspondence between exo-objects and ego-objects is not available due to the class-agnostic nature of SAM-2. Therefore, at inference time, the all-object mask is only available for the first frame. As observed in the below Table, using all-object masks does not lead to better performance, primarily because it is limited to masks from only the first frame, which fail to provide accurate motion guidance. Our model benefits from cross-view mask prediction, enabling the prediction of HOI masks for all frames. In future work, we aim to extend this approach to include a broader range of objects.

[6] Kwon, Taein, et al. "H2o: Two hands manipulating objects for first person interaction recognition." CVPR 2021.

[7] Damen, Dima, et al. "Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100." IJCV 2022.

[8] Grauman, Kristen, et al. "Ego4d: Around the world in 3,000 hours of egocentric video." CVPR 2022.

[9] Zhan, Xinyu, et al. "OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion." CVPR 2024.

Q5. How alpha was annealed during training.

A5. Recall that our feature $Z''=\alpha*Z'+(1-\alpha)*Z$, where $Z'$ and $Z$ denotes ego and exo features, respectively. At training stage, we adopt a step decay mechansim for $\alpha$ representated as:

where $t$ and $T$ refers to the current iteration and total training iteration, respectively. Such strategy helps the model training in the early stage, and eventually learns to predict the mask features for all egocentric frames at inference time.

Table 4 in our manuscript compares our approach with methods that use (1) ego feature only (i.e. $\alpha$=1.0) and (2) exo feature only (i.e. $\alpha$=0.0), here, we also add a comparison that adopts cosine decay for $\alpha$. The results show that cosine decay performs worse than step decay, possibly because introducing exo features too early during the training phase might interfere with the model's fundamental segmentation ability. We will add this result on the final paper.

Q2. The dynamics the model is learning / the performance of only using the HOI condition of first Ego video frame / using the first Ego video frame directly as the predictions

A2.1 HOI mask of the first frame. We include the additional results of only using the HOI mask of the first frame. Our approach achieves superior performance than only using the HOI mask of the first frame, indicating the HOI masks in subsequent frames also provide valuable structual guidence.

A2.2 Using first ego frame as the predictions. We replace the predicted first frame with the ground-truth frame, and we observe no significant performance change. This indicates that the diffusion model effectively leverages the first-frame condition by predicting plausible first frame that closely aligns with the ground truth.

A2.3 How much dynamics the model is learning. To answer this question, we first analyse the dynamics within the dataset. Specifically, for each video clip, we uniformly sample 16 frames and calculate the Intersection over Union (IoU) between the hands and interactive objects in the first frame and subsequent frames. We select frames 2, 4, 8, 12, and 16 for this analysis. Results in the table below show that the IoU between the first frame and subsequent frames gradually decreases, indicating the spatial dynamic changes of hand and interactive objects.

Next, we compare the model's performance on videos with different dynamics. In particular, we calculate the averaged optical flow of the videos in the validation set using RAFT [4], and split the videos into two sets using a threshold, i.e., small-flow (SF) set and large-flow (LF) set. Compared to the baseline model, our model achieves more significant improvements on the large-flow set, demonstrating that our HOI condition effectively helps the model generate videos with dynamics.

[4] Teed, Zachary, and Jia Deng. "Raft: Recurrent all-pairs field transforms for optical flow." ECCV 2020.

Q3. Using cross-view tranformation [5] as a baseline.

A3. This work [5] focuses on the autonomous driving domain, which presents a significant gap compared to our scenario involving indoor exo-ego setups with a focus on human activity. In [5], the cross-view transformation module predicts the vehicle occupancy in the bird's-eye view (BEV) from the front-view monocular image. Despite the domain gap, we implement this approach on our exo-ego data. Here, we adopt two variants with different inputs:

• Variant-A: The input is the exocentric RGB frame and the output is the corresponding ego HOI mask;
• Variant-B: The input is the channel-wise concatenation of the exo frame and exo HOI mask and the output remains the ego HOI mask.

We train the model for 30 epochs with a batch size of 64 on 1*A100 GPU. We take the centre frame of the video clip/HOI video mask as the input/output. The results listed below reveal that this approach is not suitable for our cross-view mask prediction task. We hypothesize two reasons for this discrepancy:

• The model is image-based and lacks specific design for temporal modeling, whereas our approach incorporates temporal memory attention.
• [5] primarily focuses on the translation of single category (e.g., vehicles), while our setting involves open-world interactive objects. In our scenario, the model is required to infer ego HOI masks from exo HOI masks, posing additional challenges. Notably, we conduct an experiment by introducing exo HOI masks (Variant-B), which exhibits improvement over Variant-A. However, it still falls short compared to our method.

[5] Yang et al., Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation, CVPR 2021.

Thank you for recognising our work and the valuable comments. We hope that the following point-to-point responses address your concerns, and that you could increase the rating accordingly.

Q1. Using off-the-self HOI detectors or Exo-Ego video frames without Exo HOI mask.

A1.1 Off-the-shelf HOI detectors. We agree that using off-the-shelf HOI detectors (e.g. EgoHOS+SAM2) for producing HOI masks is a good option while training the HOI-aware video diffusion model, and this is exactly our training scheme to ensure the mask quality (Line 254-255). However, off-the-shelf HOI detectors cannot work at inference time as only the first ego-frame is visible, meaning that only the HOI mask of the first frame can be obtained. We compare X-Gen with the method that only adopts the HOI mask of the first frame in the Table below. Using only the HOI mask from the first frame as a condition improves SSIM and PSNR compared to the baseline (No mask). However, it does not lead to improvements in LPIPS and FVD. In contrast, our proposed cross-view mask predictions achieve improvements across all metrics.

A1.2 Exo-Ego frames without HOI mask. In Table 5 in our manuscript, we compare different design choices of incorporating exocentric information. Among the choices, one approach is replacing the predicted Ego HOI masks with Exo RGB frames, without introducing HOI masks in the entire process. The sub-optimal performance of Exo RGB frame condition indicates the difficulty of learning exo-ego translation and video prediction in a single video diffusion model. In contrast, HOI masks in the ego-view provide explicit pixel-aligned visual clues to improve the video prediction. We will implement more baseline approaches (e.g. Pix2PixHD[1], Vid2Vid[2], Pix2Pix-Turbo[3]) for comprehensive evaulation.

[1] Wang, Ting-Chun, et al. "High-resolution image synthesis and semantic manipulation with conditional gans." CVPR 2018.

[2] Wang, Ting-Chun, et al. "Video-to-video synthesis." ArXiv 2018.

[3] Parmar, Gaurav, et al. "One-step image translation with text-to-image models." Arxiv 2024.

Thank the authors for providing more details and ablations. The rebuttal addressed my concerns.

Q3. Performance on unpaired exo-ego videos and the usecase of paired data.

A3. This paper considers the translation from exocentric view to egocentric view, and thus the correspondence between exo-view and ego-view is required. This setting is useful in Embodied AI. For instance, a robot watches the the human demonstration of conducting an activity (e.g. cutting vegetables or washing dishes), it should map the exocentric demonstration to the egocentric view to learn and replicate the task in the same environment. The ability of AI assistants to provide visual instructions by matching third-person observations of fine-grained information from instructional videos to those in the user's first-person view is also underlined in Ego-Exo4D [2].

To see how our approach performs in the case of unpaired ego-exo data, we evaluate the cross-view mask prediction model by choosing an unpaired exocentric video during inference, which leads to poor HOI segmentation performance as listed in the Table below. This highlights the alignment between different views.

[1]. Chen, Xinyuan, et al. "Seine: Short-to-long video diffusion model for generative transition and prediction." ICLR 2023.

[2]. Grauman, Kristen, et al. "Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives." CVPR 2024.

Thank you for recognising our work and the valuable comments. We hope that the following point-to-point responses address your concerns, and that you could increase the rating accordingly.

Q1. Improvement over ConsistI2V and the adoption of HOI condition in X-Gen

A1.1 The adoption of HOI condition.

X-Gen is proposed to introduce exocentric videos to provide structural guidance for generating reliable hand-object interaction motions in video prediction tasks, whereas relying solely on the first frame and text as guidance lacks the crucial video-level motion cues necessary for generating realistic predictions. The effectiveness of HOI condition is validated by ablation experiments in Table 3 in our manuscript, and the conclusions include:

• ID1 vs. ID2, ID3 vs. ID4, introducing HOI condition significantly boost the generation performance w/ or w/o text instructions.
• ID2 vs. ID3, structural HOI control of the generation appears more effective than text instructions, as the hand-object movement can also serve as a valuable cue for inferring the current action semantics.

A1.2 The quantitative comparison between X-Gen and ConsistI2V.

Note that, the design of our diffusion model follows SEINE[1], serving as an important baseline in our experiments. The improvement of X-Gen over SEINE is shown in Tables 1 and 5 in our manuscript:

Regarding the comparison to ConsistI2V, despite a minor improvment in SSIM (0.532 vs. 0.537), X-Gen significantly outperforms ConsistI2V on perceptual metrics, i.e. LPIPS (0.351 vs. 0.311) and FVD (1109.314 vs. 1031.693).

To further validate the effectiveness of our model, we replace the base architecture of the diffusion model with ConsistI2V and fine-tuned the base model for 10 epochs. Subsequently, an HOI mask encoder is trained based on the fine-tuned ConsistI2V model. During inference, we also feed the cross-view mask predictions into the generative model as the HOI condition.

As shown in the Table below, introducing HOI to the ConsistI2V model consistently improves performance across all metrics. This demonstrates that our approach generalizes well to different diffusion models, enhancing their ability to generate reliable HOI motions.

Q2. Adding the details on computational costs.

A2. We provide additional details on the training and inference costs of the cross-view mask prediction model (Seg) and video prediction model (Gen). Regarding the Gen model, we list the computational costs of the backbone (stage-1) and the mask encoder (stage-2). During training, we use 8 A100 GPUs with a batch size of 32. For inference, we employ a single A100 GPU and evaluate the model with a batch size of 1.

The rebuttal has addressed the questions I have. I will increase my rating.