Thank you for recognizing our work and the valuable comments.

W1: Person-related design.

Thanks for your advice. We totally agree with you that person-related designs such as human hand pose prior or object states information would be very useful to improve the performance of Exo2Ego video generation. However, our method is the first work to achieve exocentric-to-egocentric video generation for daily-life real-world skilled human activities. For such challenging daily-life activities in real-world environments such as cooking in the kitchen, playing basketball on the court, the exocentric cameras have to be placed far from the human in order to capture the complete human activities. As a result, it is particularly difficult to capture detailed human hand-object regions on the exocentric cameras. Therefore, it is particularly challenging to obtain human hand pose prior information or object state information from exocentric cameras from preprocessing, and generating human-object interactions is also one of the particular challenges for our Exo2Ego video generation task.

Therefore, our method focuses on the general exocentric to egocentric video generation and also has broad applicability to novel view video synthesis with significant viewpoint variations as you suggested. In addition, our method achieves much better performances on the human-object regions compared to the baselines, as shown in Fig. 3 of rebuttal pdf. Our method can accurately generate the human hands that are operating the CPR on the model, while SVD generates much worse results. We agree that adding more person-related designs can further improve the performance and leave this as faithful future directions.

Q1: Failure case analysis.

Thanks for your advice. We provide two failure cases on Fig. 7 of rebuttal pdf. Yes, most of our failure cases are caused by the fine-grained human-object interactions, due to the significant differences between exocentric and egocentric viewpoints, as well as far-away placed exocentric cameras in the daily-life environments. Although we achieve much better performance than baselines, we believe there is room for improvements especially based on your advice of person-related designs. We leave this as faithful future directions.

Dear Reviewer,

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns? We appreciate your time and input.

Best regards,

Authors of 10425

Thanks for helpful comments.

W1: Differences with ReconFusion.

Our Exo2Ego prior is inspired by ReconFusion and based on PixelNeRF (see L199-L200, L204-L206 of main paper), but it significantly differs from ReconFusion in following aspects.

1. The task of our method is significantly different from ReconFusion. ReconFusion targets 3D static scene reconstruction, and thus it trains a diffusion prior on 3D static datasets and explores how to distill their diffusion prior into a 3D model through regularization losses. In contrast, our method is the first work to approach the challenging exocentric to egocentric dynamic video generation for skilled human activities with significant differences between exocentric and egocentric viewpoints.
2. The design of our Exo2Ego prior is different from ReconFusion’s diffusion prior. ReconFusion directly trains a diffusion prior that includes a PixelNeRF and diffusion UNet, and then explores distilling from the pretrained diffusion prior. In contrast, our method first trains a PixelNeRF-based Exo2Ego prior, and then we integrate this Exo2Ego prior into our complete Exo2Ego video generation pipeline and finetune Exo2Ego prior to provide the coarse yet spatially aligned egocentric features for our egocentric video generation. In addition, the PixelNeRF from the diffusion prior of ReconFusion is a 2-layer MLP, while our Exo2Ego prior is a 6-layer MLP.
3. We further propose the multi-view exocentric encoder to provide fine-grained multi-scale exocentric features for egocentric video generation. Our multi-view exocentric encoder and Exo2Ego prior complement with each other. Our complete Exo2Ego video generation model is the first video-level model for Exo2Ego video generation, and achieves new SOTA.

W2: Ablation on temporal-spatial.

We conduct an ablation to first perform temporal attention and then spatial attention for our model. As shown in the following table, the original spatial-temporal model is slightly better than the temporal-spatial model in terms of PSNR and SSIM, and slightly worse for LPIPS. Since previous video generation methods such as AnimateDiff [16], SVD [5] all follow the spatial-temporal pipeline, our method also follows such spatial-temporal attentions. It is noted that since our temporal layers are finetuned from AnimateDiff [16] that is a spatial-temporal pipeline, our spatial-temporal model performs slightly better than temporal-spatial pipeline. We provide qualitative results on Fig. 4 of rebuttal pdf where ours generates more accurate human-objects.

W3: Reasoning efficiency.

Please refer to common response 2.

W4: Number of exocentric videos.

We conduct additional ablations on the number of exocentric views of 4, 3, 2, 1 views on the following table. Our method’s performance slightly drops with the reduction of exocentric views. We provide qualitative comparisons on Fig. 5 of rebuttal pdf. Even with only one exocentric view, our performance is still much better than other baselines, which significantly demonstrates the effectiveness of our approach with various numbers of exocentric views. This avoids the necessity of capturing time-synchronized exocentric videos at the same time, making our method more applicable for real-world applications.

W5: Evaluation dataset.

We selected Ego-Exo4D dataset [15] as our main evaluation dataset due to its significant challenges of the complexity and diversity of daily-life scenarios, large differences between exo and ego viewpoints, complex human-object interactions, and its largest Ego-Exo data scale with 1286 hours of video. Ego-Exo4D has 8 domains: Cooking, Health, Bike Repair, Music, Basketball, Rock Climbing, Soccer, Dance. It is different from Ego4D and does not contain a wide range of activities. Since our task focuses on Exo2Ego video generation, We select 5 categories that significantly emphasize both egocentric and exocentric activities, such as human-object interactions in both Ego and Exo viewpoints (see L235-L238 of main paper). Others like Dance, Soccer, Rock Climbing are mainly exocentric activities and contain very few human or human-object interactions in the egocentric view, which weakens the usefulness of generating egocentric views and thus are not included in our experiments.

Our method can be generalized to other EgoExo datasets. Please refer to common response 1 for more results on H2O dataset.

W6: Baselines.

Our method is the first work to achieve Exo2Ego video generation for daily-life real-world skilled human activities. This task is particularly challenging and there are no available baselines to compare with. Therefore, we design three baselines including the SOTA open-sourced video generation model SVD, the image generation method SD, and the 3D-based method PixelNeRF. We modify the input and condition modules of these baselines and train and evaluate them on the Ego-Exo4D dataset. We selected SVD because it is the SOTA open-sourced video generation model. Therefore, we believe our baselines are adequate to demonstrate the superiority of our method. We additionally inference two more pre-trained image-to-video and video-to-video generation models on the Ego-Exo4D dataset as shown in Fig. 6 of rebuttal pdf, where these methods totally fail to generate egocentric videos from exocentric videos input.

W7: Writing.

We have corrected such typos.

Evaluation dataset.

We explained in our rebuttal W5 that we selected Ego-Exo4D dataset [15] as our main evaluation dataset due to its significant challenges of the complexity and diversity of daily-life scenarios, large differences between exo and ego viewpoints, complex human-object interactions, and its largest Ego-Exo data scale with 1286 hours of video. We selected the 5 out of 8 categories from the Ego-Exo4D dataset that significantly emphasize both egocentric and exocentric activities, such as human-object interactions in both Ego and Exo viewpoints.

We have demonstrated that our method can be generalized to other EgoExo datasets such as H2O dataset in common response 1, where our method achieves much better performance than SVD, with significant 30.3% improvement over SVD in terms of LPIPS. We are glad that Reviewer ANQv, who suggested this H2O dataset evaluation, has responded that our rebuttal has solved all Reviewer ANQv’s concerns, and we will add these results to provide more insights to readers.

Therefore, we believe our dataset evaluation is very challenging and extensive to evaluate the superiority of our proposed method.

Dear Reviewer qWdm,

Thank you for updating your comment in the Reviews Questions. We are glad that we have addressed some of your concerns. Here we further respond to your remaining concerns about the paper novelty and evaluation dataset.

Paper novelty.

We respectfully disagree with your statement that our paper’s novelty is primarily inspired by ReconFusion. We provided detailed responses about the differences between our Exo2Ego prior with ReconFusion in our rebuttal to address your concern about weakness 1. Here we further clarify the overall novelty and technical contributions of our paper in comparison with ReconFusion.

1. Our method is the first work to achieve exocentric-to-egocentric video generation for daily-life real-world skilled human activities. This task is particularly challenging in terms of the significant differences between the exocentric and egocentric viewpoints, as well as complex human-object interactions and real-world environments. In contrast, ReconFusion focuses on traditional 3D static scene reconstruction. Therefore, our method is fundamentally different from ReconFusion.

2. To address the above challenges of Exo2Ego video generation, we propose a new diffusion-based multi-view exocentric encoder to extract fine-grained multi-scale exocentric features, as well as an Exo2Ego view translation prior to render spatially aligned geocentric features. These two modules complement each other with fine-grained exocentric features and spatially aligned egocentric features to provide conditional information for our egocentric video generation pipeline. Our complete Exo2Ego video generation model is the first video-level model for Exo2Ego video generation, and significantly outperforms previous approaches by a large margin of 35% for LPIPS.

3. Therefore, the technical contributions of our method are three-folds: 1) Diffusion-based multi-view exocentric encoder, 2) Exo2Ego view translation prior, 3) The first Exo2Ego video generation pipeline. We conducted extensive ablation studies to demonstrate the effectiveness of each component in our main paper.

As discussed in our main paper (L199-L200, L204-L206) and rebuttal W1, only the Exo2Ego view translation prior is partially inspired by ReconFusion — both our Exo2Ego prior and ReconFusion are based on PixelNeRF, but our Exo2Ego view translation prior is significantly different from ReconFusion. Furthermore, our proposed first exocentric-to-egocentric video generation model, our multi-view exocentric encoder, and our challenging Exo2Ego video generation task definition are all fundamentally different from ReconFusion. Below we further discuss the detailed technical differences between our Exo2Ego translation prior with ReconFusion.

1. The design and functionality of our Exo2Ego prior is different from ReconFusion’s diffusion prior. Our method first trains a PixelNeRF-based Exo2Ego prior, and then we integrate this Exo2Ego prior into our complete Exo2Ego video generation pipeline and finetune Exo2Ego prior to provide the coarse yet spatially aligned egocentric features for our egocentric video generation. In contrast, ReconFusion directly trains a diffusion prior that includes a PixelNeRF and diffusion UNet, and then explores distilling from the pretrained diffusion prior.

2. Our Exo2Ego prior is specially designed for rendering egocentric views from exocentric inputs. To achieve this, we differentiate the egocentric views and exocentric views for our Exo2Ego prior model – it only extracts exocentric views’ ResNet features as condition signals to render the pixel colors and features. Our exocentric and egocentric viewpoints are sparse and significantly different from each other. In contrast, ReconFusion does not differentiate between views and is trained on dense multi-view 3D static scenes datasets that contain hundreds of views for each 3D static scene.

3. Our Exo2Ego prior’s model architecture is different from ReconFusion. We employ a 6-layer MLP as our Exo2Ego prior base model to handle the complex Exo2Ego translation with large Exo-Ego viewpoint differences, while ReconFusion only adopts a 2-layer MLP.

4. Our Exo2Ego prior’s training strategy is different from ReconFusion. We first pretrain our Exo2Ego prior on the Ego-Exo4D dataset. Then, we integrate our Exo2Ego prior into our Exo2Ego video generation pipeline and we alternatively finetune Exo2Ego prior and the complete pipeline during training. This strategy ensures that Exo2Ego prior renders spatially-aligned feature maps necessary for egocentric video generation, as well as keeps the original scene reconstruction capability.

Dear Reviewer,

Thank you once again for your feedback. We provided additional responses towards your remaining concerns. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns? We appreciate your time and input.

Best regards,

Authors of 10425

Thank you for recognizing our work and the valuable comments.

W1: Ablation on Exocentric CLIP features.

Thanks for your advice. We conduct additional ablation study by replacing our exocentric feature encoders with the CLIP exocentric features. As shown in the following table, our method achieves much better performance compared to using CLIP features across all metrics on the unseen action and unseen take evaluations. This clearly demonstrates the superiority of our multiview exocentric encoder in maintaining fine-grained details compared to the CLIP features. We also provide qualitative ablation comparison on Fig. 4 of rebuttal pdf, where our method with Exo encoder outperforms the one with CLIP features on many detailed regions such as left and right human arm and objects. We will add this ablation result to our final paper.

W2: Only 4 exocentric views are very challenging.

The 4 sparse views and significant differences between exocentric and egocentric cameras are exactly the key challenges for our Exo2Ego video generation task. Therefore, previous 3D reconstruction methods such as PixelNeRF fail in this task, as shown in the baseline PixelNeRF results in Fig. 3 of main paper. To tackle these challenges, we propose our complete Exo2Ego-V pipeline with both multi-view exocentric encoder to extract multi-scale exocentric features and Exo2Ego translation prior to render coarse yet spatially aligned egocentric feature maps. These two designs complement each other: the exocentric features are fine-grained but not spatially aligned with the ego input, while the egocentric features are a bit coarse but they are spatially aligned with the ego input. Our experiments demonstrate that our Exo2Ego-V pipeline achieves much better performance than previous reconstruction and generation baselines for such a challenging task with large Exo-Ego viewpoint differences.

W3: Visualization of rendered egocentric features.

As shown in Fig. 6 of appendix, our Exo2Ego prior rendered both the egocentric features and images. We want to clarify that we utilize two types of egocentric features: the egocentric features rendered directly from the Exo2Ego prior, and the CLIP features of the egocentric images rendered from the Exo2Ego prior. The first one is spatially aligned with the egocentric input latent and is directly concatenated with the input to provide spatial guidance, and the latter is utilized as the cross attention computation to maintain the original diffusion architecture.

Following your suggestino, we provide more visualizations of rendered egocentric images and rendered egocentric features on Fig. 2 of rebuttal pdf. As shown in the figure, the rendered feature maps can accurately encode the geometric information such as the desktop shape, and the CPR models. Our rendered features are spatially aligned with the egocentric inputs and are complemented with the fine-grained exocentric features extracted from our exocentric encoder. Although the rendered feature maps could be coarse for complex scenarios, but they are spatially aligned egocentric features (as mentioned in L150-L151 of main paper) can provide spatial guidance for our egocentric generation pipeline, and our ablation studies also demonstrate the effectiveness of our proposed Exo2Ego prior.

We conducted an ablation study on our proposed Exo2Ego translation prior in Tab. 2 and Fig. 5 of the main paper, where removing Exo2Ego prior results in worse performance for PSNR and SSIM. In addition, through the visualization of Fig. 5, removing Exo2Ego prior (second row, second column) results in the missing of right human arm. These ablations demonstrate the effectiveness of our Exo2Ego prior.

Q1: SVD training.

For fair comparisons, we provide the same 4 exocentric videos as inputs for our method and all baseline methods including SVD. Therefore, we modify the condition blocks of SVD to take 4 exocentric videos as input and finetune the pretrained SVD model on the Ego-Exo4D dataset. As such, all methods are given the same exocentric videos as inputs to generate corresponding egocentric videos, and they are all trained and tested on the same dataset to achieve fair comparisons among each other.

Q2: Generalization on other EgoExo datasets.

Thanks for your advice! Our method can be generalized to other EgoExo datasets. Following your suggestion, we additionally train and evaluate our method on the H2O dataset, and compare our method with the SOTA baseline - SVD on the following table. As shown in the following table, our method achieves much better performance than SVD, with significant 30.3% improvement over SVD in terms of LPIPS. We provide qualitative comparisons of our method against SVD on Fig. 1 of rebuttal pdf, where our method achieves much more photorealistic and accurate results than SVD, such as the details of human hands, interacted objects, and the environments. These can demonstrate our method can be generalized to other datasets, and we will add this result on the final paper.

Dear Reviewer ANQv,

Thanks for your valuable comments! We are glad that our rebuttal solves all your concerns! We will add these additional experiments to the final paper to provide more insights to readers!

Regarding your additional question, “Given the GT ego frame/rendered feature map/rendered rgb image, is it possible to map one pixel (e.g. center of the table) back to each of the four exocentric frames given the ego and exo camera poses?”

Yes, it is possible to map one pixel from the ego frame back to each of the four exocentric frames given the ego and exo camera poses. To achieve that, we also require egocentric depth information $\mathbf{D}$ which can be conveniently obtained by additionally rendering the sampled points depths with their density values $\sigma$ along each ray using our Exo2Ego prior.

Specifically, we can first backproject one egocentric pixel $\left(u, v\right)$ to the 3D space, denoted as $\mathbf{X}$, using egocentric cameras pose (to provide projection direction) together with the egocentric depth at that pixel (to provide the projection distance) as follows,

$\mathbf{X}=E_{\mathrm{ego}}^{-1}\cdot K_{\mathrm{ego}}^{-1}\cdot\mathbf{D}\left[u, v\right]\cdot\left[u, v, 1\right]^{\mathrm{T}}$,

where $E_{\mathrm{ego}}$ and $K_{\mathrm{ego}}$ are the extrinsics and intrinsics of egocentric camera, and $\mathbf{D}$ is the depth map for egocentric frame.

After that, we can project $\mathbf{X}$ to each of the four exocentric frames using the exocentric camera poses $E_{\mathrm{exo}}$ and $K_{\mathrm{exo}}$.

$\left(u_{\mathrm{exo}}, v_{\mathrm{exo}}, 1\right)=K_{\mathrm{exo}}\cdot E_{\mathrm{exo}}\cdot\mathbf{X}$,

where $\left(u_{\mathrm{exo}}, v_{\mathrm{exo}}\right)$ are the projected exocentric pixels.

As you suggested, this could provide more fine-grained understandings of how each pixel in the rendered feature map/rgb image leverages the pixel-wise exocentric information from each exo camera.

Thank you very much for recognizing our work and valuable comments!

Best regards,

Authors of 10425

Thank you for recognizing our work and the valuable comments.

W1.1: Network architecture.

The base network architecture of the exocentric encoder and the spatial modules of the egocentric video diffusion model are extensions of the Stable Diffusion [44]. Therefore, they consist of four downsample layers, one middle layer, and four upsample layers, where each layer includes 2D convolution, self-attention, and cross-attention. Since Stable Diffusion operates in the latent space, the feature dimension of input encoded noisy latents is $\mathbb{R}^{N\cdot F\times C\times\frac{H}{8}\times\frac{W}{8}}$.

The exocentric attention features’ dimensions are different for different downsample, middle, and upsample layers with channel dimension varying at [320, 640, 1280], and the spatial dimensions $\frac{H}{8}\times\frac{W}{8}$ are downsampled by 2x2 or upsampled by 2x2 for each downsample and upsample layers, respectively. Eq. (1) and (4) are the denoising processes of diffusion models that predict the added noise of input noisy latents, except that in Eq. (1) we extract the exocentric features from each denoising layer. We will add these details to the main paper.

W1.2: Appendix results.

Thanks for your advice. We will move the unseen scenes results to the main paper and move some qualitative results to the appendix.

W1.3, Q1: Exo2Ego prior training.

We only train one single generalizable Exo2Ego prior model for all exocentric videos and egocentric videos from the training set. For each training iteration, we randomly extract synchronized 4 exocentric frames and 1 egocentric frame to supervise our Exo2Ego prior (L197-L200). During the complete training, each iteration will load different Exo-Ego videos from the training set to train our single generalizable Exo2Ego prior. Therefore, we only need to train one single Exo2Ego prior from all videos from the training set.

During testing, our Exo2Ego translation prior can generalize to test scenes that are not seen during training without retraining. Therefore, we do not need to train the Exo2Ego translation prior for novel test scenes. As shown in our experiments, we evaluate our Exo2Ego-V on unseen actions, unseen takes, and unseen scenes without retraining our Exo2Ego prior on any of the test sets, which is very challenging. We will explain it clearer in our final paper.

W2: Unseen scenes results.

The evaluation of the unseen scenes is particularly challenging for two reasons. 1) We only train one single Exo2Ego-V model, including the Exo2Ego prior, from the training dataset and evaluate its performance on the proposed 3 challenging test setups without any retraining or finetuning. 2) The number of different scenes from Ego-Exo4D is relatively limited, but the geographic diversity of the captured scenes is particularly significant across the world. Therefore, all methods perform worse on the unseen scene evaluation.

Despite these challenges, our method still outperforms the baseline methods in most metrics for the unseen scene evaluation. As shown in Tab. 3 of the main paper, PixelNeRF only achieves the best PSNR metric for the Covid Test category, but it generates very blur results as shown in Fig. 7 since PSNR favors blurry images [37]. SVD occasionally achieves better metrics results but it only generates wrong scenes without any human-object interactions, as shown in Fig. 7 of main paper. In comparison, our method is the only method that tries to generate both scenes and human-object interactions, such as the background and human arms and basketball in Fig. 7. Although the performance on unseen scenes is not yet optimal, we believe jointly training our method with a more diverse multi-view dataset has the potential to improve performance and we leave this as future work.

Q2: Exocentric feature dimension.

The exocentric attention features’ dimensions are different for different downsample, middle, and upsample layers with channel dimension varying at [320, 640, 1280], and the spatial dimensions $\frac{H}{8}\times\frac{W}{8}$ are downsampled by 2x2 or upsampled by 2x2 for downsample and upsample layers.

As explained in L183-L187, to encode the relative camera poses information, we do not directly concatenate the camera poses with exocentric features. Instead, we first encode the camera pose to 1280 dimension with a 2-layer MLP, and then add it to the denoising timestep embedding as the residuals for the noisy latents in the following denoising blocks. In these following denoising blocks, there are additional MLP layers to map the channel dimension of the timestep and camera embedding to the same dimension as the corresponding noisy latents, so that these embeddings are added to the noisy latents as residuals.

Q3: Egocentric camera poses.

Yes, we directly use the camera poses provided by the Ego-Exo4D dataset.

Q4: EgoGT Results.

Thanks for your advice, we will remove the repeated EgoGT which was intended for symmetrical layout.

Q5: Performance over time.

Following your suggestion, we compute the PSNR for each frame of the generated videos over time, and then average these metrics across all test videos for each time step respectively. We evaluate this on the three test sets of the Cooking category and compute all metrics. We report PSNR due to space limits, and SSIM and LPIPS show similar results. As shown in following table, the performance remains relatively stable over time, and the intermediate frames perform slightly better than the final frames due to larger motion in the latter.

Tab. 1 Quantitative metrics (PSNR) over time for Cooking category.

L1: Inference time.

Please refer to common response 2.

Dear Reviewer,

Thank you once again for your feedback. As the rebuttal period is nearing its conclusion, could you kindly review our response to ensure it addresses your concerns? We appreciate your time and input.

Best regards,

Authors of 10425

Dear authors,

Thanks for your hard work and detailed responses.

Dear reviewers,

the authors have responded to your questions and comments, please read them and further provide feedback: Whether your concerns addressed? After reading all the reviews and responses, are there more questions about the clarity, contributions, results, etc?

Thanks!

Best, Your AC