EgoSim: Egocentric Exploration in Virtual Worlds with Multi-modal Conditioning

Thank you for your insightful feedback and valuable suggestions.

Imbalanced data: In Appendix A1.1, we mention that although the RealEstate dataset contains more videos, the video durations in the Epic-Field dataset are significantly longer. In fact, the total amount of usable training data, measured in the number of frames, is in the range of 10 million for both datasets. Therefore, the number of training samples that can be constructed from each dataset is comparable in scale. When training the version of our model using both datasets, we sampled training clips at a 50:50 ratio from each dataset.

ViewCrafter: ViewCrafter is indeed a highly relevant and very impressive work, and we will cite it in the related work section. However, directly comparing the preciseness of camera controls is somewhat unfair and infeasible for now for the following reasons:

1. ViewCrafter utilizes an additional pretrained dense stereo model DUST3R to estimate the point cloud. As such, its 3D world modeling is essentially a reflection of the dense stereo model's capabilities, rather than being extracted from video pretraining.
2. The camera trajectories used in the two approaches are different. Our method assumes an ego-centric exploration trajectory, whereas ViewCrafter follows an object-centric trajectory. As a result, the camera poses in these two works are defined using different coordinate systems and scales. Since their current open-source code does not cover how they handle camera poses from RealEstate, it is currently difficult to directly convert camera poses from one to another, making effective comparison infeasible. We attempted to directly use the c2w matrix from Viewcrafter; however, we found that despite both methods using the same c2w matrix as input, the resulting camera motions were unmatched. This confirms that the two approaches rely on different camera coordinate systems.
3. Although RealEstate may seem to feature only static scenes, as mentioned in lines 92–96, our approach enables the imagination of unobserved parts using supplementary text prompts—an ability that ViewCrafter lacks. This highlights that our task differs slightly from traditional novel view synthesis. We can illustrate this point with the following example. Case 1 was generated by ViewCrafter. Case 2 and Case 3 were generated by EgoSim. In Case 2, we specified that the model should generate plants on the left side, while in Case 3, we specified the generation of a floor-to-ceiling window. As a comparison, in Case 1, ViewCrafter cannot create meaningful content on the left side. Note that the camera trajectories used on these two methods are not aligned.
4. Finally, ViewCrafter was uploaded to arXiv in September, and according to the updated ICLR reviewer guidelines, comparisons are only required with works available before July 1.

Input trajectory: The input camera trajectory is egocentric rather than object-centric, and we actually showcase some generated videos with complex and extended trajectories on the project page.

1. Longer Sequence: We will first explain how to generate longer videos. Specifically, this is achieved by taking the last frame of a generated video as the first frame for the next generation step. By repeating this process, longer videos can be produced. Interactive demo video 10-1, 3-4 are examples of this case.
2. Complex Trajectory: We also provide examples of complex and unordered trajectories. For instance, in interactive demo videos 5-3 and 5-4, complex trajectories are showcased. The camera trajectory in 5-3 follows an upward, rightward spiral, while in 5-4, it moves forward in a zigzag pattern.

Artifacts: Thank you for pointing out the limitations of the current model. This aspect of generation capability is primarily constrained by the generative power and precision of the pretrained weights we used. We believe that when our method is applied to more powerful video pretraining models, it will achieve more impressive and more consistent results.

Thank you for your insightful feedback and valuable suggestions.

Clarity: Improving the paper's readability is certainly an important aspect we aimed to address. The missing values in the table were a result of combining 4 different comparison scenarios into a single large table to save space, which inadvertently caused comprehension challenges. In the revised manuscript we have submitted, we have addressed this issue by separating the data into four individual tables. We hope this modification resolves any potential misunderstandings. The result of MotionCtrl was actually already included in the ablation study serving as a starting point. We have added these results back to Tables 3 and 4.

EPIC-Fields: We didn’t evaluate T2V and I2V settings on EPIC-Fields because the primary focus of this paper is on the image + text-to-video(T+I2V) scenario. However, there is no issue with including these comparisons. The results for T2V and I2V on EPIC-Fields are provided in the tables below. EgoSim outperforms all baselines on all metrics, demonstrating the superior performance of SEAL.

CameraCtrl + SparseCtrl: Yes, you are correct. We mentioned this in lines 426–429, where we stated that we directly used the publicly available code and weights for CameraCtrl + SparseCtrl. However, it failed to generate reasonable video results. As a result, the COLMAP optimization does not converge. You can view some visual examples 1 and 2 here to observe the failure. In these GIFs, the upper part shows videos generated by CameraCtrl + SparseCtrl, while the lower part represents the ground truth.

Efficiency:

1. Sparsity: Introducing the epipolar attention mask effectively enforces sparsity, reducing unnecessary computation and storage, thereby significantly lowering GPU memory consumption. Quantitatively, with a batch size of 2 on each A100, the memory consumption without the sparsity-optimized algorithm is 78GB, whereas with the optimization, it is reduced to 42GB.
2. Pixel unshuffle: This statement is based on a comparison with CameraCtrl, which employs a CNN encoder to downsample the Plücker embedding. However, we observe that the choice of a CNN encoder has no significant impact on the results. Consequently, a simpler pixel shuffle operation proves to be sufficient. Since the additional computation from CNNs is removed, it naturally saves a small portion of computation.

Values in Eq.4: Thank you for providing such helpful suggestions! We plotted the distribution of sigmoid values from different layers, and as you pointed out, they are indeed primarily concentrated around 0 or 1. This confirms the effectiveness of the CI2V-Adapter module.

Details:

1. Data preprocessing: These details are covered in Appendix A1.1. For Epic-Field, we use 611 scenes for training and 88 scenes for testing. We found that a large number of videos in Epic-Field suffer from significant motion blur due to rapid movement, which is detrimental to model learning. Therefore, we applied NAFNet to perform motion deblurring on all frames. During the training, we use a sample stride of 4 for Epic-Field.
2. Trajectory: These details are covered in Appendix A1.3. We sorted the translation and rotation of the real camera poses from two datasets, and then randomly selected poses from the top 20% to 40% and 40% to 60% ranges to classify them into hard and medium groups, respectively.
3. Training: In the case of Epic-Field, we re-trained all baselines.

Clarification: We are happy to provide further clarifications as needed. Overall, our analyses are entirely based on the failure cases we observed in baseline models. The design improvements in EgoSim were also derived from these observations and analyses.

Starting with the camera control component, mainstream methods like CameraCtrl and MotionCtrl typically integrate camera information directly into the intermediate features. In this setup, the model does learn the correlation between camera motion and video generation. However, the model at the same time learns correlations between other conditions and video generation as well. For instance, when the input is an image of a corridor as the initial frame, the training data related to corridors only involve moving forward or backward. This creates a correlation between the corridor and forward/backward motion. When these conditions are fused together through addition, the model finds it challenging to discern which correlation should take the dominant role.

The failure cases we generated further validate this claim: videos generated by baseline methods struggle to produce out-of-distribution camera movements. When other correlations conflict with camera motion control, we naturally want camera motion to take the highest priority and override the other correlations. To achieve this, we need a control mechanism that is not entangled with the intermediate features but operates independently of them.

Meanwhile, the dynamic factors, initially governed by text and image conditions, become unstable with the inclusion of camera information. This instability often manifests as abnormal dynamics, most notably a sudden and abrupt transition from the initial frame to a text-driven, imaginary scenario.

To address this issue, it is essential to integrate camera information into the dynamical control modules of video generation. Regarding the integration approach, we can observe that as the camera moves, part of the scene can find reference from the initial frame, while the rest corresponds to unobserved regions. These unobserved areas can be extrapolated based on previously seen content or supplemented with additional structure provided by the text.

For example, consider a scenario where the camera rotates 30° to the left and reveals a bed. The right side of the future video frame remains consistent with what was previously observed in the first frame, while the left side is generated under the guidance of the text, which specifies the bed. This shows that different regions are either explained by the context frame (the first frame) or by the text. The combination of camera motion and the model's understanding of 3D depth in the scene effectively informs the model about which parts are explained by the context and which are explained by the text. Therefore to implement this, we introduce a sigmoid function to control this process, enabling a smooth and dynamic fusion of context and text-driven generation.

Thank you for the constructive comments and valuable feedback. We sincerely appreciate the reviewers for pointing out that there is room for improvement in the readability and clarity of our paper.

Clarity: We begin by providing a detailed overview of our experimental design. The experiments are organized into three distinct scenarios: (1) camera + text-to-video (Section 3.2.1), (2) camera + image-to-video (Section 3.2.2), and (3) camera + image + text-to-video (Sections 3.2.3 and 3.3). The first two scenarios were chosen to align with the specific capabilities of the baseline models, allowing for a fair comparison. In Sections 3.2.1 and 3.2.2, we evaluate our model against these baselines, showcasing its superior performance. However, the primary focus of this paper is on the camera + image + text-to-video scenario. In Sections 3.2.3 and 3.3, we conducted experiments on two different datasets, RealEstate and Epic-Field, respectively. EgoSim demonstrated more precise camera control and more effective scene interaction in these cases. The description of the experimental design has been included in the revised manuscript.

Next, we will answer each of the individual questions.

1. EgoSim is compatible with various pretrained models. In the case of camera+ image-to-video (I2V), we can use SVD instead of AnimateDiff.
2. We input camera poses, image and text for Epic-Field experiments for camera + image + text-to-video scenario.
3. Following the same protocol as CameraCtrl, we trained an additional LoRA, as both our work and CameraCtrl use AnimateDiff as the pretrained weights.
4. Both datasets inherently include camera pose information. We improved the Epic-Field dataset because it was originally unsuitable for training video diffusion models. Its videos contain extensive motion blur and lack detailed captions.
5. For the RealEstate dataset, we use 67,477 scenes for training and 7,289 scenes for testing. For Epic-Field, we use 611 scenes for training and 88 scenes for testing. It should be noted that the duration of each scene of Epic-Field is much longer than that in RealEstate. That’s why the number of scenes in Epic-Field is much smaller. These details are mentioned in Appendix A1.1.
6. In the case of SVD and AnimateDiff, we keep the original parameters of our base models fixed and only optimize the newly introduced layers and their subsequent layers using the AdamW optimizer with learning rate 2 × 10−4 . All models are trained on 8 NVIDIA A100 GPUs for 300k iterations using an effective batch size 32. We use BF16 precision for training SVD. These details are covered in Appendix A1.2.

Computation: In practice, for each query, we can compute an epipolar attention mask for keys and values. The physical meaning of this mask is that each query essentially represents a ray in 3D space, and this ray's projections in other views are captured as epipolar attention masks. The mask values are represented as 0 and 1, where 0 indicates positions that do not need to be attended to, and 1 indicates positions that the projection passes through or is near. So K* and V* refer to keys and values whose mask values are 1s.

Typo: Thank you for pointing out this typo. It has been corrected in the revised version.