PlayerOne: Egocentric World Simulator

First thanks for detailed comments.

Q1.The model,research gap,motivation,and challenges are unclear. A1: Most prior world models are limited to narrow domains or simple navigation and cannot align complex human motion or ensure scene consistency in egocentric video.Our motivation is to achieve free-form full-body motion control and immersive egocentric video in realistic environments.Our approach takes a single egocentric image and third-person motion as input and generates consistent egocentric video through two modules:PMI splits motion into codes for different body parts and injects them independently for precise part-level control;SR uses 3D point maps to ensure scene geometry remains consistent.PMI and SR together address the main challenges of motion alignment&scene consistency.

Q2.Release of dataset and code. A2: Training code is in the supplement.Full dataset&code for training&distillation will be released after acceptance.

Q3.Claim of first egocentric simulator and comparison with GEM,VISTA. A3: Egocentric here means world-consistent simulation in diverse environments with free interaction,not just driving."Egocentric" here is from person view.GEM and VISTA focus on driving and cannot support daily-life scenes or real-time user interaction.For EgoGen,it's designed for egocentric perception tasks to generate labels and isn't related to our topic of world simulator.GEM only edits non-player character motion by pose;VISTA is for driving control.Neither allows free egocentric exploration.Our model maintains scene and motion consistency and supports real user motion.Our method outperforms both:

We'll update our definition&comparison in revision.

Q4.What does this work offer the community?Why is it first? A4: We present first general-purpose egocentric world simulator enabling free-form full-body control and immersive exploration in realistic,diverse settings.Previous work lacks high-degree-of-freedom motion alignment and consistent interactive simulation from minimal input.Our code,models,and data set a new benchmark for embodied AI.

Q5.What is a world model(Line 20)? A5: A world model simulates and interacts with environments and human actions,learning a unified dynamic representation for prediction and interaction,covering spatial,temporal,and causal structure,allowing open-ended exploration.Our definition follows recent works like The Matrix[6] and MineWorld[2].

Q6.Why do these advancements help(need citations)? A6: World models allow game engines to simulate interactive environments that adapt to player actions,not just fixed scripts.This is essential for immersive,scalable games.They enable realistic motion,consistent scene generation,and efficient content creation,as shown in Oasis,MineWorld,Google Genie,Microsoft WHAM.

Q7.What is predictive modeling in Line26? A7: Predictive modeling is learning a model that generates or predicts future environment states from current and past input,typically predicting future frames,actions,or dynamics for planning,control,or simulation.

Q8.Why are these world models critical infrastructure? A8: World models are essential for next-generation interactive simulations&virtual worlds,offering dynamic simulation,generalization,and data-driven learning,which are key for building scalable interactive environments.Prototypes like Oasis&MineWorld show this potential.

Q9.Statement of topic Line29. A9: "This topic" refers to world models enabling immersive human interaction,allowing users to explore and interact as participants,not just simple actions or planning.We will clarify scope and emphasize our advances over limited-interaction models.

Q10.Works on realistic scenes Line30. A10: See A3.

Q11.Why do previous works limit users in this way(Line31)? A11: These limits come from data and model design.Prior datasets are from games with basic moves only,so models inherit these limits and cannot generate complex actions.Prior models also lack modules for high-degree-of-freedom motion.Our method addresses both aspects to enable immersive interaction.

Q12.GEM seems to allow skeleton poses. A12: GEM uses pose conditions to edit non-player characters in pre-recorded videos;it does not allow free exploration or user control as main character.See A3 for more.

Q13.Unclear problem statement&research gap(Line39-60). A13: Previous world models only support simple navigation or specific games and do not allow high-degree-of-freedom user interaction in realistic scenes.Our goal is a simulator like a VR game,where users can explore environments with full-body motion,providing a single egocentric image and exocentric video;our model generates immersive egocentric video in real time.

Q14.Why is this method first(Line61)? A14: See A3.

Q15.Release of dataset/code(Line67). A15: See A2.

Q16.What does k mean in Line106? A16: k refers to the number of frames divided by the VAE temporal downsampling ratio,that is,the length of the latent sequence after encoding.

Q17.What are problem statement,inputs,and outputs(Line99)? A17: See A13.

Q18.How does paragraph in Line109 connect to its context? A18: Lines109–120 introduce PMI and SR,crucial for part-level motion control and scene consistency,matching the motivation and pipeline description and directly addressing egocentric video generation.In revision,we will clarify their connection to the overall framework.

Q19.Goal and motivation of core modules(Line109). A19: See A1.

Q20.Role of VAE? A20: VAE is used to encode video frames and does not process human motion sequences,which are encoded separately by our motion encoder.

Q21.Why are motion latents the same shape as images?How is that ensured and why is high-resolution encoding needed? A21: We use eight-layer 3D conv motion coder for this.Same shape lets pixel-to-pixel align for motion and look,so model can set each frame pixel by pixel by motion code for fine part control.Past work[4,5] shows this concat way keeps info and gets better match than others.Matching shapes gives better motion control.

Q22.What is camera sequence? A22: Camera sequence is a series of camera extrinsic parameters(i.e., rotation matrices)from head motion over time,representing the egocentric viewpoint path in video.

Q23&Q24.What's 4D scene/point-map sequence? A23: 4D scene/point-map sequence is modeled as a temporal sequence of 3D point clouds,where each point cloud encodes scene geometry at a given frame.This captures spatial geometry over time,ensuring consistent context in generated videos.

Q25.Why t in U(0,1) instead of U(1,T) in Eq.1 A25: In Eq.1,t is sampled from a uniform distribution in [0,1],as is standard in flow matching.The notation t=1,...,T in Line121 will be unified in revision.

Q26.Where does alpha^2+delta^2=1 come from A26: We follow common practice in previous diffusion models,alpha^2+delta^2=1 ensures the trade-off between signal and noise is normalized during forward diffusion.

Q27–Q29.Empirical analysis in Line130 and why three-separate-latents/separate-processing is better A27–A29: See A1 for Reviewer-jaf9.

Q30.Write axis angles differently in Line143 A30: Thanks for the suggestion.We'll revise axis angle notation in revision.

Q31.Unclear how 4D scene is represented in Line152 A31: See A23.

Q32.Introduction of CUT3R in Line154 A32: CUT3R generates per-pixel 3D point maps from images online and is used to create point-map sequences as labels.We will expand details in revision.

Q33.How is alignment in Line158 happening A33: See A2 of Reviewer-WmZe.

Q34.How is scene consistency kept during inference if point maps do not update A34: During inference,the model generates point-map and video-frame latents auto-regressively,each depending on previous results.This ensures the evolving point-map sequence always guides scene geometry,keeping spatial context consistent.

Q35.Claim of no public datasets A35: We acknowledge for the oversight of EgoSim and clarify EgoSim and the subset of EgoExo4D with annotated poses are public but too small.To achieve good generalization on diverse domains, we built an automatic pipeline for more large-scale data.Models trained only on these public data perform much worse.

We will add detailed comparisons in revision.

Q36.Is exoview used for other purposes A36: No,exoview data is only used to extract SMPL motion sequences.

Q37.Why do ablation studies come before main results A37: No existing methods match our setting,we show ablation studies first to validate contributions before main comparisons.

Q38.Complexity of joint motion encoder and latent shape A38: Three split motion encoders(2.31M,1.29M,1.97M parameters)are close in size to the joint encoder(4.5M).Both encode SMPL parameters as arrays of shape R(k × 3 × h × w),where k is a quarter of frames(4× downsample),and h,w are spatial sizes.

Q39.Why were SOTA models like GEM not included in comparisons? A39: GEM is designed for autonomous driving,not general environments,and does not support interactive user control.Based on feedback,we add GEM as a baseline(see A3)and will expand comparisons in revision.

Q40.Discussion and limitations. A40: See A1 of Reviewer-cVUd.

[1]OASIS.Yang et al[2]MineWorld.Guo et al[3]GeNIE.Wang et al[4]CatVTON.Chong et al[5]UniAnimate.Wang et al[6]The Matrix.Feng et al

Q3: Superiority Over Previous Video Diffusion Models

A3: Our approach advances egocentric video generation and simulation in several key ways that previous video diffusion models do not address:

1. Scene Consistency and World Modeling:
   Previous video diffusion models typically generate frames with limited consideration for long-term spatial or geometric consistency, often leading to visual artifacts such as background flicker, drifting objects, or abrupt scene changes. In contrast, our method explicitly reconstructs and maintains a unified 3D world throughout the video sequence. The scene-frame reconstruction module leverages point cloud geometry to align each generated frame with an evolving, coherent scene representation. This ensures that the world remains stable and immersive, which is critical for any true simulation of first-person experience and has not been realized by prior works.

2. Accurate and Fine-Grained First-Person Control:
   Existing methods often use a single entangled pose vector for motion control, treating all body parts equally. However, egocentric scenarios demand nuanced handling: head movement is critical for viewpoint and world orientation, hand movement is central to object interaction, and foot placement influences navigation and realism. By employing part-disentangled motion injection, our model provides independent and precise control of the head, hands, and feet. This design captures the true complexity and agency of first-person interaction, enabling the model to accurately reflect user-driven actions and maintain correct alignment between the user’s motion and the virtual environment. Such fine-grained control is essential for natural egocentric simulation and is not achievable with prior, entangled motion injection approaches.

3. Efficient and Real-Time Generation:
   Many existing models rely on heavy, computationally intensive modules—such as 3D point cloud encoders—during inference, which greatly limit their practical use for real-time applications. Our method, inspired by the latest advances in lightweight latent-space alignment, uses efficient adapter structures within the scene-frame reconstruction, eliminating the need for bulky inference-time modules. This significantly reduces computational overhead and enables our system to generate high-quality, immersive egocentric videos in real time—crucial for interactive and practical deployment (e.g., virtual reality or gaming).

4. Comprehensive Validation:
   We have thoroughly validated our approach by directly adapting several leading video diffusion models to the egocentric setting, with the same data and input motion injection schemes. Our results consistently show that these adapted baselines suffer from scene inconsistencies, poor alignment, and lower visual fidelity, while our method delivers stable, realistic, and precisely controlled egocentric video. This is further supported by both quantitative metrics and qualitative results provided in the manuscript.

5. Empirical Superiority: To further verify our superiority, we adapt the baselines in the manuscript (Cosmos-14B, Cosmos-7B, Aether) to our setting and apply our disentangled scheme with the same training data, our model still achieves notably better scene stability and action alignment:

In summary, our method is not simply an incremental improvement but addresses core limitations of previous video diffusion frameworks. By ensuring stable world modeling, enabling accurate and interactive egocentric control, and maintaining a lightweight, real-time capable design, we provide a practical and substantial advance for the field of egocentric simulation and video generation.

Q2: Model design motivation

A2: Thank you for raising the need for more explicit design motivation. Our model architecture is purpose-built for egocentric simulation and tackles specific issues that make prior conditional diffusion models insufficient in this setting:

• Lightweight, Efficient 3D Modeling: Previous methods[1] that attempt 3D scene modeling often use heavy point cloud encoders during inference, greatly limiting scalability and real-time use. Inspired by works[2,3]showing that spatial consistency can be enforced via lightweight 3D latent adapters, our Scene-Frame Reconstruction achieves efficient, real-time scene alignment without such heavy inference overhead. This design choice is crucial for supporting long, interactive sequences, as required in VR-style egocentric scenarios.

[1] Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation

[2] Video World Models with Long-term Spatial Memory.

[3] AETHER: Geometric-Aware Unified World Modeling.

• Part-Disentangled Control: Traditional pose-driven diffusion models were designed for third-person, fixed-camera tasks and do not account for the varying impact of different body parts in first-person scenarios. In egocentric simulation, the head, hands, and feet each play unique, critical roles for maintaining realism and interaction. Our model encodes and injects these parts separately, allowing independent and precise control. This decoupling enables—for example—a user to turn their head while reaching out with their hand, with both motions remaining stable and realistic in the generated video.

  • Concrete Example: If head and hand motion are entangled, moving the hand could unintentionally “drag” the viewpoint, resulting in dizziness or loss of immersion. Decoupling prevents such artifacts.

• On Complexity: Our architecture’s complexity reflects these needs: real-time egocentric simulation with stable world modeling and fine-grained control cannot be solved by simple 2D or monolithic architectures. Each module serves a specific purpose toward this goal.

Summary: Our design directly addresses the limits of prior work for egocentric, interactive tasks. We will expand architectural motivation, provide ablation examples, and clarify each component’s contribution in the revised version.

Thank you very much for your detailed follow-up. We sincerely appreciate your thoughtful feedback and acknowledge your continued concerns. Below is our further clarification of the mentioned points:

Q1: Is PlayerOne truly a "world model," or just a pose-conditioned video generator?

A1: Thank you for raising this fundamental question. We fully agree that not all pose-driven egocentric video generators qualify as world models. Here, we clarify why PlayerOne goes beyond simple video generation:

• Continuous 3D World Consistency: The core difference is that PlayerOne maintains an evolving, consistent 3D world throughout video generation. Our Scene-Frame Reconstruction module is not just a frame-level denoiser, but a mechanism for online world modeling: it reconstructs 3D geometry for each frame using egocentric cues, ensuring the virtual scene remains stable, consistent, and spatially correct even as the viewpoint and body motion evolve. This is fundamentally different from prior models, which often produce temporal drift or scene incoherence (e.g., backgrounds that “jump,” objects that move or vanish between frames).

• Part-Specific Motion Control: Unlike exocentric (third-person) video generators (such as dance models), which encode all joints as a single vector, we found that in egocentric settings, body parts contribute very differently to scene stability and realism:

  • Head motion is directly tied to camera movement and viewpoint alignment. If the head motion is imprecise, the virtual world appears unstable or the geometry is mismatched.

  • Hands often interact with the scene (e.g., grasping, gesturing); misaligned hands break the realism and the sense of agency.

  • Feet anchor the user and determine movement; poorly aligned feet can result in floating or gliding artifacts. Existing pose-based video models, not designed for egocentric tasks, fail to capture these distinctions and typically produce visible artifacts when applied directly (see quantitative results below).

• Empirical Verification: To further verify our findings, we adapted multiple exocentric pose-driven methods (RealisDance, Magic-Me, MimicMotion, AnimateAnyone) to the egocentric domain, that is using their injection scheme in our method. Results show clear improvements with our part-disentangled approach:

  Method	DINO↑	CLIP↑	MPJPE↓	MRRPE↓	PSNR↑	FVD↓	LPIPS↓
  Ours	67.8	88.2	127.16	151.62	52.6	226.12	0.0663
  RealisDance-Scheme[1]	62.3	83.9	137.44	161.20	50.9	246.25	0.0711
  Magic-Me-Scheme[2]	61.6	82.8	139.10	163.42	50.6	250.71	0.0724
  MimicMotion-Scheme[3]	61.1	82.2	140.34	164.55	50.3	252.66	0.0735
  AnimateAnyone-Scheme[4]	60.8	81.8	141.22	165.37	50.1	254.10	0.0741

• User Control: Our present interface, accepting only an initial frame and motion sequence, was chosen to emulate the VR scenario where a user can freely explore with minimal manual input. We fully agree that richer controls (e.g., text, object, or goal-driven instructions) are valuable, and we plan to expand user interactivity in future work.

• Summary: PlayerOne is not merely pose-conditioned; it models the evolving world, achieves part-level motion realism, and ensures persistent scene geometry—key traits of a world model. We will further emphasize and visualize these distinctions in the revised paper.

[1] Equip controllable character animation with realistic hands

[2] Magic-Me: Identity-Specific Video Customized Diffusion

[3] MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance

[4] Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

However, I am still uncertain about your claim regarding the world model. Your model takes point map latents during training, yet you state, "During inference, the system simplifies the process by using only the first frame and the corresponding human motion sequence to generate world-consistent videos." This suggests to me that the component intended to ensure world consistency is removed during inference.

Additionally, your description of the method lacks equations and formal definitions, making it difficult to follow the precise inputs and outputs of your modules.

How many point map latents are you using? What is the maximum sequence length before the point map latents start losing information? Do you retain point map latents from previous frames? How is this information concatenated during training and inference when no point map latents are available?

In general, the lack of clear definitions, ambiguity about scene consistency, which would be required for a world model, and missing information about the exact inputs and outputs of the modules, poses significant challenges. Although I believe the method could have great and valuable potential for a future submission, I believe it is too much for a single revision.

Q3.How many point map latents are you using? What is the maximum sequence length before the point map latents start losing information? Do you retain point map latents from previous frames? How is this information concatenated during training and inference when no point map latents are available?

A3: Thank you for these detailed questions. Our point-to-point clarifications are shown below:

• How many point map latents are you using?
  The point map sequence matches the length of the video frames. For a video with $K$ frames, we use $K$ point map latents. The point map sequence can be understood as a progressive reconstruction of the world: that is the $n$-th point map is rendered from frames 1 to $n$ using CUT3R.

• What is the maximum sequence length before the point map latents start losing information?
  In our experiments, scene consistency remains good for videos up to about 30 seconds. When the video exceeds one minute, occasional scene inconsistencies appear, likely due to imperfect point map reconstruction/information loss. We plan to further refine this aspect in future work.

• Do you retain point map latents from previous frames?
  The point map latents are inherently and jointly reconstructed with the video latents in an autoregressive manner when inference. During the training process, each ground truth point map is indeed rendered based on all previous frames.

• How is this information concatenated during training and inference when no point map latents are available?
  As described in A1, during both training and inference, the model treats the video latent and point map latent equally: both are noised and denoised in parallel, so there is no need to provide extra ground truth point map sequence as input at inference time. The model predicts both the video and the point map sequence jointly, ensuring scene consistency throughout generation.

If further details are needed, we are happy to expand or clarify more in our revision.

Q1.When inference, the component intended to ensure world consistency is removed?

A1: Thank you for raising this important point. We would like to clarify that, contrary to the impression that the scene consistency module is removed during inference, our framework maintains the same mechanism for ensuring world (scene) consistency during both training and inference.

• Role of Point Maps: During training, the CUT3R module is used only to generate ground-truth point map sequences corresponding to each video frame. These ground-truth point maps serve as supervision for our scene-frame reconstruction loss and the generation of noised point map latents. However, the model’s true inputs, both in training and inference, are always just the initial egocentric first frame and the human motion sequence (The prediction targets are always the video frames and point map sequence). The point maps themselves are never given as input at inference time. Our proposed framework can simultaneously generate the video frames and corresponding point map sequence at the inference stage.

• Training Pipeline Details: Let us clarify the process more precisely:

  • The raw video sequence is denoted as $V \in \mathbb{R}^{B \times K \times 3 \times H \times W}$, and its corresponding point map sequence as $P \in \mathbb{R}^{B \times K \times N \times 3}$.

  • After VAE encoding, the video becomes $z_{video}^0 \in \mathbb{R}^{B \times k \times c \times h \times w}$, where $h = H//8, w = W//8, k = K//4$.

  • The point map sequence is encoded via a 3D encoder and adapter to $z_{point} \in \mathbb{R}^{B \times k \times 64 \times h \times w}$, matching the video latent’s shape.

  • Both video and point map latents $z_{video}$ and $z_{point}$ are noised for training.

  • The motion sequence, encoded by our motion encoder, yields $z_{motion} \in \mathbb{R}^{B \times k \times 3 \times h \times w}$.

  • The first-frame latent ($z_{first} \in \mathbb{R}^{B \times 1 \times c \times h \times w}$) is repeated k times as ($z_{first} \in \mathbb{R}^{B \times k \times c \times h \times w}$) and then concatenated with motion, video, and point-map latents as follows:

    $$z_{input} = \text{concat}(z_{first}, z_{motion}, z_{video}', z_{point}') \in \mathbb{R}^{B \times k \times (3+64+2c) \times h \times w}$$

    (where $'$ denotes latents with added noise).

  • The DiT backbone takes this concatenated latent, denoises it to $z_{output}$ (same shape), and the MSE loss is computed between the predicted ($z_{video}^{\star}$, $z_{point}^{\star}$) and ground-truth ($z_{video}$, $z_{point}$) video/point map latents.

• Inference Process: During inference, the model still generates both the video and the point map sequence simultaneously. The only difference is that no ground-truth point maps are available; instead, the network predicts both the video and its corresponding point map latents frame by frame in an autoregressive manner, ensuring that scene consistency is maintained internally.

• Key Point: Thus, the point map sequence and the video are always treated equally in our latent modeling, and the scene consistency module is fully active during inference—the model predicts the evolving point map sequence jointly with the video, guaranteeing scene consistency without any need for extra input or for removing any module.

We hope this clarifies that our approach does not remove the scene consistency mechanism during inference; instead, it is seamlessly integrated into both phases, with point map prediction acting as a consistent latent scaffold for world modeling.

Q2.Description of the method lacks equations and formal definitions.

A2: Thank you for highlighting the need for more formal definitions and equations. We agree that explicit mathematical notation and clear input-output specification will greatly improve the clarity and replicability of our work. In A1, we have now included the precise mathematical forms, dimensions, and roles of each input and output in our main modules. We will integrate these formalized descriptions and key equations into the revision to make our architecture, module connections, and the flow of information in our method fully transparent for readers. If there are any further clarifications or additional information you would find helpful, we are open to your suggestions and will be happy to further optimize the paper based on your feedback. Thank you again for this valuable advice, which will help us substantially improve the quality and accessibility of our work.

Q1. How to conduct comparison without Point Map Latents during Inference

A1: We clarify (see also RE Q1-Rebuttal Part1 A1) that point map latents are indeed present during inference. Both point map latents and video latents are jointly denoised in the DiT model; the only difference is that, at inference, we do not require a separate point map encoder or adapter to obtain the point map latents (Since we do not have the ground truth point map as input). Instead, random noise is used to initialize both the video latents and the point map latents, and the model jointly predicts the sequence, maintaining scene consistency throughout generation.

Q2. Visual and Quantitative Difference with the 'No Adapter' Setting. Similar for "No Recon".

A2: The large performance gap for 'no adapter' comes from a mismatch in the feature spaces: our point map encoder is fixed, and its latent space is not directly aligned with the VAE-encoded video latent space. When these are simply concatenated without a dedicated adapter, the model struggles to reconcile the different statistics and semantics between the two types of latents, resulting in degraded video quality and scene breakdown. Our adapter module learns to map point map features into the appropriate latent space for effective fusion with video latents, thus improving both fidelity and scene consistency. This design is crucial for effective information transfer between geometry and appearance.

For 'no recon', we acknowledge that in Table 2, the quantitative results for 'no recon' do not seem significantly different from the full model. This is primarily due to two reasons:

• Strong Pretraining: Our model is first pretrained on a large-scale egocentric video dataset. This pretraining stage may equips the model with a weak ability to model scene dynamics and maintain a certain level of spatial consistency, even before introducing explicit scene-frame reconstruction.
• Lack of Direct Metrics: Most commonly used metrics (e.g., DINO, CLIP, PSNR, FVD, LPIPS) are not designed to specifically measure scene (world) consistency across frames. These metrics primarily evaluate frame-wise visual similarity, semantics, or overall diversity, but do not directly capture temporal geometric consistency, which is the core advantage of our reconstruction module.

Metric Recommendation: To more objectively evaluate scene consistency under large viewpoint changes, we perform multi-view 3D reconstruction (using COLMAP) on generated videos. We then compare the overlap ratio and Chamfer Distance between the reconstructed point clouds for different methods. Our method yields significantly higher point cloud consistency and structural stability, demonstrating superior 3D scene coherence across time.

Example Results:

In summary: The relatively small difference in traditional quantitative metrics is due to both strong pretraining and the lack of a direct metric for scene consistency. However, with a more targeted consistency metric, the effectiveness of our reconstruction module is much more evident. We will include such an analysis and corresponding visualizations in the revised paper to clarify this point.

---

Q3. DUST3R Ablation Implementation

A3: For the DUST3R ablation, we implemented it exactly as for CUT3R: one point map latent is generated per frame and processed by our 3D encoder and adapter. During inference, neither CUT3R nor DUST3R are used for input—our model predicts both video and point map latents from noise, ensuring all scene information is generated in a self-consistent way. Our method relies on no external point map inputs at test time.

Dear Reviewer X28J,

Thank you again for your comments. Your review is very valuable, and we would appreciate your further feedback. Of course, considering the deadline is approaching, please let us know if this or any other point remains unclear, as we would be happy to elaborate further.

Thank you for your valuable time!

Sincerely,

Authors of Submission1028

Thanks for the valuable feedback. We appreciate your insights and suggestions. Below, we address the points raised and clarify the related issues.

---

Q1.Why is the camera translation removed. A1: Camera translation is removed because SMPL head-parameters only provide rotation,not translation.When converting SMPL head params to camera extrinsics,only rotation is available—so translation can't be included.This matches the SMPL definition and keeps the camera centered at the head in the egocentric-frame.

Q2.How the scene-frame alignment step works.How does the adapter have the signal to align the point-map latents with video latents?(L158). A2: Thank you for your question.In our framework,the adapter maps point-map latents into the same latent space as video latents.We do not use an explicit alignment loss;instead,we follow prior works like[1-2],which show that simple linear or convolutional modules can effectively project point cloud latents to the video latent space. This lets the model leverage spatial structure from point-maps while conditioning video generation efficiently.Thus,alignment is achieved through learned mapping and implicit end-to-end training.

[1]Wu,T.,et al.Video World Models with Long-term Spatial Memory. [2]Aether.AETHER: Geometric-Aware Unified World Modeling.

Q3.More explicit and precise call outs about what is distinct and what is important+distinct in this work. A3: Thank you for your question.We acknowledge our previous wording may have been unclear on this point.To our knowledge,no prior work at submission used human body pose as the main condition for egocentric video generation in real-world scenarios.A recent related work[1]also uses human pose,but their method does not ensure scene consistency or accurate action-pose alignment;their model also struggles to generalize beyond its training domain.

By “specific directional motions,” we mean prior models only support basic navigation(forward,backward,turning),without enabling complex,VR-like or full-body actions such as picking up objects or interacting naturally with NPCs. These limits are due to both the restricted motion in game datasets and the lack of modules for fine-grained action alignment and scene consistency.In contrast,our method uses full-body motion as condition,explicitly addresses part-level action alignment and scene consistency,and generalizes well to diverse,realistic settings.

[1]Bai,Y.,et al.Whole-Body Conditioned Egocentric Video Prediction.

Q4.L130:Movement is still camera(head) motion. A4: We'd like to clarify while the output video is the first-person(head/camera)view,our method uses real-world,full-body human motion as driving condition—not just head pose—so the generated video captures both camera motion and detailed body part movements in sync.This results in more natural,immersive egocentric movement(validated by full-body evaluation results in A10).

Q5.Technical novelty—the interesting nature of the specific setting. A5: We appreciate your recognition of our problem setting and application scenario.Although our technical choices build on established methods for robustness,our main contribution is to introduce and solve a highly interactive egocentric world simulation task—supporting flexible,full-body,realistic user exploration,which prior works have not addressed.We will further highlight the significance and potential of this scenario in the revision.

Q6.Reproducibility for the steps towards end of Sec 3(e.g.,distillation and finetuning). A6: For finetuning,we have included all necessary code in the supplement for reproducibility.For distillation,we strictly follow the Causvid protocol,which is standard in long video generation and reproducible with public tools.We will add more details about our distillation process(Also shown in Q3-A3 of Reviewer cVUd),including key parameters and scripts,in the revision.After acceptance,we'll release the complete codebase with full documentation for both distillation&finetuning.

Q7.Whether it’s expected input exo body poses are “projected” to the nearest visually suitable poses in the ego scene thanks to the diffusion model?Is this intended and is it observed? A7: In training, motion poses and the first-person frame are strictly aligned. At test time, however, we observe our model shows some robustness when the input pose doesn't exactly match the ego one in target scene. In these cases, the model often adapts the motion, effectively “projecting” it to the most visually reasonable pose in the generated result, as you described. We will add visual examples of this behavior in revision.We use self-taken videos for qualitative results mainly to demonstrate we can offer immersive egocentric experiences(Sunglasses as VR headset in demo),not because strict pose-to-scene matching is required.

Q8.Why Cosmos&Aether were chosen.Why they are complete set of baselines to support the claims. A8: Most world models differ from our setting,as they focus on specific games(e.g.,Minecraft)or tasks(e.g.,autonomous driving).Cosmos and Aether are the closest general-scene simulators to our approach.Therefore,we emphasize thorough ablation studies of our modules rather than direct comparison with domain-specific baselines(That's the reason that we place the ablation-study before the other-baseline-comparisons.).Our experiments evaluate all modules&variants,demonstrating their necessity&effectiveness.

Q9.Why is Wanx 2.1 the right base model?How does that interact with the choice of baselines as well in terms of fairness,apples-to-apples. A9: Thanks for your question.Wanx2.1 is the best open-source video-generation model currently available,so we use it as our backbone.Our approach is not tied to Wanx—when using stronger models,performance further improves:

Wanx as backbone is independent of baseline-selection.Compared to Cosmos(7B/14B)&Aether(5B),our method achieves better results with fewer-parameters(1.3B),demonstrating both effectiveness&efficiency.

Q10.Why benchmark-samples only from Nymeria? A10: Thanks for your suggestion.In response,we have reconstructed our evaluation benchmark to provide a more comprehensive assessment.Specifically,we selected 100 videos featuring full-body actions from the test-sets of four different datasets: EgoExo-4D,Nymeria,FT-HID,EgoExo-Fitness. The results,as shown below,demonstrate that our method consistently outperforms both existing approaches and various ablated variants of our method.

Q11.Unclear how qual and quant results match up. A11: Thank you for your question.For the qualitative comparisons in our supplementary materials,we used exo videos recorded by ourselves.The main reason was to better showcase that our model can simulate a "virtual reality"experience.To highlight this,the subjects in our videos wore sunglasses to mimic the appearance of VR headsets.Please note that for all quantitative evaluations and benchmarking,we strictly relied on the official test sets from Nymeria and other datasets.We will make this distinction clearer in the revised version to avoid any confusion between demonstration videos and quantitative results.

Q12.Where do text descriptions come from that are used to evaluate ground truth? A12: For the comparison methods,the text descriptions were generated using the Qwen2.5-VL model based on the corresponding ground truth video frames(Line210-211 in the manuscript).

Q13.What about arms or any other visible body parts for the MPJPE metric,vs.just hands? What if the hands are not visible in the frame of the proposed method’s output?. A13: In most cases from the test sets,only the hands are visible in the first-person view videos,so our previous evaluation of motion-related metrics(such as MPJPE) primarily focused on the hands.Following your suggestion,we have reconstructed our benchmark by selecting cases where the full body is visible in the frame for comprehensive evaluation.The results in A10 show that our method still consistently outperforms existing approaches across all metrics.We will revise the manuscript with these results to provide more comprehensive demonstration.

Q14.Show ego-frame input for each one and one-video at a time throughout and typos. A14: Thanks for your suggestion.We appreciate your feedback and will update related text and demo video in revision.

Q15.Aspects of model design that may cause issues. A15: Please refer to the last paragraph of Q1-A1 for Reviewer cVUd(Due to the space-limit).

Thank you for the valuable feedback. We appreciate the reviewers’ insights and suggestions. Below, we address the individual points raised and clarify relevant aspects of our work.

---

Q1.Consistency in long-video generation and whether the point cloud encoding scheme can ensure consistency across the generated video sequences?

A1: Thank you for your comment. We would like to clarify that our method is not limited to generating only short (e.g., 6-second) videos. After distillation with Causvid, our model can generate long videos (i.e., more than 30s) in an auto-regressive fashion, producing each frame sequentially conditioned on previous outputs on-the-fly, rather than simply concatenating multiple short clips. This sequential generation, together with our current point cloud encoding approach, allows us to maintain strong scene consistency throughout long video sequences. We acknowledge that the 6-second horizon in the demo was an oversight, and we will include longer video generations in the revised version.

To address your question directly, we have extended our benchmark by increasing the video length to 24 seconds and evaluated our method under this longer horizon. The results below show that our model maintains comparable performance and visual quality, demonstrating its ability to generate consistent long-term video sequences.

---

Q2.Limited Discussion of Failure Cases.

A2: Thank you for your suggestion. We acknowledge that our current manuscript provides limited analysis and visualization of limitations in game-like scenarios. This is mainly due to the relatively small amount of available training data from such domains. As a result, our model is more likely to exhibit issues such as inaccurate hand alignment with the given motion conditions or background blurring during rapid movements in game environments.

To better illustrate this, we separately evaluated our model on data from game-like and real-world domains. The results are shown below:

We will include more visualizations and an in-depth discussion of these failure cases in the revised version to provide a more comprehensive and transparent evaluation of our method’s limitations in different domains.

---

Q3.Clarify the actual latency between motion input and video output.

A3: Thank you for highlighting this important point. After distillation, the measured end-to-end inference latency is approximately 119 milliseconds per generated-frame, corresponding to about 8.4 FPS. This latency is measured from the moment the motion input is received to the generation of the corresponding video frame, and includes all major processing steps. We will include more comprehensive inference performance metrics and latency breakdowns in the revised version to better illustrate the real-time capability and deployment feasibility of our model in interactive scenarios.

Thank you for the valuable feedback. We appreciate your insights and suggestions. Below, we address the individual points raised and clarify relevant aspects of our work.

---

Q1.Lack of analysis for failure mode.

A1: Thank you for your suggestion. We acknowledge that our current manuscript provides limited analysis in this area. In particular, due to the relatively small amount of training data from game scenarios, our model may more likely exhibit results with slightly inaccurate hand alignment with the given condition or background blurring. To better illustrate this, we separately evaluated our model on data from game-like and real-world domains. The results are shown below:

Besides, our models may generate results with noticeable artifacts, particularly during extremely fast or complex motion, which can be observed in the provided cases (e.g., jumping from a high place in 1:03 of the demo). These artifacts are mainly due to the limitations of the current adopted base model and replacing it with a stronger backbone can alleviate such issues. Specifically, when using the Wan 2.1 14B as our backbone, we can achieve better performance:

Here we also provide the limitation analysis on our model structure. From the view of the model, one current limitation of our model is the lack of explicit physical interaction modeling between the human body and the environment. Structurally, our framework conditions video generation on SMPL motion sequences and a static point cloud reconstruction of the scene, but does not incorporate physical constraints or interaction modules such as collision detection, physics engines, or contact reasoning. Consequently, our model may generate unrealistic results, such as hands or body parts penetrating objects, or a lack of proper occlusion and collision feedback. We acknowledge this limitation and consider it a promising direction for future work. As recommended, we will substantially expand the Discussion/Limitation section and the related visualization of the failure cases in the revision, providing a more thorough analysis of both model and data limitations.

---

Q2.Limited temporal horizon of generation and related discussion.

A2: Thank you for pointing out this issue. While our original experiments and visualizations focused on 6-second video sequences, this does not reflect a fundamental limitation of our method. After distillation with Causvid, our approach generates video in an autoregressive manner, allowing for the synthesis of arbitrarily long video sequences. The 6-second horizon in the demo was an oversight, and we will include longer video generations in the revised version.

To address your question directly, we have extended our benchmark by increasing the video length to 24 seconds and evaluated our method under this longer horizon. The results below show that our model maintains comparable performance and visual quality, demonstrating its ability to generate consistent long-term video sequences.

We believe that generating longer videos is crucial for this task for several reasons:

• (1) it enables modeling of more complex and realistic user interactions.
• (2) it allows thorough evaluation of temporal coherence and scene consistency.
• (3) it better supports applications such as virtual reality and embodied AI, where prolonged, immersive experiences are essential.

While our implementation uses Causvid for long video generation, there are many emerging approaches[1-3] focused on long-term consistency and video synthesis, and we agree that pushing the boundaries of temporal horizon will be an important direction for future research.

[1] Lin, J., et al. "Long-Context Autoregressive Video Modeling with Next-Frame Prediction." arXiv preprint arXiv:2403.09603, 2024.

[2] Yu, X., et al. "Long-context State-space Video World Models." arXiv preprint arXiv:2405.11590, 2024.

[3] Chen, H., et al. "FreeLong++: Training-Free Long Video Generation via Multi-band Spectral Fusion." arXiv preprint arXiv:2406.03855, 2024.

---

Q3.Insufficient details about distillation.

A3: Thank you for your question regarding the distillation process. We strictly follow the Causvid distillation procedure in our implementation. During the distillation stage, we use the entire SMPL-Video paired dataset from the second phase of training. Specifically, we first generate 8,000 ODE pairs and train the student model for 30,000 iterations using the AdamW optimizer with a learning rate of 5 × 10⁻⁶. We then continue training with our asymmetric DMD loss for an additional 30,000 iterations, using AdamW with a learning rate of 2 × 10⁻⁶. A guidance scale of 3.5 is used, and we adopt the two time-scale update rule from DMD2 with a ratio of 5. The entire distillation process takes about 5 days on 32 A800 GPUs.

Regarding quality, the distillation stage is mainly aimed at improving inference efficiency, but we observe that the generation quality is largely maintained compared to the non-distilled model. In some cases, the distilled model even achieves slightly better temporal consistency. The comparision between the non-distilled model and distilled one for 6s video generation is shown below:

We will include a detailed comparison and qualitative side-by-side visualizations between distilled and non-distilled models, and discuss in which scenarios quality differences may be most obvious (e.g., rare actions, very long videos) in the revised version to make this clear to readers.

Thank you for the valuable feedback. We appreciate your insights and suggestions. Below, we address the individual points raised and clarify relevant aspects of our work.

---

Q1.Why use 3 body parts, not more?

A1: Thank you for your question. We choose to divide the human body into three parts—head, hands, and body/feet—based on the following considerations:

• Empirical Results: As shown in our ablation studies below, using fewer than three partitions (such as treating the whole body as a single entity) results in insufficient precision in aligning generated motions, especially for actions involving complex hand or head movements. Conversely, dividing into more than three parts provides negligible improvement but increases computational cost and memory requirements.

• Intuitive Reasoning: This three-part division aligns naturally with the semantic structure of typical human interactions: head movements primarily control viewpoint direction, hands execute detailed interactions, and the body/feet handle overall positional and movement dynamics.

• Why separate processing contributes to precise alignment: Splitting the latent representation into three separate parts—head, hands, and body/feet—enables the model to more effectively capture the unique and often semi-independent motion characteristics of each region. Human body motion is inherently high-dimensional and highly articulated, with different parts frequently moving independently or exhibiting distinct dynamics. Encoding all motion information into a single, entangled latent can cause fine-grained cues—especially from subtle or fast-moving regions like hands or head—to be overshadowed by larger body movements, making precise alignment difficult. By disentangling the motion into part-specific latents, each body region’s movement can be independently modeled and aligned with the corresponding input. This modular approach allows the model to better capture local nuances, improves compositionality, and enhances control over complex motions, resulting in higher motion fidelity and visual quality.

Hence, our chosen partition strikes an optimal balance between computational efficiency, precision, and intuitive semantic clarity.

---

Q2.Failure of the second motion condition in Fig.7.

A2: Thank you for bringing this to our attention. We carefully reviewed Figure 7 multiple times but could not identify a clear failure case in the second motion condition on either the left or right side. We would greatly appreciate it if the reviewer could clarify which specific motion sequence or frame is being referred to, so that we can better understand the issue and address it appropriately in our revision. We are committed to improving our work and welcome any further details or suggestions.

---

Q3.Examples of actions beyond handshakes or basic interactions.

A3: Thank you for your question. While our current demo primarily showcases simple interactions with people or animals—such as handshakes, high-fives, or lifting a dog(in 1:03 of our demo)—our method is not limited to these basic actions.

(1) Diverse Interaction Scenarios in Benchmark:
Our benchmark includes a broad range of interaction scenarios involving people and animals. Typical examples include:

• Passing objects to another person (i.e., passing basketball/football)
• Dancing with someone
• Petting or playing with animals (i.e., riding a horse and spinning in circles with a dog)
• Feeding animals

On these diverse and realistic human-human and human-animal interaction cases, our method achieves a significant performance advantage compared to existing baselines. This demonstrates the strong generalization capability and robustness of our approach. We will highlight these benchmark results and add more qualitative and quantitative examples in the revised version.

(2) Complex and Challenging Environmental Interactions:
Beyond interactions with people and animals, our method also handles a wide variety of complex actions involving the environment. For example:

• Figure 7: Challenging actions such as diving
• Figure 5: Climbing over a wall and kicking a ball
• Figure 8: Precise actions like picking up milk from a cabinet

Our demo video provides further examples, including:

• Driving by operating the steering wheel (see 0:32 and 0:38)
• Riding a bicycle, swimming, climbing over a wall (1:03), and swinging a sword (0:47)

(3) Ongoing Enrichment:
We are committed to further expanding the diversity and complexity of our demonstrations. In the revised version, we will provide additional visualizations featuring even more complex and varied interactions, including richer cases involving people and animals, to better illustrate the generalization and flexibility of our approach.

---

Q4.Noticeable artifacts, particularly during fast or complex motion.

A4: Thank you for your valuable feedback. The artifacts are mainly due to the limitations of the current adopted base model and replacing it with a stronger backbone can alleviate such issues. Specifically, when using the Wan 2.1 14B as our backbone, we can achieve better performance:

Thus if more powerful backbone architectures become available, we will replace our current backbone in future work. This upgrade could further reduce potential artifacts and improve the fidelity of the generated results.

---

Q5:Potential benefits of incorporating more images or scene information as input to the model.

A5: We would like to clarify that our current framework is intentionally designed to operate with only a single egocentric first frame and related human motion provided by the user. This design choice was made to simplify the pipeline and ensure a lightweight, user-friendly experience, but it does limit our ability to directly utilize additional images or scene inputs in the present version.

We agree that introducing richer scene information—such as additional frames, multi-view images, or more detailed environmental cues—could potentially improve the temporal consistency and visual realism of the generated video. Exploring how to effectively integrate such supplementary scene inputs is a promising direction for future work, and we appreciate the reviewer’s suggestion. In future versions, we plan to extend our framework to support and benefit from richer scene context, and we expect this will further enhance the fidelity and robustness of egocentric video generation.