We thank the reviewer for the clear and constructive comments, and appreciate the recognition of both the novelty of our approach and the breadth of our experiments. We appreciate the suggestions to improve the paper and we address these in depth below:

1. Comparison to Text-Conditioned Video Generation Methods

Thank you for the thoughtful comment. Our goal with PEVA is to explore action-conditioned egocentric video generation using physically grounded full-body motion actions that are given at every timestep. This is fundamentally different from text-conditioned methods such as Cosmos [1], which represent actions using text (e.g., “walk forward”) and thus lack precise and continuous physical control of the generated video.

To further validate the benefits of motion-based control, we conducted a comparison with Cosmos on a controlled hand-motion subset. We moved the hand in defined directions (e.g., “move the hand to top-right”, “move the hand to bottom-right”) and evaluated the alignment between intended and generated motions using PCK@0.1. PEVA achieves 85%, while Cosmos reaches only 22%.

While text prompts are intuitive and flexible, they often lack the precision required for fine-grained, spatially grounded motion control. Our experiments suggest that translating subtle physical intentions into language is inherently ambiguous—especially when pixel-level accuracy is needed. Expressing such intent in natural language would require awkward approximations (e.g., “move the hand slightly upward at a 73-degree angle”), which are rarely interpretable or faithfully executable by text-conditioned models. This likely contributes to Cosmos's lower performance on tasks requiring precise action execution.

While we are unable to include the visualizations at this stage, we will provide full results and qualitative comparisons in the final version. These preliminary findings suggest that PEVA offers significantly more accurate and controllable motion grounding than text-conditioned alternatives.

2. Performance Improvement Over Baselines (DF, CDiT)

We respectfully disagree with the claim that the improvement is marginal. While LPIPS gains may appear small numerically, they are consistent across all horizons and paired with significantly better controllability.

Below is a comparison of 1-step DreamSim perceptual loss over various prediction horizons:

DreamSim 1-step Perceptual Loss Across Time Horizons

3. Clarification on SMPL-X vs. Our Representation

Thank you for the request for clarification.

Our motion representation is based on the Xsens skeleton [2][3], which shares the kinematic tree structure with SMPL but differs in joint set, ordering, and lacks body shape parameters. SMPL-X extends SMPL with hand and facial joints and facial expression parameters, while our representation is a flattened 72D vector of joint angles and root translation, without any mesh or expression components. We will revise Section 3.1 to clarify this distinction.

[1] Agarwal, Niket, et al. "Cosmos world foundation model platform for physical AI." arXiv preprint arXiv:2501.03575 (2025).

[2] Movella Xsens MVN Link Motion Capture, https://www.movella.com/products/motion-capture/xsens-mvn-link

[3] Ma, Lingni, et al. "Nymeria: A massive collection of multimodal egocentric daily motion in the wild." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

Thanks for the responses. Overall, the authors have adequately addressed my main concerns. I am glad to keep my positive score.

We sincerely thank the reviewer for their thoughtful and constructive feedback, which helped us significantly improve the clarity and depth of our work.

1. Generalization to Unseen Subjects or Environments

We thank the reviewers for highlighting this important issue. We fully agree that evaluating generalization across subjects and environments is essential for embodied video prediction.

While our main experiments focus on within-subject generalization on Nymeria, we took several steps to ensure meaningful diversity in motion and scenario distribution. First, our train/eval split is designed to cover distinct sets of action episodes. The evaluation set contains many tasks and motion patterns not seen during training, including interactions with objects (e.g., balloons, boxes, mats), fine-grained hand manipulations, and multi-stage sequences.

For example, the evaluation video 20231214_s0_jeremy_allen_act5_m10nnd contains a full sequence of making a balloon animal, which involves a series of fine-grained, compositional actions that are not seen during training. These include:

• 3.35–3.47: pick and place, arranging materials
• 4.59–5.02: pumping air into the balloon
• 5.25–5.28: tying the balloon, stretching it
• 6.09–6.15: folding and twisting the balloon with both hands
• 9.01–9.04: additional pumping in a new context

These action combinations and their temporal structure are not present in training, yet the model generates coherent predictions—demonstrating generalization to unseen, complex, and multi-step interactions.

That said, while we believe the diversity in Nymeria is meaningful, it is inherently difficult to quantify how diverse is “diverse enough.” To further ensure generalization and provide a more systematic evaluation, we conduct cross-subject experiments on EgoBody [1], which features identities, body shapes, action styles, motion dynamics, and indoor environments that are entirely unseen during training. We evaluate PEVA and baselines on this dataset with single-step prediction, with completely novel scenes and activities.

PEVA-XL outperforms both DF-XL and CDiT-XL, matching LPIPS while achieving significantly better DreamSim and FID. The XXL variant further improves performance on DreamSim and FID.

These results suggest that PEVA produces semantically coherent and perceptually realistic future predictions, even under substantial distribution shift across subjects and environments.

Due to Program Chairs' guidelines, we are unable to include generated videos in the rebuttal, but we will include more visualizations of generated videos in the camera-ready version.

2. Use of Lower-Body Joints

Thank you for pointing this out. The model does in fact use full-body joints, including the lower body. The confusion stems from an error in how the degrees of freedom (DoF) were reported—we mistakenly excluded the contribution of lower-body joints in the DoF count. We will correct and clarify this in the revised manuscript. No joints were excluded in training or inference.

3. Efficiency: Training and Inference Time

We appreciate the request for computational details. The model was trained for a total of 57.9 hours on 16 H100 nodes, each equipped with 8 GPUs. For inference, the average time per frame is 23728 ms ± 207 ms, measured on a single A6000 GPU. We will include these numbers in the final version to clarify the model’s efficiency and real-time applicability.

4. Main Failure Modes, Challenges in Long-Term Prediction, and Future Directions

The model tends to fail in scenarios involving ambiguous egocentric motion, such as rapid head turns with no clear landmarks or asynchronous movement of body parts. These cases challenge the model’s ability to temporally disambiguate motion and infer global structure from a first-person view.

Moreover, long-horizon degradation is driven by error accumulation: small inaccuracies in early timesteps compound over time, leading to drift or overshooting. This is a common limitation of open-loop generative models. While some of this degradation is due to future uncertainty, it also reflects the lack of corrective feedback during inference.

In future work, we plan to address this via training in interactive environments that allow feedback, or by incorporating goal-conditioned or text-conditioned supervision to help anchor predictions over extended rollouts.

5. Qualitative Visualization

Due to Program Chairs guidelines, we are unable to include generated videos in the rebuttal. However, we have included extensive visualizations in the supplementary material—such as sampled frames, temporal rollouts, and attention maps —to illustrate the model’s behavior. We hope these help convey the qualitative aspects of the generated sequences. We will include more visualizations of generated videos in the camera-ready version and we are happy to provide additional video results in an online repository upon request if there is a change in the conference guidelines.

[1] Zhang, Siwei, et al. Egobody: Human body shape and motion of interacting people from head-mounted devices. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

We sincerely thank the reviewer for their detailed and insightful review. Your feedback is incredibly valuable. Based on your review, we have conducted additional experiments for clarity. We discuss each point in depth below.

1. Evaluation Beyond Image Space: Measuring Action-Following Accuracy

We appreciate the reviewer’s suggestion to evaluate action-following accuracy beyond image-level metrics. In response, we introduce two additional evaluations:

• Wrist Position Evaluation We compute PCK@0.2, precision, recall, and accuracy on 2D wrist keypoints across generated frames to measure local hand alignment.

• Pose Estimation Evaluation We estimate egocentric camera pose using VGG-T [2] and compare it to ground truth using ATE, RPE (Translation), and RPE (Rotation) [1].

These metrics directly evaluate whether the generated video faithfully reflects the conditioned action signal—both locally (e.g., visible hand positions) and globally (e.g., head-driven camera motion). PEVA outperforms all baselines, demonstrating its superior capacity for tracking embodied actions.

Wrist Position Evaluation across Models

Camera Pose Evaluation across Models

2. Necessity of Whole-Body Conditioning

We thank the reviewer for raising this important question. To investigate whether head and hand pose alone are sufficient, we conduct an ablation study using only the 6-DoF poses of the head, left hand, and right hand (relative to their previous states).

Although this subset captures visible body parts and head orientation, it leads to substantial degradation in both wrist and pose evaluations—highlighting the importance of full-body context.

Wrist Position Evaluation by Conditioning Type

Camera Pose Evaluation by Conditioning Type

These results indicate that non-visible body joints, such as those in the torso and legs, meaningfully affect egocentric visual dynamics—e.g., through head stabilization, balance, and intent. Full-body conditioning improves both spatial grounding and long-term consistency.

3. Lack of Qualitative Results / Videos

Due to Program Chairs guidelines, we are unable to include generated videos in the rebuttal. However, we have included extensive visualizations in the supplementary material—such as sampled frames, temporal rollouts, and attention maps —to illustrate the model’s behavior. We hope these help convey the qualitative aspects of the generated sequences. We will include more visualizations of generated videos in the camera-ready version and we are happy to provide additional video results in an online repository upon request if there is a change in the conference guidelines.

4. Clarity of Metrics and Terminology (e.g., Table 2 and Random Time Skips)

Thank you for this valuable feedback.

• Table 2: The evaluation metric is LPIPS (Learned Perceptual Image Patch Similarity), which measures perceptual distance between generated and ground truth frames. We will clarify this in the table caption and main text.

• Random Time Skips: Instead of training with a fixed timestep (e.g., every 0.5 s), we randomly sample time intervals during training and condition the model on the time delta. This improves generalization across short- and long-term predictions.

To assess the effect of random time skips, we compare models trained at fixed frame rates (0.5 Hz, 1 Hz, 2 Hz, 4 Hz) against one trained with random time skips. We evaluate 1-step predictions at 1, 2, 4, 8, and 16 seconds using DreamSim perceptual loss.

Ablation: Random Time Skips vs Fixed Rate (↓ DreamSim)

Fixed-rate models tend to overfit to specific temporal patterns and degrade at longer horizons. In contrast, random time skips encourage temporally invariant learning, improving both short- and long-term prediction quality.

We will clarify this motivation and experiment in the revised paper.

[1] Sturm et al. "A benchmark for the evaluation of RGB-D SLAM systems." IROS 2012

[2] Wang et al. "VGG-T: Visual Geometry Grounded Transformer." CVPR 2025

We thank the reviewer for the constructive feedback and for acknowledging the clarity of our model presentation and experiments. We appreciate the suggestions and have addressed the raised concerns as follows:

1. Novelty of the Contribution

In this work we address a new and underexplored problem: how embodied physical actions causally shape first-person visual experience over time. This problem is challenging, and so far has only been tackled with either simple action space (e.g., navigation) or in synthetic environments (e.g., computer games).

To address this, we proposed PEVA, a novel autoregressive video model that generalizes to variable context lengths and can condition on physical actions. PEVA outperforms recent state-of-the-art action-conditioned diffusion models like NWM and DF across perceptual metrics such as LPIPS and DreamSim. Additionally, in a hand-controlled motion task, PEVA achieves 85% PCK@0.1, compared to only 22% for the text-based model Cosmos, highlighting its ability to capture precise low-level control that language struggles to specify.

Unlike prior diffusion-based methods, which typically use flat or semantically abstract conditioning (e.g., text, labels, global motion), PEVA leverages hierarchically structured full-body kinematic trajectories—aligned with the human joint graph. This enables visual prediction driven by specific embodied actions (e.g., head tilts, shoulder rolls, fine arm motions), and supports:

• Disentangled action-perception analysis
• Atomic counterfactuals
• Goal-conditioned planning in pixel space

The novelty lies not only in the modeling technique but in proposing and solving a new class of controllable, embodied egocentric prediction problems.

2. Justification for Random Time Skips

We thank the reviewer for highlighting the need for clarification.

To evaluate the importance of random time skips, we conduct an ablation comparing models trained with fixed frame rates (0.5 Hz, 1 Hz, 2 Hz, 4 Hz) to our approach that was trained using random time skips. All models are evaluated on 1-step predictions at 1, 2, 4, 8, and 16 seconds using DreamSim perceptual loss.

Fixed-rate training often overfits to specific temporal dynamics and degrades at longer horizons. In contrast, random time skips encourage learning temporally invariant dynamics and improve generalization across both short- and long-term predictions.

We will clarify this motivation and experimental design more explicitly in the revised paper.

3. Visualization of Generated Videos

Due to Program Chairs guidelines, we are unable to include generated videos in the rebuttal. However, we have included extensive visualizations in the supplementary material—such as sampled frames, temporal rollouts, and attention maps—to illustrate the model’s behavior.

We hope these help convey the qualitative aspects of the generated sequences. We will include more visualizations of generated videos in the camera-ready version, and we are happy to provide additional video results in an online repository upon request if there is a change in the conference guidelines.