We greatly appreciate Reviewer gRv6 for providing a detailed review, insightful questions and forward-looking feedback.

---

Q1: Utility of Generated Videos

First, SAMPO is designed as a control-oriented world model, where the goal is to produce short but accurate rollouts to support planning and reinforcement learning, rather than generating long-horizon videos. This design choice aligns with the nature of robotic manipulation tasks in BAIR and RoboNet, which typically involve short-horizon interactions in constrained tabletop environments with fixed camera views.

Second, the brevity of predicted sequences does not diminish their utility. In practice, 10 to 15-frame predictions from 1–2 context frames have proven sufficient for downstream tasks like visual planning (Tab. 3) and MBRL (Fig. 5). SAMPO also supports action-conditioned generation, reward modeling, and exhibits strong zero-shot transfer to unseen manipulation tasks, without being restricted by sequence length. Thus, even short clips are effective for training and deploying intelligent agents.

Third, generating longer sequences introduces greater uncertainty due to error accumulation, a challenge that remains an open problem in long-horizon prediction. Instead of relying on long open-loop rollouts, our method emphasizes producing short but accurate predictions that are recomputed at every step. This approach is consistent with many prior works in Model Predictive Control (MPC) [1, 2].

Nevertheless, we agree that extending prediction horizons and modeling more diverse motion patterns, such as mobile manipulation and outdoor scenarios, is a valuable future direction.

[1] Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv 2018.
[2] Deep Visual Foresight for Planning Robot Motion. ICRA 2017.

Q2: Frame Rate & Duration

The temporal length of our generated videos follows the standard rollout settings used in prior work (iVideoGPT, MAGVIT, MaskViT, etc.). Specifically, the BAIR dataset operates at 20 FPS, so predicting 15 frames results in a 0.75s video. In contrast, RoboNet and 1X operate at 10 FPS, and our setting of 10 predicted frames yields a 1.0s video.

Q3: Large Variation in Success Rates

Based on our understanding and experiments, the performance gap across VP2 tasks is mainly due to:

• Task complexity. Tasks like Push blue Button are visually explicit, while others like Open Slide require fine-grained contact modeling and involve occlusion, making them harder to model.

• Visual ambiguity. At low resolution, it is often unclear whether key interactions (e.g., contact) occur, leading to underestimation of visually plausible predictions.

• Noisy reward evaluation. VP2 relies on pretrained classifiers rather than ground-truth rewards, which can misjudge success and amplify inter-task variance.

The "100%" aggregate score for the simulator baseline results from normalizing each model's average task success by the simulator's average, making the simulator's score always 1.0 (100%). This follows the evaluation convention in [3]. We will clarify this normalization step in the revised version, both in Sec. 4.2 and the captions of Tab. 3.

[3] A Control-Centric Benchmark for Video Prediction. ICLR 2023.

Q4: Occluded Red Small Object

The key comparison lies in motion consistency. In iVideoGPT's result (Fig. 4, second row right), the robot arm's predicted trajectory deviates from the ground truth, causing occlusion of the red object and indicating a failure in maintaining accurate object-track continuity. In contrast, our method preserves spatial coherence between the robot and the object, avoiding unnatural occlusion and ensuring consistent interaction. This demonstrates that our model generates more realistic and task-consistent motion predictions.

Q5: Plans in Real-world

The experiments reported in Sec. 4.2 (Visual Planning and Reinforcement Learning), though conducted in simulation, are performed under low-level control settings, bridging the gap to real-world deployment. Although new hardware experiments are not feasible within the rebuttal period, we are actively transitioning our system to real robots.

Inspired by recent real-world deployments of world models (e.g., CoTVLA, WorldVLA), we believe SAMPO is well-suited for deployment on physical robots. Concretely, we identify two integration paths:

• Future-State Priors for VLA Policies: SAMPO can be integrated with Vision-Language-Action (VLA) models like RT-2 and OpenVLA by providing short-horizon predictions that help ground actions in anticipated dynamics. This improves motion consistency and decision reliability.

• Synthetic Rollouts for Data Augmentation: SAMPO generates physically coherent future states, enabling efficient data augmentation for robotic policy learning.

We will highlight these real-world deployment strategies in the revision to better convey SAMPO’s practical impact. We sincerely appreciate your forward-looking feedback!

Dear Reviewer gRv6,

We sincerely appreciate your time and thoughtful consideration of our responses.

Value of action-conditioned world model

We clarify that both the BAIR and RoboNet datasets are collected by executing random actions, which naturally results in a diverse set of short trajectories. Some clips contain only the robot arm moving in free space, while others involve contact with objects that lead to visible changes in the scene. This diversity is intentional and follows the standard setup used in prior work [1, 2, 3, 4]. A general-purpose world model should be able to predict the outcomes of arbitrary actions, including both contact-free and contact-rich interactions.

We believe this design choice is meaningful. Accurate predictions in both types of sequences are essential for building a reliable world model. This capability is especially important when the model is paired with Model Predictive Control (MPC), where many action candidates are evaluated at each step. Even small motions, such as moving the arm a few centimeters without immediate contact, must be modeled precisely to support safe and effective downstream control. From this perspective, the ability to handle a wide range of dynamics is not a weakness, but a necessary property for real-world deployment.

To avoid potential misunderstanding, we will include additional examples in the revised version that show contact-induced changes in the scene, further illustrating the model’s ability to handle diverse types of motion.

Motion consistency

Thank you for your continued attention to the visualization in Fig. 4. We believe there may be a misunderstanding in the interpretation of this example. As shown in the top row (ground truth), even at frame T = 8, the red object in the background remains clearly visible and unoccluded. In contrast, iVideoGPT, when conditioned only on the first two observed frames (T = 0, 1), produces a prediction that already deviates significantly from the ground-truth trajectory as early as T = 2. By T = 4, the robot arm has shifted such that it incorrectly occludes the red object, which is a clear artifact of erroneous motion prediction.

This is not an incidental occlusion due to camera perspective, but rather a consequence of inaccurate trajectory modeling. The predicted arm motion in iVideoGPT is misaligned with the observed dynamics, leading to premature overlap with the red object and diverging from the intended action sequence. In contrast, SAMPO maintains a trajectory that closely matches the ground-truth motion throughout the rollout, preserving both the interaction intent and visual layout consistency.

This example is representative of a broader trend: our method yields more physically plausible and temporally coherent predictions under action conditioning. A quantitative result in RoboNet is shown below, and additional qualitative results demonstrating this consistency can be found in Appendix Fig. 15. We will revise the manuscript to better emphasize this point and prevent further confusion in interpreting the example.

We hope this response addresses your remaining concerns, and we remain open to further discussion. Thank you again for your time and effort.

Best regards,

Authors

---

[1] iVideoGPT: Interactive VideoGPTs are Scalable World Models. NeurIPS 2024.
[2] MAGVIT: Masked Generative Video Transformer. CVPR 2023.
[3] MaskViT: Masked Visual Pre-Training for Video Prediction. ArXiv 2022.
[4] MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation. NeurIPS 2022.

We greatly appreciate Reviewer 2sYs for the careful reading of our paper and constructive comments in detail. We also appreciate the pointed-out typos, which we will correct in a future revision.

---

Q1: Clearer Comparison

We clarify that SAMPO is fundamentally different from iVideoGPT, rather than a straightforward extension with incremental components. As detailed in our response to Reviewer cg6d (Q1), SAMPO introduces a novel hybrid autoregressive framework that performs multi-scale bidirectional spatial modeling within frames while maintaining causal temporal modeling across frames. This design addresses key limitations in iVideoGPT, including the loss of spatial structure and inefficient inference due to raster-scan decoding. In addition, the asymmetric tokenizer and motion prompt modules are introduced to tackle challenges specific to action-conditioned world models. These motivations are elaborated in L174–179 and L190–193 of the main text. We will make the distinction between the two frameworks clearer in the revised version.

Q2: AR vs Hybrid

Following the decoder-only architecture implemented in VAR, we designed our hybrid autoregressive framework to support scaling across model sizes. Due to the time constraints of the rebuttal period, we were unable to retrain the asymmetric multi-scale tokenizer and hybrid AR model from scratch under strict parameter parity. However, we present the following results from existing experiments with motion prompt disabled, which provide partial insight into parameter efficiency.

SAMPO-B uses 25% fewer parameters than iVideoGPT while achieving slightly better performance in both FVD and SSIM. This suggests that the hybrid autoregressive framework improves parameter efficiency. Furthermore, SAMPO supports larger-scale deployment via its decoder-only architecture, as evidenced by the even stronger performance of SAMPO-L. Based on our empirical results, the hybrid AR design appears to reduce computational overhead rather than increase it. Unlike token-level autoregression, which suffers from sequential decoding bottlenecks, our intra-frame bidirectional attention mechanism is fully parallelizable. As demonstrated in the ablation study provided in response to Reviewer Auas (Q3), the hybrid AR framework not only improves modeling capacity but also enhances runtime efficiency, making it particularly suitable for scalable world modeling.

Q3: Trajectory Rollout Error

We thank the reviewer for the thoughtful question. While our current submission does not explicitly include quantitative plots showing how the prediction error evolves over the rollout horizon, we provide qualitative and indirect evidence of temporal stability in Fig. 4 and Fig. 15 (in Appendix). Compared to prior autoregressive baselines such as iVideoGPT, SAMPO demonstrates consistently better spatial-temporal fidelity across extended horizons, particularly in long rollout settings like RoboNet and 1X World Model (10 frames). This can be attributed to our hybrid spatiotemporal autoregression and the use of coarse-to-fine generation, which better preserves high-level motion early on and gradually refines details at finer scales. In contrast, we observed that standard raster-scan autoregressive models tend to accumulate more low-level noise over time.

While qualitative visualizations indirectly reflect temporal degradation, we agree that directly plotting error versus timestep curves would provide more intuitive insight. We intend to include such visualizations in the revised version. If including the full curve in the main text is not feasible, we will highlight this analysis and treat it as an important direction for future work, aimed at better assessing and improving long-horizon prediction stability.

Q4: Motion Prompt Usage

We apologize for insufficient clarity in our experimental setup. In BAIR experiments (1 context frame), we omitted motion prompts to ensure a fair comparison with baseline methods. In RoboNet experiments (2 context frames), we utilized CoTracker to extract point trajectories from the first two frames. The extracted trajectories were filtered by motion magnitude and tracking confidence, and subsequently projected onto the image plane. We will clarify these details in the paper to avoid confusion.

We would like to sincerely appreciate Reviewer Auas for the comprehensive review and insightful questions.

---

Q1: Novelty Reformulation

We emphasize that SAMPO introduces a novel hybrid autoregressive architecture , which seamlessly unifies intra-frame scale-wise generation with inter-frame causal modeling, tailored for efficient and action-conditioned world modeling.

Although inspired by the coarse-to-fine decoding paradigm in VAR, SAMPO is not a direct adaptation; rather, it is a task-specific redesign that extends the idea from static image generation to temporally causal, action-driven generative rollouts in robot-centric settings. Moreover, SAMPO introduces an asymmetric multi-scale tokenizer (Sec. 3.2), which encodes observed frames densely to preserve semantic context and future frames sparsely to focus on dynamic content. This design significantly improves inference efficiency while retaining visual fidelity (Tab. 5), and is not addressed by VAR, which applies uniform scale-wise encoding without considering temporal asymmetry or efficiency in rollout.

We empirically observed that naively extending VAR's residual decoding to sparse future frames yields poor reconstruction quality. To address this, we devised a scale-specific weighting scheme during training (Appendix A.2), which adaptively balances loss contributions across scales. This adjustment is crucial for stabilizing generation when tokenizing future frames sparsely and ensures that both coarse dynamics and fine details are faithfully captured across rollout steps.

Regarding motion prompts, SAMPO and TraceVLA differ in both objective and mechanism. TraceVLA overlays offline trajectories as visual prompts to enhance policy learning but lacks a generative component. By contrast, SAMPO integrates motion trajectories directly into the generation pipeline via a trajectory-aware prompt module (Sec. 3.3), which provides spatiotemporal priors that guide the model toward dynamic regions during rollout. This integration improves temporal consistency and physical realism, as confirmed by both quantitative gains (Tab. 4) and qualitative results (Fig. 4).

We hope this clarifies that SAMPO is a task-specific, modular redesign that advances beyond prior works by unifying coarse-to-fine generation, autoregressive rollout, and dynamic prompting in a single, scalable framework for robot-centric world modeling.

Q2: Clarified Motivation

As described in Lines 289–292, our residual quantization scheme assigns each scale to model the residual between its coarser‑scale prediction and the ground truth. This decomposition naturally delegates large, structural appearance shifts in space and time to the coarse scales (e.g., scales [1–2]), which capture the dominant dynamics, such as an arm moving from left to right. The finer scales (e.g., scales 6 or 8) then refine the output by adding texture and spatial detail. This principle aligns with the perceptual observation that motion signals often reside in low-frequency components (as also supported by [1, Sec. 3.1]; [2, Sec. 3.2]) and that residual quantization naturally enforces a progression from global to local dynamics. In robotic tabletop settings, where backgrounds are mostly static, such separation is especially effective. As shown below, modeling coarser spatial scales (e.g., [1–4]) improves dynamic understanding, while removing coarse levels leads to worse performance.

Although we predict only a subset of finer-scale tokens, each frame is still represented by 91 tokens (e.g., scales=[1,2,3,4,5,6]), which is significantly richer than prior work (e.g., iVideoGPT's 16-token setup). These predicted tokens can be directly reused as input for the next step in the rollout. Moreover, Tab. 5 shows that this configuration achieves a strong balance between speed and fidelity, and supports visual planning and RL tasks where rollout is essential. We plan to clarify these design choices in the final version.

[1] Generating Diverse High-Fidelity Images with VQ-VAE-2. NeurIPS 2019.
[2] Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample. NeurIPS 2020.

Q3: Additional Ablations

We conducted an additional ablation study to assess how the number of predicted spatial scales affects video quality and downstream planning performance, following the setting of Tab. 5 and Tab. 3. Due to time constraints, we conducted two additional configurations and did not re-train a full-scale encoder ([1–16]) as it is computationally expensive.

We find that using [1–5] scales achieves comparable planning success as [1–6], with slightly worse video quality. Reducing to [1–4] leads to a moderate drop in fidelity, but still captures the key dynamics. The primary difference lies in texture detail, not motion quality. Notably, [1–4] achieves 3.2× faster rollout. This highlights a clear trade-off: fewer scales improve efficiency, while more scales enhance visual quality. We believe this flexibility allows users to choose the best setting for their application. We will include this result in the revision.

Q4: Clarified Details

Zero-shot experiments setup. Following the setting used in iVideoGPT, Fig. 6 reports results from an action-free zero-shot setting, as further detailed in Appendix Fig. 13 for clarity. We will update the caption and main text to avoid confusion. We adopt this design because cross-robot generalization under action conditioning is inherently difficult, primarily due to heterogeneous embodiment. Different robots often differ in action space dimensionality, control semantics, and actuation properties. As discussed in our response to Reviewer cg6d (Q3), transferring actions across such differences remains a significant challenge. By removing the need for action tokens, we avoid the action-space mismatch and instead evaluate the model's ability to generalize visual dynamics such as object trajectories and arm motion across varied robot embodiments and manipulation scenes. We agree that exploring action-conditioned zero-shot generalization is a valuable future direction. In future work, we plan to investigate action-space alignment and conditioning strategies to support this goal. We will clarify this design decision in the revised manuscript.

Reward Prediction Module. In our implementation, reward prediction is integrated into the transformer's token-level modeling pipeline. After tokenizing video frames and inserting start tokens [S], each action embedding is linearly projected and added to the corresponding [S] token. We attach a lightweight linear head to the hidden state of the token following [S] and the compressed visual tokens, mapping it to a scalar reward via regression. This head is trained jointly with the next-token prediction loss using an MSE objective, enabling frame-aligned, token-synchronized reward prediction without auxiliary branches. In MBRL tasks such as Meta-World, SAMPO can directly generate both future observations and rewards, enabling data augmentation in the replay buffer. This simplifies the coupling between world model and policy learning, and supports efficient, rollout-based policy optimization.

Q5: Real-world Experiment

We apologize that the limited hardware access during the rebuttal period, we were unable to conduct real robot evaluations. However, we plan to include such evaluations in future work by estimating the expected rewards of real-world robot policies using SAMPO and comparing them with actual execution outcomes. Prior work such as DreamerV3 has demonstrated that learned world models can simulate imagined trajectories for offline policy learning and evaluation. Given SAMPO's strong performance in both video prediction and planning, we believe it is well-suited for this purpose, and we will report results in future revisions. We greatly appreciate Reviewer Auas for this valuable suggestion.

We sincerely thank Reviewer cg6d for providing a detailed review, valuable suggestions, and a positive evaluation of our work.

---

Q1: Inherent Complexity

We would like to clarify that SAMPO is intentionally designed to be modular and extensible, serving as a flexible framework rather than a rigid system.

The core of SAMPO lies in its hybrid autoregressive framework, which unifies spatial and temporal modeling via scale-wise generation and causal decoding. This design directly mitigates the spatial degradation and temporal drift seen in earlier AR world models. In fact, several components of SAMPO, such as the tokenizer and the prompt module, can be replaced or extended. For instance, the asymmetric tokenizer can be substituted with 3D VQ-VAE, MaskViT-style tokenization, or ViT-VQ, while the motion prompt can incorporate trajectory embeddings or CLIP-style visual goals. Additionally, the causal modeling can be upgraded with diffusion-style decoders to improve sample diversity without compromising consistency [1].

As evidenced in Tab. 4, these loosely coupled, problem-driven modules make SAMPO a flexible foundation that readily absorbs advances from diverse generative paradigms. Consequently, SAMPO remains compatible with emerging world-modeling approaches while providing a unified structure for systematic comparison and improvement.

[1] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. ArXiv 2025.

Q2: Sparse Assumption

Our asymmetric tokenization design is based on the nature of our data: large-scale robotic manipulation datasets (e.g., Open X-Embodiment), where most scenes are tabletop, with fixed camera views and limited motion from the robot arm and manipulated objects. In such settings, the assumption of largely static backgrounds is valid, and sparse tokenization allows the model to focus on dynamic regions without wasting computation on redundant, static areas.

Even in highly dynamic scenes, maintaining sparsity remains essential for ensuring computational tractability. Full-resolution tokenization over multiple future frames quickly becomes infeasible in terms of memory and latency. Our method provides a flexible framework for dynamic environments by focusing tokens on motion-related regions. This approach is aligned with recent findings in video generation, such as [2], which demonstrate that spatial-temporal sparsity can retain high visual fidelity even in complex scenarios. Moreover, our design is compatible with adaptive refinement strategies. For instance, techniques like progressive token refinement [3] or adaptive spatial masking [4] could be incorporated into our framework to further improve the modeling of fine-grained spatial details in dynamic regions. We believe that integrating such mechanisms with our asymmetric tokenization would allow SAMPO to scale gracefully to more challenging domains.

[2] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. ICML 2025.
[3] PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models. CVPR 2025.
[4] AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders. CVPR 2023.

Q3: Scalability Beyond Compute

We argue that building scalable world models, particularly in the context of robotics, requires more than simply increasing model size or dataset volume. Below, we outline several key factors beyond cost that we consider essential for building more robust and generalizable systems:

1. Heterogeneous embodiment and control abstraction. Robotic datasets are often heterogeneous in embodiment (e.g., arm type, degrees of freedom, gripper morphology) and control interfaces. To scale beyond specific robot morphologies, SAMPO's architecture must accommodate variable control tokens and state embeddings. Future extensions can integrate low-rank control factorization or policy-conditioned dynamics modeling to generalize across robot bodies without explicit retraining.

2. Viewpoint and context invariance. Scaling to real-world deployments requires robustness to diverse camera viewpoints, lighting, and scene compositions. We believe this demands view-invariant tokenization and context-decoupled motion representations. The SAMPO framework can naturally support such invariances by conditioning future frame tokenization on shared multi-view context (e.g., via cross-attention), allowing consistent dynamics modeling across cameras.

3. Curriculum pretraining and transfer. We advocate a structured curriculum that begins with Internet-scale open-domain videos to acquire basic physical knowledge, progresses to first-person human-object interaction data (e.g., EPIC-Kitchens) to ensure operational diversity, and finally specializes on robotic datasets. This staged progression aligns with the "Data Pyramid" proposed in [5].

4. Imagination fidelity under open-world uncertainty. As world models are used for planning and long-horizon reasoning, scaling must also address uncertainty estimation and semantic consistency. We envision extending SAMPO with retracing rollouts, behavioral priors, or diffusion-based latent refinement to enhance fidelity under distribution shift, enabling safer policy training and evaluation in the wild.

In summary, we view scaling not as a brute-force endeavor, but as a systems-level challenge involving representation, generalization, and modularity. SAMPO's design opens up promising avenues in all three directions, and we plan to investigate these dimensions in future iterations.

[5] GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. ArXiv 2025.