Thank you for the positive feedback and thoughtful questions. We provide answers to each question below.

Q1: Ablation study on the performance of FM and MGM model with different architectures.

We evaluated three Transformer architecture variants: DiT [Peebles et al. 2022 ICCV 2023], STDiT (our default), and CDiT [NWM Bar et al., CVPR 2025] - with both the FM and MGM models. All models were trained in a small-scale setting for one day on 32 A100 GPUs and we report FVD scores. As shown in the table below, Orbis-FM model consistently outperforms Orbis-MG across all three architectures in terms of FVD. We evaluated both methods using 12 and 30 inference denoising (unmasking for MGM) steps. The Orbis-FM model also shows relatively stable performance across different architectures, whereas Orbis-MG shows larger variance. ST architecture gives the best performance for both model types across different architectures. Overall, Orbis-FM with ST architecture achieves the best performance with 30 steps, as used in the paper. FVD scores are computed over 200 generated videos on the BDD validation set, each over 60 frame rollouts at 5 fps. We will include this ablation table in the revised version of the paper.

Table: FVD scores for small-scale versions of Orbis-MG (discrete) and Orbis-FM (continuous) with different architectures. The FVD is calculated over 60 generated frames on 200 samples of BDD validation set.

Q2 Additional metrics, such as inference speed, would be helpful for understanding the effectiveness of continuous tokens.

As shown in the table below, we compare the inference speed of our discrete Orbis-MG model and the continuous Orbis-FM model. We also report the GPU (VRAM) memory requirements for both methods during inference. Orbis-MG shows better inference speed compared to the Orbis-FM model. However, since Orbis-FM achieves significantly better video generation performance, it remains the default choice despite the speed advantage of the discrete model.

Table: Inference speed and VRAM requirements of the Orbis-MG and Orbis-FM models.

Thanks for the rebuttal. It has addressed some of my concerns. However, a few issues remain — particularly that the core modeling approaches (e.g., autoregressive, flow-matching, MaskGIT) are primarily adaptations of existing techniques rather than novel contributions. Therefore, I will maintain my original score.

Thank you for your positive feedback and thoughtful questions and comments. We address each question and provide detailed discussions below.

W1: It is not clear why the proposed method performs well for turns compared to other methods. Please add analysis or discussion to motivate why this happens.

• We speculated in the discussion section about two possible reasons for the failure of previous continuous driving models. We further investigated the hypothesis that parallel multi-frame prediction would impair long horizon generation, by training our smaller scale model to predict 12 frames at once (i.e. 2.4s ahead, like Vista and GEM). We found our model to work just as well with this setup, ruling out the hypothesis.

• We could not verify the impact of pretraining biases due to involved heavy compute cost. However, we speculate that prior driving world models such as GEM and Vista, which initialize using general-purpose Stable Video Diffusion model, tend to inherit biases from pretraining data lacking scenarios like sharp turns that are critical for driving tasks. We also observe that the Cosmos model tends to produce smooth, drone-like trajectories, which may similarly stem from biases in the training data.

• Additionally, we believe that it is not a single factor that boosts our model's performance. For instance, training with a flow-matching objective rather than a diffusion objective, such as the EDM used in Vista, can be advantageous. For example, previous works (e.g. Diffusion Autoencoders are Scalable Image Tokenizers, Cascaded Diffusion Models for High Fidelity Image Generation') show that v-parameterization (used in flow matching) is more robust than $\epsilon$-parameterization of diffusion: the most related is "GameNGen" (Valevski et al., 2024), where one focus is long-horizon roll-outs in gaming environments. We also incorporated techniques such as adding noise to the context, GameNGen (Valevski et al., 2024) and dropping the context. We observed adding noise to the context improves FVD from 280 to 246 in a small-scale setting.

• The exact reasons stay speculative, since direct comparisons that remove a single potential cause require training an entire new model, which costs 9k GPU hours.

W2: Suboptimal visual quality of dynamic objects relative to static background.

• We acknowledge that the current quality of the generated dynamics objects is suboptimal based on visual inspection. However, we emphasize that the dynamic agents tend to appear in plausible places at a plausible time and with a realistic speed. This suggests that their internal representation seems to be better than their generated visual appearance suggests. We believe that seeing more of these agents in the data and concurrently scaling the model is necessary to achieve higher quality. Another possible direction could be training a larger diffusion decoder post-hoc (i.e., independent of the predictive model) to enhance visual fidelity. However, even such a decoder would need to see more of these dynamic agents during training to be effective.

W3: Vista paper already had a proposed trajectory metric, this is not discussed and compared to the new trajectory metric.

• The trajectory metric proposed in Vista measures the control effect of action conditioning. We will give proper credit to this. However, our proposed trajectory metric operates at a distribution-level, evaluating the quality and realism of ego-motion in the generated videos without action conditioning (Table 3 in paper).

• We also evaluate the effect of conditioning on trajectories (Table 5 in paper) with the trajectory difference metric proposed by the authors of Vista, but we use a state-of-the-art off-the-shelf-model for inverse dynamics (VGGT, Wang et al. CVPR-25). This is consistent with the procedure used in GEM, where the SOTA DroidSLAM was used for inverse dynamics. We refer to both Vista and GEM while using this metric for evaluating action conditioning, in Line 296.

Thank you for your positive feedback and thoughtful comments. We provide detailed discussion for all comments and answer each question below.

W1: [Limited Theoretical Insights] Further insights into potential failure factors for other models and future directions.

• We speculated in the discussion section about two possible reasons for the failure of previous continuous driving models. Meanwhile, we further investigated the hypothesis that parallel multi-frame prediction would impair long horizon generation, by training our smaller scale model to predict 12 frames at once (i.e. 2.4s ahead, like Vista and GEM). We found our model to work just as well with this setup, ruling out this hypothesis. We will add this result to the paper.

• We could not verify yet the impact of pretraining biases and other architectural design choices due to a limited compute budget (every full training run costs 9k GPU hours). A few additional studies could be conducted in isolation in future work include: 1) comparing the v-parameterization of flow-matching with the $\epsilon$-parameterization used in of diffusion, as discussed in GameNGen paper [Valevski et al., 2024], and 2) investigating context augmentation strategies, such as injecting noise [Diffusion forcing Chen et al. 2024] or context dropout.

W2: [Generation Fidelity Limitations] Discussion on realism of the generated videos and model's planning ability.

• Fully agreed that we are still some distance away from the point where the world model matches the quality needed for planning and training an agent in real driving scenarios. Our work is just another step in this direction.
• We could only experiment at a resolution 288 x 512 due to a limited compute budget. Scaling up the resolution of this model input by $2\times$ should resolve some of the issues related to missing details. Also increasing the amount of training data would improve the model’s ability to capture more detailed state features. So far, our focus was on improving design choices before solving remaining issues by scaling the model, the resolution, and the data.
• In this work, our focus is on two key challenges in learning driving world models - long-horizon video prediction and generalization to complex real-world scenarios, such as turning maneuvers and urban traffic. Long-horizon prediction serves as a proxy for evaluating how well the world model learns state transitions, while generalization to complex scenarios shows the model's ability to capture a diverse range of real-world dynamics.

W3 [Insufficient Downstream Validation] Discussion on the scope of the paper.

Most existing visual world models which can perform downstream decision-making tasks are limited to restricted or simplistic environments such as Atari, DMLab, or Minecraft. It is feasible to perform closed loop tasks in such settings either due to simpler requirements from the perception module or from the constrained environment setting. So far none of the current real video-based driving world models (including ours) are ready yet to be used in a closed-loop setting. In our work we focus on building a driving world model with robust state dynamics, which allows handling of difficult situations like turning maneuvers or urban driving. We observe that prior methods often struggle in such non-trivial yet common situations in an open-loop setting. We argue that developing models with stable and realistic transitions is critical before they can be effectively trained for planning and decision making. Our additional metrics beyond short-term FVD are early pointers, which we think are relevant for indicating the readiness, such as the degradation rate with increasing roll-out length and diverse plausible trajectories derived from the predicted video.

W4: [Benchmarking and Baseline Clarifications] Comparison with closed-source GAIA-1 model

• GAIA-1 is not only closed-source; it is not even possible to run the model on new data. All evidence we have is from their paper and the few published videos. Thus, all discussion becomes quite speculative.
• GAIA-1 having a large number of parameters (6.5B) may provide an advantage in enabling long rollouts. Additionally, they used a 2.6B-parameter decoder to reduce flickering.
• Although it is not explicitly stated, the shift to a flow-matching objective in GAIA-2 may indicate the advantage of a continuous latent space over a discrete one for the GAIA model.

Thank you for the positive feedback and thoughtful questions. We address each question below and will incorporate the feedback in the revision.

L1 & Q2: Clarification on high-resolution terminology.

• The term "high resolution" in the paper may be misleading, as it is used only to distinguish between the lower resolution (256×256) and the higher resolution (512×288) in our experiments. We will revise this terminology to avoid confusion.
• The primary limitation for generating HD videos is the computational cost: training at HD resolution would be roughly six times more expensive due to having 6x more tokens, exceeding our current computational budget.

L4 & W2: Discussion on the purpose of the driving world model.

We think generating realistic synthetic data can be a useful application, but it is generally not the primary objective of world models. As demonstrated in a number of influential works (e.g. World Models Ha et al. 2018, Dreamer Hafner et al. 2019, Cosmos, VJEPA-2 Assran et al. 2025), the main purpose of world models is representation learning, planning, and policy learning. In this context, the quality of the state representation and the prediction of next states distribution is key. The decoded video is only one way to roughly assess to which degree this is successful, e.g. whether the model can successfully complete a turn without becoming unstable and without running into a steady state. The baselines we compare with (Vista, GEM, etc.) are conceived with planning as a possible application, however we observe that they often struggle in non-trivial yet mundane situations. In our work we prioritize building a world model with robust state representation and dynamics, which allows handling such situations. One could train a more sophisticated high-resolution diffusion decoder post-hoc to improve visual quality, but this is not the scope of this paper.

W3: Training compute budget and inference costs for Orbis and other models.

• Orbis training over 5 days on 72 A100 amounts to 9k A100 (40GB) GPU-hours. This is less than all baselines included in this work. For example, Vista was trained for 25k A100 (80GB) GPU-hours, DrivingWorld model for 18k A100-GPU-hours, and the general purpose Cosmos model for more than 20000k H100-GPU-hours.
• The table compares the average times (fps) needed to generate a frame, and the required GPU memory. Orbis has the best throughput compared to all other methods. This advantage can be attributed to the smaller size of the Orbis model. By parallelizing Orbis' computation in batches we can achieve an higher throughput. GEM and Vista are based on the same architecture but use different sampling protocols trading off FPS and VRAM.

Table: Inference speed and VRAM requirements for different methods. All numbers are calculated on a H200 GPU.

W1 & L3: Pixel-wise video quality metrics.

• We benchmarked Orbis using video quality metrics as requested. As shown in the tables below, we computed average PSNR and SSIM metrics over two shifted video windows of 10 frames (i.e. 2 seconds at 5fps) over 400 generated videos on the nuPlan-turns evaluation benchmark. We resized the generated videos of all methods to the same resolution for a fair comparison. Orbis performs marginally better than other methods over the first window. However, all methods converge to similarly low numbers in the second window.

Table: PSNR metrics computed over two shift windows (2 seconds each) from predicted frame 0 to 9 and frame 10 to 19.

Table: SSIM metrics computed over two shift windows (2 seconds each) from predicted frame 0–9 and frame 10–19.

• We would like to remark that pixel-wise VQA metrics like PSNR and SSIM are not well-suited for evaluating the quality of generated videos. These metrics rely on exact correspondence between the real and the generated video, which is not sensible, as large parts of the future are non-deterministic and can take different plausible paths, especially on a pixel level. This is reflected in the tables above, where all methods show similarly low numbers in the second window just after 2 seconds, both for PSNR and SSIM metrics.

• Additionally, the visual fidelity of our model can surely be improved using a more powerful decoder or refiner network can be trained post-hoc, without having to change the world model itself -- similarly to what is done in other works (e.g. Cosmos, GAIA-1).

Q5 & L2: Observations on longer video rollouts.

We are not allowed to provide longer clips for rebuttal due to the text-only format. In many cases, the model is able to produce longer drives, however, open-ended roll-outs expose two main failure modes: 1) the ego-vehicle stops at a red light, before an obstacle or a stationary vehicle, and does not restart. 2) the model hallucinates a scene change after too rapid shifts in context appearance. Restarting behavior becomes more common when training on more data. We observed this in previous scaling steps. The cause of the hallucinations is likely the limited context size. We agree that this is relevant for the community, and will add it to the discussion. We well also publish longer videos. Moreover, we will release the code and the model upon publication, so everybody can make all experiments they are interested in.

Q4: Orbis model loss function plot.

We are unable to provide a loss function plot due to the text-only rebuttal format. The training loss curve follows the same trend as Figure 13 in the DiT paper (Peebles et al. 2022). We can include the loss curve in the camera-ready version.

Q1 & Q3: Other clarifications.

• Our model predicts at 5 fps. Therefore, 20s clips include 100 frames.
• In the camera-ready version, we will label our method as "Ours" in Fig. 2 to prevent any potential confusion.