COME: Adding Scene-Centric Forecasting Control to Occupancy World Model

We thank the reviewer for recognition of novelty, writing and experimental results. We sincerely thank the reviewer for suggesting evaluating the proposed method under more diverse driving scenarios and adversarial conditions. We have supplemented initial verification on Occ3D-Waymo[1] and are currently working with the new dataset. Additionally, we have added analyses explaining how to ensure generated results closely align with input condition changes to meet simulation needs.

• [W1]: The method is only evaluated on nuScenes, which is relatively constrained in terms of driving diversity. Ego-motion is often limited to driving in-lane following the road, and the environment contains few truly complex or rare events. This limits insight into how the model might generalise to more diverse or interactive settings.

[A1]: nuScenes has limited diversity in ego-motion and complex objects, so we add experiments on the more complex Waymo Open Dataset (WOD). Occ3d-Waymo, built on WOD perception data, includes 798 training and 202 test sequences—richer and more diverse than nuScenes’ training scenes—with 10Hz sensor data and annotations. Our task is the same as nuScenes: predict 3s of future using 2s of past occupancy data at 0.5s intervals.

We retrain OCC-VAE, UNet, DOME [2], COME world model, and COME ControlNet on Occ3d-Waymo, initializing from nuScenes-trained components to save cost and time, with unchanged model sizes and training schedules.

Currently we report a COME variant with the same structure and with stronger geometric controls from the scene-centric forecasting module, differing by additionally replacing noisy BEV latents with UNet-encoded latents in future visible areas during denoising. This variant converges faster, yields slightly better quantitative results but risks reducing prediction multimodality. Due to computational limits, the default COME model requires a few more training days (Current status: mIoU=29.35 and IoU=46.46 in epoch 20/1000, already surpassing DOME and COME world model).

VAE reconstruction quality: mIoU=74.88, IoU=82.57.

The results of Waymo Occ3D are shown in the following tables. In general comparison (Table. 13) , our reproduced DOME achieves similar performance as in nuScenes dataset. COME world model has marginally better performance with DOME by 2.38 mIoU and 0.32 IoU. Scene-centric forecasting module outperforms the generative world model by a large margin (63% mIoU and 19.5% IoU). COME ControlNet, which built on control features provided with UNet module, maintains the same 3-s average mIoU and further improves 3-s average IoU to 23.7% improvement.

Table 13. Comparisons of general 4D occupancy forecasting performance between DOME and different stages of COME on Waymo Occ3D dataset.

We also compare the per-category 4D-occupancy forecasting performance of moving objects in Waymo Occ3D dataset in Table. 14. COME outperforms DOME on all moving categories by a large margin.

Table 14. Comparisons of 4D occupancy forecasting performance of moving categories between DOME and COME on Waymo Occ3D dataset.

To demonstrate the prediction of truly complex or rare events, we hope to further train and test our method on industrial-level private autonomous driving data and corner cases.

• [W2]: Furthermore, the paper does not explore the generation of fictional or counterfactual scenarios, which is a key promise of using world models for training and evaluation data. For example, it is unclear whether the model could simulate a crash scenario or how it would react to structural changes in the environment (e.g., the insertion or removal of a lane or a pedestrian crossing). It would strengthen the paper to evaluate or at least qualitatively discuss the model's capacity for reasoning about interventions or rare events, which are crucial for planning and decision-making.

[A2]: Thanks for pointing out the potential application of COME. World models for simulation typically use obstacle box positions and maps as control conditions, generating content that adapts to changes in these conditions. Our BEV Condition control scheme mirrors UniScene [3], which yields good rendering with customized BEV layout. Unlike UniScene, our method adds external controls for physics-consistent prediction, with adaptive scene-centric forecasting control (details in A3.2).

We test static (by randomly removing half map patches) and dynamic (by halving box-to-car center distances) scene distortions. In both, COME prioritizes BEV layout guidance rather than conditions from the scene-centric forecasting module, generating occupancy maps qualitatively matching the layouts. Per new NeurIPS guidelines, results can’t be visualized but will be added to supplements in revision.

• [Q1]: **[Q1.1]: Is it possible to verify that the method works on more diverse driving scenarios? **[Q1.2]: How does the model handle user-specified changes to the BEV-layout? **[Q1.3]: How does the model handle adversarial scenarios or scenarios where a crash is imminent?

[A3.1]: We verify the method on the Waymo Open Dataset, which has more diverse driving scenarios.

[A3.2]: The model accepts user-specified changes or adversarial scenarios via BEV-layout modifications, validated in UniScenes. A potential issue is conflicts between the Scene-Centric Forecasting module and BEV-layout control conditions (e.g., other vehicles' behavior conflicting with historical inertia). An elegant fix: mark adversarial simulation-focused agent areas, mask the corresponding Scene-Centric Forecasting results, ensuring only BEV-layout covers these regions to maintain global physical consistency and local customization.

[A3.3]: COME's simulation model takes adversarial BEV layouts (e.g., from adversial traffic simulators like Challenger [4]) as input conditions. When agents crash, COME merges them into a single occupancy instance. World models for recognizing and predicting such instances are left for future work.

[1] Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving, NeurIPS 2023.

[2] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024.

[3] Uniscene: Unified occupancy-centric driving scene generation, CVPR 2025.

[4] Challenger: Affordable Adversarial Driving Video Generation

We thank the reviewer for recognition of experimental results. We sincerely thank the reviewer for suggestions to enhance originality clarification. We supplement comparison experiments under identical camera and ego trajectory inputs, experiments with varying input conditions to demonstrate the effectiveness of scene-centric forecasting for downstream generation, and statistics on dynamic categories.

We respectfully disagree with originality. Prior works (DOME [10], OccWorld [1]) remain in ego-centric coordinates, suffering performance drop under perspective shifts. Our novelty lies in recognizing that **short-term scene-centric forecasting of moving regions is sufficient and necessary**; we are the first to formalize this as a ControlNet condition. The ablation in Table 10 shows that removing this component leads to a 31 % performance drop, confirming our nontrivial contribution and novelty. Below are our responses to other specific concerns:

• [W1]: The same occupancy prediction model with camera input should be used across all baselines for a fair comparison.

[A1]: Thank you for highlighting the need to address experiments with potentially different setups and inputs. We prepare two occupancy predictors that are the same as prior works and demonstrate COME's performance in Table 8. COME with both inputs receives the best performances.

Table 8. Comparisons of Camera-input occupancy world models

We add a quantitative comparison between DOME and COME with strictly the same input, as shown in Table. 9. Compared to DOME, COME has 17.5% mIoU and 8.5% improvement under BEV-Planner[11] ego trajectories, and 17.3% mIoU and 15.7% IoU improvement under ground-truth ego trajectories.

Table 9. Comparison with aligned camera input and ego trajectory.

• [W2]: Given many static voxels, what is the specific advantage of using predicted future occupancy prediction from a simple Unet architecture as a condition?

[A2]: The scene-centric forecasting module handles not only static voxels but also dynamic agents, retaining static voxels while accurately predicting future dynamic voxels. A key advantage is that pre-processing rigid body coordinate transformations simplify predicting from ego-centric to scene-centric views. Results for dynamic categories (Table. 11) demonstrate the module’s prediction ability.

We add an experiment (Table 10) where the COME world model is controlled solely by static voxels. This yielded marginal improvements (0.37 mIoU, 1.23 IoU), showing static voxels as controls are less effective than the proposed UNet module.

Table 10. Model performance when COME world model is controlled by static voxels.

• [W3]: How moving objects are accounted for in a world model?

[A3]: The scene-centric forecasting module perceives static surroundings and strongly predicts moving objects’ evolution, providing a geometric prior for generative models. We compare moving object prediction across the stage-1 world model, stage-2 scene-centric forecasting, and stage-3 COME ControlNet, listing IoU metrics for moving categories below. Table 11 reports the average IoU of moving categories across 6 future frames (0.5s to 3.0s). The UNet module outperforms DOME in predicting all moving categories, so using UNet as a prior significantly enhances COME-ControlNet’s dynamic object prediction.

Table 11. Comparison of model performance on dynamic categories on nuScenes dataset.

• [Q1]: L158 and L160: Definitions for D_t and D_τ are missing. -> This creates confusion regarding how the transformed sequence x' is handled. Are all sequential voxels concatenated into x' ?

[A4]: We will thoroughly rewrite the UNet section. D_t denotes the number of input historical frames, and D_τ the number of output future frames; both are set based on the task formulation. The transformation from x to x' is a simple rigid transformation: each voxel’s center point in x is converted to the current frame’s coordinates, resulting in x' having the same shape as x.

• [Q2]: Would directly using the transformed voxels as conditions yield a similar effect? This approach may be analogous to the use of warping in the image domain to leverage information from previous frames, as shown in [1]. [1] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion

[A5]: We add an experiment that all visible voxels are transformed and then inputted to COME ControlNet as conditions. Table 12 shows this variant improves marginally over the COME world model (0.88 mIoU, 0.81 IoU) but is far less effective than the UNet module.

Table 12. Model performance when COME world model is controlled by all transformed voxels.

[1] OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving, ECCV2024.

[2]Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction, CVPR2023.

[3] OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving, Arxiv 2024.

[4] FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation, Arxiv 2023.

[5] OccVAR: Scalable 4D Occupancy Prediction via Next-Scale Prediction, Openreview 2024.

[6] An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training, Arxiv 2024.

[7] Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Models, ICRA2025.

[8] RenderWorld: World Model with Self-Supervised 3D Label, ICRA2025.

[9] BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereo, AAAI2023.

[10] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024.

[11] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving, CVPR2024.

Thank you for the rebuttal. All of my concerns have been addressed. Therefore, I will raise my final rating.

We thank the reviewer for recoginition of novelty and experimental results. We sincerely thank the reviewer for pointing out clarification issues and suggesting potential improvements to the introduction and methodology sections. We have revised our draft accordingly to enhance its comprehensiveness. Below, please find our responses to other concerns:

• [W1.1]: Unclear motivation and applicability to real-world scenarios. The introduction lacks a clear explanation of why generating occupancy maps is critical for autonomous driving.

[A1.1]: Thank you for pointing this out. We will add explanations in the revised draft. Occupancy is widely recognized as a key intermediate representation for understanding drivable space in autonomous driving, effectively handling spatial perception of general obstacles. The generative occupancy world model has multiple potential applications:

(1) Predicting future occupancy grid maps based on planned trajectories and validating collision risks for the current trajectory [1].

(2) Serving as a critical intermediate result for synthesizing simulation data [2][3]. Generated occupancies can be rendered to the image perspective via Gaussian splatting [4], acting as accurate geometric controls for driving video generation.

• [W1.2]: More detail is needed on why perspective shifts from ego-centric systems are problematic, ideally with statistics or realistic examples.

[A1.2]: Perspective shifts in ego-centric systems are problematic due to the challenge of relocating visible dynamic agents and static environments. We classify scenes as having perspective shifts based on whether the vehicle moved more than 5 meters in 5-second prediction clips. We test DOME [5] and COME in these scenarios, with quantitative results in Table 7. DOME performs well in ego-static scenes (where other agents are static) but poorly in ego-dynamic ones, achieving only 47% mIoU and 52.9% IoU (vs. ego-static). In contrast, COME shows larger improvements in ego-dynamic scenes (36.3% mIoU, 26.88% IoU) than in ego-static ones (8.8% mIoU, 4.2% IoU), demonstrating the effectiveness of our scene-centric forecasting control for the world model.

Table 7. Model performance on scenes classified according to ego movement.

• [W2]: Some lack of clarity and organization in the writing: 46-57: This section is difficult to follow, especially for readers unfamiliar with the methods. These details would fit better in the methods section. 58-61: Discussion of input sources and prediction horizons is misplaced in the introduction. 39-45, 66-68: Repeatedly foreshadowing SOTA results feels redundant. 153-163: The purpose and operation of the scene-centric forecasting UNet are not explained clearly enough.

[A2]: Thanks for pointing out paragraphs that are hard to understand. We will revise the manuscript based on the instructions above. For l46-57, we will simplify the module introduction and place the original paragragh in methodology. For l58-61, we will place the test details in the experiment section. For l39-45, we will delete the redundant description. For l153-163, we will thoroughly rewrite this section to clarify its purpose and operation.

• [W3]: Minor editing issues: 134: Periods for e.g. 543: Confusing sentence.

[A3]: Thanks for pointing out editing errors, we will check and correct editing errors in the manuscript. Line 543 should be revised as: We try to perform masks on conditions before COME ControlNet rather than masks on controls after COME ControlNet, but the qualitative results are poor, so we do not pose the masks in condition features.

• [Q1]: 161-163: Can you explicitly explain what inputs the scene-centric forecasting module receives and what it is predicting? Additionally, how are these predictions propagated to future timesteps? This section is not entirely clear.

[A4]: The scene-centric forecasting module takes historical occupancy sequences and ego poses as input. Historical occupancy sequences are transformed to the current ego pose, resulting in semantic occupancy sequences of shape [D_t, H, W, L] under the same coordinate system. The module outputs future scene-centric occupancy sequences of shape [D_τ, H, W, L], which are concatenated with the input sequences to form a tensor of shape [D_t + D_τ, H, W, L]—serving as the input condition feature for COME ControlNet.

[1] Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving, AAAI 2025.

[2] Uniscene: Unified occupancy-centric driving scene generation, CVPR 2025.

[3] Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation, CVPR2025.

[4] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, ACM Transactions on Graphics (TOG) 2023.

[5] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024

We thank the reviewer for recoginition of clarity and experimental results. We sincerely thank the revieweer for the insightful comments. We supplement more comparisons between proposed COME and the baseline DOME[1], including computational cost and latencies, ablations on hyper-parameter, area partition, ego trajectories. Moreover, Occ3d-Waymo[2] is not used by prior occupancy world models as a 4D occupancy benchmark. We supplement the comparison of DOME and COME on Occ3d-Waymo. Below are our responses to other specific concerns:

• [W1]:Increased computational cost: Compared to Baseline, COME’s combined UNet + DiT + ControlNet pipeline raises GFLOPS from ~1,375 to ~2,066, challenging real‑time deployment. Provide a comparison with the GFLOPS of DOME.

[A1]: In comparison to the baseline DOME, our COME approach incorporates an additional lightweight UNet and a relatively heavy ControlNet, naturally leading to an increase in computational cost under the same world model settings. To highlight the effectiveness of our method, we introduce COME-small, a streamlined variant with only 222.7 GFLOPs—substantially less than both DOME and the full COME model. Specifically, we reduce the number of attention heads and hidden dimensions by half and decrease the transformer blocks from 28 to 12. As shown in Table. 1, COME-small outperforms DOME even with 80% fewer computations, demonstrating efficiency without compromising performance. We sincerely thank the reviwer for the insightful comments and will include the results in our new draft.

Table 1: Parameters and computational complexity of DOME and COME variants.

• [W2]:Narrow evaluation scope: Experiments are confined to nuScenes‑Occ3D.

[A2]: In our initial draft, COME was evaluated solely on occ3d-nuscenes following the settings of the baseline approaches. However, we acknowledge the feedback from reviewers (2AyB and Gif3) that evaluating on a new benchmark would enhance the study. Consequently, we have extended our evaluation to include occ3d-waymo. Please refer to our response to the reviewer Gif3 for our detailed settings and experimental results. In summary, COME consistently outperforms the baseline in this new benchmark.

• [Q1]:Inference Latency & Throughput: What are the end‑to‑end inference times and achievable frame rates for 10, 20, and 40 denoising steps.

[A3]: Thanks for the comment. We report the results on Table. 2. In a nutshell, the overall time is proportional to the number of denoising steps, and lower latency can be achieved with smaller models.

Table 2: Latencies of models with different denoising steps.

• [Q2]: Visibility Mask Sensitivity: How sensitive are final forecasts to the visibility‑mask threshold \varepsilon ? Please provide an ablation study over a range of \varepsilon values.

[A4]: Table. 3 below presents the ablation study on the ($\varepsilon$) values. When ($\varepsilon = 0$), meaning only visible BEV patches are used for control, the performances are not satisfying. Conversely, when ($\varepsilon = 1$), utilizing all BEV patches, the model achieves peak performances. However, as illustrated in Fig. 8 of our draft, this setting leads to a diminished ability to imagine invisible regions due to the injection of control signals. Therefore, we employ ($\varepsilon = 0.5$) in our paper for a balance between model performance and imaginative ability.

Table 3: Ablations of hyper-parameter $\varepsilon$ of COME.

• [Q3]: How is the performance of DOME in Tables 2, 3b and 4c? Is the conclusion consistent with COME ablation study?

[A5]: Yes, the conclusion aligns with our ablation study. We provide additional results in the table below. In the first two tables (Table. 4 and Table. 5), our implemented base world model currently performs worse than DOME despite having similar structures, as we only introduced extra skip connections. However, the COME counteract this issue, resulting in significantly better performance compared to DOME. We anticipate that if our base world model could replicate DOME's performance, our COME would further amplify its advantages.

Table. 6 illustrates the performance across different trajectories. Both COME and DOME exhibit similar trends when settings are modified. When ground-truth occupancies align with predicted trajectories, DOME experiences a more significant decline (-2.77 mIoU and -2.73 IoU) in performance compared to COME (-0.23 mIoU and -0.44 IoU), demonstrating COME's superior geometric consistency under perspective shifts.

Table 4: Ablation studies of model performance in various stages (Tab. 2 in manuscript).

Table 5: Model performances under different computational complexity. (Tab. 3(b) in manuscript).

Table 6: Effects of the used trajectories.. (Tab. 4(c) in manuscript).

[1] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024.

[2] Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving, NeurIPS 2023.