Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving

I appreciate the authors' efforts in clarifying the issues in the rebuttal. I would like to raise the score to 8.

Furthermore, the model performance on the RayIoU metric also helps us understand why our model may not outperform baseline models in some categories. Our method do not achieve the SOTA performance in five categories, all of which are concentrated in the large static categories. The best performance in three of the categories is achieved by RenderOcc [2], which, as mentioned earlier, has a tendency to predict thicker surfaces for better mIoU scores. Here, we provide a visualization result comparing our model with RenderOcc and SparseOcc: https://www.hostize.com/v/Ta5ND-IuCV

As shown in the figure, RenderOcc can not generate reasonable scene structure even with a high mIoU score. The remaining two SOTA results are achieved by OccFlowNet [3]. Given that OccFlowNet's model framework is very similar to RenderOcc, we can reasonably speculate that OccFlowNet also faces similar issues. Additionally, OccFlowNet is not part of publicly published works in academic journals or conferences.

Therefore, we have surpassed all published baseline methods in almost all categories and explained why we may not perform as well as RenderOcc on this metric, for our PreWorld tends to generate a more reasonable scene structure.

[1] Liu, et al. "Fully sparse 3d panoptic occupancy prediction." ECCV 2024.

[2] Pan, et al. "Renderocc: Vision-centric 3d occupancy prediction with 2d rendering supervision." ICRA 2024.

[3] Boeder, Simon, Fabian Gigengack, and Benjamin Risse. "Occflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow."

But one remaining question is pre-training on 700 scenes and downstream to 700 scenes brings more (or comparable) improvement (0.9 mIoU) than downstream to 450 scenes (0.87 mIoU), which is a bit confusing because generally as the ratio of pre-training data to fine-tuning data increases, the performance gain also increases. Is there any analysis on this?

Thank you for bringing this up. We believe the reason for this still lies in the fact that, for large categories, existing evaluation metric does not adequately reflect the model's actual performance. Therefore, we have detailed the corresponding mIoU for large categories and small objects as follows:

It can be observed that the mIoU for large categories does not always effectively reflect the performance improvement of the model. For the model fine-tuned with 450 scenes, pre-training leads to a 0.64 increase in mIoU for large categories, while the model fine-tuned with 700 scenes sees an increase of 0.89. In contrast, the increase in mIoU for small objects can better reflect the effectiveness of pre-training, aligning with the expectations: 2D pre-training yields more significant performance improvements for smaller 3D fine-tuning datasets.

In order to better showcase the effectiveness of pre-training, we use RayIoU as the evaluation metric, and the results obtained are as follows:

When using RayIoU as the evaluation metric, the improvements in overall RayIoU and RayIoU for large categories follow a similar trend, indicating that as the scale of the 3D fine-tuning dataset increases, the benefits of 2D pre-training do indeed gradually diminish.

The performance improvement is minimal. For example, the mIoU improvment for 3d occupancy prediction is only about 2% compared to OccFlowNet. And there is no significant improvement in other experiments and metrics

It is important to note that OccFlowNet [9] is not part of publicly published works in academic journals or conferences. We included it in the comparison just to ensure a more comprehensive presentation of occupancy-related works. In comparison with published works like SparseOcc, our model achieved around a 12% performance improvement.

On the other hand, we believe that the current mIoU metric does not effectively showcase the advantages of our method. Common practices use visible masks during evaluation, calculating mIoU only for voxels within the visible region. As a result, existing models tend to predict thicker surfaces for large static categories, such as driveable surface and manmade. Since the current metric does not penalize predictions outside the visible region, models often achieve higher scores through this strategy. However, this can result in the generation of unreasonable scene structures in predictions. Here we provide a visualization result comparing our model with RenderOcc and SparseOcc: https://www.hostize.com/v/Ta5ND-IuCV

As shown in the figure, RenderOcc can not generate reasonable scene structure even with a high mIoU score. Given that OccFlowNet's model framework is very similar to RenderOcc, we can reasonably speculate that OccFlowNet also faces similar issues.

To address this, SparseOcc introduces the RayIoU metric, which we believe to be a more reasonable evaluation metric. As shown in Table 7 in the supplementary material, when utilizing RayIoU as the evaluation metric, our approach sets new SOTA performance by 2.6 RayIoU, with the proposed pre-training stage boosting model performance from 36.4 to 38.7, reaffirming the effectiveness of our training paradigm.

Furthermore, we believe that our method has made significant advancements in other tasks as well. For 4D occupancy forecasting, PreWorld surpasses existing 3D occupancy world model with a rather simple module design. For motion planning, PreWorld achieves comparable L2 performance to some planning models even without ego-state information. By incorporating ego-state information, the motion planning capabilities of our model are significantly enhanced, surpassing other existing 3D occupancy world models by a notable margin.

[9] Boeder, Simon, Fabian Gigengack, and Benjamin Risse. "Occflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow."

The proposed two stages can be trained jointly, why train them in two stages? Can the authors provide any motivation on this?

Our ultimate goal is to achieve superior performance by conducting large-scale self-supervised pre-training of occupancy models using abundant unlabeled 2D image data. This approach aims to excel in performance while minimizing the cost of annotating 3D occupancy data. Due to the limitations of the nuScenes dataset, we are only able to preliminarily validate the effectiveness and scalability of our training paradigm through Table 5. However, as shown in the table above, our training paradigm can achieve even more significant performance improvements on larger-scale datasets.

Based on this motivation, we believe it is necessary to divide the training process into two stages: one using 2D image data (self-supervised pre-training stage) and the other using 3D occupancy data (fully-supervised fine-tuning stage). This approach ensures the decoupling of our requirements for these two types of training data, such as using data with imbalanced scales, which joint training cannot achieve. Furthermore, we aim for the 2D data to be sourced from various datasets, different acquisition environments, and diverse camera settings to truly enhance the generalization of the pre-training process. Subsequently, only a small amount of 3D data would be needed for fine-tuning in specific scenarios, a feat that joint training also cannot accomplish.

The introduction of RayIoU as an evaluation metric lacks proper justification. The authors do not explain in the main text of the paper way RayIoU is superior to the widely accepted mIoU. The authors should provide clear and convincing rationale in the paper for adopting this metric.

In our revised manuscript, we conduct a more detailed analysis of the RayIoU metric and include additional experimental results to complement our findings. Moreover, it is necessary to emphasize that the introduction of the RayIoU metric is intended to provide a clearer explanation for the performance decline of our model under mIoU and why our model lags behind baseline models in certain categories. Therefore, to align with common practices [2-5], we continue to utilize the mIoU metric in the main text while offering a detailed analysis of the results under the RayIoU metric in Section A.1.

[2] Zhang et al., "Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction", ICCV 2023.

[3] Tian et al., "Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving", NeurIPS 2023.

[4] Li et al., "Fb-occ: 3d occupancy prediction based on forward-backward view transformation", CVPR 2023.

[5] Pan et al., "Renderocc: Vision-centric 3d occupancy prediction with 2d rendering supervision", ICRA 2024.

In both Table 7 and the supplementary figure (https://www.hostize.com/v/Ta5ND-IuCV), the absence of results for OccFlowNet prevents a comprehensive comparison. Without including OccFlowNet’s performance metrics, it is difficult to assess how the proposed method stands against existing approaches, thereby weakening the paper’s claims of superiority or innovation.

In our revised manuscript, we have added the RayIoU performance of OccFlowNet to Table 7.

To robustly evaluate predictions outside the visible region, as the aurhors mentioned, the authors should have included results from the SSCBench [1] dataset, which provides ground truth labels beyond visible areas. The omission of this evaluation leaves a critical gap in demonstrating the method’s effectiveness in handling those scenarios.

We need to clarify that Occ3D also provides ground truth labels for occluded regions. However, during evaluation, we follow common practices [3-9] by using visible masks to exclude these regions. Our analysis of occluded regions is simply to illustrate why RenderOcc and OccFlowNet can achieve higher mIoU scores in certain categories. To maintain consistency with common practices, we will not include experiments with the SSCBench [1] dataset, as it is irrelevant to our objectives and not a dataset widely used in previous works [3-11].

To better demonstrate the model's prediction results for both visible and occluded regions, we provide visualized comparisons with RenderOcc: https://www.hostize.com/v/cnpCwid2c4

Consistent with our analysis, RenderOcc tends to predict thicker surfaces for large static categories. However, while this approach may lead to higher mIoU scores, its predictions for occluded regions are chaotic, indicating a lack of true understanding of the scene structure. On the contrary, our PreWorld makes more cautious predictions for occluded regions, demonstrating a more comprehensive understanding of the holistic scene structure.

[1] Li et al., "Sscbench: A large-scale 3d semantic scene completion benchmark for autonomous driving." 2023.

[6] Tong et al. "Scene as occupancy." ICCV 2023.

[7] Liu et al. "Fully sparse 3d panoptic occupancy prediction." ECCV 2024.

[8] Huang et al., "Selfocc: Self-supervised vision-based 3d occupancy prediction", CVPR 2024.

[9] Zhang et al., "Occnerf: Self-supervised multi-camera occupancy prediction with neural radiance fields".

[10] Zheng et al. "Occworld: Learning a 3d occupancy world model for autonomous driving." ECCV 2025.

[11] Wei et al. "Occllama: An occupancy-language-action generative world model for autonomous driving."

The rebuttal fails to provide additional explanations or clarifications for the figures. This lack of detail makes it difficult for readers to fully understand the proposed method, assess its validity, or replicate the results.

Without comprehensive explanations accompanying the figures, the methodological contributions remain vague. This lack of clarity hinders the ability to evaluate the robustness and originality of the proposed approach, raising concerns about the paper’s overall rigor and contribution to the field.

In our revised manuscript, we have provided more detailed captions. For example, in Figures 1, 2, and 3, we have made the meaning of the dashed arrows clearer either within the figures or in the captions.

The ego state-conditioned module lacks of sufficient analysis and comparison.

Thank you for bringing that to our attention. In Section 3.1 and Section 3.2, we compare our proposed forecasting module with the modules utilized in existing 3D occupancy world models. Current methods separate occupancy prediction and forecasting into two distinct procedures: the former relies on a frozen occupancy network, while the latter depends on a complex module involving extensive encoding and decoding steps to obtain future occupancy from current occupancy. In contrast, we consider unifying the procedures of occupancy prediction and forecasting for joint training to be a more natural and end-to-end approach. Therefore, our choose to obtain future volume features from current volume features, which is achieved with just a few simple MLPs without the need for excessive design. The results in Table 2 demonstrate that our simple design has already surpassed existing 3D occupancy world models, validating the soundness of our approach.

In Table 6, we conduct an analysis of the module's impact. The comparison between Row 1 and Row 2 demonstrates that this straightforward design can still introduce future forecasting capabilities to our models. Rows 3 and 4, on the other hand, show that this capability can be further enhanced by incorporating ego-state information and leveraging self-supervised pre-training.

In general, we introduce this module to offer a more rational framework for 3D occupancy world models, and the experimental results indicate that we achieve promising outcomes without employing complex designs. We will provide a more detailed analysis of our current experimental results to better articulate this idea, and thank you for pointing this out.

"Our Copy&Paste" in Tab.6 is confusing for me. What is the setting of row 2 in Tab. 6 and what is difference between row 2 and copy & paste?

"Our Copy&Paste" refers to the process of taking our best prediction result on the 3D occupancy prediction task (as shown in the last row of Table 1) and calculating the mIoU between these results and the ground truth of the future 1s, 2s, and 3s. This can serve as a lower bound to showcase the performance of a model without any future forecasting capability on the 4D occupancy forecasting task.

The setting of Row 2 is that we utilize our forecasting module without introducing ego-state information and self-supervised pre-training. The performance difference between Row 2 and Copy&Paste demonstrates that simply by introducing a strightforward forecasting module and applying temporal 3D occupancy supervision to the model, we can equip the model with future forecasting capabilities and yield non-trivial results (with an average mIoU of 8.25 over 7.79).

How to correctly interrept the result in Tab.3? The method can achieve a comparable result even without ego-state information? Can you elaborate more about the result for ours with ego-state information.

When we do not incorporate ego-state information as input , the model relies solely on historical image inputs to predict the ego trajectory. Introducing ego-state information provides the model with additional scene information, such as velocity and acceleration, enhancing the motion planning capabilities further.

For fair comparison, we provide the results of these two settings. The results without ego-state information illustrate that our model can achieve comparable L2 performance to some planning models, albeit slightly higher collision rates. By incorporating ego-state information as input, the motion planning capabilities of our model are significantly enhanced, surpassing other existing 3D occupancy world models following the same configuration by a notable margin.