$\alpha$-OCC: Uncertainty-Aware Camera-based 3D Semantic Occupancy Prediction

We appreciate your constructive comments and feedback. Now we will explain your concerns and questions point by point in the following. We have highlighted the modified sections in the revised paper for clarity.

W1. Thank you for your suggestion. Now we are working on conducting experiments on the NuScenes dataset. We will show the results before the discussion deadline.

W2. Thank you for your valuable question. As mentioned in L512-L516 of our paper, the class-specific error rate $\alpha^y$ is set by multiplying the original class error rate of OCC models with the scale $\lambda < 1$, to raise the coverage requirement for each OCC model and class in our experiments. In the ablation study (Figure 6), we consider five settings with $\lambda \in \{0.86, 0.89, 0.92, 0.95, 0.98\}$. For the results in Table 2, we use $\lambda = 0.86$.

In practical application, deciding the error rate $\alpha$ for conformal prediction depends on the specific use case and the trade-off between prediction set size and risk tolerance [1]. In high-risk applications, such as medical diagnosis, lower error rates are often chosen to ensure high coverage and minimize critical errors. For instance, $\alpha = 0.01$ is used for achieving a 99% confidence level. In low-risk applications, such as recommendation systems, a higher error rate, such as 0.1 or 0.2, may be acceptable if smaller prediction sets are preferable.

Q1. Thank you for your valuable question. The SemanticKITTI dataset has known annotation defects for dynamic objects, primarily due to inaccuracies in labeling. These defects arise from the use of LiDAR temporal fusion for annotations, which can lead to ghosting effects for dynamic objects. This phenomenon is thoroughly discussed in Figure 2 of the SSCBench paper [2]. In the third row of Figure 4, the car is in motion on the road, which results in incorrect labels filling the entire middle row with cars. We added a footnote in Figure 4 of the revised paper, for this problem.

SSCBench[2] has acknowledged this issue, and thus KITTI360 proposed in SSCBench does not suffer from ghosting problems. This may also explain why our Depth-UP enhances VoxFormer significantly more on KITTI360 compared to SemanticKITTI, showing a 1.64 mIoU improvement versus a 1.01 mIoU improvement.

Q2. Yes, that is correct. In HCP, "Hierarchical" refers to first determining whether a voxel is occupied, and if it is, subsequently identifying its specific category.

Q3. Thank you for your valuable question. We think it is common to use the depth prediction on the semantic prediction, such as works in [3,4,5]. As mentioned in L246-L254 of our paper, to utilize the depth uncertainty information on the semantic features, we extract depth features from the concatenated depth mean and standard deviation. These newly acquired depth features are then seamlessly integrated with the original 2D image features, constituting a novel set of input features $\{**F**_I, **F**_D\}$. This integration strategy capitalizes on the extensive insights gained from prior depth predictions, enhancing the OCC performance with enhanced semantic understanding.

References.

1. Conformal prediction for reliable machine learning: theory, adaptations and applications, Newnes 2014.

2. SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views, IROS 2024.

3. Indoor semantic segmentation using depth information, ICLR 2013.

4. Semantic Scene Completion From a Single Depth Image, CVPR 2O17.

5. Depth-guided Texture Diffusion for Image Semantic Segmentation, arXiv 2024.

W1. Thank you for your suggestion on doing experiments on the nuScenes occupancy dataset. We have conducted experiments on the Occ3D-nuScenes dataset [1] (occupancy version of nuScenes). The Occ3D-nuScenes dataset consists of 1,000 outdoor driving scenes captured using six surround-view cameras. All results are in the Subsection A.11 of the revised paper. The OCC model we used here is BEVStereo [2]. BEVStereo serves as a commonly used OCC baseline for the Occ3D-nuScenes dataset. Due to time and computational constraints, both the base BEVStereo model and BEVStereo with our Depth-UP were trained for 12 epochs and 20 batch sizes on the server with 4 Tesla V100 GPUs.

Table 6 presents the mIoU across all classes and the IoU for each individual class for both the base BEVStereo model and BEVStereo enhanced with our Depth-UP on the Occ3D-nuScenes dataset. Depth-UP demonstrates notable improvements over the base OCC model, achieving a 1.61 (8.77%) increase in mIoU. Furthermore, our Depth-UP significantly enhances performance for small, safety-critical classes, including a 9.43 IoU improvement for the motorcycle class and a 1.09 IoU improvement for the person class. These improvements are attributed to the effective integration of uncertainty information from the depth model into the OCC model.

Table 7 compares our HCP method with SCP and CCCP on the Occ3D-nuScenes dataset using the BEVStereo model. The results demonstrate that HCP consistently achieves robust empirical class-conditional coverage while generating smaller prediction sets. Compared to SCP, it reduces the set size by up to 87% and the coverage gap by up to 97%. Similarly, compared to CCCP, it achieves reductions of up to 10% set size and 6% coverage gap. These findings are consistent with experimental results on the SemanticKITTI and KITTI360 datasets, further validating the scalability of our approach.

Reference.

1. nuScenes: A multimodal dataset for autonomous driving, CVPR 2020.

2. BEVStereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo, AAAI 2023.

We appreciate your constructive comments and feedback. Now we will explain your concerns and questions point by point in the following. We have highlighted the modified sections in the revised paper for clarity.

W1. Thank you for your valuable question. Upon reviewing every image in the validation set, we identified the reason for the mIoU decrease in the person and bicyclist categories on the SemanticKITTI dataset. These issues primarily stem from annotation defects, particularly for dynamic objects such as persons and bicyclists. The SemanticKITTI dataset generates annotations using LiDAR temporal fusion, which introduces ghosting effects for moving objects. This problem has been documented in Figure 2 of the SSCBench paper [1]. While cars are also affected, most are stationary, so the impact is minimal. However, nearly all persons and bicyclists in the SemanticKITTI validation set are moving, leading to erroneous annotations. SSCBench [1] has acknowledged these issues, and thus KITTI360 proposed in SSCBench does not suffer from ghosting problems. Our Depth-UP shows mIoU improvements in both person and bicyclist categories on KITTI360, aligning with our expectations. This may also explain why our Depth-UP enhances VoxFormer significantly more on KITTI360 compared to SemanticKITTI, showing a 1.64 mIoU improvement versus a 1.01 mIoU improvement. We have added the explanation in the Subsection A.7 of the revised paper.

In the paper, we did not explicitly show the relationship between recall and mIoU for specific categories. However, if you are referring to the relationship between recall and IoU, we illustrate this in Figure 5. To clarify the relationship between higher recall and a decrease in IoU, the explanation is as follows: Recall measures the model's ability to capture true positives relative to false negatives, while IoU also takes false positives into account [1]. High recall indicates that the model effectively identifies true occupied voxels but does not penalize over-predictions (false positives). In contrast, IoU provides a more balanced metric by penalizing both false negatives and false positives. Therefore, when a model over-predicts (generates too many occupied voxels), recall may remain high since true positives are well captured, but IoU will decrease due to the increased number of false positives. This trade-off highlights the differing focuses of the two metrics.

In summary, the slight mIoU drop in certain categories on SemanticKITTI is influenced by annotation quality, and the relationship between higher recall and decreased IoU is due to the trade-off between true positives and false positives.

W2. Thank you for your suggestion. Now we are working on conducting experiments on the NuScenes dataset. We will show the results before the discussion deadline.

References.

1. SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views, IROS 2024.

W2. Thank you for your suggestion on doing experiments in the multi-cameras dataset. We have conducted experiments on the Occ3D-nuScenes dataset [1] (OCC version of nuScenes), which is a multi-cameras dataset. All results are in the Subsection A.11 of the revised paper. The Occ3D-nuScenes dataset consists of 1,000 outdoor driving scenes captured using six surround-view cameras. The OCC model we used here is BEVStereo [2]. BEVStereo serves as a commonly used OCC baseline for the Occ3D-nuScenes dataset. Due to time and computational constraints, both the base BEVStereo model and BEVStereo with our Depth-UP were trained for 12 epochs and 20 batch sizes on the server with 4 Tesla V100 GPUs.

Table 6 presents the mIoU across all classes and the IoU for each individual class for both the base BEVStereo model and BEVStereo enhanced with our Depth-UP on the Occ3D-nuScenes dataset. Depth-UP demonstrates notable improvements over the base OCC model, achieving a 1.61 (8.77%) increase in mIoU. Furthermore, our Depth-UP significantly enhances performance for small, safety-critical classes, including a 9.4 IoU improvement for the motorcycle class and a 1.09 IoU improvement for the person class. These improvements are attributed to the effective integration of uncertainty information from the depth model into the OCC model.

Table 7 compares our HCP method with SCP and CCCP on the Occ3D-nuScenes dataset using the BEVStereo model. The results demonstrate that HCP consistently achieves robust empirical class-conditional coverage while generating smaller prediction sets. Compared to SCP, it reduces the set size by up to 87% and the coverage gap by up to 97%. Similarly, compared to CCCP, it achieves reductions of up to 10% set size and 6% coverage gap. These findings are consistent with experimental results on the SemanticKITTI and KITTI360 datasets, further validating the scalability of our approach.

Reference.

1. nuScenes: A multimodal dataset for autonomous driving, CVPR 2020.

2. BEVStereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo, AAAI 2023.

We appreciate your constructive comments and feedback. Now we will explain your concerns and questions point by point in the following. We have highlighted the modified sections in the revised paper for clarity.

W1 & Q1. Thank you for pointing this out. As outlined in the contributions part of the introduction, in the depth model, the uncertainty refers to the error in depth estimation. For the occupancy prediction (OCC), uncertainty is defined as the prediction set under a given class coverage guarantee for each voxel. In the introduction, we explain the importance of considering depth uncertainty propagation and OCC uncertainty quantification in Figure 1. Figure 1(a) illustrates the relationship between depth estimation uncertainty and OCC performance: as the uncertainty (error) in depth estimation increases, the accuracy of OCC decreases. This underscores the significant influence of depth uncertainty on OCC performance and motivates our Depth-UP. By incorporating depth uncertainty into both the geometry completion and semantic segmentation components of OCC, Depth-UP enhances overall OCC accuracy. The importance of uncertainty quantification in OCC is further demonstrated in Figure 1(b). Using our HCP to quantify uncertainty enables improved occupied recall for rare classes and the generation of prediction sets for each occupied voxel, reducing the likelihood of potential crashes. This ability to quantify and manage uncertainty in OCC is crucial for enhancing the planning and control processes of safety-critical autonomous systems, ensuring both reliability and safety. We have modified the revised paper in L051 and L082.

W2 & Q2. Thank you for your suggestion. First, it is important to note that VoxFormer[1] and OccFormer[2], which we used in our paper, were recently proposed (in 2023) on SemanticKITTI and KITTI360. They were widely regarded as baselines in OCC at the time we began this work. VoxFormer[1] has achieved 185 citations and OccFormer[2] has achieved 121 citations, highlighting their relevance in the field. Second, the main contribution of our paper is to recognize the OCC problem from a fresh uncertainty quantification perspective, as emphasized in the introduction. We do not aim to propose a state-of-the-art OCC model or compare it directly against others. Instead, our focus is on enhancing existing OCC models through uncertainty propagation (via Depth-UP) and achieving better uncertainty quantification under the high-class imbalance challenge on OCC (via HCP). Due to limitations in time and computational resources, we were unable to conduct additional experiments with the latest state-of-the-art OCC models.

References.

1. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion, CVPR 2023.

2. OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction, CVPR 2023.

We appreciate your constructive comments and feedback. Now we will explain your concerns and questions point by point in the following. We have highlighted the modified sections in the revised paper for clarity.

W1 & Q1. Thank you for your suggestion. In the revised paper, we have provided the number of parameters for each model in Table 3. Specifically, adding PGC introduces 0.11M parameters (0.2%) for introducing the additional head to estimate depth uncertainty. Adding PSS increases the parameter count by 11.54M (19.25%), due to the additional head for depth uncertainty estimation and the extra ResNet backbone for depth feature extraction. Consequently, the entire Depth-UP module also results in an increase of 11.54M parameters (19.25%). To reduce the added parameters, we could explore using a simpler backbone for depth feature extraction; however, this might affect overall performance. Investigating alternative feature extraction backbones will be a focus of our future work.

W2 & Q2. Thank you for pointing this out. As highlighted in the introduction, the primary contribution of our work is to recognize the OCC problem from a fresh uncertainty quantification perspective. Both Depth-UP and HCP methods are designed to tackle the uncertainty in OCC and improve its overall performance: Depth-Up is the uncertainty propagation framework to improve the OCC performance, where the uncertainty quantified by the direct modeling is utilized on both geometry completion and semantic segmentation. HCP is the uncertainty quantification method to solve the high-level class imbalance challenge on OCC, where on geometry completion, a novel KL-based score function is proposed to improve the occupied recall of safety-critical classes with little performance overhead. While they may appear relatively independent, they complement each other by demonstrating how uncertainty can influence both the internal mechanisms and the external outputs of an OCC model. Together, they form an integrated pipeline for addressing OCC challenges. Moreover, employing Depth-UP and HCP simultaneously in a model provides significant improvements in both accuracy and uncertainty quantification. As illustrated in Figure 5 of our paper, both modules independently enhance IoU at the same occupied recall for the rare class. When used in combination, they achieve the best performance. For instance, in the bottom subfigures of Figure 5(a), the Depth-UP & HCP model maintains an IoU above 30 even at an occupied recall of 70 for person class, while other models achieve, at most, an IoU of around 10. This clearly demonstrates that the joint application of Depth-UP and HCP delivers superior performance.

W3 & Q3. Thank you for your suggestion. We provide the evaluation of how the estimated uncertainty of depth and uncertainty of OCC vary with distance separately in Figure 10 of the revised paper. We explain the results in detail in Subsection A.9. Figure 10(a) shows the correlation between the estimated standard deviation (uncertainty) of depth and the distance from the camera to the object. The results reveal that the depth uncertainty is highest when the object is very close to the camera. This phenomenon arises because, in stereo vision systems, objects at close range result in minimal disparity between the two images, making depth estimation inherently challenging [1]. The uncertainty reaches its lowest point at approximately 15 meters, beyond which it increases with distance. This trend aligns with the inverse relationship between depth and disparity, as well as the reduced pixel resolution available for objects further away from the camera [1]. These observations confirm that the depth uncertainty estimation in our model is consistent with theoretical expectations.

Figure 10(b) presents the relationship between the Expected Calibration Error (ECE) metric of VoxFormer outputs and the distance to the voxels. In this case, we applied the voxel-based ECE computation method described in Pasco[2] as you mentioned. The results show that the OCC uncertainty is minimized at approximately 15 meters, consistent with the depth uncertainty trend observed in Figure 10(a). When voxels are very close to the camera, the OCC ECE is relatively high, likely due to depth estimation errors. Similarly, when voxels are far from the camera, the OCC ECE increases, attributed to the limited pixel resolution for distant objects.

Notably, the similarity in the shapes of the curves in Figure 10(a) and 10(b) highlights the significant influence of depth uncertainty on OCC performance, as discussed in Section 1. These findings reinforce the importance of utilizing the depth uncertainty in improving final OCC performance.

To help this paper receive a higher score and increase the likelihood of acceptance, do you have any additional suggestions on how to further improve it? We would sincerely appreciate your constructive feedback!

We appreciate your constructive comments and feedback. Now we will explain your concerns and questions point by point in the following. We have highlighted the modified sections in the revised paper for clarity.

W1. Thank you for pointing this out. To estimate the uncertainty of the depth model, we freeze all parameters of the original depth model and only retrain the additional header for $\hat{\mathbf{\Sigma}}$. So in terms of the performance of the depth estimation $\hat{\mathbf{D}}$, there is no difference between the newly trained depth model and the basic depth model. The joint training does not improve the depth prediction. We have added more explanation in L211 of the revised paper.

W2. Thank you for pointing this out. The base models and models with our Depth-UP use the same depth estimation model, producing identical depth estimation outputs $\hat{\mathbf{D}}$ for the same input data. The sole difference in the depth estimation model is that the base model does not include the additional header for $\hat{\mathbf{\Sigma}}$ (uncertainty branch).

Other Minor Mistakes. Thank you for pointing them out. We have corrected them in L050 and L354-355 of the revised paper.