OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries

W1. More details of the segmentation model

Segmentation models are trained using homologous data and then integrated into the network as fixed components, being inferred together rather than as additional separate modules. In this setup, a UNet architecture is used, where the encoder part is shared with the backbone's ResNet, and the decoder consists of a three-layer deconvolution. This design minimizes additional memory overhead.

Thank you for the suggestion. We will include these details in the revised version.

W2. Comparison between OctreeOcc without extra semantic information and other methods

The input and output of our network are the same as those of other methods (i.e., input surround-view images and output occupancy prediction result). We trained a sub-network using the same data and incorporated it into the overall framework without relying on additional semantic information. The segmentation network is an integral part of our model, not a separate component. Theoretically, we do not use extra information.

Semantic segmentation is useful for initializing the octree structure; however, even without this component, our network's performance remains comparable to the dense voxel method (as shown in the table below).

Q1. How the high-confidence and low-confidence regions are determined

As shown in Section 3.5 of original paper, a split probability is maintained throughout the process, dividing high and low confidence areas using this probability and top-k. In this context, confidence actually refers to the split probability.

Q2. How does OctreeOcc handle the conflict between the octree queries initialized by current semantic information and the octree queries aligned from the previous frame?

During temporal fusion, the octree query of the historical frame must first be converted into the same structure as the current frame during ego-motion conversion to ensure that fusion can be performed.

W1. The image resolution and visible mask should be marked

In Table 1 of original paper, we use "$\star$" to indicate whether a method uses a camera mask during training. Methods marked with "$\star$" are the latest SOTA methods.

The image sizes used in training for each method are as follows:

All methods use settings close to full-size images within their respective ranges, ensuring a fair comparison.

W2. Temporal Fusion Details

Yes, the temporal fusion method is the 3D version of BEVFormer’s TSA, using a fusion of four frames, consistent with PanoOcc.

W3. The metrics of the Symponies in the tables should be updated

Thanks for your reminding, we will follow the number in its updated version and update it in our final version.

W4. Memory presented in the tables

Yes, the table shows memory consumption during training.

W5. The number of the attention layers of the 3D voxel

As shown in implementation details, octree encoder consists of three layers, each composed of TSA, ICA, and ISR modules.

W1. Too many manually set hyperparameters affect generalisability

Octree-related hyperparameters focus on selecting the top-k operation’s K during octree query sparsification and rectification.

These hyperparameters are statistically derived from the dataset and are not difficult to determine.

The experiments conducted on two datasets in the original paper used the same set of octree hyperparameters. Table 2 shows that applying the octree hyperparameters from nuScenes to SemanticKITTI resulted in SOTA performance on the IoU metrics and performance comparable to SOTA on the mIoU metrics. These experiments validate the generality of our approach.

W2. Updating processing of octree queries

When the octree mask changes, the strategy for adjusting queries in the affected region is to first convert to a dense form and then adapt to the new sparse structure based on the updated octree mask.

W3. Octree queries focus only on their centers, neglecting granularity effects

Thank you for the suggestion. We have compared the sampling scheme you mentioned (as shown in the Table below), where we assign different numbers of reference points to queries of varying granularity during cross-attention. For example, the finest granularity query uses only its centroid, while the medium granularity query samples four additional points around its centroid.

This approach can lead to performance gains but also increases memory overhead due to the effective increase in the number of queries. Our original design balances memory overhead with performance; regions with coarse query granularity are likely part of the same object, so sampling features only at the query's center is adequate. Overall, finding better ways to utilize octree queries will be our next step in enhancing our work.

W1. The proposed method seems difficult to optimize and relies on segmentation and historical information

Regarding optimization, our method uses the same optimizer settings as other methods and is trained for 24 epochs as well. Under the same training conditions, our approach achieves SOTA performance on multiple benchmarks.

Our segmentation component is integrated into the model as a whole and is used to obtain a more accurate initialization. Importantly, the input to the framework remains unchanged, continuing to use surround-view images. As shown in Table 4 of the original paper, even without semantic-guided initialization, our approach still outperforms the baseline (34.17 vs. 35.88) while reducing memory consumption (27200M vs. 18500M).

Furthermore, incorporating historical information is standard in all occupancy prediction methods, including ours. Including historical information is optional, and using only the current frame does not affect the model's inference.

W2. Results on the SemanticKITTI

In our experiment on SemanticKITTI, we used the same set of octree hyperparameters as for nuScenes, achieving SOTA IoU, which highlights the effectiveness of our approach in capturing the overall spatial structure.

For smaller objects (e.g., bicyclists, poles), our IoU is limited by the three-level octree structure. The 32 × 32 × 4 resolution of the first level may not fully capture small objects, leading to early information loss. This represents a trade-off between performance and computational resources; increasing the number of queries generally improves performance but significantly raises memory overhead.

Regarding memory overhead reduction, since model sizes for different methods on SemanticKITTI are small, the potential for memory reduction is limited. However, our memory reduction percentage on SemanticKITTI remains considerable.

Overall, the current results sufficiently validate the effectiveness of our approach. We will continue to fine-tune the hyperparameters for SemanticKITTI and explore better trade-offs to achieve improved performance across different datasets.

W3&Q1. The training time

We present the results of the comparison of the training time of each model on the nuScenes dataset, which show that our method does not significantly increase training time compared to other methods.

Q2. Memory consumption of VoxFormer

The memory consumption reported in the VoxFormer paper accounts only for the stage-2. For a fair comparison, our calculations include both the stage-1 and stage-2. Additionally, the memory consumption we report for our method also includes the segmentation part of the model.

Q3. Implementation details for SemanticKITTI

The setup for SemanticKITTI remains essentially the same as in the previous method. We use ResNet50 as the backbone with an image size of 1220x370. The Adam optimizer is employed, with a learning rate of 2e-4 and a weight decay of 0.01. The octree-related hyperparameters are consistent with those set for the nuScenes dataset. We will include these details in the final version.

Q4. Results on SSCBench-KITTI360

We provide experiment results on SSCBench-KITTI360, and with the same efficient framework and settings, our method also outperforms other methods in both metrics.