SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving

Dear Reviewer FWLb,

Thanks for your precious time and considerate suggestions to improve our manuscript.

Q1: Discussion about the class-balancing strategies with existing studies [1] [2].

A1:  Thanks for your comments, both methods present excellent self-training category balancing methods. In the process of self-training, Class-balanced self-training (CBST) [1] sets different thresholds for different classes, and then balances classes according to the confidence of false tags, so that false tags with low confidence of rare classes can participate in the training. Based on the CBST, Class-range-balanced Self-training (CRB-ST) [2] adds range-based considerations,  and sample pseudo-labels both from both category and range dimensions. Unlike them, we are mainly from a global perspective to alleviate the problem of class imbalance in the pre-training process. First, to ensure the integrity of occupancy labels in the scene, class sampling is based on frame-level. Secondly, considering that foreground information is more important than background information, and a large amount of background information can easily drown out scarce foreground information, we only rebalance for foreground categories, which can bring benefits to both downstream detection and segmentation tasks.

Q2: Discussion about using different sampling strategies in downstream task to potentially achieve better downstream subsets.

A2:  We strongly agree with you that better sampling for downstream datasets leads to better performance. Since we do not use unlabeled data in the downstream dataset, we only perform fine-tuning experiments and fair comparison experiments on the same batch of randomly sampled data. Also, we have added some experimental results with uniform sampling. In the future, we will also integrate different sampling methods, such as the ST-RFD method [3], into the fine-tuning pipeline for better performance.

Table 19. Different sampling strategies on downstream task.

Q3: Discussion about task-specific semi-supervised learning/active learning [2,3,4].

A3:  [2, 3, 4] are excellent semi-supervised or weakly supervised methods in the field of 3D point cloud segmentation, and they use the labeled data and other pseudo-labels of unlabeled data to achieve excellent performance, but most of these semi-supervised methods are task-specific or dataset-specific, e.g., they can only work in segmentation tasks or detection tasks. In contrast, SPOT focuses on the pre-training part, aiming to bring better transferable prior knowledge to all other tasks and datasets through just one pre-training process, which is decoupled from the downstream datasets and tasks. Moreover, we have not used downstream unlabeled data, meaning that we can combine SPOT with various semi-supervised or weakly-supervised methods in downstream fine-tuning to further improve the performance, by using SPOT as the downstream initialization. We think it's going to be a very effective combination.

Q4: Lack of related work, e.g., [1-4].

A4:  Thank you for pointing these out! We has added all of them in the revision.

We hope our reply and additional experiment results address your concerns. Look forward to further discussion on any remaining concern about our paper.

Best regards,

Authors of Paper 496

[1] Zou, Y., Yu, Z., Kumar, B. V. K., & Wang, J. (2018). Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV) (pp. 289-305).

[2] Unal, O., Dai, D., & Van Gool, L. (2022). Scribble-supervised lidar semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2697-2707).

[3] Li, L., Shum, H. P., & Breckon, T. P. (2023). Less is more: Reducing task and model complexity for 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9361-9371).

[4] Kong, L., Ren, J., Pan, L., & Liu, Z. (2023). Lasermix for semi-supervised lidar semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21705-21715).

Dear Reviewer FWLb,

Thanks again for your great efforts and constructive advice in reviewing this paper! As the deadline of discussion period is approaching, we sincerely hope that our response can address your concerns and we are looking forward to further discussion on any other questions or issues regarding the paper or our response.

Best regards,

Authors of Paper 496

Q4: How to make pre-training self-supervised via SPOT?

A4: We totally agree that using fewer or no labels for pre-training, namely semi-supervised or self-supervised pre-training, is essential and more practical. However, previous studies [3,4,5] that employ the fully self-supervised paradigm suffer from the inability to attain the task or dataset scalability: they perform the 3D pre-training and downstream on the same dataset. In contrast, SPOT is the first to explore the so-called one-for-all setting, meaning that the baseline model is only pre-trained once using the proposed SPOT and fine-tuned on different downstream datasets and tasks.

Also, as demonstrated in image domain, where the most popular pre-training model is fully-supervised pre-trained on ImageNet [6], it is not necessary for pre-training methods to be full self-supervised. Besides, for 3D pre-training on LiDAR point cloud, AD-PT [7] is a successful exploration on semi-supervised pre-training. We perform semi-supervised pre-training experiments with SPOT in Q1 and General Response Part 2, which demonstrates the effectiveness of SPOT with few labeled data.

We hope these can cover your concerns.

Q5: Pre-training on a different dataset via SPOT?

A5: To verify that SPOT is able to pre-train on other datasets, we utilize the model which is pre-trained on Waymo to predict occupancy labels on 5% NuScenes dataset and generate pseudo occupancy labels. Next, we pre-train SPOT from scratch on these NuScenes data, and then fine-tune on the 20% KITTI data. As shown in Table 10 below, SPOT achieves significant gains compared to baseline results on KITTI dataset, demonstrating the effectiveness and generalization of SPOT.

Table 10: Pre-training via SPOT on NuScenes and downstream to 3D object detection on KITTI.

We hope the discussion and experiments address your concerns. We are happy to discuss any remaining concerns about the paper.

Best regards,

Authors of Paper 496

[1] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896. Atlanta, 2013.

[2] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.

[3] Runsen Xu, Tai Wang, Wenwei Zhang, Runjian Chen, Jinkun Cao, Jiangmiao Pang, and Dahua Lin. Mv-jar: Masked voxel jigsaw and reconstruction for lidar-based self-supervised pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13445–13454, 2023

[4] Zhiwei Lin and Yongtao Wang. Bev-mae: Bird’s eye view masked autoencoders for outdoor point cloud pre-training. arXiv preprint arXiv:2212.05758, 2022

[5] Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhenguo Li, and Luc Van Gool. Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3293–3302, 2021.

[6] Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical image database[C]//2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009: 248-255.

[7] Jiakang Yuan, Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Ad-pt: Autonomous driving pre-training with large-scale point cloud dataset. arXiv preprint arXiv:2306.00612, 2023

Dear Reviewer y7L9,

Thank you for your review. We conduct new experiments on semi-supervised pre-training and provide detailed analysis both in General Response Part 2 and here. We also provide experiment results on pre-training on other tasks and pre-training on NuScenes dataset via SPOT to cover your concerns. Besides, we provide discussion on supervised, semi-supervised and self-supervised pre-training here.

Q1: Label-intensive nature for pre-training via occupancy prediction.

A1: We thank you for acknowledging SPOT as a promising pre-training method for LiDAR perception and pointing out this aspect. We totally agree that using fewer or no labels for pre-training, namely semi-supervised or self-supervised pre-training, is essential and more practical. To address this concern, we conduct semi-supervised pre-training with SPOT. To be more specific, we pre-train the backbones with SPOT using only 5% sequence-level labeled data and different amount of sequence-level unlabeled data (5% and 15%) via pseudo-labeling [1] and mean-teacher approach [2]. The results are shown in Tables 1, 2, 3, 4 and 5 in General Response Part 2.

It can be found in Tables 1, 2, 3, 4 and 5 that pre-training with 5% sequence-level labeled data and 15% sequence-level unlabeled data achieves comparable downstream fine-tuning performance as that of pre-training with 20% sequence-level labeled data. Meanwhile, when compared to pre-training with 5% sequence-level labeled data and 5% sequence-level unlabeled data, it can be found that more unlabeled data brings better downstream performance. These results verify that (1) SPOT can learn general representations for different downstream datasets, tasks and architectures with small part of labels. (2) When incorporating more unlabeled data into pre-training phase, SPOT also shows scalability.

We hope these results can cover your concerns on the label-intensive nature for fully-supervised pre-training via SPOT.

Q2: Given annotations on Waymo Dataset, the improvements and results are anticipated

A2: We would like to clarify that the proposed SPOT is a more general pre-training paradigm. This means that only one pre-trained is conducted via SPOT, and we applied the pre-trained backbones to improve performance in various datasets and tasks. As shown in Table 8 below, it can be found that pre-training via detection or segmentation task on fully-annotated Waymo data suffers from two gaps when fine-tuned in downstream tasks (1) the gap between pre-training task and downstream task. Pre-training on semantic segmentation task harms the performance a lot. (2) the gap between different LiDARs used to collect datasets. When pre-training via detection task, the performance on downstream dataset also drops due to different sensors used in these two datasets. In comparison, SPOT achieves general improvements, which demonstrate the superiority of SPOT as pre-training task.

Table 8. Pre-training with different tasks with annotations on Waymo Dataset and fine-tune the pre-trained backbones for 3D object detection on NuScenes dataset.

Q3: Using only binary occupancy labels for pre-training

A3: We conduct experiments where we pre-train backbones via SPOT with only binary occupancy labels. The results are as shown below:

Table 9: Pre-training via SPOT with binary labels indicating Occupied and Empty.

It can be found that pre-training only with binary labels harm the performance. This might stem from that pure occupancy information provides little semantic information and thus offer little help for downstream detection and semantic segmentation tasks.

Dear Reviewer y7L9,

The deadline of discussion period is approaching. We hope that our response can address your concerns and we are looking forward to further discussion on any remaining question about our paper and response.

Best regards,

Authors of Paper 496

Dear Reviewer y7L9,

Thank you for your precious time on the review. As the deadline of discussion period is approaching, we sincerely hope that our response can address your concerns and we are looking forward to further discussion on any other questions or issues regarding the paper or our response.

Best regards,

Authors of Paper 496

Dear Reviewer y7L9,

Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any.

Best regards,

Authors of Paper 496

Q5: Low train from scratch baseline performance: As shown in Fig 7, with 100% NuScenes, training from scratch leads to mAP ~ 50, which is far from SOTA (>70).

A5: Thanks for pointing this out. Our motivation is to improve label-efficiency performance on various downstream tasks and datasets via one pre-trained model. We totally agree that it is also important to improve SOTA models. Most models that achieve >70 NDS performance on NuScenes Leaderboard is multi-modality methods with camera inputs but we focus on LiDAR perception. To demonstrate SPOT can improve SOTA model, we perform fine-tuning experiments with 100% full data on the VoxelNext [2] model, which is one of SOTA LiDAR-only models on NuScenes benchmark. And the results in Table 17 prove that SPOT can still achieve gains on SOTA models.

Table 17. Fine-tune the pre-trained backbones for VoxelNext on 100% NuScenes 3D object detection dataset.

Q6: Could the baseline model be undertrained or that it is using a weaker data augmentation?

A6: The data augmentations we used all follow the OpenPCDet settings, including gt sampling, random flip/rotate/translate, etc., which is a consistent approach for current 3D detection models. Besides, we conduct experiments on extending the training schedule on NuScenes dataset, and the results are shown as follows. From the results, it can be seen that even when the training schedule is extending by 5 times at least, the initialization by our pre-trained model can achieve higher performance. Especially for the Transformer-based DSVT detector, the result of just 20 epochs is even superior to the result of 150 epochs.

Table 18. Extending training schedule on NuScenes dataset.

We hope our additional experiment results and discussion address your concerns. And we are glad to discuss anything unclear about our paper.

Best regards,

Authors of Paper 496

[1] Chen Min, Liang Xiao, Dawei Zhao, Yiming Nie, and Bin Dai. Occupancy-mae: Self-supervised pre-training large-scale lidar point clouds with masked occupancy autoencoders. IEEE Transactions on Intelligent Vehicles, 2023.

[2] Chen Y, Liu J H, Zhang X Y, et al. VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking. arXiv 2023[J]. arXiv preprint arXiv:2303.11301.

Dear Reviewer 4p2k,

Thanks for your time and constructive suggestions! We provide additional experiments on semi-supervised pre-training via SPOT and make more fair comparisons to the baselines. Also a detailed discussion on the results are presented.

Q1: Unfair comparisons with BEV-MAE and AD-PT

A1: The motivation of SPOT is to pre-train a general backbone for different datasets and tasks, which is not achieved by BEV-MAE and AD-PT. For BEV-MAE, they are only able to pre-train and fine-tune on the same dataset. Tables 11, 12, 13 and 14 below show this phenomenon. For AD-PT, fine-tuning on semantic segmentation task harm the performance. Tables 12 and 13 show that when fine-tuning on LiDAR semantic segmentation task, AD-PT brings smaller improvement or even harm the performance.

To make more fair comparisons to AD-PT and also demonstrate the scalability of SPOT, we perform semi-supervised pre-training with SPOT. Results are provided in Tables 11, 12, 13 and 14 below. It can be found that pre-training with 5% sequence-level labeled data and 15% sequence-level unlabeled data achieves comparable downstream fine-tuning performance as that of pre-training with 20% sequence-level labeled data. Although 20% data in Waymo is much less than that used in AD-PT, it can be found that SPOT still achieve comparable or surpass AD-PT in various downstream datasets and tasks. We are also running experiments with more unlabeled data but due to the time limitation the experiments are not finished yet. We would update the results as soon as they finish. However, scalability of SPOT can also be discovered when we compare pre-trainings with different amount of unlabeled data (5% and 15%). It can be found that more unlabeled data brings better downstream performance. These results verify that (1) SPOT can learn general representations for different downstream datasets, tasks and architectures with small part of labels. (2) When incorporating more unlabeled data into pre-training phase, SPOT also shows scalability.

Table 11. Comparison to different pre-training methods. The downstream task is 3D object detection on NuScenes. We provide additional semi-supervised pre-training via SPOT here.

Table 12. Comparison to different pre-training methods. The downstream task is 3D object detection on KITTI. We provide additional semi-supervised pre-training via SPOT here.

Table 13. Comparison to different pre-training methods. The downstream task is LiDAR semantic segmentation on NuScenes. We provide additional semi-supervised pre-training via SPOT here.

Dear Reviewer 4p2k,

Thank you for the prompt reply and the acknowledgement of our additional experiments and discussions. We are writing to provide further discussion on your remaining concerns.

Q1: The seemingly marginal improvement for VoxelNext.

First of all, in our pre-training, both 3D and 2D (BEV) backbones are used for occupancy prediction but VoxelNext [1] only uses 3D backbone for encoding. Thus, we are only able to load weights of 3D backbone when fine-tuning on NuScenes, which losses useful information from pre-training. Under such circumstances, we also achieve 0.52% mAP improvement. It is a well-known phenomenon in previous works [2,3,4] that improving over SOTA models with 100% downstream labels is hard and 0.52% mAP is non-negligible improvement.

To further demonstrate the ability of SPOT to improve SOTA performance on NuScenes, we perform two additional experiments: (1) training VoxelNext on train and validation set and then test it on Test set. (2) fine-tuning on CenterPoint [5], which is another SOTA model on NuScenes, and it has both 3D and 2D backbone for embedding point clouds, leading to less information loss.

(1) We train VoxelNext on train and validation set and then test it on Test set, which is a standard setting for test set. It can be seen from the following table that, although only pre-trained parameters of 3D backbone are loaded, SPOT can boost the performance of VoxelNext on test set by 1.0% mAP and 0.7% NDS, respectively. More than 200% improvement are gained compared to previous results and this improvement is significant on SOTA models as stated in previous works [2,3,4].

Table. Fine-tune the pre-trained backbones for VoxelNext on 100% NuScenes 3D object detection trainval set and evaluate on test set.

(2) Furthermore, to avoid information loss, we employ the CenterPoint [5] (voxel_size=0.075), which utilizes both 3D and 2D backbones for encoding, for 3D object detection task on NuScenes. We load pre-trained weights for both 3D and 2D backbones from SPOT and fine-tune CenterPoint with full training data. In the Table below, it can be found that compared to training from scratch, initializing CenterPoint by SPOT brings 2.63% mAPs and 1.25% NDS improvement. Thus, without loss of pre-trained information, that is loading the full amount of pre-training parameters, SPOT is able to achieve far more significant performance improvement on SOTA models.

Table. Fine-tune the pre-trained backbones for CenterPoint (voxel_size=0.075) on 100% NuScenes 3D object detection dataset.

We hope these discussions and experiment results can cover your concerns on the improvement on SOTA models. Thanks.

---

---

Dear Reviewer 4p2k,

We hope that our response can address your concerns. As the deadline for discussion is approaching, we would greatly appreciate it if you could let us know if there are any other questions or issues regarding the paper or our response. We are looking forward to further discussion.

Best regards,

Authors of Paper 496

Dear Reviewer 4p2k,

Thank you for your precious time on the review. As the deadline of discussion period is approaching, we sincerely hope that our response can address your concerns and we are looking forward to further discussion on any other questions or issues regarding the paper or our response.

Best regards,

Authors of Paper 496

Dear Reviewer 4p2k,

Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any.

Best regards,

Authors of Paper 496

It can be found in Table 1, 2, 3, 4 and 5 that pre-training with 5% sequence-level labeled data and 15% sequence-level unlabeled data achieves comparable downstream fine-tuning performance as that of pre-training with 20% sequence-level labeled data. Also, when compared to pre-training with 5% sequence-level labeled data and 5% sequence-level unlabeled data, it can be found that more unlabeled data brings better downstream performance. These results verify that (1) SPOT can learn general representations for different downstream datasets, tasks and architectures with small part of labels. (2) When incorporating more unlabeled data into pre-training phase, SPOT also shows scalability.

Thus, semi-supervised occupancy prediction is promising in LiDAR perception pre-training and we believe that SPOT would pave the way for future exploration in LiDAR pre-training.

[1] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896. Atlanta, 2013.

[2] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.