Pre-training LiDAR-based 3D Object Detectors through Colorization

nuScenes dataset. We thank the reviewer for the suggestion. We have provided some preliminary results of segmentation in our first response, Downstream tasks. Meanwhile, we have tried our best to conduct 3D object detection experiments of pre-training on NuScenes and fine-tuning on 5% NuScenes, using CenterPoint. The results are summarized in the table below. The performance of GPC is close to the SOTA, TriCC, and better than previous algorithms. Please note that the performance shown here could be suboptimal due to the time constraint of the rebuttal period.

TriCC: Pang et al. "Unsupervised 3D Point Cloud Representation Learning by Triangle Constrained Contrast for Autonomous Driving." CVPR. 2023.

SLidR: Sautier et al. "Image-to-lidar self-supervised distillation for autonomous driving data." CVPR. 2022.

In light of our clarification and additional results, we would like to ask if the reviewer will reconsider the rating. Thank you.

Downstream tasks. Thanks for the comment. In our humble opinion, our colorization-based pre-training (i.e., GPC) does not just learn semantics but which nearby points belong to the same instance (e.g., cf. the last paragraph on page 4). In essence, the same semantic class can have different colors, even in the same scene (e.g., multiple cars of different colors in Fig. 4). Our method can accurately colorize each car instance (cf. Fig. 4 (d)), implying that it learns the notion of object instances, which we believe crucial for 3D object detection.

That said, during the rebuttal, we followed your suggestion to perform a preliminary study of GPC on LiDAR-point semantic segmentation. We use a UNet-like model as in PointContrast and perform experiments on NuScenes. While we did not have sufficient time to perform hyper-parameter tuning, we have already obtained a 6% gain on mIoU against training from scratch. We will include the comprehensive results in the final version.

Using SOTA detectors. We provide additional results of pre-training on Waymo and fine-tuning on KITTI (cf. Table 2), using PV-RCNN as the 3D detector. (We note that many SOTA results use PV-RCNN.) As shown in the following table, GPC consistently outperforms existing algorithms. Please also see our response More comparisons in Table 1 and Table 2 to Reviewer Hnni for further discussions.

STRL: Huang et al. "Spatio-temporal self-supervised representation learning for 3d point clouds." ICCV. 2021.

ALSO: Boulch et al. "Also: Automotive lidar self-supervision by occupancy estimation." CVPR. 2023.

GCC-3D: Liang et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." ICCV. 2021.

PatchContrast: Shrout et al., "PatchContrast: Self-Supervised Pre-training for 3D Object Detection." arXiv preprint arXiv:2308.06985.

Writing: decorative math. We appreciate your feedback. We will try to squeeze Sect. 3 so we can include some rebuttal responses in the main paper.

That said, we respectfully think the math on page 4 is meaningful. While the concepts of color variation (e.g., the same object can have different color options) and hints may sound intuitive in hindsight, we think a more formal description of the problem and the solution is needed for the soundness of our paper. In particular, Sect. 3.1 justifies why the color variation cannot be resolved by designing a more complex neural network or training objective and why adding hints is an effective solution. Our ablation studies in Sect. 4.3, in particular, Table 8 and Fig. 4, further validate the importance of adding hints for pre-training with colorization.

Lack of validation in instance-level 3D object detection tasks. We respectfully think there might be a misunderstanding. While we pre-train with point-level colorization, our main experimental results (cf. Table 1–9) are about instance-level 3D object detection. We will make sure we clarify this in the final version. In short, during pre-training, we leverage colors and the hint mechanism to equip the model backbone with the semantic cues that facilitate the subsequent fine-tuning for 3D object detection.

Multimodal pretraining approach. Thank you for the comment. In our original submission, we indeed compared with other multimodal pre-training in Table 3. Please also see the paragraph Learning with Images on page 8 of the main paper.

Here, we further provide the results of fine-tuning on 100% KITTI, using the PV-RCNN model. All the compared methods in the following table use multimodal pre-training, including the current SOTA: TriCC. GPC performs the best overall.

TriCC: Pang et al. "Unsupervised 3D Point Cloud Representation Learning by Triangle Constrained Contrast for Autonomous Driving." CVPR. 2023.

SLidR: Sautier et al. "Image-to-lidar self-supervised distillation for autonomous driving data." CVPR. 2022.

Linear probing. Thanks for the comment. We note that while linear probing is usually used to evaluate pre-training on classification tasks, it may not be suitable for the downstream task of 3D object detection. For 3D object detection, the detection head added on top of the backbone contains multiple neural network layers and is trained for multiple tasks (e.g., regression and classification). In this paper, we follow the convention in this branch of study (e.g., PatchContrast, ProposalContrast, etc.) to perform full fine-tuning to compare with existing SOTA algorithms.

Details about the proposed method. Thank you for pointing out the problem. Due to the space limit, we keep some details in the supplementary. We will incorporate more details and discussion in the final version.

Inconsistency of camera and point cloud. Thanks for the question. We provided a detailed discussion regarding this question in the first response (i.e., color assignment) to Reviewer Hnni.

Specifically about the KITTI dataset, since the 3D object detection metric only considers objects that appear within the frontal camera's field of view (Ref. [1]), it is a convention to input only the corresponding frontal-view LiDAR points into the 3D object detector in both training and testing (Ref. [2]).

Regarding why using colorization for pre-training is effective, we have provided several analyses (cf. Sect. 4.3, S2, and S4) and arguments (cf. Sect. 1 and 3) in the main paper and supplementary. We will be happy to conduct additional analyses in the final version. We note that pre-training with naive colorization is suboptimal (cf. Table 6 with the MSE loss and Table 8 without hints). Our key contribution is identifying the intrinsic difficulty (cf. Sect. 3.1 and 3.2) and proposing GPC as the appropriate way to use color for pre-training.

[1] https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

[2] https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/dataset_configs/kitti_dataset.yaml#L17

Comparison to LaserMix at 20% fine-tuning. Thanks for providing the reference. We carefully read it (Ref. [3]) and found that the focus differs from ours. LaserMix focuses on LiDAR semantic segmentation, while we focus on 3D object detection. We could not find its performance of 3D object detection. The fine-tuning subsets used in our paper are randomly sampled. However, we provide the results of repeated experiments in Appendix S1, showing that the improvement is consistent regardless of which data are sampled for fine-tuning. Our claim about the 20% fine-tuning is just to explain the effectiveness of the proposed method and demonstrate the potential benefits of reducing labeling efforts. It is worth noting that, at the time of our paper submission, the SOTA algorithm, ProposalContrast, achieved only an overall moderate performance of 66.2% when fine-tuned on 20% labeled data (cf. Table 2). In contrast, GPC achieved 70.1%, better than 69.5% achieved by training from scratch using the entire dataset. The same trend is found in our new results using PV-RCNN as the 3D model (cf. the response More comparisons in Table 1 and Table 2 to Reviewer Hnni): GPC performs fairly well when fine-tuning with 20% data.

[3] Kong et al. "Lasermix for semi-supervised lidar semantic segmentation." CVPR. 2023.

In light of our clarification and additional results, we would like to ask if the reviewer will reconsider the rating. Thank you.

Comparison to ProposalContrast and DepthContrast. We will surely include more discussions. In general, we found GPC particularly useful for the label-scarce setting (e.g., 5% and 20% in Table 2). We see a similar trend in our newly added PV-RCNN results above. Under the label-scarce setting, GPC performs particularly well on pedestrians and cyclists, while the gain of the latter case diminishes when more labeled data are available. We hypothesize that GPC performs particularly well on pedestrians due to the intrinsic difficulty. Compared to cars and cyclists, whose shapes and layouts are (partially) rigid, pedestrians have larger variations not only in colors but also in poses. This makes propagating the seed colors to the remaining points challenging, encouraging the backbone to learn more knowledge, e.g., establishing stronger connections between points that would in turn benefit downstream fine-tuning. In contrast, cars contain some parts that the model does not need additional hints to predict their colors. For example, as shown in Fig. 5(b) of the main paper, the model can learn the red tail lights without giving it any hints. We will include the discussion in the final version.

Joint training. Thank you for the interesting question. In this paper, we mainly follow the pre-training and fine-tuning convention to study pre-training for 3D object detection. Joint training, or more precisely, multi-task learning, is another way to leverage multiple objectives. In our experience, joint training typically could outperform training the downstream task from scratch, but may not outperform pre-training and then fine-tuning. This is because, in joint training, the model backbone has to perform well on both tasks simultaneously, which may distract from the main objective. That said, we will try to include joint training in the final version.

The introduction about data and labels. We apologize for the confusion. We mainly want to convey the idea that the connection between the input point cloud and the bounding boxes can not be straightforwardly learned by a black-box model. One way to make their connection more explicit is through first segmenting objects and then reading out their poses. We will carefully modify this sentence in the final version.

Grammar mistakes. Thank you very much for pointing out the issue. In the final version, we will carefully review the entire paper to enhance its readability.

Loss ablation for pre-training. We apologize for the confusion. Indeed, different losses require different hyperparameter tuning. Actually, we conducted comprehensive grid searches for pre-training learning rates and fine-tuning learning rates (i.e. 0.0001 to 0.01), as well as other optimization-related hyperparameters. What we reported in Table 6 of the main paper is the best performance of each loss. We will clarify the ablation and provide more discussion in the final version. Thank you for bringing this to our attention.

Regarding regression, it is known to be sensitive to outliers. In contrast to MSE, SL1 is more commonly used in many computer vision tasks such as colorization and regressing the bounding box locations or offsets. Your suggestion of a balanced SL1 loss sounds interesting. If we understand it correctly, you mean to balance the SL1 loss similarly to the BalancedSoftmax. Unfortunately, unlike classification where one can easily calculate the balance factors based on the number of samples per discrete class, calculating such a factor for continuous values is not trivial (e.g., some binning may be needed).

Instead, we explore the use of BalancedL1 (Ref [1]), which attempts to reduce the effect of outliers. The results are shown in the table below. To provide a snippet of how we did the grid search in our main paper, we have reported the top-5 performance from the grid search of more than 30 pairs of learning rates on the experiments of fine-tuning on 5% KITTI.

In our case, the color distribution highly depends on the point distribution. For example, the gray ground and the larger objects like cars contain the majority of points. They are inliers for regression. MSE learns more towards outliers (smaller objects like pedestrians and cyclists), leading to better performance on them after fine-tuning (c.f. Table 6 in the main paper). In contrast, if this effect is reduced by SL1 or BalancedL1, the performance of cars can get better. Naive cross-entropy is also known to be biased towards major training examples in class-imbalanced learning. In our paper, the per frame re-weighting in BalancedSoftmax can reduce the effect of imbalance point distribution, leading to the best fine-tuning results for all classes.

[1] Pang, Jiangmiao, et al. "Libra r-cnn: Towards balanced learning for object detection." CVPR. 2019.

Point Sampling for hints. Thank you for the interesting suggestion. In the main paper, the hints provided in the pre-training are completely random. We agree that, during pre-training, it may not be necessary to provide as many hints for points located on the ground, and instead, we should focus more on foreground points. One potential solution is to sample fewer points from the major color classes. For example, the number of sampled points per class is set proportional to the square root of the total number of points per class. We have implemented the idea and reported the results of fine-tuning on 5% KITTI in the table below. We found no significant difference with more balanced sampling. However, we agree that smarter sampling is indeed interesting for future research and exploration.

More related work about colorization. We sincerely appreciate your valuable comments and references. We will certainly cite and discuss them, and include more papers on image colorization-based pre-training, image colorization with seeds, and general point cloud colorization. In the following, we provide a short summary and discussion of the literature.

Image colorization has been an active research area in computer vision [1-13], predicting colorful versions of grayscale images. Typically, this task relies on human-related cues such as scribbles [9], exemplars [7], or texts [8], as well as external priors like pre-trained detectors [3], generative models [10], or superpixels [11, 18], to achieve realistic and diverse colorization. [14-17] have extended the task to 3D point clouds. Specifically, [13, 16] uses a GAN-style loss, learning a discriminator to differentiate real colorful images and colorized images and using it to train the colorization model. This provides an alternative to using hints to resolve the color variation problem.

Instead of treating colorization as the ultimate task, [1, 12, 13] have explored colorization as a pre-training task for 2D detection and segmentation. We note that [1, 12] do not specifically address the color variation problem.

In our paper, we introduce colorization as the pre-training task to equip the point cloud representation with semantic cues useful for the downstream 3D LiDAR-based detectors. We identify color variation as the intrinsic challenge and propose inserting color hints into the output of the backbone to ground the colorization taking place in the color decoder (cf. Figure 2). We note that such a design is crucial—we cannot add color hints as input to the backbone as the downstream task takes only the LiDAR points as input. In contrast, for colorization methods not for pre-training, the hints can be directly added to the input gray-level image [9]. [18] proposed a hybrid method to first predict the color of each superpixel pixel using a neural network, and propagated them to nearby pixels using a hand-crafted method (e.g., 2D-PCA). In contrast, our GPC learns to propagate colors in an end-to-end fashion by directly minimizing the colorization loss, which is the key to benefit downstream 3D detection.

[1] Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." ECCV, 2016.

[2] Larsson, Gustav, Michael Maire, and Gregory Shakhnarovich. "Learning representations for automatic colorization." ECCV. 2016.

[3] Su, Jheng-Wei, Hung-Kuo Chu, and Jia-Bin Huang. "Instance-aware image colorization." CPVR. 2020.

[4] Kumar, Manoj, Dirk Weissenborn, and Nal Kalchbrenner. "Colorization transformer." arXiv. 2021.

[5] Weng, Shuchen, et al. "CT 2: Colorization transformer via color tokens." ECCV. 2022.

[6] Ji, Xiaozhong, et al. "ColorFormer: Image colorization via color memory assisted hybrid-attention transformer." ECCV. 2022.

[7] He, Mingming, et al. "Deep exemplar-based colorization." TOG. 2018.

[8] Chang, Zheng, et al. "L-CoIns: Language-Based Colorization With Instance Awareness." CVPR. 2023.

[9] Levin, Anat, Dani Lischinski, and Yair Weiss. "Colorization using optimization." ACM SIGGRAPH 2004 Papers. 2004. 689-694.

[10] Kim, Geonung, et al. "Bigcolor: colorization using a generative color prior for natural images." ECCV. 2022.

[11] Xia, Menghan, et al. "Disentangled image colorization via global anchors." TOG. 2022.

[12] Larsson, Gustav, Michael Maire, and Gregory Shakhnarovich. "Colorization as a proxy task for visual understanding." CVPR. 2017.

[13] Treneska, Sandra, et al. "Gan-based image colorization for self-supervised visual feature learning." Sensors 22.4 (2022): 1599.

[14] Cao, Xu, and Katashi Nagao. "Point cloud colorization based on densely annotated 3d shape dataset." MultiMedia Modeling: 25th International Conference. 2019.

[15] Liu, Jitao, Songmin Dai, and Xiaoqiang Li. "Pccn: Point cloud colorization network." ICIP. 2019.

[16] Shinohara, Takayuki, Haoyi Xiu, and Masashi Matsuoka. "Point2color: 3D point cloud colorization using a conditional generative network and differentiable rendering for airborne LiDAR." CVPR. 2021.

[17] Gao, Rongrong, et al. "Scene-level Point Cloud Colorization with Semantics-and-geometry-aware Networks." ICRA. 2023.

[18] Wan, Shaohua, et al. "Automated colorization of a grayscale image with seed points propagation." IEEE Transactions on Multimedia 22.7 (2020): 1756-1768.

Baseline approaches. In the main paper, we aimed to present our colorization idea in a clear and accessible way, allowing readers to grasp the core concept easily. Point-based detectors (e.g., Point R-CNN and IA-SSD) were chosen for the purpose. That said, our GPC can definitely be applied to voxel-based detectors, after a voxel-to-point operation. (Such an operation has been used in LiDAR point segmentation when a voxel-based backbone is adopted.) To illustrate this, we have conducted 3D object detection experiments with PV-RCNN on Waymo -> KITTI. The results are summarized in the table below. GPC consistently outperforms the existing algorithms. We appreciate your suggestion and we will incorporate these new results into the final version.

STRL: Huang et al. "Spatio-temporal self-supervised representation learning for 3d point clouds." ICCV. 2021.

ALSO: Boulch et al. "Also: Automotive lidar self-supervision by occupancy estimation." CVPR. 2023.

GCC-3D: Liang et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." ICCV. 2021.

PatchContrast: Shrout et al., "PatchContrast: Self-Supervised Pre-training for 3D Object Detection." arXiv preprint arXiv:2308.06985.

More comparisons in Table 1 and Table 2. We provide more experimental results of pre-training on Waymo and fine-tuning on KITTI (cf. Table 2), using PV-RCNN as the 3D detector. (We note that many SOTA results use PV-RCNN.) As shown in the following table, GPC consistently outperforms existing algorithms. The existing SOTA algorithms primarily utilize contrastive learning (e.g., ProposalContrast and PatchContrast). They require specific designs (i.e., proposal encoding module, patch extraction module, etc.) to identify meaningful regions to contrast. GPC, in contrast, offers a fresh perspective on pre-training for 3D detectors. It is straightforward in its approach yet remarkably effective. GPC is particularly advantageous when the labeled data are scarce. Regardless of the 3D model (i.e., PointRCNN or PV-RCNN) and the pre-training dataset (i.e., Waymo or KITTI), GPC’s performance of fine-tuning on just 20% of labeled data is consistently better than training from scratch with the entire dataset (cf. Table 1 and Table 2 and the following table). This advantage holds the potential to reduce the annotation effort required. We will include the new results and more discussion in the final version.

STRL: Huang et al. "Spatio-temporal self-supervised representation learning for 3d point clouds." ICCV. 2021.

ALSO: Boulch et al. "Also: Automotive lidar self-supervision by occupancy estimation." CVPR. 2023.

GCC-3D: Liang et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." ICCV. 2021.

PatchContrast: Shrout, Oren, et al. "PatchContrast: Self-Supervised Pre-training for 3D Object Detection." arXiv preprint arXiv:2308.06985. 2023.

We appreciate your positive feedback and your insightful questions. To clarify them, we have conducted more experiments and ablations, as shown below.

Color assignment. Thanks for the question. For LiDAR points that are outside the camera's field of view and cannot be projected onto the image, we do not calculate loss on them during pre-training (cf. Eq 1 and Eq 4). Specifically for the KITTI dataset, since the 3D object detection metric only considers objects that appear within the camera's field of view, it is common to input only the corresponding LiDAR points into the 3D object detector (Ref. [1]).

For LiDAR points that are within the camera’s field of view, we would require a good calibration between the camera and LiDAR to ensure correct color assignments. To our knowledge, existing datasets do provide well-calibrated data (e.g., for studying sensor fusion). Even if one wants to collect pre-training data on their own, there is well-documented code to calibrate LiDAR and cameras with checkerboards.

We thank the reviewer for pointing out another issue: some LiDAR points may be projected onto the same pixel location, for example, due to sensor errors or decimal precision. We examined the KITTI dataset and found that, on average, about 0.18% of points have this issue. Compared to the label error rate in common datasets (e.g., 5.83% in ImageNet, 5.85% in CIFAR-100) (Ref. [2]), it is relatively low. To address this issue, one can remove these confusing points in pre-training. As shown in the following tables (with fine-tuning on 5% KITTI), we see no notable difference between removing these points or not. However, it is worth keeping this issue in mind if applying GPC to other datasets or tasks.

[1] https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/dataset_configs/kitti_dataset.yaml#L17

[2] Northcutt, Curtis G., Anish Athalye, and Jonas Mueller. "Pervasive label errors in test sets destabilize machine learning benchmarks." arXiv preprint arXiv:2103.14749 (2021).

Ablation study regarding the number of color bins. Thanks for the question. We conducted a study on fine-tuning with 5% KITTI (cf. top row of Table 1 in the main paper), using 32, 64, and 128 color bins in pre-training. The results are summarized in the following table. We found no significant difference in the downstream detection task with different color bins in pre-training. We hypothesize two reasons. First, our pre-training goal is to equip the backbone with the knowledge to identify which subset of points should possess the same color and be segmented together. In this sense, it could be less sensitive to the number of color bins as long as we have enough of them. (As in Fig. 3, images with 32 bins still show high contrast between objects and the background.) Second, the pre-trained backbone will still be optimized with the downstream task data. We will include the ablation in the final version.

The reweighted loss vs. transferability. This is an interesting question. We indeed have results with pre-training on Waymo and fine-tuning on KITTI (cf. Table. 2 in the main paper). GPC consistently outperforms baselines and performs similarly to pre-training with KITTI, demonstrating the transferability. We will provide more results with PV-RCNN in the next response.

We thank you for the insights about transferability and the distribution shifts between datasets. (This issue also exists in 2D recognition, e.g., ImageNet pre-training transferred to COCO detection.) We respectfully think the reweighting loss could mitigate the distribution shift to facilitate transferability. As discussed in the literature on class-imbalanced learning and long-tailed recognition, learning a model without re-weighting would result in a model attempting to predict the major classes in the dataset—essentially capturing the dataset-specific bias. Learning with the re-weighting loss (cf. Eq 5) would reduce this effect. When a huge distribution shift exists between datasets, one can use the quantization bins obtained from the target dataset for pre-training with the source data.

Thank you for all the additional comments. I'm voting for this paper to be accepted and hope the AC agrees! Looking forward to more papers like this in the future :)