MixSup: Mixed-grained Supervision for Label-efficient LiDAR-based 3D Object Detection

We sincerely appreciate your generous suggestions and the thoughtful questions you've provided. We highly value your feedback and will provide detailed explanations to address your concerns.

W1: The illustration seems not very relevant to the motivation and the concept of the study.

We sincerely apologize for any potential confusion caused by the introduction. In the upcoming revised version, we will present it in a clearer and more precise manner. It's important to clarify that the images in the introduction are intended to offer readers a more vivid and intuitive depiction of the distinctive properties of point clouds.

W2: The detail of pilot study is not quite clear.

Firstly, we sincerely apologize for any lack of clarity in the pilot study. We indeed trained an spconv-based detector on well-classified regions using varying amounts of data to validate our claim. The purpose of using well-classified regions for training is to factor out the impact on semantic learning, while only focusing on the impact on geometry estimation. Certainly, as you rightfully noted, compared to scene-level point clouds, region-level data does contain less contextual information. However, we believe such a difference does not weaken our conclusion in the pilot study for the following two reasons.

• Geometry estimation (regression) does not necessitate too much spatial context. The second stage detector LiDAR-RCNN [R1], employs region-level point clouds to predict accurate object poses. It only incorporates context by enlarging the proposal dimension by 1.5 meters on each side. Its success demonstrates that precise pose estimation does not necessitate an excessive amount of contextual information. Other two-stage detectors, such as FSD[R2], also leverage points inside proposals to make accurate pose refinement with a small proposal enlargement (e.g., 0.5m). Therefore, our experiments based on region-level data to assess the impact of data amount on regression performance are reasonable.
• Our region-based network also utilizes necessary context information. We adopted a proposal enlargement strategy similar to LiDAR-RCNN, providing sufficient contextual information for regression.

W3: The motivation is simply another way of expressing a common phenomenon in object detection, which is the high recall often contribute more on high average precision (AP).

We acknowledge that this phenomenon is reasonable in 2D object detection, and we truly thank the reviewer for pointing it out. But it's crucial to note that 3D detection is more sensitive to regression quality compared to its 2D counterpart, due to the following two reasons:

• In the 3-dimensional space, the additional spatial dimension makes achieving high 3D IoU more challenging than achieving high 2D IoU.
• 3D object detection requires the consideration of orientation angles, which introduces an additional freedom in the matching process between predictions and GTs.

Taking the widely used autonomous driving dataset Waymo as an example, Waymo sets the official IoU threshold as 0.7 for vehicle, and proposes the metric APH that incorporates heading as a weighted factor, which is a strict metric for evaluating a 3D object detector.

To vividly illustrate the rigorous evaluation of geometry attributes in 3D object detection, we conduct experiments with a pre-trained CenterPoint model. Specifically, we introduce different noises to the predicted boxes.

The introduced noise (e.g., 0.05m) is minor because vehicles are usually more than 4 meters long. Experimental results demonstrate that even minor geometry attribute deviations can lead to significant performance drops under the strict metric. This highlights the need for a good 3D detector with strong geometry estimation capabilities, as high recall alone may not guarantee good performance.

References

• [R1] Li, Zhichao, et al. "Lidar r-cnn: An efficient and universal 3d object detector." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
• [R2] Fan, Lue, et al. "Fully sparse 3d object detection." Advances in Neural Information Processing Systems 35 (2022): 351-363.
• [R3] Meng, Qinghao, et al. "Weakly supervised 3d object detection from lidar point cloud." European Conference on computer vision. Cham: Springer International Publishing, 2020.
• [R4] Meng, Qinghao, et al. "Towards a weakly supervised framework for 3d point cloud object detection and annotation." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.8 (2021): 4454-4468.
• [R5] Shi, Shaoshuai, et al. "Pv-rcnn: Point-voxel feature set abstraction for 3d object detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

W4: Why not just using a region-based network (e.g. the second stage PointNet in LiDAR-RCNN) to refine these boxes as precise box-level supervision?

We greatly appreciate your insightful suggestion. It is an insightful suggestion, but there are some potential drawbacks:

• Pretrained region-based detectors cannot be directly utilized. There are domain gaps between the human-labeled cluster labels and the input regions of the region-based detector. For example, the input of LiDAR-RCNN comes from the output proposals of other detectors with being enlarged, which is quite different from our cluster labels. So it is infeasible to directly utilize an off-the-shelf pretrained region-based detector. Moreover, the diverse data distributions of different datasets make it more infeasible to use pretrained region-based detectors from other datasets. How to improve the transferring performance of region-based detectors remains open.

• Transforming coarse labels to precise boxes for supervision introduces inherent errors, reducing the performance upper bound. This is because the predicted precise boxes cannot be as good as human annotations. In contrast, although our coarse cluster labels do not have accurate geometry information, the semantic label for each cluster remains precise. Furthermore, the regression learning in MixSup only relies on accurate box-level labels. Thus, although we adopt coarse labels, the supervision in our method does not have such inherent errors and potentially offers a higher performance upper bound.

Although there are these drawbacks, your suggestion is indeed very insightful. We will try to make it more feasible in the future, for example by improving the transferability of region-based detectors.

W5: MV-JAR can achieve very close performance to author's with 10% annotations.

Thank you for expressing your concerns, and we appreciate the opportunity to clarify this point. It's essential to note that in Table 4, MixSup adopts SECOND as the base detector rather than SST, which is a better detector than SECOND. Therefore, the marginal improvement of MixSup over MV-JAR in Table 4 is reasonable.

To further address your concerns, we also take SST as the base detector to compare with MV-JAR fairly. As shown in the following table, MixSup outperforms MV-JAR with the same base detector, showing that using extra coarse labels leads to improvement.

Q1: What is the difference between the settings of 10% boxes annotation and 10% frames annotation?

“10% frames annotation” indicates using 10% of all annotated frames, where all objects in each used frame are labeled. “10% boxes annotation" means that the annotated boxes account for 10% of all boxes. It is noteworthy that, due to random sampling and the large scale of driving datasets (e.g., Waymo contains millions of annotated objects), 10% boxes annotation and 10% frames annotation involve almost the same annotation cost.

Q2: I suggest author can perform another experiment by training the semantic head by leveraging voxel-to-point interpolation (eg. used in PVCNN, SASSD) and imposing point-level supervision.

We sincerely appreciate your insightful suggestion. Your proposal is indeed quite reasonable.

In fact, our method has already employed such supervision in certain situations, such as PV-RCNN [R5] (tested on KITTI validation split with moderate difficulty). Specifically, PV-RCNN generates features for key points through the voxel set abstraction module, which is very similar to voxel-to-point interpolation, as you pointed out, from neighboring voxels.

Moreover, we supplement a point-based method FSD [R2], which adopts a segmentation network in the first stage to obtain foreground points with point-level supervision and test it on Waymo validation split with L2 APH.

Indeed, as you pointed out, MixSup incorporating point-level supervision has shown excellent performance. We sincerely appreciate your valuable suggestion!

*: Using coarse cluster-level labels and 10% box-level labels. The percentage in parentheses indicates the performance ratio to the fully supervised counterpart

Thanks for your follow-up question, and truly sorry for causing such a misunderstanding. We clarify our settings and answer this question in the following aspects.

• We list some results in the following table, where the experiments adopt the same "annotation cost". Here the “annotation cost” is a comprehensive metric taking both accurate box annotation and coarse cluster annotation into account. Its details are also listed below.

• We further explain the details of “10% annotation cost”, which consists of the cost of box labels and coarse cluster labels. The following table shows the statistics.

As illustrated in Sec 5.3, we conducted a human labeling study by asking three experienced annotators to label 100 frames, containing thousands of objects. Based on the statistics gathered from the annotators, the average time cost of a coarse cluster label is 14% of an accurate box label. In this case, the overall annotation cost is evaluated using the following equation.

Thus, at the same annotation cost, the MixSup also shows superiority.

• Finally, we would like to clarify that there are two reasons why we put “10% boxes annotations + coarse labels” on the table in our paper.
  • The first reason is that we want to imply that the proposed MixSup could utilize “additional” coarse labels, which is the core feature of MixSup.
  • The second reason is that we would like to offer readers a more comprehensive understanding of our performance.

We hope our explanation could resolve your concern. If not, we are open to any further discussion. If it does, we would greatly appreciate it if you could kindly offer us a positive rating. Many thanks in advance.

References

[R6] Sohn, Kihyuk, et al. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." Advances in neural information processing systems 33 (2020): 596-608.

[R7] Yin, Junbo, et al. "Semi-supervised 3D object detection with proficient teachers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[R8] Xu, Runsen, et al. "MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

References

• [R1] Zhang, Dingyuan, et al. "SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model." arXiv preprint arXiv:2306.02245 (2023).
• [R2] Liu, Youquan, et al. "Segment Any Point Cloud Sequences by Distilling Vision Foundation Models." arXiv preprint arXiv:2306.09347 (2023).
• [R3] Chen, Runnan, et al. "Towards label-free scene understanding by vision foundation models." Thirty-seventh Conference on Neural Information Processing Systems. 2023.
• [R4] Meng, Qinghao, et al. "Weakly supervised 3d object detection from lidar point cloud." European Conference on computer vision. Cham: Springer International Publishing, 2020.
• [R5] Meng, Qinghao, et al. "Towards a weakly supervised framework for 3d point cloud object detection and annotation." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.8 (2021): 4454-4468.
• [R6] Chen, Kai, et al. "Hybrid task cascade for instance segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
• [R7] Xia, Qiming, et al. "CoIn: Contrastive Instance Feature Mining for Outdoor 3D Object Detection with Very Limited Annotations." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

We sincerely appreciate your questions and suggestions for our work. It requires considerable effort to meticulously read the paper and provide insightful and detailed questions and feedback. We believe these suggestions hold significant value for our work.

W1: A number of assertions lack rigorous justification or analytical backing.

Thanks for your insightful suggestion regarding the need for rigorous justification and analytical backing. In response, we will enhance the manuscript to ensure a more comprehensive and transparent presentation of the underlying rationale. The revision will be reflected in both the responses to Questions and the upcoming edited version of the manuscript.

W2: The paper omits certain pivotal implementation specifics.

Due to the page limit, we couldn't elaborate on all implementation details. However, we appreciate your suggestion, and we will carefully include detailed implementation of crucial steps. In addition, we commit to open-sourcing the code for reference in the future.

Q1: Elucidation about the term “data size” in pilot study.

We sincerely apologize for the confusion caused by the term “data size”. We will replace it with “data amount” to convey that it refers to the amount of cropped regions used in training the SparseConv Detector, rather than the size of objects or instances. By changing the amount of training data, we demonstrate that geometry estimation does not necessitate a large amount of data.

Moreover, we supplement experiments to enhance the claim that “massive data is necessary for semantic learning”. More details can be found in Response to Reviewer yY5M (Part 1) W2.

Q2: The mechanics of the “clicking annotation”.

We will address your questions regarding the mechanics of the “clicking annotation” as follows.

• “Clicking annotation” for label-efficient 3D detection is first proposed by Meng et al. [R4], [R5]. They require annotators to click object centers in the BEV perspective, which are used to supervise the center prediction.
• In the process of labeling, there is no difference made for different instance types.
• For the last question about “Was manual labeling extended to all point clouds and how were the crucial clicking points discerned”. We are not sure we understand the reviewer’s question accurately. Our understanding is that the reviewer wants to know if the “click annotation” annotates all the objects in the training dataset. If we understand accurately, the answer is no. Using their implementation as an example, they randomly choose 500 frames from KITTI, where the centers of all instances are labeled in each frame. If our understanding is not accurate, we are open to any further discussions at anytime.

Below we attach the comparison with Meng et al. [R4], [R5] to showcase our superiority. The evaluation results on KITTI val set (Car) with 534 box-level labels along with weakly labeled annotations.

Q3 & Q4: SAR and CCL

Thank you for raising questions about contents that were not clearly elucidated. We sincerely apologize for any inconvenience caused by this. Since SAM's over-segmentation and nuScenes' imprecise point-pixel projection result in mediocre 3D instance masks, we leverage the spatial separability of objects in point clouds to refine the segmentation with Separability-Aware Refinement (SAR), which is demonstrated in the response to Q7. In SAR, the Connected Components Labeling (CCL) is a conventional operation using Euclidean distance for connectivity as outlined in https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csgraph.connected_components.html. Our code will be open-sourced for reference.

Q5: pre-trained semantic segmentation model

The model we employed is the HTC [R6] pre-trained on nuImages dataset. In automated coarse labeling, we solely utilized its semantic segmentation head to obtain semantics for each pixel. There is no need for concern regarding data leakage, as the author of nuScenes have explicitly stated that the overlap between nuImages and nuScenes is negligible in https://github.com/nutonomy/nuscenes-devkit/issues/550#issuecomment-769571712

Q6: Distinct advantages when juxtaposed against other methodologies utilizing SAM for segmenting objects from LiDAR point clouds.

To the best of our knowledge, it is the first initiative harnessing SAM for instance segmentation in outdoor scenes. Notably, ACL achieves performance on par with the recent fully supervised panoptic segmentation models without the need for any 3D annotations. Compared to these methods, our Automated Coarse Labeling (ACL) innovatively leverages the inherent spatial separability of point clouds to refine instance segmentation, enabling the mitigation of the projection error (shown in response to Q7).

In contrast, [R1] is confined to detecting the "car" category, while [R2] and [R3] focus on pretraining via pixel-point projection for semantic segmentation without addressing instance segmentation. It is crucial to emphasize that ACL is proposed to provide coarse labels for MixSup.

We are delighted to see exceptional works like [R2] and [R3] playing a similar role and potentially integrating with MixSup. Finally, we appreciate your reminder and will add a detailed comparison with them in the revised version.

Q7: Inaccurate calibration

We sincerely appreciate your suggestion for experiments regarding the inaccurate calibration. Throughout our experimentation, we identified inherent inaccuracies in the calibration of nuScenes, leading to discrepancies in the pixel-point projection of foreground objects. Hereby, we introduce SAR module to mitigate the degradation in segmentation caused by projection error.

To further investigate the impact of projection discrepancies on automated coarse labeling, we introduced random noise to each camera's position. The performance for foreground objects on nuScenes val split panoptic segmentation is summarized in the table below. The results indicate that SAR-derived coarse instance masks exhibit a certain degree of resilience to calibration inaccuracies, attributed to the refinement introduced by the SAR module.

Q8: Analyze some of the latest entrants in the domain of weakly and semi-supervised 3D object detectors

• We highly value your suggestion and have enriched the comparisons with relevant works. To incorporate more entrants, we find a new work CoIn [R7] published in ICCV2023, with a 5% annotation cost on the KITTI validation split, evaluating Car-3D AP R40. It's worth noting that the 3.3% annotation cost for MixSup in the experiment is borrowed from other experiments. Despite having a lower annotation cost than CoIn, we achieve higher AP in both moderate and hard difficulties. We will include the comparison in the revised manuscript.

• It's also worthwhile to note that MixSup is orthogonal and complementary to semi-supervised learning, such as a simple self-training as highlighted in Sec 5.5 Table 6 in the paper. We will delve into the combination of MixSup and other semi-supervised training schemes in future work.

Q9: Formatting tables.

Thank you for your generous suggestion. We will modify tables in the subsequent manuscript.

Q10 Discussing the limitations and potential negative social impact of this work.

Regarding the limitations of MixSup, as a novel label-efficient paradigm, it is orthogonal to other label-efficient methods. Our work has not explored integration with semi-supervised methods, providing an avenue for potential performance enhancements. Moreover, our proposed automated coarse labeling, when combined with other exceptional 3D segmentation models, could yield higher-quality coarse labels. We believe MixSup holds significant promise in reducing annotation costs for the community, contributing to the conservation of both human and environmental resources. However, due to the cost savings in annotation come with a certain performance trade-off, practical deployment may raise risks of compromising driving safety.

We are more than willing to continue engaging in in-depth discussions. If you have any further questions, please feel free to comment at any time.

We have updated the response to Q2 and Q8. We are open to any further discussions at any time.

References

• [R1] Meng, Qinghao, et al. "Weakly supervised 3d object detection from lidar point cloud." European Conference on computer vision. Cham: Springer International Publishing, 2020.
• [R2] Meng, Qinghao, et al. "Towards a weakly supervised framework for 3d point cloud object detection and annotation." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.8 (2021): 4454-4468.
• [R3] Fan, Lue, et al. "Fully sparse 3d object detection." Advances in Neural Information Processing Systems 35 (2022): 351-363.
• [R4] Xu, Runsen, et al. "MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
• [R5] Fan, Lue, et al. "Embracing single stride 3d object detector with sparse transformer." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
• [R6] Liang, Hanxue, et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
• [R7] Yin, Tianwei, et al. "Center-based 3d object detection and tracking." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

W3: The study on the labeling costs is not fully grounded.

We highly value your questions and try to resolve your reasonable concerns from the following aspects.

• As described in Sec 5.3, we conducted a study by asking three experienced annotators to label 100 frames from different sequences of nuScenes. “The average time cost of a coarse cluster label is only 14% of an accurate box label” is based on the statistics gathered from the annotators.

• In the original paper, we simulate the coarse labels by randomly expanding GT boxes 0~10% in each dimension without evaluating the similarity with human coarse annotation. Thank you so much for pointing this out! To be more rigorous, here we first evaluate the similarity between human-annotated coarse labels (100-frame subset) and ground-truth labels, which is 81.22% in terms of segmentation mIoU. Then we add noise to ground-truth labels to simulate more rigorous coarse labels, controlling coarse labels possessing 80% to 82% mIoU with ground-truth labels, just like the human-annotated coarse labels. In particular, we use the following two noises:

  • Randomly shifting center locations $-0.1\sim 0.1$ m and expanding 0%$\sim$50% in each dimension (Noise ①). This noise has 81.91% mIoU with the ground-truth label.
  • Randomly shifting center locations $-0.2\sim 0.2$ m, expanding 0%$\sim$ 20%in each dimension, and rotating $-15^\circ\sim15^\circ$ in heading (Noise ②). This noise has 80.60% mIoU with the ground-truth label.

  We then conduct experiments on nuScenes using such noisy coarse labels, results shown in the following table. Performance with noise ①, ② is similar to the performance in the original paper. Thus, our simulated coarse labels are reliable for the experiments in our paper.

• As for the center-click annotation, we unintentionally overlooked the human labeling study because it is not a core part of our method. We cannot launch and finish a labeling study during the limited rebuttal period without the labeling tools in WS3D. We are truly sorry for that. However, we demonstrate that center-click annotation has some limitations.
  • For the inevitable instances with very partial point clouds, accurately clicking the center is challenging. Conversely, MixSup's cluster-level label only requires clicking around the visible part of an object with three points, ensuring that the generated parallelogram encompasses the partial point clouds.
  • Some methods need more than centers, such as the recent state-of-the-art FSD [R3]. It adopts a segmentation network in the first stage to obtain foreground points, requiring point-level supervision. Here we attach the performance on Waymo validation split with L2 APH using FSD as the base detector to show the effectiveness of MixSup, where the percentage in parentheses indicates the performance ratio to the fully supervised counterpart.

W4: Presentation.

We sincerely appreciate your meticulous reading of our work. Your understanding of "semantic," "geometry," and "assignment" is highly accurate. We definitely believe good papers necessitate high-quality writing and are truly sorry for the unclear expressions. We highly value your suggestions and will make revisions in the upcoming manuscript.

Q1: PointContrast and the ProposalContrast have negative performance gains.

We apologize for any confusion caused. In our paper, we took a recent method MV-JAR [R4] (CVPR 23) as a compared method. The results of PointContrast and ProposalContrast in our paper are reported by MV-JAR. For a direct comparison, we compare MixSup with the results in the original ProposalConstrast paper as follows. Using 1% boxes and coarse labels, MixSup is slightly better than them with 20% frame annotations. Due to the different settings, we cannot perfectly align the annotation cost with them, but the following results suggest MixSup could achieve promising results.

We are more than willing to continue engaging in in-depth discussions. If you have any further questions, please feel free to comment at any time.

We sincerely appreciate your insightful questions and we believe they hold significant value for our work. We try to resolve your concerns below.

W1: Further discussion and comparison between the proposed work [R1, R2] and this work.

We apologize for unintentionally overlooking the comparison as our focus was primarily on Waymo and nuScenes. Here we supplement a discussion and experimental comparison with WS3D [Meng et al. 2020/2021] as follows.

• WS3D has a specifically designed detection pipeline and cannot be generalized to other detectors. In contrast, MixSup can be integrated with various detectors.
• The coarse cluster-level labels proposed in our method can be obtained through foundational models such as SAM, as we demonstrated in Automated Coarse Labeling.
• For the annotation cost, we are truly sorry that we cannot provide a rigorous quantitative comparison with center-click in WS3D because the cost largely depends on labeling tools and protocols, but we do not have the labeling tools in WS3D. However, we provide the analysis and discussion in the following W3.
• At the same annotation cost, our method has better performance. In the following Table, we report the performance of Car detection in KITTI val split since WS3D does not use Waymo and nuScenes dataset.

W2: "Massive data is only necessary for semantic learning" seems not fully grounded.

We greatly appreciate your insightful feedback. We supplement the experiments to enhance this claim.

Specifically, we decrease the data amount (#. training frames) from 100% to 10% in Waymo and train a popular detector CenterPoint [R7]. Such a decrease in data amount has a negative impact on both geometry learning and semantic learning. To mitigate the impact on geometry learning and only focus on semantic learning, we aggressively relax the IoU thresholds in evaluation from 0.7 / 0.5 / 0.5 to 0.5 / 0.25 / 0.25 for vehicle/pedestrian/cyclist, respectively.

Let’s first look at the following table, there are huge performance gaps between using 100% data and using 10% data. Obviously, here these gaps are caused by the degradation of both semantic learning and geometry estimation.

Then let’s look at the following table, where we relax the IoU thresholds for evaluation to mitigate the negative impact introduced by degradation of geometry estimation. The overall performance becomes much higher after relaxing IoU thresholds. However, there are still significant gaps between using 100% data and using 10% data. Especially, for Pedestrian and Cyclist, the gaps (between 10% and 100%) do not even get smaller after we relax the IoU thresholds. Thus, the performance gaps between 100% data and 10% data should be mainly caused by the degradation of semantic learning. In other words, semantic learning is sensitive to the data amount.

From another point of view, in the main paper, we have decreased the data amount of well-classified patches from 100% to 10% to reveal the impact on geometry estimation, as shown in the following table. Compared with the aforementioned performance change caused by semantic learning degradation, the performance change in the following table is negligible. Thus, we draw the conclusion that “LiDAR-based detectors indeed only need a very limited number of accurate labels for geometry estimation. Massive data is only necessary for semantic learning”.

The following table: Performances with varying data amounts on the well-classified dataset (Copy from main paper).

The discussion above will be included in the edited version.