Unleashing the Potential of Unlabeled Data: Bidirectional Collaborative Semi-Supervised Active Learning for 3D Object Detection

Dear Reviewer Dsaa,

We sincerely appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below to clarify any misunderstandings. Please let us know if you have any further questions or concerns.

Q1: The difference between our CPSP and SS3D[1].

A1: While our CPSP shares some conceptual similarities with SS3D, it differs fundamentally in both design and purpose. The key distinction between our CPSP and existing SS3D lies in how pseudo-objects are selected for training. SS3D is designed for a sparse-supervised setting, focusing on extracting as many unlabeled objects as possible, whereas CPSP is specifically tailored to support the AL process. This fundamental difference leads SS3D to overlook two critical aspects that are central to CPSP: the box bank deletion mechanism and the use of backgrounds likely to be false positives (FPs) for training. Let’s delve into these two factors to illustrate their importance.

• Box Bank Deletion Mechanism:

  No matter how confident or refined the filtering process is, pseudo-labels in the box bank inevitably contain errors and noise. Without a mechanism to delete these noisy labels, they propagate throughout training, resulting in confirmation bias that adversely affects uncertainty measurement. CPSP incorporates a robust deletion mechanism to address this issue, ensuring cleaner pseudo-labels and better AL outcomes.

• Use of Backgrounds Likely to Be FPs:

  CPSP utilizes background regions that are likely to be FPs, providing negative signals to the model. These negative signals help the model avoid overconfidence in incorrect predictions, thereby improving the reliability of uncertainty measures.

To further demonstrate these differences, we conducted experiments by directly applying SS3D within our framework. The results, presented below, compare the mAP and D-ECE (calibration, where lower is better) metrics for various configurations:

Table I: mAP comparison across settings on Waymo

Table II: D-ECE comparison across different pre-training methods on Waymo

The results indicate that SS3D negatively impacts calibration (Table II), particularly for harder classes (pedestrians and cyclists), thereby limiting its effectiveness in supporting the AL process (Table I). CPSP, on the other hand, demonstrates significant improvements in both performance and calibration, validating the necessity of its design choices.

Q2: The motivation of the conflict between SSL and AL.

A2: Actually, there are many evidences in our paper showing the conflict between SSL and AL. In Tables 1 and 2, various SSL pre-training methods combined with AL methods often perform worse than their counterparts (Normal + AL). We also provide a detailed analysis in Section 4.4.3 and Appendix E.1, showing that many SSL methods can negatively impact model calibration, leading to unreliable uncertainty measures that affect AL performance.

Additionally, we provide results below for a naive combination of SSL and AL on KITTI, further showcasing this conflict, which is often overlooked by existing SSL + AL frameworks.

Tabel III: Results of naive combinations of SSL and AL on KITTI

These results demonstrate that directly combining SSL—whether using the SOTA HSSDA method or a simpler approach like 3DIoUMatch—with AL often leads to performance degradation, highlighting the need for a carefully designed framework like ours to effectively resolve this conflict.

Q3: About the figure.

A3: Thank you for your feedback on the figures. We appreciate your suggestion and will carefully revise and enhance the figures to improve their clarity and aesthetic quality in the final version of the paper.

Dear Reviewer BLqE,

We sincerely appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below to clarify any misunderstandings. Please let us know if you have any further questions or concerns.

Q1: The difference between our CPSP and SS3D[1].

A1: While our CPSP shares some conceptual similarities with SS3D, it differs fundamentally in both design and purpose. The key distinction between our CPSP and existing SS3D lies in how pseudo-objects are selected for training. SS3D is designed for a sparse-supervised setting, focusing on extracting as many unlabeled objects as possible, whereas CPSP is specifically tailored to support the AL process. This fundamental difference leads SS3D to overlook two critical aspects that are central to CPSP: the box bank deletion mechanism and the use of backgrounds likely to be false positives (FPs) for training. Let’s delve into these two factors to illustrate their importance.

• Box Bank Deletion Mechanism:

  No matter how confident or refined the filtering process is, pseudo-labels in the box bank inevitably contain errors and noise. Without a mechanism to delete these noisy labels, they propagate throughout training, resulting in confirmation bias that adversely affects uncertainty measurement. CPSP incorporates a robust deletion mechanism to address this issue, ensuring cleaner pseudo-labels and better AL outcomes.

• Use of Backgrounds Likely to Be FPs:

  CPSP utilizes background regions that are likely to be FPs, providing negative signals to the model. These negative signals help the model avoid overconfidence in incorrect predictions, thereby improving the reliability of uncertainty measures.

To further demonstrate these differences, we conducted experiments by directly applying SS3D within our framework. The results, presented below, compare the mAP and D-ECE (calibration, where lower is better) metrics for various configurations:

Table I: mAP comparison across settings on Waymo

Table II: D-ECE comparison across different pre-training methods on Waymo

The results indicate that SS3D negatively impacts calibration (Table II), particularly for harder classes (pedestrians and cyclists), thereby limiting its effectiveness in supporting the AL process (Table I). CPSP, on the other hand, demonstrates significant improvements in both performance and calibration, validating the necessity of its design choices.

Q2: Validation of our CAL module.

A2: In fact, the effectiveness of our active learning (CAL) module has already been demonstrated in various settings, as shown in Tables 1 and 2 in the paper, where it outperforms other active learning methods, including KECOR[2] and CRB[3].

Additionally, we provide a direct comparison with CRB and KECOR below, further showcasing the superiority of our method.

Tabel III: Performance comparison of AL on KITTI

Tabel IV: Performance comparison of AL on Waymo

Q3: Detailed explaination about conflicts and bidirectional collaboration.

A3.1 (Conflicts):

Clarification of the conflicts: The conflicts primarily arise from learning with unconfident objects, which are characterized by high uncertainty. Assigning incorrect pseudo-labels to these objects in SSL introduces significant noise, making it difficult for AL to reliably assess their uncertainties.

Consequences of the conflicts: As demonstrated in Section 4.4.3 and Appendix E.1, previous SSL methods negatively impact model calibration, resulting in unreliable uncertainty measures that hinder AL performance. This is evident in Tables 1 and 2, where these SSL pre-training methods combined with AL perform worse than their counterparts (Normal + AL), proving that these conflicts significantly affect performance.

Solution to the conflicts: Our solution is to selectively learn only from confident objects. Specifically, we generate pseudo-scenes from unlabeled data by including only confident objects while excluding those with high uncertainty. Pre-training on these pseudo-scenes ensures that unconfident objects are not influenced by their own noisy pseudo-labels, thereby significantly mitigating the negative impact of mislabeling and noise. As shown in Section 4.4.3 and Appendix E.1, our approach achieves better calibration across all classes. This improved calibration supports AL methods in achieving superior results, as demonstrated in Tables 1 and 2.

A3.2 (Bidirectional Collaboration):

The design of our method is bidirectional collaborative, reflecting the two-way interaction between SSL and AL:

• From SSL to AL: We address the conflicts discussed above, ensuring SSL pre-training effectively supports AL by focusing on confident objects and aligning uncertainty measures with AL strategies.
• From AL to SSL: The primary contribution lies in the integration of AL with our pre-trained CPSP model. One key challenge is that the CPSP pre-trained model may "forget" some original true and confident objects. To address this, we propose an ensemble strategy to provide more reliable uncertainty measures. Additionally, our approach tackles class imbalance, a critical issue in outdoor scenes. Our design helps reduce FPs, enabling the model to sample objects from the correct classes more effectively. The importance of this can be further understood in Reviewer VC1w's Q2.

Q4: Validation across different semi-supervised schemes.

A4: Firstly, HSSDA is a state-of-the-art SSL method, and our success with it demonstrates the effectiveness of our approach. Secondly, we have already validated the proposed scheme across different SSL methods in the Final Model Delivering (FMD) stage: Table 1 uses HSSDA, Table 2 uses CPSP, and Table F in the appendix also uses CPSP. These experiments provide strong evidence of our method’s effectiveness. Lastly, conducting additional experiments with multiple SSL methods in the FMD stage is extremely time-consuming due to the combination of various SSL and AL methods we evaluated (e.g., Table 1 and Table 2). Given resource constraints, we focused on the most representative settings.

Thank you for your response. I have carefully read the comments of the other reviewers as well as your reply. I agree with the majority of the reviewers that the proposed method is quite similar to existing methods, so I will maintain my rating.

Dear Reviewer wW9A,

We sincerely appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below to clarify any misunderstandings. Please let us know if you have any further questions or concerns.

Q1: The clarification about ‘Bidirectional Collaboration’.

A1: The design of our method is bidirectional, reflecting the two-way interaction between SSL and AL:

• From SSL to AL: We analyze why previous SSL methods often fail to integrate well with AL. The conflicts primarily stem from learning with unconfident objects, which introduces noise and unreliable uncertainty measures. To address this, our approach selectively learns only from confident objects by generating pseudo-scenes from unlabeled data that include confident objects while excluding those with high uncertainty. Furthermore, our uncertainty measure is aligned with the AL methods, ensuring consistency and better integration.
• From AL to SSL: The primary contribution lies in the integration of AL with our pre-trained CPSP model. One key challenge is that the CPSP pre-trained model may "forget" some original true and confident objects. To address this, we propose an ensemble strategy to provide more reliable uncertainty measures. Additionally, our approach tackles class imbalance, a critical issue in outdoor scenes. Our design helps reduce FPs, enabling the model to sample objects from the correct classes more effectively. The importance of this can be further understood in Reviewer VC1w's Q2.

Q2: The contributions and the difference between our CPSP and GT sampling.

A2: First, one of the key contributions of our paper is identifying and addressing the conflict between AL and SSL, a critical issue overlooked by many previous SSAL frameworks. Ignoring this conflict often leads to inferior performance, as shown in our analysis and experiments.

Furthermore, we believe our modules are not simple combinations of existing works but are specifically designed and carefully integrated to effectively resolve these conflicts. Each component is tailored to address the challenges inherent in combining AL and SSL, ensuring they work collaboratively to achieve superior performance. For example, in the case of CPSP, the key contribution lies in how we select suitable pseudo boxes for training during iterations. Unlike standard GT sampling, which is limited to GT objects , CPSP introduces a more nuanced and dynamic selection process, enabling the effective use of pseudo-labels while minimizing noise and ensuring robust performance.

Q3: The clarification about improvement.

A3: The goal of our method is to improve the overall mAP across all classes, especially in challenging caterories like pedestrians and cyclists, and our method consistently outperforms others on both KITTI and Waymo datasets. Considering the number of objects, especially in Waymo, the test samples across all classes are sufficient to validate the effectiveness of our approach. We believe the average mAP improvement shown in Tables 1 and 2 is substantial (such as compared to 3DIoUMatch, our method bring an improvement 2.6% mAP on KITTI and 1.9% mAP on Waymo) , achieved by only changing the training data.

Although 3DIoUMatch outperforms our method in the Car category on KITTI, this primarily stems from class imbalance. Specifically, in the setting with 350 labeled objects, 3DIoUMatch samples 264 Cars, 63 Pedestrians, and 21 Cyclists, while our method samples 181 Cars, 128 Pedestrians, and 38 Cyclists. This gives 3DIoUMatch an advantage in the Car category due to over-sampling but results in weaker performance for other classes. We believe their sampling distribution is not ideal for object detection tasks where all classes are important. When we adjusted our sampling strategy by assigning significantly higher weights to the Car class (~260 Cars), our method achieved an 81.6 mAP in Car_mod, surpassing 3DIoUMatch (80.8). This result demonstrates that our method can achieve competitive performance in individual categories while maintaining balanced performance across all classes.

Dear Reviewer VC1w,

We sincerely appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below. Please let us know if you have any further questions or concerns.

Q1: The difference between our CPSP and REDB [A].

A1: While our CPSP shares some conceptual similarities with REDB, it differs fundamentally in both design and purpose. The key distinction between our CPSP and REDB lies in how pseudo-objects are selected for training. REDB is designed for a cross-domain setting, focusing on identifying objects that remain stable across source and target domains. In contrast, CPSP is specifically tailored to support the AL process in a single-domain setting. This fundamental difference results in two critical distinctions that are central to CPSP: the box bank deletion mechanism and the use of backgrounds likely to be false positives (FPs) for training. Let’s delve into these two factors to illustrate their importance.

• Box Bank Deletion Mechanism:

  No matter how confident or refined the filtering process is, pseudo-labels in the box bank inevitably contain errors and noise. Without a mechanism to delete these noisy labels, they propagate throughout training, resulting in confirmation bias that adversely affects uncertainty measurement. CPSP incorporates a robust deletion mechanism to address this issue, ensuring cleaner pseudo-labels and better AL outcomes. In contrast, REDB filters objects by inserting them into random sampling scenes (e.g., CDE). This approach, while reasonable in cross-domain scenarios, may not work effectively in single-domain settings. Randomly placing objects into unrelated scenes (e.g., a car in a forest) can generate unreasonable predictions, reducing the stability of pseudo-labels and potentially discarding true and confident objects during training.

• Use of Backgrounds Likely to Be FPs:

  CPSP utilizes background regions that are likely to be FPs, providing negative signals to the model. These negative signals help the model avoid overconfidence in incorrect predictions, thereby improving the reliability of uncertainty measures. Conversely, REDB lacks this design, which limits its effectiveness in supporting the AL process.

To further demonstrate these differences, we conducted experiments by directly applying REDB and its CDE component into our framework with the same threshold. The results, presented below, compare the mAP and D-ECE (calibration, where lower is better) metrics for various configurations:

Table I: mAP comparison across settings on Waymo

Table II: D-ECE comparison across different pre-training methods on Waymo

The results indicate that while CDE provides some improvements in calibration for vehicles, it fails to improve calibration for the more challenging classes, pedestrians and cyclists (Table II), thereby limiting its overall effectiveness in supporting the AL process (Table I).

Finally, we acknowledge that some modules from REDB, such as OBC, may be beneficial for better training of objects. We will consider incorporating such modules into our framework in future work.

Q2: The Novelty of our CAL.

A2: The primary novelty of our CAL lies in its integration with our pre-trained CPSP model. While we acknowledge that entropy is a widely used metric, our approach goes beyond its simple application. One key challenge is that the CPSP pre-trained model can "forget" some original true and confident objects. To address this, we propose an ensemble strategy, ensuring more reliable uncertainty measures. Moreover, our AL method is highly flexible and can easily incorporate future advancements in uncertainty estimation. Additionally, our approach effectively addresses class imbalance, a critical issue often underestimated in outdoor scenes. The co-occurrence of multiple objects in a scene complicates the selection process, and FPs can result in sampling irrelevant scenes that lack the intended objects, particularly for rare classes like cyclists. This challenge is further exacerbated by many pre-trained models that generate excessive FPs. Unlike previous methods, our class balance strategy explicitly tackles these issues, enabling more effective and balanced sampling across all classes.

Q3: Typos.

A3: We appreciate your suggestions for improving the presentation of our paper. We will revise our paper carefully.

Q4: About the thresholds for confident object extraction.

A4: This is a great question! Noise is indeed a major reason why many SSL methods struggle with calibration and performance. Initially, we considered setting a high confidence threshold, but score distributions vary significantly across classes and models, making it difficult to determine a universally appropriate threshold. As a result, we abandoned this approach. We also explored top-k selection; however, this method can easily introduce noise and requires careful tuning of the value k, which is also challenging to determine.

Instead, we opted for a score clustering method, setting a high number of clusters (KITTI: 20, Waymo: 50, which we found to be effective compared to fewer clusters (<10) or excessive clusters (>100)). By extracting objects from the highest-confidence cluster, this approach effectively minimizes noise while providing a sufficient number of pseudo-objects for training.

Thanks for your thorough response. Most of my concerns have been addressed.