Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection

We sincerely thank your comments and for recognizing the contributions of our rebuttal. We would like to address the concern regarding the comparison between UET and AKD, particularly regarding the motivation and conceptual differences in our uncertainty learning approach.

The fundamental difference between UET and AKD lies in their conceptualization and utilization of uncertainty in KD. AKD focuses solely on addressing the diversity of uncertain knowledge from the teacher model by proposing an uncertainty-aware adaptive weighting mechanism. While this approach is valuable, it operates under the assumption that uncertain knowledge alone is sufficient for effective distillation, neglecting a critical aspect: the complementary role of deterministic knowledge.

In contrast, UET is motivated by the insight that deterministic and uncertain knowledge are complementary in guiding the student’s learning process. Deterministic knowledge, carefully trained and precision-driven, captures the core structure and high-confidence predictions of the teacher model, which are indispensable for stabilizing and enriching the student’s training. Uncertain knowledge, on the other hand, reflects the nuanced understanding of ambiguous or diverse scenarios, providing additional flexibility. By integrating these two perspectives, UET seeks to strike a balance, leveraging both types of knowledge to achieve more robust and effective distillation.

Key Differences Between UET and AKD:

1. Complementary knowledge integration vs. uncertainty weighting. AKD emphasizes handling diverse uncertain knowledge, overlooking the deterministic knowledge from the teacher model. UET explicitly integrates both deterministic and uncertain knowledge to leverage their complementary strengths, as validated by our ablation studies (Table 6 in the manuscript). For example, incorporating deterministic knowledge improves the student’s mAP from 43.5 to 44.0, demonstrating its critical importance.

2. Loss function design: AKD employs an uncertainty-weighted loss to adaptively weigh uncertain predictions during distillation. UET, in contrast, incorporates a novel feature distillation mechanism that jointly captures deterministic and uncertain knowledge, enhancing the transfer of fine-grained and global information.

3. We acknowledge the reviewer’s observation that the empirical improvements over AKD may appear modest. However, it is important to note that achieving significant performance gains in KD has become increasingly challenging due to the maturity of existing methods. If we further examine the distinctions between UET and AKD, our method achieves superior results with a more straightforward design. For instance, as shown in Table 3, UET improves the baseline mAP by 0.9, compared to AKD’s 0.7, demonstrating a relative performance gain of 28.5% over AKD. Moreover, our method narrows the gap to the teacher model to only 0.8 mAP, which is a substantial improvement in the context of KD. These results highlight the efficacy of UET and reinforce the significance of the proposed approach in advancing KD performance. We further would like to argue that UET’s strength lies in its fundamentally different perspective and motivation, which redefines how deterministic and uncertain knowledge can work together in KD. The introduction of complementary knowledge integration is not merely incremental; it represents a new way of approaching uncertainty in KD, which has been empirically validated through our ablation studies and performance comparisons.

We hope this clarifies the key distinctions between UET and AKD and underscores the conceptual novelty and effectiveness of our approach. We are looking forward to further discussion and appreciate your constructive feedback.

We sincerely appreciate your recognition and valuable feedback. We will incorporate the discussion from our exchange into Section 4.2 of the main paper, as suggested. Thank you for helping us refine our motivation and discussion, making our work more comprehensive and robust. Your thoughtful input has been instrumental in improving our paper, and we are deeply grateful for your support.

R3: On Comparisons with Transformer-Based Models

Thank you for your thoughtful feedback and for highlighting the importance of evaluating our method in the context of Transformer-based models, given their prominence in state-of-the-art machine learning tasks.

a) Heterogeneous Backbone Experiments

To clarify, we have already conducted distillation experiments using Swin Transformer-Tiny as the teacher backbone and ResNet-50 as the student backbone, as shown in Table 11 of our paper. The results demonstrate that our proposed UET performs effectively even in Transformer-to-CNN heterogeneous configurations (FGD: 37.4 vs. FGD+UET: 37.7).

b) Additional Experiments on Transformer-to-Transformer Distillation

We appreciate the insightful references [6-7] you provided and have extended our experiments to include Transformer-to-Transformer configurations. Specifically, we used Swin Transformer-tiny as the teacher backbone and Swin Transformer-nano as the student backbone. The results are shown in follows table. When employing PKD for distillation, the student network achieves an mAP of 32.2. By integrating the proposed UET module, the mAP of the student model is further improved, reaching 32.5. These findings further confirm the generalization capability of UET in Transformer-based settings.

Table: Experiments for detectors with transformer backbone on the RetinaNet detector.

[6] Miles et al., VkD : Improving Knowledge Distillation using Orthogonal Projections, CVPR 2024

[7] Song et al., Vidt: An efficient and effective fully transformer-based object detector (under Token matching setting), ICLR 2022

R4: Calibration Analysis

Thank you for highlighting the importance of calibration in evaluating the reliability of the detector. In response to your suggestion, we performed additional calibration analysis using Expected Calibration Error (ECE) and Brier Score, alongside mAP, to comprehensively assess both prediction reliability and detection performance.

The results, summarized in the table below, show that our method achieves improved calibration (lower ECE and Brier Score) while significantly enhancing detection performance (mAP: 44.0 vs. 43.4). These findings demonstrate that our approach not only improves accuracy but also provides more reliable predictions, achieving a better balance between performance and calibration compared to FGD.

We believe these results provide valuable insights into how UET improves both prediction reliability and accuracy, further emphasizing the robustness and practicality of our method. Thank you again for your suggestion, which helped us strengthen our evaluation.

Table: Calibration Analysis Results

R5: Clarifications on Timing Performance and MC-Dropout

Thank you for your insightful comments. We address your concerns as follows:

a) On Training Time and Forward Passes.

To clarify, our implementation does not perform multiple forward passes during training. Instead, we simulate Monte Carlo (MC) sampling by applying multiple independent dropout operations within a single forward pass. This significantly reduces computational overhead, ensuring that the training time remains almost unchanged, as reflected in our experiments (Appendix A.5). Specifically, without introducing uncertainty knowledge, the training time per iteration for FGD is 0.7571 seconds. With MC-Dropout sampling set to 1, 5, 10, and 15, the training times are 0.7579, 0.7618, 0.7673, and 0.7720 seconds, respectively. We also revised the manuscript to make this implementation detail more explicit.

b) On MC-Dropout During Inference.

MC-dropout is not applied to the student model during inference. We have explicitly clarified this in the Methods section to avoid any potential misunderstandings.

R2: Clarifying the Role of UET

Thank you for your valuable feedback. We would like to address your concerns as follows:

1). Generalization Across KD Methods and Detectors

While FGD is indeed a strong baseline for distillation, the consistent improvements achieved with UET across various KD methods demonstrate that our approach is not limited to FGD but provides complementary benefits to a wide range of frameworks. To showcase the generalizability of UET, we conducted experiments using the GFL detector with four distinct KD methods (e.g., FGD, PKD, MGD, and LD), as presented in Table 7. Additionally, based on your suggestion, we extended our analysis to include experiments on the RetinaNet detector, as discussed in R1.3. These results consistently show that UET enhances the performance of various KD methods. These findings further reinforce that UET is not tied to any specific KD framework or detector, highlighting its broad applicability and complementary nature.

2). Complementary Role of UET

The primary objective of UET is not to compete directly with existing KD methods but to act as a complementary enhancement. While methods like FGD focus on deterministic knowledge distillation, they often overlook the inherent uncertainty in teacher knowledge, which serves as a valuable additional source of guidance for the student model. UET bridges this gap by introducing uncertainty as a novel dimension of knowledge in the distillation process. By integrating deterministic and uncertain knowledge, UET enables existing KD methods to unlock their full potential.

We greatly appreciate your feedback, as it allowed us to clarify the positioning of UET and its role in complementing existing KD methods. These insights have strengthened the focus and presentation of our work.

R3: Generalization of UET Across Detectors and KD Methods

To address your concern, we provide additional experimental evidence demonstrating UET's effectiveness across different detectors and KD methods.

Specifically, we added experiments embedding UET into multiple KD methods (MGD and PKD) on the RetinaNet detector. The results are presented in the following Table. When using MGD for deterministic knowledge distillation, the student model achieves a performance of 39.3 mAP, representing a 1.9 improvement over the original student model. By adapting MGD to follow the UET paradigm, the student model's performance further increases to 39.6 mAP, achieving a total improvement of 2.2 points over the original student. Similarly, with PKD, the student model achieves 39.6 mAP, which is a 2.2 improvement over the original student. When PKD is adapted to the UET paradigm, the student model's performance further improves to 39.8 mAP, resulting in a total improvement of 2.4 points over the original student.

These extensive experiments demonstrate that UET is a versatile and robust framework, capable of improving various KD methods across multiple detectors and configurations.

R4: Clarification of the Student's Knowledge Uncertainty (S)

We sincerely thank the reviewer for pointing out the need for clarification regarding the student's knowledge uncertainty (S). The student's knowledge uncertainty represents the inherent uncertainty in the student model's knowledge, derived using Equation 5 in the paper. The key insight of our work is that both deterministic knowledge and uncertain knowledge from the teacher should be simultaneously considered during the knowledge distillation process. Naturally, the student model also can capture and leverage both types of knowledge. Therefore, we conducted specific experiments to explore the student's uncertain knowledge, as presented in Table 6.

Thanks for the response, which partially answer my questions. I greatly appreciate the authors’ effort in providing additional KD results on other detectors and KD approaches. However,

1. Regarding uncertain knowledge, UET is relatively similar to AKD. The authors do not provide relevant discussions to show the difference between UET and AKD. Empirically, I would be looking for significant mAP gains of UET without deterministic knowledge (i.e., without the original teacher), which may better reflect the advantages of UET over AKD.

2. Given the point raised in 1, compared with AKD, the superiority of UET is actually attributed to deterministic knowledge, i.e., the original teacher. This causes UET to be more like an incremental work. The simple combination (key insight of UET, as the authors stated) of uncertain and deterministic knowledge for distillation may not sufficiently underpin the novelty of this work. To be clear, as recognized by Reviewer kB7D and extensively investigated in the past, multi-teacher KD is bound to deliver better KD results.

I have also gone through the other reviewers’ feedback and authors’ responses to them. Based on all these, I will maintain my original recommendation of 5.

Sorry for the late update. I think there are some misunderstandings.

1. I totally agree that dropout is a simple yet effective scheme for measuring knowledge uncertainty. This is a strength indeed. My concern is that UET and AKD share the idea of generating uncertain knowledge with the dropout for distilling detectors. In other words, you only need to clarify the essential differences between UET and AKD regarding how to estimate and leverage knowledge uncertainty in KD. It would be better if UET (w/o deterministic knowledge) also has significant performance advantages over AKD.

2. If the concern in 1 is valid, combining the uncertain knowledge (already explored in AKD) and deterministic knowledge (the common practice in KD) for distillation is an incremental work, from where I stand.

A kind reminder: using NeurIPS instead of NIPS, to avoid unintentional issues.