ELDET: Early-Learning Distillation with Noisy Labels for Object Detection

We'd like to thank all of our reviewers for taking the time to give us constructive reviews. Your reviews will help us improve our methods and manuscripts.

... the performance improvements vary depending on the type and amount of noise (less than 1% in some cases, and not the best in some cases). The improvements for the MS COCO dataset are typically less than for the Pascal VOC dataset.

We appreciate this observation regarding performance variations. While some improvements may appear modest in absolute terms, they represent meaningful gains in challenging noisy scenarios. Specifically, our method achieves consistent improvements across different noise conditions: on PASCAL VOC, ELDET shows an average improvement of 2.29 AP across all detectors and noise settings, with 87.5% of cases (35/40) demonstrating positive improvements. On MS COCO, we achieve an average improvement of 1.27 AP with 72.5% of cases (29/40) showing positive gains.

The smaller improvements on MS COCO compared to PASCAL VOC are expected due to the dataset's inherent complexity and scale. Even modest AP improvements (1-2%) are significant in object detection, particularly under noisy conditions where performance typically degrades substantially. The few negative cases occur only in clean settings for specific detectors, which is acceptable as our method specifically targets noisy scenarios.

... it will be good to evaluate how the proposed approach will work when objects are occluded.

This is an excellent suggestion for future work. While our current approach focuses on annotation noise (categorization and localization errors), the phenomenon we leverage could potentially extend to occlusion scenarios. The robustness principles underlying our method—using early-stage models as teachers before overfitting occurs—may help models maintain better representations of partially visible objects. However, this would require specialized adaptations, such as occlusion-aware metrics and potentially different early-learning termination criteria. We plan to investigate this extension in future research.

How well the adaptation to infrared images will work?

We believe our approach will adapt well to infrared images. While we haven't tested specifically on infrared imagery, we demonstrated strong cross-domain performance on the VinDr-CXR medical dataset, which represents X-ray images with significantly different visual characteristics from natural RGB images. Our method showed substantial improvements across all detectors (e.g., FCOS improved from 25.32 to 32.74 AP under 40% combined noise), indicating robust generalization capabilities. The plug-and-play nature of our framework makes it readily applicable to infrared imaging scenarios, as the early-learning phenomenon is a fundamental property of deep networks that should manifest similarly across different image modalities.

Have the authors considered other kinds of noise that can affect the performance of object detectors?

Thank you for this important question. While our work focuses on the two most prevalent annotation noise types (categorization and localization errors), we acknowledge that object detectors face various other noise sources, including image-level noise (Gaussian, salt-and-pepper), sensor noise, missing annotations, and bogus bounding boxes. The early-learning principle we leverage is quite general and could potentially be extended to handle these additional noise types. For future work, we plan to investigate extending our framework to address missing annotations and bogus boxes, incorporating image-level noise robustness, and developing unified approaches for multiple noise sources simultaneously.

I thank the reviewers for their response. Given that object detection is a well-studied problem, challeges due to noisy lables is not as challenging as being able to handle occlusions. The impact is low and I will keep my original score.

We sincerely thank the reviewer for valuable feedback. We would like to emphasize that, as discussed in our related works section, the problem of noisy labels in object detection has not been comprehensively studied. Our paper proposes a novel framework that simultaneously addresses both localization and categorization noise, which are prevalent challenges in practical object detection tasks, and we believe this contribution has significant impact.

While we acknowledge that occlusion is an important and distinct problem in object detection, it lies outside the scope of our current work which focuses specifically on label noise. We aimed to clarify that while occlusions may be interpreted as a form of image-level noise, our work specifically targets label noise — particularly the decoupled dynamics of categorization and localization noise during training.

We appreciate the suggestion regarding occlusion as a valuable direction for next research.

We'd like to thank all of our reviewers for taking the time to give us constructive reviews. Your reviews will help us improve our methods and manuscripts.

Why do the reported performances on clean data in Table 1 and Table 2 vary for different methods, and what factors contribute to these discrepancies?

Thank you for this important observation. The performance variations on clean data between different methods can be attributed to several key factors.

First, noise-robust training methods inherently introduce regularization effects that can act as random perturbations during optimization. These regularization mechanisms, while designed to handle noisy scenarios, can introduce variability in clean data performance due to their stochastic nature and different optimization dynamics.

Second, methods like ORSOD and ADELE employ different loss weighting and sample selection strategies that can interact differently with various detector architectures, leading to performance variations even on clean data. Additionally, the initialization and convergence behavior of these methods can vary across different random seeds, as robust training methods often modify the optimization landscape in ways that affect final performance.

Our ELDET method shows consistent improvements on clean data (e.g., +1.45 AP for RetinaNet on PASCAL VOC) because the early-learning distillation acts as beneficial regularization that enhances generalization rather than simply addressing noise.

Could you elaborate on why the relative performance drop between noisy and clean data appears to be larger for ELDET compared to the baseline method (Tab. 2)?

Thank you for this insightful observation. We would like to emphasize that the difference in average drop between ELDET and the baseline narrows considerably to only 0.52 AP (2.65 AP for ELDET vs. 2.13 AP for baseline). This indicates that the overall degradation due to noisy labels is not substantially worse for ELDET. This phenomenon is primarily attributable to ELDET's superior baseline performance on clean data, which is actually a positive indicator rather than a limitation. It is noteworthy that ELDET consistently outperforms baseline methods on clean data across all detectors, with an average improvement of +1.74 AP (ranging for Faster R-CNN to +2.32 AP for GFL).

We would like to emphasize that our ELDET framework successfully achieves effectiveness across all noise settings. For instance, under 40% combined noise—the most challenging scenario—ELDET shows absolute improvements of +1.04 AP (RetinaNet), +1.46 AP (FCOS), +0.73 AP (Faster R-CNN), and +0.77 AP (GFL) compared to baselines. These consistent absolute gains demonstrate that our method successfully addresses both categorization and localization noise while maintaining superior performance across varying noise conditions.

In Figure 5, the performance of ELDET initially declines before subsequently improving. What are the potential underlying reasons for this observed trend?

This initial performance decline followed by improvement is a well-documented characteristic pattern in knowledge distillation frameworks[1,2]. Several factors contribute to this phenomenon.

First, the introduction of the distillation loss creates a multi-objective optimization problem where the model must balance learning from both noisy ground truth and teacher guidance. During early epochs, there can be conflicting signals between these two sources, leading to temporary performance degradation.

Second, our EMA-based teacher update mechanism requires time to stabilize and provide consistent guidance. The dynamic interaction between teacher and student creates a moving optimization target that can cause initial instability before convergence.

Third, knowledge distillation research has shown that this initial decline represents a transition period where the student model moves away from simply memorizing noisy patterns toward learning more generalizable representations guided by the early-learning teacher. The subsequent improvement demonstrates that the distillation process successfully helps the model focus on clean learning patterns, ultimately achieving better performance than baseline methods.

[1] Cho, J. H., & Hariharan, B. (2019). On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 4794-4802).

[2] Sun, Z., Liu, Y., Meng, F., Chen, Y., Xu, J., & Zhou, J. (2025). Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation. arXiv preprint arXiv:2502.11766.

We'd like to thank all of our reviewers for taking the time to give us constructive reviews. Your reviews will help us improve our methods and manuscripts.

"Lacking essential experimental results from several SOTA transformer-based detectors employing denoising training to enhance robustness."

We appreciate the valuable observation regarding transformer-based detectors and their denoising training strategies. We would like to clarify that our main experiments focused on convolution-based architectures, and thus a direct comparison with transformer-specific denoising methods, which involves adding random noise to ground truth as an input to transformer decoder, was not feasible.

To address concerns about transformer architectures, we conduct additional experiments applying ELDET to Deformable DETR on PASCAL VOC:

Table: Performance of ELDET on Deformable DETR under combined noise conditions on PASCAL VOC (AP@50)

While improvements are modest, these preliminary results indicate that ELDET can enhance performance on DETR-based models. The limited gains are partly due to restricted hyperparameter tuning during the rebuttal phase and the use of CrossKD, a distillation method initially designed for convolutional models rather than transformers. In the camera-ready version, we plan to incorporate distillation methods better tailored to DETR architectures[1] and anticipate further performance improvements. Additionally, applying the aforementioned transformer decoder denoising strategies could further boost results.

Importantly, ELDET’s plug-and-play design ensures its broad applicability across various backbone architectures, including both convolutional and transformer-based detectors.

"Unclear explanation regarding the calculation of the transition point and the relationship between the proposed CA and GTBA, as well as Eq (1) and Eq (2)."

Thank you for highlighting these clarifications. Below is a concise explanation of the transition point calculation and the relationship between CA and GTBA metrics:

Transition Point Calculation

We model training performance $f(t)$ (e.g., AP over epochs $t$) with the exponential function:

$f(t) = a \left(1 - e^{-b \cdot t^c}\right)$

where parameters $a,b,c$ are fitted to observed data.

The transition point $t^*$ from early learning to memorization is identified when the relative change in the derivative crosses a threshold $\gamma$:

$\frac{|f'(1) - f'(t^*)|}{|f'(1)|} > \gamma$

This detects when learning slows down significantly, marking onset of memorization of noisy labels.

CA and GTBA Metrics

• Class-Agnostic (CA) treats all objects as a single class to isolate localization performance by ignoring classification errors. CA tracks how well the model learns precise object locations despite noisy labels.

• Ground Truth Box Allocation (GTBA) replaces predicted boxes with ground truth ones when IoU exceeds threshold (\tau\), removing localization error influence to isolate classification performance.

Together, CA and GTBA provide complementary views into memorization dynamics — CA tracks localization noise memorization, GTBA tracks classification noise memorization — enabling task-specific transition detection.

"Lack of sufficient experimental results for different datasets and detectors ... regarding the different patterns of fitting noise labels for localization and classification tasks."

Due to space constraints, we provide a detailed response to this point in our reply to Reviewer RwS3. Please kindly refer to that response for a comprehensive explanation.

"Lacks of the scale of the datasets such as larger datasets (e.g., Object365) as well as some smaller datasets (e.g., Oxford Pets)."

We acknowledge this important point regarding dataset scale diversity. Our experiments already demonstrate effectiveness on datasets of varying scales: VinDr-CXR represents a relatively small dataset (18,000 images), which showed substantial improvements (e.g., FCOS: 25.32→32.74 AP under 40% noise), indicating ELDET's effectiveness even on limited data.

We have conducted additional experiments on Oxford Pets, a smaller dataset with ~7,400 images across 37 categories:

Table: Performance on Oxford Pets Dataset (AP@50)

Regarding Object365, our method's consistent improvements across existing diverse scales (small: VinDr-CXR, medium: PASCAL VOC, large: MS COCO) provide strong evidence for scalability. We aim to include Object365 evaluation results in the camera-ready submission, ensuring a thorough demonstration of the method’s applicability to large and diverse datasets.

"In the FCOS experiments presented in Table 1, under the ELDET framework, why does the performance using the 20% localization noise dataset outperform that of the clean dataset?"

This interesting phenomenon can be attributed to ELDET's regularization effect through knowledge distillation. When FCOS processes 20% localization noise, our early-learning teacher provides beneficial guidance that acts as implicit regularization, similar to how data augmentation can sometimes improve generalization. The distillation process helps the model learn more robust spatial representations that generalize better than training on clean data alone. This effect is specific to ELDET because other methods lack the early-learning guidance mechanism—they either ignore the regularization benefits (baseline) or employ different noise handling strategies that don't provide the same regularization effect (ORSOD, ADELE). The magnitude of this improvement (+1.40 AP) suggests that the early-learning teacher's spatial knowledge provides valuable inductive bias for FCOS's anchor-free detection paradigm.

"In Table 4, using GTBA alone actually decreases the model's performance. However, performance improves only when it is combined with CA."

We appreciate this important observation. Our key finding is that memorization of localization noise occurs earlier than categorization noise. When only GTBA is applied, our framework may provide early-learning guidance focused on classification while the model's performance on localization task is still in its noisy memorization stage. This mismatch can lead to suboptimal learning and decreased overall performance.

By combining GTBA with CA, we provide coordinated early-learning guidance for both classification and localization tasks. CA specifically targets localization noise by treating all classes as one category, helping the model accurately identify the proper transition point before localization memorization begins. This allows the model to respect the sequential memorization order from localization to classification. This balanced approach enables the model to effectively handle both noise types, leading to significant performance gains compared to using GTBA alone.

"The paper lacks the analysis of the training cost."

We acknowledge this important practical consideration and provide comprehensive training cost analysis. Our experiments were conducted on a single NVIDIA A100 GPU.

Table: Training Cost Analysis

The additional computational overhead of our method mainly comes from three sources: (1) exponential curve fitting to detect early learning transitions, (2) the computation of the knowledge distillation loss, and (3) exponential moving average (EMA) updates of the teacher model’s parameters.

A noticeable part of the increased training time arises from per-epoch validation performed during the early learning phase. To detect the timing of early-learning termination, validation on the entire training set occurs in more than half of the initial training process, which contributes to the overhead on the training time. However, since validation is limited to this early-stage, its relative impact diminishes with longer training schedules.

While ELDET does increase training time and memory usage due to maintaining a frozen teacher network for additional forward passes, this cost can be viewed as acceptable because it potentially replaces more expensive and complex noise detection or correction procedures, making the overall approach efficient and practical.

[1] Wang, Y., Li, X., Weng, S., Zhang, G., Yue, H., Feng, H., ... & Ding, E. (2024). Kd-detr: Knowledge distillation for detection transformer with consistent distillation points sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16016-16025).

We thank the reviewer for the constructive feedback and the score adjustment. We appreciate the opportunity to clarify the experimental settings for DINO, the rationale behind removing query denoising, and the performance differences observed across architectures.

The denoising mechanism in DINO is designed to accelerate the convergence of the decoder by appending additional queries constructed from ground-truth boxes and labels with artificial perturbations—such as adding noise to bounding box coordinates or replacing the class label with a background class. These denoising queries are trained to recover the original ground-truth, which helps align queries with target objects more quickly in the early stage of training. However, this mechanism is not primarily designed to improve robustness to actual label noise: when the ground-truth annotations themselves contain noise, the denoising queries are generated from such noisy annotations, and the model may still memorize the noise. In contrast, ELDET leverages a teacher model from an early learning stage for distillation, allowing the student to learn from cleaner signals and thus avoid overfitting to noisy labels.

Regarding the relatively larger improvement in the DINO experiment compared to Deformable DETR, we attribute this to a well-tuned knowledge distillation (KD) loss weight that considered the loss scale in DINO. We expect that Deformable DETR would also benefit from similar tuning of the KD loss weight, potentially leading to further improvements.

We also agree that reducing training costs is a promising direction. Exploring lighter alternatives to EMA, such as LoRA-based teacher updates, and further optimizing the transition point calculation are interesting extensions that we plan to investigate in future work. Another possible approach is to perform validation on a sampled subset of the training data, which may further reduce computation without significantly impacting performance estimation.

We again thank the reviewer for the thoughtful comments and valuable suggestions.

We'd like to thank all of our reviewers for taking the time to give us constructive reviews. Your reviews will help us improve our methods and manuscripts.

"The paper is based on the observation that 'models tend to memorize localization noise earlier than categorization noise.' This observation is empirical. It may change with different levels of noise or different classes of objects to detect. It is questionable whether the observation can always hold."

We acknowledge this important concern regarding the generalizability of our empirical observation. To address this, we conduct extensive validation experiments across diverse settings and present comprehensive evidence demonstrating the consistency of this phenomenon. We will provide the following additional experimental validation:

Table: Early Learning Termination Epochs for Localization vs. Classification Tasks Across Diverse Settings

These comprehensive experiments across different datasets, detectors, and noise levels will demonstrate that the differential memorization timing between localization and classification tasks is a consistent phenomenon across diverse experimental conditions, validating the robustness of our core observation.

"It is questionable how the 'two-stage' distillation would work for models without two separate classification and detection heads."

We appreciate this important architectural consideration. To clarify, we evaluated our method on both one-stage detectors such as FCOS and GFL, and two-stage detectors including RetinaNet and Faster R-CNN, covering a broad range of representative architectures.

We would like to emphasize that most modern state-of-the-art object detection models indeed employ separate classification and regression heads, which naturally accommodate our two-stage distillation approach[1,2,3]. Such separation allows each head to specialize and optimize for its respective task—classification or localization—leading to more accurate and stable learning. Prior works[4,5,6] have empirically demonstrated that distinguishing these heads improves both convergence and final detection performance, supporting the architectural rationale behind our distillation method. Therefore, applying distillation strategies that leverage this separation is consistent with established design principles in object detection.

In the rare cases where a single head is shared by both classification and regression tasks, our framework can be adapted with minor modifications. A possible approach might be to apply task-specific masking to the gradients of the final output head during backpropagation. This would allow selective updating of weights relevant to each task (classification or regression) while masking out the others, enabling different training procedures and EMA updates.

[1] Wang, A., Chen, H., Liu, L., Chen, K., Lin, Z., & Han, J. (2024). Yolov10: Real-time end-to-end object detection. Advances in Neural Information Processing Systems, 37, 107984-108011.

[2] Zhao, Y., Lv, W., Xu, S., Wei, J., Wang, G., Dang, Q., ... & Chen, J. (2024). DETRs beat YOLOs on real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16965-16974).

[3] Nan, Z., Li, X., Dai, J., & Xiang, T. (2025). MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4703-4712).

[4] Wu, Y., Chen, Y., Yuan, L., Liu, Z., Wang, L., Li, H., & Fu, Y. (2020). Rethinking classification and localization for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10186-10195).

[5] Fang, Z., Chen, N., Jiang, Y., & Fan, Y. (2024). Decouple and align classification and regression in one-stage object detection. The Visual Computer, 40(11), 7773-7786.

[6] He, L., & Todorovic, S. (2022). Destr: Object detection with split transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9377-9386).

Thank you to the authors for the detailed rebuttal response and additional experiments. However, considering the generalization capabilities of the work and overall originality, I think there still is a gap to a publication in NeurIPS. (I raised my rating slightly)