DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer

W1: The method introduces extra hyperparameters, such as the 200-iteration interval for GSNR computation, which seems empirically chosen and may depend on various factors. The authors do not fully analyze or justify these choices.

Thank you for raising this important point regarding hyperparameter justification. We would like to clarify that the GSNR computation interval (e.g., every 200 iterations) is not a hyperparameter involved in the actual training process, but rather a setting used exclusively for our offline theoretical analysis. The GSNR plots in our paper (e.g., Figure 1a, Figure 2) are generated to empirically validate our hypothesis about gradient signal quality, and the interval simply determines the sampling frequency for these visualizations. This is a standard approach for analyzing training dynamics and does not impact the method’s practical application.

Our findings show that the theoretical conclusions drawn from the GSNR analysis are robust to the choice of sampling interval, as long as the interval is not extremely small (e.g., >50 iterations). A smaller interval may introduce more noise, while a larger interval improves statistical reliability at the cost of higher computational overhead. We chose 200 as a balanced value for clear visualization and efficient computation. Importantly, this choice does not affect the training or require tuning for different datasets, architectures, or optimization settings.

In summary, our method does not introduce any GSNR-related hyperparameter that needs to be tuned in practice. The 200-iteration interval is solely for offline analysis and does not influence the training process. The actual hyperparameters used in our method are principled, practical, and robust, as demonstrated in our experiments. We will further clarify this distinction in the revised manuscript.

W2: Why not use stronger teacher models (e.g., ViT, DETR) in experiments? How does your method perform on these architectures?

Thank you for your question regarding the diversity and strength of teacher models in our experiments. Our experimental design follows established practice in the knowledge distillation literature, using widely adopted baselines such as ResNet and Faster R-CNN to ensure fair and direct comparison with prior state-of-the-art methods. In addition, we have included strong teacher models such as ResNet50 and RegNetY-16GF on large-scale datasets like ImageNet.

We fully agree that evaluating on modern Transformer-based architectures is essential. In fact, our original submission already includes results on Vision Transformers: Table 3 demonstrates that DeepKD, when combined with MLKD, significantly improves the performance of a DeiT-Tiny student on ImageNet, with up to +1.90% Top-1 accuracy gain. We also evaluated strong teacher models, including RegNetY-16GF, across a range of modern architectures.

To further strengthen our validation, we have conducted additional experiments on Swin Transformer (ICCV 2021) and ViT-to-ViT (ICLR 2021) distillation. As shown in Table 1 below, DeepKD achieves notable Top-1 accuracy improvements of +1.12% and +0.92%, respectively, on ImageNet-1K.

Table 1: Results on Larger Transformer Architectures (ImageNet-1K)

In response to your specific question about DETR, we have also performed new experiments on the DETR architecture for object detection on MS-COCO. As shown in Table 2, DeepKD provides consistent improvements over standard distillation methods, achieving up to +1.0 AP gain on DETR-based models.

Table 2: DETR Distillation Results on MS-COCO

These results demonstrate that DeepKD generalizes well to both CNN-based and Transformer-based architectures, including challenging tasks such as object detection with DETR.

We believe these comprehensive experiments provide strong evidence of DeepKD’s scalability and effectiveness across a wide range of modern architectures. All new results will be included in our revised manuscript.

Q1: Why was 200 chosen for GSNR computation, and are there results on stronger teacher models like ViT or DETR?

Thank you for these questions. Please find a concise summary below, with detailed explanations provided in the main rebuttal text above:

• On Hyperparameter Justification: As discussed in the section "On Hyperparameters and Theoretical Justification," the GSNR computation interval is used only for offline analysis and does not affect training. Our method’s actual hyperparameters are principled, practical, and robust.

• On Experiments with Stronger Models: As detailed in "On the Strength and Diversity of Teacher/Student Architectures," we have conducted extensive experiments on modern and strong architectures, including DeiT, Swin Transformer, ViT, and DETR.

We hope these clarifications fully address your concerns, and we will incorporate all new experimental results into our revised draft.

W1: The main concern is the introduction of new hyperparameters, including Δ, β₁, β₂, and the DTM schedule (Phase-1 and Phase-2).

Thank you for raising this important point regarding the hyperparameters introduced by our method. We address each of the parameters as follows:

1. Loss Weights (β₁, β₂):
   These are not tuned hyperparameters in our framework. To ensure fair and consistent comparisons, we directly adopt the fixed coefficients for TCKD and NCKD as proposed in the original DKD (CVPR 2022) method, following established best practices.

2. Momentum Difference (Δ):
   This parameter is principled and robust. Our GSNR analysis provides strong theoretical motivation to simply set Δ > 0. As shown in our ablation study (Table 5, Left), the final performance is not sensitive to the exact value of Δ as long as it is positive; all settings with Δ > 0 significantly and consistently outperform the baseline with Δ=0.

3. DTM Schedule (k-value, Phase-1, Phase-2):
   We acknowledge that the DTM schedule introduces new hyperparameters, which is expected for a new mechanism. However, our empirical results show that the DTM schedule, implemented as a three-phase curriculum with a heuristic linear growth strategy inspired by curriculum learning, is highly robust and practical. This is validated by extensive experiments on the large-scale ImageNet-1K dataset, where the DTM component consistently improves performance across all tested teacher-student pairs, confirming both its stability and practical value.

In summary, the key hyperparameters in DeepKD are either fixed by standard practice, guided by strong theoretical principles and shown to be robust, or empirically robust on large-scale tasks. This ensures that our framework remains practical and easy to apply.

W2: The ablation studies suggest DTM schedule parameters may cause large performance fluctuations. For example, changing Phase-1 from 60 to 40 (with k=55, Phase-2=170) leads to a >7% drop, even below the student-only baseline. This raises concerns about the method's stability and practicality.

We sincerely appreciate your careful and rigorous review. Your observation prompted us to conduct a thorough audit of our experiment logs, during which we discovered a critical data compilation error in Table 5 of our manuscript.

We apologize for this oversight. The error was due to a manual mistake in transcribing results for two experimental settings. To ensure transparency and reproducibility, we will release our complete code and final model checkpoints soon.

The two erroneous data points that led to concerns about model stability have now been corrected (see Table 1).

Table 1: Correction of erroneous data.

With these corrections, all tested configurations in our ablation study now significantly outperform the student-only baseline. The results are now logically consistent and demonstrate that our DTM framework is highly robust across a wide range of parameterizations. The previously reported "high performance swings" and "instability" were artifacts of the data error and are now resolved. The corrected results provide clear and consistent support for our claims.

Thank you again for your diligence, which has been instrumental in identifying and correcting this issue, thereby strengthening the accuracy and reliability of our work.

Q1: What are the phase-1 and phase-2 values for ImageNet?

Thank you for your question regarding the DTM schedule parameters for our ImageNet experiments. Due to the high computational cost of exhaustive hyperparameter search on ImageNet, we adopted a principled two-step approach:

1. Principled Schedule Transfer from CIFAR-100:
   We first transferred insights from our extensive CIFAR-100 ablations, which indicated that a well-proportioned three-phase curriculum is effective. Accordingly, for the 100-epoch ImageNet training, we set the phase transitions at the 25% mark (epoch 25) and the 75% mark (epoch 75).

2. Experimental Verification on ImageNet:
   We then conducted a targeted ablation study on ImageNet, exploring variations around these initial values. The results are summarized below:

Table 2: ImageNet DTM Schedule Verification (ResNet34 teacher, ResNet-18 student; static top-k: k=100 yields best performance).

This ablation confirms that our initial, principled choice is indeed optimal. The best-performing schedule used in our main ImageNet experiments is: k-value=100, Phase1 ends at epoch 25, and Phase2 ends at epoch 75. Importantly, the results demonstrate the high robustness of our DTM schedule on ImageNet, as performance remains stable across all tested configurations. This confirms that our method is not sensitive to its hyperparameters on large-scale tasks.

Minor Comments: Including recent works [A-E] in the appendix would provide a more complete comparison and better context for our improvements.

Thank you for these valuable literature suggestions. We agree that including these recent works [A-E] will help contextualize our contributions within the rapidly evolving field of knowledge distillation.

In the revised manuscript, we will:

• Update the Related Work section to thoroughly discuss these papers.
• Expand the comparison tables in the appendix to include reported results from these methods where possible, providing a more comprehensive view of the state-of-the-art and better highlighting the improvements of our work over time.

We appreciate your suggestions and are confident these additions will further strengthen our paper.

I thank the authors for their detailed clarification! Thank you for sharing the rectified results on the DTM schedule ablation studies. That makes the proposed idea more robust than thought before.

the final performance is not sensitive to the exact value of Δ as long as it is positive; all settings with Δ > 0 significantly and consistently outperform the baseline with Δ=0.

I did not see any results that confirm "all settings with Δ > 0 significantly and consistently outperform the baseline". There is only one setting with all \Delta_{*} > 0, the last row. Do you have other results that looked at other positive combinations and found those accuracy results to be similar? Something like all $\Delta_{*}=0.05 and all \Delta_{*}=0.1 also yielded similar performance as \Delta_{*}=0.075 as shown in the last row? Note: I am using \Delta_{*} to represent all of \Delta_{TOG}, \Delta_{TCG}, \Delta_{NCG}

Thank you for sharing the additional experiments. It would be good to add in the paper that experiments across Delta values are stable and add this experimental table in the appendix of the paper. I thank the authors for engaging in the review period. I have raised my score to reflect my view after the our discussions.

W1: The framework mainly discusses logit-based distillation, and lacks concrete results or methods for feature-based scenarios.

Thank you for highlighting the scope of our framework. To directly address this concern and further demonstrate the flexibility of DeepKD, we have conducted additional experiments on feature-based distillation.

Specifically, we integrated DeepKD with two widely adopted feature-based methods, FitNet (ICLR 2015) and CRD (ICLR 2020), as shown in Table 1. Our objective was to evaluate whether DeepKD’s GSNR-driven optimization could enhance a combined feature-and-logit distillation pipeline, beyond what is achievable with standard KD.

Table 1: Feature Distillation Experiments on the CIFAR-100 validation

These results provide clear evidence that DeepKD is not restricted to logit-based distillation. Our GSNR-based optimization can be effectively combined with feature-based approaches, consistently yielding state-of-the-art results. This demonstrates the broad applicability and strength of DeepKD as a general-purpose distillation optimizer.

W2: The framework introduces several components and a dynamic masking mechanism, which may raise concerns about implementation complexity and hyperparameter tuning.

Thank you for raising concerns about implementation complexity and hyperparameter tuning. We address these aspects separately below:

1. Loss Weights (β₁, β₂):
   These coefficients are not tuned in our framework. We follow the standard settings from DKD (CVPR 2022) to ensure fair and consistent comparisons, using their fixed values for TCKD and NCKD loss components.

2. Momentum Difference (Δ):
   This parameter is theoretically motivated and robust. Our GSNR analysis justifies simply setting Δ > 0. As shown in our ablation (Table 5, Left), the final performance is not sensitive to the exact value of Δ—any positive value following the GSNR principle consistently outperforms the baseline (Δ=0).

3. DTM Schedule (k-value, Phase-1, Phase-2):
   While the DTM schedule introduces new hyperparameters, our experiments on large-scale datasets (e.g., ImageNet-1K) show that the schedule is highly robust and practical. The DTM consistently improves performance across all tested teacher-student pairs, confirming its stability and utility.

In summary, DeepKD’s key hyperparameters are either fixed by standard practice, guided by strong theoretical principles, or shown to be robust in large-scale settings. This ensures the framework remains practical and easy to adopt.

W3: Concerns about scalability and potential overfitting: The scalability of the method to larger models and more diverse tasks, as well as the risk of overfitting due to decoupling and masking strategies, require further validation.

We appreciate your comments on scalability. To provide a comprehensive validation, we have conducted new experiments covering dataset scale, model scale, task diversity, and different distillation paradigms.

Beyond classification and detection on ImageNet-1K and MS-COCO (see Tables 3 and 4 in our main paper), we further evaluated DeepKD on the challenging object detection task using the Transformer-based DETR architecture (see Table 2 below).

Table 2: DETR Distillation Results on MS-COCO

We also evaluated DeepKD with strong teachers (e.g., RegNetY-16GF) and a diverse set of modern Transformer architectures. For example, DeepKD achieves a +1.90% gain when combined with MLKD on DeiT. Further, our new experiments on Swin Transformer (ICCV 2021) and ViT-to-ViT (ICLR 2021) distillation show notable Top-1 accuracy improvements of +1.12% and +0.92%, respectively (see Table 3).

Table 3: Larger Transformer Experiments on ImageNet-1K

Additionally, we validated DeepKD on semantic segmentation, comparing with KA (Tong He, CVPR 2019), as shown below:

Table 4: Pascal VOC 2012 Semantic Segmentation (val set, trained on trainaug)

In summary, our extensive experiments—spanning large datasets, diverse tasks, strong teachers, a variety of modern Transformer architectures (DeiT, Swin, ViT, DETR), and multiple distillation paradigms—provide robust evidence of DeepKD’s scalability and generality.

W4: Overfitting risk: Could the decoupling and masking strategies cause overfitting if not well regularized? The paper does not discuss this in detail.

Thank you for raising the important issue of overfitting. We believe that the core mechanisms of DeepKD—GSNR-driven decoupling and Dynamic Top-k Masking (DTM)—are inherently regularizing and actually help mitigate overfitting.

1. GSNR-Driven Decoupling as Regularization:
   Our GSNR-based momentum allocation stabilizes training by assigning higher momentum to more reliable gradient signals (TOG and NCG). This leads to convergence at flatter minima in the loss landscape, as shown in our loss landscape analysis (Figure 1(b)). Flatter minima are well-known to correlate with better generalization and reduced overfitting (Hao Li, NIPS 2018).

2. DTM as Curriculum-Based Regularization:
   DTM filters out low-confidence, potentially noisy non-target logits, especially early in training. This prevents the student from overfitting to unreliable aspects of the teacher’s dark knowledge. By gradually expanding the knowledge set, DTM provides a curriculum learning effect, further enhancing generalization.

Together, these mechanisms ensure that the student learns robust, generalizable features, actively guarding against overfitting.

Q1: How is the k value scheduled in DTM, and how does this affect training efficiency and model performance?

Thank you for your question regarding the scheduling of the k value in DTM.

1. DTM Scheduling Strategy:
Our DTM schedule is inspired by curriculum learning and is implemented in three phases using a linear growth heuristic:

• Phase 1 (Easy Learning): K increases linearly from a small value (e.g., 5% of total classes) to an optimal static value (k-value). This allows the student to focus on the most confident, semantically relevant non-target classes at the start.
• Phase 2 (Transition): K is held constant at k-value, giving the model time to consolidate knowledge from this core set.
• Phase 3 (Hard Learning): K increases linearly from k-value to the full class count, exposing the student to the complete distribution of dark knowledge for final refinement.

Empirically, we find it effective to end Phase 1 at ~25% of total epochs (e.g., epoch 60/240) and Phase 2 at ~70-75% (e.g., epoch 170-180/240).

2. Impact of DTM Scheduling:

• Training Efficiency: The scheduling logic is a simple epoch-based if/else and adds negligible computational overhead.
• Model Performance: The curriculum-based schedule consistently improves performance, as shown in all main paper tables. For example, in Table 1, DTM improves DKD+Ours from 77.25% to 77.54%. The approach prevents the student from being overwhelmed by noisy logits early on, leading to more stable and effective learning.

Q2: How does the dual-level decoupling strategy dynamically set the momentum coefficients for different knowledge components during training?

Thank you for this technical question. To clarify, our dual-level decoupling strategy does not dynamically adjust momentum coefficients during training. Instead, our GSNR analysis provides a one-time, principled basis for a fixed momentum allocation.

1. Theoretical Basis:
   As shown in Figure 2(a) of the main paper, throughout training, the gradients for TOG and NCG consistently have higher GSNR than TCG. This indicates that TOG and NCG provide more reliable learning signals.

2. Practical Mechanism:
   Based on this observation, we assign a fixed higher momentum (μ + Δ) to TOG and NCG, and a fixed lower momentum (μ - Δ) to TCG, as described in Equation 7. The value of Δ is constant and, as shown in our ablation, the method is robust to its precise value.

This design achieves the benefits of GSNR-informed optimization without the complexity of dynamic adjustment, making DeepKD both effective and practical.

We thank the reviewer again for their thoughtful feedback. We will revise the manuscript to clarify these points and are confident that these explanations address the concerns raised.

Thank you for the author's detailed response, which has addressed some of my concerns, so I will keep my final rating unchanged.

W1: DeepKD introduces additional computational complexity, which may increase training time and memory usage compared to other KD methods.

Thank you for raising this important and practical concern regarding computational complexity. We analyze DeepKD's overhead on CIFAR-100 and ImageNet-1K. Results are shown in the tables below.

Table 1: Overhead analysis. For CIFAR-100, we use ResNet32×4 as the teacher and ResNet8×4 as the student, training on a single 2080Ti GPU for 240 epochs (480 for MLKD). For ImageNet-1K, we use ResNet34 as the teacher and ResNet-18 as the student, training on two RTX4090 GPUs for 100 epochs.

As shown, the practical overhead introduced by DeepKD is well-controlled and represents a favorable trade-off for the performance improvements achieved.

Memory Consumption is Minimal:
Table 1 demonstrates that DeepKD incurs only a negligible increase in memory usage—less than 1% on CIFAR-100 and less than 0.2% (about 10–25MB per GPU) on ImageNet-1K. This minor increase is primarily due to the storage of student gradients during distillation, as the teacher model remains frozen. The efficient CUDA memory allocator further mitigates any additional memory footprint, confirming that DeepKD is suitable for large-scale training scenarios.

Training Time and Efficiency:
Although DeepKD introduces a moderate increase in per-epoch training time, Table 2 demonstrates that it achieves superior accuracy with substantially fewer epochs than baseline methods. This results in a more favorable overall time-accuracy trade-off. All DeepKD results are from independent full training runs at the specified epochs for fair comparison.

Table 2: DeepKD achieves higher accuracy with fewer training epochs. For fair comparison, all baseline methods are trained for 240 epochs on CIFAR-100 (480 for MLKD) and 100 epochs on ImageNet-1K, using ResNet32×4/ResNet8×4 and ResNet34/ResNet-18 as teacher/student pairs, respectively.

Our framework enables faster convergence and higher final accuracy than baseline methods, even with fewer training epochs. For example:

• On CIFAR-100, KD + Ours achieves 76.83% accuracy in just 160 epochs, outperforming the 240-epoch KD baseline.
• On ImageNet-1K, CRLD + Ours reaches 72.98% accuracy in only 65 epochs, surpassing the 100-epoch CRLD baseline.

Baselines Cannot Match DeepKD Even with Extended Training:
Table 3 further demonstrates that simply increasing the training duration for baseline methods does not close the performance gap with DeepKD.

Table 3: Baseline performance after extended training. For fair comparison, all methods are trained for 480 epochs on CIFAR-100 (960 epochs for MLKD) and 200 epochs on ImageNet-1K, as reported in the main paper.

For instance, even after 480 epochs, the CRLD baseline on CIFAR-100 only reaches 77.99%, which is still below our CRLD+Ours result of 79.25% achieved in half the epochs. The same trend is observed on ImageNet-1K.

The Training Cost is Justified by Substantial Gains:
The primary goal of knowledge distillation is to maximize the student model’s inference performance. In practice, it is standard and reasonable to invest more resources in a one-time training process if it results in a superior model, as inference will be performed many times over the model’s lifecycle. Importantly, this additional training cost does not affect the final inference speed of the student model. In summary, DeepKD provides a highly efficient route to a better-performing model.

W2: The experiments mainly focus on CNNs. More evidence is needed to show DeepKD's effectiveness on Transformer-based architectures.

Thank you for highlighting the importance of evaluating DeepKD on diverse architectures. We appreciate this point and have taken steps to provide more comprehensive evidence of DeepKD’s effectiveness on modern Transformer-based networks.

To clarify, our original submission already included experiments on Vision Transformers, where DeepKD improved the Top-1 accuracy of a DeiT-Tiny student by up to +1.90% (see Table 3 in the main paper).

To further address your concern, we conducted two additional sets of experiments, focusing on pure-Transformer architectures—Swin Transformer (ICCV 2021) and Vision Transformer (ViT, ICLR 2021)—on the ImageNet-1K dataset.

Table 4: Transformer Experiments on ImageNet-1K

As shown, DeepKD consistently and significantly outperforms the standard KD baseline in these challenging Transformer-to-Transformer distillation scenarios.

W3: DeepKD is mainly evaluated together with other logit-based distillation methods, making it hard to isolate its individual effect. More evidence is needed to show its independent contribution and its impact when combined with feature distillation.

Thank you for this insightful question regarding the independent assessment of DeepKD and its applicability beyond logit-based methods. We would like to clarify the nature of our framework and provide new experimental results to directly address your concerns.

First, DeepKD is a training framework or optimization strategy, not a standalone loss function. This is a common characteristic among many state-of-the-art knowledge distillation methods, such as DKD (CVPR 2022), DOT (ICCV 2023), and LSKD (CVPR 2024). These methods are designed to enhance a primary distillation signal, either by reformulating the loss, modifying the optimization process, or pre-processing the logits. As such, none of these methods—including DeepKD—can be meaningfully evaluated in complete isolation from a base teacher-student setup.

Second, the independent effectiveness of our two main contributions—Decoupled Gradients and Dynamic Top-k Masking (DTM)—is already quantified in Table 6 of our main paper, which reports the individual performance gains from each component across various logit-based baselines.

Finally, to further demonstrate the versatility of DeepKD, we conducted new experiments by integrating DeepKD with two widely used feature-based distillation methods: FitNet (ICLR 2015) and CRD (ICLR 2020). The results, summarized in Table 5, show that DeepKD provides substantial additional gains over standard feature distillation and logit-based KD combinations.

Table 5: Feature Distillation Experiments on the CIFAR-100 validation.

These results provide strong evidence that DeepKD is not limited to logit-based distillation. Its GSNR-driven optimization principles can be effectively combined with feature-based methods, demonstrating the generality and strength of our framework. We will include these findings in the appendix of our revised manuscript.

We hope these clarifications and new results comprehensively address your concerns. We will revise our manuscript to make these points more explicit and appreciate your thoughtful feedback.