Single Teacher, Multiple Perspectives: Teacher Knowledge Augmentation for Enhanced Knowledge Distillation

Thank you for your valuable feedback. We appreciate your thoughtful comments and suggestions.

Additional Experiments:

Table-1: The effects of TeKAP on the SOTA methods DKD [1], MLD [2], TAKD [3], CA-MKD [4], and DGKD [5].

Table 2: Effect of multiple original teachers.

Table 3: Effect of different noise techniques.

1. Comparison to Other Ensemble Methods: We have added a new set of experiments comparing TeKAP against CA-MKD, and DGKD. We also perform an ensemble comparison in Table 2. When we employed three teachers on DKD[2] and employed our approach, TeKAP, we experienced that our TeKAP uplifted the performance of the ensemble version as shown in Table 2.

2. Comparison with SOTA Methods: We acknowledge your comment about the comparison with outdated baselines, such as TAKD, and the absence of newer methods like DKD and MLKD. In response, we have added more experiments to compare with existing SOTAs, as suggested by the reviewers. The updated results are shown in Table 1.

3. Details on Teacher Models Used for TAKD: Thank you for pointing out this important concern. We re-run the official implementation of TAKD and use the same corresponding teacher network with lower number of layers, where TeKAP (Ours) denotes (F+L) i.e., (KD+CRD). We will add the experimental details in the supplementary. However, in our additional experiment, we run TAKD in our system for ResNet32x4-ResNet8x4 and WRN_40_2-WRN_40_1 in table 1. We have used only two blocks as the assistant teacher for every corresponding teacher whereas the original vanilla teacher has 3 blocks.

4. Explanation of $\mathcal{L}_{cel}$ in Equation 5: The term $\mathcal{L}_{cel}$ in Equation 5 represents the cross-entropy loss for the student model during training.

5. Summation of Perturbation Loss in Equations 2 and 4: In our work, we did not adjust the $\lambda$ according to the number of perturbations. As we first add the losses of all the perturbations, then use $\lambda$, the effects of noisy and true labels are always scaled by $\lambda$ and $(1-\lambda)$, respectively.

6. CAMs of TeKAP vs. Teacher in Figure 5: We wholeheartedly appreciate for pointing out this important issue. We will update the figure in the revised manuscript.

7. Experimental Details (Learning Rate and Epochs): We have updated the experimental details in the supplementary of the revised manuscript. We appreciate this important suggestion.

8. Feature-Level Perturbation: Which Features Are Selected for Noise?: Regarding feature-level perturbation, we have followed the same setups described and implemented by CRD. We have selected the features immediately before the FC layer by following the work described in CRD for a fair comparison.

Again, thank you very much for these insightful and effective comments. We believe these revisions address the concerns and further enhance the clarity. We have updated the manuscript accordingly.

References

Thank you to the authors for their reply.

1: What do the "original teachers" refer to in Table 2? Were they trained from different initializations?

2: Using only two blocks may not be a fair comparison. How about using three blocks with more shallow networks like WRN-22-2 or WRN-16-2?

5: In Equation 3, $\alpha$ is set to 0.1, which means the perturbed logits are smoother than the original ones. Using a fixed $\lambda$ for various numbers of perturbations does not make sense, as it influences the mean of the logits distribution. Is there any ablation study regarding $\alpha$ and $\lambda$? Additionally, in Equation 5, $\alpha$ is used for a different purpose.

8: I did not find any configuration files in the supplementary. Do the authors mean the default settings in train_student.py?

9: In Table 2 of the authors' reply to Reviewer NkEk, more perturbations seem to harm the student's performance. Can you explain why increasing perturbations destroys the teacher's knowledge pattern? Since the mean of the gradients is converging with the increasing number of perturbations, and based on the theoretical part of the paper, more perturbations should benefit performance.

Results-B: Effect of TeKAP for different variance $\sigma$ on the performance:

Table 6: Effect of TeKAP with different variance $\sigma$. KD is used as the baseline distillation approach. We have used mean zero in all the cases.

Section E (of the supplementary)

Table 6 (table 5 of the supplementary and section E) summarizes the impact of different variances ($\sigma$) on the performance of TeKAP, using the CIFAR-100 dataset. The baseline distillation approach, Knowledge Distillation (KD), is used for comparison. As shown in the results, the accuracy of the model remains relatively stable across varying values of $\sigma$. Specifically, when $\sigma = 0.5$, the model achieves an accuracy of 74.89%, slightly higher than the accuracy at $\sigma = 1$ (74.79%) and $\sigma = 1.5$ (74.35%). These results suggest that, within the range of variances tested, increasing the noise variance does not significantly degrade performance. In fact, the accuracy only decreases marginally as the variance increases from 0.5 to 1.5, which indicates the robustness of TeKAP with respect to noise. This behavior suggests that TeKAP can maintain competitive performance even with varying levels of noise in the teacher models, highlighting its resilience to noise during distillation. The consistent results across different variances also support the idea that TeKAP is stable and less sensitive to slight perturbations in the teacher’s logits. This stability is critical for practical applications where noise may be present in the data or models.

The improvements we have made:

1. (Reviewers: NkEk, pUYy, zdP5, FGoU): Additional comparison with state-of-the-art: Added to the revised manuscript (Table 2, page 7)

2. (Reviewers: NkEk, pUYy, zdP5, FGoU) multi-teacher: The results discussion for the recent SOTA multi-teacher approach is added to section 4.1, Table 2 (page 7) of the revised manuscript.

3. (Reviewers: NkEk): explanation of usage scenarios between the feature level and logit level: Added in section 3.1. Page 4 of the main manuscript. (Please find the changes marked highlights in the supplementary)

4. (Reviewers: NkEk, pUYy) potential benefits of increasing the number of augmented teachers Updated Figure 6 (Now Figure 5 of the main manuscript, Table 2 of this response). We have trained more teachers (till - 10) and provided the potential benefits of increasing the number of augmented teacher models in Table 1 of the supplementary.

5. (Reviewers: NkEk) Evaluation of TeKAP on ensemble learning. Added to the supplementary: Table 2, Section B. Table 3 of the last response.

6. (Reviewer: pUYy): Theoretical Depth: We have extended the theoretical analysis in the supplementary (Section K in details). more theoretical discussion in the supplementary (section D).

7. (Reviewer: pUYy, FGoU, zdP5) effect for different Gaussian noise parameters: We have used mean = 0 and variance = 1 as the default. Additionally, we added the effect for variance $\sigma$ = [0.5, 1, 1.5] in the supplementary (Table 5, section E).

8. (Reviewer: pUYy) comparative computation complexity: Added to section H of the supplementary.

9. (Reviewer: pUYy, FGoU) Description and explanation of every mathematical term on page 5: We have carefully gone through and added the description and explanation of every mathematical term used in the paper.

10. (Reviewer: pUYy, FGoU) Experiments of the class imbalance data: Added to the supplementary Table 4, section D.

11. (Reviewer: pUYy, FGoU) fixed noise experiments: Experiments are running and will be added to the final version and we will also report here with the deadline.

12. (Reviewer: pUYy. zdP5) how inter-class diversity works: Discussion added in the supplementary section I.

13. (Reviewer: zdP5) effect for different values of $\lambda$: Added in the supplementary Table 3, Section C.

14. **(Reviewer: zdP5) Meaning of $L_{cel}:** We have added the meaning of$L_{cel}$ in line 209, page 5 of the main manuscript.

15. (Reviewer: zdP5) More experiments on TAKD with WRN-22-2 or WRN-16-2?: Experiments are running. We will be added in the final version and report here soon.

16. (Reviewer: FGoU) clarification of random distortion and inter-class relationships: Added in the supplementary in section I.

We appreciate your valuable time and effort. These suggestions help the manuscript improve a lot. Again thank you very much for your valuable and insightful comments. We would love to respond if there are any further queries.

8. Question-8: I did not find any configuration files in the supplementary. Do the authors mean the default settings in train_student.py?

*Response-8: The supplementary and revised manuscript is available now. We have added details of experimental setups in section K in the supplementary document.

Note: Please note that we will add more results in the supplementary: (1) TAKD with WRN-22-2 and WRN-16-2, (2) Effect of TeKAP with fixed random noise. The experiments will be reported in the response soon on the CIFAR100 dataset (here before the discussion period ends). As the revised version submission time has ended. We will add these results in the final version and report here.

9. Question: In Table 2 of the authors' reply to Reviewer NkEk, more perturbations seem to harm the student's performance. Can you explain why increasing perturbations destroys the teacher's knowledge pattern? Since the mean of the gradients is converging with the increasing number of perturbations, and based on the theoretical part of the paper, more perturbations should benefit performance.

Response:: We agree with this comment. We are very grateful to the reviewer for pointing out this very important issue. The reported result was confused with the class-imbalanced experiments. However, inspired by these comments we carefully went through again the experiments and evaluations. We have updated Table 2 (reposenses of Review NkEk). The updated results are reported in Figure 5 and Section 4.9 of the revised manuscript.

Table 4: Effect of number of the augmented teachers.

Table. 4 (Fig. 5 of the main manuscript) shows the effect of the number of augmented teachers. We use ResNet32x4-ResNet8x4 as the teacher-student setups on the CIFAR100 dataset to examine the effect of the hyper-parameters. From Table.4 (Fig. 5 of the main manuscript) we see that TeKAP is robust to the number of augmented teachers. For every number of augmented teachers, TeKAP achieves better accuracy than baseline and DKD students. The best performance is achieved when the number of the augmented teacher is $3$. We have used three ($3$), and one $(1)$ augmented teacher along with the original teacher, respectively. During feature and logit distortion, the weights for noise and teacher output are $0.1$, and $0.9$, respectively.

Additional Ablation Study:

*Results-A: Class Imbalance Dataset:

Table 5: Significance of TeKAP on class imbalance dataset. We have used the class distribution of the CIFAR100 dataset that is described in Table 6 (of the supplementary)

Section D (of the supplementary) The results presented in Table 5 (4 of the supplementary) highlight the effectiveness of TeKAP in addressing class imbalance in knowledge distillation tasks. TeKAP consistently improves the performance of all three teacher-student model pairs (ResNet32x4-ResNet8x4, WRN_40_2-WRN_16_2, and VGG13-VGG8) compared to the baseline Knowledge Distillation (KD) approach. Specifically, TeKAP boosts accuracy by 4.71% for ResNet32x4-ResNet8x4, 0.64% for WRN_40_2-WRN_16_2, and 3.73% for VGG13-VGG8. These results indicate that TeKAP is particularly effective in enhancing performance for models with lower baseline accuracy, though it also provides improvements for models with higher baseline accuracy. This suggests that TeKAP can effectively mitigate the effects of class imbalance, leading to improved generalization in knowledge distillation tasks.

We are very grateful for these insightful suggestions. These comments help to improve our paper significantly. We have addressed all the concerns step-by-step and performed additional experiments based on the suggestions:

Important: The supplementary documents and revised versions with both the changes marked and clean copy are available now: We were working on the manuscript and supplementary to reflect all the concerns raised by the reviewers. We are sorry for the inconvenience that we uploaded the revised version late. We have provided supplementary documents and the changes marked by yellow in the supplementary zip file. In the final version, we will remove the highlighted copy.

1. Question: What do the "original teachers" refer to in Table 2? Were they trained from different initializations? Response-1: Yes! We have used different seeds and performed training multiple times. We have added this discussion in the supplementary file as follows:

Table 2 (of the supplementary): The effects of multiple original teachers. We deploy three augmented teachers to every original teacher. ResNet32x4 and ResNet8x4 are considered teacher-student. Here, "Original Teacher" represents the teacher which is trained with 240 epochs and different random seeds. Three different original teachers use three different initializations (i.e., random seeds). AugT denotes the augmented teacher achieved by distorting the original teacher logits with random noise.

Section B (of the supplementary)

Table. 2 (of the supplementary) shows the effect of the number of augmented teachers. We use ResNet32x4-ResNet8x4 as the teacher-student setups on the CIFAR100 dataset to examine the effect of the hyper-parameters. From Table. 2 (of the supplementary) we see that TeKAP is robust to the number of augmented teachers. For every number of augmented teachers, TeKAP achieves better accuracy than baseline and DKD students in every scenario. The best performance is achieved when the number of the augmented teacher is $3$. We have used three ($3$), and one $(1)$ augmented teacher along with the original teacher, respectively. During feature and logit distortion, the weights for noise and teacher output are $0.1$, and $0.9$, respectively.

2. Question: Using only two blocks may not be a fair comparison. How about using three blocks with more shallow networks like WRN-22-2 or WRN-16-2?

Response-2: Thank you very much for this suggestion. We are experimenting on this. The experiments are running. The evaluation will be reported soon here. Please note that we could not at these results in the revised version. But we promise to add this evaluation in the final version as soon as the experiments are finished.

5. Question 5: In Equation 3, is set to 0.1, which means the perturbed logits are smoother than the original ones. Using a fixed for various numbers of perturbations does not make sense, as it influences the mean of the logit distribution. Is there any ablation study regarding and ? Additionally, Equation 5, is used for a different purpose.

Response-5: We appreciate this concern and acknowledge the need for evaluation for different values of $\lambda$ and $\sigma$ for various numbers of perturbations. We have added additional experiments in supplementary documents.

Table 3 (of the supplementary): Effect of the different values of λ (the weights of the noise terms). AugT denotes augmented teachers.

Section C of the supplementary

The results in Table 3 (of the supplementary) demonstrate the effect of varying the noise weight ($\lambda$) and the number of augmented teachers (AugT) on the performance of the student model. ResNet32x4-ResNet8x4 are considered as the teacher and the student. We use $\sigma = 1$ for this experiment. For AugT=5, the accuracy consistently improves as $\lambda$ increases, starting from $74.26\%$ at $\lambda=0.2$ and reaching $75.12\%$ at $\lambda=0.8$. This trend indicates that higher noise weights contribute positively to the student’s generalization by introducing greater diversity. Similarly, for AugT=10, the performance improves from $74.29\%$ at $\lambda=0.2$ to $74.98\%$ at $\lambda=0.8$, but the gains are less pronounced compared to AugT=5, suggesting a saturation effect with a larger number of augmented teachers.

Thanks to the authors for their reply.

6: I reviewed the latest revision, and the CAMs of TeKAP vs. Teacher in Figure 1 (Supplementary) still appear to be the same.

8: I did not find the configuration in the latest provided code. Did authors use $\alpha=0.8$, $\beta=0.2$ and $\lambda=1.0$ in Equation 5 for all teacher-student pairs? Additionally, how many AugTs were used in the experiment? From the provided code, it seems the number is set to 3. If so, could you clarify why, as increasing the number of AugTs generally leads to better performance? Furthermore, there are two hyperparameters labeled as $\alpha$ in both Equation 1 and Equation 5.

9: Why did the authors update the results in Table 4: regarding the effect of the number of augmented teachers? The results for T + (7–10) AugTs were revised, while others remain the same as in the previous version. For example, the performance of T + 10 AugTs improved from 71.4 to 75.11. Could you explain this update?

10: I am unable to match the results for ResNet32x4–ResNet8x4 across Table 1, Table 4, Table 2 (Supplementary), and Table 4 (from the reply). Based on Table 1, Table 4, and Table 2 (Supplementary), I assume the authors used 1 Original + 3 AugTs for their experiments. However, the results in Table 4 (from the reply) differ for 1 Original + 3 AugTs. Could the authors clarify this inconsistency?

Dear Reviewer zdP5,

We are truly grateful for your thoughtful feedback.

We have included further results during the rebuttal, which we have detailed point-by-point for your review. We kindly request you to take a look at them at your convenience.

If these responses address your concerns, we would be grateful if you could consider reassessing the score. Your time and effort are greatly appreciated.

Best regards,
The Authors

Thank you for your constructive feedback. We appreciate the insights from the reviewers. Below, we address each of the points step-by-step:

Additional Experiments:

Table-1: The effects of TeKAP on the SOTA methods DKD [1], MLD [2], TAKD [3], CA-MKD [4], and DGKD [5].

Table 2: Effect of multiple original teachers.

Table 3: Effect of different noise techniques.

Responses:

1. Validation of the Proposed Module: We have conducted additional experiments comparing TeKAP with several state-of-the-art single-teacher and multi-teacher KD methods in Table 1. These results, which include advanced methods like DKD[1], MLKD[2], TAKD[3], CA-MKD[4], and DGKD[5]. Also, we have added additional evaluations in terms of ensemble, multi-teacher augmentations and so on in Tables 2 and 3.

2. Details on Dynamic Noise Perturbation: The generation of random noise happens on every epoch which we addressed as dynamic noise perturbation.

3. Scale of random noise: We have used zero mean and 1 std. to produce random noise on every epoch and perform a weighted combination with the original teacher logits (noise weights with 0.1 and teacher weights with 0.9). We also added detailed guidelines on how to set the noise parameters for different configurations.

4. Meaning of h in Eq 9: h represents a function from the hypothesis class H, which is a set of functions under consideration. Each h maps inputs x_i​ (from the dataset) to real numbers, often representing predictions, scores, or decisions. This measures the capacity or complexity of H.

5. Clarification of Random Distortion and Inter-Class Relationships: While the noise is random, it serves to introduce variability that prevents the student from overfitting to the teacher’s exact logits (i.e. single perspectives). If two classes are strongly correlated in the teacher logits, random distortions will not eliminate this correlation but may perturb its exact magnitude or direction, leading to diverse interpretations of the relationship. Imagine teaching a concept by showing slightly varied examples, this helps learners generalize the concept rather than memorize specific instances. Similar to techniques like dropout (which can be considered implicitly network ensemble learning because every random dropping creates a different network structure), random feature distortion (considered as a diverse network as the outputs are slightly different so it is assumed they come from different networks) can force the model to adapt to a broader range of conditions. This diversity helps the student model avoid collapsing into a rigid interpretation of the teacher’s outputs.

6. Fig. 3(b) vs Fig. 3(c): Thanks for pointing this out. Usually in knowledge distillation feature and logits level knowledge distillation are divided into different categories. So to express that our approach is applicable to both the feature and logits level, we have drawn both feature and logits level figure. Actually, the figure describes that our TeKAP is a applicable to both feature 3(b) and logit level 3(c).

We hope these revisions address your concerns and improve the clarity of the paper. We believe that these additional experiments and clarifications strengthen our work. Thank you again for your valuable feedback.

The improvements we have made:

1. (Reviewers: NkEk, pUYy, zdP5, FGoU): Additional comparison with state-of-the-art: Added to the revised manuscript (Table 2, page 7)

2. (Reviewers: NkEk, pUYy, zdP5, FGoU) multi-teacher: The results discussion for the recent SOTA multi-teacher approach is added to section 4.1, Table 2 (page 7) of the revised manuscript.

3. (Reviewers: NkEk): explanation of usage scenarios between the feature level and logit level: Added in section 3.1. Page 4 of the main manuscript. (Please find the changes marked highlights in the supplementary)

4. (Reviewers: NkEk, pUYy) potential benefits of increasing the number of augmented teachers Updated Figure 6 (Now Figure 5 of the main manuscript, Table 2 of this response). We have trained more teachers (till - 10) and provided the potential benefits of increasing the number of augmented teacher models in Table 1 of the supplementary.

5. (Reviewers: NkEk) Evaluation of TeKAP on ensemble learning. Added to the supplementary: Table 2, Section B. Table 3 of the last response.

6. (Reviewer: pUYy): Theoretical Depth: We have extended the theoretical analysis in the supplementary (Section K in details). more theoretical discussion in the supplementary (section D).

7. (Reviewer: pUYy, FGoU, zdP5) effect for different Gaussian noise parameters: We have used mean = 0 and variance = 1 as the default. Additionally, we added the effect for variance $\sigma$ = [0.5, 1, 1.5] in the supplementary (Table 5, section E).

8. (Reviewer: pUYy) comparative computation complexity: Added to section H of the supplementary.

9. (Reviewer: pUYy, FGoU) Description and explanation of every mathematical term on page 5: We have carefully gone through and added the description and explanation of every mathematical term used in the paper.

10. (Reviewer: pUYy, FGoU) Experiments of the class imbalance data: Added to the supplementary Table 4, section D.

11. (Reviewer: pUYy, FGoU) fixed noise experiments: Experiments are running and will be added to the final version and we will also report here with the deadline.

12. (Reviewer: pUYy. zdP5) how inter-class diversity works: Discussion added in the supplementary section I.

13. (Reviewer: zdP5) effect for different values of $\lambda$: Added in the supplementary Table 3, Section C.

14. **(Reviewer: zdP5) Meaning of $L_{cel}:** We have added the meaning of$L_{cel}$ in line 209, page 5 of the main manuscript.

15. (Reviewer: zdP5) More experiments on TAKD with WRN-22-2 or WRN-16-2?: Experiments are running. We will be added in the final version and report here soon.

16. (Reviewer: FGoU) clarification of random distortion and inter-class relationships: Added in the supplementary in section I.

References

1. Zhao, Borui, et al. "Decoupled knowledge distillation." Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.
2. Jin, Ying, Jiaqi Wang, and Dahua Lin. "Multi-level logit distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
3. Mirzadeh, Seyed Iman, et al. "Improved knowledge distillation via teacher assistant." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.
4. Zhang, Hailin, Defang Chen, and Can Wang. "Confidence-aware multi-teacher knowledge distillation." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
5. Son, Wonchul, et al. "Densely guided knowledge distillation using multiple teacher assistants." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript. Your suggestions have significantly enhanced its quality. We are happy to address any additional queries you may have.

We sincerely thank the reviewer for the constructive feedback. Below, we address the concerns and questions step-by-step.

Additional Experiments:

Table-1: The effects of TeKAP on the SOTA methods DKD [1], MLD [2], TAKD [3], CA-MKD [4], and DGKD [5].

Table 2: Effect of multiple original teachers.

Table 3: Effect of different noise techniques.

Responses

1. Theoretical Depth of Perturbation Methods: We appreciate this insightful comment. We agree that the theoretical depth needs to improve. Actually, Gaussian noise was chosen for its simplicity and general applicability across various domains. However, we also show the effect of uniform distribution noise in Table 3. We have used zero mean and 1 std. to produce random noise on every epoch and perform a weighted combination with the original teacher logits (noise weights with 0.1 and teacher weights with 0.9). We also added detailed guidelines on how to set the noise parameters for different configurations in the revised manuscript.

2. Comparison with SOTA Methods: We agree that the evaluation could benefit from a broader scope. We have added comparisons with additional multi-teacher distillation methods in Table 1.

3. Scalability and Computational Efficiency: The training of a teacher ResNet32x4 in CIFAR100 using KD takes 16 seconds per epoch approximately. As we run for 240 epochs then the total time taken by is 24016 = 64 minutes using 2, 3080 NVIDIA GeForce GPUs. For multi-teacher or ensemble learning we need to train multiple teachers, let's assume 2 teacher assistants of equal size which takes 642 = 128 minutes (approx) for DGKD. In our approach, TeKAP takes 18 seconds per epoch which is in total: 72 minutes only. We will add the complexity in the supplementary.

4. Overclaimed Statements: Thank you very much for this insightful comment. We will improved the literature review in the final version.

5. Gaussian Noise Parameters (Page 4): We have used zero mean and 1 std. (standard Gaussian) to produce random noise on every epoch and perform a weighted combination with the original teacher logits (noise weights with 0.1 and teacher weights with 0.9). In future work, we will work on optimizing these hyperparameters.

6. Incomplete Explanation of Terms (Page 5): We have included the explanation in the revised manuscript.

7. Overfitting Risk with Static Noise Parameters: We agree with this comment. The static noise will create inductive bias shifts or over-fitting. This is why we generated random noise at every epoch which is considered as dynamic noise which creates diversity and balance. However, we are experimenting with static noise. The results will be reported here soon.

8. Handling Class Imbalance: We have included the results below Update: Response to Reviewer pUYy (4).

9. Inter-Class Correlation Discussion (Page 8): If two classes are strongly correlated in the teacher logits, random distortions will not eliminate this correlation but may perturb its exact magnitude or direction, leading to diverse interpretations of the relationship. Imagine teaching a concept by showing slightly varied examples, this helps learners generalize the concept rather than memorize specific instances. Similar to techniques like dropout (which can be considered implicitly network ensemble learning because every random dropping creates a different network structure), random feature distortion (considered as a diverse network as the outputs are slightly different so it is assumed they come from different networks) can force the model to adapt to a broader range of conditions. This diversity helps the student model avoid collapsing into a rigid interpretation of the teacher’s outputs.

10. Number of Augmented Teachers (Figure 6): Empirical results showed that three augmented teachers offer the optimal trade-off between diversity and stability. For future applications, we recommend using three augmented teachers as a default, balancing performance and computational cost. The observed performance with two teachers aligns with expectations due to reduced diversity compared to three.

We are committed to incorporating these changes to strengthen the theoretical and experimental rigour of our work. The revised manuscript will provide a clearer understanding of TeKAP’s capabilities, limitations, and broader applicability. Thank you again for your valuable feedback.

References

1. Zhao, Borui, et al. "Decoupled knowledge distillation." Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.
2. Jin, Ying, Jiaqi Wang, and Dahua Lin. "Multi-level logit distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
3. Mirzadeh, Seyed Iman, et al. "Improved knowledge distillation via teacher assistant." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.
4. Zhang, Hailin, Defang Chen, and Can Wang. "Confidence-aware multi-teacher knowledge distillation." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
5. Son, Wonchul, et al. "Densely guided knowledge distillation using multiple teacher assistants." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

The improvements we have made:

1. (Reviewers: NkEk, pUYy, zdP5, FGoU): Additional comparison with state-of-the-art: Added to the revised manuscript (Table 2, page 7)

2. (Reviewers: NkEk, pUYy, zdP5, FGoU) multi-teacher: The results discussion for the recent SOTA multi-teacher approach is added to section 4.1, Table 2 (page 7) of the revised manuscript.

3. (Reviewers: NkEk): explanation of usage scenarios between the feature level and logit level: Added in section 3.1. Page 4 of the main manuscript. (Please find the changes marked highlights in the supplementary)

4. (Reviewers: NkEk, pUYy) potential benefits of increasing the number of augmented teachers Updated Figure 6 (Now Figure 5 of the main manuscript, Table 2 of this response). We have trained more teachers (till - 10) and provided the potential benefits of increasing the number of augmented teacher models in Table 1 of the supplementary.

5. (Reviewers: NkEk) Evaluation of TeKAP on ensemble learning. Added to the supplementary: Table 2, Section B. Table 3 of the last response.

6. (Reviewer: pUYy): Theoretical Depth: We have extended the theoretical analysis in the supplementary (Section K in details). more theoretical discussion in the supplementary (section D).

7. (Reviewer: pUYy, FGoU, zdP5) effect for different Gaussian noise parameters: We have used mean = 0 and variance = 1 as the default. Additionally, we added the effect for variance $\sigma$ = [0.5, 1, 1.5] in the supplementary (Table 5, section E).

8. (Reviewer: pUYy) comparative computation complexity: Added to section H of the supplementary.

9. (Reviewer: pUYy, FGoU) Description and explanation of every mathematical term on page 5: We have carefully gone through and added the description and explanation of every mathematical term used in the paper.

10. (Reviewer: pUYy, FGoU) Experiments of the class imbalance data: Added to the supplementary Table 4, section D.

11. (Reviewer: pUYy, FGoU) fixed noise experiments: Experiments are running and will be added to the final version and we will also report here with the deadline.

12. (Reviewer: pUYy. zdP5) how inter-class diversity works: Discussion added in the supplementary section I.

13. (Reviewer: zdP5) effect for different values of $\lambda$: Added in the supplementary Table 3, Section C.

14. **(Reviewer: zdP5) Meaning of $L_{cel}:** We have added the meaning of$L_{cel}$ in line 209, page 5 of the main manuscript.

15. (Reviewer: zdP5) More experiments on TAKD with WRN-22-2 or WRN-16-2?: Experiments are running. We will be added in the final version and report here soon.

16. (Reviewer: FGoU) clarification of random distortion and inter-class relationships: Added in the supplementary in section I.

Thank you for taking the time to provide detailed and thoughtful comments. Your feedback has been instrumental in improving the manuscript. We are deeply grateful for your insights and are ready to respond to any further questions or concerns.

We have added new results on class imbalance dataset, effects of different values for variance $\sigma$ for Gaussian, and various valus of $\gamma$.

*Results-A: Class Imbalance Dataset:

Table 5: Significance of TeKAP on class imbalance dataset. We have used the class distribution of the CIFAR100 dataset that is described in Table 6 (of the supplementary)

Section D (of the supplementary) The results presented in Table 5 (4 of the supplementary) highlight the effectiveness of TeKAP in addressing class imbalance in knowledge distillation tasks. TeKAP consistently improves the performance of all three teacher-student model pairs (ResNet32x4-ResNet8x4, WRN_40_2-WRN_16_2, and VGG13-VGG8) compared to the baseline Knowledge Distillation (KD) approach. Specifically, TeKAP boosts accuracy by 4.71% for ResNet32x4-ResNet8x4, 0.64% for WRN_40_2-WRN_16_2, and 3.73% for VGG13-VGG8. These results indicate that TeKAP is particularly effective in enhancing performance for models with lower baseline accuracy, though it also provides improvements for models with higher baseline accuracy. This suggests that TeKAP can effectively mitigate the effects of class imbalance, leading to improved generalization in knowledge distillation tasks.

Results-B: Effect of TeKAP for different variance $\sigma$ on the performance:

Table 6: Effect of TeKAP with different variance $\sigma$. KD is used as the baseline distillation approach. We have used mean zero in all the cases.

Section E (of the supplementary)

Table 6 (table 5 of the supplementary and section E) summarizes the impact of different variances ($\sigma$) on the performance of TeKAP, using the CIFAR-100 dataset. The baseline distillation approach, Knowledge Distillation (KD), is used for comparison. As shown in the results, the accuracy of the model remains relatively stable across varying values of $\sigma$. Specifically, when $\sigma = 0.5$, the model achieves an accuracy of 74.89%, slightly higher than the accuracy at $\sigma = 1$ (74.79%) and $\sigma = 1.5$ (74.35%). These results suggest that, within the range of variances tested, increasing the noise variance does not significantly degrade performance. In fact, the accuracy only decreases marginally as the variance increases from 0.5 to 1.5, which indicates the robustness of TeKAP with respect to noise. This behavior suggests that TeKAP can maintain competitive performance even with varying levels of noise in the teacher models, highlighting its resilience to noise during distillation. The consistent results across different variances also support the idea that TeKAP is stable and less sensitive to slight perturbations in the teacher’s logits. This stability is critical for practical applications where noise may be present in the data or models.

*Results-C: Effect of various $\lambda$:

Table 3 (of the supplementary): Effect of the different values of λ (the weights of the noise terms). AugT denotes augmented teachers.

Section C of the supplementary

The results in Table 3 (of the supplementary) demonstrate the effect of varying the noise weight ($\lambda$) and the number of augmented teachers (AugT) on the performance of the student model. For AugT=5, the accuracy consistently improves as $\lambda$ increases, starting from $74.26\%$ at $\lambda=0.2$ and reaching $75.12\%$ at $\lambda=0.8$. This trend indicates that higher noise weights contribute positively to the student’s generalization by introducing greater diversity. Similarly, for AugT=10, the performance improves from $74.29\%$ at $\lambda=0.2$ to $74.98\%$ at $\lambda=0.8$, but the gains are less pronounced compared to AugT=5, suggesting a saturation effect with a larger number of augmented teachers.

Result D Static (Fixed) vs Dynamic Noise (We will add this response to the supplementary of the final version)

Results D: Additional Experiments

Table D: Evaluation of the comparative effects between static and dynamic noise. KD has been used as the baseline distillation approach. The experiment is conducted with three augmented teachers. We use $\sigma = 1$, $\lambda = 0.8$, and three augmented teachers. Gaussian noise is used to generate the noise.

The results in Table C demonstrate the effectiveness of dynamic noise over static noise and the baseline Knowledge Distillation (KD) approach across three teacher-student pairs: ResNet32x4-ResNet8x4, WRN_40_2-WRN_16_2, and VGG13-VGG8 on CIFAR100 dataset. Baseline KD provides solid performance, achieving 73.33%, 74.92%, and 72.98% accuracy, respectively. Incorporating static noise (TeKAP Static-L) shows minor improvements for ResNet32x4-ResNet8x4 and VGG13-VGG8, achieving 73.74% and 73.29%, but performs slightly worse (74.66%) for WRN_40_2-WRN_16_2, indicating its limited adaptability. Conversely, our proposed dynamic noise strategy (TeKAP Dynamic-L) consistently outperforms both static noise and baseline KD, achieving significant gains with accuracies of 74.79%, 75.21%, and 74.00%, respectively. This superiority stems from dynamic noise's adaptability enabling robust generalization. These findings underscore the robustness and efficacy of dynamic noise in enhancing knowledge transfer during distillation, providing a compelling case for its application in improving student network performance across diverse architectures.

We will add this response (ablation study: the comparative effect between static vs dynamic noise on TeKAP in the supplementary).

Thank you for your detailed and insightful comments. Your feedback has significantly contributed to improving the manuscript. We are happy to address any additional questions or concerns you may have.

Dear Reviewer pUYy,

Thank you for your thoughtful comments and suggestions. We have added further results with detailed, point-by-point explanations for your review. We hope these enhancements meet your expectations, and if you feel they merit reconsidering your rating, we would be truly grateful.

Best regards,
The Authors

We thank you for the valuable feedback and suggestions on our submission. We have addressed the comments and questions step-by-step below:

Additional Experiments:

Table-1: The effects of TeKAP on the SOTA methods DKD [1], MLKD [2], TAKD [3], CA-MKD [4], and DGKD [5].

Responses:

1. Weakness-1: More comparison with recent multi-teacher work: We have included more comparisons with DKD, MLKD, TAKD, CA-MKD, and DGKD, evaluated on the CIFAR-100 dataset. TeKAP outperformed all approaches under every scenario.

2. Weakness-2: Insufficient explanation of the difference in usage scenarios between feature-level and logit-level: Thanks for this insightful suggestion. Logit-level augmentation primarily diversifies the inter-class relationships, providing alternative supervisory signals that regularize the student network. Feature-level augmentation, on the other hand, introduces diversity in intermediate feature representations, exposing the student to a broader spectrum of variations (like dropout or data augmentation). Both augmentations target distinct aspects of teacher knowledge: logits focus on prediction diversity, while features address internal representation diversity.

3. Question-1: Inclusion of more distillation methods for a more convincing study: We appreciate the suggestion. In our revised manuscript, We will add more comparisons with SOTA techniques as shown in Table 1 (here).

Table 2: Effect of number of the augmented teachers.

4. Question-2: Potential benefits of increasing the number of teacher models: In Figure 6, we show that TeKAP consistently benefits from additional synthetic teachers up to a certain threshold. Increasing the number of augmented teacher models will not further improve the performance because too much noise will destroy the teacher knowledge pattern instead of regularizing (similar to dropout or data augmentation). The same thing happens in ensemble learning where excessive models may introduce noise. We have run additional experiments till 10 augmented teachers.

Table 3: Effect of multiple original teachers.

5. Inspired By: More teachers with augmentation of every teacher: We have run additional experiments where we use multiple teachers (2 and 3) of ResNet32x4 (training using different seeds and lr). We have augmented each teacher with three noise sets. We experience performance improvements while increasing the number of teachers as shown in Table 3.

Again, thank you very much for these insightful suggestions. We experience that these suggestions help improve the script. We will add these responses to our revised manuscript accordingly.

References

1. Zhao, Borui, et al. "Decoupled knowledge distillation." Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.
2. Jin, Ying, Jiaqi Wang, and Dahua Lin. "Multi-level logit distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
3. Mirzadeh, Seyed Iman, et al. "Improved knowledge distillation via teacher assistant." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.
4. Zhang, Hailin, Defang Chen, and Can Wang. "Confidence-aware multi-teacher knowledge distillation." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
5. Son, Wonchul, et al. "Densely guided knowledge distillation using multiple teacher assistants." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

11. (Reviewer: pUYy, FGoU) fixed noise experiments: Experiments are running and will be added to the final version and we will also report here with the deadline.

12. (Reviewer: pUYy. zdP5) how inter-class diversity works: Discussion added in the supplementary section I.

13. (Reviewer: zdP5) effect for different values of $\lambda$: Added in the supplementary Table 3, Section C.

14. **(Reviewer: zdP5) Meaning of $L_{cel}:** We have added the meaning of$L_{cel}$ in line 209, page 5 of the main manuscript.

15. (Reviewer: zdP5) More experiments on TAKD with WRN-22-2 or WRN-16-2?: Experiments are running. We will be added in the final version and report here soon.

16. (Reviewer: FGoU) clarification of random distortion and inter-class relationships: Added in the supplementary in section I.

We appreciate the effort you put into reviewing our manuscript. Your suggestions have been invaluable in refining and improving its quality. Thank you for your thoughtful comments, and we would be glad to address any further queries.

We appreciate the valuable time and efforts offered by the Reviewer NkEk. Please note that we have updated the results in Table 2 of the previous response. We have added more results and a details analysis of our paper based on concerns raised by all the reviewers.

We would like to request to have a look again at our revised manuscript and overall responses. We would be happy if the reviewer again go through the responses, revised manuscript, supplementary, and reassess our updated manuscript.

The updated Table 2 can be found as: Correction: Table 2

The reported result was confused with the class-imbalanced experiments. However, inspired by these comments we carefully went through again the experiments and evaluations. We have updated Table 2 of the previous responses. The updated results are reported in Figure 5 and Section 4.9 of the revised manuscript.

Table 4: Effect of number of the augmented teachers.

Table. 4 (Fig. 5 of the main manuscript) shows the effect of the number of augmented teachers. We use ResNet32x4-ResNet8x4 as the teacher-student setups on the CIFAR100 dataset to examine the effect of the hyper-parameters. From Table.4 (Fig. 5 of the main manuscript) we see that TeKAP is robust to the number of augmented teachers. For every number of augmented teachers, TeKAP achieves better accuracy than baseline and DKD students. The best performance is achieved when the number of the augmented teacher is $3$. We have used three ($3$), and one $(1)$ augmented teacher along with the original teacher, respectively. During feature and logit distortion, the weights for noise and teacher output are $0.1$, and $0.9$, respectively.

Concerns

We have updated our manuscript based on the reviews added supplementary documents, and revised the manuscript with changes highlights. The updated manuscript's clean version, changes marked with yellow, and supplementary are available now.

The improvements we have made:

1. (Reviewers: NkEk, pUYy, zdP5, FGoU): Additional comparison with state-of-the-art: Added to the revised manuscript (Table 2, page 7)

2. (Reviewers: NkEk, pUYy, zdP5, FGoU) multi-teacher: The results discussion for the recent SOTA multi-teacher approach is added to section 4.1, Table 2 (page 7) of the revised manuscript.

3. (Reviewers: NkEk): explanation of usage scenarios between the feature level and logit level: Added in section 3.1. Page 4 of the main manuscript. (Please find the changes marked highlights in the supplementary)

4. (Reviewers: NkEk, pUYy) potential benefits of increasing the number of augmented teachers Updated Figure 6 (Now Figure 5 of the main manuscript, Table 2 of this response). We have trained more teachers (till - 10) and provided the potential benefits of increasing the number of augmented teacher models in Table 1 of the supplementary.

5. (Reviewers: NkEk) Evaluation of TeKAP on ensemble learning. Added to the supplementary: Table 2, Section B. Table 3 of the last response.

6. (Reviewer: pUYy): Theoretical Depth: We have extended the theoretical analysis in the supplementary (Section K in details). more theoretical discussion in the supplementary (section D).

7. (Reviewer: pUYy, FGoU, zdP5) effect for different Gaussian noise parameters: We have used mean = 0 and variance = 1 as the default. Additionally, we added the effect for variance $\sigma$ = [0.5, 1, 1.5] in the supplementary (Table 5, section E).

8. (Reviewer: pUYy) comparative computation complexity: Added to section H of the supplementary.

9. (Reviewer: pUYy, FGoU) Description and explanation of every mathematical term on page 5: We have carefully gone through and added the description and explanation of every mathematical term used in the paper.

10. (Reviewer: pUYy, FGoU) Experiments of the class imbalance data: Added to the supplementary Table 4, section D.