Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity

Dear reviewer KCJt,

Thank you for your time and efforts in reviewing our manuscript. We greatly appreciate your recognition of the strengths of our work, including the well-motivated and novel methodology, the easy-to-follow presentation, and the theoretical results that help explain how our approach improves accuracy.

[W1, Q2] Request for results averaged over three runs.

We fully agree that averaging over multiple random seeds provides a more reliable evaluation. Our initial submission did not include three-run average results in Table 1 due to computational constraints (Table 1 spans 18 settings across architectures, and our experiments were conducted on limited resources of 2080 Ti GPUs). In the table below, we additionally report three-run averaged results under two representative configurations from Table 1: (1) a ResNet-32x4 teacher -> a ResNet-8x4 student, and (2) a ResNet32x4 teacher -> a WideResNet-40-2 student. These are presented alongside results from TeKAP, a SoTA knowledge augmentation method.

We will include the full results of Table 1 across all settings with three-run averages in the final version, if accepted.

[W2, Q2] More comparisons with other KD methods beyond TeKAP.

Thank you for the reviewer’s valuable comments. We understand that limiting the comparison to only TeKAP may not sufficiently demonstrate the general effectiveness of our approach across diverse knowledge distillation (KD) paradigms.

To address this concern, we emphasize that our method is framework-agnostic and designed to be plug-and-play, making it compatible with a wide range of KD frameworks. In our experiments, we applied our method on top of several representative and diverse KD frameworks, including DKD [57], MLKD [20] and TAKD [33] in addition to conventional KD [17] and CRD [45], and observed consistent improvements over their baseline performances (see Table 2 and Table 11).

Additionally, the table below shows further plug-and-play results from applying our methods to Review-KD [6], in comparison with TeKAP. Once again, our method achieves consistent improvements, and this suggests that our approach can complement, rather than replace, existing KD methods.

We will revise the manuscript to clarify this point and better highlight the broad compatibility and effectiveness of our approach across various KD frameworks.

[W3] Typo on line 314.

Thank you for pointing out the typo. We will correct “Hiton” to “Hinton” on line 314 in the final version.

Dear reviewer KCJt,

We truly appreciate your recognition of our efforts in addressing the reviewers’ critiques, and we’re glad to hear that our responses were clear and satisfactory.

In the final version, we will make sure to present the relevant results and discussions more explicitly.

Please don’t hesitate to reach out if you have any further questions or suggestions.

Thank you very much,

Authors.

Dear reviewer SW6m,

Thank you for your time and efforts in reviewing our manuscript. We appreciate your recognition of our paper’s strengths, particularly our novel approach using angular diversity, the compatibility of our methods, and a well-structured paper. Below, we would like to provide clarifications and address the concerns and limitations you raised.

[W1] The assumption that intra-angle diversity loss promotes an even spread of views is unreasonable.

We acknowledge the concern that varying similarities between different views and the teacher output could challenge the assumption of an even spread. However, in our empirical analysis (Table 15 in the supplementary), we observe that the intra-angle diversity loss significantly increases pairwise intra-angles between views. We interpret larger intra-angles as indicating that the views are more evenly distributed around the teacher representation, i.e., they are less clustered in a single direction, as also illustrated in Figure 1b. We believe this supports our claim that the intra-angle diversity loss encourages directional diversity among the views.

Furthermore, we clarify that in this work, the term “even spread” is used in the sense of promoting representational diversity and reducing directional redundancy among views, rather than enforcing equal similarity to the teacher output. We will revise the final version to make this interpretation more explicit and avoid potential confusion.

[W2] The first term on the right-hand side of the bound increases with the number of augmented views N.

We would like to clarify that the first term on the right-hand side of the expected ensemble loss represents the mean of the losses across all augmented views, not their sum. Therefore, this term does not increase with the number of views N; rather, it reflects the average quality of individual views (e.g., each view’s cross-entropy loss with respect to the ground-truth label).

We will revise the explanation around Eq. (6) in the final version to clearly distinguish this averaging behavior and avoid any potential ambiguity.

[W3] The training time comparisons.

Thank you for your great suggestions. We agree that including a comparison of training time would be valuable. The following table compares the training time of our methods with 1) naive KD + CRD without augmentation, 2) the augmented one by TeKAP (a SoTA knowledge augmentation method), and 3) multi-teacher methods (e.g., Ensemble distillation [3]), including computational demands (FLOPs and Params; reported in Table 5). This experiment is conducted with a WideResNet40-2 teacher and a WideResNet-16-2 student on a 2080 Ti GPU over 50,000 training images and a total of 240 epochs.

We observe that our multiple branches incur only a marginal increase in both training time and computation cost (FLOPs and Params), since the branches are composed of lightweight linear heads. In the revision, we will include these results to clearly present the lightweight nature of our approach.

[W4] Ablations on the loss coefficients.

Yes, we appreciate your valuable suggestion. The table below reports an ablation study over the three loss coefficients.

We observe that setting all coefficients to 1 yields the best performance. In contrast, reducing the coefficients for both inter‑ and intra‑angle diversity losses to 0.5 results in a substantial decrease in performance, suggesting that these two angular losses are critical. We will include this result in the final version to highlight the effect of each loss component.

[W5, L1] Experiments on other tasks and ImageNet-1k.

We appreciate the reviewer’s interest in evaluating broader applicability. However, we would like to clarify that results on ImageNet-1K are already included in our main manuscript—specifically in Table 10 of the supplementary material, where our approach outperforms the naive KD method without augmentation and a SoTA knowledge augmentation method, TeKAP.

To further validate that our approach is effective in other tasks, we are currently conducting object detection or image segmentation experiments and exploring training recipes optimized for our environments. Given the time required for thorough evaluation, we kindly ask whether it would be acceptable to share the results during the reviewer–author discussion phase. We will update the results as soon as the experiments are complete.

Dear Reviewer SW6m,

Thank you for your patience and understanding while we completed the additional experiments on other tasks. To provide quantitative results within our computational constraints using NVIDIA 2080 Ti GPUs during the reviewer–author discussion phase, we conducted a binary segmentation task on the Carvana Image Masking dataset (5,088 training images and 100,064 test images) using a teacher U-Net (depth 32) and a student U-Net (depth 16) in the table below.

Our approach shows clear improvements across all metrics over the naive knowledge distillation. In the final version, we will extend these results to COCO-stuff for the semantic segmentation or MS-COCO for the object detection, and incorporate implementation details for each task.

If you have any further questions or suggestions, please do not hesitate to let us know.

Thank you again for your time and feedback.

Sincerely,

Authors.

Dear Reviewer SW6m,

We appreciate your constructive comments for helping us to improve our paper in many aspects.

This is a gentle reminder since the reviewer-author discussion window is coming to a close.

We would like to ask if you have any further questions regarding our submission paper so that we can still respond.

Thanks.

Authors.

Dear reviewer TLMU,

Thank you for your time and efforts in reviewing our manuscript. We appreciate your recognition of our paper’s strengths, particularly our novel and compatible methods, comprehensive experiments on multiple datasets, and our theoretical analysis and codes. Below, we would like to provide clarifications and address the concerns and questions you raised.

[W1] The comparison is limited to only one data augmentation method, lacking comparisons with other advanced methods.

Thank you for the reviewer’s valuable comments. We understand that limiting the comparison to only TeKAP may not sufficiently demonstrate the general effectiveness of our approach across diverse knowledge distillation (KD) paradigms.

To address this concern, we emphasize that our method is framework-agnostic and designed to be plug-and-play, making it compatible with a wide range of KD frameworks. In our experiments, we applied our method on top of several representative and diverse KD frameworks, including DKD [57], MLKD [20] and TAKD [33] in addition to conventional KD [17] and CRD [45], and observed consistent improvements over their baseline performances (see Table 2 and Table 11).

Additionally, the table below shows further plug-and-play results from applying our methods to Review-KD [6], in comparison with TeKAP. Once again, our method achieves consistent improvements, and this suggests that our approach can complement, rather than replace, existing KD methods.

We will revise the manuscript to clarify this point and better highlight the broad compatibility and effectiveness of our approach across various KD frameworks.

[W2] Compared multi-teacher methods are not up-to-date.

We acknowledge that the compared multi-teacher methods (TAKD [33], published in 2023; and DGKD [40], published in 2021) may not appear very recent. However, to the best of our knowledge, they remain the latest and SoTA multi-teacher KD methods available to date. If any newer approaches in this category emerge, we will incorporate them into our revision accordingly.

[W3] Additional Visualization.

Thank you for your helpful suggestion. We analyzed two visualizations for our augmented views and the original view, compared to those of TeKAP: 1) UMAP embeddings, and 2) the decision boundaries. Although we cannot add figures during rebuttal due to the NeurIPS 2025 rebuttal policy, we observe the following: 1) In UMAP, TeKAP's augmented logits substantially overlap with the teacher's logit, whereas our method yields more dispersed views, aligning with TeKAP's lower ensemble diversity scores (Table 16, supplementary); and 2) In decision boundary plots, our augmented views produce clearly more discriminative boundaries, as our higher accuracy of each view than TeKAP's each view (Table 14, supplementary).

We will include additional visualizations of UMAP and decision boundaries, along with a detailed implementation process, in the final version, if accepted.

[Q1] Results on larger-scale models.

We appreciate the reviewer’s thoughtful recommendation. To assess scalability, in the following table, we conduct experiments with larger teacher and student configurations: 1) a ResNet-110 teacher and a ResNet-32 student, and 2) a ResNet-110 teacher and a ResNet-20 student.

Across all settings, our methods yield consistent improvements and outperform TeKAP (a SoTA knowledge augmentation method), indicating that our approach remains effective at larger model scales. We will include these results in the final version.

Furthermore, in Table 14 of the supplementary, integrating our approach into the multi-teacher framework (TAKD [33]), which has high complexity due to the multi-teacher network, improves distillation performance, underscoring the robustness of our method under high-complexity methods.

Dear Reviewer TLMU,

Thank you once again for your time and effort in reviewing our work.

As the reviewer-author discussion phase is approaching its end, we would like to kindly check if you have any additional questions or points for clarification.

We would be grateful for the opportunity to address any remaining concerns during the discussion period.

Sincerely,

Authors.

Dear Reviewer TLMU,

We appreciate your constructive comments for helping us to improve our paper in many aspects.

This is a gentle reminder since the reviewer-author discussion window is coming to a close.

We would like to ask if you have any further questions regarding our submission paper so that we can still respond.

Thanks.

Authors.

Dear reviewer tUi1,

Thank you for your time and efforts in reviewing our manuscript. We appreciate your recognition of our paper’s strengths, particularly its clear writing, well-structured mathematical approach. Below, we would like to provide clarifications and address the concerns and questions you raised.

[W1, L1] Limited Novelty on mimicking the concept of negative pairs from contrastive learning.

We would like to clarify that the novelty of our approach lies in promoting a broadened coverage of augmented views via intra- and inter-angular diversity maximization. Although the inter-angle loss, particularly the constraint term in Eq. (3), may superficially resemble conventional contrastive learning, our core formulation, especially the diversity term, is fundamentally different. Rather than treating each view as a negative pair and pulling it indiscriminately from the teacher view, we encourage angular separation among all pairs of multi-views, while preserving their alignment with the teacher. This promotes diversity without sacrificing representational consistency. Intra-angle diversity loss further ensures an even distribution across views around the teacher output. We believe that this fundamental distinction sets our formulation and ideas apart from conventional contrastive learning objectives, which focus on negative separation.

Moreover, our work presents two novel findings: 1) Angularly diversified knowledge from a single teacher can serve as, and even outperform multi-teacher networks (Table 5), and 2) Higher angular diversity theoretically and empirically improves distillation effectiveness by increasing ensemble diversity and lowering ensemble loss (Tables 15 and 16).

To the best of our knowledge, these analyses on the effect of angular diversity are not covered in existing knowledge distillation and ensemble learning studies, underscoring the value of our contributions.

[W2] Minimal accuracy improvement.

We believe that our improvements are substantial and meaningful, particularly given the strength of the underlying baselines. To better quantify this, we compare the average improvements of our method and the current SoTA augmentation method, TeKAP, relative to the non-augmented baselines across all 18 settings in Table 1. As shown in the “Average Gains” column of the following table, our augmentation method yields an average gain of +1.02pp, whereas TeKAP achieves only +0.36pp from baseline methods.

Our average gain is 2.86 times larger than TeKAP (+1.02pp vs. +0.36pp). We believe this consistent performance gain across diverse settings demonstrates the practical effectiveness of our approach.

[Q1] Why is our view augmentation better than the data augmentation?

Thanks for your insightful question. We believe that our view-augmentation provides two major benefits over the data augmentation.

1. Efficiency (1 forward pass vs. N forward passes); our method produces multi-views with a single forward pass, whereas data augmentation typically requires N forward passes for N augmented inputs, leading to significantly higher computational cost.
2. Evenly dispersed representations; by explicitly maximizing the angular diversity in the representation space, our method ensures that augmented views are evenly dispersed and exhibit minimal redundancy. In contrast, heuristic data augmentation influences embeddings only indirectly, often resulting in redundant representations or ones that drift away from the teacher’s original view, making it difficult to achieve consistent and balanced dispersion in the representation space.

To further investigate this, we applied standard data augmentations (random flip, color jitter, and random rotation) to generate multiple views and ensemble them for student distillation, similar to our view augmentations. The table below shows the CIFAR-100 accuracy for our view-augmented logits vs. the data-augmented logits, where a ResNet-32x4 teacher and a ResNet-8x4 student are used.

We observe a substantial performance drop when multiple views are generated via data augmentation. To analyze this degradation, we check the accuracy of each augmented view in the table below.

Each augmented view by data augmentation shows significantly lower accuracy due to the noise caused by the data perturbation, leading the overall performance to decline.

We believe that this highlights the effectiveness of our approach in producing consistent and diverse representations without computational overhead. We will include these results and a detailed comparison with the data‑augmentation approach in the final version if accepted.

[Q2] How can the diversity introduced by view augmentation be measured quantitatively?

Yes, we quantify diversity using the ensemble-diversity metric [35]. As shown in the supplementary materials (Table 16), our method achieves 11.633, whereas TeKAP obtains 6.650, demonstrating a substantially higher degree of diversity. Table 6 further shows that each of our proposed loss functions contributes to increasing this diversity metric. Finally, in Figure 4 of the supplementary, we visualize the correlation matrix (i.e., similarity matrix) between augmented views compared to TeKAP; our approach yields significantly lower similarity between views than TeKAP, further validating that our methods produce more diverse augmented logits.

[Q3] Which dataset was used for Table 7?

Thank you for catching this omission. Table 7 presents results on CIFAR-100. We will update the manuscript to indicate this clearly.

[Q4] Can our method control the diversity at the class level?

Thank you for your valuable suggestion. Yes. We do so as follows:

First, we compute per-class variance to quantify each class's diversity by measuring the variance over all samples of each class in the table below. As you mentioned, we observe that the diversity varies across classes.

Then, we normalize these variances via the mean value over those of all classes to obtain class-wise weights. Finally, we apply these weights to the inter-angle and intra-angle diversity losses. Formally,

$L^{aug}=\frac{1}{C}\sum_{c=1}^{C}w_c(L^{inter}_c + L^{intra}_c) + L^{GT},$

where C is the number of classes, L^{inter}_{c} is the constrained inter-angle diversity loss on the sample x with class c, L^{intra}_{c} is the intra-angle diversity loss on the sample x with class c, and L^{GT} is the ground-truth loss. With this implementation, classes with larger diversity receive larger loss weights, further promoting their views to spread (reducing overlapping). The table below reports the performance of this class-wise re-weighting on CIFAR-100.

We observe that this yields no significant improvement in performance. However, since we have confirmed that diversity indeed varies across classes, we will leverage this insight in future work to explore its potential.

[L2] Additional analytical experiments based on diversity margins or the number of views.

Thank you for the suggestion. As noted in the main submission, we already report Top-1 accuracy ablations for the angular margin (Table 9b) and the number of views (Table 7), but we agree that further analysis would be valuable for our paper. Below, we present extended ablations over additional values of angular margin and the number of views.

We will include these additional results in the final version, if accepted.

[L3] Review of CIFAR-100 training sample count.

As described in Section 4.1 (line 210), CIFAR-100 contains 50,000 training and 10,000 validation images across 100 classes. We will revise the manuscript to ensure this is more clearly stated where CIFAR-100 is mentioned.

[L4] Meaningless subsections (a) and (b) in Table 9.

Thank you for pointing this out. We will update subsections in Table 9 to: “(a) Effect of diversity and constraint terms” and “(b) Effect of angular margin 𝛾” to ensure each subtable is clearly presented.

Dear Reviewer tUi1,

Thank you once again for your time and effort in reviewing our work.

As the reviewer-author discussion phase is approaching its end, we would like to kindly check if you have any additional questions or points for clarification.

We would be grateful for the opportunity to address any remaining concerns during the discussion period.

Sincerely,

Authors.

Dear Reviewer tUi1,

We appreciate your constructive comments for helping us to improve our paper in many aspects.

This is a gentle reminder since the reviewer-author discussion window is coming to a close.

We would like to ask if you have any further questions regarding our submission paper so that we can still respond.

Thanks.

Authors.

I would like to express my sincere gratitude to the reviewer for their thoughtful and dedicated responses. It is truly encouraging to see that the author not only addressed every question in detail but also took the extra step of conducting new experiments to support their feedback. I am glad and appreciative of the effort and care reflected in these responses. At the same time, I believe that several of the responses to specific questions should be further refined in the manuscript to ensure a meaningful improvement in the overall quality of the paper. I will raise my rating score.