Right Time to Learn: Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation

Thank you for your valuable comments. We provide a point-to-point response as follows. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.

Q1: The additional overhead that Spaced KD incurs compared to vanilla KD.

The computational and parameter overhead of our Spaced KD is essentially identical the counterparts of online KD and self KD. Regardless of whether or not Spaced KD is introduced, the entire model (both teacher and student) employs the same network architecture and is trained with the same total number of epochs. In practice, since we need to train the teacher s epochs in advance and wait for the student to follow, this results in a slight delay in the runtime (around 30%) but does not increase the computational overhead (i.e., the waiting teacher is frozen, not computing).

Q2: Lower update frequency of the teacher model.

Our claim "less frequent" means that, the teacher is distilled every s epochs in Spaced KD, rather than every iteration in online KD. We will clarify this in the final version.

Q3: How Spaced KD applies to self KD and the additional parameter/training overhead.

In self KD, the deepest layer (as the teacher) transfers knowledge to the shallow layers (as the student) at each training time step [1]. In its spaced version, we first train the model using the cross-entropy loss between the deepest layer's output and the ground-truth label for $s$ steps. We then train the model using the standard self KD loss [1] between the deepest layer's output and each shallow layer's output for $s$ steps. Therefore, our spaced version of self KD does not store additional parameters and does not increase the training overhead (i.e., the total number of epochs is not changed). We will make this clearer.

[1] Be your own teacher: Improve the performance of convolutional neural networks via self distillation. ICCV, 2019.

Q4: The accuracy of the teacher model and its comparison to the student model.

We would point out that in all experiments, "w/o KD" denotes the teacher's performance, "w/o Ours" and "w/ KD" denote the student's performance after online KD or self KD without our spacing effect, while "w/ Ours" denotes the student's performance after online KD or self KD with our spacing effect (referred to as the Spaced KD). This is because existing online KD and self KD methods often employ (two copies of) the same network as both teacher and student to improve generalization of the model itself. Therefore, the student's performance will exceed the teacher's performance if the online KD and self KD methods work well.

Q5: In Table 5, the authors should provide a comparison with other distillation methods.

In addition to online KD in Table 5, we add more experiments of self KD with different corruption types and network architectures. As shown in the following table, Spaced KD can also largely improve the robustness of self KD in generalizing to different noisy scenarios.

Q6 & Q7: Include citations and comparisons to more recent KD methods.

Following your suggestion, we have included more recent KD methods, especially those for 2023-2024. For example, TSB [1] constructs superior "teachers" with temporal accumulator and spatial integrator. CTKD [2] controls the task difficulty level during the student’s learning career through a dynamic and learnable temperature. LSKD [3] employs a plug-and-play Z-score pre-process of logit standardization before applying softmax and KL divergence.

As shown in the following table, our Spaced KD can be combined with these methods and provide significant improvements under two datasets and two architectures.

[1] Online Knowledge Distillation by Temporal-Spatial Boosting. WACV, 2022. (requested by Reviewer C5Ts) [2] Curriculum Temperature for Knowledge Distillation. AAAI, 2023. [3] Logit Standardization in Knowledge Distillation. CVPR, 2024.

Thank you for your valuable comments. We provide a point-to-point response as follows. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.

Q1: Review, comparison, and compatibility experiment on latest KD methods.

Following your suggestion, we have included more recent KD methods. Here we discuss some representative ones: TSB [1] constructs superior "teachers" with temporal accumulator and spatial integrator. CTKD [2] controls the task difficulty level during the student’s learning career through a dynamic and learnable temperature. LSKD [3] employs a plug-and-play Z-score pre-process of logit standardization before applying softmax and KL divergence.

As shown in the following table, our Spaced KD can be combined with these methods and provide significant improvements under two datasets and two architectures.

[1] Online Knowledge Distillation by Temporal-Spatial Boosting. WACV, 2022. [2] Curriculum Temperature for Knowledge Distillation. AAAI, 2023. [3] Logit Standardization in Knowledge Distillation. CVPR, 2024.

Q2: Selection of intervals for self KD and online KD.

We theoretically demonstrate that there exists a desirable temporal interval to improve generalization of online KD (as well as self KD) and empirically investigate the selection of specific values (see online KD in Table 6, and self KD in the following table). We find that the desirable temporal interval for a given KD paradigm is relatively stable across different network architectures and benchmark datasets. Since the SGD-induced variability of online KD (targeting the entire network) is larger than that of self KD (targeting a few network blocks), the latter requires a larger temporal interval to aggregate such variability to obtain an appropriate teacher-student gap. We therefore select s=1.5 for online KD and s=4.0 for self KD as the default implementation.

Temporal interval of self KD:

Q3 & Q4: More discussion of ``Teacher-Student Gap'' in online KD and its failure analysis.

We agree that the capacity gap between teacher and student is critical to the online KD performance. In previous studies of (online) KD, the teacher-student gap is regulated in various dimensions, such as the training data, network architecture, innate randomness (random initialization and SGD), etc. In this work, our main motivation lies in improving generalization of the model itself with identical network architectures, random initialization and data sources between teacher and student. We therefore characterize the teacher-student gap in online KD into the interplay of the SGD-induced variability and the temporal interval that aggregates such variability.

Theoretically, we demonstrate that a proper temporal interval between teacher and student (i.e., a proper teacher-student gap) helps the model to find flatter local minima using SGD. This is because the teacher that is slightly ahead in training provides a well-defined trajectory, ensuring low errors along so-called informative direction to improve generalization. In contrast, the naive SGD only ensures low errors in random directions around the convergence point, which limits the "radius" of loss flatness (see Sec.4.2).

Empirically, we perform extensive experiments to validate our proposal in online KD across a range of datasets and architectures. Aligned with our theoretical analysis, the proposed Spaced KD is most effective only when the temporal interval is set to an appropriate value (Fig.2 and Table 6). When the temporal interval is too large, the spaced version of online KD is closer to offline KD and the improvement is compromised. This is because the too large teacher-student gap makes them converge to different local minima, which violates the assumption of our theoretical analysis.

In the absence of differences in other spatial elements that produce the teacher-student gap, the effectiveness of our Spaced KD is regulated by the temporal interval in SGD. We believe that the effectiveness of our Spaced KD will be affected if the teacher-student gap produced by such spatial elements is too large, which is out of scope of the current focus.

Thank you for your valuable comments. We provide a point-to-point response as follows. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.

Q1: ImageNet-1K results are less comprehensive in Table 7.

We would respectfully point out that the ImageNet-1K results are presented in both Table 7 and Table 8, including different KD paradigms (online KD and self KD) and network architectures (ResNet-18 and Deit-Tiny). Our Spaced KD provides improvements in all cases.

Q2: Evaluation of more recent KD methods with more scenarios.

As shown in the response to Reviewer bqdc's Q1, our proposed Spaced KD brings significant improvements to a range of more recent KD methods under two datasets and two architectures. We will add these results in the final version.

Q3: The proof (Appendix A.1) assumes over-parameterized models and linearized dynamics, which may not fully capture real-world DNN training.

We would respectfully argue that state-of-the-art DNNs are often over-parameterized to improve generalization. This property results in multiple local minima with similar training errors but different testing errors, often reflected in the flatness of loss landscape. We therefore adopt the over-parameterization assumption to analyze how to converge to a flatter loss landscape.

Also, we would point out that we assume local linearization around the convergence point, rather than global linearization. As shown in previous work [1], SGD eventually selects a loss minima with linear stability (i.e., low errors with moderate linear disturbance), therefore ensuring flat loss landscape and improving generalization. We will add more explanations to these two assumptions in the final version.

[1] The alignment property of SGD noise and how it helps select flat minima: A stability analysis. NeurIPS, 2022.

Q4: Some details (e.g., hyperparameters for transformer training) are missing.

For transformer training, we adopt the well-established training pipeline tailored for the benchmark datasets [1-3]. We will add more detailed descriptions in the final version.

[1] Efficient Training of Visual Transformers with Small Datasets, NeurIPS. 2021. [2] Locality Guidance for Improving Vision Transformers on Tiny Datasets. ECCV, 2022. [3] Logit Standardization in Knowledge Distillation. CVPR, 2024.

Q5: One related work [1] is not cited.

Thanks. We have conceptually analyzed this work and empirically demonstrated Spaced KD's plug-in benefit on it (see response to Reviewer bqdc's Q1).

Q6: The pseudo-code (Appendix A.10) lacks implementation details (e.g., gradient accumulation for interval s).

We would clarify that our proposed Spaced KD inherently avoids gradient accumulation through its dual-loop design. The hyperparameter s (i.e., temporal interval) operates as a temporal decoupler rather than a gradient accumulation window. With the outer loop, the teacher model updates its parameters immediately after each batch (Algorithm 2, line 5), with no gradient retention. With the inner loop (when $|\mathcal{R}=s$), the teacher’s parameters remain frozen while the student updates using the cached samples, ensuring no backward passes occur through the teacher in this phase. This design strictly segregates the gradient flows: the teacher’s gradients are computed and applied instantly in the outer loop, while the student’s gradients are confined to the inner loop without cross-interval persistence. We will add more explanations to make this clearer.

Q7: The "temporal interval" is more precise than "space interval".

Following your suggestion, we will modify our expression for better clarity.

Q8: Why is the optimal interval s = 1.5 epochs (Page 7)? Is this dataset-dependent? A theoretical justification is missing.

We theoretically demonstrate that there exists a desirable temporal interval to improve generalization of online KD (as well as self KD) and empirically investigate the selection of specific values (see online KD in Table 6, and self KD in the response to Reviewer bqdc's Q2). We find that the desirable temporal interval and its effectiveness is related to the strength of SGD-induced variability (target of KD paradigms, learning rate, batchsize, etc.), and is relatively insensitive to different network architectures and benchmark datasets.

Since the SGD-induced variability of online KD (targeting the entire network) is larger than that of self KD (targeting a few network blocks), the former requires a smaller temporal interval to aggregate such variability to obtain an appropriate teacher-student gap. We therefore select s=1.5 for online KD and s=4.0 for self KD as the default implementation. We further investigate the impact of learning rate and batch size in Fig.5, which together with the temporal interval affect the overall performance.