Right Time to Learn: Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation

Thank you for your valuable comments. Below, we provide a point-to-point response to each of the weaknesses (W) and questions (Q), and summarize the corresponding revisions in the final version. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.

W1: New fundamental mechanism within the KD framework.

As recognized in your Strength 1, the spacing effect in KD is ``relatively unexplored in existing literature''. In fact, our work reveals an innovative mechanism that the spacing effect in KD improves generalization of the student model via encouraging the convergence to a flat loss landscape. The innovation is reflected in two aspects:

(1) Unlike previous understandings [1,2] that attribute the efficacy of KD to the knowledge capacity gap between the teacher and the student (where a spacing-like effect should be more effective in the early stage of training), our Spaced KD works in the later stage of training (see Fig.3) around the convergence point (see Sec.4.2). In Appendix A.5, we devise a naive baseline that keeps constant $s$ training steps between the teacher model and the student model, which exhibits no significant improvement over online KD. This suggests that our Spaced KD mainly affects the convergence process of the student model by providing informative directions from the stable teacher model after training $s$ steps.

(2) As shown in a pioneering theoretical analysis [3], KD shares a similar mechanism as ensemble learning in improving generalization. However, current efforts have focused on better ensembling separately trained teacher and student models from a spatial scale. Although online KD and self KD attempt to ensemble jointly trained teacher and student models from a temporal scale, the effect of temporal interval remains under-explored. Our work demonstrates that the implementation of the spacing effect in KD is highly non-trivial (see the first aspect above), and a specially designed temporal interval provides great improvements to KD, applicable to various architectures and datasets.

[1] Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks, TPAMI, 2022.

[2] Knowledge Distillation: A Survey, IJCV, 2021.

[3] Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. ICLR (Outstanding Paper Nomination), 2023.

W2: Interval sensitivity and adaptability.

As shown in Fig.2, $s$ is relatively stable and a range of $s$ can provide significant improvements, with $s=1.5$ having comparably the strongest performance. Therefore, we set $s=1.5$ for all experiments if not specified, making it an easy-to-use strategy in applications. We agree that the adaptive selection of $s$ is a promising future work, and we have discussed this in our original submission (see Sec.6).

W3 & Q1: Applicability across model sizes and architectures.

Following your suggestion, we conduct a series of experiments with large-size teachers and small-size students, using similar or different architectures. As shown in the following tables, our Spaced KD clearly outperforms the online KD across model sizes and architectures. We have added more explanations (lines 362-364) and empirical results (Tab.7 of Appendix A.4) in our revised manuscript.

Results of large-size teacher with small-size student:

Results of transformer teacher with ResNet student:

Q2: Difference between Spaced KD and traditional gradient-based methods.

As shown in our theoretical analysis in Sec.4.2, our Spaced KD serves as a well-designed regularization to improve traditional gradient-based methods such as SGD. In particular, unlike the stochastic directions of gradient descent in SGD, our Spaced KD encourages the student model to converge to a more generalizable loss landscape by providing stable informative directions from the teacher model.

Dear Reviewer v8iU,

If you have any further feedback or concerns, please feel free to let us know. Your input is highly appreciated, and we look forward to hearing from you. Thank you once again for your time and consideration.

Best wishes,

Authors

Dear Authors,

Thank you once again for your efforts in providing additional details, explanations, and experiments to enhance your paper. I have carefully reviewed the comments from the other reviewers as well.

Taking into account all the reviews, feedback, and your thorough rebuttal, I find this work to be both interesting and beneficial to the community. While some of the improvements may not be highly significant, I still believe the contribution is valuable and worthy of recognition. Accordingly, I would like to raise my score to 8.

Thank you for your valuable comments. Below, we provide a point-to-point response to each of the weaknesses (W), questions (Q), and limitations (L), and summarize the corresponding revisions. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.

Q1: Connection between the proposed Spaced KD and its bio-inspiration.

Our proposed Spaced KD for machine learning is inspired by the underlying computational principle of the spacing effect in biological learning:

In machine learning, KD aims to optimize the parameters of a student network with the help of a teacher network by regularizing their outputs to be consistent in response to similar inputs. As shown in a pioneering theoretical analysis [1], KD shares a similar mechanism as ensemble learning (EL) in improving generalization from the training set to the test set. In particular, online KD performs this mechanism at temporal scales, and self KD can be seen as a special case of online KD. In comparison, the biological spacing effect can also be generalized to a kind of EL at temporal scales, as the brain network processes similar inputs with a certain time interval and updates its synaptic weights based on previous synaptic weights, which allows for stronger learning performance at test time [2-4].

The proposed Spaced KD draws inspiration from the biological spacing effect and capitalizes on the underlying connections between (online) KD and EL. It incorporates a space interval between the student network and the teacher network to improve the performance of generalization, validated through extensive empirical and theoretical analyses. We have added more explanations to make it clearer (see Sec.3.2, lines 193-207).

[1] Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. ICLR (Outstanding Paper Nomination), 2023.

[2] The Phosphatase SHP2 Regulates the Spacing Effect for Long-Term Memory Induction. Cell, 2009.

[3] It’s All About Timing. Cell, 2009.

[4] The Right Time to Learn: Mechanisms and Optimization of Spaced Learning. Nature Reviews Neuroscience, 2016.

Q2: Experiments where the teacher's architecture is significantly larger than the student's.

Following your suggestion, we conduct a series of experiments where the teacher's architecture is significantly larger than the student's, including larger width (from ResNet-18 * 1 to ResNet-18 * 8) and larger depth (from ResNet-18 * 1 to ResNet-101* 1). As shown in the following table, the proposed Spaced KD consistently outperforms the online KD, validating its effectiveness and generality. We have added it in our revised manuscript (see lines 362-364 and Tab.7 in Appendix A.4)

Performance of online KD and Spaced KD on CIFAR-100:

Q3: Experimental setting in Fig.3. How these optimal settings were selected?

We use CIFAR-100 and $s=1.5$ for all experiments in Fig.3. This is because, in our empirical investigation, we have validated that $s$ is relatively stable and a range of $s$ can provide significant improvements (see Fig.2), with $s=1.5$ having comparably the strongest performance. Therefore, for ease of implementation and fairness of comparison, we set $s=1.5$ in all experiments if not otherwise specified (see lines 411-413). This property reduces the effort of hyperparameter tuning and further enhances the applicability of our Spaced KD.

Q4a: Consistency between the theoretical analysis and the empirical results in Fig.3.

We respectfully argue that the theoretical analysis and the empirical results are consistent. Our theoretical analysis assumes that the network parameters $\theta$ are close to their convergence point $\theta^*$, rather than far away (see lines 265 and Theorem 4.4). The empirical results in Fig.3 highlight that the proposed spacing effect exerts its greatest influence during the later stage of training which is close to the convergence point.

Q4b: Online KD baseline in Fig.3.

We have added it in Fig.3, where our Spaced KD provides great improvements on it.

Dear Reviewer bFLi,

If you have any further feedback or concerns, please feel free to let us know. Your input is highly appreciated, and we look forward to hearing from you. Thank you once again for your time and consideration.

Best wishes,

Authors

W3: Comparison with other self KD methods.

As an easy-to-use and compatible strategy, the proposed Spaced KD can be combined with other self KD methods, such as DLB [5] and PSKD [6]. We reproduce all results with their official implementations and author-provided codes. As shown in the below table, our Spaced KD brings significant improvements to both self KD methods. We have added it to our revised manuscript (see lines 374-375 and Tab.4).

Performance of self KD methods on ResNet-18 and CIFAR-100:

[5] Self-Distillation from the Last Mini-Batch for Consistency Regularization. CVPR, 2022.

[6] Self-Knowledge Distillation with Progressive Refinement of Targets. ICCV, 2021.

Q1: What are the numerical values of x and y in Fig.2a?

In Fig.2a, $x$ denotes inter-stimulus intervals between training sessions, and $y$ denotes the strength of long-lived memory trace. We have modified Fig.2a with numerical values of $x$ and $y$.

Q2 : As 0.5 epochs mean training on half of the dataset, would this cause a bias towards that half of the dataset? Is the training dataset shuffled?

As you understand it, 0.5 epochs mean training on half of the dataset. However, this would not cause a bias towards that half of the dataset. First, the training dataset is shuffled for each training epoch. Second, our specific implementation ensures that the teacher and the student receive the same data flow. For example, if $s = 0.5$, the teacher is trained on the first half of the dataset, and then the student is trained on the same first half of the dataset. After that, the teacher is trained on the second half of the dataset, and then the student is trained on the same second half of the dataset. But notice that the dataset is shuffled every time a new epoch starts, the specific images in half the dataset will also change with training iterations. Third, our proposed Spaced KD provides great improvements over a range of space intervals (i.e., 0.5, 1.0, 1.5, 2.0 epochs, see Fig.2b), further ruling out the possibility of dataset bias. We have added more explanations in our revised manuscript (see lines 220, 342-343).

Q3: How can the proposed method be applied to self-distillation in [7], where the teacher is the deepest layer of models and the students are the shallow layers?

In fact, the default implementation of ``self KD'' in our work is essentially identical to [7], i.e., the deepest layer (as the teacher) transfers knowledge to the shallow layers (as the student). To make this clearer, we have added more explanation (lines 346-350) and a pseudo code to Appendix A.9.

[7] Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. ICCV, 2019.

Q4: What dataset was used for Fig.3, and which settings were used for the experiments?

We use CIFAR-100 in Fig.3 and $s=1.5$. In these experiments, we empirically investigate the impact of different initiating times of Spaced KD, which identifies a critical period of Spaced KD that is more effective at the later stage rather than the early stage of training. We have added more explanations to Fig.3.

Thank you for your valuable comments. Below, we provide a point-to-point response to each weaknesses comment (W), and summarize the corresponding revisions. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.

W1 : Connection between the proposed Spaced KD and the spacing effect in biological learning. Why does training for several additional epochs cause a spacing effect?

Our proposed Spaced KD for machine learning is inspired by the underlying computational principle of the spacing effect in biological learning:

In machine learning, KD aims to optimize the parameters of a student network with the help of a teacher network by regularizing their outputs to be consistent in response to similar inputs. A pioneering theoretical analysis [1] shows that KD shares a similar mechanism with ensemble learning (EL) in improving generalization from the training set to the test set. In particular, online KD performs this mechanism at temporal scales, and self KD can be seen as a special case of online KD. In comparison, the biological spacing effect can also be generalized to a kind of EL at temporal scales, as the brain network processes similar inputs with a certain time interval and updates its synaptic weights based on previous synaptic weights, which allows for stronger learning performance at test time [2-4].

The proposed Spaced KD draws inspiration from the biological spacing effect and capitalizes on the underlying connections between (online) KD and EL. It incorporates a space interval between the student network and the teacher network to improve the performance of generalization, validated through extensive empirical and theoretical analyses. We have added more explanations to make it clearer (see Sec.3.2, lines 193-207).

[1] Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. ICLR (Outstanding Paper Nomination), 2023.

[2] The Phosphatase SHP2 Regulates the Spacing Effect for Long-Term Memory Induction. Cell, 2009.

[3] It’s All About Timing. Cell, 2009.

[4] The Right Time to Learn: Mechanisms and Optimization of Spaced Learning. Nature Reviews Neuroscience, 2016.

W2 : Theoretical analysis of Spaced KD and its connection to offline KD.

Here we provide more explanations on our theoretical analysis of Spaced KD, as well as its connection to offline KD in ideal and practical conditions:

The basic idea of our theoretical analysis lies in the discussion around the local optimum $\theta^*$ of the student network $f$, where the teacher network $g(\cdot, \phi)$ should also converge to $\phi^*$ although trained beforehand for a certain number of steps. With an ideal condition where $g$ and $f$ converge to the same local minima, offline KD and Spaced KD should perform identically best. However, this ideal condition hardly exists in practice, especially given the nature of over-parameterization in representative deep neural networks and the complexity of real-world data distributions. These two challenges result in a highly non-convex loss landscape of both $g$ and $f$ with a large number of local minima. Therefore, using a well-trained teacher in offline KD tends to be sub-optimal since $g$ and $f$ can easily converge to different local minima with SGD. In contrast, our Spaced KD introduces an appropriate space interval, which makes $g$ and $f$ somewhat distant but not too far apart. This design enjoys the theoretical benefits in terms of generalization while remaining effective in practice.

To validate the above explanations, we present empirical results of offline KD and Spaced KD in the following table. It can be seen that our Spaced KD also slightly outperforms offline KD, and the performance lead tends to be more significant when using a wider network, suggesting the impact of over-parameterization on the mismatched assumption. Besides, the results of the critical period (see Fig.3) suggest that the benefit of Spaced KD is highly relevant to the later training stage (i.e., close to the convergence point), consistent with our theoretical analysis around $\theta^*$.

Our theoretical and empirical analyses collectively demonstrate the particular role of Spaced KD in improving the generalization capability of the obtained solution. We have added these explanations in our revised manuscript (see lines 309-319 and Tab.8 in Appendix A.4).

Thanks authors for their reply.

W1: I still cannot see the connection between the proposed method (a learning process) and the biological spacing effect (which reduces forgetting of information). As mentioned in the authors' reply, "the brain network processes similar inputs with a certain time interval and updates its synaptic weights based on previous synaptic weights." How is this related to the proposed KD methods where teacher and student have different weights?

W2: The loss functions for the teacher and student are different, which means they have different loss landscapes. How do they share the same local minima even in the ideal condition? Authors updated Section 4.2 (Line 309-311) and stated, "Spaced KD guides the student f with a well-defined trajectory established by the teacher g that is slightly ahead in training" I would like to highlight some related works [4] [5].

W3: In PSKD, the teacher is one epoch before. Did the authors mean they used 1.5 epochs ahead when combining their method with PSKD? I cannot match the results in Tables 1, 2, and 4. For example, the results for ResNet18 on CIFAR-10 without KD are the same in Tables 1 and 2 but differ from the result for PSKD without KD in Table 4.

[4] Follow Your Path: a Progressive Method for Knowledge Distillation.

[5] Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher

Dear Reviewer 5J6i,

If you have any further feedback or concerns, please feel free to let us know. Your input is highly appreciated, and we look forward to hearing from you. Thank you once again for your time and consideration.

Best wishes,

Authors

We appreciate your insightful comments and provide a point-to-point response below.

W1: Biological Connection

In general, spaced learning is a paradigm of the biological learning process, which improves the learning performance from the training condition to the similar testing condition, similar to the generalization from the training dataset to the test dataset in ANN, rather than directly handling the memory forgetting issue (as we described in the Sec.1).

In line 204, we describe the biological learning process as "updates the synaptic weights based on previous synaptic weights" to set the stage for the discussion with ensemble learning from a temporal perspective, where the learner evolves its capability on the basis of its earlier state.

Throughout the manuscript, we mainly focus on the online condition where the teacher and student have the same parameter space and initialization. In addition, the proposed KD is a regularization term for updating the student network by the teacher's output instead of constraining the parameters directly. So, it still works when extending to the KD methods where the teacher and student have different weights.

W2: Theoretical Analysis

As you correctly pointed out, the teacher and student models indeed have different loss landscapes, which raises the concern about how they can share the same minima. The "ideal condition" mentioned in lines 309-311 refers to the situation where the student model, initialized with the same parameters as the teacher and following the teacher's training trajectory, is able to converge to a well-trained minima akin to that achieved by offline KD. In this idealized case, the student can fully track the teacher’s learning path, allowing it to reach a similar final minima.

However, as we discuss further after line 312, this ideal condition is rare and often difficult to achieve in practice, especially in dynamic training scenarios. Therefore, we introduce spaced intervals into the distillation process, which provides the student model with sufficient room to explore multiple convergence directions. This flexibility allows the student to find a better solution space, enhancing the distillation process and leading to performance improvements that often exceed traditional methods. The reason we introduce the "ideal condition" here is to set the stage for the subsequent comparison between the performance of offline KD and spaced KD. We will further refine this section of the explanation in the final draft.

In line with the progressive learning paradigm, both ProKT [4] and Pro-KD [5] propose a similar learning framework to online KD, where the student model learns from an actively updating teacher during training. In ProKT, the teacher also receives a reverse distillation loss from the student, a concept we briefly mention in lines 158-160. We will cite these valuable works in the final draft to support our discussion.

[4] Follow Your Path: a Progressive Method for Knowledge Distillation.

[5] Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher

W3: Comparision to other KD methods

As you kindly pointed out, the soft label from teachers in PSKD and DLB is one epoch earlier and one batch iteration earlier, respectively, where the teacher-student gap is pre-defined. Thus, we initiate a student network identical to the teacher, then we train the teacher model utilizing PSKD or DLB, and the student model is trained either online or in a spaced style with an interval of 1.5 epochs. Specifically, the results w/o KD of PSKD and DLB in Table 4 are the performance of the teacher model, w/ KD is the performance of online students, and w/ Ours corresponds to spaced students. It is worth mentioning that, we follow the exact training pipeline (including learning rate scheduler, optimizer, and dataset transformation, etc) of corresponding works when reproducing their results and incorporating our proposed Spaced KD framework, which is different from that of Table 1 and 2, and that explains the difference in numbers. We will add more details about the implementation of the comparative methods in our revised manuscript.

Thank you again for your constructive feedback. We hope this response adequately addresses your concerns. Please feel free to reach out if you have any further questions or require additional clarification.

Thank you for your further reply.

Thank you for your valuable comments. Below, we provide a point-to-point response to each of the weaknesses (W) and questions (Q), and summarize the corresponding revisions. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.

W1: Implementation of online KD, self KD and its spaced version.

In standard online KD, the teacher model transfers knowledge to the student model at each training time step. In our spaced version, we first train the teacher model for $s$ steps ahead, and then train the student model for $s$ steps. This two-step training process alternates between the two models. For example, if $s = 1.5$ epochs, the teacher model is updated using cross-entropy loss for 1.5 epoch, and then the student model is updated using the KD loss for 1.5 epoch.

In self KD, the deepest layer (as the teacher) transfers knowledge to the shallow layers (as the student) at each training time step [1]. In our spaced version, we first train the model using the cross-entropy loss between the deepest layer's output and the ground-truth label for $s$ steps. We then train the model using the standard self KD loss [1] between the deepest layer's output and each shallow layer's output for $s$ steps.

We have added more explanations to make it clearer (see lines 178-186, lines 214-215, lines 346-350). We further provide a pseudo-code of online KD, self KD, and its spaced version in Appendix A.9.

[1] Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. ICCV, 2019.

W2: Advanced architectures and large-scale datasets.

In our original submission, we mainly focus on online KD, evaluating it and its spaced version across a variety of architectures (ResNet and Transformer) and datasets (CIFAR-100, Tiny-ImageNet and ImageNet-100/1k). As analyzed in Sec.4.2, we treat self KD as a special case of online KD and perform validation experiments using ResNet-based architectures on CIFAR-100, Tiny-ImageNet and ImageNet-100. Following your suggestion, we evaluate self KD and its spaced version with ViT and ImageNet-1k.

Results of self KD using transformer-based architectures:

Results of self KD using ResNet-18 on ImageNet:

Due to the limited time to prepare the rebuttal, the experiments of self KD using ViT on ImageNet-1k are still under training. Based on the current training curve, our proposal is expected to largely improve self KD in this large-scale setting. We hope to finish it before the discussion period and will add it to our revised manuscript.

W3 & Q1: Explanation of Appendix A.5.

In Appendix A.5 we consider a naive baseline of implementing the proposed spacing effect. Specifically, we train the teacher model for $s$ steps before transferring knowledge to the student model at each step during the following training time. In other words, the teacher model keeps constant $s$ steps ahead of the student model. However, such a naive baseline exhibits no significant improvement over online KD, consistent with our empirical analysis (see Fig.3) and theoretical analysis (see Sec.4.2): The teacher model of Spaced KD can provide a stable informative direction for optimizing the student model after each $s$ steps, whereas the teacher model of the naive baseline fails in this purpose due to its ongoing changes when optimizing the student model. Such different effects also suggest that the implementation of the spacing effect is highly non-trivial and requires specialized design as in our work. We have added more explanations to Appendix A.5 to make it clearer.

W2 (part 2): Advanced architectures and large-scale datasets.

We have finished experiments on self KD and online KD using ViT on ImageNet-1k, and the results are summarized in the table below. These experiments further validate the effectiveness of our proposed method in improving performance on large-scale datasets over standard online KD and self-KD.

These results, which we have finalized since the initial submission, demonstrate significant improvements in performance across the training epochs when applying our method. We believe these additional experiments further strengthen the validity of our proposed approach for large-scale datasets like ImageNet-1k.

If you have any further questions or concerns, please feel free to let us know at any time.

Dear Reviewer Ac8Y,

If you have any further feedback or concerns, please feel free to let us know. Your input is highly appreciated, and we look forward to hearing from you. Thank you once again for your time and consideration.

Best wishes,

Authors

Dear reviewers,

We sincerely appreciate your great efforts in reviewing our manuscript and all your valuable feedback. We are pleased to see many positive comments, and have tried our best to address the remaining concerns. We know and understand that the reviewers are very busy, and sincerely hope you could take a little time to review our responses, while let us know whether you have further questions. We are willing to try our best to answer these questions. Great thanks!

Best wishes,

Authors