Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate

Thank you for acknowledging the novelty of our method and the comprehensiveness of our experiments.

W1: further evidences of the proposed hypothesis.

Thank you for your request for further evidence supporting our hypothesis.

We have provided additional experimental results to validate our hypothesis using ResNet-50 and ResNet-18 as both the teacher and student models on CIFAR-100, following the same methodology described in Section 3.1, but with different model architectures. The training and test KL-divergence of the sophisticated and deceptive students are shown in Figure 5 of the PDF in the General Responses. We observe that the sophisticated students achieve lower final KL losses compared to the deceptive students with fewer training epochs, which further supports our hypothesis.

Due to the time constraints of the rebuttal process, we have included additional results for two models on CIFAR-100. We promise to provide more extensive results in the final version of our paper.

Q1: The hypothesis on Section 2.1 have been proposed before? I better discussion on that will help the overall reading and to better position the paper when compared to other approaches.

Thank you for your insightful feedback. We agree that providing more context and discussion on the motivations and background of our hypothesis is crucial for better positioning our paper.

For in-depth discussions, please refer to [GR1] and [GR2].

The hypothesis presented in Section 2.1, in the context of human language acquisition and emergence, is a widely accepted concept in cognitive sciences and linguistics [17, 18, 19, 20]. In the AI community, similar hypotheses have been proposed concerning artificial language emergence [16]. However, to the best of our knowledge, we are the first to propose that this principle is widely applicable across different domains and can be used to deduce a practical regularizer.

We will incorporate these discussions to better position our paper and provide a clearer comparison with other approaches.

Q2: For all the experiments N (student steps ratio) was 1? If not the authors compared LoT with other methods for the same number of steps? If not the superior number of steps can partially be the cause of the performance gains.

Thank you for your question.

Yes, in all our experiments $N = 1$, ensuring that the total training steps (Teacher + Student) for LoT are the same as for the Teacher-only approach. This setup maintains an equal training steps budget for a fair comparison. Therefore, the performance gains observed are not due to additional training steps, as both LoT and Teacher-only utilize the same total number of steps.

Additionally, we present results in Table 7 of our paper where we further increased the training steps for both LoT and Teacher-only. These results show that further increasing the training steps does not significantly improve performance, which further underscores the effectiveness of LoT in achieving performance gains under the same training steps as the Teacher-only approach.

Q3: Regarding the choice of metrics the authors chose KL-divergence, did the authors experimented with other losses?

Thank you for your question regarding the choice of metrics.

We have experimented with different metrics for the “imitability” measurement, such as L2 loss. However, we found that using KL-divergence achieves better performance compared to L2 loss. The results of utilizing L2 loss for the LoT regularizer with ViT-B/16 and ViT-L/16 on CIFAR-100 are presented in Table 13 in the PDF. These results show that using L2 loss for the LoT regularizer also brings performance improvements, further indicating the effectiveness of LoT regularization.

Q4: LoT on ImageNet-R [1] and ImageNet-Sketch [2] will be interesting for a broader set of results.

Thank you for your suggestion.

We conducted additional experiments by fine-tuning models on ImageNet-1K and evaluating them on ImageNet-R and ImageNet-Sketch using ViT-B/16 and ViT-L/16 models to investigate the out-of-distribution robustness of LoT. The results, shown in Table 8 of the PDF, demonstrate that LoT also brings performance improvements to these datasets, indicating the robustness of LoT across a broader set of scenarios.

Q5: The current set of baselines can be expanded. Methods like [3] and [4] can provide a better understanding on how well LoT is positioned.

Thank you for your suggestion regarding expanding the set of baselines.

In our paper, we have already demonstrated that LoT achieves better performance than the distillation method BAN [23] (Table 4). To further validate LoT’s effectiveness, we conducted additional experiments using ResNet-50 and ViT-B/16 on CIFAR-100, comparing LoT to distillation methods such as BAN [23], DKD [21], and ReviewKD [22]. The teacher weights utilized in these methods were the best checkpoint for Teacher-only. The results, shown in Table 9 of the PDF, indicate that LoT outperforms these distillation baselines, underscoring the effectiveness of LoT's unique interactive learning process. Sure, here are the references in markdown format with each on a different row:

References:

[16] Ease-of-Teaching and Language Structure from Emergent Communication, ICLR 2019
[17] Iterated Learning: A Framework for the Emergence of Language, Artificial Life, 2003
[18] Iterated Learning and the Evolution of Language, Current Opinion in NeuroBiology, 2014
[19] Cumulative Cultural Evolution in the Laboratory: An Experimental Approach to the Origins of Structure in Human Language, PNAS, 2008
[20] Spontaneous Evolution of Linguistic Structure: An Iterated Learning Model of the Emergence of Regularity and Irregularity, IEEE Transactions on Evolutionary Computation, 2001
[21] Decoupled Knowledge Distillation, CVPR 2022
[22] Distilling Knowledge via Knowledge Review, CVPR 2021
[23] Born Again Neural Networks, ICML 2018

Thank you for recognizing the novelty of our regularization method and acknowledging that our experiments demonstrate the effectiveness and efficiency of LoT in identifying generalizable information. Below, we provide detailed responses.

W1: previous studies have shown that more complex data and correlations contain richer information, where models need to obtain better accuracy to learn, making them more helpful for generalization. These two views seem to be contradictory.

Thank you for pointing out the potential confusion. We appreciate the opportunity to clarify our hypothesis. Our hypothesis may be better expressed as follows: "Given a dataset D with rich enough information, if there are two learned correlations A and B that both perfectly explain D, but A is easier to imitate than B, then A is more likely to be generalizable than B."

We agree that complex data containing richer information can indeed aid generalization. However, it's important to note that the complexity of the dataset and the simplicity of the learned correlations are independent factors. To clarify:

1. Why does high complexity in the dataset help? Real-world cases are inherently complex, so the dataset should be complex enough to capture this complexity. Otherwise, models might rely on shortcuts that don't work in real scenarios. For instance, memorizing the results of a simple addition rule, like c = a + b when a and b are within ten, is easier than understanding the rule of addition itself.

2. Why are generalizable correlations easier to imitate given a complex dataset? Intelligence involves finding simple, teachable rules and correlations within complex real-world data. Learning the rules of addition is simpler than memorizing 1 million addition results. Please refer to [GR2] for further explanation.

In other words, AI can only emerge when an artificial learner effectively understands simple correlations, rules, and concepts of low Kolmogorov complexity that can successfully explain complex datasets. This idea, widely accepted by the deep learning community [14], can be seen as an implementation of Occam’s Razor, a principle dating back to the 14th century.

W1-2: the author constructed their hypothesis based on human cognition, but the relevant theoretical basis is lacking, which makes its reliability questionable. This distillation is similar to meta-learning, but its support for generalization is confused. As the author said, "both the teacher and student models may lack or possess different task-specific knowledge", it is difficult to understand why it improves generalization. I think it seems to be "joint learning", but this learning mechanism and optimization details are missing. I hope the author can further provide LOT's insight.

Thank you for your insightful feedback. For a comprehensive review of related ideas and theories in psychology and evolutionary linguistics, please refer to [GR1] and the references therein. Additionally, [GR2] provides further insights into the concept of Learning from Teaching (LoT).

It is worth noting that our method is orthogonal to the meta-learning paradigm, which involves a random distribution of tasks in the meta-learning setup. The learning paradigm most closely related to our method is iterated learning, where “ease-of-teaching” is widely accepted as an important concept [15]. Our contribution can be viewed as a generalization of the iterated learning idea to broader machine learning tasks.

Furthermore, please refer to [GR3] for a discussion on the challenges of developing comprehensive mathematical theories in this context.

W3: The description of the training process in this article is a bit hard to follow. The author mentioned that generalization is improved through "joint learning", but in Section 2.2, the teacher-learner and student-learner are optimized alternately. In the introduction, the author mentioned using the student learner as an auxiliary task to improve generalization, but it is not clear how to capture the generalization connection, what this generalization connection is, and how to guide the model learning. More details may be better.

Thank you for your feedback. We apologize for any confusion regarding the description of the training process.

To clarify, in LoT, the teacher and student are optimized alternately. The student learns from the teacher (Eq. 2), and the teacher is optimized based on both the regular task loss and the LoT regularizer, which is the feedback from the student (Eq. 1). The detailed procedure of LoT is outlined in Algorithm 1 in the paper.

The generalization connection is embodied in the LoT regularizer. According to our hypothesis, a smaller imitation loss indicates a more generalizable teacher model. By incorporating the LoT regularizer as an additional loss to the teacher, the optimization process encourages the teacher to achieve a smaller imitation loss. Consequently, the generalization of the teacher model is enhanced compared to models that do not employ LoT.

References:

[14] Ilya Sutskever, An Observation on Generalization, Simons Institute, 2023
[15] Ease-of-Teaching and Language Structure from Emergent Communication, ICLR 2019

We appreciate your recognition of the novelty and significant improvements brought by LoT, as well as the clearly articulated presentation. Below, we provide detailed responses.

W1: It will be the best to Include more comparisons with other regularization methods, or other recent advances in knowledge distillation.

As detailed in Appendix D, LoT is orthogonal to existing regularization methods such as weight decay, dropout, batch normalization, and layer normalization. For instance, in Table 1, we applied dropout to Transformer-XL models following [9]. Additionally, in Table 3, the teacher and student models are equipped with regularizations other than LoT, specifically: ViT (layer normalization, dropout, weight decay) as per the setups in [10]. LoT demonstrates effectiveness across all these settings, thereby adding value to the existing pool of regularization methods.

Regarding knowledge distillation methods, we have shown that LoT outperforms the distillation method BAN [13] (Table 4). To provide stronger validation of LoT’s effectiveness, we conducted additional experiments using ResNet-50 and ViT-B/16 on CIFAR-100. We compared LoT to distillation methods such as BAN [13], DKD [11], and ReviewKD [12], with the teacher weights in these methods being the best checkpoint of Teacher-only. The results, shown in Table 9 of the PDF in GR, indicate that LoT achieves better performance than these distillation baselines, further underscoring the effectiveness of the unique interactive learning process of LoT.

W2: a more thorough analysis of the scalability and computational cost would be beneficial.

We have provided a detailed comparison of the computational budget for LoT and Teacher-only in Table 12 in the PDF. Our analysis shows that LoT uses the same number of CPU cores as Teacher-only, with GPU usage being 12% to 55% higher. Despite this, LoT exhibits lower training times compared to Teacher-only (except in RL tasks) when subjected to the same total training epochs/steps, while still achieving significant performance improvements.

W3: Exploring the impact of different types of student models could provide deeper insights into the robustness and versatility of LOT.

We appreciate your suggestion. In fact, our findings demonstrate that LoT remains effective across a wide range of student model architectures and capacities, as detailed in Section 3.4 and Table 3 of our paper. Key observations include:

• Enhanced Teacher Performance with Weaker Students: For instance, a MobileNetV2 student model significantly boosts the performance of stronger models like ResNet-18 and ResNet-50 by over 1% on CIFAR-100, despite its smaller size and lower capacity.
• Improved Generalization with Stronger Students: Incorporating a ResNet-50 student with a ResNet-18 teacher in LoT enhances the teacher's performance by 1.99%.
• Benefits of Diverse Architectures in Transformer-Based Models: For transformer-based models such as ViTs and Swins, utilizing different architectures for the teacher and student yields better performance than using the same architecture. For example, with a ViT-B/16 student, a ViT-L/16 teacher achieves 0.27% higher accuracy compared to a ViT-L/16 student.

W4: the theoretical foundations behind why generalizable correlations are easier to imitate could be elaborated further.

To provide a more comprehensive understanding, we refer you to the theoretical foundations of LoT as discussed in the fields of cognitive sciences and evolutionary linguistics in reference [GR1]. For a detailed mathematical theoretical framework, please refer to [GR3].

Q1: what is the performance of those NLP model or CV model on the validation set of trained datasets?

In our experiments, we did not employ a separate validation dataset from the training dataset. We found that LoT performs effectively without the need for hyperparameter tuning on a validation set. This is particularly advantageous given the substantial resource demands associated with hyperparameter tuning on large datasets. Consequently, we believe the test performance remains reliable even without a separate validation set.

To address your concern, we provided additional results on the official validation datasets for PTB and WikiText-103 in Table 10 of the PDF. These results demonstrate that LoT consistently outperforms the Teacher-only approach on both the validation and test datasets for PTB and WikiText-103, further validating the effectiveness of LoT.

Q2: my understanding is that the LOT only train half number of epochs comparing to the teacher-only approach?

Yes, your understanding is correct. To maintain an equivalent total number of training steps (teacher + student) as the Teacher-only setting, LoT is trained for half the number of epochs or steps compared to the Teacher-only approach (with the exception of RL tasks). Our hypothesis, supported by results presented in Figure 1 and Figure 3 of our paper, as well as Figure 5 and Table 12 in the PDF, demonstrates that under the same total training steps, LoT converges faster and achieves better final performance compared to the Teacher-only approach.

Q3: what is performance for those students model?

We would like to emphasize that LoT is designed primarily to enhance the generalization capabilities of the teacher model. However, we also provided results for the student models in our experiments, as shown in Table 11 of the PDF. Our observations indicate that when the student and teacher share the same architecture, the student models can achieve performance comparable to that of the teacher models.

References:

[9] Transformer-XL: Attentive language models beyond a fixed-length context. ACL 2019

[10] An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2020

[11] Decoupled Knowledge Distillation. CVPR 2022

[12] Distilling Knowledge via Knowledge Review. CVPR 2021

[13] Born again neural networks. ICML 2018