Meta-Collaboration in Distillation: Pooled Learning from Multiple Students

We express our heartfelt gratitude to the reviewer for their dedicated time and thorough evaluation of our paper. The valuable insights they offered are highly appreciated, and we are grateful for the constructive feedback. We are devoted to addressing the reviewer's concerns and integrating their suggestions into the final version, with the goal of elevating the overall quality of our paper. In the following section, we present comprehensive responses to the reviewer's comments, queries, and recommendations, aiming to enhance the paper's overall rating.

Lack of novelty

Thank you for your insightful comments and for highlighting this clarity concern. We want to address a potential misunderstanding. This paper does not involve multi-teacher distillation; rather, it focuses on a single pre-trained teacher model, with all other models being student models that are concurrently undergoing training.

A significant contribution of this paper is the enhancement of knowledge distillation efficacy through the implementation of instance-wise weights, particularly in scenarios where a substantial learning capacity gap exists between the teacher model and the student model. In the conventional knowledge distillation framework, the final loss is a combination of cross-entropy loss and KL-divergence-based KD loss. Our objective is to learn weights for each of these components in an instance-wise manner. This approach allows us to concentrate on hard labels for specific data points and soft labels from the teacher for another set of points, thereby achieving an optimal trade-off between learning from the teacher model and the hard labels.

Additionally, beyond learning instance-wise and loss component-wise weights, this work aims to capitalize on the learning patterns of student cohorts. This is realized through the utilization of C-Net.

• meta-optimization is very difficult to implement and not very effective* Meta-learning is often used in setting to improve a model's generalization and denoise setting. We point to a set of relevant literature in the Related work section.

Lack of thorough evaluation

We utilized teacher-student pairs from other papers addressing significant capacity differences between teacher and student models. While we were unable to complete the additional ImageNet experiments within the review period, we expect our method would enjoy significant gains over tranditional KD even on large scale datasets.

Lack of discussion on relevant studies

Thanks for pointing out the additional literature. However, again there seems to be some misunderstanding. The primary objective of this paper is to enhance the effectiveness of knowledge distillation through the utilization of instance-wise weights. In scenarios characterized by a substantial learning capacity disparity between the teacher model and the student model, knowledge distillation efficacy is frequently suboptimal and, at times, inferior to learning with hard labels. The papers referenced seem to improve distillation by introducing additional information or modifying the student model itself. Nevertheless, we will conduct a thorough comparison and contrast of our work with these referenced papers.

We express our sincere gratitude to the reviewer for investing their time and energy in reviewing our paper. The invaluable and insightful feedback they offered is highly appreciated, and we are grateful for the constructive comments. We are dedicated to addressing the reviewer's concerns and integrating their suggestions into the final version, with the goal of improving the overall quality of our paper. In the following section, we offer comprehensive responses to the reviewer's comments, questions, and suggestions, with the aim of enhancing the paper's evaluation.

We present comparison with the additional baselines requested. We found “[C] Bidirectional Distillation for Top-K Recommender System, WWW'21 and on consensus learning with heterogeneous models.” to be specific to recommender systems and therefore compare against [1], another Bidirectional knowledge distillation method with is similar to our setup.

[1] Li, Lujun, and Zhe Jin. "Shadow knowledge distillation: Bridging offline and online knowledge transfer." Advances in Neural Information Processing Systems 35 (2022): 635-649.

In Eq.7, in my opinion, the optimization of \phi should be the outer loop, since Eq.7 is the optimization for C-Net.

Due to the bi-level optimization formulation, the C-Net parameters undergo updates based on the most recent model parameters. Initially, we adjust the model (student) parameters using loss gradients derived from the weighting parameters acquired from C-Net. Consequently, the updated parameters depend on the weights obtained from C-Net. Subsequently, employing these updated parameter values, we calculate loss gradients for C-Net.

We express our genuine gratitude to the reviewer for investing their time and energy in reviewing our paper. The valuable feedback offered is highly appreciated, and we are grateful for the constructive comments. We are dedicated to addressing the concerns raised by the reviewer and integrating their suggestions into the final version to improve the overall quality of our paper. In the following section, we offer thorough responses to the reviewer's comments, questions, and suggestions, aiming to elevate the rating of our paper.

The novelty is limited in my view. This work follows the widely used online distillation framework, except that the proposed C-NET part is trained with meta-learning. There are no clear improvements to the framework.

A primary contribution of this paper lies in enhancing the effectiveness of knowledge distillation by incorporating instance-wise weights in situations characterized by a significant disparity in learning capacity between the teacher and student models. In the typical framework of any knowledge distillation, the final loss is a combination of cross-entropy loss and KL-divergence-based KD loss. We aim to learn weights for each of these components on an instance-wise basis. This approach allows us to emphasize hard labels for specific data points and soft labels from the teacher for another set of points, facilitating an optimal trade-off between learning from the teacher model and the hard labels. This trade-off is crucial as diverse students exhibit varying learning capacities, and when a student model is considerably smaller than the teacher model, some learning from hard labels proves beneficial.

Beyond learning instance-wise and loss component-wise weights, our work seeks to leverage insights from student cohorts, inspired by online distillation. However, unlike traditional online distillation, there is no explicit matching imposed among the student models through a loss function. Instead, this information is implicitly shared among the student models through the utilization of a common C-Net.

Can you provide further theoretical analysis or insights into how the meta-collaboration process influences the learning patterns of the student models?

The effect on learning patterns of different student models while training with weights from C-Net like neural network is well studied in [1]. The same setting is applicable here as well.

[1]Shu, Jun, et al. "Meta-weight-net: Learning an explicit mapping for sample weighting." Advances in neural information processing systems 32 (2019).

We extend our sincere appreciation for the reviewer's dedicated time and effort in evaluating our paper. The insightful feedback provided is invaluable to us, and we are thankful for the constructive comments. We are committed to addressing the reviewer's concerns and incorporating their suggestions in the final version, aiming to enhance the overall quality of our paper. Below, we provide detailed responses to the reviewer's comments, questions, and suggestions with the hope of improving the rating of our paper.

- The novelty looks incremental. It looks to me that the paper combines offline KD and online KD by training multiple students simultaneously while learning the loss weights with the meta-learning strategy (bi-level optimization). 
- It misses some ablation studies to show how the learned weights by C-NET are better than other weight strategies, e.g., manually set.

Thank you for your insightful comments and for bringing attention to the clarity issue. We wish to address a potential misunderstanding. A key contribution of this paper is the enhancement of knowledge distillation efficacy through the incorporation of instance-wise weights. In scenarios where there exists a significant learning capacity gap between the teacher model and the student model, knowledge distillation efficacy is often suboptimal, and in some cases, inferior to learning with hard labels.

To alleviate this issue, our proposal involves the utilization of instance-wise and loss component-wise weights. Typically, in any knowledge distillation framework, the final loss is a combination of cross-entropy loss and KL-divergence based knowledge distillation (KD) loss. Our objective is to learn weights for each of these components on an instance-wise basis. This approach enables us to focus on hard labels for specific sets of data points and soft labels from the teacher for another set of points. This allows us to achieve an optimal trade-off between learning from the teacher model and the hard labels, a crucial consideration given the diverse learning capacities of different students. When a student model is significantly smaller than the teacher model, some learning from hard labels proves to be beneficial. In addition to learning instance-wise and loss component-wise weights in this work we also want to benefit from learning patterns of the student cohorts. This is achieved via using C-Net.

Why use PooledStudent logits as the supervision? Theoretical or empirical explanation is supposed to be provided.

In the context of handling a knowledge distillation scenario characterized by a substantial disparity between the student and teacher models, it has been observed that incorporating outputs from intermediate models—those with learning capacities between the teacher and student models—enhances the knowledge distillation process. Consequently, we introduce the PooledStudent logits in conjunction with the teacher logits to amplify the knowledge transfer from the teacher model to the student model. Here again, we control the participation of the PooledStudent logits in the training process using instance-wise loss-component-wise loss. Further in Appendix B.1 we study the effect of the addition of PooledStudent logits.