Enriching Knowledge Distillation with Intra-Class Contrastive Learning

Suggestions:

1. more comparisons with recent knowledge distillation methods, especially those that use contrastive learning.
2. provide ablation studies to assess the contributions of margin loss and intra-class losses.
3. more evaluation datasets and network architectures.

1. Could you provide any theoretical or experimental demonstrations to support your assumption that cross-entropy loss is equivalent to inter-class contrastive loss?
2. If your assumption holds, could you explain why you did not directly use the inter-class contrastive loss for training the teacher model instead of cross-entropy loss?
3. Could you show that your two distance metrics (inter-class and intra-class distance) satisfy the four conditions required for a valid distance function (metric space)?
4. In section 3.1, you define the sample space($\mathcal{X}$), the label space($\mathcal{Y}$) and the hypothesis space($\mathcal{F} : \mathcal{X} \rightarrow \mathcal{Y}$) then refer to the classifier as $f \in \mathcal{F}$. However, shouldn't it be represented as $\mathcal{F} : \mathcal{X} \rightarrow \mathbb{R}^d(d>c)$, similar to the feature embedding function $\varphi$ described in Section 4.1?
5. Could you report the results of the teacher model trained with your loss?
6. Could you compare the student model’s improved performance when combined with other KD methods (beyond RKD) against the additional parameters that need to be trained?
7. Could you provide experimental results for directly training the student model using your proposed losses (CE and margin) along with knowledge distillation loss, rather than just distilling knowledge through the teacher model trained with those losses?
8. Could you verify whether Sohn (2016a) and Oord et al. (2018) on page 2 indeed employ augmented samples as positive samples and other samples from the same class as negative samples?
9. Could you report the test accuracy and test loss curves for at least one type of teacher model to illustrate instability and slow convergence during training?

Things to improve the paper that did not impact the score:

1. Please follow the formatting instructions of ICLR regarding citations within the text, ensuring that \citep{} and \citet{} are used appropriately.
2. Possible typo: In line 5 of the Contrastive Learning section on page 3, "Constrastive earning" should be corrected to "Contrastive learning."
3. Please differentiate between scalar and vector forms in your mathematical notation.
4. Please clarify whether $f$ indicates $f_t$ or $f_s$ in Equations (4) and (7).
5. To my knowledge, existing methods utilize KL-divergence for knowledge distillation. Strictly speaking, While KL-divergence and cross-entropy loss may function similarly in knowledge distillation, they are mathematically different. Please clarify the mathematical differences in Equations (2) and (3).
6. What type of augmentation techniques did you use for positive samples in the intra-class loss?
7. what values did you set for $\delta$ in the margin loss?
8. Please unify the notation for referencing equations, such as whether to use "Equation 5," "Eq. 5," or simply "5."

1. Why not use the common hyperparameter settings for KD? For example, for CIFAR-100, use 240 epochs with a learning rate of 0.05 and a batch size of 64 [3].

2. What teacher models were used for the baseline methods?

3. What is the performance of the proposed method when combined with other baseline methods?

4. The experiments lack CRD [3], which proposed contrastive learning for KD.

5. please address W1-4.

6. It would be better to have a comparison to a teacher trained with inter-class loss.

[1] Li, Z.; Li, X.; Yang, L.; Zhao, B.; Song, R.; Luo, L.; Li, J.; and Yang, J. 2023. Curriculum temperature for knowledge distillation. In Association for the Advancement of Artificial Intelligence (AAAI)

[2] Sun, S.; Ren, W.; Li, J.; Wang, R.; and Cao, X. 2024. Logit Standardization in Knowledge Distillation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).

[3] Tian, Y.; Krishnan, D.; and Isola, P. 2020. Contrastive representation distillation. Proc. Int. Conf. on Learning Representation (ICLR)

Here are some concerns that need to be addressed.

1. The idea of ​​enhancing knowledge distillation via contrastive learning is not innovative.
2. Based on the experimental results, the improvement of the proposed method is not significant.
3. Following the previous work, the author needs to add experimental analysis results on the ImageNet and MS COCO datasets.
4. Moreover, the authors need to add relevant analysis regarding the efficiency of the proposed method compared to the state-of-the-art baselines, including training time.

Although the paper mentions that margin loss improves training stability, if unstable phenomena are observed in experiments, it is recommended that the authors provide more details on training stability analysis and coping strategies, including experimental results under different hyperparameter settings. The experiments mainly focus on specific image classification datasets, and it is suggested to expand the scope of experiments to cover text, audio, or time series data to verify the generality and applicability of the method. In addition, it is recommended to provide more detailed parameter tuning guidelines to help researchers effectively select weight parameters (such as α and λ). Finally, it is suggested to delve into the theoretical basis of intra-class contrastive loss to fully understand its specific impact on the model's learning mechanisms.

1. How does the computational complexity compare to standard knowledge distillation?
2. What is the sensitivity of the method to the choice of λ and δ?
3. How does the pipeline-based caching mechanism affect training time?
4. Can you provide ablation studies showing the individual contribution of intra-class contrastive loss and margin loss?