VRM: Knowledge Distillation via Virtual Relation Matching

The weaknesses of the work is listed as below: (1) There are many techniques and details in VRM framework. It makes VRM seem complex and hard to reproduce. For instance, it has many hyper-parameters to tune. (2) As the main contribution of the work is "Virtual Relation Matching", we supposed virtual relation to be an important component for final performance gain. However, according to the ablation study, other techniques including ZSNorm and L2Norm contribute significantly. It weakens the contribution of this work. (3) Similar to (2), graph pruning shows marginal performance gain, though it is claimed as a main contribution of this work. (4) Important citation missed: Knowledge Distillation via Instance Relationship Graph.

1. While this study has the potential to bring the relation-matching distillation back into attention, this method sacrifices the elegance of simplicity and instead introduces numerous new modules and hyperparameters, including the balancing loss weight of each module, the pruning criterion and the number of augmentation views. The additional hyperparameters would impair the simplicity and hinder its application in many other tasks.
2. Although the authors state that the current method is not a simple view-augmented relation distillation in the section on further analysis, the proof does not seem convincing enough. As the number of nodes is doubled in VRM, which brings in 1x more additional computational cost, the question would arise if the weakly augmented relation matching network would achieve the same performance given 1x more training budget.
3. The presentation of the two drawbacks of the previous relation-matching network is still intuitive, and the attempted mathematical proof is less vigorously formulated. For example, in the analysis of the effect of the spurious sample on the gradient of other samples, both the input x and guidance y are treated as random variables which leads to 0 correlation between x and y, which clearly does not hold in reality.

1. Previous GNN-based KD methods, such as [1][2], have shown strong results on various tasks. Without comparisons in the related work or experimental sections, it is difficult to fully assess VRM's effectiveness.
2. Currently, each prediction is assigned only one virtual view. Further analysis of how the number of virtual views affects KD performance and computational cost would provide a more complete evaluation of VRM’s design.
3. Although VRM includes a strategy for pruning unreliable edges, there is a lack of visualization or analysis of the pruned samples to confirm that these edges were indeed unreliable. Comparing VRM’s pruning approach with alternative methods would also clarify its importance in the framework.
4. VRM has been evaluated solely on image tasks. Further exploration into how VRM might be adapted to non-image domains would enhance its potential applicability.

[1]Zhou, Sheng, et al. "Distilling holistic knowledge with graph neural networks." ICCV. 2021.

[2]Zhang, Chunhai, et al. “Multi-scale distillation from multiple graph neural networks.” AAAI. 2022.

1. My biggest concern regarding this paper is whether it delivers a meaningful contribution to the community in 2024 given its current experimental setup. Specifically:

• The proposed methods have only been validated on weak baselines, whose training configurations (described in Appendix A.3) are from several yeas ago. In recent years, lots of advancements have been made in developing better training methodologies for training high-performing image classification models, including improved optimizers like AdamW and data augmentation techniques like RandAug. It is well-known that improving weak baselines is relatively straightforward, while achieving improvement on strong baselines is more challenging. The paper does not convincingly show if the proposed method can surpass simple knowledge distillation (KD) techniques, such as basic logit or feature matching, under these modern training settings.
• The architectural choices used in the experiments are outdated. To better demonstrate the effectiveness and generalizability of the proposed method, it is recommended that the authors incorporate experiments with more contemporary baselines, such as ConvNext and Swin Transformer. Even for mobile-optimized models, newer architectures like MobileViT should be evaluated.
• While the authors may claim that they simply follow the experiment settings of some recently published previous works, to me, those works (including this paper) are more like "playing toy KD games on a small scale while completely overlooking the recent advances in a broader scope. Also, it should be noted that there are indeed KD works with up-to-date experiment settings, e.g., [a].
• It is essential to emphasize that this critique does not undermine the potential significance of the work. I acknowledge that knowledge-distillation remains crucial, particularly for deploying models in resource-limited environments. However, to convincingly position this research as a contribution in 2024, significant updates to the experimental settings are necessary.

2. It is unclear whether the reported improvements are primarily from the increased training data introduced by the additional views. A valid question arises as to whether similar performance gains could be achieved by simply incorporating these augmented views during baseline training or while using conventional instance-matching methods?

3. Some proposed techniques yield only minor gains over the baseline. For instance, the primary enhancement in the inter-sample matching component is from employing a new distance metric, which results in only marginal performance improvement (as shown in Table 6).

4. While the paper claims that relational knowledge is particularly valuable for specific architectures, modalities, and tasks (line 48), the effectiveness of the proposed method in these contexts has not been demonstrated. On the contrary, Section 3.2 convincingly illustrates the drawbacks of relation-based methods, raising doubts about the necessity of developing more relation-matching approaches.

[a] Beyer, Lucas, et al. "Knowledge distillation: A good teacher is patient and consistent." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.