1. The technical contribution/novelty is quite limited. Firstly, applying feature-based distillation on the shallow layers is not novel since this is common in CNN distillation, e.g., FSP [1], AT [2], OFD [3], and VID [4]. One can easily apply them on all the layers. Second, the main discovery is the performance degradation when distilling the deep layers. However, the proposed method, ViTKD, mainly adopts MGD [5] as the solution to this problem, which is a direct application of CNN's distillation. If we look at Sec. 3.1 and 3.2, there is no fundamental difference between the listed equations and the ones in Sec.3 of the MGD paper. This makes the paper more like a technical report rather than a scientific paper.

2. There are inappropriate statements in the contributions listed at the end of the Introduction section. 1) "We reveal that the feature-based KD method for CNNs is unsuitable for ViTs." However, the proposed methods (Sec. 3.1, 3.2) are all CNN's distillation methods. 2) "The distillation on the shallow layers is also important for ViT, which differs from the conclusion in KD for CNNs." Can you provide some references that explicitly make that conclusion on CNN KD? Is there any evidence before claiming that shallow layer distillation is not important for CNN KD?

3. The necessity of using random masks is low. As shown in Table 10, when $\lambda=0$, the method boosts the performance by +1.35 Acc. However, it can be further improved by only +0.29 Acc. if we carefully tune $\lambda$. This significantly undermines the motivation of deep layer’s Generation distillation, because one can see that the main improvement comes from the feature-based distillation just with one more additional block to process the student's features.

[1] Yim J, Joo D, Bae J, et al. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. CVPR 2017.

[2] Zagoruyko S, Komodakis N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ICLR 2017.

[3] Heo B, Kim J, Yun S, et al. A comprehensive overhaul of feature distillation. ICCV 2019.

[4] Ahn S, Hu S X, Damianou A, et al. Variational information distillation for knowledge transfer. CVPR 2019.

[5] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. ECCV 2022.

Weakness

1. Downstream results are only compared to the baseline, not other distilled or pruned small ViTs.
2. The combination with logit-based distillation methods such as NKD is mentioned without sufficient explanation. It remains unclear whether more hyperparameters will be introduced on the basis of many hyperparameters (α, β, λ), or make the method more complicated for other reasons.
3. In the main text, the author only compares the proposed method with a few baselines, and the logits-based methods compared are all from papers before 2021. The relatively new articles in Appendix A.2 are easily overlooked, so the comparison results of this part can be mentioned in the main text.
4. More analysis could be provided to better understand how and why ViTKD works. For example, in Figure 4 the authors could add attention maps of some middle layers (such as layers 5 and 6 aligned with Figure 1) to show how they change after distillation.
5. The paper would benefit from a more detailed discussion of the limitations of the proposed approach. Providing insights into the potential weaknesses of the method would strengthen the paper's contribution.

1. The technical contribution is limited. $L_{lr}$ is proposed by FitNet [1]. $L_{gen}$ is borrowed from masked language (image ) modeling. The authors claim that they use NKD as the logit-based distillation method. This make the novelty weak.
2. The motivation is not clear. The paper do not explain why feature generation (i.e. masked modeling) is effective to vit-based architecture. If Table 1 and Table 5 are the support for they method, then the motivation is empirical and not generalized. When it comes to different teacher-student pair, the motivation may not be met. For instance, in Table 3, the performance on swin transformer (+0.52) is limited compared to DeiT (>=1.4).
3. In 5.2, the authors use first two layers for feature disillation and the last layer for feature geneartion. However, they analysis do not provide enough insight to explain such decision. Hence, the experimental result looks random. When the student network architecture is changed, the same recipe can not guarantee the performance. In contrast, logit-based distillation is model-agnostic and general.
4. By my understanding, the result in Table 2 uses the technique of logit-based method NKD. The ablation study in Table 6 does not show the performance of ViTKD without such a loss, making the claimed performance unclear.
5. Regarding the transformer and knowledge distillation, this work misses some releated work such as DistillBert.

[1] Fitnets: Hints for thin deep nets. ICLR 2015

[2]DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

• The intuition behind why the generation mechanim was chosen to fill the gap between student’s deep layers features and teacher’s deep layer features is not very clear.

• The initial empirical analyses is just based on visualization and perhaps comparison based on quantification of the disparity between two corresponding set of features could be more convincing.

• What happens to student’s deep layer features after applying the method of ViTKD is not clearly discussed?

• How does the generative process between student’s deep feature and teacher’s deep feature allows student’s deep features to obtain better semantic information?

• Since there is a gap between the student’s deep feature layers and teacher’s deep feature, instead of resorting to generative way, wouldn’t some relaxed loss formulation give similar or even better performance?