Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation

We sincerely thank you for the constructive comments and suggestions, which are very helpful in improving our paper. The following responses will be incorporated into the revised paper.

Q1. The impact of over-parameterization on student model performance.

Reply: Thank for your excellent comment. Since the performance of student models are typically limited by their number of parameters, increasing the number of model parameters can significantly enhance performance (e.g. TinyViT-21M (91.9) v.s. TinyViT-5M (87.6) in top-5 accuracy on Imagenet V2). Thus, increasing parameters is beneficial for enhancing the performance of the student model. We will include information about the impact of over-parameterization on the performance of student model in the revised version of the preliminary section.

Q2. The meaning of "think independantly".

Reply: Great remark! We will rephrase this statement to be "learn independently". In the OPDF, auxiliary tensor alignment allows the student model to learn task-relevant information. During the distillation process, learning through the central tensor not only enables the model to imitate the teacher model, but also equips it with the capability to learn independently from the original labels, detached from the teacher model. Consequently, this endows the model with the potential to surpass the teacher model.

Conversely, if the entire parameter matrix is directly aligned, then the student model can only emulate the teacher model. The effectiveness of this method is further evidenced by the experimental results. As shown in Table 1 in our paper, after incorporating OPDF, the student model outperforms the teacher model by 1.2% on the RTE dataset.

Q3. The interpretation of central and auxiliary tensors.

Reply: Thanks for your comment. The definition of central and auxiliary tensors is shown in lines 152-153 in our paper (please kindly refer to the Figure 1b). MPO allows for the decomposition of parameter matrices into a series of tensors:

{MPO}~(\mathbf{M})= \prod_{k=1}^n \mathbf{T_{(k)}}[d_{k-1},i_k,j_k,d_k].\tag{1}

d_k = \min\bigg(\prod_{m=1}^k i_m\times j_m, \prod_{m=k+1}^n i_m\times j_m\bigg).\tag{2}

Following Refs. [1,2], the tensor right in the middle is termed as central tensor, and the rest as auxiliary tensor.

References:

[1] P. Liu, et al. Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators.ACL2021

[2] Z.F. Gao, et al. Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models. COLING2022.

Q4. Reasons why MPO outperforms SVD and the justification for not using MPO over other tensor networks.

Reply: This is an insightful question. MPO generally surpasses SVD because adding eigenvalue dimensions in SVD improves very little of the model capacity, due to the fixed degrees of freedom, expanding parameters through SVD has an upper limit. In contrast, MPO employs matrices of eigenvectors without increasing eigenvalue dimensions, enabling potentially unlimited growth in model parameters.

We also have considered other tensor network format like CP decomposition (CPD) and Tucker decomposition. Generally, the algorithm capacity is larger as the number of tensors $n$ increases (denoting more tensors). When $n > 3$, MPO has smaller time complexity than Tucker decomposition. It noted that SVD can be considered as a special case of MPO when $n = 2$ and CPD is a special case of Tucker when the core tensor is the super-diagonal matrix. The specifics are illustrated in Table A. We will investigate other tensor networks as part of our further work.

Table A. Inference time complexities of different low-rank approximation methods. Here, $n$ denotes the number of the tensors, $m$ denotes $max({{{\{i_k\} }^n_{k=1}}})$ means the largest ${i_k}$ in input list, and $d$ denotes $max({{{ \{d_k^{'} \} }^n_{k=0}}})$ meaning the largest dimension $d_{k}^{'}$ in the truncated dimension list.

Q5. The impact of TT-Ranks on performance.

Reply: Excellent comment! Indeed, we have conducted experiments regarding the impact of $d_k$ on the model performance (please kindly refer to Appendix Table S.4). We can observe that the performance of our approach consistently stabilizes around certain values, indicating that our method is not sensitive to the specific MPO techniques used. Therefore, when over-parameterizing, we should focus primarily on the decomposition scale rather than the MPO method employed.

Q6. The limitations of OPDF.

Reply: Great remark! As discussed in Section 5.3 in our paper, while OPDF can enhance the performance of the student model through over-parameterization, there are inherent limits to these benefits. Additionally, due to over-parameterizing the KD model, OPDF results in higher memory consumption and longer training cost. The memory usage and training cost before and after using OPDF can be found in Table 1 in the one-page PDF rebuttal file.

While OPDF increases the memory usage and training time, this impact diminishes as the dataset size grows, meaning that the ratio of additional memory and training time to the original requirements decreases. We will include an enhanced discussion on the limitations of OPDF in the revised paper.

Concluding remark: We sincerely thank you for putting forward excellent comments. We hope the above responses are helpful to clarify your questions. We look forward to addressing any additional questions. Your consideration of improving the rating of our paper will be much appreciated!

Dear Reviewer XV5e,

We're keen to know if there are any remaining concerns that require attention or if there are additional discussions that should take place. Your insights are greatly appreciated as they contribute to the refinement of the paper. Looking forward to your feedback. Thank you.

Best regards,

The Authors

Dear Reviewer XV5e:

As the author-reviewer discussion period will end soon, we would appreciate it if you could kindly review our responses at your earliest convenience. If there are any further questions or comments, we will do our best to address them before the discussion period ends.

Thank you very much for your time and efforts!

Sincerely,

The Authors

Dear Reviewer XV5e:

As the author-reviewer discussion period is soon ending, we would appreciate it if you could review our responses and provide your feedback at your earliest convenience. If there are any further questions or comments, we will do our best to address them before the discussion period ends.

Thank you very much for your time and efforts!

Sincerely,

The Authors

We sincerely thank you for the positive feedback along with constructive comments and suggestions, which are very helpful in improving our paper. We are also grateful that you recognized the strengths and contributions of our paper. Moreover, the following responses will be incorporated into the revised paper.

Q1. Time Consumption for decomposing and reconstructing the parameter matrix.

Reply: This is a great remark. We listed the time of overparameterization using MPO and the contraction of decomposed matrices into the original matrix in Table A as follows. It can be observed that the time required for decomposition and reconstruction is acceptable compared to the training duration (please alse see Appendix C).

Table A. The spending time (s) of decomposition and reconstruction.

Q2. Applying OPDF to model with billions of parameters.

Reply: This is an insightful question. We have implemented OPDF on the GPT-2-760M, OPT-6.7B, and LLAMA-7B models, with corresponding teacher models of GPT-2-1.5B, OPT-13B, and LLAMA-13B, respectively. The Rouge-L scores of these models on five instruction-following datasets are presented in Table B (see below). Our results indicate that OPDF significantly enhances the distillation efficiency for larger models across all datasets, demonstrating its efficacy even for models with billions of parameters.

Table B. Distillation results on larger models with OPDF.

Dear Reviewer Dajm,

Thank you for your positive feedback. We will include the new experiments and texts in our revised paper.

Thank you very much for your time and effort!

Best regards,

The Authors

Reply to Reviewer fkms

We sincerely thank you for the constructive comments and suggestions. The following responses will be incorporated into the revised paper.

Q1. Results on CV tasks are different from original TinyVit paper. Compared to the distillation results of the original method, OPDF does not offer any advantages.

Reply: Great comment! However, we would clarify that the tasks (TinyViT-5M⚗ achieves 80.7% top-1 with 5.4M parameters) in the original TinyViT paper differ from those in Table 2 of our paper. In lines 230-231 (please see our paper), we report CV task results based on the performance of models tested directly without fine-tuning, whereas the original TinyViT paper presents results after fine-tuning.

Typically, the more parameters a pretrained model has, the greater the performance improvement after fine-tuning, which likely explains why larger models exhibit greater performance gaps. In our replication results, our method improved TinyVit's result from 77.4 to 80.0 under the same conditions, demonstrating OPDF's efficacy.

Q2. Using OPDF in CNN distillation models.

Reply: Thanks for your comment. Ref. [1] affirms MPO's ability to decompose CNNs, confirming its applicability. To show OPDF's effectiveness in CNN distillation, we conducted extra experiments using the methodology from Ref. [2]. This involved distilling a WideResNet from a larger to a smaller version, incorporating OPDF into OFD and classical knowlesge distillation (KD) frameworks. Results and parameters are detailed in Table 2 and 3 of the one-page PDF rebuttal file.

Our results demonstrate that OPDF enhances distillation performance in classical KD and OFD through over-parameterization, underscoring its adaptability in CNN models. This finding will be included in the revised paper. Additionally, as shown in Table 3 of the one-page PDF rebuttal file, OPDF does not increase inference parameters, thus preserving inference time.

References:

[1] Z.F. Gao et al. Compressing deep neural networks by matrix product operators. Physical Review Research, 2020, 2(2): 023300.

[2] Heo, B. et al. A comprehensive overhaul of feature distillation. ICCV2019.

Q3. The difference between MPO and SVD. Necessity and Effectiveness of MPO.

Reply: Thanks for your question. MPO and SVD are different. Since adding dimensions of eigenvalues in SVD does not increase the degrees of freedom, directly expanding the model parameter quantity through SVD improves very little of the model capacity. Moreover, the over-parametrization process via SVD is subjected to an upper limit on the number of parameters. Conversely, MPO uses matrices of eigenvectors without expanding eigenvalue dimensions, potentially allowing unlimited model parameter growth.

For example, for a matrix ${W} \in \mathcal{R}^{I\times J}$, the parameter limit via SVD is $I^2 + J^2$. In contrast, over-parameterizing ${W}$ using MPO with $n$ tensors allows more parameters, as given by follwing equation, proving more effective than SVD.

$N = \sum_{k=1}^{m} i_kj_kd_{k-1}d_k.$

Besides allowing the decomposition of the parameter matrix into an arbitrary number of effective parameters, OPDF demonstrates more significant performance improvements compared to SVD, as detailed in Section 5.2 of our pape. According to Tables 1 and 2 in our paper, while incorporating SVD generally enhances the knowlesge distillation (KD) model's performance across most datasets, it does not perform as well as OPDF.

Q4. The meaning of "think independantly".

Reply: Great remark! We will rephrase this statement to be "learn independently". In the OPDF, auxiliary tensor alignment allows the student model to learn task-relevant information. Through distillation with the central tensor, the student model not only can imitate the teacher model, but also equips itself with the capability to learn independently from the original labels, detached from the teacher model. Consequently, this endows the model with the potential to surpass the teacher model.

Conversely, if the parameter matrix is fully aligned, the student model merely emulates the teacher model. Experimental results validate this, with Table 1 showing that after adopting OPDF, the student model exceeds the teacher by 1.2% on the RTE dataset.

Q5. The training time for TinyViT is excessively long.

Reply: Excellent comment! One GPU day refers to running a single GPU for one day, and actual distillation time can be calculated using multi-GPU parallelism. By utilizing four parallel servers (each equipped with 8 NVIDIA A100 GPUs) for training, the actual training time needed amounts to 5 days. Initially, the TinyViT paper stated a distillation time of 140 GPU days, which increased to 160 GPU days (14% rise) after using the OPDF method, significantly enhancing effectiveness.

Table 1 of our paper shows the parameter changes after parameterization, e.g., DBKD(53M) vs. DBKD+OPDF(83M), with performance increased from 74.4 to 77.6. It is evident that introducing additional training parameters incurs a certain training cost increase, yet it notably boosts the distillation performance.

Q6. The meaning and significance of normalization.

Reply: Great question! During MPO decomposition, normalization distributes information evenly across tensors. As depicted in Algorithm 1 in our paper, after $n$ SVD iterations, all eigenvalue information accumulates in the final tensor, while the preceding $n-1$ tensors are derived from unitary matrix decompositions and do not contain significant information. Normalization effectively distribute information across each tensor.

Concluding remark: We sincerely thank you for putting forward excellent comments. We hope the above responses are helpful to clarify your questions. We look forward to addressing any additional questions. Your consideration of improving the rating of our paper will be much appreciated!

Dear Reviewer fkms,

We're keen to know if there are any remaining concerns that require attention or if there are additional discussions that should take place. Your insights are greatly appreciated as they contribute to the refinement of the paper. Looking forward to your feedback. Thank you.

Best regards,

The Authors

Thank you for your thorough response and additional experiments. Most of my concerns have been addressed and in light of the other reviewers remarks I am happy to raise my score. However, I am still unsure about 2 main parts.

Firstly, although I do understand and appreciate that the authors only report the linear probe performance and thus a comparison to TinyViT, DeiT etc is unfair, I would like to know why this is done? The authors don't seem to do this for the NLP related tasks. Are there other KD works that train models and evaluate in this fashion? (specifically for vision-tasks). Vision KD is a lot more more mature than NLP KD, especially for feature distillation, and so I believe a comparison here is important.

Secondly, the experiments on CNN models is greatly appreciated and does strengthen this submission. It is important to show that the generalisation has practical utility rather than just for the sake of generality. The authors have addressed my concern over this point theoretically too. These CIFAR100 results are fully fine-tuned which is nice to see and partly addresses my previous concern. OFD is argueably quite an old paper. Is there any chance for a comparison to some other architecture pairs provided in the CRD [1] benchmark? and using some more recent KD methods. There are some simple methods with code here [2].

Hope the authors can address my concerns.

[1] "Contrastive Representation Distillation" ICLR 2020 [2] https://paperswithcode.com/sota/knowledge-distillation-on-cifar-100

We greatly appreciate your feedback along with two additional questions. Please see our reply to your first question as follows.

Q1. The application of linear probe performance metrics; distillation results affter fine-tuining.

Reply: Thanks for your question. In fact, Table 5 in original TinyVit paper [1] has already utilized linear probe performance to assess the efficacy of TinyVit. Hence, we opted it for comparison to effectively demonstrate the validity of OPDF.

Moreover, we have fine-tuned models previously distilled following the tasks set in TinyViT (where TinyViT-5M⚗ achieves 80.7% top-1 accuracy with 5.4M parameters), and the results are showed in Table A below to show the efficiency of our proposed OPDF.

The experiments results show that even when employing the same experimental setup in TinyVit, OPDF consistently yields a significant enhancement in the performance of TinyVit. We will include these results in the revised paper.

Table A. Models are pretrained on ImageNet-21k and then finetuned on ImageNet-1k, Imagenet-Real and Imagenet-V2 (top-1/top-5).

BTW, the code of applying OPDF to other recent CNN KD methods (e.g., CRD) is currently executing. We will supplement the corresponding results as soon as the model training is completed. Hence, we will respond to your second question soon. Thank you for your patience.

References:

[1] Wu, et al. TinyViT: Fast Pretraining Distillation for Small Vision Transformers. ECCV 2022, pages 68–85.

We sincerely thank you for the constructive comments and suggestions, which are very helpful for improving our paper. We are also grateful that you recognized the strengths and contributions of our paper. Moreover, the following responses will be incorporated into the revised paper.

Q1. Additional distillation cost (memory and time) incurred by introducing OPDF.

Reply: Great remark! We listed the distillation cost (memory and time cost) of the original model and the model after applying OPDF in Table 1 in the one-page PDF rebuttal file. We can observe that as the number of parameters obtained from MPO decomposition increases, both the training time and memory cost increase. However, as the dataset size increases, the ratio of additional time and memory required for training by OPDF to the original training requirements generally exhibits a decreasing trend (e.g., 0.6/0.4 for RTE vs 0.3/0.1 for MNLI in BERT-of-Theseus model). Therefore, the additional time and memory introduced by our method become less of a critical bottleneck affecting the training speed as the dataset size increases. Hope this clarifies your question.

Q2. OPDF may conflict with mainstream research directions aimed at reducing trainable model parameters (e.g., LoRA).

Reply: Excellent comment! We would like to clarify that our approach has different goal compared with LoRA. Whereas LoRA is designed to reduce the number of parameters during the lightweight fine-tuning process, OPDF aims to enhance the capabilities of existing knowledge distillation models through an over-parameterization procedure.

Nonetheless, we performed tests to assess the effect of over-parameterization through MPO on model performance during the lightweight fine-tuning process. The results are presented in Table A. We observe that the implementation of MPO can augment the efficacy of BERT-base models without extending the inference duration or expanding the parameter volume. Conversely, the adoption of LoRA necessitates the parameter volume for inference, which consequently lengthens the inference time.

Table A. The result of lightweight fine-tuning on GLUE by using MPO and LoRA.

Q3. Distillation results on larger models.

Reply: Great comment! We have implemented OPDF on the GPT-2-760M, OPT-6.7B and LLAMA-7B models, with the corresponding teacher models of GPT-2-1.5B, OPT-13B and LLAMA-13B, respectively. We have reported the Rouge-L scores of these models on 5 instruction-following datasets, with the results displayed in Table B. We can observe that, after implementing OPDF, the efficiency of distillation on large models is improved across all datasets. This demonstrates that our method is also highly effective on larger models.

Table B. Distillation results on larger models with OPDF.

Concluding remark: We sincerely thank you for reviewing our paper and putting forward thoughtful comments/suggestions. We hope the above responses are helpful to clarify your questions. We will be happy to hear your feedback and look forward to addressing any additional questions. Your consideration of improving the rating of our paper will be much appreciated!

Dear Reviewer Qtmi,

We're keen to know if there are any remaining concerns that require attention or if there are additional discussions that should take place. Your insights are greatly appreciated as they contribute to the refinement of the paper. Looking forward to your feedback. Thank you.

Best regards,

The Authors

Thank you for your positive response. We would like to clarify that during the inference with LoRA, the parameters that need to be computed are $W+\Delta W$, where $W$ represents the weights of the pre-trained model (PLM) and $\Delta W$ the additional parameters introduced by LoRA. This is why we mentioned in our reply that applying the LoRA method during inference increases the number of parameters, thereby increasing the inference time.

Lightweight fine-tuning (LoRA) and model compression (KD) represent two different tracks for deploying large PLM. While LoRA is cost-effective, it slightly increases the inference time due to the added $\Delta W$ parameters. On the other hand, KD results in a compressed model that reduces both inference time and computational overhead while increasing the training cost.

Moreover, compressed models obtained through KD may have diverse applications on the edge. For instance, they are particularly suitable for deployment in resource-constrained environments, such as mobile devices and embedded systems.

Again, thank you very much for increasing the rating of our paper!