Adaptive Pruning of Pretrained Transformer via Differential Inclusions

Weakness 1: The results are insufficient.

As shown in Table 1, for Deit-Small, with the same number of flops 2.6, WDPruning has an accuracy of 78.4% compared to 78.9% for our method. For Deit-Base, we outperform all methods with 81.2% accuracy and keep fewer flops, only 6.9.

As we explained to Reviewer 1jDf, our method not only provides better numerical results but also introduces a new pruning approach based on differential inclusion. We identify the sparse structure of the model in the inverse space and map it to the solution path. The most important parameters are learned first, followed by the less important ones, allowing us to prune based on each parameter's importance. Therefore, our approach learns a regularization path, eliminating the need to re-prune the network for each sparsity level.As shown in Figure 2, our method is very consistent, ensuring that adaptive pruning is done effectively.

Weakness 2: Figure 1 and tables lack legends and comments.

We will rewrite the comments for Figure 1 and Table 1 to make them clearer and easier to understand.

Weakness 3: Typo error.

Thank you for pointing out the typo . We will correct this and improve our writing when revisiting the manuscript.

Question 1: The application of low-rank compression to paired modules of transformers.

The detailed application equations can be found in Section 3.1.

Question 2: The actual running time of the proposed method.

The latency times for the uncompressed and compressed CLIP models are shown in Table 7. The search epochs and training epochs are listed in Table 6.

Dear Reviewers,

We have worked hard to address your comments in detail and would greatly appreciate any additional feedback or clarification. If anything remains unclear or if more information is needed, please let us know how we can assist. Your feedback is important to us, and we look forward to hearing from you whenever it is convenient.

Best regards,

Weakness 1: Marginal Improvement Over Existing Methods

Traditional mask-based pruning methods, while effective in fixed-ratio compression, are not efficient in adaptive pruning, which is important for deployment in many scenarios. This paper provides a comprehensive solution. First, by adopting the perspective of differential inclusion, we can efficiently obtain a solution path from sparse to dense, making it well-suited for adaptive training. Moreover, our dynamics enjoys the inverse scale space property: important features will select at earlier epochs, and while be consistently maintained in later epochs (figure 2). Finally, we establish the convergence results, ensuring the reliability of our algorithm.

Weakness 2: Lack of Broad Comparison with Other Mask-Based Methods

In Table 1, we have compared with several masked based pruning methods, such as UPop [1] and VitSlimm [2], which have achieved state-of-the-art results.

[1]Shi, Dachuan, et al. "Upop: Unified and progressive pruning for compressing vision-language transformers." International Conference on Machine Learning. PMLR, 2023.

[2]Chavan, Arnav, et al. "Vision transformer slimming: Multi-dimension searching in continuous optimization space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Question 1: How does the method scale to very large transformers？

The detailed algorithm is shown in Algorithm 2. As stated in line 762-765, we apply zero shot pruning to LLMs with our method. Since there is no weight update, our pruning method is efficient while preserving the generalization ability.

Question 2: What is the stability of the solution path across different random seeds?

Our method is robust across seeds.

Question 3: How sensitive is the method to the choice of hyperparameters κ and λ?

Our method is not sensitive to hyperparameters as is shown in Table 8. In fact, as long as κ and λ are set in a way that allows the inverse space to learn the important parameters, the method will work effectively.

Question 4: Can this method be extended to dynamic/runtime pruning scenarios?

In theory, we could add weight updates during algorithm iterations to achieve runtime pruning. However, this would result in additional training costs and make the model's loss function too complex, which could prevent proper convergence. Besides, It would increase the training time. Starting from a fixed, pretrained checkpoint is more efficient.

Dear Reviewers,

We have worked hard to address your comments in detail and would greatly appreciate any additional feedback or clarification. If anything remains unclear or if more information is needed, please let us know how we can assist. Your feedback is important to us, and we look forward to hearing from you whenever it is convenient.

Best regards,

Weakness 1: Training cost of the proposed method.

The training cost of our proposed method is provided in detail in Table 6 and Table 8 of the Appendix. We have listed both the search cost and the fine-tuning cost. Our fine-tuning cost for the Deit models is 300 epochs, which is the same as other methods such as WDpruning[1] and UPop[2].

[1]Yu, Fang, et al. "Width & depth pruning for vision transformers." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 3. 2022.

[2]Shi, Dachuan, et al. "Upop: Unified and progressive pruning for compressing vision-language transformers." International Conference on Machine Learning. PMLR, 2023.

[3]Chavan, Arnav, et al. "Vision transformer slimming: Multi-dimension searching in continuous optimization space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Weakness 2: Comparison between weight-based pruning and mask-based pruning.

Weight-based pruning has already been shown to be less effective compared to mask-based pruning in the VitSlimm[1] method. DessiLBI[2] method has the same pruning metric with weight-based pruning. The comparison with DessiLBI method can be tought to be comparison with weight-based pruning. While Lasso can be used for mask-based pruning (line 246, Eq.7), it is computationally expensive to adaptively train nonlinear models which do not have a closed-form solution path, as we have for linear models (line 249-250). That means, we need to optimize the loss in Eq. (7) for each regularization parameter \lambda. Moreover, UPop and VitSlimm are advanced mask-based pruning methods that existed before our method. We compared these methods with ours in Table 1.

[1]Chavan, Arnav, et al. "Vision transformer slimming: Multi-dimension searching in continuous optimization space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2]Fu, Yanwei, et al. "Dessilbi: Exploring structural sparsity of deep networks via differential inclusion paths." International Conference on Machine Learning. PMLR, 2020.

Question 1: How our method extends to pruning large language models.

The detailed algorithm is shown in Algorithm 2. For large language models, the fine-tuning cost is higher than for models like ViTs and CLIP. As mentioned in line 762, it is important for LLMs to maintain their generalization performance even after pruning. Current pruning methods for ViTs cannot preserve generalization. Therefore, we extended our method to zero-shot pruning with no weight update, where the RIA [4] method performs best.

[4] Zhang, Yingtao, et al. "Plug-and-play: An efficient post-training pruning method for large language models." The Twelfth International Conference on Learning Representations. 2024.

Question 2: Typo error.

Thank you for pointing out the typos. We will make sure to improve our writing when revisiting the manuscript.

Dear Reviewers,

We have worked hard to address your comments in detail and would greatly appreciate any additional feedback or clarification. If anything remains unclear or if more information is needed, please let us know how we can assist. Your feedback is important to us, and we look forward to hearing from you whenever it is convenient.

Best regards,