PPT: Token Pruning and Pooling for Efficient Vision Transformers

We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your concerns and questions one-by-one in what follows:

"Lack of discussion and comparison with the highly related work "joint Token Pruning & Squeezing (TPS)..."

Thank you for the complement to our comparative methods, which is a great work. we provide additional comparative results based on the DeiT-S baseline as shown in the table below:

It should be noted that the dTPS* is fine-tuned under 100 epochs as mentioned in the paper, while the eTPS and our method both are fine-tuned under 30 epochs. We can find that our method achieves better performance than eTPS on the trade-off between FLOPs and accuracy.

"Lack of experiments of hybrid ViT such as PVT..."

As we mentioned in the conclusion section, the classification token plays an important role in our framework, so we cannot apply our method to PVT directly. However, we are committed to addressing this limitation and finding a solution in our future research endeavors. We sincerely appreciate your suggestions.

"I think the author should make the "adaptive" point more convincing, such as comparing other non-adaptive pruning and pooling methods (pruning and pooling at the same time), or comparing the naive non-adaptive baseline (pruning only before layer x and pooling only after layer x, layer x is a fixed parameter such as half or two-thirds of the depth)."

We highly appreciate your viewpoints, and in Section 4.2 of our paper, titled "Effectiveness of techniques integration (Figure 8)" and "Different policy selection mechanisms (Table 3)", we have discussed and provided further evidence for the two approaches you mentioned. These discussions aim to demonstrate the effectiveness and convincing of the "adaptive" strategies we proposed.

"I think more metrics other than the "variance" of the distribution of importance score of image tokens should be considered. Since variance is shift-invariant and absolute value of the score is not considered. The absolute value of the score distribution should be somehow helpful to the choice of pooling or merging."

Thanks for your suggestion, we have explored different metrics for redundancy discrimination in Section C of Appendix due to the space limitations. Specifically, we have explored another metric, the average of token similarity, which also reflects the redundancy level of tokens in different layers. And we find its performance is inferior to our chosen metric (The data placement in Table 7 is filled incorrectly, and we will refine the typo in refined version).

"See weakness 3, how to make this method work on hybrid ViT(including regular spatial conv) and what's the improvement?" "How to make this method work on dense prediction tasks such as segmentation and what's the performance?"

Thank you for your suggestions. Applying our method to models without class token is indeed one limitation of our method, as we also discuss in Page conclusion. We will leave it for future work.

"Is it a possible case where pooling at the early stage and pruning at the final stage? What's the performance if the adaptive policy is reversed? (change pooling to pruning and pruning to pooling)

As shown in Section 4.2 of our paper, titled "Different policy selection mechanisms (Table 3)", we have explored the inversion of our policy decision, wherein pruning was performed instead of pooling, and vice versa. This significantly reduces the accuracy of the model, further reinforces our claims that token pooling techniques are typically applied in shallow layers, while token pruning techniques are preferably employed in deeper layers.

We hope that our response adequately addresses your concerns. We genuinely appreciate your valuable discussion and suggestions, and we will refine the relevant content in the revised version accordingly.

We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your concerns and questions one-by-one in what follows:

"The novelty is relatively limited. The token pruning or token pooling has been widely investigated in the literature [DynamicViT, Evo-ViT, Self-Slim ViT, etc]. The adaptive design used in the paper is simply the combination of both pruning and pooling without much insightful modifications."

We emphasize the novelty of our work in the following aspects:

• The proposed method is the heuristic framework that acknowledges the complementary potential of token pruning and token merging techniques for effifient ViTs, i.e., token pruning for inattentive redundancy while token pooling for duplicative redundancy. The effectiveness of our approach is visually demonstrated in Figure 1 of the paper.
• Our method is not a simple combination of existing techniques. As detailed in Section 3.4, our method is derived based on a closer look at the characteristics of token pruning and token merging techniques, as well as a thorough analysis of the importance scores in different layers.
• Additionally, we heuristically design a redundancy criterion, i.e., the variance of the significance scores, to guide adaptive decision-making on prioritizing different token compression policies for various layers and instances, as shown in Figure 7.

"The results are actually comparable with the state of the art methods, with a little improvement. As shown in Table 1, the proposed method is actually comparable to Evo-ViT, in terms of all the DeiT model settings. Similarly, it shows the marginal performance improvement, compared with EViT-LV-S. All of these indicate that, the effectiveness of the proposed design is not quite convincing."

We followed the settings of many previous works and reported the accuracy in Table 1 based on the given FLOPs. However, in this comparatively high FLOPs scenario, various methods have already approached the accuracy of the original model. Although our method achieves state-of-the-art performance, the improvement does not appear to be significant.

To provide a more comprehensive evaluation, we show the comparison between our method and other methods under different FLOPs (based on the DeiT-S as baseline) and provide corresponding data in the Figure 4.

As analyzed in paper, the superiority of our method is demonstrated in various FLOPs and off-the-shelf (without fine-tuning) scenarios. Specifically, it includes three aspects:

• At lower FLOPs (2.5G below), our method exhibits a noteworthy performance improvement of 0.7%-3.8% compared to other pruning methods.
• At higher FLOPs (3.0G above), our method demonstrates superior performance compared to the baseline model, surpassing the capabilities of the comparison methods.
• When used in an off-the-shelf (without fine-tuning) setting, PPT demonstrates significant improvement (0.1%-2.3%) compared to other methods. Notably, PPT also achieves competitive results with other fine-tuned models.

We hope that our response adequately addresses your concerns. We genuinely appreciate your valuable discussion and suggestions, and we will refine the relevant content in the revised version accordingly.

We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your concerns and questions one-by-one in what follows:

"The inference is hard to implement to be really act as what the authors have claimed. I do not think the throughputs are experimental numbers but theoretical numbers. This is the common issue for adaptive token pruning methods, such as A-ViT. I think in practical this paper is useless, not only not decreasing the real computation cost but increasing the cost. I do not think the codes would be released for inference."

Compared to A-ViT, which employs dynamic compression ratios based on different images, our approach utilizes a compression strategy that is more conducive to batch processing by progressively reducing the Top-K redundant tokens in each stage. This strategy significantly reduces the computational workload with only a minimal amount of additional computations required.

As we mentioned in paper, we measure the throughput of the models on a single NVIDIA V100 GPU with batch size fixed to 256 and we indeed decrease the real computation cost. The inference codes with throughputs calculation are updated in Supplementary Material. You can check it by running the eval.sh (dependencies refer to requirements.txt).

"The results are not promising. In recent CVPR2023, there are several new works that achieved better results than this one. For example, Making Vision Transformers Efficient From a Token Sparsification View."

Thank you for the complement to our comparative methods, which is a great work. In comparison to the method you mentioned, our method excels in that it does not require additional trainable parameters and can achieve impressive results even in off-the-shelf scenarios, where no customization or fine-tuning is required. We believe that this characteristic is particularly beneficial for huge models and scenarios with limited computational resources.

"Further finetuning based on pretrained models is not fair comparisons."

Fine-tuning based on pretrained models is a widely adopted practice in this field. To ensure a fair comparison, we applied the same fine-tuning strategy as the previous methods, which include DynamicViT, EViT, PS-ViT, ATS, and others.

We hope that our response adequately addresses your concerns. We genuinely appreciate your valuable discussion and suggestions, and we will refine the relevant content in the revised version accordingly.

We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your concerns and questions one-by-one in what follows:

"Missing a pretty relevant reference: Unified Vision Transformer Compression. ICLR 2022 Please cite and compare with it on Deit-S, Deit-B, and Deit-Tiny."

In our paper, we primarily compare acceleration techniques for Vision Transformers (ViTs) that focus on token-level compression. However, as you mentioned, the UVC method you referred to primarily compresses the model parameters. We appreciate your reminder and would like to cite it in paper and incorporate a comparison of its performance as follows:

"The backbones used in the paper are Deit and LV-VIT. How about the Swin-Transformer, which also already has a patch merging module for each stage of blocks? I am curious about the generalization of the proposed mechanism in this kind of structure and its performance."

In the Swin-Transformer, the patch merging module is designed to reduce the resolution, similar to pooling, by merging local features in a regular manner. However, our pruning and pooling techniques are actually instance-aware to reduce the computation cost, and these two approaches are not contradictory. In our future work, we will investigate experiments combining our methods with the Swin-Transformer. We sincerely appreciate your suggestions.

"The target compression ratio seems to be hard to directly and accurately map to each layer's reduction percentage contributed by token pooling and token pruning. How to balance the K threshold in pruning and pooling and how to choose the decision threshold to match a user-given compression ratio? Please clarify it in more detail."

As illustrated in Figure 4, we showcase the performance of our method across various FLOPs by adjusting the hyperparameter K. For more comprehensive details, please refer to Section B in the appendix, where we provide specific settings and explain how to select the appropriate K to achieve a desired compression ratio. Additionally, in Section 4.2 of our paper, we delve into a thorough discussion on the impact of the decision threshold and present Figure 9, which outlines the process for determining the optimal threshold value.

We hope that our response adequately addresses your concerns. We genuinely appreciate your valuable discussion and suggestions, and we will refine the relevant content in the revised version accordingly.