Frequency-Aware Token Reduction for Efficient Vision Transformer

We sincerely thank the reviewer for their insightful summary and positive assessment of our work. We are grateful that you recognized our method's solid theoretical foundation, state-of-the-art results, and practical efficiency. Your valuable suggestions regarding clarity, visualization, and computational analysis are well-taken, and we will address each point to further strengthen the manuscript.

• Clarity and Token Selection with Windowing

We thank the reviewer for this suggestion. To clarify the procedure, our method first performs a global identification of High-Frequency (HF) and Low-Frequency (LF) tokens based on the full attention map, as described in Section 3.2 . Subsequently, the identified global LF token set is partitioned into spatial windows, and tokens within each windowed subset are aggregated into a local DC token . HF tokens are always preserved globally.

To improve clarity and ensure reproducibility, we will add a dedicated algorithm block with pseudo-code in the Appendix of the revised paper.

• Visualization

Following your valuable suggestion, we performed an experiment to visualize the selected High-Frequency (HF) and Low-Frequency (LF) tokens on sample images. While we cannot attach figures directly to this rebuttal, we will describe our key findings here and commit to adding these visualizations in the main manuscript. Our analysis confirmed that in early to mid-layers, the token selection aligns well with human intuition: HF tokens predominantly correspond to patches with object boundaries and complex textures, while LF tokens are clustered in smooth, uniform background areas. Interestingly, we also observed 'outliers' where some background patches were selected as HF tokens, which we found often correlates with high-norm features in the embedding space. In deeper layers, we observed that where tokens corresponding to the salient foreground of an object began to be classified as LF tokens. Since rank collapse becomes more severe, the unique signals of even the most important tokens in human perception, get 'smoothed out' and converge toward the low-frequency, DC-like representation of the entire feature map.

• Role of DC Subtraction in Token Ranking

You are correct that the argmax ranking is mathematically unchanged by the subtraction. Our reasoning for this formulation is to maintain a clear and consistent theoretical framework throughout our method. This initial decomposition is crucial for consistency with our subsequent attention modification step in Equation (12), where we use a learnable parameter ω1 to amplify high-frequency signals . We will clarify this in the revised manuscript.

• Precise Definition of “Relative Magnitude”

The term “relative magnitude (weight)” in Eq.7 is calculated as the column-wise sum of the high-pass attention map corresponding to a particular token, which is then averaged across all attention heads. We will revise the text in Section 3.2 to make this connection explicit and avoid any confusion.

• Computational Cost Analysis

Thank you for this valuable suggestion. Following your recommendation, we have performed a more concrete computational cost analysis.

A naïve method that involves calculating the DC signal and then measuring feature similarity between all tokens can have a complexity of up to $O(nd+nd^ 2 )$. In contrast, our proposed method, which simply post-processes the already computed attention map, requires only an additional $O(n^ 2 )$ complexity per head.

For a typical model like DeiT-S with n=197 and d=384, the cost of our $O(n^ 2 )$ operation is significantly lower than that of the naïve approach. Even with large n, for instance, down stream task such as object detection, cost is still lower since $n << d^2$. This analysis confirms that our frequency-aware selection is not only highly effective, as demonstrated by our experiments, but also computationally very efficient. We will add this detailed cost analysis to the revised manuscript.

• Typo

Thank you for your careful reading and for pointing out the inaccuracies. We will correct all identified typos and unclear phrasing in the final version of the paper.

We sincerely thank the reviewer for the thorough and insightful feedback. Your suggestions have been invaluable in helping us strengthen our work. We address each of your points below with new experimental results and analyses.

• On Performance Improvement and Downstream Task Generalization

While the accuracy improvement is most visible on the ViT-S and ViT-S-21K models, we would like to clarify the goal of our method from a broader perspective, which demonstrates its effectiveness across all experiment settings.

The primary objective of any token reduction method is to significantly reduce computational cost while minimizing performance degradation. From this viewpoint, the results for models like DeiT represent a major success. For instance, on DeiT-B, our method matches the baseline accuracy of 81.8% exactly, while reducing the computational cost from 17.6 to 11.6 MACs. This result is the ideal outcome, demonstrating that our core objective—maximizing efficiency without sacrificing performance—was successfully achieved. We consider this a significant contribution to the field of efficient models.

The notable performance increase on ViT-S and ViT-S-21K can be seen as a significant benefit, highlighting a unique strength of our approach. As our paper analyzes, standard ViTs are prone to over-smoothing and rank collapse, where models lose token diversity.

Following your valuable suggestion, we conducted additional experiments on object detection. To ensure a fair comparison, we adopted the experimental settings from SViT (Revisiting token pruning for object detection and instance segmentation) [1] and applied the same pruning configuration used in our main classification experiments (removing 30% of tokens at the 4th, 7th, and 10th layers). These experiments were performed on an A100 GPU. In the results table, values in bold represent our own measurements, while others are taken from the SViT paper for reference.

The results show our method achieves a highly competitive performance-efficiency trade-off. Our method's APbox of 45.0 (-0.8 drop) is significantly better than other pruning-based methods like EViT (-1.3). When compared with SViT, which shows better performance preservation (-0.3 drop), it is important to consider the fundamental architectural differences that lead to this specific trade-off.

SViT recycles information from pruned tokens in all subsequent layers, which helps preserve accuracy but introduces overhead. In contrast, our method is designed to be more lightweight by recycling this information only once at the final layer of the ViT-Adapter block. This key design choice is why our method achieves a notably higher throughput, running over 7% faster than SViT (21.0 vs 19.6 FPS), making it a compelling alternative for latency-sensitive applications.

Furthermore, this trade-off is highly task-dependent. It is worth noting that on ImageNet classification with DeiT-S, our method outperforms SViT on both accuracy and efficiency. Our method achieves 79.8% accuracy (-0.0% drop), while SViT scores 79.4% (-0.4% drop).

We hypothesize that adopting a more complex, SViT-style recycling mechanism could further improve our detection performance. While we were unable to complete this experiment due to time constraints, we aim to include this important analysis in the main manuscript of our final paper.

[1] Liu, Yifei, et al. "Revisiting token pruning for object detection and instance segmentation." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

• Visualization of HF/LF tokens

Following your valuable suggestion, we performed an experiment to visualize the selected High-Frequency (HF) and Low-Frequency (LF) tokens on sample images. While we cannot attach figures directly to this rebuttal, we will describe our key findings here and commit to adding these visualizations in the main manuscript. Our analysis confirmed that in early to mid-layers, the token selection aligns well with human intuition: HF tokens predominantly correspond to patches with object boundaries and complex textures, while LF tokens are clustered in smooth, uniform background areas. Interestingly, we also observed 'outliers' where some background patches were selected as HF tokens, which we found often correlates with high-norm features in the embedding space. In deeper layers, we observed that where tokens corresponding to the salient foreground of an object began to be classified as LF tokens. Since rank collapse becomes more severe, the unique signals of even the most important tokens in human perception, get 'smoothed out' and converge toward the low-frequency, DC-like representation of the entire feature map.

• Ablation on Different Image Size

Thank you for this excellent suggestion. Following your advice, we have conducted an extensive ablation study on various image and patch sizes to analyze our method's effectiveness on different token lengths. For these experiments, we used the ViT-21k-finetuned-on-1k model, as it provides pre-trained weights across the various configurations. The results, summarized in the table below, strongly support our hypothesis and demonstrate the remarkable scalability of our approach. (GFLOPS are shown in parentheses).

As you hypothesized, the benefits of our method are most prominent with longer token sequences. The results clearly show that our approach not only improves efficiency but can also enhance performance, especially when dealing with high token counts.

For the high-resolution case (16/384, 576 tokens), our method reduces the computational cost by a substantial 35% (from 55.5 to 35.9 GFLOPS) while also improving the accuracy from 86.6% to 87.0%.This trend is even more pronounced with the longest sequence tested (8/224, 784 tokens), where we achieve a +0.7% accuracy gain while cutting GFLOPS by 36% (from 78.3 to 50.0 GFLOPS).

We thank you for pushing us in this valuable direction. These new experiments clearly demonstrate the effectiveness and scalability of our method. We will add this full ablation study to the final version of our paper.

We thank the reviewer for the feedback. Below, we aim to address you concerns regarding our methodology and experimental validation to further clarify the contributions of our work.

• On Experimental Setup, Baselines, and SOTA Comparisons

We would like to address the concerns regarding the suggested recent works, datasets, and training epochs.

First, we thank you for suggesting the recent papers. However, we would like to clarify that the mentioned works—PYRA and DyT focusing on 'Efficient Task Adaptation,' and OFB on 'Model Compression'—belong to research fields that are orthogonal to our work on 'token reduction for pre-trained models'. These methods aim to improve the efficiency of 'parameter-efficient fine-tuning' or 'compression' via adaptor or network sparsity search, whereas our work concentrates on dynamically reducing the number of input tokens during 'inference'. In fact, the DyT paper itself suggests this complementary relationship by stating, "Then, we explore the compatibility of our method with token pruning methods," indicating that our fields are not direct comparable methods.

As these research fields differ, so do their standard evaluation protocols. As the reviewer noted, papers in those fields often evaluate on the VTAB-1K dataset with 100 training epochs. However, the standard for our subfield—as followed by recetn SOTA methods (e.g., DTEM('24), Zero-Tp('24)) in Table 1 —is 30-epoch fine-tuning on ImageNet-1K. We adhered to this protocol to ensure a fair and direct comparison. Furthermore, we chose DeiT as our primary comparison model among recent SOTA methods, because it is the standard model used in recent token reduction literature. Therefore, we compared the proposed method with recent SOTA methods, in DeiT. The inclusion of the ViT model was not reported in the most of the previous works, but was a necessary choice to validate our core hypothesis and proposed method; that the degree of rank collapse differs depending on the training recipe, thus requiring verification under different conditions. In addition, the practical challenge of some recent works not having publicly available code also factored into our decision.

In conclusion, our reason for selecting ToMe / DynamicViT, EViT as primary baselines is that they represent the two foundational pillars of token reduction: 'merging' and 'pruning'. While most subsequent works are incremental improvements on their ideas, our research introduces a fundamental difference by approaching the problem from a new aspect based on 'frequency'. Our goal was to extensively validate the efficacy of this novel methodology across various models, and we respectfully submit that our experimental design was therefore reasonable and well-justified. We hope this explanation helps to address the reviewer's concerns.

Nevertheless, to further address the reviewer's concern and further demonstrate the effectiveness of our method, we have conducted new experiments with DyT and AdaptFormer. To ensure fairness, we replicated the experimental environment from Table 5 of the DyT paper (using AdaptFormer on VTAB-1K). Since the DyT paper lacks a precise description of its token pruning setup for that table, we followed the original ToMe experiment settings by pruning 13 tokens in each block. Values measured by us are marked in bold; otherwise, they are from Table 5 of the DyT paper. VTAB Accuracy refers to the grouped-mean accuracy, and the parameter count denotes the number of trainable parameters in the backbone.

The result clearly shows that our frequency-aware approach is a effective solution that surpasses existing token reduction methods, even when integrated with the latest SOTA adaptation frameworks. These experiments on VTAB-1K, together with our results on ImageNet-1K and the ADE20K segmentation task (in supplementary materials), clearly demonstrate the robustness and effectiveness of the proposed method across diverse datasets and tasks.

• On the Validity of Mean Filtering

Our use of mean filtering ($A^{LP}$) to isolate low-frequency components is a direct application of a fundamental signal processing principle: a signal's mean is its DC (zero-frequency) component, a quintessential low-pass operation (see Appendix C for a formal proof ). In addition, the mean filter is widely used for a lowpass frequency filter, effectively removing high-frequency content from an image, or token, in our papers [1,2].

While our selection method is computationally simple, its effectiveness is empirically validated in Figure 2, which confirms that our selected high-frequency (HF) tokens are indeed richer in high-frequency content and more critical to model accuracy. Furthermore, its effectiveness is validated by the high fidelity of our aggregated DC token to the original DC signal (Figure 4) and the notable performance drop upon its removal (Figure 5). We believe this strong theoretical and empirical evidence demonstrates the soundness of our approach.

[1] Stanford University. (n.d.). CS279 Notes.

[2] Fisher, R., Perkins, S., Walker, A., & Wolfart, E. (n.d.). Hypermedia Image Processing Reference (HIPR2). University of Edinburgh.

We sincerely thank the reviewer for their thorough feedback and insightful comments. We are encouraged that they found our method to be simple, theoretically sound, and experimentally effective. Below, we address each point raised and believe these clarifications will help strengthen the final manuscript.

• Effects of self-distillation

We adopted self-distillation to follow the standard protocol for a fair comparison. The main prior works we compare against, such as EViT ,Evo-ViT , and DynamicViT, also report their performance using this same self-distillation method. This approach is standard practice in the field.

To address the concern, we ran an additional experiment without any distillation. The results confirm its impact is minimal: the performance DeiT-S w/ distillation (79.8%) and w/o distillation (79.7%). The similar behavior can be observed in the other methods, for instance, DynamicViT w/ distillation (79.3%) and w/o distillation (79.2%).

This ablation study demonstrates that the performance gains stem from our proposed frequency-aware method, not the distillation setup. We will add this analysis to the final paper and thank you for helping us strengthen our work.

• Reduction rates of LF and HF tokens.

We use a fixed reduction rate, to precisely control the computational budget (MACs). This is a standard practice in the field that ensures a fair comparison with prior work. Our method then preserves the top-r high-frequency (HF) tokens.

Our paper's primary contribution is the novel frequency-aware selection criterion, which is orthogonal to the scheduling method (fixed vs. dynamic) . While we are aware of dynamic methods like A-VIT and ATS , our focus was on first validating this new criterion. We agree that a dynamic approach is a promising direction. However, dynamic methods introduce implementation challenges, particularly for batched training and inference, as token lengths can vary per sample. We are actively exploring how to efficiently integrate our criterion with a dynamic scheduler, and if the experiments yield meaningful results in time, we will include them in the final manuscript. This investigation builds upon our initial exploration of optimal reduction schedules via NAS (Appendix F) .

• Dense Prediction tasks

We thank the reviewer for this excellent suggestion. The semantic segmentation results were moved to Appendix G primarily due to the main paper's strict page limits. We prioritized a detailed analysis of our core methodology on the primary benchmark of image classification.

However, we agree that these results are an important demonstration of our method's generality. In the revised manuscript, we will add a summary of the dense prediction results to the main paper.

• Paper Formatting

Thank you for your review and for catching these details. We appreciate you helping us improve the clarity of our paper. We will correct all the formatting issues you pointed out in the final version.