Connectivity-based Token Condensation for Efficient Vision Transformer

• Unclear motivation: Based on the introduction and results, it is not clear to me which is the main gap that the paper tries to adress:

  • For instance, when comparing to other token pruning methods, the introduction states that "only discarding tokens will incur performance loss as even those less important tokens" -> however, one could argue that merging tokens may also result in loss of information. Furthermore, there are already existing methods such as "EViT: Expediting vision transformers via token reorganizations, Liang et al, ICLR 2022" which proposes to fuse all pruned tokens into one token, rather than discarding them.
  • Similarly, about token merging methods, the introduction mention that "They also overlook the potential influence of other tokens on the relationships between two tokens when analyzing their interactions" -> However, when looking at the ablation experiments in Table 8, it seems that using a connectivity factor of $K = 1$ actually yields better performance than most of the higher connectivity values $K \in \{4, 8, 32\}$. This would indicate that higher-order relationships between tokens are not so meaningful for token pruning. Similarly, from the ablation in Table 3, it seems that the dominance score (i.e. the same attention to class token used in other token pruning methods) has a more significant impact on accuracy than connectivity.

• Missing training hyperparmeters: The experimental section is not very clear on the training setup of baselines. In particular, on whether (i) all baselines are trained from scratch or finetuned, (ii) how long they are trained for and (iii) whether they include distillation or not. Because of this and the fact that the accuracy gap is generally small, it is hard to interpret the results from Table 1; Furthermore, many results seem to directly come from their respective original paper, which might have a different training pipeline; To give some examples of potential different training pipelines:

  • For DeiT-S The results reported for EviT (79.5% accuracy at 3.9GFlops) corresponds to the version trained from scratch for 300 epochs without distillation. If finetuned from a pretrained model, the original work reports 79.8% accuracy. Similarly for ToME, the results seems to correspond to training from scratch from the original paper.
  • According to the original paper, HeatViT also applies 8-bit quantization to weights/activations after token pruning, so the accuracy might not be directly comparable.

• The writing should be more concise.
• The novelty may not be sufficient for this conference. Although the framework has been proposed earlier, the main novelty comes from the CONNECTIVITY-BASED metric. A more detailed analysis of this metric should be provided, and a stronger explanation for how it captures "high-order relationships between tokens" is needed.
• In Equation 2, the dot product convention is denoted by \cdot, making further explanation redundant.
• The related work section is loosely written. The author should include several efficient ViT works following the same "selection then fusion" framework (e.g., [1]). The significant differences between these approaches should be highlighted.
• There is a lack of justification for the generalization of connection-based metrics for tasks other than classification.

[1] https://arxiv.org/abs/2304.10716

• The authors conducted several ablation experiments to validate the design choices mentioned in Section 3, but there are no direct comparisons between different token fusion strategies. As mentioned in the abstract, the authors claim that previous methods "do not consider carefully how to select tokens". To support this claim, I think it is necessary to provide direct evidence instead of the final performance of the whole system to show the superiority of the proposed designs. For example, ToMe (Bolya et al., 2023) presents a token fusion solution with bipartite matching. The method can achieve 79.3% with 2.7 GFLOPs without training based on AugReg ViT-S (Table 4). If the method can better select and fuse tokens, will the proposed method achieve better performance? Since obtaining the results doesn't require training, the comparisons can also reduce the effects of training techniques and settings.

• Why are there some missing numbers of FLOPs or throughput in Table 1 and 6? Are these results better than the proposed method? The code and models of most of these methods like ToMe and DynamicViT are publically available and widely used in previous work. I think it is easy to obtain the throughput of these models. It is also strange that you cannot obtain FLOPs of Evo-ViT while can measure the throughput of this model.

• Several recent token fusion/pruning methods [r1,r2,r3,r4] have extended this group of methods to more complex scenarios like convolution backbones and dense prediction tasks. Only conducting experiments on ImageNet based on DeiT and LVViT may make the work less competitive.

[r1] Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers, CVPR 23 [r2] ToMe (Bolya et al., 2023) [r3] Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks, TPAMI 23 [r4] Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation ICCV 23

The proposed method appears to be the 'select and fuse' type, which seems to require more computation than the existing methods, and I can't figure out at which part the speed improvement has been made. I can't find how many percent of the tokens were pruned finally by selecting and fusing the tokens compared to other models. It would be better to report the token reduction ratio and compare it with those of SoTA methods.

Minor points

presentation and writing

• In the Introduction, the explanation of the method proposed in the last paragraph seems to be cut off as if it were interrupted. For completeness, It needs to explain the matching and fusing part in the rest of the part.
• In page #7, “Distillation” paragraph is located among tables, which looks bad.