SparseFormer: Sparse Visual Recognition via Limited Latent Tokens

W3. Ablation study on the RoI adjusting mechanism

We have included the ablation study on the RoI adjuting mechanism from two aspects in Table 3 (b) and Table 3 (c) in the submission paper. We repeat these two tables here.

Table 3 (b) shows the effectiveness of the focusing transformer to adjust RoIs and extract features to build 49 tokens, compared with the dense counterparts to produce 49 tokens. All entries in the table use the same cortex transformer configuration. The 'ViT/32' produces 49 tokens simply by patchifying an image into $32\times32$ patches, rather than using the focusing transformer. The 'ViT/32*' add two more transformer blocks to 'ViT/32'. The 'conv×4' uses 4 convolution+relu after the early convolution, where each convolution is with stride 2 and doubled output channel to produce 49 tokens. The 'swin' exploit the shifted local attention used in [4] to produce 49 tokens, similar to the 'conv×4'.

The focusing transformer shows its importance to adjust RoIs to prioritize foregrounds and exclude backgrounds, and therefore enables SparseFormer to perform more accurate recognition with fewer latent tokens, compared with dense counterparts to produce tokens.

Table 3 (c) studies the number of iteration of the focusing transformer to adjust RoIs, $L_f$. The 'nil' stands for the token RoIs are just learnable parameters of the model and do not adapt to different image content. In other words, the RoI for a token does not adjust at all and keeps the same for all images in the 'nil' entry, and as expected, it is with an inferior result. We can see that the iteration number is vital to the final performance, as the focusing transformer with the insufficient iteration may not effectively adjust tokens to foregrounds.

[1] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. ICLR 2023.

[2] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Trans. Mach. Learn. Res., 2022,

[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020.

[4] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. CVPR, 2021

W1. Comparison with token pruning

Thank you for your valuable suggestion. We here compare SparseFormers with one of the most typical and effective token pruning method, ToMe [1], which can be used off-the-shelf to trained models. The throughput is measured on a single NVIDIA A5000 with batch size 32. Note that AugReg [2] models here are pre-trained on ImageNet-21K and we use them as strong baselines.

The table shows that even with ImageNet-21K pre-training, the post-training ToMe with AugReg-B only just matches the same performance as SparseFormer-B, which was trained from scratch on ImageNet-1K. The smaller SparseFormer-T also offers a better trade-off over AugReg-S w/ ToMe.

W2. Why the detection results are not more favorable

This could be because we did not include any form of positional information in the latent token embeddings, as described in Section 4 model configurations. As a result, the attention between these token embeddings is not aware of their positional relationships. Possessing positional information is crucial for achieving optimal detection performance, as validated in the original DETR [3]. The detection transformer pipeline relies on positional information for accurate localization and to suppress redundant detections by utilizing attention between tokens. Meanwhile, the segmentation is simple in sense that we can perform 'classification' on latent tokens that correspond to specific areas in an image, and map this 'classification' back into nearby pixels in the original image space using a simple operator (the location-aware cross-attention we used). Besides, using the exact same SparseFormer for the classification and its weights might be a reason, and we expect that introducing more newly initialized transformer blocks could alleviate this gap.

Thank you for reviewing our paper. The concerns are addressed in the following:

W1. Results in Appendix.1 and generalization to more complex scenarios

We admit that the detection performance of SparseFormer is lagging behind DETRs. We suspect that this is because we do not inject any positional information into latent tokens while the detection transformer training pipeline may require strong positional encodings to localize and suppress nearby redudant detections. Besides, we directly use the SparseFormer architecture for classification (without the introduction of extra transformer blocks) and its pre-trained weights to perform object detection, and this may not be the optimal design.

We agree that the sparse attention patterns might limit the long-term and precise positional dependency, which is important to the object detection task. However, this does not mean that SparseFormer cannot handle complex scenarios. As a fact, the SparseFormer segmentation model performs well on much more complex ADE20K scenarios in Appendix. 2. This is because SparseFormer follows a spatially divide-and-conquer manner when dealing such scenarios, where the location-aware cross-attention operator enables a token to only respond to a specific area.

W2. Concerns about RoI localization on dense prediction tasks

We believe that the SparseFormer does not require the strong explicit spatial RoI localization supervision on dense prediction tasks. Indeed, for segmentation tasks, the spatial distribution of classification labels can serve as a coarse localization supervision. A token RoI in SparseFormer can be adjusted to focus on a consistent label area by such spatial label distributions and the end-to-end location-aware cross-attention discussed above. Besides that, the segmentation task does not necessitate well-localized token RoIs to perform well since the final classification is performed on the dense map, which is projected back from latent tokens and one pixel in the dense map is a mixture of nearby latent tokens with the semantic similarity.

We greatly appreciate your positive rating of our paper. Thank you for your reviewing and assessment.

Thank you for your reviewing our paper. We will address your concerns in the following:

W1. Similarity to Perceiver and DeformableDETR architectures
First, we would like to emphasize again that the aim of SparseFormer is to build a sparse architecture for visual recognition tasks, and many efforts of this paper are put on how to perform visual recognition with much fewer tokens in a transformer-like architecture. This distinguishes SparseFormer from either Perceiver or DeformableDETR, where both latters do not attempt to reduce the computation.

Difference with Perceivers

We agree with that the primary distinction lies in the focusing transformer since its introduction enables SparseFormer to perform accurate recognition with a highly limited number of tokens. From Table 3(b) in the paper, where illustrate how 49 tokens are produced, one can see the effectiveness of the focusing transformer comparing with dense counterparts.

As noted in the paper, Perceivers use a naive cross attention layer to traverse the pixel or feature space in a dense manner $\mathcal{O}(H\cdot W\cdot C)$, where the focusing transformer in our SparseFormer extract a few tokens from an image in a sparse way $\mathcal{O}(N\cdot P\cdot C)$, regardless of the input height and width. Also, the size of latent space is also sparse, the typical Perceiver use 512 tokens with 1024 dimension, resulting in a total 512*1024 capacity, which is more than $8\times$ the SparseFormer-B uses (81 tokens with 768 dimension).

Difference with DeformableDETR

Building a sparse architecture necessitates a sparse variant of the basic operators on the image space. We must admit that DeformableDETR and SparseFormer share the similar idea to use bilinear interpolation to extract image features, but the motivation and details differs a lot: DeformableDETR aims to achieve the shift invariance in detection transformers (DETRs), where the naive cross and self attention layers in DETRs struggle to handle, while SparseFormer uses bilinear interpolation to efficiently sample image features. Also, deformable attention in DeformableDETR is typically performed on the highly semantic feature map (e.g., after ResNets) and in a multi-scale manner, while SparseFormer uses a single-level early conved feature map to perform sampling in the very beginning of the network. From this view, SparseFormer can be seen as a decoder-only architecture for visual modeling except for the early convolution, which much differs from DeformableDETR that requires deformable self attention encoders on dense units upon the backbone.

W2. Dense prediction tasks and information loss

SparseFormer can perform dense prediction tasks like object detection and semantic segmentation. We have included experimental results and discuss them in Appendix A.1 and Appendix A.2. Specifically, Table 7 in the Appendix shows the detection results on MS COCO on a simple SparseFormer-S with only an extra linear classifier head and an extra MLP for regression, and the training recipe and loss also following the DETR. Given that no positional information is injected into latent tokens and no multi-scale features are used, SparseFormer can still achieve meaningful detection results with about $0.3\times$ FLOPs. Table 8 in the Appendix illustrates the segmentation performance of SparseFormer on ADE20K. One can see that SparseFormer reaches the result on par with the dense Swin-T w/ UperNet with $0.18\times$ GFLOPs.

We deeply understand your concern about information loss introduced by SparseFormer since the latent token capacity (e.g., 81*768) is indeed much smaller than the image size (e.g., 3*224*224). However, we believe that this is not a significant issue for visual recognition tasks. Natural images contain a lot of pixel-level redundancy, such as backgrounds, which contributes little the final recognition. In fact, our sparse paradigm is exactly based on this redundancy and the proposed SparseFormer learns how to discard the redudancy, or how to perform 'information loss', with the recognition supervision. As for generation tasks like with diffusion models, this information loss issue becomes somewhat critical, and it is appealing to investigate how to incorporate SparseFormers into generation models with the least information loss. However, this is beyond the scope of this paper.

We greatly appreciate for your kind reassessment on our work. Till now, we have not managed to deal with the type conversion bug (the issue is already mentioned by others due to customized ops, but not solved, https://github.com/pytorch/TensorRT/issues/2113). We will report FP16/INT8 in future revisions once this bug is resolved.

Regarding the novelty, we agree that our method follows the same idea of indexing feature maps based on query-associated coordinates with Deformable-DETR and its follow-ups. But the important thing is, that we show this query coordinate feature map indexing scheme can lead to sparse vision transformers with much fewer tokens, which is not explored at all to our best knowledge. Therefore, we believe that our SparseFormer per se has it novelty in exploring sparse vision architectures.

We are extremely grateful for your valuable and detailed suggestions on how to enhance the impact, the comprehensiveness, and the applicability of our work, especially for results in Table 8. In future versions, we will reorganize this paper and cover as much experiments as possible per your suggestions.