SCFormer: Spatial Coordination for Efficient and Robust Vision Transformers

In Sec.4.4, the authors provide detailed roadmap from baseline to proposed method. However, it mainly focuses on the macro design of backbone. Micro-design ablation study (e.g., SCD or pixel gating in SCA module, kernel sizes of ILR and the pooling kernel in SCA ) are also needed.

i. This work only conducts experiments on large-scale datasets. How is the performance on the small-scale datasets, such as CIFAR-10, and Tiny ImageNet, by training from scratch or finetuning on the pretrained weights from ImageNet?

ii. Is the proposed network effective on multi-modal applications, such as CLIP [1] and LLaVA [2]?

iii. Which kind of pooling layer, max pooling, average pooling, or another variant, is used in the SCD? Was an ablation study made to make this choice?

iv. In the SCA module, how are tokens from pooling descriptors with different areas aligned along the channel dimension, given that they produce different numbers of tokens? Could you provide more details on the implementation of this alignment process and its impact on the model's performance?

[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PMLR, 2021: 8748-8763.

[2] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. Advances in neural information processing systems, 2024, 36.

Major: The manuscript lacks in-depth analysis of how ILR and the SCA improve robustness.

1. In the introduction, key arguments need more grounding than referencing related works. For example, the statement “robustness in lightweight architectures is closely tied to channel width, with wider channels offering greater capacity to capture diverse spatial features, such as textures and frequency patterns” is overly vague and lacks detailed justification or clear evidence.
2. Section 3 focus on the design of modules without discussing how they address the robustness challenges.

Minor:

1. The introduction is unclear in its use of terms like “lightweight ViT” and “hybrid ViT” (Lines 47-53), which are used interchangeably. This section would benefit from refinement to clarify the paper's focus.
2. The Inceptional Local Representation (ILR) block appears incremental, as inception-inspired methods are well-established. While the ILR provides practical improvements, these modifications lack novelty.
3. Robustness evaluation is restricted to image classification and should be extended to more diverse tasks, such as object detection, to better substantiate SCFormer's robustness claims across applications.

1.In the paper, it is unclear why Spatial Condition Deconstruction works, the experiment part can not provide support. Compared to the attention block in Restormer[1], the SCA seems additionally conduct a multi-scale enhancement when generating Query and Key. If not conduct SCD, Matmul(Q,K) also achieves interaction between all the tokens. From this perspective, the novelty of SCA is limited and its function is unclear. It suggested to illustrate why process the features at these three shapes in the SCD. More experiments and analysis are welcomed. [1] Restormer: Efficient transformer for high-resolution image restoration. 2.The operations to enhance the feature diversity in attention-based models are well explored, such as Local Context Enhancement module in [2]. The contribution of ILR is limited. It is suggested to discuss the main difference and potential advantages of ILR compared to the existing local enhancement methods. What is the effect of other enhancement manner to the SCFormer? [2] Biformer: Vision transformer with bi-level routing attention.
3.Some related methods [3-4] are missed for comparison. It is suggested to include TransNeXt[3] and RMT[4] in the comparison. As reported in [3], TransNeXt-T achieves better performance compared to SCFormer-L including IN1K,IN-A. A comprehensive comparison of all relevant indicators and datasets can bring a clearer understanding of the performance of SCFormer compared to these state-of-the-art methods. [3] TransNeXt: Robust Foveal Visual Perception for Vision Transformers. CVPR 24. [4] RMT:Retentive Networks Meet Vision Transformers. CVPR 24. 4. The method section should focus more on the core contribution. More content about the function of SCA need to be added. The content for Pixel Gating, ILR and SFFN can be reduced. It is suggested write a brief introduction of the three components and present details in the appendix.

The paper does a good job on the efficiency side highlighting relevant works in the literature and comparing to them. However, on the robustness side that part is lacking. There is no treatment on how robustness has been addressed in the literature nor comparisons to more relevant works.

The paper ove rfocuses on efficiency and does not compare to actual robust methods. Robustness is barely considered in the related work or experiments. There have been a large number of efficient architectures, and some of them are discussed in the paper. There are also many papers focused on robustness, particularly understanding the benefits of transformer architectures for zero-shot noise. Those are entirely omitted in the paper. There are also recent papers explicitly addressing the trade-off between robustness and efficiency. For instance:

• Zhao et al. Fully Attentional Networks with Self-emerging Token Labeling ICCV 2023 (Robustness in transformers)
• Bair et al. Adaptive sharpness-aware pruning for robust sparse networks ICLR 2024 (Efficient-Robustness Trade-off for CNN)
• Zhou et al. Understanding The Robustness in Vision Transformers. ICML 2022.

For robustness, a quick look to Imagenet-A and imagenet R results at ICCV2023 gives STL 38.2 and 51.8 respectively while the current submission yields 27.9 and 48.7 respectively. Please, clarify