Alias-Free ViT: Fractional Shift Invariance via Linear Attention

We thank the reviewer for their endorsement and constructive feedback. We are happy the reviewer found our work clearly explained and well-motivated. In the following, we address the questions that have been raised. We will incorporate these clarifications and additional results into the paper (or its appendix).

It would be good to include a vanilla ViT as an additional baseline or reference to illustrate the gains from linear attention and lack of invariance to translation for the dominant ViT architecture.

We thank the reviewer for this suggestion. We find that vanilla ViT is indeed less robust to crop shifts than the hybrid models, and our AFT in particular. For example, ViT-Base adversarial accuracy for crop shifts with max-shift 3 (see Figure 2a) declines from 80.4 to 71.7 vs. 80.0 to 76.4 in our model. We will include complete results in the revision of the paper.

The proposed method comes with several tradeoffs. First, compared to APS the method achieves lower test accuracy for both the nano and small XCiT architectures. Second, the method achieves slightly lower integer shift consistency, particularly for the nano architecture (Table 1).

Table 1 had an issue which gave APS an unfair advantage, but we already corrected this issue in Section B of the SM (Table 4). Specifically, upon the main paper submission, we detected an error in our implementation of the APS model (in the results presented in Table 1 and Figure 2), leading to training with an effective patch size of 8 instead of 16, as was used in the Baseline and AF models. This led to better accuracy for the APS model and an unfair comparison vs. the other methods.

The corrected results in Table 4 (in the SM) show that the APS model has lower accuracy than the AF model (APS 68.7 vs. AF 70.6 in the Nano versions and APS 80.1 vs. AF 80.7 in the Small versions). Consequently, the APS and AF models have similar adversarial accuracy under the integer grid (APS 68.7 vs. AF 68.9 in the Nano versions and APS 80.1 vs. AF 80.0 in the Small versions). Notably, the AF model has better adversarial accuracy under the half-pixel grid, with a larger margin than reported in Table 1 (APS 62.9 vs. AF 69.2 in the Nano versions and APS 76.0 vs. AF 79.9 in the Small versions).

Third, and perhaps most important, is that the method comes with a much higher training cost (509 hours versus 303 for APS and only 77 for the baseline XCiT small model). This is a significant cost that must be clearly highlighted to make the trade-off clear to readers, particularly in the abstract, which seems to ignore these important trade-offs.

Indeed, the proposed model currently comes with a high additional computational cost. As mentioned in the discussion, we believe this issue may be alleviated by carefully optimizing components that are currently not commonly used within neural architectures. For example, [A] reports a custom CUDA kernel for their alias-free activation function, which uses an approximation of sinc-interpolation upsampling, yielding a 10× training speedup. Nevertheless, we will make sure this trade-off is clearly noted.

One other suggestion to bolster the experiments is to consider also adding object detection and segmentation experiments to further validate the proposed architecture, as was done in prior work such as XCiT.

We thank the reviewer for the suggestion. We also find it intriguing to implement translation-robust detection and segmentation models using the anti-aliasing approach. XCiT is indeed used for detection and segmentation as a backbone to Mask R-CNN. However, this includes additional modules that are not trivial to convert to be fractional-shift equivariant under our framework (for example, RoI Align, which uses bilinear interpolation). Therefore, we leave this as an interesting direction for future work and focus this paper on classification tasks.

I acknowledge resource constraints may not make this possible and believe the current experiments on their own are certainly interesting, but I would be curious what happens when the both the model and training data is scaled (to say ImageNet-22k or larger). The original ViT paper for example demonstrated the performance of the transformer architecture in vision can change significant with scale (and that ViT is generally data hungry). If the authors are resource constrained, I would recommend at least noting the scale of the experiments (model sizes and training data size) as limitations.

Unfortunately, we were unable to scale the size of the experiments due to limited resources. We will mention this in the limitations section of the camera-ready version.

Can you include additional figures for what the natural shifts used in Figure 2 are? It would be great to show for example an original image and transformed variants in contrast to the fractional / integer shifts that the authors present as more artificial. This would help illustrate to readers the benefit of the method as well.

We thank the reviewer for this suggestion. Unfortunately, we are unable to attach figures in the rebuttal, but we will include a visual example in the revision. We kindly refer the reviewer to visual examples in [B], where Figure 6 demonstrates circular shift and crop shift, and Figure 5 demonstrates bilinear fractional shift. Note that, unlike in Figure 5, we interpolated the translated images while leaving an edge of one pixel instead of padding with zeros, to avoid any edge artifacts

[A] Alias-Free Generative Adversarial Networks, NeurIPS, 2021.

[B] Alias-free convnets: Fractional shift invariance via polynomial activations. arXiv, 2023.

We thank the reviewer for their constructive feedback. We are happy the reviewer found our work well-motivated. In the following, we address the concerns that have been raised. We will incorporate these clarifications into the paper (or its appendix).

Based on [A], this work seems to be incremental. Even though this paper focuses on transformer architecture, there are many insights originating from [A], e.g., MLP, layer normalization.

Indeed, many shared components between ViTs and convnets were already “solved.” Therefore, this work is focused on the attention mechanism, as this is the main difference between the two architectures. We believe this work may also have a broader impact, as the attention mechanism is commonly used in many architectures. For example, [B] used existing alias-free components to improve shift equivariance, yet the attention mechanism was overlooked (i.e., it was not modified and, therefore, was not alias-free).

Moreover, we report in the paper additional methodologies and findings that did not originate in [A]. For example, we found that we need to remove the positional encoding and class attention, and to use GeLU instead of polynomial activations in the patchifier. Also, we found that the AF-LayerNorm performs better in the AFT compared to the AFC (in contrast, in AFC it was the main cause of performance degradation). Combining all these small modifications and insights was critical to achieving good performance, and their discovery required careful and extensive experimentation.

The experimental results are not enough to illustrate the superiority of Alias-Free ViT. For example, the authors only evaluate XCiT, while there are many ViT variants. Also, the baselines (only APS) are not sufficient to make the result convincing.

We assume this comment relates to Table 1, where we compare three variants (Baseline, APS, and AF (ours)) of XCiT, as this is our baseline model. The main idea of this table is to show that: (1) our model maintains classification accuracy, and (2) it is almost perfectly robust to circular shifts, as theoretically claimed. We compare our model to other ViT variants (Swin and CvT) in Figure 2, where we evaluate “standard shifts” that better represent practical cases. Note that common models usually perform worse on circular shifts than on standard shifts, as shown in [C].

We are unaware of any other relevant baselines (i.e., techniques to improve vision transformers’ translation robustness) with public implementation, other than [C] and [D], represented by APS. We would be happy to add additional baselines to the camera-ready version in case we have missed any.

As shown in Table 1, the Alias-Free version of XCiT cannot beat XCiT-AFS in 3 aspects out of 5.

Table 1 had an issue which gave APS an unfair advantage, but we already corrected this issue in Section B of the SM (Table 4). Specifically, upon the main paper submission, we detected an error in our implementation of the APS model (in the results presented in Table 1 and Figure 2), leading to training with an effective patch size of 8 instead of 16, as was used in the Baseline and AF models. This led to better accuracy for the APS model and an unfair comparison vs. the other methods.

The corrected results in Table 4 (in the SM) show that the APS model has lower accuracy than the AF model (APS 68.7 vs. AF 70.6 in the Nano versions and APS 80.1 vs. AF 80.7 in the Small versions). Consequently, the APS and AF models have similar adversarial accuracy under the integer grid (APS 68.7 vs. AF 68.9 in the Nano versions and APS 80.1 vs. AF 80.0 in the Small versions). Notably, the AF model has better adversarial accuracy under the half-pixel grid, with a larger margin than reported in Table 1 (APS 62.9 vs. AF 69.2 in the Nano versions and APS 76.0 vs. AF 79.9 in the Small versions).

[A] Alias-free convnets: Fractional shift invariance via polynomial activations. CVPR, 2023.

[B] Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space. CVPR, 2025

[C] Making Vision Transformers Truly Shift-Equivariant. CVPR 2024

[D] Reviving Shift Equivariance in Vision Transformers, arXiv 2023

We thank the reviewer for their endorsement and constructive feedback. We are happy they found our work novel, practical and useful, well-motivated, clearly presented, and supported by solid experimental results. In the following, we address the questions that have been raised. We will incorporate these clarifications and additional results into the paper (or its appendix).

Were you able to reproduce your findings on other datasets?

We evaluated our model solely on ImageNet-1K, as this is the only classification benchmark reported in our baseline model paper (XCiT [A]). This allowed us to avoid expensive hyperparameter tuning. In [A], the authors report results on additional detection and segmentation datasets, but these require training Mask R-CNN, which includes modules that are not trivial to convert to be fractional-shift equivariant under our framework (e.g., RoI Align, which uses bilinear interpolation). Therefore, we leave this as an interesting direction for future work and focus this paper on classification tasks.

In response to the reviewer’s request, we evaluate our model on additional classification datasets. We fine-tune the models reported in the paper (pre-trained on ImageNet-1K) following the regime of [B]. Note that the ImageNet pre-training regimes of [A] and [B] are identical. For each dataset, we report the accuracy, integer shift consistency, and half-pixel shift consistency, as in the left part of Table 1 in the paper. Although the hyperparameters may not be optimal, the AF models maintain competitive accuracy while exhibiting near-perfect shift consistency. Note that we do not report results for the XCiT APS models, as they diverged with the current hyperparameters.

How exactly was HP tuning performed?

We refrained from performing full hyperparameter tuning due to limited resources. Thus, we retained the same hyperparameters as the XCiT paper (reported in Appendix C), with the following exception: We observed instability in training the AF models, which was resolved by reducing the batch size (and consequently the learning rate, scaled with the same ratio, as in [A] and [B]). We trained all Nano models (Baseline, AF, and APS) using batch sizes of 1024 and 512, and chose the best-performing configuration for each. We then used the same configuration for the Small versions of each model (i.e., 1024 for the baseline and 512 for APS and AF). Below, we report the accuracies for each of the Nano models under these batch sizes.

[A] XCiT: Cross-Covariance Image Transformers, NeurIPS 2021.

[B] Training data-efficient image transformers & distillation through attention, ICML 2021.

We thank the reviewer for their endorsement and constructive feedback. We are happy that the reviewer found our work novel. In the following, we address the concerns that have been raised. We will incorporate these clarifications and additional results into the paper (or its appendix).

The proposed model can only outperform existing models when evaluated on fractional translations. However, based on the results reported in Table 1, the difference decreases substantially for the larger model, raising the question of relevance on even larger ViTs.

Unfortunately, we were unable to evaluate our method on larger models due to limited resources. However, in the following, we evaluate the shift consistency of larger variants of the baseline model (XCiT [A]), as in the third and fourth columns of Table 1 in the paper. Seemingly, the improvement in shift consistency by increasing model size is limited and never reaches the consistency levels of our AF model (>98).

Fractional shifts are evaluated synthetically, e.g., using bilinear interpolation, which may not be representative of fractional shifts observed in real-world distributions. However, it is noted that this is a general problem within this domain and not particular to this work.

Indeed, “natural translations” (e.g., due to physical camera movements) are hard to simulate precisely given a pixel representation of an image.

[A] XCiT: Cross-Covariance Image Transformers, NeurIPS 2021.