When Semantic Segmentation Meets Frequency Aliasing

Thank you for the valuable feedback and encouragement.

W1, Q1, Q2: More visualizations

[Authors’ response]

We acknowledge the importance of visualization in elucidating how the proposed methods operate and rectify errors. In response to your valuable advice, we have included comprehensive visualizations and analyses in the revised appendix. As the figure cannot be added in the comment box, we recommend referring to Section J of the revised appendix for a detailed visual representation. Section J.1 discusses how the FAD improves the feature during downsampling (Figure 5), Section J.2 discusses how the FreqMix suppresses unnecessary high frequency in the background and object centers (Figure 6), and Section J.3 provides a more visualized analysis of how the FAD and FreqMix improve the feature and prediction while relieving three types of errors (Figure 7). Section K provides additional frequency analysis of features (Figures 8 and 9). For your convenience, we have also attached our description of the visualizations below.

In Figure 7 of the revised appendix, we randomly selected image patches with a high aliasing score from the Cityscapes dataset validation set. We observed that aliasing leads to a jagged phenomenon[1,2], disrupting object shapes and boundaries, resulting in false responses (see Figure 7(b) and (l) in the revised appendix) and displacement errors (see Figures 7(c), (d), and (e) in the revised appendix). DAF and FreqMix relieve this phenomenon and improve the feature representation thus resulting in lower false responses and displacement errors.

Besides, when high-frequency information is aliased, it transforms into false low-frequency information. For example, when two objects are close to each other, their high-frequency boundaries can be aliased to lower frequency during downsampling, causing the two objects to appear connected and their boundaries to be merged. This leads to merging errors (see Figures 7(a), (g), and (j) in the revised appendix). DAF/FreqMix solve this by removing/suppressing these high frequencies during downsampling/encoder block, leading to lower merging errors.

Moreover, high-frequency components in the object center or background can result in false responses (see Figures 7(i) and (k) in the revised appendix). FreqMix addresses this issue by suppressing the high frequency in the object center or background while preserving the high frequency at the boundaries (see Figures 6(a) and (b) in the revised appendix).

Thanks for your valuable comments that help us improve our manuscript and clarify the proposed method more clearly!

[1] Qian S, Shao H, Zhu Y, et al. Blending anti-aliasing into vision transformer. Advances in Neural Information Processing Systems, 2021, 34: 5416-5429.

[2] Zou, X., Xiao, F., Yu, Z., Li, Y., & Lee, Y. J. (2023). Delving deeper into anti-aliasing in convnets. International Journal of Computer Vision, 131(1), 67-81.

Thank you for the valuable feedback!

W1: Time complexity analysis

[Authors’ response]

Here, we conduct a comprehensive evaluation of the time complexity during both the training and testing phases. The results are presented below. We use a batch size of 16, a resolution of 512$\times$512, and train for 40k iterations on a single RTX 3090 for evaluating training time. The test time is evaluated on 1024$\times$2048 image patch with a batch size of 1.

The inclusion of DAF and FreqMix increases the training time from approximately 4.2 hours to around 5.9 hours. Similarly, it leads to an increase in inference time from approximately 45ms to around 82ms. Notably, the majority of this increase is attributed to the FFT/iFFT operations. When these operations are excluded, the inference time is reduced to 49ms. This gap is primarily due to the limited speed optimization in the engineering implementations of frequency transformations, such as FFT/iFFT. This phenomenon has also been observed in previous works [1]. This issue can be resolved by better engineering implementations of frequency transformations.

[1] Huang, Z., Zhang, Z., Lan, C., Zha, Z. J., Lu, Y., & Guo, B. (2023). Adaptive Frequency Filters As Efficient Global Token Mixers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6049-6059).

W2: Experiments on COCO and Pascal VOC

[Authors’ response]

Following your advice, we present additional quantitative results on the PASCAL VOC and COCO datasets, encompassing semantic segmentation, object detection, and instance segmentation tasks. On the PASCAL VOC, the proposed method demonstrates improvements in semantic segmentation results, with an increase of +1.8 mIoU, +2.5 BIoU, +2.1 BAcc, and a notable reduction in displacement error (DErr) by 2.5.

On the COCO, our proposed method enhances the average precision (AP) of object detection boxes by +1.6. Regarding instance segmentation, it results in a corresponding boost of +1.4 in box AP and +1.3 in mask AP. These results demonstrate that the proposed method consistently leads to improvements by addressing the aliasing degradation, thereby verifying the generalization of our approach. They also highlight that the aliasing degradation widely exists in many fundamental computer vision tasks and needs to be urgently resolved.

We thank the reviewer for encouraging us to fully verify the effectiveness on various datasets, making our work more solid.

Q3: Incorrect figure reference.

“Aligning with the observations in Figure 1 that false responses and merging mistakes predominantly exist in areas with relatively low aliasing scores.” — Could you clarify how we are associating aliasing scores with a region of an image?

[Authors’ response]

Thank you for your careful comments. The reference to the figure was indeed incorrect. We have rectified this to "Align with the observations in Figure 2, where false responses and merging mistakes predominantly exist in areas with relatively low aliasing scores."

In Table 1 of the main paper, we provide evidence indicating that increasing the blur kernel leads to a reduction in the overall aliasing score. Consequently, there is a decrease in displacement error, while the occurrences of false responses and merging mistakes exhibit an upward trend. This aligns with the statistical findings presented in Figure 2 of the main paper, illustrating the distribution of three errors. In this figure, we observed that the aliasing score of false responses and merging mistakes is lower than that of displacement error, providing consistent support for our observations.

We also have included comprehensive visualizations and analyses in the revised appendix (Figure 7 in the appendix). As the figure cannot be added in the comment box, we recommend referring to Section J of the revised appendix for a detailed visual representation.

Q4: How does aliasing lead to a loss of object response

[Authors’ response]

Thank you for your inspiring comments. We provide visualizations in Figure 5 of the revised appendix. As the figure cannot be added in the comment box, for your convenience, we have also attached our description of the visualization below. Please refer to Section J and Figure 5 of the revised appendix for a detailed visual representation.

In Figure 5 of the revised appendix, we have marked areas with high aliasing scores in the features with red boxes. Owing to the aliasing, the downsampled features exhibit a severe jagged phenomenon[2, 3], resulting in the degraded representation of object shapes and boundaries (see Figures 5(c) and (e) in the revised appendix). When downsampling increases to 4×, the response of some objects fades or is lost (see Figure 5(e) in the revised appendix). It is noteworthy that widely used state-of-the-art models, such as Swin Transformer and ConvNeXt, adopt a total downsampling stride of 32$\times$, potentially leading to an even more severe loss of response. After applying the proposed DAF to address the aliasing artifact, the jagged phenomenon is relieved (see Figures 5(d) and (f) in the revised appendix) and the response of objects still remains strong in the 4× downsampled features (see Figure 5(e) in the revised appendix). This indicates a strong correlation between the aliasing effect and a loss of object response.

Recent work [2] also observes a similar phenomenon, where an increase in image noise raises the high frequency in features (leading to aliasing), resulting in deeper features losing the response of interested objects. Applying a low-pass filter (which can be regarded as an anti-aliasing filter) can alleviate this problem and recover the response of interested objects.

[2] Qian S, Shao H, Zhu Y, et al. Blending anti-aliasing into vision transformer. Advances in Neural Information Processing Systems, 2021, 34: 5416-5429.

[3] Zou, X., Xiao, F., Yu, Z., Li, Y., & Lee, Y. J. (2023). Delving deeper into anti-aliasing in convnets. International Journal of Computer Vision, 131(1), 67-81.

[4] Chen, L., Fu, Y., Wei, K., Zheng, D., & Heide, F. (2023). Instance Segmentation in the Dark. International Journal of Computer Vision, 1-21.

Q1: Discussion about aliasing in a transformer-based architecture.

[Authors’ response]

I sincerely appreciate your encouragement and valuable feedback. I would like to further discuss the question of whether aliasing remains a concern in a transformer-based architecture.

We think aliasing is still a problem in existing transformer-based architecture. Taking the renowned Vision Transformer (ViT) as an example, ViT [1] tokenizes images by splitting them into non-overlapping patches, which are then fed into transformer blocks. The tokenization and self-attention operations performed on these discontinuous patch embeddings can be viewed as downsampling operations, introducing a potential side effect of aliasing. It is essential to note that this downsampling operation is virtually unavoidable due to the spatial redundancy nature of the image [2] and huge computational costs without downsampling (increasing by 256$\times$ without downsampling in ViT).

Several existing studies have acknowledged this concern. A straightforward solution to alleviate aliasing is to increase the sampling rate. Similar trends are observed in vision transformers, where the use of overlapped tokens [3] and smaller patch sizes [4] contributes to improved performance. However, escalating sampling rates incur quadratic computational costs. Consequently, I hypothesize that integrating appropriate anti-aliasing filters into the 'attending' process could offer a viable solution. In fact, existing work has empirically explored blending anti-aliasing filters into the vision transformer, reporting observed improvements[5].

In conclusion, aliasing persists as a potential concern in transformer-based architectures, and prior studies have endeavored to address it empirically by enhancing the sampling rate or integrating anti-aliasing filters into the attention mechanism. Our work represents a step forward in quantitatively assessing and addressing aliasing in contemporary models, supported by theoretical foundations. This issue still presents ample opportunities for further investigation and exploration.

We appreciate your insightful questions that delve into the depth of our work and highlight its broad prospects.

[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.

[2]He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. Masked autoencoders are scalable vision learners. In CVPR, 2022

[3] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. in ICCV, 2021.

[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers.in ICCV, 2021.

[5] Qian S, Shao H, Zhu Y, et al. Blending anti-aliasing into vision transformer. Advances in Neural Information Processing Systems, 2021, 34: 5416-5429.