UNIP: Rethinking Pre-trained Attention Patterns for Infrared Semantic Segmentation

Thanks for your effort in reviewing our paper and giving detailed suggestions. We are very pleased to receive such high praise for our work, including the motivation, writing, methods, and experiments. We sincerely appreciate it! We hope the following responses can solve your concerns.

Q1: About the title.

Thanks for your valuable suggestion. We modify the title from "Semantic Segmentation" to "Infrared Semantic Segmentation" in our revised draft. Additionally, the NMI-HAD and LL-FPN components of our method are also applicable to other image modalities, such as RGB. For further details on this aspect, please refer to our response to Reviewer FdMz, specifically in Q1.

Q2: About the abbreviations.

Thanks for your detailed suggestion. We recognize that there is a lexical error. Given that our research encompasses both supervised and self-supervised pre-training methods, we replace "SSL" with "pre-training" in line 146 of the revised draft and include citations of both supervised and self-supervised approaches.

Q3: About the unaligned letters in Fig. 8.

Thanks for your meticulous finding. We correct this typo in the revised draft.

Q4: About the Ethics Review.

Our constructed dataset InfMix is derived from the extraction and integration of 25 publicly accessible RGB and infrared datasets. These datasets are available for public use. During the extraction process, we employ data deduplication methods only, ensuring that no personal biases were introduced. We adhere to all usage requirements of the original datasets and use them solely for scientific research purposes. Importantly, the dataset does not include any additional metadata or labels that could be used to identify individuals. Detailed information regarding the composition and construction process of our dataset is provided in Appendix E of our paper. In accordance with the usage agreements of the original datasets, when publicly sharing our dataset, we will only release the index list corresponding to the images we utilized, without redistributing the original datasets.

We would be more than happy to discuss any further questions!

Q4: Comparison between independent training and joint training.

Thanks for your valuable question. We conduct experiments involving a two-stage training process using MAE-Large as the teacher model. In the first stage, the model is distilled using the RGB component of InfMix, i.e., the subset of ImageNet and the training set of COCO. In the second stage, the model is subsequently distilled using the infrared component of InfMix, i.e., the constructed InfPre dataset.

As indicated in the table below, benefiting from the hybrid pattern distillation, the model of the RGB training stage surpasses MAE-Small by a large margin. After the infrared training stage, the model's average segmentation performance improves further, attributed to both the hybrid pattern distillation and the mitigation of distribution shift issues for infrared images. However, we observe a slight decline in performance on the SODA dataset. We attribute this to the problem of data distribution mismatch. Notably, half of the images in SODA depict indoor scenes, which are scarce in our infrared pre-training dataset InfPre. In contrast, such scenes are more prevalent in the ImageNet and COCO datasets. We believe this discrepancy also accounts for the inferior performance of the two-stage training compared to joint training. Joint training benefits from a wider data distribution, which contributes to improved generalization performance.

We would be more than happy to discuss any further questions!

Q3: Comparison with distillation strategies in domain-adaptive methods.

Thank you for your suggestions. We think that our work differs from domain-adaptive methods in the following three main aspects:

• Different Application Scenarios: Domain-adaptive methods are typically tailored for specific downstream tasks and corresponding network structures, such as semantic segmentation [1,2] and object detection [3,4,5], with the requirement that the target labels in the source and target domains should overlap as much as possible. In contrast, our study focuses on adapting pre-trained models without being tied to any specific visual task. The adapted model can be applied to various visual tasks.

• Different Implementation Approaches: In domain-adaptive methods, the source model is usually trained on labeled data from the source domain, which equips it with the basic ability to perform specific tasks like semantic segmentation. The model is then adapted using labeled/unlabeled data from the target domain, allowing the adapted target model to perform these tasks without further fine-tuning. In contrast, our approach involves a source ViT model pre-trained on labeled/unlabeled datasets from the source domain, which is unable to complete specific tasks. We then adapt the pre-trained model using unlabeled data from the target domain. However, the adapted pre-trained model still lacks the ability to perform specific tasks and requires supervised fine-tuning with task-specific networks and labeled datasets.

• Different Evaluation Methods: In domain-adaptive methods, the adapted model is evaluated directly on the test set. Conversely, in our approach, after adapting the pre-trained model, we conduct supervised fine-tuning on downstream tasks for evaluation.

Regarding other distillation methods, in our early experiments, we tried several different approaches, including masking the input images followed by feature or attention distillation, as well as distilling multiple layers from the teacher to multiple layers of the student. However, these methods did not perform as well as our current approach. To be honest, we have not identified suitable distillation methods within the domain-adaptive framework to compare against. If you have any suggested methods, we would be very interested in conducting experiments in this area.

[1] Hoyer L, Dai D, Wang H, et al. MIC: Masked image consistency for context-enhanced domain adaptation. CVPR, 2023.

[2] Wang K, Kim D, Feris R, et al. CDAC: Cross-domain attention consistency in transformer for domain adaptive semantic segmentation. ICCV, 2023.

[3] Cao S, Joshi D, Gui L Y, et al. Contrastive mean teacher for domain adaptive object detectors. CVPR, 2023.

[4] Li J, Xu R, Ma J, et al. Domain adaptive object detection for autonomous driving under foggy weather. WACV, 2023.

[5] Weng W, Yuan C. Mean Teacher DETR with Masked Feature Alignment: A Robust Domain Adaptive Detection Transformer Framework. AAAI, 2024.

Thanks for your effort in reviewing our paper and providing insightful suggestions. We hope the following responses can address your concerns.

Q1: About the title.

Thanks for your constructive suggestion. Our primary motivation is to enhance the performance of the infrared semantic segmentation task from the perspective of pre-trained models. Therefore, in the revised draft, we update the title from "… Semantic Segmentation" to "… Infrared Semantic Segmentation". Additionally, the NMI-HAD and LL-FPN components of our method are also applicable to other image modalities, such as RGB. For further details on this aspect, please refer to our response to Reviewer FdMz, specifically in Q1.

Q2: About the texture bias in distillation.

This is a noteworthy question. In this paper, we elucidate two key factors that enhance the performance of pre-trained models in infrared semantic segmentation. The first factor is the presence of hybrid attention patterns, and the second factor is the reduction of biases towards texture. Therefore, the reasons for improved performance are two-fold. For example, using MAE-Large as the teacher to distill UNIP-Small.

• Firstly, UNIP-Small exhibits hybrid attention patterns in the deep layers through the NMI-HAD method, as illustrated in Fig. 11 of our paper, leading to better segmentation performance.
• Secondly, we utilize 541,088 infrared images for the distillation of UNIP-Small, enabling the model to adapt to the characteristics of weak textures in infrared images. As a result, the distilled UNIP-Small significantly reduces its bias toward texture information, as illustrated in Fig. 12 of our paper, leading to better transfer performance on infrared datasets.

When comparing UNIP-Small with MAE-Small, both reasons are significant. However, in comparison with MAE-Large, we believe that the second reason may have a greater impact, as the hybrid pattern is also present in MAE-Large. Additionally, when compared to models distilled by other models, such as DINO-Base, we think that the quality of attention maps produced by the teacher model is a critical factor. Generally, larger models tend to generate higher-quality attention maps. Therefore, models distilled from MAE-Large and iBOT-Large demonstrate superior performance compared to those distilled from DINO-Base.

Dear Reviewer nH3J,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission2787

Thanks for your effort in reviewing our paper and giving kind suggestions. We hope the following responses can solve your concerns.

Q1: About the underlying motivation and contribution.

Motivation:

Our motivation is to explore the transfer performance on infrared segmentation tasks of various RGB pre-trained models, and further improve their infrared segmentation performance based on some insightful findings. Our work’s Chain-of-Thought (CoT) is illustrated in Fig. 1 of the paper, following the sequence from step1 to step3. We first evaluate the infrared segmentation performance of different pre-trained models and then find two key phenomena: (1) The hybrid attention pattern is essential for semantic segmentation. (2) The texture bias hinders the model’s generalization on infrared images. For the first phenomenon, we introduce the NMI-HAD and LL-FPN to utilize the importance of the hybrid attention pattern. For the second phenomenon, we construct a large-scale dataset InfMix for pre-training to mitigate the distribution shift issue. Notably, all the proposed methods and datasets are dedicated to improving the infrared segmentation performance, which is our primary goal. In the revised draft, we have changed the title of our paper from "… Semantic Segmentation" to "… Infrared Semantic Segmentation" to better emphasize our motivation and to prevent any potential misunderstandings.

The applicability of the proposed methods to other infrared visual tasks and image modalities is a bonus. (1) The UNIP can be applied to other infrared visual tasks, such as object detection, since UNIP focuses on enhancing segmentation performance from the perspective of pre-training, without relying on architecture specifically designed for segmentation tasks. (2) The NMI-HAD and LL-FPN are applicable to other image modalities like RGB and depth, since the hybrid attention pattern is also essential in these modalities, as indicated in Tab. 10 of our paper. Moreover, we distill the hybrid pattern from MAE-Large on the ImageNet-1K dataset using NMI-HAD and fine-tune the distilled model on ADE20K using LL-FPN. The results in the table below demonstrate the effectiveness of NMI-HAD and LL-FPN in RGB datasets.

Contribution:

There are three main differences between our paper and previous works [1,2] in knowledge distillation for pre-trained models.

1. We evaluate the transfer performance of six popular pre-training methods, including supervised training, contrastive learning, and masked image modeling, while previous works mainly focus on a single pre-training method.

2. We conduct a thorough analysis of the performance differences among various pre-trained models, revealing distinct distributions of attention patterns across these models and correlating them with their pre-training tasks. These valuable findings provide insight into the intrinsic characteristics of different pre-training methods and models, as appreciated by Reviewers ctU1, nH3J, and gRqm.

3. We perform the distillation in the domain transfer setting, while previous works mainly focus on the RGB domain. Our work demonstrates that selective knowledge distillation, combined with unlabeled domain-specific data, can significantly unleash the potential of the large models pre-trained on extensive RGB datasets. It offers a viable route to improve the performance on tasks in domains with insufficient labeled data, from the perspective of model pre-training.

We believe that our analysis and experiments can catalyze further exploration, encouraging more researchers to focus on and continue improving this area. For instance, future work could involve multi-layer distillation, multi-model distillation, or more fine-grained head-wise distillation, as suggested by Reviewer ctU1.

[1] Bai Y, Wang Z, Xiao J, et al. Masked autoencoders enable efficient knowledge distillers. CVPR, 2023.

[2] Xiong Y, Varadarajan B, Wu L, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. CVPR, 2024.

Q2: About the attention patterns.

Thanks for your valuable questions. We respond to them one by one.

• The attention patterns in this paper are classified into local, hybrid, and global. These three attention patterns are qualitatively summarized by the visualization of attention maps (Sec. 3.1) and are quantitatively measured by NMI (Sec. 3.2). Our paper does not describe the attention pattern as “local-global”, do you mean to refer to “hybrid”? As shown in Fig. 3 (a), in the hybrid attention pattern, query tokens attend to both nearby and foreground tokens, while in the global pattern, all query tokens focus on the same foreground tokens, resulting in nearly identical attention maps.

• Previous works like DMAE [1] indeed suggest that the last layer may not be the best choice for single-layer distillation. However, their findings are limited to a specific type of pre-trained model and do not provide an in-depth analysis of why the last layer is suboptimal. On the contrary, our study thoroughly analyzes the representation of different pre-trained models, revealing that the underlying reasons for the last layer's inefficacy may differ significantly across models. For example, MAE models exhibit local patterns in the final layer, while DINO models exhibit global patterns. Moreover, previous works are unable to identify the layer best suitable for distillation, while we provide a straightforward yet effective method to address this issue.

• For multiple hierarchical feature distillation, we conduct experiments as shown in Tab.16 and discuss them in lines 1109-1113 of our paper. We also perform additional experiments, and the consolidated results are presented in the table below. The experimental results indicate that increasing the number of layers used for distillation leads to a decline in performance. As stated in our paper, we believe that employing multiple layers for distillation complicates the distillation objectives and introduces unnecessary redundancy, thereby negatively impacting the distillation performance. An adaptive selection of attention maps might be a potential solution to reduce noise and redundancy. Combined with the head-wise distillation suggested by Reviewer ctU1, the selection and aggregation of head-wise attention maps in multiple layers is precisely one of the directions we intend to explore in the future.

  $\enspace$Layers	SODA	MFNet-T	SCUT-Seg	Avg FT
  18 (ours)	70.99	51.32	70.79	64.37
  16+18	69.73	50.88	70.68	63.76
  17+18	69.59	51.33	69.47	63.46
  18+24	69.51	49.93	69.37	62.94
  17+18+19	69.13	49.96	67.73	62.27

Q3: Whether only the attention map of a single layer is used for distillation?

Yes, only the attention map of a single layer is used for distillation. As discussed in response to Q2, vanilla multi-layer distillation does not perform as well as single-layer distillation.

We would be more than happy to discuss any further questions!

[1] Bai Y, Wang Z, Xiao J, et al. Masked autoencoders enable efficient knowledge distillers. CVPR, 2023.

Dear Reviewer FdMz,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission2787

Q3: About the head-wise distillation.

Thanks for your insightful suggestion. We think it is a valuable idea and conduct corresponding experiments as follows. The experimental setup involves using MAE-Large (16 attention heads for each layer) as the teacher to distill UNIP-Small (6 attention heads for each layer). First, we calculate the Normalized Mutual Information (NMI) for attention maps of each attention head in MAE-Large and observe that not all attention heads within the same layer exhibit the same attention pattern. Therefore, we categorize these attention heads into three patterns: local (l), hybrid (h), and global (g).

We then select six attention heads (the total number of heads in UNIP-Small) as distillation targets and do not use the head misalignment strategy in Appendix B.4. For the 18th layer in MAE-Large, there are 5 global heads, 4 local heads, and 7 hybrid heads. We experiment with three different combinations: one containing only hybrid patterns (row 2), one containing only local and global patterns (row 3), and one containing all three patterns (row 4). The average NMI values for these combinations are comparable. Notably, the combination containing only hybrid attention patterns achieves the best performance, demonstrating the effectiveness of hybrid attention patterns even in head-wise distillation. Furthermore, using just 6 hybrid attention heads for distillation even surpasses the performance of distilling all 16 heads in the 18th layer (row 1). This phenomenon is also observed in the 24th layer. This suggests that there may be redundancy in the attention maps within a single layer. Therefore, we believe that more fine-grained distillation, such as head-wise distillation, is a highly promising research direction. We will discuss this in the future work section of our final version. We greatly appreciate your insightful suggestion and will continue to explore this avenue in the future.

The number(pattern) in the target column denotes the index of the selected head and the abbreviation of the corresponding pattern.

We would be more than happy to discuss any further questions!

Q2: Ablation studies for distillation of combining the features and attention.

Thanks for your valuable suggestion. We supplement the experiments in Tab. 6 of our paper by simultaneously using feature and attention distillation, as shown in the table below. Regardless of whether the distillation is performed using the 18th or 24th layer of the teacher model, the combined use of feature and attention distillation outperforms using features alone, but is inferior to using attention alone.

From the perspective of feature constraints, feature distillation imposes restrictions on each token's features and inherently restricts the relationships between features, whereas attention distillation only constrains the relationships between features. In the context of this paper, the issue of distribution shift is relatively significant. Consequently, employing feature distillation makes the distilled model less capable of effectively adapting to new data distributions. In contrast, attention distillation only requires the preservation of the relationships between features, making it more suitable for knowledge distillation in scenarios with significant distribution shifts.

Therefore, compared to using feature distillation alone, when both feature and attention distillation are employed together, the additional attention distillation objective reduces the emphasis on constraining each token's features during training, thereby overall decreasing the constraints on features. Conversely, compared to using attention distillation alone, the additional feature distillation objective increases the constraint on features, leading to a decline in performance.

Thanks for your effort in reviewing our paper and providing valuable feedback. We are pleased to hear your positive comments regarding our writing, experiments, and methods. We hope the following responses address your concerns effectively.

Q1: About the attention pattern in RGB and infrared datasets.

Thanks for your appreciation of our analysis. This paper mainly discusses two insights about the attention maps in pre-trained ViT models: (1) The hybrid attention pattern is more important for semantic segmentation than other attention patterns; (2) The bias towards texture impairs the transfer performance on infrared semantic segmentation. The first insight is evident not only in infrared datasets but also in RGB and depth datasets, as mentioned in Lines 505-522 in the original paper. In contrast, the second insight is specific to the infrared modality, and this is where the attention maps have different impacts on RGB and infrared datasets. Further explanations are as follows:

• Insight1: For experiments, Fig. 5 indicates the importance of the hybrid pattern in infrared datasets, while Tab. 10 of our paper demonstrates its effectiveness in RGB and depth segmentation datasets, including ADE20K. For visualization, we add the illustration of the attention maps on RGB image inputs in Fig. 13 of our revised draft. A comparison between Fig. 3 and Fig. 13 reveals that pre-trained models exhibit nearly identical distributions of attention patterns for both RGB and infrared images. Therefore, we believe that the difference in attention patterns is important in both RGB and infrared semantic segmentation. We will further highlight the scope of this insight in our final version. We further conduct the experiment about distilling on ImageNet using NMI-HAD and fine-tuning on ADE20K using LL-FPN, which also demonstrates the importance of hybrid attention patterns in RGB datasets. For more details on this experiment, please refer to our response to the Q1 of Reviewer FdMz.
• Insight2: As stated in Sec. 3.4 of our paper, the bias towards texture would amplify the distribution shift and hinder the model’s generalization on infrared tasks, since texture information is scarce in infrared images. For example, MAE-Base outperforms DeiT-Base and DINO-Base on RGB datasets like ADE20K, while lags behind them on infrared datasets, as indicated in Tab. 2 of our paper. Therefore, the texture bias of attention maps has different impacts on RGB and infrared datasets.

Additionally, compared to hierarchical architectures like the Swin Transformer which integrates both local and global attention mechanisms in their architectural design, this paper focuses on the pre-trained vanilla Vision Transformer (ViT), which does not incorporate such prior knowledge. Based on the thorough experiments and analysis, our study aims to convey that, despite the non-hierarchical nature of this architecture, different layers may exhibit distinct and hierarchical attention patterns, which have varying impacts on the model’s transfer performance. Furthermore, in light of these findings, we propose NMI-HAD and LL-FPN to leverage the significant influence of hybrid attention patterns on semantic segmentation tasks.

Dear Reviewer ctU1,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission2787

Thank you for your comment. Most of my concerns have been addressed, though, it appears to me that this paper does not have an emphasized motivation, which is also pointed out by reviewer FdMz. Although the key factor that affects the infrared semantic segmentation pretraining performance has been pointed out, I have to agree with the comment from FdMz that 'the proposed approach is not specific to infrared semantic segmentation tasks. After all, I believe this paper needs more analysis to emphasize the connection between the proposed NMI-HAD and infrared semantic segmentation. As a result, I decided to maintain my rating.