Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation

1. Compare with other frequency-based methods

We answered this question in the global response. We hope this could resolves your concerns.

2. The complexity analysis

We present the results of the complexity analysis, showing that our APM and ACPA are extremely lightweight, modules with minimal parameters and computational overhead. Our experiments were conducted on a single 4090 GPU. We also compared it with a lightweight frequency-based method (DFF [1]), which further highlights our advantages in terms of computational overhead and parameters.

[1] Deep Frequency Filtering for Domain Generalization, CVPR2023

3. Our method could benefit the model in reducing the overfitting

We believe that overfitting in few-shot fine-tuning arises from two main reasons: 1) The extremely limited samples (few-shot) prevent the model from fully learning each feature, making it prone to fitting extreme features (noise) and overly relying on feature correlations (e.g., if the samples are red apples, the model binds red color and round shape, and fails if the test sample is a green apple). 2) Intra-class variations (such as viewing angles, transparency, and distances) hinder the model's ability to recognize the same features accurately.

Our APM addresses the first issue by reducing the correlation between features and eliminating channel bias (eliminating extreme features). ACPA tackles the second issue by leveraging the phase's invariant information to minimize intra-class variations. Consequently, our approach effectively mitigates overfitting in few-shot fine-tuning.

1. Our method can be applied to other tasks

Our method can also be applied to cross-domain few-shot learning (CDFSL). Following BSCD-FSL[1] we implemented our method under this task setting (5-way 1-shot), and experimental results show that our method is effective in CDFSL as well.

[1] A Broader Study of Cross-Domain Few-Shot Learning

2. Sensitivity analysis on the choice of frequency components to filter

First, we visualized the average masker results for each domain to observe the filtered frequency components, as shown in the global rebuttal PDF. We found that the masker effectively adjusts to filter different frequency components according to different domains.

Then, we validated the robustness of APM by adding gaussian noise during its adaptive process. Even with the added noise, APM could still dynamically adjust and filter out the frequency components detrimental to the current domain, demonstrating its robustness.

APM's initialization We also explored different initialization strategies for APM. A value of 0 means no frequency components pass through, while a value of 1 means all frequency components pass through. "Rand" indicates random values uniformly distributed in [0,1], "gauss" indicates values drawn from a normal distribution, "clamp" indicates values clipped to [0,1], and "line" indicates values scaled linearly to [0,1]. The experimental results show that our APM is robust, quickly adjusting and adapting even with an initial value of all zeros. Our default initialization strategy is all ones, meaning all frequency components pass through initially, which also facilitates the dynamic adjustment and learning of APM.

3. Compare with other frequency-based methods

We answered this question in the global response. We sincerely hope this could resolve your concerns.

Here, we provide a more detailed explanation of the differences between our work and GFNet (experimental results are in the global rebuttal table):

1. The motivation of GFNet is to use global frequency filters to replace self-attention or MLPs, reducing computational overhead while removing inductive biases and maintaining a large receptive field (which helps capture long-term dependencies). This is reasonable because a spatial location in the frequency domain represents global information. In contrast, our work is motivated by the observation that different frequency components play different roles in different domains; a frequency component beneficial in domain A might be harmful in domain B. Therefore, we designed an adaptive masker to dynamically filter different frequency components according to different domains. We also explored and validated the relationship between frequency and feature correlation.

2. GFNet's "filter" refers to a convolutional filter, which can be seen as a stack of multiple convolution operations (a multiplication operation in the frequency domain can be replaced by multiple convolution operations in the spatial domain), with values in the range (-∞, +∞). In contrast, our masker is used to filter frequency components, with values in the range [0,1].

1. Compare with directly constraining the model

We answered this question in the global response. We sincerely hope this could resolve your concerns.

2. Comparing with [27]Channel Importance Matters in Few-Shot Image Classification

We compared with [27] in the global response, and here we provide a more detailed explanation.

We implemented the two transformation methods from [27] under our task setting. The performance slightly declines on the FSS dataset, which is similar to the source domain. However, on the other three datasets that are more distant from the source domain, there is a slight performance improvement. Nonetheless, our method has an advantage in terms of performance, with improvements significantly surpassing those of [27].

We further elaborate on the differences between our work and [27]:

1)[27] found that different channels recognize different patterns, and the channel bias present between channels can affect the model's recognition ability. They improved performance by eliminating this channel bias through feature transformation functions. In contrast, our work posits that different frequency components play different roles in different domains. We dynamically filter out detrimental frequency components based on the domain, thereby reducing channel correlation and improving performance.

2)Compared to [27]'s spatial operations, our frequency operations have the advantage of better representing global information. A spatial position in the frequency domain represents information from the entire spatial domain, giving frequency domain operations a natural advantage in capturing long-term dependencies and maintaining a large receptive field. Additionally, compared to feature transformation and convolution operations in the spatial domain, frequency domain operations remove inductive biases.

4. Compare with other frequency-based methods

We answered this question in the global response. We hope this could resolve your concerns.

5. Could other frequency-based works also achieve the correlation reduction

We tested the MI of the aforementioned frequency-based methods, and not all of them were able to achieve correlation reduction. They achieved correlation reduction in certain domains because they are used during training to enhance the model's generalization. However, due to the domain gap, feature extraction patterns that perform well on the source domain may not benefit the target domain. For example, DFF explores frequency components beneficial for generalization during training but its results show it filters out a lot of high frequencies and retains low frequencies. Therefore, its performance improvement might be due to filtering out noise in high frequencies. We visualized how our masker filters frequency components across different domains (the global rebuttal PDF displays these results). The frequency components to be filtered differ among target domains. Additionally, phase and amplitude need to be considered separately rather than being treated as a single entity. Hence, not all frequency-based methods can achieve correlation reduction.

6. More details on the experimental setup

Here, we provide a more detailed explanation of our data processing method. We adopt the same setup and data processing as PATNet[22]. For FSS-1000, the official split for semantic segmentation is used in our experiment. We report the results on the official testing set, which contains 240 classes and 2,400 testing images. For Deepglobe, the images have a fixed resolution of 2448 × 2448 pixels. To increase the number of testing images and reduce their size, each image was cut into 6 pieces. This cutting has minimal effect on segmentation due to the irregular shapes of the categories. After filtering out single-class images and the 'unknown' class, we obtained 5,666 images, each with a resolution of 408 × 408 pixels, for reporting the results. For ISIC, the images have a spatial resolution around 1022 × 767. We down-size the images to 512 × 512 pixels. For Chest X-ray, due to the large size of the image, we down-size the images to 1024 × 1024 pixels.

7. Why is frequency operation more advantageous in reducing correlation

The aforementioned (global response, answer2) orthogonality constraints, whitening, MMC, and MI Loss (discussed in the main text) all use spatial operations to reduce correlation. Here, we elaborate on the advantages of frequency operations compared to spatial operations.

1)The frequency domain inherently offers finer granularity compared to the spatial domain, facilitating more precise feature disentanglement. When a spatial domain channel (feature) is transformed into the frequency domain, each point in the frequency domain represents the global information of the feature. This results in a finer granularity transformation from 1 to hw points.

2)The frequency domain inherently provides a more lightweight operation compared to the spatial domain. A simple multiplication in the frequency domain can be equivalent to multiple convolutions in the spatial domain. This makes modules operating in the frequency domain more lightweight, easier to adapt to different domains, and more advantageous when data is scarce.

3)The frequency domain inherently has a larger receptive field and better helps capture long-term dependencies, making it more effective for learning global information. This enables operations in the frequency domain to capture more independent channel patterns, leading to expanded activation regions and more generalized representations.

1. Filtering on features damages the original feature structure? Filtering certain frequency components does not damage the original feature structure; instead, it is beneficial. Since not all frequencies are advantageous for the current domain, we dynamically adjust the mask and take its inverse (1-mask). We find that performance decreases compared to the baseline, indicating that the frequencies we filter out are indeed detrimental components.

2. Sensitivity analysis on the choice of frequency components to filter Since different domains require different weights, our method adapts directly to the target domain without the need for source-domain training. We visualized the average masker results for each domain to observe the filtered frequency components, as shown in the global response PDF. We found that the masker effectively adjusts to filter different frequency components according to different domains.

3. Differences between our approach and Deep Frequency Filtering for Domain Generalization (DFF): We compare with DFF in global response answer 1. Here we provide a more detailed explanation.

1. Motivation: DFF aims to explore and retain frequency information beneficial for generalization during training, while filtering out frequencies that are not. However, we found that useful frequency information varies across different domains; frequencies beneficial to one domain may be harmful to others. Therefore, we focus on adaptively selecting beneficial information for different domains.

2. Amplitude and Phase: DFF does not distinguish between amplitude and phase, using attention mechanisms to filter out non-generalizable frequency components during training. However, amplitude and phase play different roles: amplitude contains domain-specific information, while phase contains domain-invariant information. Our APM independently adjusts amplitude and phase, filtering out detrimental frequency information separately. ACPA leverages the domain-invariant characteristic of phase to reduce intra-class variance between support and query.

3. Effectiveness: DFF performs well when input distributions differ but the label space remains the same. However, its effectiveness is limited in our task, where both input distributions and label spaces differ. We implemented DFF in our task, and our method demonstrated superior performance (see Q1 in global response).

4. Performance could be further improved after segmentation refinement To highlight the effectiveness of our method, we did not employ methods such as data augmentation, or segmentation refinement. We used the segmentation refinement of PANet [37], which led to further performance improvements. Our method already surpasses the existing SOTA and shows significant improvement over the baseline, even without using any additional methods.

5. The different initialization strategies of APM We presented this experiment in our response to question 2 for reviewer H2xr. Please refer to the AMP initialization experiments in question 2 for reviewer H2xr. We sincerely hope this could resolve your concerns.

6. Why does ACPA only use the phase information Previous interpretability studies have shown that phase is an invariant representation, while the amplitude varies between samples and contains specific information. To alleviate intra-class variations (such as viewing angles, transparency, and distances, which hinder the model's ability to recognize the same features accurately), we leverage the invariant nature of phase to align the feature spaces of support and query.

7. Compare with other methods for reducing correlation/Compare with other frequency-based methods/More detailed about the individual contributions of APM and ACPA We answered these questions in the global response. We sincerely hope this could resolve your concerns.

8. Other metrics validate that our method achieves feature disentanglement We normalize the feature map channels with L2 normalization and then compute the L1 norm to measure their sparsity. A smaller value indicates higher sparsity. After masking certain frequency components, the sparsity value decreases, indicating sparser features. Sparse features imply lower feature redundancy, which benefits feature disentanglement and thereby enhances the model’s generalization capability.

9. The complexity analysis We answer this question in the reviewer upoB's question 2. We sincerely hope this could resolve your concerns.