Learning Frequency-Adapted Vision Foundation Model for Domain Generalized Semantic Segmentation

Q1: The adaptation for low-frequency and high-frequency components are different. The paper could provide more intuition to explain the particular adaptations chosen for low and high frequency components.

R: Thanks for your general positive comments on our work and your insightful comment, so that we could have the chance to provide a more in-depth analysis on the particular adaptations.

In general, the adaptation on high-frequency components is further implemented by an instance normalization when compared to the low-frequency components. Instance normalization, which computes the channel-wise mean and standard deviation, is effective to eliminate the domain-specific styles [44,27].

To further validate this point, in the attached file in general response, we visualize the t-SNE feature space of the samples from four unseen domains. The results without and with implementing the instance normalization on the high-frequency components are displayed in the first and second row of Fig.R1, respectively. Fig.R1 is attached in the 1-pg PDF in the general response. Notice that, when without instance normalization, the method is the same as using the same adaptation on the low-frequency component.

It can be seen that, the use of instance normalization alleviates the existence of the domain-specific clusters (in the same color) and allows samples from different domains to be more uniformly distributed (in different color). The domain-specific clusters indicate that samples from the same domain have less distance and higher similarity in the feature space. It indicates that the different adaptation for high-frequency component is more effective to develop the style-invariant property.

Finally, should you have further suggestions and questions, we are glad to address during the discussion stage.

Q1: Writing & logic. 1) the relationship among three DGSS category methods; 2) Why the style-invariant properties of VFM is important and urgent? 3) The introduction is not accompanied by an explanation of Fig. 1.

R: Thanks for your valuable feedback so that we could have the chance to introduce a native speaker to thoroughly polish the writing and logic. The mentioned problems are clarified as follows and will be incorporated accordingly.

1. Existing three types of DGSS methods, either decouple the style, augment the style or stabilize the content, are all full-training paradigm (on a CNN or Vision Transformer encoder). In contrast, the realm of VFM provides a new paradigm for DGSS, which only fine-tunes on a minor part of model parameters instead of full-training. The motivation of this work is to allow the VFM features to stabilize the content while at the same time be invariant to the style shift, which aligns the advantages of different types of existing DGSS methods.

2. An ideal representation for DGSS is to learn stable content representation despite the cross-domain style shift [44,26,13,45,63,60,47]. However, the fine-tuning of VFM is highly dependent on the low-dimensional intrinsic embeddings [58], which is subtle and fragile to the distribution shift, which is posed by the style shift in the context of DGSS. Therefore, an ideal VFM representation for DGSS is supposed to demonstrate robust style-invariant property, so that it is able to demonstrate robust scene representation from different domains.

3. We would like to kindly raise the reviewer's attention that, the explanation of Fig.1 is in the third line of the introduction; while the caption of Fig.1 further details the the domain generalization property of the high-/low- frequency components of Haar wavelet transformation.

Q2: Advantages of 1) Haar wavelet for domain generalization, 2) enhancing generalization of VFM features by Frequency Decoupling over existing related works?

R: Thanks for your constructive comments. To clarify:

1. Both FFT based methods and Haar wavelet can decouple a scene representation into low- and high- frequency components. However, compared with FFT, Haar wavelet transformation consists of orthonormal basis, and warrants the orthonormality between the low- and high- frequency components. Orthonormality is essentially a type of de-correlation, which provides a better separation between the low- and high- frequency components. In the context of DGSS, the orthonormality of Haar wavelet transformation allows a better seperation between the scene content and the style. Meanwhile, from the experimental side, Table R1 (in general response) shows that our Haar wavelet based method shows a superior performance on all unseen target domains than the FFT based FourierFT [a]. This outcome further demonstrates the advantage of Haar wavelet over existing Fourier transformation based methods.

2. We would like to kindly raise the reviewer's attention that, harnessing VFM for DGSS is an emerging research line. By the time of submission, the closest work is REIN [58]. REIN [58] directly implements low-rank adaptation to learn the entire scene representation. It does not decouple the content and style, which is necessary in visual domain generalization. In contrast, the advantage of the proposed method is that, it provides a feasible path to decouple the style from the content by the separation between high- and low- frequency components. This decoupling allows the representation learning to handle the style and content in a divide-and-conquer manner, where the style representation is supposed to be invariant to the domain, while the scene content is supposed to be stable.

3. The research line of frequency decoupling based methods is focused on generic domain generalization (classification task), not for DGSS or VFM based DGSS.

Q3: What is the difference between the paper with [1]?

[1] Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning. CVPR 2024.

R: Thanks for your reference suggestion, so that we could have a chance to discuss the difference between the recent CVPR 2024 work BlindNet. We would like to kindly raise the reviewer's attention that, this paper is essentially different from the proposed method in many critical aspects.

1. Completely different methodology design. BlindNet first augments the style diversity from the image, and constraints the scene representation by designing style similarity and content consistency based losses. In contrast, the proposed method decouples VFM features in the frequency space by Haar wavelet to decouple the style and content separately. It does not reply on any types of style augmentation or augmented domains, which is more universal and more generalized than BlindNet.

2. Completely different path to eliminate the style. BlindNet first augments the styles from the original image, and then mitigates the style variation by computing and reducing the style similarity between the per- and post- augmented styles. In contrast, the proposed method decouples the style and content from the frozen VFM features by high- and low- frequency components, respectively. Afterwards, the low-rank adaptation on the high-frequency representation is implemented with an instance normalization transformation, so that the high-frequency representation is invariant to the style shift and better generalized to unseen target domains.

3. Significant performance improvement. The official report of BlindNet on $C \rightarrow \{C, B, M, S\}$ is 38.56%, 34.51%, 40.11% and 25.64% mIoU, while this paper achieves 68.23%, 61.94%, 68.09% and 50.36% mIoU. The official report of BlindNet on $C \rightarrow \{B, M, G, S\}$ is 59.27%, 71.10%, 58.11% and 40.43% mIoU, while this paper achieves 65.12%, 75.86%, 63.78% and 49.75% mIoU.

We will discuss this work accordingly.

Q1: Is the assumption of "similar content yet varied style" practical? The joint distribution of content and style? Can we assume that they are independent across the dataset?

R: Thanks for your insightful comment so that we could have a chance to further clarify the basic assumption in domain generalized semantic segmentation (DGSS). In the context of DGSS and autonomous driving, it has been well established and recognized by the prior DGSS works that the key challenge is 'similar content yet varied style' [13,28,33,47,66,67].

Just as the reviewer acknowledged, despite the style shifts greatly in driving scenes, it is reasonable that in driving scenes the categorical distribution is nearly the same.

On the other hand, we definitely agree with the reviewer that, it is a great perspective to discuss if the joint style and content distribution (i.e., the driving scenes) is independent or not. It should be highlighted that, the joint style and content distribution in DGSS, same as the assumption in generic visual domain generalization, is not (rigorously) independent. As extensively discussed in [5,67], the style in driving scenes and DGSS is impacted and jointly determined by multiple extrinsic factors such as weather, lighting, urban landscape and etc. An example is that, two datasets collected from two cities with different landscapes are under the same weather, which makes the style vary but not rigorously independent between each other.

To sum up, given that domains in DGSS are not rigorously independent and generally have the distribution shift, which in lines with the generic visual domain generalization, we would like to raise the reviewer's attention that, the established assumption 'similar content yet varied style' [13,28,33,47,66,67] is practical in DGSS.

Q2: Results on the other VFMs (e.g. SAM).

R: We would like to kindly raise the reviewer's attention that, the results on other VFMs have been provided in Table 4 and we attach it as follows.

Table R1: Performance Comparison of the proposed FADA with baseline on other VFMs. Evaluation metric mIoU in %.

In short, when using SAM as the image encoder, the proposed method outperforms REIN [58] by 1.4%, 1.2% and 1.3% mIoU on C, B and M unseen target domains; when using CLIP as the image encoder, the proposed method outperforms REIN [58] by 1.6%, 1.1% and 1.6% mIoU on C, B and M unseen target domains; when using EVA02 as the image encoder, the proposed method outperforms REIN [58] by 1.4%, 1.4% and 1.3% mIoU on C, B and M unseen target domains. These outcomes further demonstrate that the proposed method is robust and generalized on a variety of VFM image encoders.

Finally, should you have further suggestions, we are glad to address during the discussion stage.

Q1: Compare recent LoRA methods [a,b].

[a] Gao, Z., et al. (2024). Parameter-Efficient Fine-Tuning with Discrete Fourier Transform. ICML.

[b] Yang, Y., et al. (2024). Efficient low-rank backpropagation for vision transformer adaptation. NeurIPS.

R: We've compared both methods, namely, FourierFT [a] and WHT [b], with the baseline, REIN [58] and the proposed FADA under the G$\rightarrow$ C, B, M setting. For fair evaluation, both [a] and [b] are attached to process the VFM features of each layer as the common low-rank adaptation paradigm does. The results in Table R1 show that the proposed FADA shows its superiority than the recent methods [a, b] on all unseen target domains with a $\textgreater$ 2% mIoU improvement.

Table R1: Performance Comparison of the proposed FADA with recent PEFT methods [a,b]. Evaluation metric mIoU in %.

Q2: Sec.5.2, the authors are encouraged to add more insights towards the proposed method in the experiment analysis section.

R: Thanks for your constructive suggestions, so that we could have the chance to provide more insight in the experimental section. We will enrich the following analysis accordingly.

(1) From the perspective of representation. Vision Transformer based encoder is more capable to capture the scene content owing to the long-range dependencies than the CNN based encoder, which explains the significant improvement of Transformer based methods than CNN based methods. On top of it, both the proposed method and REIN [58] use Transformer based foundation model, which not only possesses the inherited representation ability from the Transformer, but also occupies the inherited out-of-distribution generalization ability from the large-scale data pre-training. Most importantly, compared with REIN [58], the proposed method further advances low-rank adaptation in the frequency space, which handles the scene content and the style variation in a divide-and-conquer manner, therefore further improving the generalization ability on unseen target domains.

(2) From the perspective of domain gap. The first two experiments use GTA5 and SYNTHIA as source domains, respectively. Both source domains are synthetic data. Even if trained on such synthetic source domain, the proposed method still shows a generalization performance of 68.2% on real-world CityScapes target domain. This outcome is very close to some modern CNN and Transformer segmentation models when full-trained on CityScapes, which further indicates the generalization ability of proposed method. In contrast, the compared DGSS methods still shows a significantly inferior performance when the domain gap is huge.

(3) From the perspective of low-rank adaptation in frequency space. In visual domain generalization and DGSS, an ideal representation is to be stable to the scene content when the cross-domain styles shift greatly. As many prior works [25,36,32,62,56] reflect, the decoupling of high- from low-frequency component is an important research line to decouple the style from content. The proposed method takes advantage of this aspect, and implements Haar wavelet transformation to realize this objective. The individual low-rank adaptation for low- and high- frequency components provides a divide-and-conquer path for VFM features to handle the style and content, respectively, and therefore significantly improves the performance on unseen target domains when compared with REIN [58].

Finally, we would like to kindly raise the reviewer's attention that, in Sec.5.3 we have provided the ablation studies from four different aspects to analyze why the proposed method achieves a better performance than the VFM baseline and REIN [58].

Q3: The ground truth images are not listed on Figure 5 and Figure 6, which hinders the comparison regarding the qualitative performances.

R: Thanks for your constructive suggestions. We have accordingly provided the ground truth map for each sample in Fig.5 and Fig.6. The updated Fig.5 and Fig.6 have been provided in the attached PDF file in the general response, named as Fig.R2 and Fig.R3, respectively.

Q4: Lack of description of the implementation details. What losses are used during the experiments? Which lr scheduler is used?

R: We are sorry that the implementation details of the proposed method is extensive than intended. In general, all the loss functions, learning scheduler and other hyper-parameters keep the same as REIN for fair comparison. Specifically, the optimizer uses AdamW. The initial learning rate for the backbone is $1\times10^{-5}$. The initial learning rate on the decoder and the proposed FADA's parameter is set $1\times10^{-4}$. The final loss function $\mathcal{L}$ directly inherit the losses from the Mask2Former decoder [12], given by

$\mathcal{L} = \lambda _ {ce} \mathcal{L} _ {ce} + \lambda _ {dice} \mathcal{L} _ {dice} + \lambda _ {cls} \mathcal{L} _ {cls}$

where $\mathcal{L} _ {ce}$, $\mathcal{L} _ {dice}$ and $\mathcal{L} _ {cls}$ denote the cross-entropy loss, dice loss and classification loss. Here the hyper-parameters $\lambda _ {ce}$, $\lambda _ {dice}$ and $\mathcal{L} _ {cls}$ are 5.0, 5.0 and 2.0, respectively. For more details of the classification loss, please refer to [12]. We will enrich these details accordingly.

Finally, should you have further questions and suggestions, we are glad to address during the discussion stage.

Q1: Justify: 1) domain-specific style and domain-invariant content are demonstrated by high- and low-frequency components 2) how the proposed method stabilizes domain-invariant content while mitigating domain-specific style.

R: Thanks for your valuable comment. To clarify:

1. In visual domain generalization, an important research line is to learn a domain-invariant content representation from the frequency feature space. It has been observed and well-documented that the low- and high- frequency components rest more domain-invariant content and domain-specific style properties [25,36,32,62,56].

2. To validate this point in our context, we extract the frozen VFM feature from driving scene samples containing the same semantic categories (i.e., content) but from four unseen domains (i.e., style), and implement the Haar wavelet transformation on them. According to the t-SNE feature visualization in Fig.R1 Row 1 (attached in the 1-pg PDF file), the high-frequency features ($HH$, $HL$, $LH$) exhibit some domain-specific clusters, where driving scenes from the same domain (points of the same color) lean to cluster together. In contrast, the low-frequency features ($LL$) allow the driving scenes from different domains (point of different color) to be more uniformly mixed and distributed.

3. To inspect the effectiveness of the proposed method on mitigating the domain-specific features, we implement the instance normalization transformation on the high-frequency features ($HH$, $HL$, $LH$) from the VFM. After that, the t-SNE feature visualization is conducted and the results are shown in Fig.R1 Row 2. It can be seen that, compared with the original high-frequency features in Fig.R1 Row 1, the domain-specific clusters are alleviated, and the samples from different domains (points of different colors) are more uniformly distributed.

Q2: More analysis on instance normalization for style invariance.

R: Thanks for your insightful comments. Please refer to the 2) and 3) response to Comment#1 for the detailed analysis.

In addition, we would like to kindly raise the reviewer's attention that, Fig.3 and Fig.4 in the main text provide analysis on how the proposed method eliminates the domain-specific styles in both feature space and channel-wise activation patterns.

Q3: L250 & Table 2. Eliminate high-frequency components or preserving all components.

R: To clarify, our objective is not to eliminate the high-frequency components that rest in domain-specific styles, but to be invariant to cross-domain style variation (L172) & to decouple the high-frequency component from the impact of cross-domain styles (L250). The reason behind is that the high-frequency components contain not only styles, but also other information such as structure and object boundary. A naive removal of all high-frequency components loses such information, which degrades a scene representation and declines the segmentation performance. Therefore, our objective is to allow the high-frequency representation to be invariant to the cross-domain styles by instance normalization, not to entirely eliminate all the high-frequency representation. It aligns with the ablation study in Table 2, where the low-rank adaptation with instance normalization on high-frequency components contribute positively to the performance. We will clarify this point accordingly when revising.

Q4: Discuss/compare the advantages of Haar wavelet over FFT methods.

R:

1. Both FFT and Haar wavelet decouple a scene representation into low- and high- frequency components. However, compared with FFT, the orthonormal basis of Haar wavelet warrants the orthonormality between the low- and high- frequency components. Orthonormality is essentially a type of de-correlation, which provides a better separation between the low- and high- frequency components. In the context of DGSS, the orthonormality of Haar wavelet transformation allows a better separation between the scene content and the style.

2. From the experimental side, please refer to Table R1 in the general response for the comparison. It is observed that, the proposed method, with the aid of Haar wavelet, shows a predominant performance improvement than FFT based [a] on all unseen target domains. Meanwhile, we respectively ask for the reviewer's understanding that, as most frequency decoupling based domain generalization methods are devised for classification task , it would be difficult and impractical to adapt all these methods for VFM based DGSS.

[a] Gao, Z., Wang, Q., Chen, A., Liu, Z., Wu, B., Chen, L., & Li, J. (2024). Parameter-Efficient Fine-Tuning with Discrete Fourier Transform. ICML 2024.

Q5: What specific style-invariant properties?

R: Indeed, VFM has an inherited ability to generalize to out-of-distribution. However, the performance of current large segmentation models still lacks in specific down-stream scenarios, so additional training is needed. Consequently, fine-tuning VFM on DGSS encounters the unique challenge of DGSS, that is, an ideal representation for the task of DGSS is to learn stable pixel-wise content representation despite the cross-domain style shift [44,26,13,45,63,60,47]. Here the style shift can be posed by different types of weather, illumination, urban landscape and etc [67]. However, the fine-tuning of VFM is highly dependent on the low-dimensional intrinsic embeddings [58], which is subtle and fragile to the distribution shift, which is posed by the style shift in the context of DGSS. Therefore, an ideal VFM representation for DGSS is supposed to demonstrate robust style-invariant property to generalize well on different types of weather, illumination, urban landscape and etc [67].

On the other hand, as shown in Table 1, 2, 4 and 5, the modified VFM representation from the proposed method shows a significantly better generalization on unseen target domains, in different weather and day-night conditions.

Q1: 1) Intuition on adapting in all the transformer layers. 2) Ablate the layer location.

R: Thanks for the constructive comment.

1. The high-level idea of our intuition focuses on low-rank adaptation (LoRA) in the frequency space. As existing LoRA methods [24,58] implements the adaptation on each of the transformer layer, and we further introduce the frequency space. Therefore, we treat the frequency space as a whole, and embed it into each of the transformer layer.

2. But we definitely agree with the reviewer that, it would be meaningful to have an in-depth analysis on the learning behaviour from shallow to deep. To this end, apart from the baseline (REIN [58]) and the proposed method, we further provide two experiments, where the frequency adapter is attached in the first half seven layers (denoted as shallow) and the second half seven layers (denoted as deep), respectively. Results in Table R1 show that:

• using the frequency adapter on the first half layers (shallow) shows a slightly better performance than on the second half layers (deep). It may be explained that the shallower features contain more cross-domain styles, such as illumination, landscape and etc.
• using the frequency adapter on all layers (ours) achieves the best performance, indicating its effectiveness on all layers.

Table R1: Ablation study on the positions of the frequency adapters. GTA5 as the source domain. CityScapes, BDD and Mappilary are unseen target domains. Evaluation metric mIoU in %.

Q2: Compare FADA with REIN under similar tunable parameters.

R: Thanks for your constructive suggestion. The feature dimension of MLPs in LoRA impacts the parameter number. We therefore minimize the feature dimension in FADA, from 1024 to 256, so that the parameters are reduced significantly. Results in Table R2 show that, even if the feature dimension is only one fourth of REIN [58], FADA still shows a 1.1% mIoU improvement on three unseen domains.

Table R2: Comparison between FADA and REIN under similar tunable parameters. GTA5 as source domain.

Q3: 1) prioritize low- over high-frequency features v.s. Table 2 adapting to all features. 2) suggest an experiment design.

R: Thanks for your constructive comments, so that we could have a chance to have a more in-depth analysis between the low- and high- frequency representation.

1. In general, the intuition of the proposed method is to allow the high-frequency components to be invariant to the style shift, instead of eliminating the entire high-frequency components. The reasonable behind is that, high-frequency components of an image representation consist of not only styles, but also other types of information such as the scene structure, object boundaries and etc. Scene structure and object boundary is also important for scene segmentation. Therefore, our objective is to adapt the high-frequency components so that they are invariant to the cross-domain shift, not to directly eliminate all the high-frequency components.

2. We definitely agree with the reviewer that, it would be beneficial to specify the high-frequency features from multiple domains. Following your suggestion, we follow the evaluation setting where GTA5 and Synthia are both used as the source domains, while CityScapes, BDD and Mapillary are used as unseen target domains, and REIN [58] as the baseline. After Haar wavelet transformation on the per-batch image from each domain, their low-frequency components are fused into a single low-rank adaptation module. Instead, two low-rank adaptation modules are used to process the high-frequency components from the two source domains, respectively (denoted as $HH_{G+S}$). Both high-frequency components are implemented with an instance normalization after the adaptation. It is compared with the scenario where one high-frequency component is used, denoted as $HH_G$ and $HH_S$. The results in Table R3 show that, handling the high-frequency component from each source domain indeed improves the performance on unseen target domains. Handling the high-frequency components from both source domains shows a further improvement.

Table R3: Impact on the learning paradigm of high-frequency components. GTA5 and SYNTHIA are both source domains. CityScapes, BDD and Mapilary are unseen target domains.

Q4: Compare the inference time.

R: We compare the inference time of the proposed FADA along with REIN [58] and a vanilla VFM. All the experiments are conducted on a single RTX 3090Ti GPU. As reported in Table R4, the proposed FADA, implementing Haar wavelet transformation with additional parameters, only leads to a slight decrease on the inference time than the previous REIN [58].

Table R4: Inference Time Comparison. Evaluation Metric: Frame Per Second (FPS).