Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation

1. The computational efficiency of SAM-SVN：

Our SAM-SVN is used only during source domain training and not during fine-tuning or inference, so it does not affect the computational efficiency during inference. Regarding efficiency during training in the source domain, although it requires double backpropagation, it is applied only to the singular value matrix of DFN, resulting in negligible additional computation. We demonstrate the computational efficiency of SAM-SVN by measuring the efficiency of the baseline, PATNet (which adopts the same baseline as ours), SAM applied to the entire model, SAM applied to DFN, and SAM applied to SVN (singular value matrix of DFN). The results are shown in the table below:

[1] Cross-Domain Few-Shot Semantic Segmentation

2. References：

We promise to cite the listed references in the final version.

1. Our method can fit different structures：

Our structure and the APM[1] structure are both based on HSNet, while our ViT structure is based on FPTrans[2]. Additionally, in the appendix, we used the SSP-based architecture for comparison with IFA. Thus, our method can be applied to networks with various architectures.

2. The reason for the different performance between ResNet and ViT：

The difference is due to the datasets' bias towards local recognition cues or global ones.

We use KL divergence (DisFB) to measure the similarity between the foreground and background of datasets. A higher DisFB suggests a greater requirement for the model's global resolution capability, with ISIC having the highest DisFB. ResNet, due to its convolution-based design, has local prior characteristics, making it more effective for datasets like DeepGlobe that demand detailed local features. On the other hand, ViT, based on attention mechanism, excels in long-range dependencies and global resolution, offering better performance for datasets like ISIC that rely more on global perception.

3. Validating adapter choice impact on sharpness

In the main text, we measured loss fluctuations by adding Gaussian noise to observe the changes in the sharpness of loss landscapes before and after integrating DFN. Here, we further measure the impact of different adapter choices on sharpness. Consistent with Fig3 and Fig5 in the main text, we analyze based on position and structure (res: residual, ser: serial, BKB: backbone, enc-dec: encoder-decoder):

The results above show that, from the perspective of sharpness, significant changes in loss fluctuations occur only when residual links are satisfied and the position is deep within the backbone (with the adapter structure not being a determining factor). This indicates that the adapter has captured domain-specific related information, which is consistent with the conclusions drawn in the main text.

4. Our methods can benefit general few-shot segmentation tasks

We tested the performance of DFN on the source domain (Pascal). Pascal consists of 20 classes and is set to a 4-fold configuration in the FSS setup. This means training is conducted on 5 classes, while testing is performed on 15 classes that were not seen during the training phase. Due to its ability to enhance the model's adaptation to new domains/classes, it also provides benefits for general few-shot segmentation. After fine-tuning DFN, its positive impact becomes even more pronounced.

5. More settings to DFN：

Here, we include more experiments on the DFN training and fine-tuning settings：1）DFN is involved in source domain training but removed in the target domain. 2）DFN is involved in source domain training, but its original parameters are discarded and it learns from scratch in the target domain (using different initialization methods).

For the setting that DFN is removed in the target domain, DFN captures domain-specific information during source domain training and guides the model to learn domain-agnostic knowledge, resulting in a significant performance improvement compared to the baseline. However, due to a lack of target-specific knowledge, this approach is suboptimal. In the setting that DFN learns from scratch, its performance is influenced by the initialization method, yet it still performs well under various initializations. Our approach (i.e., DFN+SAM-SVN) is seen as using DFN to guide the model to focus on domain-agnostic knowledge in the source domain, while simultaneously providing DFN with a reasonable initialization value beneficial for adapting to various domains. Therefore, it achieves the best performance.

1. Deeper theoretical analysis for “natural decoupling”：

Due to space limitation, please refer to reviewer ueDb's reply 1 for theoretical analysis.

2. Analysis of trade-off between decoupling and domain knowledge retention:

For any $\rho>0$ and any distribution $\mathscr{D}$, with probability $1-\delta$ over the choice of the training set $\mathcal{S}\sim \mathscr{D}$, SAM bounds the generalization error as

$$L\_\mathscr{D}(\boldsymbol{w}) \leq \max\_{\|\boldsymbol{\epsilon}\|\_2 \leq \rho} L\_\mathcal{S}(\boldsymbol{w} + \boldsymbol{\epsilon}) + \sqrt{\frac{k\log\left(1+\frac{\|\boldsymbol{w}\|\_2^2}{\rho^2}\left(1+\sqrt{\frac{\log(n)}{k}}\right)^2\right)+ \frac{4\log\frac{n}{\delta} + \tilde{O}(1)}{n-1}}{n-1}}$$

where $n=|\mathcal{S}|$, $k$ is the number of parameters, and we assume $L\_\mathscr{D}(\boldsymbol{w}) \leq \mathbb{E}\_{\epsilon\_i \sim \mathcal{N}(0,\rho)}[L\_\mathscr{D}(\boldsymbol{w}+\boldsymbol{\epsilon})]$. The trade-off is controlled by $\rho$. For $\rho$, a larger value indicates more perturbations are added, emphasizing domain knowledge retention (experiment is in Appendix Tab11 left)；For $w$​, a larger perturbation range signifies a greater focus on knowledge retention (experiment is in Tab7 right). All experiments regarding SAM-SVN are presented in Tab7 left, Fig10, Tab8, and Tab11 left.

3. Clarity for Figure 1:

In fact, the feature does not contain domain-agnostic and domain-specific knowledge in equal parts; our diagram merely indicates that the feature includes both specific and agnostic information. Regarding DFN, which is a structure-based decoupler rather than a loss-based one like current approaches, it captures domain-specific information, thereby directing the model’s attention towards domain-agnostic knowledge (described in abstract and discussed in section 2). We will add a brief explanation of DFN in the caption to make it clearer for readers. Thank you very much for your suggestion.

4. More explanations for why cka decrease means "domain-specific"

CKA is a metric used to measure domain similarity by comparing the distances between kernel centers of two sets of data. If the kernel centers of the representations extracted by neural networks from the two data sets are closer (resulting in a higher CKA value), it indicates that their data distributions in the feature space are more consistent, meaning their feature representations are more similar and they share more patterns. Conversely, a lower CKA value implies lower domain similarity, with the kernel centers being farther apart in the feature space. Lower domain similarity suggests that the extracted features are less similar, indicating that the two sets of representations share fewer common patterns and have more characteristics specific to their own data distributions (more domain-specific).

To verify it is the domain gap that influences CKA, we divided the data in the Pascal into 20 groups based on categories and measured the CKA between different groups. We reported the CKA between the two most divergent groups, as well as the mean CKA and standard deviation (std) CKA. The results are as follows: It can be seen that even for the most divergent feature groups within the same dataset, the CKA is above 0.85, indicating the validity of using CKA to measure domain similarity.

5. Reliability of the CKA metric

Absolute change is used here. We choose CKA as a measurement metric for several reasons: 1) CKA employs kernel center alignment, which eliminates measurement errors caused by extreme feature representations (noise); 2) Compared to other metrics like MMD and cosine similarity, CKA removes the effects of scaling differences; 3) CKA is more sensitive to differences in data distribution, allowing it to more accurately reflect distribution changes (detailed theory in the appendix).

The discrepancy in change between FSS and DeepGlobe is consistent due to: 1) CKA's sensitivity to data distribution differences; 2) The smaller domain gap between FSS and Pascal (both being natural image datasets) results in smaller changes, whereas the larger domain gap between Deepglobe (remote sensing images) and Pascal leads to greater gains and changes.

Additional metrics: In the experimental section (Figure 8), we also used MMD to validate our viewpoint, which is consistent with CKA. Furthermore, in the appendix, we employed relative CKA to mitigate the impact of data distribution (Figure 15).

1. Deeper theoretical analysis for “natural decoupling”：

The behavior of adapters as decouplers can be analyzed through the Information Bottleneck (IB) theory. The IB objective is:

$\mathcal{L}_{IB} = I(X;Z) - \beta I(Z;Y)$

where $I(\cdot;\cdot)$ is mutual information, X is the input, Y is the final output label, $Z$ is the intermediate representation, and $\beta$ is the balance parameter. Decomposing the input as $X = X_{inv} + X_{spec}$, the IB objective becomes:

$\mathcal{L}_{IB} = I(X\_{inv} + X\_{spec};Z) - \beta I(Z;Y)$

For encoder-decoder network (ED) parameters $\theta_f$ and adapter parameters $\theta_g$ where $\theta_f \gg \theta_g$, the adapter's information capacity is much smaller than the ED, causing it to selectively absorb source domain-specific information that better optimizes the current training objective, resulting in $I(X_{spec};Z) \gg I(X_{inv};Z)$, which promotes gradient flow differentiation. In a network $f$ with residual adapter $g$, the forward propagation is: $F(x) = f(x) + g(f(x))$

For ED parameters $\theta_f$, the gradient is: $\frac{\partial \mathcal{L}}{\partial \theta_f} = \frac{\partial \mathcal{L}}{\partial F(x)} \cdot \frac{\partial F(x)}{\partial f(x)} \cdot \frac{\partial f(x)}{\partial \theta_f}$, expanding the middle term: $\frac{\partial F(x)}{\partial f(x)} = I + \frac{\partial g(f(x))}{\partial f(x)}$

where $I$ is the identity matrix (from the direct residual path), and the second term is the Jacobian matrix of the adapter function.

For adapter parameters $\theta_g$, the gradient is proved to be $\frac{\partial \mathcal{L}}{\partial \theta_g} = \frac{\partial \mathcal{L}}{\partial F(x)} \cdot \frac{\partial g(f(x))}{\partial \theta_g}$

Differentiated gradients: Due to the adapter's selective absorption of domain-specific information and the residual adapter structure, gradient flow naturally separates network optimization into two complementary learning objectives.

2. Further justification for perturbing singular values in SAM-SVN:

For any $\rho>0$ and any distribution $\mathscr{D}$, with probability $1-\delta$ over the choice of the training set $\mathcal{S}\sim \mathscr{D}$, SAM bounds the generalization error as

$$L\_\mathscr{D}(\boldsymbol{w}) \leq \max\_{\|\boldsymbol{\epsilon}\|\_2 \leq \rho} L\_\mathcal{S}(\boldsymbol{w} + \boldsymbol{\epsilon}) + \sqrt{\frac{k\log\left(1+\frac{\|\boldsymbol{w}\|\_2^2}{\rho^2}\left(1+\sqrt{\frac{\log(n)}{k}}\right)^2\right)+ \frac{4\log\frac{n}{\delta} + \tilde{O}(1)}{n-1}}{n-1}}$$

where $n=|\mathcal{S}|$, $k$ is the number of parameters, and we assume $L\_\mathscr{D}(\boldsymbol{w}) \leq \mathbb{E}\_{\epsilon\_i \sim \mathcal{N}(0,\rho)}[L\_\mathscr{D}(\boldsymbol{w}+\boldsymbol{\epsilon})]$. This condition implies that adding Gaussian perturbations should not reduce the test error, which generally holds for the final solution but not necessarily for all $\boldsymbol{w}$. $w$ represent the parameters influenced by SAM. Applying SAM to the entire network or the entire DFN introduces excessive perturbations, which can hinder DFN’s ability to capture domain-specific information. SAM-SVN strikes a balance by decomposing $w_{DFN}$ and applying perturbations only to the singular value matrix of DFN. The singular value matrix governs the representation space of DFN, thus limiting perturbations to a reasonable spatial range. This prevents DFN from overfitting to the source domain while maintaining its ability to capture domain-specific information during training.

We also quantitatively compared the perturbation on the singular values and other weights in Tab.7.

3. Comparisons with IFA use a batch size of 96:

IFA is set to a batch size (bsz) of 96, so we adopt the same setting of bsz=96 for comparison to ensure fairness, as stated in appendix section D. Moreover, we also found using different bsz leads to different performance in IFA. For example, for Deepglobe, the performance is 50.1 when bsz=96, while it is 44.7 when bsz=1.

4. Discussion on GPRN (AAAI'25)：

Since the work had not been published at the time of our submission, we did not include a comparison. We are now offering a discussion on GPRN. GPRN exploits SAM’s generalizability by converting SAM-extracted masks into semantic prompts, aggregating prompt information through graph-based reasoning, and adaptively choosing feedback points. Our approach highlights that adapters naturally function as decouplers. We explore this concept further, proposing a decoupling strategy that is applicable to a variety of models.

5. Impact statement in page 9:

Thank you very much for your kind reminder, but the official guidelines indicate that the impact statement is not subject to page limitations. Here is the original wording: “Papers must be prepared and submitted as a single file: 8 pages for the main paper, with unlimited pages for references, the impact statement, and appendices."