Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation

1. Design for the cross-domain part

Our method design also highlights the cross-domain part of CDFSS in the following aspects.

(1) Intuitively, for the cross-domain transferability, the overall semantics, like a whole bat in Fig.1, is much harder to transfer than the separated parts of it, like wings, claws. Based on this intuition, our method disentangles the overall semantics into small and transferable parts, and recomposes them on the target domain through efficient tuning of AFW.

(2) Quantitatively, we validated in Fig.4 and Tab.1 that it is the mismatch between layers that harms the transferability between source and target domains by the domain similarity via CKA metrics. Therefore, our design addresses the mismatch problem and encourages the model to focus on more meaningful matches (typically with higher CKA value), and is verified in Fig.9 to show that our model can indeed highlight these matches, improving the cross-domain transferability.

2. Portability of our methods

Our method has excellent adaptability to other approaches. To verify this, we combined our method with APM and PATNet. Additionally, in response to reviewer HFkz and PmpS, we applied our method to Swin Transformer, demonstrating its suitability for different transformer architectures.

3. Failure cases

Here https://anonymous.4open.science/r/Self-Disentangle-Rebuttal-for-ICML25-706E/failure_cases.PNG, we analyze some failure cases. We found that in DeepGlobe, areas with large contiguous regions tend to experience incomplete segmentation, particularly at the edges, where the segmentation granularity may not be fine enough. The main reasons are twofold: 1) DeepGlobe consists of remote sensing images, which are typically high-resolution, yet for computational efficiency, we standardize the images to 400x400; 2) DeepGlobe demands high local feature recognition capability from the model. ViT-based methods, due to their attention mechanism, are strong in global modeling but relatively weaker in recognizing local features compared to models like ResNet, which have inherent local priors. Therefore, we believe that future research could focus on enhancing ViT's ability to recognize local features in cross-domain scenarios.

4. Background prototype

(1) Role of the background prototype

The final result of segmentation is based on the probability map derived from both the support-set foreground prototype and the support-set background prototype compared with the query feature, rather than relying solely on the background prototype. Thus, the background prototype is complementary to the foreground prototype.

(2) Support-set image and query-set image in the same background class

Although the background of support and query image are not in the same class, most current prototype-based methods（PANet, SSP) utilize a single background prototype to model the background patterns, and have been verified to have good performance in both current works and our paper.

Indeed, it is better to consider different background classes for different images. Therefore, here we further utilize clustering to obtain multiple background prototypes. By comparing the single-background prototype versus multi-background prototypes, we observed a slight performance improvement. However, the gains were not substantial enough to justify the computational overhead of clustering. Although this is not the main focus of our paper, we still appreciate your suggestion.

5. Utilization under 5-shot setting

For the 5-shot setting, the methodology is consistent with the 1-shot approach, with the following differences: 1) During fine-tuning, each task has 5 support images available for learning; 2) Five support foreground prototypes are calculated for the same class and then aggregated by averaging.

6. Minor writing error

Thank you very much for your careful correction. Equation 11 should indeed be $W_{out}$ instead of $W_{in}$​. We will take extra care to correct this in the final version to prevent any minor writing errors.

7. FSS performance

In our response to reviewer PmpS's third point, we validated the effectiveness of our method in the FSS setting. Thank you for your suggestion.

1. Compare with more disentangle-based methods and Discussion on slot attention

Our approach differs from slot attention-based methods [1, 2] in two fundamental aspects: 1) Slot attention primarily disentangles distinct objects through object-centric representation optimization, exhibiting coarser granularity, whereas our method focuses on disentangling different patterns within the same object at a finer granularity level. 2) While slot attention employs additional iterative attention modules (external networks) for disentanglement, we leverage the inherent property of ViT layers that naturally attend to distinct spatial regions, augmented with orthogonality constraints to reinforce semantic separation, without requiring additional networks. To validate our method's effectiveness, we conduct comparative evaluations with two representative slot attention-based disentanglement approaches [1, 2]：

We promise to add the discussion on Slot Attention in the final version for both the related work and experiments.

[1] Locatello, Francesco, et al. "Object-centric learning with slot attention." Advances in neural information processing systems 33 (2020): 11525-11538.

[2] Seitzer, Maximilian, et al. "Bridging the gap to real-world object-centric learning." arXiv preprint arXiv:2209.14860 (2022).

2. Misunderstanding of our cross-comparison design

We would like to point out that we did not deny the effectiveness of cross-comparison of layers. Instead, we surely acknowledge the importance of cross-comparisons, and our method is majorly built upon cross-comparisons. Specifically,

(1) We hold that different layers tend to focus on different patterns that are complementary to each other. Our analysis is based on this characteristic, utilizing it to achieve self-disentanglement, and the complementary of patterns is achieved by the recomposition. The comparison of feature-similarity distributions measured by CKA is intended to demonstrate this dynamic nature, indicating that different layers may still have meaningful connections, making it possible to complement different layers by the cross-comparisons, instead of denying it.

(2) It is because of the meaningful connections across different layers that we design AFW to dynamically assign weights for these cross-comparisons for recomposition, instead of simply forcing the position-wise comparison that ignores the cross comparison. In our experiments (Fig.9 and Tab.5), we also validated the effectiveness of the cross comparison.

(3) Also, as validated in Fig.9, our OSD does not suppress non-diagonal comparisons; its role is to encourage layers to focus on more complementary patterns instead of those repeated ones that have been focused on by other layers. With the AFW module, our model still encourages the cross-comparisons across layers, while the complementary of patterns is enhanced by the OSD module.

3. Computational efficiency

We compared our method with PATNet, HSNet, and SSP. Our method demonstrated higher computational efficiency compared to the other methods. This is because we do not require additional networks but instead leverage ViT’s inherent structure for feature separation.

1. Orthogonal loss weight

We validated the impact of the weight of the orthogonal loss on performance. The results indicate that the optimal choice of the weight is in a wide interval, which means the tuning of this hyper-parameter is not difficult. Additionally, we used the same orthogonal loss weight in the Swin Transformer architecture as we did in the ViT architecture. Its performance indicates that our method design is not sensitive to this weight.

2. Apply to Swin Transformer

Thank you for your suggestion. We further validated the effectiveness of the method on Swin Transformer. Since the layers in Swin Transformer do not reside in the same feature space, two additional steps are required: 1) For features from stage 2 to stage 4, we upsample them spatially to H/4*W/4; 2) For features from stage 1 to stage 3, we add three mapping linear layers (c,8c), (2c, 8c), and (4c, 8c) to map them to the same feature space as that of stage 4. The mapping layers are trained alongside the model during the source domain training phase. The performance results are as follows: our method is well-suited to Swin Transformer, and Swin Transformer shows a significant improvement in performance compared to ViT. Thank you again for your suggestion!

1. The theoretical analysis of the effectiveness of ViT disentanglement

In cross-domain few-shot segmentation tasks, models need to transfer knowledge from the source domain $\mathcal{S}$ with abundant annotations to the target domain $\mathcal{T}$ with limited data. Let $\mathcal{H}$ represent the hypothesis space of the segmentation model. The upper bound of the generalization error for target domain risk $\epsilon_{\mathcal{T}}(h)$ is defined as:

$$\epsilon_{\mathcal{T}}(h) \leq \epsilon_{\mathcal{S}}(h) + d_{\mathcal{H}}(\mathcal{S}, \mathcal{T}) + \lambda,$$

where $h$ denotes features extracted by the encoder, $\epsilon_{\mathcal{S}}(h)$ is the source domain risk, $d_{\mathcal{H}}(\mathcal{S}, \mathcal{T})$ represents the $\mathcal{H}$-divergence (domain gap) between the source and target domains, and $\lambda$ is the irreducible ideal joint risk.

Our approach reduces $\epsilon_{\mathcal{T}}(h)$ through two mechanisms:

1）Adaptive Fusion Weights (AFW): adaptively assigns higher weights to semantically appropriate matches, leading to better alignment (as confirmed by the experiments on the source domain in Answer 3), thereby optimizing source domain output and reducing $\epsilon_{\mathcal{S}}(h)$.

2）Domain-Invariant Component Isolation: Minimizing $d_{\mathcal{H}}(\mathcal{S}, \mathcal{T})$ by isolating domain-invariant patterns (e.g., object parts) via:

$$d_{\mathcal{H}}(\mathcal{S}, \mathcal{T}) \approx \sum_{i=1}^L \sum_{j=1}^L w_{ij}d_{\mathcal{H}}^{(ij)}(\mathcal{S}, \mathcal{T}) \leq \sum_{i=1}^L \sum_{j=1}^L d_{\mathcal{H}}^{(ij)}(\mathcal{S}, \mathcal{T})$$

where $d_{\mathcal{H}}^{(ij)}$ denotes the inter-layer domain discrepancy. By leveraging the self-disentangle property and the orthogonal constraints from the OSD module, inappropriate matches are learned to have small $w_{ij}$, reducing the inter-layer mutual information $\mathbb{I}[\mathbf{h}_i \mathbf{h}_j^T]$, thereby tightening the boundary.

2. Effectness in Swin Transformers

Thank you for your suggestion. We further validated the effectiveness of the method on Swin Transformer. Since the layers in Swin Transformer do not reside in the same feature space, two additional steps are required: 1) For features from stage 2 to stage 4, we upsample them spatially to H/4*W/4; 2) For features from stage 1 to stage 3, we add three mapping linear layers (c,8c), (2c, 8c), and (4c, 8c) to map them to the same feature space as that of stage 4. The mapping layers are trained alongside the model during the source domain training phase. The performance results are as follows: our method is well-suited to Swin Transformer, and Swin Transformer shows a significant improvement in performance compared to ViT. Thank you again for your suggestion!

3. Our methods can benefit general few-shot segmentation tasks and domain generalization tasks

We measure the FSS performance of our methods on Pascal. Pascal consists of 20 classes and is set to a 4-fold configuration in the FSS setup. This means training is conducted on 5 classes, while testing is performed on 15 classes that were not seen during the training phase. The experimental results show that our method can also effectively improve the performance of general FSS tasks.

Under the domain generalization setting, our method, trained on Pascal and tested on FSS1000, demonstrates an improvement in domain generalization.

4. Discussion on computational efficiency

We compared our method with PATNet, HSNet, and SSP. Our method demonstrated higher computational efficiency compared to the other methods. This is because we do not require additional networks but instead leverage ViT’s inherent structure for feature separation.

5. Method section

Thank you for your suggestion; we will work on improving it in the final version.