InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective

We sincerely appreciate your thoughtful feedback. In the following sections, we will provide a detailed response to each of your comments.

---

Q1:Literature reviews for domain-invariant information.

The concept of domain-invariant information was first introduced in prior works on domain adaptive segmentation (DAS), which explored cross-domain invariant features such as edge and structural information (Hoffman et al., 2018). DAS aims to learn domain-invariant representations across multiple domains and follows two main approaches: (1) extraction and refinement of domain-invariant features, where methods like feature disentanglement (Chang et al., 2019) or analysis (Xu et al., 2022) decompose images into domain-invariant (e.g., shapes, edges) and domain-specific (e.g., textures, colors) components, aiming to enhance the former while suppressing the latter; (2) GAN-based domain-invariant feature generation, which employs adversarial training to align domains at different levels: image (Li et al., 2022), feature (Ma et al., 2024), and output (Huang et al., 2022). For example, GLGAN (Ma et al., 2024) integrates multi-scale global and local features to improve cross-domain transferability in remote sensing.

With the introduction of SAM, this domain-invariant concept has gained further attention. SAM's large-scale segmentation pretraining on 11 million images inherently encodes cross-domain commonalities, enabling strong zero-shot generalization. Recent works leverage these universal visual patterns for downstream tasks (Li et al., 2024; Peng et al., 2024). However, these methods rely on complex designs or external data to learn representations. In contrast, we focus on preserving the domain-invariant information in pre-trained SAM for fine-tuning.

Thanks the thoughtful review again, we will further add this discussion in the related work in the revised paper.

---

Q2:Loss and preservation of information

Many recent studies leverage SAM's pretrained capabilities for downstream tasks by fine-tuning. However, when the fine-tuning data distribution is narrow, the model tends to overfit task-specific local features (Wang et al., 2024). We argue that this is mainly because task-specific optimizations will cover or suppress domain-invariant features learned during pre-training.

To substantiate this assumption, we have conducted experiments in Sec. 5.4 to illustrate that the extracted relation works (see Tab. 5) and is domain-invariant (see Tab. 6) .

• Tab. 5: Extracted relations boost other distillation methods (e.g., TinySAM) by 1.7%–5.2% IoU, indicating the preserved information's effectiveness.

• Tab. 6: Applying RM trained on one domain to a completely different domain preserves its effectiveness, suggesting that these transferable relations are domain-invariant and beneficial for fine-tuning.

We further explore the nature of domain-invariant information. We employ relations to represent domain-invariant information, which serves as an implicit yet generalizable characterization that may inherently encode various domain-agnostic properties. Here, we showcase and evaluate structural edge information using the Boundary F1 Score (BFS) (Peng et al., 2023). As shown in Fig.2 (https://anonymous.4open.science/r/InfoSAM-7D61/README.md), InfoSAM with the relation module outperforms other fine-tuning baselines in boundary preservation, demonstrating that this implicit relational encoding effectively extracts richer structural edge features.

Boundary F1 Score comparisons on leaf dataset (threshold=3):

---

References:

1. Hoffman et al. Cycada: Cycle-consistent adversarial domain adaptation. ICML, 2018.
2. Chang et al. All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation. CVPR, 2019.
3. Xu et al. DIRL: Domain-Invariant Representation Learning for Generalizable Semantic Segmentation. AAAI, 2022.
4. Li et al. A stepwise domain adaptive segmentation network with covariate shift alleviation for remote sensing imagery. TGRS, 2022.
5. Ma et al. Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation. TGRS, 2024.
6. Huang et al. MLAN: Multi-level adver sarial network for domain adaptive semantic segmentation. PR, 2022.
7. Li et al. Domain-invariant Representation Learning via Segment Anything Model for Blood Cell Classification. Arxiv, 2024.
8. Peng et al. Learning to Adapt SAM for Segmenting Cross-domain Point Clouds. ECCV, 2024.
9. Zhang et al. Learning Shape-Invariant Representation for Generalizable Semantic Segmentation. TIP, 2023
10. Wang et al. SAMCL: Empowering SAM to Continually Learn from Dynamic Domains. Arxiv, 2024.

We sincerely appreciate your thorough review of our paper and the valuable insights you provided. In the following sections, we will provide a detailed response to each of your comments.

---

Q1:Risk of trivial solutions in relation module (RM).

We clarify that the teacher's RM and student's RM share identical parameters, as described in the Problem Formulation in Sec. 4.2 (lines 185–187).

Moreover, our proposed loss function ($L_{info}$) includes several regularization terms ($\log\_2 \|| G^{T}\_{imr} \||\_F^2, \log\_2 \|| G^{T}\_{r} \||\_F^2 , \log\_2 \|| G^{S}\_{r} \||\_F^2$), which are elaborated in Sec. 4.2 after Eq. (11) and Eq. (13). These terms explicitly promote diversity in the feature distribution and prevent it from converging to trivial solutions.

To further verify the effectiveness of these regularization terms, we conducted an ablation study to assess their impact both qualitatively (through the visualization of relation maps) and quantitatively (through performance on downstream tasks). Both results indicate that the proposed loss with regularization terms effectively extracts domain-invariant features, rather than domain-specific noise, thereby enhancing downstream performance and alleviating the problem of trivial solutions.

• Visualization: We visualize the relation maps and their corresponding statistical distributions evolving from early to late epochs. As shown in Fig. 1 (https://anonymous.4open.science/r/InfoSAM-7D61/README.md), without the regularization terms, the distribution of relation maps becomes increasingly narrow during training, and the domain-invariant information captured by the relation maps becomes less distinct. In contrast, the RM trained with regularization terms maintains a broad relation distribution and a more representative relation map.

• Performance: The regularization terms benefit our method by improving performance, as demonstrated by a 1.0% and 1.8% increase in IoU on the Leaf and Road datasets, respectively.

  Method	Agriculture	Remote Sensing
  	IoU (Leaf)	IoU (Road)
  w/o RT	74.6 ± 0.12	59.6 ± 0.69
  w RT	75.6 ± 0.27	61.4 ± 0.30

---

Q2:Typographical error of $L_{info}$.

Thank you for your careful review. Eq. (14) should indeed be $L_{info}=\lambda_1 L_r+\lambda_2 L_d$. We will correct this typographical error in the revised paper.

---

Q3:Ablation study of various relation modules.

We conducted an analysis to compare different model architectures and explore the number of attention layers for relation module (RM). We compare direct dot product, a linear layer, multiple attention layers, and our proposed RM across multiple experiments on two distinct domains.

The experimental results show that: (1) attention-based RM outperforms other other architectures designs. This indicates that attention mechanism effectively assess the correlations between the input features (i.e., image and mask features), thereby adaptively filtering and enhancing the useful information (e.g., edge details) while reducing redundancy. (2) If we stack an appropriate number of attention layers (e.g., 3 layers) in the RM can be beneficial for capturing key information. However, stacking too many (e.g., five layers) increases training difficulty and risks overfitting. In a nutshell, the current RM design is a trade-off between performance and computational overhead, and it effectively captures the relationships between image and mask features.

Ablation study of various RM:

We appreciate your detailed and valuable review on our paper. We will address each of your comments thoroughly in the following sections.

---

Q1:Insufficient justification for $\alpha=2$.

The core reasons for choosing $\alpha=2$ in matrix-based Rényi's $\alpha$-entropy are as follows:

(1) The primary practical motivations are computational efficiency and alignment with prior works. By setting $\alpha=2$, we enable direct computation of matrix-based Rényi entropy through Frobenius norm operations (Eq.11), eliminating the necessity for eigenvalue decomposition. This optimization reduces time complexity from $O(n^3)$ to $O(n^2)$ ($n$ represents the sample numbers) (Dong et al., 2023), substantially reducing computational costs while maintaining theoretical rigor, particularly advantageous for high-dimensional data analysis (Yu et al., 2019). Additionally, prior research (Miles et al., 2023) has successfully applied Rényi entropy with $\alpha=2$ in segmentation tasks, to align with the established practices in this field, we adopt $\alpha=2$.

(2) For theoretical reasons, if the application requires emphasis on tails of the distribution (rare events) or multiple modalities (distributions with multiple peaks), $\alpha$ should be less than 2 and possibly approach to 1 from above. If the goal is to highlight the dominant mode (the most probable region), $\alpha$ should be greater than 2 to emphasize central tendencies. $\alpha=2$ provides neutral weighting (Yu et al., 2019). Moreover, the Frobenius norm's differentiable and strongly convex properties guarantee rapid convergence in gradient-based optimization algorithms (Boyd, 2004).

(3) Furthermore, we conducted an analysis to evaluate the performance of different $\alpha$ values ($\alpha=1.01, 2, 3$). Following with prior work (Yu et al., 2019), we set $\alpha=1.01$ to asymptotically approach Shannon entropy. The results indicate that $\alpha=2$ achieves the highest verification accuracy while reducing computational overhead by an order of magnitude. This computational gain stems from its exclusive reliance on Frobenius norm operations (Eq. 11), whereas $\alpha=1.01$ or $3$ require eigenvalue decompositions, which are computationally more expensive.

Experiments of different $\alpha$ values in $L_{info}$ :

---

Q2:Unclear definition of $L_{info}$

Thank you for your careful review. Equation (14) should indeed be $L_{info}=\lambda_1 L_r+\lambda_2 L_d$. We will correct this typographical error in the revised paper.

---

References:

1. Dong et al. Optimal Randomized Approximations for Matrix-based Renyi’s Entropy. TIT, 2023.
2. Yu et al. Multivariate Extension of Matrix-Based Rényi's α-Order Entropy Functional. TPAMI, 2019.
3. Miles et al. MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation. CVPR, 2023.
4. Boyd S. Convex optimization[J]. Cambridge UP, 2004.