Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer

Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.

---

W1: The triplet loss contributes only marginal improvements, raising questions about its necessity.

The triplet loss, although contributing marginally to numerical performance, plays a critical theoretical role in stabilizing the training process. It ensures feature consistency between the pre-trained SAM, teacher, and student networks, preventing excessive divergence that could lead to noisy predictions or unstable optimization. From a theoretical perspective:

1. The triplet loss acts as a regularizer, preserving the reference model’s general-purpose knowledge during adaptation.

2. It prevents overfitting by maintaining meaningful feature relationships across networks.

3. Its synergy with reverse distillation and direct alignment losses ensures robust training.

While its numerical gains may appear small, its role in stabilizing the mutual learning process is essential. This theoretical significance will be clarified in the revised manuscript.

W2: The method is heavily tailored to SAM, and its generalizability to other foundational models is unclear.

Our framework is not inherently tied to SAM and can be generalized to other foundational models. Theoretical principles such as dynamic teacher-student assignment, reverse distillation, and triplet loss are broadly applicable:

1. Dynamic Role Assignment: Reliance on real-time reliability scoring allows adaptation to any pre-trained model.

2. Reverse Distillation: Promotes diversity and avoids overfitting, critical for models fine-tuned on new domains.

3. Triplet Loss: Ensures consistent alignment with the foundational model while adapting to target domains.

We will explicitly discuss this generalizability and propose extending the framework to other models as future work.

W3: The framework’s sensitivity to hyperparameters (e.g., thresholds for role assignment, loss weights) is not discussed, which could impact reproducibility.

We conducted additional experiments analyzing the impact of key hyperparameters.

1. Threshold for Role Assignment: The values of $\mathcal{T}$ around 0.5 consistently yielded the best or second-best results across datasets, confirming robustness to moderate variations.

2. Loss Weights: Reverse distillation and direct alignment losses were set to the same weight based on their design purpose. Optimal weights were determined as 0.1 for reverse and direct alignment losses and 0.001 for the triplet loss. Extremely high or low weights resulted in significant performance degradation. These values were validated across datasets and generalized to all experiments.

We will include these findings in the revised manuscript and supplementary material to enhance reproducibility.

W4: The triplet loss formulation could be made more intuitive.

To improve clarity, we will revise the explanation of the triplet loss in the manuscript with step-by-step derivations and an intuitive diagram illustrating its role in aligning feature representations between the reference, teacher, and student networks. This will help readers better understand its purpose and implementation.

Q1: Could the dynamic role-switching mechanism be extended to more than two networks?

Our current framework is designed for two networks to simplify the reliability-based role assignment process, as it allows for a straightforward comparison of reliability scores. However, the mechanism can be extended to more than two networks by generalizing the reliability metric to rank multiple networks and dynamically assign roles (e.g., multiple teachers and students). This extension could potentially enhance performance further by leveraging a broader ensemble of models. While we have not implemented this in the current work, we consider it an exciting direction for future research and will discuss this possibility in the revised manuscript.

Q2: Should the original SAM data be added to the table in the supplementary material to make the role selection schemes more intuitive?

We agree with this suggestion. Adding the original SAM data will provide a clearer baseline for understanding the improvements achieved by our role selection schemes. We will update the supplementary material to include this data and highlight the relative gains of each scheme.

Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.

---

W1: Lack of analysis on role switching frequency and its Impact on convergence and performance

We conducted an additional quantitative analysis of the role-switching frequency across training iterations. Specifically, we tracked the number of role switches for both networks during training on three datasets. The results show that the switching frequency decreases as training progresses, stabilizing after several epochs. This behavior aligns with our design goal: early in training, frequent switches allow both networks to explore the target domain, while later, more stable role assignments aid convergence.

We also evaluated the impact of role switching on performance by comparing source SAM, a fixed-role method (WeSAM), and our dynamic switching framework. The dynamic role assignment consistently outperformed the fixed-role method. These results confirm that dynamic switching is crucial for effective domain adaptation.

W2: Lack of visualization of the dynamic Teacher-Student Process

We generated a visualization of the reliability scores for both networks over training iterations. This visualization demonstrates how the reliability-based role assignment mechanism dynamically adjusts roles based on the real-time performance of each network. Early in training, roles alternate frequently as both networks adapt to the target domain. Over time, the network with consistently higher reliability assumes the teacher role more often, reflecting a stabilization in the mutual learning process. This visualization will be included in the revised manuscript or open-source platform to illustrate how roles evolve during training and to provide a clearer understanding of the dynamic teacher-student process.

W3: Lack of detailed visualizations of segmentation improvements

We have prepared detailed visualizations of segmentation results on challenging samples from the CAMO, COD10K, and OCID datasets. These visualizations highlight the improvements achieved by our method, particularly in cases involving occlusion, complex object boundaries, and cluttered scenes. For example, on the OCID dataset, our method produces cleaner and more precise segmentation masks compared to both the baseline SAM and WeSAM, especially when using polygon prompts. These improvements are attributed to the combination of our proposed loss functions, which enhance feature alignment, promote diversity, and ensure robust training. We will include these visualizations in the revised manuscript and provide additional examples on the open-source platform for further clarity.

Q1: Could you provide more theoretical insights or empirical evidence to justify why reverse distillation enhances diversity?

The reverse distillation loss is designed to encourage output diversity by promoting a divergence between the teacher and student networks. We summarize its theoretical and empirical justifications:

Theoretical Insights:

1. Balancing Exploration and Exploitation: Reverse distillation introduces controlled divergence between the teacher and student, forcing the teacher to explore diverse predictions. This ensures the teacher provides richer and more comprehensive guidance.

2. Regularization Effect: It prevents knowledge collapse by maintaining teacher-student diversity, critical for adapting to domains with significant distribution shifts.

3. Optimization Perspective: This loss expands the teacher’s solution space, avoiding local minima caused by excessive alignment with the student.

4. Model Ensemble Parallel: Similar to ensemble learning, diversity between teacher and student predictions strengthens generalization.

Empirical Evidence:

We conducted an ablation study comparing the performance of our method with and without the reverse distillation loss. On the CAMO dataset, removing the reverse distillation loss led to a 0.99 drop in average mIoU, while on COD10K, the performance dropped by 0.91 average mIoU.

In summary, reverse distillation enhances diversity, improves robustness, and synergizes with other loss terms to ensure effective knowledge transfer. These insights will be added to the revised manuscript.

Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.

---

Claims And Evidence: Clarification on "Catastrophic Preservation" in line 256

We appreciate your observation regarding the term "catastrophic preservation". In the revised manuscript, we will rephrase this term to "preventing catastrophic forgetting" to align with standard terminology and clarify its conceptual basis.

Experimental 1: Attribution of improvements to the mutual learning framework

We conduct an ablation study to evaluate the impact of different role assignment strategies (learning paradigms) in our framework. Experimental results are as follows. These results confirm that the performance improvements are directly attributable to our dynamic role assignment mechanism in mutual learning.

1. Role Assignment Matters: Random or reverse assignments disrupt learning, leading to performance drops, while our dynamic role assignment achieves the best results by leveraging real-time reliability evaluation.

2. Reverse Assignment Impact: Incorrectly assigning the less reliable model as the teacher severely hampers knowledge transfer, highlighting the importance of proper role allocation.

3. Dynamic Role Assignment Effectiveness: Our approach ensures stable collaboration and effective knowledge transfer, significantly outperforming fixed or random assignments.

We also counted the number of roles exchanged at different training epochs in several adaptation tasks:

The results align with our design goal: early in training, frequent switches allow both networks to explore the target domain, while later, more stable role assignments aid convergence.

Experimental 2: Performance differences with Prompts (Table 4)

OCID images have dense objects with severe occlusion and overlap, making segmentation difficult. Sparse supervision from box and point prompts often includes multiple overlapping or similar instances, leading to ambiguous boundaries and imprecise masks. Polygon prompts explicitly describe instance boundaries, reducing ambiguity and enabling more accurate segmentation.

WeSAM’s fixed teacher provides stable guidance, which is beneficial when sparse prompts introduce noise or ambiguity. Our dynamic role assignment can amplify noise in challenging scenarios, leading to suboptimal role allocations. WeSAM may be better tuned for sparse prompts, while our framework emphasizes bidirectional learning.

W1: Mutual learning is inherently more robust but comes with higher computational and design complexity, which is a limitation of the proposed method.

We conducted experiments comparing the training time and memory usage and performance of WeSAM, a single LoRA adapter, and our dual LoRA adapter framework on the source$\to$Kvasir-SEG(box) task. The results are as follows:

Our method achieves a balance between computational efficiency and performance. Compared to the single LoRA adapter, the dual adapter only increases training time by 0.2 seconds per iteration and requires 4.4% more memory, while delivering significantly better results. Importantly, our algorithmic optimizations make our method faster than WeSAM (1.3s vs. 1.8s per iteration) despite incorporating a more advanced mutual learning framework.

W2: Insights into loss design motivations

Our loss functions are designed to enhance the effectiveness of the mutual teaching framework by facilitating knowledge transfer, increasing model diversity, and stabilizing training.

1. Direct Alignment Loss: This loss aligns the student network’s predictions with the teacher’s outputs, enabling effective transfer of target domain knowledge from the teacher to the student. It ensures the student learns domain-adapted features from the teacher.

2. Reverse Distillation Loss: Unlike direct alignment, this loss increases the divergence between the teacher and student networks, encouraging the teacher to explore more diverse outputs. By providing the teacher with greater optimization freedom, this loss enhances the robustness of the teacher network.

3. Triplet Relationship Loss: This loss reinforces the feature relationships among the reference model (pre-trained SAM), the teacher, and the student. By aligning their relative feature representations, ensures consistent and robust training, improving target domain adaptation.

Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.

---

Claims And Evidence: About the motivation

We appreciate your detailed understanding of teacher-student networks. Our motivation is not directed at the use of EMA or the exploration role of the student network but rather at the fixed-role assignment in traditional teacher-student frameworks. These methods assume that the teacher consistently provides more reliable predictions than the student, and this fixed-role assumption limits their adaptability.

To address this, we propose a dynamic role assignment mechanism, where teacher and student roles are reassigned based on real-time reliability, enabling more flexible and adaptive knowledge transfer. Additionally, we emphasize robust knowledge distillation by comparing each network's output with the frozen pre-trained SAM. The network retaining more pre-trained knowledge is assigned as the teacher, ensuring cautious and stable updates. This approach reduces error accumulation and catastrophic forgetting.

We will revise the motivation section to clarify these points and include references to related CTTA methods to better contextualize our contributions.

Experimental Designs Or Analyses: About the threshold $\mathcal{T}$ in Eq. (3)

To set the base value for this threshold, we referred to common practices in SAM applications, where thresholds of 0.5 or 0.7 are often used. To further refine this selection, we conducted cross-validation experiments on domains with significant distribution shifts, specifically the CAMO and COD10K datasets. We present more ablation study experiments on $\mathcal{T}$ in the following table:

We observed that 0.5 consistently produced the best or near-optimal results across both datasets, balancing precision and recall. To ensure fairness and consistency across domains, we used 0.5 as the default threshold for all experiments. It generalizes well, achieving significant performance improvements across diverse datasets without domain-specific tuning. We will include these results and their analysis in the revised manuscript to justify the threshold selection and its impact on performance.

Q1: How is $f_{\gamma}$ obtained?

Yes, $f_{\gamma}$ is directly derived from the frozen pre-trained SAM encoder, without any LoRA modifications. It represents the general knowledge learned during SAM’s large-scale pre-training. By using a frozen reference, we ensure that $f_{\gamma}$ provides a stable baseline for evaluating how much pre-trained knowledge the collaborative networks retain.

Q2: Why does computing similarity with $f_{\gamma}$ estimate model confidence, especially if $f_{\gamma}$ is frozen?

The similarity to $f_{\gamma}$ measures how much pre-trained knowledge is preserved during adaptation. Although $f_{\gamma}$ remains frozen throughout, it encodes rich and reliable pre-trained knowledge learned from large-scale datasets. Using $f_{\gamma}$ as a reference allows us to measure how much pre-trained knowledge each network retains during adaptation. Networks with higher similarity to $f_{\gamma}$ are considered more reliable, as their updates are more cautious and less prone to overfitting to noisy, domain-specific features.

This approach has two advantages:

1. Compared to relying on the teacher network determined in the previous step, which may have already drifted from the pre-trained knowledge, $f_{\gamma}$ provides a stable and consistent baseline for evaluating reliability.

2. It is more computationally efficient, as it avoids the need for iterative teacher updates. By dynamically assigning the teacher role to the network closer to $f_{\gamma}$, we ensure robust and cautious knowledge transfer, reducing error accumulation and catastrophic forgetting.

Our experiments validate the effectiveness of this method, demonstrating that it achieves stable and efficient domain adaptation while leveraging the reliability of pre-trained knowledge. We will further elaborate on this in the revised manuscript to clarify its rationale and efficiency.

Relation to CTTA methods

Thank you for pointing out the connection to CTTA methods. While CTTA methods address challenges like error accumulation and catastrophic forgetting, our approach introduces key innovations:

1. Dynamic Role Assignment based on real-time reliability, allowing flexible teacher-student role switching;

2. Robust Knowledge Preservation via the frozen pre-trained SAM, preventing overfitting to noisy features;

3. Bidirectional Knowledge Exchange, enabling mutual learning rather than unidirectional transfer.

We will expand the related work section to highlight these distinctions and include the provided references([1-4]).