Gate to the Vessel: Residual Experts Restore What SAM Overlooks

We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:

W1, 5 “Extending the approach to other structures or modalities to validate its generalizability.”:

We agree with the concern regarding the importance of validating the generalizability of our proposed framework beyond vessel segmentation. In response, we have extended our evaluation to multi-class segmentation on the Synapse Multi-Organ CT dataset, which includes eight anatomically diverse abdominal organs. Following [1, 8, 9], we split the dataset into 18 training samples and 12 test samples, and perform corresponding preprocessing and data augmentation methods.

We present the quantitative results in Tab.XII. As shown, our method achieves the highest average Dice score (87.97%) and the lowest Hausdorff Distance (HD) of 7.89 among all compared methods, outperforming strong baselines such as H-SAM and nnU-Net. Notably, our method maintains high segmentation accuracy on challenging small organs like the pancreas, demonstrating that FineSAM++ preserves its segmentation strength and robustness even when extended to more delicate anatomical structures. We will include this supplementary analysis in the final submission to better showcase the generalization capability of our approach and further support the robustness of the method.

Table XI. Comparison to state-of-the-art models on Synapse multi-organ CT dataset.

W2 “Parameter efficiency”:

Thanks for your valuable comment. To address your concern regarding the efficiency of our proposed method FineSAM++ in terms of learnable parameters among SAM-based segmentation methods, we have conducted a detailed comparison. Specifically, we computed the number of parameters, FLOPs (GAMCs), and inference latency under a standard input resolution of (1, 3, 1024, 1024). The results are summarized in the Tab.X below:

Table X. Comparison of model size, computational cost, latency, and segmentation accuracy (Dice score) across segmentation methods using an input of size (1, 3, 1024, 1024).

As shown in the Tab.X, FineSAM++ introduces only 1.4M additional learnable parameters on top of the SAM backbone. Compared to other SAM-based segmentation methods, FineSAM++ demonstrates significantly higher parameter efficiency. Although the total parameter count is higher than lightweight models like Unet, FineSAM++ achieves the highest Dice score (0.8231). These results clearly indicate that FineSAM++ strikes an excellent balance between parameter efficiency and segmentation performance. We will include this comparison in the revised version.

W3 “Ablation study on threshold $\delta$ in the Gating module”:

Thanks for pointing this out. To assess the sensitivity of the Gating module to the pre-defined error threshold $\delta$, we conducted an ablation study on the DRIVE dataset by varying $\delta \in \{0.3, 0.4, 0.5, 0.6, 0.7\}$. The results are summarized in the Tab.XI below:

Table XI. The Ablation Result of threshold δ in the Gating module

As shown, while some metrics are slightly higher, $\delta=0.5$ achieves the best overall performance across all metrics. Overall, $\delta=0.5$ provides a meaningful routing threshold:

• If $\delta$ is too low, almost all pixels are considered uncertain, resulting in unnecessary refinement and loss of gating sparsity.
• If $\delta$ is too high, only a few pixels are routed, making the refinement module underutilized.

Thus, $\delta=0.5$ strikes a balance, activating residual experts in genuinely ambiguous regions while maintaining efficiency. We will include this explanation and the ablation results in the final version.

W4 “A more detailed description of the method, including the design of $g_\theta$ in Equation 2 and the architecture of the expert modules”:

We thank the reviewer for emphasizing the importance of methodological clarity. We will revise the manuscript to provide a more detailed description.

1. Gating Module $g_\theta$ (Eq. 2):

$g_\theta$ is implemented as a lightweight CNN without downsampling. It contains three convolutional layers with kernel size $3 \times 3$, followed by BatchNorm and ReLU activation. It produces two outputs through separate heads:

• Uncertainty map: A $1 \times 1$ convolution followed by a sigmoid activation outputs a spatial confidence map.

• Routing weights: The feature map is globally pooled and passed through an MLP to produce per-image expert routing weights.

2. Expert Modules:

Each expert is a compact 3-level U-Net, where each level contains 2 down-sampling and 2 up-sampling blocks. All experts share architecture but are independently trained to specialize in different error patterns. The final prediction is a weighted sum of expert outputs, gated by the spatial uncertainty and routing weights. We will update the final paper and appendix to support reproducibility.

L1 “Societal impact discussion”:

We appreciate the reviewer’s suggestion to address societal impact. FineSAM++ targets localized failures in medical images, especially in fine-grained regions like vessels with blurred boundaries, by introducing a structure-aware, sparsely activated expert mechanism that improves segmentation fidelity with minimal overhead. This design offers two key benefits: (1) improved reliability of automated tools to support clinical decision-making, and (2) a scalable, resource-efficient adaptation strategy that facilitates deployment in low-resource healthcare settings where retraining large models is impractical.

References

[1] Chen J., et al (2021). Transunet: Transformers make strong encoders for medical image segmentation.

[2] Cao H., et al (2022). Swin-unet: Unet-like pure transformer for medical image segmentation.

[3] Azad R., et al (2022). Transdeeplab: Convolution-free transformer-based deeplab v3+ for medical image segmentation.

[4] Azad R., et al (2023). Dae-former: Dual attention-guided efficient transformer for medical image segmentation.

[5] Rahman M M., et al (2024). Multi-scale hierarchical vision transformer with cascaded attention decoding for medical image segmentation.

[6] Hu X., et al (2023). How to efficiently adapt large segmentation model (sam) to medical images.

[7] Chen T., et al (2023). Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more.

[8] Zhang K., et al (2023). Customized segment anything model for medical image segmentation.

[9] Cheng Z., et al (2024). Unleashing the potential of sam for medical adaptation via hierarchical decoding.

[10] Isensee F., et al (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.

We are grateful for your thoughtful reconsideration and for acknowledging the clarifications we provided. Your constructive comments have been valuable in improving the paper.

We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:

W1 “While it is intuitive to perform refinement on top of coarse masks from a foundation model, it would be important to acknowledge and analyze potential failure cases. In my own experience, refinement modules can sometimes overfit to thin structures, leading to false positives and degrading the initial mask quality. Did the authors observe any such failure modes where refinement worsened the output?”:

We thank the reviewer for the insightful comment. We agree that refinement modules can sometimes overfit to thin structures, potentially leading to false positives or degraded mask quality—especially in regions with weak boundaries or ambiguous textures. This is an important and realistic concern that we have also observed in our experiments.

To address this, we adopt a Degrade strategy in FineSAM++. Instead of feeding all experts the same coarse mask, we introduce randomized degradations:

where $\text{Degrade}(\cdot)$ applies random masking, noise injection, or occlusion perturbations to promote coarse mask input diversity. This encourages specialization across experts and improves robustness to structural uncertainty. We conducted an ablation experiment to evaluate the effectiveness of the degradation mechanism. The results are summarized in Table XV.

Table XV. Ablation study of the Degrade strategy for multi-expert training.

The Degrade strategy consistently improved robustness, reduced false positives, and encouraged specialization across experts. We will include these results and visual examples in the revised submission and supplemental material.

W2: The ablation of progressive optimization strategy

Thank you for the insightful suggestion. We agree that analyzing the effect of the progressive optimization strategy with dynamic weighting is important. To this end, we compared three strategies:

1）Naïve Joint Training: end-to-end optimization with uniform loss weights.

2）Stage-wise (Independent): freeze coarse module, train refinement separately.

3）Ours (Progressive): progressive training with dynamic uncertainty-based loss weighting.

The results are summarized in Table XVI. Our method consistently outperforms both baselines. The progressive setup encourages interaction between modules and focuses learning on uncertain regions, which improves refinement without overfitting. Our method consistently outperforms both baselines. The progressive setup encourages interaction between modules and focuses learning on uncertain regions, which improves refinement without overfitting. We will include these results and discussion in the revised paper. Thank you again for the helpful suggestion. We will include these results and discussion in the revised paper. Thank you again for the helpful suggestion.

Table. XVI: Ablation study on training strategies.

W3 “The Residual Experts appear to be relatively independent from the base model. Have the authors explored lightweight, orthogonal integration methods such as adapters? This could offer insights into whether the refinement process is generalizable”:

We thank the reviewer for the insightful comment. We have indeed explored adapter-based refinement as a baseline, inspired by recent methods such as SAM-Adapter [1]. While adapters provide a parameter-efficient integration mechanism, we found them suboptimal for our task: they struggle to correct fine-grained structures such as thin vessels, bifurcations, and topological continuity.

As shown in Tab. L and Fig. 4 (in manuscript), adapter-based methods often fail to recover fragmented branches or subtle boundaries, especially in high-precision vessel segmentation. These limitations motivated us to explore a mechanism that activates refinement paths only when needed—allowing the model to focus capacity on challenging regions while maintaining overall efficiency.

Table. X: Comparison with SAM-Adapter across vessel datasets.

Mixture-of-Experts (MoE) architectures are well-suited for modeling spatially sparse and heterogeneous error patterns [2–4], which are characteristic of SAM’s failure cases in medical imaging. Our confidence-guided routing module enables spatially adaptive specialization by activating lightweight Residual Experts only in uncertain regions. Unlike bottlenecked adapter layers, these CNN-based experts retain local structural priors critical for topology-aware refinement.

Moreover, while SAM-Adapter [1] focuses on global adaptation in natural scenes (e.g., camouflage, shadows), our method is designed for localized, structure-aware correction—essential in clinical settings where boundary precision and topological integrity are crucial. FineSAM++ consistently outperforms both adapter-based and task-specific baselines across five vessel benchmarks, demonstrating the effectiveness and generalizability of this sparse, modular correction paradigm.

References

[1] Chen T., et al (2023). Sam-adapter: Adapting segment anything in underperformed scenes.

[2] Jacobs R A., et al (1991). Adaptive mixtures of local experts[J]. Neural computation.

[3] Fedus W., et al (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.

[4] Hwang C., et al (2023). Tutel: Adaptive mixture-of-experts at scale.

We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:

W1 “Sparse related work section”:

We appreciate the reviewer’s observation regarding the breadth of the related work. We would like to clarify that our Related Work section (Sec. 2) will include dedicated subsections that cover foundation models, uncertainty modeling, specialized approaches, and vessel-specific segmentation literature:

1)Foundation Models & Uncertainty modeling: We will expand Section 2.1 to cover additional efforts on adapting foundation models for medical image segmentation, including LoRA-based fine-tuning [1], prompt-driven adaptation [2], and SAM-Adapter methods [3]. To address the reviewer’s interest in uncertainty and boundary quality, we will add HQ-SAM [4], which improves boundary precision via high-quality priors and edge-aware refinement. We will also discuss topology-aware uncertainty modeling [5,6], which enhances consistency by quantifying structural uncertainty. These additions better contextualize the potential and limitations of foundation models in segmenting fine-grained medical structures like vessels.

2) Specialized Models for Vessel Segmentation: We will expand Section 2.3 to include recent progress in both traditional and foundation model-based vessel segmentation. Beyond CNN and Transformer methods [7,8], we will add vesselFM [9], a foundation model for universal 3D vessel segmentation, and a retinal disease foundation model [10], which supports generalizable vessel-pathology detection. These works underscore the increasing relevance of foundation models in vascular imaging for both segmentation and clinical insight.

Q1, 3, 4, 5 & W2.1, 2.3, 2.4, 2.5, 2.6 “Baseline selection, data augmentations, missing import baselines and standard deviations”:

We thank the reviewer for raising this important point. For each dataset, we selected baselines that are both highly cited on Google Scholar and representative of state-of-the-art performance, aiming to reflect the most influential methods specific to each task. Importantly, we included several SAM-based methods (SAM-Adapter, H-SAM, AutoSAM, SAMed) across all datasets, along with a vessel-specific baseline (Gupta et al. [5]). We agree that a unified suite of baselines would improve consistency, and we will revise our experiments to add some important baselines:

1. HQ-SAM [4]: A SAM variant specifically designed for fine structural segmentation.
2. nnUNet [11]: A standard network for medical image segmentation.
3. RETFound (modified) [10]: A foundation model for vessel disease detection. We adapt it for segmentation by replacing the classification head with a lightweight decoder.

Furthermore, to ensure fairness, all methods were trained with the same data augmentation strategies, following [5], which includes random rotation, flipping, elastic deformation, and brightness/contrast jittering.

As shown in Table XII and Table 1, HQ-SAM improves over other SAM-based models (e.g., H-SAM, SAM-Adapter, SAMed) via lightweight token-level refinement. However, its frozen encoder and global adaptation lack domain-specific priors and struggle with fine-grained structures such as thin vessels and bifurcations. RETFound performs better structurally due to pretraining, while nnUNet remains stable due to its self-configuring pipeline and strong inductive bias in medical segmentation. Our method achieves balanced and robust performance across datasets, especially on Dice and AUC, critical for fine vessel structures. A more detailed analysis will be given in the revised version.

To better evaluate the robustness of our method, we report the mean ± standard deviation in Table XII. Minor revisions will be made in the final version to include these statistics for completeness.

Table XII. Quantitative comparison on DRIVE, DCAI, CHUAC and ROSE datasets.

W2.2 “Why are the UNet results so variable depending on the dataset?”:

Thank you for the question. The performance variability of UNet reflects both dataset characteristics and inherent model limitations. UNet relies heavily on low-level intensity cues and lacks global context modeling, making it sensitive to modality shifts and low-contrast scenarios. For instance, it performs well on DRIVE (high-contrast RGB) but struggles on CHUAC (grayscale, noisy angiography). This highlights UNet’s limited robustness across diverse medical domains.

Q2 & L1 “Parameter Count Comparison”:

We thank the reviewers for their comments. To better illustrate the computational properties of our method, we compare its performance against SAM-based methods, UNets, and nnUNet. While training time is a relevant consideration, it is not the primary bottleneck or design objective of our approach. FineSAM++ is built upon a frozen backbone and introduces two lightweight adaptation stages, both of which converge efficiently in practice. Accordingly, we focus on parameter efficiency, inference cost, and segmentation accuracy. The results are summarized in Table X below:

Table X. Comparison of model size, computational cost, latency, and segmentation accuracy (Dice score) across segmentation methods using an input of size (1, 3, 1024, 1024).

As shown above, FineSAM++ achieves a Dice score of 0.8231, the highest among all SAM-based methods, while introducing only 1.4M additional learnable parameters. Compared to UNet (1.1 ms latency) and SAM (116.33 ms latency), our method maintains a comparable inference time (117.6 ms) and significantly improves performance over Unet. In contrast to nnUNet, which requires heavy computation (1864.9G FLOPs), FineSAM++ achieves similar accuracy (0.8231 vs. 0.8220) with over 4× fewer parameters and over 30× faster inference.

References:

[1] Cheng Z., et al (2024). Unleashing the potential of sam for medical adaptation via hierarchical decoding.

[2] Chen Z., et al (2025). UN-SAM: Domain-adaptive self-prompt segmentation for universal nuclei images.

[3] Wu J., et al (2025). Medical sam adapter: Adapting segment anything model for medical image segmentation.

[4] Ke L., et al (2023). Segment anything in high quality.

[5] Gupta S., et al (2023). Topology-aware uncertainty for image segmentation.

[6] Huang J., et al (2024). Representing topological self-similarity using fractal feature maps for accurate segmentation of tubular structures.

[7] Qi Y., et al (2023). Dynamic snake convolution based on topological geometric constraints for tubular structure segmentation.

[8] Mou L., et al (2024). CS-Net: Channel and spatial attention network for curvilinear structure segmentation.

[9] Wittmann B., et al (2025). vesselFM: A Foundation Model for Universal 3D Blood Vessel Segmentation.

[10] Zhou Y., et al (2023). A foundation model for generalizable disease detection from retinal images.

[11] Isensee F., et al (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.

We appreciate the reviewer’s feedback and addresss the concerns as follows:

“1.Standard deviation”:

We have calculated the standard deviation for baseline methods. This addition allows for more reliable comparisons in terms of both performance and stability (See Tab.XII).

Table XII. Quantitative comparison.

“2.Statistical significance”:

We performed paired t-tests to assess statistical significance (Tab.XIII), where most improvements are significant (p < 0.05 are bold). Our method demonstrates superior overall performance in segmentation accuracy, consistency (lower standard deviation), and statistical significance.

Table XIII. Statistical significance.

“3.Examination of Tab.X”:

We address the reviewer’s concerns on Tab.X below.

a.Parameter and Latency:

Compared to nnUNet (1864.9G GAMCs), FineSAM++ is ~4.9× more computationally efficient (376.8G GAMCs). The original “30×” was a typographical error based on preliminary theoretical FLOP ratios across scales. We will revise the manuscript.

b.Dice:

The Dice in Tab.X are averaged over all test images per dataset. While our method has higher latency than nnUNet, it outperforms SAM-based baselines with a similar parameter budget.

We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:

Q1&W1 “Parameter efficiency vs. SAM-based method”:

Thanks for your valuable comment. To address your concern regarding the efficiency of our proposed method FineSAM++ in terms of learnable parameters among SAM-based segmentation methods, we have conducted a detailed comparison. Specifically, we computed the number of parameters, FLOPs (GAMCs), and inference latency under a standard input resolution of (1, 3, 1024, 1024). The results are summarized in the Tab.X below:

Table X. Comparison of model size, computational cost, latency, and segmentation accuracy (Dice score) across segmentation methods using an input of size (1, 3, 1024, 1024).

As shown in the Tab.X, FineSAM++ introduces only 1.4M additional learnable parameters on top of the SAM backbone. Compared to other SAM-based segmentation methods—such as SAM Adapter (104.3M), H-SAM (111.3M), and AutoSAM (135.29M)—FineSAM++ demonstrates significantly higher parameter efficiency. Although the total parameter count is higher than lightweight models like Unet, FineSAM++ achieves the highest Dice score (0.8231). These results clearly indicate that FineSAM++ strikes an excellent balance between parameter efficiency and segmentation performance. We will include this comparison in the revised version.

Q2&W2 “Ablation study on threshold $\delta$ in the Gating module”:

Thank you for pointing this out. To assess the sensitivity of the Gating module to the pre-defined error threshold $\delta$, we conducted an ablation study on the DRIVE dataset by varying $\delta \in \{0.3, 0.4, 0.5, 0.6, 0.7\}$. The results are summarized in the Tab.XI below:

Table XI. The Ablation Result of threshold δ in the Gating module

As shown, while some metrics (e.g., SE at $\delta=0.3$) are slightly higher, $\delta=0.5$ achieves the best overall performance across all metrics. Overall, $\delta=0.5$ provides a meaningful routing threshold:

• If $\delta$ is too low, almost all pixels are considered uncertain, resulting in unnecessary refinement and loss of gating sparsity.
• If $\delta$ is too high, only a few pixels are routed, making the refinement module underutilized.

Thus, $\delta=0.5$ strikes a balance, activating residual experts in genuinely ambiguous regions while maintaining efficiency. Therefore, we adopt a fixed threshold $\delta=0.5$ for all five public datasets for consistency. We will include this explanation and the ablation results in the final version.

Q3&W3 “There is no study on the hyperparameters used in the gating module and residual experts, such as normalized routing weights {w_j}. A hyperparameter study would strengthen the contribution”:

Thank you for the helpful suggestion. We address the reviewer's concerns as follows:

1. Routing weights $\{w_j\}$ are dynamically learned.

The weights $\lbrace w_j \rbrace^J_{j=1}$ are not manually defined hyperparameters. They are predicted dynamically by the Gating module $g_\theta$, which outputs both the uncertainty mask $m \in [0,1]^{H \times W}$ and expert routing weights $\{w_j\}$, based on input $x$ and coarse prediction $\hat{y}_{\mathrm{SAM}}$.

2. Softmax normalization ensures stability.

To enable robust and balanced routing, we apply softmax normalization to the predicted weights across experts at each spatial location $\sum_{j=1}^{J} w_j = 1$. This prevents expert collapse and allows smooth specialization without hard routing decisions.

3. Explanation of the indicator function $\mathbb{I}(\cdot)$.

The Gating module is trained using a binary supervision mask $g_t$ defined as:

where $\mathbb{I}(\cdot)$ denotes the standard indicator function, returning 1 if the condition holds and 0 otherwise. This provides supervision for pixels with high prediction error, helping the module identify uncertain regions.

4. On the number of residual experts $J$.

As shown in Tab.3 of our manuscript, we performed an ablation study on $J \in \{1, 2, 4, 6\}$. While $J=6$ achieved the highest accuracy, we chose $J=4$ in the final model to strike a better trade-off between segmentation performance and model complexity. This reduces the number of parameters and computational cost, while still maintaining strong performance.