Stable Segment Anything Model

Q5. Explaination on the performance improvement of SAM+DSP.

A5. Thanks for your helpful comment.

In Table 7 of the main paper, SAM+DSP improves segmentation performance from 43.3 mIoU to 46.8 mIoU with the one-point prompt, where the 3.5 mIoU improvement is not as significant compared to the larger 33.6 mIoU improvement achieved by our full model.

The large performance improvement with the noisy box prompt likely results from training on the HQSeg-44K dataset, a trend observed in other comparison methods, as shown in Table 1 of the main paper.

The one-point prompt is more ambiguous than the noisy box prompt for indicating the desired segmentation areas. Thus, without RTS, the performance improvement with the one-point prompt is much smaller than with the noisy box prompt. Nevertheless, our DSP is effective in handling the challenge posed by the one-point prompt, outperforming other methods. DSP's flexibility enables the model to shift attention precisely to the intended region, compensating for the limited information provided by a single point. Thus, DSP helps guide the attention mechanism toward a broader region that is contextually consistent with the high-quality prompt used during training. This adaptability can produce substantial gains even without RTS, which aligns with the observed results.

Q6. Potential segmentation bias and ambiguity.

A6. Thank you for pointing out this interesting aspect.

We emphasize that our method is designed without inherent bias towards large or small objects. Instead, the DSP adaptively redirects attention to align with the user’s intended target, whether a large or small object. Furthermore, the DRP toggles SAM between deformable and regular grid sampling modes to further mitigate concerns about potential bias. If the prompt explicitly targets small objects, our approach does not constrain the model to prioritize large objects, but instead adapts to the prompt’s ambiguity and specificity.

While our approach does not introduce method bias, we acknowledge that the unbiased model may still be influenced by dataset bias. For example, if the dataset predominantly features large objects during training, the model may skew its predictions towards these large regions, potentially neglecting small objects. Thus, we also highlight the flexibility of our method. If users need to personalize their segmentation targets, such as focusing on specific small object regions, Stable-SAM can be easily fine-tuned to accommodate this requirement for better performance and stability. This fine-tuning process requires only a small amount of training data and minimal computational resources, as demonstrated in L512-L525 of the main paper. This adaptability is a key strength, enabling our approach to effectively mitigate dataset bias and address a wide range of user needs and scenarios.

We have included this discussion in the revised supplementary material.

References

[1] Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey, arXiv 2024

[2] Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey, arXiv 2024

Q3. Analysis of DRP behavior under different prompt qualities.

A3. Thanks for your insightful comment.

We have conducted experiments to analyze the behavior of the DRP across different prompt qualities, as shown in Table 5 of the supplementary material. We copy the table here for your reference.

The results show that the DSP routing weight, α_1 (learned and controlled by DRP), decreases from 0.614 to 0.359 as the number of point prompts increases from one to twenty. This indicates that lower-quality prompts rely more on DSP features to direct attention to the desired regions.

Reviewer UZpL also appreciates that The inclusion of a dynamic routing mechanism is practical and well-justified.

Following your suggestion, we conduct additional experiments to manipulate the output strength of the DRP and examine its impact on segmentation quality for different prompt qualities. Specifically, we evaluate segmentation performance at different α_1 values for both ambiguous (one point) and precise (ten points) prompts. For a fair comparison, we use the same set of point prompts for each α_1 value.

The results show that larger α_1 values (indicating stronger DSP activation) lead to better segmentation performance for the ambiguous one point prompt. This suggests that increasing the output strength of the DRP is particularly beneficial for less informative prompts. In contrast, segmentation performance is less sensitive to variations in α_1 for precise prompts, suggesting that when prompts provide clearer guidance, the system achieves satisfactory results even with lower activation levels.

We have included the discussion and experiments in the revised supplementary material.

Q4. Method comparisons with/without RTS.

A4. Thanks for your constructive suggestion.

We have conducted experiments to analyze the behavior of the RTS for different methods, as shown in Table 8 of the main paper. We copy the table here for your reference.

Robust training is critical for improving model’s segmentation stability, but is usually overlooked in previous works. RTS can guide the model, including our DSP, to accurately segment target objects even when provided with misleading low-quality prompts. The results show that RTS substantially improves the segmentation stability of all the methods, albeit with a slight compromise in performance when dealing with high-quality prompts. Note that our Stable-SAM benefits the most from the application of RTS, which can be attributed to our carefully designed deformable sampling plugin design.

We would like to sincerely thank you for your efforts and valuable comments to improve our work!

Below we address your concerns.

Q1. Motivation behind DSP (Dynamic Sampling Plugin).

A1. Thanks for your valuable comment.

Our motivation has been elaborated in L187-L206, with both quantitative and qualitative analyses (Table 1, Figure 1, and Figure 2) in the main paper. Here, we provide further explanation of our motivation.

Problem motivation: The primary motivation for adjusting the attention sampling positions through learnable offsets is to overcome the inherent limitations of traditional fixed-grid attention, especially in the presence of noisy prompts. In the original SAM architecture, the mask decoder uses a regular grid sampling strategy, assuming that the initial prompt provides an accurate indication of the target region. However, in real-world scenarios where prompts are imprecise, fixed-grid sampling can cause the attention mechanism to focus on unintended areas, often capturing background or partially correct features.

Method motivation: Our approach introduces learnable offsets to adapt the sampling positions dynamically, enabling attention to more precisely target the correct regions, even with ambiguous or noisy prompts. The learnable offsets enable the attention mechanism to "deform" the regions it focuses on in a data-driven manner, akin to how a person might adjust their focus when given unclear instructions. Essentially, the model learns from diverse training examples how to "interpret" and refine ambiguous or noisy prompts, using the learned offsets to direct attention to the most contextually relevant regions of the image.

We have included this discussion in the revised supplementary material.

Q2. Parameter-efficient fine-tuning (PEFT) v.s. full retraining.

A2. Thank you for highlighting the trade-offs between retraining all components of SAM and adopting a parameter-efficient fine-tuning (PEFT) approach.

The decision to use PEFT rather than retraining all components of SAM primarily depends on the scale and type of the training dataset. SAM was pretrained on the SA-1B dataset, containing 1 billion masks and 11 million images, which endows it with a generalized ability to segment a wide range of objects across various domains. In contrast, our downstream dataset, HQSeg-44K, contains only 44,000 images. Due to the significant difference in dataset scale, PEFT is a strategic choice to prevent overfitting while effectively adapting the pre-trained SAM to our target segmentation tasks with a small training budget.

Numerous studies have demonstrated that PEFT methods are particularly effective when adapting large pretrained models to downstream tasks with limited data. This approach retains the generalizable features learned during pre-training, fine-tuning only a small subset of additional parameters. This paradigm is well-established in parameter-efficient fine-tuning research (as summarized in the PEFT surveys [1, 2]). By using PEFT, we aim to retain SAM's foundational representation power while mitigating overfitting, which is more likely to occur if we were to retrain the entire model on a relatively small dataset.

In our case, finetuning only the DSP and DRP modules ensures that SAM retains its generality and adaptability while becoming more robust to noisy and ambiguous prompts, maintaining computational efficiency. In contrast, retraining SAM’s entire model compromises its integrity and significantly impairs performance, as shown by the large performance drop of FT-SAM (finetuning SAM’s whole model) in Table 2 of the main paper.

We have included this discussion in the revised supplementary material.

Thanks for the author's patient response. I understand the author's explanation, but I still retain some different opinions. I think that for the box prompt that encloses multiple objects, it is better to provide user multiple options. For example, for pillows/quilt that are all on the bed, users may still want to segment the bed separately, so providing multiple masks is always beneficial. I really appreciate the seriousness of the author. I decide to raise the score to 6.

We would like to sincerely thank you for your efforts and valuable comments to improve our work!

Below we address your concerns.

Q1. Clarification on the results for SAM in the SBD tests of Table 6.

A1. Thank you for your careful attention to the reported metrics in Table 6.

The performance inconsistency can primarily be attributed to differences in experimental settings.

In our experiments, we set multimask_output = False, while other methods likely evaluate SAM with the default multimask_output = True setting.

As noted in SAM's original code, setting multimask_output = False can yield better results for non-ambiguous prompts, such as multiple input prompts, as shown in https://github.com/facebookresearch/segment-anything/blob/dca509fe793f601edb92606367a655c15ac00fdf/segment_anything/predictor.py#L115C1-L122C38:

multimask_output (bool): If true, the model will return three masks. For ambiguous input prompts (such as a single click), this will often produce better masks than a single prediction. If only a single mask is needed, the model's predicted quality score can be used to select the best mask. For non-ambiguous prompts, such as multiple input prompts, multimask_output=False can give better results.

For a fair comparison, we reset multimask_output = True, retrained our method, and reevaluated the performance of both SAM and our method.

The results show that, with multimask_output = True setting, our reproduced SAM yields results consistent with those reported by other methods. Our method still performs better or comparably to other methods. We have updated Table 6 in the revised main paper.

Q2. Effectiveness of Stable-SAM with increasing number of points.

A2. Thank you for your valuable comment.

We have already conducted ablation studies on the number of prompt points, as presented in Table 4 of the supplementary material. We copy Table 4 here for your reference.

The results show the performance curves of SAM, PT-SAM, and Stable-SAM for varying numbers of prompt points. We also report the performance standard deviation to assess segmentation stability.

The results demonstrate that Stable-SAM achieves larger performance gains when handling lower-quality prompts, i.e., fewer prompt points. As the number of prompt points increases, all methods begin to converge in performance, suggesting that while more prompts generally improve segmentation, the improvements become marginal beyond a certain number, e.g., 7 points in this case. However, our method consistently outperforms others across a wide range of prompt point numbers, highlighting the robustness and adaptability of Stable-SAM in diverse prompting scenarios.

Q3. Potential segmentation bias when targeting the small objects.

A3. Thank you for pointing out this interesting aspect.

We emphasize that our method is designed without inherent bias towards large or small objects. Instead, the DSP adaptively redirects attention to align with the user’s intended target, whether a large or small object. Furthermore, the DRP toggles SAM between deformable and regular grid sampling modes to further mitigate concerns about potential bias. If the prompt explicitly targets small objects, our approach does not constrain the model to prioritize large objects, but instead adapts to the prompt’s ambiguity and specificity.

While our approach does not introduce method bias, we acknowledge that the unbiased model may still be influenced by dataset bias. For example, if the dataset predominantly features large objects during training, the model may skew its predictions towards these large regions, potentially neglecting small objects. Thus, we also highlight the flexibility of our method. If users need to personalize their segmentation targets, such as focusing on specific small object regions, Stable-SAM can be easily fine-tuned to accommodate this requirement for better performance and stability. This fine-tuning process requires only a small amount of training data and minimal computational resources, as demonstrated in L512-L525 of the main paper. This adaptability is a key strength, enabling our approach to effectively mitigate dataset bias and address a wide range of user needs and scenarios.

We have included this discussion in the revised supplementary material.

Q4. Computational efficiency analysis.

A4. Thank you for your constructive suggestion.

We clarify that our method runs the SAM decoder only once, with the DSP and DRP directly integrated into it. This design ensures minimal computational overhead, as the new modules require only a small amount of additional processing time.

We have already conducted a speed comparison (evaluated on the NVIDIA GeForce RTX 3090 GPU) in Table 3 of the main paper. We also include additional comparisons on model performance, parameters, inference speed, and GPU memory usage for different backbone variants in Table 3 of the supplementary material. We copy these tables here for your reference.

The results demonstrate that our approach is lightweight and efficient, with the negligible addition of 0.08M parameters having no impact on the efficiency of the original SAM model.

We have emphasized the efficiency advantages of our method in the revised main paper.

Q5. Writing typo.

A5. Thank you for your helpful comment.

We have already corrected this typo and will conduct further proofreading to improve the overall quality of the manuscript.

References

[1] PA-SAM: Prompt Adapter SAM for High-quality Image Segmentation, ICME 2024

[2] CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Model, ECCV 2024

[3] RobustSAM: Segment Anything Robustly on Degraded Images, CVPR 2024

[4] SAM 2: Segment Anything in Images and Videos, arXiv 2024

[5] AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition, NeurIPS 2022

[6] LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022

[7] Hiera: A hierarchical vision transformer without the bells-and-whistles, ICML 2023

[8] Window attention is bugged: How not to interpolate position embeddings, ICLR 2023

Q3. Review, Comparison, and Extension to SAM 2.

A3. Thanks for your insightful suggestion.

Segment Anything Model 2 (SAM 2) is a unified model for video and image-based promptable segmentation. SAM 2 is a natural extension of SAM to the video domain, incorporating a memory attention module that leverages object and interaction information from previously observed frames. When applied to images, SAM 2 behaves similarly to SAM, with the memory module empty. SAM 2 also collects the largest video segmentation dataset for training. Therefore, for image-based promptable segmentation, SAM 2 benefits from a larger training dataset, a stronger hierarchical backbone [7, 8], and multi-scale feature decoding in the mask decoder.

Our method can be seamlessly integrated into SAM 2 to enhance segmentation stability and performance under prompts of varying quality. We train the original SAM 2 model with the Hiera-B+ backbone on the HQSeg-44K dataset and evaluate its performance on four fine-grained segmentation datasets, using the same experimental settings as in our main experiments. We further apply our Stable-SAM method to SAM 2, resulting in the Stable-SAM 2 variant. We compare Stable-SAM 2 with SAM 2 and other state-of-the-art SAM-based methods in the table below.

SAM 2 outperforms SAM, due to its stronger backbone and larger pretraining dataset. However, SAM 2 still suffers significantly from low-quality prompts, owing to the overlooked segmentation stability problem. Stable-SAM 2 exhibits substantial improvements in segmentation quality and stability, outperforming the original Stable-SAM. This confirms the effectiveness and applicability of our method in addressing segmentation stability across different SAM-based methods.

We have included these discussions and experiments in the revised main paper.

Q4. Inference Speed Comparisons.

A4. Thank you for your constructive suggestion.

We have already conducted a speed comparison (evaluated on the NVIDIA GeForce RTX 3090 GPU) in Table 3 of the main paper. We also include additional comparisons on model performance, parameters, inference speed, and GPU memory usage for different backbone variants in Table 3 of the supplementary material. We copy those results here for your reference.

The results demonstrate that our approach is lightweight and efficient, with the negligible addition of 0.08M parameters having no impact on the efficiency of the original SAM model. We will emphasize the efficiency advantages of our method in the main paper.

We have emphasized the efficiency advantages of our method in the revised main paper.

We would like to sincerely thank you for your efforts and valuable comments to improve our work!

Below we address your concerns.

Q1. Comparisons with more relevant SAM-based methods.

A1. Thank you for your valuable suggestion.

In our initial submission, we focused the comparison on the original Segment Anything Model (SAM) as the primary baseline to isolate and highlight the effectiveness of our method in a controlled, well-understood environment.

Following your suggestion, we compare our method with several recent SAM-based approaches with open-source code repositories. For a fair comparison, we train and evaluate these methods using the same datasets and experimental settings on four fine-grained segmentation datasets, as in the main paper.

The results show that advanced SAM-based methods still suffer significantly from low-quality prompts. This comparison not only highlights the strengths of our approach but also underscores the largely overlooked stability challenges inherent in SAM-based methods under varying prompt qualities.

We have included these experiments in the revised main paper.

Q2. More implementation details of LoRA and adapters in SAM.

A2. Thank you for your constructive feedback.

We introduce Adapter/LoRA modules to the feed-forward network (FFN) of each ViT layer in SAM’s encoder for tuning. During training, we fine-tune only the adapter/LoRA modules and SAM’s prediction layer, with all other parameters frozen. In line with AdaptFormer [5], the adapters consist of small bottleneck layers inserted in parallel into the FFN, containing two MLPs and a GELU activation function between them. The bottleneck’s middle dimension is set to 64 to balance model performance and computational efficiency. In line with LoRA [6], the module uses an encoder-decoder structure to impose a low-rank constraint on FFN weight updates, injecting small trainable rank decomposition matrices into each layer. In our experiments, the rank of LoRA is set to 4 for efficiency and performance optimization. All other experimental settings remain the same as those of the baseline and full model.

We have included these implementation details in the revised supplementary material.

Q3. Inference Speed Comparisons.

A3. Thank you for your constructive suggestion.

We have already conducted a speed comparison (evaluated on the NVIDIA GeForce RTX 3090 GPU) in Table 3 of the main paper. We have also included additional comparisons on model performance, parameters, inference speed, and GPU memory usage for different backbone variants in Table 3 of the supplementary material. We copy those results here for your reference.

The results demonstrate that our approach is lightweight and efficient, with the negligible addition of 0.08M parameters having no impact on the efficiency of the original SAM model.

We have emphasized the efficiency advantages of our method in the revised main paper.

Q4. Model scalability to noisy training data.

A4. Thank you for your insightful question.

Increasing the amount of training data could improve the model’s robustness and generalization. More noisy data could help the model learn to handle a wider variety of input prompts, especially in real-world scenarios where user-provided prompts are often imprecise or ambiguous. As the model is exposed to more diverse and challenging inputs, it may become better at distinguishing relevant features and handling uncertainty in segmentation. However, introducing excessive noisy data could cause the model to overfit to the noise, leading to instability in some cases. Our method, which dynamically calibrates attention based on prompt quality, mitigates some risks of noisy data by guiding the model to focus on relevant image regions without being overwhelmed by noise.

We also understand your concerns about the scalability of our method when scaling up the training data. While our method is designed to efficiently handle noisy data with limited resources, it can still benefit from larger training sets. As the training dataset grows, we can fine-tune additional parameters of Stable-SAM to further improve segmentation stability. Our method’s lightweight design, with only 0.08M learnable parameters, allows for fine-tuning additional parameters without significantly increasing model complexity or computational cost. This additional fine-tuning of more parameters applied to larger datasets, can further boost performance and stability, as the model adapts more precisely to the increased diversity of data and ambiguous prompts. This highlights the flexibility of our approach, which performs well with limited data but can also benefit from additional fine-tuning of more parameters when the training dataset increases in size and noise.

We have included this discussion in the revised supplementary material.

We would like to sincerely thank you for your efforts and valuable comments to improve our work!

Below we address your concerns.

Q1. Potential segmentation bias when targeting the background.

A1. Thank you for pointing out this interesting aspect.

We emphasize that our method is designed without inherent bias towards large or small objects. Instead, the DSP adaptively shifts attention to match the user’s intended target, whether a foreground object or background area. Additionally, the DRP toggles SAM between deformable and regular grid sampling modes to reduce concerns regarding potential bias. If the prompt explicitly specifies the background, our method adapts accordingly, without constraining the model to prioritize the foreground.

Although our method avoids introducing bias, we acknowledge that the model may still be influenced by dataset bias. For instance, if the training dataset predominantly contains foreground objects, the model may skew predictions towards the foreground, potentially neglecting background regions. Thus, we also highlight the flexibility of our method. If users wish to personalize segmentation targets, such as focusing on specific background regions, Stable-SAM can be easily fine-tuned to meet this requirement. This fine-tuning process requires minimal training data and budget, as shown in L512-L525 of the main paper. This adaptability is a key strength, enabling our approach to effectively mitigate dataset bias and address a wide range of user needs and scenarios beyond typical foreground segmentation.

We have included this discussion in the revised supplementary material.

Q2. User study of noisy boxes for realistic application scenarios.

A2. Thank you for your insightful comment.

In response to your suggestion, we conduct a user study to provide a more realistic evaluation of our method, using noisy box annotated from users across four fine-grained segmentation datasets: DIS (validation set), ThinObject5K (test set), COIFT, and HR-SOD. Five participants are asked to provide box annotations for the highlighted target object in each image. Participants are instructed to complete each box annotation within 5 seconds to ensure consistency throughout the process. Annotations are collected using the Label Studio platform. The user-annotated boxes are subsequently used as prompts to evaluate the performance of each segmentation method.

The results show that user-annotated boxes provide better segmentation performance across all methods compared to generated noisy boxes. We also assess the quality of the user-annotated boxes by comparing them to the ground truth boxes derived from the mask annotations. The average IoU between user-annotated boxes and ground truth boxes is approximately 0.753, indicating that user-annotated boxes are more accurate than generated noisy boxes. Under the user-provided box prompt, our method continues to outperform other methods by a large margin.

We have included this experiment in the revised supplementary material.