UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model

We thank the reviewer for the feedback and positive assessment.

To further support the reviewer’s inclination toward acceptance, we would like to highlight the linked code repository in the main paper, which demonstrates the ease of use of our proposed framework. We remain open to any additional questions, remarks, or suggestions to refine our proposed paper during the author-reviewer discussion.

We acknowledge the in-depth review that recognizes our strengths and provides valuable directions for addressing flaws, particularly regarding related methods. Below, we address questions and concerns. Issues already covered in other responses are only referenced.

---

Claims

Thank you for pointing out that this claim in the abstract is not sufficiently elaborated in the introduction. A more detailed discussion of SAM and its implications for UQ is provided in Section 2.3, "UQ in SAM". Most UQ methods are based on MC strategies and motivated by Bayesian principles (see Equation 1 ff.). While these methods work well in practice, their theoretical justifications rely on assumptions that are often unrealistic and cannot be guaranteed. For example, test-time augmentation (Wang et al., 2019), used to estimate aleatoric uncertainty, assumes that the sample is in-distribution. Similarly, epistemic UQ estimators, such as model ensembles, assume that uncertain samples are out-of-distribution. These issues, which already exist in well-established UQ approaches, are further exacerbated by the task-agnostic nature of SAM. To the best of our knowledge, there is no general discussion in the literature on SAM's implications on UQ. However, we acknowledge that our formulation of this claim is unclear. To address this, we suggest adding the following to line 126 (right column):

"While the presented methods work well in practice, the theoretical foundations for separating aleatoric and epistemic uncertainty remain debatable and under active discussion (Gruber et al., 2023). The task-agnostic nature of SAM further amplifies this challenge, as discussed next. More..."

Weakness 1

As the reviewer mentioned, several papers propose UQ methods, such as [1] and [2]. The first encodes variance into the latent to sample ambiguous segmentations, while [2] estimates pixel-wise epistemic uncertainty to improve finetuning. They are designed for specific tasks and not directly comparable to us. USAM neither samples mask nor estimates pixel-wise uncertainty, making it unsuitable for both.

However, our UQ comparison already aligns with the comparisons performed in [1] and [2]. In [1], UQ performance is evaluated by applying prompt perturbations (similar to our H_XP) to SegGPT and image augmentations (similar to our H_Y) to SEEM. Note that by changing the frameworks to SegGPT and SEEM, the ambiguous segmentation task is evaluated but UQ is not evaluated in isolation. In [2], a pixel-wise equivalent to our "Entropy" is used, termed SUM Confidence-Uncer, alongside a pixel-wise equivalent to our H_A, termed SUM Discrepancy-Uncer. These study designs underline that standard UQ methods are suitable for evaluating new UQ approaches. Nevertheless, we acknowledge that our paper does not sufficiently address the variety of UQ approaches used in conjunction with SAM. We propose to expand Section 2.3 ("UQ in SAM") and include a discussion of more related works that propose or employ UQ with SAM, such as [1] and [2].

---

Q1

"For several tasks, quantifying the uncertainty of SAM is of particular interest". We mention some tasks in the proposed paper: For example, Vujasinovic et al. (2024) use UQ to estimate if supervision is required to correct tracking errors, or in the healthcare domain to gain trust and improve prediction (Deng et al., 2023; Zhang et al., 2023). The papers [1] and [2] mentioned by the reviewer employ UQ in SAM to sample prediction or to improve finetuning. We will add these additional references to the camera ready.

---

Q2

A similar concern was raised by reviewer nnaG. We agree that the simplicity of our MLP does not sufficiently support claims of superior efficiency. To address this, we conducted runtime experiments demonstrating that USAM incurs minimal computational overhead and is significantly faster than MC-based methods. For details, see our response to reviewer nnaG, Weakness 3, or run the experiment in our anonymous repository https://anonymous.4open.science/r/UncertainSAM-C5BF/scripts/speed_test.ipynb. We propose adding these findings to the camera-ready version.

---

Q3 and Q4

Our submitted paper contains ambiguities regarding our use of SAM vs. SAM2. For a detailed clarification, please see our introduction of the response to reviewer 6x8E.

---

Q5

We are uncertain how our method could be applied to the ambiguous segmentation benchmark. As stated by [X], the goal of this benchmark is to achieve "better agreement between prediction and the ground truth". In contrast, our method estimates the variance of the distribution (i.e., uncertainty). Since we do not modify SAM’s mask prediction, USAM’s segmentations would be identical to vanilla SAM’s.

[X] Rahman, Aimon, et al. "Ambiguous medical image segmentation using diffusion models." CVPR. 2023.

We want to thank the reviewer for highlighting our strenghts and asking questions that are valuable for eliminating weaknesses and enhancing our paper in terms of readability and evaluation.

---

Before addressing the questions, we need to clarify an issue that affects this review as well as reviewer 3EoJ. After carefully re-reading the paper with temporal distance, we acknowledge that our description of the usage of SAM2 is misleading and needs to be improved:

• First, line 254 (right column), "We combine the evaluation of SAM and SAM2," is misleading because we do not combine the models during evaluation. Rather, we combine the datasets evaluated in the corresponding papers. We do this to provide a broader overview that also includes the latest data publication from SAM2.
• Second, line 242 (right column), "We use the pretrained SAM2," suggests that we use the video segmenation mechanism of SAM2, which is designed for segmenting temporally consistent masks in videos. In reality, while we use the pretrained SAM2 model, we only use its single-image implementation, which is equivalent to SAM and does not include the video extension. We do this because the authors of SAM2 provide more stable code, better interfaces, and slightly better accuracy.

Potentially, our MLPs could be used ad hoc to estimate uncertainty from the refined tokens of SAM2 applied to videos, but that is not evaluated and is beyond the scope of this paper. To prevent misunderstandings for future readers, we will avoid the term SAM2 and suggest the following modifications in the camera-ready version:

Replace: "In our experiments, we use the pretrained SAM2 (Ravi et al., 2024)" with "In our experiments, we use SAM models pretrained by Ravi et al. (2024)."

Replace: "We combine the evaluation of SAM and SAM2 ..." with "Combining the dataset selection of Kirillov et al. and Ravi et al., we ..."

---

Weakness and Question 1: Why the mask and IoU tokens can reflect uncertainty

(Un)certainty refers to the likelihood that the prediction is correct. In our case, correctness is measured by the IoU metric, making the expected IoU an ideal proxy for uncertainty. In general, estimating uncertainty does not require ground truth. For example, predictive uncertainty is often described as the variance of the predictive distribution. As seen in Equations 1 and 2, this distribution does not depend on the ground truth. Therefore, the absence of ground truth during inference does not invalidate the expressiveness of our method that uses tokens for uncertainty estimation.

Additionally, the SamScore in SAM (Kirillov et al.) is based on the same principle and also does not have access to ground truth during inference. However, the question of whether and how the mask and IoU tokens reflect uncertainty is an interesting one:

Essentially, the MLPs attempt to map patterns from the object’s latent space (i.e., the tokens) to the expected IoU gap (i.e., uncertainty) based on statistical evidence. The already given effectiveness of the SamScore suggests that such patterns exist in general.

To further investigate and provide more details for future readers, we conducted an additional ablation study. In this ablation, we set either the IoU token or the mask token to 0 to remove potentially informative patterns. We evaluated using the tiny model across the three tasks described in the submitted paper on the COCO dataset. The results indicate that both tokens contribute to the predictive capability, yet they remain accurate as standalone features. Both in combination lead to the best results. The results are:

Token to Uncertainty Ablation in relative AUC in %

We will add this interesting findings to the camera ready version.

Weakness 2: The MLPs' design choice

The reviewer’s remark about our MLP design choice is similar to the comment from reviewer nnaG, indicating that our current explanation lacks clarity. There was no dedicated architecture design process for the UQ estimator. Instead, the architecture is motivated by SAM’s original MLP design for predicting the SamScore. We differ only in the input dimension and hidden states (increasing from 256 to 512) because we use and concatenate both tokens instead of using just one.

The sigmoid activation function is a reasonable choice since the expected IoU gap ranges between 0 and 1. To improve clarity, we suggest modifying the method description and adding a figure to illustrate the MLP architecture. Please see our response to reviewer nnaG for details on the suggested changes.

Thanks to the reviewer for the detailed feedback and comprehensible remarks that will help enhance the paper. Our responses to the proposed improvements are as follows:

---

Weakness 1

We only partially agree with the statement that we do not compare to SAM-specific UQ. For example, our Prompt UQ is closely related to Deng et al. (2023, see main paper for reference). Furthermore, SAM-specific UQ is usually application-specific and not readily translatable to our holistic view of UQ. However, this weakness will be addressed with a modification of the paper, which is further elaborated in our response to reviewer 3EoJ, who identified a similar concern. Please see response to reviewer 3EoJ for the details.

---

Weakness 2

We agree that the architecture is underdescribed. It is motivated by SAM's original MLP design for predicting the SamScore. We differ only in the input dimension and hidden state (from 256 to 512) because we employ and concatenate both tokens. We suggest the following modifications to clarify this in the paper:

• Line 237: "The simple design of the MLPs aligns with the architecture of SAM’s inherent MLP for predicting the SamScore, as described in the Appendix of Kirillov et al. (2023). They have three layers and a sigmoid activation, but we use 512 input dimensions and hidden states due to the concatenated input tokens. The architecture is visualized in Figure XX."
• In addition, we will include a figure presenting an MLP with the input tokens.

Regarding SMAC3: Due to space limitations, we presented the upper and lower hyperparameter bounds optimized using SMAC3 in ll. 247–252 (right column). We suggest adding a more detailed description and presenting the per-iteration results in tabular form. Since the optimization is not the primary focus of the paper, we will include this in the Appendix and refer to it in line 252. The SMAC3 results will provide greater transparency regarding the robustness of the proposed method.

---

Weakness 3

Our proposed MLPs are very small compared to the entire SAM model and require only a single forward pass, unlike MC-based methods. However, as mentioned by the reviewer, an explicit performance comparison should be included in the final version of the paper to demonstrate the impact of our method on efficiency. To address this, we designed hardware-dependent runtime experiments and added them to our anonymous code repository at https://anonymous.4open.science/r/UncertainSAM-C5BF/scripts/speed_test.ipynb for reproducibility.

To illustrate the practical efficiency impact, we conducted tests on a small consumer GPU (GeForce RTX 3050) using setups described in the paper. We compare us to vanilla SAM, entropy, and MC sampling (image and prompt) as employed in our paper. Notably, we performed the test with all nine (!) MLPs presented in our paper. Thus, the efficiency of our method could be increased by up to a factor of nine if only specific MLPs are necessary for a given application. It shows that our method is superior to all other extended UQ methods. We thank the reviewer for this suggestion and propose adding the following results to the camera ready:

Large Model

Base+ Model

Small Model

Tiny Model