Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM

Thanks for appreciating our paper as addressesing a critical challenge, contributing to the advancement of robust and adaptable segmentation models. We provide pointwise responses to your concerns below.

Q1. Applicability to real-world non-synthetic and non-medical datasets

As shown in Fig. 3 & Tab. 1 in initial submission, we evaluated our method on the Sim10k dataset, featuring synthetic non-medical street scenes with ambiguous semantics. This challenging context demonstrates our approach's broader applicability beyond medical imaging.

Following your suggestion, we further conducted experiments on KINS dataset for instance segmentation in real-world street scenes. More details and results are shown in Gloabl Table 2 and Global Response 2. Our method performs competitively on KINS, still achieving the superior performance. We will include the results in the revision. Thank you for your insightful suggestions!

Q2. In Table 6, the performance of "No Prompt Ambiguity" is comparable to the full model?

We respectfully point out that this is a factual error based on the empirical evidence presented in our paper. In Table 6, the results for the "No Prompt Ambiguity" variant are significantly lower across all metrics compared to our full model $\mathcal{A}-SAM (Ours)$, clearly demonstrating the necessity and effectiveness of all components in our proposed design. For example, GED↓: 0.308 vs. 0.228, HM-IOU↑: 0.674 vs. 0.717, etc. These substantial differences consistently observed across various performance indicators provide strong support for the critical role of each element in our model, including the prompt ambiguity mechanism.

Q3. Where are the ensembling weights in Eq. 13 obtained?

The original SAM outputs multiple predictions for each prompt to address ambiguity. We manually design an ensemble weight for each prediction and make this weight vector learnable. This approach allows our model to dynamically adjust the importance of each prediction, optimizing the combination of multiple outputs to produce more accurate final segmentations for ambiguous cases.

Q4. Inference speed

Thank you for your valuable suggestion! We've added the inference speed for different methods, as shown below. We find that our method achieves better performance while using less or comparable inference time, demonstrating the superiority and practicality of our approach. This balance of high accuracy and computational efficiency highlights the real-world applicability of our $\mathcal{A}$-SAM model.

Q5. What percentage of the images contain reasonable masks?

These diverse ambiguous masks are obtained from expert multi-annotations provided with the established datasets. Currently, there are no existing standards or methods to evaluate their reasonableness. This lack of standardized evaluation metrics for ambiguous annotations is a recognized challenge in the field. Exploring methods to assess the validity and quality of ambiguous masks could indeed be valuable future work. We appreciate your insightful suggestion, as it highlights an important area for further investigation in ambiguous image segmentation.

Q6. Why select these datasets?

Following the current common setting established in existing wisedom [1-4], we selected datasets specifically designed for evaluating ambiguous segmentation tasks. This approach ensures our work aligns with the standard practices in the field and allows for meaningful comparisons with existing methods. These datasets are widely recognized in the research community for their ability to challenge models with ambiguous segmentation scenarios.

[1] A probabilistic u-net for segmentation of ambiguous images.

[2] A hierarchical probabilistic u-net for modeling multi-scale ambiguities.

[3] Phiseg: Capturing uncertainty in medical image segmentation.

[4] Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty.

Q7. What datasets used in Ablation and Robustness studies?

Ablation and Robustness studies are conducted on the LIDC dataset. This dataset was chosen for its comprehensive collection of lung CT scans, each annotated by multiple expert radiologists, providing a rich source of ambiguous segmentations. These studies evaluate the contribution of each model component and assess performance stability under various conditions.

Dear Reviewer V2bw,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,

We are very glad and appreciate that you had a positive initial impression. Thanks for appreciating our paper as the first work that leverages the inherent properties in vision foundation models for ambiguous image segmentation, demonstrating impressive advantages and value in practical applications. We provide pointwise responses to your concerns below.

Q1. Task setting of ambiguous segmentation

Your assessment of the ambiguous segmentation task setting is entirely correct, namely, providing multiple segmentation hypotheses for ambiguous images.

The key insight of this paper stems from revealing an inherent characteristic of SAM: its non-robust output in response to input prompt perturbations. We innovatively leverage this apparent weakness, transforming it to serve the objective of ambiguous segmentation tasks. Relevant experiments are presented in Figures 3, 4 and Tables 1, 2, 3, 4, 5 in our submission.

Furthermore, we discovered that this design for ambiguous segmentation tasks also incidentally improves the model's robustness to prompt perturbations. Related experiments are shown in Figure 5 of our submission.

Q2. Performance comparison with Mose

Thank you for your valuable suggestion! We implemented Mose [1] in conjunction with SAM as you recommended. The results of quantitative comparsion for ambiguous segmentation on the LIDC datasetare presented as below. We also include the qualitative comparison in Figure 1 in uploaded PDF of Global Response. We can see that our method still achieves superior performance compared to this approach, demonstrating the advantages of our strategy design beyond the use of SAM.

[1] MODELING MULTIMODAL ALEATORIC UNCERTAINTY IN SEGMENTATION WITH MIXTURE OF STOCHASTIC EXPERTS

Q3. Writing structure

Thank you for your suggestion. We agree with this perspective and will reorganize the paper structure as recommended. We will create a new section titled "Training and Inference" in the revision. We will ensure this new section clearly articulates both the training and inference processes, highlighting the connections between them while maintaining the integrity of each part. This change will facilitate readers' understanding of our approach. We appreciate your valuable feedback once again, as it will significantly enhance the overall quality and clarity of our paper.

Q4. Results of original SAM without point/box/mask shift?

Table 1 (Sec. 4.2) compares SAM variants adapted for ambiguous segmentation. As suggested, we also compared these with the original SAM, as shown in Global Table 1 in Global Rebuttal. Results show the original SAM's multi-output capability is insufficient for complex, diverse scenarios, leading to suboptimal performance.

Q5. Trained from scratch or finetuned?

We strongly agree with your assertion that fine-tuning a pre-trained SAM would be superior to training from scratch, as it would be impractical to expend enormous resources to build a large model like SAM from the ground up. Therefore, our approach indeed involves fine-tuning a pre-trained SAM.

Dear Reviewer 2dZP,

Thank you for your comment. Regarding line 213 of the main paper, when we stated "training the model from scratch with randomly initialized weights," we meant to convey that we train all weights outside the SAM component (e.g., PGN, IGN, posterior PGN, posterior IGN, etc.) from scratch, while the weights used for the SAM component (e.g., prompt encoder, image encoder, mask decoder, etc.) are initialized with SAM's original weights. We will clarify this point in the revision. Thank you for bringing this to our attention!

Best,

Authors of Submission 830

Dear Reviewer 2dZP,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,

We appreciate your acknowledgment on our method and that this is the first work leveraging the inherent properties of vision foundation models (SAM) for ambiguous image segmentation*. We will make the following revisions in our updated submission:

• According to your suggestions, we are commited to fine-tuning the structure in Method, by intergrating the related contents of our training and inference pipelines into an unified sub-section ``Training and Inference''.
• Also, we are committed to modifying the statement in Line 213 by: "We train all weights from scratch except for the SAM components, including the modules of PGN, IGN, posterior-PGN, and posterior-IGN. For the components including prompt encoder, image encoder, and mask decoder in SAM, we initialize them with SAM's original weights before commencing training."

We sincerely look forward toincorporating these suggested changes from you and hope that this paper will inspire a broader audience thanks to your constructive feedback.

Yours,

Authors

We appreciate your positive initial impression and valuable feedback. We look forward to revising our manuscript based on your suggestions. Below are our point-by-point responses to your concerns. For brevity, we address recurring issues only once.

Q1. General remarks

<Applicable to real-world non-synthetic and non-medical datatset?> As shown in Fig. 3 & Tab. 1 in initial submission, we evaluated our method on the Sim10k dataset, featuring synthetic non-medical street scenes with ambiguous semantics. This challenging context demonstrates our approach's broader applicability beyond medical imaging.

Following your suggestion, we further conducted experiments on KINS dataset for instance segmentation in real-world street scenes. More details and results are shown in Gloabl Table 2 and Global Response 2. Our method performs competitively on KINS, achieving the superior performance.

<More Details about metrics> We appreciate your suggestion. In the revision, we will include the following content in the appendix:

Generalized Energy Distance (GED): A metric for ambiguous image segmentation comparing segmentation distributions:

$

D^2_{GED}(P_{gt},P_{out}) = 2\mathbb{E}[d(S,Y)]-\mathbb{E}[d(S,S')]-\mathbb{E}[d(Y,Y')]

$

where $d(x, y) = 1 - IoU(x, y)$, $Y, Y'$ are samples from $P_{gt}$, and $S, S'$ from $P_{out}$. Lower energy indicates better agreement between prediction and ground truth distributions.

Maximum and Mean Dice Matching ($D_{max}, D_{mean}$): For medical diagnoses, we define Dice score as:

$

Dice(\hat{Y},Y) = \begin{cases} \frac{2 |Y \cap \hat{Y}|}{|Y|+|\hat{Y}|}, &\text{if } Y \cup \hat{Y} \neq \emptyset
1, & \text{otherwise} \end{cases}

$

We calculate $D_{max}$ for each ground truth $Y_i$ as:

$

{D}_{max} = max \{Dice(\hat{Y}_1,Y_i),Dice(\hat{Y}_2,Y_i),..., Dice(\hat{Y}_N,Y_i) \}

$

Hungarian-Matched Intersection over Union (HM-IoU): This metric calculates the optimal 1:1 match between annotation and prediction using the Hungarian algorithm, better representing sample fidelity. It uses $IoU (Y, Y')$ to determine similarity between samples.

<Insights of modeling granularity within image embedding?> As noted in Lines 185-189, SAM generates multiple candidate masks at different granularities, demonstrating inherent ambiguity related to object granularity. This inspired us to enhance the model's perception of ambiguous objects. We introduce an image generation network (IGN) for object embedding to model this distribution, injecting granularity-aware priors.

Q2. Implementation

<Trade-off coefficients> As noted in Lines 238, the trade-off coefficients are set as α_P = α_I = 1.

<How three masks from SIM10k dataset were obtained?> As noted in Lines 236-237, for the SIM 10k dataset, we select images where pixels from two instances overlap (e.g., instances A and B), and derive the three potential segmentation masks (e.g., only region A, only region B, the union region of A and B).

<Model selection> Following current best practices [1], we employ a 6:2:2 ratio for training, validation, and test sets. Results are reported based on best performance on the validation set. We will include this information in our revision.

[1] A probabilistic u-net for segmentation of ambiguous images

Q3. Fig1

<What’s full granularity?> Full granularity refers to the ensemble of various granular outputs within SAM. This approach leverages the multi-scale nature of SAM's internal representations, allowing for a more comprehensive and nuanced segmentation result.

<Bboxs that are not covering whole region are not considered?> We respectfully disagree, as our approach accounts for this scenario. Due to potential bounding box offsets, the complete segmentation region may not always be fully contained. Our method ensures robust performance even when the initial box doesn't perfectly encapsulate the entire region of interest

Q4. More details

<How GT generated?> We will add the following details to the revised appendix:

The LIDC dataset features manual annotations from four domain experts, accurately reflecting CT imaging ambiguity. Twelve radiologists provided annotation masks. We use the version after the second reading, where experts reviewed and adjusted annotations based on peer feedback, ensuring comprehensive and diverse expert opinions.

The BraTS 2017 dataset includes 285 cases of 3D MRI images (155 slices each) in four modalities (T1, T1ce, T2, Flair). Expert radiologists annotated four classes: background, non-enhanced/necrotic tumor core, oedema, and enhanced tumor core. We overlay these annotations to create binary masks, simulating ambiguous segmentation scenarios.

The ISBI 2016 dataset contains 900 training and 379 testing dermoscopic images. A dermatology expert annotated lesion boundaries in all images, providing a gold standard for segmentation tasks.

For the SIM 10k dataset, we focus on images where pixels from two instances overlap (e.g., instances A and B). From these, we derive three potential segmentation masks: the region exclusive to A, the region exclusive to B, and the union region of A and B.

<What’s SAM amalgamating different granularities? > We acknowledge that the original SAM outputs multiple predictions per prompt to address ambiguity, without integration. Ensemble integration was introduced in Per-SAM [1]. We will clarify this distinction in our revision.

[1] Personalize segment anything model with one shot.

Q5. Evaluation

<Why original SAM not included in the comparison from Sec. 4.2?> Table 1 (Sec. 4.2) compares SAM variants adapted for ambiguous segmentation. As suggested, we also compared with the original SAM, as shown in Global Table 1 in Global Response 1. Results show the original SAM's multi-output capability is insufficient for complex, diverse scenarios, leading to suboptimal performance.

Dear Reviewer YPmC,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,

Last but not least, we have to acknowledge the challenge of elucidating all details of the proposed method regarding data, model and metrics within the confined space of a submission. Nevertheless, we beg to point out that the majority of the content addressed in this rebuttal was $\textcolor{blue}{**already existed in our initial submission**}$, both in the main text and the appendix. For instance, questions regarding <more details about metrics> were addressed in the $\textcolor{blue}{**appendix, Section A.2, Lines 513-525**}$. The inquiry about <How multiple GT generated?> was covered in the $\textcolor{blue}{**appendix, Section A.2, Lines 485-500**}$. It's worth noting that most of the datasets used in this paer, including LIDC, BraTS, ISIBI, etc., are widely recognized benchmark datasets for ambiguous segmentation, following established practices [1-5]. We simply utilized the pre-defined multiple GTs provided by these datasets. Additionally, <Trade-off coefficients> were mentioned in $\textcolor{blue}{**Line 238**}$ in main text. Our responses to these queries primarily $\textcolor{blue}{**emphasize where they are in the original text**}$ rather than introducing new clarification or content.

Hence, we sincerely believe that the areas requiring further clarification in the revision are primarily limited to the following several aspects:

• Adding a definition & clarification of full granularity in Figure 1's caption and in the introduction.
• Including a sentence in Section 4.1 (Dataset, Line 230) to tell the readers to refer to more details in the appendix.
• Expanding Section A.2 in the Appendix (Dataset, after Line 500) with a sentence to account for how the ambiguous segmentation masks are made. We will also add a figure to clarify potential misunderstandings about spatial overlap.

$\textcolor{blue}{**Most other clarifications are extracted from the existing content in the initial submission and appendix.**}$ Furthermore, we sincerely appreciate $\textcolor{blue}{**your ``Applause" that this paper is interesting**}$ and we sincerely hope that some "Flaws" related to clarification issues in this paper can be effectively addressed thanks to your constructive feedback.

Yours,

Authors

[1] A probabilistic u-net for segmentation of ambiguous images.

[2] Phiseg: Capturing uncertainty in medical image segmentation.

[3] Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty

[4] Pixelseg: Pixel-by-pixel stochastic semantic segmentation for ambiguous medical images.

[5] Ambiguous medical image segmentation using diffusion models

Thank you for your thoughtful comments and the opportunity to clarify our work. We are pleased that our initial response addressed some of your concerns. We will now respectfully address the points you further emphasized.

C1: Full granularity refers to the ensemble of various granular outputs within SAM - how is this ensemble computed?

Regarding the full granularity ensemble within SAM, as shown in Eq. (13) and Lines 199-205, we implement a ensemble strategy to $\textcolor{blue}{**weighted summing the multi-granular outputs of SAM**}$ to get an ensembled output map. The texts in initial submission is provided as follow: “*Specifically, given multiple candidate outputs from SAM, represented as {M¹, M², ..., M^n}, where n is the number of scales, we introduce a set of learnable mask weights W = {w₁, w₂, ..., w_n} ∈ R^n. The final mask output is obtained through a weighted sum calculation:

$$\tilde{M}=\Sigma_{i=1}^n w_i \odot \tilde{M}^i,$$

where w₁, w₂, ..., w_n are initialized to 1/n and subsequently fine-tuned to enable the model to effectively perceive object scales. By adaptively integrating masks at multiple scales, the model's perception and modeling capabilities for complex target diversity are further enhanced.*”

C2: I do not think I stated 'Bboxs that are not covering whole region are not considered' in any part of my review so I am quite confused about this reponse

We apologize for the confusion. This response actually corresponds to Question 11 in your initial review regarding Figure 1: “The prompt variation experiment depicted in Figure 1…making the bounding box smaller than an object impacts segmentation more than making it larger”. We initially misinterpreted your point, as it requires some efforts to accurately understand. You were inquiring about the impact of bounding box scaling (both enlargement and reduction) on SAM's segmentation performance, particularly when the box may or may not cover the entire object, right?

First, in Table 1 of the initial submission, both SegGPT and SAM use box shifts that include random pixel offsets in all directions and size scaling from [0.8, 1.2], as stated in Lines 253-254 and 259-260. We found that our method $\mathcal{A}-SAM$ can achieve superious results compared with them.

To further specifically address your interest in how box scaling affects segmentation performance when the box may not fully cover the object, we conducted additional experiments in LIDC dataset (which follows the same setting in Fig. 1 in initial submission). We scale down and up the canonical box prompt for foreground in each image by different scaling factors from 0.8~1.20. The results are as follows. We observed that when the box doesn't fully cover the entire object (i.e., $\textcolor{blue}{**scaling factor < 1**}$), the performance change is more dramatic compared to the standard box. This aligns with intuition. We appreciate your insightful suggestion!

C3: I do not understand the methodology of creating masks for Sim10k and KINS... . 'the region exclusive to A, the region exclusive to B, and the union region of A and B' - why is the intersection of A, B excluded from the first two masks? …

Regarding the methodology for creating masks for Sim10k and KINS, we refer to region of instance A, region of instance B, and their union A∪B. We didn't extensively discuss the intersection of A and B in the last response, because, for segmentation, there typically aren't strictly overlapping regions at the pixel level. If a local region of A spatially overlaps and occludes a local region of B, only the local region of A is visible. Thus, we approach this from a 2D segmentation, pixel classification perspective when referring to 'the region exclusive to A, the region exclusive to B, and the union region of A and B'. From a spatial relationship viewpoint, $\textcolor{blue}{**the overlapping region belongs to the occluder**}$ rather than the occluded object. Thank you for pointing this out!

Thanks for appreciating our paper as harnessing SAM's sensitivity, redeemed as a weakness, to address ambiguous and uncertain predictions. We provide pointwise responses to your concerns below.

Q1. Method details

<How to extract the mean and standard deviation from networks?> As noted in Lines 155-157, the mean and standard deviation of the Axis Gaussian distribution are the two vectors output by the neural network. The input to the network can be the image or prompt embedding, and the network is trained to directly output the mean and standard deviation parameters. This allows for efficient extraction of the distributional statistics from the network outputs, which is crucial for downstream probabilistic modeling tasks.

<How to utilize the ground truth labels in posterior prompt generation network?> As described in Eq.(7), for each iteration of the training process, the method randomly samples one of the possible ambiguous labels from the label set. This sampled label is then input into the posterior prompt generation network, which is responsible for producing prompts that capture the semantics of the given ambiguous label. By iteratively sampling from the set of ambiguous labels and feeding them to the prompt generation network, the model is able to learn to produce effective prompts that can handle the inherent ambiguity in the training data.

Q2. Symbol details

Thank you for your pointing out. The parameters Θ and Φ that you mentioned represent the hyperparameters characterizing two distributions, such as the mean and variance of a Gaussian distribution. $N_i$ and $N_p$ are hyperparameters denoting the dimensions of the vectors. We will provide a more detailed description of these parameters in the revision. We appreciate your attention to detail and the opportunity to clarify this point.

Q3. Missing reference

Thank you! The authors will add the line "Previous research [1] has described..." at Line 169 to provide more context on how the current approach builds upon prior work in this area, as outlined in the referenced publication [1].

[1] A Probabilistic U-Net for Segmentation of Ambiguous Images

Q4. Comparison with more existing models

We appreciate your insightful suggestion! It's worth noting that our method is designed for ambiguous segmentation, while the comparison approach you proposed is designed for conventional deterministic segmentation. To enable a fair comparison, we have adapted our method for deterministic segmentation by ensemble averaging the segmentation results from three sampling iterations into a single output.

We have conducted further comparisons on the overlap dataset and the BraTS 2017 dataset, as mentioned in the references OM-Net [1], DC-UNet [2], and CE-Net [3] you cited. The results of these comparisons are listed as below. As we can see from these results, our method still achieves superior segmentation performance under this setting. We believe this additional evaluation addresses your concern and provides a more comprehensive assessment of our method's performance across different segmentation paradigms.

[1] https://arxiv.org/pdf/1906.01796v2 [2] https://arxiv.org/pdf/2006.00414v1 [3] https://arxiv.org/pdf/1903.02740v1

Q5. Difference between PGN and posterior PGN?

As noted in Lines 171-173, a posterior version for the prompt generation network $F^{post}_{PGN}$, parameterized by $\Theta^{\mathcal{T}}$, is further introduced during the training process. This posterior prompt generation network learns to generate the effective distribution for the prompt embedding when accessing the ground-truth label distribution. The introduction of this additional network component allows the model to better capture the semantics of the ground-truth labels and generate more targeted prompts, improving the performance of models.

Dear Reviewer vXP5,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,