Segment Anything without Supervision

Dear Reviewer ecwU, thank you for your insightful comments, and we really appreciate that you are willing to increase our score if we can answer the questions positively. We will provide detailed responses to each of them below.

[W1] It Has Not Been Experimentally Proven that Unsupervised Training of UnSAM Outperforms Fully Supervised SAM by Continuously Increasing the Size of the Dataset.

Thanks for your question. We respectfully argues that surpassing the heavily supervised segmentation model SAM with an unsupervised model like UnSAM is not trivial. We are pleased to report that UnSAM's performance improves with an increase in training samples—from 0.1% of SA-1B to 0.4% of SA-1B. We found that by increasing the training samples to 0.4% and using a larger backbone, UnSAM's performance already surpasses that of SAM.

Due to limitations in computing resources, completing the model training on the full 100% SA-1B dataset could take over two months, which may not be feasible within a short rebuttal period. However, the current results already show that UnSAM performs better than SAM on average across seven datasets. We are confident that these performance gains over SAM will further increase as we continue training with more samples.

[W2] Difference in Design Compared to MIS

• High-level Main Differences in Methods: Unlike MIS, which employs top-down clustering primarily for selecting proposals WITHOUT the capability to discover new objects missed by bottom-up clustering methods. In contrast, both bottom-up and top-down clustering in our divide-and-conquer strategy contributes to discovering new objects and parts within an image. This leads to two significant distinctions: 1) Bottom-up clustering relies heavily on low-level features like color and texture, resulting in lower-quality pseudo-masks for instance and semantic segmentation, particularly when dealing with disconnected or overlapping instances. 2) The bottom-up clustering in MIS merges only adjacent groups, often resulting in incomplete segmentation of instances with disconnected parts.
• Minor Differences in Bottom-up Clustering: While MIS utilizes a cut-based method for bi-partitioning the affinity matrix, our approach employs mean-shift clustering, which leverages cosine similarity for merging groups into larger entities. This makes our method more resilient to noise and outliers in the training data.
• Model Performance: To ensure a fair comparison, we trained UnSAM using the same model as MIS, ViT-Base, and present the NoC@85 results below:

The results for MIS and SimpleClick are copied directly from MIS. Our unsupervised model UnSAM significantly surpasses the previous state-of-the-art in unsupervised interactive segmentation and delivers performance comparable to the supervised SimpleClick. And the semi-supervised model UnSAM+ greatly surpasses the SimpleClick model.

It's important to note that datasets such as GrabCut, Berkeley, SBD, and DAVIS are easier and primarily focus on instance and semantic-level masks, typically labeling only a few dominant instances in each image. Consequently, we observe larger performance improvements on the MSCOCO dataset, as detailed in the following table.

[W3] Comparisons with SimpleClick

Thank you for introducing us to SimpleClick! Beyond the results shared earlier, we have also measured the 1-IoU for SimpleClick on the MSCOCO dataset and present the results below:

Our unsupervised model, UnSAM, outperforms the supervised SimpleClick by over 7%, and these gains increase to 17.2% under a semi-supervised setting. The more significant performance improvement on MSCOCO, compared to datasets like GrabCut, Berkeley, SBD, and DAVIS, can be attributed to MSCOCO's complexity with numerous small and heavily overlapped instances, which presents a greater challenge than the datasets frequently used for interactive segmentation studies.

Hope our explanation and experiments can address your inquiries. We will integrate all your valuable comments into our revision!

Dear Reviewer wivK, we appreciate your invaluable insights and thoughtful comments. In the following sections, we address the questions you have raised:

[W1.1] Technical Contributions

Please check our answers in the global rebuttal. Thank you!

[W1.2] Differences with Prior Works

We explained the key differences between UnSAM and prior works (e.g. CutLER and SOHES) in lines 162-173, and described the distinctions in the global rebuttal. Thank you!

[W2] Performance Gain Comes From Additional Thresholds Used?

Since SOHES didn't release its codes, models and data, we reproduced SOHES's results and reported the results of SOHES using exactly the same threshold settings as below:

The performance gain from using additional thresholds is marginal, at only about 0.3%. These results indicate that it is our divide-and-conquer strategy, not the threshold settings, that contributes to the significant performance improvements over SOHES.

[W3.1] UnSAM Uses a Smaller Backbone (RN-50 / Swin-Tiny) and May be Unfair

The results for UnSAM reported in the paper were obtained using a significantly smaller backbone and fewer training samples than those used for SAM, placing our method at a disadvantage. To fully address your question, we conducted additional experiments using a larger backbone, ViT-Base, and have presented the comparative results on SA-1B below:

UnSAM demonstrates superior performance with a larger backbone, outperforming SAM in terms of both Average Precision (AP) and Average Recall (AR), despite being trained on significantly fewer samples.

[W3.2] ViT-Base in Line 139 is a typo. It will be addressed in the revision, thank you for pointing it out!

[W4.1] Inconsistent Figure Caption: Thank you for pointing this out! We will correct the figure caption accordingly. UnSAM's results are displayed in row 3, while SAM's results are in row 2.

[W4.1] Clarification on Key Distinctions with SOHES

Compared to SOHES, UnSAM has 1) better segmentation quality for coarse-grained instance/semantic-level masks. 2) better segmentation quality for fine-grained sub-part masks. For detailed explanations of these improvements, please refer to our responses in the global rebuttal.

[W5] Additional Comparisons with U2Seg

Nice question! We have included the comparisons with U2Seg as below:

UnSAM outperforms CutLER, U2Seg and SOHES by a large margin on all experimented benchmarks.

[Q1 Q2] Did UnSAM and SOHES Use the Same Set of 0.2M Images? If Different Sets of 2% Unlabeled Training Data From SA-1B Are Used, Are The Results Stable?*

Unfortunately, SOHES has not released their codes, models, and training data, preventing us from using the identical set of 0.2M images. However, our results remain robust when training the model with different subsets of 0.2M images.

As shown in the table, we observed that the model's performance remains stable across different sets of 2% unlabeled training data (sampled with 3 different seeds). All three models, each trained with a distinct set, consistently outperformed previous state-of-the-art methods by a large margin.

[Q3] What Does “the two extreme cases” in Line 169 Refer To?

The term "two extreme cases" refers to the semantic/instance-level masks (the coarsest granularity), and the subpart-level masks (the finest granularity). We will clarify it in the revision.

[L1] Limitation Section

We discussed UnSAM's limitations in Section A6 of our submission and will highlight this more prominently in the main paper. Specifically, UnSAM struggles with images containing dense, fine-grained details, often missing repetitive instances with similar textures. Additionally, it tends to over-segment images due to the unsupervised clustering method mistaking details like folds and shadows on clothing as distinct entities—a contrast to human annotators who use prior knowledge to disregard such information. This underscores the ongoing challenge for unsupervised methods to match the performance of supervised approaches.

Hope our explanation and experiments can address your inquiries. We will integrate all your valuable comments into our revision!

Dear Reviewer p86w, thank you for your thoughtful comments. We will provide detailed responses to your questions below:

[W1] What is "UnSAM"?

Nice suggestion. In the paper, UnSAM stands for Unsupervised Segment Anything Model. The pseudo-labeling strategy discussed in our paper is named as divide-and-conquer. I agree that we can make this clearer, and we plan to designate the terms UnSAM (pseudo), UnSAM (whole-image), and UnSAM (promptable) to distinctly refer to the pseudo-labeling method, whole-image segmentation model, and promptable image segmentation models, respectively. We would appreciate any suggestions you might have!

[W2] Why Using AR as One of The Main Evaluation Metrics?

Excellent question. We explained the rationale for using AR as a primary evaluation metric in lines 234-238, following previous works like CutLER (CVPR2023) and SOHES (ICLR2024). But why is AR more appropriate than AP for unsupervised segmentation? The main reason is that all human-labeled datasets, including SA-1B, only label a subset of objects, and AR doesn't penalize models for detecting objects not labeled in these datasets.

Below is a comparison of results on the MSCOCO dataset that further illustrates why AR is preferred:

Despite SAM's overall better performance, its lower AP compared to CutLER highlights that AP can unfairly penalize models for detecting unannotated objects in evaluation datasets, failing to truly reflect a model’s capabilities. I agree that AR is not without its imperfections, but it is a more suitable metric than AP for unsupervised segmentation. Future research on developing new metrics is needed.

[W3.1] The Proposed Model Seems to Generate an Extremely Large Amount of Masks.

Sorry for the confusion—the 2000 queries mentioned in L407 refer to the number of learnable queries in Mask2Former, not the number of masks predicted. Before outputting the final model predictions, we apply non-maximum suppression, confidence score thresholding, etc. Consequently, the average number of masks our model predicts is around 500.

[W3.2] What is The Number of Masks Used for Assessing Recall? Why Using AR$_{1000}$?

We assessed recall using AR$_{1000}$, following the approach used by SOHES. The maximum number of masks per image is set at 1000 for UnSAM and CutLER, and 32*32 (=1024) for SAM. We set the maximum number of masks as 1000 in CutLER's config.

[W4 W5] What's the Difference Between UnSAM and SOHES? Why UnSAM Performs Better Than Prior Works?

Good question! We have outlined the key differences between UnSAM and prior works (e.g. CutLER and SOHES) in lines 162-173. These distinctions are further detailed in the global rebuttal. Thank you for checking!

[W6] Are Different Two Different Models Required?

It is indeed feasible to use the same model for both tasks. However, we choose to employ two distinct models primarily due to differences in inference time. While the Semantic-SAM-based model can be prompted similarly to SAM (using a grid of points) with only a minor performance disparity (< 2~3%), its processing speed is significantly slower—at least 5 times slower than a specialized whole-image segmentation framework like Mask2Former. In future research, we plan to utilize models such as FastSAM or SAM-2 to enhance the speed of interactive segmentation models.

Additionally, we can still effectively construct the hierarchical structure using the masks from Mask2Former by post-processing the results based on mask overlaps.

[W7] CascadePSP

Great question! We follow previous works CutLER (CVPR 2023) and SOHES (ICLR 2024), and utilize CascadePSP (from SOHES) or CRF (from CutLER) for mask refinement (Table 3). We opted for CascadePSP as our default method primarily for two reasons:

1. Efficiency: The primary advantage of CascadePSP is not actually in terms of model performance, but rather in terms of inference speed. In our local small scale experiments, we discovered that the performance difference (in terms of AR) between training UnSAM with CascadePSP-refined pseudo-masks and training UnSAM with CRF-refined masks is only about 2-3%, but the speed difference in refining pseudo-masks is about 5-10 times. Due to our limited computing resources, refining pseudo-masks using CRF would take approximately 3-4 months, making it unfeasible.

2. Consistency with Previous Works: To maintain consistency with SOHES, we used the same refinement method.

To fully answer your question, how can we speed up the overall process for CRF-based mask refinement? One strategy could be to train a CascadePSP model using the "ground-truth" generated by CRF to achieve "fully-unsupervised" mask refinement quickly.

[W8] #images Should Include ImageNet

Since ImageNet was actually used by all methods for pre-training the backbone, including SAM, SOHES and CutLER, and UnSAM was never trained on pseudo-labels from ImageNet, we didn't include it as training data. We'll explain it in our paper.

[W9] Typos and Minors Issues will be addressed in the revision! Thank you for pointing them out!

Hope our explanation and experiments can address your questions. We will integrate your valuable comments into our revision!

Dear Reviewer S6W7, we appreciate your invaluable insights and thoughtful comments! In the following sections, we address the questions you have raised:

[W1] Contributions and Insights

1. Novel and Simple Pipeline: We introduce a simple yet effective divide-and-conquer pipeline for producing high-quality pseudo-masks for unsupervised image segmentation. As no previous work has used this pipeline for unsupervised image segmentation, we believe that our overall pipeline is novel and adds new knowledge to the field. We believe in simple science: we think a simple change leading to substantial performance gains is far more valuable and intriguing than a complex method yielding marginal gains.
2. UnSAM, for the first time, demonstrates that unsupervised image segmentation method can achieve competitive results with the supervised counterpart SAM. In addition, UnSAM achieves over 12% better AR than the previous unsupervised segmentation methods.
3. UnSAM is also the first to show that the state-of-the-art supervised segmentation method, SAM, can benefit from our self-supervised labels—a discovery that has not been previously reported. UnSAM+ exceeds SAM’s AP by 3.9% and AR by over 6.7% on SA-1B.

[W2.1] Model Backbones

The results for UnSAM reported in the paper were obtained using a significantly smaller backbone and fewer training samples than those used for SAM, placing our method at a disadvantage. To fully address your question, we conducted additional experiments using a larger backbone, ViT-Base, and have presented the comparative results on SA-1B below:

UnSAM demonstrates superior performance with a larger backbone, outperforming SAM in terms of both Average Precision (AP) and Average Recall (AR), despite being trained on significantly fewer samples.

[W2.2] Number of Masks Per Point

Great question! We employed additional granularity levels (i.e., more masks per click) because our unsupervised pseudo-labeling method can create a hierarchical structure with more granularity levels than the ground-truth masks from SA-1B. We increased the number of masks per click to fully utilize the advantages of our hierarchically structured data. Despite increasing the number of masks for SAM (two clicks producing six output masks), the performance improvement on MSCOCO was relatively marginal.

As indicated in the table, UnSAM+ surpasses SAM by over 1.4% despite being trained with 100 times fewer samples.

[W3] Typos

Thank you for pointing out these typos! We will correct them and thoroughly review the manuscript before finalizing the paper.

Hope our explanation and experiments can address your inquiries. We will integrate all your valuable comments into our revision!