The Devil is in the Object Boundary: Towards Annotation-free Instance Segmentation using Foundation Models

Thank you for your detailed review. In order to facilitate a prompt discussion and exchange of ideas, we now address your most critical concerns first. We will respond to all questions and revise the paper as soon as possible. Thanks for your understanding!

W1: The statement used - The devil is in the object boundaries; is it not in line with a part work, which hypothesized - "all you need are priors"?

Thank you for your valuable suggestions! In fact, "all you need are priors" is among the title candidates we consider. However, we ultimately choose "the devil is in the object boundaries" because we want to highlight that discerning object boundaries is the crucial aspect of how to empower foundation models for annotation-free instance segmentation.

While we recognize that "the devil is in the object boundaries" may not be the optimal choice, we will continue to explore a more suitable title (such as "Unveiling The Object Boundary: Empower Foundation Models Towards Annotation-free Instance Segmentation"). If you have any suggestions or recommendations, please feel free to share them with us. Thank you!

W2: Although the analytics presented in Eqns. (1) - (7) are clear, unable to find much of novelty in them. eg Cosine similarity, boundary score etc.

Although the implementation of our Zip may seem straightforward, involving simple calculations in Equations (1)-(7), the observations, motivations, technical contributions, and innovations behind Zip are far from trivial.

Moreover, it is precisely due to its simplicity that our method not only demonstrates effectiveness in terms of performance but also remarkable efficiency. When compared to the previous best-performing method, CutLER, under default settings, our algorithm typically delivers a 10 times increase in processing speed.

In addition, our novelies are summary as follows:

• We are the FIRST to effectively utilize foundation models (CLIP and SAM) for annotation-free instance segmentation. Such effective utilization is not trivial and cannot be achieved by a naive combination. Simply combining SAM and CLIP results in unsatisfactory performance due to SAM's semantic-unaware issue of generating masks with varying degrees of granularity yet lacking instance-aware discernment (see Appendix B) and CLIP's misclassification issue (see Appendix F). Based on these observations, we propose that the key to effectively utilizing foundation models for the instance segmentation task is to probe foundation models to delineate the boundaries between individual objects!
• We are the FIRST to discover CLIP's special intermediate layers can outline object edges through the proposed novel semantic-aware feature clustering. By incorporating this discovery, we address challenges in complex scenarios with multiple instances, under annotation-free instance segmentation. To ensure stable and general clustering of object edges, we devise a semantic-aware clustering initialization technique, which is crucial.
• We are the FIRST to propose a novel classification-first-then-discovery pipeline for annotation-free instance segmentation task. Mere identification of boundary information does not inherently and straightforwardly aid instance segmentation without our classification-first-then-discovery pipeline. In the pipeline, we introduce a boundary metric to identify boundaries and transform the instance segmentation challenge into a fragment selection task, a novel approach not explored by previous work.

Dear Reviewer 5mwu,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

W1: Limited technical novelty. I feel that leveraging the features on particular middle layers of CLIP and applying K-Means clustering on CLIP's features are not technically novel but well-engineered. Of course, the proposed annotation-free instance segmentation pipeline is interesting. This paper seems to be an engineering paper that addresses how to effectively use CLIP and SAM and is not far from an academic paper.

Thank you for your thoughtful review and your comments regarding our technical novelty. We understand your perspective and acknowledge that the application of K-Means clustering to CLIP’s features may seem like an engineered technique when viewed solely from the algorithm execution process. We would like to clarify the technical novelty and contribution that underpin our work as follows:

• Semantic-Aware Initialization: we propose a novel semantic-aware initialization technique to leverage semantic activation of CLIP to automatically initialize clustering centers and determine the number of clusters. Without our semantic-aware initialization, K-Means fails to produce general and reliable clustering results for delineating object boundaries. As shown in Figure 5, despite clustering the same intermediate features in CLIP, it is only when utilizing the proposed initialization technique that stable clustering and boundary discovery are achieved. The core motivation behind semantic-aware initialization is that by initializing certain clustering centers near the outer edges of the semantic activation map, boundaries between objects within the activation map can be clustered and outlined (Please refer to Appendix L for a comprehensive explanation). Semantic-aware initialization is a unique technical contribution that sets our work apart from conventional clustering approaches.
• Classification-First-Then-Discovery Pipeline: we propose a classification-first-then-discovery pipeline, which represents a novel paradigm in the field of annotation-free instance segmentation. In the pipeline, we introduce a boundary metric to identify boundaries and transform the instance segmentation challenge into a fragment selection task. This pipeline has not been explored in prior work and offers a fresh perspective on solving the instance segmentation problem.
• The Boundary Discovery Property of CLIP: we are the first to discover that CLIP's specific intermediate layers can delineate object edges, marking an interesting academic finding rather than just an engineering achievement. As discussed in Appendix C.1, we believe that the hierarchical representation structure, distinctive pretraining approach, and training data may be the reasons why CLIP's intermediate features have the potential for instance-level localization. Similar to the salient object discovery capability of the DINO model and the presence of in-context learning and prompting learning in LLM, we hope that this discovery can assist the academic community in better understanding CLIP and exploring its various features and underlying principles of contrastive language-image pre-training.
• Clear Motivation and In-depth Analysis with Academic Significance: Instead of simply combining SAM and CLIP, we conduct a comprehensive analysis of DINO, SAM, and CLIP. This allows us to clearly illustrate our motivation for effectively leveraging foundation models, rather than superficially combining them without thorough consideration. We believe that the effective utilization of CLIP and SAM for annotation-free instance segmentation, as demonstrated in our paper, is a valuable contribution to both the engineering and academic communities.
• Simple yet Super Effective and Efficient: Through our in-depth analysis and well-explored motivation, we were able to propose a method that is both super simple but also super effective. Furthermore, thanks to its simplicity, it is also highly efficient, achieving a 10-fold increase in processing speed compared to previous state-of-the-art unsupervised object discovery methods like CutLER.

W2: I wonder if the target dataset was human part segmentation or medical segmentation, the CLIP features are still valid for annotation-free instance segmentation.

In addition to the COCO dataset, we apply the same clustering algorithm and exactly the same hyperparameters on the Pascal VOC dataset (as shown in Table 3), and achieve consistently positive results.

Following your suggestion, we also conducted an evaluation of our method on the human part segmentation task, as depicted in the newly added Figure 12. We employ the same clustering method and hyperparameters for this evaluation, and the results further demonstrate the generality of our discovery and the robustness of our approach.

Dear Reviewer CLaV,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Thank you for your detailed review. In order to facilitate a prompt discussion and exchange of ideas, we now address your most critical concerns first. We will respond to all questions and revise the paper as soon as possible. Thanks for your understanding!

W1+W2+Q2: Figure 1's Layout, Figure 2's Clarity, and Visualization in Figure 2

Thanks for your suggestion. We will improve the alignment in Figure 1 and enhance the clarity of boundaries in Figure 2 to enhance visualization.

W3: Understanding Figure 3.

Thank you for pointing this out. We will revise the caption as follows: ZIP follows a classification-first-then-discovery approach, consisting of four steps: 1) Classification first to obtain semantic clues provided by CLIP, where the semantic clues indicate the approximate activation regions of potential objects. 2) Clustering on CLIP's features at a specific intermediate layer to discover object boundaries with the aid of our semantic-aware initialization. The semantic-aware initialization leverages semantic activation to automatically initialize clustering centers and determine the number of clusters. 3) Localization of individual objects by regrouping dispersed clustered fragments that have the same semantics, all while adhering to the detected boundaries. 4) Prompting SAM for precise masks for each individual object.

W4: Discussion on DINO and SAM.

Thank you for your suggestion. We will relocate some of the discussions about DINO and SAM to the supplementary materials and place a stronger emphasis on our primary contributions in the introduction.

Q1: Semantic-Aware Initialization: Could you provide a more in-depth explanation of the Semantic-Aware Initialization mentioned on page 17?

Thanks for pointing this out. We have added an in-depth explanation of the semantic-aware initialization in Appendix L. Please refer to it for a detailed understanding.

The core motivation behind semantic-aware initialization is that by initializing some clustering centers near the outer edges of the semantic activation map, boundaries between objects within the activation map can even be clustered and outlined. Despite the semantic activation map's inability to differentiate instances with the same classes, its edges continue to signify the boundaries between objects of distinct classes. By initializing some clustering centers near these edges (specifically, the outer edges of the semantic activation map), we can effectively group the boundaries between objects from BOTH the same and different classes into one cluster (i.e., the "boundary" cluster) due to their high feature similarity on the specific intermediate layer of CLIP. This results in the boundaries of objects from the same class also being identified by the "boundary" cluster, thereby enabling the distinction of these objects through the "boundary" cluster. As shown in Figure 5, even though the middle three figures cluster the same intermediate features in CLIP, only clustering with our semantic-aware initialization can deliver stable clustering and boundary discovery.

Dear Reviewer g6s6,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Thank you for your detailed review. In order to facilitate a prompt discussion and exchange of ideas, we now address your most critical concerns first. We will respond to all concerns and revise the paper as soon as possible. Thanks for your understanding!

W1: Weak novelty as the method is simply utilizing other foundation models without proposing anything specific to the instance segmentation task. The clustering techniques are not really showing anything novel in its formulation. Novelty is more in the pipeline.

Thank you for your rigorous consideration.

First, we attempt to directly address the reviewer's question regarding the novelty of our utilization of the foundation model and clustering technology.

• Our method goes far beyond the simple utilization of foundation models without proposing anything specific to the instance segmentation task. On the contrary, one of our key contributions and novelties lies in how effectively we probe foundation models for annotation-free instance segmentation.

  • SAM itself is not suitable for instance segmentation on COCO, exhibiting high recall but low precision rates primarily because of its semantic-unaware and mostly edge-oriented approach. Also, DINO may be suitable for salient object detection, but not for instance segmentation. Please refer to the second paragraph of the Introduction and Appendix B.
  • The simple utilization of CLIP+SAM results in a less-than-ideal performance (see Table 1), highlighting the non-trivial nature of utilizing CLIP and SAM for annotation-free instance segmentation.
    • CLIP+SAM: CLIP first generates a semantic activation map, and then SAM directly selects points from the activation map to prompt SAM in instance segmentation. This simple utilization results in clustered objects of the same class being unable to be distinguished as individual instances, as the semantic activation map only identifies these objects as a whole.
    • SAM+CLIP: SAM first produces numerous valid segmentation masks, and CLIP then recognizes these proposals. However, this simple combination yields poor performance due to the SAM's semantic-unaware issue of generating masks with varying degrees of granularity yet lacking instance-aware discernment (see Appendix B) and CLIP’s misclassification issue (see Appendix F).

  Based on these observations and motivations, we propose that the key to effectively utilizing foundation models for the instance segmentation task is to probe foundation models to delineate the boundaries between individual objects!

• For clustering, we propose a novel semantic-aware initialization technique to automatically initialize clustering centers and determine the number of clusters. Without our novel semantic-aware initialization, clustering techniques (such as the K-means we used in our clustering) CANNOT generate general and reliable clustering results for outlining object boundaries. As shown in Figure 5, even though the middle three figures cluster the same intermediate features in CLIP, only clustering with the proposed initialization technique can deliver stable clustering and boundary discovery.

  • The core motivation behind semantic-aware initialization is that by initializing some clustering centers near the outer edges of the semantic activation map, boundaries between objects within the activation map can even be clustered and outlined. Despite the semantic activation map's inability to differentiate instances with the same classes, its edges continue to signify the boundaries between objects of distinct classes. By initializing some clustering centers near these edges (specifically, the outer edges of the semantic activation map), we can effectively group together the boundaries between objects from BOTH the same and different classes into one cluster (i.e., the "boundary" cluster) due to their high feature similarity on the specific intermediate layer of CLIP. This results in the boundaries of objects from the same class also being identified by the "boundary" cluster, thereby enabling the distinction of these objects through the "boundary" cluster.
  • We have added a detailed explanation of the clustering process and motivation in Appendix L. Please refer to it for a comprehensive understanding.

Second, we emphasize our novelty from a comprehensive perspective.

• We are the FIRST to effectively utilize foundation models (CLIP and SAM) for annotation-free instance segmentation. Such effective utilization is not trivial and cannot be achieved by simply combining foundation models.
• We are the FIRST to discover CLIP's special intermediate layers can outline object edges through the proposed semantic-aware feature clustering. By incorporating this discovery, we address challenges in complex scenarios with multiple instances, under annotation-free instance segmentation. To ensure stable and general clustering of object edges, we devise a semantic-aware clustering initialization technique, which is crucial.
• We are the FIRST to propose the classification-first-then-discovery pipeline for annotation-free instance segmentation task. Mere identification of boundary information does not inherently and straightforwardly aid instance segmentation without our classification-first-then-discovery pipeline. In the pipeline, we introduce a boundary metric to identify boundaries and transform the instance segmentation challenge into a fragment selection task, a novel approach not explored by previous work.

W2: there are no quantitative results to support their claim that DINO is worse than CLIP in identifying the boundaries as far as I have seen but if there are please clarify in the response. Also is CLIP better than Up-DetR or better than DINO, DINOv2 in identifying boundaries. Meaning if they apply same clustering and everything on these models' features how would it fair against CLIP.

Thank you for your suggestions. However, we cannot apply the same clustering and everything on features from DINO, DINOv2, or the Up-DetR models due to the following reasons:

• Our clustering relies on the proposed semantic-aware initialization (see Appendix L), which uses CLIP's semantic activation map to effectively initialize clustering centers and determine the number of clusters. However, DINO, DINOv2, or the Up-DetR models cannot produce such a semantic activation map for a class. As a solution, we utilize CLIP to acquire the semantic activation map and subsequently apply the same clustering method on DINOv2. The results are depicted in Figure 13, clearly illustrating that CLIP's clustering results can identify edges, such as the object boundaries of the "person", a task that DINOv2 does not work well.
• We reformulate the instance segmentation task as a fragment selection task, and the progress of fragment selection also depends on the semantic activation map (see Equ.3). Therefore, we cannot leverage DINO, DINOv2, or the Up-DetR models to accomplish fragment selection.

DINO and DINOv2 can detect salient objects in object-centric images from ImageNet. However, they are not suitable for localizing multiple objects in more complex scenes from COCO. In object detection scenes, DINO and DINOv2 struggle to distinguish between instance-level objects and often miss potential objects within the background.

W3: Up-DetR is not clearly stated in the related work

Up-DetR is an unsupervised pre-training method for object detection. Without annotated data for object detection, Up-DetR cannot solely perform annotation-free object detection or instance segmentation. Up-DetR requires fine-tuning on an object detection dataset after unsupervised pre-training to enable effective object detection. In contrast, our ZIP is entirely annotation-free and does not rely on object detection or instance segmentation annotations.

Thank you for your detailed review. In order to facilitate a prompt discussion and exchange of ideas, we now address your most critical concerns first. We will respond to all questions and revise the paper as soon as possible. Thanks for your understanding!

W1+Q1: It is challenging to follow the explanation of how the clustering works. It would be helpful if the authors could include some figures to illustrate this process more clearly.

Thank you for your valuable suggestions! We have added an in-depth explanation of the clustering process in Appendix L. Please refer to it for a detailed understanding. The core of the clustering process is our semantic-aware initialization, which leverages the semantic activation map to automatically initialize clustering centers and determine the number of clusters for K-Means.

The core motivation behind semantic-aware initialization is that by initializing some clustering centers near the outer edges of the semantic activation map, boundaries between objects within the activation map can even be clustered and outlined. Despite the semantic activation map's inability to differentiate instances with the same classes, its edges continue to signify the boundaries between objects of distinct classes. By initializing some clustering centers near these edges (specifically, the outer edges of the semantic activation map), we can effectively group together the boundaries between objects from BOTH the same and different classes into one cluster (i.e., the "boundary" cluster) due to their high feature similarity on the specific intermediate layer of CLIP. This results in the boundaries of objects from the same class also being identified by the "boundary" cluster, thereby enable the distinction of these objects through the "boundary" cluster. As shown in the Figure 5, even though the middle three figures cluster the same intermediate features in CLIP, only clustering with our semantic-aware initialization can deliver stable clustering and boundary discovery.

W2: The robustness of the clustering algorithm is concerning when applying to other datasets or settings.

In addition to the COCO dataset, we apply the same clustering algorithm and exactly the same hyperparameters on the Pascal VOC dataset (as shown in Table 3), and achieve consistently positive results.

We also conduct an evaluation of our method on the human part segmentation task, as depicted in the newly added Figure 12. We employ the same clustering method and hyperparameters for this evaluation, and the results further demonstrate the robustness of our approach.

Q2: How stable is the clustering algorithm? Will the results change drastically if optimal hyperparameters are not found? For example, which intermediate layer should be used? Is there any experiment on how the choice of  K will affect the results? Are there any guidelines? Please also provide some insights for Eq.7.

Thanks for the valuable feedback. We agree that hyperparameter tuning and large search space are challenging in most unsupervised methods. As a reference, the previous SOTA CutLER has nine hyperparameters.

Essentially, our clustering algorithm has two hyperparameters. Most importantly, these hyperparameters are relatively independent, easy to determine, and consistent across different datasets and experimental settings.

• The setting of threshold $\tau$=0.7 in Equ (5) is consistent across different datasets. The $\tau$ determines whether the pixel blocks are activated given a category, i.e., the threshold to judge whether pixel blocks are the category's foreground or background. It is an unavoidable hyper-parameter to binarize a similar/confidence map to a binary mask in the previous unsupervised object detection methods. Fortunately, our approach only needs a coarse binary mask to help initialize cluster centers instead of a precise one for final prediction, increasing the robustness to $\tau$ and setting $\tau$ consistently over different tasks and datasets.
• The threshold in Equ (7) is set to 0.15 in all the experiments. Equ (7) roughly estimates the number of clusters based on estimating the number of object categories in single images. In reality, our method only needs to set the number of clusters within a certain range (15~30) to work effectively, thanks to our reformulation instance segmentation by fragment selection. Our method is even robust to over-segmentation caused by a slightly larger number of clusters (as long as it is not too small) because our subsequent fragment selection (Sec 3.4) can regroup over-segmented fragments into an instance. | Pascal VOC | AP50 | AR100 | | --- | --- | --- | | K=15 | 21.9 | 25.5 | | Ours | 22.4 | 28.4 | | K=30 | 21.1 | 26.6 |

Thanks for the detailed replies. I have no more questions and willing to raise my score. Here are some minor issues.

1. In Appendix L, should Figure 13 be Figure 11 instead?
2. I assume the outer edge at Figure 11(d) should be the contour of the oranges not the brown box of the images? It is somewhat confusing.
3. The discussion of which layer to use in different scenarios is still lacking.
4. The explanations of semantic aware initialization should be intergrated for the final version. I can't understand how it work without looking at the appendix.

Dear Reviewer WVZD,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair