SOHES: Self-supervised Open-world Hierarchical Entity Segmentation

We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement of our motivation, experimental results, and overall presentation. We provide the following clarifications in response to your concerns:

1. Sum of features

• In fact, summing and averaging the features when merging two regions are equivalent. The reason is that, we adopt the cosine similarity when measuring the semantic closeness between two regions, which normalizes the features’ magnitude in computation. The sum of patch-level features within a region leads to the same cosine similarity as the average of patch-level features.

• We chose the sum over average due to its calculation and implementation simplicity. Equivalent to computing and storing the feature sum $f_k=f_i+f_j$, we can also use the feature average $\bar{f_k}=\frac{N_i\bar{f_i}+N_j\bar{f_j}}{N_i+N_j}$, where $N_i,N_j$ are the numbers of patches in region $i$ and $j$, respectively. The sum formation does not require keeping track of the patch numbers.

2. Impact of local re-clustering

• We designed the local re-clustering to refine the pseudo-labels for small entities, because they are the most challenging to be discovered. The local re-clustering has minimal impact on the large entities. As shown in Table 2 of the original manuscript, the average recall for large entities (AR-Large) is slightly improved from 17.2 to 17.5 by the local re-clustering step. The reason for this change is that “large” entities considered by the MS-COCO evaluation metric are regions that are larger than $96\times 96$ pixels (https://cocodataset.org/#detection-eval), which could still fall under our relative area threshold $\theta_\text{small}=\frac{1}{1024}$ in some high-resolution images from SA-1B (e.g., $1500\times 8000$). Visually, these entities are not “large” compared to the whole image, but are still counted by MS-COCO style AR$_L$.

• To better understand the impacts of the hyper-parameter $\theta_\text{small}$, we run additional ablation study experiments. We observe that a threshold larger than $\frac{1}{1024}$ does not further significantly improve the coverage of small entities but would introduce more processing time. | $\theta_\text{small}$ | 0 (No local re-clustering) | $\frac{1}{2048}$ | $\frac{1}{1024}$ | $\frac{1}{512}$ | | --------------------- | -------------------------- | ---------------- | ---------------- | --------------- | | AR$_S$ | 1.9 | 4.3 | 5.1 | 5.1 | | Time/Image (sec) | 0.0 | 6.0 | 8.4 | 10.0 |

• We appreciate the reviewer’s feedback and will integrate the explanation for the impact of $\theta_\text{small}$ into the revision.

3. Ablating the ancestor prediction head

• We proposed the ancestor prediction head to perform the additional task of hierarchical segmentation, which could be considered as an add-on extending the Mask2Former segmentation model. We did not expect it to help with mask prediction, since this ancestor prediction head operates in parallel to the mask prediction head. In fact, we trained a SOHES model without the ancestor prediction head, and its performance is very close to the standard SOHES model (e.g., on SA-1B, 33.0 Mask AR w/o ancestor prediction vs. 33.3 Mask AR w/ ancestor prediction).

We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement of our novel task, “method to generate hierarchical proposals,” “novel ancestor prediction head,” and good performance. We provide the following clarifications in response to your concerns:

1. Multi-phase framework

• We agree with the reviewer that such a multi-phase learning paradigm (e.g., discovering pseudo-labels and then learning a detection/segmentation model) is a general framework in the existing self-supervised object localization/discovery literature. Recent methods which adopt this framework include LOST [R1], TokenCut [R2], FreeSOLO [R3] (as suggested by the reviewer), FOUND [R4], and CutLER [R5] (as mentioned by Reviewer PuvF). The contributions of these different methods do not lie in the introduction of a new framework, but instead in proposing unique components to improve this established framework for self-supervised object localization/discovery.

• Our SOHES similarly adopts this general multi-phase framework, but makes significant contributions through novel designs for each phase involved in the learning process, including a global-to-local clustering algorithm for pseudo-label generation, a hierarchical relation learning module, and a teacher-student self-correction phase. In particular, none of the previously mentioned methods [R1-5] consider hierarchical structures among objects/entities, while learning such structures is involved throughout the learning process of SOHES. Given these technical novelties and advancements, our SOHES substantially improves performance over previous methods.

• We appreciate the reviewer’s feedback and will revise the Related Work section and the beginning of the Approach section, to better reflect this general framework and our novel components proposed in this work with a hierarchical perspective.

2. Hierarchical segmentation model

• We would like to first clarify that in the original manuscript, we stated that “existing segmentation models cannot predict the hierarchical relations among masks” in the context of general segmentation model architectures like Mask2Former (Section 3.2). This drove us to design a novel module that could predict hierarchical structures among entities. Indeed, some specialized segmentation models (e.g., GroupViT [R6]) are capable of making hierarchical predictions, but do not directly fit into our context.

• More importantly, although our SOHES and GroupViT both aim to produce hierarchical segmentation masks, there are some vital differences between the two approaches:

  • SOHES directly learns from unlabeled raw images, but GroupViT is weakly supervised by text.
  • SOHES can produce a variable number of hierarchical levels, but GroupViT is limited to pre-defined levels (e.g., 2 levels).
  • SOHES can produce a variable number of whole entities, parts, and subparts, but GroupViT is limited to a fixed number of pre-defined group tokens (e.g., 64 and 8 group tokens in the first and second grouping stages, respectively).

• We appreciate the reviewer’s feedback and will revise this sentence in Section 3.2 to more precisely describe the context about the widely-applied segmentation architectures. In addition, we will cite GroupViT [R6] and include the discussion above in the revision.

I have read the responses from the authors. Most of my issues have been addressed. However, the "ancestor prediction head" does not have significant improvements. I decided to keep my original rating.

We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement of our “unsupervised segmentation” task, “approach to generate unsupervised hierarchical masks,” and paper writing. We provide the following clarifications in response to your concerns:

1. Technical contributions

• We agree with the reviewer’s perspective, but we would like to point out that such a multi-phase learning paradigm (e.g., discovering pseudo-labels and then learning a detection/segmentation model) is a general pipeline widely-adopted in the existing self-supervised object localization/discovery literature. Indeed, not only does CutLER [R5] (as suggested by the reviewer), but all recent methods also follow this pipeline including LOST [R1], TokenCut [R2], FreeSOLO [R3] (as mentioned by Reviewer vWUT), and FOUND [R4]. That means, the contributions of these different methods do not lie in the introduction of a new pipeline, but instead in proposing unique components to improve this established pipeline for self-supervised object localization/discovery.

• Our SOHES similarly adopts this general multi-phase pipeline, but makes significant contributions through novel designs for each phase involved in the learning process, including a global-to-local clustering algorithm for pseudo-label generation, a hierarchical relation learning module, and a teacher-student self-correction phase. In particular, none of the previously mentioned methods [R1-5] consider hierarchical structures among objects/entities, while learning such structures is involved throughout the learning process of SOHES. Given these technical novelties and advancements and our substantial performance improvements over CutLER, we respectfully disagree with the reviewer that the distinctions between our SOHES and CutLER are merely minor implementation differences.

• We appreciate the reviewer’s feedback and will revise the Related Work section and the beginning of the Approach section, to better reflect this general pipeline and our novel components proposed in this work with a hierarchical perspective.

2. Model performance

We would like to first clarify that, regarding the open-world segmentation task on MS-COCO/LVIS, the computed AP number is misleading and cannot accurately reflect the model’s true segmentation performance. The observed lower AP in our results compared with CutLER’s on MS-COCO/LVIS is a consequence of limitations in the annotations within these datasets. In contrast, on the SA-1B dataset which mitigates these annotation limitations, our AP (12.9) significantly surpasses CutLER’AP (7.8). Below we explain in detail.

• We chose average recall (AR) over average precision (AP) for the following reason: When evaluating open-world segmentation models (which try to segment “everything”) on datasets with closed-world annotations (such as MS-COCO/LVIS which only include a pre-defined, limited set of entity classes), AP would penalize model predictions that are actually valid entities, but just not annotated by the dataset. Therefore, when the dataset annotations cannot cover all entities, AP becomes misleading for judging the performance of open-world models. Note that this choice of AR over AP is also commonly adopted in the open-world literature. Examples include (but are not limited to) detection [R6], segmentation [R7], and tracking [R8].

• As an example (shown in Table 5), SAM’s AP is lower than CutLER on MS-COCO (6.1 vs. 9.8) and PartImageNet (3.4 vs. 5.3), which definitely cannot imply SAM is a model inferior to CutLER. The lower AP of SAM is due to insufficient annotations in these two datasets, not the capability of SAM. Meanwhile, as the annotated entities increase, AP becomes more and more trustworthy. For instance, AP on SA-1B (the most densely annotated dataset evaluated in our work) matches qualitative comparison and follows the same trend as AR (FreeSOLO < CutLER < SOHES < SAM).

3. Unfair comparison

• We chose Mask2Former as our segmentation model mainly for making hierarchical predictions. In Cascade Mask R-CNN, each proposal is predicted independently, and therefore, analyzing hierarchical relations between entities would be challenging in Cascade Mask R-CNN. In contrast, the attention modules in Mask2Former allow information exchange among queries, so we can build our ancestor prediction head on Mask2Former.

• For a more comprehensive comparison with CutLER, we train a Cascade Mask R-CNN segmentation model using SOHES without the entity hierarchy. This model still significantly outperforms CutLER with the same segmentation model architecture. | Method | Segmentation Model | LVIS | Entity | SA-1B | | ------ | ------------------ | ---- | ------ | ----- | | CutLER | Cascade Mask R-CNN | 20.2 | 23.1 | 17.0 | | SOHES | Cascade Mask R-CNN | 27.0 | 29.2 | 31.9 | | SOHES | Mask2Former | 29.1 | 33.5 | 33.3 |