Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels

We thank reviewer 83hz for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below:

What role does the text encoder play in the clustering process?

Table G. Ablation on the usage of text encoder for clustering during training.

We use the text encoder to match the inference strategy where classes are passed through the text encoder, as well as for leveraging its pre-trained knowledge to aid the semantic clustering process. The reviewer is correct in that the clustering process is possible without the text encoder by initializing $f_C$ as random vectors. We show such ablations in Table G where we verify that although the clustering is possible without the text encoder, it largely benefits with the text encoder. We thank the reviewer for pointing out an important factor for ablation, and we will add the results and discussions to the paper.

Why use clustering technology to solve the problem of too fine initial mask granularity?

We would like to highlight that our clustering is done at a global level: we cluster masks across all images in the batch. GroupViT and SegCLIP in contrast cluster at a local level: they cluster regions within a single image using the caption for that image. Therefore, it is not obvious how we can use the GroupVit or SegCLIP grouping blocks in our framework. We will update the Related Work in our revision with this discussion. We emphasize that although the use of clustering is not novel, to the best of our knowledge,  our use of clustering across images for unsupervised representation learning is.

Can the ”SAM initialization + clustering” scheme proposed by MCLIP obtain competitive performance in the O3S experimental setting?

Table H. Results on COCO-20i under Z/FS setting from Yang et al.[d] Following LSeg[b], we used ‘others’ for the background class and prompt ensembling was used during inference.

To validate the performance in O3S setting, we directly evaluate MCLIP trained on SA-1B on COCO-20i without training with any COCO images or annotations. Interestingly, we find MCLIP to show reasonable performance, surpassing LSeg despite not being trained on other folds of the COCO-20i dataset. We thank the reviewer for suggesting the evaluation, and we will add it to our revision.

So, in the case of keeping the complete inference strategy of the previous method, how much improvement can be achieved by using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP?

Table I. Results from FC-CLIP with the complete inference strategy

We report the results with the full pipeline from FC-CLIP, when incorporating the “out-of-vocabulary” branch with MCLIP in Table I. We notice that due to the fusion strategy, the gains from the “out-vocabulary” branch are not completely transferred to the fused scores, but observe improvements over all datasets. We will add these results to the revision.

Why is the performance so poor without Semantic Clustering? The conclusion obtained is that with the increase of k, the performance tends to saturation rather than decreasing to the level of ”w/o Semantic Clustering”.

Table J. Additional results for increasing the number of clusters k. Performance starts decreasing after k=256

We clarify that ‘w/o Semantic Clustering’ refers to supervising CLIP directly with fine-grained masks from SAM, i.e. f_C=f_M, which is to the reviewer’s understanding. This would enforce CLIP to distinguish small regions within the image, despite having near identical semantic meanings. We find this to conflict with the coarse, semantic understanding of CLIP when fine-tuning, eventually losing the pre-trained knowledge of CLIP, i.e. catastrophic forgetting, which is critical when fine-tuning foundation models[11, 43, 65].

Furthermore, as the reviewer pointed out, ‘w/o Semantic Clustering’ would virtually be k=50M, (5% of SA-1B), and we agree that further increasing k should start to decline the performance, eventually reaching 50M. To verify this, we additionally provide results with k=512 in Table J which is the maximum number of k within our GPU memory constraints, and observe the performance decreasing for all datasets. We greatly thank the reviewer for providing the insight, and we will add the discussions to the revision.

References:

[b] Language-driven Semantic Segmentation. B. Li et al., ICLR 2022

[c] Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models, C. Ma et al., BMVC 2022

[d] Multi-Modal Prototypes for Open-World Semantic Segmentation, Y. Yang et al., IJCV 2024

We thank reviewer kepm for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below:

It would be helpful to quantify the costs of performing the online clustering at training time. What are the costs of performing online clustering at the dataset level during training?

Table C. Results on open-vocabulary semantic segmentation with additional training data.

We compute the cluster assignment through the Sinkhorn-Knopp algorithm [13], which only adds around 1ms in our training every iteration, as it can be performed on GPU. We opted for a 5% split of SA-1B due to the large scale of the full dataset, which is similar in scale with SAM-CLIP [43]. To study the scalability of our approach, we provide results when doubling the amount of training data in Table C, showing that we do observe modest improvements from more training data. We will add this experiment to the revision.

What is the performance of MCLIP when trained on the benchmark datasets and tested in- and out-of-distribution? How does it compare against some baseline methods?

Table D. Results on open-vocabulary semantic segmentation for in- and out-of-distribution analysis. *: indicates in-distribution results where the training splits of the datasets were seen.

We thank the reviewer for providing an interesting point for discussion. To study in- and out-of-distribution, we provide “in-distribution” results by training MCLIP with COCO-Stuff images, and “out-of-distribution” results by zero-shot evaluating on other datasets. For comparison, we provide MaskCLIP as “out-of-distribution” and MaskCLIP+ suggested by reviewer Dx6v as “in-distribution” baselines, where MaskCLIP+ is trained with images from the target dataset COCO-Stuff and PC-59 respectively. We observe that our MCLIP outperforms MaskCLIP+ in COCO-Stuff, but the in-distribution performance of MaskCLIP+ outperforms MCLIP in PC-59 with DINO masks. However, when incorporating stronger SAM masks, MCLIP can improve largely, outperforming MaskCLIP+ despite being out-of-distribution for PC-59. We will add this experiment and discussion to the revision.

What is the model performance when changing the template at inference time? What is the performance without the template? What about training without the template?

Table E. Results on open-vocabulary semantic segmentation with different prompts. Each row show results from different prompts in training, while each column is in inference. For brevity, we report the mean over 5 benchmarks from Table D. “itap” is a common abbreviation of “I took a picture”

We thank the reviewer for suggesting ablation with different prompts. We provide results when trained with no prompt(“{}”)  and with different prompts in Table E. We initially used “a photo of a {} in the scene” following other methods [11, 14, 21] but surprisingly, the prompt “itap of a {}” shows significant improvements for both training and inference. Given that “itap of a {}” is one of the well-performing prompts originally curated from CLIP, we speculate the results to reflect the preference of prompts from CLIP. We will add the results and the discussions, and re-conduct our experiments with better prompts.

Does performance improve with prompt ensembling (i.e., similar to what CLIP does with N prompts that average to get better class centroids)?

Table F. Results on open-vocabulary semantic segmentation with prompt ensembling strategy.

We provide results for ensembling in Table F, where we ensemble the default prompt “a photo of a {} in the scene” with 7 additional prompts originally curated in CLIP. We observe that not only can prompt ensembling largely boost the performance during inference time, it also shows additional gains when applied during training. We thank the reviewer for the valuable suggestion, and we will add the results and discussion to the paper.

Using CLIP as a baseline for the ablation studies may be misleading due to the mismatch between its application and open vocabulary semantic segmentation

We clarify that the baseline results from CLIP in the ablation studies are in fact obtained from applying MaskCLIP [61], which slightly modifies CLIP for open-vocabulary semantic segmentation. We apologize for the confusion, and we will revise the baseline as MaskCLIP in the ablations.

We thank reviewer rHvB for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below:

Compared with methods using similar VFM, such as SAM-CLIP, the proposed method performs much worse.

Table B. Additional comparison on open-vocabulary semantic segmentation including COCO-Object, evaluated on 80 instance classes of COCO-Stuff.

Thank you. We would first like to point out that SAM-CLIP actually reported results on COCO-Object and not COCO-Stuff, which we have verified with the authors of SAM-CLIP. We therefore present the corrected comparisons in Table B, which we will also include in the revision. We can see that our proposed MCLIP does indeed outperform SAM-CLIP on COCO-Object, consistent with PC-59 and VOC, and is only marginally behind by 0.4 points on ADE. We achieve this strong performance despite SAM-CLIP additionally using 40M image-text pairs, showing the benefits of our approach.

If simply adding caption could boost the performance so much, why should we stick to a setting without caption?

We would like to highlight the main contribution to be our exploration for effectively leveraging unlabeled masks, which has also been acknowledged by other reviewers to be a “neat idea” (Dx6v), “motivation is clear and sound” (kepm) and “interesting” (83hz), hence focus on masks instead. The framework can be potentially enhanced by incorporating captions along with unlabeled masks, which we anticipate for future exploration.

Compared with other methods [ 50 , 30 , 43 , 51 ] leveraging image caption as supervision, this paper actually uses stronger DINO generated pseudo masks

We emphasize that we do not need human labels within our training as DINO is trained in a self-supervised manner with only images. In contrast, the mentioned methods leverage human-annotated captions in addition to images, which makes it hard to consider that our method is leveraging stronger supervision. Furthermore, we also highlight that SAM is leveraged to demonstrate scenarios with higher quality unlabeled masks, and MCLIP still shows strong performance with DINO generated masks.

Table 1 looks more like an ablation study to me.

We thank the reviewer for the suggestion, and will adjust the tables accordingly.

We thank reviewer Dx6v for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below:

Comparison with MaskCLIP+

Table A. Additional comparison on open-vocabulary semantic segmentation with other methods.

We provide comparison with MaskCLIP+ in Table A, with results from ViT-B/16 CLIP backbone. Despite MaskCLIP+ having an advanced decoder for segmentation and requiring training on the target dataset, we demonstrate that MCLIP outperforms MaskCLIP+ in both COCO-Stuff and PC-59, demonstrating the effectiveness of our approach.

Some related works should be discussed and compared

We thank the reviewer for pointing out related works with our work. We provide results from ZeroSeg[a] as mentioned by the reviewer in Table A, and we will add discussions and comparison to ZeroSeg and MaskCLIP+ to the paper.

References:

[a] Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only. J. Chen et al., ICCV 2023