Test-time Contrastive Concepts for Open-World Semantic Segmentation

We thank the reviewer for the insightful feedback and address all the concerns below. We would however like to emphasize that our method is a test-time improvement. In our comparisons, even if a method requires training, we apply our $CC$s only at test time. Secondly, we would like to highlight that our method considers each query concept separately at test time and generates $CC$s per concept. There is, therefore, no "dataset-level" adjustment of contrastive concepts $CC^{L}$ and $CC^{D}$.

W1: In-depth analysis

We thank the reviewer for the remark. Our initial statistical analysis was aligned with the findings of Miller et al., which motivated and guided our work. We however agree with the reviewer that adding such an analysis would be interesting for the community as most of the literature on CLIP space is focused on the task of image classification.

In Fig.13 (appendix) of the revised version of the paper, we provide an analysis of patch-level CLIP space, using MaskCLIP features. The figure shows histograms of patch-level maximum text similarities (in cosine similarity) across 100 randomly sampled images from VOC (a) and ADE20k (b). We notice an overall concentration of cosine similarity scores in [0.1,0.3], suggesting that the feature space is not easily separable. We will integrate a discussion on separability in the paper.

W2: Conceptual diagram missing

We have added a conceptual diagram in the revised version of the paper, please see Fig. 14. We will integrate it in the final version of the paper. We also welcome any suggestions for further improvement of the figure.

W3: Comparison with supervised baselines

Our method is a test-time improvement. For CAT-Seg, which is a method supervised with mask annotations, we apply $CC$ in a zero-shot manner and show significant improvement even without retraining. Besides, trying to improve CAT-Seg and retraining it, possibly playing on candidate classes, goes beyond the use of existing methods as baselines.

Tab.1 in the paper serves as a comparison between our $CC$s generation methods and was not meant to compare segmentation methods, as they use different levels of annotation. We could make this clearer in the table if the reviewer thinks it would be useful. When using our $CC^D$ and $CC^L$, compared to the baseline "background", we show significant improvements for all segmentation methods, no matter their level of supervision.

W4: Comparison to SAN

As requested by the reviewer, we run experiments with the SAN method on 2 datasets, ADE20K (presented in Tab.11) and VOC (presented in Tab.12).

Similarly to other OVSS methods, our $CC$ generation methods applied to SAN provide consistent improvements over the $CC^{BG}$ baseline. We will include SAN results in the final version of our paper.

W5: Additional experiments

We ran the two experiments requested by the Reviewer. In Tab.13, we present results on VOC when using all concepts from LAION dictionary $T$.

Table 13. Experiments on VOC with all concepts from LAION as $CC$, noted as $CC^D(T)$. Reported results are with IoU-Single%.

We observe a significant drop in performance across all methods when using all $T$ as contrastive concepts.

Next, we study how different numbers of LLM-generated contrastive concepts influence the results. We use CLIP-DINOiser, TCL and MaskCLIP (with OpenAI weights) models, and experiment on two datasets: Pascal VOC and Cityscapes.

Table 14. IoU-Single results on VOC dataset with $CC^L$. Experiment with LLM-generated contrastive captions ($CC^L$) for different numbers of generated concepts.

Table 15. IoU-Single results on Cityscapes dataset with $CC^L$. Experiment with LLM-generated contrastive captions ($CC^L$) for different numbers of generated concepts.

Tables 14 and 15 present results for 50, 100, and 200 LLM-generated concepts. We compare them to the original setup from the paper ($CC^L$ column). We can see that using more concepts decreases performance, as measured by single-IoU across all models and datasets. We will include the ablation in the final version of the paper.

Q1: When to use $CC^L$ and when to use $CC^D$

As discussed in the main paper, we notice that $CC^L$ generalizes better to domain-specific datasets. In contrast, for both VOC and ADE20K, $CC^D$ outperforms most of the time the LLM-based $CC^L$. However, this trend does not hold for Cityscapes, where $CC^L$ gives the best results. We believe it is due to the limited availability of such driving scenes in large pretraining datasets.

$CC^L$ can also yield better results for concepts that are rare or missing from the pretraining dataset. Moreover, thanks to our similarity-mapping mechanism, $CC^D$ can be used for any query concept. Yet, its effectiveness still relies on the nearest-neighbor mapping in $T$ and on the frequency of co-occurrences of $q$.

It must also be kept in mind that our approach is not to be thought of for a target dataset. Indeed, if only a specific dataset is targeted, the setting is then more like a close-world scenario, and other approaches might then be more meaningful. Our approach is more appropriate for "in the wild" settings.

Q2: Multi-domain evaluation

Our single-query scenario with IoU-Single could indeed be applied to multi-domain evaluation. On the other hand, applying our $CC$ generation methods would not be as straightforward. The considered domains in the proposed multi-domain benchmark [G] span a large spectrum of very specific domains, such as agriculture, medicine, etc.

In the case of $CC^L$, we are bounded by the capabilities of the LLM - in particular its generalization to very specific domains. In the case of $CC^D$, the contrastive concepts directly depend on the training data, which is not as specialized as the evaluation datasets.

[G] Blumenstiel Benedikt, et al. "What a mess: Multi-domain evaluation of zero-shot semantic segmentation." NeurIPS 2024.

W1: Problem setting: potential overlap with goals of referring segmentation and expression localization

We thank the reviewer for this interesting remark and suggestion. As mentioned in the related work of our paper, there are substantial differences between referring segmentation and OVSS, and thus between their corresponding methods.

Methods for referring segmentation, such as GRES, are highly supervised, including object relationship information, which is not the case for OVSS. While CAT-Seg has access to pixel-level class information, most OVSS methods rely solely on a model pre-trained with only image/caption annotation. Moreover, input text prompts are much more complex in referring segmentation, requiring spatial and relational reasoning, which are too complex for CLIP-like models [C], whose text encoder is limited to short texts [D].

We agree, however, that methods such as GRES could be applied to OVSS. We therefore run an evaluation of GRES in a single-query scenario for OVSS on two datasets: VOC and Cityscapes. We compare GRES against CLIP-DINOiser and CAT-Seg with $CC^D$ in Tab.10.

Table 10. IoU-single% of different OVSS methods with $CC^D$, compared to GRES.

On VOC, we notice that GRES outperforms the open-vocabulary segmenters even with $CC^D$. On the other hand, the results on Cityscapes point out that the reliance on strong supervision makes GRES less suitable for domain-specific datasets and for complex datasets featuring many objects. Perhaps fine-tuning on Cityscapes could address the issue.

[C] Mert Yuksekgonul et al. When and why vision-language models behave like bags-of-words, and what to do about it? ICLR 2023.

[D] Beichen Zhang et al. Long-CLIP: Unlocking the Long-Text Capability of CLIP. ECCV 2024.

W2: Re-defining some existing concepts: the term "open-world" is already used

We thank the reviewer for this important remark. It is indeed crucial to avoid potential confusion on terminology.

The original "open-world" setting proposed in [E] is a pragmatic approach for applications - requiring models to identify novel (unknown, out-of-distribution (OOD)) classes, have them progressively labelled and further learned incrementally. In recent years, the novel class identification issue has been addressed with VLM-based zero-shot, open-vocabulary or OOD methods [F].

Here, we also aim for a practical and realistic scenario, going beyond the typical open-vocabulary benchmarks that still assume access to a pre-defined and static list of dataset-specific classes, which is ultimately a closed-world setup. With "open-world", we want to highlight this difference, stressing that our contrastive concepts are automatically suggested for each new user query, with concepts not tied to any specific dataset or benchmark. (When we evaluate on a benchmark, we do not assume that the list of classes appearing in the benchmark is known). In contrast, just saying "open-vocabulary" could offer freedom in the formulation of concepts while nevertheless restricting to a domain, which we want to exclude here.

We propose to rename our task "open-class" segmentation, as the goal of our work is to break the dataset-level dependencies and to consider classes as independent concepts. We are also gladly open to other suggestions.

[E] Abhijit Bendale & Terrance Boult. Towards Open World Recognition. CVPR 2015.

[F] Jianzong Wu et al. Towards open vocabulary learning: A survey. T-PAMI 2024.

W3: Paper presentation

We apologize for the confusion and provide a revised version of Tab.1 in the attached revision of our paper. We also corrected the mentioned sentence and removed any typos we could find.

Q1: $CC^L$ sufficient for open-world segmentation?

As discussed in Sec. 4.3 of our paper, we found $CC^L$ more suitable for "domain-specific" datasets, such as Cityscapes. We believe this is due to the limited availability of driving scenes in large pretraining datasets. On the other hand, on "general-domain" images such as images in ADE20K, $CC^D$ consistently outperforms $CC^L$. This suggests that each approach has specific merits, which may benefit one setting more than another. We also observe in the qualitative examples in Fig.4 that multiple contrastive concepts (columns $CC^D(all)$ and $CC^L(all)$) can be useful for good segmentation.

As the reviewer phrases it, $CC^L$ can indeed be thought of as a way to find "hard" negatives, i.e., the most relevant concepts to contrast with. But $CC^D$ is also a way to do so, relying on CLIP training-time concepts rather than LLM-based open concepts. Finding multiple good negatives is exactly the motivation for our Contrastive Concepts. In case we misunderstood the question, we kindly ask the reviewer to let us know.

W1: What about image-based contrastive concepts?

Our constrastive concepts are currently only based on the given query, not on the image. Using the image as well, which contains more specific information, is definitely an interesting research direction. Thank you for the suggestion. To try the idea, we experiment with LLAVA-1.5 7B [A] on VOC and present the results in Tab.9. As suggested by the reviewer, we prompt LLAVA using both $q$ and an image. We instruct the model to generate a list of visual concepts surrounding $q$ on the given image. As for LLM prompting, we ask the model explicitly not to return synonyms and parts of $q$. We notice, however, that the model has trouble following the instructions, quite frequently returning an empty list. The model also does not fully follow the requirement of ignoring synonyms.

Table 9. IoU-single% of different OVSS methods on VOC. We note $CC^{VLM}$ the VLM-based contrastive concept setting described in the text.

Our results show that the VLM-based approach underperforms $CC^L$ and $CC^D$. We attribute it to the fact that our prompt requires additional steps of reasoning (removing synonyms, etc.) for which the language model in the considered VLM can be too weak. We believe it is still a promising direction which we leave for future work. One might consider improving the prompt, using a VLM with stronger language and reasoning capabilities, or using both a VLM and an LLM, etc.

[A] Haotian Liu et al. Improved baselines with visual instruction tuning. CVPR 2024.

W2: What happens when querying the part of an object?

This is an interesting question. We reuse here the proposed example where we query alternatively the whole object "person" or the part "arm". We start by discussing a single-query scenario, which is the main focus of our work.

In the case of the query $q$ being "parent class", such as "person" in the given example, we have the following mechanism:

• For $CC^L$, we explicitly encode in the prompt an instruction not to include parts of query objects (please see Fig.8 of our paper).
• For $CC^D$, we also considered an explicit part removal, but we found it was not impacting the overall performance on tested datasets (see Sec. C.3 of our appendix).

In the case of $q$ being a part of a parent object ("arm" in our considered example), we did not design any specific mechanism, therefore if both are present the method is bounded by the segmentation method itself and its feature separability. We qualitatively investigate such an example in Fig. 16 of our revised pdf. We present the results when $q$="arm" (top row) and $q$="person" (bottom row) for both $CC^D$ and $CC^L$.

• In the case of "person", as described above, none of the $CC$ interfere with $q$.
• In the case of "arm", we notice that both methods produce $CC$ that interfere with $q$, $CC^D$ includes "coat" which covers part of "arm", leaving only the hand properly segmented. In $CC^L$, "sleeve" unfortunately covers the whole "arm".

Please note however that there is an intrinsic ambiguity in the task regarding the interpretation of queries. For instance, if part of a person is occluded by an object, the occluded part should not be segmented. On the contrary, if "occluded" by clothes, the clothes should probably be segmented along with the visible parts of the body.

A potential improvement could be to incorporate another filtering mechanism which removes potential occluding concepts. Another alternative would be to consider the top $K$ concepts and allow for multi-label segmentation. We believe that considering hierarchies is an important direction for future work.

When considering both prompts at once, we would follow our multi-query process described in Sec 3.1. and combine $CC$s from both queries with the filtering mechanism. With our filtering mechanism, we ensure that $CC_{q=arm}$ does not interfere with $q=$"person" and the other way around. We however emphasize that the final segmentation quality of "arm" vs "person" is dependent on the CLIP space. A final segmentation result is thus determined by the clip-similarity score, meaning that the closest concept wins. In the case of our example, this would be "person".

Following the reviewer's request, we experiment with SigLIP ViT-B/16 backbone. We present results on the datasets VOC in Tab. 16, ADE20k in Tab. 17 and Cityscapes in Tab. 18. We report results using our metric IoU-Single when employing different $CC$s, as well as when using a non-contrastive Sigmoid-based thresholding strategy (noted Sigmoid) for SigLIP. We consider different thresholds which we observe lead to a high variability in results. We compare the results with those of MaskCLIP with different $CC$s as is it the closest method which does not include any refinement mechanism. Overall, the results obtained with SigLIP (either using $CC$ or sigmoid) are significantly worse than those obtained with MaskCLIP on VOC. On ADE20k and Cityscapes, SigLIP with sigmoid obtains closer results to MaskCLIP; however, it is always outperformed by MaskCLIP when using $CC^D$ and/or $CC^{L}$. Additionally, we observe that the sigmoid results are highly sensitive to the threshold, with a drop of between -4 and -7 pts (eg. from 20.6 to 13.8/12.1 on Cityscapes) when the delta changes by as little as 0.005. We will integrate this discussion in the final paper, as we believe it is insightful.

Table 16. Results (IoU-Single%) on VOC with SigLIP backbone.

Table 17. Results (IoU-Single%) on ADE20k with SigLIP backbone.

Table 18. Results (IoU-Single%) on Cityscapes with SigLIP backbone.