A Simple Framework for Open-Vocabulary Zero-Shot Segmentation

Overview

We thank the reviewer for the detailed and constructive feedback. Below, we address your questions and concerns in detail.

Usage of a predefined concept bank & open-vocabulary capabilities

There appears to be a general misunderstanding regarding the role of the concept bank. Its primary purpose is to filter out noun phrases that may not appear in the image, addressing noise in the captions of large-scale web datasets like LAION-400M. For more curated image-caption pair datasets, the concept bank is optional. To support this, we trained SimZSS without the concept bank (see Table 1 and 2 below). As expected, the scores are slightly lower compared to using the concept bank. However, the results consistently outperform the baseline (LiT), which focuses solely on enforcing consistency between captions and image representations. Moreover, we observe that the performance of SimZSS without the concept bank is closer to the version with the concept bank than to LiT. This demonstrates that SimZSS's performance is driven more by the consistency objective between noun phrases and localized image regions than by the concept bank itself. Considering the results achieved by concurrent baselines on the same tasks (Tables 1 and 2 in the main text), we conclude that SimZSS yields great performance compared to existing methods, irrespective of whether a concept bank is used, which shows that a concept does not need to appear in the concept bank to be properly localized at test time.

Table 1: Zero-shot segmentation with and with out the bank of concepts on datasets without background.

Table 2: Zero-shot segmentation with and with out the bank of concepts on datasets with background.

Performance with the ViT-L model

We encountered significant issues with the ViT-L checkpoint from DINOv2, which arise from a misalignment between the CLS token and the spatial tokens. This is discussed in detail in lines 846–855 of the draft. Notably, LiT exhibits the same challenges with this specific model, leading us to conclude that the issue is inherent to the checkpoint itself rather than the methods being evaluated.

Scalability of the bank of concepts and generalization

The weights of the linear classifier are computed by summing the representations of the unique text concepts within a given batch. Consequently, the method's complexity depends solely on the number of unique concepts in a batch, rather than the total number of concepts in the entire bank. This ensures that scaling the concept bank does not introduce additional computational cost.

To shed some light on the relation between generalization and the choice of the concept bank we evaluate SimZSS on a standard open-vocabulary zero-shot classification benchmark (see https://github.com/LAION-AI/CLIP_benchmark). In Table 3, we report the average top-1 accuracy obtained therein. We observe that using the bank of concepts does not decrease the average zero-shot classification performance, which shows that using a relatively small set of concepts does not negatively affect the generalization. Additionally, we observe that in average SimZSS does not compromise the zero-shot classification accuracy compared to a LiT pre-training with the same backbone (ViT-B/14 pre-trained with DINOv2) and dataset.

Table 3: Average zero-shot classification top-1 accuracy.

Overview

We thank the reviewer for the detailed and constructive feedback as well as the appreciation of our work in terms of soundness, presentation and contribution. Below, we address your questions and concerns in detail.

Lack of details about the training and data

Based on your feedback, we have relocated the "Experimental Setup" subsection from the Appendix to the main text.

Usage of a predefined concept bank & open-vocabulary capabilities

There appears to be a general misunderstanding regarding the role of the concept bank. Its primary purpose is to filter out noun phrases that may not appear in the image, addressing noise in the captions of large-scale web datasets like LAION-400M. For more curated image-caption pair datasets, the concept bank is optional. To support this, we trained SimZSS without the concept bank. As expected, the scores are slightly lower compared to using the concept bank. However, the results consistently outperform the baseline (LiT), which focuses solely on enforcing consistency between captions and images representations. Moreover, we observe that the performance of SimZSS without the concept bank is closer to the version with the concept bank than to LiT. This demonstrates that SimZSS's performance is driven more by the consistency objective between noun phrases and localized image regions than by the concept bank itself. Considering the results achieved by concurrent baselines on the same tasks (Tables 1 and 2 in the main text), we conclude that SimZSS yields great performance compared to existing methods, irrespective of whether a concept bank is used, which shows that a concept does not need to appear in the concept bank to be properly localized at test time.

Table 1: Zero-shot segmentation with and with out the bank of concepts on datasets without background.

Table 2: Zero-shot segmentation with and with out the bank of concepts on datasets with background.

Regarding the open-vocabulary capabilities of SimZSS, we argue that any zero-shot segmentation method using free-form text as a prompt qualifies as open-vocabulary. Moreover, SimZSS differentiates itself from LiT by incorporating an additional loss at the concept level, complementing the cross-modal contrastive objective introduced by CLIP. Since both CLIP and LiT are regarded as open-vocabulary models, there is no reason to consider SimZSS differently, particularly in light of the results from the preceding experiment. Furthermore, in the following section, we demonstrate that the additional loss introduced does not compromise the zero-shot classification capability.

Table 4: Zero-shot classification top-1 accuracy.

Missing related works

Thank you for pointing the missing entries from the related works. We recognize the relevance of these works in this context and will include these in the updated draft that will be uploaded before the end of the discussion period. Furthermore, we will add a subsection specifically dedicated to the comparison with training-free approaches.

Conclusion

We hope we've effectively addressed your concerns and encouraged a reconsideration of your rating. If you have any additional questions or feedback, please don't hesitate to reach out, we're happy to engage in further discussion.

Overview

We thank the reviewer for the detailed and constructive feedback. Below, we address your questions and concerns in detail.

Explanation behind SimZSS good performance

Visual representation learning vs vision-language alignment

As mentioned in Section 1 and identified by LiT, the cross-modal contrastive objective used in CLIP is a suboptimal approach for learning visual representations, both in terms of efficiency and effectiveness. Achieving zero-shot segmentation relies on two key requirements:

• The vision encoder must be capable of clustering groups of pixels with shared semantics.
• The text encoder must be sufficiently aligned with the visual modality to accurately label these clusters.

We argue that the second task is significantly easier to accomplish than the first and that the first task can be addressed independently of the second. This is the main reason behind the strong performance we report. Indeed, most concurrent methods begin with pre-aligned vision and text encoders (task 2) and attempt to transfer image-level cross-modal alignment to pixel-level alignment. As shown in the following tables, CLIP demonstrates limited cross-modal alignment at finer levels of granularity.

Table 1: Zero-shot segmentation with a ViT-B/16 pre-trained with CLIP on LAION-400M for 32 epochs on datasets without background.

Table 2: Zero-shot segmentation with a ViT-B/16 pre-trained with CLIP on LAION-400M for 32 epochs on datasets with background.

Finetuning/freezing the visual encoder

As opposed to previous methods relying on CLIP (which does not exhibit good dense vision-language alignment), our approach starts with a pre-trained vision encoder that demonstrates semantic coherence at the pixel level, and we focus on aligning a text encoder to it. This is illustrated in Table 11 of the main text. We show that the same vision backbone (ViT-B/14 pre-trained with DINOv2) equipped with a linear layer can achieve over 40 mIoU on ADE20K. Zero-shot segmentation can be interpreted as classification with a frozen linear layer that is initialized with embeddings of textual labels. This result highlights that with a better alignment of the text encoder, SimZSS could reach over 40 mIoU on ADE20K, showcasing that the quality of the vision encoder is far from being the bottleneck.

This design choice is further supported by the strong performance of LiT on zero-shot segmentation tasks. The remaining performance gap can be attributed to SimZSS's ability to identify concept pairs across modalities and enforce consistency between them. Specifically, the consistency objective is implemented via a linear projection layer, where the weights are derived by summing the concept representations (of the same concept) within the batch, combined with a cross-entropy loss. As such, the training of the concept-level loss is completely aligned with the way classification is done at inference time, further explaining the observed strong performance.

We will make this analysis explicitly clear in the paper, add an entry for CLIP zero-shot segmentation in Table 1 and 2 of the main text, and add pointers to Table 11 which is located in the Appendix.

We thank you for taking the time to review our paper. We address all your points below:

The comparison is unfair. The proposed method is trained on COCO Captions (Lin et al., 2014; Chen et al., 2015) and LAION-400M (Schuhmann et al., 2021) and the image encoder is pretrained on LVD-142M dataset, as shown in L764-L774. However, other methods are not trained on these large-scale dataset. Even trained on these large datasets, the performance improvement is limited.

This statement is incorrect. In fact, methods such as CLIP-DINOiser, CLIP-DIY, and TCL utilize models pre-trained with CLIP on web-scale datasets like LAION-2B for multiple epochs (see https://github.com/mlfoundations/open_clip). Similarly, CLIPpy is pre-trained on HQITP-134M, a dataset containing 134 million samples. For OVDiff, we report results obtained using DINO+CLIP+StableDiffusion pre-trained image encoders, where both CLIP and StableDiffusion models are trained on web-scale datasets. Moreover, OVSegmentor employs a DINO pre-trained image encoder and a BERT pre-trained text encoder which was pretrained with BooksCorpus (800 million words) (Zhu et al., 2015) and English Wikipedia (2,500 million words). Additionally, they perform alignment on CC4M, a dataset with 4 million samples.

We would like to emphasize that assigning a confidence score of 5 necessitates a deep familiarity with related works, which does not appear to be reflected in this assessment. We have the impression the reviewer is not well aware of the literature on this topic and therefore may not be well placed to review our paper.

The novelty and contribution is limited.

We would be grateful if you could point us toward any existing zero-shot segmentation works that significantly overlap with SimZSS.

First, I prefer simple method. I believe that the simple method is more valuable for applications and research. However, simple method requires more deep analysis and insights. Unfortunately, this work only presents a simple method without any insights. The method is designed without motivation or explanation. This paper looks like a experiment report, rather than a research paper. For a good research paper, the authors need to tell new insights, rather than just propose a model and conduct some experiments.

We believe that scientifically driven insights are best derived through rigorous experimentation, which is why we meticulously designed a comprehensive set of experiments, each accompanied by detailed conclusions. We would appreciate it if you could highlight specific aspects of SimZSS where additional insights or explanations might be needed.

The proposed method even performs worse than previous methods on Pascal VOC dataset. For example, the proposed method SimZSS trained on LAION-400M achieves 48.6 on Pascal VOC, which is much lower than CLIP-DINOiser (Wysoczanska et al., 2023), CLIP-DIY (Wysoczanska et al., 2024), OVSegmentor (Xu et al., 2023a), OVDiff (Karazija et al., 2023), CLIPpy (Ranasinghe et al., 2023) and TCL (Cha et al., 2023).

The datapoint you selected is only one of the many evaluation dataset / alignment dataset combination we report in the paper. If you had picked our best performing setting (i.e. alignment on COCO-Captions), you would observe that only OVDiff (Karazija et al., 2023), CLIP-DIY (Wysoczanska et al., 2024) and CLIP-DINOiser (Wysoczanska et al., 2023) outperform it. We also provide insights in the paper regarding this (see L395 before paper update).

Overall, we outperform all other methods on 7 out of 8 evaluations and all other methods on average over these 8 evaluations. The aggregate performance of SimZSS is superior to all other baselines which is what we state in the paper.

Thanks for your reply! The revised version is better. However, I disagree that the motivation is only derived from experimental improvement. The authors should provide convincing evidence to support their argument, rather than just saying the performance is improvement. The behind reason of improved performance can be diverse. Besides, the following statement is overclaimed:

The vision encoder must be capable of clustering images or groups of pixels with shared semantics.
The text encoder must be sufficiently aligned with the visual modality to accurately label these clusters.

So I change my rating to 5.