MaskCLIP++: A Mask-Based CLIP Fine-tuning Framework for Open-Vocabulary Image Segmentation

We sincerely thank reviewer LpvX for acknowledging our presentation and contributions, as well as for pointing out typos in our manuscript. We will provide further clarification regarding the concerns raised by Reviewer LpvX.

(1) Closed-Vocabulary Performance: The table below summarizes the corresponding performance of MaskCLIP++ on COCO-Stuff and COCO-Panoptic, as reported in Tables 5 and 6.

(2) Regarding Additional Mask Generators: Because the mask generator and CLIP operate as two independent modules in our framework, any mask generator capable of producing a set of mask proposals from an input image can be integrated into the inference process. This adaptability is demonstrated in both Fig. 1b and Table 3 of the main text. As a complement to Table 3, we further evaluate the performance upper bound using ground truth (GT) masks. Additionally, we incorporate SAM (Segment Anything Model) in a simple way: Using SAM under its default configuration to provide mask proposals, then applying our model for mask classification. The results are presented in the table below, under the same experimental setup as Table 3:

Although SAM generates accurate masks, it is observed that under the default configuration, SAM has a lower mask recall compared to Mask2Former, leading to inferior overall performance.

(3) Regarding FLOPs: Existing FLOPs calculation tools are primarily designed for end-to-end network computations. However, in various OVS inference modes, certain custom computations within the models are not easily accounted for automatically. Therefore, we provide below the average inference latency on ADE20K for recent methods tested on the same device (a single RTX 3090, using the best-performing configurations from Table 5), as shown in the table below:

Due to the complete decoupling of CLIP and the mask generator, our inference typically incurs additional computation in CLIP-V compared to methods where the mask generator and CLIP share a backbone (e.g. FC-CLIP).

However, it is important to note that the shared-backbone design may limits the applicability across various VLMs: For example, FC-CLIP does not perform well with ViT-based CLIP architectures. In contrast, the decoupled-backbone inference framework supports both CNN and ViT architectures of CLIP, and offering greater flexibility by allowing the mask generator to be plug-and-play.

(4) On the Implementation of Class Inclusion/Exclusion:
Integrating CLIP and mask generator classification scores based on vocabulary overlap with the training set is a common technique in prior mask-based OVS work [1,2,3]. In our approach, we inherit the implementation of this strategy from FC-CLIP but apply it only to seen categories. Below are our responses to related questions:

• Is this classification manual or based on text overlapping of super-categories?
  It is manual. FC-CLIP expands vocabulary with synonyms for several datasets. For example, in COCO-Stuff, "wall-other" is expanded to include "wall," which consequently marks "wall" in ADE as seen and uses the score of "wall-other" from the mask generator during ensemble.

• What are the potential impacts of super-categories appearing in the training source while fine-grained classification is required in zero-shot datasets?
  Since overlaps are manually specified, the impact depends on whether the test category appears in the synonym list of any training category.

• What is the extent of the impact?
  Our use of the ensemble aims to leverage knowledge from the mask generator appropriately, as the additional computational cost is negligible. However, this is not the focus of our method, and its minimal impact on benchmarks in Table 5. When the relationship between test and training categories is unclear, we recommend disabling this strategy.

We appreciate Reviewer n1UQ’s recognition of the effectiveness of our work. We will address the concerns raised by n1UQ in further detail.

(1) Clarification on the Neck structure: Our goal is to improve the mask classification performance of CLIP across various architectures, which is why we do not illustrate a specific CLIP architecture in Fig. 3. However, we have clarified in the first paragraph of Section 3.2 that the neck is part of CLIP-V and summarized its role in spatial dimension aggregation in Equation (1). We then provide a detailed classification discussion based on the original CLIP's pooling methods in the rest of the paragraph.

(2) Clarification on the PSM structure and consistency alignment: We have provided the design principles and structural details of PSM in the second paragraph of Section 3.2. In summary, our PSM structure is very simple, often requiring just a single linear layer. For CLIP models originally using average pooling, we supplement global information during the PSM process as described in Eq. 3. The key design principle behind these simple PSM structures—consistency alignment—is what we aim to emphasize in Fig. 3.

(3) Clarification on "not needing a mask generator": This paper operates within the inference paradigm of mask-based OVS. By analyzing the potential of mask generators and CLIP, we conclude that the bottleneck in CLIP's mask classification limits the full utilization of pretrained mask generators, and joint training may even negatively affect CLIP's mask classification performance. Our innovation lies in the fine-tuning approach, without altering the inference paradigm of mask-based OVS.

(4) Response to "the consistency alignment module can substantially reduce overfitting issues is not well-supported by the experimental results". In Table 1 and the first paragraph of Section 4.2, we demonstrate the importance of maintaining consistency alignment in PSM design for alleviating overfitting during fine-tuning. Furthermore, in our response to Reviewer gVpU under "Regarding the design principles of PSM," we expanded Table 1 with experiments that highlight the phenomena caused by severely violating consistency alignment.

(5) Response to "There is no evidence presented regarding overfitting": As described in the experimental setup in Section 4.1 and the first paragraph of Section 4.2, we partitioned the validation set categories into seen and unseen based on their overlap with the vocabulary of the training set. The baseline in Table 1 represents the mask classification performance of the original CLIP on seen and unseen categories. Evidence of overfitting is demonstrated by the significant drop in performance on unseen categories after fine-tuning, which falls far below the baseline.

(6) Response to "how well this work enhances mask classification compared to the original mask generator.": In Table 3, the $P_s$ column reports the original performance of various mask generators, while the $P_c$ column presents the mask classification performance achieved using MaskCLIP++ on these mask generators. The subsequent $P_\gamma$ column shows the integrated results calculated according to Equation (4). As shown, the primary performance improvement stems from mask classifation abillity of MaskCLIP++.

(7) Response to "original similarity relationships are noisy and require improvement": We agree that this statement summarizes the goal of most OVS methods. However, the motivations and approaches differ significantly. Our research does not aim to explore how to adapt mask shapes to CLIP or relearn pixel grouping and aggregation relationships. Instead, we focus on enhancing CLIP's mask classification capabilities using limited image segmentation data. The key to avoiding overfitting lies in ensuring that the alignment remains consistent before and after applying additional parameters $\theta$.

Dear Reviewer,

Thank you once again for your valuable feedback and comments on our manuscript. We have made revisions based on your suggestions and addressed the concerns raised. We kindly ask if you could confirm whether our responses have sufficiently addressed your concerns.

We appreciate your time and effort in reviewing our work.

Thank you to Reviewer gVpU for acknowledging the simplicity and effectiveness of our method, as well as the presentation of the paper. We will provide further clarification on the concerns raised by gVpU.

[About Methods]

(1) Regarding the design principles of PSM: We will include the following experiments to verify whether PSM significantly violates the consistency alignment constraints and to explain the relationship between maintaining consistency constraints and overfitting. After fine-tuning with a specific PSM, we will compare performance when PSM is retained or removed during inference. As shown in the table below, PSM designs that significantly violate consistency alignment lead to better performance when PSM is removed during inference, but perform much worse on unseen categories after PSM application compared to the baseline. In contrast, PSM designs that largely maintain consistency alignment show consistent improvements over the baseline, both in the presence or absence of PSM, with gains observed in both seen and unseen categories.

(2) Regarding the similarity with the CDT module: Cross-attention structures similar to Content-Dependent Transfer (CDT) are widely used in multimodal alignment. However, we aim to highlight that, when fine-tuning with limited data, the key to maintaining consistency alignment lies in the $\langle E_m, \operatorname{Norm}(E_t) \rangle$ part, whereas $\langle E_m, \operatorname{Norm}(P_t)\rangle$ does not utilize the pre-alignment between $E_m$ and $E_t$. We aim to emphasize the importance of the consistency alignment design principle through Table 1, rather than proposing a specific architecture for the PSM. As for the gains brought by $P_t$ in $\langle E_m, \operatorname{Norm}(E_t+E_p) \rangle$, as discussed in Appendix Table 13, they primarily supplement the global visual information missing from $E_m$.

(3) Regarding multi-granularity open-vocabulary segmentation: To the best of our knowledge, existing mask-based OVS methods train mask generators on the COCO dataset. While their mask generation is class-agnostic to some extent, the extent to which they can achieve multi-granularity segmentation masks remains an open research question. Our work primarily focuses on enhancing CLIP's mask classification capability without making improvements to the mask generator.

Dear Reviewer,

Thank you once again for your valuable feedback and comments on our manuscript. We have made revisions based on your suggestions and addressed the concerns raised. We kindly ask if you could confirm whether our responses have sufficiently addressed your concerns.

We appreciate your time and effort in reviewing our work.

We sincerely apologize for not fully addressing your concerns in our previous response. Thank you for your patience and for providing a more detailed explanation of your doubts. Allow us to provide additional clarifications in a structured manner:

(1) We add an explanation and Figure 5 in the second paragraph of Appendix C.1 to clarify the phenomenon of the similarity deviation. Table 11, presented thereafter, statistically demonstrates the prevalence of this phenomenon across different PSMs.

(2) As stated in the first paragraph of Section 4.2:

when fine-tuning CLIP models with attention pooling (ViT and ResNet), we still use the PSM of $\operatorname{Linear}\langle E_m, E_t \rangle$.

The Large VLMs presented in Table 5 and Table 6 are based on the ViT architecture, which demonstrates that MaskCLIP++ still outperforms other methods based on the simpler $\operatorname{Linear}\langle E_m, E_t\rangle$.

(3) To explore how our method performs on partial segmentation tasks in open-vocabulary settings, we present the following insights:

• Fine-tuning MaskCLIP++ on part-level segmentation data is not required. The following experiment demonstrates this: using groundtruth masks from the same MHP-v1[1] dataset, MaskCLIP++ fine-tuned on COCO-Stuff still achieves a significant improvement in mask classification capability on this dataset. (The 19 categories in MHP-v1, such as left leg, face, and sunglasses, are all novel to categories in COCO-Stuff.)

  	mIoU
  Origin CLIP	30.6
  MaskCLIP++	48.6

• For the mask generator, it may be necessary to retrain it on masks with a similar granularity. Our findings regarding the potential of the existing mask generator are as follows: if the model has seen an example like an elephant during training, it can still generate masks for a similar-granularity object like a giraffe. However, it is known that mask generators, such as Mask2Former, trained at the semantic level typically perform poorly for instance-level segmentation. Therefore, it is hypothesized that they would also struggle to generate effective mask proposals at the part level. In addition to retraining the mask generator, more unified mask generators, such as OneFormer[2], may be worth exploring in the future.

Following your suggestion, we have revised the abstract to emphasize the improvements we made in mask classification.

(4) To the best of our knowledge, previous works using EVA-CLIP include CLIPSelf and Cat-Seg. In our initial response (which has been added to Table 7), we compare the mask classification ability of the CLIPs from MaskCLIP++, CLIPSelf, and Cat-Seg. In addition, CAT-Seg provides their performance using EVA-CLIP in the appendix of their paper. Under the Cat-Seg method, there does not appear to be a clear advantage of EVA-CLIP over OpenAI's CLIP.

We hope that the revisions and responses provided have addressed your concerns clearly. Please let us know if any further clarification is needed.

[1] Li, Jianshu, et al. "Multiple-human parsing in the wild." arXiv preprint arXiv:1705.07206 (2017).

[2] Jain, Jitesh, et al. "Oneformer: One transformer to rule universal image segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

(1) First, we appreciate your pointing out the inconsistencies between the Figure 5 and the main text. We have made the necessary revisions, using $\Theta$ updating the mask embeddings as an example.

(2) We did not provide extensive background for introducing $\langle E_m, \operatorname{Norm}(E_t + P_t) \rangle$ because we are not proposing a specific PSM structure (we also referenced MAFT+ when introducing it). Instead, it arises from our understanding of PSM design. Clearly, there are multiple reasonable PSM structures, but they should all adhere to the consistency alignment constraint as closely as possible.

(3) For the clarification of core idea of our paper, and the differences between our work and CAT-Seg in terms of motivation, explanation, objectives, and effects, please refer to our response to reviewer n1UQ.

We sincerely appreciate your patient and thorough review.

Thank you to Reviewer mMjr for acknowledging the motivation and contributions of our work. We will provide further clarification on the concerns raised by mMjr.

(1) Clarification regarding the relationship with MaskCLIP: We apologize for any confusion caused by the method name "MaskCLIP++." Three recent works have used the name "MaskCLIP" [1, 2, 3], but MaskCLIP++ is not an improved version of any of these works. Since our work focuses on obtaining a CLIP model with improved mask classification capabilities, we named our method MaskCLIP++.

(2) Regarding fairness: As described in related work, there are four types of OVS implementations based on data sources. Our method uses a setup with limited image segmentation data (image, masks, and category annotations). The methods compared in Table 5 and Table 6 are all under this setup.

(3) The combination with SOTA: To the best of our knowledge, all mask-based OVS methods follow a similar "inference" paradigm: mask generation + mask classification. We focus on improving mask classification rather than mask generation. Therefore, after obtaining fine-tuned CLIP through different fine-tuning strategies, our inference pipeline remains largely consistent with recent works such as FC-CLIP and MAFT+. The mask generators used in Table 5 and Table 6 are derived from the publicly available models of MAFT+ and FC-CLIP.

(4) Discussion on data quantity and quality: We understand the reviewer’s concern: “The masks used during inference are often of lower quality compared to the GT masks used during training, and obtaining large-scale GT mask annotations for fine-tuning can be challenging.” We address this concern step-by-step as follows:

• Impact of generated masks: Fine-tuning with the generated masks from FC-CLIP (selecting those with the highest IoU to the GT masks) results in a noticeable performance drop, as shown in Table 2. This indicates that high-quality masks are indeed crucial for fine-tuning.
• data efficiency: We randomly sampled subsets of COCO at different scales (10%, 1%, and 0.1%) as the training set for fine-tuning. The results, presented in the table below, demonstrate that enabling pre-trained VLMs to learn localized recognition does not require extensive image segmentation annotations. This is because we do not rely on segmentation annotations to learn pixel grouping or category information beyond CLIP’s capabilities.

In summary, we believe that for our fine-tuning approach, small but precise annotations are more effective than large-scale noisy annotations. Data scalability efforts should be directed toward the pre-training of CLIP or mask generators rather than fine-tuning CLIP.

We sincerely thank Reviewer mMjr for valuable feedback and thoughtful suggestions. We look forward to further discussions. In the revised paper, all changes will be highlighted in blue and carefully aligned with our responses.

[1] Zhou, Chong, Chen Change Loy, and Bo Dai. "Extract free dense labels from clip." ECCV. 2022.

[2] Ding, Zheng, Jieke Wang, and Zhuowen Tu. "Open-vocabulary universal image segmentation with maskclip." ICML. 2023.

[3] Dong, Xiaoyi, et al. "Maskclip: Masked self-distillation advances contrastive language-image pretraining." CVPR. 2023.