SemPLeS: Semantic Prompt Learning for Weakly-Supervised Semantic Segmentation

Thank you for all the constructive comments. Here are our replies:

1. Complex training pipeline:

The whole WeakTr pipeline has two stages: 1) CAM generation for pseudo-label, and 2) online retraining for final segmentation. The whole CLIP-ES pipeline also has two stages: 1) CAM generation with refinement for pseudo-label, and 2) segmentation model training with their proposed confidence-guided loss. Therefore, replacing the mask generator with CLIP-ES will not reduce the training pipeline complexity.

2. Dependency of the mask generator: We have also built our method upon CLIMS (we use the better version provided by the official repo). The results are as follows:

The above table demonstrates that our proposed SemPLeS exhibits effectiveness with both a CNN-based mask generator, such as CLIMS, and a Transformer-based mask generator, like WeakTr, and improves both CLIMS and WeakTr, respectively.

3. More meaningful comparison:

As shown in the above table, our proposed SemPLeS can work different segmentation models (e.g., CLIMS and WeakTr), and improve both performances, respectively.

COCO 2014 is a challenging dataset for the WSSS problem. The average mIoU for WSSS papers published at top conferences (e.g., CVPR, ECCV, NeurIPS, etc.) was 43.11 in 2022 and increased to 43.58 in 2023. Notably, the overall improvement in WSSS within the community over the past year is estimated at around 0.47. Hence, the achieved 0.5 improvement is a non-negligible advancement in the context of WSSS performance.

Thank you for all the constructive comments. Here are our replies:

1. Citation missing:

Thanks for sharing. We will cite both ICD [1] and SPML [2] in our revised paper. Our difference from the above two papers:

• ICD [1] and our method both aim to separate the foreground and the background, but they achieve this by learning intra-class boundaries while we learn the class-associated prompts embedded with semantic knowledge.
• SPML [2] and our method both conduct contrastive learning, but they focus on pixel-segment contrastive learning while we focus on image-text and text-text contrastive learning.

2. Confusion about class-specific background prompts: The whole $L_{prompt}$ in Eq. (2) is used to learn class-associated background prompts, including $L^b_{prompt}$ and $L^f_{prompt}$:

• $L^b_{prompt}$ aims to attract class-associated background prompts to predicted background images
• $L^f_{prompt}$ aims to repel class-associated background prompts from foreground class labels

After the class-associated background prompts are learned, we use the learned prompts to refine the mask with $L_{refine}$. Therefore, without $L_{refine}$, we cannot know how $L^b_{prompt}$ and $L^f_{prompt}$ affect the final mIoU performance. That is to say, training with only $L_{match}$ and $L^f_{prompt}$ will provide the same results as training with $L_{match}$.

3. Comparison w/ SAM & Why not SAM:

Thanks for the insightful comments. We acknowledge the effectiveness of SAM [5] as an established segmentation model and we agree that combining SAM with open-vocabulary semantic segmentation methods, such as OVSeg, leads to enhanced performance. However, our paper focuses on the challenging WSSS problem, specifically in the scenario where only class labels are available, without the use of mask annotations during training. SAM's training on the SA-1B dataset including over 1 billion mask annotations, places it beyond the scope of the WSSS problem. Nonetheless, we find it intriguing to explore how our SemPLeS framework could enhance SAM's semantic understanding capabilities, offering valuable insights into WSSS under different constraints. We consider this as a potential direction for our future work.

4. Why CLIP instead of OVSeg/MaskCLIP+:

This paper tackles the WSSS problem under the constraint of using only class labels during training, excluding the use of mask annotations. In light of this restriction, CLIP emerges as a favorable option for external knowledge incorporation due to its training without reliance on mask annotations. Notably, CLIP is designed for general purposes rather than being tailored solely for segmentation tasks. A recent trend involves leveraging knowledge within the CLIP space to enhance performance in dense prediction tasks. We aim to harness the semantic knowledge ingrained in the CLIP space, contributing to the improvement of WSSS training.

The reason why OVSeg [4] and MaskCLIP+ [3] are not better choices for this paper:

• OVSeg (CVPR’23) [4]: it is trained w/ COCO-Stuff segmentation annotations, which violates the WSSS protocol.
• MaskCLIP+ (ECCV’22) [3]: it uses the transductive setting, so the testing data is seen during training, which also violates the WSSS protocol.

Nonetheless, there is an intriguing avenue for exploration in integrating certain concepts from OVSeg (e.g., learnable mask prompt) or MaskCLIP (e.g., prompt denoising) into our SemPLeS framework. We consider this as a potential direction for our future work.

---

References:

• [1] (ICD) Junsong Fan, et al. “Learning Integral Objects with Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation”, CVPR, 2020.
• [2] (SPML) Tsung-Wei Ke, et al. “Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning”, ICLR, 2021.
• [3] (MaskCLIP) Chong Zhou, et al. “Extract Free Dense Labels from CLIP”, ECCV, 2022.
• [4] (OVSeg) Feng Liang, et al. “Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP”, CVPR, 2023.
• [5] (SAM) Alexander Kirillov, et al. “Segment Anything”, ICCV, 2023.

We will add missing citations in the revised paper.

Re: Re: 2. Confusion about class-specific background prompts

Thanks for the valuable feedback. We will change the notation as follows:

• $L^b_{prompt}$ $\rightarrow$ $L^I_{prompt}$: the loss is computed using class-associated background prompts and predicted background images.
• $L^f_{prompt}$ $\rightarrow$ $L^T_{prompt}$: the loss is computed using class-associated background prompts and foreground class labels.

We will revise the whole paper accordingly.

Re: Re: 3. Comparison w/ SAM & Why not SAM

Thank you for the comments.

The annotation for semantic segmentation includes two aspects: localization and classification. Fully-supervised semantic segmentation (FSSS) relies on full mask annotation for training, with pixel-level localization and class labels. Methods without full supervision can be categorized as WSSS. However, the current convention of the WSSS community defines Weak only in terms of localization, which means that the ground truth does not contain precise pixel-level location information, but with (sparse) point-level, box-level, or image-level supervision instead. Our focus is exclusively on image-level supervision, lacking any location information.

SAM, although falling under the WSSS umbrella according to the aforementioned definition due to the absence of class annotation, surpasses the focus of this study by incorporating foreground/background annotations. We are sorry for the confusion caused by our previous comments. While SAM aligns with the broader WSSS definition, it extends beyond the scope of WSSS with image-level supervision in this work. This work focuses on the problem with only image-level supervision without any location information.

We appreciate the discussion prompted by your inquiries. WSSS with SAM is an under-explored research avenue, involving class-agnostic mask annotation and image-level labels. We find it intriguing to explore how our SemPLeS framework could benefit this direction, offering valuable insights into WSSS under different constraints. We see this as a prospective direction for our future work. Thank you for initiating this valuable discussion.

Thank you for your insightful feedback.

We agree that SAM reshapes the research directions in the community, especially for segmentation-related tasks, and SAM can solve various types of WSSS problems. However, WSSS with image-level supervision remains a formidable challenge. Recent investigations [2-5] have explored the potential of leveraging SAM for this specific problem. Notably, most of them integrate SAM with additional foundation models (e.g., BLIP-2 [6], Grounding-DINO [7], RAM [8]) to enhance SAM’s ability to incorporate semantic information. The results in the below table demonstrate that even with SAM, the WSSS with image-level supervision problem is still challenging, and our proposed SemPLeS has already outperformed several SAM-based methods.

We chose VOC and COCO since they are widely recognized benchmark datasets for this problem at the time of submission to ICLR’24. We acknowledge your suggestion to focus future research directions on tasks where SAM faces challenges or is inapplicable.

In accordance with the feedback you provided, data domain emerges as a prominent challenge, particularly when data distributions deviate significantly from SAM's original image training data, encompassing scenarios such as concealed objects [1] and glass objects [11]. Addressing this issue requires the adoption of additional methodologies. Moreover, model efficiency is also another issue. SAM's application in resource-limited or time-sensitive scenarios, such as mobile applications, is hindered by its large model size. Exploring strategies to enhance SAM's efficiency without compromising accuracy represents a compelling and practical avenue for future research. Furthermore, similar to most deep models, robustness is another issue that SAM is facing [9-10]. We consider these directions as our future work.

Thank you for continuing this valuable discussion. We hope our responses address your concern. We would also be appreciative if you could consider re-evaluating the rating of this work. Please let us know if there are any concerns or questions, and we are eager to provide further clarification or discussion.

---

References:

• [1] Chunming He, et al. “Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping”, NeurIPS, 2023.
• [2] Tianle Chen, et al. “Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation”, NeurIPS Workshop, 2023.
• [3] Peng-Tao Jiang, et al. “Segment Anything is A Good Pseudo-label Generator for Weakly Supervised Semantic Segmentation”, arXiv, 2023.
• [4] Zhaozheng Chen, et al. “Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Models”, arXiv, 2023.
• [5] Weixuan Sun, et al. “An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problems”, arXiv, 2023.
• [6] Junnan Li, et al. “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models”, ICML, 2023.
• [7] Shilong Liu, et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection”, arXiv, 2023.
• [8] Youcai Zhang, et al. “Recognize Anything: A Strong Image Tagging Model”, arXiv, 2023.
• [9] Yuqing Wang, et al. “An Empirical Study on the Robustness of the Segment Anything Model (SAM)”, arXiv, 2023.
• [10] Yu Qiao, et al. “Robustness of SAM: Segment Anything Under Corruptions and Beyond”, arXiv, 2023.
• [11] Dongsheng Han, et al. “Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects Cannot Be Easily Detected”, arXiv, 2023.

Thank you for all the constructive comments. Here are our replies:

1. (i) End-to-end WSSS:

Most WSSS papers follow the standard 3-stage pipeline. There are a few end-to-end WSSS papers, but their performances are worse than standard 3-stage methods. The quantitative comparison in the below table shows that our method significantly outperforms current SOTA end-to-end methods, including ToCo [4] and MCC [14]. We will introduce more end-to-end methods in Related Works.

(ii) Contribution of using CLIP:

There are not many WSSS papers based on CLIP. Most CLIP-based papers are related to the open-vocabulary setting which requires mask annotations for training. We summarize recent CLIP-based WSSS papers and explain how they exploit CLIP as follows:

• CLIMS (CVPR. 2022) [3]: refine masks by manually prompting the pretrained CLIP model.
• CLIP-ES (CVPR, 2023) [5]: extract attention maps from the CLIP-pretrained ViT model
• MMCST (CVPR, 2023) [7]: use text features from the CLIP-pretrained ViT model to perform their proposed contrastive loss.

Different from the above, we learn the class-associated prompts embedded with semantic knowledge, which can be used to refine masks. The quantitative comparison in the below table shows that our method outperforms current CLIP-based WSSS methods.

2. Symbols in Figure 2 (a):

Thanks for the suggestion, and we will revise the figure as suggested.

3. (i) details of text prompt t_k (how to initialize & shape):

The learnable prompts are randomly initialized with the shape K by D, where K is prompt length, and D is the prompt embedding dimension of CLIP.

(ii) difference from current prompt learning methods:

As mentioned in Related Works, our Contrastive Prompt Learning aims to capture class-associated semantic knowledge for segmentation purposes rather than replacing the manual-defined text template like “a photo of {}” for classification purposes as in current prompt learning methods.

4. More SOTA comparison (including MARS):

We are not sure if it’s reasonable to ask for a comparison with publications where the conference dates are after the ICLR’24 submission deadline, such as ICCV 2023.

Nevertheless, we list all WSSS papers published in CVPR and ICCV 2023 in the below table with the heatmap and segmentation backbones. Our method outperforms almost all other methods in the table except for MARS (ICCV’23) [10]. MARS [10] builds upon a strong baseline RS-EPM (arXiv’ 22) [1] with a strong backbone DeepLabv3+ (ResNet101), and it also proposes to exploit an additional strong Unsupervised Semantic Segmentation model, STEGO (ICLR’ 22) [2] with ViT-B, to remove biased objects and achieve the current SOTA for WSSS. This method is interesting and we envision its potential integration with our automatic prompt learning method for further improvement. We leave this prospect as part of our future work.

5. Labels presented in Figures 3 and 4:

Thanks for the suggestion, and we will revise the figure as suggested.

6. “All BG Prompts” in Figure 4:

“All BG Prompts” means the visualization is generated from ALL background prompts manually defined in CLIMS [3].

7. Table 4 clarification:

$L_{prompt}^b$ and $L_{prompt}^f$ are used to learn the background prompts, and then we use the learned prompts to refine the mask with $L_{refine}$. Therefore, without $L_{refine}$, we cannot know how $L_{prompt}^b$ and $L_{prompt}^f$ affect the final mIoU performance. Training with $L_{match}$ + $L_{prompt}^b$ + $L_{prompt}^f$ will be the same as training with $L_{match}$.

Thank you for all the constructive comments. Here are our replies:

1. SOTA comparison:

The better SOTA by WeakTr uses different settings, including a better ViT backbone pre-trained on ImageNet-21K [1] and a larger resolution (384) for fine-tuning. For the classification performance on the ImageNet-1K dataset, DeiT-S (our backbone) has 79.9 accuracy while ViT-S (SOTA WeakTr’s backbone) has 83.7 accuracy. We choose not to use the better WeakTr as the backbone because we would like to follow the standard protocol of the WSSS benchmark for fair comparison.

2. Panoptic segmentation:

To our best understanding, although there are some weakly-supervised panoptic segmentation papers like WSSPS [2] (box-supervised), PSPS [3] (point-supervised), and Panoptic-FCN [4] (point-supervised), there is no panoptic segmentation work with class supervision. It would be an interesting research direction to explore, and we see it as a potential future work.

---

References:

• [1] Andreas Steiner, et al. “How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers”, TMLR, 2022.
• [2] (WSSPS) Qizhu Li, et al. “Weakly- and Semi-Supervised Panoptic Segmentation”, ECCV, 2018.
• [3] (PSPS) Junsong Fan, et al. “Pointly-Supervised Panoptic Segmentation”, ECCV, 2022.
• [4] (Panoptic-FCN) Yanwei Li, et al. “Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision”, TPAMI, 2022.

We will add missing citations in the revised paper.