IPSeg: Image Posterior Mitigates Semantic Drift in Class-Incremental Segmentation

Thank you for your positive feedback and insightful comment! In regard to the weaknesses and questions raised, we provide our point-to-point responses below:

Q1: Question about the decoupling implementation and detailed comparisons with existing works.

A1: Previous works, such as SSUL, use pseudo-labels and saliency maps to decouple background regions and mitigate the background shift challenge. However, they do not fully address the challenge of semantic shift due to incomplete pseudo labeling and decoupling.

1. Based on the observation in L233 (L234 in the revised version), an image can be divided into four distinct parts: the region of past classes $\mathcal{C}_{1:t-1}$, target classes $\mathcal{C}\_{t}$, unknown foreground $c'_u$ and pure background $c'_b$. This strategy is adopted by both previous works [1-3] and ours.
2. SSUL identifies unknown classes $c'_u$ from the background and employs extra parameters for prediction, which is optimized together with other target class heads. It inevitably introduces noise and affects the learning of target classes, as discussed in L148-154 in our paper.
3. Essentially, we fundamentally analyze the characteristics of noisy semantics and applies a decoupled, divide-and-conquer strategy to learn the noisy semantics. Specifically, we introduce two separate branches to decouple the learning of these classes:

• Temporary branch: This branch learns the target classes $\mathcal{C}_t$ and non-target foregrounds $c_f$, which change dynamically across each incremental step.
• Permanent branch: This branch learns the pure background $c'_b$ and unseen foreground $c'_u$, which persist throughout the incremental learning process.

We also provide a visualization of semantic decoupling in Figure 7.

Q2: Details of memory buffer with image-level samples.

A2: We use a shared memory buffer for both the image posterior branch and the segmentation branch. Below, we provide a detailed explanation of how this memory buffer is constructed, updated, and how it storages samples：

1. Memory Construction and Update: Given the memory size $\mathcal{M}$ and the number of already seen classes $|\mathcal{C}\_{1:t}|$, the memory buffer is constructed before step 1 with $\mathcal{M} // |\mathcal{C}_{1:t}|$ samples per class. Once initialized, it is updated before the start of each new step, as done in previous works [1-3]. The update of our memory buffer follows a class-balanced sampling, which is explained in detail in Section 3.5.
2. Data Storage: For raw data, IPSeg directly stores the image paths in a JSON file, as done in previous works [1-3]. For image-level labels, IPSeg stores the class labels of the images as arrays in the same JSON file with multi-hot encoding, where $1$ indicates the presence of a class and $0$ indicates absence. The memory cost for this is negligible. For pixel-level labels, instead of storing full-class annotations (with data type of uint8 ) as prior approaches, IPSeg only stores the salient mask, where the background and foreground are labeled as $0$ and $1$, respectively (with data type of bool ). Theoretically, the storage space could be reduced to $1/8$.

Q3: The figure presentations are inconsistent. These should be presented by pdf, etc.

A3: Thanks for your valuable feedback and careful check regarding the quality of our figure presentations. We replace all blur figures you point out with newer high-resolution PDF version in the the revised manuscript.

Q4: Moreover, the authors shall further make some comparisons in terms of training efficacy with the previous method, since they use a memory buff and constitute a two-branches architecture.

A4: Please refer to our official comments, we provide a detailed and comprehensive analysis on training and inference cost, thank you.

[1] SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning, NeurIPS 2021.

[2] Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation, NeurIPS 2022.

[3] CoinSeg: Contrast Inter-and Intra-Class Representations for Incremental Segmentation, ICCV 2023.

Thank you for your positive comments and recommendation to our work. We look forward to any further discussions and will make all essential improvements based on your and other reviewers' feedbacks.

Best wishes!

Q4: The training and inference costs, such as training time and memory.

A4: Please refer to our official comment, we provide a detailed and comprehensive analysis on training and inference cost, thanks.

Q5: The effect of the salient maps supervision and the quality of the salient maps on the ADE20K dataset.

A5: It indeed poses a challenge when using saliency maps on ADE20K to identify target regions, specifically for objects of small scale and at the edges of images. According to your concerns, we conduct the ablation study on the ADE20K 100-10 task with the following settings: "w/o Sal" uses no saliency map supervision, "w/ Sal" uses saliency map supervision as we do in paper, and "w/ SAM" uses saliency maps extracted by SAM [3] model. We use the official Grounded SAM [4] code with all class names in ADE20K as text prompts to extract corresponding masks in implementation. Additionally, we also report performance differences on "thing" and "stuff" classes defined in ADE20K panoptic segmentation[5] to investigate the bias of saliency maps on different semantic regions.

Basically, we find the following conclusions:

1. Using the default saliency map supervision to implement knowledge decoupling strategy, IPSeg gets a performance improvement by $+1.6$% mIoU. And using saliency maps extracted by SAM further improves IPSeg performance by $+0.7$% mIoU.
2. The default setting performs well in identifying "Things" classes but struggles with "Stuff" classes, resulting in a performance gain of $+1.9$% on "Things" classes but merely $+0.1$% on "Stuff" classes. Furthermore, the SAM-based saliency maps provide better supervision for both "Things" and "Stuff" classes, with improvements of $+0.6$% on "Things" and $+1.1$% on "Stuff" compared to "w/ Sal".

In summary, the analysis result confirms that the quality of saliency maps affects model performance on the ADE20K dataset. We will report this conclusion in our supp to inspire future works.

In our paper, IPSeg uses the same saliency map extraction method without more information leakage to ensure a fair comparison with baseline methods. Despite this, IPSeg still achieves SOTA performance compared to SOTA methods like MicroSeg and CoinSeg, which utilize region proposals from more advanced models like Mask2Former as auxiliary information. Thanks for your interesting suggestion.

Q6: Why is the replay-based IPSeg (33.6) worse than data-free-based PLOP+NeST (34.8)? Is there any insight or explanation?

A6: We find the PLOP+NeST outperforms IPSeg in the ADE20K 50-50 setting mainly because of two reasons:

1. Unfair Training Epochs: Compared with IPSeg, NeST [6] requires additional 15 warm-up epochs to initilize new classifiers for each step. These extra training epochs allow NeST to better adapt to new classes, leading to improved performance.
2. Characteristics of the 50-50 Task: NeST aligns new classifiers with the backbone and adapts to the new class data with extra 15 warm-up epochs, which relies on sufficient training data. In the "50-50" or "100-50" settings, there are a large amount of new classes and training data to help NeST better warm up. However, in long-term challenging tasks such as "100-5" and "100-10", there is not enough new task data to achieve similar warm-up effect, and its performance is not ideal as in "50-50" setting.

Generally speaking, the performance advance on the ADE20K 50-50 setting is highly related to the inconsist training setting in NeST.

[1] SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning, NeurIPS 2021.

[2] Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation, NeurIPS 2022.

[3] Segment Anything, ICCV 2023.

[4] Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks, Arxiv 2024.

[5] Semantic Understanding of Scenes Through the ADE20K Dataset, IJCV 2019.

[6] Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation, ECCV 2024.

Thank you for your positive feedback and constructive comments. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:

Q1: Why the classification subnetwork does not suffer from catastrophic forgetting? Is there any evaluation of the performance of the classification subnetwork?

A1: Evaluation on classification subnetwork: We provide an evaluation of the performance of the classification subnetwork after all steps as below. We hold three experiment settings: "Seq" refers to training the subnetwork sequentially without any additional tricks, indicating the worst case suffering from catastrophic forgetting, "Ours" refers to the training setting used in our paper, and "Joint" refers to training with all task data jointly as the upper bound.

Analysis: It is obvious that the classification subnetwork suffers little forgetting. We achieve this performance mainly owing to two specific settings:

1. Mixed training data: As mentioned in L204-207 (L206-209 in the revised version), the classification subnetwork uses data from both the memory buffer and the current task dataset for training. Compared to fine-grained task heads, it is easier for the subnetwork to learn image-level knowledge from memory buffer.
2. Image-level pseudo-label for supervision: As claimed in L210-213 (L212-214 in the revised version), IPSeg introduces image-level pseudo-label on past classes as supervision to mitigate catastrophic forgetting challenge. The ablation result in Table 4 partially reflects the effectiveness of this design.

Q2: What is the pre-trained knowledge of the backbone from, the first training step or the ImageNet?

A2: Both. We use the ImageNet pretrained Resnet-101 and Swin-B to intilize our backbone. Then, we finetune the backbone at the first step and freeze it in the subsequent steps following previous works [1-2].

Q3: The differences between the mechanism of image posterior probability and transformer-based mask prediction, like Incrementer.

A3: Thanks for your reminder. We list the commons and differences between Incrementer and IPSeg as follows:

Commons:

1. Mask generation: The processes of mask generation in Incrementor and IPSeg are similar. They all generate mask prediction by fusing dense visual information with sparse class information.
2. Class information: The class tokens in Incrementer and image posterior probabilities in IPSeg both contain classes information.

Differences:

1. Inplementation details: Incrementer computes cosine similarity between visual embedding and class embedding to generate mask prediction. While IPSeg multiplies dense pixel-level class predictions with image-level class predictions to generate the final outputs.
2. Mechanism: In Incrementer, class embeddings are responsible for class information while visual embeddings are class agnostic. But in IPSeg, both the pixel-level and image-level predictions are class-related, and the latter plays a role in introducing image-level class guidance to rectify the former.

Thanks for the authors' response. The authors have clearly demonstrated that image-level classification suffers less forgetting than semantic segmentation, which verifies the motivation of IPSeg. Additionally, the authors show that IPSeg can perform better than PLOP + NeST using the same training epochs. Hope these results can appear in the revision and I decide to raise my rate from 6 to 8.

Q1: please calculate the image-level accuracy for both tasks.

A1: Based on your concerns, we hold two experiments to answer your questions.

1. First, we evaluate the image level accuracy performance of the base 15 classes using the image posterior (IP) branch and the segmentation (Pixel) branch at each step on Pascal VOC 15-1 to further investigate their forggetting on base classes. "IP" refers to only using the IP branch, "Pixel" refers to only using the segmentation branch, where the class $\mathcal{C}$ exists if a pixel is predicted as $\mathcal{C}$. "Pixel+IP" denotes using them both as our paper does.

The ablation shows that: the image-level classification (IP) suffers less forgetting than the segmentation (Pixel), and our method (Pixel+IP) shows similar property against forgetting with the help of IP.

2. Additionally, we also evaluate the image-level accuracy on all seen classes at each step to analyze their performance on both keeping old knowledge and learning new knowledge.

For the segmentation branch, the image-level accuracy of it on all seen classes gradually degrades after learning new classes, performing worse than its accuracy on base classes. This indicates the segmentation branch performs poorly on new classes, which is consistent with our description about separate optimization (L160-161 and L180-182 in the revised manuscript). In contrast, the IP branch experiences less deterioration from separate optimization and help our method maintain a good balance between retaining old knowledge and learning new knowledge. Therefore, the ablation evidently shows that the IP branch learns image-level knowledge better than the segmentation branch.

In summary, these experiments demonstrate that, on one hand, the image classification (IP) branch exhibits higher accuracy after all steps and suffers less forgetting. On the other hand, the IP branch mainly helps our method mitigate the sparate optimization, effectively improving overall performance.

Please let us know whether our response solve your questions and concerns.

Q2: For Q6, the authors' response is not so convincing. I suggest that the authors retrain the proposed IPSeg in the 50-50 setting using any preferred training schedule to show that the performance can be further improved.

A2: Thanks for your suggestion. We run an experiment on ADE20K 50-50 setting with a longer schedule as same as PLOP+NeST uses and report the result as below. Using the same training schedule, our method shows slight advantage over NeST.

Q3: Regarding Q5, Grounding SAM can not be used, because it has data leakage when using all class names as prompts. This is only my reminder, not my concern.

A3: The SAM model segments all regions of a given image indiscriminately, making it difficult to distinguish between foreground and background areas. Consequently, this poses a challenge for using SAM as salient maps within our methods. To address this issue, we employ the Grounded SAM model instead but introduce information leakage problem as you mentioned. We will further explore related techniques. Thanks for your kind reminder.

Thank you for your positive feedback on our work and for increasing the score. We greatly appreciate your constructive suggestions for improving our work.

Best wishes.

Q1: I don't see the rationale why it is called the permanent branch. Generally, only consistent class definitions across all steps can be regarded as permanent semantics.

A1: In our work, the terms "permanent" and "temporary" do not refer to specific classes but depend on their learning cycle throughout the incremental stages. For the concepts existing across all incremental steps, we use the term "permanent" to describe them and the "permanent branch" to indicate the branch consistently learning them across the whole incremental steps.

There are three components needed to be identified once a new incremental step begins: target classes set $\mathcal{C}\_{t}$, the pure background $c_b'$, and the unknown set $c_u'$. Across all incremental steps, $\mathcal{C}_{t}$ changes drastically, $c_b'$ keeps fixed and $c_u'$ shrinks but will not disappear. Compared with the ever changing target class set $\mathcal{C}_t$, the pure background $c_b'$ and the unknown set $c_u'$ exist across all incremental steps.

Further, based on your example, what happens in our IPSeg is as follows: "cat" is included in the unknown set in step 1 to be learned by the permenant branch. In step 2, "cat" is removed from the unknown set and transformed into target class set. After step 2, "cat" is only regarded as a seen class. This transformation process happens on all target classes across all incremental steps.

Q2: How are the final prediction maps generated from the permanent and temporary branches?

A2: As shown by the green lines in Figure 2, the permenant branch $\phi_p$ outputs prediction for the background $c_b'$, and the temporary branch $\phi_i$ (i=1,2,...,t) outputs predictions for the target classes $\mathcal{C}\_t$. And the pixel-level prediction $\phi_{0:T}(h_\theta(x_i))$ can be writen as: $\phi_{0:T}(h_\theta(x_i))= Con( \phi_{p,bg}(h_\theta(x_i)) , \phi_{1:T,target}(h_\theta(x_i)) ).$ Where $Con$ is the concatenate operation, $\phi_{p,bg}(h_\theta(x_i))$ and $\phi_{1:T,target}(h_\theta(x_i))$ represent the background prediction from the permenant branch and the target classes prediction from the temporary branch. And this pixel-level prediction is then producted by image posterior probability to form the final predition maps as Eq.3 and Eq.4.

In the revised manuscript, we provide a detailed description of the prediction map generation process.

Q3: The previous-class image labels are not available at the current step. How to supervise the image posterior in each step?

A3: As mentioned in L204-207 (L206-210 in the revised version), image posterior branch uses mixed data from memory buffer and current training dataset of each step. The supervision is derived from the knowledge of all seen classes in the mixed data, which mainly comes from:

1. Samples from memory buffer are saved with their image-level labels of corresponding stages, contributing to knowledge of previous classes.
2. IPSeg uses image-level pseudo-label of current data to capture the knowledge of old classes.
3. The ground truth of current task data on target classes is available.

To investigate how these forms of supervision improve performance, we provide detailed ablation in Table 4.

Q4: Does the temporary branches predictions contribute to the final prediction or just the target class scores are used to multiply by the corresponding confidence score from the image posterior branch?

A4: Yes, there are two roles the temporary branch plays in contributing to the final prediction. First is the target class predictions $\phi_{1:T,target}(h_\theta(x_i))$, and it is used as our answer to Q2 above. Second is the prediction of other foreground regions $c_f$, and IPSeg utilizes $c_f$ from temporary branches of each step to filter out erroneous outputs during inference as shown in L265–271 (L277-285 in the revised version). While the pure background prediction within each temporary branch does not contibute to the final predictions, and it only helps the model distinguish the target classes during training.

[1] ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning, CVPR 2024.

[2] SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning, NeurIPS 2021.

[3] Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation, NeurIPS 2022.

[4] CoinSeg: Contrast Inter-and Intra-Class Representations for Incremental Segmentation, ICCV 2023.

[5] Label-guided knowledge distillation for continual semantic segmentation on 2d images and 3d point cloud, ICCV 2023.

Thank you for your positive feedback and insightful comments. In regard to the weaknesses and questions raised, we provide our point-to-point responses below.

W1: The claim that "separate optimization does not attract any attention" is inaccurate.

A1: We sincerely thank the reviewer for pointing out this oversight. We notice that ECLIPSE [1] also focuses on this challenge and proposes the logit manipulation method. We also revise the corresponding statement. Besides, though focusing on the same challenge, IPSeg differs ECLIPSE from its solution:

1. ECLIPSE is a prompt-based method built on the Mask2Former network, which encounters an error propagation problem after freezing the old prompts. To address this, ECLIPSE incorporates logit manipulation to leverage common knowledge across the classes.
2. IPSeg is an architecture-based approach built on DeepLabV3 network. We provide a detailed analysis of the issues arising from freezing the old classification heads. IPSeg introduces an image posterior branch to explicitly introduce informative image-level class knowledge into segmentation network and directly overcome separate optimization challenge.

W2: Efficiency comparisons, such as computational complexity, iterations required for convergence, and finetuning time cost.

A2: Please refer to our official comment, we provide a detailed and comprehensive analysis on training and inference cost, thanks.

W3: Salient object detector will incur additional computational costs and brings additional information and is unfair to other competing methods that do not use it.

A3:

1. Computational costs：To make saliency maps as auxiliary information, we use an off-the-shelf salient object detector to pre-compute the saliency maps for the entire training set. This process is performed only once, takes less than 10 minutes for Pascal VOC and 30 minutes for ADE20K, and has no impact on inference time.
2. Fair comparison: We notice SOTA methods [2-4] all introduce additional information to enhance the model's recognition capability. Following these methods, IPSeg utilizes the same saliency maps to ensure a fair comparison.

W4: The memory buffer may raise scalability and privacy concerns, especially when scaling to larger datasets and storing sensitive user data.

A4: We appreciate the reviewer's concern regarding memory buffers, scalability, and privacy.

As for the size of memory buffer and scalability, IPSeg sets $\mathcal{M}=100$ for VOC and $\mathcal{M}=300$ for ADE20K, strictly following the same setting as SSUL, MicroSeg and CoinSeg.

In scenarios with privacy constraints or limited data scalability, we also provide the data-free version of IPSeg (denoted as "IPSeg w/o M" in Table 1) as an alternative with minor performance loss. For privacy concerns, we discuss the limitations of using the memory buffer and its potential social impact in our "Conclusions" section. We agree with that the privacy issues must be handled with caution in artificial intelligence applications.

W5: This paper missed several SOTA methods in experimental comparison.

A5: We provide an experimental comparison on Pascal VOC 2012 and ADE20K below, where IPSeg still achieves competitive performance, particularly in long-term incremental tasks （ADE 100-5 and ADE 100-10）.

These two methods do not hold comprehensive incremental experiments as IPSeg does. Though they have performance advances in their respective domains, differences in focusing task scenarios may limit their direct comparability with IPSeg in the context of our study. Upon this point, it is unfair for them to compare with ours. We also add them into the main results in Table 1 and Table 2 for comprehensive comparisons as your suggestions.

Q: Please do not list the results without denoting whether memory is used or not in the same table in A5. It is not fair to do so. Additionally, I would like to request for Data-Free IPSeg w/o M results on ADE20k.

A: Thanks for your tips. Here, we provide the results denoting whether memory is used or not below. Besides, we reorganize Table 2 and add the results of "IPSeg w/o M" for comprehensive and fair comparison.

From the results, we can draw two key conclusions:

1. Comparable Performance Without Memory Buffer: Even without a memory buffer, our data-free version demonstrates close performance to our data-replay version, exhibiting only a minor performance loss (up to 1.1% when using ResNet-101 on ADE20K). Our method maintains robust and consistent performance across all settings, whether using memory buffer or not.

2. Competitive Performance in Long-Term Incremental Scenarios: The data-free version of IPseg (IPseg w/o M) achieves competitive performance with both ResNet-101 and Swin-B backbones, especially in challenging long-term incremental scenarios (e.g., ADE 100-5, ADE 100-10).

Dear Reviewer wQeR.

As the author-reviewer discussion period is nearing its end, and since other reviewers have actively engaged in discussions, we would greatly appreciate it if you could review our responses to your comments at your earliest convenience.

This will allow us to address any further questions or concerns you may have before the discussion period ends. If our responses satisfactorily address your concerns, please let us know. Thank you very much for your time and effort!

Sincerely,

The Authors of Submission #731

I would like to thank the authors for their efforts in providing additional experiments on ADE20K. However, the results bring me some concerns about the key contribution "the image-level classification (IP) suffers less forgetting than the segmentation".

Concerns:

• IPSeg w/o M performs inferior compared to LGKD+PLOP on the three standard settings ADE20K 100-10, 100-50 and 50-50, when no memory is used. Could the author explain the underlying reasons? As mentioned by Reviewer zxHa, the image-level classification (IP) suffers less forgetting than the segmentation is regarded as the key contribution of this paper. If this is the case, I believe IPSeg w/o M should definitely outperform LGKD+PLOP [5] on ADE20K without any memory. This is my main concern, and I hope the authors can provide convincing response.

• The computational cost is considerable compared to SSUL (137.1G vs. 94.9G), indicating a 44.5% increase. I am not convinced by the cost-efficiency.

Questions:

• From my perspective, ECLIPSE does not suffer from error propagation (L157). Could the reviewer comment on this?
• The subscript $\phi_{1: t-1}\left(h_\theta\left(x_i^{m, t}\right)\right)$ used for $\tilde{\mathcal{Y}}$ is inconsistent to the subscript $i$ used for $\mathcal{Y}^t$. I suppose i to be the i-th input image. However, $\phi_{1: t-1}\left(h_\theta\left(x_i^{m, t}\right)\right)$ is related to the class index predicted by previous heads.

[5] Yang, Ze, et al. "Label-guided knowledge distillation for continual semantic segmentation on 2d images and 3d point clouds." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.