Continual Gaussian Mixture Distribution Modeling for Class Incremental Semantic Segmentation

We greatly appreciate the reviewer's comprehensive review and recommendation of our work. Our responses to each point are as follows:

1. Empirically assuming fixed GMM components for per class in latent features.

Considering the generalizability of our method, we chose to use a fixed number of components (K=3) per class for all incremental scenarios, based on the results presented in Table 4 of the paper. We show additional results on VOC 15-1 using more values of K as follow:

The results suggest that the overall performance of our method remains stable across most K values, indicating that it is not highly sensitive to the selection of this hyperparameter. Additionaly, the results reveal a slight decline in performance when the number of Gaussian components beyond three. We speculate that using a large number of Gaussian components may lead to partial overfitting, particularly for classes with simple feature distributions in the training set. Therefore, we choose K=3 for the final model, holding performance and computational efficiency simultaneously. Obviously, using a fixed number of GMM components is unlikely to be fully robust across all classes, because the complexity of feature space varies significantly between different classes, as discussed in Appendix A.6 of our manuscript. We look forward to addressing this issue in future work by assigning an adaptive number of Gaussian components to each class, enabling the model to better adapt to varying levels of feature distribution complexity across different incremental scenarios. We will include these results and give more disscussion in the final version.

2. Mathematical notations.

We apologize for the unclear explaination of some mathematical notations in the manuscript. We will review these notations and provide detailed explanations in the final version. Following your valuable suggestion. Below is a summary table of all symbols with subscripts/superscripts, along with their corresponding notations, presented as comprehensively as possible. Please note that some vector symbols in the table below may be displayed unnormally due to platform limitations. We will present the standard symbols in the final version. In addition, some typos of formulas and symbols might result in misunderstanding (e.g. $\mu_{n_o}$ should be ${\mu}_{n_c}$ in Eq. 10.). We will modify them in the final version.

3. Usage of GMM features during classification.

We appreciate your thoughtful recommendations. At step $t+1$, we utilize the storaged GMMs to generate pseudo-features for the old classes. Specifically, for each old class $c$, we reproduce the $N_c$ pseudo-features by per training epoch: $\mathcal{F}\_c = \mathcal{S}(\phi_c^*, N_c)$, where $\phi\_c^{\*}$ denotes the parameters of GMMs and $\mathcal{S}(\cdot)$ represents Gaussian sampling function. Then, pseudo-features are concatenated with the real features $F$ extracted from the input training images by the feature extractor $f^{t+1}$, and jointly fed into the classifier $g\^{t+1}$. Consequently, we obtain the corresponding pseudo-prediction $p$. We compute a standard mBCE loss by: $\mathcal{L}\_{mbce}=-\frac{1}{N}\sum_{j=1}^{N}\sum_{c \in \mathcal{C}^{t+1}}w\cdot y_{j, c}\cdot\log p_{j, c}+(1-y_{j,c})\cdot\log(1-p_{j,c})$ Here, $N$ is the number of reproduced pseudo-features, $j$ is the index, $\mathcal{C}^{t+1}$ is the set of foreground classes at $t+1$ step, and $w$ is the positive weight. $y_{j,c}$ is the ground-truth of the $j$-th features sample (0 means it does not belong to class $c$, 1 means it does). In this way, the classifier $g^{t+1}$ can capture discriminative characteristics between new and learned classes in the feature space, achieving a good balance between stability and plasticity. We will include the such mathematical formulations and description in the final version.

We thank the reviewer for the thoughtful response.

As you mentioned, when $y_{i,c}=0$, the value of the first term in $\mathcal{L}\_{mbce}$ will be $0$ and the classfier $g^{t+1}$ is only supervised by the second term. This reflects our expectation that the classifier $g^{t+1}$ should assign lower probability scores when pseudo-features of old classes are treated as negative samples of new classes, thereby ensuring a clear decision boundary between old and new classes. Since we use the same mBCE loss function to calculate the losses of all classifier outputs, the form of $\mathcal{L}\_{mbce}$ that presented in the initial response can be regarded as the pseudo-prediction loss part, which is not affected by the weight $w$. The weighting parameter $w$ mainly works during the learning of new class pixels in the real images of $\mathcal{D}^{t+1}$. We show whole $\mathcal{L}\_{mbce}^{all}$ for pseudo-features and real pixels as follow: $\mathcal{L}\_{mbce}^{all}=-\frac{1}{H\cdot W+N}\sum_{j=1}^{H\cdot W+N}\sum_{c \in \mathcal{C}^{t+1}}(w\cdot y_{j, c}\cdot\log p_{j, c}+(1-y_{j,c})\cdot\log(1-p_{j,c}))$

where $H\cdot W$ indicates the spatial size of the input image and $j$ is the index of pseudo-features or real pixels. Considering the imbalance between the number of new class samples and that of other samples (real background, and pixels or pseudo-features from other classes, we set the parameter $w$ to balance the weight of the two terms.

We sincerely thank you for precious time and valuable comments on our work agin.

I thank the authors for preparing the rebuttal.

I have a follow-up question for #3: For each pixel location on image, only one class is positive among all $c\in C$. Therefore, $y_{i,c}=0$ may dominate $L_{mbce}$.

How do you set the weighting parameter $w$?

We appreciate your detailed comments and suggestions. Our responses to each point are as follows:

1.Inconsistent statement and typos.

We apologize for the misunderstanding caused by our oversights regarding the writting of Eq. 10. The goal of $L_{grc}$ is to maintain the discriminative distance between new classes and their most similar old counterparts. We will correct all identified issues in the final version:

a. We will change $\mu_{n_o}$ to $\mu_{n_c}$ and modify Eq.10 to $L_{grc}=\frac{1}{|\mathcal{C}^{t+1}|}\sum_{n_c\in\mathcal{C}^{t+1}}\frac{1}{\min_{o_c\in\mathcal{C}^{1:t}}\||\mu_{n_c}-\mu_{o_c}\||_2}$ for consistant statement. b. We will change $\pi_j$ to $\pi\_{cj}$ in Eq. 3 to refer to the specific class $c$. c. We will modify the font style of $\pi\_{ck}$/$\pi\_{cj}$ from vector to scalar across the paper, which denotes a probability value blong to $k$-th Gaussian component for class $c$.

2.How to ensure a sufficiently clear decision boundary if some classes are widely distributed in the feature space? Will this also cause overlap?

Thank you for your thoughtful comment. We believe that a well-trained feature extractor enables features of the same class to be distributed in a unified and compact latent space. Consequently, the distances among the K mixture components of a given class tend to be closer to each other than to those of different classes. We argue that computing the geometric centroid provides a more intuitive and comprehensive representation of old class feature distributions than relying on the means of individual mixtures. For new class features, supervised by $L_{grc}$, it is easier to maintain an appropriate distance outside the distributions of old classes rather than being closely embedded within them. Based on the above analysis, we believe that using the geometric centroid formulation effectively avoids the issue of overlap. The t-SNE visualization results of Figure 5 in our manuscript further intuitively demonstrate that our method can ensure a sufficiently clear decision boundary. Additionally, we report the results of different $L_{grc}$ settings regarding the formulation of old class centroids on VOC 15-1 below (as shown in Appendix Table 7), to further verify the effectiveness of our design. Here, 'original' represents that using the K mean vectors derived from the stored GMMs as centriods of each learned class to compute the L2 distance between these centriods and new counterparts.

3.Wheter dynamically adjusting the distribution of old classes also be considered a form of forgetting?

There might be a misunderstanding. In our view, the primary cause of forgetting is that the model fails to capture the complete distribution of previously learned classes due to lacking the past data during the incremental learning process. Our method of Gaussian mixture distribution modeling utilizes stored GMMs to reproduce the complete distribution of previously learned classes, which has significantly alleviated the issue of forgetting. However, the distribution of the old classes may change because the model parameters are updated to accommodate new classes. This leads to a mismatch between the fixed distribution obtained by the stored GMMs and the evolving distribution of previously learned classes reflected in the continually updated model. Thus, we propose a dynamic adjustment (DA) strategy to update the GMMs, allowing them to continuously adapt to the changes in the distribution of previously learned classes within the model. Based on the above analysis, we tend to argue that our method of dynamically adjusting the distribution of previously learned classes addresses the issue of distribution mismatch, rather than representing a form of forgetting. The results presented in the table below demonstrate the effectiveness of our DA strategy.

4.Usage of prototype and specific setting of the baseline.

Usage of prototype: In continual learning process, single-prototype based methods utilize stored prototypes obtained from previous learning steps to generate old class features. This is typically done by repeatly adding random Gaussian variances to the mean vectors of old classes. The generated features are then combined with real features and fed into the classifier to facilitate the learning of new classes.

Specific setting of the baseline: In the ablation studies, we selected the segmentation network implemented with DeepLabv3 and ResNet-101 pretrained on ImageNet as our baseline model, which was surpervised by standard a multiple Binary Cross-Entropy (mBCE) loss in the initial step. During the incremental learning process, we incorporate the knowledge distillation loss from STAR [1], along with an uncertainty loss from Adapter [2], as additional supervision.

We will clarify the usage of prototype and specific setting of the baseline in the final version to help the readers to follow.

[1] Saving 100x storage: Prototype replay for reconstructing training sample distribution in class-incremental semantic segmentation. In NIPS 2023.

[2] Adaptive prototype replay for class incremental semantic segmentation. In AAAI 2025.

Dear Reviewer zb2d,

The author-rebuttal phase is now underway, and the authors have provided additional clarifications and performance results in their rebuttal. Could you please take a moment to review their response and engage in the discussion? In particular, we’d appreciate your thoughts on whether their revisions adequately address your initial concerns. Thank you for your time and valuable contributions.

Best, Your AC

We greatly appreciate the reviewer's insightful review of our work. Our responses to each point are as follows:

1.Different joint result of CNN compared with previous methods.

As you mentioned, some methods reported different joint results (e.g., 77.4 in MiB [1] and MBS [2]). In our manuscript, we shown the joint results of CNN directly cited from CoinSeg [3] for the consideration of revealing the performance gap between the current CISS methods and the best upper bound. Based on our observations, these methods differ from each other in experimental settings (e.g., learning rate, batch size, and training epochs). Additionally, the segmentation network in Coinseg is trained with the assistance of proposals generated by Mask2Former, further enhancing the performance of joint training. Thus, we reproduce as many results on VOC 15-1 and joint results with correspond official code as possible as follow:

These results show that our approach achieves the smallest performance gap (4.7%) to ours own upper bound. Moreover, our joint result is largely consistent with other methods, except for CoinSeg. Therefore, we argue that our evaluation provides a fair comparison environment, and the joint result of Coinseg serves as a reference for the best upper bound. We will provide more such results of other incremental scenarios in the final version.

2.Limited intuitive experiment results showing the anisotropy problem. (W2 & Q2)

We are grateful for the reviewer's insightful comments. We roughly treat the change in old class features during the learning of new classes as an anisotropy problem, which was analyzed in FeCAM[4]. Considering your recommendation, we conducted visualization experiments on the feature distributions of some old classes at incremental steps on VOC 15-1. We observed that the shapes of some feature distributions have undergone certain changes, indicating that part of the old class features have deviated from their initial positions during the model's updating process. We truly want to show these visualizations but there is no way to add new figures during the rebuttal process. Therefore, we will add such visualizations with more analysis in the final version so please understand it.

3.Experiments on computational and memory cost with previous exemplar-based methods.

As suggested, we provide a comparison of the overall performance (mIoU) on VOC 15-1, computational complexity (GFLOPs), model size (#Params), and storage cost for exemplars or prototypes across the three methods, as follows:

The results show that our method achieves performance gains of 1.2% at similar model sizes and GFLOPS. In addition, we use 0.12M memory for storaging GMMs, which is far lower than the SSUL that storage raw images. Even when compared with the prototype-based method (STAR), our storage cost only increase 0.08M that almost be ignored.

4.Effectiveness of method for Mask2Former-based structures.

We appreciate your thoughtful recommendations. Our current research primarily focuses on Deeplab-based continual learing architectures. The Mask2Former-based structures in MBS [2] differ from the DeepLab-based architecture used in our manuscript, requiring us to make several adjustments to adapt the method for implementation on such a framework. This might take a relatively long time. Thus, we look forward to exploring the effectiveness of our method for Mask2Former-based structures in future work, and we kindly ask for your understanding.

[1] Modeling the background for incremental learning in semantic segmentation. In CVPR 2020.

[2] Mitigating Background Shift in Class-Incremental Semantic Segmentation. In ECCV 2024.

[3] CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation. In ICCV 2023.

[4] FeCAM:Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning. In NIPS 2023.

Thank you for your thorough review and insightful questions again. Regarding your Q2 ('Is this method effective for Mask2Former-based structures compared to [1][2][3]?'), we preliminarily attempted to implement our method on such a structure used in MBS [2] and conducted the experiments on VOC 15-1 compared with other methods as follow:

The results demonstrate the effectiveness of our method on such a structure, achieving competitive performance. We also believe that with further exploration, our method has the potential to perform even better on this structure.

[1] Modeling the background for incremental learning in semantic segmentation. In CVPR 2020.

[2] Mitigating Background Shift in Class-Incremental Semantic Segmentation. In ECCV 2024.

[3] Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class. In CVPR, 2023.

[4] Rbc: Rectifying the biased context in continual semantic segmentation.In ECCV, 2022.

Dear Reviewer dzLn,

The author-rebuttal phase is now underway, and the authors have provided additional clarifications and performance results in their rebuttal. Could you please take a moment to review their response and engage in the discussion? In particular, we’d appreciate your thoughts on whether their revisions adequately address your initial concerns. Thank you for your time and valuable contributions.

Best, Your AC

Thank you very much for your valuable feedback, time, and attention. We sincerely appreciate your encouragement. Your feedback and engagement throughout the review and discussion process has greatly improved our paper, and we will be sure to incorporate these relevant results and clarifications into our final version.

We sincerely appreciate the reviewers' thorough evaluation and constructive feedback. The major comments are answered below.

1.Difference between our method and STAR.

As you mentioned, some prior works have employed prototypes to generate the features of learned classes, with a representative being STAR [1] that you referred to. However, there were some limitations and challenges in the previous methods, which have been discussed in our submitted manuscript. Our main contribution is to address these problems in prototype-replay based CISS methods. We highlight clear differences between ours method and STAR from three points as follow:

• Gaussian Mixture Models (GMMs) vs. single prototype: Firstly, STAR utilizes a single prototype to represent the complete distribution of each old class, which may be insufficient when the incoming data stream for a class is inherently multimodal. Secondly, the method of obtaining prototypes by directly computing the mean vectors of old class features is not easily extendable from a single prototype to multiple prototypes. Different from STAR, to address the aforementioned limitations, we are the first to ingeniously incorporate the multimodal characteristics of GMMs into the CISS framework, enabling accurate modeling of old class distributions through iterative learning with the Expectation-Maximization (EM) algorithm. The results from the table in the response '3.Results for different values of K.' also demonstrates the effectiveness of our use of multiple Gaussians.

• Dynamic GMMs adjustment vs. fixed prototype replay: During incremental learning steps, the model needs to update its parameters to accommodate new classes. Inevitably, some old class features that are sensitive to parameter changes deviate from their original positions. This is more likely to result in a mismatch between the fixed prototypes in STAR and the changed feature distributions of the old classes. To address this issue, we propose a Dynamic Adjustment (DA) strategy that continuously updates the parameters of stored GMMs using the real-time features of old classes at each learning step, thereby adapting to the changed representations of old class distributions. The results on VOC 15-1 (Table 3 in our manuscript) below suggest that our DA strategy further improve the performance of the model on both of new and old classes. Baseline|GMMs|DA|0-15|16-20|all :-|-|-|-|-|-| ✔|✔||79.1|50.5|72.3 ✔|✔|✔|79.9|51.8|73.2

• Gaussian-based representation constraint (GRC) loss vs. similarity-aware discriminative (SAD) loss: As you mentioned, we adopt a contrastive loss formulation similar to that of SAD. However, SAD selects a single prototype as the centroid of each old class, which ignores the non-uniform density and multimodal characteristics of feature distributions. In contrast, we refine the computation of old class centroids by adaptively combining multiple Gaussian means, weighted by the respective probability of each mixture component. This refinement enables a more faithful representation of the old class distributions. To further illustrate the impact of our GRC loss, we present additional experimental results on VOC 15-1 below, comparing the final models trained with either SAD or GRC loss. The results demonstrate that our GRC loss provides more effective guidance for learning new classes and significantly improves performance. Method|0-15|16-20|all :-|-|-|-| With $L\_{sad}$|79.8|52.1|73.2 With $L_{grc}$|80.1|53.6|73.8

In summary, our method diverges from STAR both in features modeling (single prototype vs. multiple Gaussian mixture components) and features replaying (fixed prototype vs. Dynamic GMMs). Moreover, our modification of anchors of old class in GRC loss is line with the disign of our method, fundamentally distinguishes it from previous SAD loss in STAR. From the results of our extensive experiments and ablation study, it can be seen that our method of continual Gaussian mixture distribution modeling (CoGaMiD) is effective, particularly in challenging multi-step scenarios (e.g. VOC 1-1 (20 steps) and ADE 100-5 (11 steps)).

2.The specifics of the refined knowledge distillation loss $L_{kd}$ in Eq. 11 are unclear. Is this referring to the effectiveness of the Old-Class Features Maintaining (OCFM) loss in STAR?

We apologize for unclearly notation of $L_{kd}$ in Eq. 11. As you guessed, we trained our baseline using OCFM loss in STAR as the form of $L_{kd}$. Since our focus is not on improving the knowledge distillation techniques that widely used in CISS, we hope to adopt the most effective distillation loss from existing methods to establish a stable baseline. We provide the results for baseline using different distillation losses on VOC 15-1 as follow:

The results demonstrate the effectiveness of OCFM, which is why we adopted it as the knowledge distillation component and used it to train the model as our baseline. We will clarify the notation of $L_{kd}$ in the final version.

3.Results for different values of K.

Following your suggestion, we show results for different values of K on VOC 15-1, ranging from 1 to 3, for both the baseline and final models as follows:

The results suggest that using multiple Gaussians simultaneously improves the performance of the baseline and our final model, which demonstrates the superiority of our Gaussian Mixture distribution modeling approach compared to the use of a single prototype.

4.The results for the 'joint' method appear to be higher than those for DKD, and present the reproduced results of some baseline models.

As you mentioned, some methods reported different joint results (e.g., 77.6 reported in DKD [4]). In our manuscript, we reported the joint results of CNN directly cited from CoinSeg [5], in order to highlight the performance gap between existing CISS methods and the best upper bound. Based on our observations, the differences in the joint results of these methods mainly stem from variations in the experimental settings (e.g., learning rate, batch size, and training epochs). Additionally, the segmentation network in CoinSeg is trained with the assistance of proposals generated by Mask2Former [5], which further enhances the performance of joint training. Following your suggestion, we conducted joint training using our method and reproduced as many results with the corresponding official code as possible on VOC 15-1. The results can be seen in the response to Reviewer dzLn ('1.Different joint result of CNN compared with previous methods.'). These results suggest that our joint result is almost consistent with previous methods, ensuring a fair comparison and to highlight the effectiveness of the proposed methods without modifications to the architecture or minor techniques. We will provide more such reproduced results for other incremental scenarios in final version.

[1] Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation, In NIPS 2023.

[2] Incremental learning techniques for semantic segmentation. In ICCVW, 2019.

[3] Plop: Learning without forgetting for continual semantic segmentation, In CVPR 2021.

[4] Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation, In NIPS 2022.

[5] Masked-attention mask transformer for universal image segmentation, In CVPR 2022.

Dear Reviewer vdnJ,

The author-rebuttal phase is now underway, and the authors have provided additional clarifications and performance results in their rebuttal. Could you please take a moment to review their response and engage in the discussion? In particular, we’d appreciate your thoughts on whether their revisions adequately address your initial concerns. Thank you for your time and valuable contributions.

Best, Your AC

It would have been helpful to include additional experiments to justify the need for Multiple Gaussians over a Single Gaussian, beyond the observed performance improvements — for example, by examining whether the actual feature distribution is indeed multimodal. In addition, the concept of updating memory when storing features or distributions is not novel, as similar ideas have already been applied in works such as ALIFE [S1]. Furthermore, the proposed Dynamic Adjustment (DA) method itself also appears to lack technical novelty.

Therefore, it would be beneficial to provide further explanations or additional experiments to more convincingly justify the use of GMMs.

[S1] ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation. NeurIPS 2022.