POC: Preventing the Over-Collapse of Classes for Class-Incremental Learning

Dear Reviewer UhWh,

We are grateful for your valuable time and opinions. We would like to answer your questions according to specific contents in our paper to help you have a more comprehensive understanding of our work.

Why over-collapse leads to forgetting is not clear.

The over-collapse indirectly causes forgetting by first leading to overlapping between seen and future classes in the feature space, which will then result in forgetting. As detailed in line 14, line 46 and Figure 1, over-collapse is a phenomenon that the classification model maps originally separated seen class and its adjacent regions in the input space to be mixed in the feature space. This phenomenon has been observed and analyzed in previous works, such as [1] and [2]. For future classes in these adjacent regions in the input space, the areas covered by them and the seen class after being mapped into feature space will then overlap, as shown both empirically and experimentally in line 46, Figure 3 and Figure 8. As stated in line 53, according to Masana et.al (2022), the overlapping will then result in forgetting since the model will classify the overlapped areas belonging to future classes to minimize classification loss. Therefore, over-collapse will indirectly cause catastrophic forgetting.

Our approach primarily addresses future classes residing in the adjacent regions of seen classes, as they are more prone to overlapping. While seen and future classes distant in the input space could also overlap in the feature space, since most neural networks are equivalent to continuous functions and according to [2], it is much more possible for the future classes that are in the adjacent regions of seen class in the input space to overlap with the seen class in the feature space. Therefore, we mainly focus on the future classes residing in the adjacent regions of seen classes in the input space. We will generalize our POC to deal with other possible situations in our future work.

DCL is not well-motivated and contradictory to Proposition 3.2

DCL is designed to make the features of one seen class and its adjacent regions close to protect model’s generalization. Differently, Proposition 3.2 aims to ensure that learnable transformations produce samples within the adjacent regions of seen class in the input space. Therefore, they operate in different spaces with distinct purposes so that are irrelevant and not contradictory to each other.

As proved in [3], a classification model will maximize the distance between two classes after being mapped into the feature space to improve its generalization. In this way, as detailed in line 73 and line 274, $L_{Mod-Cls}$ will make the model map one seen class and its adjacent regions to be distant in the feature space. This reduces generalization, as testing samples in these adjacent regions may cross the classification boundaries, resulting in misclassification. Therefore, DCL is proposed to make the features of the seen class and adjacent regions close to protect model’s generalization. We have added the referenced paper to line 77 for support.

[1] On the decision boundary of deep neural networks

[2] Empirical study of the topology and geometry of deep networks

[3] Large margin deep networks for classification

Dear Reviewer UhWh,

Thank you once again for taking the time and effort to review our submission. We sincerely appreciate your valuable feedback and have made a concerted effort to address all your concerns, including clarifying the relationship between over-collapse and forgetting, and the motivation of DCL.

To address your main concerns, we would like to draw your attention to the following results and updates:

• Relationship clarification: As in line 46, Figure 3 and Figure 8, we both empirically and experimentally demonstrate that over-collapse will lead to overlapping between seen and future classes in the feature space. And the phenomenon that overlapping between seen and future classes in the feature space will result in forgetting is proved by Masana et.al (2022). In this way, it is shown that over-collapse will indirectly cause forgetting. According to [2], the future classes in adjacent regions are more prone to overlapping so that we mainly focus on them.

• Motivation of DCL: According to [3], the model without DCL will map one seen class and its adjacent regions to be distant in the feature space so that the model’s generalization will be harmed. As a solution, DCL is proposed to make the features of the seen class and adjacent regions close to protect model’s generalization. Differently, Proposition 3.2 aims to ensure that learnable transformations produce samples within the adjacent regions of seen class in the input space. Therefore, there is no relationship between DCL and Proposition 3.2 so that they are not contradictory to each other.

With the rebuttal period extended by a week, we kindly request your attention to our response. Your assessment is vital to the improvement of our work, and we greatly value your insights. We are also fully prepared to provide any additional clarifications or analysis you may require to facilitate the re-evaluation of our paper. Your understanding and guidance mean a great deal to us, and we remain hopeful for your feedback during this extended period.

Best Regards,

The Authors of Submission 5728

[2] Empirical study of the topology and geometry of deep networks

[3] Large margin deep networks for classification

Dear Reviewer UhWh,

Thank you once again for your valuable comments and suggestions. We have carefully responded to each of your concerns according to the specific lines in our paper and revised our paper based on your suggestions.

We understand that this is a particularly busy time. However, it is less than 10 hours to the last time that the reviewers can post message to the authors. Therefore, we deeply appreciate it if you could take a moment to review our responses and let us know if they adequately address your concerns.

Moreover, we would greatly appreciate it if you could re-evaluate the overall score and other aspects of our work based on the responses we have submitted.

Best regards,

The Authors of Submission 5728

Dear Reviewer rQbH,

We sincerely thank you for your valuable comments, which have helped us improve our work. In the following, we will solve your concern with more experimental results.

Whether the method can be applied to other tasks.

Since our POC only modifies the optimization objective of the classification model, it can be seamlessly integrated into other incremental learning tasks where classification serves as a subtask. To demonstrate it, we supplement experiments when our POC is integrated into 2 incremental object detection methods, 2 incremental semantic segmentation methods and 1 few-shot incremental instance segmentation method to show the generalization ability of POC. We only apply our POC to the classification branch of these methods, and do not apply the learnable transformations to the “background” category in these tasks since it is a collection of multiple classes.

For the incremental object detection task, we evaluated POC using ABR [1] and BPF [2] as baseline methods on the PASCAL-VOC 2007 dataset. Performance is measured using mean average precision (mAP) at a 0.5 IoU threshold. In each incremental setting A-B(C-D), A represents the number of classes in the initial stage, B indicates the number of classes introduced at each subsequent stage, C-D means the mAP from C-th class to D-th class. As shown in the table below, incorporating POC consistently improves mAP across all settings, demonstrating its effectiveness in incremental object detection.

For the incremental semantic segmentation task, we adopt DKD [3] and STAR [4] as baseline methods, with performance evaluated using mean Intersection over Union (mIoU) on the PASCAL-VOC 2012 dataset. Similar to the detection task, A-B(C-D) specifies the incremental setting. The results presented in the table below show consistent improvements in mIoU across all configurations after adopting POC, highlighting its effectiveness in this context.

For few-shot incremental instance segmentation, we use iMTFA [5] as the baseline method. Following the referenced work, we report both AP and AP50 on the COCO dataset, considering 60 base classes and 20 novel classes. The results include performances with K=1, 5, and 10 shots per novel class. As shown in the table below, the incorporation of POC leads to improved AP and AP50 values across all scenarios, further validating its effectiveness for few-shot incremental instance segmentation.

In conclusion, our POC is generalizable to other incremental learning tasks where classification serves as a subtask. We have supplemented the experiments into Section C.7 of our new version of paper.

[1] Augmented box replay: Overcoming foreground shift for incremental object detection

[2] Bridge past and future: Overcoming information asymmetry in incremental object detection

[3] Decomposed knowledge distillation for class-incremental semantic segmentation

[4] Saving 100x storage: prototype replay for reconstructing training sample distribution in class-incremental semantic segmentation

[5] Incremental few-shot instance segmentation

Dear Reviewer rQbH,

Thank you once again for taking the time to review our submission. We deeply appreciate your thoughtful feedback and have worked diligently to address your problem by supplementing more experiments and analysis on other incremental learning tasks.

To address your primary concern, we would like to highlight the following results and updates:

• Generalize POC to other tasks: We integrate our POC into 2 incremental object detection methods, 2 incremental semantic segmentation methods and 1 few-shot incremental instance segmentation method to show that POC is generalizable to other incremental learning tasks where classification serves as a subtask. The results supplemented in Table 17, 18 and 19 in Section C.7 show that incorporating POC consistently improves performance across all tasks and settings, demonstrating its effectiveness in other incremental learning tasks.

With the rebuttal period extended by a week, we kindly request your attention to our response. Your evaluation is crucial to improving our work, and we greatly value your insights. We are fully committed to addressing any questions or providing additional analysis that might assist in re-evaluating our paper. Your understanding and guidance are deeply appreciated, and we remain hopeful for your feedback during this extended period.

Best Regards,

The Authors of Submission 5728

Discuss the issue of increased training costs

To address concerns regarding training costs, we primarily use training time as a representative metric. With the same experimental settings as in Section 4.2.1, we report the training GPU days of various methods on CIFAR-100 and ImageNet, utilizing a single GeForce RTX 3090 GPU. The tables below show the results on CIFAR-100 and ImageNet-100 respectively. As shown in the table, while adopting our POC slightly increases the training time, the difference remains minimal. Furthermore, when the dataset is larger, the proportion of increased time is smaller.. This is due to the parallel computing capability of GPU. Although the batch size increases due to the classification of augmented images, parallel computation ensures that the overall training time does not scale proportionally with the batch size. We have supplemented the experiments in Section C.9 in new version of paper.

Compare with results when augmented images are used as positive samples.

Following the setting of Section 4.2.1, we evaluate the performance on CIFAR-100 and ImageNet-100 when images produced by transformations are used as positive samples. The tables below are the results on CIFAR-100 and ImageNet-100 respectively. According to the results, the performance gain is lower than that when the augmented samples are regarded as negative.

We analyze the Inter-Class Distance (ICD) and Intra-Class Generalization (ICG), and the results are in Figure 4 in new version of paper. The results show that when augmented samples are considered positive, both ICD and ICG increases. It results from that rotated samples have similar distribution to the original ones as discussed in line 258. Additionally, the transform loss $L_{Trans}$ makes the transformations similar to rotations. Therefore, the transformations will converge to rotations to minimize both $L_{Mod-Cls}$ and $L_{Trans}$, reducing $L_{Total}$. This effectively mirrors training with rotation augmentations so that the decision boundary of one class will extend more broadly. While this enhances model's generalization, it exacerbates the over-collapse phenomenon, which in turn worsens the overlap between seen and future classes. As a result, the overall performance gain is limited. Instead, POC is designed to both prevent over-collapse and protect the generalization so that it obtains better performance. The experiments have been supplemented in Section C.2 in new version of paper.

Dear Reviewer dVqr,

We sincerely appreciate your valuable feedback and the opportunity to clarify our work. Below, we provide detailed responses to your concerns, supported by additional explanations and supplemented experimental results.

Insufficient analysis of over-collapse and its relationship with catastrophic forgetting.

In our paper, we primarily demonstrate that over-collapse will lead to overlapping between seen and future classes in the feature space. By citing previous work (Marana et.al 2022) to prove that overlapping can lead to catastrophic forgetting, we can indirectly show that over-collapse is one of the causes of forgetting. And as in line 46, Figure 3 and Figure 8, we both empirically and experimentally show that over-collapse will lead to overlapping, supporting our motivation to prevent over-collapse to avoid catastrophic forgetting.

Regarding the link between overlapping and catastrophic forgetting, as outlined in lines 12 and 53, when only limited samples from seen classes are available during the learning of future classes, the classification model tends to misidentify overlapping samples from seen classes as belonging to future classes. This behavior minimizes classification loss but results in knowledge forgetting. This relationship has been previously demonstrated in works such as Masana et al. (2022) so that we cite it for support, rather than to claim the relationship between over-collapse and forgetting. We apologize for the ambiguity in line 184 of the original text and have revised it in our new version of paper.

The over-collapse phenomenon is analogous to the well-known generalization behavior of classification models in other tasks, and has been analyzed in previous studies. For instance, references [1] and [2] reveal that models will extend their decision boundaries beyond the distribution of training data, mapping one class and its adjacent regions in the input space to be mixed in the feature space to improve generalization. While this behavior is beneficial in other classification tasks, we find it detrimental in class-incremental learning. To emphasize its negative impact, we rename the phenomenon. We have added the two references to line 49 for support.

We provide both empirical and experimental evidence to establish the relevance between over-collapse and overlapping. Empirically, as shown in line 50 and Figure 1, over-collapse causes the classification model to map one seen class and future classes in its adjacent regions to overlap within the feature space. Experimentally, baselines adopting POC demonstrate reduced ICD (as shown in Figure 3), which quantitatively measures the extent of overlapping between seen and future classes. Additionally, the visualization in Figure 8 illustrates the separation of seen and future classes after adopting POC, qualitatively supporting the same conclusion. These results collectively show the relevance between over-collapse and overlapping so that over-collapse will indirectly cause forgetting, validating our motivation to prevent over-collapse.

[1] On the decision boundary of deep neural networks

[2] Empirical study of the topology and geometry of deep networks

Dear Reviewer dVqr,

Thank you once again for your time and effort in reviewing our submission. We sincerely appreciate your valuable feedback and have worked diligently to address all your concerns, including clarifying our motivation, developing training time analysis, comparing with positive sampling and revising the manuscript accordingly.

To address your main concerns, we would like to draw your attention to the following results and updates:

• Motivation clarification: We have modified line 49 and line 184 to clarify our motivation to prevent the over-collapse, which is that it will lead to overlapping between seen and future classes. This can be shown both empirically and experimentally by line 46, Figure 3 and Figure 8. By citing the work of Masana et.al (2022) to prove that overlapping can result in catastrophic forgetting, we indirectly show that over-collapse is one of the causes of forgetting, supporting our motivation.

• Training time analysis: We supplement the training time analysis of POC with the results in Table 21 and 22 in Section C.9. Thanks to the parallel computing ability of GPU, the increased training time of backbones after adopting POC is minor and less than 10% in most cases.

• Compare with positive sampling: We supplement the performance on CIFAR-100 and ImageNet-100 when images produced by learnable transformations are used as positive samples. The results in Table 11, 12 and Figure 4 in Section C.2 show that although the generalization of models is enhanced when using positive sampling, the over-collapse is exacerbated so that the performance gain is minor.

As the rebuttal period has been extended by a week, we humbly and kindly request your attention to our response. Your assessment is critical to the improvement of our work, and we deeply value your insights. Moreover, we remain fully committed to providing any clarifications or additional analysis you might find necessary to help re-evaluate our paper. Your understanding and guidance would mean a lot to us, and we remain hopeful for your response during this extended period.

Best Regards,

The Authors of Submission 5728

Dear Reviewer dVqr,

Thank you once again for your valuable comments and suggestions. We have carefully responded to each of your concerns, provided additional experimental results to further clarify our points, and revised our paper based on your suggestions.

We understand that this is a particularly busy time. However, it is less than 10 hours to the last time that the reviewers can post message to the authors. Therefore, we deeply appreciate it if you could take a moment to review our responses and let us know if they adequately address your concerns.

Moreover, we would greatly appreciate it if you could re-evaluate the overall score and other aspects of our work based on the responses we have submitted.

Best regards,

The Authors of Submission 5728