Prospective Representation Learning for Non-Exemplar Class-Incremental Learning

Thank you for your constructive feedback! We hope the following responses can address your concerns.

---

W1 & Q1: What are the advantages of PRL over other forward-compatible methods?

A1: First, unlike previous works, PRL targets CIL in exemplar-free scenarios (NECIL). NECIL requires the algorithm to learn a unified model without access to any data from previous tasks, which means that the commonly used memory buffers are not available thus more challenging.

Second, compared to previous works, PRL considers how to resolve conflicts between new and old classes during the incremental phases, in addition to reserving space in the initial phase. Zhou et al [2] also mentioned that such conflicts need to be taken into account when it transforms into a CIL problem (i.e., there are enough instances of the new classes). Compared to the work of Zhou et al [2], which presupposes unseen classes by mixing instances from the initial task, our approach improves the forward compatibility of the model without interfering with the learning of the initial task itself.

The reference works you provided are inspiring to our research. We will add a discussion of the works on forward-compatible incremental learning in the final version of our paper.

---

W2 & Q2: Lack of evaluation on large datasets like Imagenet1000.

A2: The following table compares the average incremental accuracies of the different methods for the 10 phases setting on ImageNet-1k. We will provide more experimental results on ImageNet-1k in the final version of the paper.

We are looking forward to answering any follow up questions during the discussion period.

Thank you so much for the insightful questions! We will revise our manuscript accordingly and address your questions below.

---

W1: Few analysis was conducted on the loss weight α in formula 14.

A1: We set $\alpha_1=10$, $\alpha_2=10$ , and $\alpha_3=2$ by default. Due to rebuttal limitations, the experimental results are shown in the pdf file attached to the Author Rebuttal. In Eq.14, $\alpha_1$ and $\alpha_2$ are common in previous NECIL methods and represent the weights of distillation loss and prototype loss, respectively. The main role of these two loss functions is to maintain the pre-existing knowledge of the model. Therefore, as shown in Fig.(d) and Fig.(e), as $\alpha_1$ and $\alpha_2$ get larger, the optimization of the model will be biased towards maintaining stability, resulting in the model performing better on the old task and worse on the new task. For the consideration of comprehensive performance, we set $\alpha_1 = 10$ and $\alpha_2=10$ for PRL.

Then $\alpha_3$ controls the loss of the Prototype-Guided Representation Update (PGRU) proposed in this paper. In Figure (c), as $\alpha_3$ increases PGRU comes into play. The effect of growing $\alpha_3$ on the overall performance fluctuates, which may be caused by overly strict constraints on the learning of new class representations. Overall, our algorithm is relatively robust to the choice of hyperparameters.

---

W2 & Q2: Some grammar or writing errors.

A2: Thank you for pointing out our error. As you guessed, $\mathcal{L}_ {IIC}$ was a writing error, and should actually be $\mathcal{L}_ {PES}$. We will carefully scrutinize the main text and the appendix, and ensure such errors are eliminated in the final version of the paper.

---

Q1: Why not perform Preemptive Embedding Squeezing (PES) on new classes during the incremental phase? Will this not cause overlap in the feature space of the new classes?

A3: First, going into the incremental phase, the main problem faced by the model shifts from adjusting the relationship between new classes to dealing with the overlap between the new classes and the old ones, especially in the absence of samples from the old classes. We therefore propose Prototype-Guided Representation Update (PGRU) to alleviate this problem. Due to the shift of the main problem, the improvement from performing PES in the incremental phase is not obvious.

We further reflect on this issue. This could be caused by the fact that as incremental learning goes on and on, the number of new classes is relatively small compared to the old ones. Therefore we try to increase the number of classes in the incremental phase for our experiments on CIFAR-100. In the following table, we performed only one incremental phase and set the number of classes $C$ in the incremental phase to 5, 10, 20, and 50, respectively. It can be seen that the effect of PES is gradually apparent when the number of new classes increases.

Certainly, the ratio of the number of new classes to the number of old classes naturally decreases as the tasks learned by the model accumulate, so we only employ PES in the initial phase. Previous work [1] that considered forward compatibility also optimized the learning of the model only in the initial phase.

Second, in the incremental phase, due to the sufficient number of samples in the new classes, the cross-entropy loss also makes the features of the new classes distinguish from each other and reduce the overlap of the new classes.

---

Q3: What do the old class prototypes of formula 11 and the potential spatial features of formula 12 specifically refer to, and is there a relationship between them?

A4: We have described the old class prototypes and the features in the potential space in Sections 3.2 (line154-158) and Section 3.3 (line195-200), respectively. At the end of each phase, a prototype is computed and saved for each class of the current task. As shown in Eq.4, the prototype is usually the average of all the features of the class. This is widely used in existing NECIL methods [2, 3]. The potential spatial features refer to features $\mathcal{P}_ {\phi_ t}(\mathcal{F}_ {\theta_ {t-1}}(x^i))$ that have been projected, where $x^i$ denotes the sample of the current class, $\mathcal{F}_ {\theta_ {t-1}}(x^i)$ is the feature extracted by the teacher model (the model trained from the previous phase).

Old class prototypes cannot be updated after saving due to the absence of old class data. Since the teacher model $\mathcal{F}_ {\theta_ {t-1}}$ is frozen, $\mathcal{F}_ {\theta_ {t-1}}(x^i)$ also does not change in the current phase of training. Therefore, we introduce a projector $\mathcal{P}_ {\phi_t}$ to project the prototypes $\boldsymbol{p}^c$ and $\mathcal{F}_ {\theta_ {t-1}}(x^i)$ into a potential space. Optimizing the projector via Eq.11 can change the relationship between $\boldsymbol{p}^c$ and $\mathcal{F}_ {\theta_ {t-1}}(x^i)$ in the potential space thus avoiding the new class features to be too close to the region where the old class prototype is located. Eq.12 is to align the features $\mathcal{F}_ {\theta_ {t}}(x^i)$ extracted by the current updated model $\mathcal{F}_ {\theta_ {t}}$ with the potential spatial features $\mathcal{P}_ {\phi_ t}(\mathcal{F}_ {\theta_ {t-1}}(x^i))$ that have been adjusted by the projector.

We will optimize the expression to make this clearer in the final version.

---

Reference:

[1] Shi et al. Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning.

[2] Zhu et al. Prototype augmentation and self-supervision for incremental learning.

[3] Shi et al. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incremental learning.

---

We are looking forward to answering any follow up questions during the discussion period.

Thank you for your response. We suppose that you may have misunderstood our response and our experiments in rebuttal.

Our proposed PES aims to construct a better initial embedding space. Therefore, in our initial submitted version, PES is only performed in the first phase. We have done extensive experiments in our submitted version (Table 2 and Figure 3) to demonstrate the effectiveness of this manner.

We also list the results in the following. Results demonstrate that PES can significantly improve the forward compatibility of model since it reserves space to prepare for future classes. For continual learning on incremental new classes, to make new classes embedded in the space previously reserved, PES was not included in the incremental phase in our submitted version.

In the first round of review, as you suggested, we add experiments that using PES in the incremental phase, shown in the rebuttal, which lead your misunderstandings. The results have verfied that adding PES in the incremental phase indeed does not bring in further improvement. It means that our original design in submitted version is reasonable and effective. We do not need PES in the incremental phase.

We hope this response clarifies your misunderstandings.

Thank you for your thoughtful questions! We are glad that you found the paper easy to read and affirm our experiment. We hope that our response below will address your concerns.

---

W1: What is the meaning of IIC in equation 9? Is it a writing error?

A1: We are sorry for our writing error. It should be $\mathcal{L}_{PES}$ in Eq. 9. We will carefully check and ensure such errors are eliminated in the final version of the paper.

---

W2 & Q1: Discussion of the differences and advantages of the method proposed in this paper over previous studies on mapping class centers to different subspaces.

A2: First, unlike previous works, PRL targets CIL in exemplar-free scenarios (NECIL). PRL does not need to store exemplars of past tasks to promote subspace orthogonality.

Second, previous works start to deal with the conflict between old and new tasks only when a new task arrives. However, due to the lack of samples from old tasks in NECIL, it is intractable to deal with this conflict using only the new task data. Instead, we consider reserving space for unknown new classes in the initial phase to prepare for conflict resolution in advance.

The references you provided is enlightening to our research. We will include a discussion of the above methods in the Related Work section.

---

W3 & Q2: Explain the advantages of using prototype augmentation in the baseline compared to PASS.

A3: Since only one prototype is saved for each class, using one prototype to represent the old class distribution would lack diversity. Prototype augmentation helps to maintain the discrimination between old and new classes and prevents the decision boundary from being biased in favor of the new classes. From the perspective of our paper, both prototype augmentation and the knowledge distillation technique commonly used in NECIL are prompting the model to achieve backward compatibility, i.e., to make the updated model compatible with classes that have already been learned. The PRL proposed in this paper, on the other hand, is prompting the model to achieve forward compatibility, i.e., enhancing the ability of the model to be compatible with unseen classes. Therefore, better performance can be achieved when both forward and backward compatibility of the model are improved.

After PASS [1], many other prototype augmentation strategies have been proposed, such as [2, 3]. We adopt the prototype augmentation method called Prototype Reminiscence (PR) from [2] as our baseline, which is described in Section 3.2 (line 162).

The experiments in the Appendix (Table 6) also demonstrate that our PRL can be combined with PASS or other methods in a plug-and-play manner and enhance its performance. We complement the performance of PASS combined with PRL on the TinyImageNet dataset in the following table, where 'P' denotes the number of incremental phases. It can be seen that the PRL brings a significant boost to PASS as well.

---

W4: More sensitivity analysis of the hyperparameters.

A4: We set $\alpha_1=10$, $\alpha_2=10$, and $\alpha_3=2$ by default. When a sensitivity analysis is performed on one of the hyperparameters, default settings are used for the remaining hyperparameters. Due to rebuttal limitations, the experiment results are shown in the pdf file attached to the Author Rebuttal. The figures in the left column show the effect of changing the value of each hyperparameter on the average incremental accuracy of our method. The figures in the right column show the effect of changing the value of each hyperparameter in the last phase on the accuracy on the new and old tasks, respectively.

Among the three hyperparameters in Eq. 14, $\alpha_1$ and $\alpha_2$ are common in previous NECIL methods and represent the weights of distillation loss and prototype loss, respectively. The main role of these two loss functions is to maintain the pre-existing knowledge of the model. Therefore, as shown in Figure (d) and Figure (e), as $\alpha_1$ and $\alpha_2$ get larger, the optimization of the model will be biased towards maintaining stability at the expense of plasticity, resulting in the model performing better on the old task and worse on the new task. It can be seen that as the value of $\alpha_1$ and $\alpha_2$ increases to a certain level its performance improvement on old tasks slows down. Excessively large values of $\alpha_1$ and $\alpha_2$ will bring much less gain on the old task than they will hurt performance on the new task. For the consideration of comprehensive performance and with reference to previous works [1, 3], we set $\alpha_1 = 10$ and $\alpha_2=10$ for our method.

Then $\alpha_3$ controls the loss of the Prototype-Guided Representation Update (PGRU) proposed in this paper. In Figure (c), as $\alpha_3$ increases PGRU comes into play. The effect of growing $\alpha_3$ on the overall performance of the algorithm fluctuates, which may be caused by overly strict constraints on the learning of new class representations. Overall, our algorithm is relatively robust to the choice of hyperparameters.

We will add a clarification on hyperparameters in the final version.

---

Reference:

[1] Zhu F et al. Prototype augmentation and self-supervision for incremental learning. CVPR 2021.

[2] Shi W et al. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incremental learning. ICCV 2023.

[3] Wang S et al. Non-Exemplar Class-Incremental Learning via Adaptive Old Class Reconstruction. ACM MM 2023.

---

We are looking forward to answering any follow up questions during the discussion period.

Thank you for your positive response! We are delighted that the reviewer has found our method clear and sound. We have addressed the main points and questions below.

---

Q1：Lack of discussion with other methods that employ feature space compression for incremental learning.

A1: Thanks to your suggestion, we will add a separate subsection to the Related Work section to discuss methods on feature space compression in incremental learning.

---

Q2: The standard deviation of the results between different runs that change the order of the classes.

A2: We perform three runs and use a different random seed to set the class order for each run. The table below shows the standard deviation of the three runs, where 'P' denotes the number of incremental phases and 'Tiny' denotes the TinyImageNet dataset. The experiments show that our method is robust to different class orders.

---

Q3: Comparison on the number of model parameters, training time and testing time.

A3: The introduction of a projector in our method introduces additional parameters, but the number of parameters added is minimal compared to the network as a whole. And the forward propagation does not need to go through the projector during the test. The following table shows the model parameters of our method, the training time for one epoch in the last incremental phase and the testing time (on all classes) on CIFAR-100. The experimental results are performed with in the same environment.

Model parameters：

Testing time:

Training time for one epoch:

We are looking forward to answering any follow up questions during the discussion period.