Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learning

Response to Reviewer V6Ai

Thank you for your constructive and detailed feedback. We provide detailed responses to your concerns below. The discussion and experiments below will be properly included in the manuscript.

Q1: Elaborate the advantages of proposed methods over existing works [R1,R2,R3].

Response: The limitations of existing works [R1,R2,R3] can be summarized as follow respectively. 1. Toldo et al [R1] propose to train multi-layer perceptrons or variational autoencoders to estimate the semantic shift, intensifying the training cost. EFC [R2] utilizes the Gaussian kernels to capture the shift by estimating the translation of feature means, potentially neglecting other types of shift transformations including scaling. PRL [R3] proposes a feature projection to align the current embedding space to a pre-defined latent space, rather than estimating the semantic shift. Also, PRL does not consider different classes within a training task. Our DPCR provides a DP to estimate the semantic shift in both efficient and effective way via a linear projection can be calculated directly without iterative BP-training. Also, the CIP in DP includes class-specific knowledge. To demonstrate the effectiveness of DPCR, we further include the experiments compare with EFC and PRL (we did not find the released code of [R1]). The results show the high performance of our DPCR.

Q2: Does CIP need to learn and store a linear projector for each class at each subsequent incremental phase?

Response: CIP is a training-free projection to include the class-specific knowledge and it does not need to be stored. Actually, during the training of DPCR, CIP is calculated from the uncentered covariance via SVD (see Eq. (6) and (7)) and it will be used to calibrate the old information set with TSSP and then discarded. Only the covariance and prototype ${\hat{{\Phi}}{c}^{\theta{t}}, \hat{{\mu}}{c}^{\theta{t}}}$ need to be stored in DPCR. For the operations in Eq. (8) and Eq (9), they are used to demonstrate how the old features of far previous task are calibrated of without loss of generality and this process does not be necessarily performed in the implementation of DPCR. In our implementation, the calibration is conducted across two adjacent tasks (see Algorithm 1). We will further clarify this to avoid potential misunderstanding.

Q3: Using SVD seems to result in information loss.

Response: CIP is used to inject class-specific information via projecting the projector in TSSP to significant directions. The construction of projector for CIP maintains all the singular vectors corresponding to non-zero singular values and only discards the information of null space, which we believe not necessarily needed. To validate the effect of information of null space, we perform an experiment on CIFAR-100 with all the singular vectors to construct the projector of CIP (denoted as DPCR-wn). The results are shown below and we find that including null space information does not affect the performance and thus the information loss may not have large impact on the performance.

Q4: Lack of details of training.

Response: As indicated in line 310 (left), the implementation details are provided in Appendix B. We will move them to the manuscript to avoid possible omission.

Q5: Experiments on larger datasets are beneficial.

Response: As suggested, we conduct the validation on ImageNet-1000 with T=10 using ResNet-18 under the same seed of 1993. The hypeparameters are the same as those in ImageNet-100 experiments. The results shows our DPCR is still competitive on ImageNet1000.

[R1] Bring evanescent representations to life in lifelong class incremental learning. CVPR 2022.

[R2] Elastic feature consolidation for cold start exemplar-free incremental learning. ICLR 2024

[R3] Prospective Representation Learning for Non-Exemplar Class-Incremental Learning. NeurIPS 2024.

Response to Reviewer YwAB

Thank you for your constructive and detailed feedback. We provide detailed responses to your concerns below. The discussion and experiments below will be properly included in the manuscript.

Q1: Lack of evaluation on lager datasets like ImgaNet1000.

Response: As suggested, we validate DPCR and compared methods on ImageNet1000 with T=10. We use the same backbone of ResNet-18 and same under the same seed of 1993 to conduct the experiments. The hypeparameters are adopted the same as those in ImageNet-100 experiments. The results are reported below and our DPCR is still competitive on ImageNet1000. These results will be included in the manuscript.

Q2: Lack of complexity analysis

Response: As suggested, we will include the complexity analysis. Since the incremental representation learning is common in the CIL realm, here we analyze the complexity of DP and RRCR. The complexity analysis will be included as follow.

Suppose $F$ is the FLOPs of the model ($F \approx 1.8\times10^{9}$ for ResNet-18), the time complexity of DP $\mathcal{O}(FN_t + N_td^2 + tC(d^3 + C^2d))$ is the sum of those of the feature extraction $\mathcal{O}(FN_t)$, calculation of the task-wise projection matrix $\mathcal{O}(N_td^2+d^3)$, and the rectification of each class $\mathcal{O}(d^3+C^2d)$. Similarly, the time complexity of RRCR is $\mathcal{O}(d^2N_c + dtC + td^2+d^3)$.

Q3: Some figures lack of horizontal labels.

Response: Thank you for pointing out this. We will check and include the horizontal labels for all the figures.

Response to Reviewer spyP

Thank you for your constructive feedback. We provide detailed responses to your concerns below. The discussion and experiments will be properly included. Also, we will check and revise all the writing mistakes, and include references of ridge regression.

Q1: Comparison between RRCR and NCM/Mahalanobis-distance classifier in FeCAM.

Response: RRCR is a parametrized classifier where the decision boundaies can be directly modified by training the parameters. NCM and Mahalanobis-distance classifiers are non-parametric classifiers that utilizes feature distribution and absence of trainable parameters in limits their adaptability across task. Although they can avoid decision bias and achieve better performance over existing parametrized classifiers, RRCR mitigates this via reconstruction then keep the benefit as a parametrized approach. To validate this, we compare RRCR with NCM and Mahalanobis on CIFAR-100 (T=10) using the same frozen backbone after task 1 (following FeCAM). RRCR surpasses both.

RRCR also better adapts to semantic shift during CIL via reconstruction. FeCAM freezes the backbone to maintain consistent covariances, but this impedes plasticity. Once the backbone evolves, prior class covariances become outdated and hard to update without storing previous data due to centralization to obtain covariance. RRCR, by contrast, recalibrates using ${\hat{{\Phi}}{c}^{\theta{t}}, \hat{{\mu}}{c}^{\theta{t}}}$ via matrix operations, avoiding the need to retain prior data or embeddings.

Q2: Motivation of RRCR.

Response: In this paper, we propose RRCR to address decision bias in parametrized classifier during CIL. Non-parametric classifiers like NCM lack lack adaptability due to their static structure and inability to directly modify decision boudaries. While techniques like ADC/LDC try to offset semantic shift, they remain limited by non-learnable classifiers. Existing parametrized classifiers can directly learn from labels but tend to become biased without old data. RRCR fills this gap, offering the benefits of a learnable model while correcting bias via reconstruction. We will clarify this motivation in lines 066–069.

Q3: Evaluation on fine-grained datasets.

Response: As suggested, we include the results on fine-grained dataset CUB-200 with T=5 in cold-start setting with the same seed. Results shows that DPCR also leads the compared methods.

Q4: Covariance vs. Gram Matrix.

Response: We will replace "uncentered covariance" to "gram matrix".

Q5: Compare with FeCAM.

Response: We compare DPCR with FeCAM and obtain the results of FeCAM via the official implementation under the same seed of 1993. Our DPCR outperforms FeCAM and this can be attributed to that FeCAM needs to freeze backbone thus limits the plasticity. Our DPCR can take advantage of incremental representation learning with semantic shift estimation then a good stability-plasticity balance.

Q6: Acknowledge LDC and add discussion of LDC, ADC, and FeTrIL.

Response: We will include the discussion as follow. To estimate the semantic shift, ADC uses adversarial samples to estimate the translation of prototypes, and LDC introduces a learnale projection. However, ADC only considers the shift of translation, neglecting other transformation including scaling, and LDC only estimates the shift across tasks without class-specific information. Moreover, LDC needs iterative BP-training to get the projector, inducing more computation cost. Inspired by the learnable projector in LDC, DPCR achieves shift estimation via DP with low-cost closed-form solution and class-specific information. FeTrIL uses prototype-based pseudo-features to rebalance classifier training and employs a LinearSVC. However, it freezes the backbone, reducing plasticity. Also, LinearSVC's effect in CIL is not thoroughly studied.

Q7: Decision bias vs. Task-recency bias?

Response: Task-recency bias refers to the model's tendency to favor new tasks. Several works attribute it to the bias in classifier. We use “decision bias” to clarify that the bias originates in the classifier. While similar in outcome, we prefer the more specific term decision bias for better understanding and will clarify this in the text.