Replies to Reviewer tsV2

Thank you for your constructive and detailed feedback. We provide detailed responses to your concerns below.

W1. Why the avg accuracy on CIFAR-100 is much lower than the last accuracy?

Response to W_1: As mentioned in Appendix G, tasks on CIFAR-100 are notably more complex and intricate, resembling a few-shot learning scenario. Consequently, the accuracy in the initial tasks tends to be lower, leading to a relatively low average accuracy.

W2. Explain more regarding "weight-invariance" in Theorem 3.1.

Response to W_2: The weights of the classifier obtained by this recursive update are exactly the same as the weights obtained by training the classifier from scratch on the entire data set. This weight-invariant property achieves a near “complete non-forgetting” and makes the GACL outperforms all EFCIL methods and most replay-based methods.

W3. It would also be interesting to see the scenario where the pre-trained backbone yields a large domain gap.

Response to W_3: As a demonstration, a 5-phase experiment is conducted in table below, with DeiT-S/16 backbone pretrained on ImageNet-1k, and CIL with DTD [1].

[1] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, Andrea Vedaldi; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 3606-3613.

Q1. Since the experiments only focus on generalized class-incremental learning, it would be better to adjust the title accordingly.

Response to Q_1: Thank you for pointing out that. We will modify our title into "Exemplar-Free Generalized Analytic Class Incremental Learning".

Q2. Why is the proposed method specifically designed for generalized class-incremental learning (GCIL)?

Response to Q_2: As demonstrate in Section 3.2 and 3.3, we introduce the ECLG module that corresponds to the exposed-class gain since classes can reappear in GCIL setting.

Q3:Will GCIL still work if the data stream contains only new classes?

Response to Q_2: Yes. We employ Si-Blurry[2] to generate GCIL tasks for our experiments. As demonstrated in Section 4.4, when the disjoint class ratio $r_{\text{D}}$ is 0, it indicates that the data stream contatians only new classes.

[2] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, Gyeong-Moon Park; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11731-11741

I thank the authors for providing the rebuttal. After reading the rebuttal and other reviewers' comments, all my previous concerns have been adequately addressed. I will keep my positive rating.

Replies to Reviewer qU32

Thank you for your constructive and detailed feedback. We provide detailed responses to your concerns below.

W1. Were the experiments in this paper also conducted in an online scenario?

Response to W_1: Yes, we follow the settings in Si-Blurry [1], which is a online setting with blurry task boundaries.

[1] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, Gyeong-Moon Park; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11731-11741

W2. "The technical improvements on the previous work of ACL are somewhat incremental." and "What are the technical improvements and contributions of this paper over previous ACL approaches such as ACIL and DS-AL?"

Response to W_2 and Q_3: We are very sorry that our delivery of GACL gives a "trivial extension" impression. In fact, the GACL is developed with nontrivial and sophisticated manupulations beyond existing ACL techniques. According to the derivation in lines 457-484, the solution CAN NOT be trivially extended from ACIL or its variants (the derivation takes 2 whole pages in the appenix). We deliberately show the connection to the existing ACL techniques, to better illustrate the weight-invariant property (which has been recognied in ACIL, GKEAL). We will mark additional highlight in this part to avoid such an "incremental" impression.

Q1. the motivation for introducing ACL into GCIL.

Response to Q_1: Many real-world datasets follow GCIL tasks, such as the autonomous driving dataset SODA10M [2], the IoT dataset described in [3], and the real-world e-commerce service dataset described by [4]. GCIL simulates real-world incremental learning, where the distributions of data categories and sizes can be unknown in a given task. ACL techniques emerging as a new CIL branch, have a very appealing weight-invariant property in the CIL community, which works magically esepcially for large-phase CIL tasks. GCIL tasks (e.g., Si-blurry setting) are usually online (very large-phase), where ACL techinique can thrive. Existing ACL methods are specifically designed for CIL tasks only, and CAN NOT process GCIL ones, and the extension is NOT trivial (see derivations in lines 457-484), hence the motivation of GACL.

[2] Han, Jianhua, et al. "SODA10M: A large-scale 2D self/semi-supervised object detection dataset for autonomous driving." arXiv preprint arXiv:2106.11118 (2021).

[3] Wen, Zhenyu, et al. "Fog orchestration for IoT services: issues, challenges and directions." IEEE Internet Computing 21.2 (2017): 16-24.

[4] Bang, Jihwan, et al. "Rainbow memory: Continual learning with a memory of diverse samples." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

Q2. What backbone network was used to obtain the reported results?"

Response to Q_2: As mentioned in Section 4.1, all experiments were conducted using the same pre-trained backbone. We utilize the DeiT-S/16 as our backbone and pre-train the backbone on 611 ImageNet classes after excluding 389 classes that overlap with CIFAR and Tiny-ImageNet to prevent data leakage.

Thanks to the authors for the response. This rebuttal addresses my concerns well. It fully explains the motivation for introducing ACL to address GCIL and the improvements this paper makes to existing ACL methods in response to the GCIL problem. Taking into account the comments of other reviewers and the authors' rebuttal, I decide to increase my rating.

Replies to Reviewer Xb3R

Thank you for your constructive and detailed feedback. We provide detailed responses to your concerns below.

1. Comparing GACL with those methods (especially RanPAC)derives more solid results."

Response to W_1: Thank you for pointing out this important reference as they both adopt closed-form solutions! The RanPAC emphasizes on random project while our GACL focuses on recursive formulation. As RanPAC's codes are for CIL tasks, and our settings (e.g., SI-Blurry) are very different, and they utilize different backbones, we are working on reproducing its results while facing several technical issues. We are sorry that we may miss the rebuttal deadline before providing the comparison. We will try to provide them during the disucssion period. Thanks!

2. Comparison to the joint-traning. Although it is argued that GACL can bridge the gap between continual learning and joint-training, such result is not provided in the experiments.

Response to W_2:

As suggested, we have included the last-phase accuracy of GACL and its joint-training accuracy on the CIFAR-100 dataset with different buffer sizes. Results in the table below show that the accuracy of GACL is indeed the same as that of joint training (the mile differences are caused by quantization).

3. Does GACL run more fast than other (ACL) methods?

Response to Q_1: In general, our GACL gives rather strong performance regarding speed. The table below further records the GACL's training time in seconds compared with EFCIL methods and replay-based methods with the memory size of 2000.

The GACL runs faster than many baselines except a few candidates such as SLDA on three dataset. For SLDA, since only the classifier and autocorrelation memory matrix $\mathbf{R}$ are updated, leading to smaller numbers of trainable parameters compared with those baselines in back-propagation manner.

Replies to Reviewer vKCe

Thank you for your constructive and detailed feedback. We provide detailed responses to your concerns below.

W1. Most of the content on page 4 and page 5 (e.g., Theorem 3.1) in Section 3 is overly similar to existing ACL works. The main difference claimed by the authors, i.e., the ECLG module that corresponds to the exposed-class gain, is a trivial extension of the original ACL."

Response to W_1: Thank you for your valuable comment! We ackownledge that there are certain simlarities between ACIL and the GACL, mostly in the begnining part. However, this part is to generate embedding (e.g., Eq. 2, Eq. 3) and the joint-learning objective function formulation (e.g., Eq.4, Eq. 5), both of which are NOT the contributions of this paper. There are more like a Preliminary contents (e.g., like introducing what is loss function before designing a new one). Our key contributions start from Eq. 6 and Eq. 8 that separate the derivation into an exposed-unexposed pair. The derivation holds the key difference from ACL techniques (see lines 457-484). We apologize that appendixing these items have led to such confusion, and will try to highlight the difference by moving some of them in the main context.

Regarding the concern of the ECLG module being a trivial extension, we respectfully disagree. According to the derivation in lines 457-484, the solution CAN NOT be trivially extended from ACIL or its variants. We deliberately extracted the ECLG to show the connection to the existing ACL techniques, to better illustrate the weight-invariant property (which has been recognied in ACIL, GKEAL). We are very sorry to receive the "trivial extension" impression, and will mark additional highlight in this part to avoid such an impression.

W2. Experiments are needed to compare the performance of GACL with RanPAC.

Response to W_2: Thank you for pointing out this important reference as they both adopt closed-form solutions! The RanPAC emphasizes on random project while our GACL focuses on recursive formulation. As RanPAC's codes are for CIL tasks, and our settings (e.g., SI-Blurry) are very different, and they utilize different backbones, we are working on reproducing its results while facing several technical issues. We are sorry that we may miss the rebuttal deadline before providing the comparison. We will try to provide them during the disucssion period!

W3. Comparison of the memory cost of weights in the buffer layer and other replay-based method.

Response to W_3: GACL operates without the need for prior knowledge regarding the total number of classes. We denote the size of features acquired after the feature extractor and buffer layer as "feature_size," and the number of classes learned at phase $k$ as "n_class." During the continual learning process, the classifier can progressively increase the number of classes in the head and simultaneously augment the dimension of $W_k$ (feature_size $\times$ n_class), while maintaining the size of $R_k$ as feature_size $\times$ feature_size.

Compared with other replay-based methods, such as MVP [1] with a memory size of 500, the memory requirement is 500 $\times$ feature_size $\times$ feature_size $\times$ n_class, which far exceeds the storage needed by GACL. This is without even accounting for model parameters and prompts.

[1] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, Gyeong-Moon Park; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11731-11741

W4. How and why the dimensionality of the random projection affects the performance.

Response to Q_1: Based on Cover's theorem [2], a promising approach to enhancing the separability of features from different domains is to project the features extracted by the pre-trained model into a higher-dimensional space using a non-linear projection. This non-linear, higher-dimensional projection can improve performance by increasing the separability of the features.

[2] T. M. Cover, "Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition," in IEEE Transactions on Electronic Computers, vol. EC-14, no. 3, pp. 326-334, June 1965, doi: 10.1109/PGEC.1965.264137.