Response to Reviewer XB6P

Thank you for the thorough review. We provide more detailed response below. Hope this can help you with your concerns.

W1&L1: The formula No.4 seems to be wrong.

Thank you for pointing out the typo. We revise $ϕ(X)Y$ to $ϕ(X)^⊤Y$ in equation 4.

L2: Why does the formula No.4 show the optimal solution? Is W too idealized and may even overfit the data of the current batch?

We appologize that our way of delivering the analytical solution can be misleading. In fact, Eq. 4 is the optimal LS solution to loss function in Eq. 3 in the non-CIL sense (like a preliminary in the case with all the data). That is, $W$ here does NOT indicates the weight for any batch, but for the one computed using all data.

On the other hand, Eq. 5 indicates the same loss function as that of Eq. 3, but in the sense of breaking the dataset into $k$ segments (hence the "$1:k$" symbols). This leads to the least-squares solution in Eq. 7 (the same as Eq. 4 but in the sense of breaking the dataset into $k$ segments). Here the optimal solution $W$ does not guarantee Eq. 5 being 0. The "optimal" here indicates that the obtained $W$ allows the objective function in Eq. 5 to be the smallest (not necessarily 0 though).

The use of Eq. 5 (with many "$1:k$") prepares us for the subsequent recursive derivations (i.e., from "$1:k-1$" to "$1:k$"). Based on this, we are able to derive the F-OAL mainly indicated by Eq. 10 and Eq. 12 in recursive form. Note that the $W$ recursively computed using Eq. 10 or Eq. 12, obtains the identical result to that of Eq. 7. That is, the update formula in Eq. 10 or Eq. 12 considers both the current-batch data (e.g., $k$) and the past-batch data (e.g., $1:k-1$). In a sense, the F-OAL does no suffer from catastrophic forgetting (CF) at all!

L3: Can recursive least square exceed the back-propagation?

Our F-OAL beats BP because it does not suffer from CF (see analysis in L2). According to [1], using BP to update the model is the reason why CF happens, which leads to incomplete feature learning and recency bias [2]. Existing baselines are still based on BP and manage to alleviate CF. We redefine the OCIL into a recursive learning problem to avoid BP. According to [3], the recursive update paradigm could obtain results that are identical to their joint-learning counterparts.

To give further proof, we provide a quick experiment on MNIST to show that BP suffers from CF, while recursive methods do not. A two-layer MLP is trained on first 5 classes of MNIST (i.e., number 0 to 4) via BP and we test it on these 5 classes. Then the MLP is trained on rest of the 5 classes (i.e., number 5 to 9) via BP incrementally for each class per phase and we test it on the old 5 classes (i.e., number 0 to 4). We apply the identical experiment on our F-OAL using the same MLP. The results are reported below. Please check the source code of the quick experiment in General Response.

[1] Online continual learning through mutual information maximization, ICML 2022

[2] Learning a unified classifier incrementally via rebalancing, CVPR 2019

[3] Blockwise recursive Moore–Penrose inverse for network learning. IEEE TMSC-S, 2021

L4: Is the optimization process recursive on all batches of data? Why is the recursive method more efficient?

Yes, the optimization process is on all batches of data. For space efficiency, our method is forward-only and exemplar-free, eliminating the need for gradients and additional memory buffers, thereby significantly reducing the GPU footprint.

For computational efficiency, F-OAL does not have backward pass, resulting in a faster training. Accroding to [1], in BP, the backward pass spends 70% of the entire time.

[1] Decoupled Parallel Backpropagation with Convergence Guarantee. ICML 2018

L5: The innovation of this paper is insufficient. This method faces many disadvantages, such as overfitting.

Our main contributions are 1) pinpoint BP as the main cause of CF, 2) introducing F-OAL, the recursive method instead of BP that well handles OCIL problem, and 3) fusion module and smoothed projection that enhance the performance of F-OAL.

Regarding the overfitting disadvantage, we must respectably disagree. The F-OAL is mainly linear regression, which usually invites "under-fitting" instead of "over-fitting" in nature. In addition, we avoid possible over-fitting (due to small data), we have introduced an $L_{2}$ regularization in Eq. 5. However, the F-OAL needs a fixed backbone. This is an existing disadvantage that will be discussed.

L6：Many models that appeared in the comparative experiments did not have annotated references, such as DVC in Table 1.

Thank you for pointing out the issue. We will add citations for each method in Table 1 and also review the rest of the paper to ensure there are no other similar problems. The table 1 will be revised in the following form with the right citation:

[1] Not just selection, but exploration: Online class-incremental continual learning via dual view consistency. In CVPR 2022.

Based on these additional results and clarifications, we hope you could consider increasing your score in support of this work. If not, could you kindly let us know what additionally needs to be done in your assessment to make this work ready for publication?

Response to Reviewer UyXG

Thank you for your valuable time in reviewing. We provide detailed information for your concerns below

W1: Give more precise definition for all notations, such as their dimensionality.

Thank you for your suggestion. We have included a following notation table.

W2: There are some confusing notations that can be improved.

Thank you for your comments. We will revise the confusing notions as instructed. For instance, we will use $k$ to represent the task index and $n$ to represent the batch index respectively to avoid confusion.

W3: There are some writing issues like typos.

Thank you for pointing out these writing issues! We have corrected "class" to "classes" and "plan" to "plans" in line 17, changed "a" to "an" in line 117, and revised $ϕ(X)Y$ to $ϕ(X)^{T}Y$ in equation 4. Additionally, we will carefully review the paper to correct any other typos.

Q1: Does the time recorded in Table 2 for F-OAL include the ViT feature extraction time?

Yes, we have documented the entire training process, from the image entering the model to the completion of the model updates.

L1: The limitations of the method are not discussed.

Thank you for the suggestion. The major limitation is that, our method relies on well pre-trained backbones such as ViT and ResNet pre-trained on ImageNet. However, there are many open-source pre-trained backbones available within the deep learning community, which are relatively easy to obtain. Additionally, leveraging pre-trained models for fine-tuning downstream tasks has become a mainstream approach. Hence, although the need for backbones is an existing limitation of F-OAL, it is feasible and in line with mainstream.

Response to Reviewer nHHU

Thank you for your positive reviews and helpful suggestions. We provide detailed responses to your concerns below.

W1: Abuse the notion of "data privacy".

Thank you for raising this important concern. We agree that the use of “data privacy” is less appropriate in this paper. We will remove the use of "data privacy" and stick to “exemplar-free” (i.e., data need not be stored for replay in production).

Q1: Ablation study results with frozen are obvious and unnecessary.

Thank you for the suggestion. We will remove this unnecessary experiment and add more important ablation experiments based on your question.

Q2: Simple ablation that could've helped would be to compare their block-averaging + smoothed projection approach.

Thank you for pointing out the missing items in our ablation study. We have included a new set of experiments as follows according to your suggestions. The results are shown below.

We conduct ablation study to prove the contribution of block-averaging fusion module and the smoothed projection module. The average accuracies are reported. As the table shows, without two modules, the results of F-OAL are already competitive. The two modules further improve F-OAL's performance, especially on fine-grained datasets (e.g., DTD, FGVCAircraft and Country211).

Q3: Is there any work or experiment they can point to that suggests feature fusion works?

The idea of feature fusion is inspired by DenseNet [1], which suggests that hidden features are still helpful for generating more representative final output. This approach can be considered equivalent to an ensemble of multiple backbones, which can provide feature diversity for F-OAL with a frozen encoder.

[1] Densely Connected Convolutional Networks, CVPR 2017

Replies to Reviewer Pgch

Thank you for your constructive and detailed feedbacks. We provide detailed responses to your concerns below.

W1: Dependence on the quality of the pre-trained encoder.

Thank you for the suggestion. Indeed, our method relies on well pre-trained backbones such as ViT and ResNet pre-trained on ImageNet. We acknowledge this as a limitation and will discussion this limitation in the manuscript so readers can fully understand our technique.

On the other hand, there are many open-source pre-trained backbones available within the deep learning community, which are relatively easy to obtain. Additionally, leveraging pre-trained models for fine-tuning downstream tasks has become a mainstream approach. Hence, although the need for pre-trained backbones is an existing limitation of F-OAL, it is feasible and in line with mainstream.

W2: Challenges that might arise in different data scenarios.

Thank you for the suggestion. Indeed, we focus on coarse-grained data scenarios, such as CIFAR-100, Tiny ImageNet, and Core50, as well as fine-grained data scenarios, including DTD, FGVC Aircraft, and Country211. However, there are other data scenarios, such as long-tail distributions, which we have not addressed in this work. We shall include this discussion in the manuscript.

W3: Including some recent exemplar-free works.

Thank you for pointing out these baselines. However, they seem to be specially designed for non-pre-trained ResNet, and changing their backbones will compromise their performance. Therefore, we have included several alternatives, i.e., EASE [1], LAE [2] and SLCA [3], which are designed with pre-trained ViT and are the SOTA exemplar-free CIL approaches. Please refer to the uploaded PDF in the general response.

In addition, the mentioned baselines [4-7] are good references, and we shall include them in our literature review to complete our review.

[1] Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning, CVPR 2024

[2] A Unified Continual Learning Framework with General Parameter-Efficient Tuning, ICCV 2023

[3] SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model, ICCV 2023

[4] Divide and not forget: Ensemble of selectively trained experts in continual learning, ICLR 2024

[5] R-dfcil: Relation-guided representation learning for data-free class incremental learning, ECCV 2022

[6] Self-sustaining representation expansion for non-exemplar class-incremental learning, CVPR 2022

[7] DiffClass: Diffusion-Based Class Incremental Learning, ECCV 2024

L2: A complexity analysis.

Thank you for the suggestion, we have included the complexity analysis as follows.

In terms of space complexity, our trainable parameters are only R and W. The R matrix is of size D × D , where D is the output dimension of the encoder. In our paper, the encoder output dimension is 1000. Therefore, according to Equation 9, the size of the R matrix is 1000 × 1000 . The W matrix has dimensions of C × D , where C is the number of classes in the target dataset. For example, with CIFAR100, its size is 100 × 1000 . The total number of trainable parameters is relatively small and does not require gradients. This results in our method using less than 2GB of memory, as shown in Figure 2.

In terms of computational complexity, we denote the batch size as S, encoder’s output size as D, and class number as C. Therefore, the dimensions of $X$, $Y$, $R$ and $W$ are S×D, S×C, D×D and C×D, respectively. Thus, the calculation is shown below:

The computational complexity for updating R is dominated by the matrix multiplications, thus: $\text{max}${$\mathcal{O}(SDC), \mathcal{O}(SC), \mathcal{O}(SDC), \mathcal{O}(D^2C), \mathcal{O}(DC)$}≈ $\text{max}${$\mathcal{O}(SDC), \mathcal{O}(D^2C)$}

The computational complexity for updating W is dominated by the matrix multiplications and the matrix inversion: $\text{max}${$\mathcal{O}(SD^2), \mathcal{O}(S^2), \mathcal{O}(S^3), \mathcal{O}(DS^2), \mathcal{O}(D^2S), \mathcal{O}(D^2)$} ≈ $\text{max}${$\mathcal{O}(S^3), \mathcal{O}(DS^2), \mathcal{O}(D^2S)$}

In the OCIL setting, the batch size is relatively smaller. Therefore, the overall computational complexity is primally controlled by $D$.

L3: Overhead comparison.

In terms space overhead, compared to the conventional backbone + classifier structure, F-OAL introduces an additional linear projection to control the output dimension D of the encoder, and a matrix $R$ , where only $R$ is trainable. According to Equation 9, the dimension of $R$ remains a fixed size of D × D. Other methods require more extra space. For instance, LwF employs knowledge distillation, necessitating the storage of additional models, while replay-based methods require extra storage to retain historical samples. In contrast, the overhead introduced by F-OAL, consisting of an additional matrix and a linear layer, is smaller.

In terms of time overhead, our method primarily consists of a forward pass and matrix multiplication, which is determined by output dimension of encoder. By changing the output dimension of encoder, we can balance the accuracy and time. According to [8], the backward pass in backpropagation (forward pass + backward pass) accounts for 70% of the time. Therefore, our method's time overhead is also relatively small.

[8] Decoupled Parallel Backpropagation with Convergence Guarantee. ICML 2018

Based on these additional results and clarifications, we hope you could consider increasing your score in support of this work. If not, could you kindly let us know what additionally needs to be done in your assessment to make this work ready for publication?