Compact Memory for Continual Logistic Regression

We thank the reviewer for their comments. We have tried to address them below and hope to discuss more with the reviewer. If the reviewer's concerns are addressed appropriately, it will be great if they can consider increasing their score. Thanks for reading our work!

Q1. It is better to include larger databases such as imagenet-1k [1] into the experiments to make the results more convincing.

Thanks for the suggestion. We have now included ImageNet-1k where we find our method to again perform better. Following [1], we split the dataset into 10 tasks each with 100 consecutive classes and consider memory size of 0.5% (600 per task), 1% (1200 per task), and 2% (2400 per task) respectively. Below are the results:

Q2. Please clarify what "compact memory" entails. Does it mean representing more sample information within the same buffer size? If so, what is the quantitative gain (e.g., how many times more information) compared to storing raw images?

Yes, you are absolutely right. Quantitative gain can be seen in Fig. 4 and 5, for example, for tinyImagNet, to match the performance of our compact memory of 1% we need around 5% of the data for replay and K-prior. Similar results hold for other datasets.

Q3. In comparisons with Experience Replay (ER) [2], was the buffer size held constant (allowing for more represented sample information), or was the number of stored samples kept constant? This distinction is crucial for assessing the fairness of the comparison.

We set the memory size (buffer size) per task and increase it whenever the model learns a new task. For example, if a continual learning experiment consists of 5 tasks and the memory size is set to 10 per task, the total memory size increases as [10,20,30,40]. This configuration of the memory size is applied equally to both ER and our method.

Q4. Since Nearest Class Mean (NCM) classifiers are effective and widely used for continual learning with frozen feature extractors [3,4,5,6], please justify why storing compact memory representations of samples (features) is preferred over maintaining single class prototypes, especially if NCM yields comparable or superior performance.

Though NCM has been empirically effective for continual learning, to the best of our knowledge, the rationale for how NCM yields superior performance, is not entirely clear. Our work follows the more usual trend to represent the past features in a principled way that aims to recover the gradient of past task loss using a K-prior regularizer, and this could also be considered complimentary to NCM. We are happy to learn more about reviewers point of view and add discussion on such methods in the paper.

Q5. Equation (1) seems problematic. Additionally, some equations (7, 8, 9) are missing punctuation (e.g., periods or commas) at their conclusion.

Thanks for pointing this out. We will fix these mistakes in the next versions.

Q1. Why is SVD worse in Figure 2 (a)? I might be missing something, but SVD should result in the optimal low-rank representation of the current task's data combined with the compact memory. How can PPCA be better for a small number of data points?

Thanks for the question. This is most likely due to the piling up of numerical errors in SVD. This can be clearly seen by looking at the averaged test accuracy over Task $1$ to $t$ for $t=1,..,5$ during training. The following table shows that SVD performance gets suddenly worse after Task $3$; for example, the column of Task $3$ shows the averaged test accuracy over Task $1$ to $3$. This holds for all memory sizes 10, 20, 40, and 160, especially when the smaller memory is used.

The EM algorithm is more stable because it relies on matrix inversion where an identity matrix is added (see update of $S$ in Algorithm 1). It is possible that these issues will be fixed by using a better numerical algorithm for SVD or also by tuning the delta in Eq. 3, although the latter is tricky for the sequential setting in continual learning.

Q2. How does the cost of PPCA EM compare to other numerical linear algebra methods to compute top singular vectors and values, e.g. randomized SVD? Can you add more discussion around the choice of PPCA EM and its computational cost?

computation cost of PPCA EM compared to top-K SVD: In our problem, SVD is applied to the concatenated matrix $[ \Phi_{t+1} , U_t ] \in R^\{P \times (T+K)\}$ with $P$ being the number of parameters, $T$ being the size of new data, and $K$ being the size of memory $U_t$. The computational complexity of SVD is $\mathcal{O}( (T+K) P K )$ for top-$K$ SVD.

The computational cost of the EM algorithm is $\mathcal{O}(K^{3} )$ per iteration and is dominated by inverse of $K$-size square matrices $S$ and $(S^{-1}+ MM^{\top})$; see Algorithm 1. Since $K\ll T$ and $P$ both, this is faster if the number of iterations is reasonable.

In our experiments, sometimes EM took a long number of iterations to converge in which cases the cost saving compared to SVD are not obtained, but the method was still more numerically stable. It is also true that there might be other versions of SVD that could compete with the EM but for our purpose EM worked well and we did not try to find another alternative and only compared to the standard SVD.

Discussion regarding choice of PPCA EM: Thanks for the suggestion. We will highlight the numerically stability under accumulation of error as new tasks as the main reason for choosing EM over SVD.

Q3. How is convergence of EM determined? How many steps of EM are run in practice?

We keep track the norm of $|| U_{t+1} U_{t+1}^T - U_{t+1}^{old} (U_{t+1}^{old})^T ||$ and stop when it is small enough. However, we sometimes found EM to use maximum iterations (see Appendix for the max iterations for each experiment).

Q4. Is experience replay defined anywhere?

Yes, the equation in the right hand side of Eq. 2 is experience replay.

Q5. Are there any other relevant works on regression besides Khan and Swaroop [2021]? What about methods for selecting subsets of training data and other works using pseudo-data for continual learning?....... Comparing baselines with non-randomly chosen subsets of the training data would be more informative.

Regression tasks: to the best of our knowledge, we have not found relevant work that takes a similar approach to ours, as most continual learning research has focused on image classification task.

Existing sample selection methods: [1], one of the representative works, uses gradients to maintain the gradient diversity within the memory. [2] further devises a gradient-based scoring function that also considers adaptation to new tasks. [3] chooses memory samples that are most interfered with by the current task. [4] uses an information-theoretic metric within a Bayesian framework to balance information gain and learnability based on the estimated uncertainty. [5] samples memory per class using prototype vectors defined in the feature space.

In contrast to these existing approaches, our method directly extracts memory without requiring labels and therefore leads to a more compact representation of past data. We will include a discussion in the main text to clarify these points and add citations to relevant works.

References:

[1] Gradient-based Sample Selection for Online Continual Learning – NeurIPS 2019

[2] Online Coreset Selection for Rehearsal-based Continual Learning – ICLR 2022

[3] Online Continual Learning with Maximally Interfered Retrieval – NeurIPS 2019

[4] Information-theoretic Online Memory Selection for Continual Learning – ICLR 2022

[5] iCaRL: Incremental Classifier and Representation Learning – CVPR 2017

Suggestion: we have added a comparison to the method of [1] Aljundi (2019). Please refer to our response to Question 3 (Q3) from Reviewer CdwX.

Thank you for your additional comments and for updating your evaluation of our work. If you intended to change the score but did not, we kindly ask that you update it. We will incorporate your additional comments into the revised manuscript, including wall-clock time of the ImageNet-1k experiment.

We thank the reviewer for their encouraging comments.

Q1. Why does SVD outperform EM at large data sizes (e.g., in Figure 2(a) at 320 samples)? A brief comment on this would help clarify the method’s relative strengths.

Thanks for the question. The difference in performance is mostly due to numerical issues (see our response of Question 1 (Q1) to Reviewer zWmk). For larger memory, SVD can get close to near perfect reconstruction and have less numerical issues.

Thanks also for pointing out all the mistakes. We will fix them in the next version of our paper.

Q1: What is the computational complexity of each step of the algorithm of PPCA_EM_for_logistic? What is the complexity improvement compared to using PCA? Could you please use some specific examples (e.g., for the last layer of NN, there is a 2048 -> 10 classification) to illustrate how significant the complexity improvement is?

PCA is based on SVD which is costly compared to the EM iterations (both aim to solve Eq. 7). In our problem, SVD is applied to the concatenated matrix $[ \Phi_{t+1} , U_t ] \in R^\{P \times (T+K)\}$ with $P$ being the number of parameters, $T$ being the size of new data, and $K$ being the size of memory $U_t$; for example, we use $T=12000$, $P=784$, and memory size $K\in \\{10,20,30,40\\}$ (10 memory per task) in our experiment of Figure 2.

The computational complexity of SVD is $\mathcal{O}( (T+K) P^{2} )$ for full SVD and $\mathcal{O}( (T+K) P K )$ for top-$K$ SVD.

The computational cost of the EM algorithm is $\mathcal{O}(K^{3} )$ per iteration and is dominated by inverse of $K$-size square matrices $S$ and $(S^{-1}+ MM^{\top})$; see Algorithm 1. Since $K\ll T$ and $P$ both, this is faster if the number of iterations is reasonable.

We note that the main reason to use EM is not always the speed but mainly its numerical stability compared to the version of SVD we used. It is possible to use other variants of SVD that can handle the numerical errors piling up as the new tasks are observed.

Q2: Is there any time analysis or ablation study on the time cost and performance comparison of your PPCA-based method and the one with PCA? Is experiment necessary?

The performance of SVD was much worse (see Fig 2) which is why we did not conduct the time-analysis but we are happy to add it in the final version of the paper. EM was generally faster but not always, but it was definitely more numerically stable which is the reason why it worked better.

Q3: In experiments, I see that you compare with K-prior and standard replay with random sample selection. Could you please also compare with other sample selection methods, like [Aljundi, 2019]?

Thanks for the suggestion. Below we report some comparison to Aljundi (2019)‘s method called Gradient-based Sample Selection (GSS) and find it to be worse than our method. We adapt the implementation provided by the authors into our setup of Figure 5. We consider several memory sizes from 1 to 20% and compared it to Experience Replay using GSS (ER-GSS) and Random sample selection (ER-RN). Additionally, we compare the ER using class-wise random sample selection (ER-RN-C) that balances the label within the memory buffer.

On CIFAR-10, the performance of GSS is slightly worse than that of random selection, but on CIFAR-100, it performs significantly worse, especially when the memory size is small (1% or 2%).

We hypothesize that this is because GSS fails to preserve label balance within the memory buffer because GSS is known to select gradient-diverse samples. This issue is likely exacerbated when each task contains a large number of classes (e.g., 2 classes per task for CIFAR-10 vs. 10 classes per task for CIFAR-100). Indeed, the comparison between ER-RN-C and ER-RN on CIFAR-100 shows that the performance of ER-RN, which also does not consider label balance, was worse than that of ER-RN-C with 1% and 2% memory. This result supports our hypothesis on GSS’ performance degradation.

Q4: The algorithm is tested on datasets including TinyImageNet, with a similar number of images compared to MNIST and CIFAR. What is the reason for setting the experiment scale to this level? Is it possible to run it on the standard ImageNet?

Thanks again for the suggestion. We used TinyImageNet-200 as the larger dataset, but now have also done experiments on ImageNet-1k where we find our method to again perform better. Following [1], we split the dataset into 10 tasks each with 100 consecutive classes and consider memory size of 0.5% (600 per task), 1% (1200 per task), and 2% (2400 per task) respectively. Below are the results:

We hope that our explanation and new experiments address your concerns. We are happy to further discuss these with you.

[1] Improving Continual Learning by Accurate Gradient Reconstructions of the Past - TMLR 23

Thank the author for their rebuttal. My concerns are addressed. I will increase my score from 4 to 5 and increase my confidence from 3 to 4.

I agree with most of Reviewer zWmk's suggestions. In particular, you might want to cite some theoretical CL work to justify the Hessian matching, and I leave some suggested papers:

[1] Evron et al. How catastrophic can catastrophic forgetting be in linear regression? COLT'22

[2] Li et al. Fixed Design Analysis of Regularization-Based Continual Learning. CoLLAs'23

[3] Evron et al. Continual Learning in Linear Classification on Separable Data. ICML'23

[4] Li et al. Memory-statistics tradeoff in continual learning with structural regularization. arXiv:2504.04039.