Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning

We would like to extend our sincere gratitude to the reviewer for the time and effort invested in reviewing our paper. We are also grateful to read that the reviewer thinks our work is significant and novel in an underexplored area with comprehensive and well-conducted experiments and clearly written paper. Here, we give our response to the points the reviewer mentioned.

 

It appears that S-EWC does not have too much of a gap with UPGD judging from Figure 7: it entirely avoids catastrophic forgetting, and the only setting where it loses plasticity where UPGD does not is on MNIST

While S-EWC is effective in catastrophic forgetting evaluation, it suffers from loss of plasticity, which was the aspect we evaluated in the input-permuted MNIST task. Only UPGD addressed both loss of plasticity and catastrophic forgetting, which is shown by its performance on both evaluations.

 

S-MAS outperforms UPGD on miniImagenet at the end of training, and does not have a large gap overall

While we do not expect a new method to outperform in every single task, UPGD still achieves higher average accuracy in the first half of the experiment. It is desirable in streaming learning where the online accuracy at each time step is more important than the accuracy only at the end of training.

 

The ablation of figure 8 checks the contribution of each component of UPGD sequentially as they are added to regular SGD. Ideally, the ablation would study how each component affects UPGD when they are individually removed (e.g. UPGD without WP).

We conducted the requested ablation. Please see this figure [link]. The requested ablation bolsters the contribution of utility gating even more. Using utility gating on top of sgd makes sgd maintain its performance instead of dropping on input-permuted MNIST and makes sgd improve its performance continually on label-permuted problems. The role of weights decay and weight perturbation is not significant in label-permuted problems but including both with utilty gating improves performance and plasticity on input-permuted MNIST.

 

Minor comments Figure 7 is referred to before Figure 6; ideally, their order would be swapped. I see in Section 4 that the results are averaged over 20 trials, but the meaning of the error margins in some of the figures is not made clear (e.g. figure 2). I would also suggest increasing the number of trials to smooth out the curves if possible.

Thank you for pointing out the order of mention for Figures 6 and 7; we will make the necessary amendments. The error margins represent standard error. Our results are averaged over 20 trials. We use non-overlapping intervals to average over. Therefore, our plots can have variabilities even though we have a large number of trials. We don’t average by using an exponential moving average, which will smoothen out the curves since we are interested in what each point in the plot means. For example, each point in Figure 3 represents the average performance over a single task. Smoothing over time wouldn’t give the same meaning.

 

Answers the Questions:

Is it possible to show theoretical performance guarantees for UPGD?

We think this would be interesting direction that is left for future work.

How much more significantly does UPGD improve upon baselines S-EWC and S-MAS?

Based on our results, we can see that UPGD outperforms S-EWC in all experiments and outperforms S-MAS in 3 out of 4 experiments.

How does UPGD-W perform with WP and WD removed individually?

Please see this figure [link], which includes the requested ablation.

We would like to extend our sincere gratitude to the reviewer for the time and effort invested in reviewing our paper. We are also grateful to read that the reviewer thinks our method is straightforward and practical, our metrics are reasonable, our paper is well-written, and our method has solid performance. Here, we give our response to the points the reviewer mentioned.

 

Utility of an individual weight is almost always nothing...it's more about combinations of weights working together...and the proposed utility doesn't measure that

We agree with the reviewer that a better utility definition would include the interaction between weights (as we pointed out in Appendix M). However, including such interactions makes the method intractable, as the reviewer mentioned. Most methods addressing forgetting (e.g., EWC) use individual weight measures, and yet they are effective. While we are interested in learning about the kind of representations learned by each method, we believe this merits a dedicated paper to address fully. From our experiment, however, we found that UPGD improves performance over SGD even in the stationary case (see Figure 9 in Appendix F), likely due to finding better representations.

 

the proposed method is just another weight consolidation method, and it's a bit hard to believe it beats Elastic Weight Consolidation.

While there are strong connections and similarities between utility-gating and regularization-based methods, our approach differs from Elastic Weight Consolidation (EWC) in two major ways: 1) In EWC, a regularization term is essential, while our method has no such term., and 2) our proposed utility measure is different than EWC's squared gradient metric. The results from Figure 2 show the empirical superiority of our utility measure over the squared gradients, which partially explains the performance difference. More future theoretical work is needed to link between these two classes of methods.

 

It am not 100% sure that the proposed method doesn't favour plasticity over forgetting nor that the forgetting evaluation isn't biased towards methods that favour plasticity.

In our catastrophic forgetting (CF) evaluation, methods that only address CF can achieve high performance (e.g., S-MAS) since the problem is designed to favor methods that keep useful representations and improve on them continually. We do not think our CF evaluation is biased towards plasticity. Plasticity-inducing methods (e.g., Shrink and Perturb) achieve low performance in CF evaluation.

 

Answers:

Just because utility of a given weight is, say, low for the current sample, it doesn't mean it's low for previous samples.

The utility measure we use is based on an exponential moving average of the sample utilities, as we explain in the last sentence of the second paragraph of section 4.2. Thus, it does not suffer from the issue the reviewer mentioned about sample instantaneous utilities.

how 4.3 measures catastrophic forgetting. Permuting labels of CIFAR10 with the new tasks suggests to me that it's all about plasticity.

Section 4.3 indeed evaluates catastrophic forgetting, as previously learned representations transfer and remain useful from one task to another. Hence, methods that forget previously learned representations should suffer in performance. Our results also confirm that methods that protect useful representations and improve them with more data achieve higher performance (e.g., S-MAS). A method that does not protect useful representations (e.g., SGD) forgets them by overwriting useful information and, therefore, does not improve its performance continually.

Why are the accuracy results of training so poor?

In our streaming non-stationary problems, achieving state-of-the-art results as in stationary settings is unexpected due to their incomparability. Please see Figure 2b (rightmost of the three) in [1], where accuracy decreases with increased permutation frequency: from 99% (permuting every 1M samples) to about 88% (every 10K). Our experiment, with permutations every 5K samples, achieves a reasonable best accuracy of around 78%, aligning with this trend.

does the proposed method prevent forgetting at the high end?

In streaming learning, the evaluation is done on-the-fly at every time step, so maximizing the online performance (accuracy) at every step by reducing forgetting is desired. It’s a different problem setting from the one the reviewer described, where the final performance is what is important.

does the per batch normalisation of utility make the performance of the method variable with different mini-batch size settings?

In our paper, we used a batch size of 1 (streaming case). Our method can also work with mini-batches where batch normalization might help.

 

References:

[1]. Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Sutton, R. S., & Mahmood, A. R. (2023). Loss of Plasticity in Deep Continual Learning.

We would like to extend our sincere gratitude to the reviewer for the time and effort invested in reviewing our paper. We are also grateful to read that the reviewer thinks our paper is well-written and that our empirical evaluation verifies the central claim of the paper. Here, we give our response to the points the reviewer mentioned.

 

I do not think this paper makes a significant contribution. The methodology is incremental in that it combines two well-known ideas (perturbed gradient descent + keeping active neurons unchanged).

We respectfully disagree with the reviewer. First, our method does not work by keeping active neurons unchanged. Our method uses a utility-gating mechanism to protect useful weights from change and perturb less useful ones. This idea is compatible with weight perturbation and weight decay, which makes the method stronger. Second, our novel utility-gating mechanism presents an instance of a new class of methods that has not been introduced before. We kindly ask the reviewer to refer us to works to which UPGD is considered incremental.

 

It is tested on very toy setups.

Our tasks are widely used in continual learning for evaluating forgetting and plasticity (see [1,2,3,4]), where input-permuted tasks were used for evaluating plasticity and output-permuted tasks were used for evaluating forgetting. We certainly acknowledge the reviewer’s valuable suggestion for real-world setups, such as webcam data. However, such benchmarks are unavailable, especially for streaming learning, due to its complexity, and requires a separate dedicated paper. As you recognize that our experiments verify the paper’s central claim and reviewer KmBd highlights their scale, solidity, and comprehensiveness, we believe they sufficiently demonstrate the core contribution of our work.

 

Third, theoretical properties/implications of the method should be carefully examined.

We provided theoretical work in the Appendix. Please refer to Appendix A for convergence analysis, Appendix B for how utilities can be propagated in the network, and Appendix D for the connection between weight and feature utilities.

Regarding the comment on Taylor expansion, our first and second-order Taylor approximations require ignoring higher-order terms, which is common and valid for our case when weights are proper fractions, making them sufficiently small. Our empirical evaluation shows the effectiveness of this approximation in continual learning. For future work, one can consider a different utility definition based on the change in loss from sample to sample, which depends on the difference between the values of the weight from two consecutive time steps, making the approximation more accurate (see [5] and Appendix M).

 

gradient descent is no longer steepest descent but some approximation to it. Investigating why it works is important.

In Appendix A, we provided convergence analysis for our method, where we showed that our method converges to a stationary point, similar to other methods (e.g., SGD). Our empirical evaluation in Figure 9 shows that UPGD also outperforms SGD in stationary problems.

 

Answers to the Questions:

First, we thank the reviewer for the suggestions. We will incorporate them in our revision.

reference needed for this claim.

Thank you for letting us know. We will add these references in the paper revision (see [6,7]).

What does "a Hessian diagonal approximation in linear complexity" mean? Linear in the number of parameters?

This is correct: linear in the number of parameters.

Does "each learner is trained for 1M samples, one sample each time step" mean gradient descent using one sample only? Is this realistic?

That is correct. The update is made based on one sample at a time. This is often used as the more challenging constraint of streaming learning [e.g., 8]

 

References:
[1]. Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., & Dabney, W. (2023). Understanding plasticity in neural networks.
[2]. Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Sutton, R. S., & Mahmood, A. R. (2023). Maintaining Plasticity in Deep Continual Learning.
[3]. He, X., Sygnowski, J., Galashov, A., Rusu, A. A., Teh, Y. W., & Pascanu, R. (2019). Task agnostic continual learning via meta learning.
[4]. Kumar, S., Marklund, H., & Van Roy, B. (2023). Maintaining plasticity via regenerative regularization.
[5]. Zenke, F., Poole, B., & Ganguli, S. (2017, July). Continual learning through synaptic intelligence.
[6]. Hayes, T. L., Kanan, C. (2022). Online continual learning for embedded devices.
[7]. Van de Ven, G. M., Siegelmann, H. T., & Tolias, A. S. (2020). Brain-inspired replay for continual learning with artificial neural networks.
[8] Sahoo, D., Pham, Q., Lu, J., & Hoi, S. C. (2017). Online deep learning: Learning deep neural networks on the fly.

We would like to extend our sincere gratitude to the reviewer for the time and effort invested in reviewing our paper. We are also grateful to read that the reviewer thinks our paper tackles an underexplored but challenging problem and our paper is well-written with sound mathematical derivations. We are also happy that the reviewer appreciated our new plasticity metric contribution. Here, we give our response to the points the reviewer mentioned.

 

The authors claim that by adding the perturbation we are making the unimportant weights more plastic however, I am not really convinced

Noise injection has been shown in the literature of plasticity as a way of introducing plasticity (see [1]). In Figure 3 of our work, S&P maintains its plasticity compared to SGD with weight decay, mainly because of noise perturbation (which is unscaled). While noise perturbation alone might not be effective (see PGD in the same figure), its role is pronounced when used with weight decay (S&P) or when used on a per-feature level [1]. We will add a sentence to make this clearer in our discussion of the method.

 

all of proposed perturbation are getting drawn from the same standard normal distribution...adding an unscaled random perturbation to all of the weights we are ignoring this scale difference

First, we note that the standard deviation of the noise perturbation is a hyperparameter that can be tuned, which is effectively a scaling of the perturbation (see Table 2 and Table 3 for the values used). Second, we agree that noise perturbation can also use explicit layer-wise scaling. However, our utility scaling takes into account the importance of weight, meaning that weights in the earlier layers generally have higher utility and, therefore, perturbed less. Finally, we acknowledge that there are always ways to improve any method (we discussed some of them in Appendix M). Yet, our simple design was found to be very effective compared to other methods.

 

It is also necessary to have an additional ablation study, investigating the role of having and not having the perturbations in the update rule.

We conducted the requested ablation. Please see this figure [link]. The requested ablation bolsters the contribution of utility gating even more. Using utility gating on top of sgd makes sgd maintain its performance instead of dropping on input-permuted MNIST and makes sgd improve its performance continually on label-permuted problems.

 

It is not as obvious as the authors' claim that after the permutation of the input space the previously learned representations are not relevant anymore

We agree with the reviewer on the comment about the phrase “previously learned representations are not relevant anymore”. We meant to use the word "features" instead of "representations". We will correct it in our revision.

 

Including more diverse experiments

The problems we use are extensively adopted by the continual learning community to evaluate forgetting or plasticity in a controlled manner. We tried split-based tasks, as the reviewer kindly suggested, with CIFAR 100, but we found that these problems become very easy to the extent that all methods achieve an online accuracy of above 98% because, in streaming learning with long sequences, tasks have to re-occur which makes the problem easy (see [2]). Therefore, we chose label-permuted tasks since task re-occurrence probability becomes very small. Finally, due to the complexity and significance of designing realistic benchmarks suitable for streaming learning, we believe it merits a dedicated paper to address fully.

 

I believe the visualization needs several improvements.

We thank the reviewer of this note. We will improve visualization in our revision.

 

Answers:

A1: The weight perturbations in UPGD are still scaled by utility. They are just scaled by the same utility the gradients are scaled by. Future work can study or improve the relative scaling between weight perturbation and gradient information.

A2: In streaming learning, the learner receives a data stream and outputs a stream of predictions that are either correct (sample accuracy of 1) or incorrect (sample accuracy of 0). The average accuracy on a certain task is the average of these sample accuracies, which is also the percentage of correct predictions since the sample accuracies are either zero or one. We show the online average accuracy on each of the sequential tasks we have, so it is an average over each task separately.

References:
[1]. Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Sutton, R. S., & Mahmood, A. R. (2023). Loss of Plasticity in Deep Continual Learning.
[2] Lesort, T., Ostapenko, O., Rodriguez, P., Arefin, M. R., Misra, D., Charlin, L., & Rish, I. (2023). Challenging Common Assumptions about Catastrophic Forgetting and Knowledge Accumulation.