STAR: Stability-Inducing Weight Perturbation for Continual Learning

Thank you for your constructive feedback and for appreciating our proposed solution, evaluation, and the clarity of presentation. Regarding your feedback:

Figure 3: We thank the reviewer for the suggestion on measuring the KL Divergence after a gradient ascent step. We think that this does indeed further verify our claims, and we have added the modified experiment to the revised pdf (Fig. 3). We take 5 maximizing gradient ascent steps, as detailed in sec 4.3, and plot the KL Divergence. For comparison, we also plot the KL Divergence without applying any maximizing steps. We observe that STAR reduces the KL divergence in both scenarios.

Smoothness of loss landscape: Figures 1 & 2 are indeed hypothetical; we use them to illustrate the intuition behind STAR and to enhance the clarity of the method description, as the reviewer kindly noted. We thank the reviewer for the suggestion of empirically measuring the smoothness of the local loss landscape. We will include it in the camera-ready version.

Mathematical derivation: We thank the reviewer for pointing out the confusion regarding the linearity approximation. “It is important to note that this is an approximation, and that $\mathcal{L}_{FG}$ is, in practice,non-linear. Otherwise, the minimizing gradient step would be equal to the negative of the maximizing gradient step in eq. 10”. We have added the quoted statement to the revised manuscript for further clarification.

Why focus on stability: The plasticity-stability dilemma is a well-established challenge in continual learning [A]. The standard neural network loss induces plasticity. Without measures to preserve stability, neural networks tend to be excessively plastic when introduced to new classes or tasks, due to the nature of the learning objective, and lack the stability necessary to prevent forgetting previous knowledge. Therefore stability preservation has been a key focus of continual learning approaches to prevent catastrophic forgetting [B, C]. The regularization provided by STAR attempts to balance this trade-off, and does not strongly prevent model plasticity and its ability to learn new tasks.

[A] Mermillod et al. The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 2013.

[B] Mirzadeh et al. Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 2020.

[C] Lin et al. Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 2022.

We would like to thank you again for your constructive feedback, and give a gentle reminder that the discussion period will close in less than a week. We would be happy to further discuss any unresolved questions that you may have!

We are currently in the progress of including a figure depicting empirical validation for fig 1. and fig 2. and we will certainly include it in the supplement of the camera ready version.

We thank the reviewer for acknowledging our efforts and their positive comments. With that said, we would like to gently remind the reviewer to increase the score on openreview.

We thank the reviewer for praising our work as being “easy to follow”, and “thorough”.

Theoretical justification: We would like to reiterate the connection of our formulation to forgetting based on empirical error (i.e. accuracy). Our method uses weight perturbation to reduce the KL-Divergence of the output with that of future output distributions. We refer to [A], which shows that empirical error can be bounded by some function of a classification calibrated loss function, such as KL-Divergence. The theoretical justification of weight perturbation, beyond our formulation, is a subject of extensive research: AWP [B] mentions a relationship with PAC-bayes generalization bounds but does not investigate thoroughly. Finally, the seminal work of [C] claims the PAC-bayes relationship is incomplete and provides convergence proofs for a worst-case perturbation method in the stochastic and non-convex scenario.

Perturbation norm ratio: We would like to clarify and reiterate our formulation from the paper. The ratio gamma is not defined as the norm ratio, but we design the perturbation $\delta$ such that the norm ratio $\frac{\|\delta^{(l)}\|_2}{\|\theta^{(l)}\|_2}$ for each layer $l$ is approximately equal to $\gamma$. The normalization is done on a per layer basis, and then scaled by the hyperparameter $\gamma$ for controlling the magnitude of the perturbation. This is motivated by different layers having different numerical distributions, as well as the scale invariance of the neural network layers [D] (i.e. multiplying the weights of one layer by a number and dividing the weights of the next layer by that number leads to the same network). We have added further clarification of this to the revised manuscript.

Finally, we’d like to thank the reviewer for pointing out the algorithm inconsistency and we have fixed the notation. We would like to reiterate that the notion of time in our algorithm is not necessarily epoch and can be thought of as an optimization step.

[A] Bartlett, Peter L., Michael I. Jordan, and Jon D. McAuliffe. "Convexity, classification, and risk bounds." Journal of the American Statistical Association 101.473 (2006): 138-156.

[B] Wu, Dongxian, Shu-Tao Xia, and Yisen Wang. "Adversarial weight perturbation helps robust generalization." Advances in neural information processing systems 33 (2020): 2958-2969.

[C] Andriushchenko, Maksym, and Nicolas Flammarion. "Towards understanding sharpness-aware minimization." International Conference on Machine Learning. PMLR, 2022.

[D] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018

We would like to thank you again for your constructive feedback, and give a gentle reminder that the discussion period will close in less than a week. We would be happy to further discuss any unresolved questions that you may have!

We thank the reviewer for their comments,suggestions, and constructive criticism. Here are our responses:

Scaling to more tasks: We would like to kindly note that we have extensive experiments on three different datasets, which are standard and adopted for the works of [A,B]. As for Omniglot (50 tasks), while increasing the number of tasks naturally makes the CL scenario more challenging, we have no reason to believe our method would be particularly disadvantaged compared to other CL/enhancement methods. Furthermore, Omniglot is a one-shot learning dataset with limited data samples (~1600) and additional challenges which we believe to be outside the scope of this paper.

Notion of epochs: We would like to reiterate that our formulation does not necessarily equate time to an epoch, but the notion of time can be considered to be a single optimization step, making it easily extendible to single-epoch/online scenarios. Our optimization technique is performed on each batch regardless of how many epochs there are. As a proof of concept, we’ve added a limited experiment at the end of this comment (table 1). We perform single-epoch training of the S-CIFAR100 dataset with 3 training steps per batch for all settings and average over 5 seeds. The results suggest that STAR leads to performance improvements in the single-epoch setting as well. With that said, the online CL setting comes with its own challenges and may require additional considerations (such as additional augmentations and repeated rehearsal [E]), there are a variety of works tackling the online CL scenario, such as [C] which we mention in our related works. In our added results, we experiment with repeated rehearsal as well as without it. We believe a dedicated, thorough extension of our method to the online scenario could be a task for future works.

Running time: We thank the reviewer for the suggestion and will report the running times for table 1 in the revised version in the supplement. We would like to note that while our method does indeed involve extra computational expense, it is not a prohibitive cost (only two additional forward/backward steps per batch) and leads to an improvement in performance. Improving the efficiency of our method (and perhaps that of other worst-case weight-perturbation methods) can be the subject of future works.

Large-scale datasets and models: We utilized traditional CL benchmark datasets and would like to note that we conducted experiments on miniImagenet, which is a benchmark large-scale CL dataset. The additional computational cost introduced by STAR is not prohibitive for larger models as it only adds two additional forward/backward passes during training.

Hessian approximation: The approximation utilized is a first-order approximation of the KL loss (eq. 7). This approximation is widely used by existing works in the literature which leverage worst-case weight perturbation [C,D], and works well in practice, as shown in our experiments. Furthermore, the work of [F] provides convergence guarantees for Sharpness Aware Minimization which utilizes a first-order approximation on worst-case weight perturbation.

Table1. Proof of concept results for single-epoch training on S-CIFAR100, all methods train for 3 steps on each batch. Averaged over 5 seeds.

[A] Bonicelli, Lorenzo, et al. "On the effectiveness of lipschitz-driven rehearsal in continual learning." Advances in Neural Information Processing Systems 35 (2022): 31886-31901.

[B] Wang, Zifeng, et al. "DualHSIC: HSIC-bottleneck and alignment for continual learning." International Conference on Machine Learning. PMLR, 2023.

[C] Foret, Pierre, et al. "Sharpness-aware minimization for efficiently improving generalization." arXiv preprint arXiv:2010.01412 (2020).

[D] Wu, Dongxian, Shu-Tao Xia, and Yisen Wang. "Adversarial weight perturbation helps robust generalization." Advances in neural information processing systems 33 (2020): 2958-2969.

[E] Zhang, Yaqian, et al. "A simple but strong baseline for online continual learning: Repeated augmented rehearsal." Advances in Neural Information Processing Systems 35 (2022): 14771-14783.

[F] Andriushchenko, Maksym, and Nicolas Flammarion. "Towards understanding sharpness-aware minimization." International Conference on Machine Learning. PMLR, 2022.

We would like to thank you again for your constructive feedback, and give a gentle reminder that the discussion period will close in less than a week. We would be happy to further discuss any unresolved questions that you may have!

We would like to thank the reviewer for their comments and for describing our method as “novel” and “interesting” and our evaluations as “concise and to the point”.

LiDER: We would like to start our response by highlighting the major difference between our work and that of LiDER [A]. First, LiDER focuses on regularizing the Lipschitz constant of the neural network, while STAR (ours) optimizes the KL-divergence of the output distribution between current parameters and the potential future parameters. Second, and more importantly, computing the Lipschitz constant is non-trivial and requires approximations and relaxations based on the activation functions and possibly the architecture of the network. To quote the authors of LiDER in the limitations section “our approximation cannot be applied to not-Lipschitz continuous layers (e.g., cross-attention)”. STAR, on the other hand, is general to probabilistic modeling of the output classes and can be used with attention-based architectures, e.g. the transformer.

Prompting-based methods: First, the usage of prompting methods in CL requires pre-trained models, often with a large number of parameters, and furthermore, comes with certain limitations in terms of expressiveness [C]. Our method, on the other hand, is applicable to a general training scheme. Second, we would like to note that our method is a complementary approach that can also be combined with prompting methods with the inclusion of a small rehearsal buffer (as is done in some existing CL prompting works [B]).

Results on CIFAR100 and miniImageNet: We would like to note the results in table 1 and that our improvements over most existing baselines are significant (up to a ~10% increase in accuracy across the two datasets). Regarding the results in Table 2, out of the 12 experiment settings presented for CIFAR100 and miniImagenet in Table 2, we achieve best or second best accuracy compared to competing enhancement methods for 10 of them.

Mismatched results in Table 1: We thank the reviewer for the keen eye and noticing the difference, however, we use the same hyperparameters as [A, D] and reported the same results for the X-DER [E] (meaning [A,D] also suffer from this inconsistency). Upon further investigation, we suspect this is due to using a different batch size (64) than that of the original paper (32). We have rerun the experiments with the original batch size and have updated the results in the revised manuscript in Table 1. (Note that the batch size used for this experiment is not reported in [E]).

Minor errors: Thank you for indicating these, we have revised them in the manuscript.

[A] Bonicelli, Lorenzo, et al. "On the effectiveness of lipschitz-driven rehearsal in continual learning." Advances in Neural Information Processing Systems 35 (2022): 31886-31901.

[B] Wang, Zifeng, et al. "Learning to prompt for continual learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[C] Wang, Yihan, et al. "Universality and limitations of prompt tuning." Advances in Neural Information Processing Systems 36 (2024).

[D] Wang, Zifeng, et al. "DualHSIC: HSIC-bottleneck and alignment for continual learning." International Conference on Machine Learning. PMLR, 2023.

[E] Boschini, Matteo, et al. "Class-incremental continual learning into the extended der-verse." IEEE transactions on pattern analysis and machine intelligence 45.5 (2022): 5497-5512.

Thank you to the authors for taking the time to thoroughly address my questions. I appreciate that they corrected the results in Table 1 with the right batch size, and suggest to check all the other hyperparameters for a fair comparison.

I still emphasize that the results in Table 2 are not particularly impressive. However, Table 1 demonstrates a notable improvement over other state-of-the-art replay baselines, and I acknowledge that prompting techniques, which depend on pre-trained models, are beyond the scope of this work.

After the last changes, I now believe this is a strong submission and have accordingly raised my score to 8.