The Importance of Being Lazy: Scaling Limits of Continual Learning

Thank you for taking the time to read and review our paper. We are glad to hear that you found our paper well-written and that it offers a novel contribution to the space of continual learning.

---

Edge of Laziness and Phase Transition

Thank you for your thoughtful comment, and we are sorry for the confusion caused by the terminology of Edge of Laziness.

Firstly, we would like to take the opportunity to clarify our position on the observed phase transition. In our results, it is clear that a phase transition happens, however, it does not always appear as a sharp, discontinuous transition between the two regimes, but more like a non-linear continuous function of $\gamma_0$ in the area where the transition occurs. For this reason, we do not make any claims about the nature of the transition, being it discrete or sharp but still continuous.

Secondly, our choice of terminology, "Edge of Laziness," was not intended to rigorously characterize the phase transition. Instead, it was intended to be evocative of a boundary between an effectively lazy region of the $\gamma_0$-space.

Thirdly, we are currently unable to demonstrate the nature of the phase transition theoretically, a result that we believe to be highly non-obvious. Nevertheless, we are actively working towards this result, and we were recently able to characterize the infinite-time solution of CF in the rich regime of the linear case. In this setting, we observe a highly non-trivial relation between the task similarity $\rho$ and $\gamma_0$, where the second-order coefficient of CF is a sixth-grade polynomial in $\rho$. The complexity of this relationship - even in the linear case - points to the non-triviality of the behavior and thus to the difficulty of getting a general characterization of the phase transition theoretically. Nevertheless, we will continue working to extend this result to the non-linear setting for future work.

Following your comment, and determined to avoid any confusion or misunderstanding with future readers, we have rephrased the parts of the paper that introduce and discuss the concept of the edge of laziness. We decided to refer to it as lazy-rich transition, avoiding the word "edge". We are also in the process of changing the title to reflect this change of terminology accordingly. We are grateful for the opportunity to improve the clarity of our claims and paper.

Other Comments

• DMFT with one hidden layer NN

  We plan to extend the DMFT to deeper networks in future work. In the meantime, however, we would like to stress the striking overlap between the results in the shallow and deep networks, suggesting that the one-layer derivation is already able to represent well the behaviors of deep networks.

• Edge of Laziness, tradeoff, and optimal $\gamma_0$

  Thanks for the interesting question, which allows us to touch upon two distinct phenomena that influence the overall optimal level of richness.

  Firstly, we have observed that the task similarity has a direct and powerful impact on the feature evolution itself, meaning that a higher similarity effectively reduces the amount of feature learning for fixed $\gamma_0$ (Fig. 5a). This suggests that the task similarity (or in general the degree of (non-)stationarity of the data) non-trivially interacts with the regimes of the network. Including this perspective into the DMFT formalism is non-trivial, and we have just started exploring this avenue (see Appendix E3, F). However, we believe this direction would be very interesting and valuable for future work.

  Secondly, the optimal $\gamma_0^\star$ reflects the plasticity-stability tradeoff, and therefore, $\gamma_0^\star$ can shift from the $\gamma_0$ at which the lazy-rich transition happens. Generally, we find that when the tasks are close to stationary, the optimal $\gamma_0^\star$ is always 1, i.e. the maximum of the range tested. In other words, when the data is stationary, higher plasticity helps reach better performance. By contrast, as the amount of non-stationarity is increased, stability is necessary to avoid losing performance on the old data and the optimal $\gamma_0^\star$ is consistently lower than 1. From this point of view, the $\gamma_0$ at which the lazy-rich transition happens represents the minimal $\gamma_0^\star$ as any value below it will not increase stability (because the network is effectively lazy) and it will not increase plasticity.

  While we do observe the plasticity-stability tradeoff even in the theoretical experiments at infinite width (Fig. 4a), we have not specifically investigated it theoretically.

• Fig. 6 labels

  Thanks for pointing out the cut-off labels of Fig. 6, and for suggesting an increased size of the Figure. We will correct these in the final version of the paper.

---

We hope that we have adequately addressed your questions. If you have further questions or comments, we remain at your disposal.

We are grateful for your time reading our paper, and thank you for your thoughtful review.

---

The role of DMFT in the paper

We agree that the results of the DMFT are somewhat abstract, however we believe this is an inherent limitation of this theory: it is hard to simplify the results and reach interpretable results without sacrificing its completeness. Committed to this direction for future work, we are currently working on integrating minimal assumptions that could help us simplify the results without trivializing them.

Nevertheless, we would like to address your opinions that "(1) the DMFT results do not substantially strengthen the claims based on experiments", and that "(2) the technical novelty here is limited".

1. The DMFT equations allow us to actually reach the infinite-width limit, something that would have been impossible otherwise. Although it is unusual to lay the theory at the service of the experimental evidence, we believe that the infinite-width curves crucially strengthen our finite-width insights of the experiments.

2. We agree that the DMFT techniques are a known approach to analyze the network behavior. Nevertheless, we believe that our extension to the Continual Learning scenario is notable and highly valuable for the community, as we

   • introduce the NTK across tasks as a new entity of the DMFT formalism
   • are the first work to analyze feature learning in CL
   • propose a perturbative approximation which is different from previous approaches (e.g., Bordelon and Pehlevan), as we also expand the residuals in powers of $\gamma_0$, and we apply it to the CL scenario. This allows us to study the complex dependence between $\gamma_0$ and the task similarity $\rho$, obtaining a closed-form solution for the coefficients of CF, up to the second order, in the linear case. Seeking the "conceptual meat", we were recently able to obtain further insights on the role of $\gamma_0$ in the linear case, where we find that in the infinite-time solution, the second-order coefficient of CF is always positive, certifying that larger $\gamma_0$ yields higher forgetting. Moreover, we find that this relationship has a non-trivial dependence on $\rho$ through a sixth-order polynomial, where the maximum CF is obtained at richness-dependent levels of $\rho$. You can find the preliminary plot here.

Discussion of related work

• Petrini et al., 2022., Vayas et al., 2023., Yang et al., 2023. Thanks, they should definitely be included in the related works and cited accordingly.

• Atanasov et al., 2025. We would like to point out that this work is concurrent and will be presented at ICLR 2025. Although the paper is related to ours, we feel it is unfair to list as a weakness the missing discussion of such a recent work.

  Nevertheless, their findings are interesting and might be a good starting point for future work. As you suggest, the observed lazy-rich transition might be related to our choice of LR scaling (LR scales quadratically with $\gamma_0$): they observe that this scaling is optimal for the lazy regime, but not anymore for the rich and ultra-rich regimes (Fig. 1b), where instead sub-quadratic LR scaling is optimal. Our transition might therefore be due to a change of the effective LR, shifting from the optimal LR towards a too-large LR and thus approaching divergence. Note that this is just an intuition that requires further study.

Other comments

• Title: following also the review of DJW3, we have decided to avoid referring to Edge of Laziness, and will therefore opt for a more informative title, as you suggest.

• Representational Changes: We have quickly implemented this interesting experiment, and you can find the results for the Permuted-MNIST, restricted to samples 0 and 1, here. The drop in the alignment after the tasks switch represents an intriguing insight that serves as an interesting starting point for future analysis. We will add this experiment to the Appendix.

• Implementation of DMFT: thanks for pointing out the missing details. We use a modified version of the one by Bordelon & Pehlevan in the L=1 case, extended to support multiple tasks as per our training setting. We will clarify this in the final version.

• Network Capacity: we refer to network capacity as the network size (i.e. the width), irrespective of $\gamma_0$. In l. 352 we state that "the increased capacity (i.e. size) of the network does not benefit performance".

• Figure lables: we agree that the labels are slightly notation-heavy, but we believe that the CKA measure might be obscure to readers not familiar with the kernel literature; the labels provide a useful guidance to such a reader.

• Claim in the discussion: we agree and will adapt this claim.

---

We hope that we have adequately addressed your questions. If you have further questions or comments, we remain at your disposal.

Thank you for taking the time to read and review our paper. We are glad to hear that you found our paper clearly written, and that you consider our experiments well-designed, providing a systematic study of the topic and supporting well both claims and theoretical results.

---

1. Do the claims in this paper hold for the Transformer architecture trained with Adam optimizer?

   Despite the success of transformers in DL, their adoption in CL is still limited and fairly understudied [1,2,3]. If transformers in CL are already rare, their optimization with adaptive optimizers (e.g., Adam) is even more so, whereas vanilla SGD is the standard optimizer even when training transformers [4,5]. For these reasons, in our work we decided to study the ResNet architecture with SGD. Nevertheless, we agree that exploring our claims even for transformers and for adaptive optimizers is an interesting avenue for future research.

2. Can the DMFT be extended to an adaptive optimizer like that in the TP 4b paper?

   The DMFT could potentially be extended from [6]. However, in this work, we were more interested in understanding the dependence of the dynamics on $\gamma_0$, which requires lengthy derivations even for a two-layer linear network trained under gradient flow.

3. In practice, we may train a neural network with $O(width)$ steps by the scaling law. Can we extend the experiments and the DMFT theory to that case?

   The DMFT relies on the assumptions of the number of datapoints and timesteps of order O(1), beyond which the theory may break down. In this study, an extension to width-dependent training times is out of scope. However, we point to the experiments presented in Appendix B.1, in which we investigate the effect of training time in the NTP. We find that longer training time systematically yields high feature evolution even in the NTP, and therefore it leads to CF.

4. Can we predict the existence of a phase transition theoretically?

   We are currently unable to predict the phase transition theoretically. Nevertheless, we are actively working towards this result, and we were recently able to characterize the infinite-time solution of CF in the rich regime of the linear case. In this setting, we observe a highly non-trivial relation between the task similarity $\rho$ and $\gamma_0$, where the second-order coefficient of CF is a sixth-grade polynomial in $\rho$. The complexity of this relationship - even in the linear case - points to the non-triviality of the behavior and thus to the difficulty of getting a general characterization of the phase transition theoretically. Nevertheless, we will continue working to extend this result to the non-linear setting for future work.

5. In my opinion, the $\gamma_0$ is just an output multiplier hyperparameter in existing mup papers (e.g., TP 5), so it should transfer across different widths. Am I right?

   Thanks for the interesting question, which allows us to clarify the role of $\gamma_0$ as well as the novelty of our findings.

   Firstly, $\gamma_0$ is not only an output multiplier, instead it also quadratically modulates the LR (cfr Tab 1 in our paper). This implies that one cannot infer transfer properties of $\gamma_0$ from those of the LR, as the two are non-trivially related.

   Secondly, and crucially, the transfer properties of LR (and other hyperparameters) have only been shown - but not proven analytically - in the stationary setting, and not in the non-stationary and CL scenario. We therefore deem it non-trivial and surprising to observe that, with $\mu$P, $\gamma_0$ shows these transfer properties even in the non-stationary scenario.

   In hindsight, given that width scaling does not affect forgetting under mean field scaling, it is intuitive to expect that $\gamma_0$ should transfer. However, if width scaling were to affect the dynamics (e.g., by lowering forgetting due to increased capacity), then one could in principle expect $\gamma_0$ to scale differently with width. We hope this clarifies the subtlety of these results and the coherence of our findings.

---

We hope we have adequately addressed your questions and reservations that hindered your recommendation for acceptance. We remain at your disposal in case you might have further comments or doubts.

---

References

• [1] Ramasesh et al., "Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics", ICLR 2021
• [2] Mirzadeh et al., "Wide Neural Networks Forget Less Catastrophically", ICML 2022
• [3] Lu, et al. "Revisiting neural networks for continual learning: An architectural perspective." arXiv preprint arXiv:2404.14829 (2024).
• [4] Ramasesh et al., "Effect of scale on catastrophic forgetting in neural networks", ICLR 2021
• [5] Mirzadeh et al., "Architecture matters in continual learning." arXiv preprint arXiv:2202.00275 (2022).
• [6] Bordelon et al., "Infinite Limits of Multi-head Transformer Dynamics.", Neurips 2024

We sincerely thank you for taking the time to review our paper and for the thoughtful feedback. We are glad to hear that you found the work really interesting and stimulating. We are also grateful to hear that you appreciate our contribution to the literature.

---

The Pretraining Effect

We are glad you found the pretraining effect particularly interesting and intriguing; we share your enthusiasm for this phenomenon. We agree that these observations offer an important perspective on our results, in particular as the setting approaches training scenarios closer to the real world. In this regard, we think you might be interested in inspecting also the interplay between the pretraining effect and width scaling, as reported in Figures 17-19 of the appendix.

Before answering your specific questions, we would like to clarify that the design choices for our experiments targeted the main goals of the paper. However, we agree with you that looking into the pre-training effect more thoroughly is certainly very interesting.

Split-CIFAR

Is there a specific reason not to consider the CIFAR100 task?

We find no specific reason for not considering CIFAR100 in our experiments. We have chosen CIFAR10 as a representative dataset for our experiments for computational convenience, and then directly extended our analysis to Split-TinyImagenet, aiming to provide stronger empirical evidence on a more challenging dataset. Nevertheless, we agree that CIFAR100 might be an interesting in-between dataset and we thank you for pointing it out.

Wouldn't the pretraining effect be expected for it to occur in CIFAR10 as well?

In Split-TinyImageNet, we observe the pretraining effect happening for a larger number of classes per task, namely when the tasks have enough data diversity, and thus a greater overlap between task distributions. Our intuition is therefore that the reason why this behavior is not observed in CIFAR10, is that the number of classes is too small to allow for a significant overlap between task distributions.

Other Questions on the Pretraining Effect

Is it sensible to measure the pretraining effect with the forward transfer metric?

We haven't considered measuring the pretraining effect with the forward transfer metric, instead, we have focused on disentangling the feature evolution of the first and later tasks. However, we agree that the forward transfer metric could be a valuable addition to our analysis and we will look into integrating it.

How do you control the hierarchy of class similarities in Split-TinyImageNet?

We naively split sequentially the classes of TinyImageNet, without accounting for the semantics of the classes. However, when we vary the number of classes per task, we ensure to keep fixed the classes-to-task assignment, to have a fair comparison across runs with different nr. of classes per task.

Exploring the effect of the semantical hierarchy is, however, a very interesting avenue. For example, one interesting aspect one could explore in future work is whether one can reproduce our "pretraining effect" even with a fixed number of classes per task, and only by modifying the variety of the data from a semantical perspective.

---

Other Comments

• Tractability of forgetting expression at $t \to \infty$:

  The short answer is yes, albeit the formula is not present in the current version of the paper. Indeed, we were recently able to extend our perturbation theory approach for the infinite-time solution in the two-layer linear network setting, where we have a non-trivial relation between $\gamma_0$ and similarity $\rho$.

• Cubic complexity of infinite-width simulations:

  The fields h and z are composed of sums over P data points, and discretized integrals over T time steps, i.e., h and g are $\mathcal{O}(PT)$. The NTK is a sum of $\Phi$ and $G$, which are the inner product of respective fields and thus $\mathcal{O}(P^2T^2)$. Finally, the output is the matrix-vector product of the NTK and $\Delta$ of dimension P, and then integrated over T time steps following the PDE of Eq. 3. This leads to $\mathcal{O}(P^3T^3)$.

• Noise of low-width networks:

  The training process is stochastic (over initialisation), and quantities like the NTK rely on the self-averaging properties characteristic of the $N\to\infty$ limit. Concretely, this means that in the limit, the network's dynamics become deterministic and instead, at finite widths, these observables are a partial/incomplete snapshot of the underlying process, introducing more stochasticity.

• Typos and notation inconsistencies

  Thanks for pointing these out. We will correct these in the final version of the paper.

---

We hope that we have adequately addressed your questions. If you have further questions or comments, we remain at your disposal.