Slight changes in tasks, CL Effective Model Capacity (CLEMC) is defined in terms of forgetting cost which in turn is defined in terms of the loss. When capacity increases it is actually an increase in the loss. An increased loss is usually indicative of degradation in model performance and that is what we demonstrate using our experiments, without actually quantifying a usability threshold. Furthermore, the loss threshold after which a model becomes unusable is very task-dependent and coming up with a general threshold is difficult.

One example is Fig. 6, where we also plot the validation cost. This indicates the generalization performance of the model. Lower the validation cost better the model. We show that increase in capacity leads to increased validation cost - again indicative of degraded model performance. 

Four Modalities The experiments are to validate various aspects of the theory. The common theme in each experiment is that we define tasks by changing the data distribution.  Depending on the modality of the experiment, the features we use to change the data distribution varies. In some experiments, we demonstrate the change in capacity with introduction of new tasks by directly plotting $\epsilon^*_k$ or the forgetting cost $J_F$. The choice of using the gradient of the value function w.r.t the weights was to show the forgetting is associated with large changes in these gradient values. This is in line with the result in Theorem 2. 

Detailed explanation on figure 1 We have performed a major revision of the text to make the presentation simple and easy to understand. For instance, we have added a detailed caption explaining Fig. 1 We have also included text describing parts of Fig. 1 in various subsections of Section 3. 
Hard visualizations We have made the subsequent changes to the plots for additional clarification. |ΔT(p)| For each new task, we introduce a $\delta$ change in the amplitude and frequency. For K subsequent tasks, the change is $K \times \delta.$  

Improved forgetting results In Theorem 3, we show that capacity divergence happens with change in task distribution even with regularization. In our experiments, we show that even runs with regularization show an increasing trend in capacity (higher forgetting cost) as more tasks arrive. However, note that we have not explicitly analyzed the effect of regularization on the accuracy in this paper. This could be a potential future work.

Fig 4A and fig 2B This is an expected behavior, where for each particular tasks, the model keeps trying to learn and reduces the forgetting cost. The main theoretical and experimental conclusion of this work is that, in spite of such learning, the task behavior eventually becomes too much for the network to handle, thus resulting in diverging capacity.

How is γ(k) For our experiments we chose $\gamma^{k}$ as 1. The hyper-parameters used are specified in the  setup for each experiment. We have also added Appendix F with parameter details for case study 4. If you need information about a specific setting please let us know and we will be happy to provide more details.

Thank you for your comments, please let us know if any of these comments need additonal clarification, we are happy to elaborate.

However, I feel some of my concerns have not been fully addressed. I understand that ek is used in some and gradient of value function in others. My concern is why not use a common set of experiments across all modalities.

We apologize for the unsatisfactory response.

We have now updated all the figures to consistently plot $\epsilon^*_k$ across Figs. 2-6. To provide additional insight, we plot $\epsilon^*_k , d \epsilon^*_k$ and $\partial_w V_k^*$ In Fig. 2, 3 and 4.

Moreover, we plot $\partial_x V_k^* , \partial_{\phi} V_k^*$ and $\partial_x V_k^*$ in Fig, 5 and visualize how weights change for the transformer model. We hope that these plots provide additional clarity and introduce uniformity to our experiments. \

To provide more rationale behind these choices, note that, we performed four experiments with different model architectures and data
(1) FNN with Sine wave data,
(2) CNN with image data,
(3) GNN with graph data and
(4) LLM with text data.

In each experiment, we measured the capacity through the forgetting cost, gradient of the value function with respect to weights $\frac{ \partial V^* }{\partial w}, and$ gradient of the value function with respect to input data. In case of graph data, which consists of both node and edge features, the gradient of the value function with respect to input data has two components: \

• one is w.r.t node features, $\frac{ \partial V^* }{\partial x}$
• another is w.r.t. edge features, $\frac{ \partial V^* }{\partial \phi}.$

Now, in each experiment, our goal was to highlight the capacity degradation due to distribution shift introduced by incoming tasks and also validate our theoretical results. For experiments (1), (2) and (4) [Figs. 2, 3, 4, 6], we used $\epsilon^*_k$ through $J_F$ because $\epsilon_k^*$ is upper bounded by the forgetting cost $J_F.$

For experiment (3) [Fig. 5], in addition to demonstrating that the capacity deteriorates with an increase in $|| d \epsilon^*_k ||$, we perform a fine-grained analysis and breakdown this increase in terms of the sum of the gradient of value function w.r.t weights ($\frac{ \partial V^* }{\partial w}$), and the gradient of the value function w.r.t data ($\frac{ \partial V^* }{\partial \phi}$, and $\frac{ \partial V^* }{\partial x}$ ) (as derived in Lemma 1). The intention was to provide different perspectives of the same capacity degradation problem.

I understand the theoretical insights align with the experimental results from fig 4A and 2B. But my original question was to know the rationale behind the difference in the patterns of 4A and 2B. Could you explain this more clearly?
There are two aspects to our experiments, first in Experiment 1, we introduce different degree of noise to see how the capacity reacts to different shifts in the distribution, this is shown in Fig. 2B. Next, in Fig 4A, 5 and 6 we use standard datasets and neural network architectures to elucidate, how capacity changes in typical CL problems without introducing drastic noise values. In Fig. 4A, the change in capacity is not drastic as in Fig. 2. Nonetheless, there are two important observations.

• First, $\frac{ \partial V^* }{\partial w}$ increases with each new task.
• This change influence the place at which the capacity reaches after repeated learning, as observed, the end point of capacity at 2000 is slightly smaller than 3000 and so on.

In other words, the model fails to converge to the same precision with each new task. The differnce between Fig. 4A and 2 is that is "how drastic the effect looks and how close to zero the model gets after repeated updates?" This key difference is introduced because of the cross entropy loss function in Fig. 4, where the model learn much faster than the case with MSE in Fig. 2 and therefore reaches much closer the zero than the case of MSE. Thus the effect of distribution shift is more prominent in Fig. 2 than Fig. 4. The only indication in Fig. 4 is the jumps in the gradient of the value function with respect to the weights.

We hope that these responses were satisfactory and you would consider revising your scores for them.

Please let us if there is any additional clarification, you might need.

Will do. We thank you for the reviews and for your time !!

Key theoretical study We thank the reviewer for pointing us to the reference. We have included it in the related work section in the revised manuscript. As mentioned [1] explores catastrophic forgetting (CF) in the presence of task similarity..

Task Ordering The formulation of CLEMC does not consider any particular ordering of the tasks. To obtain Theorem 2 and Theorem 3, we only assume that the changes introduced by any task are bounded. In our experiments, especially the one with LLMs (8M and 134M parameter models) we did, however, observe that forgetting cost depended on the ordering of the tasks. We could definitely provide empirical results demonstrating the effect of task ordering in the final version of the paper. Finally, we agree that task ordering and its impact on CLEMC is an interesting question and will be a good topic for future study.

Fig 1 Figure 1 is a hierarchical visualization of the Dynamic Program based formulation of continual learning and the computation of the effective model capacity. At the lowest level we have the tasks. The model weights are updated based on the tasks using appropriate loss function minimization. The forgetting cost, $J_F$, for any task $k$ is obtained by minimizing the loss over tasks in the interval $[0, k]$. This is shown by the arrows in green. The value function, $V^*$, is the forgetting cost over all the tasks in $[1, K]$. This is shown by the arrows in black. The optimal capacity is computed as the sum of the effective capacities over tasks $[1, K]$ (shown in red). 
In response to this comment, we have updated the caption of Figure 1. We have also added text in Section 3, referencing Figure 1, as we discussed different parts of the figure.

Fig 6 This was an interesting observation from our experiments. When we performed  experience replay (ER) for previous tasks along with a new task, we continued to observe an increase in forgetting cost in general, which supported our theory. Also, the cost with ER was slightly higher than without ER. We attributed this to the model learning a harder task due to the data mix. However, for the books task, when we performed an experience replay (and we repeated this experiment multiple times) we observed that the "with ER" cost was lower than the "without ER" cost. We believe that the model checkpoint from the previous task provided a better initialization for the books task, leading to lower cost. Also, it is possible that the books tasks had similarity with the previous  tasks dataset which helped in lowering the forgetting cost. 

[1] Doan, Thang, et al. "A theoretical analysis of catastrophic forgetting through the ntk overlap matrix." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

Please see the collective response to all reviewers at the top of the page for clarification on these points. We hope this response has provided some clarity and you would reconsider your score.

We thank you for the reviews and for your time. Since, the author response window closes tonight, we will not be able to answer any more questions.

We hope that, our responses were satisfactory.

We thank the reviewer for a positive review of the paper and for the valuable feedback and suggestions. Please find our responses below. 

W1: In Fig 4(a), we can see that the value of the CLEMC, $\epsilon^*_k$, grows as new tasks arrive -- indicating reduced capacity and supporting our theoretical claims.  In Figure 5, the changes in the task data occurs due to both node and edge feature changes in the graph. As new tasks arrive, we observe a change in weights $\frac{\partial V^{*}}{ \partial w}$ to accommodate changes in either or both node features and edge features, that is, the partial derivative of the value function with respect to edge and node features.

W2: We agree, but would like to humbly submit that in the CL community, Omniglot is a common dataset used for classification. The computational requirements for running the GNN on a large graph dataset makes it hard for us to run such experiments. Furthermore based on our experiments with relatively large language datasets we believe that the conclusions would most likely hold. 

W3: We completely agree with this observation. The claims by themselves are intuitive and unsurprising. Our goal was to create a theoretical framework (in the form of the Dynamic Program-based formulation of the CL problem) within which we could validate our intuition regarding these results and pin them to concrete terms. This connection would allow us to then further study the effect of these terms on capacity and forgetting and improve overall model performance.

W4: In Figure 6, the pre-training cost curve is added as a reference. For pre-training a mix of all the data sets is used.Wiki is the first task on which the model is trained as mentioned in the setup. This is followed by git, arxiv and books. As we can see for both the 8M and the 134M parameter models, when the model is trained on wiki data alone, the forgetting cost is comparable to the pre-training cost (mostly because data is similar). As new tasks arrive, initially the cost continues to reduce, showing that the model is able to learn without forgetting. Eventually as more tasks arrive, with bigger differences in data we see that the forgetting cost increases. You can also observe that when data replay based regularization is performed the forgetting cost for validation improves. 

Q1: Figures 7 and 8 are meant to visually bring out the impact of new tasks impact the model weights (in terms of how much their magnitude changes). We have increased the size of the plots to improve readability. But, we can also add summary statistic about the distribution of weight changes in the final version of the paper.  

Q2 We have revised our text based on suggestions and comments received. The different headings in Section 4.1 have been modified to better highlight the claims and act as a summary of results.

Q3 In this setup, big values are bad and small values are good. Based on these comments, we have now explicitly added a line in the introduction to make it clear that higher capacity is associated with larger forgetting cost and lower representational capability of the model.

For Fig. 5, my question is asking the following: The authors are trying to suggest that ∂V∗/∂x precedes the change in weights. However while this is clear from the big bump, where are unexplained dynamics in weights. How do these relate to dV/dx ? what weight changes are related and what are not? In practice, we would update the weights based on the data. Therefore, each jump in the edge or the node feature are followed by the jump in weights. This can be understood by the following three incremental steps
- Any change in task, brought about by a change in the edge $\phi$ and node $x$ features, reflects in a large jump in the forgetting cost $J_F$.
- A large jump in $J_F$ then leads to a change in the weights by virtue of the gradient necessary for minimizing $J_F,$
- This change is then reflected in the weight updates.

However, the dependence between weights and the tasks is recursive and mathematically inseparable (the first difference in capacity cannot be independently decomposed into the task and weight changes). Mathematically, this can be seen by writing the first order KKT (Karush Kahn Tucker) conditions on $V^*$, which is equivalent to $-\langle \partial_w V^*, dw \rangle = (V^*(k+1)-V^*(k)) + J_F +\langle \partial_x V^*, dx \rangle + \langle \partial_{\phi} V^*, d~\phi \rangle.$ where $k$ is the task index. it is clear from this equation that the dynamics between $x, \phi, w$ are not separable, i.e. $V^*$ implicitly depends on the weight, edge and node features. This recursive equation is how $x, \phi$ and $w$ relate to each other.

We have put a common response to all the reviewers on these as they concern the novelty and contributions of this paper.

We hope this response have provided some clarity and you would reconsider your score.

Thank you for your feedback and your time.

line 294, Eq. 15(a) to Eq. 15(b): You are absolutely correct, the equality in Eq.15(a) to Eq. 15(b) and inequality in Eq. 15(b) to Eq. 15(c) are switched. We have corrected this now. Note, that we use the Cauchy's inequality to make these deductions and these ideas should be clearer now. (Supplementary File, Section C, Eqs. 7(a) - 7(d)).

ine 307, Eq. 16: That factor should be dependent on m and not p. We have corrected this. We also realized in the derivation that, making this contribution explicit, introduces confusion, therefore, we have revised the derivation to indicate this.

line 323, Eq. 17(b) You are correct, there is an implicit dependency. However, we assume that all past information about this implicit dependency can be directly understood/observed by looking at $w^*_{j+1}.$ This is our primary assumption where a Markovian behavior is assumed in the weights and in the loss function. Note that, without these assumptions, the derivations and theorems in this paper do not work. To ensure validity of this assumption, we write the forgetting cost in the inner sum as a repeating term for each new task. This constructions allows the validity of this assumption as seen in numerous reinforcement learning works. 

Definition of "Effective Capacity" or "Capacity" in the Paper

line 123, (EMC): This is to state the traditional manner in which prior works have defined effective model capacity by optimizing over possible data. The following paragraph also highlights that EMC is inadequate in the CL setting. Note here the $\epsilon$ does not have a task-specific subscript. 

line 179: "as capacity at each k and line 181: We have revised our text to separately define EMC, Forgetting EMC and Continual Learning EMC and clarify the intermediate steps. At any given task, we perform minimization across all observed task~(the extra minimization), to get the forgetting cost. This is the summation over all previous tasks in $J_F$.  We use this to define the Forgetting EMC (FEMC) in Definition 2, which is the maximum forgetting cost over all the observed tasks. Finally, we define  CLEMC for a task k, in Definition 3, as the sum of FEMC over all tasks. 

line 243: The statement above refers to the term $\sum_{k}^{K} d \epsilon*_{k}$  where $d \epsilon_k*$ is the first difference in CLEMC. These notions have been clarified now.

Empirical Observations and Their Alignment with Text and Theorems

Figures. 2, 3, 4,.. This behavior is actually expected because we are to an extent trying to reduce forgetting during training. However, the goal of these plots is to show that,  for any new task, the cost increases showing there is forgetting happening. As more tasks arrive we observe greater divergence in $\epsilon^*_k$. The model's learning is interrupted by the constant changes in the tasks. 
Figure 2:: They are calculated as the Euclidean norm between the previous and the next task and these two notions are synonymous. In the revised text we have removed all references to $\Delta x$.
Line 333, "blue curve is This recovery is expected because the model is learning and trying to reduce forgetting and in this process, you observe the stability gap. However, in the long run, the model is overwhelmed by the constant changes introduced by the tasks. This is the behavior being shown here.

Minor comments on notations and typos

line 136-140 We have changed this notation. Now, both (k) and k refer to specific indexes, and $\mathbf{k} = [1, k]$ to the collection of observed tasks. Note, we now label tasks from $1, \cdots, K$ instead of $0,\cdots, K$. 
line 089, Equation (CL) We have modified these proofs and therefore the notations for clarity.
Why are the Lipschitz constantsThis is basically the multiplication of the learning rate $\alpha$ with the Lipschitz constant. We have fixed this now with a parenthesis instead of square brackets. (Supplementary Files, Equations 24 and 25)

Line 239 main text, Thank you for pointing this out. We have corrected it now. 

We hope that your questions were clarified. If you have any further questions, we are happy to explain more. Thank you again for your care and attention in reviewing this paper. Our paper has greatly benefitted from addressing the concerns raised and we genuinely appreciate your time.

We thank the reviewer for the extremely detailed and careful review of the manuscript. We realize that a lot of the confusion stemmed from the use of cumbersome notation and have tried to simplify it. Below we provide detailed responses to the various questions.

Lemma 1 and Theorem 1

line 167, Eq. 6(a): Please observe that the derivation to Lemma 1 is motivated from the standard Hamilton Jacobi Bellman (HJB) equation's derivation and a nice intuitive version can be found in [1], Chapter 7. Indeed, in Eq. 6(a), the term $\Delta V_{(j)}$, does refer to all the higher order terms, i.e., terms corresponding to order second derivative and above. This is a standard practice in HJB derivation where the higher order terms are assumed negligible~[1] and the absence of these terms introduces an order of $2$ error into the approximation (Taylor approximation). Using $\Delta$ to denote the higher order terms was an error in notation. We have now replaced this notation with $\mathcal{O}(2)$ to indicate terms of order two and above (Supplementary Files, Eq. 1).

Appropriateness of Taylor Expansion: The derivation in Lemma 1, follows the standard HJB derivation from the optimal control perspective, where it is assumed that all future information regarding the state space are known and kept constant. The only quantity that changes is the current cost. Under this construction, the additional term that comes in V(k+1) from k is $J(k).$ This is rather intuitive in the continuous time formulation as in [1]. To illustrate this,  without burdensome notation, we have the following alternative informal derivation of the HJB equation. Let 

$V^*(k) = \underset{ \{ w(t), t = k, k+1, k+2, \dots \} }{min} \sum_{t=k}^{K} J(w(t))$

where $w(t)$ represent the model weights and $J$ represent the forgetting cost. Then, 

$V^*(k+1) = \underset{ \{ w(t), t = k+1, k+2, \cdots K \} }{min} \sum_{t=k+1}^{K} J(w(t))$

By this structure, $V^*$ is a function of the weights starting at index $j$ and goes all the way to $K$. Just by subtracting, $V^*(k+1)-V^*(k),$ we can get 

$\Delta V^* = V^* (k+1)-V^* (k) = \underset{ \{ w(t), t = k+1, k+2, \cdots K \} }{min} \sum_{t=k+1}^{K} J(w(t))  - \underset{ \{ w(t), t = k, k+1, k+2, \dots \} }{min} \sum_{t=k+1}^{K} J(w(t))$

which provides 

$\Delta V^* = V^* (k+1)-V^* (k) = \underset{ \{ w(t), t = k+1, k+2, \cdots K\} }{min} \left[ \cancel{\sum_{t=k+1}^{K} J(w(t))}   + J(w(k)) -  \cancel{ \sum_{t=k+1}^{K} J(w(t))} \right].$

This is under the assumption that all future information required to calculate $J$ at every future instance is known. Therefore, the exact difference between the optimal value function at $k$ and $k+1$ is the forgetting cost $J(k).$ Ideally, making the optimal decision (weight value) at each $k$ should lead to an optimal sequence of weights for all task from $k+1$ to $K.$ However, in practice, exact cancellation will not happen and there will be a $\delta,$ a small perturbation around the current optimal cost. In particular, 
$\Delta V^* = V^* (k+1)- V^* (k) = \underset{ \{ w(t), t = k+1, k+2, \cdots K\} }{min} \left[ \cancel{\sum_{t=k+1}^{K} J(w(t))}   + J(w(k)) -  \cancel{ \sum_{t=k+1}^{K} J(w(t))} + \delta \right]$
This $\delta$ are the perturbations to $V^*$ by addition of task at $k.$ These perturbations can be obtained by Taylor series of $V^*$ around $k$ given as $\langle \partial_x V^*, dx \rangle + \langle \partial_w V^*, dw \rangle + \mathcal{O}[2]$ terms, ignoring the higher order terms provides the HJB. Please see if the revised derivation in the manuscript (different from above) introduces clarity (Supplementary Files, Section B, Lemma 1).
line 214, Eq. 9. Transition from Eq. 7 to Eq. 9: Eq. 9 is actually not coming from Eq. 7 but rather a function of Eq. 8(b). When you multiply Eq. 9 with -1 and then, take the negative inside the first max and the second min, we have the substitution and eventually the final result. Multiplying by negative one does flip the mins and the max. We had omitted these steps in the current draft leading to this confusion. Reference to Lemma 3 is a typo. We have re-written the proof in a step-by-step manner to make the above steps explicit and have removed any spurious references. (Supplementary Files, Eqs. 9 and 10).
line 252, Lemma. Definition of k. k is used as an index for the incoming tasks throughout the text. $\mathbf{k} = [0, k]$ has been defined on line 84. However, we acknowledge that this could be done better. In the revised version, we state this definition upfront when we set up the notation. (Section 3, paragraph on notation setup).
line 263, <a,b>=||a||||b||cos(θ) This was a typo, the inequality is $<a,b> \leq \|a\| \| b\|$. We have fixed it now. (Supplementary File, Section C, Cauchy-Schwarz inequality)

Thank you very much for your insights and sustained engagement with the paper. We have further added a clarification on the contributions and novelty of the paper through a common post to all authors.

My concern is whether this dynamic in optimization—where the stability gap occurs then followed by this decreasing loss—is fully captured by the theorem. It seems to me that the theorem primarily supports the overall increasing trend of loss, rather than detailed CL training dynamics. Is this interpretation correct?

The main conclusion of the theorem is essentially about the increasing trend of loss function, however, stability gap is the reason we see this increasing trend and this is observed in the proofs. We show in the proofs that, the stability gap introduced by a particular task will lead to divergence if the change in the task is too large and you can update only a finite number of times. This can be informally observed as follows. Note that the first difference in capacity is given (through Eq. FD in the paper) as

$d \epsilon^*_{k} = \langle \partial_T V^*, dT \rangle + \langle \partial_w V^*, dw \rangle.$

We have ignored the max's and the min's and assumed number of tasks to be one to make notational simplifications (the generic form of this proof with less assumptions can be found in the paper, our main results). Now, the change in capacity is a direct function of change introduced by $T, w$ which is the stability gap. That is, As soon as there is a change in T, there is a change in w and there is a change in capacity. If you add the left and right hand side for K steps (one task, $I$ updates), we get

$\sum d \epsilon^*_{k} = \sum \langle \partial_T V^*, dT \rangle + \sum \langle \partial_w V^*, dw \rangle.$

The left hand side is a telescopic sum across different updates on the same task and this reveals

$\epsilon^*_{k+1} - \epsilon^*_{k} = \left[ \sum \langle \partial_T V^*, dT \rangle + \sum \langle \partial_w V^*, dw \rangle \right].$

Whether the stability gap introduced by a new task (the first term on the right hand side) is eliminated or not depends on whether the sum in the right hand side converges to zero or not. if it does not converge to zero, in otherwords, the change introduced by the tasks is so large that finite number of updates are insufficient for convergence, say $\left[ \sum \langle \partial_T V^*, dT \rangle + \sum \langle \partial_w V^*, dw \rangle \right] = d_k$, then, $\epsilon^*_{k+1} - \epsilon^*_{k} = d_k.$

then, for a finite number of updates $T,$ this sum from 0 to K will diverge.

We hope these responses have introduced clarity into the various aspects of the paper.

We thank you for the reviews and for your time. Since, the author response window closes tonight, we will not be able to answer any more questions.

We hope that, our responses were satisfactory.