The Dual Nature of Plasticity Loss in Deep Continual Learning: Dissection and Mitigation

We sincerely thank the reviewer for their time, thoughtful feedback, and positive assessment of our work. We are particularly grateful for the encouraging remarks regarding the novelty and significance of our contributions. We address the questions in detail below and are happy to further discuss any remaining concerns.

Weaknesses 1: We are grateful to the reviewer's suggestion on adding the comparison between Mixup and G-Mixup to the main text, which is indeed crucial for demonstrating our method's true effectiveness. As pointed out in the main text mixup promotes smooth transitions in the repr. space by generating interpolated inter-class samples and labels, which can mitigate Type-2 LoP (i.e., over-stretch). However, it retains unchanged intra-class labels (i.e., fixed one-hot targets), which limits its ability to prevent Type-1 LoP. In contrast, G-mixup introduce intra-class label difference through Eq. 15.

Therefore, to clearly demonstrate the difference between the effect of the two mixup-style methods. We conduct an additional experiment that restrict mixup and G-mixup to intra-class sample pairs only, removing the inter-class smoothing effects for both party, and focus on the intra-class repr. space where mixup and G-Mixup provide the same amount of sample diversity but G-Mixup further promotes intra-class representation enhancement.

As shown in the table above, mixup rapidly leads to collapsed representations and stagnated learning, whereas G-Mixup continues to support learning subsequent tasks. This result clearly demonstrates the critical role of G-Mixup in addressing LoP, apart from providing sample diversity. We will include these results in the main text of the revised manuscript.

Weaknesses 2: We thank the reviewer for raising this insightful point. As correctly noted, the stability–plasticity dilemma is fundamental in continual learning. Our study explicitly focuses on plasticity loss, a relatively new but increasingly important perspective that emphasizes the degradation of a network’s capacity to adapt to new tasks, even in the absence of explicit constraints to retain prior knowledge.

Benchmarks such as Continual ImageNet, which involve thousands of sequential tasks (up to 5,000), push the network into regimes where backward consistency is often infeasible or undesirable. In such settings, many methods—including CBP—explicitly reset parts of the model after each task, sacrificing stability in favor of forward plasticity. Similar motivations are found in reinforcement learning, where preserving previously learned value functions from outdated policies may hinder rather than help future performance. Therefore, in our work, the primary objective is to ensure that the network retains sufficient plasticity to continue adapting to novel tasks, rather than preserving accuracy on past tasks.

That said, we fully agree that empirical assessments of stability are important. In response to the reviewer's suggestion, we report two key metrics of stability—Average Forgetting (AF) and Relative Forgetting (RF)—as defined in the reviewer's referenced work [1], to better illustrate how performance on previous tasks may deteriorate. Furthermore, since Continual ImageNet is formulated as a binary classification problem, we additionally report the average deviation from chance level (i.e., 0.5) after learning all 5000 tasks, reflecting the network’s retained knowledge on previously encountered tasks.

Table A Stability metrics on Continual ImageNet using SmallConvNet

Table B Stability metrics on Continual ImageNet using ConvNet

As shown in the tables above, all methods experience some degree of forgetting on previously learned tasks. When evaluating the final model—after training on all 5000 tasks—on earlier tasks, we observe different behaviors in terms of variance from the 0.5 chance-level baseline:

• CBP exhibits the lowest variance, suggesting that its representations are largely decorrelated from earlier task classifiers. This is consistent with the fact that CBP resets part of the network during each task, leading to representations that are less aligned with previously learned classification heads.
• L2 init achieves the highest variance. This reflects the fact that L2 init constrains weights to stay close to their initialization, which helps maintain representational stability, but its performance largely depends on how well the representations induced by the initial weights align with the current task.
• G-Mixup shows higher AF and RF values, as it allows the representation space to adapt more freely in order to optimize performance on the current task. While this may introduce some deviation from earlier representations, it supports effective continual learning by maintaining sufficient plasticity.

Question: Regarding the reviewer’s thought-provoking question about the relationship between regularization-based stability methods (such as EWC and LwF) and our dual-nature framework of plasticity loss:

While these methods also lead to impaired plasticity, the underlying mechanisms are fundamentally different. Regularization-based techniques impose constraints on network dynamics—e.g., freezing important weights (EWC) or forcing new predictions to remain close to prior outputs (LwF)—which restrict the network's ability to adapt. In contrast, loss of plasticity as defined in our work refers to a spontaneous decline in adaptability that emerges even without any external constraints, purely due to cumulative learning and optimization dynamics.

Therefore, the plasticity reduction caused by stability-focused methods does not neatly fall into the Type-1 (representational collapse) or Type-2 (representational chaos) categories we describe. Instead, they represent a separate, externally induced mechanism that trades plasticity for stability by design.

Nevertheless, there is an interesting conceptual overlap. Some recent studies in the plasticity loss literature have borrowed ideas from stability methods to preserve plasticity. For example:

• L2Init: a plasticity-preserving counterpart to EWC that penalizes deviations from initial weights, under the hypothesis that the initial network state is maximally plastic.
• Initial Feature Regularization (e.g., Lyle et al. 2022): constraining representations to remain close to those from random networks (conceptually similar to LwF), preserving expressivity and adaptability.

We sincerely thank the reviewer for these important references and suggestions. We will include this discussion in the revised manuscript to clarify the relation between our framework and stability-based methods, and to enhance the theoretical richness of our work.

First of all, we'd like to thank the reviewer for the invaluable suggestions, especially the ones regarding soft labels, LoP mitigation vs. mixup-style augmentation and Forgetting Measure, together they help improve the quality and impact of our work. We will make sure to include the results here into the revised manuscript.

W1, W2 & Q1, Q2 In our original experiments (expt's), we did not preprocess the baseline methods with soft labels. To address this concern, we have now conducted additional expt's on both ImageNet and CIFAR-100 datasets, where all baselines are also trained using soft labels. On ImageNet with MSE loss, the impact of soft labels is negligible. On CIFAR-100, where cross-entropy(CE) loss is applied, soft labels offer a modest improvement during the learning of later tasks. This is because that soft labels help prevent the representational (repr.) collapse or over-expansion that can occur when logits are trained to match one-hot labels. Due to character limitations, we present results using CE loss as a representative case in this section. However, we have conducted all the experiments on the benchmarks presented in the main text, and would be happy to share those results upon request. For all methods, the intra-class soft label smoothing factor was consistently set to 0.1 throughout our experiments. With these additional expt's, we can now show that G-Mixup consistently outperforms all baselines regardless of whether soft labels are used, as shown in the following table:

Resnet-18 Acc. on CIFAR100 with SoftLabel

Regarding the use of soft labels in G-Mixup (assuming the reviewer refers to our method): G-Mixup was originally designed for MSE loss, using one-hot interpolation for inter-class mixing and (1 + $\alpha$,0)–style labels for intra-class mixing. To enable the application of G-Mixup under the CE loss, we adopt soft labels to ensure the label remains a valid probability distribution (i.e., sums to one). This allows same-class samples to have different targets based on the mixing ratio m (see Eq. 15).

W3 & Q3

Indeed, real-world DNN contain layers with nonlinear activation where matrix multiplications cannot be directly applied. To circumvent this complication, we adopt unconstrained feature model (UFM, Eq. 4), where the post-activation representations are treated as free, trainable variables. This widely adopted simplification have been proven effective and necessary to advance the theoretical study of DL, especially for works on neural collapse and repr. dynamics (Zhu et. al., 2021 NeurIPS (ref. 34)). It allows us to reduce DL to an optimization problem between the last two layers, enabling analytical investigation of repr. dynamics. Finally, we have justified this with numerical experiments of both toy models and real-world datasets.

As the reviewer points out, nonlinearities—including BatchNorm, residual scaling, and the choice of optimizer—can all affect the learned representation. While our theoretical framework does not explicitly model these, we did conduct empirical validations under these settings, suggesting the simplified approach still works in the real-world setting. Specifically:

• CIFAR-100 benchmark uses both BatchNorm and ResNet, thereby incorporating BN and skip connections.
• On ImageNet, we test our method under various optimizer, including SGD, Adam, and AdamW.

W4 & Q5

In our setting, we consider a continual learning (CL) scenario where for each new task $T$, the two units in the final-layer are reset to zeros. It could be said that these are new labels (the later case), but actually they are still $[1, 0]$ or $[0, 1]$ in the end (the former case). However, we want to emphasize that the notation $k$ does not concern with the labels, rather, they refer to the input distributions associated with the class. Note that in our CL setups, we allow the samples from a class to reappear and participate in multiple tasks. These scenarios occurs frequently in real-world CL when the definition of a single concept is learnt gradually against other concepts to form the map of knowledge, it is also frequently encountered in RL when policy rules changes under different context while the object class remains the same.

Therefore, what's important is that any given pair of input samples of the same class $k$ share the same label in each task regardless of the vector being $[1, 0]$ or $[0, 1]$. This ensures their representations are consistently driven toward alignment and away from samples of other classes.

In conclusion, the use of $\lambda_0^{k,k}$ and $\lambda_{trained,T}^{k,k}$ is well justified regardless of label encoding. The differences in labels across tasks primarily affect the classifier weights, but not the underlying repr. structure in the feature space.

W5, W6 & Q6 Presenting FTLE curves for other methods would indeed strengthen our claims. However, due to the NeurIPS rebuttal format restrictions and character limitations, we only report the FTLE distribution statistics for baseline methods in the table below. Results for other methods are also available and can be provided upon request.

Table: FTLE Distribution Across Tasks for Different Methods (ConvNet on Continual ImageNet)

As shown above, methods that aim to address LoP also maintain FTLE within a healthy range, albeit with different mechanisms. In contrast, baselines exhibit significantly lower(higher) FTLE values, indicating repr. collapse and expansion. Theoretically, we've also demonstrated with theorem 3.1.2 that extreme FTLE values characterize the collapse and over-stretch of repr. space, which would lead to LoP. Although it is hard to rigorously prove that a system will not experience LoP at all when FTLE looks health, we consider these evidences to be enough to claim that such events are highly unlikely.

W7

We note that the smoothing effect of mixup-style augmentations align well with our theoretical framework, which highlights the importance of maintaining a well-conditioned (avoiding extreme FTLE values) repr. space during continual learning, but it could be overshadowed by the effect of sample diversity. Mixup promotes smooth transitions in the repr. space by interpolation of intra-class samples and labels, which can mitigate Type-2 LoP. However, mixup retains unchanged intra-class label, i.e, no smoothing, which limits its ability to prevent Type-1 LoP. Therefore, to empirically demonstrate that most of the performance improvement from G-mixup comes from mitigating LoP, rather than from the increased sample diversity, we restrict both mixup and G-mixup to intra-class sample pairs, thereby removing their inter-class smoothing effects and equalizing sample diversity. Whereas, G-Mixup still employs label enhancement, which systematically varies intra-class labels (Eq. 15), offering smoothing for the intra-class repr. space. As shown in the table below, mixup rapidly goes into type 1 LoP, whereas G-Mixup consistently mitigate type 1 LoP, supporting continual learning.

W8 & Q7

Indeed, the stability–plasticity dilemma has been a central topic in the CL community. However, our study is positioned within the emerging line of research that focuses on plasticity loss—a perspective that emphasizes the opposite concern: maintaining the network's ability to learn new tasks effectively.

For a typical LoP benchmarks like Continual ImageNet which involves up to 5,000 tasks, backward consistency cannot be guaranteed. E.g, methods like CBP explicitly reset parts of the network at each step, fully embracing the loss of old task performance in favor of forward generalization. A related example is in reinforcement learning, where preserving value functions learned under outdated policies can hinder new learnings.

That said, we agree with the reviewer that empirical measurement of stability is valuable. To this end, we have evaluated our method using the Forgetting Measure, and include the results in the following table:

ConvNet on Continual ImageNet

SmallConvNet on Continual ImageNet

Q6 Although our current theoretical analysis is primarily developed for binary classification tasks to ensure clarity and analytical tractability, we have empirically validated our approach on the Class-Incremental CIFAR-100 benchmark (see L299), where the number of classes increases to 100. We think that there are no fundamental obstacles to extending our framework to the multi-class classification setting.

Q8 We appreciate the reviewer’s emphasis on hyperparameter robustness, which is indeed a crucial aspect for practical deployment. To examine this, we conducted additional experiments using both SmallConvNet and ConvNet on Continual ImageNet, exploring the amplification factor $M \in [0.1, 0.3]$ and Beta prior $\in [0.5, 1.5]$. Results show only mild performance variation: SmallConvNet ranged from 0.875 to 0.886, and ConvNet from 0.895 to 0.904, indicating that G-Mixup is reasonably robust without fine-tuning.

We sincerely thank the reviewer for their thorough reading, thoughtful feedback, and constructive suggestions. We deeply appreciate the time and effort the reviewer devoted to evaluating our work. We have carefully addressed the reviewer’s comments in the following rebuttal, and we hope our clarifications successfully resolve the raised concerns.

Weakness 1 In our work, FTLE is introduced only to quantify rather than to understand how the geometry of representation evolves during continual learning. Therefore, we'd like to argue that this is not a weakness of our work since our main contributions and emphases have been on the understanding of how the geometry of representational space evolves during continual learning, rather than FTLE itself.

That said, we still want to justify the application of FTLE in our work. FTLE is a widely adopted metric used in physics for describing the spatio-temporal dynamics of various systems, and was first introduced to deep learning analogously by Storm et. al., 2024 (ref 27) to describe how distance in the representational space relates to the distance in the input space. Here, we further develop a theoretical framework utilizing this metric, specifically for the dynamical characterization of the local geometry of representational space during continual learning. It has successfully described how the distance between representations of same-class samples and different-class samples evolves as learning continues. Therefore, we think it is fair to say that FTLE is very essential to our work as a quantifying tool.

Weakness 2 We thank the reviewer for this insightful suggestion, which indeed helps strengthen the connection between the theoretical framework and empirical observations in our study.

We address this concern from both an intuitive and an empirical perspective:

1. Theoretical Intuition

The penultimate layer representation $h(x)$ is fed into a linear classifier to perform final prediction. For the linear classifier to perform well, representations must be linearly separable. Therefore:

• In compressing regions of representational space with lower effective dimensions (prone to Type 1 LoP), if representations from different classes are mapped to nearby or overlapping points (i.e., the compression collapses multiple classes into the same representation space), linear classification becomes impossible, leading to accuracy degradation.

• In stretching regions of representational space of higher effective dimensions (prone to Type 2 LoP), small input perturbations are mapped to significantly different representations. When same-class samples are send apart in various directions, the local geometry become distorted, making it difficult for the classifier to assign consistent predictions, again leading to lower accuracy.

These effects suggest that regions with extreme FTLE values (on either end) correspond to representational spaces where the local geometry make same-class alignment and inter-class separation progressively infeasible, both are detrimental to classification performance.<!-- Due to time and character limit, we are unable to provide a detailed proof, however we believe a rigorous rewrite of these intuitive understandings into mathematical proofs is possible. -->

2. Empirical Verification

To further substantiate this link, we visualize classification accuracy over training time for samples grouped by FTLE ranges. Initially, when LoP has not emerged, FTLE values are small and classification accuracy is high. As LoP progresses, FTLE values become either too large or too small (strong stretching/compression), and classification accuracy for these regions drops significantly. These findings empirically support the theoretical link between FTLE, local representational distortion, and classification accuracy.

Accuracy and Sample Distribution within Different FTLE Ranges across Tasks

We thank the reviewer again for pointing out this gap, and we will include these new results in the appendix to clarify this connection between LoP dynamics and performance impact.

As for the other empirical support suggestion: "if one could empirically show that when the sample follows in the stretching/compressing region, and will cause the FTLE to increase or decrease", we believe there's a misunderstanding. As proved by theorem 3.1.2, same-class samples decrease the FTLE values for the region of interest while different-class samples increase them. This happens regardless of whether the samples are following the stretched or compressed region (through the force of back-propagation). Then based on which pair of samples (same-class or different-class) are put into these regions, the samples become less likely to be correctly classified, as supported by the theoretical intuition and empirical support above.

Weakness 3 We thank the reviewer for this highly insightful suggestion. Indeed, when initially designing our approach, we also considered directly penalizing the FTLE in the loss function to preserve plasticity. However, we ultimately opted for G-Mixup due to practical feasibility.

Directly computing FTLE involves evaluating the local Jacobian of the feature extractor and estimating its maximum singular value, which is computationally expensive and may introduce gradient stability issues during training. This may make it hard and costly to use FTLE as a regularizer in large-scale settings like ImageNet.

That said, inspired by the reviewer’s suggestion, we implemented a proof-of-concept FTLE regularization experiment in a toy model. Specifically, we approximated the FTLE for each sample using the following procedure:

Let $h(x)$ denote the feature representation of input $x$, and let $J = \nabla_x h(x)$ be the local Jacobian. We approximate the FTLE using power iteration:

\\begin{aligned} &\\text{Initialize } v \\sim \\mathcal{N}(0, I), \\quad v \\leftarrow \\frac{v}{\\|v\\|} \\ &\\text{Iteratively update:} \\quad v \\leftarrow \\frac{J^\\top (Jv)}{\\|J^\\top (Jv)\\|} \\ &\\text{Then:} \\quad \\text{FTLE}(x) = \\log \\left( \\| Jv \\| + \\varepsilon \\right) \\end{aligned}

We then add the following regularization term to the loss:

$$L_{\\text{FTLE}} = \\lambda_{\\text{FTLE}} \\cdot \\left( \\text{FTLE}(x) \\right)^2$$

This term penalizes large deviations of the FTLE from 0, encouraging local smoothness in representation dynamics.
In practice, to reduce computational cost, we used a 20-layer network in the toy experiment, which tends to collapse easily under the setup of Toy Experiment 1 as shown in Figure 2 of the main text. In contrast, when FTLE regularization is applied, the network is able to continually adapt to new tasks without collapsing. To make the FTLE computation more efficient, we performed 5 iterations per sample during the power iteration procedure. Additionally, we employed a linearly increasing FTLE regularization coefficient, set as $ \lambda_{\text{FTLE}} = 0.05 \times \text{epoch} $ (per task), to balance task learning and the structure of the representational space.

As shown in the table below, the FTLE regularization effectively mitigates LoP in toy settings. We thank the reviewer for the insightful suggestion and will discuss this alternative approach in the revised main text.

Classification Accuracy over Task Ranges

We thank the reviewer for this valuable suggestion. We will include these results in the appendix and highlight the discussion in the final version. Exploring scalable and efficient FTLE-regularization is an exciting direction for future work.

We sincerely thank the reviewer for the constructive feedbacks. We would like to start by an overall reply to provide general clarifications about the issues raised here, given that some of the central idea of our work might have not come through. Then, we will follow with responses to the specific concerns.

Overall reply

This paper uncovered the fundamental mechanisms underlying LoP in continual learning (CL), i.e., the compressing/stretching of the representational(repr.) space. This primary contribution is a theoretically grounded diagnosis of the emergence of two types of LoP though a-posteriori. It is based on how CL compress and over-stretch the repr. space by repeated exposure to overlapped input distribution from intra-class and inter-class samples, respectively (as proved in theorem 3.1.2). The dynamics of the manipulation of repr. space geometry is characterized by FTLE. Recently, LoP has been associated with various observations, such as gradient vanishing or an increase in dead neurons. However, as pointed out in Lyle et al., (2024), these singular phenomena are insufficient to fully explain LoP on their own. This points to the lack of unified theoretical understanding of LoP since those phenomena are not the direct causes. Despite this, numerous methods have been proposed to mitigate LoP. Their strategies can broadly be classified into two categories, as the following.

One approach targets specific manifestations of LoP, such as large weight norms. For instance, Dohare et al. (2024) investigated the use of L2 regularization to constrain weight norms and found that it can partially mitigate LoP. Our theoretical framework provides a deeper understanding of how these measures work: the network used in their study is relatively wide and hence susceptible to Type 2 LoP. In such cases, L2 regularization prevents excessively large weights, thereby mitigating Type 2 LoP. However, in networks with initially lower repr. dimensions (narrow networks), which are more prone to Type 1 LoP, L2 regularization can actually impair performance. (We want to emphasize here that the size of repr. dimensionality does not equal to the network capacity, as initially large/small repr. space can still be easily stretched or compressed extremely during CL.)

Another approach builds on the observation that plasticity is highest at initialization. Techniques such as continual weight re-init (e.g., CBP) or regularization toward initial weights (e.g., L2-init) aim to keep the network in a more plastic regime. While effective in alleviating LoP, these methods suffer from several drawbacks: weight re-init can erase previous learning, while regularization toward initial weights can constrain the network's expressivity. Our work, in contrast, mitigate LoP based on the theoretical understanding that the plasticity can come from maintaining a reasonably sized (for Type-1 to not collapse) and smoothed (for Type-2 to not become chaotic) repr. space without going back to the initialized state, thus avoiding the drawbacks.

(relates to )While we appreciate the reviewer’s insightful attempt to relate Type 1 and Type 2 LoP to limited capacity and overfitting respectively, we respectfully note that this interpretation does not fully capture the mechanisms we aim to describe. This is precisely why we introduced the FTLE framework to characterize the dynamics of repr. space rather than relying on conventional notions of capacity or overfitting. We appreciate the chance to reiterate the essence of our work here.

Our theory demonstrates how repr. geometry crucially drive the network dynamics into two regimes. On the higher level, the dimensionality of the repr. space governs the general direction of the dynamics, i.e., as the dimensionality (which can be set by network width only initially) increases, the collapse of intra-class space (related to type-1 LoP) weakens and the chaotic expansion of inter-class space (related to type-2 LoP) dominates. On the lower level, only the local geometry can determines , effectively measured by FTLE. In addition to the theory described in the main text, we support this claim with empirical evidence shown in the following table:

In the following two tables, we report the FTLE stats for networks of varying widths. In Table A1, input samples are uniformly distributed, conformed to our theoretical assumption. As network width increases, the minimum FTLE quickly approaches zero much more quickly than its imbalanced counterpart.

However, when the input distribution is highly imbalanced—as shown in Table A2, where one class dominates the majority of the space—the intra-class representation vectors become less orthogonal and more prone to collapse, demonstrated by more negative min FTLE vs. the ones for uniform input distribution.

• Table A1: FTLE statistics under uniform input distribution

• Table A2: FTLE statistics under class-imbalanced input distribution

Beyond this simplified toy model setting, data distributions are structured, various activation functions and optimizer introduces additional constraints in real training scenarios. These factors modulate the global and local dynamics of the collapse and expansion of the repr. space, but do not alter the general prediction from our theory:

• In low-dim spaces, repr. space collapses (Type 1 LoP), though expansion can still occur.
• In high-dim spaces, repr. space expands chaotically (Type 2 LoP), though collapse may still be present.

Our empirical observations also align with this: we never observed pure Type 2 LoP in low-dimensional networks, nor pure Type 1 LoP in high-dimensional ones. In Figure 4 of the main text (detailed experiment settings can be found in Appendix E), we chose two widely used activation functions—ReLU (scale-invariant) and tanh (saturating)—to illustrate all four theoretically predicted regimes. One can observe that the narrower ReLU networks in Figure 4C exhibit significantly faster onset of Type 1 LoP than the wider network in Figure 4D, again consistent with our theoretical expectations.

Therefore, Type 1 and Type 2 LoP are the emergent behavior driven by the evolving geometry of the repr. space during CL:

• Type-1 LoP arises from repr. collapse. While narrower networks are more susceptible to such collapse, wider networks tend to collapse more slowly, though they may still exhibit collapse depending on various factors.
• Network capacity alone cannot explain Type 1 LoP. Increasing capacity (width) may slow down collapse but cannot eliminate it.

Regarding overfitting, it typically refers to excessively tailoring a model to a specific dataset, impairing generalization. While overfitting can account for some aspects of Type 2 LoP, it is not the primary cause in our CL setup. In our setting, the model learns 5000 tasks of comparable difficulty, and performance degradation emerges gradually, unlike the abrupt generalization loss associated with overfitting.

Generalization in DNN is supported by mapping similar inputs to similar representations. At initialization, FTLE values are approximately zero, indicating distance-preserving mappings conducive to generalization. However, as training proceeds, high-FTLE boundaries proliferate the repr. space, stretching and distorting these local mappings and degrading generalization. This happens regardless even when the learnt boundaries are perfectly generalizable for standalone tasks. Therefore, in contrast to overfitting—which is characterized by overly complex boundaries—Type 2 LoP reflects widespread instability caused by the aggregation of the stretching effect by more and more learnt boundaries (regardless of their generalizability) that finally distort the repr. space in various direction causing chaotic behavior.

We deeply appreciate the opportunity to engage in this discussion. The reviewer’s thoughtful comments helped sharpen our own understanding, which we will integrate into the final manuscript to improve the clarity and impact of the work.

Regarding specific concerns

• W1: With the overall reply above, we'd like to argue that the provided theory offer much more than a-posteriori diagnosis of LoP, and it reaches the direct cause of the two types of LoP, which also prompt the design of G-mixup that mitigate LoP in advance.

• W2 & Q1: Our core motivation is to disentangle the mechanisms behind LoP. We deliberately selected toy models that isolate Type 1 and Type 2 LoP. Therefore, we used a narrower network with 10 hidden units which is prone to Type 1 LoP vs. a wider network of 300 hidden units which is prone to Type 2 LoP. These choices of setups serve as archetypes to illustrate the distinct underlying mechanisms. Networks of all widths follow our theory; what differs is the dominance and the onset speed of each type of LoP.

• Q2 & W1 & 3: As discussed, collapse/expansion dynamics depend on activation function, data distribution, and training details. Figure 4 employs ReLU (Figure 4a,d) and tanh (Figure 4b,c) to depict typical real-world cases where both types can co-occur.

• Q3 & W4: While on the surface-level similarities exist between Type 1/2 LoP and capacity/overfitting, the former reflect dynamic changes in representation during CL, instead static model properties that governs single task learning.

We are willing to further clarify during discussion.