Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters

Thanks for your constructive feedback! We address your questions and concerns as follows.

Q1. The current analysis focuses on binary classification with two-layer CNNs. Could you elaborate on how the framework might extend to multi-class classification or deeper architectures? A clear discussion on this could enhance the practical impact and generalizability of your work.

While our current analysis focuses on two binary tasks for clarity and tractability, the angle-based framework naturally extends to multi-class and multi-task settings. Specifically, complex configurations can be decomposed into pairwise class-level interactions across tasks, with angular relationships capturing the core learning dynamics. This forms the basis for our analysis. We plan to extend our theory by studying how the accumulation of such pairwise interactions drives forgetting.

Additionally, Jiang et al. recently employed a feature learning theory based on signal-noise decomposition to study benign overfitting in Vision Transformers [1]. Our exploration of continual learning in CNNs through this lens may serve as a foundation for extending such analysis to more complex models like Transformers.

Q2. The mid-angle sampling strategy appears promising. Can you provide additional insights on how sensitive the method is to the choice of cosine similarity thresholds? Is there any theoretical or empirical guidance on selecting these thresholds optimally?

In fact, our mid-angle sampling strategy does not require threshold selection. The experiment follows the classical iCaRL framework for CL based on CNNs [2], which consists of a feature extractor and a classification layer. Since iCaRL allows only a fixed number of replay samples to be stored, our Mid-angle sampling strategy selects the most intermediate examples by first sorting all examples within each class based on their cosine similarity to the class prototype, and then selecting those closest to the median. We conduct experiments on the CIFAR100-5 (5 tasks with 20 classes each) and CIFAR100-10 benchmarks. As shown in Table 1, our mid-angle sampling outperforms herding—a nontrivial result given that herding is a commonly used sampling method in CL with replay.

Table 1: Experimental Results with std and Average Forgetting on CIFAR100.

Q3. Your proofs rely on a signal-noise decomposition of network weights. Could you offer more intuition or illustrative examples (e.g., visualizations) that clarify how this decomposition relates to neuron behavior and forgetting in continual learning?

We first provide an intuitive explanation of the relationship between signal-noise decomposition and neuron behavior. In our setting, the signal vector $\mu$ is orthogonal to the noise $\xi$, forming a basis for a plane. After training, the CNN weights evolve as $\mathbf{w}_{j,r} = j\gamma\mu + \rho\xi$, where $\gamma \gg \rho$, indicating that the weights grow predominantly along the signal direction. This suggests that the network has effectively learned the signal. In multi-task scenarios, the weights adjust according to the signal vectors of tasks.

Regarding forgetting in CL, we show that harmful forgetting mainly arises in the obtuse-angle case. As stated in Lemma 4.3 (page 5), if $\sum\_{r = 1}^{m}[\sigma(\langle\mathbf{w}\_{y\_{1},r}^{(T\_{2},t\_{end})}, y\_{1}\mu_{1}\rangle)-\sum\_{r = 1}^{m}\sigma(\langle\mathbf{w}\_{-y\_{1},r}^{(T\_{2},t\_{end})}, y\_{1}\mu\_{1}\rangle)] \geq C\_{3}$, then the CNN will achieve benign forgetting. We further derive that $\sum\_{r = 1}^{m}\sigma(\langle\mathbf{w}\_{y\_{1},r}^{(T\_{2},t_{end})}, y\_{1}\mu\_{1}\rangle)$ and $\sum\_{r = 1}^{m}\sigma(\langle\mathbf{w}\_{-y\_{1},r}^{(T\_{2},t\_{end})}, y\_{1}\mu\_{1}\rangle)$ can be characterized by $\Theta(m\overline{\gamma(\mu\_{1})\_{y\_{1}}}(1 - \cos^{2}\theta\_{1,2})^{q})$ and $\Theta(m\overline{\gamma(\mu\_{2})\_{-y\_{1}}}\frac{||\mu\_{1}||\_{2}^{q}(-\cos\theta\_{1,2})^{q}}{||\mu \_{2}||\_{2}^{q}})$ respectively. Then we obtain the angle range corresponding to benign forgetting; the analysis for harmful forgetting follows similarly.

References

[1] Jiang J, et al. Unveil benign overfitting for transformer in vision: Training dynamics, convergence, and generalization. In NeurIPS 2024

[2] Rebuffi S A, et al. icarl: Incremental classifier and representation learning. In CVPR 2017

Thanks for your constructive feedback! We address your questions and concerns as follows.

Q1. Your theory focuses primarily on two binary tasks. Could you outline how you would expect the angle-based framework and replay analysis to extend if there were multiple sequential tasks or multi-class tasks for each session?

We validate the relationship between forgetting and angle in multi-task settings using the synthetic dataset. We conduct two sets of experiments, each consisting of three tasks. In Experiment E1, Task 3 forms angles $\theta_{1,3} = 150°$ and $\theta_{2,3} = 100°$ with Tasks 1 and 2, respectively. In Experiment E2, these angles are $\theta_{1,3} = 170°$ and $\theta_{2,3} = 80°$. Let $Acc_{1}$ and $Acc_{2}$ denote the accuracy on Tasks 1 and 2, respectively, after learning all three tasks. As shown in Table 1, the conclusion consistently holds.

Table 1: Experimental Results in multi-task settings on the synthetic dataset.

While our current analysis focuses on two binary tasks for clarity and tractability, the angle-based framework naturally extends to multi-class and multi-task settings. Specifically, complex configurations can be decomposed into pairwise class-level interactions across tasks, with angular relationships capturing the core learning dynamics. This forms the basis for our analysis. We plan to extend our theory by studying how the accumulation of such pairwise interactions drives forgetting and how subproblems influence one another in more general settings.

Q2. Do you see a straightforward way to relax the “signal + noise” assumption to capture more varied real-world data distributions? Or do you view the current model as primarily a stepping-stone for further exploration?

The signal-noise data model takes inspiration from image data, where the inputs are composed of various patches, and only certain patches are relevant to the class label of the image. This model has been widely adopted in recent theoretical studies, including our references [1,2]. Furthermore, the two-patch setting can be extended to a multi-patch model by techniques similar to those in Allen-Zhu et al. (2020) [3].

Jiang et al. recently employed a feature learning theory based on signal-noise decomposition to study benign overfitting in Vision Transformers [4]. Our exploration of continual learning in CNNs through this lens may serve as a foundation for extending such analysis to more complex models like Transformers.

Q3. How significant is the computational cost of measuring and comparing angles (or proxies for them) in mid-angle sampling, particularly for large networks or large-scale datasets?

We primarily implement the mid-angle sampling strategy by computing cosine similarity, whose computational cost should be comparable to the original herding strategy adopted in the iCaRL framework (which relies on Euclidean distance) [5].

Typos and Minor Clarifications

We appreciate the thoroughness of the review and the opportunity to improve the accuracy and professionalism of our paper. We commit to fixing all minor writing issues and ensuring consistent notation to further improve clarity.

References

[1] Cao, Y., et al. Benign overfitting in two-layer convolutional neural networks. In NeurIPS 2022

[2] Kou, Y., et al. Benign overfitting in two-layer ReLU convolutional neural networks. In ICML 2023

[3] Allen-Zhu, Z., et al. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv, 2020

[4] Jiang J, et al. Unveil benign overfitting for transformer in vision: Training dynamics, convergence, and generalization. In NeurIPS 2024

[5] Rebuffi S A, et al. icarl: Incremental classifier and representation learning. In CVPR 2017

Thanks for your constructive feedback! We address your questions and concerns as follows.

Is the grey area a significant forgetting region ?

The grey area is a region for uncertainty, either harmful or benign forgetting, ensuring that our claims remain rigorous. The yellow area is for harmful forgetting.

Reporting the std and Average Forgetting in Table 1.

Due to space limitations, the answer is provided in our response to Reviewer #4 (z4gT), Comment Q2.

Experimentally validating Theorem 3.2 and 3.3.

As Reviewer #2 (qzcp) noted, "the authors present a coherent chain of reasoning plus experiments that affirm their principal angle-based explanations for forgetting". Figure 2 shows that benign forgetting leads to near-zero error, while harmful forgetting approaches one, validating our Theorem 3.2 and 3.3. Figure 3 confirms both on MNIST. The second plot in Figure 2 shows that our over-parameterized model avoids the lazy training regime, with significant weight increase in the signal direction.

I may have misunderstood the rightmost plot in Figure 2.

Blue indicates the maximum value, and green indicates the second-largest value, rather than a single continuous line. The intersection of the blue and green lines shows where the maximum value shifts between the two variables.

Why is x_k subdivided into two vectors ?

Due to space limitations, the answer is provided in our response to Reviewer #2 (qzcp), Comment Q2.

What are the assumptions about the distribution D_k ?

We assume $\xi_k \sim N\left(0, \sigma_{p_k}^2 \cdot \left(I - U (U^T U)^{-1} U^T\right)\right)$. The label $y_k$ is a Rademacher random variable. One of $x^{(1)}_k$ and $x^{(2)}_k$ is $y_k\cdot\mu_k$, the other $\xi_k$, with $(x_k, y_k)\sim D_k$.

Explain the choice of the logistic loss. Is it restrictive ?

Logistic loss is a special case of cross-entropy loss for binary classification. We use $L_{CE}=-\frac{1}{n}\sum_{i = 1}^{n}[y_i\log(\hat{y}_i)+(1- y_i)\log(1-\hat{y}_i)]$, where $\hat{y}_i$ is softmax-normalized. After softmax calculations in $L\_{CE}$, we get the logistic loss used in our analysis.

Theorem 3.1 : is the upper bound on cos theta a typo, shouldn't it be 1 ?

Theorem 3.1 does not appear in the paper. If the intended reference is Theorem 3.2 or 3.3, the range should be $1 \geq \cos\theta_{1,2} \geq 0$ for Theorem 3.2 and $-\frac{1+C_2}{2} \leq \cos\theta_{1,2} \leq 1$ for 3.3.

Does the network fall in the lazy regime ?

Our approach avoids the lazy training regime by using smaller initialization, in contrast to the NTK setting. This allows the weights to move significantly and learn the signal ($\gamma_{j,r}^{(t)}$), rather than staying near initialization. Large initialization typically induces NTK-like behavior with minimal updates, while small initialization enables meaningful parameter growth along the signal direction, thus avoiding lazy regime.

Why is the $Relu^q$ activation chosen ? Is it restrictive or does it translate to commonly used activations ?

Polynomial RELU can speed up both signal learning and noise memorization, to further boost the gap between them. Our theoretical framework can be extensible to RELU activation by techniques similar to those by Kou et al. (2023) [1].

Does the analysis apply to non convolutional models ?

The answer is provided in our response to Reviewer #2 (qzcp), Comment Q2.

Why the network is a CNN ? What happens if the convolutions are not split between the signal and noise ?

Our network structure, a common choice for theoretical analysis (including our references [1,2]), retains the core features of a CNN. We use a dual-channel input model $\mathbf{x}=[\mathbf{x}^{(1)},\mathbf{x}^{(2)}]$ with shared weights across channels. The output is: $f=\frac{1}{m}\sum_{r = 1}^m\left[\sigma(\mathbf{w}\_{+1,r}^\top\mathbf{x}^{(1)})+\sigma(\mathbf{w}\_{+1,r}^\top\mathbf{x}^{(2)})\right]-\frac{1}{m}\sum_{r = 1}^m\left[\sigma(\mathbf{w}\_{-1,r}^\top\mathbf{x}^{(1)})+\sigma(\mathbf{w}\_{-1,r}^\top\mathbf{x}^{(2)})\right]$. Here, $m$ is the number of filters, $\sigma(z)=(\max\{0,z\})^q$ ($q > 2$), and $\mathbf{w}\_{j,r}$ are filter weights. The structure approximates $\sigma(\mathbf{w}\_{j,r}^\top(\mathbf{x}^{(1)}+\mathbf{x}^{(2)}))$, which is commomly used in feature learning theory [1,2]. Furthermore, We can extend the two-patch data to multi-patch data model similar to Allen-Zhu et al. (2020) [3].

In the multitask setup, is the final accuracy impacted by the angle ?

The answer is provided in our response to Reviewer #2 (qzcp), Comment Q1.

References

[1] Kou, Y., et al. Benign overfitting in two-layer ReLU convolutional neural networks. In ICML 2023

[2] Cao, Y., et al. Benign overfitting in two-layer convolutional neural networks. In NeurIPS 2022

[3] Allen-Zhu, Z., et al. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv, 2020

Thanks for your constructive feedback! We address your questions as follows.

Q1. How the mid-angle sampling strategy relates to the theoretical theorems

We sincerely appreciate the opportunity to clarify the connection between our theoretical findings and mid-angle sampling. Our theoretical results show that smaller angles between task signal vectors lead to benign forgetting, while larger angles cause harmful forgetting. In practice, we treat each class prototype—the mean feature of its examples—as the signal vector.

We focus on the case where the angle between task prototypes is obtuse, as harmful forgetting arises only in this setting. If a sample’s feature forms a larger angle with its own prototype, it tends to form a smaller angle with the second task’s prototype, likely falling within the benign forgetting range in standard CL (i.e., it can be remembered without replay). Conversely, if the angle with its own prototype is smaller, the angle with the second task’s prototype may be larger—possibly beyond the benign range under CL with replay (i.e., it will be forgotten despite replay).

In contrast, samples with mid-range angles are more likely to fall outside the benign range of standard CL but within that of replay-based CL—making them the most effective candidates for replay. Thus, mid-angle sampling offers a more efficient and targeted replay strategy.

Q2. Discuss the choice for a single head setup.

We adopt the single-head setting primarily to facilitate theoretical analysis. However, our theoretical framework can extend to the multi-head setting. In fact, our final forgetting results are derived by analyzing the behavior of individual neurons, through which we characterize the angle-dependent antagonism between tasks. While the multi-head setup may increase the number of neurons involved in learning the feature of the second task, the core antagonism between tasks (driven by the angle) still persists.

Moreover, We emphasize that slight differences between theoretical settings and practical setups are common to ensure analytical robustness. As Reviewer #2 (qzcp) noted, "For a primarily theoretical analysis, using a simplified architecture that captures core nonlinear effects (rather than a purely linear or kernelized model) is a rational choice." In this regard, our framework goes beyond linear and NTK-based models, making it more aligned with practical scenarios.

Q3. The noise that is orthogonal to all signal vectors seems restrictive.

We adopt the orthogonality assumption primarily to simplify the proof and reduce the length of the manuscript. In fact, by techniques similar to those used by Kou et al. (2023) [1], we can extend our theoreical results to non-orthogonality case.

The signal and noise vector are processed separately by the network.

This setting, commonly used in analyses of feature learning theory [1,2], mainly serves to simplify the proof. In practice, we can extend to multi-patch model using techniques similar to those of Allen-Zhu et al. (2020) [3].

The definition of Forgetting is weird.

As you noted, the approximation using $L_{D_1}^{0-1} (W^{(T_2)})$ has limited impact since $L_{D_1}^{0-1} (W^{(T_1)})$ approaches zero. We commit to clarifying this to avoid any misunderstanding.

Experiment 3 lacks details with mid-angle sampling a slightly positive effect

The experiment follows the classical iCaRL framework for CL based on CNNs [4], which consists of a feature extractor and a classification layer. Since iCaRL allows only a fixed number of replay samples to be stored, our Mid-angle sampling strategy selects the most intermediate examples by first sorting all examples within each class based on their cosine similarity to the class prototype, and then selecting those closest to the median. As shown in Table 1 in our response to Reviewer #4 (z4gT), Comment Q2, our mid-angle sampling outperforms herding—a nontrivial result given that herding is a widely used sampling method in CL.

Condescending to refer to methods such as EWC as "empirical methods".

Here, we follow the description from Ding et al. (2024) [5]. We will remove the misleading term "empirical methods" and revise the description for clarity.

The structure of the paper should be revised.

Thanks for your suggestion regarding the structure of our manuscript. We will move Section 2 before 1.2 to improve readability.

References

[1] Kou, Y., et al. Benign overfitting in two-layer ReLU convolutional neural networks. In ICML 2023

[2] Cao, Y., et al. Benign overfitting in two-layer convolutional neural networks. In NeurIPS 2022

[3] Allen-Zhu, Z., et al. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv, 2020

[4] Rebuffi S A, et al. icarl: Incremental classifier and representation learning. In CVPR 2017

[5] Ding, M., et al. Understanding forgetting in continual learning with linear regression. arXiv, 2024