Plastic Learning with Deep Fourier Features

Thank you for your insightful comments, we particularly appreciate you referring to the paper “as a breath of fresh air.” The updated submission includes several new results and clarifying sentences (all changes and new content is in blue, except for minor typos which have been corrected). We address your comments below, Weaknesses:

A discussion about catastrophic forgetting--the most prominent challenge of continual learning--is largely absent from the paper. To be fair, trainability/plasticity is also often absent from papers focusing on catastrophic forgetting, and it’s good to have a deeper exploration on trainability given the lesser attention to it. [...] worst case a model can just be re-initialized and retrained from scratch.

Forgetting and plasticity: As you mention, this paper is specifically about the more recently established loss of plasticity problem. We have added some additional discussion around forgetting in Section 1 (see “on forgetting:” in the shared reply).

On reinit: You raise a good point. Note that we are considering a much harder problem setting where the learner has no information about a task change (see “on task-boundary:”). Thus, a reinitialization method is not applicable because knowing when to reinitialize requires knowing when the task changes. Notice that this is in fact one of the most appealing aspects of our proposed method: our results using deep Fourier features show that expensive reinitialization is not necessary to sustain good performance, this additional information is not even required!

A similar “why not reinit” argument can be applied to the catastrophic forgetting setting, and continual learning more generally: a strong but computationally infeasible baseline could store all past data and retrain a new network on every new datapoint (e.g., see [1]). This has motivated recent works on memory and compute limitations as part of the continual learning problem formulation.

[1] Prabhu et al. (2020). Gdumb: A simple approach that questions our progress in continual learning. ECCV.

W2. [...] Theorem 2 assumes a deep diagonal linear network. [...] How much do these insights shed here transfer to the more general deep linear network setting?

You raise a good point, and although our theory was developed in the same settings as other recent papers that we referenced, we share the reviewer’s concerns. This is exactly why we provide, in Figure 2, experimental evidence showing that our trainability result holds for general deep linear networks. The theoretical analysis we can perform is limited by the existing techniques in the field (and it was beyond the scope of our paper to develop a theoretical framework for this), and that’s exactly the role of the empirical results in our paper: to empirically validate our insights when the assumptions we make for our theoretical results do not hold. If the theory was able to perfectly capture our empirical methodology, we wouldn’t necessarily need to run experiments, as the theoretical results would support themselves.

W3. Double width vs half representation: [...] The experiments however are fair in this regard however, as they choose to incur the half effective penalty rather than unfairly double model width.

We are glad that the reviewer acknowledges that we are actually giving the baselines an advantage relative to the half-width of deep Fourier features. However, we wonder why the reviewer considers this to be a weakness. Please also see the new results in Appendix D.2. where we consider Wide Residual Networks to investigate the effect of width.

W4. Page Length:

Thanks for raising this point, we have moved some of the extended discussion on deep linear networks to the appendix so that the paper fits in 9 pages.

Line 419: Class incremental learning definition: This is not the traditional class incremental learning setup usually shown in most continual learning papers. Notably, it seems like addressing catastrophic forgetting is completely optional, as new tasks still contain the classes from the previous tasks. I would suggest naming this something else to distinguish it from the more common (and harder) setting.

We use the nomenclature introduced by [1], in which they say class-incremental learning involves a learning algorithm that must “learn to discriminate between a growing number of classes”. Also note that our class incremental experiment is equivalent to the class-incremental experiment in [2]. In both their experiments and in ours, each task corresponds to a distribution over all classes seen so far, which is well-represented by “a growing number of classes.” We have added a clarifying sentence to Section 5.1 to reflect this.

[1] Van de Ven et al. (2022). Three types of incremental learning. Nature Machine Intelligence (4).

[2] Dohare et al. (2024). Loss of plasticity in deep continual learning. Nature (632).

[3] Lyle et al. (2023). Understanding Plasticity in Neural Networks. ICML.

I thank the authors for the effort they put into their responses, including updating the paper (including getting under 9 pages) and adding experiments. I read through the other reviews when they were first released, as well as the authors’ responses.

Plastic Learning vs Re-Init: The authors’ response is helpful, and I’m a little more open to the value of studying plastic learning on its own now, but I still have some remaining doubts:

• The proposed method not requiring knowledge of task boundaries is a good point and a valid advantage. On the other hand, task-free continual learning methods [a,b] also deal with this problem, so there are existing solutions. Additionally, I wonder if even more basic solutions like outlier detection or simply monitoring for decreased training efficiency or lower accuracy would also solve the problem.
• I disagree that the “why not reinit” argument can be similarly applied to catastrophic forgetting. Most continual learning settings explicitly do not allow storing all the data from past tasks (the replay class of methods store a subset, but I personally view this as cheating—but I digress). Thus, the brute force solution you propose is generally not an allowable solution. In contrast, I see no reason why re-init is not a valid solution for plasticity; with pre-trained backbones (especially with recent advances in self-supervised learning or growing availability of “foundation” models), it may not even be that expensive to do.

This remains one of my largest concerns. Without a resolution, I’m hesitant on whether plasticity, without being coupled with catastrophic forgetting, is by itself a topic worth solving.

Strong assumptions: Fair, point taken. On the other hand, I would modify the text a bit to make it clear that these are strong assumptions, and that while the theory doesn’t prove that modern neural networks also behave similarly, you provide empirical results indicating that deep Fourier features might. For example, the paragraph right after Section 3 makes the connection between the theory and the more general case seem a little to easy.

Why is half-representation a weakness?: The experiments are fair, but that doesn’t change the fact that given a model size budget, this method halves the effective width of the representation. For the experiments in this paper, this didn’t turn out to be a problem, but in other settings, it could be. For example:

• I suspect this choice may be a problem for more complex data distributions or learning problems than CIFAR or tiny Imagenet, where the model is less over-parameterized and actually needs all the representational power it can get.
• Going in the opposite direction, my guess is that halving of the representation for the same number of parameters negatively impacts one’s ability to compress the model, which is often important when deploying a model in the real world.

I thus stand by my characterization that this is a weakness, but I will note that it didn’t factor that significantly in my score.

[a] Aljundi et al. “Task-Free Continual Learning”. CVPR 2019
[b] Lee et al. “A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning” ICLR 2020

We thank the reviewer for their comments, particularly praising the submission as well-written, with clear motivation, problem statement and theoretical analysis. We address your main concern below,

Nevertheless, the most critical issue for this manuscript is the small-scale experiments.

We appreciate the sentiment, as it is now common in machine learning papers to have bigger and bigger datasets. One thing that is easier to overlook, though, is the different empirical methodology used in continual learning research. We are not training a neural network once, but many many times across different tasks. Because of that, experiments in continual learning take much longer than those in more traditional supervised learning literature.

In any case, we share the reviewer's concern for scalability and generality of the proposed approach, and this is exactly why we evaluated our approach in all the major datasets used in the papers published in top-tier venues in the loss of plasticity literature, including Nature. These long training processes have been evaluated primarily using tiny-imagenet, CIFAR100 and CIFAR10. For example, CIFAR10 is used in [1], CIFAR10 and MNIST variants are used in [2,4], and more recently tiny-imagenet is used in [3,5]. The large performance differences in our experiments, particularly on tiny-ImageNet and across different wide residual networks, indicate that the scale of our experiments effectively showcases our contribution and validates the theoretical motivation we provided in Sections 3 and 4.

In contrast, there is no published work studying loss of plasticity on imagenet. Taking your estimate of 2 days for training to convergence on a single task, a single run using imagenet and the label noise problem that we considered would involve 20 days of training (due to training to convergence on 10 tasks). We have added additional results on another problem setting (Continual Imagenet, which involves binary classification of two imagenet classes, Appendix D.4), as well as additional architectures (Wide Residual Networks, Appendix D.2.)

[1] Lyle et al. (2023). Understanding Plasticity in Neural Networks. ICML.

[2] Dohare et al. (2024). Loss of plasticity in deep continual learning. Nature (632).

[3] Elsayed et al. (2024). Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning. ICLR.

[4] Galashov et al. (2024). Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset. NeurIPS.

[5] Lee et al. (2024). Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks. ICML.

We appreciate the reviewer’s thoughtful questions regarding our submission. Several suggestions have been used to add clarifying sentences (all changes and new content is in blue, except for minor typos which have been corrected). We address your comments below,

Although they compare the proposed method with ReLU-based architectures, it could include comparisons with other advanced techniques in continual learning to validate the advantages further.

We consider baselines of all prominent approaches for dealing with loss of plasticity, as described in lines 428-431. Could the reviewer clarify what they mean by “other advanced techniques”?

There is limited discussion on the computational and memory costs associated with the model compared to standard activation functions.

Thanks for bringing this up, we have added a sentence describing this in Section 4. The resnet that uses our proposed deep Fourier features has approximately half the computation and memory when compared to the vanilla resnet. This is because we use half the output width in each layer due to concatenating sine and cosine.

The authors claim that the proposed method focuses on the loss of plasticity, however, there seems to be something confusing about the experiments on label noise and pixel permutations (especially label noise).

The performance curves for the label noise experiments demonstrate that the baselines are unable to learn as effectively when trained on noisy labels that are progressively made into clean labels. This indicates loss of plasticity, where the plasticity present at initialization decreases on later tasks. We are happy to adjust the text accordingly to clarify any potential confusion, but it is unclear to us what is confusing right now.

The manuscript could benefit from more discussion of the overfitting problem.

If the reviewer refers to the fact that Fourier features can overfit, then this is a well-documented spectral bias, in which Fourier features can learn high-frequency features that are more prone to overfitting. We have added clarification on this point in Section 5.2

The manuscript could benefit from more practical insights or guidelines for implementing deep Fourier features.

We have added pseudocode in Appendix A.4. We would like to highlight that deep Fourier features are straightforward to implement. It requires i) concatenating the sine and cosine activation functions at every layer, and ii) that the layers have half their output width to accommodate the concatenation. We also added a sentence describing the implementation to Section 4.2.

More continual learning methods related to plasticity should be included, like [1,2]

We appreciate the references, but neither of these papers deal with the loss of plasticity phenomenon. Our experiments already include representatives of all prominent baselines: including regularization, weight-reinitialization and activation functions. The references provided include completely different architectures, which would not provide any meaningful comparison as a baseline relative to the results presented in our paper.

However, we have added additional experiments using more architectures, please see “Wide Residual Networks:” in shared reply.

We are pleased that the reviewer enjoyed reading our submission and appreciate the suggestions for improvement. The updated submission incorporates several of these suggestions, including experiments in the single task setting, additional architectures and results on continual imagenet (all changes and new content is in blue, except for minor typos which have been corrected). We address individual comments below,

While a new activation function is introduced, a proper analysis of its behaviour and the performance of the network should be demonstrate here we only see results on continual learning and later figure out that the results were with a regularization.

The sensitivity analysis in Section 5.2 explicitly investigates the use of deep Fourier features with and without regularization, which allows us to fully characterize the impact of each component. We find that unregularized deep Fourier features can sustain trainability, but can lack comparatively in generalization. While it is well-documented that Fourier features overfit without regularization due to their spectral bias, we find the surprising result that the increased trainability of deep Fourier features enables training with very large regularization strengths. This combination of increased trainability and high regularization strength leads to much stronger generalization, as demonstrated by Figures 5 and 6.

It is not clear which metrics are reported, e.g., test accuracy measured by 3what? performance on the latest task or all tasks? Forgetting is never discussed.

Our evaluation is consistent with the problem setting we describe in Section 2 (lines 124-126): The network is evaluated on data from the latest task as is commonly done in the loss of plasticity literature.

Please refer to “On forgetting:” in the shared reply.

In addition, note that the experiment setting that we use isolates the loss of plasticity phenomenon and does not involve catastrophic forgetting. For example, each task in class incremental learning involves all previous classes, as in [1]. The label noise experiment also does not involve catastrophic forgetting, in fact the goal is to forget the initial noisy labels as they become progressively clean [2].

[1] Dohare et al. (2024). Loss of plasticity in deep continual learning. Nature (632).

[2] Lee et al. (2024). Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks. ICML.

Only one network is studied, what about other architectures?

Thanks for bringing this up. For clarity, and to avoid too much clutter in the results, we had decided to present, in the main paper, results mostly for only one large architecture configuration. Nevertheless, notice that our results in Sections 3 and 4, and in the Appendix, investigate MLPs of varying widths and depths. In any case, we have now added an additional experiment studying additional large architectures, particularly Wide Residual Networks [3], to study the effect of width on Deep Fourier Features.

[3] Zagoruyko and Komodakis. (2016). Wide Residual Networks. ArXiv

Why loss of trainability is used, does it differ from the loss of plasticity defined in https://www.nature.com/articles/s41586-024-07711-7 and if not why using different terms?

Loss of plasticity is used ambiguously in the literature, and can refer to loss of trainability or to loss of generalization. We use the term loss of trainability to eliminate this ambiguity. Our theoretical focus is on trainability using an optimization perspective. In contrast, a theoretical understanding for loss of generalization would require understanding the mechanisms by which neural networks generalize. Understanding generalization of neural networks is an active research area, without guiding theory even outside of continual learning. We have added a sentence in Section 2 clarifying this. Note, however, that our experiments evaluate both trainability and generalization.

What is the performance compared to ReLU or LeakyReLU activations on offline setting (training on full dataset with no continual learning)?

We have added an additional figure to Appendix D.3. that compares performance on the full tiny-ImageNet dataset without any nonstationarity, using different activation functions. We see that deep Fourier features are highly trainable, and can generalize well with regularization. Note that the performance curve in Figure 5 on the first task is similar to learning in a non-continual setting (albeit, with some noisy labels making the problem harder). In this setting, we see that early performance across all methods is similar.

We thank the reviewer for their thoughtful comments and questions regarding our submission. We have added several clarification sentences to the submission in light of your feedback (changes and new content is in blue). Below, we address your individual questions.

the authors [...] do not examine whether these networks are resistant to catastrophic forgetting.

Please see “On forgetting:” in the shared reply.

The proposed activation function lacks clear implementation details [...] it’s ambiguous how the function applies to each unit in a layer and whether each pre-activation value is mapped to both sine and cosine or alternates between the two.

Thanks for bringing this up, we did not realize our presentation was ambiguous. We have added a clarifying sentence in Section 4.2 and pseudocode . Shortly, when talking about the implementation: In Figure 1 and line 344, we define the Fourier activation function as mapping each pre-activation to both sine and cosine. Having the pre-activation mapping to both sine and cosine is critical for approximate linearity discussed in the paragraph on line 399: when one of the activation functions is in a nonlinear region, the other activation function is in a linear region.

Periodic linear functions might also be a candidate for balancing linearity and non-linearity, so further justification or comparative analysis would strengthen the argument for cosine and sine.

On triangle wave: The reviewer’s suggestion of using linear periodic functions (triangle wave) is interesting. The main downside of triangle waves is that they are unstable to train because of exponentially many non-differentiable points. While relu has a non-differentiable point at the origin, the origin admits a reasonable choice of 0 for the subdifferential. For a triangle wave, however, there are exponentially many non-differentiable points away from zero which do not have a reasonable choice of subdifferential. For example, if the triangle wave has a peak at x = 1, then the subdifferential can be any number between [-1,1]. This leads to a lack of stability during training, which explains why they have not been used before.

In contrast, sine and cosine are smooth and have the property of closure under differentiation (the gradients are also sines and cosines). They also have a rich existing connection with previous neural network architectures through shallow Fourier features.

The theory in Section 3.1 suggests that linear networks can adapt to new data, but does not address resistance to forgetting.

Please see “On forgetting:” in the shared reply.

In addition to that, note that the experiment setting that we use does not involve catastrophic forgetting. This experiment setting is commonly used in loss of plasticity literature, see class-incremental setting in [1] and the label noise experiment in [2]. For example, the class incremental tasks involve all previous classes, as in [1]. The label noise experiment also does not involve catastrophic forgetting, in fact the goal is to forget the initial noisy labels as they become progressively clean.

[1] Dohare et al. (2024). Loss of plasticity in deep continual learning. Nature (632).

[2] Lee et al. (2024). Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks. ICML.

Can the authors explain how the Fourier activation function, is applied in practice? [...] Pseudo-code or a mathematical expression would be highly helpful.

We have added pseudocode to Appendix A.4. The expression provided, $\text{Fourier}(z) = [\sin(z), \cos(z)]$, summarizes the implementation which involves concatenating the element-wise sine and element-wise cosine of the pre-activation. Specifically, given a $d$ dimensional pre-activaiton, $z \in \mathbb{R}^d$, the Fourier activation is a mapping to the higher dimensional space: $\text{Fourier}: \mathbb{R}^d \rightarrow [-1,1]^{2d}$. This is why we comment in Section 4 that our baseline actually has approximately half the parameters, every layer has half the width to accommodate the concatenation.

Why did the authors choose cosine and sine for the activation function?

Thanks for bringing this up. We describe some of the rationale in Section 4 (line 361), with sine and cosine as natural choices that have been explored in other contexts (e.g., shallow Fourier features). To elaborate: A single periodic function does not provide approximate linearity in the parameters. Other concatenated activation functions involving piecewise linearity, like concatenated ReLU, provide linearity but are not adaptive due to a low unit sign entropy. In contrast, a pair of periodic activation functions achieves both i) approximate linearity and ii) high unit entropy (with many units active for any range of pre-activations, due to periodicity). We have added a few sentences to Section 4.2 before proposition 1 for additional clarity.

Dear Reviewer,

We would like to thank you for the time spent reviewing our submission.

The main discussion phase will be ending soon, and we wanted to send this gentle reminder. We have done our best to answer the comments you raised, as well as incorporate your suggestions. We would love to hear back from the reviewer and whether we have addressed their concerns.

Note: We now have updated the submission with several new results demonstrating deep Fourier features can also provide benefits in avoiding catastrophic forgetting (see Appendix D.7). Our experiment uses the setting described by [4], in which the labels for a dataset are permuted at the beginning of each task, and learning is online, meaning that each data point is seen exactly once. Moreover, and unlike the experiments in [4], we use more large-scale datasets (+CIFAR100/tiny-ImageNet) and train a ResNet end-to-end rather than just the last two layers. A good continual learning algorithm on this problem should both i) improve as more tasks are seen and ii) maintain performance on previously seen tasks. For all methods considered (ReLU, CReLU and deep Fourier features), we use elastic weight consolidation to regularize towards the parameters of the previous task [5]. Our results demonstrate that deep Fourier features can sustain high accuracy on the current and previous task, and that deep Fourier features are robust to the hyperparameter strength across online label-permuted tiny-ImageNet, CIFAR10 and CIFAR100.