Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks

Thank you for appreciating our work and providing valuable comments. We respond to your comments and questions as follows.

1. About scalability to many tasks.

Our experiments verified effectiveness on common datasets with many tasks: PMNIST (10 tasks), CIFAR-100 (10 tasks), miniImageNet (20 tasks), 5-Datasets (5 tasks), also with similar or distinct input domains and online or multi-epoch training settings. So it can scale to many tasks.

Actually, although there can be slight noise for PCA, our method enables unbiased streaming PCA that leverages all data, while previous gradient projection methods either only use a small batch of data to perform SVD and PCA [1,2] which can have larger noise and bias, or use recursive least square algorithm [3] that introduces a parameter $\alpha$ for calculation of inverses which will have bias too. Our method has improvements over them.

2. The lateral connectivity does not influence forward propagation.

Yes, our lateral connections with subspace neurons mainly modify the activity traces (i.e., eligibility traces for SNNs under some training algorithms) of feedforward neurons for weight update, and do not influence the model’s forward propagation. While it mainly influences the calculation of gradients, this does not mean that it is only active during the backward pass – as discussed in Section 4.1, since HLOP only modifies presynaptic activity traces, this lateral projection can be parallel with the forward and backward propagation of the main network. That means, during the forward propagation of the main network, lateral connections can be simultaneously activated to modify traces of feedforward neurons, and once global error signals reach, weight updates are calculated based on the error signal and modified eligibility traces. We may make some conjectures that with some mechanisms such as the refractory period or adaptive threshold, the (fast) lateral connection may not induce spike signals for forward propagation but influence eligibility traces. It can be interesting future work to further consider if there can be biological correspondence and it is currently out of scope for this work.

3. About evaluations on non-spiking networks.

Thank you for your valuable suggestion. Here we supplement all the corresponding results of ANNs that replace the spiking neuron by ReLU activation.

As shown in the results, the performance of some methods can differ considering ANNs and SNNs. Particularly, on 5-Datasets, HLOP has more improvements for SNNs. It may imply that our method can be better combined with spikes, for example probably because subspaces expanded by spike signals are harder to learn and therefore our Hebbian learning performing streaming PCA has more advantages over GPM that only performs SVD/PCA on a small batch of data which can be biased. Also, our method is promising for ANNs as well. As there is no systematic study on the difference between the continual learning performance of ANNs and SNNs, it can be interesting for future work to study it.

4. About associative memory module.

As discussed in Appendix F, our Hebbian learning to extract principal subspaces of streaming data is to some extent similar to memory but is not in an associative approach, and can well fit into our projection formulation. Currently, we have no idea if an associative memory module can fit into the projection formulation, because in the task we mainly require some information about the general subspace for restriction rather than specific associative memory for a given input.

Thank you for your valuable comments. We respond to your comments and questions as follows.

1. Method details.

In Section 4, we mainly illustrate the procedure of our method in Figures 1 and 2, and formulas are in the texts due to the space limit. More detailed descriptions of HLOP can be found in Appendix C, and details of the combination with different SNN training methods are presented in Appendix A.

2. Baselines.

Thank you for the suggestion. Since not many continual learning methods consider SNNs and we do not find SNN baselines on these common datasets, we mainly implement and compare representative methods of different kinds of methods (i.e., memory replay, regularization, and gradient projection) based on their released codes under our SNN settings, as we cannot exhaustingly reimplement all ANN continual learning methods. Note that the performance of some methods can differ considering ANNs and SNNs (please see our response to Reviewer Me3Q).

There are some recent works [1] improving the gradient projection method GPM [2], which is our sota baseline. We try to test TRGP [1] based on their released code under our setting. On PMNIST and CIFAR-100, it does not outperform GPM as shown below, while on miniImageNet and 5-Datasets, we encounter failures that training fails after one or two tasks (i.e., the performance for new tasks is only random guess, but old tasks are maintained). It may probably be the difference between ANN and SNN, or hyperparameters and training settings, or some bugs in the code. Given the limited time, we cannot thoroughly analyze their method and codes. We think that current comparison is enough to show our superiority.

3. Subspace neurons and data storage.

Yes, our subspace neurons to some extent represent memory of some information of past tasks, and previous gradient projection method is also called Gradient Projection Memory [2], because we need information (of past tasks) to guide the restriction on weight update to solve forgetting. However, it is not explicit memory and thus does not require storing raw data which may bring concerns about data privacy [2].

4. Subspace neurons do not participate in the model’s forward process.

Our lateral connections with subspace neurons mainly modify the activity traces (i.e., eligibility traces for SNNs under some training algorithms) of feedforward neurons for weight update, and do not influence the model’s forward propagation. While it mainly influences the calculation of gradients, this does not mean that it is only active during the backward pass – as discussed in Section 4.1, since HLOP only modifies presynaptic activity traces, this lateral projection can be parallel with the forward and backward propagation of the main network. That means, during the forward propagation of the main network, lateral connections can be simultaneously activated to modify traces of feedforward neurons, and once global error signals reach, weight updates are calculated based on the error signal and modified eligibility traces. We may make some conjectures that with some mechanisms such as the refractory period or adaptive threshold, the (fast) lateral connection may not induce spike signals for forward propagation but influence eligibility traces. It is interesting future work to further consider if there is biological correspondence and it is currently out of scope for this work.

5. Why weight matrices obtained through Hebbian learning can serve as a substitute for orthogonal projection matrix.

As mentioned in Section 4.1, this is the known result that Hebbian learning can extract principal components of streaming inputs [3,4]. It has been shown that our considered formulation enables weights to converge to a dominant principal subspace [4]. In Appendix C, we also mentioned an intuitive explanation: Hebbian learning can be viewed as stochastic gradient descent for minimizing the objective function

$\min_{\mathbf{H}_l'} \mathbb{E} \left\lVert \mathbf{x}_l - \mathbf{H}_l^{\prime\top}\mathbf{H}_l'\mathbf{x}_l \right\rVert^2$,

and in our setting that updates new lateral weights with Hebbian learning, it can be viewed as minimizing $\min_{\mathbf{H}_l'} \mathbb{E} \left\lVert \mathbf{x}_l - \mathbf{H}_l^\top\mathbf{H}_l\mathbf{x}_l - \mathbf{H}_l^{\prime \top}\mathbf{H}_l'\mathbf{x}_l \right\rVert^2$ to extract the new principal subspace not included in the existing subspace neurons.

[1] TRGP: Trust region gradient projection for continual learning. ICLR, 2022.

[2] Gradient projection memory for continual learning. ICLR, 2021.

[3] Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 1982.

[4] Global analysis of oja’s flow for neural networks. IEEE Transactions on Neural Networks, 1994.

Thank you for your valuable comments. We respond to your comments and questions as follows.

1. About Hebbian learning and other approximations of PCA.

It is true that there are many PCA approximation algorithms. However, a basic focus of our work is neuromorphic computing, and the biologically inspired Hebbian learning well fits neuronal computation.

We first introduce more background on neuromorphic computing. Neuromorphic computing [1] aims at the computation inspired by the structure and function of human brains. At the hardware level, neuromorphic chips are designed to imitate biological neurons, such as their spiking property and synaptic connections with local storage of weights for in-memory computation, for highly energy-efficient and parallel event-driven computation with avoidance of frequent memory transpose (the computation architecture is different from the commonly used hardware with von Neumann architecture such as CPU or GPU). At the algorithm level, we are interested in developing methods compatible with properties and operations of neurons so that they are possible for deployment on hardware. As neuromorphic hardware is under development considering software-hardware co-design, most algorithms are simulated on common computational devices while considering neuromorphic properties.

For PCA algorithms, most of them require a lot of operations not corresponding to neuronal activities and connections, and the learning/update of the matrix does not follow the (biological) learning rule that may be designed for neuromorphic computing considering the locality property.

On the other hand, biologically inspired Hebbian learning well fits neural computation, and can also achieve sota convergence rate in streaming PCA [2]. From the perspective of general computation, neuroscience-backed algorithms will indeed boil down to matrix-vector arithmetic as other methods. But considering neuromorphic computing, Hebbian learning can be mapped to the computation of neurons and synapses and implemented on neuromorphic hardware, while other methods do not. So Hebbian learning is more suitable for SNNs.

Additionally, Hebbian learning enables unbiased streaming PCA from large data for better subspace construction than PCA methods adopted by previous works. This also has improvement and leads to better performance.

2. About the task-incremental setting.

We would like to clarify that we consider both task-incremental and domain-incremental settings, as what previous gradient projection methods do. Continual learning is classified into three fundamental types: task-incremental, domain-incremental, and class-incremental [3]. Task-CL requires task id at training and testing, domain-CL shares the whole network and does not require task id, while class-CL requires comparing new tasks and old tasks and inferring context without task id. Many previous works call both task- and domain-incremental as task-incremental, and here we distinguish them as in [3] to emphasize that our method does not necessarily need task id. Following experimental setups of previous gradient projection methods [4,5], we consider both task- (CIFAR-100, miniImageNet, 5-Dataets) and domain-CL (PMNIST) settings. As for class-CL, as discussed in Related Work, it inevitably requires some kind of replay of old experiences for good performance since it expects explicit comparison between new classes and old ones [6,4]. So we also do not focus on it and it can be future work to study if HLOP can be combined with some replay methods or context-dependent processing modules similar to biological systems [7] (e.g., with a task classifier) for better class-incremental tasks.

Our PMNIST experiment in Table 2 is domain-CL that does not require task id, since the classifier is shared. Here we further supplement the domain-incremental setting results on the larger 5-Datasets with a larger network (reduced ResNet-18) to show the scalability. The comparison results are below (where HAT requires task id and is not feasible) and the results also show the superior performance of HLOP.

[1] Towards spike-based machine intelligence with neuromorphic computing. Nature, 2019.

[2] ODE-inspired analysis for the biological version of Oja’s rule in solving streaming PCA. COLT, 2020.

[3] Three types of incremental learning. Nature Machine Intelligence, 2022.

[4] Gradient projection memory for continual learning. ICLR, 2021.

[5] TRGP: Trust region gradient projection for continual learning. ICLR, 2022.

[6] Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 2020.

[7] Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 2022.

loss.backward()
optimizer.step()

Thank you for appreciating our work and providing valuable comments. We respond to your comments and questions as follows.

1. Writing issues.

Thank you for your kind advice. Following your suggestions, we reorganize/modify descriptions of the final paragraph of the intro as well as Section 3.1, and move the sentence in Section 5.2 to the footnote. Please check the updated version.

2. Limitations/challenges of HLOP and evaluation.

As for our method, HLOP introduces additional computational costs for learning the lateral weights of each layer during training, which will increase training costs in our current implementation codes. While this process can theoretically be parallel to the normal forward-backward propagation of network training as discussed in the paper, such parallelization may not be easily realized in established deep learning libraries like PyTorch. So for our implementation of GPU training, the training time would be slightly longer. It can be future work on low-level code optimization or consideration of parallel neuromorphic computing hardware. Additionally, HLOP currently requires a manual specification for the number of subspace neurons, which may be improved for automatic and adaptive allocation. For example, the Generalized Hebbian Algorithm can perform Gram-Schmidt orthonormalization on the rows of weight matrices, which may help to sweep out unnecessary neurons and connections, but it requires some non-local information and may not be directly suitable to our method. It can be interesting for future work to study improvements.

As for the evaluation, similar to other gradient projection methods, this work mainly focuses on task-incremental and domain-incremental continual learning settings. There is another class-incremental setting which, as discussed in Related Work, inevitably requires some kind of replay of old experiences for good performance since it expects explicit comparison between new classes and old ones [1,2]. So following previous works, our evaluation mainly focuses on task- and domain-incremental settings. It can be future work to study if HLOP can be combined with some replay methods or context-dependent processing modules similar to biological systems [3] (e.g., with a task classifier) for better class-incremental tasks.

We have added the above discussions in Appendix F in the revised version.

3. Why previous methods cannot be implemented by neuronal operations for neuromorphic computing.

We would like to first introduce more background on neuromorphic computing. Neuromorphic computing aims at the computation inspired by the structure and function of the human brain. At the hardware level, neuromorphic chips are designed to imitate biological neurons, such as their spiking property and synaptic connections with local storage of weights for in-memory computation, for highly energy-efficient and parallel event-driven computation with avoidance of frequent memory transpose (the computation architecture is different from the commonly used hardware with von Neumann architecture such as CPU or GPU). At the algorithm level, we are interested in developing methods compatible with some properties and operations of neurons so that they are possible for deployment on the hardware. Also, note that neuromorphic hardware is under development considering software-hardware co-design, so most algorithms are simulated on common computational devices while considering neuromorphic properties.

When it comes to previous methods mentioned in Section 3.2, they do not consider neuromorphic properties and require complex computation beyond neuronal operations. First, for calculation of the projection matrix, they require operations such as SVD on a batch of data or calculating inverses of matrices, which needs a lot of (general) operations not corresponding to neuronal activities and connections, and the learning/update of the projection matrix does not follow the (biological) learning rule that may be designed for neuromorphic computing considering the locality property. Instead, we propose to leverage biologically inspired Hebbian learning which can also be directly integrated into the lateral circuit. Second, they do not consider how projection can be implemented. They require matrix multiplication of a projection matrix with weight gradients, which cannot correspond to operations in neurons and synaptic connections. While neurons can perform matrix multiplication considering neuronal activations and local connection weights, the above calculation cannot be directly mapped to such computation and it is unclear how the projection matrix can be locally stored. Instead, we propose a lateral neural circuit to modify the presynaptic activity traces for projection. The previous methods can be implemented on common devices as in our experiments, but considering neuromorphic computing, they are difficult to realize and we seek methods that have more similarity to biological neurons.