Towards Energy Efficient Spiking Neural Networks: An Unstructured Pruning Framework

We appreciate the time and effort you invested in reviewing our paper. We will address all the issues you have raised.

The difference between ANNs and SNNs in pruning (especially unstructured neuron pruning) is not fully explained. It is not clear why unstructured neuron pruning is not used in ANNs. Why is it more suitable for SNNs?

Thanks for pointing it out! The reason why unstructured neuron pruning is more suitable for SNNs lies in the sparse computing manner of SNNs and neuromorphic hardware. In ANNs, neurons primarily function as stateless activation functions, such as Sigmoid or ReLU. These activation functions typically yield limited output sparsity. Consequently, ANNs heavily rely on dense matrix computations and are commonly implemented on hardware architectures optimized for dense computing, such as GPUs. These architectures are not designed to effectively exploit the unstructured sparsity of neurons. In contrast, SNNs employ sparse event-driven computing as their neurons communicate with each other by firing sparse, discrete spikes. Thus, SNNs are typically implemented on neuromorphic hardware that is highly optimized for sparsity. Therefore, unstructured neuron pruning is more suitable for SNNs.

It is important to note that the above analysis is only for the common case. In fact, some work has successfully utilized sparsity in ANNs by designing novel hardware [1], utilizing AVX instructions on CPUs [2], or designing new algorithms for GPUs [3]. We highly appreciate these efforts. However, it is crucial to note that these methods of exploiting sparsity are not widely adopted yet. Therefore, the analysis presented above is generally reasonable as it applies to most cases.

We have added this detailed analysis in the appendix.

The authors mainly illustrate how to optimize the penalty, but there was no explanation of how to optimize the first term of the loss function in Eq.11.

If we understand correctly, the first term of the loss function here refers to $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, and you are asking why the element-wise multiplication form in this term isn't improved as the penalty term does. In fact, the form of this term does not require further improvement, since the element-wise multiplication here is not problematic.

This is because the loss function term is different from the penalty term. The original form of the penalty term in Eq. 8 is a simple summation of the products of the three types of elements. For each of these products, if one of the three elements takes 0, the product achieves the minimum value of 0. Therefore, this penalty term tends to prune only one type of element, making it an ill-posed problem that is very sensitive to initial conditions. In contrast, the loss function term measures the loss between the target and the model output. From the perspective of the term $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, the loss function term is a complex nonlinear term with respect to $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, thus there is not a trivial solution where $\boldsymbol \alpha_w$ or $\boldsymbol w$ being 0 would make the loss function term achieve the minimum value of 0. Thus using element-wise multiplication here is not problematic.

For how to solve the optimization problem in Eq. 11, please refer to the pseudo-code using the stochastic gradient descent (SGD) method in Section A.1 on page 14.

How to choose the parameter $\beta$ for different datasets?

We would like to clarify that there is no necessity to choose different values of $\beta$ for different datasets, as the parameter $\beta$ is robust across different datasets. We consistently utilize the settings of $\beta_0=5$ and $\beta_T=1000$ in all of our experiments conducted on different datasets. Experimental results show that our parameter setting achieves good performance across diverse datasets, demonstrating the robustness of parameter $\beta$. Moreover, this robustness facilitates the application of our method to different tasks without the need for additional hyperparameter searches.

[1] Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016). EIE: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3), 243-254.

[2] Kurtz, M., Kopinsky, J., Gelashvili, R., Matveev, A., Carr, J., Goin, M., ... & Alistarh, D. (2020, November). Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine Learning (pp. 5533-5543). PMLR.

[3] Wang, Z. (2020, September). SparseRT: Accelerating unstructured sparsity on GPUs for deep learning inference. In Proceedings of the ACM international conference on parallel architectures and compilation techniques (pp. 31-42).

Great to see the difference between ANNs and SNNs in unstructured neuron pruning, which is helpful to better assess the contributions of the paper. It was a pleasure to read the manuscript, and I am happy to increase my rating.

Thank you for your valuable feedback on our work. Your insights on accelerator architectures are greatly appreciated and will help us to improve our paper. We agree with you that our model may not be suitable for all architectures, since many architectures claim that they take advantage of sparsity but have not fully exploited it, leading to additional costs for managing the sparsity. In the following, we discuss in detail which architectures are suited to use SOPs as an energy consumption metric and which are not.

First, we would like to emphasize the significance of SOPs as an important metric in both the hardware and software domains. Many works on neuromorphic hardware provide the energy consumption per SOP (or equally, energy per synaptic spike op, energy per connection, SOPs per second per Watt (SOPS/W), etc.) as an important metric for assessing hardware energy efficiency. Notable examples include Loihi [1] (23.6pJ per synaptic spike op), TrueNorth [2] (46 GSOPS/W (typical), 400 GSOPS/W (max)), and TianJic [3] (649 GSOPS/W (max)). As stated in [4], "Since synaptic processing dominates the system energy cost, this (denotes energy per (synaptic) connection) is the best guide to the overall energy efficiency of the system." For this reason, many SNN works use SOPs as the metric for estimating their energy consumption estimation, such as ESLSNN [5] compared in our manuscript. Therefore, it is reasonable to use SOPs as a meaningful energy consumption metric.

While SOPs is an important metric, we recognize that it does not cover all aspects of hardware energy consumption like memory access and data movement. Therefore, for architectures where other aspects of energy consumption, such as memory accesses, dominate and are not proportional to SOPs, it may not be accurate to use SOPs as the only metric of energy consumption. We list two types of these architectures. The first is the memory access-dominated architectures, such as SATA [6]. As demonstrated in [7], the energy consumption associated with memory access in SATA is significantly higher than that of computing and significantly depends on the dataflow. The second is the analog-based architectures such as [8]. Architectures with analog synapses typically consume much more energy to generate a spike compared to synaptic computation, thus their energy consumption may not be proportional to SOPs.

However, we believe that there are architectures that match our energy model due to their full exploitation of sparsity. By fully utilizing sparsity, the energy consumption of the whole system may be reduced to a level that is linear to the sparsity of synaptic operations, if the other aspects of energy consumption are also sparsely triggered by the synaptic operations. However, due to the wide variety of hardware architectures, it is almost impossible to distinguish which architecture can fully exploit sparsity by summarizing certain characteristics like many-core, weight reuse, the role of dataflow, etc. Thus, we list two different architectures that we believe may match our energy model and discuss which kind of architectures may match our energy model by analyzing their characteristics.

1. Loihi [1]. Loihi is a highly optimized neuromorphic chip designed for efficient neuromorphic computing. It incorporates two key features to enhance energy efficiency: local memory access and asynchronous spike communication. By colocating memory and computing within the core, Loihi minimizes the need for global memory access and reduces data travel distances. Although the detailed architecture design of Loihi has not been made public, the Loihi team claims that Loihi’s neuromorphic mesh and cores are built with asynchronous circuits and, at the transistor level, communicate event-driven tokens of information between logic stages, which helps utilize the sparsity [9]. While there is no direct evidence specific to SNNs, experimental results on other tasks indicate that the energy consumption of Loihi is roughly proportional to the computing.
2. SpinalFlow [10]: SpinalFlow differs significantly from Loihi. It is based on the DCNN accelerator Eyeriss [11], making its overall architecture more similar to an ANN accelerator rather than neuromorphic hardware. It doesn't have local memory access and asynchronous communication features like Loihi. Instead, it employs a global buffer and is built with the synchronous circuit. SpinalFlow employs an SNN-tailored dataflow and increases the reuse of neuron potential, input spikes, and weights. The architecture is designed to effectively leverage the inherent sparsity in spike-based computations. Experimental results demonstrate that SpinalFlow successfully takes advantage of sparsity and reduces energy consumption to a level proportional to sparsity.

Dear reviewer BChF, we hope that our response has adequately addressed your concerns regarding our paper and serves as a reference for your reassessment of our work. If you have any further comments or questions, please do not hesitate to let us know. We would be glad to provide a follow-up response. Thank you for your time and effort in reviewing our work.

We sincerely appreciate your approval of our work and helpful feedback. We will incorporate all your suggestions and answer your questions.

Figure 1 is not sufficiently elucidated.

Thanks for pointing it out. The purpose of Figure 1 is to provide readers with a visual representation of fine-grained pruning. The left figure shows neurons, weights, and synaptic connections in a convolutional layer with $5\times 5$ input and a $3\times 3$ filter. In order to avoid visual clutter caused by an excessive number of synaptic connections, we have chosen to display only 9 synaptic connections that correspond to one of the postsynaptic neurons. The right figure shows the effect of different pruning techniques on the number of synaptic connections. Each subfigure in the right figure is labeled with the number of synaptic connections that remain after pruning. Similarly, due to the excessive number of synaptic connections, we have not visualized the synaptic connections, which seems to have caused confusion. We have added these notes in the revised version.

Why there is not an element-wise multiplication for $m_n$ in Equation (6)? Why it is different from the masks of weights $m_w$?

This is because neurons are not considered as learnable parameters within the network, and as a result, they do not appear in Equation 6. Therefore, we only include $\boldsymbol m_n$ to denote the mask of neurons in Equation 6.

some variables are not clearly defined, such as $d_n$ and $d_w$ in Equation (6).

Thanks for your comment. In Equation (6), $d_n$ and $d_w$ denote the number of neurons and weights in the network $f$, respectively. We have added these notes in the revised version.

There are some grammar errors that require improvement. Before Equation (6), it appears that the order may be reversed.

Thank you for pointing out these errors. We have fixed them in the revised version.

We sincerely appreciate your thorough review of our paper and the invaluable insights you provided. We are grateful for the opportunity to address the points you raised.

Although combining energy factors into training is a novel approach for SNNs, it has previously been utilized in other contexts, such as artificial neural networks, for various objectives. Such as in the following paper [1].

We would like to clarify that the concept of "energy" in the energy consumption model proposed in our paper and the "energy" referred to in [1] are distinct and unrelated. In our paper, the energy consumption model is a model for estimating the energy consumption of SNNs, with "energy" specifically referring to the energy consumption of the network itself. On the other hand, the Energy-Based Model (EBM) mentioned in [1] is a type of machine learning model where "energy" represents a mathematical function that measures the compatibility between input data and the model's parameters, without any connection to energy consumption. Therefore, it is important to note that our proposed method and [1] are entirely different approaches with distinct goals. Our method mainly focuses on reducing the energy consumption of SNNs, while the approach in [1] is designed to reduce the model parameters.

The comparison with the state-of-the-art, especially concerning the ImageNet dataset, lacks comprehensiveness. Only one previous work is considered.

Thanks for pointing it out! Actually, there is limited existing work on SNN pruning, and currently, only STDS [2], the state-of-the-art SNN pruning method, provides experimental results on the ImageNet dataset. As a result, we primarily focus on comparing our method with STDS. It is worth noting that the original paper of STDS mentions the results of other pruning methods (GradR [3] and ADMM [4]) on the ImageNet dataset, which were implemented by the authors themselves. However, the performance of these methods significantly lags behind STDS, and no relevant code implementations are provided. Consequently, we have not included these methods in our comparison.

To provide a more comprehensive evaluation, we are working on reproducing the results of these pruning methods (GadR and ADMM) by our implementation. Unlike the comparison in STDS, our comparison requires additional metrics such as the number of SOPs and connections, in addition to the accuracy and number of parameters already provided. Therefore it is necessary to reproduce these experiments and obtain the trained models to measure these metrics, which is a time-consuming process. We have reproduced the experiments of ADMM and have presented the results in Table R1. The experimental results indicate that compared to both STDS and ADMM, our method achieves comparable performance while significantly reducing energy consumption.

We have added the comparison in the appendix. The GradR experiments are still in progress and we will report the results as soon as they are available.

Table R1. Comparison on ImageNet dataset

Dear reviewer gonx, we sincerely hope that our response to your review will assist in addressing your concerns and serve as a reference for your reassessment of our work. If you have any further comments or questions, please do not hesitate to let us know. We will be glad to provide a follow-up response. Thank you for your time and effort in reviewing our work.

Dear Reviewer gonx,

We notice that your initial review can be grouped into an overall concern and four detailed concerns.

1. The main concern is about the central idea of our paper. We have clarified that the concept of "energy" in the energy consumption model proposed in our paper and the "energy" referred to in [1] are distinct and unrelated in our response. Our proposed method and [1] are entirely different approaches with distinct goals.
2. The secondary concern is the comparison with the state-of-the-art methods. We have added more comparisons with ADMM on ImageNet dataset in our response and the revised version.
3. The third concern is about the implementation on neuromorphic hardware. Unfortunately, we currently do not have access to neuromorphic hardware that would allow us to deploy the sparse models to measure their actual energy consumption. Thus, we use SOPs as a proxy metric for energy consumption, which aligns with previous works. Additionally, we would like to highlight that the majority of research papers on SNNs compare energy efficiency using SOPs as the metric.
4. The fourth concern is about the unclear statements. We have clarified our intended meaning in our response and altered the expression in the revised version.
5. The last question is about the difference in definition compared with previous works. We have explained the distinction and why we use a different definition in our response.

Based on these facts and positive feedback from other reviewers, we sincerely hope you can reconsider your initial rating. If you still have any further comments or questions, please let us know and we will be glad to address your further concerns.