A Progressive Training Framework for Spiking Neural Networks with Learnable Multi-hierarchical Model

To Reviewer LLZy

How the proposed LM-H model addresses the issues of gradient vanishing and exploding faced by the LIF neuron?

Thanks for your comments! We have provided a thorough analysis and a detailed explanation for this concern. Please refer to To All Reviewers.

It would be good if the authors can also integrate the radical version of the LM-H model into these figures.

Thanks for your suggestion! We have revised the figure and submitted a new version.

Please provide a more comprehensive explanation and analysis of the radical version of the LM-H model, and discuss its advantages compared to the vanilla LM-H model.

As shown in Eq.6, the overall proportion term for historical information is represented as $(\mu_D)^{t-k}$, indicating that the extracted proportions at different time-steps follow an exponential distribution. In contrast, in Eq.7, the corresponding term changes to $\prod_{j=k+1}^{t} \mu_{D,j}$, allowing for more freedom and randomness in the extracted proportions. In addition, in the radical version, relevant factors that regulate the membrane potential ($\mu_{S,t}, \lambda_{S,t}$) and input current ($\lambda_{D,t}$) are independently assigned at each time-step, making the information representation range of the radical version wider than that of the vanilla version. We have added more details of the comparsion in Section A.5 of the revised version.

The author used the conversion framework based on the IF model during the ANN-SNN conversion phase. How about LIF neurons? Does it fit into this framwork?

Yes, it also fits into this framwork. According to our proposed hybrid training framework, if we initially convert a pre-trained ANN to the IF model, then set the membrane leaky constant $\lambda^l$ in each layer as a learnable parameter and adopt STBP for training, we can also incorporate the LIF model into the same framework. However, as shown in Eq.6, the LIF model can be regarded as a special case of the LM-H model, with its representation range being limited to a subset of the LM-H model's representation range. Consequently, we are inclined to believe that the significance of hybrid training based on the LIF model is quite limited.

The authors have illustrated the distinction between their work and that of TC-LIF in the related works section. I suggest adding it into the introduction as well.

Thanks for pointing it out. We have added the relevant content in the introduction of our revised version.

To Reviewer T4SQ

What are the significant deficiencies in deep layer gradient calculation? Does the author address the gradient vanishing & exploding problem in deep residual architectures with the new LM-H model? How?

Thanks for your comments! We have provided a thorough analysis and a detailed explanation for this concern. Please refer to To All Reviewers.

What are the significant deficiencies in capturing global information on the time dimension, as mentioned in this paper?

In terms of capturing global information, the LIF model is limited compared to the LM-H model, as depicted in Eq.6. The LIF model can only consider the second term of the equation, which represents the current representation. This limitation arises from the nature of spiking neurons (LIF model), which can transmit binary information only once per time-step. If the membrane potential and spike firing information from different time-steps are not comprehensively utilized, the representation capability of the output spike sequence becomes restricted. Consequently, this limitation can have an impact on the performance of SNNs. Table 1 in this paper, along with Table R1, can demonstrate the significant performance gap between the vanilla LIF model and our LM-H model, particularly when applied to large-scale datasets.

The paper could benefit from a more detailed discussion of why they propose the new model, how they differ from existing methods, and what specific contributions they make to the field of Spiking Neural Networks.

Thanks for your valuable feedback! We would like to clarify that our motivation for proposing the LM-H model primarily stems from the limitations of the existing LIF model in addressing deep gradient computation and global information extraction challenges. While other advanced models such as GLIF and TC-LIF have been proposed by researchers in the SNN community, we believe that the LM-H model offers four distinct contributions that previous works have not encompassed:

(1) The relationship between the LM-H model and the LIF model is elucidated through rigorous mathematical analysis, highlighting the superiority of the LM-H model. We demonstrate that the LM-H model encompasses the LIF model as a special case (Eq.5-Eq.6), while also showcasing the superior information extraction capabilities of the LM-H model. This analysis ensures that the theoretical performance of the LM-H model is at least on par with the LIF model, as acknowledged by Reviewer LLZy.

(2) The LM-H model possesses a remarkable capability to dynamically and flexibly extract global information across different time steps. Through detailed analysis, we identify the specific functions of membrane-related parameters and set appropriate learning ranges for them. This allows the LM-H model at the $t$-th timestep to effectively incorporate relevant information, including input current and membrane potential from previous $t-1$ steps. This capability to consider information from multiple time steps simultaneously sets the LM-H model apart from other advanced models.

(3) The LM-H model demonstrates superior scalability and transferability. From a computational perspective, the LM-H model is a more general form of the LIF model. In terms of neuronal structure, the LM-H model incorporates a dendrite layer to store historical information for each time-step. When $\mu_D=\mu_S=0, \lambda_D=1$, the dendrite layers become invisible, and the LM-H model degenerates into a single-layer state consistent with the LIF model. These two aspects establish a close relationship between the LM-H model and the LIF model, facilitating easy migration of various training frameworks associated with the LIF model (such as hybrid training and online training) to the LM-H model.

(4) The progressive STBP training framework is novel. This novel training framework starts with the parameter settings of the LIF model as the initial state and dynamically optimizes membrane-related parameters during the training process to achieve the optimal state of the LM-H model. This design concept of "from special to general" allows for efficient and effective training of the LM-H model.

The authors listed results of cifar data set on resnet, can you provide results on vgg as well?

Thanks for your suggestion! We have conducted additional experiments using the VGG-13 architecture on the CIFAR-10 and CIFAR-100 datasets. As shown in Tab. R4, our results demonstrate superior performance compared to previous SOTA methods, even when employing smaller network structures and fewer time-steps. We have also included these content in the Appendix A.6 of our submitted version.

Table R4: Comparison with previous SOTA works on CIFAR datasets, VGG structure.

[1] Nitin Rathi and Kaushik Roy. DIET-SNN: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, pp. 1–9, 2021.

[2] Yufei Guo, Liwen Zhang, Yuanpei Chen, Xinyi Tong, Xiaode Liu, YingLei Wang, Xuhui Huang, Zhe Ma. Real spike: Learning real-valued spikes for spiking neural networks. European Conference on Computer Vision. 2022.

[3] Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. RecDis-SNN: Rectifying membrane potential distribution for directly training spiking neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 326–335, 2022.

To Reviewer uzHG

LIF neurons have a gradient vanishing or exploding problem in deep residual architecture, how did LM-H solved this problem?

Thanks for your comments! We have provided a thorough analysis and a detailed explanation for this concern. Please refer to To All Reviewers.

ImageNet and CIFAR are all static image classification datasets, but LH-M involves historical information and performed better. Could you elucidate the underlying reasons for this enhanced performance in the context of LH-M's utilization of historical information?

Thanks for your insightful comment! While ImageNet and CIFAR are static datasets, our experimental process, as consistent with previous works, involves inputting each image for multiple time-steps. Since spiking neurons (model) can only emit binary spikes (either firing or not firing) at each time-step, the representation range of information transmitted at a single time-step is inherently limited. Therefore, in order to enhance the representation ability of spiking neurons, it becomes necessary to consider the membrane potential ($\forall j\in[1, t-1], \boldsymbol{v}^l[j]$) and spike firing status ($\forall j\in[1, t-1], \boldsymbol{s}^l[j]$) at each previous time-step when deciding whether to fire a spike at the current time-step. This utilization of historical information allows for a comprehensive decision-making process. In fact, regardless of the input data type (static or neuromorphic), as long as we adopt multi-step training, the LM-H model can observe a broader range of information compared to the LIF model, thereby demonstrating its superior performance, as illustrated in Eq.6 in the main text.

How did you perform ANN2SNN conversion? ANN do not include temporal information, but in LM-H historical information is an important property.

Thanks for pointing it out. We would like to clarify that our hybrid learning method can be divided into two stages. In the first stage, we adopt traditional ANN-SNN Conversion framework to obtain a LM-H model under the special case, which indeed does not contain temporal information. As shown in Eq.11 and its corresponding analysis, we derived that the most accurate mathematical mapping relationship between the converted LM-H model and the pre-trained ANN model (i.e., the minimum conversion error) is established under the condition $\mu_D=0,\lambda_D=1,\mu_S+\lambda_S=1$. Therefore, in the first stage of hybrid training, we replace the activation functions in the pre-trained ANN model with the LM-H models, adhering to the special case ($\mu_D=0,\lambda_D=1,\mu_S+\lambda_S=1$) layer by layer.

However, in the second stage of the training process, all the membrane-related parameters of the LM-H model ($\mu_D,\lambda_D,\mu_S,\lambda_S$) are set as learnable parameters. Throughout the progressive training process, these parameters are dynamically optimized and updated to effectively extract temporal information. The detailed description and specific procedure of the algorithm can be found in Appendix A.4 of the revised version, where a comprehensive explanation is provided.

Dear Reviewer uzHG,

As the Author-Review Discussion period will be closed within a few hours, we would like to briefly summarize your concerns and our relevant responses as follow:

1. The first concern is about the gradient calculation problem in deep residual architecture. We have provided a thorough analysis to discuss the specific difference of the gradient calculation between the vanilla LIF and LM-H model, as shown in To All Reviewers.
2. The second concern is about the LH-M model's utilization of historical information on static datasets. We have made a detailed explanation to point out that the LM-H model can observe a broader range of information compared to the LIF model as long as we adopt multi-step training, thereby demonstrating its superior performance.
3. The third question is about the utilization of temporal information in our hybrid training framework. We have clarified that our hybrid learning method can be divided into two stages. Throughout the progressive training stage, the membrane-related parameters are dynamically optimized and updated to effectively extract temporal information, the detailed description can be found in Appendix A.4 of our revised version.

Based on these facts and positive feedback from other reviewers, we sincerely hope that you can reconsider your initial rating. If you have any further comment, please let us know and we are glad to address your concern.

Best regards,

The Authors

I appreciate the authors' thorough and well-structured rebuttal, which has addressed my concerns effectively. It is evident that they have made solid progress in refining their work and providing convincing arguments to support their claims. Additionally, I believe that authors proposed a novel and effective spiking neuron model for SNN community. Taking these revisions into account, I am inclined to raise the paper's score.

To Reviewer SiwN

Why only small datasets were experimented upon, was it because the proposed approach has scalability issues on larger and more practical datasets?

Thank you for your suggestion! We have conducted additional experiments on the ImageNet-1k dataset and compared the performance of our model with previous SOTA approaches using the ResNet-34 architecture. Due to the time constraints during the rebuttal period, we directly evaluated the performance of our radical version. As illustrated in Tab. R2, it is evident that our method outperforms other SOTA approaches by at least 2% with the same (or fewer) time-steps, which clearly demonstrates the superior scalability of the LM-H model on large-scale datasets. Due to page constraints, we have included these content in the Appendix of the revised paper (Tab. S3 of Section A.5). In the final version, we will incorporate it into the main text.

Table R2: Comparison with previous SOTA works on ImageNet-1k dataset.

Why the performance of the radical version was only presented for ImageNet while missing for other datasets, was it because that it didn't show better accuracy?

Thanks for pointing it out. We have expanded our comparative experiments to include both the vanilla version and the radical version on CIFAR-10 and CIFAR-100 datasets, as shown in Tab. R3. Given the inherent capability of the residual structure to enhance accuracy, particularly on relatively simple datasets, we have also incorporated the commonly used VGG structure to facilitate a clearer comparison between the two versions. As depicted in Table R3, the radical version, by regulating gradient calculation and information extraction, continues to exhibit better performance than the vanilla version on the CIFAR-10 and CIFAR-100 datasets. Furthermore, during the experimental process, we observed that the radical version also displayed faster convergence in terms of training accuracy, particularly when dealing with large-scale datasets. Due to page constraints, we have included these content in the Appendix of the revised paper (Tab.S3 of Section A.5, Tab.S4 of Section A.6). In the final version, we will incorporate it into the main text.

Table R3: Comparison of vanilla and radical versions on CIFAR-10/100 datasets.

Grammar errors scatter through the paper, further proof-reading is suggested.

Thanks for your advice！We have carefully corrected the grammar errors in this paper and uploaded a new version.

[1] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In AAAI Conference on Artificial Intelligence, pp. 11062–11070, 2021.

[2] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. International Conference on Learning Representations, 2022.

[3] Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. RecDis-SNN: Rectifying membrane potential distribution for directly training spiking neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 326–335, 2022.

[4] Yufei Guo, Yuhan Zhang, Yuanpei Chen, Weihang Peng, Xiaode Liu, Liwen Zhang, Xuhui Huang, and Zhe Ma. Membrane potential batch normalization for spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.

[5] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Advances in Neural Information Processing Systems. 2022.

[6] Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. GLIF: A unified gated leaky integrate-and-fire neuron for spiking neural networks. In Advances in Neural Information Processing Systems, 2022.

Dear Reviewer SiwN,

As the Author-Review Discussion period will be closed within a few hours, we would like to briefly summarize your concerns and our relevant responses as follow:

1. The first concern is about the scalability of our proposed method on larger datasets. We have conducted additional experiments on the ImageNet-1k dataset and compared the performance of our model with previous SOTA approaches. The experimental results in Tab.R2 have demonstrated the superior scalability of the LM-H model on large-scale datasets.
2. The second concern is about the comparison between the vanilla version and the radical version. We have expanded our comparative experiments in Tab.R3 to include both the vanilla version and the radical version on CIFAR-10 and CIFAR-100 datasets. The experimental results can verify the effectiveness of the radical version.
3. The third question is about the further proof-reading of this paper. We have carefully corrected the grammar errors in this paper and uploaded a new version.

Based on these facts and positive feedback from other reviewers, we sincerely hope that you can reconsider your initial rating. If you have any further comment, please let us know and we are glad to address your concern.

Best regards,

The Authors