LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Model

To Reviewer 59n9

Thanks for your valuable and constructive feedback. We are encouraged that you found our paper "significantly enhances SNN performance", "presents a seamless integration with ANN-SNN Conversion frameworks" and "has extensive experimental validation". We would like to address your concerns and answer your questions in the following.

1. Checklist should be placed after appendix.

Thanks for pointing it out. We will make relevant revision in the final version.

2. The authors can conduct experiments on more backbones (MS-ResNet [3] and Spikformer [4, 5]) to demonstrate the effectiveness of the proposed LM-HT SNN. The authors can also compare with papers [1, 2] which obtain really good results by MS-ResNet-34 backbone on large imageNet datasets.

Thanks for your suggestions! As shown in Table R1-R2, we have made comparison with related advanced works [1, 2, 3, 4, 5] on CIFAR and ImageNet datasets. One can find that our LM-HT model has superior scalability and can demonstrate its effectiveness on multiple different backbones.

For example, for CIFAR-100 and ImageNet-1k datasets, compared to GAC-SNN [2], we achieve accuracy improvements of 1.35% and 1.13% on MS-ResNet-18/34, respectively. For Transformer-4-384 architecture, our method also outperforms Spikformer [5] and Spike-driven Transformer [4] in terms of performance.

Table R1: Comparison with MS-ResNet series works [1, 2, 3].

Table R2: Comparison with Spikformer series works [4, 5].

3. While the LM-HT model can be reparameterized to a single-threshold model for inference, the paper may not fully explore the implications of this transformation for deployment on neuromorphic hardware.

Thank for the question. As shown in Fig.3, the main differences before and after reparameterization are the adjustments to the bias terms of Conv&BN layers, the shape of T-GIM layers, and the membrane leakage parameters，which aims to keep the sum of the input currents within each time window during the inference phase consistent with the expected input currents at the corresponding time-step during the training phase, thus further ensuring that the sum of spike sequences emitted by each layer within each time window is consistent with the multi-bit spikes emitted at the corresponding time-step during the training phase.

Besides these, there have been no changes in other parts of the network. Therefore, the reparameterized LM-HT SNN has consistent network architecture and firing mechanism with the SNN based on the vanilla LIF model, and we believe it can be effectively deployed on neuromorphic hardware. The experimental results in Table 2 also confirm that after being reparameterized and converted to a vanilla single-threshold model, the LM-HT model can maintain consistent performance and SOPs with the multi-threshold model before the reparameterization procedure.

[1] Yao M, Zhao G, Zhang H, et al. Attention spiking neural networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2023.

[2] Qiu X, Zhu R J, Chou Y, et al. Gated attention coding for training high-performance and efficient spiking neural networks[C]. Proceedings of the AAAI Conference on Artificial Intelligence. 2024.

[3] Hu Y, Deng L, Wu Y, et al. Advancing Spiking Neural Networks Toward Deep Residual Learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[4] Yao M, Hu J, Zhou Z, et al. Spike-driven transformer[J]. Advances in Neural Information Processing Systems, 2024.

[5] Zhou Z, Zhu Y, He C, et al. Spikformer: When Spiking Neural Network Meets Transformer[C]. The Eleventh International Conference on Learning Representations, 2022.

To Reviewer wxgA

Thanks for your insightful and valuable feedback. We are glad that you found our work "offers a nice perspective", "has solid theoretical analysis" and "the writing is generally good". We would like to address your concerns and questions in the following.

1. The authors are encouraged to provide mean accuracy and standard deviation over at least three different seeds for statistical significance.

Thanks for the question. Due to time constraints, we conducted 5 rounds of validation using different random seeds on CIFAR-10/ResNet-18 and obtained an statistical accuracy result of $96.20$% ± $0.10$%, which can verify the superiority of our method.

2. The use of multi-precision spikes in SNNs has been reported in several previous studies. To cater to a broader background of readers, the authors should better clarify their motivations.

Thanks for your suggestion. We think that the core motivation and contribution of this paper lies in revealing the relationship between multi-threshold models, vanilla spiking neurons and quantized ANNs, thus establishing a bridge between ANN-SNN Conversion and STBP training, which are two core training algorithms in the SNN community, while previous works focused more on using multi-precision spikes from the perspective of enhancing the richness of information transmission or reducing inference errors in ANN-SNN Conversion learning. We think that this point might be overlooked in previous works based on multi-precision spikes.

3. The authors should correct the name of DVSCIFAR-10 and several typos.

Thanks for pointing it out. We will revise them in the final submitted version.

4. The proposed methods seem to be hard to extend to a broader range of sequential learning tasks.

Thanks for the question. As shown in Eqs.(8)-(9), we designed the the back-propagation calculation chain of the LM-HT model in a form similar to quantized ANNs (detach the term $\frac{\partial m^l(t)}{\partial m^l(t-1)}$ and establish the correlation calculation of gradients at different time-steps solely through the T-GIM layer), which means that the LM-HT model is more inclined to learn global sequential information on the time dimension.

We think that if there is a need to learn towards long-term or local temporal information, one possible solution is to consider adjusting the back-propagation mode of the LM-HT model to the computational mode of vanilla LIF model in STBP training (consider the temporal back-propagation term $\frac{\partial m^l(t)}{\partial m^l(t-1)}$), as shown in Eq.(3).

5. In Table 3, the types of learning methods should be well explained in the caption. What is the meaning of efficient training?

Thanks for pointing it out. The term "efficient training" here actually refers to online training, which means that the backward gradient will be updated immediately after the forward propagation is completed at a specific time-step, thereby keeping the GPU memory overhead at a constant level. we will make relevant modification in the final version.

6. It is not straightforward to understand the time-steps in CIFAR10-DVS, as it is normally assumed to classify a gesture sequence. Given such high performance with only two time steps, have the authors tested the results with a single time step?

Thanks for the question. As we mentioned in Appendix A.2, we will still divide CIFAR10-DVS into 10 frames, while the T-GIM layer located before the first LM-HT neural layer will compress these 10 frames into the required time-steps $T'$ ($\Omega^1\in \mathbb{R}^{T'\times T}$). The experimental results in Table 3 indicate that good performance can be achieved on CIFAR10-DVS with just such a simple treatment.

We find through subsequent experiments that when $T'=1$, LM-HT SNN can still maintain an accuracy of $81.0$%. We think that the above phenomenon might also hint that some sequence classification tasks can be effectively completed through global information on the time dimension.

To Reviewer zyi4

Thanks for your constructive and valuable feedback. We are encouraged that you found our proposed method "effctive", "superior" and "provides a mathematical bridge between multi-threshold SNNs and quantized ANNs". We would like to address your concerns and questions in the following.

1. What are the advantages of this approach compared to directly training a single threshold model based on T-GIM modules?

Thanks for the question. Firstly, there is an advantage in terms of computational cost. During the training phase, compared to single-threshold models, using LM-HT models saves $L \times$ in both time and memory overhead ($O(LT)$ v.s. $O(T)$). Next is the enhancement of performance. The LM-HT model enhances the uniformity of spike sequences within each time window, combined with the gradient calculation mode of Eqs.(8)-(9), making LM-HT SNN have an equivalent training mode to quantized ANN, significantly improving its learning ability. Ultimately, our method can also reparameterize LM-HT SNN to vanilla SNN at no cost during the inference stage.

2. The authors still need to analyze the equivalence of gradient calculation between the LM-HT model and quantized ANN from the perspective of back-propagation.

Thanks for your comment. According to Eqs.(8)-(9), we can have

$\frac{\partial \mathcal{L}}{\partial m^l(t)} = \sum_{j=1}^T \frac{\partial \mathcal{L}}{\partial m^{l+1}(j)} \omega^{l+1}_{jt} \theta^l {W^{l+1}}^{T} \frac{\partial s^l(t)}{\partial m^l(t)}$,

and $\frac{\partial \mathcal{L}}{\partial W^l} = \sum_{t=1}^T \frac{\partial \mathcal{L}}{\partial m^l(t)} \sum_{j=1}^T \omega^l_{tj} s^{l-1}(j)\theta^{l-1}$.

When $\forall i,j\in T, \omega^l_{ij}=\frac{1}{T}$ and $\forall t\in T, m^l(t)\in[\frac{\theta^l}{2}, (L+\frac{1}{2})\theta^l ]$, as $\frac{\partial s^l(t)}{\partial m^l(t)} = \text{sign}(\frac{\theta^l}{2} \leq m^l(t) \leq (L+\frac{1}{2})\theta^l )$, we will find that

$\forall t\in [1,T], \frac{\partial \mathcal{L}}{\partial m^l(t)} = \frac{\theta^l }{T} \sum_{j=1}^T \frac{\partial \mathcal{L}}{\partial m^{l+1}(j)} {W^{l+1}}^{T}$,

which means that $\frac{\partial \mathcal{L}}{\partial m^l(1)} = ... = \frac{\partial \mathcal{L}}{\partial m^l(T)}$.

In additon, we can derive that $\frac{\partial \mathcal{L}}{\partial W^l} = \sum_{t=1}^T \frac{\partial \mathcal{L}}{\partial m^l(t)} r^{l-1}, r^{l-1} = \frac{1}{T} \sum_{j=1}^T s^{l-1}(j)\theta^{l-1}$, which is equivalent to the back-propagation mode of quantized ANN.

3. Can the authors make a more detailed explanation about the parameter rectification for Conv&BN, T-GIM and neuron layers?

Thanks for the question. For the bias terms in Conv&BN layers, due to the $L\times$ time-steps expansion in the inference stage, the addition operation at each time-step needs to be divided by $L\times$ (to ensure $I^l(t) + b^l_t = \sum_{j=L(t-1)+1}^{Lt} I^l_{IF}(j) + \hat{b^l_j}$, we set $\forall j\in [L(t-1)+1, Lt], \hat{b^l_j} = b^l_t / L$).

For T-GIM layers, due to the precondition that the multi-threshold model in the training stage is based on the uniform distribution of the input current within each time window, the values at each position within the $L\times L$ windows need to be divided by $L\times$ while expanding the shape of T-GIM in the inference stage (to ensure $\sum_{i=1}^T \omega^l_{ti} I^l(i) = \sum_{j=L(t-1)+1}^{Lt} \sum_{i=1}^{LT} \hat{\omega^l_{ji}} I^l_{IF}(i)$, we set $\forall i\in[1,T], \forall j\in [L(t-1)+1, Lt], \forall k\in [L(i-1)+1, Li], \hat{\omega^l_{jk}} = \omega^l_{ti} / L$).

For spiking neural layers, the multi-threshold model during the training phase has no membrane potential leakage within each time window, and the leakage degree between windows can be directly copied from the leakage parameters $\lambda^l$ during the training phase ($\forall t \in [1,T], \forall j\in [L(t-1)+1, Lt), \hat{\lambda^l_j} = 1; \forall j=L,...LT, \hat{\lambda^l_j} = \lambda^l_{j/L}$).

To Reviewer JrEi

Thanks for your insightful and valuable feedback. We are glad that you found our work "provides a rigorous mathematical analysis", "offering a new perspective of SNNs" and "significantly outperforms previous SOTA methods". We would like to address your concerns and questions in the following.

1. While the paper introduces the Temporal-Global Information Matrix (T-GIM), it is unclear how it affects the learning process and performance of the LM-HT model.

Thanks for the question. From the perspective of forward propagation, as shown in Fig.2, we can observe that the T-GIM layer can encompass two training modes: vanilla STBP training (when $\Omega^l = \text{diag}(1,...,1)$) and quantized ANN training (when $\forall i,j\in[1,T], \omega_{ij}^l=\frac{1}{T}$). In the training stage of LM-HT SNN, we set the T-GIM layer parameters to be learnable, while combining the learnable leakage parameters $\lambda^l$ of the multi-threshold model, so that the LM-HT model can be dynamically optimized to any parameter state between vanilla STBP training and quantized ANN training.

From the perspective of back-propagation, when we want to enhance the learning properties of LM-HT SNN simliar to that of quantized ANN, we can detach the temporal gradient term $\frac{\partial m^l(t)}{\partial m^l(t-1)}$, then we will have $\frac{\partial \mathcal{L}}{\partial W^l} = \sum_{t=1}^T \frac{\partial \mathcal{L}}{\partial m^l(t)} \sum_{j=1}^T \omega^l_{tj} s^{l-1}(j)\theta^{l-1}$.

Here $\sum_{j=1}^T \omega^l_{tj} s^{l-1}(j)\theta^{l-1}$ denotes the comprehensive global information that extracted from different time windows; while $\frac{\partial \mathcal{L}}{\partial m^l(t)} = \sum_{j=1}^T \frac{\partial \mathcal{L}}{\partial m^{l+1}(j)} \omega^{l+1}_{jt} \theta^l {W^{l+1}}^{T} \frac{\partial s^l(t)}{\partial m^l(t)}$, which means that the gradients at each time-step is the comprehensive weighted result of the gradients at each time-step in the post-synaptic layer. At this point, we can observe that LM-HT SNN maintains computational equivalence with quantized ANN in both forward and backward propagation, while T-GIM layer plays an important regulatory role in it.

2. While it is interesting to combine SNNs with quantization, it is unclear what advantages the proposed models have over traditional SNNs and quantized ANNs.

Thanks for the question. The essence of the LM-HT model is to extract information from spike sequences in units of time windows with specific length, while the introduction of the T-GIM layer ensures that the LM-HT model can simultaneously focus on global information on the time dimension. In addition, the precondition that the input current is uniformly distributed within the time window has enabled the performance of LM-HT SNN to reach the same level as quantized ANNs.

By comparison, vanilla spiking models can only extract local temporal information unidirectionally and have a certain degree of loss in learning accuracies, while quantized ANNs can only learn global average information on the time dimension. As shown in Fig.2, we can note that the LM-HT model has the advantages of these two training modes and can effectively avoid the related disadvantages of them.

3. From Fig.1(d)-(e), it seems that the surrogate function used in this work is the Rectangle Surrogate Function. How about other types of surrogate functions?

Thanks for pointing it out. We have tried other types of surrogate functions before, but we tend to think that Rectangle Function is the most ideal choice. As the membrane potential of multi-threshold models before firing spikes actually represents comprehensive information within a time window, therefore, its surrogate gradients cannot be directly calculated by fitting Heaviside Function.

For example, when $m^l(t)\in(k \theta^l, (k+1)\theta^l), k=0,...,L-1$, for vanilla surrogate functions (e.g. Sigmoid Function, Triangle Function) which aim to fit Heaviside Function, $\frac{\partial s^l(t)}{\partial m^l(t)} \rightarrow 0$, thereby leading to the gradient vanishing problem, although they can more richly reflect membrane potential information at each time-step. Therefore, we tend to think that this series of surrogate functions might be more suitable for vanilla single-threshold models.