Faster and Stronger: When ANN-SNN Conversion Meets Parallel Spiking Calculation

To Reviewer mMJc

We are pleased that you recognize the relevant content of this work in terms of theoretical claims and experimental validation, as well as pointing out that our method provides a new perspective for SNN supervised learning. We will elaborate on your questions and comments.

The authors can compare the specific computational overhead of vanilla IF model and parallel model in more detail from the perspective of operands (ADD, MUL)

A1: Thanks for the question. For the vanilla IF model, the inter-layer computation (assuming the synaptic layer is a fully-connected layer $[C,C]$) involves $O(T_\text{IF} NC^2)$ multiplication/addition operations, here $N,T_\text{IF}$ respectively denote the total number of tokens and time-steps. The intra-layer calculation involves charging, firing and resetting, with a total of approximately $2T_\text{IF}$ addition and $T_\text{IF}$ comparison operands for each neuron.

For the parallel spiking model, the inter-layer computation is consistent with that of the IF model, while the intra-layer calculation can be combined with various optimization techniques proposed in Section 4.3. Specifically, when we convert $\mathbf{\Lambda}^l_\text{PC} \mathbf{I}^l$ to $[\frac{1}{T_\text{PC}},...,\frac{1}{T_\text{PC}-x+1},...,1]\odot\sum_t\mathbf{W}^l\mathbf{s}^{(l-1)}$, only $T_\text{PC}$ addition operands are involved, where $[\frac{1}{T_\text{PC}},...,\frac{1}{T_\text{PC}-x+1},...,1]$ can be further fused with $\theta^{l,t}$ at each time-step. This step involves $T_\text{PC}$ comparison operands. However, due to the existence of sorting property, the comparison operand can actually be further reduced to $O(\log T_\text{PC})$. Overall, using parallel conversion framework can save approximately $2T_\text{IF}-T_\text{PC}$ addition and $T_\text{IF}-O(\log T_\text{PC})$ comparison operands for each neuron layer, $O(T_\text{IF} NC^2)-O(T_\text{PC} NC^2)$ multiplication/addition operations for each synaptic layer, as well as achieving more superior inference performance and speed. Generally speaking, when traditional ANN-SNN Conversion reaches the same level of performance as Parallel Conversion, one can find that $T_\text{IF}\gg T_\text{PC}$.

What are the differences between the DA-QCFS based calibration method proposed in this work and SNNC-LP(AP)?

A2: Thanks for the comment. SNNC-LP(AP) is committed to extremely low-cost error calibration from pre-trained ANNs to converted SNNs, where the average spike firing rate used for calibration is based on the assumption of uniform input current distribution. Due to SNNC-LP(AP) still using the vanilla IF model in the inference stage, there is still a gap between the estimated firing rate after calibration and the actual firing rate layer by layer. In comparison, the motivation of our calibration method is to achieve efficient inference of SNN under any time latency. Since parallel spiking calculation can satisfy the assumption of input current distribution, the process of replacing the rectified DA-QCFS module with parallel spiking neurons is lossless. The goal of calibration is to regulate the distribution of input current within a specified number of time-steps, so that the estimated firing rate of each layer is aligned with the highest learning accuracy as much as possible, thereby preparing for the parallel conversion process in advance.

To Reviewer nBwe

We are delighted that you think that our method is well written and organized, as well as being validated in both theoretical and experimental dimensions. We will discuss your questions in detail in the following content.

The difference between the conversion method in this paper and I-LIF.

A1: Thanks for the question. From the perspective of conversion error, I-LIF transmits integer spikes during the training stage and can switch to vanilla LIF neurons during the inference stage. However, this can only ensure that I-LIF satisfies the input current assumption consistent with the QCFS function in the sub-slices composed of every $D$ time-steps ($D$ denotes the maximum integer spike value), and cannot fully achieve lossless conversion. In addition, I-LIF is commonly used in the field of STBP training. In comparison, parallel conversion is a learning framework specifically designed for ANN-SNN Conversion and has the property of lossless conversion.

From the perspective of calculation mechanism, I-LIF, as an enhanced version of the vanilla LIF model, still consists of three processes: charging, firing and resetting. The overall computing process is serial and has a temporal direction. In comparison, our scheme adopts parallel spiking calculation, saving the processes of charging and resetting, and enabling more efficient SNN inference.

Whether the spike trains in this paper can be adjusted at will?

A2: For ANN-SNN Conversion, the key to learning lies in the average spike firing rate. As shown in Eq.8, our $\Lambda_\text{PC}^l$ is formed by the fusion of $\Lambda_\text{PRE}^l$ and $\Lambda_\text{POST}^l$, thus possessing the ability to homogenize the distribution of input current. In other words, theoretically, arbitrarily shuffling the order of the output spike sequence $\mathbf{s}^{(l-1)}$ will not affect the prediction of the average spike firing rate $\mathbf{r}^{l,T}$ in the next layer.

From the perspective of adjusting the length of the spike sequence and the number of firing spikes, by combining the calibration scheme discussed in Sec 4.2 and Eq.9, we can arbitrarily adjust the length of the spike sequence and ensure that the converted SNN model has advanced performance within any time period.

It remains an open question whether the proposed parallel transformation can be applied to the Transformer architecture.

A3: Thanks for the comment. Assuming that the input currents through $\mathbf{Q}^l,\mathbf{K}^l,\mathbf{V}^l$ are respectively $\mathbf{I}_Q^l,\mathbf{I}_K^l,\mathbf{I}_V^l\in\mathbb{R}^T$. If the input current is directly passed through the corresponding parallel spiking neurons $\text{SN}(\cdot)$ and the attention score is calculated, it will introduce potential computational complexity of $O(T^2)$. Therefore, one viable solution is to pre-calculate the average input current $\mathbf{I}_K^{l,\text{avg}},\mathbf{I}_V^{l,\text{avg}}\in\mathbb{R}$, then complete the calculation for $\left(\text{SN}(\mathbf{I}_Q^l){\mathbf{I}_K^{l,\text{avg}}}^{\top}\right)\mathbf{I}_V^{l,\text{avg}}$.

At this point, one can note that the computational complexity of each step is at the level of $O(T)$ and always maintains $\mathbf{r}^{l,T}=\sum_t\mathbf{s}^{l,t}$. When $\mathbf{r}^{(l-1),T}=\sum_t\mathbf{s}^{(l-1),t}$, $\mathbf{I}^{l,x}=\Lambda_\text{PRE}^l\mathbf{W}^l\mathbf{s}^{(l-1)}=\mathbf{W}^l\mathbf{r}^{(l-1),T}, \forall x\in[1,T]$ holds, ensuring that the input current entering parallel neurons satisfies the distribution assumption and guarantees the precision of predicting the average spike firing rate layer by layer during the conversion process.

To Reviewer HMDR

We would like to thank for your acknowledgement about our approach in terms of theoretical analysis and performance advantages, we will provide further answers and clarifications for your questions and concerns.

Can this method be generalized to other tasks?

Thanks for this comment. To validate the generalization ability of our method on visual tasks, we further conduct experimental verification on semantic segmentation tasks. We attempt training-free parallel conversion based on DA-QCFS for Pascal VOC dataset [1] and ResNet50-FCN/DeepLabv3 structure [2, 3], as shown in Tab. R1. Experimental results indicate that our scheme can also achieve approximately lossless parallel conversion in segmentation tasks.

Table R1: Experimental results of training-free conversion on Pascal VOC dataset.

The analysis of Parallel Conversion for Transformer-based SNNs?

A2: Thanks for the comment. The most critical difference between Transformer and CNN is the matrix multiplication between multi-branch inputs (e.g. $\mathbf{Q}^l{\mathbf{K}^l}^{\top}, \text{Attn}^l\mathbf{V}^l$). If we directly insert QCFS functions after $\mathbf{Q}^l$ and $\mathbf{K}^l$ weight layers in the pre-training stage of ANN, and then replace QCFS modules with parallel spiking neurons $\text{SN}(\cdot)$ in the SNN inference stage, this may introduce $O(T^2)$ computational complexity within the self-attention modules. Therefore, we can consider pre-calculating the average input current for one of the branches (e.g. $\mathbf{I}^{l,\text{avg}}_Q$) and then calculating the attention score $\mathbf{I}^{l,\text{avg}}_Q \text{SN}({\mathbf{I}^l_K}^{\top})$ to maintain the computational complexity at $O(T)$. It is worth noting that at this point, parallel conversion will still maintain its unique lossless and sorting properties on Transformer-based SNNs.

The proposed method needs to train an ANN with quantization activation function, which brings training costs.

A3: For parallel conversion based on QCFS ANN, we usually need to complete the pre-training process of the quantized ANN model. However, our method also validate its effectiveness in training-free conversion cases. We can directly obtain the corresponding converted SNN model through utilizing quantization, error calibration, and parallel neuron replacement on an open-source and training-free ANN checkpoint.

The specific differences between Conversion Rectification and Parallel Conversion from the perspectives of performance and calculation overhead?

A4: Thanks for the question. The concept of Conversion Rectification usually refers to the relevant schemes for secondary optimization of the converted SNN model in the inference stage, aiming to reduce the representation gap between pre-trained ANNs and converted SNNs. The previous related works were generally based on IF neuron and its variants, and most of the works cannot guarantee the complete elimination of conversion errors in theory. Compared to these methods, our parallel conversion has faster inference speed and the property of lossless conversion . Additionally, due to its small number of time-steps, it can always keep the calculation overhead within a reasonable range.

[1] The PASCAL Visual Object Classes (VOC) Challenge, IJCV, 2010.

[2] Fully convolutional networks for semantic segmentation. CVPR, 2015.

[3] Rethinking Atrous Convolution for Semantic Image Segmentation, 2017.

To Reviewer 72NY

We sincerely appreciate your recognition for the novelty and experimental effectiveness of this work. We will strive to address your concerns in detail in the following section:

The discussion on threshold recording and error calibration techniques is not sufficient.

A1: Thanks for the question. In this work, threshold recording is used to confirm the initial maximum threshold and effectively utilize the distribution of input current, as QCFS explicitly provides learnable thresholds during the pre-training stage of ANN, threshold recording is only used in training-free conversion based cases. The motivation of error calibration lies in the fact that QCFS can only eliminate conversion errors ($T\neq\tilde{T}$) when the input current follows a uniform distribution, as shown in Theorem 4.1(ii). However, since the assumed condition may not be fully satisfied in practical situations, additional error calibration can further enhance the performance of converted SNNs under low time latency (where ClipReLU can also be considered as a case of $\tilde{T}\gg T$).

Which calculation step does the “sorting property” save in terms of computational cost?

A2: The sorting property actually effectively saves multiplication operations in $\Lambda_\text{PC}^l\mathbf{I}^l+\mathbf{b}^l$ and comparison operations between $\Lambda_\text{PC}^l\mathbf{I}^l+\mathbf{b}^l$ and $\theta^l$. Specifically, when we utilize the sorting property, it actually only involves calculation related to $\Lambda_\text{PC}^{l,\text{idx}}\mathbf{I}^l+\mathbf{b}^{l,\text{idx}}\geq\theta^{l,\text{idx}}$, where $\text{idx},\text{len(idx)}=O(\log T)$ is the index time-step set selected by the sorting property.

How are the corresponding throughput rates calculated?

A3: We randomly select a subset of data (1000 images) to calculate the average inference time for the above two schemes. We choose scientific tools to calculate the precise time and the specific process has been submitted into the supplementary materials. For serial inference, we feed data into the SNN backbone at each time-step, while parallel inference packages consecutive $T$ time-steps into a batch and feeds it into the network all at once.

For ResNet-50 and ResNet-101, it seems that their Acceleration Ratio do not further increase like other cases?

A4: Thanks for this insightful comment. Due to the fact that $T$ time-steps will be passed into the corresponding network backbone at once during parallel inference, for experimental cases with large model parameter quantity or time-steps, the required hardware memory is large and the utilization rate tends to be saturated. Therefore, under a environment with limited hardware support, the corresponding acceleration ratio may not further increase. However, even so, the advantage of parallel conversion is very obvious when $T\geq 32$ (e.g. the corresponding acceleration ratio exceeds $17.5\times$ for Fig 2.c-d). In addition, due to the theoretically lossless property of our method, in practical scenarios, we usually only need a small number of time-steps to achieve the same level of performance as the pretrained ANN for the converted SNN.

The authors' proof of the theorem in Appendix is somewhat concise.

A5: Thanks for the suggestion. We will further improve the relevant proof process in the final submission.