TAB: Temporal Accumulated Batch Normalization in Spiking Neural Networks

1. A detailed analysis of the impact of TAB on energy efficiency

Response 1: Thank you for your valuable feedback. In response to your suggestion, we have examined the average firing rate, specifically focusing on activity sparsity, to illustrate the network's energy efficiency. Detailed plots illustrating this analysis can be found here.

Upon careful examination of the plots, it is evident that the average spiking firing rate hovers around 10%-20%. This observation implies that merely 10%-20% of the neurons are actively firing spikes. In practical terms, this means that only a fraction of the neurons, approximately 10%-20%, are utilized for transmitting information between layers. This level of sparsity underscores the network's remarkable efficiency in transmitting information through the utilization of a minimal subset of neurons. We believe these findings significantly contribute to the energy-efficient nature of our proposed approach.

2. Can you provide a more detailed analysis of the computational complexity of TAB and how it compares to other methods?

Response 2: Thanks for the advice. The Table summarizes the Complexity Cost of different BN methods, including computational complexity and Memory Cost.

• (1) We have obtained this Complexity analysis based on the statistics and parameters of different BN methods in Table 1 in the main paper. We use here T to refer to the number of time-steps, B to refer to the batch size, C,H,W are the number of channels, height, and width. So the input at time-step t, $X_t$, to the normalization layer in SNNs has a dimension of $\mathbf{R}^{C \times H \times W}$.

• (2) In the conventional BN method in ANNs, there is no time dimension, therefore, the traditional BN method in ANNs has the computational complexity of $\mathcal{O}(BCHW)$, and a Memory Cost of $\mathcal{O}(CHW)$ to save the mean and variance which have the same dimension as $X$.

Memory Cost:

• a) As BNTT independently normalizes data at each time-step, it needs $\mathcal{O}(TCHW)$ space to save the mean and variance at each time-step, which is T times more than the traditional BN method, tdBN, and TEBN.
• b) The tdBN and TEBN jointly normalize data across all time-steps, so they only need $\mathcal{O}(CHW)$ space to save one overall mean and variance which is the same as the traditional BN method for saving mean and variance.
• c) The total memory cost of tdBN is $\mathcal{O}(CHW)$, as it needs only shared overall batch parameters just as the traditional BN, so there is no additional memory cost for saving the batch parameters.
• d) The total memory cost of TEBN is $\mathcal{O}(CHW) + \mathcal{O}(T)$, as it needs to save batch parameters just as the traditional BN and additional $T$ values to scale the data at each time-step. But $T$ is usually small compared to $\mathcal{O}(CHW)$, this memory cost can be approximated by $\mathcal{O}(CHW)$, but we still keep the original one in order to show directly the difference of using the additional $T$ memory cost.
• e) The total memory cost of our proposed TAB method is $\mathcal{O}(CHW) + \mathcal{O}(T)$, as it needs to scale the data as well as TEBN.

Computational Complexity:

• All the BN methods for SNNs need to deal with all the data spanning the time-dimension, no matter how to get the mean and variance,, all at once (e.g. tdBN and TEBN) or get that in a moving averaging way as in our proposed TAB method, the Computational Complexity is always $\mathcal{O}(TBCHW)$.

[1] Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, 2021.
[2] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, 2021.
[3] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Advances in Neural Information Processing Systems, 2022.

3. Can you provide evidence that TAB solves TCS? I think the illustration of distribution is capable of demonstrating the solution.

Response 3: Thank you for your insightful suggestion.

• (1) In response, we have thoroughly investigated the distribution of presynaptic activities both before and after the application of TAB batch normalization.

• (2) For our experimentation, we trained a 7-layer CNN model on DVS-CIFAR10 using the TAB method. The temporal distribution of normalization layers was visualized across various time-steps. Notably, all selected time-steps exhibited an almost identical distribution.

• (3) This observation has shown evidence that TAB solves TCS, leads us to the conclusion that our TAB method effectively contributes to the adjustment of the current distribution in DVS data. These findings provide valuable insights into the adaptive capabilities of TAB in normalizing presynaptic activities in SNNs.

2. Recent work all did experiments on ImageNet, while the paper ignore this.

Response 2: Thanks for your comments! We have conducted experiments on ResNet34 on ImageNet dataset.
The following table summarises the results. Please refer to To All Reviewers for the details.

• Due to the time limit of the rebuttal period, we have enabled to train 80 epochs of ResNet-34 on ImageNet with $T=2$, and 10 epochs for $T=4$.
• Although we only train our model for 80 epochs for $T=2$, the TAB method achieves a $1.09\%$ increment on ResNet-34 with smaller latency over TEBN on $T=4$ (i.e. TAB=$65.38\%$ with $T=2$ v.s. TEBN=$64.29\%$ with $T=4$).
• Even for only 10 epochs, the accuracy of our TAB method on ImageNet already reaches $65.98\%$ for $T=4$, with $1.7\%$ increment compared to TEBN with an accuracy of $64.29\%$ for $T=4$, which shows our TAB method is very promising.

3. Some references are missing.

Response 3: We will include all the listed references in the revised version of the main paper.

1. The authors should provide an ablation study of the TAB, so we can know if the TAB can really increase the accuracy and how much the method improves.

Response 1: Thanks for the advice. In order to understand how the our TAB method increases the accuracy and how much the method improves, we have conducted ablation studies. We gradually added each part of TAB method and checked the performance change.

• (1) First, we conduct the ablation study on the effect of BN methods. As we are training SNNs, we need to deal with the time dimension, so we train the network without using any BN method. We conduct experiments using the same setting as TAB method, and compare the results with our TAB method. We can see that the models are not trained at all for all datasets with all network architectures.

  Data set	Architecture	BN method	Time-steps	Accuracy (%)
  CIFAR-10	VGG-9	No BN	4	10
  CIFAR-10	ResNet-19	No BN	6	10
  CIFAR-10	ResNet-19	No BN	4	10
  CIFAR-10	ResNet-19	No BN	2	10
  CIFAR-100	VGG-11	No BN	4	1
  CIFAR-100	ResNet-19	No BN	6	1
  CIFAR-100	ResNet-19	No BN	4	1
  CIFAR-100	ResNet-19	No BN	2	1

• (3) Finally, we conduct the ablation study on the effect of learnable scaling weights $w[t]$ by removing the learnable scaling weights (i.e. setting $w[t]=1$) for all time-steps.

  Dataset	Architecture	BN method	Time-steps	Accuracy (%)
  CIFAR-10	VGG-9	No scaling, $w[t]=1$	4	91.93
  CIFAR-10	ResNet-19	No scaling, $w[t]=1$	6	94.78
  CIFAR-10	ResNet-19	No scaling, $w[t]=1$	4	94.3
  CIFAR-10	ResNet-19	No scaling, $w[t]=1$	2	93.83
  CIFAR-10-DVS	CNN7	No scaling, $w[t]=1$	4	74.2
  CIFAR-100	VGG-11	No scaling, $w[t]=1$	4	73.05
  CIFAR-100	ResNet-19	No scaling, $w[t]=1$	6	75.83
  CIFAR-100	ResNet-19	No scaling, $w[t]=1$	4	75.06
  CIFAR-100	ResNet-19	No scaling, $w[t]=1$	2	74.16

  Results show that removing the scaling weights, the performance is worse than using the scaling weights.

• (4) Finally, we show the results of our proposed TAB method for comparison.

  Dataset	Architecture	BN method	Time-steps	Accuracy (%)
  CIFAR-10	VGG-9	TAB	4	93.41
  CIFAR-10	ResNet-19	TAB	6	94.81
  CIFAR-10	ResNet-19	TAB	4	94.76
  CIFAR-10	ResNet-19	TAB	2	94.73
  CIFAR-100	VGG-11	TAB	4	75.89
  CIFAR-100	ResNet-19	TAB	6	76.82
  CIFAR-100	ResNet-19	TAB	4	76.81
  CIFAR-100	ResNet-19	TAB	2	76.31

  Results show that the performance of our TAB method indeed increases when using batch normalization and including the scaling weights at the same time.

1. The authors mention the phenomenon of covariance concept drift encountered by SNN models, so why don't they verify it on time-domain related datasets?

Response 1: Thanks for your advice.

• (a) The phenomenon of covariance concept drift encountered by SNN models is raised directly by the LIF spiking neuron model and its neuron dynamics, where each neuron maintains its membrane potential dynamics $u(t)$ over time, "integrates" the received input with a leakage, and fires a spike if the accumulated membrane potential value exceeds a threshold. SNNs use binary spike trains to transmit information between layers. The spike train is a series of 0's and 1's like spike-train=[0,0,0,1,0,0,1] with 0 indicating the neuron is silent, and 1 indicating the neuron is active.

• (2) The neuron dynamics refer to the changes in the membrane potential of a neuron over time as it integrates input signals and generates spikes. It can be formulated as a first-order differential equation (ODE), $\tau \frac{d u(t)}{d t} = -u(t) + I(t)$.

• (3) Within SNNs, synaptic currents $I(t)$ are sequentially fed into spiking neurons, with spike-triggered asynchronous currents accumulating in the membrane potential. Whenever this accumulated membrane potential exceeds a threshold, a spike is generated. This temporal dependency on membrane accumulation has the potential to amplify the internal covariate shift across the temporal domain which is termed the Temporal Covariate Shift (TCS) phenomenon.

• (4) In ANNs, the internal covariate shift (ICS) is incurred due to changes in the distribution of layer inputs caused by updates of preceding layers. In SNNs, this covariate shift does not only come from the layer updating, but also the temporal updating. The Temporal Covariate Shift phenomenon is due to updates of preceding layers and prior time-steps, which specially transpire along the additional temporal dimension compared to ANNs. Therefore, the intertwining of this temporal dependency of membrane potential dynamics $u(t)$ with the TCS phenomenon, presents a significant challenge in direct training of SNNs.

2. We can see that the improvement on the CIFAR-10 and CIFAR-100 datasets is very small, even just 0.6%. Can the current approach be regarded as a theoretically feasible strategy?

Response 2:

• (1) Yes, our proposed TAB approach provides a theoretically feasible strategy of Batch Normalization in training SNNs, as it addresses the temporal covariate shift issue by aligning with neuron dynamics and utilizing temporal accumulated statistics for data normalization. A detailed theoretical analysis has been provided in the main paper.

• (2) Meanwhile, as for the improvement of our proposed TAB method with other SOTA methods, for example on the CIFAR-10 and CIFAR-100 datasets, results of the statistical hypothesis have shown that there is a statistically significant improvement of our TAB method over other SOTA methods.

• (3) As suggested by Reviewer ZLsc, we have conducted a statistical hypothesis testing to check if the improvement of our TAB method is statistically significant over other SOTA methods. We have also done the Wilcoxon signed-rank test. Results show p-value=0.000244140625 which is less than 0.001 (the critical value), indicating that there is a significant difference in the performance between TAB and TEBN, statistically. Please check Response to All Reviewers (part 1/3) for the details.

• (4) We also checked the rank of the performance of our TAB method and TEBN, our TAB method always ranks 1 among all the paired TAB-TEBN performance results.

3. In addition, the authors have not looked at the impact of other network skeletons, such as ResNet34.

Response 3:

• (1) Thanks for your comments! We have provided the experiment results of ResNet34 on the ImageNet dataset. The following table summarizes the results. Please refer to Response to All Reviewers (part 3/3) for the details.

• (2) Due to the time limit of the rebuttal period, we have enabled training 80 epochs of ResNet-34 on ImageNet with $T=2$, and 10 epochs for $T=4$. Although we only train our model for 80 epochs for $T=2$, the TAB method achieves a $1.09\%$ increment on ResNet-34 with smaller latency over TEBN on $T=4$ (i.e. TAB=$65.38\%$ with $T=2$ v.s. TEBN=$64.29\%$ with $T=4$). Even for only 10 epochs, the accuracy of our TAB method on ImageNet already reaches $65.98\%$ for $T=4$, with $1.7\%$ increment compared to TEBN with an accuracy of $64.29\%$ for $T=4$, which shows our TAB method is very promising.

4. Another issue reviewer is concerned about is the relationship between TAB and neuron dynamics. Does the exponential nonlinear operation of the SNN-based model on the time coefficient lead to an error in approximating a first-order linear ODE?

Response 4:

• (1) Yes, but there will be an approximation error, but this is due to the discretization process.

• (2) The analytical closed-form solution for the first-order Initial Value Problem of the LIF dynamics ODE is

  $$U(t) = \exp{(-\frac{t}{\tau}) } \left( \int_0^t \frac{R}{\tau} I(s) \exp{ (\frac{s}{\tau}) } ds + U_0 \right) \ .$$

  When the neuron initiates at the value $U_0$ with no further input, i.e., $I(t)=0$, the closed-form solution of the ODE shows that the membrane potential $U(t)$ will start at $U_0$ and exponentially decay with a time constant $\tau$, $U(t) = U_0 \exp{ (-\frac{t}{\tau}) }$. However, for the exponential term, it is absorbed in $\lambda$, as $\lambda = \frac{U(t+\Delta t)}{ U(t)} = \frac{U_0 \exp{ (-\frac{t+\Delta t}{\tau}) }}{ U_0 \exp{ (-\frac{t}{\tau}) }} = \exp{ (-\frac{\Delta t}{\tau}) }$. This relationship enables us to formulate the discretization scheme $U[t+1] = \lambda U[t]$.

• (3) Through further rigorous derivation, we derive another equivalent form of the closed-form solution for the LIF dynamics,

  $$U(t) = (U_0 -R I_0) \exp{ (-\frac{t}{\tau}) } + R I(t) -\int_0^t R \exp{(\frac{s-t}{\tau}) } dI(s)$$

  The commonly used discrete LIF model, as denoted by $U[t] = \lambda U[t-1] + X[t]$, is derived from the first two terms of the discretization version of the closed-form solution.

• (4) Recalling our TAB method normalizes data utilizing temporal accumulated batch statistics ($\mu_{1:t}, \sigma^2_{1:t}$) across an expanding window $[1, t]$, where $\mu_{1:t}$ and $\sigma^2_{1:t}$ represent the temporal accumulated information up to time-step $[t]$. Computation of these temporal accumulated statistics is performed dynamically, employing a moving averaging approach, obviating the need to store batch data from all previous time-steps. The utilization of the temporal accumulated batch statistics aligns well with the accumulation mechanism of the membrane potential through the above equation.

Q2. Taking into account all the added complexity regarding trainable weights for sequential neurons, etc.

Response 2: Thanks for your comments! We have provided a thorough analysis and a detailed explanation for this concern. The following table summarizes the Complexity Cost of different BN methods, including computational complexity and Memory Cost. Please refer to Response to All Reviewers (part 2/3) for the details.

Q3. You have used trainable weights for the each point in the neurons sequence. Explain how the trainable weights for a new data can be used. Imagine you see a shift in the data distribution are these weights still valid?

Response 3:

• (1) The trainable weights are introduced in the TAB normalization and in the implementation, they will be used in the TAB-layer in the same way we use them in the conventional BN layers. In conventional BN layers, the batch-normalized inputs are scaled and shifted using learnable/trainable parameters $\gamma, \beta$ to preserve model expressiveness, $BN(x_i) = \gamma \hat{x}_i + \beta$, and $\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}$.

• (2) For the network training phase, the trainable BN weights $\gamma, \beta$ need to be learned together with the network's weights $W$. When there is a shift in the data distribution, the network adjusts the trainable weights $\gamma, \beta$ so as to alleviate the shifting phenomena in the data distribution.

• (3) Our TAB method employs the same normalization logic to the trainable weights $\gamma, \beta$, with an additional trainable scaling parameter $w[t]$ to scale along the temporal dimension, so the final parameters are $\hat\gamma = w[t]\gamma, \hat\beta=w[t]\beta$. They need to be learned in the TAB-layer.

Q1. regarding performance improvement over other SOTA methods (for example TEBN), suggest including a demsar plot to show the improvement

Response 1: Thank you for your insightful comment! we have included a demsar plot regarding our TAB method over other SOTA methods, refer here for the plot.

• (1) For the demsar plot, the difference in the average ranking on the axis of two algorithms is greater than the CD (critical difference) value, indicating that there is a statistically significant difference between our TAB method and the TEBN method. And our TAB method's performance is always ranking the first, so our TAB has a consistent improvement among all the results.

• (2) For the statistical hypothesis test, we have done the Wilcoxon signed-rank test to see whether the performance of our proposed TAB method is statistically different from other SOTA methods. Results show p-value=0.000244140625 which is less than 0.001 (the critical value), indicating that there is a significant difference in the performance between TAB and TEBN.

• (3) As among other SOTA methods, TEBN performs the best, so TEBN is seen as the most competitive method to TAB. We have conducted a statistical hypothesis test, the Wilcoxon signed-rank test on the paired TAB-TEBN performance results. By employing Wilcoxon signed-rank test, we have obtained p-value=0.000244140625 which is less than 0.001 (the critical value), indicating that there is a significant difference in the performance between TAB and TEBN.

• (4) We also checked the rank of the performance of our TAB method and TEBN, our TAB method always ranks 1 among all the paired TAB-TEBN performance results.

Details:

• (1). For the statistical hypothesis test

  The Wilcoxon signed-rank test is a non-parametric statistical test used to determine whether there is a difference between paired observations. As the Wilcoxon signed-rank test does not assume normality in the data and can be widely used, here we use Wilcoxon signed-rank test to understand whether there is a difference in the performance from different BN methods, typically between our proposed TAB method and its most competitor TEBN. Here are the Null Hypothesis and the Alternative Hypothesis.

  Null Hypothesis (H0): there is no difference in the performance between TAB and TEBN. 
  
  Alternative Hypothesis (H1): There is a significant difference in the performance between TAB and TEBN. 
  
  

  We collect the observed paired data, for example, for VGG-9 on cifar10 with time-step=4, the results of [TAB, TEBN] are [92.81, 93.41]. From table 2 and table 3 in the main paper, we have 13 such paired data.

• (2). For the demsar plot

  The value on the x-axis is the CD (critical difference) value. The value on the x-axis is the average ranking of each algorithm on multiple data sets. If the difference in the average ranking of the two algorithms is greater than the CD (critical difference) value, it indicates that there is a statistically significant difference between these two algorithms.