ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks

Thanks for your efforts in reviewing our paper and your recognition of our novel method and effective results. The responses to your questions are given piece by piece as follows.

Question 1: Does the ReverB network has a reset mechanism, and how does it function?

A1: Thanks for the question. ReverB network has a reset mechanism. When the membrane potential exceeds the firing threshold, it will emit itself and then reset to 0; otherwise, it will decay to a new value and then be updated with next-time input.

Question 2: Does the ReverB network use the Batch Normalization (BN) layer, and can ReverB eliminate it during inference similar to SNN?

A2: Thanks for the question. We use the BN layer in the network and it can be eliminated in the inference. At inference time, the batch statistics (mean and variance) are no longer needed. Instead, we can fold the normalization, scaling, and shifting operations into the weights and biases of the previous layer. Then $W′=σ/γ⋅W$ and $b′=γ(b−σ/μ)+β$. In our paper, we propose the re-parameterization method to fold the $σ/γ$ into the previous layer activation (see Eq.16). Thus the BN can be removed still in the ReverB.

Question 3: Why does ReverB exhibit performance advantages over traditional SNNs, and can the authors provide an explanation for this?

A3: Thanks for the question. This is because ReverB quantizes the weight while traditional SNNs quantize activation. Highlight greater accuracy degradation from quantizing activations compared to weights. This is due to several key reasons:

First, activations often have a much wider and more varied dynamic range compared to weights. Due to the large dynamic range and possible non-uniform distribution of activations, quantizing them requires compressing a broader set of values into a smaller bit-width, which can result in greater precision loss.

Second, Activations change frequently during the forward pass, as they are directly influenced by the input data and the weights. The values can vary significantly between different layers, and this variability can be much greater in deeper layers. Such frequent and complex changes make activations more sensitive to quantization errors. On the other hand, weights are stable and unrelated to input, making them less prone to large errors when quantized.

Third, Here, we will also use the entropy concept to show that ReverB-SNN has a higher information capacity than vanilla SNN. thus exhibiting performance advantages over traditional SNNs.

We first perform a theoretical analysis using the concept of entropy. The representational capability $\mathcal{C}(\mathbf{X})$ of a set $\mathbf{X}$ is determined by the maximum entropy of $\mathbf{X}$, expressed as: $\mathcal{C}(\mathbf{X}) = \max \mathcal{H}(\mathbf{X}) = - \sum_{x \in \mathbf{X}} p_{\mathbf{X}}(x) \log p_{\mathbf{X}}(x),$ where $p_{\mathbf{X}}(x)$ is the probability of a sample $x$ from $\mathbf{X}$. Then it is clear for the following proposition:

For a set $\mathbf{X}$, when the probability distribution of $\mathbf{X}$ is uniform, i.e., $p_{\mathbf{X}}(x) = \frac{1}{N}$, where $N$ is the total number of samples in $\mathbf{X}$, the entropy $\mathcal{H}(\mathbf{X})$ reaches its maximum value of $\log(N)$. Hence, we conclude that $\mathcal{C}(\mathbf{X}) = \log(N)$.

Using the proposition, we can evaluate the representational capacity of vanilla SNN and our model.

Let's consider two connected neuron layers. Since the connectivity of the neurons is the same no matter for vanilla SNN or our model, let us focus on an arbitrary two connected neurons from different layers.

For the vanilla SNN, the values output from one neuron to another are {0, 1} x v, where v is the fixed weight between the two neurons. Thus one neuron from vanilla SNN could transform two samples into another one, and $\mathcal{C}(\mathbf{X}) = \log(2) = 1$.

For our model, the values output from one neuron to another are u x {-1, 1}, where u is the real-valued spike and could be changed. u requires 32-bits. Thus the number of possible samples from u is 2^{32} and the value samples from one neuron to another is 2^{32+1}. $\mathcal{C}(\mathbf{X}) = \log(2^{32+1}) = 33$.

This highlights the limited representational capacity of vanilla SNN compared to our model.

Thanks for your efforts in reviewing our paper and your recognition of our novel bit-reversal strategy and adaptive weight scaling. The responses to your weaknesses and questions are given piece by piece as follows.

Weakness 1: The paper does not clearly explain the rationale behind reversing the bits of weight and activation. This lack of clarity could make it challenging for readers to fully grasp why this approach is beneficial.

R1: Thanks for the advice. Here, we will use the entropy concept to show that ReverB-SNN has a higher information capacity than vanilla SNN.

we first perform a theoretical analysis using the concept of entropy. The representational capability $\mathcal{C}(\mathbf{X})$ of a set $\mathbf{X}$ is determined by the maximum entropy of $\mathbf{X}$, expressed as: $\mathcal{C}(\mathbf{X}) = \max \mathcal{H}(\mathbf{X}) = - \sum_{x \in \mathbf{X}} p_{\mathbf{X}}(x) \log p_{\mathbf{X}}(x),$ where $p_{\mathbf{X}}(x)$ is the probability of a sample $x$ from $\mathbf{X}$. Then it is clear for the following proposition:

For a set $\mathbf{X}$, when the probability distribution of $\mathbf{X}$ is uniform, i.e., $p_{\mathbf{X}}(x) = \frac{1}{N}$, where $N$ is the total number of samples in $\mathbf{X}$, the entropy $\mathcal{H}(\mathbf{X})$ reaches its maximum value of $\log(N)$. Hence, we conclude that $\mathcal{C}(\mathbf{X}) = \log(N)$.

Using the proposition, we can evaluate the representational capacity of vanilla SNN and our model.

Let's consider two connected neuron layers. Since the connectivity of the neurons is the same no matter for vanilla SNN or our model, let us focus on an arbitrary two connected neurons from different layers.

For the vanilla SNN, the information output from one neuron to another are {0, 1} x W, where W is the fixed weight between the two neurons. Thus one neuron from vanilla SNN could transform two samples into another one, and $\mathcal{C}(\mathbf{X}) = \log(2) = 1$.

For our model, the information output from one neuron to another are o x {-1, 1}, where o is the real-valued spike and could be changed. o requires 32-bits. Thus the number of possible samples from o is 2^{32} and the value samples from one neuron to another is 2^{32+1}. $\mathcal{C}(\mathbf{X}) = \log(2^{32+1}) = 33$.

This highlights the limited representational capacity of vanilla SNN compared to our model.

Weakness 2: The section detailing the contributions is somewhat lengthy and could be streamlined. Consolidating similar ideas might improve readability and focus.

R2: Thanks for the advice. We will further polish our contributions in the final version.

Weakness 3: The comparisons in Table 2 and Table 4 appear to rely on methods from 2022, missing more recent literature.

R3: Thanks for the advice. We have added more comparisons as below. It can be seen that our method also performs on par with or better than state-of-the-art methods.

Weakness 4: The paper could benefit from additional visualizations. Graphs or schematic diagrams illustrating the architecture and the re-parameterization process would enhance understanding and provide clearer insights into the proposed method.

R4: Thanks for the advice. We have added the visualizations for the re-parameterization process. Please see it from https://imgur.com/GiJYTke

Question 1: Could you clarify the theoretical motivation and advantages behind "Reversing Bit"?

A1: Thanks for the question. Please see our response to Weakness 1.

Question 2: The paper lacks visualizations; could you add diagrams of the network architecture or re-parameterization process for clarity?

A1: Thanks for the advice. We have added the visualizations for the re-parameterization process. Please see it from https://imgur.com/GiJYTke

Thanks for your efforts in reviewing our paper. We will try to make the work clearer for you. The responses to your concerns and questions are given as follows.

Concern 1: The only required addition is not for the learnable version.

R1: Sorry for the confusion. Since the $\alpha$ will be fixed after training, it can be folded to the activation function before inference. Then only requires additions too.

Concern 2: the model may not have the low power consumption benefits.

R2: Thanks for the question. I agree that on many low-power neuromorphic hardware, the low-power consumption is thanks to the use of binary spikes. However, there are also many hardware that realize low-power consumption based on replacing multiplications with additions like [1,2]. With this hardware, our model can keep low power consumption too. What’s more, there is also no available hardware that could support SNN-based transformer architectures which become popular now. Our model and SNN-based transformer architectures could drive further hardware development like the emergence of other hardware like Lohoi and TureNorth.

[1] A systolic SNN inference accelerator and its co-optimized software framework.

[2] Tianjin chip.

Concern 3: The evaluation of memory consumption.

R3: Thanks for the question. For memory consumption, the vanilla SNN’s weights adopt 32 bits while our model’s weights could adopt 1 bit. Thus the memory consumption of our model is much less than that of the vanilla SNN.

Concern 4: Provide the derivation for Equations 11, 14, and 15.

R4: Thanks for the question. For Equations 11, from Equation 5 and $\mathbf{W_l}^b = {\rm sign}(\mathbf{W_l})$, based on chain rule, we could know that $\frac{\partial {L}}{\partial {\mathbf{W}_l}} = \sum_t \frac{\partial {L}}{\partial {U^t_l}}\frac{\partial {U^t_l}}{\partial {\mathbf{W}^b_l}}\frac{\partial \mathbf{W}^b_l}{\partial {\mathbf{W}_l}}$. From Equation 4 and Equation 1, we know that $U^t_l$ will affect $O^t_l$ and $U^{t+1}_l$, thus $\frac{\partial {L}}{\partial {U^t_l}} = \frac{\partial {L}}{\partial {O^t_l}} \frac{\partial {{O^t_l}}}{\partial {{U^t_l}}} + \frac{\partial {L}}{\partial {{{U^{t+1}_l}}}} \frac{\partial {{U^{t+1}_l}}}{\partial {{U^t_l}}}$. Combine all these, we can get Equations 11.

The proof of Equation 14 is the same as Equation 11.

For Equation 15, the $\alpha$ only affects $U^t$, thus based on the derivation of Equation 11, it is easy to derive it.

Concern 5: Equations 14 and 11 are the same?

R5: Sorry for the confusion. $\mathbf{W}^b_{\rm trainble} = \alpha \mathbf{W}^b$, and we calculate $\alpha$ and $W$ separately in Equations 14 and 15. Thus Equations 14 and 11 are the same in form.

Concern 6: The firing activity becomes differentiable" should be further justified.

R6: Thanks for the advice. Compared to binary activation, our real-valued activation becomes differentiable in more intervals. We overclaimed this in the paper. We will correct this in the final version. Thanks.

Concern 7: The firing threshold is said to be set to 0?

R7: Sorry for the confusion. In these static datasets, it is 0 all the time, since static datasets can not provide timing information. For neuromorphic datasets, we set it to 0.25.

Concern 8: the experimental settings are missing.

R8: Thanks for the question. We will clarify the code settings in detail in the final version.

Concern 9: In Table 5, the figures for learnable ReverB are missing.

R9: Thanks for the question. We add the results for learnable ReverB below.

Concern 10: Why $\alpha \in \mathbb{R}^{C \times 1 \times 1}$

R10: Sorry for the confusion. $\alpha$ is in a channel-wise manner for convolution layers.

Concern 11: Some elements in Algorithm 1 are not clear.

R11: Sorry for the confusion. The for loop is over mini-batches. With the $\alpha$ folded to i-1 function, then we can obtain a standard ReverB-SNN. We will correct the Inference to Evaluation in the Algorithm. Thanks.

Concern 12: Other Comments Or Suggestions.

R12: Very much thanks for these kind reminders. We will carefully correct these in our final version.

Question 1: Why considering binary weights not ternary weights.

A1: Thanks for the question. Using ternary weights is better than binary weights in our experiments. However, we only report the binary weight and real-valued activate results for a fair comparison of vanilla real-valued weight and binary activate SNNs.

Question 2: How to deploy the proposed model to neuromorphic hardware.

A2: Thanks for the question. Please see our response for Concern 2.

Thanks for your efforts in reviewing our paper and your recognition of our novel method and notable results. The responses to your concerns and questions are given piece by piece as follows.

Concern 1: It would have been useful to also see evaluations on sequence tasks since SNNs are inherently recurrent.

R1: Thanks for the advice. Our method also performs well in sequence tasks. We have added the result on the CIFAR10-DVS and DVS-Gesture as below. It can be seen that our method also performs on par with or better than state-of-the-art methods.

Concern 2: Why do the authors choose to use STBP to train the network rather than standard BPTT?

R2: Sorry for the confusion. Considering the similarity in computational mechanisms between SNNs and Recurrent Neural Networks (RNNs), SNN researchers transferred the Back-propagation Through Time (BPTT) method from RNNs to the supervised learning field of SNNs, which is also called the STBP training algorithm. Thus BPTT is the same as STPB in SNNs.

Concern 3: One minor weakness is the framing of the method within the current literature.

R3: Thanks for the advice. We will add more related literature and reframe them in the final version, such as:

[1] Subramoney, A., Nazeer, K. K., Schöne, M., Mayr, C. & Kappel, D. Efficient recurrent architectures through activity sparsity and sparse back-propagation through time. in The Eleventh International Conference on Learning Representations (2023).

[2] Maass, W. Networks of spiking neurons: The third generation of neural network models. Neural Networks 10, 1659–1671 (1997).

Concern 4: It's not clearly described if a value of $\alpha$ is shared across an entire layer.

R4: Sorry for the confusion. $\alpha$ is not shared across an entire layer. It is applied in a channel-wise manner across our models. We will make this description clearer in the final version.

Questions  1: What is the activation used in between layers?

A1: Sorry for the confusion. In addition inherently recurrent of our activation, the output of our activation is a ReLU-like function defined as follows: ${\rm O}= {\rm U} \\ {\rm if U} \\ \ge V_{\rm th}, \\ { \rm otherwise} \\ 0$

In Relu activation, the $V_{\rm th}$ is 0, while in our activation, the $V_{\rm th}$ can be adjusted across different datasets or scenes. Our activation will decay through time, while ReLU not.