Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks

Thanks for your efforts in reviewing our paper and your recognition of our novel method, notable results, and good writing. The response to your questions is given piece by piece as follows.

W1: The author should add more mathematical proof to demonstrate that the mentioned residual structure in SNN is not very effective? The introduction of shortcut branches might add complexity to the network architecture, which could affect the interpretability of the model.

A1: Thanks for this advice. The residual structure cannot solve the gradient vanishing problem of SNNs completely. For standard skip connections, it can be expressed as $o=g(f(x)+x)$, where f(x) is convolutional layers and g() is the activation function. The standard ResNet network is composed of multiple skip connection blocks cascaded together. In ANN, ReLU is used for g(), since ReLU is unbounded for the positive part, the gradient can be passed to the input of the block directly. However, in the case of LIF neurons in SNNs, the gradient will be reduced through the surrogate gradient. Thus, the skip connections in standard ANNs still suffer the gradient vanishing problem in SNNs, as Figure 2 in the paper also visually illustrates this problem. However, our shortcut back method can transfer the gradient from the output to the input of the block directly in SNNs.

Furthermore, the proposed methods are only alive in the training phase and will not affect the inference phase of SNN.

---

W2: Some recent SOTA works should be compared with too. The authors can also compare with paper [1][2] which obtains really good results by MS-ResNet-18 backbone with 1 or 6 timesteps on large imageNet datasets.

A2: Thanks for this advice. We will add comparisons of these recent methods in the revised version as your suggestion. Thanks.

---

Q1: Why are the bolded values not always the best values?

A1: Sorry for the confusion. With the same model and same timesteps, our method reaches the best. We will further clarify this in the revised version.

Thanks for your efforts in reviewing our paper and your recognition of our simple method, notable results, and good writing. The response to your questions is given piece by piece as follows.

W1: In this paper, the author only demonstrates a change in gradient distribution in the first layer. Presenting the changes in the men and variance of the absolute gradients for each layer would provide a more direct proof of their argument.

A1: Thanks for this advice. Here we add the changes in the men and variance of the absolute gradients for the first 10 layers in vanilla ResNet18 our ResNet18 respectively on the CIFAR-100. It can be seen that our ResNet18 could handle the gradient vanishing problem well.

---

W2: The author should provide a more detailed mathematical proof to explain why the use of surrogate gradients in deep SNN would lead to gradient vanishing, as well as why direct use of residual learning will not address the problem.

A2: Sorry for the confusion. The residual structure cannot solve the gradient vanishing problem of SNNs completely. For standard skip connections, it can be expressed as $o=g(f(x)+x)$, where f(x) is convolutional layers and g() is the activation function. The standard ResNet network is composed of multiple skip connection blocks cascaded together. In ANN, ReLU is used for g(), since ReLU is unbounded for the positive part, the gradient can be passed to the input of the block directly. However, in the case of LIF neurons in SNNs, the gradient will be reduced through the surrogate gradient. Thus, the skip connections in standard ANNs still suffer the gradient vanishing problem in SNNs, as Figure 2 in the paper also visually illustrates this problem. However, our shortcut back method can transfer the gradient from the output to the input of the block directly in SNNs.

---

W3: The author has not demonstrated their method on much deeper network architectures where the gradient vanishing problem is more severe.

A3: Thanks for this advice. We have added more experiments for deeper network architectures. It can be seen that our method also works well with these architectures.

Q1: How is the network divided into multiple blocks? Are there any additional rules for the insertion position and number of early classification heads?

W1: Thanks for the question. These modern architectures are usually partitioned into different stages corresponding to a downsample of the input feature map and an increase of the channel numbers. We add the branchs to the exit of these stages.

---

Q2: The results of using short-BP to train ResNet 18 in Table 1 and Table 2 are quite different. There may be a transcription error here.

W2: Thanks for the advice. We confused the results of the Evolutionary Training and the results of the Shortcut Back-propagation. We will correct it in the revised version. Thank you again.

Thanks for your efforts in reviewing our paper and your recognition of our interesting ideas, notable results, and good writing. The response to your questions is given piece by piece as follows.

W1: The proposed method will increase the training time.

A1: Thanks for this question. The additional training time is trivial. Here we add the training time cost comparison of Vanilla SNN and our Shortcut Back-propagation SNN for 1 epoch training with batchsize as 128 on CIFAR10 using ResNet20 with a single V100. Only about 10% of training cost is induced.

---

Q2: In the experimental section, some newer methods should be compared with this method.

A2: Thanks for the advice. We will add more recent methods like these in the below papers in the revised version.

[1]Yao M, Zhao G, Zhang H, et al. Attention spiking neural networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2023.

[2] Qiu X, Zhu R J, Chou Y, et al. Gated attention coding for training high-performance and efficient spiking neural networks[C]. Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(1): 601-610.

---

Q3: Figure 2 lacks horizontal and vertical coordinates, and the readability and comprehensibility of the picture need to be improved.

A3: Thanks for the advice. We have improved the picture as you suggested. Please see the general response.

---

W1: Does the proposed method lead to an increase in the calculation of gradient backpropagation? Howmuch is the increased training time.

A1: Thanks for this question. The additional training time is trivial. Here we add the training time cost comparison of Vanilla SNN and our Shortcut Back-propagation SNN for 1 epoch training with batchsize as 128 on CIFAR10 using ResNet20 with a single V100. Only about 10% of training cost is induced.