EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature

Thank you for your efforts in reviewing our paper and your recognition of our good writing, technical soundness experimental results, and novel methods. The response to your questions is given piece by piece as follows.

W1: How can SNN representation be converted to float activation in eq 1? Is it averaged?

A1: Sorry for the confusion. Yes, it is averaged through timesteps. We will further clarify this as per your suggestion in the revised version. Thanks again.

---

W2: Improvement with technique 2 is expected, and it makes the comparison with pure SNNs unfair.

A2: Sorry for the confusion. Though the technique 2 will increase the energy consumption， it is very simple and effective. In the appendix, we show that the method just brings about a 1.0% energy consumption uplift. Thus the additional energy consumption could be ignored. In this sense, the comparison could be seen as fair.

---

W3: The method needs to train an ANN in addition to the SNN. So the total training time is longer.

A3: Sorry for the confusion. The additional training cost is trivial. We take the ResNet20 with 4 timesteps on CIFAR10 to compare. The training time of ANN is about 6.25% of that of SNN on the same GPU, as follows: since the memory consumption of SNN is 4 times that of ANN, ANN can use a batch size that is 4 times that of SNN, which means that the number of iterations required for ANN training is 25% of that of SNN. In addition, since timestep affects SNN, the training time of ANN in the same iteration is 25% of that of SNN. Overall, the training time of ANN is 6.25% of that of SNN. We report the training cost for 400 epochs running using spiking ResNet20 (batch=128) on CIFAR10 for our methods and the training cost for 100 epochs running using ResNet20 (batch=512) with the same single V100 here. The costs are 8348.87s and 561.77s. It can be seen that the additional training cost is trivial.

Thank you for your efforts in reviewing our paper and your recognition of our simple but effective method. The response to your questions is given piece by piece as follows.

---

W1: To illustrate the advances made by EnOF and RepAct, the authors can provide the network Hessian matrix eigenvalues or convergence speed comparisons.

A1: Thanks for the advice. We will add the convergence speed comparisons as per your suggestion in the revised version. Thanks again.

---

W2: I suggest that the authors move the overhead comparison section from the appendix to the main text.

A2: Thanks for the advice. We will add the overhead comparison section to the main text as your suggestion in the revised version. Thanks again.

---

W3: The main text lacks the definition of the method name "EnOF.”.

A3: Sorry for the confusion. “EnOF” comes from the first letter of “Enhancing the Output Feature” from the title. We will further clarify it in the revised version. Thanks again.

---

Q1: Does distillation learning have a temperature coefficient?

A1: Sorry for the confusion. The distillation learning has a temperature coefficient. We set it as 0.1 for CIFAR dataset and 1 for ImageNet dataset.

---

Q2: Is the optimal $\lambda$ obtained through search, and does different $\lambda$ have a significant impact on the final results?

A2: Thanks for the question. The hyperparameter we used in the paper is 0.1 for CIFAR dataset and 1 for ImageNet dataset. We get it just from a simple search strategy that chooses several hyperparameters and compares their results. Here we also report the results of 0.5 and 1.0 for CIFAR10. It can be seen that the hyperparameter will affect the results. Nevertheless, Using the KD is better than the Baseline.

---

Thank you for your efforts in reviewing our paper and your recognition of our good writing and technical soundness experimental results. The response to your questions is given piece by piece as follows.

W1: The paper could benefit from a deeper exploration of how these specific adaptations contribute uniquely to the SNNs' context.

A1: Thanks for your advice. The starting point of our research is that a rich output feature representation is important for improving the accuracy of SNN first. Based on this perspective, we propose the L_{AF} Loss and RepAct.

For the L_{AF} Loss, it is not merely a distillation method. The primary motivation behind its proposal is that many ANN-SNN works do not alter the weights of neural networks yet achieve high accuracy. This insight has inspired us to incorporate certain ANN modules into SNN. However, direct migration and training are incompatible. Hence, we introduce the concept of utilizing the classifier of ANN to optimize the output feature of SNN. We have chosen to formalize our idea through distillation and, in the supplementary materials, we provide evidence of its significant advantages over direct distillation. Thus, from a systemic standpoint, the method we propose is a distillation approach specifically tailored for SNN, which fundamentally differs from simple distillation despite their superficial similarities.

For the RepAct, this method is very simple but effective. Maybe it is not very novel, but will provide a novel insight for the SNN field that enhancing the output feature is very useful for accuracy.

We will further clarify this in the revised version as your suggestion. Thanks.

---

W2: The paper does not adhere to NeurIPS citation format requirements.

A2: Thanks for the reminder. We will further recheck and polish our paper in the final version.

---

W3: The paper lacks a comprehensive comparison with other SNN training methods that use similar KD approaches.

A3: Thanks for your advice. Here we add some comparison results for the same models with [1] and [2] based on their reported results. It can be seen that our method works well on ImageNet with ResNet-18 and on CIFAR10 with ResNet-19, but a little worse on CIFAR10 with ResNet-18. We will report more comparisons in the revised version. Thank you.

---

W4: The improvements, while statistically valuable, may not represent a paradigm shift in SNN training that the field might expect for a high-impact publication.

A4: Sorry for the confusion. We think our most valuable contribution is providing insight for the SNN field that enhancing the output feature is much useful for accuracy. Our $L_{AF}$ and RepAct both are means, while our intention is to improve the output feature and prove that it is beneficial for boosting accuracy. With verified experiments and under the perspective, we believe this work is much beneficial for the following SNN work. And many other methods to improve the output feature would be proposed, like special temporal contrastive loss or simply increasing channels for SNNs. Yet, this is not to say our methods are not novel. For RepAct, it is very simple, but effective. With a trivial additional cost, it can improve the SNN greatly. From a practical perspective, this is a really good choice for the industry. For $L_{AF}$ it is a simple and effective KD method designed for SNN, different from these straightforward KDs. In specific, we introduce its novelty in two-fold. First, providing a simple but effective KD for SNNs is difficult. Most of KDs perform poorly without well-designed hyper-parameters, which have been comprehensively and systematically studied in [1] (see Tab.1 and Tab.2 in [1]. This paper gets more than 700 cites and 1.8k stars). Most KD methods only work well on small datasets, especially in the SNN field. Second, our $L_{AF}$ comes from both the KDs and ANN-SNN methods. Some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN to homogeneous SNN with minor changes can obtain a well-performed SNN. This inspires us that utilizing homogeneous ANN's some elements to guide the learning of SNN may be useful. Along with considering the classification and representation, we utilize ANN’s pre-trained classifier to train the SNN’s feature and it works well. Thus, we think $L_{AF}$ is new, simple, designed specifically for SNNs.

[1] Tian, Y., Krishnan, D., and Isola, P. Contrastive representation distillation. In International Conference on Learning Representations, 2020.

---

Q1: Could the authors extend their comparisons to include seminal works in SNN+KD such as those by Guo et al. [1] and Deng et al. [2]?

A1: Thanks for your advice. Please see our response for the W3.

---

Q2: The ablation study in Table 6 is critical, and the details of the hyperparameters used in the vanilla KD need to be listed.

A2: Thanks for your advice. The hyperparameter we used in ablation is 0.1. Here we also report the results of 0.5 and 1.0. It can be seen that the hyperparameter will affect the results. However, the effect is smaller on ours.

---

Thank you for your efforts in reviewing our paper and your recognition of our effective method and good results. The response to your questions is given piece by piece as follows.

---

W1: The proposed method requires training the ANN counterpart first and setting it aside as a pre-trained model. Subsequently, the SNN needs to be trained. This approach essentially involves training the same framework twice, once with ANN activations and once with SNN activations.

A1: Sorry for this confusion. The additional training cost is trivial. We take the ResNet20 with 4 timesteps on CIFAR10 to compare. The training time of ANN is about 6.25% of that of SNN on the same GPU, as follows: since the memory consumption of SNN is 4 times that of ANN, ANN can use a batch size that is 4 times that of SNN, which means that the number of iterations required for ANN training is 25% of that of SNN. In addition, since timestep affects SNN, the training time of ANN in the same iteration is 25% of that of SNN. Overall, the training time of ANN is 6.25% of that of SNN. We report the training cost for 400 epochs running using spiking ResNet20 (batch=128) on CIFAR10 for our methods and the training cost for 100 epochs running using ResNet20 (batch=512) with the same single V100 here. The costs are 8348.87s and 561.77s. It can be seen that the additional training cost is trivial.

---

W2: The method includes replacing the last LIF activation neuron layer in the SNN with a ReLU activation layer, which results in significant improvement. However, it is unclear whether the final improvement is primarily due to this replacement of last-layer's activation function or the feature alignment process.

A2: Sorry for the confusion. From the ablation study, it can be seen that both the RepAct method and the alignment method can gain huge improvements alone. However, the accuracy upper of a network is limited, and the difficulty will increase quickly with the accuracy reaching the upper. Thus using them both can not get twice the profit. The same phenomenon can also be seen in many works, e.g., IM-Loss (NeurIPS2022), using IM-Loss and ESG simultaneously would not bring much gain compared with a single.

Guo Y, Chen Y, Zhang L, et al. IM-loss: information maximization loss for spiking neural networks[J]. Advances in Neural Information Processing Systems, 2022, 35: 156-166.

---

Q1: it seems the paper trains both the ANN counterpart and the SNN, the training resources will be doubled, how to solve this problem?

A1: Sorry for this confusion. Please see our response to W1.

---

Q2: Does the improvement mainly come from this replacement of LIF with ReLU?

A2: Sorry for this confusion. Please see our response to W2.

---

Q3: In order to check the improvement for the applications, can the authors provide the performance of "w/ RepAct" and "w/ L_{AF}", "w/ L_{AF} & RepAct", the baseline, and the ANN-counterpart for CIFAR10, CIFAR100, ImageNet, CIFAR10-DVS datasets?

A3: Thanks for the advice. Here we add more experiments as your suggestion. Note that we use the ResNet20 architecture on CIFAR10 (with 4 timestep), CIFAR100 (with 4 timestep), and CIFAR10-DVS (with 10 timestep) and ResNet18 architecture on ImageNet (with 4 timestep).

---

Q4: Grammer errors, for example Line 42-44, "Despite the SNN being more energy-efficient compared with the ANN, it suffers unsatisfactory task performance, due to binary spike feature maps of the SNN will result in limited expressiveness and large accuracy degradation compared with full-precision feature maps of the ANN",

A4: Thanks for the reminder. We will further recheck and polish our paper in the final version.