We are cheerful that our contribution is well recognized. And thanks for your valuable suggestions that truly enhance the quality of our paper and make it more understandable for the wider community.

Responses to your concerns are presented as follows:

1.ImageNet Dataset (W1, Q2)

Thanks for your suggestion. We have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows:

  |    $16.81$     |   $70.24$%    |

| Spikformer w/ CPG-PE $ours$ | $15.66$ | $71.17$% |

Specifically, we set the depth to 8 and the dimension of representation to 384. For other experimental settings, we have faithfully followed the guidelines in $1$ . From the table, we can see that CPG-PE performs well on large-scale image datasets. However, due to time constraints, we were unable to conduct experiments on Random PE or Float PE. We believe that the above results demonstrate the effectiveness of our proposed CPG-PE in positional encoding. We will add these experiment results to Table 3 in our revised manuscript.

2.Discussion on temporal coding and rate coding (W2, Q1)

Thanks for your suggestion.

(1) We carefully reviewed the literature you provided and found your suggestions valuable for improving the quality of our paper. In the revised manuscript, we will include the following discussion in the related work section (due to character limitation, we omit some references but will include them in the revised manuscript):

"Spiking Neural Networks (SNNs) employ several coding methods to encode input information, each offering unique advantages. Direct coding $1$

$2$ , the simplest form and widely-used in image tasks, directly associates spikes with specific values or events, providing straightforward and interpretable outputs but often lacking efficiency for complex tasks. Rate coding $3$

$4$ , where the input is represented by the frequency of spikes within a given timeframe, is more robust and widely used but can be less precise due to its reliance on averaged spike rates. Temporal coding $5$

$6$ (a.k.a latency coding) encodes information based on the timing of individual spikes, allowing for high temporal precision and efficient representation of dynamic inputs, though it can be computationally demanding. In addition, Delta coding $7$ represents changes in input signals through spikes, focusing on differences rather than absolute values, which can enhance efficiency and response times but may introduce complexity in decoding. Each of these methods contributes to the versatility and applicability of SNNs in various domains, from neuroscience to artificial intelligence. The SNNs we considered in this paper should fall into the category of rate coding since back-prop is conducted on spike rate. Meanwhile, CPG-PE can be considered converting temporal information into spike rate of a group of neurons (Equations 11-12), and this is why CPG-PE can improve performance for sequential data. It is possible to introducing learning algorithms of temporal coding for the CPG neurons to tackle more complex sequence structure, which remains as future work."

(2) In Related Works Section, we have discussed the positional encoding used in SNNs.

3.Typo (Q3)

Thanks for your reminding. We will add commas after formulas 4-10 to ensure readers are not misled.

Reference

[1] Zhou Z, Zhu Y, He C, et al. Spikformer: When Spiking Neural Network Meets Transformer[C]//The Eleventh International Conference on Learning Representations (ICLR). 2023.

[2] Yao M, Hu J, Zhou Z, et al. Spike-driven transformer[J]. Advances in Neural Information Processing Systems (NeurIPS), 2023.

[3] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75.

[4] Lv C, Xu J, Zheng X. Spiking Convolutional Neural Networks for Text Classification[C]. // The Eleventh International Conference on Learning Representations (ICLR). 2023.

[5] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404.

[6] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.

[7] Yoon Y C. LIF and simplified SRM neurons encode signals into spikes via a form of asynchronous pulse sigma–delta modulation[J]. IEEE transactions on neural networks and learning systems, 2016, 28(5): 1192-1205.

We appreciate your comments and suggestions that enhanced the quality of our paper. Responses to your concerns and questions are hereby presented:

1.Typographical error (W1)

Apologies for the confusion caused by this typo error. In line 158, the $X$ is the input spike matrix and it belongs to the binary domain $\{0,1\}$ rather than real number domain $\mathbb{R}$, i.e., $X \in \{0,1\}^{T \times B \times L \times D}$. This error will be corrected in our revised manuscript.

2.Similarities and differences between the decay mechanism of CPG-PE and that of LIF neurons (W2)

(1) If we understand your question correctly (please kindly let us know if we misunderstood), the question is about the differences between the decay mechanisms of CPG neurons and LIF neurons instead of CPG-PE . To be clear, CPG indicates that the neuron (together with other neurons in a CPG circuit) plays the role of rhythmic output generator, while LIF is a specific dynamics of spiking neuron model. CPG-PE is our propose method which leverages the spikes of CPG neurons for encoding positional information in sequential data. Therefore, CPG-PE and LIF neurons are orthogonal concepts, and we have proved that CPG-PE could be composed of LIF neurons (as shown in Appendix C), or be manually set (for simplicity, as used in our experiments).

(2) Regarding the decay mechanism, as shown in Equation (2), the $\beta$ control the decay rate of LIF neurons. However, our proposed CPG-PE does not have a decay mechanism. We would greatly appreciate it if you could provide further clarification on this issue, and we are happy to address it.

Please let us know if you have any remaining questions or concerns, we are commited to addressing the issues.

Thank you for your comments and suggestions, which are valuable for enhancing our paper. We are pleased that our contributions are well recognized.

Responses to your concerns and questions are hereby presented:

1.Ablation experiments on parameters (W1).

Thanks for your suggestion.

(1) Firstly, while our method increases the parameter count compared to the version without positional encoding (w/o PE), this increase is relatively small. For example, on CIFAR10, the paramter numbers of Spikformer with CPG-PE are only $0.17$M(~$2.17$% of the total parameters) larger than that of Spikformer w/o PE.

(2) Secondly, we conducted ablation experiments by reducing the parameter count of Spikformer with CPG-PE to be comparable to Spikformer w/o PE, allowing for a more direct performance comparison. The results are as follows:：

Specifically, we adjusted the representation dimension of Spikformer with CPG-PE to make the parameter counts similar to those of Spikformer without PE. For instance, on CIFAR10, we changed the dimension from $384$ to $380$, resulting in a parameter count of $7.99$M, which is almost equal to $8.00$M. As shown in the table above, we can conclude that when the parameter counts are equal, Spikformer with CPG-PE significantly outperforms Spikformer without PE across all image classification datasets, proving our proposed CPG-PE is effective.

2.ImageNet Dataset (W2).

Thanks for your suggestion. We have conducted experiments with Spikformer without positional encoding (PE), Spikformer with relative positional encoding (RPE), and Spikformer with our proposed CPG-PE on the ImageNet dataset. The results are as follows:

  |    $16.81$     |   $70.24$%    |

| Spikformer w/ CPG-PE $ours$ | $15.66$ | $71.17$% |

Specifically, we set the depth to 8 and the dimension of representation to 384. From the table, we can see that CPG-PE performs well on large-scale image datasets. However, due to time constraints, we were unable to conduct experiments on Random PE or Float PE. We believe that the above results demonstrate the effectiveness of our proposed CPG-PE in positional encoding. We will add these experiment results to Table 3 in our revised manuscript.

3.Similarities and differences between CPG-PE and addition-based PE in ANNs (Q1).

In ANNs, especially in language tasks, the difference between using addition and concatenation for positional encoding is not significant $2$

$3$ . However, in spiking neural networks (SNNs), addition is not suitable for spike matrices because it can result in non-binary values, which is one of the motivations for designing SNN-PE. This paper addresses this issue by first concatenating the spike matrix and then using a projection layer to project it back to the initial dimension.

Reference

[1] Zhou Z, Zhu Y, He C, et al. Spikformer: When Spiking Neural Network Meets Transformer[C]//The Eleventh International Conference on Learning Representations (ICLR). 2023.

[2] Rosendahl J, Tran V A K, Wang W, et al. Analysis of positional encodings for neural machine translation[C]//Proceedings of the 16th International Conference on Spoken Language Translation. Association for Computational Linguistics. 2019.

[3] Ke G, He D, Liu T Y. Rethinking Positional Encoding in Language Pre-training[C]//International Conference on Learning Representations (ICLR). 2021.

Thank you for your detailed response. The experiments have alleviated my concerns about the overhead of CPG-PE. The further validation on the large-scale ImageNet has demonstrated its generalization and effectiveness. Therefore, I would like to increase my score.

Thanks for your comments and suggestions, which are valuable for enhancing our paper. The follows are responses to your individual concerns:

1.Why is positional encoding important to SNNs (W1)?

Currently, SNNs have been applied to a variety of tasks beyond image processing, including time-series forecasting [1] and text classification [2]. However, due to lack of suitable positional encoding (PE) method, it remains challenging for SNNs to capture indexing information, rhythmic patterns, and periodic data. Existing PE methods for SNNs do not meet the essential criteria: uniqueness of each position and formulation in spike-form. Inspired by CPG neurons, we propose CPG-PE, which performs well on both sequential and image tasks. Additionally, we ensure its hardware-friendliness and provide insights into the role of CPGs in neuroscience (Section 6).

2.The method is simple and does not provide enough inspiring insights; contribution is relatively limited. (W2).

"Everything should be made as simple as possible, but no simpler." – Albert Einstein

(1) We believe simplicity is a strength of our method (as it is effective), not a weakness. As it is easy to implement, there is a higher probability that more researchers can benefit from leveraging our method on their model to improve performance. Furthermore, since CPG-PE is novel to the SNN community, complex models and algorithms with even better capacity may emerge following our work, but it is necessary that we first comprehensively address that a simple CPG-PE can consistently improve the performance of SNNs.

(2) Our contributions can be summarized as: (1) Pioneer in the exploration of spike-version PE for SNNs. To the best of our knowledge, our proposed CPG-PE is the first work on spike-version PE in SNNs, which are both brain-inspired and hardware-friendly. We corrected the inappropriate PE designs previously used in SNNs. (2) Consistent Performance Gain. CPG-PE enhances the performance of SNNs across a wide range of tasks, including time-series forecasting, text classification, and image classification. (3) Insights on neuroscience and neuromorphic hardware. We mathematically proved that the traditional PE in Transformers is a particular solution of the membrane potential variations in a specific type of CPG, and we also proved that CPG-PE can be implemented with 2 LIF neurons so that it will not introduce any burden of redesigning hardware.

3.Frequent updates of neuron membrane potentials bring additional floating-piont operations and energy on hardware (W3).

We fully agree, and indeed, we have discussed this aspect in our paper.

(1) Physical Implementation of CPG-PE We have proposed an approach to physically implement CPG-PE with only 2 LIF neurons, introducing no extra efforts on chip designs. A CPG-PE neuron can be viewed as an autonomic neuron that will emit a burst of $K$ spikes after resting for $R$ time steps. Through rigorous derivation (see Appendix C), we prove that the system can behave periodically with a period $T = (R + K)∆T$, i.e., CPG-PE can be implemented with 2 LIF neurons. This approach eliminates the need for additional floating-point computations and memory read-write operations on neuromorphic hardware to frequently update neuron membrane potentials.

(2) Support from Existing Circuit Implementations Existing circuit implementations on hardware [3][4] strongly support our proposed CPG-PE, as these works are based on the similar idea of coupled LIF neurons. Circuits based on complementary metal-oxide-semiconductor technology [3] simplify membrane potential calculations, reduce power consumption significantly, and eliminate the need for additional memory to store membrane potentials. Furthermore, memristor-based CPG circuits [4] have been shown to minimize circuit complexity and increase energy efficiency. Therefore, we believe that our proposed approach is hardware-friendly, without requiring additional operations on hardware, which has also been acknowledged by other reviewers (3Lmi, s3xa, Pjez).

4.Misleading in Section 3.1 (W4).

We apologize for this typo and the resulting confusion. It should be: "$\mathbf{F}(\mathbf{x})=b>0, \mathbf{H}(\mathbf{y})=d>0$." After our thorough inspection and verification, we confirm that this does not affect the derivation results, and Equations 6-10 remain valid. We will correct this typo in our revised manuscript.

5.Why is the performance of CPG-Full not as good as that of CPG-PE (W5).

(1) CPG-Full stands for replacing all linear layers with CPG-Linear layers. CPG-Linear was introduced in Appendix D. The original CPG-PE models the inputs, while CPG-Full models the hidden states.

(2) The difference between the two is not significant. In terms of metrics, the average $R^2$ is identical, and the $RSE$ difference is not statistically significant (p-value=$0.9351$, student t-test).

(3) We conducted this experiment for two main reasons: Firstly, to explore how to implement modular CPG, hypothesizing that simply replacing Linear with CPG-Linear would allow the PE functionality to work effectively. Secondly, this experiment is supplementary, aiming to confirm that the performance improvement of SNNs is due to the functionality of PE rather than an enhancement in representation capability, as CPG-Full does not affect representation capability.

Please kindly let us know if there is any remaining concern. We are committed to addressing your concerns.

[1] Lv C, et al. Efficient and Effective Time-Series Forecasting with Spiking Neural Networks. ICML, 2024.

[2] Bal M, et al. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. AAAI, 2024.

[3] Vogelstein J, et al. Dynamic control of the central pattern generator for locomotion. Biological Cybernetics, 2006.

[4] Dutta S, et al. Programmable coupled oscillators for synchronized locomotion. Nature Communications, 2019.