ASRC-SNN: Adaptive Skip Recurrent Connection Spiking Neural Network

We thank the reviewer's feedback. We also hope the reviewer will take note of other strengths of the paper, such as the positive points mentioned by Reviewer 3VEv、 Reviewer keA4 and Reviewer 9KzL. Below, we provide point-by-point responses to the weeknesses.

Weakness 1: The ASRC model does not outperform existing methods such as PMSN [1] and TC-LIF [2] on the PS-MNIST dataset. This suggests that the ASRC approach may face challenges in handling certain types of spatiotemporal data, indicating a need for further optimization to enhance its generalizability across diverse tasks.

Response 1: I. We would like to clarify that, as shown in Table 1 of this paper, our method outperforms TC-LIF. We acknowledge that it does not surpass PMSN on the PS-MNIST dataset. However, we want to emphasize that there are many works on neurons in this field, and our approach is based on a new perspective that focuses on the synergistic role between recurrent structures and neurons.

II. We would like to clarify that this paper focuses on long-term sequence modeling. While extending it to other spatiotemporal tasks is valuable, it is beyond the scope of this paper.

Weakness 2: The ASRC raise the model's computational complexity and increase training time and escalating computational resources.

Response 2: Below, we present the computational overhead of our models on PS-MNIST, the most complex dataset in our study, as shown in the two tables. The increase in memory consumption with larger values of $T_{\lambda}/\lambda$ is significant, while the increase in training time is relatively marginal. The training time of ASRC-SNN is approximately 17% longer than SRC-SNN. When $T_{\lambda}$ and $\lambda$ are close, the memory consumption of ASRC-SNN is slightly higher than SRC-SNN. Considering the trade-off between computational overhead and performance, we recommend selecting relatively small values of $T_{\lambda}/\lambda$ whenever possible. In this case, selecting $T_{\lambda} = 11$ and $\lambda = 8$ results in good model performance, while the additional computational overhead remains acceptable compared to the vanilla RSNN.

Table 1. computational metrics of ASRC-SNN on PS-MNIST

Table 2. computational metrics of SRC-SNN on PS-MNIST

We sincerely thank the reviewer 9KzL for his review. Below we provide point-by-point responses to the Weaknesses.

Weakness 1: The length of the temporal can be marked below the dataset to make it more intuitive.

Response 1: We thank the reviewer for the helpful suggestion and will implement them in our final manuscript.

Weakness 2: The accuracy is not better than PMSN PS-MNIST datasets, and the network is also not recurrent.

Response 2: Yes, you're right. However, we clarify that there is a lot of works on neurons in this field, and our approach is based on a new perspective that considers the synergistic role between recurrent structures and neurons.

Weakness 3: It could be more additional experiment by incoporating other neuron on ASRC architecture to prove effectiveness and generality. The authors claim that PLIF and GLIF have no effect on ASRC, which can be illustrated by the specific data.

Response 3: Considering that if the neurons differ too much from LIF neurons, the gradient propagation in the time dimension would need to be reanalyzed and would likely lead to different conclusions than those presented in this paper, we have only considered PLIF and GLIF neurons.

Below, we present the neuron ablation experiment results for SRC-SNN and ASRC-SNN. First, consider Table 2. When replacing LIF with PLIF in SRC-SNN, performance improves on PS-MNIST but decreases on SSC. The skip span $\lambda$ is set to 12 for PS-MNIST and 3 for SSC. On PS-MNIST, where the skip span is larger, an appropriate membrane potential decay factor can better regulate the flow of information transmitted through the membrane potential within the skip span, especially between the first skip connection linked to the current time step, leading to improved performance. In SSC, where the skip span is smaller, controlling membrane potential transmission is likely less critical than in larger spans, making a slight performance drop reasonable. GLIF introduces additional complexity, significantly complicating the backpropagation topology in the temporal dimension of SRC-SNN. Learning proper temporal dependencies is challenging, so replacing LIF with GLIF in SRC-SNN leads to a performance drop.

In ASRC-SNN, replacing LIF with PLIF does not improve performance, possibly because simultaneously learning both an optimal match between the skip span and the membrane potential factor while ensuring global optimization is challenging. Similarly, the complex gating mechanism of GLIF introduces more learnable parameters, making it even harder for the model to learn a good match between the skip span and GLIF parameters, which explains the significant performance degradation when using GLIF. Additionally, we note that replacing LIF with GLIF increases the training time on PS-MNIST by approximately 60%.

Table 1. Ablation study on neurons in ASRC-SNN

Table 2. Ablation study on neurons in SRC-SNN

Weakness 4: This work lack more deeper theory insight or more additional ablation experiment to explain more essentially. It could expolre the performance relationship bewteen the decay term $\alpha$ of LIF and $\lambda/T_{\lambda}$ on the SRC/ASRC, there may show be some nature pattern or trend, because the LIF suffer from the temporal gradients vanishing problem, and the different $\alpha$ could influence the how fast the gradient in the temporal dimension disappears, and how $\lambda$ will solve this problem.

Response 4: We thank the reviewer for the helpful suggestions. I. We are training our model on sequential CIFAR(timesteps=1024), which takes a lot of time. We will provide the training results later. Additionally, we found that ASRC-SNN is more effective than SRC-SNN under sparse connectivity. Please refer to "Response 4" to Reviewer 3VEv.

II. Expolring the performance relationship bewteen the decay term $\alpha$ of LIF and $\lambda/T_{\lambda}$ on the SRC/ASRC requires extensive additional experimentation. We will include these detailed analyses in our final manuscript.

Weakness 5: The increased GPU memory and training time of the corresponding w SRC and w ASRC of LIF can be given correspondingly as a reference.

Response 5: Please refer to "Response 2" to wx2U.

We sincerely appreciate the reviewer's thorough and insightful comments. Below we provide point-by-point responses to the Questions.

Questions 1: What is the computational overhead of learning adaptive skip spans compared to using fixed skips or standard recurrent connections? A runtime comparison would clarify whether ASRC is practical for large-scale problems.

Response 1: Please refer to "Response 2" to Reviewer wx2U.

Questions 2: How does ASRC-SNN perform on larger, real-world spiking datasets such as event-based vision tasks? Would the method still be effective when applied to more complex sequences?

Response 2: I. This is a valuable direction to comprehensively assess the generalizability of our approach. However, to the best of our knowledge, the mainstream architectures used for event-based vision benchmarks, such as CIFAR10-DVS and DVS-Gesture, are not recurrent. This suggests that we may need to explore suitable new recurrent architectures. Additionally, our study focuses on long-term temporal modeling, whereas the timesteps for CIFAR10-DVS and DVS-Gesture are typically set to no more than 20. In summary, this is beyond the scope of this paper.

II. We are training our model on sequential CIFAR(timesteps=1024), which takes a lot of time. We will provide the training results later.

Questions 3: How sensitive is ASRC-SNN to hyperparameter tuning? Does the softmax temperature decay require careful selection, or is the method robust across different datasets?

Response 3: I. In our final setup, the learning rate of the softmax kernel is set to 100 times the global learning rate. Under this setting, ASRC-SNN is robust to the choice of the softmax temperature decay. Additionally, we find that setting the softmax kernel's learning rate to 1× or 10× the global learning rate requires careful decay rate adjustment.

II. We did not carefully select the softmax temperature decay. As mentioned in the last part of Section 3.4.1 of this paper: "In our experiments, the exponential decay factor is set to 0.96." Fine-tuning the decay precisely across different datasets could further improve ASRC-SNN.

Questions 4: How does ASRC compare to alternative approaches such as spiking GRUs or gated recurrent SNNs, which also attempt to mitigate gradient vanishing? Would incorporating gating mechanisms alongside skip connections improve performance further?

Response 4: I. The gating mechanism can create temporal shortcuts that help prevent gradient vanishing[1]. However, we believe these shortcuts are difficult to observe and inherently uncertain. SRC can creat fixed temporal shortcuts, while ASRC allows for flexible adjustment of the shortcut span.

II. Since this is not the core focus of this work and spiking GRUs do not have publicly available code, we have not explored this in SNNs. We will release our code after this paper is accepted, and we believe it is easy to extend.

[1] Dampfhoffer M, Mesquida T, Valentian A, et al. Investigating current-based and gating approaches for accurate and energy-efficient spiking recurrent neural networks[C]//International Conference on Artificial Neural Networks. Cham: Springer Nature Switzerland, 2022: 359-370.

Questions 5: Is there a risk that learned skip spans converge to suboptimal solutions? How does the model ensure that it selects useful temporal dependencies rather than defaulting to local minima?

Response 5: Thanks to the reviewer for the insightful questions. We've also given these issues some thought. First, we emphasize that the softmax kernel initially assigns equal weights to skip connections of different spans, with no manual bias. During training within an epoch, we do not guide the distribution of the softmax kernel’s weights. Only after completing an epoch do we slightly sharpen this distribution by reducing the temperature parameter. Therefore, these issues can be seen as related to whether certain parameters in BP-based neural networks will converge to local optima. Of course, we acknowledge that the convergence of the softmax kernel parameters is quite unique. This question may require knowledge from other fields, such as non-convex optimization theory, to answer. We can only state that our experimental results show that the convergence of the skip spans is good. Finally, we point out that the clear dynamic changes in the softmax kernel during training might provide material for research in other fields.

We sincerely appreciate the reviewer's comments, especially some valuable questions raised. We also hope the reviewer will take note of other strengths of the paper, such as the positive points mentioned by Reviewer keA4 and Reviewer 9KzL. Below, we provide point-by-point responses to the questions.

Questions 1: Experiments proved the effectiveness of SRC but not its generalization. For example, experiments lacking event-drive datasets, DVS-Gesture, CIFAR-DVS, and so on.

Response 1: Please refer to "Response 2" to Reviewer keA4.

Questions 2: SRC is simple and effective. Can each timestep have a different $T_{\lambda}$?

Response 2: Is the reviewer suggesting that different time steps adapt to different skip spans? If so, this would be a further extension of ASRC-SNN. We would like to point out that this extension introduces additional parameters: $T_{\lambda} \times \text{number of layers} \times \text{timesteps}$, whereas ASRC-SNN has $T_{\lambda} \times \text{number of layers}$ additional parameters compared to SRC-SNN. We will incorporate this experiment in our final manuscript as a reference. Additionally, we will release our code after the paper is accepted, and we believe it is easy to extend.

Questions 3: This paper lacks an ablation experiment where the performance of SRC increases with the increase of timesteps.

Response 3: We clarify that the timesteps for the datasets used in our paper are: GSC (101), SSC (250), SMNIST (784), and PS-MNIST (784). We will later provide experimental results of our models on sequential CIFAR with 1024 timesteps.

Questions 4: In terms of power consumption considerations, since each neuron remains dense for different timesteps of links, the power consumption remains the same as before. So, can Neuron's connections for different timesteps be sparse?

Response 4: Thank you for raising this great question! The experiments we conducted further demonstrate the advantage of ASRC-SNN. Below, we present our experimental results. Across different sparsity levels, ASRC-SNN consistently outperforms SRC-SNN. This advantage is particularly evident on the PS-MNIST dataset, which has complex temporal dependencies—where increasing sparsity further amplifies ASRC-SNN’s superiority over SRC-SNN. A possible reason is that sparse connectivity demands stronger temporal modeling capabilities, and ASRC-SNN is better suited to handle this challenge. Additionally, we observed that as sparsity increases, our model’s performance on SSC does not degrade significantly, which may be related to the tendency to overfit easily on this dataset.

Table 1. The performance of ASRC-SNN under different sparsity rates

Table 2. The performance of SRC-SNN under different sparsity rates

Questions 5: This paper lacks ablation experiments on different neurons with SRC. You may not simply describe "and the results show no performance improvement with these substitutions." The reasons behind it should be analyzed.

Response 5: Please refer to "response 3" to Reviewer 9KzL.