Spatio-Temporal Dependency-Aware Neuron Optimization for Spiking Neural Networks

Weakness 4: The proposed neuron models focus on long/short-term dependency. In this context, the results of the experiment on memory tasks are more desirable. Previous research such as Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks uses the sequential/pixel classification to evaluate the learning ability of proposed models. I suggest that the authors report these results with comparison to previous neuron models.

Thank you for this suggestion. Our method is based on the BPTT + direct encoding approach, which we have explicitly stated in the Introduction as:

However, this combination (BPTT + direct encoding) introduces varying requirements for temporal dependencies across the network spatial dimension—an aspect overlooked by existing research…. (Starting from line 54)

Therefore, the design of our approach and the targeted performance improvements are inherently tied to the BPTT + direct encoding methodology. To the best of our knowledge, long time-step memory tasks do not employ direct encoding methods, that is why we did not choose this dataset. We attempted to implement our method on the S-MNIST dataset reported in the paper you mentioned to evaluate its performance. However, since our approach relies on BPTT, and the S-MNIST data requires T=784 time steps for full representation, the runtime for BPTT becomes unacceptably long. This can be inferred from the BPTT backward propagation equation as follows,

$

\nabla_{\mathbf{X}}\mathbf{S} = \sum_{t=0}^T \sum_{i=0}^t \frac{\partial S[t]}{\partial V[i]} \frac{\partial V[i]}{\partial X[i]} = \sum_{t=0}^T \Theta'(V[t]-V_\text{th}) \cdot (1+\sum_{i=0}^t \exp(-\frac{i}{\tau}) \cdot \prod_{j=1}^i \kappa_h (V[t-j]))

$

where it involves T consecutive multiplications. Consequently, we were unable to provide corresponding performance results within the limited rebuttal period, and we apologize for this. Nonetheless, your suggestion presents a promising direction for future work, and we express our gratitude for this valuable insight.

The number of time-steps in the sequential classification task is the width of the image, e.g., T=32 for the sequential CIFAR10. I think that T=32 is not too large for BPTT?

Weakness 2: In lines 232-247, the authors try to explain how the increasing of v_th will influence the gradient. This explaination is very weak and ambiguous. It is better to use a quantitative analysis. For example, the authors could show the derivative of Eq.4 with respect to V_th to prove the monotonicity of the gradient.

Thank you very much for pointing out this issue, as it is highly beneficial for improving the quality of our work. In our previous manuscript, we adopted a qualitative analysis, and we acknowledge that this approach was not sufficiently rigorous. Therefore, in the revised version, we have incorporated your suggested approach by deriving Eq. 4 with respect to $V_{th}$ and proving its monotonicity. The conclusion indicates that LLIF neurons can enhance long-term dependencies on input data. However, due to the length of the derivation, we apologize for not including it here; please refer to Appendix C, lines 882-1065, which is an added section with highlighted in blue, of the revised manuscript for the detailed analysis.

Weakness 3: The design of Eq.6 and Eq.7 for the Long-LIF is discussed clearly that the negative inputs are regarded as the inhibitory signals and will increase the threshold. However, the motivation of Eq.9 for the Short-LIF neuron is not introduced. According to Eq.8, a large V_p will cause a smaller tau. How the short-term dependency is reflected in the neuronal dynamics?

Thank you for your valuable feedback. In fact, our design in SLIF contrasts with that of the LLIF model. In LLIF, we considered negative membrane potential as inhibitory signals and thus adapted the threshold accordingly. In SLIF, however, we regard overly high positive potentials as over-activation signals. Such potentials lead to higher residual membrane potentials under a soft reset mechanism, where the magnitude of the residual membrane potential reflects the strength of recent neuronal excitation. Furthermore, based on this excitation strength, we leverage the phenomenon of synaptic potentiation observed in biological neurons [1]—where neurons temporarily increase sensitivity to recent signals under strong stimulation while moderately reducing the influence of historical activations—and correlate this with the adaptation of the membrane time constant. This mechanism resembles a form of short-term memory effect.

Specifically, we have enhanced the description of this motivation as follows:

Short-LIF (SLIF). Strengthening short-term dependency requires neurons to enhance their sensitivity to recent time steps. Based on the synaptic potentiation phenomenon (Zucker & Regehr, 2002) observed in biological neurons—where a neuron temporarily increases its sensitivity to recent signals when subjected to strong stimulation while moderately reducing the influence of historical activations—the design in SLIF neurons contrasts with the inhibitory signal design in LLIF. In SLIF, we treat membrane potentials exceeding the threshold as signals of over-excitation, linking this signal to the sensitivity of signal reception in future time steps. This implies that in SLIF neurons, the rate of decay for past accumulated potentials at the current time step is positively correlated with the intensity of recent stimulation, enabling the neuron to pay more attention to recent inputs. Specifically, in SLIF, we employ a soft reset method (detailed in Appendix A, where the degree of recent stimulation is represented by the residual membrane potential, which refers to the neuron's potential after a soft reset. The decay rate of the potential is controlled by $\tau$. Thus, the SLIF process, as depicted in Fig. (1)(b), defines the residual membrane potential $V_p$ as per Eq. (8), while the dynamic relationship between the membrane time constant $\tau$ and the residual membrane potential is given by Eq. (9).

In the revised manuscript, this section has been highlighted in blue, located on lines 233-245.

[1] Zucker, Robert S., and Wade G. Regehr. "Short-term synaptic plasticity." Annual review of physiology, 2002.

(Continue to response Weakness 1)

From the above experiments, we observe that the algorithm's performance is only minimally affected when the tanh input retains one or two decimal places, and this impact is not necessarily detrimental to precision. In some cases, such as CIFAR-100 (T=4) with a 0.75% increase, Tiny-ImageNet (T=2) with a 0.12% increase, and DVS-Gesture (T=16) with a 0.35% increase, the model's performance even surpasses the baseline when retaining one decimal place. For cases where three or four decimal places are retained, the model consistently maintained its performance across all datasets and time step configurations. This demonstrates that, using the Loihi 2 chip as an example, our model can utilize the lookup table approach for computing tanh within the constrained memory size, enabling lossless deployment of the model. Actually, this phenomenon is understandable given the characteristics of SNNs. The forward propagation process in SNNs involves multiple conversions between continuous and discrete values, leading to unavoidable cumulative errors during forward propagation. By using different decimal precision levels for tanh, we introduce only a slight increase in cumulative error, which is dwarfed by the substantial inherent error accumulation in SNN forward propagation. Therefore, this behavior has an extremely minimal impact on the overall model performance. Next, we present existing works that utilize neuromorphic hardware to implement complex nonlinear operations, further supporting our conclusion.

4. Existing cases of complex nonlinear function computation using lookup tables. For example, the 5-th page in [3]:

"Alternatively, Tianjic [4] (a hybrid neuromorphic chip) discretized and iterated the ODE solving process with time steps and adopted a lookup table to realize nonlinear function solving, leading to the improvement in performance."

Other examples include [5, 6].

5. Summary. Based on the above analysis, we conclude that, considering the products currently available on the market, the tanh computation in our method can be implemented on neuromorphic hardware using a lookup table approach. Furthermore, taking the Loihi2 chip as an example, the discrete lookup table method introduces no accuracy loss in model deployment observed in our experiments above.

[1] Guo, Yufei, et al. "Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks." CVPR, 2022.

[2] https://download.intel.com/newsroom/2021/new-technologies/neuromorphic-computing-loihi-2-brief.pdf

[3] Yu, Fangwen, et al. "Brain-inspired multimodal hybrid neural network for robot place recognition." Science Robotics, 2023.

[4] Pei, Jing, et al. "Towards artificial general intelligence with hybrid Tianjic chip architecture." Nature, 2019.

[5] Siddique, Ali, Mang I. Vai, and Sio Hang Pun. "A low cost neuromorphic learning engine based on a high performance supervised SNN learning algorithm." Science Robotics, 2023.

[6] Siddique, Ali, Mang I. Vai, and Sio Hang Pun. "A low-cost, high-throughput neuromorphic computer for online SNN learning." Cluster Computing, 2024.

First and foremost, we sincerely appreciate your positive evaluation of the strengths of our work. Below, we have provided responses to the concerns you raised.

Weakness 1: The proposed neurons use tanh activation in neuronal dynamics (Eq.7 and Eq.9), which involves exponential operations and can hardly be implemented in neuromorphic hardware. While previous methods often use different train/inference behaviors to avoid this problem. For example, the attention SNN uses sigmoid in training and Heaviside in inference. (Please refer to Temporal-wise Attention Spiking Neural Networks for Event Streams Classification). I suggest that the authors consider similar solutions.

We understand your concerns. Our next step is to deploy the model onto neuromorphic hardware, and therefore, we conducted extensive research in this area prior to designing our approach. Below, we will elaborate on several aspects that demonstrate the feasibility of deploying our method on neuromorphic hardware without any loss in performance.

1. Exponential operations over a limited domain can be implemented on neuromorphic hardware using a lookup table approach. In our method, both the membrane potential threshold and the membrane time constant are adaptively adjusted using Eqs. 7 and 9, respectively, which involve the exponential function tanh. In Eq. 7, the variable for tanh can be approximately interpreted as the input current to the neuron, with values distributed in the range of [-2, 2] and following a Gaussian distribution [1]. For Eq. 9, the variable of tanh represents the residual membrane potential $V_p$, where $V_p$ lies within [0, 1]. On the other hand, considering the definition of tanh, it implies that to efficiently compute tanh using a table-based approach, we only need to store discrete values of $e^x$ and $e^{-x}$ over the range of [-2, 2]. This allows for fast computation of tanh without involving complex exponential operations. This leads to a related question: what is the maximum level of precision we can achieve when discretizing the interval [-2, 2]? We will provide a detailed analysis based on existing neuromorphic hardware capabilities in the following sections.

2. Neuromorphic hardware available on the market can accommodate the memory overhead required for the lookup table approach. For example, Intel's neuromorphic chip Loihi 2, introduced in 2021, features 192KB of flexibly allocatable memory, as described on page 7 of the product brief available at [2] (Link). With 192KB of memory, it is possible to store up to 24,576 double-precision floating-point numbers. This allows for discretizing the range [-2, 2] for $e^x$ and $e^{-x}$ with a maximum precision of $3.33×10^{-4}$, enabling their respective storage. Next, we will further analyze how this level of precision impacts the performance of the algorithm.

3. Result: the loss in computational precision introduced by the lookup table approach has no impact on the performance of our method. To investigate how reducing the precision of the input variables for the tanh function in Eqs. 7 and 9 to three or four decimal places affects algorithm performance, we conducted experiments during the inference phase of the model. Specifically, we rounded the inputs of the tanh function to explore the performance impact across different precision levels (with varying numbers of decimal places retained). The tables below show the changes in algorithm performance. "Text" denotes the case where the tanh input precision is unmodified (identical to the precision reported in the main text), while 1. to 4. represent scenarios where the tanh precision is controlled at 1 to 4 decimal places, respectively. Any values different from "Text" are highlighted.

Question 3: How much ST-DANO requires extra training and inference cost, such as number of parameters, or FLOPs, compared to a baseline LIF network on the same tasks?

Thank you for your valuable suggestion. Our method does not introduce any additional parameters, ensuring that its theoretical FLOPs remain the same as the baseline LIF model. In the Appendix, we provided a comparison of the spike firing rates between our model and the baseline LIF. Here, we utilize theoretical calculations of accumulation (ACs) and multiply-and-accumulate (MACs) operations using the methods described in [1] and [2], whose detailed definitions can be found in the referenced literature. In these calculations, a 32-bit floating-point AC operation consumes 0.9$pJ$ of energy per operation, while a 32-bit floating-point MAC operation consumes 4.6$pJ$ of energy per operation.

The table below shows the theoretical ACs, MACs, and inference energy consumption of our method. While our model's MACs are slightly higher than those of the baseline LIF model due to the use of LLIF or SLIF neurons in some network layers and the additional computations introduced during the adaptive process, our model achieves lower ACs by suppressing spike firing. In terms of energy consumption, our model exhibits only a 0.15$\mu J$ increase over the conventional approach on the Tiny-ImageNet dataset, while showing lower inference energy consumption on other datasets. Furthermore, when our method is deployed on neuromorphic hardware, the extra MACs introduced by LLIF and SLIF neurons can be optimized using lookup tables, which replace costly arithmetic operations with precomputed tables, further reducing power consumption.

In summary, this analysis demonstrates that our method not only enhances network performance but also reduces energy consumption compared to conventional methods.

[1] Han, Song, et al. "Learning both weights and connections for efficient neural network." NeurIPS, 2015.

[2] Huang, Yulong, et al. "Clif: Complementary leaky integrate-and-fire neuron for spiking neural networks." ICML, 2024.

Weakness 2: The method has not been tested on tasks involving long continuous time-series data (like SMNIST), which would further validate the generalizability of the findings.

Thank you for this suggestion. The design of our method is based on the BPTT and direct encoding approach, since the SMNIST dataset does not utilize direct encoding, it was not chosen in our experiments. We tried to apply our method to the SMNIST dataset; however, because SMNIST requires 784 timesteps for its representation, the BPTT process becomes extremely time-consuming. We apologize for being unable to complete this within the limited rebuttal period. Nonetheless, your suggestion has provided a valuable direction for our future work, and we sincerely thank you for your thoughtful feedback.

Question 2: Could you analyse the main differences between your model and other LIF variants in terms of neuron design?

Thank you for your suggestion. The key features of our neuron design can be summarized as follows:

1. Selective Neuron Configuration for Optimal Performance: Our approach does not aim to replace all neurons in the network with a single type. Instead, it combines different neuron types with a neuron search strategy to achieve optimal overall network configuration. This strategy avoids the limitations that may arise from a singular neuron design and allows the network to be more flexible and efficient in handling different temporal dependencies and task requirements.

2. No Additional Hyperparameters Introduced: We have avoided introducing extra hyperparameters, thereby keeping the model complexity at a low level and reducing the difficulty of model training and tuning. This design ensures the simplicity of our model and minimizes potential hyperparameter search costs, making the model more user-friendly and efficient in practical applications.

3. Energy Efficiency with Minimal Additional Computation: While there is a slight increase in computational load due to our neuron design, this is offset by the suppression of spike firing, resulting in lower overall energy consumption compared to conventional LIF neurons. Detailed data supporting this claim is provided in the response to the next question.

4. Inspiring Broader Research Directions: Our combination of neuron design and search strategy offers a valuable perspective that we hope will inspire SNN researchers to not only focus on temporal dependencies but also explore how neuron design and search can achieve better balance in other network properties that vary along spatial dimensions. We believe that such approaches can promote the development of more flexible and adaptable network architectures.

5. Applicability Beyond Our Specific Neurons: We also believe that similar neuron search strategies could be applied to other LIF variants, rather than simply replacing neuron types across the network, potentially yielding superior performance. Our work is available as an open-source project (codes are provided in supplementary materials and will be made publicly available after publication), and we sincerely welcome other researchers to build upon and extend our approach, contributing to the advancement of the field.

We sincerely appreciate your thorough review of our work and the constructive suggestions you provided to enhance its quality. Please find our responses below.

Weakness1 & Question 1: The distinction between LLIF and SLIF is redundant. LLIF alters the long-term dependency capabilities of neurons by adjusting the threshold; consequently, it should also be able to modify the short-term dependency capabilities through threshold adjustment, transforming LLIF into SLIF. Adjusting the membrane potential constant and the threshold is not contradictory. Why not combine LLIF and SLIF into one? I suggest the authors provide experiments or analyses to demonstrate the necessity of distinguishing these two types of neurons.

Yes, strengthening long- and short-term dependencies can indeed be achieved through either adaptive thresholds or adaptive membrane time constants alone. However, in designing these two types of neurons, we also took into account computational burden and the final model’s energy consumption. Firstly, adaptively increasing the membrane potential threshold intuitively suppresses spike firing while achieving our intended goals. Similarly, when we lower the membrane time constant, it increases the leakage effect of the neuron, which also suppresses spike firing. We explicitly highlighted these two reasons in lines 189-192 of the main text, as follows:

From the perspective of firing rate, we enhance long-term dependencies by adaptively increasing the membrane potential threshold and strengthen short-term dependencies by reducing the membrane time constant. Both strategies work to suppress spike firing, which not only improves network performance but also reduces energy consumption.

We provide additional experiments focusing on cases with only adaptive membrane potential thresholds or only adaptive membrane time constants. The neuron designs for this experiment are as follows:

First, we redesigned two types of neurons, named VthDown and TauUp:

• VthDown: This corresponds to LLIF, but instead of adaptively increasing the membrane potential threshold, we adaptively decrease it, thereby strengthening the neuron's short-term dependencies. VthDown follows the adaptation rule specified by Eq. 7, with the "+" sign replaced by a "-".

• TauUp: This corresponds to SLIF, but instead of adaptively lowering the membrane time constant, we adaptively increase it, thereby enhancing the neuron's long-term dependencies. TauUp follows the adaptation rule specified by Eq. 9, with the "-" sign replaced by a "+".

Next, we designed three additional control experiments on CIFAR-10 with ResNet-18, using a time step setting of 6, as follows:

• A: Only adaptive membrane potential threshold neurons (LLIF and VthDown) participate in the search, with each enhancing long-term and short-term dependencies, respectively.

• B: Only adaptive membrane time constant neurons (TauUp and SLIF) participate in the search, with each enhancing long-term and short-term dependencies, respectively.

• C: Neurons TauUp and VthDown participate in the search, with each enhancing long-term and short-term dependencies, respectively.

The experimental results are as follows, where long-term dependency neurons are marked as 1, short-term dependency neurons as 2, and LIF neurons as 0 in the optimal neuron configurations:

From these experiments, we observe that using only adaptive threshold or only adaptive time constant methods can optimize network performance to a certain extent. However, under these methods, the spike firing rate is higher compared to the baseline LIF method. Therefore, when balancing both performance and energy consumption, we believe that our combined LLIF+SLIF approach is optimal.

Question 4: Would this approach work effectively on more complex architectures, such as RNNs or Transformers, and on large-scale datasets like ImageNet?

Thank you for raising these questions. Our method is based on the design of LIF neurons in SNNs, including LLIF and SLIF neurons, and aims to optimize specific temporal dependency issues in SNNs. Since our work focuses on the unique characteristics of SNNs and given that RNN and Transformer architectures differ significantly from SNNs (While SNNs rely on event-driven, sparse computations with discrete spikes, RNNs and Transformers utilize continuous signal processing and dense, sequential or attention-based operations), the effectiveness of our approach on RNNs or Transformers remains to be explored. Extending our method to more complex network architectures would require extensive research, redesign, and adaptation, all of which are challenging and time-consuming processes.

Regarding large-scale datasets such as ImageNet, training on them poses significant time and resource challenges, making it difficult to accomplish within the limited rebuttal period. Nonetheless, we believe that with proper expansion and optimization, our approach has the potential to perform well on larger datasets, and this represents a key direction for our future work.

Question 3: Could the authors conduct ablation studies to investigate neuron combinations, such as only Long-LIF or only Short-LIF?

Thank you for your thorough review. In fact, we have already conducted ablation experiments in the original manuscript, but these experiments were placed before the main experiment section, serving as the motivation for our proposed neuron search framework. Specifically, they are located at lines 260-346 (in the revised manuscript) under the section titled "4.2 Ablation study: Why a Neuron Search Strategy is Essential?".

In our ablation experiment design, we selected representative datasets and network architectures from both static and neuromorphic data categories—Cifar10 with ResNet-18 and DVS-CIFAR-10 with VGG11. By selectively replacing LIF neurons at different layers with LLIF and SLIF, we tested their performance, including scenarios where all neurons were replaced with either LLIF or SLIF. Detailed experimental results and specifics can be found in Table 1, at lines 287-304.

Additionally, to more comprehensively address the reviewer's suggestions, we have conducted further ablation experiments on additional datasets that were not covered in the main text (CIFAR-100, DVS-CIFAR-10, and DVS-Gesture). In these additional ablation experiments, we used the same hyperparameters as specified in Table 5 of Appendix F. Here, LLIF, SLIF, and LIF indicate configurations where all neurons within the architecture are replaced with LLIF, SLIF, and LIF neurons respectively, while "Text" represents the performance under the optimal neuron configuration presented in our main text.

It can be observed that, generally, the LLIF method enhances performance, while SLIF tends to have a negative impact on performance. The reason why LLIF and "Text" yield the same result for DVS-Gesture is that the optimal configuration reported in the main text (10111111, where 1 represents LLIF and 0 represents LIF) is essentially identical to using only LLIF neurons, hence their performance is nearly the same. Additionally, due to the limited time available for the rebuttal period, we were unable to conduct a comprehensive ablation study with varying neuron configurations at different layers, as done in Section 4.2. We sincerely apologize for this limitation.

Although the above experiments may suggest that SLIF appears to offer limited benefit, this does not mean that SLIF neurons are entirely without value. In fact, these two types of neurons are complementary. To illustrate the necessity of SLIF, we modified the optimal neuron configuration reported in the main text for CIFAR-10 at timestep 6 (Text: 20000000000001011) into two alternative configurations: A (00000000000001011), where the shallowest layer SLIF neuron is replaced with LIF, and B (10000000000001011), where it is replaced with LLIF. We then compared their performance differences as follows:

On CIFAR-100, we modified the architecture at time step 6 (Text: 20002000000010111) into two alternative configurations: A (00000000000010111), where the relevant SLIF neurons were replaced with LIF, and B (10001000000010111), where they were replaced with LLIF. The experimental results are as follows:

It is evident that when SLIF is placed in the shallow layers, it makes a significant contribution to the overall network architecture's performance. Conversely, on CIFAR-10, placing LLIF in the shallow layers (Configuration B) resulted in worse performance than using standard LIF neurons (Configuration A). These experiments demonstrate that SLIF neurons, like LLIF neurons, are essential for balancing temporal dependencies and enhancing network performance. Additionally, they illustrate the rationality and necessity of each module within our method.

Question 1: Could the authors explore a generalized strategy for temporal adaptation that does not rely on predefined neuron types?

We sincerely thank the reviewer for the suggestion regarding exploring a general temporal adaptation strategy independent of predefined neuron types. We deeply appreciate this recommendation and recognize the significant research value and challenges associated with such a strategy.

Firstly, we chose to design specific predefined neuron types such as LLIF and SLIF to purposefully enhance the temporal dependency performance of SNNs. Subsequently, we developed a NAS-based framework to enable diverse dependency configurations within the network. This approach offers a direct method for optimization across different temporal scales, and it remains highly intuitive. In the worst-case scenario, our method can still search for architectures composed entirely of LIF neurons, ensuring that our approach is at least as good as the baseline. Although this method may, to some extent, rely on predefined neuron types, its performance across multiple datasets demonstrates that its generalizability is not limited by these predefined neurons. It has exhibited significant advantages in handling various types of tasks, fully showcasing its adaptability and practical value.

Regarding the reviewer's suggestion to explore a general temporal adaptation strategy, we believe this is a highly valuable research direction. In future work, we plan to investigate how to construct more flexible, data-driven temporal adaptation strategies to achieve time adjustment methods that do not rely on specific neuron types.

Question 2: How would the proposed ST-DANO method perform with other encoding paradigms?

Thank you for the suggestion. Our method is specifically designed to address the issue of temporal dependency variations across the spatial dimension brought about by the mainstream training combination of BPTT and direct encoding. This particular context has determined the focus of our current work and guided our methodological design. We have elaborated on the motivation for this work in detail in the Introduction section as follows:

However, this combination (BPTT + direct encoding) introduces varying requirements for temporal dependencies across the network spatial dimension—an aspect overlooked by existing research…. (Starting from line 54)

Regarding other encoding methods, such as rate coding and temporal coding, the temporal dependency variation issue we described does not exist. Therefore, we must acknowledge that our method may not be perfectly compatible with other encoding methods. Nonetheless, we believe that applying the design concepts of our ST-DANO approach to other encoding methods is a meaningful research direction, as our method intuitively enhances network performance.

We sincerely appreciate your thorough review of our work. We will now address each of the weaknesses and questions you raised individually.

Weakness 1: There exists extensive research on neuron design that dynamically adjusts neuron parameters, such as thresholds and membrane potential constants, to optimize SNN performance. However, the authors did not review or compare their approach against this body of work, nor do they demonstrate clear advantages over these established methods.

Thank you for this valuable comment. Here, we would like to clarify the similarities and differences between our work and existing approaches:

1. Due to space constraints, we selected a few representative cases from the literature you listed [1-3], which contained relatively comprehensive experiments, and incorporated them into our performance comparison, specifically in Table 2 (lines 383 to 431 in the revised manuscript), where added methods are highlighted in blue. The conclusion drawn is that our method consistently demonstrates advantages across various datasets.

[1] Hasssan, Ahmed, Jian Meng, and Jae-Sun Seo. "Spiking Neural Network with Learnable Threshold for Event-based Classification and Object Detection." IJCNN, 2024

[2] Wei, Wenjie, et al. "Temporal-coded spiking neural networks with dynamic firing threshold: Learning with event-driven backpropagation." ICCV, 2023.

[3] Shen, Jiangrong, et al. "Esl-snns: An evolutionary structure learning strategy for spiking neural networks."AAAI, 2023.

2. It is true that considerable research has been conducted on optimizing SNN performance by dynamically adjusting neuron hyperparameters such as thresholds and membrane time constants. However, the focus of the "Related Work" section in our review was not on these conventional dynamic adjustment studies, but rather on examining the temporal dependency issue in SNNs. This focus aligns with the core goal of our work, which is to enhance long-term and short-term temporal dependencies through LLIF and SLIF, rather than simply adjusting conventional parameters. We believe that the review of temporal dependency issues is relatively less explored in existing literature, thereby positioning our work as both unique and of significant value in this domain.

Weakness 2: The study also lacks comparisons with state-of-the-art methods in Tab.2

Thank you for your feedback. However, please allow us to clarify that the works listed in Table 2 including and not limited to Online LTL (2022) [1], Joint A-SNN (2023) [2], TET-SSF (2023) [3], OTTT (2022) [4], SLTT (2023) [5], and sparse-KD (2024) [6], etc., represent the latest SNN research, all published between 2022 and 2024.

[1] Yang, Qu, et al. "Training spiking neural networks with local tandem learning." NeurIPS, 2022.

[2] Guo, Yufei, et al. "Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization." Pattern Recognition, 2023

[3] Wang, Jingtao, et al. "SSF: Accelerating Training of Spiking Neural Networks with Stabilized Spiking Flow." CVPR, 2023.

[4] Xiao, Mingqing, et al. "Online training through time for spiking neural networks." NeurIPS, 2022.

[5] Meng, Qingyan, et al. "Towards memory-and time-efficient backpropagation for training spiking neural networks." ICCV, 2023.

[6] Xu, Qi, et al. "Reversing Structural Pattern Learning with Biologically Inspired Knowledge Distillation for Spiking Neural Networks." ACM Multimedia, 2024.

Question 3: For LLIF, the authors say ' To achieve stronger long-term dependencies, the contribution from early time steps must have a higher expected proportion in the total gradient compared to standard LIF neurons'. Could the authors give further explanations on this?

Thank you for your valuable suggestion; it has greatly helped us improve the completeness of our article. To explain more clearly the logical relationship between "increased gradient contributions from earlier timesteps" and "long-term dependency," we need to revisit the definition of long-term dependency: it refers to the ability of a neural network to effectively capture and utilize the influence of distant past inputs on the current state when processing sequential data. However, deep networks, especially SNNs with recurrent structures, often face the challenge of gradient decaying when handling long sequences. As shown in the BPTT gradient formula (Eq. 4 in the main text), it contains a multiplicative term that is less than 1 across the time dimension. This means that gradients originating from distant timesteps gradually decay during backpropagation, making it difficult for the network to retain and leverage these distant inputs.

Therefore, in our description, we aimed to convey that increasing the gradient contributions from earlier timesteps imposes a form of "memory bias" on the network, ensuring that it does not easily forget earlier signals when processing long sequences. This is precisely the essence of long-term dependency. However, we recognize that this explanation was not sufficiently clear in the original text. Thank you again for pointing out this issue. In the revised manuscript, we have supplemented this explanation as follows, located at lines 227-232 and highlighted for clarity.

We analyze why LLIF neurons enhance long-term dependencies from the perspective of gradient contribution in Appendix C. To achieve stronger long-term dependencies, the contribution from early time steps must have a higher expected proportion in the total gradient compared to standard LIF neurons. This can alleviate the inherent gradient decay phenomenon in BPTT caused by Eq. (4), thereby helping the network more effectively retain and leverage information from distant time steps during the backpropagation process.

Question 4: In table 3, Search Cost is measured in GPU days. Could the authors provide the ratio of search cost to the training time cost?

Thank you for your suggestion. The reason we listed the search time in terms of GPU days in Table 3, rather than providing a ratio of time costs, is that absolute search time more intuitively reflects the time cost associated with our search method. This provides prospective researchers with a clearer understanding of the resource requirements under different conditions, allowing them to evaluate whether our approach is suitable for their specific application scenarios or experimental environments.

In addition, we provide the ratio between the search time and the single training time of the final model for context.

The data above indicates that NAS methods indeed require more computational resources and time costs compared to conventional training. This is an inherent aspect of NAS approaches, as they involve extensive evaluations of candidate networks and hyperparameter searches.

We sincerely thank the reviewer for their constructive comments. Below, we provide our responses to all the identified weaknesses and issues.

Weakness 1: The performance on CIFAR-100 is relatively low.

We appreciate the reviewer's attention to our performance on the CIFAR-100 dataset. We acknowledge that our relative performance on this dataset is slightly lower, achieving 76.25%, while the best performance on CIFAR-100 was achieved by the Joint A-SNN method (77.39%) [1], which utilizes an ANN-to-SNN training approach that is very different from our direct training method based on surrogate gradients. Notably, Joint A-SNN's performance on CIFAR-10 (95.45%) and Tiny-ImageNet (55.39%) is significantly lower than ours, indicating that our direct training strategy demonstrates superior adaptability and generalization on these tasks. Furthermore, within the same training paradigm (direct training with surrogate gradients (SG)), our method exhibits a clear advantage, with the best performance among other SG methods being RecDis-SSF [2], which only achieved 75.48% accuracy. In the future, we will continue to focus on further optimizing our method to achieve satisfactory performance across all datasets.

[1] Guo, Yufei, et al. "Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization." Pattern Recognition, 2023.

[2] Wang, Jingtao, et al. "SSF: Accelerating Training of Spiking Neural Networks with Stabilized Spiking Flow." CVPR, 2023.

Weakness 2: Table 2 is too wide.

Following this comment, we have reduced the width of Table 2 as much as possible by using abbreviations such as A2S for ANN-to-SNN and SG for surrogate gradient, among others. However, due to the extensive amount of information contained in the table, maintaining its current width might be the most reasonable way to ensure both data completeness and readability. We are open to further suggestions and would be glad to make additional optimizations if recommended.

Question 1: In Figure 2 a1 and a2, SLIF-4 fails to converge with ResNet-18 on CIFAR-10. Could the authors give explanations on this?

Thank you very much for your valuable suggestion. We acknowledge that our previous manuscript did not provide sufficient explanation for this phenomenon, which was an oversight on our part. Therefore, in the revised version, we have added a detailed analysis of this phenomenon at lines 325-329, which is highlighted. The specific content is as follows:

This is because the closer the network is to the output layer, the stronger its temporal dynamics become, and the more abstract its temporal features. As such, near-output neurons need to integrate contextual information through long-term dependencies. If these neurons are forced to emphasize short-term dependencies, as seen with SLIF-4, it can severely damage network performance.

Question 2: The authors say that SLIF captures short-term dependencies. However, in Eq.(9), \tau[t] is decreased when a spike fires. In my opinion, instead of capturing short-term correlations, SLIF overlooks the correlations between different time steps by its mechanism.

Thank you for your valuable feedback. We understand your concern regarding the SLIF model's ability to capture short-term dependencies. Indeed, during forward propagation, SLIF accelerates neuronal leakage by reducing the membrane time constant, which might appear to diminish correlations across different timesteps. However, we wish to emphasize that the true purpose of this mechanism is to enable neurons to respond more quickly to new input signals, thereby better capturing dynamic changes in inputs over the short term.

More importantly, in the backpropagation phase, our model does not ignore these temporal correlations. In fact, during the BPTT process, all historical dependencies are computed and updated. This means that while the membrane time constant adaptively decreases during forward propagation, the flow of gradients during backpropagation captures the dependencies between different timesteps. Therefore, rather than completely disregarding temporal correlations, the SLIF model integrates the contributions of different timesteps comprehensively during the backpropagation phase.