TS-LIF: A Temporal Segment Spiking Neuron Network for Time Series Forecasting

Thank you for taking the time to review our paper and providing valuable feedback. We would like to answer your questions below.

W1: Questionable biological realism

WA1: Thank you for acknowledging the value of modeling dendritic and somatic voltages separately. We address your concern as follows:

1. The weighted summation in our model can be interpreted as representing the influence weights of two distinct types of spikes on the next layer of neurons, akin to synaptic connection strength. This weight, $\kappa$, can be seamlessly integrated into the weights of the next layer (e.g., MLP) without altering the binary spiking nature (0 or 1) of the outputs.

2. From another perspective, the dendrites and soma in our TS-LIF model can be viewed as interacting units that process information at different frequencies while influencing one another. This is analogous to sensory systems, such as auditory processing, where inner hair cells are tuned to different characteristic frequencies, or the visual system, where cone cells are tuned to different wavelengths. We believe that this perspective adds biological credibility to our approach.

A more detailed biological explanation is provided in the revised manuscript under the Temporal Segment LIF Neuron section. The modifications have been highlighted to clarify this connection and further emphasize the conceptual similarities.

W2: Is the model more efficient than other neural networks?

WA2: We apologize for any misunderstanding caused. Our original intention was to emphasize that TCN and transformer require more memory compared to GRU, rather than comparing them to SNN. The efficiency of SNNs primarily lies in their ability to significantly reduce energy consumption. Compared to traditional ANNs, SNNs reduce energy consumption by approximately 60%, as shown in the table below:

Table. The theoretical energy consumption and performance comparison during the inference stage of the Electricity dataset

W3: Missing controls.

WA3: We believe there may have been a misunderstanding regarding the parameters in our TS-LIF model. All parameters of the TS-LIF model are learnable. To address this potential confusion, we have clarified this point in the revised manuscript.

Additionally, to further validate the TS-LIF model's ability to capture long-term dependencies, we conducted additional experiments using the Delayed Spiking XOR Problem. This task involves three stages: an initial spike input, a delay period with noisy spikes, and a second spike input. The model computes an XOR operation between the first and final inputs based on firing rates.

Our experimental parameters and setup align with those described in [1]. Specifically, we used a two-layer MLP with only 20 hidden neurons, where the input feature size was set to 20. The results, which measure prediction accuracy under different delay timesteps and activation functions, are summarized as follows:

Table. The prediction accuracy of the delayed spiking XOR problem under different delay timesteps and activation functions.

• "Delay timesteps" refers to the delay period with noisy spikes.
• "ReLU" denotes the activation function
• "LIF" denotes the vanilla LIF neuron.
• "TS-LIF" denotes our proposed TS-LIF neuron.

The results clearly demonstrate the superior performance of the TS-LIF model in capturing long-term dependencies compared to both ReLU and vanilla LIF activation functions.

Additionally, to verify the effectiveness of our TS-LIF, we extended the comparison to include TC-LIF [2], LM-H [3], and CLIF [4] in the GRU Backbone with three prediction lengths (6, 24, 96) of two datasets (Metr-la and Electricity) , where TS-LIF consistently outperformed these models. The detailed results are summarized below:

Table. Comparison of TS-LIF with Other LIF Neurons

Thank you for taking the time to review our paper and providing valuable feedback. We would like to answer your questions below.

W1: The ability of spiking neuron models to capture long-term dependencies

WA1: In the submission, we compared three different types of sequence models. Among them, GRU lacks the ability to model long-term dependencies, whereas TS-LIF consistently demonstrated superior performance. This indicates that TS-LIF effectively enhances the model's capability for long-term modeling.

In the rebuttal, to further evaluate the ability of spiking neuron models to capture long-term dependencies, we conducted experiments using the Delayed Spiking XOR Problem. This task involves three stages: an initial spike input, a delay period with noisy spikes, and a second spike input. The model computes an XOR operation between the first and final inputs based on firing rates. Our experimental parameters and setup align with those described in [1]. Specifically, we use a two-layer MLP with only 20 hidden neurons, where the input feature size was set to 20. The results, which measure prediction accuracy under different delay timesteps and activation functions, are summarized as follows:

Table. The prediction accuracy of the delayed spiking XOR problem under different delay timesteps and activation functions.

• "Delay timesteps" refers to the delay period with noisy spikes.
• "ReLU" denotes the activation function
• "LIF" denotes the vanilla LIF neuron.
• "TS-LIF" denotes our proposed TS-LIF neuron.

The results clearly demonstrate the superior performance of the TS-LIF model in capturing long-term dependencies compared to both ReLU and LIF activation functions.

W2: Comparisons with existing spiking neuron models such as TC-LIF, LM-H, CLIF, and BHRF.

WA2: In the Ablation Study section of our submission, we have compared the performance of TS-LIF and TC-LIF, demonstrating that TS-LIF delivers superior results. Additionally, we extended the comparison to include LM-H and CLIF in the GRU Backbone (noting that BHRF does not provide publicly available code) on two different types of datasets (Metr-la and Electricity), where TS-LIF consistently outperformed these models. The detailed results are summarized below:

W3: It is biologically questionable whether the input current can be applied to both the dendrite and soma while producing weighted spikes.

WA3: We address this concern as follows:

1. In the Hierarchical Temporal Memory (HTM) neuron model, numerous synapses distributed along the dendrites function as pattern detectors that trigger NMDA spikes and depolarization at the soma [2]. This aligns closely with the TS-LIF model's approach, which leverages dendritic and somatic integration to process inputs across diverse temporal patterns. Furthermore, the TS-LIF model’s ability to generate both dendritic and somatic spikes is inspired by the biological capability of neurons to simultaneously process and predict signals, similar to the prediction mechanisms observed in HTM neurons.
2. The weighted summation of dendritic and somatic spikes in our model serves to reflect their differential influence on the subsequent layer of neurons, akin to the varying synaptic connection strengths observed in biological systems. This weighted influence can be integrated into the synaptic weights in the next layer (such as MLP).
3. Furthermore, from another perspective, dendrites and soma in our TS-LIF model can be viewed as interacting units, each processing different aspects of input information while influencing each other. This approach finds parallels in biological sensory systems, such as the auditory system, where inner hair cells are tuned to different characteristic frequencies, and the visual system, where cone cells are tuned to different wavelengths. These biological examples support the biological credibility of our TS-LIF model's structure and the interaction between dendritic and somatic compartments.

A more detailed biological explanation is provided in the revised manuscript under the Temporal Segment LIF Neuron section. The modifications have been highlighted to clarify this connection and further emphasize the conceptual similarities.

Thank you for taking the time to review our paper and providing valuable feedback. We would like to answer your questions below.

W1: The motivation for using a spiking model for time series tasks

WA1: We aim to propose a novel spike neuron (TS-LIF) that aligns better with the characteristics of time series forecasting tasks. We would like to provide a more detailed explanation below:

1. Spiking Neural Networks (SNNs) are inherently designed to process temporally correlated data, making them naturally aligned with time series tasks due to their event-driven and time-sensitive properties.
2. SNNs excel at handling sparse data, as evidenced by their robustness in dealing with missing values in time series data, which is a common challenge in real-world scenarios (e.g., see Table 2 and Table 9).
3. Existing time series processing models, such as Transformer, often require significant computational resources. In contrast, the sparse firing characteristic of SNNs allows for a substantial reduction in the model's energy consumption. To support this, we have conducted additional experiments demonstrating the energy efficiency of our approach (e.g., refer to Table 6 of Appendix).

As a result, TS-LIF is a novel spike neuron developed to explore the application of SNN frameworks in time series forecasting.

W2: Compared with the latest state-of-the-art time series methods (SparseTSF, SAMformer)

WA2: We conducted comparison experiments between TS-LIF and [1], [2] using three prediction lengths on two datasets (Metr-la and Electricity). SparseTSF* replaces the original ReLU activation in SparseTSF with TS-LIF, demonstrating TS-LIF's transferability and compatibility with ANN architectures, while SAMformer cannot be modified. Results show TS-former outperforms others on Metr-la, whereas SparseTSF* and SparseTSF perform slightly better than TS-former on Electricity. These findings confirm the compatibility of TS-LIF with ANN frameworks and its potential to enhance SNN frameworks in time series forecasting.

Note: Results are evaluated across 3 prediction lengths (24, 48, 96) on the Metr-la and Electricity datasets.

• SparseTSF*: Replaces the ReLU function of SparseTSF with TS-LIF.
• TS-former represent the iTransformer with our TS-LIF.

W3: Energy consumption, parameter count, training time, and FLOPs

WA3: Based on [3], we measured energy consumption on 45nm hardware and compared the parameter sizes of our TS-LIF and LIF under three different backbone structures. The detailed results are presented below. Since TS-LIF introduces only a few additional parameters, just four learnable parameters: $\alpha, \beta, \gamma$, and $\kappa$ (where $\kappa$ is a vector related to the network's dimensions, while the others are scalars) of SNN structure, TS-LIF does not significantly increase the model size or training/inference speed compared with SNN structure.

Note (Theoretic energy consumption of Electricity during the inference stage, and the results are sourced from [4], except the Train/Infer Time volume and the rows of TS-TCN, TS-GRU and TS-former):

• "OPs" refers to SOPs in SNN and FLOPs in ANN.
• "SOPs" denotes the synaptic operations of SNNs. (The definition can refer to Appendix A.5)
• "FLOPs" denotes the floating point operations of ANNs.
• "Train/Infer Time" denotes the training and inference time of 15784 training data and 5260 test data.

We thank you for recognizing the novelty and contributions of our paper and also for the positive feedback. We would like to answer your questions below.

W1: Could the authors elaborate on the biological plausibility, possibly referencing Hierarchical Temporal Memory?

WA1: We agree that the HTM neuron model can further support the biological plausibility of our TS-LIF neuron model [1].

1. Dendritic spike: In the HTM model, numerous synapses are distributed along dendrites, with small subsets of neighboring synapses acting as pattern detectors. Collectively, these thousands of synapses function as independent pattern detectors, where the detection of any pattern triggers an NMDA spike, leading to depolarization at the soma.
2. Shortcut connection: These patterns may represent sequences of bioelectrical signals across different frequencies, including identity mappings.
3. Simultaneous spike: HTM neurons are capable of making multiple simultaneous predictions, aligning with the TS-LIF model’s capability of simultaneously generating dendritic and somatic spikes.
4. Other perspective: The dendrites and soma in our TS-LIF model can be aslo viewed as two interacting units (or "neurons"). These units process information at different frequencies while influencing one another, resembling mechanisms observed in sensory systems, such as auditory processing (inner hair cells tuned to different characteristic frequencies) and visual systems (S, M, and L-type cone cells tuned to different wavelengths).

A more detailed biological explanation is provided in the revised manuscript under the Temporal Segment LIF Neuron section. The modifications have been highlighted to clarify this connection and further emphasize the conceptual similarities.

W2: Average power spectrum of the network dynamics

WA2: The temporal frequency information of the original signal is notably influenced as it passes through the encoder in the network. To illustrate this, we have plotted the power spectrum of dendritic and somatic voltages in TS-LIF after the first encoder layer of TS-former. These spectrum highlight how dendritic and somatic compartments process distinct frequency components, showcasing their unique responses. The detailed frequency power spectrum are provided in the Appendix (Fig. 5).

Q1: Energy consumption?

A1: We have carefully evaluated this aspect and present our findings in the table below. Our model demonstrates a significant reduction in energy consumption compared to traditional ANN counterparts.

Table.Theoretical Energy Consumption and Performance Comparison During the Inference Stage of the Electricity Dataset

• "OPs" refers to SOPs in SNN and FLOPs in ANN.
• "SOPs" denotes the synaptic operations of SNNs. (The definition can refer to Appendix A.5)
• "FLOPs" denotes the floating point operations of ANNs.

Q2: Are there specific types of time series data or forecasting tasks where TS-LIF might be particularly well-suited or, conversely, might struggle?

A2: Based on the results in our main experiment table, where TS-former outperforms ANN models on the Solar and Electricity datasets, we hypothesize that TS-LIF excels in time series where high-frequency and low-frequency information can be clearly decomposed. In contrast, for more challenging datasets like Metr-la, TS-LIF may struggle to perform as effectively. This is a highly insightful point, and we will conduct experiments to validate our hypothesis if time permits.

Q3: The choice of the weighting parameter $\kappa$

A3: The weighting parameter $\kappa$ controls the trade-off between dendritic spikes and somatic spikes. Instead of predefining this parameter, we treat it as a learnable parameter, allowing the model to determine its optimal value through training on the dataset.

Q4: Include new references that are relevant for the topic.

A4: Thank you for your valuable suggestion to include additional relevant references. In the revised manuscript, we have incorporated new references to enrich the discussion and provide a more comprehensive context for our work.

[1] Hawkins J, Ahmad S. Why neurons have thousands of synapses, a theory of sequence memory in neocortex[J]. Frontiers in neural circuits, 2016, 10: 174222.

[2] Lv C, Wang Y, Han D, et al. Efficient and Effective Time-Series Forecasting with Spiking Neural Networks[C]//Forty-first International Conference on Machine Learning.

We sincerely hope our clarifications above have addressed your concerns and can improve your opinion of our work.

Dear reviewer fwe4,

Thank you for your time and effort in providing such valuable feedback on our work. As the discussion period is nearing its end, we hope you’ve had the chance to go through our rebuttal. We believe it has helped clarify our work further, and if our responses have addressed your concerns, we would greatly appreciate it if you could kindly update your feedback to reflect this. We’re more than happy to continue the discussion if there are any remaining questions or points to address.

Sincerely

Authors