Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks

Thank you for your insightful feedback. We have carefully studied your comments and argue that your concerns can be addressed.

Weakness 1: One major concern is about whether learnable IMP can be implemented in neuromorphic chips since the IMP is a float number. To my experience, I do not believe it can be effectively implemented in current hardwares such as Loihi (and the whole paper does not discuss hardware implementation issues). If I am wrong, the authors please corrected me by comprehensively addressing this issue. If hardware implementation is hard or energy-ineffecient (so the main advantage of SNN disappears), then the proposed models should be compared with ANN models with similar model size (and apparently it is behind ANNs).

Answer: IMP can be implemented on neuromorphic chips. Loihi supports setting the membrane potential (a floating-point number) to a non-zero floating-point value, and the LAVA (Loihi's toolchain) also provides APIs for setting IMP.

import numpy as np
from lava.proc.lif.process import LIF

# Create LIF Neuron with constant Initial Membrane Potential (IMP)
imp = 0.
lif = LIF(
    shape=(16,),   # Number and topological layout of units in the process
    v=imp,         # Initial value of the neurons' voltage (membrane potential).
    vth=1.0,       # Membrane threshold
    dv=0.5,        # Inverse membrane time-constant
    name="lif"
)

# Create LIF Neuron with custom Initial Membrane Potential (IMP)
imp = np.random.uniform(high=0.2, low=-0.2, size=(16,))
lif_imp_version1 = LIF(
    shape=(16,),   # Number and topological layout of units in the process
    v=imp,         # Initial value of the neurons' voltage (membrane potential).
    vth=1.0,       # Membrane threshold
    dv=0.5,        # Inverse membrane time-constant
    name="lif"
)

It is worth noting that in previous research, IMP has always existed, but due to the lack of relevant research on more appropriate settings, researchers usually set it to a floating-point value of 0. Apart from that, according to our results on the ImageNet test set (Table R1), IMP does not incur significant additional theoretical power consumption overhead, but can effectively improve the performance of SNNs.

Weakness 2: The title is info-less and not proper. The are a few problems. First, it should be the dynamics of neurons, not neuron networks, since the IMP is the property of individual neurons. Second, dynamics of neurons include various aspects, such as the model choice (LIF or Hudgekin-Hoxley model), noise term, etc. The readers obtain no information from "rethinking the dynamics", and other aspsects are not disucussed in the paper. A better title might be something like "learnable initial membrane potental enhances spiking neural networks".

Answer: We appreciate your suggestions. As you mentioned, in this paper, we have investigated the dynamics of spiking neurons and its impact on the supervised learning and output representations of SNNs. We have taken your suggestions into consideration and propose to change the title to "Enhancing Spiking Neural Networks via Initial Membrane Potential and Optimized Objectives".

Weakness 3: By improving the dynamics of spiking neurons, it is reasonable to expect the model to be enhance in time series or sequential processing (e.g. NLP) tasks rather than image classifications tasks (since "dynamics" is temporal property). The lack of experiments on corresponding datasets making it unclear whether the proposed methods are effective only on CV tasks or in general.

Answer: Thank you for the suggestion to further validate the effectiveness of our method. We have conducted additional experiments on time series tasks. By incorporating IMP in SNN-delay[1], we have achieved an effective improvement from 95.10% to 96.02% (Table R2). Nevertheless, it is important to note that the neuromorphic datasets we used in our experiments (page 8 Table 2), including CIFAR10DVS[1] and NCaltech101[2], contain temporal dynamics, rather than static images.

[1] Li H, Liu H, Ji X, et al. Cifar10-dvs: an event-stream dataset for object classification[J]. Frontiers in neuroscience, 2017, 11: 309.

[2] Orchard G, Jayawant A, Cohen G K, et al. Converting static image datasets to spiking neuromorphic datasets using saccades[J]. Frontiers in neuroscience, 2015, 9: 437.

Weakness 4: When compared with baseline methods, the baseline methods should be shortly explained. Otherwise, it is unclear whether the comparisons are fair, and lack insights other than "our methods are good".

Answer: Due to the length constraint of the main text, we have provided the ablation experiments in the appendix (page 15 Table 4). We kept the experimental setup and hardware equipment completely consistent in the ablation experiments, except the parameters that needed to be compared. We hope that the results of the ablation experiments can address your concerns, and we will pay more attention to the explanation of our experimental results, providing more insights.

Question 1: Can learnable IMP can be implemented in neuromorphic hardwares such as Intel Loihi?

Answer: Yes. Please refer to the response to Weakness 1.

Question 2: Can the authors estimate the theoretical energy consumption like in other SNN papers?

Answer: Please refer to the response to Weakness 1 and the PDF we submitted.

Question 3: How is the performance on other datasets such as time series or NLP?

Answer: Please refer to the response to Weakness 3 and Table R2.

Limitation 1: The main limitation is about hardware implement, which the authors did not discuss.

Answer: Please refer to the response to Weakness 1.

Thank you for acknowledging our research direction and the innovative approach we have employed.

Weakness 1: In this paper, all experiments and illustrations are 4 time steps SNN. It remains non-clear that in the long time steps task, how much could the initial membrane potential influence the output.

Answer: To enhance the clarity and readability of the chapter content, we set T=4 to simplify the analysis process. It is worth noting that in the experimental section, we cover the cases of T values of 8, 10, and 16 (on page 8 Table 2). The experimental results demonstrate that IMP can maintain its effectiveness even at longer time steps (Table R3 and Table R11). IMP also exhibits significant improvements in time series tasks, which can be seen in Table R2.

Table R10: Accuracy on DVS128Gesture dataset.

Question 1: Since we only use the output of the last time step to give an inference, I wonder if there is any information missing in this process. Have you ever considered using the last few time steps output instead of just one?

Answer: In static tasks, using only the last time step can indeed lead to information loss, but applying the same supervision label to multiple or all time steps may also make the model difficult to fit. There may be a tradeoff between information loss and fitting difficulty. LTS only uses the last time step for supervision and output representation, which can directly verify the hypothesis. Additionally, we believe that using the mean value of a few time steps' outputs instead of just one could be a good alternative to LTS, especially when time steps is relatively large. As mentioned in the A.2 Limitations section, the advantages of LTS tend to weaken as T increases, particularly when $T > 8$, as shown in Figure 8.

Question 2: From another point of view, adjusting the initial membrane potential is just injecting a controlled noise into the membrane potential at first. Have the authors ever considered giving a controlled noise to each time step to improve the performance?

Answer: We have not yet tried related methods, this is a direction that remains unexplored. Adding controlled noise at each time step can be seen as a new method of membrane potential reduction/increase, which may have the potential to improve the capacity of the model. Similar methods like GLIF have achieved beneficial improvements by learning key parameters in the membrane dynamics. In summary, we appreciate your suggestions for future research directions.

Thank you for your insightful comments. We first list your advice and questions, then give our detailed answers.

Weakness 1: If the residual technique is altered, such as MS-ResNet, the residual connection is calculated using the membrane potential. In this instance, will a better initialization of the membrane potential result in considerable gains?

Answer: Yes. The effectiveness of our proposed IMP and LTS methods in membrane potential residuals on the MS-ResNet architecture can be evaluated on the ImageNet100 dataset.

Table R6: Accuracy on ImageNet100 dataset.

Weakness 2: If the network design is altered, such as the Spikformer with SEW residual connections and the Spike-driven transformer with MS residual connections. Are the proposed IMP and LTS still valid?

Answer: Yes, we have tried the SpikingResFormer[1] with self-attention mechanism and membrane potential residual connections. Due to the limitations of time and computational resources, we used fewer epochs compared to the original paper. Although LTS is slightly lower than SDT, it still outperforms TET, which is consistent with our analysis. Additionally, we have verified the effectiveness of the combination of self-attention mechanism and IMP on the CIFAR10DVS dataset (Table R3 and Table R8). We hope these results can address your concerns.

Table R7: Accuracy and theoretical energy consumption on ImageNet1k dataset.

Table R8: Accuracy on CIFAR10DVS dataset.

[1] Shi X, Hao Z, Yu Z. SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5610-5619.

Question 1: In the DVS datasets, each timestep has its own unique information. Will LTS lose the information of the previous time step and cause unstable recognition accuracy?

Answer: Yes, the LTS method can lead to information loss, especially in DVS tasks with a large number of time steps. We only suggest considering the use of LTS in static tasks, as the effectiveness of LTS relies on the assumption of having the same input at each time step.

Table R9: Accuracy on CIFAR10DVS dataset (resize to 48).

Question 2: Why does PSN lose dynamics and biological rationality? Aren't soft-reset SNNs dynamic? Biological rationality seems to have nothing to do with the reset method, and biological rationality is not important in deep learning.

Answer: PSN is not a soft-reset dynamics, but rather a no-reset mechanism. PSN removes the reset mechanism and achieves parallelization through linear mapping or convolution (scanning) across the time dimension. But we agree with your point that biological rationality is not important in deep learning. We will modify the statement to: "It should be noted that the removal of the reset process in PSN means that the spiking activities in the previous time steps will not affect the membrane potential values in the subsequent time steps." and change the "Dynamic" item in the table to "Reset".

Question 3.1: Hard reset will cause the model to be unable to calculate in parallel. Is the method proposed in this article only applicable to Hard-reset SNNs?

Answer: Our proposed methods and theoretical analysis do not conflict with the reset mechanism. IMP can generate new spiking patterns and pattern mappings for both Hard and Soft reset SNNs. LTS is designed specifically for static tasks, and the reset mechanism used in SNNs does not affect our hypotheses and theoretical results.

Question 3.2: Can this method scale up to a larger timestep (for example, 64)?

Answer: Yes, IMP remains effective with larger numbers of time steps (T=64), as demonstrated by experiments on the CIFAR10DVS (Table R3)，and IMP is also effective on time series tasks, as shown in Table R2.

Question 3.3: In language and autoregressive tasks, the timestep of SNN is aligned with the sequence length, which is extremely large. Using hard-reset SNNs may cause serious inefficiency. What do you think of this?

Answer: We agree with your point that the hard-reset mechanism can lead to inefficiency and affect the performance of SNNs on time series tasks, as the information from previous time steps will be completely lost when hard-reset neurons fire.

Thank you for your efforts in reviewing our article and providing constructive feedback. We’d like to reply to your concerns in detail.

Weakness 1: Some of the assumptions in the theoretical framework could be more explicitly stated and justified.

Answer: We will provide a clearer explanation of our assumptions in the subsequent version and ensure that the proof process strictly follows the format specifications, in order to improve the accuracy and readability of the content.

Weakness 2: Additional experiments, particularly in real-world scenarios, could further validate the proposed methods.

Answer: Thank you for the suggestion. We can provide the performance of our IMP method on the DVS128Gesture dataset, which captures human gesture movement trajectories using event cameras, thus being closer to real-world scenarios. Due to the current limitations in time and computational resources, we hope to try applying our method on larger-scale real-world scenarios dataset in our future work.

Table R4: Accuracy on DVS128Gesture dataset.

Weakness 3: The introduction and related work sections could provide more context to better situate the contributions within the broader literature.

Answer: We agree with your suggestion. We will supplement the introduction and related work sections to ensure they can better highlight our contributions.

Question 1: Can the authors provide more details on the assumptions made in the theoretical framework?

Answer: Sure. For static tasks, we define $f(\cdot)$ as the network computation, $\theta$ represents the network weights, $s[t]$ is the set of membrane potentials of all neurons in SNN with time step $t$, $x$ is the constant input intensity for time step $t=1,2,...,T$, and $y[t]$ is the corresponding output, the output of the SNN at each time step can be represented in the following form,

$$y[t] = f(x, s[t], \theta).$$

It can be found that the temporal variations of the output $y[t]$ is determined solely by the membrane potential set $s[t]$. Based on experimental observations, we have the following assumptions:

A1: $s[t]$ is time-varying in SNNs. A2: The change in $s[t]$ can alter the output $y[t]$. A3: Applying the same supervised label to $y[t]$ across all time steps may lead to difficulties in SNN convergence.

Question 2: How do these assumptions impact the generalizability of the proposed methods?

Answer: A1 typically holds true, except in the absence of external input. For A2, these phenomena are commonly observed in experiments, although minor perturbations may not significantly alter the output of the SNN. For A3, it is important to note that for static image classification tasks, the correct results can be obtained without perfect fitting. Therefore, when the model has sufficient expressive power to output similar $y[t]$ for different $s[t]$, and the task is simple enough, the performance of TET can approach the level of LTS.

Question 3: How does the proposed framework compare to other state-of-the-art models in terms of computational efficiency and scalability?

Answer: By combining our proposed method, the performance of SEW-ResNet can be close to the Transformer-based SNNs. However, compared to the current state-of-the-art Spiking Transformer models, there is still a certain gap (For more details, please refer to the PDF file we have submitted). Additionally, we hope to combine the proposed method with these advanced models, and explore its potential in a wider range of application scenarios.

Table R5: Accuracy and theoretical energy consumption on ImageNet test set.

Limitation 1: The discussion on the broader societal impact of the work could be expanded, particularly in terms of ethical considerations and potential negative consequences.

Answer: In this work, we found that adjusting the initial membrane potential can alter the spiking patterns of SNNs. Furthermore, in static tasks, the variation in SNN membrane potentials can influence the output. This finding may lead to the development of novel adversarial attack methods targeting SNNs. Attackers could achieve this by manipulating the membrane potential at specific future time steps. Such attacks would only require perturbation of the first few input frames, and could control the timing of errors in the SNN at a future time step. Compared to adding adversarial noise at all input time steps, this approach could be more stealthy.