A Parallel Multi-compartment Spiking Neuron For Multi-scale Sequential Modeling

Soundness 1: One of the main contributions of the paper, the parallel implementation of the algorithm, seems to hinge on the fact that they set beta_{n, n-1}. What is the effect of this on the neuronal dynamics? Can this model still be considered biophysically realistic?

Response: Thanks for your insightful comment. First of all, we would like to clarify that this work lies in the field of neuromorphic computing, which aims to design brain-inspired neural architecture that can solve real-world pattern recognition tasks with improved energy efficiency and effectiveness. This differs from the studies in computational neuroscience that aim to simulate and study the dynamics of biologically realistic spiking neuronal systems. Secondly, as you have rightly pointed out, we set $\beta_{n, n-1}$ to 0 in our model as it is a necessary setting to enable model parallelization across the temporal dimension. From the neuronal dynamics perspective, this operation eliminates the feedback effect from the $n^{th}$ compartment to the ${n-1}^{th}$ compartments. However, its impact should not be viewed in isolation. During the training process, all other compartments adapt and evolve, effectively compensating for the effects of this abstraction. As clearly demonstrated in Figure 3 and Appendix A.6, the learned dynamics within each compartment exhibit rich and varied multi-scale temporal patterns. These dynamics confirm the effectiveness of our proposed multi-compartment neuron model in capturing complex temporal dependencies across various time scales as their biological counterparts.

Clarity 1: Many parts of the paper are presented in an overly convoluted way. I believe that this paper would largely benefit from moving math that is not essential to understanding the context of the paper to the appendix, for example equations 17, 18, 19.

Response: Thanks for your helpful suggestion. We will move Equations 18 and 19 to Appendix, and thoroughly improve the clarity of the paper in our revised manuscript.

Clarity 2: In addition, it would be valuable if the authors attributed a biophysical meaning to their learnable parameters.

Response: We really appreciate your suggestions. The learnable parameters in our network can be categorized into two parts. The first part is the weight matrix $\mathcal{W}$ between network layers, which can be interpreted as synaptic strength between neurons. The second part consists of the parameters within the proposed neuron models. More specifically, as defined in Equation 3, $\beta$ can be considered as the coupling strength between different compartments, while $\tau$ is the membrane capacitance. These neuronal parameters are trained jointly with weight parameters in our SNNs. To improve the clarity, we will supplement the biophysical meaning of these parameters in our revised manuscripts to provide a more intuitive understanding of our formulation.

Clarity 3 & Question 1: Finally, I fail to understand where equation 14 comes from (although I am not exactly from the field, maybe it is clear to other reviewers). Page 6: I do not understand this sentence: we force vs to reset to a level below the firing threshold, which bears closer resemblance with biological observations. Why does this have closer resemblance to biological observations? Also closer than what other mechanism?

Response: Thank you for raising concerns on this point. In Equations 13 and 14, we have described the neuronal dynamics of the last compartment that takes charge of spike generation. Inspired by the repolarization process of biological neurons after each spike generation, the reset-by-subtraction scheme [1] has been commonly used in SNNs to reset the membrane potential after neuron firing. In particular, the membrane potential ($v_s$) is reset by subtracting the value of the threshold ($\theta$) from it. In contrast, our model modifies this mechanism to enable parallelization for the output compartment. This is achieved by subtracting the value of $\theta \lfloor v_s/\theta \rfloor$ from $v_s$ as given in Equation 14. This operation ensures the membrane potential has been reset to a level that is below the firing threshold as well as close to the resting potential ($0$). Let me provide an example to illustrate the difference between these two reset schemes. For instance, assume $v_s[t]=10.5$, $\theta=1$, and the resting potential = 0. Following the reset-by-subtraction scheme, $v_s[t+1]$ will be 9.5 after resetting, which is still much higher than the threshold. In contrast, following our reset scheme, $v_s[t+1]$ will be 0.5, which is closer to the resting potential. Therefore, our model more closely resembles the repolarization process observed in biological neurons, which reset the membrane potential to a value close to its resting potential.

Weakness 1 & Question 1, 2: In section 5.3, only the speeds of serial and parallel implementation of the PMSN are compared. Can the authors provide a speed comparison between the PSN and the PMSN? What are the theoretical FLOPs and memory consumption of the PMSN? I suggest that the authors provide a comparison between these two neurons. It would be better if a comparison between the memory consumption during training of an SNN with two neurons is provided.

Response: Thanks a lot for raising concerns about the insufficient comparison of speed, computational cost, and memory consumption between PSN and PMSN. We have already conducted both empirical and theoretical studies on the computational cost of PMSN and PSN models. Our theoretical analysis, detailed in Appendix A.5, breaks down the theoretical computational cost into two primary components: the number of Multiply-Accumulate (MAC) operations and the number of cheap Accumulate (AC) operations. In our analysis, the FLOPs correspond to the number of MAC operations. Consider a scenario with $m$ neurons across a time window of length $t$. The theoretical FLOPs for PSN and PMSN (5-compartment) are calculated as $mt^2$, and $32mt$, respectively. This indicates that as the time window lengthens, the PSN model requires increasingly more FLOPs compared to the PMSN model.

As per your suggestion, we have further conducted a comparative analysis focusing on memory consumption and training/inference speed using the following three benchmarks. In all experiments, identical training configurations and network architecture are used. The results are provided as follows:

As expected, the PMSN model requires more time and memory than the PSN model across all datasets. This increased demand is primarily attributed to a larger number of neuronal compartments being used (i.e., five in PMSN v.s. one in PSN). However, the increase in actual time and memory consumption is relatively modest, generally less than twice that of the PSN model in all experiments. This highlights the effectiveness of our proposed parallelization method. Furthermore, we believe this additional computational cost is worthwhile when considering the significantly enhanced sequential modeling capacity as demonstrated on the Sequential Cifar10 and Cifar100 datasets.

To further strengthen our manuscript, we will include these new results and discussions in our revised manuscript to provide a more comprehensive comparison between the two models.

Weakness 1 & Question 1: The implementation seems to require feed-forward structure as well as positive inputs, which seems quite restrictive (in particular the latter). I don't follow why one can assume that $I(t)$ is always positive for the ``parallel implementation of the output compartment with reset''.

Response: We greatly appreciate your time in reviewing our manuscript and providing constructive feedback. As you have rightly pointed out, we did assume that $I_h[t]$ to be non-negative in Equation 15. This is a compromise to enable the parallelization of the reset mechanism, and we provide a detailed proof of Equation 15 based on this assumption in Appendix A.4.2. However, it's worth mentioning that this assumption only applies to the last compartment. It does not require the input to our PMSN model, namely $I(t)$, to be always positive. In fact, the negative input current to other compartments (i.e., 1 to n-1) in Equation 5 is allowed. As specified below the Equation 3, its value is formulated as $I^l(t)=\mathcal{W}^l S^{l-1}(t)$. Therefore, we don’t need to impose any constraint on the input signal $I(t)$ of the PMSN model, but only apply a clamp function on the intermediate states (output of Equation 12) before it is fed into Equation 15. We will highlight this point in our revised manuscript to avoid confusion.

In the following, we would like to discuss the impact of removing this assumption. Firstly, according to our analysis, if negative inputs are allowed for the output compartment, the state update of the last compartment can only be done in a serial manner, while other compartments can still be computed in parallel. Following your suggestion, we have conducted an ablation study to quantify its impact on the simulation speed. The time elapse (seconds/epoch) for PMSN and last-compartment-serial PMSN are 62 vs 158 on sequential CIFAR-10 (T=32), and 73 vs 5182 on sequential CIFAR-10 (T=1024) tasks. These exploded simulation time underscore the necessity of the proposed parallel solution. Interestingly, we also observed that the last-compartment-serial PMSN model, which allows negative input for the last compartment, performs worse than the PMSN model. It exhibits a 3.98% and 4.5% accuracy drop on sequential CIFAR-10 and sequential CIFAR-100 (T=32) tasks, respectively. We suspect that negative input can result in negative membrane potentials that adversely affect the gradient flow. This is because when the membrane potential is far from the firing threshold, the gradient g’[t] computed from the surrogate function will become zero. This issue hinders the effective gradient propagation, therefore, leads to poorer classification accuracies than our proposed one.

Weakness 2: The neuron model does not seem to improve the benchmark results dramatically over other approaches in all tasks (ie recurrent RNNs are similar for S-MNIST) and it thus remains open how useful the model will be.

Response: We appreciate your insightful comment. While it is true that the performance enhancement of our model over recurrent SNNs may not seem dramatic in the context of the S-MNIST task, it’s worth noting that this task is relatively easy and models can perform well with limited memory capacity. The existing methods have already reached a very high accuracy and the margin for improvement is limited. Specifically, the state-of-the-art (SOTA) model TC-LIF with an RSNN architecture achieves an error rate of just 0.8%. Nevertheless, our model managed to reduce this error rate to 0.47% further.

We understand that the S-MNIST dataset alone may not fully demonstrate the potential benefits of our proposed model. Thus, to further illustrate the enhanced sequential modeling capabilities of our proposed model, we have extended our comparisons to include more challenging sequential tasks. In these scenarios, our model consistently outperforms other SOTA models by significant margins. For instance, in the SHD speech command recognition task, our model achieves an accuracy improvement of 1.8% and 5.3% over the SOTA Adaptive Axonal Delay (feedforward) and TC-LIF (recurrent) model respectively, while maintaining a comparable number of parameters. Moreover, in the Sequential CIFAR10 (T=32, 1024) and Sequential CIFAR100 (T=32) tasks, our model surpasses the accuracy of other SOTA models by 2.52%, 11.91%, and 3.87%, respectively. It is worth noting that, the accuracy improvements increase along with the increasing complexity of the task.

We believe these results underscore the potential of our model in handling more complex sequential modeling tasks, and we look forward to exploring its capabilities further in future research.

I thank the authors for the detailed response and clarification. Given that the assumptions of a positive current only applies to the last compartment and not to the others and has limited (or even detrimental) effect on accuracy, I will revise my score to 6.

Response: We appreciate your comments and would like to address your concerns with our following responses:

Weakness 1: The model is very specific, without any particular justification. When one uses spiking networks, it is usually (in my experience) because one wants to model a spiking neuron system, but then choices of compartments are no longer arbitrary.

Background – Different from the studies in computational neuroscience that aim to simulate and study the dynamics of biologically realistic spiking neuronal systems, this work lies in the field of neuromorphic computing. In particular, we aim to design brain-inspired neural architecture that can solve real-world pattern recognition tasks with improved energy efficiency and effectiveness. To this end, simplified phenomenological spiking neuron models are typically used (e.g., Leaky Integrate-and-Fire (LIF) model), which allow SNNs to be implemented in silico with extremely low energy cost. In this context, the choice of compartments in our model is also driven by these goals. We trade off its computational efficiency and efficacy in sequential modeling tasks, making the model well-suited for processing complex temporal signals with low-energy requirements.

Weakness 2: This is really a question of motivation for the paper that seems missing to me. What problem are the authors solving? Why does one need a spiking neuron model? Or is this particular spiking neuron model related to one in common use in circumstances unfamiliar to me?

Research Gap – While SNNs have achieved substantial performance improvements for tasks with limited temporal dynamics, such as image classification, their performance in handling tasks with long sequence lengths and rich temporal dynamics (e.g., audio signal processing) remains limited. This is attributed to the limited memory capacity and neuronal dynamics offered by the commonly used LIF model, which is also the one used in (Zenke & Ganguli 2018). This oversimplified model, while computationally efficient, exhibits limited capacity in representing the rich neuronal dynamics of biological neurons. To bridge this gap, we design a memory-enhanced multi-compartment spiking neuron model in this work. Our model specifically addresses these limitations by leveraging the interplay among neuron compartments.

Weakness 3: If it is already related to a model in common use, then having a fast implementation is of some importance.

Novelty - Our model draws inspiration from the multi-compartment model used to describe biological neurons. As mentioned by Reviewer 9iDX ``This model is not new, however, it is largely restricted to biological spiking networks instead of SNNs’’. In the context of SNNs, our proposed model is the first to incorporate more than two compartments, demonstrating superior performance in sequential modeling. Furthermore, we introduce a GPU-accelerated solution to enhance the simulation speed of the proposed model. This is also the first multi-compartment model in SNNs that supports parallel training on GPUs.

Weakness 4: A related issue is that the models used for comparisons seem very limited (if one is only concerned with whether the model is a spiking model). Many groups have been training spiking neural networks for a long time (Zenke & Ganguli 2018 is but one example).

Outstanding Sequential Modelling Capacity - In our experiment section, we aim to provide a holistic view of the existing SNN research by including all the state-of-the-art memory-enhanced spiking neuron models that have been studied earlier, including the LIF model that had been used in (Zenke & Ganguli 2018). Our analysis suggests that the proposed multi-compartment model can not only be trained in parallel, but also excel in modeling multi-scale temporal dependencies, thereby bridging the performance gap between biological neuron and spiking neuron models in terms of the sequential modeling capacity.

Weakness 5: If the multi-compartment model itself is not of particular importance, but I needed to train a spiking model, I could just as well use a network model from one of these groups. Does the PMCS have advantages over those models?

Practicality – As discussed above, if you're seeking to train an SNN to process temporal signals with long sequence length and rich temporal dynamics (e.g., in scenarios like speech processing), our model presents an excellent choice due to its enhanced sequential modeling capacity and rapid training speed.

Weakness 1: Regarding Figure 4: while the parallel model was compared to its serial implementation in terms of speed up, what is the ratio/training time difference between PMSN and PSN? If PSN is faster, since it is only a single compartment model, how significant is this difference? I am not referring to the computational cost but instead training acceleration only.

Response: Thanks for your insightful comment. Following your suggestions, we have conducted a comparative analysis of the training speed between the PMSN and PSN models. This involves measuring each model's training time across various tasks with different time window lengths, with identical training configurations and network architectures for a fair and direct comparison. The results, including time elapsed (seconds/epoch) and classification accuracy (%), are as follows:

The results clearly show that the PMSN model, with more complex and sophisticated multi-compartment dynamics (five compartments in PMSN vs. one in PSN), trains slightly slower than the PSN model. However, the difference in training speed is not significant (with a ratio of less than 2 across all the experiments), demonstrating the efficiency of our parallelization approach. Furthermore, this minor trade-off in speed is outweighed by the superior sequential modeling capabilities of PMSN, particularly evident in its performance on the Sequential CIFAR-10 dataset. This underlines the value of the PMSN model in applications where enhanced temporal signal processing capacity is critical.

Weakness 2: Is the parallel implementation/methodology that incorporates the reset mechanism general enough to be applied to single compartment models?

Response: We appreciate your insightful question. This reset mechanism can indeed be seamlessly transferred to the single-compartment Integrate-and-Fire (IF) model, thereby enabling parallelizable training for this model. The generalization is straightforward and involves a simple modification. Specifically, it requires replacing the input of Equation 13, denoted as $I_h[t]$, with the input of the single-compartment model. For instance, the replaced input can be denoted as $I^l[t]=\mathcal{W}^l S^{l-1}[t]$, which adheres to a common practice of single-compartment models regardless of the dynamics of synaptic decay. Consequently, Equations 13 and 14 effectively formulate a parallel IF model that incorporates our reset mechanism.

We sincerely hope that our response has addressed your concerns. We will include these additional results in the revised manuscript and Appendix.