Efficient Parallel Training Methods for Spiking Neural Networks with Constant Time Complexity

We sincerely thank you for your high evaluation of our work, seeing this paper as a milestone due to its potential to enable parallel pre-training of SNNs. We also appreciate your valuable and detailed feedback. Below, we will answer your questions.

Q1: The usage of "\cite" and "\citet"

A1: We have carefully re-checked all citations throughout the manuscript. However, since ICML does not allow manuscript modifications during the rebuttal period, we will correct any remaining formatting issues (e.g., incorrect use of \cite vs \citet) in the camera-ready version if the paper is accepted.

Q2: Running time comparison of the training process and inference process

A2: Our proposed algorithm supports parallel training, including parallel forward and backward passes. We achieve speedups in both passes. To demonstrate this, we compare the training and inference complexity of BPTT and FPT at different timesteps ($T$). The experiments are conducted on the MNIST dataset using a 3-layer MLP (784×256×128×10), batch size = 256, learning rate = 0.001, and 80 training epochs.

Here, "time" refers to the average time required to train or infer a single batch on a single A100 GPU. As shown in the table, both the training and inference time of BPTT increase with the number of timesteps $T$. In contrast, the training time of FPT increases only slightly, and its inference time remains almost the same regardless of $T$. Notably, at $T = 512$, FPT is 21 times faster to train compared to BPTT.

Q3: Extended to pre-training SNNs parallelly

A3: A key advantage of our algorithm is its flexibility - it does not impose restrictions on specific network architectures and can be applied to a wide range of SNN models. By replacing the time-consuming LIF neuron sequential computation with our proposed FPT-based parallel iterations, both the forward and backward passes during training can be significantly accelerated.

Importantly, since FPT preserves the original neuron dynamics, the pre-trained model can still be deployed and inferred using standard SNN sequential processing. This makes FPT not only suitable for accelerating training, but also highly compatible with existing SNN hardware deployments. Therefore, FPT provides a practical and general solution for efficient parallel pre-training of SNNs without sacrificing biological fidelity or inference compatibility.

We again sincerely thank you for the thoughtful and encouraging comments. We hope that our response will clarify your concerns and further demonstrate the practicality and generality of FPT.

We thank you for your encouraging feedback and for recognizing that FPT significantly accelerates SNN training, making it a scalable and efficient solution for large-scale, long-duration tasks. We also appreciate your recognition of its potential in practical applications, especially neuromorphic computing and time-sensitive spatiotemporal processing. Below, we provide detailed responses to your questions and concerns.

Q1: The network dynamics affacted across long T

A1: We replaced the LIF neurons in a pre-trained 3-layer LIF-based MLP (784×256×128×10) on the MNIST dataset with parallel LIF neurons based on FPT. The cosine similarity (%) between the original and FPT-replaced outputs for $T$=8, 64, and 512 is shown in the table below:

As $T$ increases, the outputs of the FPT-based parallel LIF and the original LIF become less consistent due to error accumulation, resulting in a slight decrease in the similarity of the final network output. However, even for $T$=512, the similarity remains around 99.5%, indicating that FPT maintains a high degree of consistency in the network dynamics. Moreover, this minor difference can be addressed by light fine-tuning.

Q2: Experiments on ImageNet1K

A2: The proposed FPT mainly focuses on improving the speed for long $T$. Thus, in the submission, we reported results on dynamic datasets such as DVS-CIFAR10 ($T$=10) and DVS-Gesture ($T$=20), static datasets including ImageNet-100 ($T$=4), and graph datasets like Amazon Photos ($T$=8, 32, 128), as well as sequential datasets such as Sequential CIFAR10 ($T$=32) and Sequential CIFAR100 ($T$=32). In contrast, static datasets like ImageNet usually use smaller timesteps (e.g., $T$=5 or 6).

In addition, due to the high computational cost of training SNNs on ImageNet-1K, many previous studies (e.g., SSNN and LocalZO) have adopted alternative benchmarks such as Tiny-ImageNet or ImageNet-100. For this reason, we also chose ImageNet-100 as the representative benchmark in the main results. We are currently actively conducting experiments on ImageNet-1K, using the same experimental settings as MS-ResNet-18 [1]. Due to time constraints, we report the preliminary results we have obtained so far below.

The training loss refers to the cross-entropy loss of the last batch in the given epoch. As seen, with the same hyperparameters and number of epochs, FPT achieves higher accuracy and lower loss. Additionally, even with $T$=6, FPT achieves approximately 1.5× faster training speed compared to BPTT on the same 4 3090 GPUs. It is important to note that, due to time and computational limitations, we have not yet fine-tuned the hyperparameters for FPT. Further adjustments may lead to even better performance.

[1] Advancing Spiking Neural Networks Toward Deep Residual Learning. TNNLS 2024

We sincerely hope that our clarifications have addressed your concerns and will help improve your opinion of our work.

We sincerely thank the reviewers for recognizing the novelty and contributions of our proposed FPT framework, including its ability to reduce the training complexity of SNNs and the fact that existing parallel spiking neuron models can be viewed as special cases of FPT. We also appreciate the constructive and insightful feedback. Below, we answer the reviewers' questions in detail.

Q1: Various complexities of FPT during training refer to T-RevSNN

A1: Thank you for your valuable suggestion to further demonstrate the various complexities of FPT. The table below compares the theoretical training and inference complexity of different algorithms:

Here, $L$ is the number of network layers, $T$ is the timestep, and $K$ is the number of iterations in FPT, which is typically 3 and does not increase with $T$. Thus, for long $T$, $K$ can be approximated to be negligible. The space complexity $O(LKT)$ and time complexity $O(LK)$ can be approximated as $O(LT)$ and $O(L)$, respectively. The coefficient $\lambda$ represents the proportion of memory attributed to LIF components, which is the only part FPT increases—other parts of the network remain unaffected.

It is worth noting that methods such as OTTT, SLTT, and T-RevSNN truncate gradients or discard most of the temporal connections, which can limit their applicability to tasks requiring fine-grained temporal dynamics. In contrast, FPT accelerates SNN training without modifying the network and retaining the original neuron dynamics, so it is applicable to a wider range of SNN models.

Due to time constraints and the limitations of OTTT, SLTT, and T-RevSNN in capturing temporal features, our experiments mainly focus on comparing the complexity of BPTT and FPT at different timesteps $T$. The experiments are conducted on the MNIST dataset using a 3-layer MLP (784×256×128×10), a batch size of 256, a learning rate of 0.001, and 80 epochs.

Here, "Time" refers to the average running time for one batch during training or inference on a single A100 GPU. As shown in the table, both training and inference time for BPTT increase with $T$. In contrast, for FPT, the training time increases slightly, while the inference time remains almost constant. At $T=512$, FPT is 21x faster than BPTT during training. FPT only slightly increases the memory usage for the LIF during training without affecting other components of the network. In larger networks, such as MS-ResNet-18 trained on ImageNet1K, BPTT occupies 22.54GB on 4 3090 GPUs, while FPT occupies 23.52GB, resulting in only about a 4% increase in memory usage.

Q2: Experiments on ImageNet1K and If possible, HAR-DVS

A2: For a detailed discussion of the ImageNet1K experiments, we respectfully refer you to our response to Reviewer gXhg (A2), due to space constraints here.

We thank the reviewer for suggesting the HAR-DVS dataset, a promising new benchmark for DVS-based action recognition. We will cite it in the introduction: For instance, neuromorphic benchmark datasets such as HAR-DVS, DVS-CIFAR10 and DVS-Gesture typically need 10 or more timesteps to reach satisfactory accuracy. However, due to time and computational constraints, and because this dataset was recently released, the main algorithms we compare have not yet reported results on it, making a direct comparison difficult in the short term.

Q3: TET Loss

A3: The baseline accuracy and metrics in our submission are from the best results in the respective papers. As TET loss is a mainstream loss function for SNNs, most of these baselines, such as T-RevSNN and LocalZO, utilize TET loss during training.

Q4: "Time" in Table 2 and Figure 3

A4: We apologize for any confusion. The "time" here refers to the average time required to train one batch on a single 3090 GPU. All baseline models were implemented with the same network architecture and hyperparameter configuration as ours, differing only in training method. FPT requires significantly lower training time than these baselines.

We sincerely hope that our clarifications have addressed your concerns and helped strengthen your confidence in our work. Thank you again for your thoughtful review.

We sincerely thank you for recognizing that FPT enhances the efficiency of SNN training without modifying the underlying network architecture and for acknowledging FPT as a novel and well-motivated approach. Below, we provide detailed responses to your questions.

Q1: How does FPT compare in memory consumption to standard BPTT?

A1: To compare the memory consumption of BPTT and FPT, we conducted experiments on the MNIST dataset using a 3-layer MLP (784×256×128×10), a batch size of 256, a learning rate of 0.001, and 80 epochs. The table below shows the comparison at different $T$:

Here, "Time" refers to the average running time for one batch during training or inference on a single A100 GPU. At $T=512$, FPT is 21x faster than BPTT during training.

FPT only slightly increases the memory usage for the LIF during training without affecting other components of the network. Thus, the overall memory consumption is not significantly higher than BPTT. For instance, in larger networks such as MS-ResNet-18 trained on ImageNet1K, BPTT consumes 22.54 GB on 4 3090 GPUs, while FPT consumes 23.52 GB, resulting in only a 4% increase in memory usage.

Q2: How does the surrogate gradient approximation affect convergence guarantees?

A2: The surrogate gradient approximation does not affect the convergence guarantees because these guarantees are based on the forward pass of the network, while the surrogate gradient approximation is applied on the backward pass. In addition, the surrogate gradient approximation is necessary for backpropagation in SNNs due to the non-differentiability of the spike activation function. The effectiveness of surrogate gradient approximation has been well verified through experiments on various datasets.

Q3: Can FPT be applied to reinforcement learning tasks or other non-spiking domains?

A3: FPT can indeed be applied to reinforcement learning tasks as long as the task uses LIF-based SNNs for temporal processing. Our approach is particularly suitable for addressing the high latency problem of sequence processing using LIF neurons. However, for other non-spiking domains that do not rely on SNNs, this is beyond the scope of our current work.

Q4: What are the potential failure cases where FPT might not converge effectively?

A4: In the submission，we have discussed this issue in Appendix F "Discussion and Limitation". FPT works well for LIF-based SNNs with temporal decay, but may not be effective for simple Integrate-and-Fire models that lack such mechanisms. However, mainstream neuron models typically incorporate a decay factor or gating mechanism, which ensures that the influence of past inputs diminishes over time.

Other Questions

• The claim that FPT maintains all critical neural dynamics is also somewhat oversimplified because the backward process does not exactly correspond to traditional BPTT.

Neural dynamics, including leakage, integration, firing, and reset, are part of the neuron's forward process. Neither the backward process of BPTT nor FPT belongs to the neural dynamics process; they are solely used to optimize network weights.

• Since one of the key motivations of SNNs is energy efficiency, a discussion of power consumption during training would be valuable.

The energy efficiency of SNNs typically refers to inference on neuromorphic hardware. Both BPTT and FPT rely on floating-point operations on GPUs during training, which does not reflect deployment energy costs. FPT does not affect sequential inference behavior of SNNs. As shown in Table 2 of the submission and in additional experiments presented in Review hARo (A1), its firing rate is comparable to or even lower than BPTT, ensuring that SNNs trained with FPT remain energy efficient during deployment.

• The paper primarily focuses on neuromorphic datasets like DVS-CIFAR10 and DVS-Gesture

In the submission, we reported results on dynamic datasets such as DVS-CIFAR10 and DVS-Gesture, static datasets including ImageNet-100, and graph datasets like Amazon Photos, as well as sequential datasets such as Sequential CIFAR10 and Sequential CIFAR100.

• Additional numerical experiments demonstrating different convergence behaviors would strengthen the argument.

In the submission, Section 6.1 "LIF Dynamics Simulation" already discusses the convergence behavior.

• Providing open-source code would improve reproducibility.

The code will be publicly available on GitHub after the publication of this work.

We sincerely hope that our clarifications above have addressed your concerns and that our responses contribute positively to your understanding of our work.