Training High Performance Spiking Neural Network by Temporal Model Calibration

Rebuttal Appendix:https://anonymous.4open.science/r/TMC-0262

1. I tend to think the proposed method can provide a new perspective for the SNN community to a certain extent. The research perspective of Proposition 3.3 seems interesting. The authors conduct research on temporal heterogeneity and gradient allocation problems for different time steps.

We sincerely appreciate your feedback and are greatly encouraged by it.

2. As shown in Tab.2 and Tab.4, the proposed method seems to be more effective for neuromorphic datasets (e.g. DVS-CIFAR10), but the performance improvement on ImageNet-1k (large-scale dataset) and SNN Transformer architecture appears to be relatively limited (<1%). For CIFAR100 dataset, the reported accuracy of this work lags behind TCL and ETC.

Here, we highlight the performance advantages of our TMC method on neuromorphic datasets, ImageNet and CIFAR100. Furthermore, we evaluate TMC on real-world applications and the results demonstrate that TMC achieves SOTA performance.

1) Neuromorphic Datasets

• Dataset Feature Description: These datasets capture the dynamic changes in pixel intensity and record the resulting spike events using dynamic vision sensors (DVS). Compared to static or frame-based datasets, they contain rich spatio-temporal components by interacting the spatial and temporal information and follow the event-driven processing fashion triggered by binary spikes.
• TMC Advantages: Our method employs temporal heterogeneous learning using a temporal logit gradient rescaling strategy. This is beneficial for capturing more task-relevant spatio-temporal dynamics features compared to existing temporal homogeneous methods, such as SDT and TET. Therefore, TMC exhibits significant advantages on neuromorphic datasets.

2) ImageNet Dataset

• Dataset Feature Description: ImageNet is known for its high-resolution and complex scenes. Current efforts to enhance the performance of ImageNet mainly concentrate on refining the model architecture, often resulting in the introduction of a large number of additional parameters to achieve performance gains.
• TMC Advantages: It is notable that our proposed method TMC is plug-and-play. When TMC is integrated into the Hierarchical Spiking Transformer (current SOTA model) in our paper, it enhances both classification and calibration performance on ImageNet, further demonstrating the effectiveness of our method.

3)CIFAR100 Dataset

• Dataset Feature Description: CIFAR-100 is characterized by its low-resolution images, which are often blurry and contain relatively less information.
• TMC Advantages: With a small number of time steps, i.e., T=2, our method, which employs a temporal heterogeneous learning approach, is capable of capturing a richer set of features. Specifically, when T=2, TMC achieves a classification accuracy of 76.35%, outperforming other methods such as TEBN (75.86%), RMP-Loss (74.66%), and ETC (75.96%).
• Limitation of the Dataset: However, when the number of time steps is increased, our method may capture noisy features due to the blurriness of the images, which is not beneficial to performance improvement. This suggests that our method has more advantages when handling dynamic and complex tasks.

4) Real-Word Application

Moreover, we evaluate our method on real-world applications, including text sequential classification and dynamic action sequential recognition tasks, where TMC achieves state-of-the-art (SOTA) performance.

• Text Sequential Classification Task. We apply our method in the direct training phase of SpikingBert model proposed in [R1] on the widely used dataset Quora Question Pair (QQP). We compare the classification performance of TMC with SDT (as used in the original paper) and TET. TMC achieves a higher accuracy of 87.86% than SDT(86.82%) and TET(87.03%).
• Dynamic Action Sequential Recognition Task. We train VGGSNN with T=10 on DVS-Gesture and compare the performance of our method with current works in Table R1 of Rebuttal Appendix. Results show that TMC achieves SOTA performance with an accuracy of 99.12%.

3.We futher evaluate the training process of TMC on VGGSNN on DVSCIFAR10 with T=10.

1)Gradient Norm Evaluation:

In Figure R3 in the appendix, we evaluate the trend of gradient norms for model parameter updates during the training of TMC, SDT, and TET. TMC exhibits the highest norm, indicating faster convergence.

2)Training Stability Evaluation:

We visualize the training loss variation during the training of TMC, SDT and TET in Figure R4 in the appendix. The results demonstrate TMC's stable training, driving loss to minimal values, while SDT and TET suffer from overconfidence-induced oscillations.

[R1] Spikingbert: Distilling bert to train spiking language models using implicit differentiation.

Rebuttal Appendix:https://anonymous.4open.science/r/TMC-0262

1.Provide more results on dynamic datasets to make the method more convincable, like action recognition datasets including DVS-Gesture, SL-Animals, etc.

1)DVS-Gesture: We train VGGSNN with T=10 on DVS-Gesture and compare the performance of TMC with current works in Table R1 in the appendix. TMC achieves SOTA performance with an accuracy of 99.12%.

2)SL-Animals-DVS: VGGSNN (instead of a larger model due to time constraints) is trained using TMC with T=16. Comparison with SDT and TET shows TMC achieving a higher accuracy of 70.05% against SDT (66.75%) and TET (68.34%), demonstrating the superiority of TMC.

2.Is $\hat{P}_t$ the maximum value of softmax on the time-averaged logits? Then how does a temporally perfectly calibrated model trained with gt increase its temporal confidence monotonically to match Definition 3.1?I didn't see much relationship between the proposed method and Definition 3.1, nor a relationship between Definition 3.1 and 3.2.

Clarification of Definition 3.1 and Our Mechanism:

1)Element Definition(lines 165-169 left)

• Temporal Predicted Confidence: $\hat{P}_t=max(Softmax(\overline{z_t}))$.
• Accumulated Logit Outputs at t-th time step: $\overline{z_t}=\frac{1}{t}\sum_i^tz_i$.
• $t \in {[1,2,...,T]}$.
• $\hat{P}_t$ is different at different time steps, because $\overline{z_t}$ varies across time steps.

2)Definition 3.1:Temporally Perfectly Calibrated SNN(lines 169-176 left)

• Introduce model calibration into SNN's time dimension, where a temporally perfectly calibrated SNN satisfies Test Accuracy is Equal to Confidence at each timestep: ${\hat{*P*_t}} = \mathbb{P}(\hat{*y*}=*y*|\hat{*P*_t})$.

3)Definition 3.2:Temporal Gradient Scaling Factor(lines 177-176 left)

Model calibration is realized using a gradient rescaling factor to rescale the gradient of cross-entropy(CE) loss in the training of ANNs. To realize the temporal model calibration in SNN, we propose the temporal gradient scaling factor $g_t$ (lines 177-180 left) to rescale the gradient of CE loss at each time step, i.e., TET loss, in Equation 6.

4)Effect of $g_t$

• Note that Definition 3.1 constructs the concept of temporally perfectly model calibration in the field of SNN. Our work focuses on the rate-coding SNNs and we provide a further instantiation specific to rate-coding SNNs (lines 196-202 left). Specifically, current work suggests that the accuracy of rate-coding SNNs increases monotonically with time steps. Thus, for the temporally perfectly rate-coding SNNs, confidence should increase monotonically with time steps.
• To match definition 3.1, $g_t$ should meet: at earlier time steps, it should shrink the effect of CE loss logit gradient ($\Delta Z_t^{TET}$) to reduce the confidence. Conversely, $g_t$ should enhance the effect of $\Delta Z_t^{TET}$ to increase the confidence.

5)Method Design (Section 3.3)

In Section 3.3, we propose TMC loss function with a new regularization term to realize the effect of $g_t$. With temporal gradient theoretical analysis (lines 244-274 left and 220-235 right), we derive the temporal gradient rescaling factor $g_t^{TMC}$ in our method, which can optimize the rate-coding SNN to increase its confidence monotonically with time steps.

3.Some sentences in section 3.1 are false.

• Thank you for your correction. We follow the loss function definition of SDT in [R13] and the reference [1] in line 139 left should be changed to [R13]. [R13] defines the loss function of SDT by calculating the cross-entropy loss between the average pre-synaptic input of the output layer and the true label.
• TET calculates the cross-entropy loss between the pre-synaptic inputs of the output layer with the true labels at each time step. To be more accurate, we should change "the membrane potential of the last layer" in our paper to "pre-synaptic input of the output layer".

4.The definition of the confidence score is quite weird, not bounded in [0, 1].

We would like to provide clarification regarding the "confidence score" in our paper.

• Before Equation 9, our analysis focuses on absolute confidence values, that is, the maximum predicted probability, which is inherently bounded within the interval [0, 1].
• For Equation 9, to enhance the loss function's sensitivity to overconfidence, we adopted the ratio of confidence/(1 - confidence) as $\theta_t$. This essentially serves as a more effective regularization of confidence.

5.Result on N-Caltech101 is not SOTA.

On N-Caltech101, our method achieves 86.03% accuracy at T=10, while [3] achieves 87.86% accuracy at T=16. Reevaluating our method at T=16, we achieve 88.24% accuracy, surpassing [3].

[R13] Deng, S.,Li,Y.,Zhang, S., andGu, S. Temporal efficient training of spiking neural network via gradient re-weighting. arXivpreprintarXiv:2202.11946,2022.

Rebuttal Appendix:https://anonymous.4open.science/r/TMC-0262

1.The relationship between temporal heterogeneity and the performance of SNNs remains unclear. It is not clear whether the so-called logit gradient should be diverse.

1)Relationship between Temporal Heterogeneity and Performance

In SNNs, the neuronal dynamics of the membrane potential $u_t$ can be formulated as:   {\tau\frac{du_t}{dt}}=-(u_t-u_{reset})+I_t.\tag{R1}

When $u_t$ reaches the threshold $V_{th}$, $u_t$ is set to $u_{rest}$ and the neuron fires a spike:

s_t=\sum_{t_f}\delta(t-t_f).\tag{R2}

The temporal heterogeneity of SNNs implies $|\frac{du_t}{dt}|>0$, yielding benefits as follows:

• Reduction of Dead Neurons: Ensuring neurons spike when necessary.
• Sensitive Response to Temporal Dependency and Input Information: Capturing spatio-temporal dynamics effectively.
• Diverse Spike Firing Frequency: Adjustable $\Delta t$ in $s_t$ allows for diverse behaviors, such as burst spikes.

Overall, accumulating temporal heterogeneous neuron responses to varied temporal inputs over time steps enhances predictive confidence and performance.

2)Temporal Heterogeneity Improvement

Direct training of SNNs involves the discrete model expression:   {u_{t+1}^i}=\lambda(u_t^i-V_{th}s_t^i)+\sum_j\mathbf{W_{ij}}s_{t+1}^j.\tag{R3}

Enhancing temporal heterogeneity can be achieved by improving either the temporal dependency dynamics $\lambda(u_t^i-V_{th}s_t^i)$ or the input mapping dynamics $\sum_j\mathbf{W_{ij}}s_{t+1}^j$ via $\mathbf{W}$ training. We concentrate on the latter, which is less explored.

3)Logit Gradient Diversity Rationality

Enhancing linear separability, that is, increasing the rank of $\mathbf{W}$ is crucial for dynamic input mapping. Increasing temporal gradient diversity is beneficial to explore the solution space and increase $\mathbf{W}$ rank. However, existing methods lack exploration of logit gradient diversity.

2.The method is a minor modification of TET.This paper fails to show the significance of such marginal changes clearly.The regularization to improve diversity is not straightforward.

From the technological perspective, TMC is a regularization term modification based on TET. However, from the temporal gradient decent perspective, TMC is a critical improvement of TET. The significance of TMC is highlighted as follows:

1)Rethinking the Temporal Logit Gradient Calculation

• TET focuses on increasing the momentum of temporal gradient descent but ignores appropriately assigning magnitude and direction to the momentum. This leads to an overconfidence issue.

• Introducing model calibration into SNNs, we rethink the temporal gradient calculation properties that SNNs should meet and propose a temporal gradient rescaling mechanism. This mechanism assigns appropriate magnitude and direction to the gradient descent momentum across time steps.

2)Effective and Adaptive Regularization Term

• TET introduces the MSE regularization to prevent the occurrence of "particular outliers" but lacks deeper analysis on the issues. The regularization term appears to be an intuitive design. Moreover, the regularization term at each time step keeps same and fixed, relying on hyperparameters.
• During training, TMC with a new regularization term rescales the logit gradient effects of TET, adaptively responding to input data, training phases, and time steps without hyperparameter dependence.
• There exist straightforward model calibration techniques like label smoothing, but these methods have been found to be suboptimal relying on prior information and hyperparameters. Moreover, the regularization-based calibration techniques achieve SOTA performance.

3.Sequential tasks are more appropriate benchmarks to be considered.

• Text Sequential Classification Task. TMC is applied during the direct training phase of the SpikingBert [R1] model on Quora Question Pair dataset. Compared to SDT(86.82%) and TET(87.03%), TMC achieves higher accuracy at 87.86%.
• Dynamic Action Recognition Task. We train VGGSNN with T=10 on DVS-Gesture and compare the performance of TMC with current works in Table R1 in the appendix. TMC achieves SOTA performance with an accuracy of 99.12%.

4.TMC works better on neuromorphic datasets than static images. Provide explanation and insights.

TMC indeed exhibits more significant advantages on neuromorphic datasets.

• Compared to static datasets, neuromorphic datasets have rich spatiotemporal components by interacting the spatial and temporal information. 
• TMC employs temporal heterogeneous training and has the advantage of capturing more task-relevant spatiotemporal dynamics features, outperforming existing methods, which employ temporal homogeneous training.

[R1]Spikingbert: Distilling bert to train spiking language models using implicit differentiation.

Rebuttal Appendix:https://anonymous.4open.science/r/TMC-0262

1.Visualization or mathematical proof of increased heterogeneity.In-depth analysis of how tmc affects neuron activations.

1)Temporal Heterogeneity Visualization

We compare VGGSNNs trained by TMC and TET on DVSCIFAR10, visualizing the cosine similarity of layer features across time steps in Figure R1. TMC exhibits higher temporal heterogeneity at all layer levels.

2)TMC Effect

The neuron model:

{u_{t+1}^i}=\lambda(u_t^i-V_{th}s_t^i)+\sum_j\mathbf{W_{ij}}s_{t+1}^j.\tag{R4}

With TMC $\mathbf{W}$'s separability for varied inputs is enhanced, boosting neuron temporal heterogeneity in response to diverse temporal inputs.

2.Discussion on computational efficiency.

We analyze TMC's time and space complex of the logit gradient calculation relative to SDT and TET. (T:Time Step, B:Batch Size, C:Class Number)

1)SDT:Time complexity $O(T\*B\*C)$, space complexity $O(B\*C)$.

2)TET:Time complexity $O(T\*B\*C)$, space complexity $O(T\*B\*C)$.

3)TMC:Based on TET, it introduces additional calculation of the regularization term $\theta_t^{\lambda_t}$. At each timestep, generating $\theta_t$ needs $O(B\*C)$ time complexity and $O(1)$ space complexity. Calculating $\theta_t^{\lambda_t}$ needs $O(1)$ for both time and space complexity. Overall, TMC's logit gradient calculation time complexity is $O(T\*B\*C)+O(T\*B\*C)+O(T\*B\*C)+O(T)$, and space complexity is $O(T\*B\*C)+O(T\*B\*C)+O(T)+O(T)$.

TMC's total gradient computation, including hidden layer and logit gradients, is close to SDT and TET in computation complexity and training time.

3.No formal proof that tmc leads to a more stable or optimal training.Theoretically analyze the convergence properties of tmc.

1)Firing Rate Evaluation:

Firing rates impact SNN gradients. As shown in Figure R2, compared with SDT and TET, TMC-trained VGGSNN on DVS-CIFAR10 exhibits the lowest firing rates in shallow layers, rising with depth, and peaking in deeper layers. This indicates TMC's sensitive response to global features in deeper layers. Higher firing rates also mitigate the gradient vanishing.

2)Gradient Norm Evaluation:

Figure R3 further compares model parameters update gradient norms of TMC, SDT and TET. TMC shows the highest norm, suggesting faster convergence.

3)Training Stability Evaluation:

We theoretically analyze TMC's temporal rescaling factor $g_t$(lines 224-274 left). $g_t$ effectively rescale logit gradients, enhancing optimization over SDT and TET. Figure R4 demonstrates TMC's stable training, driving loss to minimal values, unlike SDT and TET, which suffer from overconfidence-induced oscillations.

4.The paper does not include an ablation study to determine which components of tmc contribute most to the performance gains.Analyze hyperparameter sensitivity of tmc.

1)In lines 358-384 left and 330-336 right, we conducted ablation studies on the regularization term's components—base $\theta_t$ and exponent $\lambda_t$. Both enhance performance, but their combined effect is optimal.

2)TMC does not introduce additional hyperparameters, ensuring flexibility.

5.It does not compare tmc to alternative approaches.It does not discuss some recent works on improving training stability through adaptive gradient methods.

1)Compare to Alternative Approaches: In Related Work section and lines 149-155 left, we discussed entropy-based loss function methods and highlighted their homogeneous training effect. Notably, TMC has a different training effect by introducing temporal heterogeneous training. As the code for these methods is not available, we can only compare TMC's classification accuracy with them in Section 4.4.

2)Compare to Adaptive Gradient Methods: Recent studies[R10-R12] have introduced adaptive gradient methods, primarily focusing on adjusting hidden layer gradients and using SDT loss to calculate the logit gradient. In contrast, TMC modifies logit gradient calculations and offers plug-and-play compatibility with various hidden layer gradient mechanisms.

6.Explore tmc to much larger models,real-world applications,deeper networks or more complex architectures.

1)Larger Model Evaluation: Hierarchical Spiking Transformer used in our paper with 64.96M parameters is the large model.

2)Deeper Model Evaluation: We verify TMC on ResNet101 on ImageNet dataset with T=4 and compare it with SDT and TET. TMC achieves a higer accuracy of 70.52% than SDT(68.74%) and TET(67.98%).

3)Real-World Applications: We evaluate TMC on text sequential classification and dynamic action sequential recognition tasks and TMC achieves SOTA performance. More detailed results can be seen in Response 3 to Review ksZk.

[R10]Learnable Surrogate Gradient for Direct Training Spiking Neural Networks [R11]Sparse Spiking Gradient Descent [R12]Gradient Descent for Spiking Neural Network