Bipolar Self-attention for Spiking Transformers

Thank you very much for your recognition of our work.

Weakness 1 and Additional Comments: imprecise expression.

A: We thank you for this valuable feedback and apologize for the unclear presentation. In the revised manuscript, we will replace "SOTA Spiking Transformer" with "advanced Spiking Transformer" to ensure academic rigor. Additionally, we will clarify that in Figure 3, the dimensions N and D are 256 and 512, respectively, and update the corresponding dimensions in Table 1 accordingly. We greatly appreciate your constructive suggestions, which significantly contribute to improving the manuscript quality.

Weakness 2: The paper fails to quantify the additional theoretical energy cost incurred by BSA’s full shift operation relative to SSA.

A: Thank you for the valuable feedback. Although the operation between membrane potentials (V) and attention maps resembles matrix multiplication, the computation is fundamentally different. Our attention maps consist of powers of two, representing bit-shift positions rather than conventional weights. The following table presents energy consumption data for different operations on ASIC (45nm)[1], providing a quantitative foundation for assessing the energy advantages of this shift-based approach.

We conduct a fair comparison using standardized configurations with Q, K, V matrices of dimension B×C×N = 256×512×512. Under identical computational loads, BSA achieves energy efficiency comparable to Spiking SSA, with both methods significantly outperforming Standard SA.

Weakness 3 and Question 3 : The authors consider only SSA as a spiking self-attention mechanism, omitting variants such as SDSA[1] and DSSA[2].

A:As you pointed out, both SDSA and DSSA are improved variants of SSA. However, SDSA and DSSA primarily treat the self-attention mechanism as a token mixer, without explicitly computing the correlation between Q and K—which is a key factor behind the success of self-attention in ANNs. Therefore, we aim for spiking self-attention to retain the ability to capture such correlations, and for this reason, we only compare our method against SSA. In the revised manuscript, we will include this comparative analysis to clarify the distinction.

Question 1: Based on the three vision tasks, BSA shows limited performance improvement in classification tasks, but a significant enhancement in regression tasks. Could the authors provide an analysis of the underlying reasons for this discrepancy?

A: We appreciate the reviewer’s insightful observation. The performance gap between classification and regression tasks stems from the fact that classification generally requires coarse-grained representations and involves simpler gradient feedback, making it less sensitive to attention refinement.

In contrast, regression tasks rely on fine-grained modeling and continuous value prediction, which benefit significantly from our BSA's ability to preserve polarity interactions and apply stable attention allocation. Specifically, our BSA offers two main advantages over SSA: (1) a ternary Q-K correlation mechanism that captures richer polarity semantics; and (2) a lightweight Shiftmax normalization that approximates Softmax without introducing non-linear operations, leading to more precise and effective attention allocation.

Question 2: The authors demonstrate the method’s effectiveness on several network architectures; however, the neurons in Spike‐Driven V3 were specially designed. How does the proposed approach conform to the ILIF neuron’s dynamical properties, and can it support true spike‐driven inference on par with V3?

A: Thank you for your valuable suggestions. STD-V3 employs I-LIF[2] neurons as an innovative approach for decoupling training and inference processes. We would like to provide a detailed introduction to how ternary neurons are trained and perform inference in the V3 architecture.In the training stage, the output of an I-LIF neuron is expressed as:

$$\mathbf{s}^\ell = \frac{1}{T} \cdot \text{round}\left( \mathrm{clip}(\mathbf{x}^\ell, -T, T) \right),$$

where $\mathbf{x}^\ell$ denotes the input current at layer $\ell$, $T$ represents the simulation time steps of SNNs, $\mathrm{clip}(\mathbf{x}^\ell, -T, T)$ constrains $\mathbf{x}^\ell$ to the interval $[-T, T]$, and $\text{round}(\cdot)$ denotes rounding to the nearest integer. The computed output $\mathbf{s}^\ell$ represents the signed spike firing rate.

During inference, the firing rate $\mathbf{s}^\ell$ is decomposed into $T$ ternary spike signals. The sign of $\mathbf{s}^\ell$ determines the spike polarity: positive spikes are emitted when $\mathbf{s}^\ell \geq 0$, while negative spikes are emitted when $\mathbf{s}^\ell < 0$. It can be mathematically described as:

$$\mathbf{s}^\ell[t] = \mathrm{sign}(\mathbf{s}^\ell) \cdot \mathbf{1}(t \leq |\mathbf{s}^\ell| \cdot T)$$ $$\mathbf{s}^\ell = \frac{1}{T} \cdot \sum_{t = 1}^{T} \mathbf{s}^\ell[t]$$

where $\mathbf{s}^\ell[t] \in \{-1, 0, 1\}$. The front-loaded firing pattern concentrates all $|\mathbf{s}^\ell| \cdot T$ spikes within the first $|\mathbf{s}^\ell| \cdot T$ time steps, with spike polarity determined by the sign of $\mathbf{s}^\ell$.

Question 3: Have the authors compared their method to BNNs (with activations limited to +1 and –1)? Please support this with both theoretical analysis and experimental validation.

A: BSA introduces a "silent" state (0) that better mimics the sparse activation characteristics of biological neurons. This inherent sparsity provides significant computational advantages in QK correlation calculations—zero-value positions can be directly skipped, whereas BNN's {-1,1} system enforces computation across all positions. Comparative experiments between binary SA and BSA within BERT architecture would effectively validate this theoretical advantage.

Reference:

[1] Yao M, Qiu X, Hu T, et al. Scaling spike-driven transformer with efficient spike firing approximation training[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.

[2] You H, Chen X, Zhang Y, et al. Shiftaddnet: A hardware-inspired deep network[J]. Advances in Neural Information Processing Systems, 2020, 33: 2771-2783.

[3] Qin H, Ding Y, Zhang M, et al. Bibert: Accurate fully binarized bert[J]. arXiv preprint arXiv:2203.06390, 2022.

Your suggestions have greatly contributed to improving our quality. We will address your concerns and questions one by one.

W1 and Q2: Details for the implementation of Ternary Spiking Neuron on the SDT-V3?

A: Thank you for your valuable suggestions. STD-V3 employs I-LIF[4] neurons as an innovative approach for decoupling training and inference processes. We would like to provide a detailed introduction to how ternary neurons are trained and perform inference in the V3 architecture. In the training stage, the output of an I-LIF neuron is expressed as:

$$\mathbf{s}^\ell = \frac{1}{T} \cdot \left\lfloor \mathrm{clip}(\mathbf{x}^\ell, -T, T) \right\rceil,$$

where $\mathbf{x}^\ell$ is the input current at layer $\ell$, $T$ is the simulation time steps, $\mathrm{clip}(\mathbf{x}^\ell, -T, T)$ constrains $\mathbf{x}^\ell$ to $[-T, T]$, and the symbol $\left\lfloor \cdot \right\rceil$ denotes rounding to the nearest integer.

The output $\mathbf{s}^\ell$ represents the signed spike firing rate. During inference, $\mathbf{s}^\ell$ decomposes into $T$ ternary spike signals where spike polarity follows the sign of $\mathbf{s}^\ell$:

$$\mathbf{s}^\ell[t] = \mathrm{sign}(\mathbf{s}^\ell) \cdot \mathbf{1}(t \leq |\mathbf{s}^\ell| \cdot T)$$ $$\mathbf{s}^\ell = \frac{1}{T} \cdot \sum_{t = 1}^{T} \mathbf{s}^\ell[t]$$

where $\mathbf{s}^\ell[t] \in \{-1, 0, 1\}$. The front-loaded firing pattern concentrates all $|\mathbf{s}^\ell| \times T$ spikes within the first $|\mathbf{s}^\ell| \times T$ time steps, with spike polarity determined by the sign of $\mathbf{s}^\ell$.

W2 and Q1: This work lacks experiments on neuromorphic datasets. Giving more experiments on CIFAR10, CIFAR100, CIFAR10DVS, DVSGesture.

A: We evaluate our BSA on small-scale datasets, including CIFAR10, CIFAR100 and temporal neuromorphic datasets(CIFAR10-DVS and DVS128 Gesture). The detailed results on the QKformer structure for these small-scale datasets are presented in the following：

The experimental results demonstrate that BSA combined with QKformer can achieve overall performance improvements. However, as discussed in our analysis, the advantages of BSA are not prominently exhibited in classification tasks.

W3: The authors should provide more details about tracking and segmentation tasks

A: Due to space constraints, detailed implementation specifics for segmentation and tracking tasks are provided in the Appendix, including comprehensive hyperparameter settings and network configurations. Code will be made available on GitHub.

Segmentation Implementation: For ADE20K (150-category semantic segmentation benchmark), we employ the MMSegmentation framework [3] with 160,000 training iterations. The AdamW optimizer uses a learning rate of 2×10⁻⁴ and weight decay 0.05, with 1,500-iteration warm-up followed by linear decay. Experiments utilize 4 NVIDIA A100 80GB GPUs.

Tracking Implementation: SDTrack-Base employs C=64 channels, while SDTrack-Tiny uses C=32 with reduced Stage 4 (6 blocks) and Stage 5 (10C channels, 2 blocks). Tracking heads differ in channel dimensions: Tiny outputs 256 channels versus Base's 512 in the first layer, with progressive halving in subsequent layers. Both maintain 128×128 template and 256×256 search image inputs. Training involves ImageNet-1K pretraining followed by fine-tuning on event-based tracking datasets via pair matching tasks, without data augmentation.

Reference:

[1] Wang, Y., Shi, K., Lu, C., Liu, Y., Zhang, M., & Qu, H. (2023, August). Spatial-Temporal Self-Attention for Asynchronous Spiking Neural Networks. In IJCAI (pp. 3085-3093).

[2] Zhou C, Zhang H, Zhou Z, et al. Qkformer: Hierarchical spiking transformer using qk attention[J]. Advances in Neural Information Processing Systems, 2024, 37: 13074-13098.

[3] Contributors, MMSegmentation. "MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark." 2020.

[4] Yao M, Qiu X, Hu T, et al. Scaling spike-driven transformer with efficient spike firing approximation training[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.

Thank you very much for your recognition of our work. In response to the weaknesses and suggestions you have raised, we will provide further detailed explanations:

Weakness 1 and Questions 4:This work lacks Ablation experiments comparing RMP and TMP performance, despite both Theory 1 and Theory 2 resting on the premise that RMP outperforms TMP.

A: Thank you for your valuable suggestions. We have added performance evaluations of RMP and TMP under the Shiftmax framework. The experiments were conducted using the QKformer architecture on CIFAR-100. As shown in the table below, RMP and TMP achieve comparable performance, while RMP introduces substantial floating-point multiplication operations. Thereofre, TMP serves as a highly efficient alternative to RMP.

Weakness 2: The manuscript lacks description of the spiking neuron model within the Transformer Block.

A: Our BSA implementation comprises two primary neuron architectures. First, the ternary LIF neuron, commonly employed in QKformer and Spikeformer, governed by the following. The dynamics of the ternary spike neuron (TSN) can be concisely expressed as follows. The membrane potential is first updated by $U[t+1] = H[t] + X[t+1],$ where $H[t]$ is the previous hidden state and $X[t+1]$ is the input at the current time step. The neuron then generates a ternary spike output using $S[t+1] = \mathrm{Sign}(U[t+1]) \cdot \Theta\left( |U[t+1]| - V_{\mathrm{th}} \right),$ where $\Theta(\cdot)$ is a step function and $V_{\mathrm{th}}$ is the firing threshold. Finally, the internal state is updated by: $H[t+1] = V_{\mathrm{reset}} \cdot S[t+1] + \tau \cdot U[t+1] \cdot (1 - |S[t+1]|),$ where $V_{\mathrm{reset}}$ is the reset value and $\tau$ is a decay factor that determines the leaky integration when no spike occurs. Second, the ternary ILIF model within the STD-V3 framework, which operates in distinct modes during training and inference phases. In the training stage, the output of an I-LIF neuron is expressed as:

$$\mathbf{s}^\ell = \frac{1}{T} \cdot \left\lfloor \mathrm{clip}(\mathbf{x}^\ell, -T, T) \right\rceil,$$

where $\mathbf{x}^\ell$ is the input current at layer $\ell$, $T$ is the simulation time steps, $\mathrm{clip}(\mathbf{x}^\ell, -T, T)$ constrains $\mathbf{x}^\ell$ to $[-T, T]$, and the symbol $\left\lfloor \cdot \right\rceil$ denotes rounding to the nearest integer.

Weakness 3: The proposed BSA is validated solely on visual datasets, with no evaluation on sequential or speech processing tasks.

A: Thank you for your valuable suggestions. Following the architecture proposed by Wang et al.[], we replaced the T-SSA component with our BSA and validated the approach on the GSC-V2 dataset. Our BSA achieves a 0.85% improvement over ST-Transformer.

Weakness 4: In the semantic segmentation experiments, the authors re-implement only the 19 M-parameter model; to ensure a fair comparison, the 5 M and 10 M variants should also be re-implemented.

Thank you for your valuable suggestions. To further validate the effectiveness and scalability of our approach, we conducted multiple independent implementations and repeated experiments on 5M and 10M parameter models. The corresponding baseline experimental results are presented as follows:

Questions 1 : Appendix A shows that binary matrix multiplication partially achieves low-entropy activation：why can’t binary multiplication directly replace ternary multiplication + Softmax?

A: Low-entropy activation is an inherent property of Softmax that enables sharp contrast in attention scores—amplifying weights for relevant positions while suppressing irrelevant ones, thereby achieving clear differentiation. Although BMP generates attention matrices with low-entropy-like distributions (exhibiting similar high-low contrast patterns), the resulting high and low attention regions may not accurately identify the truly important and negligible content, potentially failing to achieve optimal attention allocation.

Questions 2 :Why is it necessary to establish local correlation? Does it complement distributional similarity?

A:We analyze local correlation because while global distributional similarity indicates overall statistical alignment, it does not guarantee accuracy at individual query-key interaction levels. Attention mechanisms fundamentally rely on precise pairwise correlations between specific tokens, making local accuracy crucial for effective attention allocation. Our analysis demonstrates that for typical spike firing rates, TMP exhibits approximately 2× stronger local correlation with RMP than BMP.

Questions 3 :In my view, the strengths of SNNs are most evident on dynamic datasets (e.g., DVS), yet the authors evaluate their method predominantly on static image tasks. I recommend validating the proposed approach on a dynamic benchmark to demonstrate its broader applicability.

A: Thank you for your valuable suggestions. To further validate the generalizability and robustness of BSA, we conducted supplementary performance evaluations on neuromorphic datasets DVS-CIFAR10 and DVS-Gesture. We employed the QKformer architecture with two Transformer blocks to ensure sufficient representational capacity for processing complex temporal dynamics. BSA and SSA were compared under identical experimental conditions to ensure fair evaluation. As shown in the table below, BSA demonstrates significant performance improvements over SSA on both challenging dynamic datasets, further supporting the viability of BSA as an efficient attention mechanism alternative.

Questions 5 :In the presented BSA, V does not activated by Spiking neurons. Does it still retain true spike‐driven behavior? It would also strengthen the paper to add theoretical energy‐consumption analyses to quantify the claimed efficiency gains.

A：Thank you for the valuable feedback. Although the operation between membrane potentials (V) and attention maps resembles matrix multiplication, the computation is fundamentally different. Our attention maps consist of powers of two, representing bit-shift positions rather than conventional weights. The following table presents energy consumption data for different operations on ASIC (45nm)[8], providing a quantitative foundation for assessing the energy advantages of this shift-based approach.

We conduct a fair comparison using standardized configurations with Q, K, V matrices of dimension B×C×N = 256×512×512. Under identical computational loads, BSA achieves energy efficiency comparable to Spiking SSA, with both methods significantly outperforming Standard SA.

Reference

[1] Yang, Qu, Qi Liu, and Haizhou Li. "Deep residual spiking neural network for keyword spotting in low-resource settings." Interspeech. 2022.

[2] Wang S, Zhang D, Shi K, et al. Global-Local Convolution with Spiking Neural Networks for Energy-efficient Keyword Spotting[C]//Proc. Interspeech 2024. 2024: 4523-4527.

[3] Wang, Y., Shi, K., Lu, C., Liu, Y., Zhang, M., & Qu, H. (2023, August). Spatial-Temporal Self-Attention for Asynchronous Spiking Neural Networks. In IJCAI (pp. 3085-3093).

[4] Yao M, Qiu X, Hu T, et al. Scaling spike-driven transformer with efficient spike firing approximation training[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.

[5] Zhou Z, Zhu Y, He C, et al. Spikformer: When spiking neural network meets transformer[J]. arXiv preprint arXiv:2209.15425, 2022.

[6] Zhou C, Yu L, Zhou Z, et al. Spikingformer: Spike-driven residual learning for transformer-based spiking neural network[J]. arXiv preprint arXiv:2304.11954, 2023.

[7] Zhou C, Zhang H, Zhou Z, et al. Qkformer: Hierarchical spiking transformer using qk attention[J]. Advances in Neural Information Processing Systems, 2024, 37: 13074-13098.

[8] You H, Chen X, Zhang Y, et al. Shiftaddnet: A hardware-inspired deep network[J]. Advances in Neural Information Processing Systems, 2020, 33: 2771-2783.

We sincerely appreciate your recognition of our work, which serves as significant encouragement for our research endeavors. Regarding the suggestions you have proposed, we provide a comprehensive overview below.

Weakness 1: I am curious whether this module can serve as a viable replacement for the corresponding components in existing LLMs, and what level of performance improvement it can bring.

A: We appreciate your valuable suggestion regarding BSA as a potential alternative to SA. While we agree that evaluating BSA on large language models such as LLaMA[1] and DeepSeek[2] would be meaningful, time and computational constraints during the rebuttal period precluded retraining or fine-tuning these architectures. Instead, we conducted comprehensive experiments using BERT[3] on GLUE benchmark tasks.

As shown in the above Table, we replace the SA components in BERT with binary SA[4] in BiBERT and our BSA (Ours). The experiments demonstrate that BERT with BSA exhibits only partial performance degradation compared to vanilla BERT and superior performance to Binary SA[4]. These results support your expectation that BSA represents a viable alternative for large language models. We plan to validate BSA's effectiveness on larger model architectures in future work.

Reference:

[1] Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint arXiv:2302.13971, 2023.

[2] Dai D, Deng C, Zhao C, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models[J]. arXiv preprint arXiv:2401.06066, 2024.

[3] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.

[4] Qin H, Ding Y, Zhang M, et al. Bibert: Accurate fully binarized bert[J]. arXiv preprint arXiv:2203.06390, 2022.