Toward Relative Positional Encoding in Spiking Transformers

Thanks for raising comments and suggestions regarding many important issues of our work. We address your concerns with the following responses.

1. Limited technical novelty: The core techniques (Gray Code, logarithmic encoding) are well-established. The main contribution is adaptation rather than fundamental innovation. (W1)

While it is true that both Gray Code and logarithmic encoding are long-standing concepts, our contribution is not a direct reuse, but a principled rethinking and repurposing of these techniques under the fundamentally different constraints posed by SNNs.

Our work tackles a largely unexplored problem: how to design relative positional encoding (RPE) that is compatible with the binary, event-driven nature of SNNs and implementable on neuromorphic hardware. To the best of our knowledge, this is the first systematic attempt to bring RPE into Spiking Transformers in a theoretically grounded and hardware-compliant manner.

Our contributions lie in both theoretical and architectural innovation:

1. Theoretical innovation: We prove (Theorem 1) that Gray Code ensures a consistent Hamming distance between relative positions differing by $2^n$, which is critical in preserving relative distance information in spiking binary attention.
2. Methodological innovation: We design two spike-compatible, non-floating-point RPE schemes that are implementable under neuromorphic principles and show superior empirical performance across multiple SNN backbones and tasks. Prior RPEs like RoPE/ALiBi are fundamentally incompatible with binary spiking attention due to their reliance on continuous or additive positional biasing.

More importantly, our work serves a pioneering role in a very nascent area: Spiking Transformers are still in their infancy, with most work in the past year focusing on building basic viability. There remains a clear gap in adapting core mechanisms from ANN Transformers—like positional encoding—to this new paradigm. The fact that seemingly "established" concepts need to be carefully reengineered for this setting speaks to the novelty and significance of our contributions.

2. This method undermines the linear attention property inherent in SSA. (W2)

We agree that preserving the low-complexity nature of SNNs is crucial for future large-scale applications. We would like to clarify and respond to this concern:

1. SSA in Spiking Transformers Is Not Strictly Linear in Practice. While the original SSA mechanism in Spikformer and SDT adopts a formulation that can theoretically be reduced to O(N) under the assumption N ≫ d, this condition is not met in most practical SNN applications, especially in the datasets and architectures we evaluate. In time-series forecasting, the sequence length N is typically 12 or 168, much smaller than the embedding dimension d = 256. In text classification, even for long sequences such as AGNews (N=1024), the dimensionality d = 768 is comparable, and the assumption N ≫ d does not hold. In vision tasks, where Spikformer operates on image patches, N is often small (e.g., 49 for 7×7 patch grids), again violating the N ≫ d condition. Furthermore, we note that even in the original Spikformer implementation, the authors compute the attention using the standard form $QK^TV$ rather than the factored $K^TV$ variant, acknowledging this practical constraint.
2. XNOR-based SSA Preserves Efficiency and Hardware-Friendliness. Our proposed XNOR-based SSA is computationally lightweight and hardware-efficient. Bitwise XNOR and summation operations over binary matrices are inherently more efficient than floating-point matrix multiplications on neuromorphic hardware or digital circuits. Importantly, our modification does not increase the asymptotic time complexity and can be efficiently parallelized at the bit level. In this sense, while the attention mechanism deviates from the "linear attention" formulation, it remains computationally tractable and well-suited to energy-efficient SNN deployment.

In summary, we acknowledge the value of linear attention in scaling models to longer sequences, but in the current spiking architectures and tasks, the practical implications are minor. Our proposed mechanism maintains the core benefits of spiking computation—namely sparsity, bit-level efficiency, and compatibility with neuromorphic hardware—while significantly improving sequence modeling through principled relative positional encoding.

3. Experimental limitations: The dataset selection in this study deviates from mainstream choices, as widely used benchmark datasets such as GLUE in the NLP domain and ImageNet in the computer vision field were not adopted in this paper. (W3)

We appreciate the suggestion regarding broader dataset usage. However, to fairly validate the cross-domain generalizability of our RPE methods in SNNs, we selected datasets following previous work on sequential tasks in SNNs. Specifically,

1. We follow [1][2][3] to conduct time-series forecasting experiments (e.g., Metr-la, Pems-bay, Solar).
2. We follow [2][4] to conduct text classification, including six diverse datasets (English/Chinese, short/long), offering rich linguistic diversity;
3. We follow [2][5] to conduct image classification, including not only standard datasets (CIFAR, Tiny-ImageNet) but also a neuromorphic dataset (CIFAR10-DVS), which is specifically relevant to the SNN community.

While GLUE and ImageNet are standard in ANN research, they are computationally intensive and less suitable for initial SNN benchmarking due to current hardware and convergence limitations. We highlight that our methodology is orthogonal to specific datasets and can be easily applied to GLUE/ImageNet in future work.

[1] Lv C, Wang Y, Han D, et al. Efficient and Effective Time-Series Forecasting with Spiking Neural Networks[C]//International Conference on Machine Learning. PMLR, 2024: 33624-33637.

[2] Lv C, Han D, Wang Y, et al. Advancing spiking neural networks for sequential modeling with central pattern generators[J]. Advances in Neural Information Processing Systems, 2024, 37: 26915-26940.

[3] Fang Y, Ren K, Shan C, et al. Learning decomposed spatial relations for multi-variate time-series modeling[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(6): 7530-7538.

[4] Lv C, Xu J, Zheng X. Spiking Convolutional Neural Networks for Text Classification[C]//The Eleventh International Conference on Learning Representations, 2023

[5] Li Y, Geller T, Kim Y, et al. Seenn: Towards temporal spiking early exit neural networks[J]. Advances in Neural Information Processing Systems, 2023, 36: 63327-63342.

4. Is directly concatenating Gray codes into Q and K for Hamming distance computation equivalent to first computing the Hamming distance between Q and K separately, then adding a fixed distance matrix? (Q1)

The short answer is: Yes, the two are equivalent under the assumption that both Q and K are binary vectors, and the Gray codes are deterministically assigned based on position.

Mathematically:

1. Concatenation form: $\text{AttnMap}\_{i,j} = \sum\_d \neg([Q\_i | G(i)] \oplus [K\_j | G(j)])$
2. Additive form: $\text{AttnMap}\_{i,j} = \sum\_d \neg(Q\_i \oplus K\_j) + R\_{i,j}$

In the binary domain, these two are mathematically equivalent since $\text{XNOR}(A | B, C | D) = \text{XNOR}(A, C) + \text{XNOR}(B, D)$ in terms of Hamming matching. Under this setting, $R_{i,j} = \sum_d \neg(G(i) \oplus G(j))$.

We chose the concatenation-based formulation for the following reasons:

1. Clarity of Presentation: Our primary motivation is to make the functionality of Gray codes as a form of relative positional encoding more transparent to readers. By embedding $G(i)$ and $G(j)$ directly into Q and K, it becomes more intuitive to see how the Hamming distance between Gray codes captures relative position. Nevertheless, we acknowledge that this could be misinterpreted as being fundamentally different from additive RPE, and we will add a clarification and mathematical equivalence discussion in the Method section of the revised version.
2. Memory Efficiency: Concatenating $G(i)$ and $G(j)$ directly avoids the need to explicitly store the full pairwise distance matrix $R$, whose size grows quadratically with the sequence length. This is particularly important for neuromorphic hardware, where memory is limited and efficient use of bitwise operations is critical.
3. No Additional Computation Overhead: Since XNOR is a bitwise operation, our approach introduces no extra computational cost. The Gray code bits are simply treated as additional spike dimensions during the existing Q-K matching. In contrast, adding a precomputed $R_{i,j}$ would require separate accumulation logic and increase implementation complexity in a hardware-constrained setting.

5. This approach is suboptimal because Gray codes can only distinguish whether two positions differ by 1. (Q2)

We believe this is a misunderstanding. Gray codes can distinguish whether two positions differ by $2^n$, and we offer a detailed proof in Appendix A.

While it is true that adjacent Gray Codes differ by 1 bit, our method does not rely solely on this property. Our core idea is that relative distances differing by $2^n$ (See Theorem 1 and Appendix A) yield consistent and distinct Hamming distances, which enables structured representation of relative distances.

Moreover, in short-to medium-length sequences, the discriminative capacity of Gray-PE is sufficient, as shown in our experiments on Metr-la, SST-2, CIFAR, etc. For long sequences, we explicitly recognize this limitation and propose Log-PE as a complementary method, which handles large positional spans via a decaying distance-sensitive bias.

We hope the responses above can fully address your concerns. Please kindly let us know if there are any remaining concerns. We will be glad to have further clarifications. Thanks again for your time and effort on the review process!

We are cheerful that our contribution is well recognized. And thanks for your valuable suggestions that truly enhance the quality of our paper and make it more understandable for the wider community.

Responses to your concerns are presented as follows:

1. Why is it challenging to incorporate relative positional encoding? (W1)

We thank the reviewer for raising this important question. While absolute positional encoding (APE) has been explored in previous Spiking Transformer designs, incorporating relative positional encoding (RPE) remains a significant challenge—not only technically, but also fundamentally, given the nascency of this research area.

First, Spiking Transformers are a newly emerging architecture, with most architectural formulations only appearing in the last one to two years. The exploration of how to adapt techniques from ANNs—especially position encoding schemes—to the spike-based domain is still in its infancy. Identifying RPE mechanisms that are compatible with spike-based representations, preserve temporal dynamics, and remain hardware-friendly requires extensive evaluation, filtering, and experimentation. Our work serves not only to provide one such solution, but also to illuminate a viable design space for future studies in this direction.

From a technical perspective, RPE is particularly challenging in SNNs due to the following:n Constraint**: Unlike standard Transformers, both Q and K in spiking Transformers are represented as binary spike matrices. This means that many conventional RPE methods—such as RoPE (which relies on sinusoidal transformations in continuous space) and ALiBi (which introduces real-valued additive bias terms)—cannot be directly applied without violating the discrete, binary nature of spike-based computation.

• Sparsity and Temporal Dynamics: Spiking models are temporally sparse and compute in an event-driven manner. Thus, injecting continuous-valued RPE biases into spike-based attention maps requires additional processing (e.g., floating-point operations), which undermines the neuromorphic compatibility and energy efficiency of SNNs.
• Hardware Considerations: Neuromorphic chips (e.g., Intel Loihi, IBM TrueNorth, and the Speck chip) are optimized for bitwise logical operations and typically lack hardware support for continuous arithmetic. Therefore, any RPE design must preserve the bitwise, energy-efficient computation pattern, which rules out most RPE mechanisms in conventional Transformers.

These factors collectively explain why designing spike-compatible RPE is non-trivial and why existing literature has so far focused primarily on absolute encodings. We will update the manuscript to further clarify our motivations. Thanks for the comment!

2. The paper lacks sufficient analysis regarding the computational speed of the newly proposed XNOR-Based SSA operator compared to the traditional SSA operator. (W2)

This is a good point! Although traditional SSA benefits from highly optimized matrix multiplication (GEMM) on GPUs, we would like to clarify that our XNOR-based SSA retains computational efficiency for the following reasons:

• Bitwise Operations: The core of XNOR-based SSA relies on XNOR and bit-count operations, which are natively supported by digital hardware and neuromorphic processors. These are much cheaper than floating-point multiplications and additions in terms of energy and hardware complexity.
• Hardware Compatibility: Many neuromorphic accelerators (e.g., Loihi, TrueNorth) natively support spike-based bitwise logic, making our XNOR mechanism better aligned with the target deployment platform than conventional floating-point matrix products.
• Parallelizability: While matrix multiplication benefits from BLAS acceleration, XNOR and summation over dimensions are also highly parallelizable, and can be efficiently implemented using tensor intrinsics (e.g., bitwise_xnor, popcount, reduce_sum).
• Empirical Evidence: We evaluate computational speed and GPU memory of SNNs on Time-series forecasting (Electricity, horizon=24, equal batch size and input length) as follows:

3. Regarding the Hamming distance calculation formula between two integers provided in Theorem 1 of Section 4.3, I don’t quite understand why the distance between $d_H(0,1)$ and $d_H(1,2)$ is 1. If they are converted into binary spike sequences, shouldn’t the Hamming distance of $d_H(1,2)$ be 2? (Q1)

Thank you for pointing this out. The confusion stems from a missing clarification in our notation. The statement in the main paper should be:

“As illustrated in Figure 2(b), the Hamming distance $d_H(G(1), G(2)) = 1$...”

That is, we are referring to the Hamming distance between the Gray code representations of position 1 and 2, not their raw binary or integer distance. The key property of Gray code is that values with a difference of 1 have a single-bit difference, i.e., Hamming distance = 1, which is leveraged to encode relative positions in a hardware-friendly way.

We will correct this notation and clarify the explanation in the final version to avoid ambiguity.

4. Is it possible that the addition of Log-PE to the attention map and Gray-PE to the QK terms could compromise the low-power characteristic of SNNs, which relies on purely additive operations? (Q2)

We designed both Gray-PE and Log-PE with energy-efficiency in mind, and do not compromise the low-power characteristics of spiking neural networks:

• Gray-PE simply concatenates precomputed binary Gray code vectors to Q and K. The resulting attention map is still computed via XNOR and bit-count operations, which are lightweight and consistent with spike-based processing.
• Log-PE introduces a precomputed integer bias map added to the integer-valued attention map. The addition is carried out once and involves no floating-point operations, which maintains low precision and energy efficiency.

Overall, our methods respect the core design principles of SNNs: binary spike operations, minimal use of arithmetic, and hardware alignment, ensuring that the proposed RPE methods remain highly suitable for low-power neuromorphic applications.

We appreciate again your deliberated review and the valuable feedbacks! In case there is any remaining concerns, we will be more than happy to make further discussion.

Thanks for your thoughtful comments and suggestions, which are valuable for enhancing our paper. We appreciate the positive assessment of our work. The following are responses to individual concerns:

1. Offering a clearer rationale for the design choices would significantly strengthen the paper. What specific advantages does XNOR offer over dot-product in spiking self-attention? (W1 & Q1)

We thank the reviewer for pointing out the need to clarify the rationale behind our architectural choices. Our overall design philosophy is to develop a coherent and spike-compatible framework that jointly considers encoding and attention mechanisms in the binary domain. Below, we outline the interconnected motivations for each design choice:

• Gray-PE: We adopt Gray code as a binary-friendly positional encoding scheme due to its uniform Hamming distance behavior (Theorem 1). Specifically, positions differing by powers of two exhibit stable Hamming distances between their Gray code representations, making it well-suited to approximate relative distances in a spike-efficient manner.
• Log-PE: Inspired by the ALiBi-style design, we initially introduced a full relative position matrix into attention. However, this naïve integration led to severe numerical imbalance in the spike domain, as shown in Table S1 (Appendix B). We therefore apply a logarithmic transformation to compress the dynamic range while still preserving coarse-grained distance patterns—effectively balancing biological plausibility and attention stability.
• XNOR-Based Attention: In standard self-attention, similarity between Q and K is computed using the dot-product. However, in spiking Transformers, Q and K are binary matrices (0/1), and dot-product loses its semantic interpretability—particularly in the context of relative positions. Instead, we leverage Hamming distance, which provides a natural dissimilarity measure between two binary vectors. Since $d_H(Q, K) = \sum (Q \oplus K), \quad \text{then} \quad \text{similarity}(Q, K) = \sum \neg(Q \oplus K),$ the XNOR operation becomes a natural surrogate for dot-product in the binary spike domain. Crucially, the effectiveness of Gray-PE depends on using XNOR/XOR as the similarity operator—Theorem 1 only holds under this operation. Moreover, XNOR produces a larger similarity scale compared to dot-product, which helps counteract the numerical dominance of relative position encodings. We also note that XNOR-based attention is gaining traction in recent spiking Transformer architectures [1, 2], further validating our design direction.

Overall, these components are not arbitrarily chosen, but rather co-designed to maintain consistency within the binary spike computation paradigm.

[1] Xiao Y, Wang S, Zhang D, et al. Rethinking Spiking Self-Attention Mechanism: Implementing a-XNOR Similarity Calculation in Spiking Transformers[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 5444-5454.

[2] Zou S, Li Q, Ji W, et al. SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and O(T) Complexity[C]//Forty-second International Conference on Machine Learning, 2025

2. What is the rationale for applying a logarithmic transformation to the relative distance in Log-PE, rather than using the raw offset directly? (Q2)

As mentioned in our response to W1 and Line 201, Section 4.4, we initially experimented with direct relative offset terms (e.g., $R_{i,j} = \frac{L-1}{|i-j|+1}$) inspired by ALiBi. However, such absolute values overpowered the attention map, leading to collapsed spike activations and degraded performance (see Table S1 in Appendix B). We chose a logarithmic transformation to compress the dynamic range of relative positions, enabling the model to:

• Encode near positions with higher resolution;
• Avoid numeric overflow in the spike-based attention;
• Preserve spike-compatibility through integer-based additive bias.

To intuitively illustrate this difference, we provide two examples: the matrix resulting from the XNOR operation on the original $Q$ and $K$ exhibits overall values around 230 (without scaling). Given the sequence length of 168, the maximum value of $R_{i,j} = \frac{L-1}{|i-j|+1}$ can reach as high as 167, which significantly disrupts the original attention mechanism. In contrast, the maximum value of $R_{i,j}$ in our Log-PE formulation is only 8, thus minimally interfering with the original attention. This compromise ensures the stability of Log-PE while still preserving effective relative position modeling.

3. The paper claims that Log-PE addresses issues associated with ALiBi, but experimental comparisons were limited to RoPE. Could the authors include ALiBi as a baseline to support this claim with experimental evidence? (Q3)

Thanks for your suggestion! We follow your advice to conduct the following results:

Numbers marked with $^*$ indicate non-convergent results. Transformers equipped with ALiBi exhibit performance comparable to those using Sin-PE and RoPE. However, directly injecting ALiBi-style bias into spiking self-attention disrupts the attention map, often resulting in training instability and convergence failure—particularly on datasets with long input sequences, such as Solar and Electricity ($L=168$).

These results will be included in the updated version of the paper and appendix.

4. Is the logarithmic operation used in Log-PE friendly to hardware implementation? (Q4)

First, we would like to clarify that it is not necessary to perform logarithmic operations directly on hardware during inference. Specifically, the relative position bias is defined as $\mathbf{R}\_{i,j} = \begin{bmatrix} \lceil \log\_{2} \left( \frac{L-1}{|i - j| + 1} \right) \rceil \end{bmatrix}$, where $\lceil \cdot \rceil$ denotes the ceiling (round-up) function, and $L$ is the maximum sequence length. Since this bias depends only on the relative positions and the predefined sequence length, the entire bias matrix can be computed offline and stored ahead of time, eliminating the need for any runtime computation.

Secondly, even if one wishes to compute the logarithmic transformation on hardware, this can be efficiently achieved using a lookup table (LUT) implementation.

• Given an unsigned integer input of $N$ bits, we partition the input range into $K$ intervals.
• Each interval is approximated using a piecewise linear function $y = ax + b$, with the parameters $(a, b)$ stored in the LUT.
• The total LUT storage cost is: $K \cdot (N + 2P) \ \text{bits} \approx \frac{K \cdot (N + 2P)}{8} \ \text{bytes},$ where $P$ is the bit width of the parameters.

This strategy is similar to existing SNN approximations for exponential/leaky functions and has been successfully deployed in many types of neuromorphic chips, such as Intel Loihi [1]. Hence, the hardware implementation of Log-PE is efficient, low-cost, and practically feasible.

[1] Davies M, Wild A, Orchard G, et al. Advancing neuromorphic computing with loihi: A survey of results and outlook[J]. Proceedings of the IEEE, 2021, 109(5): 911-934.

Thanks again for these valuable comments from your detailed review! We will be more than happy to address any further feedbacks.

Thank you for your thoughtful comments and suggestions, which are valuable for enhancing our paper. We are pleased that our contributions are well recognized.

1. Computational Overhead of Gray-PE (Q1)

1. What is the magnitude of channel dimension expansion introduced by concatenating Gray codes?

Thanks for the question. The number of additional channels introduced by Gray-PE depends on the maximum sequence length. Specifically, to uniquely encode positions up to length $L$, we require $\lceil \log_2 L \rceil$ bits. For example, for $L = 1024$, only 10 additional channels are needed. We empirically analyze this relationship in Section 5.6, Figure 3(b), showing that a small number of bits (typically 5–10) is sufficient to preserve relative distance information for common SNN sequence lengths.

2. Has the authors conducted empirical measurements to quantify the resultant increase in computational complexity (e.g., FLOPs, memory footprint, or latency)?

In response to your suggestion, we have conducted empirical measurements on the Electricity time-series forecasting task (prediction horizon = 24), using equal batch size and input length across all models. The results are summarized below:

These results show that XNOR-based SSA and Gray-PE introduce negligible additional overhead in both time and memory. The memory footprint increase is primarily due to the additional Gray code channels (≤10), and the runtime increase is <2.3% per epoch.

2. Boundary Effects in Log-PE (Q2)

1. Could Log-PE values overwhelm the original attention map magnitudes?

This is indeed an important point. The answer is No. As mentioned in Line 201, Section 4.4, we deliberately adopt a logarithmic (rather than linear or inverse) transformation to avoid overwhelming the spike-based attention map. In fact, we attempted a "complete" relative position encoding (e.g., $R_{i,j} = \frac{L-1}{|i-j|+1}$), but found it led to severe degradation in model performance due to value collapse in the attention map. We report this result in Table S1 in Appendix B, confirming that Log-PE is a balanced approximation that preserves performance and numerical stability.

2. Has the distribution of the original attention map been changed significantly after using Log-PE?

We designed Log-PE to mimic the ALiBi bias pattern that largest values on the diagonal (i.e., close positions), gradually decreasing toward the edges (i.e., distant positions). As such, it preserves the relative structure of the attention map, and empirical results across multiple tasks (see Tables 1–4) show no degradation in performance or convergence, further confirming the compatibility of Log-PE with the spiking attention mechanism.

3. Table 1 is hard to understand (Q3)

Thanks for your feedback about the potential clarity issue regarding Table 1. We would like to explain Table 1 as follows:

1. Table 1 shows the experimental results of various SNN models in time-series forecasting.
2. Special Notations: PE denotes positional encoding. "R" denotes relative PE, while "A" denotes absolute PE. "w/" denotes "with", while "w/o" denotes "without". $\uparrow$ denotes that the larger the value, the better the metric.
3. The $L$ in Metr-la ($L=12$) indicates that the input length of Metr-la dataset is $L$. The number $6, 24, 48, 96$ is the prediction length.
4. Sin-PE denotes the original sin/cos-based absolute positional encoding [1] in Transformer. RoPE denotes rotary positional encoding [2], which is a classic relative PE. CPG-PE denotes the spiking absolute PE proposed by [3]. Spikformer, SDT-V1, and QKFormer are all variants of spiking Transformers.
5. Through Table 1, we want to show several conclusions: (1) Results of "Transformer with RoPE" and "Spikformer with RoPE" show that directly applying RPE methods to spiking Transformers is ineffective. (2) Results of "Spikformer with Conv-PE" and Spikformer-XNOR with Conv-PE show that the XNOR modification does not impact the performance of the original SNN models. (3) Obviously, Gray-PE and Log-PE, enable spiking Transformers to achieve the best performance among their variants. (4) For long input sequences (Solar and Electricity), Log-PE is more effective than Gray-PE in capturing relative positional information.

We will also update the main text to discuss Table 1 more clearly. Thanks again for these valuable comments, and please kindly let us know if you have any further suggestions.

[1] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.

[2] Su J, Ahmed M, Lu Y, et al. Roformer: Enhanced transformer with rotary position embedding[J]. Neurocomputing, 2024, 568: 127063.

[3] Lv C, Han D, Wang Y, et al. Advancing spiking neural networks for sequential modeling with central pattern generators[J]. Advances in Neural Information Processing Systems, 2024, 37: 26915-26940.

attn = q @ k.transpose(-2,-1) # q and k are spike matrices of 0s and 1s

attn = torch.sum(1 - (q-k) ** 2, dim=-1)

D_new = q.size(-1)
sum_q = q.sum(dim=-1)
sum_k = k.sum(dim=-1)
qk_matmul = torch.matmul(q, k.transpose(-1, -2))
sum_q = sum_q.unsqueeze(-1)
sum_k = sum_k.unsqueeze(-2)
attn = (D_new - sum_q - sum_k + 2 * qk_matmul)