Spiking Vision Transformer with Saccadic Attention

Question 1:

Thank you for your suggestions. We have compiled the resource requirements and training durations for the SNN-ViT across various tasks, detailed as follows:

• For CIFAR-10 and DVS-CIFAR10, training was conducted using a single NVIDIA RTX 4090 GPU, with each task requiring approximately 6-8 hours.
• For CIFAR-100, we utilized a four-GPU setup with NVIDIA RTX 4090s, also completing in about 6-8 hours.
• For ImageNet1K, the training was performed using four NVIDIA A100 GPUs , taking approximately 5-7 days.

Question 2: The details of training and inference for saccadic neurons.

We apologize if Eq.8 causes any confusion in reading this manuscript. Here, we will separately introduce the training and inference processes for the saccadic neurons. A: We apologize for any confusion caused by our wording. The bolded letters $**H**$ and $**V**\_{th}$ represent the membrane potential and threshold that include temporal dimensions, whereas the non-bold letters denote $H[t]$ and $V_{th}[t]$ at a specific timestep. Additionally, to more clearly articulate the training and inference processes of our saccadic neurons, we provide separate explanations for each:

training process: The dynamic of saccadic neurons can be described as follows:

$\mathcal{P}atch \in \mathbb{R}^{T \times N}$ represents the spatial importance of different regions, $**M**\_w$ is a lower triangular matrix, which can represent **M**\_w = \left \begin{matrix} w_{11}&\cdots&0\\ \vdots&\ddots&\vdots\\ w_{n1}&\cdots&w_{nn}\\ \end{matrix} \right$$. $**S**$ are spike trains, generated by performing an element-wise comparison between the $**H**$ and $**V**_{th}$.

Inference Process:

Lemma: A lower triangular matrix is invertible if and only if all its diagonal elements are nonzero.

The diagonal elements of $**M**\_w$ represent the proportional weights of $\mathcal{P}atch$ at the current time step, and we ensure they are nonzero during training. Therefore, $**det**\left( **M**\_{w} \right) \neq 0$, then $**M**\_w$​ is Invertible.

where $**V**'\_{th}=**M**_{w}^{-1} **V**\_{th}$ represents the threshold $**V**\_{th}$ at different timesteps. Through this approach, we implement a sequential inference process and ensure the equivalence of the training and inference processes.

Question 3: Which method performs better? SSSA-V1 or SSSA-V2 .

Our subsequent experiments are conducted on SSSA-V2. In the training process, SSSA-V2 and V1 exhibit a linear scaling relationship with virtually no performance loss. The distinction between them emerges during the inference process, where V2 integrates $(\mathcal{K}^T \times L)$ into the threshold $V_{th}$ of the saccadic neurons as a constant value, whereas in V1, it is data-driven. We have also conducted experiments that show minimal performance disparity between the two versions and have successfully reduced the computational complexity from $\mathcal{O}\left(N^2\right)$to $\mathcal{O}\left(D\right)$.

Thank you for your insightful feedback. We address each of your points in turn.

Weakness 1: Expression details.

Thank you for your suggestions. We clarify Equation 1 as follows: \text{Dot-Product}$ \mathcal{Q}_i, \mathcal{K}_i $ = \sum\_{j=1}^{D} \mathcal{Q}\_{ij} \mathcal{K}\_{ij} $D$ is the dimension of both vectors, $\mathcal{Q}\_{ij}$ and $\mathcal{K}\_{ij}$ refer to the $j$-th elements of these vectors, respectively. Additionally, the bolded letters $**H**$ and $**V**\_{th}$ represent the membrane potential and threshold that include temporal dimensions, whereas the non-bold letters denote $H[t]$ and $V\_{th}[t]$ at a specific timestep.

Weakness 2: Further energy consumption analysis.

Thank you for your feedback. Compared to SSA, our SSSA exhibits lower computational complexity and consequently fewer SOP operations. The Table below presents a comparative analysis of energy consumption across different SSA configurations under the conditions of Head=8, D=512, N=196, and T=4. Notably, our SSSA demonstrates reduced SOPs and lower computational energy consumption. However, the overall energy consumption of the SNN-VIT varies due to differences in network performance and architecture, rendering the comparative results less pronounced.

Weekness 3: Experimental Validation.

Thank you for your suggestions. While we conducted multiple experiments, we inadvertently did not calculate the error bars. As shown in the table below, all experiments are conducted at least five times to ensure reliability.

Weakness & Questions 2: introduction of biological saccadic mechanisms.

A: Thank you for your suggestions. We add an explanation of biological saccadic mechanisms from three perspectives in the revised manuscript. (Appendix.C）

1.Details of biological saccadic mechanisms: Numerous neuroscience findings [1-3] confirm that the eyes do not acquire all the details of a scene simultaneously. Instead, attention is focused on specific regions of interest (ROIs) through a series of rapid saccadic called saccades. Each saccade lasts for a very brief period—typically only tens of milliseconds—allowing the retina's high-resolution area to align with different visual targets sequentially. This dynamic saccadic mechanism enables the visual system to process information efficiently by avoiding redundant processing of the entire visual scene.

2.Other similar works inspired by visual mechanisms: Zhao, Jing, et al.[4] introduce a model utilizing a retina-inspired spiking camera to enhance image clarity in high-speed motion scenarios. McIntosh, Lane, et al.[5] explore how deep convolutional neural networks can model the retina's response to natural scenes. Tanaka, Hidenori, et al.[6] discuss the use of deep learning models to understand the computational mechanisms of the retina. These advanced features of biological vision effectively inform the rational design of deep neural networks, promoting the efficient integration of biological and machine intelligence.

3.Specific examples of saccadic interaction:

For the saccadic mechanism, two key processes are briefly included: (1). It focuses on a small part of the scene at each moment while ignoring other information. (2). The eye achieves a global understanding of the entire visual scene by continuously moving its focus at different moments, combined with a history of previous focal points.

In our saccadic neurons, the salient patch selection process primarily aims to facilitate the first process by assessing the overall relevance of different patches to each other. It can be described as:

$\mathcal{P}atch = \sum_{j=1}^{n} \mathrm{CroAtt}\left(Q, K\right), **       **  \mathrm{CroAtt}\left(Q, K\right) \in \mathbb{R}^{T \times N \times N},$

 Higher-scoring patches indicate more critical spatial positions while lower-scoring ones are overlooked. This scoring provides the basis for choosing the patch at each moment.

Subsequently, saccadic neurons decide which visual regions to select based on the current patch score and historical attention, achieving the second process.

Notably, our saccadic neurons are specifically designed to integrate historical information into different thresholds at each moment, ensuring performance under full spike-driven conditions.

Reference:

[1] Melcher, David, and M. Concetta Morrone. "Spatiotopic temporal integration of visual motion across saccadic eye movements." Nature Neuroscience 6.8 (2003): 877-881.

[2] Binda, Paola, and Maria Concetta Morrone. "Vision during saccadic eye movements." Annual review of vision science 4.1 (2018): 193-213.

[3] Guadron, Leslie, A. John van Opstal, and Jeroen Goossens. "Speed-accuracy tradeoffs influence the main sequence of saccadic eye movements." Scientific Reports 12.1 (2022): 5262.

[4] Zhao, J., Xiong, R., Xie, J., Shi, B., Yu, Z., Gao, W., & Huang, T. Reconstructing clear image for high-speed motion scene with a retina-inspired spike camera. IEEE Transactions on Computational Imaging, 8, 12-27, (2021).

[5] McIntosh, L., Maheswaranathan, N., Nayebi, A., Ganguli, S., & Baccus, S. Deep learning models of the retinal response to natural scenes. Advances in neural information processing systems, 29, (2016).

[6] Tanaka, H., Nayebi, A., Maheswaranathan, N., McIntosh, L., Baccus, S., & Ganguli, S. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems, 32, (2019).

We greatly appreciate your recognition of the innovative aspects and motivation of our work. We will respond to each of your suggestions in turn.

Weakness & Questions 1: Include the performances of SOTA ANN-based ViTs.

A: Thank you for your valuable suggestions. We will revise the manuscript to include comprehensive performance comparisons with state-of-the-art ANN-based ViTs for Table 1-3. This addition allows for a more rigorous evaluation of our method's effectiveness relative to existing approaches. For example, the details of comparisons for ImageNet-1K are shown as follows：(Table 1-3)

Reference：

[1] Dosovitskiy, Alexey. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

[2] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision (2021).

[3] Han D, Pan X, Han Y, et al. Flatten transformer: Vision transformer using focused linear attention. Proceedings of the IEEE/CVF international conference on computer vision (2023).

Question 1: The explanation of distribution-based similarity correlations and SSSA's spike-driven characteristics.

**A:**As you noted, the computation of distribution-based correlations can be interpreted as evaluating the correlations between the firing rates of vectors in Q and K. This method aligns well with the principles of frequency encoding. Moreover, the SSSA-V1 model faces challenges with the outer product of integer vectors. To address this, we develope SSSA-V2, ensuring spike-driven characteristics. In the SSSA-V2 version, the calculation formulas are as followed:

$(\mathcal{K}^T \times L)$ is a integer scalar value. To avoid this integer multiplication $\mathcal{Q} (\mathcal{K}^T \times L)$，We treat $(\mathcal{K}^T \times L)$ as a learnable scaling factor $\alpha$, which is applied to the $**V**\_{th}$ of the saccadic neuron. Therefore, combining the asynchronous decoupling approach as above, the formula for SSSA-V2 in inference can be expressed as follows:

In this manner, $\frac{1}{\alpha} \left(**M**\_{w}^{-1} **V**\_{th}\right)[t]$ is the $**V**_{th}$ for saccadic neurons at timestep $t$, thus ensuring the fully spike-driven nature of SNN-ViT.

Question 2: Would V1 perform better when floating-point computations are involved?

A: Our subsequent experiments are conducted on SSSA-V2. In the training process, SSSA-V2 and V1 exhibit a linear scaling relationship with virtually no performance loss. The distinction between them emerges during the inference process, where V2 integrates $(\mathcal{K}^T \times L)$ into the threshold $V_{th}$ of the saccadic neurons as a constant value, whereas in V1, it is variable. We also conduct experiments that show minimal performance disparity between the two versions, and SSSA-V2 successfully reduces the computational complexity from $\mathcal{O}\left(N^2\right)$to $\mathcal{O}\left(D\right)$.

Thank you for your insightful feedback. We will address each of your points in turn.

Weakness 1: Unclear Expression.

A: We apologize for any confusion caused by our wording. The bolded letters $**H**$ and $**V**\_{th}$ represent the membrane potential and threshold that include temporal dimensions, whereas the non-bold letters denote $H[t]$ and $V_{th}[t]$ at a specific timestep. Additionally, to more clearly articulate the training and inference processes of our saccadic neurons, we provide separate explanations for each:

training process: The dynamic of saccadic neurons can be described as follows:

$\mathcal{P}atch \in \mathbb{R}^{T \times N}$ represents the spatial importance of different regions, $**M**\_w$ is a lower triangular matrix, which can represent **M**\_w = \left \begin{matrix} w_{11}&\cdots&0\\ \vdots&\ddots&\vdots\\ w_{n1}&\cdots&w_{nn}\\ \end{matrix} \right$$. $**S**$ are spike trains, generated by performing an element-wise comparison between the $**H**$ and $**V**_{th}$.

Inference Process:

Lemma: A lower triangular matrix is invertible if and only if all its diagonal elements are nonzero.

The diagonal elements of $**M**\_w$ represent the proportional weights of $\mathcal{P}atch$ at the current time step, and we ensure they are nonzero during training. Therefore, $**det**\left( **M**\_{w} \right) \neq 0$, then $**M**\_w$​ is Invertible.

where $**V**'\_{th}=**M**_{w}^{-1} **V**\_{th}$ represents the threshold $**V**\_{th}$ at different timesteps. Through this approach, we implement a sequential inference process and ensure the equivalence of the training and inference processes.

Weakness 2: The experiment details of SNN-ViT with YOLO-v3.

A: Feature maps from the backbone network are fed into detection heads through concatenation for multi-scale object detection. This hierarchical feature fusion strategy enables the model to capture objects at different scales and spatial contexts. The experimental framework was implemented using the PyTorch deep learning library. All experiments are conducted on a high-performance computing platform equipped with an NVIDIA RTX4090 GPU. This hardware configuration provided sufficient computational capacity for training our proposed architecture through all experimental phases.

weekness 3: Further explanations for Appendix.

A: (1) Why is the dot product used to compute similarity in attention mechanisms in ANNs?

The dot-product is widely used in ANNs for its simplicity and effectiveness in capturing vector similarity. when vectors are normalized, it becomes cosine similarity, reflecting their angular relationship. Moreover, it is computationally efficient, leveraging matrix multiplication to handle large-scale data, ensuring high performance during training and inference.

(2) Why is cross-entropy not used to compute similarity in attention mechanisms in ANNs?

Cross-entropy measures the difference between probability distributions, but ANN vectors are not probability distributions—they may include negative values or elements that do not sum to one. Converting them using Softmax operation adds computational cost and distorts the original vector properties, reducing accuracy. In contrast, SNNs use binary data (0 or 1), where firing rates directly correspond to probabilities, making conversion straightforward, computationally efficient.

Thank you for your insightful feedbacks. We will address each of your points in turn.

Question 1：Analysis of the distribution of Query and Key is unfair?

A: We apologize for the confusing statements regarding the mismatch between LayerNorm and spike trains. As you pointed out, normalization operations can be integrated with convolutional and linear layers in SNNs. We clarify that the membrane potentials of our $\mathcal{Q}$ and $\mathcal{K}$ spike trains are computed through batch normalization. The formulations for our $\mathcal{Q}$ and $\mathcal{K}$ are as follows:

where $BN$ represents batch normalization and $Conv$ denotes the convolution process. Therefore, the comparison of the magnitudes of Q and K in SNN and ANN is fair. Through comparative analysis as Fig.1, we point that differences in the magnitudes of $\mathcal{Q}$ and $\mathcal{K}$ are attributable to the discrete activation characteristics of $\mathcal{SN}$. In the revised manuscript, we have clarified the processing for Q and K, as well as Fig. 2.

Question 2 ： Why does the author insist on adding an additional temporal modeling component to the self-attention?

A: As you mentioned, SNNs possess neural dynamics capable of processing spatio-temporal information. However, many studies[1-3] propose that designing temporal components (TC) can further enhance the performance ceiling of SNNs. For instance, Yao et al.[1] introduced a temporal attention module into SNNs, allowing the model to adaptively assign importance to different time steps. Shen et al.[2] proposed learnable decay factor for the spiking neurons at each timestep.

Moreover, we conduct performance evaluations of our SSSA on the CIRAF100 dataset both with and without saccadic temporal interactions. As illustrated in the table above, it reveals that our SSSA achieves an approximate 1.2% performance improvement under identical structural conditions.

Reference:

[1] Yao, M., Gao, H., Zhao, G., Wang, D., Lin, Y., Yang, Z., & Li, G. Temporal-wise attention spiking neural networks for event streams classification. ICCV(2021).

[2] Yin, B., Corradi, F., & Bohté, S. M. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10), 905-913, 2021.

[3] Zhu, R. J., Zhang, M., Zhao, Q., Deng, H., Duan, Y., & Deng, L. J. (2024). Tcja-snn: Temporal-channel joint attention for spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems.

Question 3： Reset and decay mechanism in saccadic neurons？

A: As you highlighted, reset and decay mechanisms in spiking neurons are crucial for preventing excessive neuron firing and preserving memory capability. However, numerous studies[1-2] also tried to remove the reset and decay mechanisms during training, achieving significant performance on many long-sequence tasks. To demonstrate that our saccadic neurons also possess sparse and have better performance compared to LIF, we conducted experiments on sequence tasks.

• Better performance: In sequence tasks, models based on saccadic neurons demonstrate significantly improved accuracy and outperform those based on LIF neurons. This indicates that saccadic neurons better preserve long-term dependencies and effectively utilize historical information.
• Sparsity: As shown above, the average firing rates of saccadic neurons are comparable to those of LIF neurons, maintaining network sparsity.

Thus, our saccadic neurons possess both better performance and Sparsity characteristics.

Reference:

[1] Su Q, Mei S, Xing X, et al. SNN-BERT: Training-efficient Spiking Neural Networks for energy-efficient BERT[J]. Neural Networks, 2024, 180: 106630.

[2] Fang, W., Yu, Z., Zhou, Z., Chen, D., Chen, Y., Ma, Z., ... & Tian, Y. (2024). Parallel spiking neurons with high efficiency and ability to learn long-term dependencies. Advances in Neural Information Processing Systems, 36.

We greatly appreciate your recognition of the innovative aspects and motivation of our work. In response to the weaknesses and suggestions you have raised, we will provide further detailed explanations:

Weakness 1: Additional explanation of these heatmaps.

A: Thank you for your suggestions. Our heatmap display is noisy mainly for two reasons. Firstly, the SSSA module, driven by spikes, operates in two states at each timestep: focus or ignore. This binary operation makes its behavior less smooth compared to full-precision ViT networks but ensures highly sparse. Second, because our display shows multi-target remote sensing object detection, the network focuses on all visual areas close to the ten types of targets. Consequently, on the NWPU-10 dataset, the heatmaps also show attention to areas outside the target boxes. Following your advice, we have added visualizations for a single-target detection dataset SSDD. This updated visualization is included in the new manuscript.(Appendix.G)

Weakness 2: Lacking ablation studies on the SSSA.

A: Thank you for your valuable suggestions. In the revised manuscript, we design additional ablation experiments to address your concerns. Specifically, we evaluate the performance of the method that uses all QK values. Additionally, we evaluate the performance of our method on the CIFAR10/100 and CIFAR10DVS datasets to demonstrate its generalizability.

As demonstrated in the table, "QK-based" denotes the method that utilizes all QK values. For various datasets, the "QK-based" method increases the computational complexity to $\mathcal{O}(N^2D)$ but yields only limited performance improvements. Therefore, our SSSA module achieves an optimal trade-off between computational complexity and performance.

Moreover, the "Patch" design yields high performance and lower computation complexity across various datasets, demonstrating its generalizability. (Appenidx.E)

Questions 1: The explanation of Figure 1

A: Thank you for your suggestion. In the revised manuscript, we have added X and Y labels to Figure 1 and included details about the network architecture in the figure caption. The X-axis represents the vector magnitudes, and the Y-axis indicates the number of samples. As per your recommendation, our data are not simulated but collected from training an ANN-based ViT model and a Spikformer on the CIFAR100 dataset for 300 epochs, both configured with $D = 384$ and 12 heads. Additionally, following your advice, we observe abnormal distributions of $\mathcal{Q}$ and $\mathcal{K}$  vector in SNNs across multiple datasets, indicating that different datasets do not affect the statistical results.（Fig.1）

Questions 2: Further clarification for the statement in lines 277–278? Adding a source or further explanation here would help substantiate this claim.  

A: Your suggestions have been immensely helpful in enhancing the quality of our manuscript. Following your advice, we have added Appendix C to detail this part. Numerous neuroscience findings[1-3] confirm that the eyes do not acquire all details of a scene simultaneously. Instead, attention is focused on specific regions of interest (ROIs) through a series of rapid eye movements called saccades. Each saccade lasts for a very brief period—typically only tens of milliseconds—allowing the retina's high-resolution area to sequentially align with different visual targets. This dynamic eye movement mechanism enables the visual system to process information efficiently by avoiding redundant processing of the entire visual scene. (Appenidx.C)

References:

[1] Melcher, David, and M. Concetta Morrone. "Spatiotopic temporal integration of visual motion across saccadic eye movements." Nature neuroscience 6.8 (2003): 877-881.

[2] Binda, Paola, and Maria Concetta Morrone. "Vision during saccadic eye movements." Annual review of vision science 4.1 (2018): 193-213.

[3] Guadron, Leslie, A. John van Opstal, and Jeroen Goossens. "Speed-accuracy tradeoffs influence the main sequence of saccadic eye movements." Scientific reports 12.1 (2022): 5262.

Thank you for your recognition of our work. Regarding the M_w you mentioned, its dimension is not N×N but rather a T×T lower triangular matrix, which is exclusively responsible for temporal information interaction. In Equation 8, both H and Patch have dimensions of T×N; therefore, M_w must be T×T to ensure proper matrix multiplication. We hope this clarification helps you. Thank you again for your acknowledgment of our work, and we wish you success in your endeavors.