SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity

Dear Reviewer 9Qge,

We are grateful for your insightful feedback. Below, we provide a detailed response to each of your points.

Other Strengths And Weaknesses:

W1 Inference Time (per video clip $T\times 256\times 256\times 3$ as input)

• We report the inference time in the table below, tested on an A6000 GPU and AMD EPYC 7543 CPU for human pose tracking, averaged over 1,000 video clips with a batch size of 1. As $T$ increases from 8 to 32 (4x), GLoT (quadratic ANN attention) achieves the best performance but experiences a 9.8× increase in inference time, while VIBE (linear ANN-GRU) shows a 5.1x increase but performs the worst. Both SNN methods exhibit only 4.3× and 4.6× increases thanks to our proposed spiking space-time linear attention.

• For edge device deployment, SNNs, relying only on additions, significantly reduce power consumption and latency on neuron-computing devices [A, B]. ANNs, requiring floating-point multiplications, achieve high parallel acceleration on GPUs but at a high energy cost. On the AMD Xilinx ZCU104 [C], ANNs process at 691.2 GFLOPs/s, while SNNs reach 5529.6 GFLOPs/s—an 8x speedup. In 45nm technology [D], ANN multiplication consumes 4.6pJ, whereas SNN addition uses only 0.9pJ, a 5.1x energy reduction.

[A] Neuromorphic computing at scale. Nature 2025.

[B] Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip. Nature Communications 2024.

[C] Firefly: A high-throughput hardware accelerator for spiking neural networks with efficient dsp and memory optimization. IEEE VLSI 2023.

[D] 1.1 computing’s energy problem (and what we can do about it). IEEE ISSCC 2014.

W2 Event

• We have explored long-term event-based human pose tracking in Table 2.
• We present results on event-based action recognition (HAR-DVS [E]) and event-based semantic segmentation for scene understanding (DDD17 [G]) in the table below.

[E] Hardvs: Revisiting human activity recognition with dynamic vision sensors. AAAI 2022.

[F] Action-net: Multipath excitation for action recognition. CVPR 2021.

[G] EV-SegNet: Semantic segmentation for event-based cameras. CVPR 2019.

W3 Energy

• Please refer to our response to W1 Inference Time. We have included latency (inference time) to evaluate efficiency in long-term tasks using our linear complexity method. Additionally, we have discussed the potential of deploying our SNN on specialized neurocomputing devices to further enhance its efficiency.

W4 Ablation

• We have conducted additional analysis of time steps and hyper-parameters on Human Pose Tracking (evaluated on PA-MPJPE). The results are shown in the tables below.

W5 Clarity

• Our SpikeVideoFormer differs from existing Video Transformers in two aspects:
  • Unlike ANN-based Video Transformers, which focus on reducing quadratic complexity in space-time attention, we highlight that spike-driven attention inherently achieves linear complexity (Table 1), making our approach more efficient and scalable.
  • Unlike existing SNN-based Transformers, which focus on single-image tasks only, we introduce the first Spiking Video Transformer with an effective spike-driven attention mechanism (Proposition 3.1).

Questions For Authors:

• Q1: A video comprises a sequence of RGB images with 3×8 bits/pixel. In contrast, an event stream can be represented as a sequence of event frames with 1 bit/pixel (24x more sparse), indicating event occurrences within the time interval.

• Q2: We emphasize videos due to their practicality in real-world applications, highlighting the need for efficient and effective processing methods. Following prior works, we also incorporate event-based tasks in our experiments to further demonstrate the applicability of SNNs.

We are grateful for your suggestions and will ensure that the above results and discussions are reflected in the final revision.

Dear Reviewer 4RFZ,

We sincerely appreciate your time and effort in reviewing our paper. Please find our point-by-point responses to your comments below.

Claims And Evidence:

• Thanks for the value suggestion. Normally, Power = Watts * Time. According to [B, C], when comparing ANNs and SNNs, we typically assume the hardware operates for the same duration. Therefore, computational efficiency measured in power is equivalent to that measured in watts. As suggested by other reviewers, we have also reported latency (inference time) as an additional evaluation metric for computational efficiency in our response to Reviewer 9Qge (W1 Inference Time).
• ANNs rely on floating-point multiplications, where GPUs can accelerate in parallel but with high energy costs. In contrast, SNNs use binary spikes to propagate, requiring only additions—a special case (0/1 binary matrix) that GPUs can speed up as well. However, power consumption and processing time can be further reduced on specialized neuron-computing devices [A, B], where multiplication operator is replaced with much faster addition operators.

[A] Neuromorphic computing at scale. Nature 2025.

[B] Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip. Nature Communications 2024.

[C] Firefly: A high-throughput hardware accelerator for spiking neural networks with efficient dsp and memory optimization. IEEE VLSI 2023.

Methods And Evaluation Criteria:

• We evaluate our approach on the ImageNet-C dataset to assess object recognition performance in noisy environments. Notably, Extreme Image Transforms [B] lacks ImageNet-C results, so we compare our method with VOneResNet50 [A] and ResNet50 [C].

[D] Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations. NeurIPS 2020.

[E] Extreme Image Transformations Facilitate Robust Latent Object Representations. arXiv 2023.

[F] Deep residual learning for image recognition. CVPR 2016.

Supplementary Material & Other Strengths And Weaknesses:

• We have included ground truths for human pose tracking and the spike attention maps on the anonymous github page.

Essential References Not Discussed:

• Visual cognitive neuroscience studies how factors like attention, motivation, emotion, and expectation shape visual perception and cognition [1]. While the first three enhance relevant stimuli processing, expectation suppresses predictable inputs [2-3]. Predictive processing suggests perception is inference-driven, refining sensory input through internal models shaped by context and experience [4]. Beyond vision, the visual cortex processes object names and activates in blind individuals, indicating broader cognitive roles in memory, imagery, and language [5-7]. Though its full scope remains debated, these insights inspire the brain-inspired spiking neural network (SNN) approach for complex visual tasks, enabling high-speed, low-energy neural processing.

[1] The role of context in object recognition. Trends in Cognitive Sciences 2007.

[2] How brains beware: neural mechanisms of emotional attention. Trends in Cognitive Sciences 2005.

[3] Expectation (and attention) in visual cognition. Trends in Cognitive Sciences 2009.

[4] An integrative, multiscale view on neural theories of consciousness. Neuron 2024.

[5] Object domain and modality in the ventral visual pathway. Trends in Cognitive Sciences 2016.

[6] The human imagination: the cognitive neuroscience of visual mental imagery. Nature Reviews Neuroscience 2019.

[7] Reevaluating the sensory account of visual working memory storage. Trends in Cognitive Sciences 2017.

Questions For Authors:

• The performance gain stems from space-time joint attention. As shown in our ablation study (Table 6), using spatial-only attention results in pose tracking error (PA-MPJPE) increase of more than 36%. Moreover, GLoT is limited by the quadratic complexity of ANN-based attention, requiring image features to be represented as a high-level single vector. In contrast, our SNN's linear complexity allows for more detailed spatial and temporal feature fusion through spike-driven attention.

We are grateful for your thoughtful comments and will carefully integrate all results and discussions into the final version.

Dear Reviewer ZrRF,

We greatly appreciate your time and effort in reviewing our work. Below are our point-by-point responses to your comments.

Experimental Designs Or Analyses:

• Thanks for the constructive comment. For a qualitative comparison of video semantic segmentation, please refer to Supp Figure 3 on the anonymous GitHub page. Additionally, video results are provided in Supp Figures 5 and 6. The source code, results, and project website will be publicly released.

Relation To Broader Scientific Literature & Essential References Not Discussed:

• We have cited this paper [A] (Line 477) in our work and also applied in the experiment of Video Semantic Segmentation (Line 386). Integer-valued spike representation proposed in the suggested paper represents spikes as a set of integers, e.g. {0,1,2,3} (Integer-LIF=4), rather than {0,1} only. During inference the integer spike can be separated as a sum of {0,1} spikes, e.g., 3= 1+1+1, 2=1+1+0. This separation maintains spike-driven efficiency of SNN methods, while improving the model's representation ability.
• We further compare our method on the object detection task against this work and Meta-SpikeFormer, as shown in the table below. Following prior work, we use spiking-based YOLO as the detection head and evaluate on the COCO 2017 dataset. Our method surpasses Meta-SpikeFormer by 0.6 mAP but lags behind SpikeYOLO by 0.2 mAP. However, unlike SpikeYOLO, which is specifically designed for object detection, our method is more general and applicable to a wide range of downstream tasks.

[A] Integer-valued Training and Spike-Driven Inference Spiking Neural Network for High-Performance and Energy-Efficient Object Detection. ECCV 2024.

Other Strengths And Weaknesses:

• Thanks for the valuable suggestion. Please refer to the response above under Relation to Broader Scientific Literature for the results related to the object detection task.

• We present recent advancements in ANN-based Transformers for semantic segmentation as follows.

  • Recent advancements in ANN-based Transformers have greatly enhanced semantic segmentation. SETR [1] pioneers a sequence-to-sequence approach, employing a pure Transformer encoder without convolutional layers. Segmenter [2] builds on a ViT backbone pre-trained on ImageNet and incorporates a mask transformer decoder to capture global context. SegFormer [3] further optimizes Transformer-based segmentation with a hierarchical encoder and a lightweight MLP decoder while eliminating positional encoding. Mask2Former [4] refines segmentation by restricting cross-attention to foreground regions using a masked attention operator. Recent CFFM [5] introduces coarse-to-fine feature assembling and cross-frame feature mining to capture both local and global temporal contexts. In contrast, we explore spiking video transformers to develop a faster, more energy-efficient approach for this task.

[1] Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. CVPR 2021.

[2] Segmenter: Transformer for semantic segmentation. ICCV 2021.

[3] Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. ICCV 2021.

[4] Masked-attention mask transformer for universal image segmentation. CVPR 2022.

[5] Learning local and global temporal contexts for video semantic segmentation. IEEE TPAMI 2024.

We sincerely appreciate your feedback and will ensure that all results and discussions are thoroughly reflected in the final version.

Dear Reviewer oaZD,

We appreciate your time and effort in reviewing our paper. Below, we provide a point-by-point response to your questions.

Claims And Evidence:

• We report the latency in the table below, based on tests conducted using the same hardware setup—a single A6000 GPU and an AMD EPYC 7543 CPU. The task is human pose tracking, comparing the performance of GLoT (ANNs) and our SpikeVideoFormer (SNNs). The input is an RGB video clip of shape $T\times 256\times 256\times 3$, with a batch size of 1. We conducted tests using 1,000 samples and averaged the time consumption as the latency. As the temporal length $T$ increases from 8 to 32 (4x), GLoT (quadratic attention) experiences a 9.8x latency increase, whereas SpikeVideoFormer (linear attention) shows only a 4.6x increase. GLoT’s increase is lower than the expected ~16× due to its use of a ResNet followed by a Transformer, where ResNet has linear complexity concerning temporal length.

Methods And Evaluation Criteria:

• Q1: Yes, our proposed SDHA can also benefit spike-driven transformers in the image domain. As demonstrated in the table below (also in Appendix F, Table 8), applying SDHA leads to an accuracy improvement of 0.4% (15.1M model) and 0.2% (55.4M model) on ImageNet. In this experiment, our method follows the same model architecture as Meta-SpikeFormer, except for the inclusion of SDHA.

• Q2: We respectfully believe that our proposed method offers valuable contributions and introduces new insights compared to previous ANN-based works, specifically:
  • Space-time joint attention in ANNs suffers from quadratic computational complexity, and most related methods propose different space-time attention designs to reduce this complexity (as shown in Table 1). However, to our knowledge, we are the first attempt to show that space-time spiking attention designs share the same linear complexity. The experiments demonstrate that our spike-driven solution achieves performance comparable to recent ANN-based methods while offering significant efficiency gains, with improvements of x16, x10, and x5 on three video-based tasks.
  • Our primary contribution lies in the design of the first Spiking Video Transformer with effective spike-driven attention design (Proposition 3.1). This approach achieves SOTA performance compared to existing SNN approaches, with over 15% improvement on human pose tracking and video semantic segmentation.

Thanks for the helpful questions. We will clarify the uniqueness and novelty in the final version.

Experimental Designs Or Analyses:

• Q1: We are sorry that the statement of our method is unclear than intended. To our knowledge, we are the first to explore spiking video Transformer. Thanks for the constructive comment. We will clarify the novelty in the final version.

• Q2: Other Transformer-based SNNs are image-based approaches that only use spatial attention. For a fair comparison on video-based tasks, we adapted these approaches to incorporate space-time joint attention. This setting is applied consistently across all three tasks in our work. We will clarify this point in the final version to avoid confusion.

Supplementary Material:

• Thanks for pointing out this issue. We will include the supplementary materials as an appendix at the end of the main manuscript in the revised submission.

Other Comments Or Suggestions:

• We will carefully address the formatting errors in the table captions, correct the notations in the equations, and fix any typos in the final version.

We value your feedback and will ensure that the above results and discussions are thoroughly addressed in the final revision.