SPACT18: Spiking Human Action Recognition Benchmark Dataset with Complementary RGB and Thermal Modalities

References [1] Tong Bu, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, Tiejun Huang, Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks, ICLR 2022

[2]Bingsen Wang, Jian Cao, Jue Chen, Shuo Feng, Yuan Wang. A New ANN-SNN Conversion Method with High Accuracy, Low Latency and Good Robustness, IJCAI 2023.

[3] Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for spatio-temporal feature extraction. Neural Processing Letters, pages 1–17, 2023.

[4] Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[5] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting, ICLR 2022.

[6] Shiting Xiao, Yuhang Li, Youngeun Kim, Donghyun Lee, and Priyadarshini Panda. ReSpike: Residual frames-based hybrid spiking neural networks for efficient action recognition. arXiv 2024.

[7] A. Amir et al., "A Low Power, Fully Event-Based Gesture Recognition System," CVPR 2017

[8] Qi Wang, Zhou Xu, Yuming Lin, Jingtao Ye, Hongsheng Li, Guangming Zhu, Syed Afaq Ali Shah, Mohammed Bennamoun, Liang Zhang. DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognition. ECCV 2024

[9] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. CVPR 2021

[10] Gaole Dai, Zhenyu Wang, Qinwen Xu, Ming Lu, Wen Chen, Boxin Shi, Shanghang Zhang, Tiejun Huang. SpikeNVS: Enhancing Novel View Synthesis from Blurry Images via Spike Camera. Arxiv 2024

[11] R. Zhao et al., "Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations," CVPR 2024

[12] Kexiang Feng, Chuanmin Jia, Siwei Ma, Wen Gao. SpikeCodec: An End-to-end Learned Compression Framework for Spiking Camera. Arxiv 2023

[13] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. ICME 2019.

[14] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In CVPR 2021

[15] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In CVPR 2020

[16] Jing Zhao, Ruiqin Xiong, Jiyu Xie, Boxin Shi, Zhaofei Yu, WenGao, andTiejun Huang. Reconstructing clear image for high-speed motion scene with a retina-inspired spike camera. IEEE TCI 2021

[17] Jiyuan Zhang, Shanshan Jia, Zhaofei Yu, and Tiejun Huang. Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI, 2023

[18] Shiyan Chen, Chaoteng Duan, Zhaofei Yu, Ruiqin Xiong, and Tiejun Huang. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In IJCAI, 2022

[19]Wenming Weng, Yueyi Zhang, and Zhiwei Xiong. Eventbased video reconstruction using transformer. In ICCV 2021

W3 Overclaiming on Spike Camera superiority. The paper claims that spike.........

The main drawback of the DVS camera is that it does not capture static scenes and struggles to capture texture details of visual scenes due to their recording of only relative changes in light intensity, leading to significantly degraded visibility. In contrast, a spiking camera records the absolute light intensity which allows it to produce spikes in static or nearly static areas, effectively displaying the background, and static objects providing rich spatial resolution [9,10].

W4: The paper overstates the complexity of action recognition relative to tasks such as optical flow estimation.......

We apologize for the possible confusion in our statements, we are not overstating the complexity of video classification but we mean that most of the existing datasets using spiking cameras actually target low-level vision tasks, but there is no dataset for action classification which is the primary task for benchmarking models for video understanding. Actually, besides its usage for action video classification, our dataset is also will be a valuable resource for image reconstruction and spiking data compression tasks [11,12].

Questions

Q1: Could you clarify the advantages of using spike cameras over event cameras, RGB cameras, and thermal data?

Spike cameras compared to RGB and thermal offer several advantages, particularly in their ability to produce asynchronous spike outputs that are inherently compatible with Spiking Neural Networks (SNNs) and neuromorphic hardware in addition to high temporal resolution. Compared to the event camera has an advantage in responding to static scenes, and background, and effective texture reconstruction. We don't claim superiority of spike camera over other modalities but it distinct camera with a distinct type of spike data to benchmark SNN models.

Q2: From Table 4, it appears that performance using spike data is consistently lower, and Figure 6 shows that spike data is quite blurry, which may make it harder for models to learn meaningful dynamic information. How do you address these limitations, and in what situations does spike data excel over other modalities?

The reconstruction presented in Figure 6 is intended for visualization purposes and was done using the TFP method [13]. It is important to note that the figure shows the compressed versions of the original data, reducing 100k spikes to 10k and 1k spikes versions. This compression inevitably introduces a degree of blurriness in the visualized output, but it does not affect the model's ability to learn from the raw spiking data. Furthermore, more advanced reconstruction methods are available [14-19], and TFB is just one of the techniques employed for the visualization purpose.

Additionally, The primary purpose of our paper is not to benchmark reconstruction methods but to focus on video classification as the primary application of spike data in this paper. Reconstruction is included as a downstream task to verify and visualize that the model preserves scene information, but it is not the raw input used for training. This distinction emphasizes that while reconstruction has its uses, it does not reflect the full potential of spike cameras, which excel in tasks like video classification due to their compatibility with neuromorphic hardware.

Furthermore, as shown in Table 4, the accuracy of models trained on spiking data (10k) is nearly on par with those trained on thermal and RGB data. This indicates that the information necessary for effective learning is retained in the spiking data despite the compression.

Spiking data is preferred because of its compatibility with SNNs and neuromorphic hardware, the existence of spiking data is necessary to benchmark the SNNs model for video classification. We didn't claim that necessarily spiking data will excel over thermal and RGB data in these tasks, however, we are introducing a native dataset collected using a spike camera for benchmarking SNN models. Additionally, we have introduced the other two modalities as complementary models to accelerate the research of, including but not limited to, multimodal SNNs.

Summery We hope that we addressed all of the reviewer's concerns, and provided the required experiments. And we are so happy to provide more answers and experiments, We kindly ask the reviewer to reconsider their low score. Thanks for your feedback to enrich our paper with your valuable feedback!

Dear Reviewer,

Thank you for your review and valuable feedback. We will address the weaknesses and questions you raised below,

W1: The experimental setup and analysis are not sufficiently thorough......

ANN-SNN conversion aims to approximate the original ANN model, but it requires the activation function to be ReLU, which is uncommon in ANN-based video classification models. We attempted to replace the Sigmoid and SiLU activation functions in X3D models with ReLU. While training the modified model resulted in less than a 1% drop in accuracy, the ANN-SNN conversion produced random guesses, even after testing multiple ANN-SNN conversion techniques [1,2].

This issue arises because X3D is a very deep model, and the error propagates through the layers. By the final layer, the classification accuracy dropped to 5%, even with a high number of timesteps +1000 time step. While we provided a conversion for MC3, as noted in the paper, it still requires high timesteps. This underscores the need for ANN-SNN conversion techniques specifically designed for video classification tasks.

On the other hand, directly training SNNs for video classification remains challenging, with performance significantly lagging behind ANNs, by over 30%, as noted in [6]. Following your suggestion, we conducted additional experiments with existing SNN-based models [3,4,5] and the ReSpike hybrid model [6]. The results, summarized below, show that hybrid models effectively balance energy efficiency and accuracy. Notably, Feeding spiking data to SNNs and RGB or thermal data to ANNs, combined with cross-attention fusion, improved performance. However, our findings highlight the need for further efforts to enhance SNN training for video classification, such that Hybrid models are not compatible with energy-efficient neuromorphic hardware.

Model Accuracy for SNN and Hybrid Models

W2: The dataset's scale and diversity are limited compared to established RGB video......

We acknowledge the reviewer's observation regarding the scale and diversity of SPACT18 compared to benchmarks like HMDB-51, UCF-101, and Kinetics-400. While SPACT18 includes fewer action classes (18) and a smaller number of training clips, it is important to note that our dataset pioneers the use of spike cameras for action recognition. This novel modality inherently differs from RGB datasets and introduces unique challenges and opportunities for research in neuromorphic computing and multimodal fusion. In addition to that, we want to highlight the problem of storage and high volume of data, which is the storage for raw spike data was ~95GB per subject. Notably, DVS-Gesture [7] when it was proposed for the DVS sensor was only 11 classes with 29 subjects, however, this work was a valuable benchmark for SNN models and recently there is a dataset with 200 classes [8]. We believe that SPACT18 complements existing benchmarks by addressing a gap in the study of spike camera data, rather than directly competing with them in scale. Furthermore, as the research community adopts and builds upon this dataset, its scope can be expanded through collaborative efforts and future iterations.

Comparison of Action Recognition Datasets

Dear Reviewer YfhY,

We would like to know if we have addressed your concerns or if you have any follow-up questions.

Dear Reviewer,

Thank you for your positive feedback and valuable suggestions and questions to enrich our work. Below, we address your questions and concerns.

W1: The paper lacks a thorough comparison with existing action......

We compared our dataset with similar datasets collected by the event camera as shown in below Table, we will include it in the revised version.

Comparison of Event-Based Action Recognition Datasets

Dear Reviewer jxr8,

We would like to know if we have addressed your concerns or if you have any follow-up questions.

Q3: Due to the fact that video understanding typically adopts frame-by-frame input........

Thank you for your insightful comment. In our study, we thoroughly examined the impact of individual frames and cumulative states on overall recognition performance. Specifically, we conducted ablation experiments by selectively removing or masking certain frames to evaluate their influence on the task. This included testing different frame selections and analyzing their impact, particularly for RGB and thermal videos. We found that uniform sampling across frames yielded the best results, as it minimizes information loss while avoiding redundant frames.

For spiking camera has a resolution of 20k fps, which is more than sufficient to capture all necessary information. Unlike event-based cameras, spiking cameras can capture both static and dynamic events, making them less sensitive to frame removal. However, we conducted an ablation study on the effects of the compression algorithm (as detailed in the paper) by removing frames. The results showed that increased compression (i.e., more frame removal) led to visually degraded outputs and reduced model performance. This highlights the importance of preserving frames in spiking data, although the dependency is not tied to specific frames as it often is with event cameras. Again, we chose uniform sampling across frames as in RGB as it reduces the needed frames without hurting the performance.

Summary

We hope we answered all questions and weaknesses, conducted the required experiments and made the required comparisons to make the reviewer more confident about the contribution of our paper and we kindly ask the reviewer to increase their score. Thank you!

References

[1] Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for spatio-temporal feature extraction. Neural Processing Letters, pages 1–17, 2023.

[2] Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[3] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting, ICLR 2022.

[4] Shiting Xiao, Yuhang Li, Youngeun Kim, Donghyun Lee, and Priyadarshini Panda. ReSpike: Residual frames-based hybrid spiking neural networks for efficient action recognition. arXiv 2024.

Dear Reviewer,

We thank you for your positive assessment of our paper and for suggesting further improvements. Please find our responses to your questions and concerns below.

Weaknesses: The experimental part appears slightly thin........

We utilized X3D and UniFormer as they represent SOTA architectures for convolution-based and transformer-based models, respectively, in action recognition and video understanding.

Following your suggestion to make our evaluation more comprehensive. We have conducted more experiments using different models on both ANN and SNN direct training. For ANN model, we have conducted more experiments on C2D, I3D, SlowFast models, the results are presented below

Results for ANN Models

On the other hand, directly training SNNs for video classification remains challenging, with performance significantly lagging behind ANNs, by over 30%, as noted in [4]. Following your suggestion, we conducted additional experiments with existing SNN-based models [1,2,3] and the ReSpike hybrid model [4]. The results, summarized below, show that hybrid models effectively balance energy efficiency and accuracy. Notably, Feeding spiking data to SNNs and RGB or thermal data to ANNs, combined with cross-attention fusion, improved performance. However, our findings highlight the need for further efforts to enhance SNN training for video classification, such that Hybrid models are not compatible with energy-efficient neuromorphic hardware.

Results for SNN and Hybrid Models

Questions

Q1: Has a detailed analysis been conducted on the impact of external factors such as age, gender, and race....

Thank you for your valuable insights. We confirm that the subjects used in training and testing have no significant impact on the model's performance. Our dataset encompasses a diverse range of genders and races, with an age span of 21 to 42 years. It includes 13 different nationalities and represents both males and females. During training and testing, we employed various random splits to check the impact, which consistently yielded comparable results with a standard deviation of less than 1%.

Dear Reviewer jR4n,

We would like to know if we have addressed your concerns or if you have any follow-up questions.

W5: There is a lack of comparison of model experiments for SNNs that direct train:

Directly training SNNs for video classification remains challenging, with performance significantly lagging behind ANNs, by over 30 %, as noted in [4]. Following your suggestion, we conducted additional experiments with existing SNN-based models [5,6,7] and the ReSpike hybrid model [4]. The results, summarized below, show that hybrid models effectively balance energy efficiency and accuracy. Notably, Feeding spiking data to SNNs and RGB or thermal data to ANNs, combined with cross-attention fusion, improved performance. However, our findings highlight the need for further efforts to enhance SNN training for video classification, such that Hybrid models are not compatible with energy-efficient neuromorphic hardware.

Model Accuracy for SNN and Hybrid Models

W6: Please reference these papers [8,9]:

Thanks for providing us with these valuable references to enhance our work, we will include these references in our revised paper.

Summary

We hope that we addressed all of the reviewer's concerns, and provided the required experiments. We kindly ask the reviewer to reconsiders their low score since the reviewer recognised our contribution as an Excellent. Our dataset will be a valuable resource for accelerating research in SNN video recognition and multimodal SNN.

References

[1] Leñero-Bardallo J A, Serrano-Gotarredona T, Linares-Barranco B. A 3.6 s Latency Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor[J]. IEEE Journal of Solid-State Circuits, 2011, 46(6): 1443-1455.

[2] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. CVPR 2021.

[3] Siwei Dong, Tiejun Huang, Yonghong Tian. Spike Camera and Its Coding Methods, Arxiv 2021.

[4] Shiting Xiao, Yuhang Li, Youngeun Kim, Donghyun Lee, and Priyadarshini Panda. ReSpike: Residual frames-based hybrid spiking neural networks for efficient action recognition. arXiv 2024.

[5] Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for spatio-temporal feature extraction. Neural Processing Letters, pages 1–17, 2023.

[6] Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[7] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting, ICLR 2022.

[8] Wang X, Wang S, Tang C, et al. Event stream-based visual object tracking: A high-resolution benchmark dataset and a novel baseline[C]. CVPR 2024.

[9] Duan Y. LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising. CVPR 2024

Dear Reviewer,

Thank you for taking the time to read and carefully review our paper.

W1: The authors emphasize that the temporal.....

Thank you for pointing paper [1] to our attention. However, although the paper [1] establishes theoretically that DVS sensors can have a very low temporal resolution, DVS sensors have one fundamental drawback compared to spiking cameras, DVS struggle to capture texture details of visual scenes. DVS camera generates events only when there is a change in pixel brightness. It does not respond to static areas of the scene and, therefore, cannot represent the background. It is sensitive exclusively to dynamic changes, such as moving objects or variations in lighting. In contrast, a spiking camera simulates neural spike activity and can continue generating pulse activity even when changes in the scene are minimal [2,3]. This allows it to produce pulses in static or nearly static areas, effectively displaying the background. The spiking camera's ability to mimic neural firing patterns enables it to capture subtle, ongoing background information, even in the absence of significant changes in the scene [[2,3].

Additionally, we want to highlight that our paper presents a novel native spike dataset for action recognition using a spiking camera. Existing spiking camera datasets primarily focus on low-vision tasks, where video sequences are on a millisecond timescale, requiring high camera motion. Our dataset differs in that each sample contains approximately 100,000 frames, opening new research questions, such as how to efficiently train this high volume of spike data? how do compress this high volume of data for transmission, processing, and extracting important features?..etc. Our dataset has a wide range of potential applications, as discussed in the last section of the paper that we enlist briefly here including but not limited to: a benchmark for SNN-Video models, multimodal-SNN, Spiking data compression, reconstruction..etc.

W2: In line 60, the thermal modality...........

Thank you for your valuable suggestion and question. We will provide a detailed explanation in the revised version of the paper.

In the scope SNNs, multimodal research remains relatively limited and underexplored. To address this gap, we have included these datasets to encourage the development of multimodal architectures using SNNs, thereby increasing the versatility and applicability of our dataset.

The inclusion of a thermal camera is particularly important because it captures motion effectively at low frame rates, even under low lighting conditions. By synchronizing thermal cameras with RGB cameras, we aim to enhance the diversity and robustness of our dataset, enabling broader applications and better performance comparisons across modalities.

W3: The references are out of date......

Thanks for your observation, The revised version will include up-to-date relevant and related work.

W4: Different models should be.....

We utilized X3D and UniFormer as they represent SOTA architectures for convolution-based and transformer-based models in action recognition, respectively. However, following your suggestion, we conducted a more comprehensive set of experiments on additional ANN models as presented in Table below.

Dear Reviewer dDYr,

We would like to know if we have addressed your concerns or if you have any follow-up questions.