Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras

Thank you for your precious time and insightful comments. We first list your advice and questions, then give our detailed answers.

W1: A little confusion in Spike-Based Image Reconstruction.

Thank you for your question that has brought our attention to a potential point of confusion. We would like to clarify two things in Related Works:

(1) “Spike-Based Image Reconstruction” refers to image reconstruction using spiking cameras (a type of neuromorphic camera), which is not related to the spiking neural network (SNN, a type of network architecture).

(2) When introducing deep learning techniques in this part, we first introduce supervised methods, which can then be divided into CNN-based [1, 2] and SNN-based [3]. Then what follows are the self-supervised CNN-based methods [4, 5].

To avoid ambiguity and aid understanding, we will revise the subtitle of Related Works from “Spike-Based Image Reconstruction” to “Spike-to-image Reconstruction” and Lines 109-111 in the manuscript as follows:

Original Lines 109-111: “Furthermore, several self-supervised CNNs have also been developed. However, due to the step-by-step paradigm, the above CNN-based architectures inevitably have higher model complexity, blocking them from mobile and real-time applications.”

Revised Lines: “The above three are supervised methods, whereas several self-supervised CNNs have also been developed. However, due to the step-by-step paradigm, the above CNN-based architectures inevitably have higher model complexity, blocking them from mobile and real-time applications. In contrast, our single-stage model jointly considers temporal motion estimation and spatial intensity recovery, thus facilitating the intrinsic collaboration of spatio-temporal complementary information.”

W2: The limitations of the related work and our improvement.

The limitations of related work:

(1) SSIR [3] (the SNN-based method), though energy-efficient, performs largely below the ideal level. While CNN-based methods [1,2] achieve promising results, the step-by-step paradigm inevitably leads to higher model complexity.

(2) In spike embedding representation, previous methods relied on either explicit or implicit representation. But relying on each side leads to the drawback of not being able to balance interpretability and strong expressiveness.

Our improvement:

(1) Our single-stage architecture targets the previous step-by-step paradigm and addresses temporal motion estimation and spatial intensity recovery jointly, therefore exhibiting excellent performance while maintaining low model complexity (see Table 1 and Figure 1).

(2) We developed a hybrid spike embedding representation (HSER) to offer good certainty and strong expressive capability simultaneously while maintaining low computational cost.

We have provided an overview of the workflow of our method in Section 4.1. Combined with Figure 2, it becomes easy to understand the mechanism and role of each module.

W3: More training details to allow reproducibility.

Due to page limitations, we put more training details and loss functions in Appendices A and B, which are sufficient to reproduce. We resonate with your concern for reproducibility and will put all the training details into the main text.

Q1: Is it possible to provide the codes and model weights in the supplementary material for reviewers’ inspection?

In accordance with NeurIPS requirements, we have sent the anonymous open-source code link to the AC for reviewers’ inspection.

Q2: Experiments only on the SREDS dataset. Can the results be generalized with a larger variety of benchmarks?

Currently, there are two available benchmark datasets for the spike-to-image reconstruction task: SREDS and Spike800.

• The Spike800 training set was introduced in Spk2ImgNet [1] in 2021. The spatial resolution of images is 400x250. Each scene contains 5 GT images, with 1 GT corresponding to 13 spike planes. There are 240,000 GT images and 13×240,000=3,120,000 spike planes.
• The SREDS training set was introduced in SSIR [3] in 2023. The spatial resolution of images is 1280x720. Each scene contains 24 GT images, with 1 GT corresponding to 20 spike planes. There are 524,160 GT images and 20×524,160=10,483,200 spike planes.

Both are synthesized based on the REDS dataset [6] and by the same simulator as that in Spk2ImgNet [1]. Considering the higher resolution images and larger number of training samples in SREDS, SREDS can be considered an upgraded version of Spike800. So we ultimately chose the latest SREDS dataset.

The commonly used generalization test approach of current spike-to-image reconstruction methods is to train on simulated data and then perform generalization tests on real-world datasets. We not only did experiments on the synthesized SREDS dataset but also on a variety of real-captured datasets, including “momVidarReal2021”, “recVidarReal2019” and our newly collected spike data, which covers high-speed camera/object motion scenarios under complex indoor and outdoor conditions and is sufficient to demonstrate the generalization performance of our method. Our method significantly outperforms existing methods in terms of reconstruction accuracy on both synthetic and real datasets.

[1] Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR 2021.

[2] Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI 2023.

[3] Spike camera image reconstruction using deep spiking neural networks. In IEEE TCSVT 2023.

[4] Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In IJCAI 2022.

[5] Self-supervised joint dynamic scene reconstruction and optical flow estimation for spiking camera. In AAAI 2023.

[6] Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In CVPRW 2019.

Thank you for your precious time and insightful comments. We address each concern below.

W1 & Limit2 & Q3: The overall architecture is too complex to implement or adapt. The approach still requires significant computational resources. How is the scaling achieved, and what are the trade-offs involved?

(1) Model Complexity. We understand your concerns. But if we take a holistic perspective, it becomes clear that the model follows a typical encoder-decoder architecture. More importantly, since our motivation is to jointly address temporal motion estimation and spatial intensity recovery in a single-stage manner, we actually simplified the whole reconstruction process. Our method has achieved real-time inference on an NVIDIA RTX 3090 GPU and 400×250 inputs (see Fig. 2), which cannot be done for existing CNN-based spike-to-image reconstruction methods. In addition to FLOPs presented in Table 1 (note that our method has the lowest FLOPs among CNN-based architectures), we further profile the computational complexity on an NVIDIA RTX 3090 GPU and 1280×720 inputs, i.e.,

Hence, we respectfully contend that, compared with other existing methods, our method has already achieved excellent performance with low model complexity and low computational resources. In the future, we will explore ways to enhance the efficiency of our approach on resource-constrained platforms.

(2) How the model adapts. It is clearly stated in Section 5.3 that feature pyramid levels, model capacity (the width multiplier for the feature channel), and number of multi-motion fields can be adjusted. In the implementation, you just need to change the corresponding parameters to achieve scaling, e.g., when the number of pyramid levels is 3, our method maintains the state-of-the-art performance with the smallest number of parameters (0.832M).

(3) There is a trade-off between computational complexity and model performance. From Table 3, the increase in model size improves the reconstruction quality but leads to more parameters and computations. Our model is handy to scale to fit into diverse scenarios. For instance, in scenarios that demand high precision and have abundant computational resources, a larger model is preferred. Conversely, for mobile or real-time applications, a simpler model can be employed.

W2: Limited theoretical justification for why this approach works better.

We deeply relate to your concerns and acknowledge the importance of theoretical analysis. Yet, our primary focus was to demonstrate the practical effectiveness of our motivation, i.e., temporal motion estimation and spatial intensity recovery can be mutually reinforcing. Therefore, in the paper, we prioritized extensive experimental validation to show its feasibility. Particularly, the sub-models we used (like ResNet and cross-attention) to meet our needs are widely tested for their exceptional modeling ability in academic research and have a solid mathematical foundation. We have made modifications to fit into our model, whose effectiveness was justified by extensive ablation studies.

Q1: The concept of the "hybrid spike embedding representation" (HSER) module and how it balance interpretability with expressive power?

HSER is composed of two parts: explicit and implicit. Explicit representations using TFP provide an anchor for image reconstruction since the results of TFP can be viewed as low-quality reconstructed images that have good interpretability. Implicit representations obtained from residual blocks have strong expressive capability to map original spike data to features. Previous methods have relied on only one side without integrating the power of both. Besides, HSER is very light-weight, which can balance interpretability with expressive power in an efficient way. All of these are clearly described in Section 4.2.

Q2: Which component seemed to have the most significant impact on the model’s performance?

Ablation studies have shown that each module improves reconstruction quality in its own right and is equally important. Yet Table 3 (b) underscores the foundational role of synthesis-based intra-feature filtering since it reconstructs the intermediate intensity frame. Rather than being assembled haphazardly, all modules are designed with an overarching purpose: to address temporal and spatial information simultaneously in an interactive and joint manner. Under the guidance of this idea, we proposed a single-stage architecture and multiple customized modules, all of which synergistically enhanced the performance to the state-of-the-art.

Q4: The significance of using $F_{t_2}^{L}$ as the query and using $F_{t_0}^{L}$ and $F_{t_2}^{L}$ as key/value in the attention mechanism.

The center $F_{t_1}^{L}$ corresponds to the final reconstruction objective, while the sides $F_{t_0}^{L}$ and $F_{t_2}^{L}$ serve as auxiliary components. In doing so, we followed the standard formula of cross-attention and designed a symmetric interaction strategy to collaboratively incorporate contextual information from both sides into the center, thus improving the overall performance as demonstrated in Table 3(c).

Limit1: The performance on a wider range of real-world scenarios.

To the best of our knowledge, we have covered all the publicly available real-world datasets for testing, i.e., “momVidarReal2021” and “recVidarReal2019”. (Note: You might find some other real-world datasets, but their scenarios are either already covered by these two datasets or involve motion that is too slow to effectively demonstrate the model’s performance.) It is also worth mentioning that we tested it on our real-captured spike data, which exhibits excellent results (see Figs. 9 and 10).

Thanks for authors' reply, I will still keep my score because there is still a lacking of theoretical reasoning to explain the novelty and contributions.

Thanks for your feedback. We would like to take this opportunity to address your concern and further clarify our innovation and contributions.

Existing learning-based spike-to-image reconstruction methods [1,2] predominantly rely on a two-stage architecture that sequentially cascades temporal motion estimation and spatial intensity recovery. The two-stage design, upon closer examination, also lacks rigorous theoretical justification for its effectiveness. This absence of theoretical analysis of network architectures is a common issue in current methods; however, it does not detract from the empirical results that consistently demonstrate the effectiveness of these approaches. In future work, we plan to incorporate the principles of the spiking camera imaging and leverage interpretability theories in machine learning so as to explore stronger theoretical foundations. If you have any insights or suggestions for improving the theoretical analysis of existing methods, we would be very grateful to hear them.

Our key contribution lies in the paradigm shift from two-stage to single-stage. We recognize that motion estimation and intensity recovery are inherently a "chicken-and-egg" problem—more accurate motion estimation facilitates better image recovery, and vice versa. To address this, we propose to integrate these two independent steps through a joint interactive learning approach. This not only significantly improves reconstruction accuracy (PSNR: 38.79dB vs 37.44dB) but also demonstrates substantial advantages in computational complexity (FLOPs: 0.42T vs 3.93T) and memory usage (Memory: 9424MB vs 20180MB), all compared to WGSE [2].

While ResNet and cross-attention have been modified and integrated into the model, they are not the core contributions of our work, as demonstrated in Tables 2 and 3(c), where our model still outperforms existing methods when they are removed or replaced.

Once again, thank you for your review. We would be grateful if you could consider raising the score accordingly, as we believe in the value of our work and that the paradigm shift will introduce new perspectives and prompt a reevaluation of existing approaches in the community. We will address the aforementioned issues in the final version and release our code and model upon acceptance of the paper.

[1] Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR 2021.

[2] Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI 2023.

Thank you very much for your precious time and recognition of our work. We first list your advice and questions, then give our detailed answers.

W1: Using previous models without mathematical justification.

(1) Mathematical justification. We deeply relate to your concerns about the mathematical foundation of a machine learning model. However, rather than merely copying the model or relying on them to achieve the state-of-the-art performance, we choose them by matching their functionalities with our needs. 

• ResNet excels at robust feature extraction by allowing gradients to flow more easily during backpropagation, and it has been tested in all sorts of tasks for its exceptional modeling ability. Its modularity and flexibility also allow us to integrate it easily into the spike embedding representation module. Combined with explicit representations (TFP), we designed a hybrid spike embedding representation (HSER). After trying different methods, empirical validation in Table 2 not only demonstrated the effectiveness of using ResNet compared with Multi-dilated [1] and HiST [2] representations, but also the effectiveness of the proposed hybrid scheme.

• Cross-attention improves the model’s ability to understand and utilize contextual relationships and facilitates the alignment of features between sequences. In our setting (Section 3.2), we use three non-overlapping spike sub-streams $S_{t_0}^{N}$, $S_{t_1}^{N}$, and $S_{t_2}^{N}$ to reconstruct the intermediate intensity frame $I_{t_1}$. The center is the reconstruction objective, while the sides serve as auxiliary components. In order to better incorporate contextual information from $S_{t_0}^{N}$ and $S_{t_2}^{N}$ into the intermediate time $t_1$, we adopted the idea of cross-attention and presented a symmetric interactive attention block. This symmetric design collaboratively enhances the bilateral correlation between the intermediate feature $F_{t_1}^{L}$ and temporal contextual features $F_{t_0}^{L}, F_{t_2}^{L}$ and also injects prior motion-intensity guidance into the subsequent interactive decoder, which boosts the model performance as shown in Table 3(c).

(2) Reproducibility. The model structure and experiment details are clearly illustrated and described in the paper, which facilitates reproduction. Moreover, we have sent our anonymous open-source code link to the AC for reviewers’ inspection as requested by NeurIPS.

(3) Interpretability. We have justified above the functionalities of previous models, the needs of our method, and the modifications we make to match the two, which may help in understanding the designs intuitively.

Q1: It would be beneficial to explore whether this method can be applied to event-based cameras.

Thanks a lot for your insightful idea. We have demonstrated in Table 4 that our HSER can better adapt the event-to-image reconstruction model to spike-to-image reconstruction, which is a good indication that the key reconstruction module is transferable by adjusting the frontmost embedding representation. Moreover, the core contribution of this paper lies in the idea of spatial-temporal joint learning. Applying this design philosophy to event-based reconstruction holds significant promise, given analogies between spiking and event cameras.

However, the reconstruction of event cameras might pose more challenges and need further investigation considering the different working mechanisms of event cameras and spiking cameras. Event cameras utilize a differential sampling approach and record the change in light intensity. Whereas spiking cameras follow an integral sampling method, thus preserving the absolute value of light intensity. Though this presents a slight digression from our current work, we look forward to inventing an input-agnostic image reconstruction method that unifies the input types for neuromorphic cameras in the future.

Limit1: It is challenging to determine which information contributes most to success. Further investigation can be done by applying the methodology to various sources of input.

The key contribution is that we adopted an interactive and joint perspective to address temporal and spatial information simultaneously. It was under the guidance of this idea that we proposed a single-stage architecture and multiple customized modules, all of which synergistically enhanced the performance to the state-of-the-art. Among them, synthesis-based intra-feature filtering acts as the foundation since it is devoted to reconstructing the intermediate intensity frame (as shown in Table 3 (b)), indicating that “spatial” information is essential for intermediate frame reconstruction. In contrast, other modules improve reconstruction quality to varying degrees, e.g., warping-based inter-frame feature alignment helps to aggregate contextual “temporal” information to achieve higher quality reconstruction. So it is crucial to note that all modules are designed with an overarching purpose instead of being assembled haphazardly.

As for further investigation, we appreciate you a lot for your insight. At present, spike datasets in more scenarios like autonomous driving and drone-taken scenes are not available. In the future, we will take these scenarios into consideration and validate the model’s effectiveness and generalization across different contexts.

[1] Unsupervised optical flow estimation with dynamic timing representation for spike camera. In NeurIPS 2023.

[2] Optical flow for spike camera with hierarchical spatial-temporal spike fusion. In AAAI 2024.

Thank you very much for your precious time and recognition of our work. We first list your advice and questions, then give our detailed answers.

W1: Generalization ability under unknown or broader conditions.

The commonly used generalization test approach of current spike-to-image reconstruction methods is to train on simulated data and then perform generalization tests on real-world datasets. We adopted this approach as well, but tested it on a wider range of real-world data, including “momVidarReal2021”, “recVidarReal2019” (also used in [1, 2]), and our real-captured data, which covers high-speed camera/object motion scenarios under complex indoor and outdoor conditions and is sufficient to demonstrate the generalization performance of our method.

As for the more unknown or broader conditions, they remain to be further explored. But we will consider involving more diverse scenarios and building datasets under more challenging conditions to tap into the potential of our models in future work. Thanks for your advice.

Q1: How does the model handle image reconstruction tasks under extreme motion or lighting conditions?

With a sampling rate of 40,000 Hz, the spiking camera is inherently capable of handling ultra-high-speed motion effectively. Even in scenarios that exceed the response speed of the human eye, our model still shows excellent performance. As illustrated in Figure 10 in the Appendix, our method successfully reconstructed the instantaneous process of a water balloon bursting in great detail.

In Section Limitations, we have discussed model performance in extremely low-light scenarios. Empirical experiments have shown that the limited accumulated light intensity often leads to darker images and increased noise. However, this limitation is common to current spike-to-image reconstruction methods [3, 4, 5] and can be further explored in our future work.

[1] Capture the moment: High-speed imaging with spiking cameras through short-term plasticity. In IEEE TPAMI 2023.

[2] Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI 2023.

[3] Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR 2021.

[4] Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI 2023.

[5] Spike camera image reconstruction using deep spiking neural networks. In IEEE TCSVT 2023.