High Dynamic Range Imaging with Time-Encoding Spike Camera

Thank you for your helpful comments, summary of our paper and affirmation of the performance. The questions and my answers are as follows.

Answers to the weaknesses

1. Comparison is limited to spike camera technology. Also, advantages or motivation of using a TE spike camera instead of any of the other methods described in the related work is missing.

Thank you for your valuable comments.

• Conventional RGB cameras enhance dynamic range by increasing the full-well capacity of pixels and the bit depth of analogue-to-digital conversion. However, due to their reliance on synchronous exposure and relatively long exposure times, they are prone to motion blur, making them unsuitable for high-speed scenarios.
• Event cameras, on the other hand, record changes in light intensity with extremely high readout frequencies, enabling excellent adaptation to high-speed scenes and offering a large dynamic range. Nevertheless, they struggle to capture static objects or scenes with large-scale repetitive textures, often requiring the assistance of other kinds of cameras to reconstruct complete environmental textures.
• Spike cameras accumulate photons and generate a spike when the threshold is reached, allowing them to capture both static and dynamic objects simultaneously. Their high readout frequency—up to 40 kHz—also makes them well-suited for high-speed imaging. However, their main limitation lies in the restricted dynamic range. In this work, we aim to improve the dynamic range of spike cameras in high-speed scenarios by modifying their working mechanism.
• We have included a comparison with one of the SOTA Event-RGB hybrid method HDRev[1]. This is the best model we have found so far among the available open-source implementations. We retrained it on our own training dataset. Since HDRev generates colour images, we converted them to grayscale for a fair comparison. The results on the HDM-HDR-2014 dataset are shown in the table below. The proposed method achieves the best result.

The results on the Kalantari13 dataset are shown in the table below. The proposed method achieves the best result for the PSNR-$\mu$ metric, and HDRev achieves the best result for the SSIM-$\mu$ metric.

[1]. Yang, Yixin, et al. "Learning event guided high dynamic range video reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

2. Evaluation is limited to simulation under idealized settings where the only noise is due to intrinsic shot noise and not any other hardware inconsistencies.

Thank you for your insightful comments. Our current focus is on the methodological exploration and validation. In future work, we plan to investigate hardware implementation under this novel mechanism.

3. Many figure & table captions are short making them more difficult to understand and what the reader should take away. For instance, figure 3 is confusing because an arrow from the first box flows into (c) suggesting this is one large flow diagram, but that is not the case.

Thank you for your helpful comments. We sincerely apologise for any confusion or misunderstanding that may have been caused. The arrow from the first box to $(c)$ indicates that the spike stream also serves as input to the Light Intensity-based Refinement module. We will revise both the layout of figures and their captions for improved clarity.

Answers to the questions

1. Over what time interval are spikes passed to the reconstruction network? And how fast does the network recover full frame HDR images?

Thank you for your valuable questions. We divide the continuous 61-frame spike stream into five overlapping segments: (1–21), (11–31), (21–41), (31–51), and (41–61), which are fed into the network. The proposed method infers a $512 \times 384$ image in 0.005 seconds.

2. What is the motivation or advantage of using a spike camera for HDR imagery over the alternatives? For instance, SPADs and event cameras are equally as fast. Why is recovering a full frame necessary for applications such as autonomous driving where the application is something else like obstacle avoidance?

Thank you for your insightful questions.

• SPAD sensors are capable of photon counting and are typically designed for extremely low-light environments. However, they struggle to get the accurate photon count when the light intensity is high.
• Event cameras, on the other hand, record changes in light intensity and generally offer a high dynamic range. Nevertheless, they struggle to capture static objects or scenes with large-scale repetitive textures, often requiring the assistance of other kind of cameras to reconstruct complete environmental textures.
• Spike cameras accumulate photons and generate a spike when the threshold is reached, allowing them to capture both static and dynamic objects simultaneously. Their high readout frequency—up to 40 kHz—also makes them well-suited for high-speed imaging. However, their main limitation lies in the restricted dynamic range. In this work, we aim to improve the dynamic range of spike cameras in high-speed scenarios by modifying their working mechanism.
• For obstacle avoidance scenarios, relying solely on event cameras may lead to failure in detecting texture-similar obstacles, such as a wall ahead. Furthermore, effective obstacle avoidance often requires the detection of lane markings to ensure compliance with traffic rules. Thus, recovering a full-intensity frame is often necessary for such tasks.

3. What is the final dynamic range of the proposed system?

Thank you for your valuable questions. The dynamic range of the TE spike camera is closely related to the bit depths of both the SFC (Spiking Firing Counter) and the CCC (Clock Cycle Counter). Assuming both the SFC and CCC are configured with an 8-bit depth, and that no significant motion blur occurs within the duration of 10 spike intervals in high-speed scenes, the TE spike camera can theoretically achieve a dynamic range of up to 116 dB ($20\log_{10}^{(2^8\times (2^8-1)*10)}$).

4. What is the average bandwidth / energy requirement of the proposed system versus the ML spike camera system?

Thank you for your insightful questions. TE spike camera outputs either the SFC or CCC data. When TE spike camera employs both an SFC and a CCC with a bit depth of $B$, and the ML spike camera uses an SFC with the same bit depth $B$, TE spike camera can ensure a bandwidth similar to that of ML spike camera. Compared with ML spike camera, TE spike camera achieves an extended dynamic range at the cost of recording time information.

Answers to the limitations

1. Some spike camera crops look blurry in comparison to the GT image.

Thank you for your valuable comments. There remains potential for further improvement in handling challenging high-speed motion scenarios. We plan to explore this aspect more thoroughly in future work.

2. Only grayscale imagery is recovered, but for consumer applications where recovering a full frame is necessary, color is often important.

Thank you for your helpful comments. In theory, this mechanism is also applicable to color spike cameras. Alternatively, color information can be recovered by integrating an RGB camera. Your suggestion provides a valuable direction for further development.

Thank you for your summary of our contributions and strengths. Your comments are helpful to our paper, and my answers are as follows.

Answers to the weaknesses

1. The experimental results are performed in simulation only.

Thank you for your insightful comments. Our current focus is on the methodological exploration and validation. In future work, we plan to investigate hardware implementation under this novel mechanism.

Answers to the questions

1. I would like to see more concrete examples given in your discussion of "fast ultra-HDR" scenes to highlight the importance and impact of TE spiking cameras. From the current discussion, it's not immediately apparent which applications this method would be most beneficial for.

Thank you for your helpful comments. In certain autonomous driving scenarios, high-speed and high-dynamic-range (HDR) conditions may occur—for example, when a vehicle enters or exits tunnels or underpasses, resulting in drastic changes in illumination. In nighttime environments, most regions may be dark, while others, such as those near streetlights, can be extremely bright. Similarly, extreme weather conditions like heavy rainfall can also create challenging HDR scenes. In future work, we plan to collect more datasets to further explore and investigate these scenarios.

2. Are there any challenges from a practicality standpoint in building the proposed TE spiking camera? Are there any tradeoffs (or increase in complexity) for building a TE vs. ML spiking camera?

Thank you for your insightful comments. In practical fabrication, the integration of both the SFC and CCC modules increases the complexity of the manufacturing process. Moreover, the timing errors introduced by CCC can significantly affect the performance of TE spike camera. Compared with ML spike camera, TE spike camera achieves an extended dynamic range at the cost of recording time information.

3. Representing spike streams as a large volume $\mathbb{R}^{H\times W\times T}$is inefficient since (presumably) much of the data in the stream will be zeroes (i.e., no spikes triggered). How does this affect network training / inference at scale? Can you run directly on spike data to save memory / computation?

Thank you for your helpful question. The spike stream characterises both the number of spikes and their firing times. Therefore, it is feasible to reconstruct images by focusing only on the fired spikes (i.e., excluding the zeros) along with their corresponding timestamps. This practice can save memory/computation, especially when much of the data in the stream is zero. This practice would require modifications to the network architecture and possibly an additional mapping between spike data and their firing time. This is an insightful question, and we plan to explore it in our future work.

Thank you for your summary of our contributions and strengths. Your comments are helpful to our paper, and my answers are as follows.

Answers to the weaknesses

1. It is not entirely clear to me how to understand TE spike cameras with reference to ML spike cameras. Do TE cameras provide higher dynamic ranges at the cost of additional resources, or do they present a better use of the circuitry and balance of bit depths? To image the same scene with a fixed dynamic range using the two types of cameras, how can we compare their energy consumption, bandwidth, latency, reconstruction difficulty etc.? In addition, the operating regimes of the two types of spike cameras in terms of dynamic range or bit depth need to be clarified.

Thank you for your helpful comments.

• Spike camera encodes light intensity by accumulating photons and releasing a spike when a predefined threshold is reached. However, since only one spike can be latched during each readout interval and subsequently read out, spike camera can only indicate whether the accumulated photon count has reached the threshold by the readout time. It cannot distinguish whether the threshold has been reached multiple times within the same interval.
• Multi-Level (ML) spike camera addresses this limitation by incorporating a counter to record the number of spikes, enabling it to represent higher light intensities and thereby extend the dynamic range of spike-based imaging.
• Building upon this, Time-Encoding (TE) spike camera introduces temporal information by recording the time required for the accumulated spike count to reach a certain value. By selectively reading either the spike count or the time information, TE spike camera achieves a further enhancement in dynamic range while maintaining a bandwidth comparable to that of ML spike camera. Of course, TE spike camera comes at the cost of additional timing circuitry.
• The dynamic range of ML spike camera is closely related to the bit depths of both the SFC (Spiking Firing Counter). Assuming the bit depth of the SFC is 8 and that no significant motion blur occurs within the duration of 10 spike intervals in high-speed scenes, ML spike camera can theoretically achieve a dynamic range of up to 68 dB ($20\log_{10}^{((2^8-1)*10)}$).
• The dynamic range of TE spike camera is closely related to the bit depths of both the SFC (Spiking Firing Counter) and the CCC (Clock Cycle Counter). Assuming both the SFC and CCC are configured with an 8-bit depth, and that no significant motion blur occurs within the duration of 10 spike intervals in high-speed scenes, TE spike camera can theoretically achieve a dynamic range of up to 116 dB ($20\log_{10}^{(2^8\times (2^8-1)*10)}$).

2. The reconstruction section is somewhat disconnected from the proposal of TE spike cameras. It would be better if the authors could offer more insights into the rationale behind the method design.

Thank you for your insightful comments. A basic reconstruction result can be obtained directly based on the following equation:

where $b$ denotes the number of fired spikes, $N_{\theta}$ is the threshold, and $nT_r$ represents the spike interval . This is also the principle of the TFI_TE method used for comparison in our experiments.

To achieve higher-quality reconstruction, we leverage deep learning techniques to effectively exploit the spatiotemporal features embedded in the TE spike stream. Specifically, we first decode the TE spike stream to obtain the number of spikes generated by each pixel at different readout moments. Given that the spike count is proportional to light intensity, its gradient serves as a useful cue for motion alignment. Motivated by this observation, we extract multi-scale gradient features from the TE spike stream and perform temporal alignment using a similarity-based multi-level pyramid network. Finally, an intensity-based refinement module is employed to fuse spatial features and generate the final reconstruction result.

3. According to Fig. 4, the visible performance gain is mainly on motion mitigation instead of dynamic range recovery. This does not align with the focus of this paper as introduced in the previous sections. Furthermore, whether the test data aligns with the ultra-HDR theme is not explained. While the TE results present higher scores and fewer blur effects, the advantages of TE spike cameras over ML ones are not fully verified by the results.

Thank you for your helpful comments.

• The reconstructed images preserve details in both bright and dark regions more effectively. Please refer to Figures 5–10 in the supplementary material for more visual comparisons.
• The HDM-HDR-2014 dataset features a dynamic range of up to 108 dB. The Kalantari13 dataset features a dynamic range of up to 97 dB. These two widely used public HDR datasets are typically regarded as benchmark datasets for ultra-HDR scenarios due to their extreme dynamic range. The images in the HDM-HDR-2014 dataset are organised in OpenEXR format. The images in the Kalantari13 dataset are organised in .hdr format.

4. This core contribution of this work is the proposal of a new design of spike cameras, but no prototype camera or real-world experiment is presented to verify the effectiveness and practicality of the proposed device.

Thank you for your valuable comments. Our current focus is on the methodological exploration and validation. In future work, we plan to investigate hardware implementation under this novel mechanism.

Answers to the questions

1. Can the authors comment on the operating regimes of the types of spike cameras, i.e., what dynamic range is considered HDR where ML spike cameras are applicable and what is considered ultra-HDR where TE spike cameras are needed?

Thank you for your valuable questions.

• Building upon ML spike camera, TE spike camera introduces temporal information by recording the time required for the accumulated spike count to reach a certain value. By selectively reading either the spike count or the time information, TE spike camera achieves a further enhancement in dynamic range while maintaining a bandwidth comparable to that of ML spike camera. Of course, TE spike camera comes at the cost of additional timing circuitry. Therefore, TE spike camera can be viewed as an extension of ML spike camera with an enhanced representational modality.
• The dynamic range of ML spike camera is closely related to the bit depths of both the SFC (Spiking Firing Counter). Assuming the bit depth of the SFC is 8 and that no significant motion blur occurs within the duration of 10 spike intervals in high-speed scenes, ML spike camera can theoretically achieve a dynamic range of up to 68 dB ($20\log_{10}^{((2^8-1)*10)}$).
• The dynamic range of TE spike camera is closely related to the bit depths of both the SFC (Spiking Firing Counter) and the CCC (Clock Cycle Counter). Assuming both the SFC and CCC are configured with an 8-bit depth, and that no significant motion blur occurs within the duration of 10 spike intervals in high-speed scenes, TE spike camera can theoretically achieve a dynamic range of up to 116 dB ($20\log_{10}^{(2^8\times (2^8-1)*10)}$).

2. What is the bit depth and the dynamic range of images in the dataset used? How does it align with the HDR and ultra-HDR arguments?

Thank you for your insightful questions. The HDM-HDR-2014 dataset features a dynamic range of up to 108 dB. The Kalantari13 dataset features a dynamic range of up to 97 dB. These two widely used public HDR datasets are typically regarded as benchmark datasets for ultra-HDR scenarios due to their extreme dynamic range. The images in the HDM-HDR-2014 dataset are organised in OpenEXR format. The images in the Kalantari13 dataset are organised in .hdr format.

3. In the experiment, ML spike cameras use an 8-bit SFC while TE spike cameras use a 5-bit SFC plus a 4-bit CCC. Is this setting fair?

Thank you for your helpful question. For TE spike camera, it would be more intuitive to compare a configuration using an 8-bit SFC along with an 8-bit CCC against ML spike camera using an 8-bit SFC, since TE spike camera outputs either the SFC or CCC data. This design ensures a similar bandwidth to that of ML spike camera. Our intention in this experiment setting was to reduce the bandwidth consumption of TE spike camera. We apologise for any confusion this may have caused.

Thank you for your reply. I think my concerns on fairness of ML and TE spike cameras comparison and definition of HDR and ultra-HDR are not fully addressed. To fairly compare ML and TE spike cameras, the cost of additional circuitry for CCC should be deducted from the SFC circuitry, instead of only fixing the output bandwidth. On dynamic range, many CMOS cameras today already achieve 12- or 14-bit readout in a single capture. This is 84 dB already ($20\log_{10}{(2^{14}-1)}$). Claiming the data used in this work to be ultra-HDR is questionable, and the necessity of the proposed TE spike camera is not clearly explained. In addition, there is no prototype camera or real-world experiment presented, which is critical for such a proposal of a new sensor design. I would have to keep my rating.

Thank you for your summary of our contributions and strengths, and for providing five specific weaknesses together with constructive suggestions.

1. Lack of real-sensor evidence – all experiments are synthetic. No prototype TE chip or mixed-signal simulation is shown, so claimed energy savings remain hypothetical. (Provide at least SPICE or Verilog-A level simulations of pixel, CCC, and timing signal to quantify area/energy overhead versus ML spike counter.)

Thank you for your helpful comments. We simulate the proposed method at the software level. The simulation results show that the imaging results are good under this mechanism. The bandwidth can be determined by the bit depth of the SFC and the CCC. Assuming that the bit depth of the SFC and CCC is $B$, since TE spike camera outputs either the SFC or CCC data, the number of bits read out each time is $B+1$ (overflow flag). At present, we cannot effectively simulate the energy cost.

2. Limited baseline set – comparison omits recent hybrid-sensor HDR methods that fuse events with frames or ML-based spike counters; fairness of gains is unclear. Include newer event/frame fusion HDR networks as baselines; retrain them on the same datasets to ensure fairness.

Thank you for your insightful comments. We have included a comparison with one of the SOTA Event-RGB hybrid methods, HDRev[1]. This is the best model we have found so far among the available open-source implementations. We retrained it on our own training dataset. Since HDRev generates colour images, we converted them to grayscale for a fair comparison. The results on the HDM-HDR-2014 dataset are shown in the table below. The proposed method achieves the best result.

The results on the Kalantari13 dataset are shown in the table below. The proposed method achieves the best result for the PSNR-$\mu$ metric, and HDRev achieves the best result for the SSIM-$\mu$ metric.

[1]. Yang, Yixin, et al. "Learning event guided high dynamic range video reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

3. Reproducibility gaps – authors will release code only “after publication’’ and provide no error bars. Statistical significance is therefore unknown. (Release training scripts and checkpoints at submission time and add confidence intervals (3 seeds) for key metrics.)

Thank you for your valuable comments.

• Due to the limitations of the rebuttal process, we are unable to release our code at this stage. However, we will pay attention to this point in future submissions. Thank you for your reminder.

• Regarding the error bars, we used three different random seeds to generate our training datasets (note that different seeds affect the shot noise) and trained the proposed method, Spk2ImgNet_TE, and BSF_TE. The testing results on the HDM-HDR-2014 dataset are shown in the table below. The proposed method achieves the best result.

The testing results on the Kalantari13 dataset are shown in the table below. The proposed method achieves the best result.

4. Hardware feasibility – storing a high-frequency CCC per pixel may raise SRAM and routing pressure; silicon overhead is not analysed. (Analyse sensitivity to clock frequency choice f and overflow depth b; a plot of dynamic-range gain vs. energy would guide sensor designers.)

Thank you for your insightful comments. To support a larger dynamic range, the bit depth of the SFC or CCC needs to be increased, which will lead to a larger bandwidth and greater energy cost. The focus of this paper is not on hardware implementation, but mainly explores the feasibility of the solution from the perspective of the working mechanism and principle design, and verifies the feasibility of the solution at the software level. The overhead and energy consumption analysis at the circuit level will be further explored by colleagues responsible for circuit design in the future.

5. Generalisation to real HDR noise – Poisson shot noise and hot pixels are not modelled in the synthetic data. Hence, robustness might degrade in practice. (Test reconstruction on real spike-camera HDR sequences (e.g., hybrid SpikeCam + LED-flash dataset) to demonstrate domain robustness.)

Thank you for your helpful comments. In our simulations, we modeled Poisson shot noise as follows: we scaled each image pixel’s normalized intensity (ranging from 0 to 1) by 60,000 to obtain the average photon count per readout interval for the corresponding spike camera pixel, and then applied the function numpy.random.poisson to simulate the Poisson shot noise in the photon arrival process. Therefore, the proposed method exhibits a certain degree of robustness to Poisson shot noise. Currently, the proposed method can only simulate spike streams based on real datasets for conducting experiments.

We sincerely appreciate your thorough review and valuable time invested in evaluating our work. We fully acknowledge your primary concern regarding the absence of physical camera hardware implementation. While we recognize the importance of circuit-level simulation, we clarify that due to time constraints and interdisciplinary barriers (between computer vision and electrical engineering), such implementation remains beyond the current scope of this study. Nevertheless, in direct response to your constructive suggestion, we have:

1. Conducted comprehensive comparisons with state-of-the-art event-RGB hybrid HDR methods

2. Implemented rigorous validation through triple random-seed experiments to demonstrate method consistency

3. Expanded discussion of alternative camera technologies (as suggested by Reviewer ndvD)

4. Enhanced simulation credibility through detailed modeling of dark current noise characteristics (as suggested by Reviewer ndvD)

Regarding code availability, we respectfully maintain that in computer vision research, open-source implementation typically follows peer-reviewed publication. We regret to inform you that policy restrictions currently prevent us from sharing code links during the rebuttal phase.

We remain fully committed to addressing any additional concerns or suggestions you may have. Your expert feedback would be valuable for further improving our methodological framework and presentation quality. Thank you again for your professional engagement with our work.