A Light-robust Reconstruction Method for Spike Camera

Thank you for reading! We responded later than expected due to the substantial content additions. However, with the deadline approaching, we sincerely hope all reviewers to respond, allowing us to continue providing further information.

Thanks for your careful reading and valuable feedback!

Q1: How does the delta T affect the overall performance?

Our recurrent-based reconstruction method demonstrates flexibility in efficiently covering and extracting temporal information from spike streams. The number of spike streams, combined with delta T, determines the method's coverage of temporal information from spike streams. When employing the same number of spike streams, a too-small delta T results in insufficient coverage of the temporal information. This hinders our method of predicting and reconstructing scenes effectively. The increasing delta T is beneficial to our method of reconstructing scenes until stable. Generally, delta T is 41 (Zhao et al. ,2021); chen et al. ,2022); Zhang et al. ,2023). We have also opted for this value.

Q2: Does the number of forward path of LR-Rep reduces if we increase delta T?

The number of spike streams and delta T are independent. We can choose the different number of spike streams for inference as shown in Fig.11.

Q3: The current pipeline seems to occupy a large memory. How much the training takes for memory and time?

The time and memory required for training our method are, 18 hours and 34.4 GB which can be satisfied with 1 NVIDIA A100 GPU. This is attributed to the fact that recurrent-based networks typically consume more time and memory during training due to Backpropagation Through Time (BPTT). During testing, our method demonstrates high efficiency, i.e., our inference time is 818 ms while the inference time of WGSE is 1344 ms as shown in Table 7.

Q4: Related to display

Thanks for your careful reading. We have made relevant modifications based on your suggestions:

1. Figure 1 & Figure 3: We selected a color pair (green and red) with high contrast to represent forward and backward data streams. We have added a description of that temporary features for both forward and back are different.

2. Explanation of LISI and GISI transform: We divided the original paragraph into two blocks to describe proposed method more clearly.

3. Figure 4 & Figure 3: We have selected a more intuitive result (GISI$_t$) for display in Fig.4. The reconstruction results in Fig.3 are also updated accordingly.

4. Figure 5: We have made standardized adjustments to the formula (Not limited to Fig.5).

5. Figure 9: Thanks for your suggestion. The official source code of STP has been used for testing which is comparison a fair comparison. Therefore, we still hope to keep the initial results in the main paper. Besides, we have also provided the adjusted results of STP in Fig.20.

6. typo "camrea": We have updated the relevant typo.

We appreciate the professionalism and fairness of the official reviewers. In the process of responding to your questions, we have also gained insights on how to improve our paper. We sincerely hope that all the reviewers will make a fair and unbiased decision regarding the outcome of this article. If any of the public reviewers can disregard the results presented in the paper, the images and videos in the supplementary material, or even the promised release of model code and datasets, the quality of the open review environment would be greatly diminished.

Thank you for taking the time to read. We have made the necessary modifications based on your suggestions. Additionally, we have provided explanations regarding the questions raised by the public reviewers. We would greatly appreciate your feedback.

Thanks for your positive feedback. We calculate PSNR after scale the images to [0,255]. The result is as follows:

We have updated it to Table 1. We also keep the PSNR of raw results in our main paper for two reasons: 1. Although LLR includes low-light scenes, their the dynamic range is wide as shown in Fig.14. 2. In previous work (WGSE, SSML and S2I), PSNR of raw results is shown and we hope to be consistent with those papers.

Thanks for your careful reading and valuable feedback!

Q1: In Fig. 2, the GT scenes seem to be RGB. How are they converted to grayscale?

We employ the grayscale conversion function provided by OpenCV, i.e., cv2.cvtColor(img, cv2.COLOR_BGR2GRAY).

Q2: Why is the PSNR value so high? The S2I image looks so noisy compared to GT in Fig.8, but according to table 1, the PSNR is higher than 40dB.

When the scene is so dark, both the Ground Truth (GT) and the reconstructed images can be at a lower grayscale level. This results in a significantly lower MSE compared to normal-light scenes. Consequently, their PSNR is high.

The Peak Signal-to-Noise Ratio (PSNR) can be expressed as $PSNR = 10 \cdot \log_{10}\left(\frac{{\text{255}^2}}{{\text{MSE}}}\right),$ where MSE represents the Mean Squared Error. It can be further defined as, $MSE = \frac{1}{mn} \sum_{i=0}^{m-1} \sum_{j=0}^{n-1} \left( I(i, j) - K(i, j) \right)^2,$where m (n) is the height (width) of the image, respectively and $I(i, j)$($K(i, j)$) represents the pixel value at (x, y) in ground truth (the reconstructed image). Hence, when both the Ground Truth (GT) and the reconstructed images are at a lower grayscale level, MSE is lower (PSNR is higher). In Fig.8, the scene is exceptionally dark, and we applied the same gamma correction to the reconstructed result for better visibility to readers.

We appreciate the professionalism and fairness of the official reviewers. In the process of responding to your questions, we have also gained insights on how to improve our paper. We sincerely hope that all the reviewers will make a fair and unbiased decision regarding the outcome of this article. If any of the public reviewers can disregard the results presented in the paper, the images and videos in the supplementary material, or even the promised release of model code and datasets, the quality of the open review environment would be greatly diminished.

Thank you for reading! We responded later than expected due to the substantial content additions. However, with the deadline approaching, we sincerely hope all reviewers to respond, allowing us to continue providing further information.

Thanks for your careful reading and valuable feedback!

Q1: Reconstructed videos

We add reconstructed videos to the supplementary material where the state-of-the-art method WGSE is compared. We can find that the results of WGSE are very unstable and fluctuate greatly in the time domain, while our method can effectively handle temporal noise.

Q2: How the proposed method would perform where a significant amount of Poisson noise would be present?

Our method can deal with Poisson noise better than the state-of-the-art methods. First, the mean of the error from Poisson noise in the temporal dimension is 0. Our bidirectional recurrent-based reconstruction method can effectively aggregate temporal information. Its structure is suitable for learning to filter Poisson noise. Besides, our training set is generated from the spike camera simulator (Zhao et al. 2022a) and it considers Poisson noise and fixed-pattern noise in the spike camera.

Q3: Figure 5 & typos

Thanks for your suggestions. We aim to enhance the reader's understanding of GISI through Fig.5. Following your advice, we have included links to the algorithm. Additionally, we have addressed typos in our paper.

Q4: What were the intuitions behind selecting the chosen architecture?

This is an excellent question. We initially observed a pronounced performance degradation of reconstruction methods in low-light conditions. Subsequently, we noticed a significantly sparser distribution of spikes in low-light spike streams compared to those in normal-lighting conditions. The recurrent-based structure proves effective in aggregating information from different time. This forms the basis of our work. Finally, we design a tailored bidirectional recurrent reconstruction framework specifically for spike cameras.

Q6: Why not leverage any of the wavelet structure from WGSE?

The wavelet structure in WGSE (Zhang et al., 2023). exists solely within its representation. We replaced our representation with that of WGSE and retrained it. As shown in Table.3, the performance of our method (PSNR: 45.075 and SSIM: 0.9868) is better than that (PSNR: 42.302 and SSIM: 0.9744).

We appreciate the professionalism and fairness of the official reviewers. In the process of responding to your questions, we have also gained insights on how to improve our paper. We sincerely hope that all the reviewers will make a fair and unbiased decision regarding the outcome of this article. If any of the public reviewers can disregard the results presented in the paper, the images and videos in the supplementary material, or even the promised release of model code and datasets, the quality of the open review environment would be greatly diminished.

Thanks for your careful reading and valuable feedback!

Q1: Setting of light source power and How to achieve consistency with the real world?

We set the lighting parameters in the advanced 3D graphics software, blender, to make the lighting conditions as consistent as possible with the real world. The following are the configuration details in Blender.

In Blender, various types of lighting simulation functions, including sunlight, point lights, and area lights, have been integrated into the graphical interface. We can adjust lighting parameters to control brightness and darkness.

1. For sunlight in Blender, the watts per square meter can be modified. Typically, 100 watts per square meter corresponds to a cloudy lighting environment. For the 'Car$_L$' scene in Fig.13, we have set sunlight to 10 watts per square meter, which is deemed sufficiently low.

2. For point lights and area lights, Blender allows modification of radiant Power, measured in watts. This is not the electrical power of consumer light bulbs. A light tube with the electrical power of 30W approximately corresponds to a radiant Power of 1W. In the 'Cook$_L$' scene in Fig.13, we have set an area light with the radiant Power to 1W (the electrical power of 30W) . It already represents a very dim indoor light source.

Q2:How dark is our dataset?

The low light scene in our dataset cover various dark range. The brightness is not only determined by the light source, but also by factors such as camera distance, object occlusion, and so on. These factors are ultimately reflected in the grayscale of the rendered image. Therefore, we calculate the grayscale histograms of images in low light scenes. As shown in Fig.14 (each bar represents 5 grayscale levels). We can see that the grayscale is diverse and in a lower range.

Q3: A quantification for performance vs light intensity.

We generated spike streams under different lighting conditions by modifying the light source parameters. Due to the large time-consuming of once rendering for each type of lighting, we only use one scene to demonstrate the performance of our method. Related results are updated in Table.4. Our method demonstrates excellent performance.

Specific modification details: We set sunlight (see Q1) to 30, 50, 70, 90, 110 and 130 watts per square meter to render images respectively.

Q4: Diversity of motion

The motion in LLR are diverse. Each designed scene contains 1000 frames of images and the motion at different times is different. We generate a optical flow every 40 frames for LLR. The degree distribution of the optical flow is statistically analyzed in Fig.15. We can find that the motion in LLR covers all kinds of directions.

Q5: The significance of our overall idea

We appreciate your recognition that our idea is interesting. The bidirectional RNN itself is a meaningful structure and emerges in the NLP domain. Many other domains, such as video interpolation, multi-agent systems, and event-based vision, have been inspired and modify the structure to better suit their field. We follow the same research line, and we are the first ones that design a tailored bidirectional recurrent reconstruction framework specifically for spike cameras. Our experimental results demonstrate that RNN structure can also boost the performance in spike vision.

Q6: TFI & DSFT

Both TFI and DSFT first calculate the interval between adjacent spikes in the input spike stream, and they both employ a brute-force search approach. The difference lies in the fact that TFI takes the reciprocal of each interval.

Q7: LISI VS. GISI

GISI (our final method) not only outperform LISI (baseline (E) in Table.2) in both PSNR and SSIM on synthetic datasets but also have better generalization on real spike streams.

We update reconstruction results of two methods in Fig.17. More importantly, the cost of using GISI instead of LISI is negligible (we only need to use two 400x250 matrices to store the time of the forward spike and the backward spike, respectively), which does not affect the parameter and efficiency of the network at all.

Q8: Fig.5 & Fig.7

Thanks for your suggestions.

1. For Fig.5, we have included additional supplementary explanations for LISI in the appendix and highlighted them in the main paper. The "maintain" and "update" are unique operations in GISI. For better understanding, it is suitable to be written as pseudocode. The pseudocode has been written in the appendix. In the caption of Fig.5, we have updated the link for the specific location of the algorithm.

2. For Fig.7, we have also adjusted its positioning to better illustrate the specific modules of the network, placing it before the formulas.

Thank you for reading! We responded later than expected due to the substantial content additions. However, with the deadline approaching, we sincerely hope all reviewers to respond, allowing us to continue providing further information.

We appreciate the professionalism and fairness of the official reviewers. In the process of responding to your questions, we have also gained insights on how to improve our paper. We sincerely hope that all the reviewers will make a fair and unbiased decision regarding the outcome of this article. If any of the public reviewers can disregard the results presented in the paper, the images and videos in the supplementary material, or even the promised release of model code and datasets, the quality of the open review environment would be greatly diminished.

Thanks for taking the time to read. We have provided lots of additional results to address your concerns. Additionally, we have provided explanations regarding the questions raised by the public reviewers. We would greatly appreciate your feedback.

1. Incomplete Comparison:

We have compared our method with a sufficient number of methods to demonstrate its effectiveness.

Ref1 is from the MM conference, which has the same timeline as ICLR. This is a concurrent piece of work.

Regarding ref2, we have actually tested the method, but it struggles to handle scenes with a large motion. We have included it in the supplementary material.

As for ref3, it did not specifically design modules to handle low-light data or test low-light scenarios. Besides, ref3 is not the state-of-the-art method even for the normal-light data. No work includes the comparison with all methods in their field.

2. Concerns about Experimental Results

We have addressed and updated the information about PSNR in our response to Reviewer ajbr's comments. Please refer to that for clarification. In our user study, we adjusted the brightness uniformly first for viewing (details can be found in our paper by searching for the keyword "apply"). Then, they are presented to 30 individuals independently. We assure you of their authenticity.

3. Concerns about Experimental Results

We have provided the relevant information on training S2I and WGSE in the main paper. Here, we further clarify more details.

For TFSTP, SNM, and SSML, we use their open-source codes for testing! The dataset of Zhao et al., 2021 is also open-source. You are welcome to reproduce the results to verify the authenticity of the results instead of questioning us.

For all the supervised deep learning methods, i.e., S2I and WGSE, they were trained on the RLLR, consistent with our method to ensure fairness. S2I and WGSE offer open-source source codes. We trained and tested based on the source codes and the relevant parameter settings are the same as their paper!

Potential sources of visual differences are as follows:

1. Different ways of generating spike streams between RLLR and the training set from S2I: The training set in S2I uses an ideal simulation without noise to generate spike streams. However, in RLLR, we used the state-of-the-art spike camera simulator (R1) to generate spike streams, which introduces diverse noise including temporal noise and fixed-pattern noise. This introduces more difficulties in training for other methods to converge. However, our bidirectional recurrent-based reconstruction method can effectively aggregate more temporal information. Since our structure is suitable for addressing the noise in the dataset, our method can converge better. In addition, we also discussed the impact of the presence or absence of noise in training set on the results in our appendix (Fig.16).

2. Different frames used for comparison: The displayed results in our paper are not at the same time as those in the cited work. Many methods show temporal fluctuations in their reconstructions, while ours is more stable. You can refer to the supplementary materials for the reconstruction video.

3. Differences in position and scale: For reconstructed results of different methods, we have selected a more differentiated and representative location to enlarge and show.

We guarantee the reproducibility and authenticity of our experiments under the supervision of all public reviewers, official reviewers, AC, and PC. Besides, after publication, we will release all datasets and the checkpoints for S2I and WGSE.

4. Concerns about Experimental Results

This question overlaps with the previous ones. Please refer to Q2 for clarification on the higher PSNR of our method. Regarding reproducibility, as stated at the beginning of the paper, we will provide all codes and datasets after publication.

5. Concerns about the Speed and Domain Gap of Dataset:

The speeds in the scenes are set to mimic real-world scenarios. Taking the car scene as an example, the speed set in Blender is approximately 70 km/h. To provide more insights, we calculated the distance between two consecutive positions based on the designed action, which is approximately 5.099e-4m. Further, 5.099e-4 * 40000 * 3600 = 73 km/h. We have included an image in the supplementary material to illustrate these details.

6. Motivation & "spike cameras may be not suitable for low-light high-speed scenarios":

We cannot understand why this conclusion is inferred from our paper. Various sensors experience a decrease in data quality under low-light conditions, including DVS (R2). Following your logic, should we also conclude that R2 indicates DVS is not suitable for low-light scenes since it designed the architecture for low-light conditions?

Spike camera has shown enormous potential for high-speed visual tasks, such as reconstruction, optical flow estimation, and depth estimation (A large amount of work is accepted by high-quality meetings and you can find citations in our main paper). However, we have observed a challenge, i.e., the spike camera faces the decreasing data quality as other sensors under low-light conditions. Hence, this paper aims to address the challenges of the spike camera and our method successfully accomplishes them.

7. Limited Contribution:

In addition to the proposed contribution, you overlooked our other contribution. All the reviewers rated our contributions as 'good.' According to your argument, most papers would not be novel, as they can trace influences from others. The novelty lies in whether new problems are addressed (low-light reconstruction in spike vision), whether a reasonable framework is designed for new problems, and whether results are achieved that other methods cannot reach. If addressing an unattended problem is not considered novel, then what really is novelty?

R1. Junwei Zhao, Shiliang Zhang, Lei Ma, Zhaofei Yu, and Tiejun Huang. Spikingsim: A bio-inspired spiking simulator. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 3003–3007. IEEE, 2022a.

R2. Rui Graca, Brian McReynolds, and Tobi Delbruck. Optimal biasing and physical limits of dvs event noise. arXiv preprint arXiv:2304.04019, 2023.

3. experimental results

(1) dataset: Perhaps you have some misunderstandings here. In our first response (3.1), we mentioned that results from the same model trained on different datasets are different.

(2) temporal fluctuations: We have provided a reconstruction video to showcase this situation in the supplementary materials, and we also informed you in our first response. Ignoring our answers and asking the same question is an unprofessional and irresponsible behavior.

4. The model checkpoints:

In our first response (3 and 4), we have informed you more than once that we will provide checkpoints, codes, and datasets after publication for reproducibility!

Please wait...

1. Relevant instructions for Ref1, Ref2, and Ref3.

Please refer to the fourth paragraph of the previous comment for details. In addition, the results of Ref2 have been added to the supplementary materials. We hope you can observe their quality with an objective attitude.

2. Relevant instructions of PSNR.

We actually answered this question in our previous response. We are willing to provide a more detailed explanation once again.

(1) High PSNR:

When the scene is so dark, both the Ground Truth (GT) and the reconstructed images can be at a lower grayscale level. This results in a significantly lower MSE compared to normal-light scenes. Consequently, their PSNR is large.

The Peak Signal-to-Noise Ratio (PSNR) can be expressed as $PSNR = 10 \cdot \log_{10}\left(\frac{{\text{255}^2}}{{\text{MSE}}}\right),$ where MSE represents the Mean Squared Error. It can be further defined as, $MSE = \frac{1}{mn} \sum_{i=0}^{m-1} \sum_{j=0}^{n-1} \left( I(i, j) - K(i, j) \right)^2,$where m (n) is the height (width) of the image, respectively and $I(i, j)$($K(i, j)$) represents the pixel value at (x, y) in ground truth (the reconstructed image). Hence, when both the Ground Truth (GT) and the reconstructed images are at a lower grayscale level, PSNR can be large. In Fig.8, the scene is exceptionally dark, and we applied the same gamma correction to the reconstructed result for better visibility to readers.

(2) Calculate PSNR after scale the images to [0,255]:

Based on the comments of Reviewer ajbr, we scaled the image to [0, 255] before calculating PSNR. The relevant results are as follows:

You can see that our method has a PSNR (scale) of 38.131, while none of the other methods exceed 35! Therefore, the reconstruction results can be distinguished after scaling them to normal brightness!

(3) Quality of visualization results

We have presented the reconstruction results of all methods in the main paper. Our reconstruction results are visually better. This can be perceived by 30 invited individuals, us, and official reviewers. We hope that these results can be evaluated with an objective and fair attitude.

Please wait...

Despite the limited time available before the deadline, we have efficiently and to the best of our ability responded to your concerns. I am starting to suspect that this is an academic attack rather than a normal exchange.

First, we must express our dissatisfaction with your comments, which we find unprofessional and biased. Firstly, as you acknowledged, you are not a formal reviewer nor a representative figure in the field, which implies that your concerns might be based on misunderstandings. Therefore, your expression should be 'doubts' rather than 'serious issues,' which leans towards a negative connotation that could influence the reviewers and readers. Secondly, given that you claim to be a researcher in neuromorphic vision, your declaration of no conflicts of interest is meaningless.

Second, We are happy to answer your questions, but before we respond, the words you use, like 'hardly trust the validity of these experimental results,' 'somewhat suspect that you did not well-train other methods on these datasets,' and 'This outcome seems quite unbelievable!!' carry a very strong personal emotion. With such strong subjective beliefs, we are puzzled whether you would be willing to be convinced, even if we presented solid evidence.

Third, do you not understand the meaning of 'concurrent work'? Why isn't work from two months ago considered concurrent? Also, whether to compare concurrent works should be judged by the reviewers. You can ask questions, and we have given you the reasons. We think continuing nitpicking ignoring the previous explanation is disrespectful and carries clear offensive intent. As for R3, this work does not provide experiments to support it can work under a low-light environment itself, however, you ask us to conduct the experiments to verify this? As for R2, the main content is limited, if this work is not competitive in the settings, why should we consider it on the main content? But we can pose it.

We hope to have a polite discussion based on basic facts, not personal emotions, but scientific rationality. We believe it would contribute positively to an open review environment.

Please wait...

The results presented in Table 1, visualized in Figure 8, are obtained from the Cook Scene. Under the specified scene (Cook) and the given frame (Figure 8), $K_{\text{max}}$ is fixed.

Please note that the max pixel value varies in each scene, and the max pixel value also varies for each frame! You used a wrong assumption to come to a wrong conclusion.