Q1: What is the adjusted ReLU function for surrogate gradient?

A1: The adjusted ReLU function is defined as:

$f(x) = ln(1+e^{(x-\eta)}),$ where $\eta$ is the threshold to control the intensity in EQ. (4) of the original manuscript.

Q2: What are the differences between LIF and MP_LIF ?

A2: The output of the LIF neuron is a spike train, while the output of the MP_LIF neuron is the numerical values of membrane potentials. Naively speaking, the LIF neurons output 'where the spikes are' and MP_LIF neurons output 'how large is the membrane potentials'. The predicted membrane potentials will be set to zero after a spike is fired.

Q3: How are the parameters of the phosphene model selected? It says "fitting the experimetnal data recorded from patients with retinal protheses". I didn't see any fitting procedure or decription of the data.

A3: The hyperparameters of the phosphene model utilized in this work are based on various experimental and modeling results from different components of corresponding research:

In Eq. 2, we use $a = 0.75$, $b = 120$, $k = 1.73$, based on a fitting method of [1] on data of a human experiment from [2].

In Eq. 3, Eq. 4, $Pw = 3.3ms$, $f = 30Hz$, $threshold = 0.3 mW/mm^{2}$ are obtained from the setting and human experimental results with retinal prosthesis [3]. We use $I_{leak} = 23.9 \mu A$ based on a data fitting from [4].

In Eq. 5, Eq. 6, $a_{1}$, $a_{2}$, $b_{1}$, and $b_{2}$, $\lambda$ is adopted from a retinal phosphene model [5].

In Eq. 7, $d_{s}$, $d_{e}$, $\rho$, and $\gamma$ were obtained from a human retina model [6].

Q4: Table 2, The performance of this work is better than CNN approach (Cadena et al, 2019). But can you show the number of parameters of each approach?

A4: The number of total parameters in our proposed StimuSEE, including retina-net and PVS-net is 68.9 K, while the number of parameters in the compared CNN approach is 62 K.

Q5: Table 3 shows the energy cost of SRNN and CNN with a similar achitechture. The advange of SRNN is not surprising given the intrisically low energy cost of SRNN. However, the energy cost of SRNN should be compared to that of a CNN with similar performance (such as, Cadena, 2019) rather than similar achitechture.

A5: In the proposed StimuSEE, we predict stimulation patterns using spike trains recorded from a dynamic vision sensor as input. The frameworks designed by Cadena et al. [7] and Yan et al. [8] take images as input and output responses of V1 cells and Ganglion cells, respectively. Therefore, directly comparing power consumption under the same performance is intractable given different inputs and outputs. In Tab. 3, we demonstrate that our designed StimuSEE is energy efficient, leading to a more practical visual prosthesis.

References

[1] Polimeni J R, Balasubramanian M, Schwartz E L. Multi-area visuotopic map complexes in macaque striate and extra-striate cortex[J]. Vision research, 2006, 46(20): 3336-3359.

[2] Horton J C, Hoyt W F. The representation of the visual field in human striate cortex: a revision of the classic Holmes map[J]. Archives of ophthalmology, 1991, 109(6): 816-824.

[3] Granley J, Beyeler M. A computational model of phosphene appearance for epiretinal prostheses[C]//2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2021: 4477-4481.

[4] Fernández E, Alfaro A, Soto-Sánchez C, et al. Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex[J]. The Journal of clinical investigation, 2021, 131(23).

[5] Granley J, Relic L, Beyeler M. Hybrid neural autoencoders for stimulus encoding in visual and other sensory neuroprostheses[J]. Advances in Neural Information Processing Systems, 2022, 35: 22671-22685.

[6] Jansonius N M, Nevalainen J, Selig B, et al. A mathematical description of nerve fiber bundle trajectories and their variability in the human retina[J]. Vision research, 2009, 49(17): 2157-2163.

[7] Cadena, Santiago A., et al. "Deep convolutional models improve predictions of macaque V1 responses to natural images." PLoS computational biology 15.4 (2019): e1006897.

[8] Yan, Qi, et al. "Revealing fine structures of the retinal receptive field by deep-learning networks." IEEE transactions on cybernetics 52.1 (2020): 39-50.

Q1: The method and experimental procedure is still a bit vague to me, i.e. I’m not sure how data and gradient flow through this system and how this system trains. Correct me if I’m wrong, during training you use the DVS → spiking RNN → stimulation model → phospene model → PVS-net V1 model and then fit all parameters with the primate V1 data? Some parameters of the phosphene model are constrained by human patient data, is that also transferred to fit primate V1 data?

A1: The StimuSEE is composed of a retinal network, a phosphene model, and a primary visual system network (PVS-net). The retinal network utilizes dynamic vision sensor and spiking RNN to perceive and process the scene signals. Then, a phosphene model is designed to simulate phosphene in the retina. Finally, PVS-net is adopted to predict the firing rate of V1 neurons with the simulated phosphene. The phosphene model is fixed during training. The parameter of retinal network and PVS-net are trained to fit the neural responses of V1 cells.

Q2: Eq. 8 and Eq. 9, the notations are somewhat confusing, are capital Index and index the same thing? What does $\cdot$ and $[]$ mean in the Eq. 9?. I’m also not sure about the dimension of any of the variables… are they scalar or vector integer or what? Please clarify these in the paragraph following it.

A2: $x \in R^{1 \times 166}$ is the extracted feature of the PVS-net. $params \in R^{166 \times D}$ refers to a learnable matrix, where $D$ is set to 50 in this work. The selected parameters are further dot producted with the input value $x$ to yield the final prediction results. $\cdot$ denotes dot production and $[]$ refers to indexing operation. Eq. 8 and Eq. 9 reduce the correlation among different neurons when predicting spike firing rates, simultaneously acting as a form of regularization.

Q3: Usually you’d like to plot the scatter plot between predicted spikes and actual spikes for all images and summarize the correlation for each cell. Figure 5 a) looks nice but lack the population summary.

A3: We first present the image-level Pearson Correlation Coefficient (PCC) distribution for all 166 neurons in Table 1. Specifically, for each image, we compute the PCC of the firing rate across all 166 neurons with respect to the ground truth, providing statistical information for all testing images. Additionally, in Table 2, we offer the neuron-level PCC distribution for all 166 neurons. This involves calculating the PCC of the firing rate for each individual neuron in relation to the ground truth across all testing images, and presenting statistics for all 166 neurons. These experimental results show that our proposed StimuSEE fits the neural response of V1 cells well. Fig. 5b shows one of the best-performing ones.

Table 1: The distribution of PCC of 166 neurons to each image

Table 2: The distribution of PCC of each neuron against all test images

Q4: If the data presented Figure 5b is real biological spike data from times, it seems overly sparse. Can you clarify the image stimulation onset and offset time schedule of it? —— if the no spike period is just gray screen stimulation, then the temporal accuracy is just defined by the stimuli time course, which doesn’t mean much.

A4: The visual stimulus is presented to the subject to capture neural responses within a 60 ms time window. The spike firing rate is assessed by counting neuron firings within a fixed recording window of 60 ms. For one image, as the recording covers 166 neurons, the label consists of 166 numbers, each representing the corresponding neuron's firing count. The offset time is 940 ms. Figure 5b illustrates the firing pattern of a single neuron across the first 20 test images (x-axis). The temporal spike train of both the labeled and predicted neuron responses is modeled using a Poisson distribution. The fitting process is iterated ten times to mitigate the impact of randomness.

References

[1] Beauchamp, Michael S., et al. "Dynamic stimulation of visual cortex produces form vision in sighted and blind humans." Cell 181.4 (2020): 774-783.

[2] Walker, Edgar Y., et al. "Inception loops discover what excites neurons most using deep predictive models." Nature neuroscience 22.12 (2019): 2060-2065.

[3] Bashivan, Pouya, Kohitij Kar, and James J. DiCarlo. "Neural population control via deep image synthesis." Science 364.6439 (2019): eaav9436.

[4] Polimeni J R, Balasubramanian M, Schwartz E L. Multi-area visuotopic map complexes in macaque striate and extra-striate cortex[J]. Vision research, 2006, 46(20): 3336-3359.

[5] Horton J C, Hoyt W F. The representation of the visual field in human striate cortex: a revision of the classic Holmes map[J]. Archives of ophthalmology, 1991, 109(6): 816-824.

[6] Granley J, Beyeler M. A computational model of phosphene appearance for epiretinal prostheses[C]//2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2021: 4477-4481.

[7] Fernández E, Alfaro A, Soto-Sánchez C, et al. Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex[J]. The Journal of clinical investigation, 2021, 131(23).

[8] Granley J, Relic L, Beyeler M. Hybrid neural autoencoders for stimulus encoding in visual and other sensory neuroprostheses[J]. Advances in Neural Information Processing Systems, 2022, 35: 22671-22685.

[9] Jansonius N M, Nevalainen J, Selig B, et al. A mathematical description of nerve fiber bundle trajectories and their variability in the human retina[J]. Vision research, 2009, 49(17): 2157-2163.

[10] Granley J, Relic L, Beyeler M. Hybrid neural autoencoders for stimulus encoding in visual and other sensory neuroprostheses[J]. Advances in Neural Information Processing Systems, 2022, 35: 22671-22685.

[11] Van der Grinten M L, de Ruyter van Steveninck J, Lozano A, et al. Biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses[J]. bioRxiv, 2022: 2022.12. 23.521749.

Q1: Though the framework demonstrates reliable results in predicting V1 neurons, because of the different performance metrics (neuronal responses vs. reconstruction error), it is hard to directly compare this method against other existing methods. One potential solution is adding a decoder layer on top of the trained StimuSEE framework and see whether it indeed also facilitates reconstruction as well.

A1: We could potentially replace StimuSEE's PVS-net with a decoder for image reconstruction. However, unlike other existing frameworks (hybrid neural autoencoders) that use the original image as input for reconstruction, our proposed StimuSEE utilizes spike trains recorded from a dynamic vision sensor as input. From an information perspective, reconstructing the original image with spike train input is a more challenging task compared to using the original image as input.

Q2: Since the StimuSEE frameworks consist of multiple models, the paper perhaps should make it more clear which of the models are trained with the error signals from predicting V1 responses. For example, in Figure 2, it could be helpful to use a bounding box with a different color or the SRNN model or an arrow signifying the direction of training feedback.

A2: The parameters of the retina-net and PVS-net are updated based on the error signals during model training, and the parameters of the phosphene model are kept fixed. As suggested, we will revise Fig. 2 to better indicate the gradient flow.

Q3:The paper only briefly mentions that V1 neuron responses in blind animals are comparable to those in sighted animals, it is unclear how this approach directly transfers to blind animals.

A3: The concept of using visual cortical prostheses for vision restoration has long been proposed. Beauchamp et al. [1] demonstrated that dynamic stimulation could facilitate the recognition of letter shapes in both sighted and blind individuals. Building upon this insight, we introduce StimuSEE, a system that leverages the V1 neuron response in sighted animals as a reference to create stimulation for the blind. In parallel, researchers [2,3] have explored the correlation between stimulation and the responses of vision neurons.

Q4: Does this framework end up with different learned model based on coverage of V1 neurons in recording? Could you use two sighted animal to estimate how transferrable the model is from one animal to the other?

A4: We appreciate Reviewer Fc3Q for posing these intriguing open questions. Given that the dataset employed in this study is obtained from a single subject with a fixed electrode placement corresponding to one coverage pattern of V1 neurons, direct answers to these questions are not feasible. Considering that StimuSEE generates stimulation patterns based on V1 neuron firing, we posit that transitioning to a different V1 coverage pattern would result in a different model. The verification of transferability among animals will be addressed in our future research endeavors.

Q5: Since the frameworks require many pre-determined parameters for other non-trainable models, could the authors provide some insight on how sensitive is the SRNN network training with respect to different parameter settings among the other models?

A5: The hyperparameters of the phosphene model utilized in this work are based on various experimental and modeling results from different components of corresponding research:

In Eq. 2, we use $a = 0.75$, $b = 120$, $k = 1.73$, based on a fitting method of [4] on data of a human experiment from [5].

In Eq. 3, Eq. 4, $Pw = 3.3ms$, $f = 30Hz$, $threshold = 0.3 mW/mm^{2}$ are obtained from the setting and human experimental results with retinal prosthesis [6]. We use $I_{leak} = 23.9 \mu A$ based on a data fitting from [7].

In Eq. 5, Eq. 6, $a_{1}$, $a_{2}$, $b_{1}$, and $b_{2}$, $\lambda$ is adopted from a retinal phosphene model [8].

In Eq. 7, $d_{s}$, $d_{e}$, $\rho$, and $\gamma$ were obtained from a human retina model [9].

The phosphene model demonstrates robustness across various hyperparameter combinations, and the framework consistently produces similar prediction performance [10,11].

Q6: Could you elaborate on the purpose of the parameter index in equations (8)?

A6: Eq. 8 and Eq. 9 reduce the correlation among different neurons when predicting spike firing rates, simultaneously acting as a form of regularization.

Q1: It's unclear to me what is the novelty of the proposed phosphene model compared to Granley et al. (2022).

A1: Compared to the phosphene model in Granley et al. [1], the phosphene model of StimuSEE utilizes coordinate system transition for a more accurate phosphene location calculation. Also, the leakage $I_{leak}$ during the electrical stimulation process is also taken into account in the calculation of stimulation intensity. Furthermore, our proposed phosphene model incorporates a threshold function to determine the generation of phosphene.

Q2: How is PVS-net pre-trained? Was the same dataset utilized both for the PVS-net pretraining and the end-to-end training?

A2: We utilized CIFAR-10 dataset to pretrain the PVS-net with a classification task, and adopted the DVS-V1 dataset in the end-to-end training process.

Q3: What is the temporal resolution of the label (neural firing rate)? How many time points per image?

A3: The visual stimulus is presented to the subject to capture neural responses within a 60 ms time window. The spike firing rate is assessed by counting neuron firings within a fixed recording window of 60 ms. For one image, as the recording covers 166 neurons, the label consists of 166 numbers, each representing the corresponding neuron's firing count.

Q4: In Fig 5a, the authors only showcased 20 neurons. It would be informative if the authors could provide quantitive results on the model's fit across all 166 neurons. Is Fig. 5b showing the results from a single neuron? If so, is it the best-performing one?

A4: We first present the image-level Pearson Correlation Coefficient (PCC) distribution for all 166 neurons in Table 1. Specifically, for each image, we compute the PCC of the firing rate across all 166 neurons with respect to the ground truth, providing statistical information for all testing images. Additionally, in Table 2, we offer the neuron-level PCC distribution for all 166 neurons. This involves calculating the PCC of the firing rate for each individual neuron in relation to the ground truth across all testing images, and presenting statistics for all 166 neurons. These experimental results show that our proposed StimuSEE fits the neural response of V1 cells well. Fig. 5b shows one of the best-performing ones.

Table 1: The distribution of (PCC) of 166 neurons to each image

Table 2: The distribution of PCC of each neuron against all test images

Q5: What would be the chance-level performance for both Fig. 5a and Fig. 5b?

A5: We provide statistics Tab. 3 and 4 as in A4 based on our defined chance-level predictor.

Table 3: The distribution of PCC of 166 neurons to each image

Table 4: The distribution of PCC of each neuron against all test images

Q6: Is Fig. 5a showing spike counts from 20 neurons for a single time point or single image? If yes, why not show the overall statistics for all images in the test dataset?

A6: Please refer to A4 for the overall statistics.

Q7: Fig. 5b shows the neural spike train over time. What is the temporal resolution here? Also, why comparing the spike train randomly generated 10 times from the firing rate, instead of directly comparing the time series of the predicted and groundtruth firing rate?

A7: We apologize for any confusion caused by the x-axis of Figure 5b. When referring to 'times,' we meant images. Figure 5b illustrates the firing pattern of a single neuron across the first 20 test images (x-axis). The temporal resolution of both the labeled and predicted neuron responses is modeled using a Poisson distribution. The fitting process is iterated ten times to mitigate the impact of randomness. We will revise Fig. 5b in the camera-ready version for better presentation.

Q8: How did you get the PCC in Table 2? Is it the correlation between the predicted and ground truth firing rate along time? then averaged across all 166 neurons?

A8: Since the spike firing rate is assessed by counting neuron firings within a fixed recording window of 60 ms. The label consists of 166 numbers, each representing the corresponding neuron's firing count. Thus, we can directly calculate the PCC of all 166 neurons.