Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli

We thank the reviewer for being supportive of our work and for the constructive comments. We will try our best to address the comments. Below are our detailed responses.

1. About line 318.

We apologize for not refering to the corresponding table. The conclusion comes from the results on the right side of Table 2. We will add a reference to that table in the manuscript.

2. About spiking mechanisms?

We already have some experiments in our manuscript to compare spiking and non-spiking networks to demonstrate the importance and benefits of the spiking mechanism. What's more, according to the suggestion of reviewers, we conduct additional experiments to provide more evidence and to strengthen our conclusions (please refer to response 1 to Reviewer YZqi). We will add and emphasize these results in our manuscript.

3. About the training of our work.

Due to space constraints, we have put the detailed network training procedure and training parameters in the appendix. Please refer to Appendix C.

4. Is there a good reason for averaging? Could some regions be better fitted than others?

We use the average across brain regions as the similarity score for two reasons. First, neurons responsive to movie stimuli are found in all six cortical regions [1]. Second, the depths of the network layer with the best similarity score to most mouse cortical regions are at the same level [2]. We show the individual similarity scores of our model to each region in Table R1, suggesting that there is no significant difference across regions.

Table R1: TSRSA scores of LoRaFB-SNet to six cortical regions, respectively.

[1] Saskia EJ de Vries, et al. "A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex." Nature neuroscience 2020.

[2] Jianghong Shi, et al. "Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex." NeurIPS 2019.

I thank the authors for replying to my questions.

Additional analysis strengthens the evidence of the benefit of spiking versus non-spiking networks on capturing representational similarity in the mouse visual cortex. However, it remains somewhat unclear why spiking is actually beneficial. Authors make a valuable first step to demonstrate such benefit, even though their datasets are relatively small. Their hypothesis of the membrane potential dynamics being helpful (I suppose authors refer to the subthreshold dynamics of the membrane potential) is an interesting and viable hypothesis, but it is not proven in this paper. Future work could dig deeper to explain the benefit of spiking for the information processing of natural and time-dependent stimuli.

I have a question about the choice of the Pearson correlation coefficient to evaluate the similarity of movie frames and of Spearman rank coefficient to compute the similarity of the model and the data. I suppose that the choice of the Spearman coefficient is motivated by the ability to better capture nonlinear relations. Why is similarity of movie frames computed with Pearson correlation coefficient? Authors could justify these choices also when they introduce their method for computing the representational similarity.

Does the membrane time constant of the LIF neurons influence the TSRSA score? Authors report using tau=0.5, is this in units of milliseconds? Biologically plausible values of the membrane time constant are between 10 and 20 milliseconds. Can authors comment on that? Has the dependence of results on the membrane time constant been tested?

I find that the lack of local recurrent connections creates a major discrepancy between the proposed model and sensory networks in biological brains. It is expected that in biological networks that process sensory stimuli, local recurrent connections have a major impact on signal processing [1] and can support efficient computations on sensory features with biologically plausible neural architectures [2]. I would find it important that authors better describe this limitation of their model as they revise the paper.

[1] Bourdoukan et al., "Learning optimal spike-based representations", NeurIPS 2012 [2] Koren and Panzeri, "Biologically plausible solutions for spiking networks with efficient coding", NeurIPS 2022

We thank the reviewer for the clear and thoughtful comments. We will do our best to address the reviewer's concerns and answer the questions in the following.

1. The novelty of our model architecture.

The work of Xiao, Mingqing focuses on deriving the equilibrium state of a spiking neural network by introducing feedback connections, and in turn deriving the gradient of parameters for training the network without having to consider the forward process. However, their algorithm has only been validated on networks with shallow and stacked layers on small datasets like CIFAR10, and it is not yet known whether the equilibrium state is tractable in deeper and more complex network structures (e.g., skipping connections).

In contrast, we introduce multiple long-range feedback connections into a deeper spiking network and train it with a commonly used algorithm (BPTT with surrogate gradient) on large-scale datasets. This provides a more general structure and training paradigm for exploring feedback spiking networks.

2. The TSRSA.

We thank the reviewer for recognizing TSRSA. As suggested by the reviewer, we provide a more mathematical representation of the method here and will add it in our manuscript.

We define the representation matrix as $\mathrm{R}=(\mathrm{r}\_1, \mathrm{r}\_2, ..., \mathrm{r}\_t , ... , \mathrm{r}\_T)\in\mathbb{R}^{N\times{T}}$ of a given model layer or a given cortical region, where $\mathrm{r}\_t$ represents the population responses of $N$ units/neurons to the movie frame $t$. We first compute the Pearson correlation coefficient between the responses to a given movie frame $t$ and to its subsequent frames to obtain the representational similarity vector $\mathrm{s}\_t$. The vector is formulated as:

$\mathrm{s}\_t=(s\_{t1}, s\_{t2}, ..., s\_{tp}, ...),$

$s\_{tp}=Corr\_{Pearson}(\mathrm{r}\_t, \mathrm{r}\_{t+p}),$

where $0<p<T-t$.

Second, we concatenate all vectors to obtain the complete representational similarity vectors $S\_{model}$ for a network layer and $S\_{cortex}$ for a cortical region. Finally, we compute the Spearman rank correlation coefficient between $S\_{model}$ and $S\_{cortex}$ as the similarity score.

3. Is there a reason for not using VLMs, etc as a baseline.

We agree that exploring the information processing mechanisms of VLMs with the real visual cortex may lead to new insights for the machine learning and neuroscience communities. However, our work focuses on constructing biologically plausible deep neural network (DNN) models in terms of the structure and dynamics, to improve representational similarity to real biological neural responses, providing new insights into information processing mechanisms of movie stimuli in the visual cortex. Therefore, the baselines we chose for comparison are all bio-inspired DNNs, which are widely used in studies of neural representational similarity to the visual cortex.

SOTA vision models such as VLM on visual tasks have not been chosen as baselines in our work due to the following points.

• The core of our work is to build brain-like models, not to achieve better performance on visual tasks. However, VLMs are designed in a performance-oriented manner, without taking biological plausibility (e.g., attention mechanisms) into account [1].

• Previous studies already proved that visual task performance is not positively correlated with neural representation similarity. Instead, higher task performance may lead to poorer brain-like models [2].

• Transformer-based vision models have shown poor similarity (worse than convolutional networks) to real neural responses in the visual cortex [2, 3]. Similarly, transformer-based language models (LLMs) have been questioned for not truly reflecting the core elements of human language processing [4].

4. More bio-plausible learning mechanisms and more sophisticated neuronal models.

Despite the higher biological plausibility of the advised learning mechanism and neuronal models, introducing them in deep spiking networks and training them on large datasets suffers from problems such as difficulty in convergence and untrainability. Therefore, we have not considered them in our current work.

5. In the experiments why were the Neurons firing less than 0.5 spikes/s excluded?

When analyzing neuronal spiking activity, researchers often exclude neurons with firing rates below a certain threshold to focus on neurons that are responsive to visual inputs and to more effectively extract stimulus-related neural representations. An empirical threshold of 0.5 spikes/second is commonly used in many studies [5, 6].

[1] Paria Mehrani & John K. Tsotsos. "Self-attention in vision transformers performs perceptual grouping, not attention." Frontiers in Computer Science 2023.

[2] Drew Linsley, et al. "Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex." NeurIPS 2023.

[3] Liwei Huang, et al. "Deep Spiking Neural Networks with High Representation Similarity Model Visual Pathways of Macaque and Mouse." AAAI 2023.

[4] Ebrahim Feghhi, et al. "What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores." arXiv 2024.

[5] Lucas Pinto, et al. "Fast modulation of visual perception by basal forebrain cholinergic neurons." Nature Neuroscience 2013.

[6] Brice Williams, et al. "Spatial modulation of dark versus bright stimulus responses in the mouse visual system." Current Biology 2021.

We thank the reviewer for the notable and perceptive comments. We will do our best to address the comments and provide detailed responses point by point, below.

1. Importance of Spiking Mechanisms.

In our work, we design a deep spiking network based on rate coding and pretrain it on large-scale datasets. The training algorithms of temporal coding spiking networks are still not mature [1]. Given that our work is centered around deep neural network models, the rate coding spiking network is a better choice.

To better demonstrate the importance of spiking mechanisms, we clarify our existing results (the comparisons with CORnet in Fig. 3A, 3C and 3D) and perform more experiments for the non-spike version of LoRaFB-SNet (termed LoRaFB-CNet) as suggested by the reviewers. The results (see details below) provide strong evidences that spiking mechanisms play a crucial role, allowing our model to better capture neural representations of the visual cortex. Since our spiking network is based on rate coding, we think that the following two properties may contribute. First, our network encodes and transfers information exclusively in the form of spikes, just like the brain. Second, the membrane potential dynamics of spiking neurons help to process dynamic information, which complements the long-range feedback connections well.

The clarifications and new results about spiking mechanisms:

• LoRaFB-SNet outperforms CORnet and LoRaFB-CNet on both two movies (Movie1: 0.5202, 0.5060, 0.4975; Movie2: 0.2827, 0.2230, 0.2619).

• LoRaFB-SNet consistently outperforms CORnet on different lengths of movie clips (Fig. 3C) and shows the increasing ratio compared to CORnet (Fig. 3D). The results suggest our model performs significantly better on longer movie clips, echoing the results in Fig. 3A. Since longer movie stimuli increase the diversity of population neuronal response patterns in the visual cortex [2, 3], it is more difficult for models to capture brain-like representations. Our model has a more pronounced advantage in this case. See discussions in lines 234-243.

• We added the experiments on Movie2 in Figure 3C for the LoRaFB-CNet. As shown in Table R1, the results support the above conclusions. Besides, LoRaFB-CNet outperforms CORnet for all clip lengths, suggesting that our model structure also benefits information processing in the long movie.

Table R1: TSRSA scores with different movie clip lengths on Movie2. The standard error is omitted.

• For the experiments of static natural scene stimuli in Fig. 4C, we add the results of CORnet and the LoRaFB-CNet (0.4130, 0.3544, 0.3411), which shows that our model also outperforms them in this neural dataset.

The clarifications and new results about feedback connections:

• Fig. 4A compares the changes in similarity scores between the network with feedback and fully feedforward structures when the dynamic information is disrupted, supporting that the long-range feedback connections allow LoRaFB-SNet to represent dynamic information in a more context-dependent manner. See discussions in lines 272-275. We add the experiments for CORnet and LoRaFB-CNet in Fig. R1 in the PDF of the "global rebuttal", which solidifies our conclusion about the effectiveness of feedback connections.

• Fig. 4B does not involve comparisons of model structures and mechanisms. Therefore, we don't include CORnet and LoRaFB-CNet here.

We will add the above results and discussion to our revised manuscript.

2. Data and Model Robustness.

As mentioned above, the similarity score of models suffers from an increase in movie length, and LoRaFB-SNet shows a more pronounced advantage in longer movie experiments. The significant improvement in our model's score compared to other models, coupled with the overall lower scores of other models on Movie2, underscores the robust advantage of our model across different movies. In fact, when we use randomly selected 30s (the length of Movie1) clips from the 120s Movie2, LoRaFB-SNet achieves a score of 0.405 (about 80% of the score on Movie1). The scores of our model in six visual regions also indicate the robustness across brain regions (please refer to response 4 to Reviewer rzHc).

In summary, experiments and analysis in a variety of settings effectively demonstrate the performance and robustness of our model. In the future, despite the paucity of publicly available neural datasets under natural movie stimuli, we will try to apply our model to more datasets to solidify our conclusions.

3. Dependency on Pretraining Tasks.

We have preliminarily discussed the influence of pretraining datasets and tasks (Fig. 3B and the first paragraph of Section 4.3.3). In particular, the video recognition task better benefits model's neural similarity. Besides, temporal structures of data contribute more than the static content. Larger datasets and other video tasks may have different impacts, which we will explore in the further work.

4. Exploration of Local Recurrent Connections.

We agree that lateral connections within brain regions are useful for brain-like modeling. However, our work focuses on feedback connections across regions to build bio-inspired models. We will introduce local recurrent connections in the future for exploration.

[1] "Temporal-coded spiking neural networks with dynamic firing threshold Learning with event-driven backpropagation." ICCV 2023.

[2] "Rapid learning in cortical coding of visual scenes." Nature Neuroscience 2007.

[3] "Representation of visual scenes by local neuronal populations in layer 2/3 of mouse visual cortex." Front Neural Circuits 2011.