Modeling Focal Synaptic Degeneration and Neural Plasticity in Ventral Visual Cortex

We thank the reviewer for the discussion that they have brought up. We have tried to address all of them. Please also see the global rebuttal for common confusions around the intended purpose of the work. We reply to your concerns point-by-point below-

Current experiments focus on specific visual tasks, without covering a broader range of visual functions

As stated in the paper, we model synaptic decay within the framework of the ventral visual cortex (VVC), which is implicated in core object recognition [1, 2]. Currently, ImageNet is the best-known dataset that when trained on makes the model effective in predicting responses of real neurons to a large degree [3]. Moreover, ImageNet is not a simple dataset, and almost every paper within the neuroscience community that focuses on the ventral visual pathway uses ImageNet for model training [4, 5, 6]. It is the current best known dataset that when trained on produces more neurally-aligned models of the ventral visual cortex. If a better dataset is developed, we would be happy to test our models on it when it becomes available.

[1] DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012). How does the brain solve visual object recognition?. Neuron, 73(3), 415-434.

[2] Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23), 8619-8624.

[3] Yamins, D. L., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3), 356-365.

[4] Kubilius, J., Schrimpf, M., Kar, K., Rajalingham, R., Hong, H., Majaj, N., ... & DiCarlo, J. J. (2019). Brain-like object recognition with high-performing shallow recurrent ANNs. Advances in neural information processing systems, 32.

[5] Margalit, E., Lee, H., Finzi, D., DiCarlo, J. J., Grill-Spector, K., & Yamins, D. L. (2024). A unifying framework for functional organization in early and higher ventral visual cortex. Neuron.

[6] Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D., & DiCarlo, J. J. (2020). Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations. Advances in Neural Information Processing Systems, 33, 13073-13087.

While the model performs well on the tasks tested, its generalizability and adaptability to other visual tasks remain uncertain

We are not trying to generalize to other visual tasks. Please remember that we are working within the framework of the ventral visual cortex and not a vision world model. Could you elaborate on what “other visual tasks” we might have missed that are performed by the ventral visual cortex? Processing dynamic visual scenes or object motion usually happens in the dorsal visual cortex.

synaptic degradation is modeled as a gradual, linear process

Synaptic degradation has not been modeled as a linear process. Degradation has been modeled as an asymptotic process where there is an exponential decay in the number of synaptic connections damaged with “time”. The progression of damage is highly variable between two real patients, meaning that we could never have modeled this process in a general way to understand brain function. Several previous works published in well-reputed neuroscience journals have used the same degradation protocol that we use here [1, 2, 3, 4], and we still observe biologically relevant behaviors in the model. We have provided the same argument in the discussion section of the updated paper.

[1] Tuladhar, A., Moore, J. A., Ismail, Z., & Forkert, N. D. (2021). Modeling neurodegeneration in silico with deep learning. Frontiers in Neuroinformatics, 15, 748370.

[2] Moore, J. A., Tuladhar, A., Ismail, Z., Mouches, P., Wilms, M., & Forkert, N. D. (2023). Dementia in convolutional neural networks: using deep learning models to simulate neurodegeneration of the visual system. Neuroinformatics, 21(1), 45-55.

[3] Moore, J. A., Tuladhar, A., Ismail, Z., & Forkert, N. In Silico Modelling of Neurodegeneration Using Deep Convolutional Neural Networks. In SVRHM 2021 Workshop@ NeurIPS.

[4] Moore, J. A., Wilms, M., Gutierrez, A., Ismail, Z., Fakhar, K., Hadaeghi, F., ... & Forkert, N. D. (2023). Simulation of neuroplasticity in a CNN-based in-silico model of neurodegeneration of the visual system. Frontiers in Computational Neuroscience, 17, 1274824.

We thank the reviewer for the discussion that they have brought up. We have tried to address all of them. Please also see the global rebuttal for common confusions around the intended purpose of the work. We reply to your concerns point-by-point below-

Presentation of results in Figure 5 is chaotic

Thank you for pointing this out. We have updated the figure and the associated section for better interpretability. None of the results we present in the paper are cherry-picked, and we present effective receptive field sizes and neural predictivity scores for all model layers and damaged regions in the supplementary now.

Section 4.5

In the earlier version of paper, we had mis-analyzed results for the difference in object recognition performance recovery due to supervised versus self-supervised retraining protocols. More specifically, the legend had been erroneously swapped for the models. Rest assured, we have fixed that error in the updated version of the paper, and revised the section to reflect that change. We apologize for this.

I find the idea of plotting performance against lesion iteration rather than total fraction of lesioned synapses a bit odd.

We have fixed the x-axis to address this. Thank you for the feedback.

I do not understand how the magnitude of performance is reaching nearly 80% in Figure 4D

We incorrectly divided the performance by 100 instead of a batch size of 128, which has been fixed to reflect in the figure now. Sorry for that.

Using topographic neural networks for modeling synaptic dysfunction

We agree that a topographic neural network provides for an interesting analysis. We now discuss this in the discussion section of the updated paper. We are the first, to the best of our knowledge, to mechanistically interpret DANNs under focal synaptic dysfunction. Consequently, this is an early work. There is no model out there that implements both topographic constraints and recurrent connections (intra-layer or long-range feedback). So, it was a matter of deciding what to focus on in this paper. We are committed to the cause, and we do plan on following up with this work. Nevertheless, we hope that the analyses we present here are meaningful to the computational neuroscience community, and to the neuroscience community at large.

The role of recurrent connections in functional compensation

We analyze the role that recurrent connections play, if any, in functional compensation in the updated version of section 4.3 of the paper. A key question that we answer there is – do we see functional compensation due to the presence of intra-layer recurrent connections along with neural plasticity, or does neural plasticity alone drive such reorganization. We do so by comparing CORnet-S with intact recurrent connections against purely feedforward DANNs, which still show such reorganization behaviors with neural plasticity.

Biologically, why is it more sensible to prune connections (as done here) rather than entire units?

That is a valid point. As discussed in the introduction, in this work we focus on synaptic dysfunction, one of the earliest consequences of focal damage to a region. Synapses are highly sensitive to changes in its environment caused by nutrient depletion (oxidative stress, excitotoxicity, etc.). Their damage is seen to occur well before cell death [1]. One cause of why this might be is due to the involvement of neurotransmitters like glutamate which are quickly impaired by metabolic deficits.

Verma, M., Lizama, B.N. & Chu, C.T. Excitotoxicity, calcium and mitochondria: a triad in synaptic neurodegeneration. Transl Neurodegener 11, 3 (2022). https://doi.org/10.1186/s40035-021-00278-7

We thank the reviewer for the discussion that they have brought up. We have tried to address all of them. Please also see the global rebuttal for common confusions around the intended purpose of the work. We reply to your concerns point-by-point below-

The differences between simulated synaptic degeneration and actual stroke progression in patients may limit direct applicability.

This work is not for releasing a specific model for clinical application. Please see the global rebuttal for the motivation behind the work.

The modeling process assumes random synapse pruning, which oversimplifies the actual patterns of synaptic loss

Again, please see the global rebuttal for a discussion on this. The progression of damage is highly variable between two real patients, meaning that we could never have modeled this process in a general way to understand brain function. Several previous works published in well-reputed neuroscience journals have used the same degradation protocol that we use here [1, 2, 3, 4], and we still observe biologically relevant behaviors in the model. We have provided the same argument in the discussion section of the updated paper.

[1] Tuladhar, A., Moore, J. A., Ismail, Z., & Forkert, N. D. (2021). Modeling neurodegeneration in silico with deep learning. Frontiers in Neuroinformatics, 15, 748370.

[2] Moore, J. A., Tuladhar, A., Ismail, Z., Mouches, P., Wilms, M., & Forkert, N. D. (2023). Dementia in convolutional neural networks: using deep learning models to simulate neurodegeneration of the visual system. Neuroinformatics, 21(1), 45-55.

[3] Moore, J. A., Tuladhar, A., Ismail, Z., & Forkert, N. In Silico Modelling of Neurodegeneration Using Deep Convolutional Neural Networks. In SVRHM 2021 Workshop@ NeurIPS.

[4] Moore, J. A., Wilms, M., Gutierrez, A., Ismail, Z., Fakhar, K., Hadaeghi, F., ... & Forkert, N. D. (2023). Simulation of neuroplasticity in a CNN-based in-silico model of neurodegeneration of the visual system. Frontiers in Computational Neuroscience, 17, 1274824.

The retraining protocols used in the study involve large numbers of images between lesion iterations

A discussion on the number of images used for retraining, and why we use that number, is presented in lines 317-323 of the paper.

Although the study shows contrastive learning to be effective over supervised learning, its practical use in clinical contexts could be challenging.

Please note that in the earlier version of paper, we had mis-analyzed results for the difference in object recognition performance recovery due to supervised versus self-supervised retraining protocols. More specifically, the legend had been erroneously swapped for the models. Rest assured, we have fixed that error in the updated version of the paper, and revised the section to reflect that change. We apologize for this.

Need validation through neurophysiological experiments

That is beyond the scope of the current paper. We make it very clear in both the introduction and the discussion sections that outside of emergent reorganization that has been clinically observed in both animals and humans, the hypotheses we make do not replace in-vivo testing. To confirm such would require us to run a large neurophysiology experiment. We are not macaque electrophysiologists, so we don’t have the competence or the resource to perform what would be one of the most difficult experiments in neuroscience. However, as previously stated, why we believe that our hypotheses are worth acknowledging is due to the emergence of certain behaviors that have been clinically seen before.

Can this framework be adapted to other sensory systems or cognitive functions affected by strokes?

Yes, absolutely. Given any computation model (trained to perform audition or decision making, or predict object motion, for example), an analysis of the form we have conducted here around neural predictivity, task performance, and population receptive field sizes, after careful translation to the applied domain, can be performed.

This is not true.

I am surprised and truly perplexed by the authors' response! The question I raised was about the mapping between models units and neurons/voxels. All the studies cited by the authors have used PLSregression (as on BrainScore) which is a linear-mapping strategy. And this is exactly my point. We know linear mapping can be used to predict brain data in monkey and neuron brains. Unit averaged responses (or uniform weighting) does not do as well at brain prediction. I haven't suggested in my review that the authors have analyzed behavior on a per-voxel basis. What I do say is that the authors have created lesions (strokes) based on sampling of model units. This is akin to uniform weighted sampling.

real monkey brain optimal perturbations that produce stronger responses

This response also does not address my question/concern. I am well aware of the studies the authors reference. Notably all of these utilized neural networks as models of inference (prediction ability on new stimuli) not learning (my original point). Predicting responses to new stimuli is importantly distinct from perturbing the brain (stroke) as they invoke fundamentally different mechanisms.

We thank the reviewer for the discussion that they have brought up. We have tried to address all of them. Please also see the global rebuttal for common confusions around the intended purpose of the work. We reply to your concerns point-by-point below-

DNNs tend to be most predictive when responses are represented as a linear combination of multiple model units, as single units alone do not capture brain responses as effectively. This study’s reasoning, however, seems to hinge on an assumed direct correspondence between individual model units and brain regions, as it involves lesioning units and model weights

This is not true. We never analyze behaviors demonstrated by the model on a per unit basis. DANNs themselves compute responses to visual stimuli as successive application of a linear combination of units within a model layer, followed by a nonlinear activation function. When we introduce synaptic activity decay to the model, it is introduced within a (or multiple) model layer(s). Analyses that follow include object recognition performance, capacity to predict real neural responses in different primate visual areas, and effective receptive field sizes, none of which are conducted on individual model units.

• Object recognition performance is computed by scoring the number of correctly predicted categories for each test stimulus from the features computed by the entire model. This is done through linear probing, which is a linear combination of all output model units and not a one-to-one map between single units and categories.
• Neural predictivity scores are computed over intermediate model layer activations and their ability to predict neural responses in a target brain region. The metric is the median over all individual neural recording predictivity values, and has been extensively used by prior works in comparing models to the brain (namely, through BrainScore [1, 2]).
• Receptive fields have again been computed at the population level (i.e., quantified for a model layer as opposed to individual model “neurons”).

[1] Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., ... & DiCarlo, J. J. (2018). Brain-score: Which artificial neural network for object recognition is most brain-like?. BioRxiv, 407007.

[2] Schrimpf, Martin, Jonas Kubilius, Michael J. Lee, N. Apurva Ratan Murty, Robert Ajemian, and James J. DiCarlo. "Integrative benchmarking to advance neurally mechanistic models of human intelligence." Neuron 108, no. 3 (2020): 413-423.

DNNs are primarily viewed as models of inference rather than mechanisms for experience-dependent learning and plasticity.

DANNs are not just used for data-driven modeling. Several well-cited papers within the computational neuroscience community have established the use of DANNs in designing and implementing in a real monkey brain optimal perturbations that produce stronger responses in various areas of the VVS than any previously known natural stimulus. This is necessary for us, because when we are perturbing different areas of the model, as we have done in this work, we want the responses to be good enough that they can be optimized to drive the brain. For example, Bashivan et al. [1] used a deep network to synthesize novel “controller” images based on the model’s implicit knowledge of how the VVS works to find that the model was capable of selectively controlling an entire neural subpopulation in subjects. Similarly, Walker et al. [2] trained an end-to-end, deep-learning-based model to synthesize optimal stimuli, which, for the mouse V1, drove neuronal responses in vivo significantly better than control stimuli. Thirdly, Ponce et al. [3] used a pre-trained deep generative neural network and a genetic algorithm to allow neuronal responses to guide the evolution of synthetic images that elicited large responses in various visual areas of macaques.

Since DANNs have been shown to be effective at predicting such responses, given that synaptic dysfunction is a natural in-brain perturbation, DANNs seem that they are likely to be effective at predicting those as well.

[1] Bashivan et al. “Neural population control via deep image synthesis”. In Science, 2019.

[2] Walker et al. “Inception loops discover what excites neurons most using deep predictive models”. In Nat Neurosci, 2019.

[3] Ponce et al. "Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences." In Cell, 2019.