A data and task-constrained mechanistic model of the mouse outer retina shows robustness to contrast variations

We thank the reviewer for their time in reviewing our manuscript. We appreciate that the reviewer considered our model “accurate but also functionally relevant” and our multi-stage training process “a more structured approach to model fitting.”

To address the reviewer’s first concern about the missing Tables 3 and 4, we apologize that those references were supposed to be to Tables 1 and 2. Table 1 was designed to list the different connectivity patterns and Table 2 was designed to show the performance of each connectivity pattern on the task. We will fix that.

Another concern of the reviewer was that our model is very detailed and thus computationally expensive. We acknowledge this point, and simplifications to the model could be made but at the expense of our scientific understanding of the processing. We are currently able to look at the role of each state variable throughout task performance, and we have created additional plots that provide further insight into the contrast invariant encoding. Specifically, we found that the photopigment and phosphodiesterase activity change greatly as the contrast moves further out of distribution, but that the cGMP concentration is quite robust, and then in turn the downstream calcium concentration, photocurrent and photoreceptor voltage. Such findings would not be possible in a simpler model.

To address the concern about our baseline in Figure 5, we trained and tested some other artificial architectures in addition. We rather refer to these alternative models as “reference models”, as we are not so much interested in their absolute performance, but rather e.g. in their capability to handle out of distribution contrasts. As the reviewer suggested, we tested a two layer MLP with a strong optimization routine but no layer normalization and a linear classifier preceded by layer normalization. Both of these architectures perform better outside of the contrast training distribution than the linear classifier; however, they come with their own set of drawbacks. A two-layer MLP, for example, without layer normalization but with a strong optimization procedure, requires far more trainable parameters by design to achieve higher accuracy. We chose the linear classifier in part because it has a similar number of trainable parameters to the biophysical model, 2,010 vs 2,000. We find that the use of layer normalization in addition to linear classification performs the best, not surprisingly, and the contrast-accuracy curve has a similar shape to that of the biophysical model. Thus, this is likely the computation implemented by our biophysical system, and we can now determine which components are responsible for it. We are happy to add these reference models to our figure for more context.

To address the reviewer’s questions:

1. The last sentence in Section 4.3 describes panel b of Figure 5 where the glutamate release of three photoreceptors is plotted. The glutamate release is plotted for the full contrast stimulus (yellow) for which the model is trained, and then the reduced contrast stimulus at the different colors. These colors match the colors of the line indicating the biophysical model in panel a. This is an illustrative figure showing that as contrast is reduced, some cones release more glutamate than at high contrast whereas others reduce less glutamate than at high contrast. What we should have mentioned, however, is that cones where the digit pixels are present (illuminated cones) tend to increase their glutamate at lower contrasts and cones where the digit pixels are not present (non-illuminated cones in the background) tend to reduce their glutamate release at lower contrasts. This is expected because greater illumination results in hyperpolarization and less glutamate release. These changes in glutamate release maintain the digit representation until contrasts of below 0.3 where the classification accuracy drops as shown in panel a.
2. This is closely related to your first question, and hopefully the previous discussion was helpful. The answer is basically yes, the photoreceptors are “filtering out the contrast” in their own way.

We hope that our additional experiments and modifications further increase the confidence of the reviewer in our paper.

We thank the reviewer for their detailed and valuable feedback. The reviewer found our manuscript to be “well written” and providing sufficient details for reproducing our findings. The reviewer also positively noted that we were able to “get around previous shortcomings of detailed biophysical models” and that “overcoming this problem could open up new possibilities for modelling and understanding biophysical processes.” We further appreciate that our model was called “exceptionally realistic” and that our proposed procedure for fitting the parameters was considered novel. Given this positive appraisal of our work, we were surprised by the low score.

The reviewer’s main critique was that our biophysical model did not achieve classification accuracy on par with artificial neural networks and that the biophysical model is more complex than it needs to be to solve the task. We would like to clarify that the main goal of our work is not to compete with ANNs, but to investigate how a biophysically realistic model of the outer retina could implement contrast invariant visual processing and what biophysical mechanisms are at play.

Biophysical systems are much more constrained than artificial architectures because information is transmitted by a variety of bounded physical quantities. We also used experimental data to fit most model parameters which further constrained the model. In this biophysical system, “simple operations” employed by artificial architectures must be implemented as the change of physical quantities, and this is not trivial. We are happy to make this perspective and motivation more clear in the introduction and discussion of our paper.

To provide more biological insights, we also plotted all of the state variables in the phototransduction cascade throughout the classification of out of distribution stimuli at different contrast levels. This figure shows that the photopigment and phosphodiesterase activity change greatly as the contrast moves further out of distribution, but that the cGMP concentration is quite robust, and then in turn the downstream calcium concentration, photocurrent and photoreceptor voltage. Our Fig. 5 currently only shows the glutamate traces as contrast moves further out of distribution, but we are happy to show additional state variables that make predictions about the biophysical mechanisms supporting contrast invariant encoding in even more detail.

The reviewer also cited a lack of strong baselines. While we consider the linear model as a point of reference rather than as a baseline for our model to beat in terms of absolute classification accuracy, we agree with the reviewer that additional reference models would help to contextualize the performance of the biophysical model. We thank the reviewer for this suggestion and have run additional experiments that we compare to in the revised version of the manuscript. The additional non-biophysical reference models we explored for this rebuttal include a two-layer MLP, layer normalization followed by linear classification, and adaptive and static thresholding followed by linear classification.

Interestingly, a two-layer MLP achieves much better classification performance for all contrasts than our model, but it uses far more trainable parameters by design to achieve this. Furthermore, the computation this network implements is quite opaque, and the mechanism by which it achieves this feat is unclear.

We find that the use of layer normalization in addition to linear classification performs the best, with a contrast-accuracy curve similar to that of our model. While it provides an explicit description of the computation, it does not provide insights into how such a computation could be implemented by the outer retina, as our model does.

Using static thresholding plus linear classification does not work for classifying out of distribution stimuli in this case because the models are trained at full contrast, and thus the threshold will be too high or too low for all of the other contrast levels depending on where it is initialized. That is, there is no proper way to learn a static threshold that classifies the stimuli well outside of the training distribution. An adaptive threshold, however, that is multiplied by the maximum stimulus intensity, performs almost the same as layer normalization with linear classification and thus implements a similar computation, but likewise does not provide insights into the biophysical implementation of such a computation.

Thus, in summary, the rationale of our revised paper is:

1. We hope that our previous discussion about the biophysical realism of our system convinces the reviewer that seemingly simple computational operations are not actually so simple in constrained biophysical systems and our model helps to solve that question. In addition, it is important to note that our model considers time, and this is critical to our testing of the hypothesis that horizontal cells contribute to task performance, because their feedback would be required on an adequate timescale. While this makes our model suitable for answering a host of other interesting questions, in the context of this task, assessing the role of the different cell types is not possible without the notion of time.

2. As a reference model, we used a linear classifier to see if our biophysical model simply implements linear functions. We found that the linear model performs very well in its training distribution (87% when trained at varying contrasts), but its performance degrades much more quickly as contrast moves outside of the training distribution than for our biophysical model. We thus concluded that the biophysical mechanisms implemented in our model underlie this ability to generalize.

3. Prompted by the reviewer, we used other reference models and found that the linear model does perform much better when preceded by a simple adaptive thresholding operation on out-of-distribution stimuli, so we suppose that this is the sort of operation that the biophysical model implements, though in a biophysically plausible way with real physical quantities. We are happy to include this in our figure and discussion in the paper.

We are grateful to the reviewer for their feedback and hope that they will indeed raise their score as mentioned in the review in light of these clarifications and forthcoming additions to the paper.

We thank the reviewer for the detailed assessment of our work and for their valuable feedback. They found our work to be “novel”, inviting to “community extension” and a “technically impressive advance” in the field.

The reviewer raises an interesting point about stochasticity as discussed by Cottaris et al. 2020. We did include a citation to this paper in our discussion section and discussed how this was a limitation of our current model. However, we would like to reiterate that our model of photoreceptor activity is already highly detailed and the photocurrents are trained on actual data with a noisy stimulus. We also note that most biophysical models do not consider noise, likely because the mathematical form of realistic noise in these systems is difficult to define. This is an interesting problem for future work.

The second major concern of the reviewer was the equation of our ionotropic synapse model. This was an unfortunate typing error in our manuscript. The model synapse does in fact correctly implement the synapse as described in Abbott & Marder 1998, with $\tau = (1-s_{\infty}) / k_-$. We thank the reviewer for pointing this out and we will correct this in the text.

To address the reviewers questions:

• The reviewer asked for clarifications about Figure 1: The photocurrent y-axis is the patch-clamp measured photocurrent measured in pA but normalized by the dark current and the glutamate y-axis is the change in fluorescence of the experimental glutamate indicator from the baseline fluorescence. The green and magenta images show the images shown to the region of interest on the retina in the experiment where the data was collected. The luminance levels of the MNIST squares are converted to photoreceptor stimulus intensities in P*/s, and the model photoreceptors each have an (x, y) location that indicates what pixel of the digit they see. We show only a single horizontal cell in the diagram because this is one of the architectures we train, and showing more horizontal cells would look a bit messy. The BC readout traces are simply example voltage traces of arbitrary size and timescale, depicted only to sketch the task setup. We did indeed not reference this figure in the manuscript and thank the reviewer for pointing this out to us. We will clarify this in the main text of the paper and add references to Fig. 1.
• Regarding equations, we will add the general form of the gating variable equations to the main text. They differ slightly depending on the ion channel, but their general form is $\frac{dm}{dt}= \alpha(V)(1-m) - \beta(V)m$. We will also add the ion channel equations to the supplementary materials. We also thank the reviewer for pointing out several small formatting errors which we will of course fix in the revised manuscript.
• Local horizontal cell connections refers to an architecture where each horizontal cell is connected to its 50 nearest cones. This means that cones have the opportunity to influence one another’s activity via horizontal cells, likely being close to what is observed in the real retina. The one-to-one connectivity pattern is designed to model a case in which each cone photoreceptor gets feedback from an electrically isolated segment of horizontal cell. In this architecture, cones still receive horizontal feedback, but they are not able to influence one another via the horizontal cells. We will clarify this further in the text.
• An intuitive explanation for why horizontal cells could contribute to contrast invariant encodings is that they are positioned to provide photoreceptors with lateral feedback. Lateral feedback is in reference to the case where a strongly excited cell inhibits its neighbors, for example by exciting an inhibitory interneuron. This action can enhance contrast by creating a larger difference between the activity of the two cells. Such inhibition was observed in wet-lab experiments with photoreceptors in the 1950s and 60s; however, other findings since have conflicted our understanding of what horizontal cells do. A potential intuitive explanation why horizontal cells would not contribute to contrast invariant encoding is that highly stimulated photoreceptors themselves perform enough local luminance adaptation so that they do not elicit enough horizontal cell activity to inhibit their neighbors. We are happy to add a summary of these explanations to the abstract.

We agree with the reviewer that more complex and natural stimuli will be an interesting direction for future work. We would like to emphasize, however, that our work is one of the first to train highly-detailed biophysical models on any task such as contrast-invariant classification and our contrast-modulated MNIST provides a reasonable test case for this.

Using more complex stimuli is, however, high on our list of things to study. Because we trained our photoreceptor models on stimuli with color, we have estimated how sensitive each cone is to green and UV light (the two opsins present in mouse cones). Thus, we could also study the role of different cone types in different retinal regions in pattern discrimination. In the current paper, we restrict ourselves to the outer retina’s processing of stimuli with varying contrast and luminance.

To another point of the reviewer, we are happy to provide more discussion of previous models in the discussion section. None of the models in previous work were both biophysically detailed and made predictions about cellular mechanisms responsible for the contrast invariant encoding of visual information. Baek et al. developed a model with similar detail to ours, but its ability to perform any sort of computation was not tested (2020). Another retina simulator developed by Wohrer and Kornprobst used a phenomenological model of the outer retina. This model includes a baked-in mechanism for contrast gain control, but it cannot be used to study what existing biophysical mechanisms might be responsible for such a computation (2009). We can continue to provide detail.

We thank the reviewer for their valuable feedback and kindly ask them to raise their score given our clarifications, additional experiments and modifications to our paper.

I have another follow-up question because I saw feedback from other reviewers to the authors. Even though I raised the score, I hope the authors can continue to respond.

The one-to-one connectivity pattern is designed to model a case in which each cone photoreceptor gets feedback from an electrically isolated segment of horizontal cell. In this architecture, cones still receive horizontal feedback, but they are not able to influence one another via the horizontal cells.

Does the number of inhibitory neurons impact the response of excitatory neurons to contrast? The network features both lateral inhibition and lateral facilitation. Your rebuttal covers lateral inhibition. I would like to hear from the authors.

We thank the reviewer for taking the time to review our paper and provide helpful feedback. We are encouraged that the reviewer called our framework “creative,” our model “detailed” and “accurate,” and our paper “well-motivated.”

A main concern of the reviewer is that we do not include more simple controls. For testing out of distribution performance, we have now tested a variety of other linear and nonlinear architectures for comparison. Specifically, we have tested the linear classifier with layer normalization (which performs extremely well out of the training distribution, as expected), a two layer MLP which also performs well out of its training distribution but with a much greater number of trained parameters, the initialized biophysical model as suggested by the reviewer (see answer to Q1 below), and a couple of other references suggested by other reviewers. These experiments have shown us that our biophysical model could be implementing the computations of a linear model with layer normalization, but our biophysical detail allows us to provide an account of how this computation is implemented with the cell’s molecular components.

Similar to these artificial architectures, models such as the one proposed by Clark et al. are phenomenological models, and thus target a different level of understanding of the biophysical system. While phenomenological models can provide interesting insights, they do not provide intermediate steps of computation that can be validated as our model does. To highlight this, we have also plotted the state variables of the phototransduction cascade when performing the classification task on out of distribution stimuli. We are happy to include this as a supplementary figure for our paper, or potentially in the main text if space allows. This figure shows that the photopigment and phosphodiesterase activity change greatly as the contrast moves further out of distribution, but that the cGMP concentration is quite robust, and then in turn the downstream calcium concentration, photocurrent and photoreceptor voltage. Our Fig. 5 currently only shows the glutamate traces as contrast moves further out of distribution, but we are happy to show additional state variables that make predictions about the biophysical mechanisms supporting contrast invariant encoding in even more detail.

The reviewer also voiced concern that our classification accuracy is not high. While reasonable performance is important, the goal of our paper is rather to understand how a biophysically realistic system could implement contrast invariant visual processing and what quantities are most decisive. Biophysical systems are much more constrained than artificial architectures because information is transmitted by a variety of bounded physical quantities. The biophysical model without the horizontal cells also does not have more trainable parameters for learning the task than the linear classifier, it has 2,000 weights while the linear model has 2,010, and the biophysical model with its initialization parameters only achieves chance accuracy at any contrast level. We have, however, included other reference models as previously mentioned which provide guidance on what computations need to be implemented to solve the task.

Finally, the reviewer notes a limitation that MNIST is not naturalistic. We would like to emphasize, however, that our work is one of the first to train highly-detailed biophysical models on any task such as contrast-invariant classification and our contrast-modulated MNIST provides a reasonable test case for this. We are very excited to see in future work if our results still hold when processing other images and videos.

To address the questions of the reviewer:

1. To assure the reviewer that the out of distribution performance is not due to model architecture, we see that the model only achieves chance performance on the task at initialization, even for stimuli within the training distribution. The model weights are initialized from normal distributions centered at values chosen to promote stable dynamics throughout training. To answer the subquestion of task performance if the models are trained on a variety of contrasts: We do train our biophysical models on contrasts chosen randomly in Tab. 2. When training the linear model this way, it actually achieves a high test accuracy of 87%. The linear model is able to achieve high classification accuracy on data within its training distribution.
2. The reviewer asks about more analysis on the temporal nature of contrast and luminance adaptation: We think this is a great idea! The linear model we test actually has no concept of time; it simply takes a snapshot at the peak stimulus intensity and uses that for classification, so we would have to reformulate this reference model. However, for this rebuttal, we did briefly test whether our trained biophysical model would maintain its test accuracy on stimuli with contrast fluctuations throughout the stimulation period. The test accuracy in this case only reduced to about 48% (without any additional training). Future work could extend this analysis.
3. The reviewer suggests a “preliminary attempt to understand an aspect of the biophysical model of the cones.” We thank the reviewer for this suggestion, and we refer them to our previous discussion of the other state variables in our model, a plot that we can add to the paper.
4. We will be more explicit that our goal is to obtain biological insights into contrast-invariant visual encodings while adding our comparison to layer normalization and other architectures as discussed previously.
5. A major goal of this paper is to enhance appreciation for the computational power of the outer retina which extends beyond the linear filtering of stimuli. Horizontal cells are positioned to provide lateral feedback to photoreceptors, a phenomenon first observed in the 1950s and 60s in Nobel prize winning work by H. K. Hartline and colleagues. Lateral feedback is in reference to the case where a strongly excited cell inhibits its neighbors, for example by exciting an inhibitory interneuron. This action can enhance contrast by creating a larger difference between the activity of the two cells. Such inhibition was observed in the photoreceptors of Hartline’s horseshoe crab and in other species since; however, other findings have conflicted our understanding of what horizontal cells do. For example, Fred Rieke wrote in 2001 that contrast adaptation was likely to first occur in bipolar cell dendrites after showing that a model of horizontal cell activity did not change at different contrast levels but the bipolar cell model did. The suppression of horizontal cell input to bipolar cells also did not change contrast adaptation in this study. We only briefly discussed this history and motivation in our introduction, and we would be happy to add more information.

We hope that the additional clarifications and experiments increase the confidence of the reviewer in our paper and potentially allows them to raise their score.