Model Based Inference of Synaptic Plasticity Rules

Dear Reviewer,

Thank you for your feedback and your recognition of the interesting aspects of our paper. Regarding your concerns:

1. Machine learning versus neuroscience: We think that a machine learning conference proceeding is a better fit for our work than a neuroscience paper for two main reasons. First, typical neuroscience papers ask an interesting scientific question and then answer it through some combination of method development, experimentation, data analysis, and modeling. Answering a scientific question is not the point of this work. Instead, we aim to introduce a powerful general methodology and illustrate it with proof-of-concept applications. When neuroscientists write a paper like this, they usually publish it as a methods paper, and ours is a machine learning method that fits a machine learning conference perfectly. It is common for papers like ours to be published in machine learning conferences, as evidenced by the many machine learning conference papers that we cite in our paper. Second, our paper introduces a methodology for understanding learning in neural networks that, while demonstrated within a biological context, has broader applications. The principles and techniques we outline are relevant not only to neuroscience but also to the field of artificial neural networks (ANNs) and deep learning. For instance, our method could be used to learn a simple parameterized learning rule that approximates the learning trajectories of a more computationally intensive method. This interdisciplinary relevance makes our work a good fit for the conference, as it bridges the gap between biological neural mechanisms and computational models in machine learning. In summary, we believe that our work contributes significantly to the understanding of learning in neural networks, with implications that span across neuroscience and artificial intelligence. We are open to further discussions and clarifications on how this aligns well with the conference scope.

2. Choice of Oja's rule: The use of Oja's rule in our study is primarily for validation purposes. It serves as an arbitrary, yet well established, benchmark to demonstrate the efficacy of our methodology. Our choice of Oja's rule was driven by its recognizability and established theoretical foundations, making it a suitable candidate for ground truth comparison. However, it's important to note that our methodology is not restricted to Oja's rule; it can be applied to infer a wide range of synaptic plasticity rules. For example, the alternatives in Table 1 were not derived from any theoretical objective, they were merely many examples of learning rules that could be used to generate neural trajectories, and our method was consistently able to recover them. We do not think that it is necessary to use symbolic regression, or any other method, to find the optimal plasticity rule. Although it is interesting to relate the learning rules of single neurons to the output behavior of the whole system, that is not the topic of our paper. Instead, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model.

3. Generalized learning forms and network performance: Your suggestion to model a more generalized learning form and benchmark its performance in networks like a Hopfield network is a good idea. However, that is not the topic of this paper, which you already recognize as “really interesting.” Our current focus is on developing and validating a method for inferring learning rules, rather than on enhancing or modifying specific learning methods like Oja's rule. That our paper suggests another interesting computational problem to the reviewer lends further credence to our view that a machine learning conference proceeding fits our work well.

I would like to thank the authors for their feedback. I like the paper. However, I still don't fully understand the significance. The authors mentioned, "our method could be used to learn a simple parameterized learning rule that approximates the learning trajectories of a more computationally intensive method" and "our methodology is not restricted to Oja's rule; it can be applied to infer a wide range of synaptic plasticity rules" - I believe it is necessary to show these cases in this paper. Otherwise, I am not very convinced regarding the substantiality of the novelty of the current manuscript.

For this reason, I will stick to my original score. Thank you, and good luck!

Replies to points brought up in Questions section:

B1. Our method is simpler to implement and appears to work perfectly well for our purposes. We therefore did not explore the CMA-ES approach. We note that Tyulmankov et al. 2022 also successfully used gradient descent to meta-learn plasticity rules, and our implementation more closely follows that paper.

B2. This is a good suggestion. We did not explore this learning rule because it involves significantly more coefficients that must be fit, and hypothetical dependencies of the mushroom body’s learning rule on y can be excluded on biological grounds. We will mention this in our revision. We will also try fitting this more complex learning rule in our revision to see whether the coefficients can still be recovered.

Dear Reviewer,

We thank the Reviewer for their positive assessments of our work and suggestions for improvement. As detailed below, we will make many changes to address your suggestions.

Replies to points brought up in Weaknesses section:

B1. We agree. We will modify this sentence to be stronger.

B2. Yes. Thank you for catching this typo. We will correct it.

B3. Yes. Thank you for catching this typo. We will correct it.

B4. The coefficients were randomly initialized i.i.d. from a normal distribution of zero mean and variance 1e-4. We will add this methodological detail to the paper.

B5. We agree with the Reviewer that it would be interesting to extend our method to recurrent neural networks. However, we could perform this numerical experiment in our setup without changing Eq. 6. In particular, x would become the presynaptic neuron’s activity, y would become the postsynaptic neuron’s activity, and w would be the weight between these neurons. The bigger difference is that Eq. 1 would need to be updated to incorporate recurrent network dynamics. We have not experimented with recurrence in our model, and this change would entail significant changes to our code bank. We have therefore decided to postpone these experiments for future work.

B6. We accepted the Reviewer’s suggestion and summarize additional experiments that varied the architecture of the MLP below. We found that the performance of the MLP architecture could be significantly improved by moderately increasing the size of the hidden layer. As suggested, we will include these simulations in our revised manuscript.

Simulation information and results

Plasticity MLP architecture: 3 → h_l → 1

Network: 100 → 1000 → 1

Ground truth plasticity rule: x.(R - E[R])

Trained on simulated behavioral data

B7. We will revise the manuscript to state these details. For the experiments, hyperparameters were fine tuned by grid search, with performance averaged across three random seeds. Hyperparameters included an L1 sparsity regularizer applied to the plasticity coefficients, with the optimal value found to be 5e-3, and a moving average window for calculating the expected reward, which showed minimal impact and was set at a value of 10. In terms of optimizers, our experiments included AdaBelief, Adam, and AdamW, using their default parameters for each. The results were comparable across these optimizers, and the findings presented in the paper are based on the use of Adam.

B8. We agree with the reviewer, but we think that the choice of optimization method is highly significant. As explained in the Introduction and Related Work sections, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model. The method provided here provides an exciting opportunity to infer biological learning rules from real experimental data sets from biology, where the function of the circuit is a priori unknown. For instance, no one knows if the mushroom body is optimal for some specific task, but we were nevertheless able to estimate its learning rule from behavioral data in Figure 4.

B9. Can the Reviewer please clarify what comparison they would like to see? We already conceptually compared our work to previous papers in the Related Work section. However, since we’re optimizing different objective functions than almost all of these previous works, apples-to-apples comparisons are usually impossible. In the setting of Figure 4, we could revise the text to compare our model’s performance to the previous model in Rajagopalan et al. Would this help, or does the Reviewer have something else in mind?

Sloppiness in fit and interpretability/reliability

Although the sloppiness of fit is a real problem in our models, we again think that it is a generic problem in statistical model fitting rather than a limitation in our specific approach. Interpretability depends on the functional form of the model (or equivalently, in the case of probabilistic graphical models, its graph structure). We demonstrate our method with an interpretable plasticity model in the form of a Taylor expansion, as well as a less interpretable one in the form of an MLP. The reliability of the method, as with all statistical models, depends on the quality and the quantity of the data. Standard statistical techniques can be used in these cases – confidence intervals, bootstrapping, cross-validation, etc. Some hypotheses, however, by definition cannot be distinguished with a particular dataset.

We would also like to highlight the fact that the sloppiness occurs in the specific parameterization of our plasticity rule, rather than in a degeneracy of the rule itself (unlike the results in [2], which, again, we will discuss in the final version of our submission). We find a unique plasticity function, i.e., the shape of the input-output curve on a graph, but not a unique parametrization of that function.As a trivial example, the functions $y = x$ and $y = x^2$ are indistinguishable if the only values of $x$ which we sample are 0 and 1, so if the ground truth is $y = x + x^2$, we can fit any function $ax + bx^2$ with $a + b = 1$.

Nevertheless, regardless of the parametrization of the plasticity rule, our method is informative with regard to the inputs that are relevant for a plasticity update, e.g. whether pre- or post-synaptic activity plays a role, or both; or, as we demonstrate in Drosophila, whether a weight decay is present. Following the adage, “All models are wrong but some are useful,” we believe that our model is useful from this perspective.

No a priori hypotheses

The problem of hypothesis generation is an important and difficult one. Any fitting experiment is limited to the functional family that is being fitted. Since the goal of our work is introducing and demonstrating a new and general computational methodology, not on establishing and interpreting a novel biological finding, we simply chose a hypothesis using prior knowledge established in other work. We demonstrate that our method is reliable for a given hypothesis class. Generating a hypothesis (i.e. the functional family as well as choosing the appropriate inputs) can be done through what has been referred to as "Box's loop" [3]. We iterate on hypotheses and do model validation for each one until we reach satisfactory convergence (or a deadline).

In an extreme case, given sufficient data and computational power, we can simply consider a sufficiently broad hypothesis which includes all the possible inputs and allow the optimization routine to decide which ones should be used and in what way. If the functional family is an interpretable one, such as the Taylor series, we can gain mechanistic insights into the plasticity rule itself. For instance, consider

$$g_\theta = \sum_{\alpha,\beta,\Phi,\gamma} \theta_{\alpha,\beta,\Phi,\gamma} x_j^\alpha r^\beta y_i^\Phi w_{ij}^\gamma$$

We did not explore this learning rule because it involves significantly more coefficients that must be fit, and hypothetical dependencies of the mushroom body’s learning rule on y can be excluded on biological grounds. To address the reviewers concern, we will try fitting this more complex learning rule in our revision to see whether the coefficients can still be recovered.

In a fully general approach, we can even consider a dynamic plasticity model (e.g. an RNN) that, given all possible inputs to a synapse, can learn history-dependent terms such as expected reward (which, this time, we manually included in our function). Although such a model (or even a simpler one such as the MLP) is hard to interpret, it still gives us an input-output relationship that we can use to make predictions. It also allows us to explore which inputs are important e.g. by including/excluding reward as one of the inputs into the MLP. Moreover, it gives us a well-defined functional form of a putative plasticity rule that we can perturb and analyze in ways unavailable in a biological system. Since the input space is low-dimensional, for example, we can visualize the function’s graph.

[1] Sjostrom and Gerstner. Spike-timing dependent plasticity. Scholarpedia, http://www.scholarpedia.org/article/Spike-timing_dependent_plasticity#STDP_versus_Rate_based_learning_rules

[2] Ramesh et al. Indistinguishable network dynamics can emerge from unalike plasticity rules. bioRxiv, https://www.biorxiv.org/content/10.1101/2023.11.01.565168v1.full.pdf

[3] Blei. Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models. https://www.cs.columbia.edu/~blei/papers/Blei2014b.pdf

Dear Reviewer,

Thank you for your comments on our work. We address each of the suggested weaknesses/questions below. To summarize, while we agree with the highlighted limitations due to robustness (noise/sloppiness), interpretability, and lack of prior hypotheses, these concerns are applicable to any statistical model-fitting setup. Some of these have standard solutions, but in general they are ubiquitous problems that are outside the scope of our work. We also point out that our method indeed can account for spiking models, and the plasticity rules that we fit are, by definition, biologically plausible.

Spiking neural network models

Although our approach uses rate networks as the underlying neural network model, our approach is not limited to rate networks. In the case of stochastic spiking models, we demonstrate an analogous scenario by fitting fly behavioral data with a probabilistic model, maximizing the log-likelihood of the model’s output given the data. This can easily be applied to fitting spiking (e.g. Poisson) neural trajectories rather than stochastic behavioral trajectories.

Alternatively, we can think of rate-based neural networks as an abstraction over spiking networks. Given spiking data, we can generate a rate code by smoothing e.g. with a Gaussian kernel, which we can fit immediately with a rate model. Furthermore, STDP rules can be mapped onto Hebbian plasticity in rate networks [1], so rate networks do not preclude STDP as a potential plasticity mechanism (although we do concede that the precise timing of the spikes is lost).

Our approach can also be easily modified to account for spiking data without any preprocessing, using a spiking model. As presented in the current work, the only limitation is the optimization algorithm. If the spiking model is an Hodgkin-Huxley type model with differentiable spike waveforms, we can use our method out-of-the-box – the only limitation is compute power, since these models have extremely detailed dynamics and many CPU cycles are required to simulate a single spike. If the spiking model is comprised of binary neurons with a Heaviside step function nonlinearity, optimization techniques such as surrogate gradients can be used. In the case of non-differentiable models, e.g. integrate-and-fire neurons, we can use evolutionary algorithms.

Unknown noise sources and biological plausibility

We agree that the noise model and loss function would impact the results of the model fitting. However, this is a problem that is true for any statistical modeling, and therefore much more general than the scope of our work. The Gaussian noise assumption is a common one and is used as the noise model ubiquitously. More importantly, Gaussian noise is also the most likely, assuming the Central Limit Theorem holds, and therefore it is a valid assumption. Similarly, mean-squared error is the standard loss function for many statistical models of real-valued data, and cross-entropy for models of binary data.

If, however, there is strong prior belief that the noise is non-Gaussian, e.g. if there are correlations, bias, skew, or heavy tails in the noise model, this can be factored into the loss function appropriately. For completeness, we also note that after our initial submission, a preprint was released that directly tackles this problem by using a generative adversarial network to fit neural trajectories rather than an explicit loss function [2]. Although this approach is more general from the perspective of the noise model, it is both more demanding computationally, and a more difficult loss landscape to optimize due to the use of a GAN. Although we were unable to include this work in the initial submission, we will make this comparison in our final version.

Finally, the concern about the biological plausibility of the inferred plasticity rules in complex neural behaviors or trajectories is an important one. We would like to emphasize that our model infers plasticity rules using only local information, such as presynaptic and postsynaptic activity, current synaptic weight, and global dopaminergic reward signals. This, by definition, constrains the plasticity rules to be biologically plausible.

I thank the authors for their detailed responses to my questions.

Regarding the point on spiking neural network models, I agree that evolutionary algorithms or MLE estimation might indeed be a useful way to extend the framework spiking network models. However, I am still worried about the framework's generalizability, particularly when also accounting for non-Gaussian noise:

If, however, there is strong prior belief that the noise is non-Gaussian, e.g. if there are correlations, bias, skew, or heavy tails in the noise model, this can be factored into the loss function appropriately.

Factoring these terms into the loss function might make MLE or evolutionary methods non-trivial to implement and train to convergence. In particular, while the assumption of Gaussian noise / central limit theorem might be ubiquitous, it is not uncommon to find correlated neural trajectories (Smith and Kohn 2005, Bartolo et al 2020, Valente et al 2021) or other shifts away from purely Gaussian noise. Given these difficulties, I am still skeptical about the generalizability of the framework.

Regarding the sloppiness of fit:

We would also like to highlight the fact that the sloppiness occurs in the specific parameterization of our plasticity rule, rather than in a degeneracy of the rule itself

I am confused by this -- could the authors elaborate on this point? Following from the example in the response, wouldn't the ambiguity in the values of $a$ and $b$ qualify as degeneracy? In any case, it would be useful to include this point in the discussion in the manuscript.

Regarding a-priori hypothesis:

we will try fitting this more complex learning rule in our revision to see whether the coefficients can still be recovered.

This would be a great addition. Comparisons/Discussion of Ramesh et al or Rajagopalan et al (as pointed out by reviewer s75f) would certainly be useful.

I recognize that this may be difficult to achieve within the time-frame of discussion period. I have increased my score to a 5, on the understanding that these comparisons / discussions will be included in subsequent revisions of the paper if accepted.

However, I would like to note that my concerns remain on the generalizability as well as limited novelty of this work.

Dear Reviewer,

Thank you for your insightful comments and constructive feedback on our manuscript. We appreciate the opportunity to clarify and expand upon key aspects of our work. Below, we address each of your concerns in detail.

Methodological Choices and Theoretical Grounding

Our decision to include multiple output neurons in our model is primarily to demonstrate scalability. The model effectively functions with a single output neuron, but the use of multiple neurons showcases its applicability to larger, more complex networks. This is not a trivial problem as the loss is computed over the entire trajectory length and parameters are optimized with backpropagation through time. In the context of the behavioral task, the outputs are pooled through a fixed layer followed by a sigmoid function, determining the probability of accept/reject decisions. It's noteworthy that these output neurons typically follow unique trajectories. The convergence to the same outputs in the case of Oja’s rule is a specific instance, not a general trend that holds across learning rules. This aspect also underlines the versatility of our model in fitting diverse rules.

We will revise the manuscript to mention the link between Oja’s rule and principal component analysis, to state that the theoretical motivation for including the E[R] term is to enable covariance-based learning that leads to operant matching, and to explain that the weight decay term could explain forgetting. However, we do not think that it is necessary to provide a theoretical rationale for the other learning rules that we consider. Instead, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model. Although we agree that it is interesting to relate the learning rules of single neurons to the output behavior of the whole system, that is not the topic of our paper.

For the E[R] term, our experiments encompassed various time scales (5, 7, 10, 15, 20) trials, leading to consistent results across these variations. This uniformity in outcomes led us to exclude it as a learnable parameter in our optimization loop. Dynamics within the trial had no bearing on E[R].

Modelling Synaptic and System-Level Behavior: Recent research, particularly by Aso et al., has demonstrated that activating a specific neuron in the fly mushroom body can influence the fly's upwind movement. This finding is crucial to our experimental setup, where odors flow and the fly's reward-driven upwind movement is a key behavior. This validates our approach of modeling single neurons, as it aligns with the observed behavior in real-world scenarios. Our approach also mimics the setting of a previous biology study by Rajagopalan et al. that we wanted to extend with our methodology. We will modify the text to explain these points and specify the exact neuron referenced in this context for clarity.

Other work from Aso and collaborators has quantified how olfactory memories decay over time. In light of your comments, our revision will include an analysis of how the inferred learning rule leads weights to decay over trials, thereby leading to memory decay. We will compare the inferred learning rule to plasticity mechanisms discussed in Aso’s studies.

We parameterized the learning rule with an MLP to see whether we could capture the learning rules with a general expressive model that did not match the ground-truth generative model. However, Figure 3 and Table 1 revealed that the MLP usually underperformed the Taylor series representation. We therefore decided to focus Figure 4 on the Taylor series representation, which is both more interpretable and better aligned to the known biology. We will explain these motivations and findings in the revised manuscript. For similar reasons, we did not attempt to quantify what the MLP rules learned in Figure 3. If the MLP approach had been more empirically successful, then we would have interpreted the learned rules by directly plotting the three-dimensional function of presynaptic activity, reward, and weight that they discovered. This could have easily been compared to the exact function specified by the ground-truth generative model.

We thank the Reviwer for the follow-up questions and concerns. We address each of them below. With these clarifications and planned changes, we believe that we will have been able to address all of the reviewer’s major concerns. Please let us know if there are any lingering issues that prevent the reviewer from considering updating their overall rating.

Our main contribution is indeed optimization over a trajectory with backpropagation through time (BPTT). To be precise, however, we still optimize a loss function, although it is defined as the error between the data and model trajectories, rather than a functional relationship as in previous work [Confavreaux et al. 2021, Tyulmankov et al. 2022]. To optimize this loss function we use BPTT, which was implicit in the definition of the loss function (Eq. 3) and its derivative (Eq. 4) due to the time dependency. We agree that this should be made more clear and explicit, and we will update the text accordingly.

It might be helpful to distinguish between the optimization problem and the used optimization technique to solve it. Our approach is fundamentally different from alternative setups in terms of the optimization problem. We provide a discussion and comparison in the "Related Work" section. The only other work we are aware of that uses neural trajectories to fit a plasticity rule is [Ramesh et al. 2023], which was only published to bioRxiv after our initial submission. We will include a qualitative comparison in the final version of our manuscript nevertheless. In terms of comparison to other optimization techniques, the choice of optimizer is an implementation detail that we could revisit in future work. Indeed, alternatives for computing gradients such as real-time recurrent learning (RTRL) can be used, or approximations to the gradient such as truncated BPTT, or even non-gradient based methods such as evolutionary algorithms.

However, any approximations (whether approximations to the gradient or using only endpoints in the loss) will be strictly less data-efficient (although may be more computationally- and memory-efficient), since we would be ignoring long-term dependencies between the parameters and data. As such, we opted for the most powerful methodology because we wanted it to work for arbitrary rules, some of which might be highly nonlinear and benefit from long time dependencies more than others (see below).

The reviewer asked several related questions. Our responses are:

• Output neurons are uniformly summed (weights equal to 1), and a logistic sigmoid function is applied to that sum to generate a probability of choosing the current odor.
• Our discussion of “scalability” refers to our implementation’s ability to handle large networks, rather than any comment about data efficiency.
• In the general case, learning rules define nonlinear dynamical systems that can depend sensitively on their initial conditions as well as the particular noise instantiation. Because of this, the output neurons do not necessarily follow identical trajectories – the pooled behavioral readout is then a linear combination of their individual (unique) outputs. As such, additional output neurons do offer additional information about the plasticity rule beyond simple data availability. However, as with Oja’s rule (which learns weights that align with the first principal component), it may be the case that output neurons behave similarly. Nevertheless, Table 1 shows that the method works for many different learning rules beyond Oja’s rule, and it is inaccurate to say that we’re generalizing because our method “works for a specific case.”

The problem of hypothesis generation is indeed an important and difficult one. Here, we simply choose a hypothesis using prior knowledge established in other work. We demonstrate that our method is reliable for a given hypothesis class. Generating a hypothesis (i.e. the functional family as well as choosing the appropriate inputs) can be done through what has been referred to as "Box's loop" [Blei, 2014]. We iterate on hypotheses and do model validation for each one until we reach satisfactory convergence (or a deadline).

We do not claim that plasticity rules can be inferred without any prior knowledge. The most important prior knowledge that underlies this work is that plasticity rules in biology can only local information, such as presynaptic and postsynaptic activity, current synaptic weight, and global dopaminergic reward signals. This, by definition, constrains the plasticity rules that we fit to be biologically plausible. We note that this prior knowledge is far less restrictive than prior knowledge of the circuit’s computation, which was assumed in the previous work of Confavreaux et al. and Tyulmankov et al. Given sufficient data and computational power, one could consider a sufficiently broad hypothesis which includes all the possible inputs and allow the optimization routine to decide which ones should be used and in what way. For instance, consider

$$g_\theta = \sum_{\alpha,\beta,\Phi,\gamma} \theta_{\alpha,\beta,\Phi,\gamma} x_j^\alpha r^\beta y_i^\Phi w_{ij}^\gamma$$

We did not explore this learning rule in our submission because it involves significantly more coefficients that must be fit, and hypothetical dependencies of the mushroom body’s learning rule on $y$ can be excluded on biological grounds. However, to address the reviewer’s concerns, we will try fitting this more complex learning rule in our revision to see whether the coefficients can still be recovered.