Feedback control guides credit assignment in recurrent neural networks

Thank you for taking the time to review our work.

My main concern is that I the task chosen consists on bringing a system to a desired static target...

Thank you for flagging this - we also trained RNNs on an additional, commonly used task, where the network has to generate both the sine and cosine wave given some input frequency (see cartoon in extra figure E). Here, the relevant perturbation is a frequency shift. Our adaptation to perturbation results remain qualitatively the same (extra figure F and G), expanding the generality of our method. We will include these results as an additional supplementary figure in the final manuscript.

Also, I fail to see the "biological circuits".

Thank you for flagging this. In the discussion, we state that we are agnostic to the exact source and implementation of the feedback signal. For the RNN itself, while it does not directly correspond 1-1 to a single brain region, it has been used in the literature as a useful model of a non-linear dynamical system such as the brain. Hence, our study is more about the brain as an integrated system rather than a single brain region or circuit per say.

.. The term "inference learning" is only used there and in one of the figure, but it is not clearly defined.

Thank you for flagging this. We are referring to the inference learning (IL) definition from Alonso et al. 2022, which is based on the predictive coding (PC) literature. In IL, neuron activities are modified first using some gradient, and only after this step are the weights modified. This contrasts with backpropagation, where weights are modified first and only. As defining this term in Figure 2/introduction may not be ideal, we have omitted it in our updated manuscript and instead included the relevant citation in the discussion. We now instead refer to simply "approximate learning in the activity space", due to approximate gradient alignment (which was not explicitly shown in previous work).

Alonso, Nick, et al. "A theoretical framework for inference learning." Advances in Neural Information Processing Systems 35 (2022): 37335-37348.

For the second finding ...

Thank you for flagging this - here, we refer to accuracy in terms of approximation of the gradient, not performance accuracy, as the referenced section 3.4 outlines.

For the third, the second order gradient is not really injected ...

Thank you for flagging this - you are right! We now changed this to read: "Feedback control enables more efficient weight updates during task adaptation due to the (implicit) incorporation of adaptive, second-order gradient into the network dynamics "

Line 142: it seems natural that ...

Thank you for flagging this - we now changed this to simply read: "Thus, during the task perturbation, the network activity is increasingly driven by the feedback signal."

LInes 164 and 165. The authors say that "using a local learning rule ...

Thank you for flagging this - we corrected this and we thank the reviewer again for these important detailed comments!

In 3.4 it is a bit hard to follow. It seems as if the authors are using an eligibility trace to train the RNN through BPTT. But this intermediate step might not be real BPTT as it is commonly used.

Thank you for flagging this! In the beginning of section 3.4, our main goal is to briefly outline why biologically plausible learning is particularly hard in RNNs, due to the need for recursive computation of the past network states using the network Jacobian. This and the problems with the currently proposed biologically plausible solutions was explored in more detail in Marchall et al. 2020., where RFLO was shown to be quite a severe approximation of RTRL. This mainly serves as context for our main contribution of this section - that with feedback control, RNNs are more driven by their present than their past states, which simplifies the learning problem and allows for more accurate local learning.

The work of Meulemans et al 2022b is credited with alleviating the temporal storage required by BPTT.

Thank you for flagging this. For simplification, as we are not exploring equilibrium systems in this work, we removed this specific claim in the updated manuscript.

The connection between feedback-based learning and second order optimization ...

Thank you for flagging this citation omission! We now cite it in our updated section on related work.

A brief mention of why would one think that the learning rules are close to biological ones would be welcome.

Thank you for this suggestion and citation, which has now been added to the discussion section of our updated manuscript.

I am not 100%, but I think that the work of Gilra and Gerstner had a very similar architecture. Could you mention what are the main differences?

Thank you for flagging this! Our work is indeed similar to that of Gilra and Gerstner 2017 (eLife), with the main differences being 1) we use a rate instead of a spiking network, which simplified our analysis, such as gradients + Hessians 2) in their work, the feedback loop is trained separately from the the recurrent network, while we pretrain them together 3) in their work, in contrast to ours, the network output is close to reference from the very start of learning due to "tighter" feedback control, which further eases the learning problem.

Thank you so much for reviewing our rebuttal. We acknowledge your concern with our title. We had a couple of working titles, one of which was "recurrent neural networks" instead of "biological circuits", and we are happy to change it back to this. We initially chose "biological circuits" because the motivation for our project was/is on the biological learning side, but we agree that revising it is a good idea.

Thank you for taking the time to review our paper.

In the appendix, it is written that the learning rate is taken to be constant. To make claims about e.g. learning speed, the optimizer, in particular its learning rate, has to be tuned.

Thank you for pointing this out. To address your concern, we have included a learning rate sweep for Figure 3b in the extra figures page (see figure B).

Figure 5b: it is not clear from this experiment that RFLO-c contains some second-order information.

Thank you for raising this important point. To clarify, we are not claiming that our learning method performs exact 2nd-order optimization. Instead, it effectively interpolates between a 1st and 2nd order method, with the gradient showing a significant projection in both directions. In the extra figure A, we show the measured alignment between the 1st and 2nd order gradient as a baseline (yellow line). This baseline alignment is lower than the overlap of our gradient with both 1st and 2nd order directions, indicating that the observed alignment is not accidental. Additionally, having any curvature information allows the optimizer to better navigate the learning landscape and improve learning efficiency.

A BPTT-c/RTRL-c baseline would be an interesting add to disambiguate between the role of feedback control and approximate gradient estimation through RFLO...

Thank you for the suggestion. We have added the BPTT-c baseline to Figure 3b in the extra figures page (see D) to help disambiguate the roles of feedback control and approx gradient through RFLO. It seems that BPTT+c performs worse than BPTT (remember that the test loss is reported in the open-loop setting, i.e. no feedback is provided to the network regardless of whether it was trained with it or not). This suggests that BPTT+c overfits to feedback control here, unlike RFLO+c. However, note extra figure B, which suggests that our choice of learning rate for BPTT+c may simpy not be ideal. For Figure 5b, however, this baseline is not applicable because in it, there is no plasticity in the weight matrix yet, and both gradients are computed in the activity space. While we could perform the same analysis in the weight space, calculating the Hessians of the full weight matrix is intractable due to poor scaling.

l98-99: can the author clarify the link between the use of a local learning rule and the rapid adaptations shown in neuroscience studies?

Long-term skill learning is known to involve synaptic changes in the motor cortices (e.g. Roth et al. Neuron 2020). However, we don’t know how rapid learning is implemented in the brain, nor where it happens. In the neuroscience study that motivated our work (Perich et al. 2018), plasticity was not directly measured, due to the extreme difficulty of such measurements. Nevertheless, animals progressively learn to adapt better to persistent perturbations, which requires some form of rapid learning along the sensorimotor pathways. Thus, in the fine-tuning phase of this work, we enable plasticity in the recurrent layer of our RNNs, which serve as a model of a nonlinear dynamical system doing closed-loop control. Additionally, we use a learning rule that could feasibly be implemented in the brain, as it respects the locality constraint. Thank you for flagging this important point - we have now edited the relevant section in the main text to clarify this reasoning better.

Fig1: a, b, c legends are missing in the figure. l140-141: "that" missing after "feedback control,"? typo "outout". l167-168: "does" missing + typo for "adaptatation".

Thank you for catching these errors!

Fig2: "approximate inference learning": what do the authors mean by inference learning? I could not find any definition.

Thank you for flagging this. We are referring to the inference learning (IL) definition from Alonso et al. 2022, which is based on the predictive coding (PC) literature. In IL, neuron activities are modified first using some gradient, and only after this step are the weights modified. This contrasts with backpropagation, where weights are modified first and only. As defining this term in Figure 2 may not be ideal, we have omitted it in our updated manuscript and instead included the relevant citation in the discussion.

Alonso, Nick, et al. "A theoretical framework for inference learning." Advances in Neural Information Processing Systems 35 (2022): 37335-37348.

Appendix A.3: the authors mention that they use Adam with weight decay. The standard practice is to use AdamW instead (c.f. the AdamW paper for more detail). Can the author confirm that they are using AdamW?

We use Adam, and we explicitly add the regularization terms (i.e., weight decay) directly to the cost function. This approach is functionally different from the weight decay correction in AdamW. While we did not use AdamW, it does not impact our results as we do not report any performance gains with Adam and instead use SGD during fine-tuning. We now make this explicit in the detailed methods section.

Thank you for taking the time to review our paper.

Several sections of the paper seem to just present results from previous work ...

While our work builds upon the task adaptation findings of previous work (Feulner et al. 2022), it offers a substantially distinct focus and set of contributions, as outlined in our introduction. Our primary goal is to investigate the mechanistic interpretability of the model published by Feulner et al. 2022., and to connect their results to the broader literature of online learning in recurrent neural networks. Specifically, we address why rapid adaptation occurs without any plasticity and how this benefits approximate, local learning algorithms, whose limitations have been previously studied by Marschall et al. 2020). Beyond learning accuracy, we also analyse learning efficiency and show significant differences between learning with and without feedback during the fine-tuning stage. Thus, despite the similarity in the problem setup and the methods presented in the main text, our analysis is distinct from Feulner et al. (2022), where the emphasis was on modeling primate observations and data. Here, we ask a different question: based on what we know about local learning in recurrent neural networks, why does this model work so well?

I may be missing something, but ...

Thank you for your insightful comments. While it may seem intuitive that the feedback contribution to the overall network output would increase during perturbation, it is important to note that this behaviour is not explicitly designed in our networks. The same applies to the incorporation of second-order information. The key contribution of our work lies in empirically showing that these mechanisms naturally emerge from our model. Even if some aspects may seem anticipated, demonstrating this behaviour in a practical and previously unexplored context is a significant step forward in understanding online learning in recurrent neural networks, and specifically how it relates to biologically plausible learning.

The authors claim that feedback control guides credit assignment in biological circuits ...

Thank you for raising this important question. Yes, this approach reflects principles commonly observed in biological learning systems, specifically the different time scales of learning:

• Foundational Development: Certain core circuits and functions may develop on a slower timescale, guided by an innate basis shaped over evolutionary time. This provides a foundational framework for subsequent learning (analogous to BPTT pretaining).
• Rapid, Contextual Adaptation: Building upon this foundation, biological systems exhibit rapid adaptability to specific environments and tasks. This occurs on a much faster timescale, and is thought to occur through fine-tuning of pre-existing circuits (analogous to fine-tuning with a local learning rule).

For example, infants possess a general capacity for movement, learn to walk relatively slowly, but quickly adapt to different terrains like ice. While BPTT plays a significant role in the credit assignment during pretraining, the rest of this work focuses on alternative, biologically plausible credit assignment during rapid, contextual adaptation, on perturbation not seen during pretraining.

Given that the primary goal of this paper is to improve understanding of an existing learning algorithm ...

Thank you for raising this important point. We now include extra figures with this rebuttal that include the final adaptation to perturbation for different learning algorithms as a function of learning rate and hidden size (extra figures B+C). We also include results from an additional, commonly used synthetic task (where the network has to produce a sine and a cosine wave given some frequency, see cartoon in extra figure E), showing that our results remain qualitatively the same (extra figure F+G), expanding the generality of our method. We will add these results to the supplementary figures of the final paper.

How does this work relate to hierarchical predictive coding, and to the feedback connections introduced by Hinton in [1] (and further explored by [2])?

In the FF network architecture, the activity vectors from the layer l+1 at previous timestep contribute to the activity vectors at layer l, which is a common feature with our work. However, nature of and reason for this contribution is different. In our case, the information communicated between the layers is a global error to control the activity space online. In FF, the information communicated uses layer activities themselves and is done to sync up local learning between different layers. The follow up work links FF to the predictive coding framework, where the error neurons are explicitly represented. We link our work to other relevant literature in predictive coding in our discussion.

The learning setting presented in this work seem very similar to the setting of reinforcement learning...

Even though, in principle, feedback control studied here resembles a reinforcement learning (RL) problem due to the interaction with the environment, there are some major differences from classic RL problems. For instance, unlike many RL problems, the feedback here is an explicit function of the network's prediction. Moreover, during pretraining, we are backpropagating through the environment itself, which is generally assumed to be impossible in most RL problems. However, our results may be relevant to certain RL subfields, such as movement control in robotics, and we briefly mention this in the "Impact Statement" section of our manuscript. Moreover, it may be interesting to expand our work to settings where feedback is less explicit (as we note in our limitations section), which would more closely link it to RL problems.

I'd like to see an expanded discussion ...

Thank you for this suggestion. As explained above, here we use BPTT for pre-training of the networks only.

Thank you for taking the time to review our paper.

The training setup is rather limited; it would be interesting to see training done for other tasks and architectures (or RNN sizes).

Thank you for raising this important point. We also trained RNNs on an additional, commonly used task, where the network has to generate both the sine and cosine wave given some input frequency (see cartoon in extra figure E). Here, the relevant perturbation is a frequency shift. Our adaptation to perturbation results remain qualitatively the same (extra figure F and G), expanding the generality of our method. We also show results for different RNN sizes (extra figure C, final adaptation loss as a function of learning algorithm and hidden layer size) - where we mostly see the expected increase in accuracy post adaptation with increased network capacity. We will include both of these results as an additional supplementary figure in the final manuscript.

The paper might benefit from some theoretical analysis of why the feedback signal alings with the true gradient, although it’s not clear if that can be easily done.

Given that the feedback weights approximately align with the transpose of the forward weights and the feedback is a linear projection of the loss function derivative (the error), the (approximate) alignment of the feedback with the gradient of the network activities is somewhat expected. However, why this would aid local learning is not entirely clear. In Section 3.4 and Figure 4 of our work, we address this question empirically by demonstrating that the output of the networks with feedback control relies less on past and more on present states. This reliance reduces the bias introduced by the severe Jacobian approximation by the eligibility trace, as studied by Marchall et al. (2020), making it more accurate. We acknowledge that further theoretical analysis could provide additional insights and leave this exploration for future work.

What is the difference between RFLO and RFLO+c? Does the first lack the feedback term in Eq. 1? This should be clearly stated within Sections 2.2-2.4.

Thank you for highlighting the lack of clarity in Figure 3(b). You are correct in that +c indicates the presence of feedback in Eq.1 during adaptation to persistent perturbation via local or non-local learning. We now clarify this in both the figure caption and methods section 2.4:

"Note that when the same weight matrix $\boldsymbol{W^{fb}}$ is used for both control and learning, we denote this by adding $+c$ to the respective learning rule."

Line 141: “outout” Line 141: “ is increasingly by”

Thank you for catching these typos!