Error Forcing in Recurrent Neural Networks

We thank the reviewer for the feedback.

Toy tasks: While we acknowledge that the tasks we use are purposefully simple one should note that we deliberately used extremely long delays (more than twice that commonly used in past work, e.g. [1, 2]). Under these harder regimes BPTT succeeds on only ≈25 % of random seeds, while EF still solves >85 % (Figure 1). Since the task is so simple and only time delays increase, it is unequivocally clear that the BPTT failures are directly due to vanishing gradients and so EF owes its performance improvement from mitigating that. To push this point further, we have run further numerical experiments expanding this temporal horizon on delayed XOR task (delay=2500) with BPTT performance %12 +/-4 vs. EF performance %52 +-5, while TF-BPTT is %25 +/-4. This reinforces the idea of EF substantially improving on standard approaches for low dimensional tasks with very long temporal horizons. While it would be great to expand these results to high dimensional real world applications, we feel that the main goal of this paper is conceptual, with scalability questions best left for a separate publication.

[1] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. "A solution to the learning dilemma for recurrent networks of spiking neurons." Nature Communications 11(1): 3625, 2020. doi:10.1038/s41467-020-17236-y

[2] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown, and Uygar Sümbül. "Cell-type–specific neuromodulation guides synaptic credit assignment in a spiking neural network." Proceedings of the National Academy of Sciences 118(51): e2111821118, 2021. doi:10.1073/pnas.2111821118

Stop gradients: It is important to note that the stop gradient solution is not an ad-hoc choice but rather the formal solution implied by the Bayesian perspective of EF: Because error corrections are dynamically inferred via a coordinate descent procedure on the variational free energy, they are effectively constants with respect to network parameters (hence the use of stop gradients). We do show empirically that EF works with or without the stop gradient. Nonetheless, dropping it comes at the cost of non-local gradient calculations, which makes including the stop gradient still the preferable option from a biological modeling perspective.

Hyperparameter $\boldsymbol{\alpha}$ selection: Please see lines 198 and 199 of the text. “The optimal degree of forcing $\alpha$ was determined by a grid search (note that \alpha = 0 reduces EF-BPTT to vanilla BPTT).” This holds for both TF-BPTT and EF-BPTT and for every different task and task difficulty.

Mismatch in Fig 3c: We plot EF‑BPTT only for  $\alpha$ ≥ 0.01; at $\alpha$ = 0 the update is mathematically identical to plain BPTT, so the two curves coincide and we skipped the duplicate point. Even a tiny $\alpha$ (0.01) already gives a clear boost over vanilla BPTT. Note that small $\alpha$ values being already very effective is also noted in the appendix of GTF paper. We will make this point visually clearer in the revised version of Figure 3 and its caption to ensure there is no ambiguity.

$\boldsymbol{\alpha}$ = 1 limit: In TF, if pseudoinverse is used, $\alpha$ = 1 gives

$r^{\sim}\_{t}= W^{+}\_{\phi} y^{*}\_{t}$ while for EF, $\alpha = 1$ gives (even without the stop-gradient operation):

$r^{\sim}\_{t} = r_{t} + W^{+}\_{\phi} e_t = r_{t} + W^{+}\_{\phi} (y^{*}\_{t} - y_{t})$ (see equation 9 in the main text)

So these two algorithms are different even when we set $\alpha = 1$, and even when we don’t use the stop-gradient operation.

Benefits over BPTT in evidence integration task: We have conducted additional experiments where we trained the BPTT, TF, and EF on evidence integration tasks with limited resources (less trials per epoch). In this regime we saw the importance of the EF algorithm over the other two more prominently: delay timesteps: 1000, EF-BPTT: %84 +/-8, TF-BPTT: %73 +/-8, BPTT: %52 +/-4

We believe that for this task if enough resources are provided the learning algorithms perform similarly, while in the less resource regime EF performs better.

We thank the reviewer for appreciating that the provided "perspectives are complementary and both useful to provide intuition” and that we evaluated Error Forcing “across a variety of tasks with varying difficulties”. We addressed the raised issues below:

On the ”more exploration during learning” advantage of EF vs. TF: We agree that the demonstration of the effect could be more precise; to address this, we performed new numerical tests to empirically estimate the dimensionality of neural activity over learning, directly comparing EF and TF. The new experimental results will be added in the revised version.

On the inclusion of other relevant works: We would be happy to discuss the broader online learning from feedback work, however, we are uncertain which specific works the reviewer has in mind. Could you please clarify the works you are referring to?

More on Bayesian perspective: The connections to Kalman and EM are indeed formally tight and detailed in the Appendix; we can revise the text to make that even clearer in the main text. We are also adding more information on how to generalize the approach for a tighter treatment of uncertainty and the EF implications of that at the prompting of reviewer j5Yw.

Train/test: During training, we apply error forcing to adjust network dynamics, whereas during testing, we do not employ error forcing at all. Naively, one might assume that a network trained in the presence of error forcing might come to ‘rely’ on that error signal, and would not generalize well to unforced conditions. This is also a concern for teacher forcing, which has led to approaches that gradually reduce the amount of forcing required over training [1]. We did not find such annealing empirically necessary to reach good performance for our networks. More importantly, our results formally justify the generalization performance of our trained networks: according to the principles of the variational EM algorithm, Error Forcing maximizes a lower bound on the true log-likelihood by jointly adapting hidden states (corresponding to the E step) and weights (M step); nudging is therefore part of the exact training objective, not an external oracle: the same generative objective is optimised in training and evaluated in testing. The Bayesian perspective on EF thus provides a formal justification for the train-test difference in dynamics.

[1] Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 11297–11312. Curran Associates, Inc., 2022.

Mismatch in Fig 3c: We plot EF‑BPTT only for  $\alpha$ ≥ 0.01; at $\alpha$ = 0 the update is mathematically identical to plain BPTT, so the two curves coincide and we skipped the duplicate point. Even a tiny $\alpha$ (0.01) already gives a clear boost over vanilla BPTT. Note that small $\alpha$ values being already very effective is also noted in the appendix of GTF paper. We will make this point visually clearer in the revised version of Figure 3 and its caption to ensure there is no ambiguity.

Early training in Fig 3f: Thank you for pointing this out. When investigating the loss curves for other tasks, we did not see the same pattern, so we believe this is an idiosyncratic behavior specific to that numerical experiment.

We thank the reviewer for highlighting both its merits (“clearly the right way to do TF”, “principled Bayesian motivation”) and the areas that would benefit from additional clarification or empirical evidence. Detailed answers below.

Toy tasks: While we acknowledge that the tasks we use are purposefully simple one should note that we deliberately used extremely long delays (more than twice that commonly used in past work, e.g. [1, 2]). Under these harder regimes BPTT succeeds on only ≈25 % of random seeds, while EF still solves >85 % (Figure 3). Since the task is so simple and only time delays increase, it is unequivocally clear that the BPTT failures are directly due to vanishing gradients and so EF owes its performance improvement from mitigating that. To push this point further, we have run further numerical experiments expanding this temporal horizon on delayed XOR task (delay=2500) with BPTT performance %12 +/-4 vs. EF performance %52 +-5, while TF-BPTT is %25 +/-4. This reinforces the idea of EF substantially improving on standard approaches for low dimensional tasks with very long temporal horizons. While it would be great to expand these results to high dimensional real world applications, we feel that the main goal of this paper is conceptual, with scalability questions best left for a separate publication.

[1] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. "A solution to the learning dilemma for recurrent networks of spiking neurons." Nature Communications 11(1): 3625, 2020. doi:10.1038/s41467-020-17236-y

[2] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown, and Uygar Sümbül. "Cell-type–specific neuromodulation guides synaptic credit assignment in a spiking neural network." Proceedings of the National Academy of Sciences 118(51): e2111821118, 2021. doi:10.1073/pnas.2111821118

Training vs test in TF/EF: While Teacher Forcing (TF) indeed introduces a train–test mismatch, EF maximizes a lower bound on the true log-likelihood by jointly adapting hidden states (corresponding to the E step) and weights (M step); nudging is therefore part of the exact training objective, not an external oracle: the same generative objective is optimised in training and evaluated in testing. The Bayesian perspective on EF thus provides a formal justification for the train-test difference in dynamics.

Annealing EF: We did not try it for two reasons: 1) In the original GTF paper annealing always performed worse or the same as the GTF. 2) Given the Bayesian perspective on EF, $\alpha$ value should not be a hyperparameter but the ratio between the latent noise and the observation noise (i.e. constant over time). Nonetheless, due to Error Forcing’s approximate (variational EM) nature, reducing alpha over the training might still prove practically helpful. This potentially introduces two different hyperparameters, one for the rate of (exponential) decay, the other for the lower bound of the $\alpha$ (potentially zero). In new numerical experiments we found that there always exists a constant $\alpha$ value that performs better than the best performing annealing process. We therefore believe that if necessary, optimizing the $\alpha$ value might be a better approach than optimizing the annealing process. We will add these additional experiments in the revised version of the paper.

The use of uncertainty in EF updates: While the MAP approximation is already useful in formally justifying the form of the EF update and training/test distinction as EM, it is quite correct that discarding uncertainty information potentially comes with performance costs.

As the reviewer has noticed, tracking uncertainty through time could be accomplished via an explicit extended Kalman Filter approach, or a variational approximation of EKF that uses diagonal precision matrices for increased computational efficiency. Formally, this translates into latent dimension-specific forcing $\alpha$ values that change dynamically through time (though practically Kalman gains stabilize relatively quickly so one could remove the temporal dependence). This would produce less forcing occurring along axes with low latent uncertainty and more along axes with higher latent uncertainty, at the cost of having to track extra state variables through time. To provide guidance for future work, we will include in the Appendix the full EKM derivation under our error reparameterization, as well as a variational approximation of the extended Kalman Filter using diagonal precision matrices for increased computational efficiency.

Metrics in the experiments: All quantitative results in Figs 3 and 4 are always on a test set. We will add “test” to every y-axis label that lacks it, and state explicitly in the captions.

The reference on TF in Transformers is interesting, we will mention it in discussion.