Local-Forward: Towards Biological Plausibility in Deep Reinforcement Learning

We thank the reviewer for their comments and suggestions.

We particularly appreciate that the reviewer gave detailed, constructive feedback to help us improve our work, while acknowledging the future potential of our approach.

We are currently working on additional experiments that compare LF's performance against ANGC, and evaluate LF's performance on MNIST and CIFAR-10 to further support its robustness across tasks.

In the meantime, we want to address the other concerns and questions the reviewer raised:

• Biological insights beyond just "it can solve MinAtar." Thanks for raising this, we're happy for this opportunity to elaborate. Since this issue also provides useful context for our other responses, we'll address it first.

  Our motivation for designing LF is heavily biologically inspired. We saw a potential connection between the design principles of Forward-Forward (FF) and biological reward-based learning via striatic dopamine release; LF is our attempt to synthesize the two to improve deep Q-learning.

  FF provides an alternative to backpropagation that is more biologically plausible and computationally efficient, because it does not suffer from weight transport or update locking; Ororbia and Kifer (2022) also discusses these issues. Like LF, FF globally broadcasts the target to each layer of the network, which each computes their own local loss and updates. However, FF does not provide a biologically plausible explanation on how the target signal can be globally broadcasted in the brain. Interestingly, recent research in neuroscience by Li and Jasanoff (2020) found that in the striatum, dopamine release can be a global signal that induces local changes to reward stimulation, providing a potential answer in the case of reward-based learning. Additionally, biological reward-based learning has close ties with TD-learning, which is used as an explanatory model for dopamine-induced synaptic modifications in the striatum (Starkweather and Uchida, 2021). The alignment between FF's principle of globally-induced local learning, and evidence of biological globally-induced local modifications for reward-based learning in the brain, is the central biological insight that motivated us to develop LF as a Q-learning algorithm.

  To be clear, we do not believe that LF plausibly represents the complete reward-learning process in the striatum. Unlike backpropagation, the brain likely utilizes multiple types of learning in conjunction, both supervised and unsupervised, using local and global signals (O'Reilly et al., 2012). LF only models one type of learning, i.e. error-driven learning using global reward signals to induce local updates. Other biologically plausible algorithms, like ANGC that the reviewers cited, may provide explanations to other types of biological learning. Indeed, aspects of Hebbian learning or active inference are likely critical to achieving a full biologically plausible, efficient, and powerful learning system. In that light, we examine LF here not to replace other methods, but to demonstrate that this one principle alone is sufficient to be able to solve some complex RL tasks nearly as well as backprop. We believe that a key to achieving general, human-like intelligence will be the integration of these different learning methods and learning signals; this is an exciting direction we aim to explore in future work.

• Background on the forward-forward algorithm is insufficient. We expanded the background section on FF to include more mathematical/algorithmic details. This will be reflected in the upcoming updated manuscript. Specifically, the first paragraph of Section 2.2 now reads: "The Forward-Forward (FF) algorithm is a greedy multi-layer learning algorithm that replaces the forward and backward passes of backpropagation with two identical forward passes, positive and negative. These passes differ only in the data fed through the network and the objective they optimize. For a positive pass, the update increases the goodness $g$ of that layer. While Hinton proposes several measures of goodness, the de facto measure used is the sum of each squared hidden activation $h_{i}$ of the layer minus a threshold $\theta$, i.e. $g = \sum\_{i}{h\_{i}^2} - \theta$. In the case of a negative pass, the update decreases goodness. The positive data are real examples while the negative data is `fake.' One way to generate the fake data is by masking and overlaying real examples. For the MNIST dataset, this could be combining the top half of a 7 with the bottom of a 6. The goal of the model is to correctly classify an input as either positive (real) or negative (fake). To optimize each layer's weights, FF minimizes the binary cross-entropy loss between the sigmoid of the goodness $\sigma(g)$ and the ground truth."

• Computational cost of LF compared to DQN. The computational efficiency gains of LF over DQN come from two main sources: increased potential for parallelization, and decreased data movement costs. For DQNs (and backpropagation in general), because the error of each lower layer is dependent on higher layers, the updates must be computed sequentially layer-by-layer. In contrast, since LF computes losses and updates locally (using a global signal), the layers of LF are not dependent on each other, which allows the learning to be parallelized across the layers. Futhermore, DQNs suffers from the weight transport problem, which is not only biologically implausible, but also significantly increases its computational efficiency on neuromorphic hardware (Crafton et al., 2019). For full transparency, we want to clarify that these efficiency gains directly result from the design principles that we inherited from Forward-Forward, and have already been raised by Hinton (2022); achieving these efficiencies over backpropagation is not a novel contribution of our work. Our contribution here lies in proposing a new algorithm that extends these efficiency improvements to deep Q-learning, which is notoriously computation-heavy.

  While each layer does use more neurons and connections to generate their own Q-value predictions, we took this into consideration when selecting our network sizes-- this is why each layer of the LF network has less hidden activations. In addition, since LF does not suffer from update locking, the Q-value predictions of each layer can be performed in parallel. More specifically, in terms of number of parameters, in our experiments, we used a similar number of parameters for both the LF network and the DQN to make the performance comparisons more fair and direct. Since the MinAtar testbed slightly varies in input channels and action space across the environments, these numbers also vary depending on the environment. Given an environment with 4 input channels and 3 actions, the LF network had 1,199,619 trainable parameters, and the DQN had 1,123,200. We added the calculations of these numbers to the Appendix, which will be reflected in the updated manuscript.

References

[1] Li, Nan, and Alan Jasanoff. "Local and global consequences of reward-evoked striatal dopamine release." Nature 580.7802 (2020): 239-244.

[2] Bartunov, Sergey, et al. "Assessing the scalability of biologically-motivated deep learning algorithms and architectures." Advances in neural information processing systems 31 (2018).

[3] Crafton, Brian, et al. "Local learning in RRAM neural networks with sparse direct feedback alignment." 2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED). IEEE, 2019.

[4] Hinton, Geoffrey. "The forward-forward algorithm: Some preliminary investigations." arXiv preprint arXiv:2212.13345 (2022).

[5] O'Reilly, Randall C., et al. Computational cognitive neuroscience. Vol. 1124. Mainz: PediaPress, 2012.

[6] Ororbia A, and Kifer D. The neural coding framework for learning generative models[J]. Nature communications, 2022, 13(1): 2064.

[7] Starkweather, Clara Kwon, and Naoshige Uchida. ”Dopamine signals as temporal difference errors: recent advances.” Current Opinion in Neurobiology 67 (2021): 95-105.

Hey again Reviewer AyQt,

Just wanted to give you an update re: the additional experiments we promised to run-- they're done and in the updated manuscript :)

In Appendix C we added new experiments on 4 new RL environments, Cart Pole, Mountain Car, Lunar Lander, and Acrobot to further show the applicability of our algorithm across different RL environments, and indirectly compare LF's performance against ANGC. In Appendix D we added results on MNIST and CIFAR-10. LF's performance across all these tasks look strong!

We chose Cart Pole, Mountain Car, Lunar Lander, and Acrobot as our new environments because they are similar to those used by ANGC (Ororbia and Mali, 2022), which is the closest work to ours. We reached out to the authors for code of their agent, but unfortunately they needed more time to prepare their code for public release, which they said they will likely complete in December. In light of this, rather than comparing ANGC against LF in the MinAtar environments we used, we evaluated LF in the 3 publicly available environments that Ororbia and Mali used to evaluate ANGC, as well as a similar environment Acrobot-v1. To indirectly compare our performance against ANGC, we reimplemented the DQNs that ANGC benchmarked against, and use their performance as a reference point (we also compare against CleanRL's DQN for good measure). As a pleasant surprise, LF handily outperforms the DQN in all 4 environments. Its training process is in general more stable, and reaches higher average return on each environment.

For MNIST and CIFAR-10, LF also performed reasonably well, achieving 98.74% test accuracy on MNIST, and 57.25 on CIFAR-10. We also have results for different network sizes, check out our new Appendix D.

In addition, we thought that your question on biological insights is very important, and agree that we didn't do LF justice on this matter in our writing-- so we added parts of our response to your question to Appendix E.

Lastly, we added Ororbia and Mali (2022) and Ororbia and Kifer (2022) to our related work section. They are closely relevant to our work, thanks for bringing them to our attention.

Once again, thank you for the detailed feedback to help us improve our work!

• We have a two-part answer regarding Weakness 3, i.e. by conducting our experiments on MinAtar, we only cover 5 basic tasks. First, while solving MinAtar might not be challenging for more established deep RL algorithms, in our experience they are not certainly not basic tasks for a novel algorithm that does not utilize backpropagation. For example, Active Neural Generative Coding (Ororbia and Mali, 2022), another recently developed biologically-inspired deep Q-learning algorithm, is evaluated on four simpler environments, but their work is nonetheless a very insightful contribution that progresses the field towards stronger biologically-plausible learning algorithms. Thus, we believe that most deep RL researchers, if told that an algorithm can perform as well as backprop on MinAtar, without using backprop to coordinate learning across the layers, would find it at least interesting, if not unexpected.

  Second, previous research has shown that experiments in the smaller-scale MinAtar environment can still yield valuable insights for deep RL. Obando-Ceron and Castro (2021) showed that most results achieved by the Rainbow paper (Hessel et al., 2018) in the Atari environments can be replicated in the MinAtar environments. Therefore, they argue that Hessel et al. would likely have arrived at the same qualitative conclusions if they conducted their experiments in MinAtar. Additionally, they urge the field to change the status-quo of excessively focusing on large-scale tasks, and avoid dismissing empirical work that focus on smaller problems, as this reduces barriers of entry for newcomers from underprivileged communities, which we fully support. Given the potential of LF (also supported by the review of Reviewer AyQt), we believe that its strong performance on MinAtar is a valuable result, and would thus like to share it with the community.

• Re: your concern raised in the Questions section, we removed the misplaced $\pi(a|s)$, thank you for pointing it out.

References

[1] Starkweather, Clara Kwon, and Naoshige Uchida. "Dopamine signals as temporal difference errors: recent advances." Current Opinion in Neurobiology 67 (2021): 95-105.

[2] Li, Nan, and Alan Jasanoff. "Local and global consequences of reward-evoked striatal dopamine release." Nature 580.7802 (2020): 239-244.

[3] Ororbia, Alexander G., and Ankur Mali. "Backprop-free reinforcement learning with active neural generative coding." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 1. 2022.

[4] Obando-Ceron, Johan Samir, and Pablo Samuel Castro. "Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research." International Conference on Machine Learning. PMLR, 2021.

[5] Hessel, Matteo, et al. "Rainbow: Combining improvements in deep reinforcement learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

We thank the reviewer for their comments and suggestions.

Unfortunately, we believe there are some significant misunderstandings, particularly regarding the novelty of our algorithm and our contributions, that may have affected the reviewer’s judgment. This likely reflects that our paper does not present these aspects of our work sufficiently clearly. Thus, we have adjusted Sections 1, 2 and 3 of our upcoming updated manuscript accordingly to better communicate them, but we also want to first provide a more direct response here.

• We believe the reviewer misinterpreted our work as just an application of the existing FF algorithm. In the weaknesses section, the reviewer states that we “simply [apply] the existing FF algorithm to RL tasks.” We want to clarify that this is not true— one of our central contributions is that we introduce a novel learning algorithm for Q-learning. The novelty of our method is also supported by the review of Reviewer AyQt. Our algorithm builds upon FF, inheriting the local computation of losses and forward-in-time connections to propagate information downwards, but differ substantially to adapt it to our domain.

  Unlike FF, our algorithm does not make 2 forward passes through the network during training. We only make a single forward pass, then provide the reward globally to every cell, such that they can each locally compute their own TD error to update their parameters. Rather than comparing positive and negative data, like FF, the comparison is between the model’s current prediction of the “goodness” of its situation, and its updated prediction in the next timestep. In some sense, the models’ current prediction is the negative data, and the TD error entices it to generate positive data, i.e. the true Q-value given the current policy and environment— this is why we don’t need 2 separate forward passes. The local computation of loss using a global target is indeed like FF. However, FF is designed for classification tasks, whereas Q-learning is a regression task. To solve our task, we introduced new architecture in the form of our LF cells, which utilize an attention-like mechanism to handle regression and normalization of the activity vectors, as we show in Figure 2 and Section 3.1 of our paper. We hope this explanation addresses some of the concerns regarding Weakness 1; rather than testing existing variations of FF, we built upon it to design our own new version that is more suited to our domain, and showed that it achieves comparable performance to DQN in the MinAtar testbed.

• We also want to address Weakness 2, and explain why we chose to work on Q-learning rather than explore more broadly applying variations of FF to RL methods. There are two interesting synergies between the TD-learning setting and the properties of FF we inherited; they motivated us to develop LF specifically for Q-learning. First, it has a biologically plausible explanation. TD-learning has strong ties to reward-based learning in the brain, and is used as an explanatory model for synaptic modifications in the striatum (Starkweather and Uchida, 2021). Importantly, research suggests that the striatum uses dopamine release to globally broadcast signals that affect response to reward stimulation (Li and Jasanoff, 2020). LF's use of global reward signals to modulate local learning of Q-values imitates this process. This also answers a question that FF did not explain: how would a biological NN send the global target/label to every layer? For reward-based learning, the brain could be broadcasting the global target via dopamine. Thus, framing LF as a TD-learning algorithm is the most biologically plausible.

  Second, the forward connections in time are well-suited to take advantage of the temporal structure of the Q-values of trajectories for better learning. Given a good policy, the Q-value predictions of a well-trained model should remain stable through each state of a trajectory (assuming the dynamics are reasonably deterministic). This means that the Q-value prediction of the current timestep, and the hidden activations used to make this prediction, can often still be useful for predicting the Q-value of the next timestep. In contrast, in FF's experiments on image classification, this effect is forced-- to ensure that the activations of the current timestep is still useful for the next, FF has to repeat the same input every timestep, reducing computational efficiency. Our results empirically support the particular effectiveness of forward connections for Q-learning; they qualitatively improve performance in the more complex MinAtar environments.

Regarding your question: this is indeed an interesting and surprising observation— intuitively, it seems that a multi-layer network without an information channel from upper to lower layers should not perform well, as the upper layers cannot provide any feedback to the lower layers. We can’t claim any credit for the discovery of this unexpected behavior; it was first raised by Hinton (2022). Hinton showed that, for the MNIST classification task, the lower layers of a multi-layer network trained without any downward connections can nonetheless learn useful features for upper layers, and thus achieve good classification performance.

But as our experiments suggest, the importance of downward information channels increases with the difficulty of the task. In our results, the performance of the 3-layer network without forward connections is strongest on Freeway, where its average episodic return rivals that of LF and the backprop-trained DQN. This is likely because Freeway is the simplest of the MinAtar tasks, and unlike the other environments, it has an upper limit on episodic return, which bounds performance. In the other 4 environments, the episode length is determined by when the agent fails and dies, and the difficulty of the game increases as time goes on. In contrast, Freeway has a fixed episode length, and does not increase in difficulty: the goal for Freeway is to cross the road as many times as possible within a fixed 2500-frame window. But, even with the optimal solution, the maximum possible number of times an agent can cross the road within that window is 50-60, depending on how lucky the agent gets with the traffic patterns, which is random. Therefore, the reason that the 3-layer network without forward connections catches up to LF on Freeway is likely not because it is as good, but because LF has no more room to improve its performance. On the other hand, on Seaquest, where agents must learn a complex, multi-step rescue-and-resurface procedure to achieve higher returns, we can observe the opposite behavior. Both the 3-layer network without forward connections and the single-layer network are unable to learn the rescue-and-resurface procedure, and as a result sit flatly near 0 return throughout training.

References

[1] Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022.

We thank the reviewer for their comments and suggestions.

We particularly appreciate that the reviewer took extra time to give specific, actionable feedback. We agree with the weaknesses, and addressed each concern in our upcoming updated manuscript to improve it. Specifically:

• We modulated our claim in the first bullet point of the contributions; it now reads “We propose an alternative to backpropagation for reinforcement learning that suffers less contradictions with known biology."
• We standardized our use of the $Q$ notation in the background section.
• We rewrote our description of the cell internals to better explain the roles of the matrices $W\_{in}$ and $W\_{query}$. We experimented with adding $W_{in}$ and $W_{query}$ to Figure 2, but since they live within the ReLU and tanh weight layers, we felt that it might distract from the high-level structure the Figure aims to convey. Instead, in the new text, we included a reference to the parts of Figure 2 the matrices correspond to; let us know what you think. The paragraph now reads: "We introduce an attention mechanism to compute the Q-value prediction. Each cell first computes its hidden activation, $h^{[l]}\_t$, using the layer above's hidden activations from the previous timestep, $h\_{t-1}^{[l+1]}$, and the layer below's hidden activations at the current timestep, $h\_{t}^{[l-1]}$. Specifically, it passes the concatenation of $h\_{t-1}^{[l+1]}$ and $h\_{t}^{[l-1]}$ through a ReLU weight layer (shown in Figure~2) to get $h^{[l]}\_t$. The ReLU weight layer multiplies the concatenated input $[h^{[l-1]}\_t, h^{[l+1]}\_{t-1}]$ by its learned weight matrix $W^{[l]}\_\text{in}$, then applies the ReLU nonlinearity function. Next, the cell passes $h^{[l]}\_t$ through the tanh weight layers, each corresponding to an action $a$, which multiplies $h^{[l]}\_t$ by its learned weight matrix $W^{[l]}\_\text{query}$, then applies the tanh function. The output of the tanh layers plays a role akin to attention weights, although they do not have to sum to 1. Finally, the cell takes the inner product between the output of the tanh layers, $\text{tanh}(W^{[l]}\_\text{query} \cdot h^{[l]}\_t)$, and the hidden activations $h^{[l]}\_t$, to compute the cell's Q-value prediction $\hat{Q}^{[l]}(s\_t,a)$. This structure is heavily inspired by the successful attention architecture, and allows each cell to attend to the incoming information dynamically per action."
• We changed the first usage of argmax in Algorithm 1 to $a_{t} \in argmax$, and the rewrote the second usage with a max function instead to prevent this confusion.