Temporal-Difference Learning Using Distributed Error Signals

We sincerely thank the reviewer for their constructive feedback and detailed questions, and we’re glad to hear that you appreciate the novelty, clarity, and elegance of our work.

To address your specific questions and concerns:

Clarifications on Section 3.1.

We understand how this may be confusing. Conceptually, the Q_t function takes two parameters, the state and the action, and outputs the value for that state-action pair. There exists multiple tanh layers, one for each action; W_att represents the weights of one of these layers. In practice, for parallelization, like [1], the Q_t function is implemented to output a vector of |A| values (where A is the action space), i.e. one value for each possible action. Each tanh layer is used to compute the value of one corresponding action. We then take the argmax over that vector to determine the action to take. We see how Figure 3 is misleading; we will update it to show that the cell computes multiple Q_t values, one for each action.

10 runs are quite low for comparing RL algorithms; additional metrics like IQM/IQR would be beneficial.

While we agree that our results can be further bolstered with more runs, RL experiments are indeed computationally demanding (Appendix H), and like most researchers, we are constrained by available resources and environmental concerns. However, we hope that the number of tasks we evaluate our algorithm on helps alleviate some of your concerns; AD is evaluated on 14 standard RL tasks, with 10 runs on each task.

Further, we agree that additional metrics like IQM/IQR are beneficial for comparing the performance of RL algorithms. We’re working on additional figures that show the IQM and IQR of our data for our main results and ablation study, and will upload these before the end of the discussion period.

The algorithm is based on Q-learning, could the main ideas be transferred to other RL backbones?

Yes, we believe they can, although we haven't yet experimented with this. We chose Q-learning as its one of the most supported hypotheses for how the brain performs reward-based learning, however the same learning principles should transfer to other RL paradigms like actor-critic models.

The reward needs to be globally available to all layers to compute the local prediction errors. Which implications does this global target induce? Any theoretical or practical drawbacks?

Just making the reward signal globally available to all layers does not incur any notable drawbacks we are aware of; there is evidence that this occurs in biological learning (Section 2.1), and providing such a signal to each layer does not add any significant computational costs for modern hardware (in fact, it improves parallelization, as each layer’s update can be computed in parallel, Section 3).

However, there is a significant drawback to replacing the sequentially propagated error signal with just per-layer reward and prediction errors, namely the layers can no longer directly coordinate their learning. Specifically, the upper layers of the network can't influence the learned representations of the lower layers via sequentially propagated gradients. The problem with gradient propagation is that there is no evidence it in the brain, despite its known importance in deep learning. This is partially why we believe our results are surprising and significant: we show that just distributed, local prediction errors and updates may already be sufficient to learn many complex reward-based learning tasks, without the need to directly coordinate updates across layers during learning.

How crucial is the choice of the two activation functions in the AD cell?

We experimented with alternative choices for these two activation functions, including replacing the relu function with a leaky relu, and replacing the tanh function with a sigmoid and with a second relu function. Most combinations of different activations did not have a significant impact on performance, with the exception of using two relu functions, which made learning unstable.

For transparency, our experiments on activation function choice were not extensive; there could certainly be combinations of other activation functions that lead to better performance. However, as our primary goal is not to achieve state-of-the-art performance, we believe that any potentially unrealized gains in performance does not interfere with our contributions, and thus leave further hyperparameter tuning of AD cells to future work.

Line 101 mentions 15 tasks, while the rest of the paper mentions 14; the main paper uses 10 tasks.

We agree this can be further clarified. Line 101 is indeed a typo; we evaluated AD on 14 tasks, including 5 tasks from the MinAtar testbed, 5 tasks from the DeepMind Control Suite, and 4 classic control tasks. We relegated the 4 classic control tasks to Appendix C because they are relatively less challenging compared to the MinAtar and DMC tasks (despite still being an informative starting point for evaluating the performance of reward-based learning algorithms). As we achieved consistently strong performance on each of these 4 tasks, to avoid cluttering the main paper body (given the space limit), we moved them to the appendix.

To address this issue, we have fixed the typo, and updated the Experiments section (Section 4) to explicitly state the number of tasks.

While technical and practical limitations are mentioned in other sections, the Limitations section only discusses limitations related to biology.

We will add a dedicated paragraph in the Limitations section to discuss technical and practical limitations, including discretized action spaces, current difficulties with scaling, and the number of runs per task.

References

[1] Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." nature 518.7540 (2015): 529-533.

We sincerely thank the reviewer for their insightful feedback, and we’re grateful for the opportunity to improve and clarify our work through answering your detailed questions and concerns.

There is a long tradition arguing that it is actually the dorsal striatal subregions (caudate and putamen) which signal action values, whereas ventral striatum signals state value.

We acknowledge that there are multiple theories that provide different mappings of reinforcement learning frameworks to the brain’s reward circuitry. We want to clarify that our assumptions and findings are not in conflict with the actor-critic models by Joel et al. (2002) and O’Doherty et al. (2004). Actor-critic mappings of the reward circuitry, like Joel et al. (2002) and O’Doherty et al. (2004), argue the dorsal striatum is responsible for action selection, i.e. policy learning, whereas the ventral striatum is responsible for value learning. On the other hand, Q-learning is a simpler model that combines policy and value learning. The main difference (with respect to our work) is whether action selection occurs separate or together with value learning.

AD does not make the argument that action selection need happen in the NAc. Rather, we just take the assumption that the NAc computes the Q-value function, which takes in the state and action as parameters, and performs Q-learning. The action can be provided from another region like the dorsal striatum. This view is consitent with the findings of works by Roesch et al. (2009), Mannella et al. (2013), and Aston et al. (2024).

On the other hand, we acknowledge that there exists research like Ito and Doya (2015) and Weglage et al. (2021) that suggest the competing theory that the dorsal striatum is responsible for signaling action values, whereas the ventral striatum signals state values. For completeness and openness, we will include this in the Limitations section.

The biological evidence for the modeling framework was weak. What is the evidence that dopamine updates values locally in the NAc, and for the particular form of hierarchical message passing architecture proposed here?

The theory that dopamine updates values locally in the NAc is supported by Wightman et al. (2007), who found spatial and temporal heterogeneity of dopamine release within the NAc. This suggests that while dopamine concentration is homogeneous within each microenvironment containing a local subpopulation of NAc neurons, it can vary across different subpopulations.

The hierarchical message passing architecture is a design choice inherited from current deep learning practices, reflecting the passing of information between subpopulations of neurons. Our specific architecture is not intended to be a fully mechanistically accurate model of the NAc, but rather a computational model to demonstrate that it is possible to learn complex reward-based tasks using synchronously distributed, locally homogeneous error signals. The actual implementation in the NAc may differ. This counterintuitive and surprising finding potentially opens new avenues for explaining biological reward-based learning, which is why we argue that we make a valuable contribution.

I was unsure whether the claim here is that the NAc is a multi-layer network or one layer in a multi-layer network.

Each layer in our model represents a local subpopulation of NAc neurons that receive the same signals from a subpopulation of dopamine neurons. However, as our model is not intended as a mechanistically accurate representation of the NAc, we do not claim that the NAc is a multi-layer network.

If the authors would like their model to be taken seriously by neurobiologists, they need to do more to directly link it to empirical evidence.

We agree that additional biological connections can strengthen our work. However, the main contribution of our work is not to propose a biological model, but to show that synchronously distributed, locally homogeneous error signals observed in the mesolimbic dopaminergic system may be sufficient to teach neurons complex reward-based tasks. While we make some biological assumptions for modeling (e.g., NAc neurons encode approximations of the value function, and VTA projections may compute errors specific to these approximations), we do not claim our work proves these assumptions; they are part of our premise, not our hypothesis.

We demonstrate that if these assumptions hold true, such conditions may be sufficient for complex reward-based learning by neural networks, without the need for other mechanisms to explicitly coordinate credit assignment. Previously, it was believed that solving complex nonlinear tasks required neurons in different regions to sequentially pass error signals and explicitly coordinate their learning. This assumption is held by both biologically plausible local-learning algorithms and backpropagation. We show that this assumption may not be necessary.

Ideally the authors could make some neural predictions based on the model. It would also be helpful to understand how the predictions are different from those of other models.

We agree that making neural predictions based on our model is an important direction. As an initial step towards better alignment with neurobiology, we performed experiments showing agreement with the distributional dopamine coding model by Dabney et al. (2025) in Section 5. We plan to further develop this aspect of our work in the future.

References

Roesch, M. R., Singh, T., Brown, P. L., Mullins, S. E., & Schoenbaum, G. (2009). Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. Journal of Neuroscience, 29(42), 13365-13376.

Mannella, F., Gurney, K., & Baldassarre, G. (2013). The nucleus accumbens as a nexus between values and goals in goal-directed behavior: a review and a new hypothesis. Frontiers in behavioral neuroscience, 7, 135.

Ashton, S. E., Sharalla, P., Kang, N., Brockett, A. T., McCarthy, M. M., & Roesch, M. R. (2024). Distinct Action Signals by Subregions in the Nucleus Accumbens during STOP–Change Performance. Journal of Neuroscience, 44(29).

Weglage, M., Wärnberg, E., Lazaridis, I., Calvigioni, D., Tzortzi, O., & Meletis, K. (2021). Complete representation of action space and value in all dorsal striatal pathways. Cell reports, 36(4).

Wightman, R. M., Heien, M. L., Wassum, K. M., Sombers, L. A., Aragona, B. J., Khan, A. S., ... & Carelli, R. M. (2007). Dopamine release is heterogeneous within microenvironments of the rat nucleus accumbens. European Journal of Neuroscience, 26(7), 2046-2054.

Thank you for the review. We noticed that this review is nearly identical to an earlier review we received at a different conference. Building on the previous feedback, we have significantly updated our manuscript with additional experiments, ablation studies, and expanded discussions, including new experiments and algorithmic updates aligning with DeepMind’s distributional dopamine coding (Section 5). We are thus surprised to receive the same review with a decreased score. Assuming you are the same reviewer, we wonder if there might have been a transpositional error or if the old version of our manuscript was reviewed by mistake?

In either case, we’d like to directly respond to your concerns, and highlight the relevant updates we made since our last submission that aims to address these concerns.

The neuro premise in Sections 1 and 2.1 reads as if the roles of the brain regions involved in the brain circuit are known undebatable. This is refuted in Section 6, but the discussion is inadequate.

We acknowledge this was a weakness in our previous submission. We expanded our Limitations section to explicitly state the assumptions we make regarding the biological mapping and included additional discussions on alternative mappings of RL frameworks to the reward circuit. We additionally discussed works by Takahashi et al. (2008), Chen & Goldberg (2020), Morris et al. (2006), Niv (2009), Roesch et al. (2007), Akam & Walton (2021), and Coddington et al. (2023). We highlighted that these theories are high-level mappings and may not be mechanistically accurate. We also added a footnote and forward reference to ensure this discussion is not overlooked.

The paper's comp neuro premise limits itself to the Forward-Forward algorithm, and ignores the host of other models in the field of biologically plausible learning.

Our main investigations revolve around an architecture inspired by the Forward-Forward algorithm. This does not mean we disregard other models in biologically plausible learning. Instead, our focus was to address a specific problem in biological reward-based learning: how neurons can coordinate learning to solve complex, nonlinear RL tasks using regionally homogenous, synchronously distributed error signals. While there are many related methods, they address different aspects of biologically plausible learning.

Taking a reductionist approach allows us to tackle more complex modern RL tasks, highlighting what TD-learning with distributed error constraints can achieve relative to non-constrained current deep RL algorithms. This approach helps attribute performance differences to distributed errors and the downsides of our algorithm, without being confounded by other shortcomings in current biologically plausible deep learning.

We agree that integrating our model with other established methods, like DeepMind’s model of distributional dopamine coding (Dabney et al., 2020), is important. Since our last submission, we designed an extension of our algorithm based on the quantile regression method to learn distributions over values (Section 5). Our goal is to evaluate whether AD can learn such distributions, aligning with Dabney et al.’s work suggesting the brain’s value predictors learn distributions over values, rather than the mean value. Like Dabney et al., we use quantiles to reflect differences in optimism/pessimism across dopamine neurons. This distributional version of AD consistently achieved strong performance on DMC tasks, though it was slightly less sample efficient than the standard version of AD.

We have not yet integrated with Backpropamine (Miconi et al., 2020), which focuses on neuromodulated plasticity and is less directly relevant to our problem, but we we look forward to potentially exploring this direction in future work.

Finally, we expanded our related work section (Appendix N) to include additional works in biologically plausible learning.

A single model consistent with data doesn’t preclude the existence of multiple other models, potentially with different mechanics, leaving the chance that the proposed model has little to do with the biological reality.

We agree that our model solving the credit assignment problem in reward-based learning doesn't preclude other models from doing so. Our goal is not to claim our model is the only explanation, but to provide one potential solution to this problem, which, to our knowledge, has been lacking.

To our knowledge, no published works consistently solve all MinAtar tasks or our subset of DMC tasks without backpropagation; we are the first to do so. Ororbia & Mali (2022) proposed a similar backprop-free Q-learning algorithm, but their evaluations are limited to simpler tasks. We present results on 3 of these tasks in Appendix C (the fourth was not publicly available).

We also want to address the biological reality issue, and explain why we believe our work is biologically relevant. The central hypothesis of our work is that synchronously distributed, locally homogeneous error signals, observed in the mesolimbic dopaminergic system, may already be sufficient to teach neurons to solve complex reward-based learning tasks. While we assume that NAc neurons encode approximations of the value function (and that VTA projections may compute errors specific to these approximations) in our experiments, we do not claim that our work proves this to be true; this assumption is part of our premise, not our hypothesis (we agree that alternative theories exist).

Rather, our findings suggest that if our assumptions hold, such conditions may be sufficient for complex reward-based learning by neural networks without the need for mechanisms explicitly coordinating credit assignment. This is counterintuitive and challenges established beliefs, potentially opening new avenues for explaining biological reward-based learning, and thus we believe it is a meaningful contribution.

We sincerely thank the reviewer for their constructive feedback, and we’re glad to hear that you appreciate the significance, scientific rigor, and clarity of our work.

We have fixed the typo on L116, and reworded L134 as “…in many practical applications with large or continuous state spaces…” to avoid confusion; thank you for pointing out these issues.

The reviewer also raised an insightful question: whether the temporal dependency between time-steps due to recurrence would prevent parallelization (and thus improved runtimes). For most recurrent architectures trained with backpropagation-through-time, this is indeed the case; the updates of each layer at each time step must be sequentially computed. However, in AD, unlike backpropagation-through-time, we do not propagate any error signals back along the recurrent connections, thus there does not exist any such dependencies during learning. To elaborate, the forward-in-time recurrent connections only serve to pass activations from upper layers to lower layers; no error information is passed back through these connections. Therefore, the recurrent connections do not prevent parallelization.

Thank you for starting the discussion. I see your summary as correct.

Thanks for this thread.

In general I agree to the high-level summary, although the critique on "the extent of the biological evidence that this model could correspond to real activity/mechanisms in brains" is less an issue in my opinion. That was not the intend of the paper nor did the authors claim this (please correct me if I'm wrong), but rather aimed to show a bit more biological plausible RL algorithm that is inspired by neuroscience and recent ML results (ff), and that eliminates the need of BP.