Sufficient conditions for offline reactivation in recurrent neural networks

We would like to thank the reviewer for their review and helpful suggestions. We appreciate that the reviewer found our theory to be novel, providing interpretability, and an interesting new perspective on reactivations. We are glad that the reviewer found the paper to be well-written and our contributions clearly explained. We have addressed the points and questions in the review below:

It is unclear what is meant with tuning of the value of \tau. Is this for a fixed noise level? Or is it chosen as the optimal for all used noise levels for training?

Response: We choose $\tau$ for the RNN based on the task to ensure it performs optimally, and it remains the same for all noise levels. The intention is to ensure that the network is faster than the inputs or else it does not react quick enough to new inputs, causing the activity to remain stuck and the performance to be poor. In practice, we do not have to tune $\tau$ very carefully, we set it to a reasonable value that ensures that the network's dynamics are fast enough to process the task's inputs and perform optimally. We clarify this point in the revised manuscript.

Use some transparency in Figure 1 f and e so that overlapping points are visible. At the moment it is unclear what proportion of the points is hidden behind the first layer and what their distribution is.

Response: We apologize if the plot was not clear. We tried adding transparency to these plots but they did not aid in visualizing the distribution better, due to the interaction with different shades of color that attempt to show the structure of the attractor manifold, and its correspondence with the environment.

Nevertheless, this is why we provide the kernel density estimates below! The KDE provides an actual distribution on the data, i.e., a principled visualization of the distributions, rather than requiring visual estimation based on occupied points. We hope that this plot aids the reviewer in comparing the distributions.

The the demonstration contribution of this framework could benefit from some additional experiments. Comparison to training without noise and added noise in quiescent state is missing. Comparison to other ring attractor/head direction models is missing.

Response: We apologize if this was unclear from the text. We have compared to training without noise and with added noise in quiescent state in Suppl. Fig. A.2e-f. We have shown in this experiment that these trajectories are erratic, and do not properly correct for lack of exploration arising from training without noise. In an additional experiment, we have shown that the point-to-point distances are much higher for these erratic trajectories than even waking behavior, indicating their implausibility.

It is true that we don't compare to more traditional ring attractor models, with which we have some key differences. The critical distinction is that our model recapitulates statistics of waking navigation, whereas networks with hand-tuned weights, for instance, would only be able to produce a uniform distribution over the attractor manifold in question.

Finally, some of the implications of the proposed framework require more justification. The conclusion that reverse replay could be explained by diffusive reactivations seems like a stretch and should be substantiated better. Also, reactivations such as replay are not the type of reactivations that are found in the networks in the paper (this is admitted in the paper) even though the introduction is discussing those for a large part.

Response: We apologize if our claims were not clear. In our mathematical results, we are only claiming that the steady-state distribution of quiescent activity matches that of waking activity. Specifically, we do not make any comment on the sequential dynamics of quiescent trajectories themselves. Thus, for reverse replay, the only point we make is that for true diffusive rehearsal, reversed trajectories are as probable as forward trajectories. In additional experiments with our RNNs on the head direction task (Suppl. Fig. A.5), we biased the waking moment-to-moment transition structure to be counter-clockwise. We found that quiescent trajectories partially recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were also reflected during quiescence, although less clearly than waking. We do not yet have theoretical justification for this. However, we note that reversals in the trajectories still remain possible and do occur (Suppl. Fig. A.5).

Dear Reviewer NzXs,

We believe our response has addressed all of the reviewers' concerns and would be grateful if the reviewer would consider raising their score. As the discussion period is coming to an end soon, we would greatly appreciate a response from the reviewer. If the reviewer has any further questions, we would be happy to address them. We thank the reviewer again for their helpful suggestions and eagerly look forward to a response.

Dear Reviewer NzXs,

This is a gentle reminder that the discussion period with authors is coming to an end (<20 hours). We believe our responses in the comments and updated manuscript have addressed all of the reviewers' concerns, and we would be grateful if the reviewer would consider raising their score. If the reviewer has any further questions, we would be happy to address them in the short time remaining. We thank the reviewer again for their helpful suggestions and eagerly look forward to a response.

We would like to thank the reviewer for their review and helpful suggestions. We appreciate that the reviewer found our theory and experiments to be interesting, well-described and well-supported. We have addressed the points and questions in the review below:

The paper is potentially confusing to some readers in that the presentation (and the numerical experiments if I have followed) regard training a network with a fixed nonlinearity and trainable weight parameters (i.e. a typical RNN) but the theoretical analysis considers not a fixed nonlinearity, but rather the space of all possible dynamics that could govern the system. I suspect this may misdirect some readers regarding the basic argument (possibly me as well).

Response: We apologize if this was not clear. The idea is that the RNNs trained optimally are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18—according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993). We are thus assuming that our trained RNNs are near the optimum, and our empirical results indicate that they are. Our theoretical results do not make any assumptions about the architecture of the RNN (in fact we have added simulations with GRUs, Suppl. Fig. A.4), only that it is trained to be optimal. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Training a regular RNN on the cost function [...] In short, the authors should, I think, be direct about the assumptions on the RNN necessary for their theory.

Response: We apologize if this was not clear. We would like to state that our mathematical analysis is a theory for high capacity networks capable of legitimately approximating the optimal solution. RNNs are capable of doing this, as they are universal function approximators (Funahashi & Nakamura, 1993) as long as the activation function is nonlinear. We would like to draw attention to a new experiment we have included with GRUs thanks to the reviewer's suggestion, to highlight the universality of our results. We found that our results do hold even with continuous-time GRUs (Suppl. Fig. A.4), showing that these reactivation phenomena are not unique to a particular network architecture or activation function. We add a discussion about this important point in the revised manuscript.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Regarding the quiescent state, I think the authors are perhaps a little too cavalier with the fact that, strictly speaking, the noise term must change in order to guarantee replay. As presented at the moment, it feels to this reader like a weakness that is not adequately addressed. The authors should either supply 1) a reason to believe this sort of change happens in real networks or 2) a demonstration that the deviations are not sufficient to qualitatively change the main result. There is also the possibility that this is an opportunity for the authors. The deviations from perfect replay if the noise does not change amount to a prediction, and one which may have computational advantages (e.g. facilitating exploration in terms of a planning algorithm).

Response: We apologize if there has been a misunderstanding, but the increase in noise variance is not required to see reactivation. It is only required for an exact distributional match for the Langevin sampling. We demonstrate in Suppl. Fig. A.1 (in the updated manuscript) that reactivation occurs and matches the waking behaviour even when the noise variance remains the same during quiescence. The noise variance merely behaves like a temperature parameter for the sampling, reactivation is not precluded if it stays at the same level. As the reviewer points out, there may indeed be related computational advantages which future work could explore. We thank the reviewer for raising this point and the suggestion for the additional demonstration, which we believe improves our work.

Dear Reviewer aZpo,

We believe our response has addressed all of the reviewers' concerns and would be grateful if the reviewer would consider raising their score. As the discussion period is coming to an end soon, we would greatly appreciate a response from the reviewer. If the reviewer has any further questions, we would be happy to address them. We thank the reviewer again for their helpful suggestions and eagerly look forward to a response.

Dear Reviewer aZpo,

This is a gentle reminder that the discussion period with authors is coming to an end (<20 hours). We believe our responses in the comments and updated manuscript have addressed all of the reviewers' concerns, and we would be grateful if the reviewer would consider raising their score. If the reviewer has any further questions, we would be happy to address them in the short time remaining. We thank the reviewer again for their helpful suggestions and eagerly look forward to a response.

I found the link between the theoretically optimal solution for update dynamics and the actual application of RNNs somewhat unclear. Are we to take that (presumably backprop-trained) RNNs employ update dynamics similarly to the theoretical solution; is it biologically reasonable for e.g. equation 18 to be implement by an RNN? If so, is it possible to show that they do; e.g. by comparing these recurrent inputs to the optimal solution? If not, are the experimental results relevant to the theory?

Response: We apologize if this was not clear. The idea is that the RNNs trained optimally (following any optimization scheme) are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18---according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993). We did not show that the network dynamics explicitly align with the optimal solution, though our empirical results indicate that they do. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks. We clarify these points in the revised manuscript.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Relatedly, perhaps this is overly harsh, but I also do not find the result that RNNs without stimuli visit similar states as with stimuli very surprising.

Response: We believe that there may have been a misunderstanding---our claim is much stronger than merely saying that similar states are visited in the absence of stimuli. Instead, we prove and demonstrate that the networks visit the same states with the same, i.e., identical steady-state distribution during quiescence as active behavior. During our biased and unbiased tasks, the same path integrating continuous attractor could theoretically solve both tasks, but then rehearsal would be the same in both conditions. We see that the networks learn different solutions for the biased and unbiased tasks, and that these different solutions produce rehearsals that reflect the task statistics. This is non-trivial, and demonstrating this is, we believe, our main novel contribution. We further clarify this point in the revised manuscript.

The authors suggest an interesting discrepancy in experimental prediction for their noise-based theory of offline reactivation and generative modeling; specifically, that "while generative models necessarily recapitulate the moment-to-moment transition statistics of sensory data, our approach only predicts that the stationary distribution will be identical". I am confused by this statement and not sure if it is true. Is that to say that the proposed model would not encode sensory transitions but would simply replicate the overall probability distribution of sensory states? I would find this surprising for an RNN whose recurrent weights are trained on sensory data; that is, I would suppose that the RNN weights would themselves somehow capture transition statistics of the task variables (e.g. maybe this is valuable as a kind of denoising effect on noisy observations). Perhaps I am wrong though

Response: The reviewer is indeed correct that it is possible for our networks to capture the transition dynamics (and this is what happens in practice), it is just that our theory does not comment on the dynamics of reactivations. In a new experiment with training solely on counter-clockwise trajectories for the head direction task (Suppl. Fig. A.5), we found that the reactivations recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. This would indeed appear to be due to the RNN weights capturing the transition structure, but is not a prediction of our theory. The biased velocities were reflected during quiescence, but less clearly than during waking. However, reversals in the trajectories still occurred (Suppl. Fig. A.5b).

We thank the reviewer for their detailed review. We appreciate that the reviewer found our paper to be "well written" and the theory "well presented and interesting". We have attempted to address the reviewer's concerns and questions below:

perhaps this is not in the scope of the paper, but the authors do not provide any theoretical/numerical support for the functional benefit of offline reactivation. [...] Do the theoretical/numerical results in this paper support these possibilities, or any one in particular?

Response: This is indeed an interesting research question. However, as the reviewer themselves point out, we believe this is out of scope for our current work---our scope here is to explain theoretically the emergence of reactivation and justify this with experiments, not to identify its functional benefits. It is nevertheless an important direction for future work. Many papers have talked about the functional benefits of offline reactivation in other contexts (Mnih et al. 2015; Hayes et al. 2019; Hayes & Kanan 2022), but the reviewer is correct in noting that we would have to concretely demonstrate the benefits of this particular form of reactivation. We acknowledge that this is an immediate direction to be explored and are revising our discussion to discuss the implications of our results in this context.

Mnih et al. Human-level control through deep reinforcement learning. Nature 518, 529–533, 2015.

Hayes et al. Memory efficient experience replay for streaming learning. International Conference on Robotics and Automation (ICRA), pp. 9769–9776. IEEE, 2019.

Hayes & Kanan. Online continual learning for embedded devices. arXiv preprint arXiv:2203.10681, 2022.

Relatedly, a key feature of this study is that the RNNs considered are noisy during awake states. It's unclear to me exactly why this is necessary. The authors "found that even trajectories generated by networks that were not trained in the presence of noise, and also were not driven by noise in the quiescent phase, still generated quiescent activity distributions that corresponded well to the active phase distributions"; is the point that there's more space exploration with noise, giving rise to a functional benefit? Or that somehow the introduction of noise better captures observed neural data?

Response: Indeed, as the reviewer points out, the noise leads to more exploration. Noise has been shown to improve RNN training (Qin & Vučinić, 2018). Furthermore, neural activity in the brain is noisy (Destexhe & Rudolph-Lilith, 2012), for example, in the probabilistic nature of synaptic transmission, and in other ways.

It is critical to note that the match in distributions is across multiple trajectories (multiple initializations), and so does not entirely capture the properties of individual trajectories, such as the amount of exploration.

We thus carried out an experiment to show the amount of exploration of these noiseless trajectories. Indeed we found that individual trajectories do not explore the space well, and get stuck (Suppl. Fig. A.2c-d). Adding noise during quiescence alone does not fix this problem---it makes the trajectories erratic and implausible (Suppl. Fig. A.2e-f), with much higher point-to-point distances than even waking behavior (Suppl. Fig. A.3). So individual trajectories from noiseless networks do a worse job relative to the noisy trained networks. Noisy training stabilizes noisy quiescent activity, which in turn explores more of the task manifold than noiseless quiescent activity. We clarify this point in the revised manuscript.

Qin & Vučinić. Training Recurrent Neural Networks against Noisy Computations during Inference. 2018 52nd Asilomar Conference on Signals, Systems, and Computers (2018): 71-75.

Destexhe & Rudolph-Lilith. Neuronal Noise. Springer, 2012.

We thank the reviewer for reviewing our paper and providing helpful comments that we believe lead us to considerably enrich our work. We are glad that the reviewer found our theoretical approach, which the reviewer has stated is a rare exercise in the literature, to be interesting and addressing an important question. We have provided our responses to the reviewer's comments below:

A number of relevant studied were not cited and discussed. This include work using RNNs and attractor dynamics to model the replay and theta sequences in the hippocampus [...] The paper also misses several pieces work in training RNNs to study the grid cells systems and HD systems. [...]

Response: We thank the reviewer for pointing us to these works. We have cited these in our updated draft. We would also like to stress that the focus of our work is to demonstrate that reactivations are statistical reproductions of waking activity (i.e. the steady-state distribution of reactivations is identical to that of waking activity), and not just observed as a result of our networks learning continuous attractors. We formalize and prove this mathematically, and also show this with experiments. We apologize if this was not clear. We hope that our comments here and in the updated draft improve on this.

The main results are not surprising after considering what we know so far on this topic. Prior work has shown that training the RNNs to perform path integration or angular integration task leads to networks that exhibit attractors dynamics that is similar to continuous attractor models. Furthermore, it is well known that attractor dynamics can perform denoising. Thus it is not surprising that RNNs trained on these previous studied tasks can perform denoising.

Response: We apologize if the main message appeared to be the related to the RNNs learning continuous attractors and performing denoising, this is not our intention. In fact, our very starting point is the recognition that continuous attractor networks emerge with training, it is not one of our major claims. The central finding and novel contribution of our manuscript is to explore the conditions under which reactivation, driven by noise, matches the statistics of waking activity.

With respect to denoising, we believe that one of our novel contributions is to show theoretically how denoising comes to be, and gives rise to statistically faithful reactivations as described below, providing a principled theoretical understanding. We would like to reiterate that our focus is not to show that denoising occurs, although we build on works that explore this idea (Section 2.2--2.3).

More precisely, our novel contribution is to prove and demonstrate that the steady-state distribution of reactivations is identical to that of waking activity, under a well-defined and minimal set of conditions. While this mechanism is related to denoising, which will occur along attractor manifolds, our contribution is to demonstrate that optimal denoising under different task statistics will show rehearsal patterns that reflect those different task statistics. This is the key novel contribution, which we believe is non-trivial in and of itself. We make sure to clearly communicate our findings in the revised manuscript (Section 2.4).

The mathematical analysis relied on a set of approximations and assumptions, which were not well justified.

and

The mathematical analysis relies on a number of assumptions, to the extent that it is difficult to judge whether the conclusion would actually be applicable to the numerical experiments. Can these assumptions be justified or better motivated (beyond the sake of mathematical convenience)?

Response: We thank the reviewer for the suggestion, and we have added a section to the Supplementary Material to clarify the key assumptions for our theory. We provide the assumptions and explanations here:

1. We consider discrete-time approximations of noisy continuous-time RNNs. Using continuous-time RNNs is a common practice in the literature, and we use its discrete-time approximation to be in line with our implementation.
2. The network must be performing some variant of path integration, by integrating change-based information about environmental state variables to some function of these variables. This condition is often met in natural settings, such as spatial navigation.
3. We assume that the inputs to the network are drawn from a stationary distribution, which would translate to assuming that the effects of initial conditions on state occupancy statistics are ignored, and also assuming that the behavioral policy remains constant throughout time.
4. We consider greedy optimization of the loss at every timestep. Greedy optimization is a sensible way of partitioning effort across time in this task: the agent does the best that it can on each timestep, assuming that at each previous timestep the best possible job has been done. In the absence of noise the greedily optimal solution is equivalent to path integration, which is also a globally optimal solution.
5. We assume that the network is performing optimally in the presence of noise. In practice, we train our networks until their loss reaches very low, near-zero values.
6. We assume that our network dynamics can be decomposed into two terms with different functional dependencies. The first term depends only on the activity while the second term depends on all original dependencies—the activity, the state and the change-based inputs.
7. For the quiescent state, we assume that the change-based sensory inputs are zero. This is reasonable because for tasks like those we have considered, which involve integrating self-motion cues, $\frac{\mathrm{d}s(t)}{\mathrm{d}t}$ must be zero during periods of quiescence like sleep, where the animal is not moving and hence doesn't receive sensory inputs associated with self-motion.
8. While we assume that the noise variance is doubled during quiescence to show exact equivalence with Langevin sampling, this is by no means necessary to witness reactivation (Suppl. Fig. A.4). This noise variance is equivalent to a temperature parameter for the sampling.

RNNs trained optimally (following any optimization scheme) are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18---according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993). Our empirical results indicate that the network's dynamics align with this solution. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks. Overall, we believe our assumptions are well-justified, and the conclusions of the theory are indeed applicable to our numerical experiments as evidenced by our results.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

We hope that our response and updates to the draft have addressed all of the reviewer's concerns, and would be grateful if the reviewer would consider raising their score. If the reviewer has any additional questions, we would be happy to discuss further. We eagerly await the reviewer's response.

I appreciate the authors’ detailed response to my critiques. The response helps clarify some of the concerns. In particular, the key contribution is made more clear. I have raised my score.

A couple of issues: -- while the relevant references were added to the revised version, generally there is a lack of appropriate discussion of these studies in the context.

-- the connection to empirical research/ neuroscience data remains to be strengthened. I don’t feel this point was appropriately addressed in the revised version.

Assumptions related to RNNs for theoretical result (and universality of results)

Our mathematical analysis is a theory for high capacity networks capable of legitimately approximating the optimal solution. The idea is that the RNNs trained optimally are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18--according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993), given that the activation function is nonlinear. We are thus assuming that our trained RNNs are near the optimum, and our empirical results indicate that they are.

Our theoretical results do not make any assumptions about the architecture of the RNN (in fact we have added simulations with GRUs, Suppl. Fig. A.4), only that it is trained to be optimal. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks.

We would like to draw attention to a new experiment we have included with GRUs to highlight the universality of our results. We found that our results do hold even with continuous-time GRUs (Suppl. Fig. A.4), showing that these reactivation phenomena are not unique to a particular network architecture or activation function. We add a discussion about this important point in the revised manuscript.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Once again, we thank all the reviewers for their suggestions. We believe that we have addressed all reviewers' concerns in our comments, and eagerly look forward to their responses.