We would like to thank the reviewer for their valuable comments. They have significantly enhanced the quality of our manuscript.

signposting

We thank the reviewer for and agree with this feedback. We improved the text by:

• Changing the last paragraph of the introduction, summarizing each section
• Highlighting the main questions arising from the discussed examples
• Adding sentences at the start of each (sub)section to point out the main message

Fig 1) A was a great figure...

We now added the needed reference to Fig.1B.

Fig 2) Why were there hexagons everywhere?

This is a quality of the ring attractor in [1]. The attractor implemented by 6 neurons is made up of 6 line attractors, fused at their ends. At these fusion points there is a lack of smoothness (similar to the function abs(x)). Therefore, this piecewise straight "ring" attractor, when projected onto two dimensions looks like a hexagon. We have added clarification to this on the main text.

Further, in B and C...

We find the connecting orbits between the fixed points by identifying the slow part of simulated trajectories. From these slow trajectories, we choose the trajectory that is closest to two fixed points. These trajectories go from a saddle node to a stable fixed point. We do this for every pair of neighbouring fixed points.

Should I be using it as evidence for your big theoretical claim?

In this section, we aim to motivate the relevance of our theory by demonstrating two key points: 1) that continuous attractors are inherently fragile to noise in the parameters (a well-known fact), and 2) that all bifurcations from and approximations of continuous attractors share a common feature: a slow invariant attractive manifold. In the subsequent sections, we provide an explanation of these universal features.

What am I looking at in figure 4A2...

We appreciate the remarks and have revised the text accordingly.

1. You are correct; this figure shows multiple example trajectories. We have corrected this in the text and caption.
2. We have corrected the mistake of referring to Fig. 4B and C as a limit cycle.
3. We did not include the slow repulsive points that we found as it decreased the interpretability of the figure and this structure is not relevant for how the network solves the task.
4. The grey lines on the torus represent simulated trajectories, indicating the network dynamics after a saccade.
5. Fig. 4A1 shows the output of an integrated angular velocity, illustrating the task in addition to the solution type in Fig. 4C. Fig. 4A1 and 4A2 are different in that they show how the networks behave (in output space). The other subfigures in Fig.4 illustrate the stability structures of the networks, i.e., to which part of state space they tend over time in the absence of input.

On that last point...

We included these details to support reproducibility.

Figure 5A) Why did one of the finite time results go above the line?

This is indeed an exact theoretical result; however, our numerical methods are not exact. Because we numerically approximate the invariant manifold of each trained RNN, on which we calculate the uniform norm of the vector field, we cannot guarantee the vector field simulated trajectories follow to be exact. Additionally, the network initialization along the invariant manifold is approximate due to our parametrization (using a cubic spline). Nevertheless, it is important to note that this method has only a single violation of our theory among approximately 250 networks tested.

Seemed obtuse...

We appreciate the reviewer pointing this out. We have corrected the mistake and now observe the log-linear relationship, which aligns with the theoretical expectation. The angular error should asymptotically approach zero as the number of fixed points increases, making the log-linear plot the appropriate representation for this relationship.

Did you define the memory capacity?

We determine the location of the fixed points through points of reversal of the flow direction (the system evolves along a 1D subspace). We calculate the probability of a point converging to a stable fixed point by assessing the local flow direction (which allows us to characterize the basin of attraction). The memory capacity is the entropy of this probability distribution. The definition is in S.4.3.1., referenced in the main text. We hope these definitions are clearer in the new version of Sec. 5.

Did you need to introduce omega-limit set...

We believe that this definition is supporting the definition of memory capacity. As we explained above, the memory capacity is calculated from the omega-limit set of each of the points on the invariant ring. This idea can be more generally applied to systems with other omega-limit sets, like limit cycles or chaotic orbits and therefore included this definition.

You should definitely cite...

[2] analyzes an Ising network perturbed with a specially structured noise at the thermodynamic limit. Although their analysis elegantly shows that the population activity of the perturbed system does not destroy the Fisher information about the input to study instantaneous encoding, they do not consider a scenario where the ring attractor is used as a working memory mechanism. In contrast, our analysis involves understanding how the working memory content degrades over time due to the dynamics. We are not aware of any mean field analysis that covers this aspect. We include this work to discuss continuous attractors in mean field approaches.

What was section 5.2 trying to show?

See the shared rebuttal for our clarification of Sec. 5.2.

References

[1] Noorman, M. et al. (2022). Accurate angular integration with only a handful of neurons. bioRxiv, 2022-05.

[2] Kühn, T., & Monasson, R. (2023). Information content in continuous attractor neural networks is preserved in the presence of moderate disordered background connectivity. Physical Review E, 108(6), 064301.

We would like to thank the reviewer for their valuable comments and suggestions (and appreciate the recent edit). They have significantly enhanced the quality of our manuscript.

Weaknesses:

Some more control experiments need to be added for proving the generality (See below).

We address the specific questions below, but would like to emphasize that the main contribution of this submission is theoretical. The numerical experiments are meant to illustrate our results, rather than prove them.

Questions:

I believe the slow manifold picture the authors are introducing here is not too different from [1], specifically Fig. 7. Can the authors please clarify the differences in the main text?

We appreciate the remark, and will revise the text to include the following:

1. The two approaches are indeed similar, however there are subtle technical differences.
2. In [1], Sussillo and Barak are primarily concerned with a pointwise definition of slowness, by comparison normal hyperbolicity requires a uniform separation of timescales over the entire invariant manifold.
3. Our theory can explain why perturbations to the trained RNN (with random gaussian noise with zero mean and standard deviation) still lead to the same approximate plane attractor dynamical structure is still in place. The Persistence Theorem guarantees that for small perturbations (of size $\epsilon$) the persistent invariant manifold will be at the approximate same place (it will be at a distance of order $\mathcal{O}(\epsilon))$. See Figure 9 for the experiments of the structural perturbations in [1]. They do not provide an explanation for their observations.

The fast-slow decomposition does not seem to be specific to vanilla RNNs. Can the authors please include experiments with LSTMs and GRUs, which would support their claims on generality.

We have now trained and analyzed both LSTMs and GRUs on the angular integration task. We include our preliminary results in a separate document uploaded to OpenReview.

Similarly, the experiments on Section 4 are centered around ring-like attractors...

We appreciate the comment, and have included an additional task where the approximate continuous attractor is of higher dimension, namely a double angular velocity integration task. Please see the shared reply to all reviewers on our new findings. Specifically regarding addition or multiplication tasks, an idealized solution to either would require that the RNN represent $G = (\mathbb{R},+)$ or $G = (\mathbb{R}_{+},\times)$, which are not compact. Because this contradicts our technical assumptions, we opt to focus on tasks where the invariant manifolds are naturally compact.

similar slow dynamics arguments are made in [2]...

We thank the reviewer for pointing out this work, we will reference it accordingly. This work identifies asymptotic behaviors in dynamical systems, fixed point dynamics and more general cases cycles and chaos. We look beyond asymptotic behavior and characterize attractive invariant manifolds, thereby also identifying connecting orbits (or heteroclinic orbits) between fixed points. We would like to reiterate that we believe that the main contribution of the paper is a new theory of approximations of continuous attractors. Although we developed new analysis methods for dynamical systems to find slow manifolds in them, we do not propose a new general framework for analysis of all dynamical systems. Finally, [2] provides analysis tools for Piecewise-Linear Dynamical Systems, while our methods are generally applicable to RNNs with any activation function.

The presentation in section 5 becomes unclear, and perhaps too dense...

We agree with the reviewer, the updated manuscript will include a substantial revision of section 5. In it, we will focus on clarity. We have split the section into generalization properties of trained RNNs (all relating to Fig.5 and in relation to our theoretical prediction on the error bound) and the four conditions that guarantee that a system that approximates an analog memory system will be near a continuous attractor (which was section 5.2). Please, see the shared rebuttal for our clarification of Section 5.2.

the evidence presented in Fig. 5.

The main message of Fig.5 is to show the validity of our theoretical predictions about the bound to the memory based on the uniform norm of the flow. Besides, we can demonstrate that even though all networks learned a continuous attractor approximation, they are distinguished from one another by their fixed point topology, which determines their asymptotic behavior and hence generalization properties (Fig.5D and E). These results indicate the distance to a continuous attractor as was discussed in Sec.3.2 as measured by the uniform norm of the invariant slow manifold.

The title is not descriptive of the manuscript's content and feels like as if it belongs to a blog post. Could you please update the title to be representative of the paper's conclusions?

The title references the reversal of the Persistence Manifold theorem, i.e., how to get back to a continuous attractor. Furthermore, it references that we can return to continuous attractors as a useful concept to describe neural computation because of the deep connection of all continuous attractor approximations. We can however propose to revise the title to: "A theory of analog memory approximation: Back to the continuous attractor."

References

[1] Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks, David Sussillo and Omri Barak, Neural Computation, 2013

[2] Schmidt, D., Koppe, G., Monfared, Z., Beutelspacher, M., & Durstewitz, D. (2019). Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies. arXiv preprint arXiv:1910.03471.

We would like to thank the reviewer for their valuable comments and suggestions.

Weaknesses

The experiments and associated analyses focus solely on networks that approximate 1D ring attractors. This is quite simplistic, and at least for the numerical experiments, the authors could consider tasks like navigation where a planar 2D attractor is approximated by the networks.

We appreciate the remark, and have included an additional task where the approximate continuous attractor is of higher dimension, namely a double angular velocity integration task. Please see the shared reply to all reviewers on our new findings. The networks develop corresponding approximate continuous attractors that have the same structure as the task requires (in this case a torus).

That being said, we would like to reiterate that the primary contribution is theoretical and the numerical experiments are meant to illustrate the theory. Regarding navigation tasks, two points bear mentioning.

1. For planar attractors are diffeomorphic to $R^2$, note that they do not conform to the assumptions on normally hyperbolic invariant manifolds, since $R^2$ isn't compact. There are suitable generalizations of this theory to noncompact manifolds [1], but we do not pursue them since they require more refined tools, which would only obscure the point that we are trying to make.
2. Tangentially, we would also like to point out that we assume that neural dynamics are naturally bounded (e.g. by energy constraints) and hence sufficiently well described by compact invariant manifolds.

In the revised version of the manuscript, we will include the above limitations and provide reference to [2].

The authors have only qualitatively characterized ...

We thank the reviewer for pointing out the reference; we applied DSA to our numerical results. Our preliminary observations are that DSA reflects the fact that the geometry of the invariant manifold is preserved, but it cannot detect the emergence of fixed-points and saddles on the perturbed manifold. The DSA values clustered around two points regardless of the number of fixed points. This appears to be consistent with the results reported in the referenced paper, c.f. Figure 4 shows a gradual increase in DSA as $\alpha \to 1$ despite having a bifurcation at $\alpha = 1$.

Lastly, we would like to note that the analysis using DSA cannot be trivially automated. As pointed out by the authors of DSA:

1. The DSA 'score' is relative; one needs to compare different dynamics.
2. DSA essentially requires 'learning' or fitting a separate model, which implicitly requires performing model selection with respect to the delay embedding, rank of the linear operator.

For these reasons, we would like to adhere to our initial analyses.

For the generalization analysis, the authors could evaluate generalization performance by the nature/type of the approximate attractor as well.

We looked at the generalization performance by the nature/type of the approximate attractor (Fig.5D MSE vs number of fixed points).

Furthermore, although I may have missed this, could the authors comment on what networks hyperparameters lead to which approximations?

The only networks hyperparameters that we varied were the nonlinearity and the size. In all our figures we show which nonlinearity and size corresponds to which fixed point topology (which we characterize through the number of fixed points on the invariant ring).

The figures and presentation could be improved [...]

We appreciate the comments, and changed the manuscript accordingly.

Overall, the writing could be improved in several places to improve clarity.

We improved the writing, focusing on overall clarity. See the shared comments.

the conclusions of the generalization analysis and their implications are not very clear...

In the revised version, we will make a stronger point that connects the inherent slowness of the invariant manifold to the generalizability of the approximate solutions. We also added a longer description of the implications of our numerical experiments to the main text.

Questions:

How do the authors identify the various kinds of approximations of the attractors? Can this be automated, perhaps by using to DSA to cluster the various types?

We identify approximations by their (1) attractive invariant manifold (as motivated by the theory) and (2) asymptotic behavior (as motivated by our analysis of perturbations and approximations of ring attractors). The invariant manifold in our examples typically take the structure of a ring with fixed points and transient trajectories on it. In the supplementary document (FigR.1) we show that the identified invariant manifold indeed reflects the fast-slow separation expected for a normally hyperbolic system. We find the fixed points and their stabilities by identifying where the flow reverses by sampling the direction of the local flow for 1024 sample points along the found invariant manifold. The only example we found that is of another type is the attractive torus (Fig.4D). For this network, instead of finding the fixed points, we identifies stable limit cycles where there was a recurrence of the simulated trajectories, i.e., where the flow returned back to an initial chosen number of time steps (up to a distance of $10^{-4}$).

For the difficulties of using DSA, see above.

At what level of performance are all trained networks compared? Are they all trained until the same loss value and how close is this MSE to 0?

All networks are trained for 5000 gradient steps. We exclude those networks from the analysis that are performing less than -20dB in terms of normalized mean squared error tested on a version of the task that is 16 times as long as the task on which the networks were trained.

[2] Eldering, J. (2013). Normally hyperbolic invariant manifolds: The noncompact case (Vol. 2). Atlantis Press.

We would like to thank the reviewer for their valuable comments and suggestions.

Weaknesses:

their theoretical investigation and results are limited to a few very simple systems, low-dimensional systems [...]

We respectfully disagree with the stated limitation-- the role of the analysis, and numerical experiments is not to prove the generality of the theory but to illustrate it. Hence, we focus on visualizable low-dimensional systems and are more helpful in developing intuition. Nevertheless, we include additional results with RNNs trained on a 2D task. In the updated manuscript, we emphasize that the theory holds under broad, practically relevant conditions.

1. We added statements that assure that our theory is applicable regardless of the dimensionality or the invariant manifold (see shared rebuttal).
2. Furthermore, we will revise Theorem 1, to show that normal hyperbolicity is both sufficient and necessary for invariant manifolds to persist, see [1].

RNNs with specific activation functions and restrictive settings

We appreciate the reviewer's insightful comment. In response, we have conducted additional experiments with LSTMs and GRUs, for which the results are included in the supplementary document and discussed in the shared rebuttal.

Under what conditions the perturbation function p(x) induces a bifurcation?

Continuous attractors satisfy the first-order conditions for a local bifurcation; that is, they are equilibria, and their Jacobian linearization possesses a non-trivial subspace with eigenvalues having zero real parts. Consequently, any generic perturbation $p(x)$ will induce a bifurcation of the system. For a more comprehensive discussion on this topic, see [2].

What types of (generic) bifurcations can arise from the perturbation p(x)?

We are working to characterize codimension-1 bifurcations for a ring attractor and believe a simple polynomial normal form can be derived. Characterizing bifurcations with codimension $n > 2$ is an open problem. Perturbations in the neighbourhood of a ring attractor (in $C^1$ topology) will result in no bifurcation, a limit cycle, or a ring slow manifold with fixed points.

the functions h and g are also not clear

The essence of Theorem 1 can be restated as follows: if the function $f$ has a normally hyperbolic invariant manifold (NHIM), then there exist vector fields $h$ and $g$ that satisfy the conditions for equivalence between the systems defined by Eq.2 and Eqs.3&4. This means that the existence of these functions is guaranteed under the condition of having a NHIM, but their explicit forms is case specific.

What does sufficiently smooth mean in Theorem 1?

We mean that a system needs to be at least continuous, but some extra conditions apply if a system is not differentiable (discontinuous systems are not considered in our theory). Since all theoretical models and activation functions for RNNs are at least continuous and piecewise smooth, our theory is broadly applicable. Center manifold are unique, and generally are local in both space and time and therefore invariance under the flow generally cannot be analyzed using them.

It is unclear under what conditions RNN dynamics can be decomposed into slow-fast form to which we can apply Theorem 1.

Theorem 1 holds for all dynamical systems that have a normally hyperbolic continuous attractor. For example, RNNs with a ReLU activation functions can only have such continuous attractors. The continuous attractors and their approximations that we discuss are all normally hyperbolic. In fact, there is a huge benefit from having normal hyperbolicity as it can counteract state noise.

line 213 [...]

The transformation from the continuous-time to the discrete-time is independent on the function $f$ anf the matrix $W$. However, it is important to note that the discretization process can result in significantly different system behavior depending on the activation functions used. For instance, discretization can introduce dynamics that are not present in the continuous-time system.

In S4, the sentence "All such [...]

We will reformulate it as "all such perturbations leave the geometry of the continuous attractor intact as an attractive invariant slow manifold, i.e. the parts where the fixed points disappear a slow flow appears." The persistence of the invariant manifold under perturbations is a direct consequence of the normal hyperbolicity condition in Theorem 1. Therefore, for a normally hyperbolic continuous attractor there will remain an attractive slow invariant manifold.

Questions:

How does the selection of a specific threshold value...

We believe that the threshold value used to identify slow manifolds is robust- we used the same value for the newly added RNN models, without modification.

Could you elaborate [...] emergence of persistent manifolds?

It is unclear for us to which threshold the reviewer is referring. The emergence of persistent manifolds happens under the 4 conditions we discuss. We demonstrate that all systems with a sufficiently good generalization property (in our case, defined as networks with NMSE lower than -20 dB) must have a NHIM. The persistence of these manifolds is a direct consequence of their normal hyperbolicity.

Limitations:

The authors have discussed most limitations [...]

We appreciate the reviewer's suggestion to make the limitations of our analysis more explicit. In response, we have included a dedicated Limitations subsection in the discussion section of the manuscript. Please refer to the shared rebuttal for further details.

[1] Mané, R. (1978). Persistent manifolds are normally hyperbolic. Transactions of the American Mathematical Society, 246, 261-283.

[2] Kuznetsov, Y. A., Kuznetsov, I. A., & Kuznetsov, Y. (1998). Elements of applied bifurcation theory (Vol. 112, pp. xx+-591). New York: Springer.

I really appreciate the authors' responses and new experiments, particularly the additional experiments with LSTMs, GRUs, and RNNs trained on a 2D task.

• Regarding the first weakness I mentioned, my main concern was not about the role of numerical experiments in proving the generality of the theory. Rather, it was mostly about:

1. Whether and how the theory itself is applicable to higher-dimensional systems in the sense that the conditions of the theorems can be met for high-dimensional systems (especially as many systems in neuroscience are high-dimensional). Nevertheless, I appreciate the revision of Theorem 1 to show that normal hyperbolicity is both sufficient and necessary for invariant manifolds to persist.

2. Bifurcation and stability analysis in Appendix (S2) are not only restricted to a low-dimensional system but also to a system with specific parameters (eq. (9)). One can at least consider such analysis for a more general case, e.g., W = [w11 w12; w21 w22]. Then, one can mention that in some cases, even when having a theory for low-dimensional systems, it can be extended in a similar way to higher-dimensional systems, for instance, through center manifold theory, where the higher-dimensional system has the same low-dimensional center manifold.

• Regarding the perturbation p(x) (the second mentioned weakness), thank you for the clarification. Certainly, dynamical systems with nonhyperbolic equilibria and/or nonhyperbolic periodic orbits are not structurally stable, meaning that any ϵ-perturbation (in the sense of Definition 1.7.1 on p. 38 of [1]) will induce a bifurcation of the system. But, please note that in Sect. 3.1, based on your response, "$l$" must be nonzero. Otherwise, one can consider $l=0$, which implies there is no continuous attractor, which might lead to confusion, as it did for me. So, I suggest changing "$l$" to "$l \neq 0$" in Sect. 3.1.

• Finally, I agree with Reviewer CAVv that it might help to further clarify Sect. 4.1 to demonstrate the relationship between equations (6) and (7).

---

[1] J. Guckenheimer and P. Holmes, Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields, Springer-Verlag, New York, 1983.

Dear Authors,

Due to your new experiments and the fact that most of your responses were convincing to me, I will wait until the end of the discussion period. I will raise my score to 6 if you can address my concerns regarding the bifurcation and stability analysis; otherwise, I will raise it to 5.

Kind regards, Reviewer

[1a] Fenichel, N. (1979). Geometric singular perturbation theory for ordinary differential equations. Journal of differential equations, 31(1), 53-98.

[2a] Kirchgraber, U., & Palmer, K. J. (1990). Geometry in the neighborhood of invariant manifolds of maps and flows and linearization. (No Title).

[3a] https://docs.sciml.ai/DiffEqDocs/stable/solvers/sde_solve/

[4a] Mante, V., Sussillo, D., Shenoy, K. V., & Newsome, W. T. (2013). Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(7474), 78-84.

[5a] Sussillo, D., & Barak, O. (2013). Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural computation, 25(3), 626-649.

[6a] Tzen, B., & Raginsky, M. (2019). Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883.