We thank the reviewer for comments of our work and suggesting related work relevant to the paper. We also thank the reviewer for highlighting that the exponential capacity is novel in the context of sequential memories. We appreciate the thought of the reviewer to suggest incorporating kinetic effects to consider the dynamic properties of the network, which is required for more general settings of sequence models that generalize the Markovian sequence transition assumption we make in the paper.

Another issue is that although the capacity is exponential in N, the model requires one hidden unit per memory, so the total number of hidden units also grows exponentially with N. Consequently, when counting all neurons in the network, the overall capacity scaling is only linear. Having exponential capacity only in the visible neurons is therefore not sufficient; the authors should either devise a way to reduce the number of hidden units required or clarify this important point about scaling. Otherwise, the defined capacity is not the most relevant metric.

We thank the reviewer for the opportunity to clarify this point. The capacity argument is a common criticism of the literature of Dense Memory models and is used in many landmark paper. Here, there is a subtlety in the definition of capacity - i.e capacity is defined as the maximum number of memories that can be stored as function of the dimensions in the state space (Line 197 in the paper) and not the actual neurons in the network. This is a subtle point and we have adjusted our discussion to make this clear. We have also introduced a limitation section to enumerate this weakness in defining capacity in this manner.

The reason why capacity is defined in this way is because the hidden neurons in Eqn (1) are often not explicitly written out as separate neurons, instead these variables are assumed to be part of dynamics of the visual neurons $(v_i)$ itself (by substituting $h_\mu$ in the dynamical equation of $v_i$). This requires an assumption that the synapses have many-body interactions (see reference $1$ below). Even though this definition of capacity may seem to inflate the actual capacity, not all neural networks can accommodate arbitrary high hidden neurons for a fixed number of visual neurons, which is the main problem tackled by dense associative memory models. e.g. Classical Hopfield networks can accommodate only a small fraction of the original visual neurons as hidden neurons without introducing spurious patterns.

$1$ Krotov, Dmitry, and John J. Hopfield. "Large Associative Memory Problem in Neurobiology and Machine Learning." International Conference on Learning Representations. 2021

Finally, regarding Figure 5: to achieve exponential capacity the model assumes high-order, all-body interactions (obtained by expanding the exponential in the soft-max interaction term). While the figure nicely demonstrates the dynamical traces of the slow neurons, it is a bit of a stretch to call them time cells and ramping cells.

Thank you for raising this excellent point. The model we have is abstracted away from the biophysically realistic neural models like Hodgkin-Huxley neural networks. As such there are limitations to the extend to which the biological claims can be made. In the section, we make the case that the dynamical behavior is correlated - which is still interesting because this match to biology was not originally intended in the model. We only make the claim that the dynamical characteristics show similarities to biology. The perceived role of ramping/time cells also match some of the predictions we make from the theory. The evidence we show is only correlational and further work may be required to better quantify this connection

We thank the reviewer for engaging with our responses. We want to clarify that the term "Exponential" in the title refers to the exponential interaction terms in the model, which we indeed have in our model. i.e the hidden neurons have exponential activation with normalization in Eqn 1 of our paper.

"Also, other reviewers pointed out the highly relevant reference Karuvally 2023, which after comparison I think the novelty in the current paper mainly lies in the theoretical analysis (which I appreciate), since the two key ingredients: slow-fast dynamics, exponential interaction term, are both not new. Therefore, I think the writing should emphasize that this is a theoretical analysis on Modern Hopfield Network with Sequence Memory, rather than emphasizing that this is a new architecture call EDEN."

Thank you for pointing to the work and providing an opportunity to clarify the distinction with Karuvally 2023. The model we introduce in Eqn 1 is different from Karuvally 2023. Karuvally 2023 only considers polynomial interaction terms in their Dense GSEMM compared to the exponential terms we have in EDEN. We respectfully urge the reviewer to refer to the appropriate section of the Karuvally 2023 paper. The exponential terms are new and the theoretical analysis is a nontrivial extension of the Karuvally 2023 paper and provides rigor to the claims raised there.

Karuvally, A., Sejnowski, T. & Siegelmann, H.T.. (2023). General Sequential Episodic Memory Model. <i>Proceedings of the 40th International Conference on Machine Learning

We thank the reviewer for positive comments on the mathematical analyses in our manuscript and the clarity of our figures. We address the weakness raised by the reviewer below.

I think that, unless I am missing something, the main claim of the paper title and abstract is not entirely reflected the content, as it the authors claim exponential memory capacity, whereas...

We thank the reviewer for the opportunity to clarify this point. The capacity argument is a common criticism of the literature of Dense Associative Memory models and is used in many works. Here, there is a subtlety in the definition of capacity - i.e capacity is defined as the maximum number of memories that can be stored as function of the dimensions in the state space (Line 197 in the paper) and not the actual neurons in the network. This is a subtle point and we have adjusted our discussion to make this clear. We have also introduced a limitation section to enumerate this weakness in the defining capacity in this manner.

The reason why capacity is defined in this way is because the hidden neurons in Eqn (1) are often not explicitly written out as separate neurons, instead these variables are assumed to be part of dynamics of the visual neurons $(v_i)$ itself (by substituting $h_\mu$ in the dynamical equation of $v_i$). This requires an assumption that the synapses have many-body interactions (see $1$ below). Even though this definition of capacity may seem to inflate the actual capacity, not all neural networks can accomodate arbitrary high hidden neurons for a fixed number of visual neurons, which is the main problem tackled by dense associative memory models. e.g. Classical hopfield networks can accomodate only a small fraction of the original visual neurons as hidden neurons.

Regarding Fig.5, is it possible / straightforward to adjust the model time scales to more closely match the observed data?

Thank you for the suggestion. It is not straightforward to match the parameters of our network with the biological data. The model we have and the energy-based networks in general are abstract models of memory storage. To our knowledge, there is currently no known setting of the parameters ($\alpha$'s and the timescales) that are validated from biological data. This is why we claim only that the biological connection is qualitative. If a procedure is devised to obtain these parameters from biological knowledge and use it for prediction, we may be able to predict some high level behavior of biological systems, this is a very interesting avenue to pursue in future work.

$1$ Krotov, Dmitry, and John J. Hopfield. "Large Associative Memory Problem in Neurobiology and Machine Learning." International Conference on Learning Representations. 2021

We thank the reviewer for engaging with our rebuttal. We agree that the dynamic attractor dynamics is a central contribution of our paper. We agree that the term exponential for the capacity is misleading but, this is the terminology used in the modern Hopfield Network capacity. For further clarification, we urge the reviewer to consider the following updates we propose to the manuscript:

1. We mention in the abstract that the capacity. "The analysis of capacity for EDEN shows that it achieves exponential (in the dimensions of the feature space) sequence memory capacity ..."
2. We adjust the language in the introduction Line 79 to "The network capacity scales exponentially in the number of feature dimensions (the number of visible neurons)" and points where we write in the text "exponential in the number of neurons" to "exponential in the number of feature neurons".
3. We rewrite the bound as $P_{\max} = O(\min(\gamma^N, P))$ with $P$ as the number of hidden neurons in
4. We add a clarifying paragraph in the capacity section -"Note here that the exponential capacity as defined here takes into account only the number of neurons in the feature layer (the state space dimension) following similar capacity definition used in state associative memory models $Krotov 2021, Demircigil 2017, Ramsauer 2021$ . The capacity is still linear in the number of hidden neurons, as there is one neuron per memory stored in the network. The hidden neurons are typically removed in the count as (1) they do not count meaningfully to the state space dimension - which contains $O(N)$ variables and can be substituted out in the dynamical equations if we were to allow for many body synapses $Krotov 2021$ , (2) Not all network interactions can accommodate exponential number of hidden neurons for a fixed number of feature neurons. Eg. the polynomial interaction sequence networks considered in $Karuvally 2023, Chaudhry 2023$ can accommodate only $O(N^d)$ memories in the state space dimension, where $d$ is the polynomial interaction power. So, the scaling relationship of the number of memories with the number of visible neurons is an important consideration in the design of sequence networks."
5. We add a limitation section that includes the statement
   • "The exponential capacity we obtained requires an identical exponential increase in the number of hidden neurons, the question of how network capacity can be increased without taking hidden neurons into consideration is still a problem in both static (Hopfield-like energy) and sequential (dynamic energy) networks."

The Dense Associative Memory models paper [1] (from which the bound is taken)

We want to clarify that we do not take the bound from any paper. There have been similar use of exponential interactions in the dense associative memory models, we have tried to cover them in our introduction section. However these models are very different from the ones we study in our paper. We reworked these bounds and found them nearly identical in the form of scaling. The bounds we found, however, are not exactly identical as we had to re-derive the bounds under the new conditions of our model.

I do want to state that I do really like theoretical and empirical contributions of the time-varying dynamics in Hopfield networks....

We thank the reviewer for the positive comments and the potential impact of the contribution in the field.

The thank the reviewer for the comments and highlighting the grounding of our core contributions in rigorous theory. We also thank the reviewer for positive comments on the presentation. We appreciate the level of detail and meticulousness involved in formulating this review and the attention to the math that enabled finding typos in the document. Prior to addressing the concerns, we want to point out a key assumption of Markovian transitions we make in the theory that enables us to obtain analytically tractable results.

Markovian Assumption

In line 195, we make the assumption that the capacity is computed at the phase change boundary, but the key assumption required to obtain the analytic form of Eqn (10) for the capacity is that the state transitions encoded in the network are Markovian and the network stays in a memory state long enough for the previous memory state to dissipate. i.e. Eqn 43 in the appendix is such that $s_i(t) = \sqrt{r} \xi^{(2)}_i + \xi^{(1)}_i (1 - \sqrt{r} ) \approx\sqrt{r} \xi^{(2)}_i$, which happens when $r=\alpha_s/\alpha_c$ nears 1, but does not necessarily have to be exactly 1, where the escape time is infinite. Due to exponential decay, the information is lost in an exponential rate, so the approximation error will be exponentially bounded.

In sequential networks where we observe stable state transitions, this condition seems to be satisfied. Without this condition, the network will easily recall spurious memories .i.e the network may converge to a spurious memory that is a combination of $\xi^{(2)}$ and $\xi^{(3)}$ when transitioning from $\xi^{(2)}$ because some signal of $\xi^{(1)}$ is left in the state. We have amended our discussion to clarify this assumption better. Most of the questions posed are related to this.

Line 88 of the main paper suggests that the 'Long Sequence Hopfield Memory' paper does not use energy arguments.

Thank you for raising this concern. There are no energy arguments in the "Long Sequence Hopfield Memory" paper. i.e there is no energy function defined for their network, the starting point of the paper is an asymmetrically connected Hopfield network. This is because in their case, the authors consider a discrete sequence network where there is no separation of timescales to define an energy function. We have amended our discussion to include the discrete-continuous difference.

this work seems to exhibit a lot of similarities to...

We thank the reviewer for pointing out the work. The network and the analysis we performed go beyond the aforementioned work in two ways - (1) compared to the above paper, we provide "quantitative" and analytic guarantees on the properties of the network. (2) The networks considered in the Karuvally paper are polynomial activation networks and did not provide mathematical guarantees of capacity, while we provide both empirical and theoretical guarantees of network scalability.

I also suggest suggest adding...

We thank the reviewer for the suggestions. We have amended our introduction to include comparisons to these two paper. We identify two differences compared to the referenced works. In the above two papers, the authors consider an abstract network that learns to encode temporal stimuli, where the plasticity rules play a role in encoding temporal information. In our setting, we analyze network dynamics where the temporal information is hardcoded. There are no plasticity rules in our models, but this is an interesting future avenue to pursue. The second difference is that the model we consider is a continuous time model, which enables applying the energy formulation to understanding the network. Discrete models lack this energy formulation for dynamical behavior. The continuous-discrete difference may also be one of the reasons we find qualitative behavior similar to biological networks.

In my understanding, it appears that the method has been tested ...

Thank you for the suggestion, we will add more datasets in the final paper submission in the appendix. As rightly pointed out, the aim of the paper was to obtain analytically tractable features of the network.

Line 142 of the paper claimed that no non-convergent...

Thank you for pointing this out. It is correct that in the regimes we tested, there were no spurious states. However, if the Markovian assumption is violated and the network is forced to transition before the memory of the previous state is lost, then the low energy state becomes a combination of two encoded memories. e.g. for the sequence of transitions $\xi^{(1)} \to \xi^{(2)} \to \xi^{(3)}$, during the transition from 2 to 3, if the slow signal still has a trace of 1, then a spurious minima is formed as a combination of $\xi^{(2)}$ and $\xi^{(3)}$. Excluding this case is a key assumption we make in the theory when the capacity is analyzed close to the phase change boundary. These spurious states become relevant when the ratio $\alpha_s/\alpha_c$ is far from the phase boundary.

In line 443 (Appendix), ....

We thank the reviewer for pointing this out. The minus sign is actually an $\alpha$ that should be corrected in all the denominators of line 443 starting from $Z = \sum_{\nu \neq 3} \frac{\exp(\alpha( r \langle \xi^{(\mu)}, \xi^{(3)} \rangle + \langle \xi^{(\mu-1)}, \xi^{(2)} \rangle ))}{\exp(\alpha(1+r)(N-1))}$. We apologize for the oversight and is corrected in the manuscript. The bound is still correct.

The first line of page 15 (Appendix)...

Thank you for pointing this out, we have amended the discussion to clarify this point now. Since $\xi^{(\mu)}_i$ is Rademacher distributed, $\xi^{(\mu)}_i \xi^{(\nu)}_j, \mu \neq \nu$ is also Rademacher distributed. Next, the sum of Rademacher random variables are binomial distributed, which when $n$ is large, can be approximated by a Normal distribution. These follow from the standard properties of the distributions.

The way Equation (24) follows from Equation (23) does not seem straightforward...

For equation 24, we assume that the network transitions from 1->2 almost instantly, so in order to obtain an analytically tractable form for this, we set $t_0=0$, $s_i(t_0)=\lambda \xi_i^{(1)}$ and $v_i(s)=\xi_i^{(2)}$, a constant, for the duration. This approximation is valid as long as two conditions are satisfied (1) $\mathcal{T}_d$ is larger than $\mathcal{T}_f$, that is, there is a separation of timescales and the transient transition between the two memory states are negligible and (2) Markovian state transition, that is the network stays in $\xi_i^{(1)}$ long enough till $\xi^{(P)}_i$ is fully lost from the state. The lambda is assumed, and then later found using the energy function dynamics.

Question 1: What is $z_\mu$ in Equation 12 (Appendix)? How does Equation (15) follow from Equation (14)?

$z_\mu = \exp(h_\mu)$ is a short hand for the exp terms. I have defined it now in the manuscript. We apologize for the oversight. We appreciate the reviewer pointing out the typo in Equation 14. The summation over $i$ is is not ordered correctly. Eqn 14 is $\frac{dE}{dt} = \sum_{i} \frac{d v_i}{d t} ( v_i - \sum_{\mu} \frac{z_\mu}{\sum_{\nu} z_\nu} \xi^{(\mu)}_{i} ) + ...$ and $\frac{d E}{dt} = - \sum _{i} \mathcal{T}_f \left(\frac{d v_i}{d t}\right)^2 + ...$. There is no change in the discussion or the results because of these typos.

Question 2: In line 174, why are only ....

This is a result of the Markovian assumption. If the network operates farther from the phase change boundary, the influence of prior states become non-negligible and the network starts showing spurious memories.

Question 3: To derive capacity bounds ...

Yes, the capacity bounds are derived for Rademacher distributed patterns.

Question 4: What is $\hat{v}$  in Equation 48?

Thank you for pointing out this typo - $\hat{v}$ is the slow variable $s_i$.

Question 5: Just to confirm, in line 441, $\alpha$  is indeed $\alpha_s$, right?

Yes, that is correct. The subscript is removed only for exposition.

Question 6: With respect to Equation (59), ...?

Yes, this is added now. $\beta_x = \frac{\cosh(x)}{\exp(x)} < 1$ when $x>0$.

Question 7: In Equation (23), ....

$v_i(s)=\xi_i^{(2)}$ a constant, for the duration. The assumption here is that the the slow timescale enables ignoring the transient when transitioning from $\xi_i^{(1)} \to \xi_i^{(2)}$.

Question 8: In Equation (41),...

To derive the expression for $\lambda$ from Eqn 40, we make use of our Markovian assumption. i.e the memory of the old memory state is lost by the time the network reaches the escape time. From there, we take the coefficient of $\xi_i^{(2)}$ and equate it to $\lambda$ which is defined in Line 406 as a factor that controls to what extend the previous state is reflected in the slow variable before state transition occurs. This assumption becomes valid closer to the phase change bounday $\alpha_s/\alpha_c$ approaches 1, but not exactly 1.

Minor Typos

We thank the reviewer for finding these typos. These are corrected in the manuscript.

We thank the reviewer for highlighting that our proposed extension of the energy paradigm to the temporal domain via a slowly drifting landscape is a clean, powerful concept. and positive comments about our derivation of escape times. We first clarify our key assumption when deriving the capacity formula and address the weaknesses below.

Clarification of Markovian State Transition Assumption

In line 195, we make the assumption that the capacity is computed at the phase change boundary, but the key assumption required to obtain the analytic form of Eqn (10) for the capacity is that the state transitions encoded in the network are Markovian and the network stays in a memory state long enough for the previous memory state to dissipate. i.e. Eqn 43 in the appendix is such that $s_i(t)\approx\sqrt{r} \xi^{(2)}_i$, which happens when $r=\alpha_s/\alpha_c$ nears 1, but does not necessarily have to be exactly 1, where the escape time is infinite. Note that Eqn (10) for the capacity has $r$ as a parameter so the equation can be used to probe what happens close to the boundary (We chose 0.999 for our simulation). In this limit $(1-\sqrt{r}) = 5 \times 10^{-4}$ which is assumed as 0.

In sequential networks where we observe stable state transitions, this condition needs to be satisfied. Without this condition, the network will easily recall spurious memories .i.e the network may converge to a spurious memory that is a combination of $\xi^{(2)}$ and $\xi^{(3)}$ when transitioning from $\xi^{(2)}$ because some signal of $\xi^{(1)}$ is left in the state. We have amended our discussion to clarify this better.

Limitations

We add a limitations sections to collect the assumptions we make for the formulated theory in our paper.

As a theory of dynamical behavior of a non-linear system, we make key assumptions that simplify our mathematical analysis. (1) The synaptic strengths of the neuron interactions are fixed and does not vary during training, in actual biological systems synaptic strengths can change due to short and long term potentiation effects and consolidation (2) The timescales of symmetric and asymmetric interactions are separate - this allows use to treat the asymmetric part as slowly evolving and change the energy function of the symmetric network is response. In human brains, there are different timescales for information processing but the timescales may not be perfectly separated as a distinct slow population of asymmetric connections and fast population of symmetric population, (3) Binary memory - we assume Rademacher distributed patterns for theoretical exposition following related works in the field, although the theory can be similarly worked out for other distributions (4) Markovian State Transition - in deriving our capacity bounds, we assumed that the network spends enough time in a memory state that the historical trajectory information is lost and the state transitions are purely Markovian in nature. Further, the capacity bounds we formulated shows how the maximum number of memories (number of hidden neurons) scales with the number of visual neurons ($v$) following previous results in the field. The number of hidden neurons required for storage however grows only linearly in the number of hidden neurons.

Limitation 1: the analytical proof is performed only at the phase transition boundary (α_s / α_c → 1), a special case where escape times are infinite and the model ceases to function as a sequential memory system.  It is not at all clear that this result holds in the dynamic regime.

See our clarification above. The key assumption we use to compute the capacity is that Eqn (43) in the appendix for the slow signal can be written as $s_i(t) = \sqrt{r} \xi^{(2)}_i + \xi^{(1)}_i (1 - \sqrt{r}) \approx \xi^{(2)}_i$ which happens when $\sqrt{r}$ nears $1$ that is close to the boundary and not exactly on the boundary. The factor $r$ still occurs in our capacity equation and can be used to probe what happens slightly away from the boundary.

For example, we do not assume anything about $\mathcal{T}_d/\mathcal{T}_f$, so these can be tuned to have the capacity guarantee in a reasonable timeframe. We have amended our discussion to make this nuance clear. In our simulations, we used $r=0.999$ not exactly $1$ and the $\mathcal{T}_d$ was set at $20$, which produced sequential transitions in a reasonable time in the network which is the dynamic regime. For the dynamic regime, the ratio just needs to be less than 1.

Second, the experimental validation leans entirely on a Gaussian mean-field approximation that is only tested on tiny networks (N ≤ 35). Without either a larger-scale experiment or a theoretical bound on the approximation error, the capacity result feels more aspirational than proven.

The theory is derived in the limit of large N (asymptotic regime) so the bound becomes more valid as N is increased, The reason we constrain to N<= 35 is because the capacity for these networks are under the 1M (Million) memory limit. The 1M constraint is the limitation of the simulation hardware available to us. Our network still works for large scale tasks - for example, we simulated a 784 (28x28) unit network to obtain Fig 2 in the paper that uses MNIST digits. The capacity for the 784 network is way higher than 1M memories, MNIST has only 60k samples.

Furthermore, the work is not well-situated within the broader context. There is no task-level comparison to modern sequential models (RNNs, Transformers) or even recent sequence-Hopfield baselines (e.g., Chaudhry et al., 2023), making it difficult to assess the practical significance.

The intention of the paper was to describe salient features of an Exponential interacting network using a novel dynamic energy function. Our contributions are primarily in understanding these network mathematically in an analytically tractable way and not to establish any superiority over existing models in practical tasks. This is still an important avenue to pursue, which will be done in future works. We urge the reviewer to consider the merits of the paper within the context of our contributions.

While the use of a drifting energy landscape is a neat twist, the originality is tempered by the fact that asymmetric, multi-timescale Hopfield models have existed for decades (e.g., Kleinfeld ’86; Sompolinsky & Kanter ’86), and a more explicit discussion of what is new versus what is borrowed would strengthen the paper's positioning.

As rightly pointed out, asymmetric multi-timescale Hopfield networks have existed for decades, however, to our knowledge computation of the salient features of the network were analytically intractable, and construction of networks with exponential capacity provably has not been conducted. We thank the reviewer for suggesting the citation of Karuvally et. al. (we have added it now with discussion in the introduction). There are two key changes to the above paper we made. (1) While the above paper provided qualitative descriptions of dynamic behavior using the energy approach, we use quantitative approaches that is analytically tractable, (2) The models proposed in the above paper were polynomially interacting and hence had power law capacity with no proofs of the capacity claims, here we make the interactions exponential, and prove that the capacity is exponential in the number of feature neurons.

Questions

Question 1: Validity and Scale of the Capacity Claim

See the above discussion in the Weakness section. To further support the assumption, note that the slow neuron dynamics (in Eqn 26) shows that previous memory state decays exponentially over time. We further analyzed the capacity equation in the presence of one previous memory (non Markovian with history 1), in which case a spurious pattern can be shown to form if the escape time is not long enough. So, setting the operating point close to the boundary is necessary for the stable retrieval of sequential patterns.

Question 2: Hidden Layer Scaling

We thank the reviewer for pointing this out. The hidden population size is exponential, which is the current practice in Exponential Dense Associative Memory models. This is one of the reasons why the capacity experiments are constrained to networks <= 35 and upper bounded at 1M memories. To find the capacity, we use a binary search style algorithm where a range of patterns from 1 to 1M is searched to find the upper bound. With this change, the capacity experiment time complexity is only logarithmic in the number of memories which is favourable for such dense networks. Beyond this, there are no additional tricks, so the memory also scales linearly in the number of memories.

Question 3: Storing Multiple Sequences

The key assumption needed is Markovian sequences, storing multiple sequences does not need a significant change. circular sequence is only used for convenience in the exposition. We refer the reviewer to Karuvally et. al. where more general connectivity patterns is considered.

Question 4: Link to Neuroscience

The focus of the current paper was on understanding salient features of multi-timescale non-linear dynamical system analytically. As of now, the biological connection is only qualitative. The theory and model we developed is in a higher abstraction level compared to biophysical models like Hodgkin-Huxley models. Due to the abstraction, the high level qualitative features may become evident from the model, but not the fine quantitative details needed to fit the parameters. This is a common weakness of models neural models higher in abstraction.

We thank the reviewer for the positive comments on the manuscript and suggesting that our model clearly expands the current scope of DAM's by introducing a time-variable, the theory we developed to compute quantities like escape time is rigorous, and the connection to biological features is strong. Below we address the concerns raised by the reviewer.

Limitation 1: - ICLR is an ML conference (not a biology conference). What does this model have to say about ML? Does it explain any ML phenomenon or can it be a basis for a new time of ML model? Any experimental results in this direction?

It is our understanding that NeurIPS is an interdisciplinary conference where researchers from ML and neuroscience gather to share ideas. While we have not discussed the merits of our model in the ML domain, identical works in static Dense Associative Memory models have shown promise in both improving and understanding relevant ML ideas. Eg. DAM has strong connections to transformers and can be used as external memory stores. Additionally, recent works have generalization performance of diffusion models can be understood via phase transitions observed in energy function approach.

Limitation 2 - - What is the motication of assuming that the sequential memories are organized in a circle? How robust are the results to this somewhat adhoc (but technically convenient?) construction?

The primary motivation for the assumption is to simplify the theoretical exposition. The main assumption we use is that the sequential transitions are Markovian. i.e. the memory to transition to determined purely from the current memory and not the history of memory transitions in the network. eg. when network moves from 1->2->3, the transition 2->3 is independent of whether the network started from 1 then transitioned to 2 or the network initially started from 2. As such, all theoretical results should hold as long as the escape time is long enough for the network to forget this history.