Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective

Thank you very much for appreciating the technical novelty and efficiency achieved by our method. We are really sorry for missing several details, most of which delve into some details of the state-space model. Please allow us to provide you with a detailed response to questions 1-8.

• Q1: $\mu(ds)$ or $ds$? Typically, describing two orthogonal bases involves the measure ($\mu(ds)$). However, we have opted for the simplified representation seen in prior state space model literature, omitting the measure when the integral context is evident. In the revised version of the paper, we will incorporate the complete $\mu(ds)$ and provide an explanation. Your observation is appreciated.

• Q2: The difference between $\mu$ and $\omega$: In general, $\omega$ denotes the weight function, and $\mu(ds)= \omega(s) ds$. Expressing it in this manner within state space models enables the adaptable manipulation of basis and weight functions, facilitating the derivation of various state space model variants.

• Q3: The connection between $K(t,s)$ and $e^{tA}B$? Equations 2a-2c (line 86) represent the standard equations for state space models [1-6]. These formulas utilize the kernel $K$ to signify the integral transformation. For a differential equation of $x^{\prime}(t)=Ax(t)+Bu(t)$, its general solution is:$x(t)=e^{A (t-t_0)} x (t_0)+\int_{t_0}^t e^{A(t-s)}Bu(s) \mathrm{d}s.$ When we assume the initial value is 0, i.e., $x(t_0)=0$, we can express $K(t,s)$ as $e^{tA}B$ based on the definition of matrix exponentiation.

• Q4: Details about $A$ and $\nabla$: You are correct that an attractor for a dynamical system should be multiple, so in our paper, we use sets to represent this. The symbol $\nabla$ at line 158 signifies the distinctions between these attractor patterns. As per Theorem 3, it's crucial for $\nabla$ to be sufficiently large to maintain the stability of these attractor patterns during training (invariance). This necessity leads us to establish multiple orthogonal subspaces to store each $A_i$.

• Q5: The computation of $Ax+Bu$: We have adhered to the tensor shape representation of the state space model (Mamba), and in our open-source code, we have meticulously annotated the tensor shapes at each step of the model. Specifically, Equation 2a represents the expression for continuous states. Given $A, B, \Delta$, we first obtain the discrete versions for actual computation using Equation 8 as $\overline{A}=e^{\Delta A}$ (B, L, D, N) and $\overline{B}=\Delta B$ (B, L, D, N). We then proceed with the calculations outlined in Formula 9 (also $x[k]=\overline{A}[k]* x[k-1]+ \overline{B}[k]*u[k], x[0]=0$ where the index is in L dimension).

• Q6: Details about $\Delta$ and $\theta$:In the field of state space models, early studies such as LMU (NIPS 2019) and HIPPO (NIPS 2020) introduced the parameter $\theta$ to represent the size of the measurement window, mainly in theoretical contexts. However, in practical coding, the $\theta$ term was eventually removed through formula adjustments. Progressing from S4 (ICLR 2022), state space models began incorporating a trainable discrete step size $\Delta$ as a dynamic measurement window $\theta$, with additional insights provided in Figure 3 (a). In Remark 3.4, we say $\Delta$ as an attention mechanism. This terminology is also used in HTTYH (ICLR 2023) and Mamba (ICML 2024). By structuring $\Delta$ as (B, L, D), where each time step of the input signal has a unique discrete step size, the model can adaptively assign different weights to information at each step, revealing important sequence elements. This behavior resembles an attention mechanism (a gating mechanism) where attention scores are computed not through pairwise products but via a linear layer directly derived from the data itself (Line 120).

• Q7: Details about $\Delta B$: Given $B$ (B,L,N), $\Delta$ (B,L,D), $\Delta B$ is in shape (B,L,D,N). The corresponding code is DeltaB = Delta.unsqueeze(-1) * B.unsqueeze(2), which means (B,L,D,1)*(B,L,1,N).

• Q8: Details about $H$: $H$ denotes the hierarchical projection matrix we have defined. As detailed in line 136, the state space model allows us to estimate the dynamical trajectory using a series of $\theta /\Delta$. However, attractor structures exhibit diverse forms (points, loops, surfaces), necessitating multiple scales of $\theta (e.g., 2\theta, 4\theta, 8\theta)$ for accurate approximation. To address this, as illustrated in Figure 3 (b), we iteratively combine $\theta$ in powers of 2 (repeatedly folding the sequence to acquire representations at varied scales). The matrix $H$ signifies the projection from the prior scale to the subsequent scale. Based on the distinctions between the left and right $\theta$ intervals, it can be further segmented into $H^{\theta1}$ and $H^{\theta2}$. These concepts encompass piecewise polynomial approximation and wavelet transforms (line 466).

• W2: About Chaos System: Chaos, a characteristic of nonlinear dynamical systems, has been a focal point of extensive study. The Takens and Whitney theorems serve as valuable tools for exploring chaos. Following your recommendation, we will change Chaos perspective to Dynamic systems perspective within the document.

• W1,W3,W4: Paper Presentation:** We trust that the responses to questions 1-8 have assisted in clarifying any uncertainties related to symbol interpretations. In the revised manuscript, we will rephrase sections related to the Takens theorem and integrate a more thorough background on state space models to ensure accessibility for readers across various disciplines.

Considering reviewers zUbZ/zqBV, all agreed that our paper has good merits such as satisfactory novelty, good evaluation, and efficiency, we believe our research findings are worth sharing with the research community. If your queries have been addressed, we kindly request an increase in the rating and confidence level.

We sincerely thank you for appreciating the theoretical design of our model and its efficiency. We apologize for missing several details and would like to clarify as follows:

• W1: Unclear presentation of results: Your comment is really helpful. We have followed your comments to indicate the best/worst performance in our revision. Thank you.

• W2: Lack of several architectural details: Thank you for pointing out this issue. Please see our reply in Q1.

• W3: Missing error bars: Thank you for your correction. Subsequently, we chose 3 random seeds for experimentation. Due to time constraints, we have provided the standard deviation results （MSE）for select datasets. The comprehensive experimental error data and associated error bars will be included in the Appendix. ||ETTh1|ETTm2|Weather| |---|---|---|---| |96|0.002|0.003|0.003| |192|0.003|0.003|0.001| |336|0.001|0.002|0.001| |720|0.005|0.003|0.002|

• W4: Several non-academic words: We have revised this part based on your advice. Thanks for raising this concern.

• Q1.1: The complete architecture of Attraos: We have streamlined the architecture by removing redundant modules from Mamba (such as gated multiplication and local convolution), keeping only the fundamental SSM kernel $K(t,s)=e^{tA}B$. This refinement enhances interpretability and recent studies suggest that redundant modules can introduce side effects in time series prediction tasks (https://arxiv.org/pdf/2406.04320, https://arxiv.org/pdf/2405.16312). To facilitate a deeper grasp of the Attraos framework, we present pseudocode below for your enhanced understanding, and a model pipeline structure diagram will be added in the appendix:

• Q1.2: Details about Hopfield Evolution: The Hopfield network evolution strategy is one of the three mainstream attractor evolution methods mentioned in the paper (line 178). It is positioned alongside frequency domain evolution and direct evolution strategies, used for system evolution on the dynamic representation built on multi-resolution dynamic memory units (MDMU). A detailed introduction to the Hopfield network is provided in Appendix A2 on page 13, and the modern version of the Hopfield network(https://arxiv.org/pdf/2008.02217) is essentially a cross-attention mechanism. By employing predefined trainable tokens as attractor memory banks (Key and Value in attention), and using data as queries, the overall energy function of the system is minimized during model training to achieve stability. The trainable tokens in the stable state can be considered as the system's attractor memory.

• Q1.3: Experimental hyperparameters: The experimental hyperparameters can be located in Appendix D3 on page 21. For further details concerning SSM and mathematical aspects, please refer to my response to the third reviewer (qpQ4).

In the revision, we have carefully incorporated your comments in the revised paper. Considering reviewers zUbZ/qpQ4, all agreed that our paper has good merits such as satisfactory novelty, good evaluation, and efficiency, we believe our research findings are worth sharing with the research community.

Thank you very much for appreciating our technical novelty and the SOTA performance achieved by our method. We are really sorry for missing several details. Here we endeavor to address your questions.

• W1: Lack of discussion on model settings:

[why using this particular SSM]: The paper opts for an $A$ matrix structured as {-1, -1, -1...}, considering it a rough approximation of $Hippo-Leg$, which relies on finite measure window approximations utilizing Legendre polynomials (Line 478), aligning with our objective of approximating dynamical systems using windows $[t, t+\theta]$.

[how the hyperparams are chosen]: In the realm of SSM-related investigations, diverse $A$ matrices exhibit varying characteristics; for example, a diagonal matrix like {-1, -2, -3...} might emphasize past information, while a step matrix could function as a localized attention mechanism, treating specific time steps collectively. Table 3 illustrates the performance stability enhancement of our designated $A$ matrix compared to randomly initialized $A$ matrices. The revised article will encompass a more extensive variant analysis.

[Limitation]: Irrespective of the chosen $A$ matrix, it's crucial that its diagonal elements are negative (following left-half plane control theory) to avert gradient explosions. Moreover, the diagonal state space may limit the expressive power of the model. This paper partially addresses this issue by introducing multiple orthogonal subspaces (MDMU block).

• W2: Missing additional analysis.:

[non-toy experiment]: In Appendix E, we have delved into Chaos Modulation (E4: rectifying trajectories with real values throughout model evolution), Chaos Reconstruction (E5: visualizing the constructed model's dynamical system alongside the actual dynamical system), and Chaos Representation (E6: exploring the impacts of truncating and non-truncating trajectories). Should you necessitate additional detailed analysis, kindly inform us, and we will promptly address your queries.

[the dynamics of other baselines] As this paper is the first to restore the underlying dynamic structure in a time-series prediction benchmark using PSR technology and innovatively utilize polynomial estimation, whereas other baseline models are purely data-driven, it is challenging to access the dynamic systems built by other models. Detailed explanations regarding the real dynamical systems behind the datasets are provided in Appendix E1 and E2.

[Polynomial distribution] Given that the state-space model is initialized from specific polynomials and then adapted through open matrix gradients, making it challenging to access the specific polynomial distributions behind the model. In recent work, the state-space model has been defined to adaptively update to the most suitable polynomial space for modeling the dynamics of the time series through gradient updates. (https://arxiv.org/pdf/2405.16312)

• Q1: Notation of PSR: Takens embedding often refers to phase space reconstruction, while Takens theorem is broader, ensuring attractor structures can be recovered even in dimensions greater than twice the original sequence. These are roughly equivalent terms without a unified label in dynamic literature.

• Q2: Local Evolution: Local evolution is not actually a single-step autoregression. In lines 81-83, the adjacent dynamic trajectory points belonging to the same attractor will undergo evolution using the same evolution operator $K_i$ (for example, time steps {1, 7, 8, 11} use the $K_1$ operator, and time steps {2, 5, 67, 68} use the $K_2$ operator). In the direct evolution strategy, we employ the KNN clustering algorithm to partition the points belonging to each attractor based on the Euclidean distance in phase space. In the frequency domain evolution strategy, we directly use dominant modes in the frequency domain to represent attractors.

• Q3: Frequency Evolution: [Motivation]: We chose the frequency domain evolution strategy inspired by articles in the field of neuroscience, where attractors amplify in the frequency domain (lines 54-56). [Limitations]: Regarding interpretability, the frequency domain evolution strategy is less effective than direct evolution (dividing attractors based on Euclidean distance clustering). [Advantages]: Direct clustering can lead to incorrect attractor divisions due to the significant noise present in real-time series data. Frequency domain evolution, on the other hand, filters out some high-frequency noise, enhancing the model's robustness and consequently improving its performance.

In the revision, we have carefully incorporated your comments in the revised paper. Considering reviewers zqBV/qpQ4, all agreed that our paper has good merits such as satisfactory novelty, good evaluation, and efficiency, we believe our research findings are worth sharing with the research community. We sincerely hope that a revision is still considered.