LETS Forecast: Learning Embedology for Time Series Forecasting

Thanks for your detailed feedback. We’re pleased that you recognize the novelty of combining Empirical Dynamical Modeling with deep learning and the strong performance of DeepEDM, especially in noisy settings. We also appreciate your acknowledgment of our connection to the Attention mechanism and the flexibility of our approach. Below, we address your questions.

1. Validation of Latent Space Representation

Great question! The latent space in DeepEDM functions as a learned kernel. We hypothesize that with noise, the learned kernel retrieves nearest neighbors more faithfully in the underlying state space compared to the time-delay embedding, which, as dictated by Takens' theorem, relies on nearly noise-free data for accurate reconstruction. This is indirectly supported by our results in Table 6, where DeepEDM outperforms EDM in time series forecasting.

To provide a more direct validation, we design and conduct a new experiment using simulated time series generated from the Lorenz system, where true neighbors in the state space are known. We evaluate the ability to retrieve these neighbors using different distance metrics. Specifically, for a given state at time $t$, we first determine $k$ true neighbors in the original state space. For the same data point at time $t$, we then retrieve the top $m$ nearest neighbors in (i) the time-delay embedded space using Euclidean distance, and (ii) DeepEDM's latent space using the learned kernel. For each query point, we evaluate the recall of the retrieved neighbors, quantifying the fraction of true neighbors successfully retrieved. For a comprehensive evaluation across different settings, we conduct this experiment with varying values of $m \in \{7, 14, 28\}$, fixing $k = 7$ (the minimum number of neighbors necessary for recovering the dynamics). We also consider two noise conditions: no noise ($\sigma = 0$) and w. noise ($\sigma = 2.5$). A high recall score indicates a distance metric can successfully identify true neighbors in the state space necessary to reconstruct the dynamics. The results of this experiment can be found here:

https://anonymous.4open.science/r/icml_rebuttal-B440/nearest_neighbor_retrieval_exp_results.md

As expected, under noise-free conditions, both methods perform similarly. However, when noise is added, the vanilla time-delayed embeddings exhibit a sharp decline in recall. In contrast, our learned kernel in DeepEDM degrades more gracefully, maintaining more accurate retrieval of neighbors. This suggests that the state space reconstructed using the learned kernel more accurately reflects the underlying state space, as it preserves the local neighborhood structure.

2. Generalization as a Unified Parametric Model

Please refer to our response #1 to reviewer Ywcy.

3. Selection of baselines

We respectfully disagree with the comment, 'baselines (for synthetic dataset experiments) are predominantly EDM-based'. The following baselines were considered (for this experiment) in the paper:

• Simplex: A classical EDM approach, as the starting point of DeepEDM.
• Koopa: A recent deep model based on Koopman operator, as a representative method that combines deep learning and dynamical system modeling (Koopman).
• iTransformer: A recent deep learning-based model, as a representative method that is pure learning based using a Transformer.

This selection ensures a well-rounded comparison against strong and methodologically diverse baselines. We note that the full set of results using these baselines is reported in Table 6 of the Appendix.

4. Missing References on Chaotic Time Series Forecasting

Your suggestion to incorporate prior work on chaotic time series forecasting is well taken. While our focus has been on EDM and deep learning-based forecasting, we recognize the relevance of research in chaotic time series forecasting. We drafted the following paragraph to add to our related work section. We welcome your further feedback.

A related research direction focuses on forecasting chaotic time series via state space reconstruction, mirroring the underlying principles of EDM. Pioneering work by Farmer & Sidorowich (Phys. Rev. Lett. 1987) introduced local approximation techniques within reconstructed state spaces using delay embeddings, facilitating short-term predictions. Subsequent studies explored the application of feedforward neural networks for learning direct mappings from reconstructed phase states to future states (Karunasinghe & Liong Journal of Hydrology 2006). Recurrent neural networks, particularly Echo State Networks (ESNs), have also shown promise, with adaptations like robust ESNs (Li, Hand & Wang. IEEE TNNLS 2012) addressing the inherent sensitivity of chaotic signals to noise and outliers. However, a significant gap remains: the development of a fully differentiable, end-to-end trainable neural network architecture that seamlessly integrates dynamical systems theory with deep learning methodologies.

Thank you for your valuable feedback and recognition of our paper’s clarity, structure, and experimental design. We appreciate the acknowledgment of our contribution to nonlinear dynamical systems through EDM-integrated deep learning. Below, we address your suggestions and will incorporate these responses into the final version.

1. Noise Robustness of DeepEDM

We hypothesize that DeepEDM’s noise robustness stems from its learned kernel, which enables adaptive distance assessments for nearest-neighbor retrieval, rather than using the fixed Euclidean distance in traditional time-delay embeddings. Fixed metrics are more sensitive to perturbations, while the learned kernel adapts to the data, thus improving neighbor identification under noise.

To validate this, we conducted additional experiments (please refer to our response #1 to Reviewer XFih for more details) on nearest-neighbor retrieval. In short, we compare (i) the time-delay embedded space using Euclidean distance, and (ii) DeepEDM's latent space using the learned kernel for retrieving true neighbors in the state space using synthetic data with known dynamics. Our results show that the learned kernel in DeepEDM achieves higher recall under noise, confirming that the learned kernel indeed enhances robustness against noise. This robustness is further supported by the superior forecasting performance of DeepEDM compared to EDM w. Simplex, under noisy conditions (see Table 6).

2. Sensitivity to Time Delay and Embedding Dimension

We conducted additional experiments to study the effects of time delay and embedding dimensions. Specifically, DeepEDM has two parameters that control time delay and embedding dimensions. First, $m$ defines the number of time-delayed steps, and $\tau$ controls the interval between these steps. For example, $m=3$ and $\tau=1$ results in a 3-dimensional embedding $[y_t, y_{t-1}, y_{t-2}]$, while $m=3$ and $\tau=2$ would yield an embedding $[y_t, y_{t-2}, y_{t-4}]$. In our work, $m$ is determined empirically for each dataset, while $\tau$ is set to 1. Following your suggestion, we conduct ablation experiments to analyze the impact of these hyperparameters. For the experiments on $m$, we fix $\tau = 1$ and vary $m$ over $[3, 5, 7, 11, 15]$. Conversely, for the experiments on $\tau$, we set $m = 5$ and explore $\tau$ in $[1, 2, 3]$. The results are linked below.

Effects of time-delayed steps $m$ (i.e., embedding dimension): Our findings indicate that the impact of $m$ is dataset-dependent. In some cases, performance remains relatively stable across different values of $m$, suggesting that the intrinsic state-space dimensionality might either be very low or very high. If the state-space dimension is small, larger values of $m$ will capture the underlying dynamics and thus perform similarly well. On the other hand, if the state-space dimension is large, different small values of $m$ will not be able to model the dynamics, yielding results at a similar performance level. This behavior aligns with the intuition that effective embedding reconstruction depends on the underlying system complexity.

https://anonymous.4open.science/r/icml_rebuttal-B440/m_delay_results.md

Effects of delay interval $\tau$: Our results show that in most cases, $\tau = 1$ yields the best performance, suggesting that a small stride is sufficient. This is also the case considered in Takens’ theorem. However, it remains possible that an optimal balance between $\tau$ and $m$ could further improve results, which we leave for future investigation.

https://anonymous.4open.science/r/icml_rebuttal-B440/tau_results.md

3. Performance degradation on High-Dimensional Datasets

Please refer to our response to Q5 from Reviewer TNFN.

4. Loss Function Impact

Our ablation study (Table 4) shows that our DeepEDM significantly enhances forecasting performance (MLP vs. MLP+EDM, without any loss function optimization) and the loss optimization further reduces the errors (MLP+EDM w/o TDT loss vs. Full Model w. TDT loss). To further delineate the contributions of DeepEDM and loss optimization, we conducted an additional experiment. Specifically, we compare the DeepEDM trained with standard MSE loss against DeepEDM trained with the optimization objective incorporated in our paper. The results of this experiment are available at the table linked below. Notably, the differences remain marginal—while in some cases, the MSE loss further improves DeepEDM’s performance, in others, it results in slight declines.

https://anonymous.4open.science/r/icml_rebuttal-B440/loss_exp_results.md

Thank you for your detailed feedback and recognition of our theoretical contributions. We appreciate your acknowledgment that embedology, despite its foundational role, has been largely overlooked and that our work helps bridge this gap by integrating Takens' embedding theorem with neural networks. Your recognition of our approach as a step toward extending EDM and providing theoretical insights into Transformer-based forecasting is also greatly valued. Below, we address your suggestions and will integrate these responses into the final version.

1. Clarification on Figure 2B

The axis in Figure 2B is presented in absolute terms, which we will clarify in the final version. We chose to not normalize the data (e.g. mean subtraction, standardization) to preserve the raw performance of the model. While this increases error ranges—especially in chaotic Lorenz settings where noise sensitivity is high—it provides a more transparent evaluation. Detailed results are in Appendix A.7.

2. Consistency of Empirical Analysis

Our empirical analysis consists of experiments and results using (a) synthetic time series generated from known, challenging dynamic systems (Lorenz and Rossler); and (2) real-world time series from multiple application domains. For synthetic data, we demonstrated that DeepEDM is more robust to noise when compared to the vanilla EDM, and is more accurate when compared to other representative learning-based methods (Koopa and iTransformer). For real-world data, we showed that DeepEDM consistently outperforms prior methods across different settings (e.g., fixed vs. varying lookback length). This consistency was evidenced by the overall ranking of our method (e.g., in Table 1, DeepEDM ranks the top 1 for 36 out of 72 metrics while the next best model achieves 11) as well as the results of multiple runs (Table 5 in the Appendix). Additionally, we presented extensive ablation studies of our design and lookback length. We believe our current empirical analysis is sufficient to support the main claim of the paper. Yet we welcome specific suggestions to improve our empirical analysis.

3. Sensitivity to lookback length

Theoretically, Takens’ theorem puts constraints over the number of time-delayed steps (>2d for d-dimensional state space), yet does not speak to the number of samples needed (i.e., lookback length). Empirically, we observe that the model exhibits minimal sensitivity to the lookback length beyond a certain threshold. Once the model has sufficient historical data, additional time steps—particularly those with similar dynamics—contribute little to improving forecast performance. On the other hand, too short a lookback window can degrade performance, as there may be insufficient information to generate an accurate forecast. This observation is shown in our paper: the existence of an optimal lookback length, or "sweet spot," as illustrated in Figure 4.

4. Moving Figure 4 to the main paper

Given the additional one-page allowance in the final version, we plan to incorporate this analysis into the main paper.

5. Benefit of DeepEDM for Some Kinds of Time Series than for Others

Great question! We will include a discussion in the main paper. Our main findings from the empirical analysis are twofold. First, there is a sweet spot in the lookback length, which is dataset-dependent (discussed in our response #3). Second, DeepEDM is best suited to time series with a low to moderate number of input variates, as seen in datasets like the ETT set, Weather, ILI, and our simulated data. For datasets such as Traffic and ECL, which contain a larger number of variates, we observe a slight decline in performance. One possible reason is that DeepEDM considers a channel-wise prediction using a shared model across all channels, and thus assumes independence between input variates. With a growing number of variates, there is likely an increasing strength of correlation among these variates, which our DeepEDM can not effectively model. We plan to further explore this correlation among input variates in future work.

6. Performance Under Noise

We highlight that noisy conditions present a significant challenge for forecasting models, leading to performance degradation in strong baselines such as iTransformer, Koopa, and the vanilla Simplex. However, DeepEDM demonstrates greater robustness, exhibiting less performance degradation compared to these baselines. As discussed in Sec 5.1 and further shown in Table 6, DeepEDM achieves the best performance among strong baselines in the presence of noise. These results support our claim on noise robustness, as also noted by reviewer XFih that “[DeepEDM] remains robust even in the presence of noisy observations”.

Thank you for your thoughtful and detailed review. We now address your suggestions. Our response will be integrated into the final version.

1. Clarification on Generalization Statement

This was meant to contrast DeepEDM and EDM w. simplex: EDM trains a separate model for each time series, whereas DeepEDM learns a single shared model across all series (within same dataset). The statement is partially supported by our main results (Table 1), where models were evaluated on unseen time series. In this widely-used evaluation protocol, the training series and test ones may be extracted from the same long sequence. Due to this design, we realize that part of this statement (“generalization to new time series”) is ambiguous and will revise it in the final version.

To further clarify, we conducted an additional experiment on ETT, Exchange, and Weather datasets to evaluate DeepEDM's ability to generalize across “different” time series (within the same dataset). Specifically, for each dataset, we trained the model on a subset of sequences (seqs.) and tested it on a different subset.

• For the ETT datasets, which consist of 7 seqs., we trained on the seq. 0-2 (using only timesteps from the standard training split) and tested on seq. 4-6 (using timesteps from the standard test split). We use this 3:3 split as several baseline models are unable to handle different channel dimensions between training and testing.
• Similarly, for the Exchange dataset (8 seqs.), we trained on the first 4 sequences and tested on the latter 4.
• For the Weather dataset (21 seqs.), we trained on sequences 0-9 and tested on sequences 10-19.

We compared DeepEDM against strong baselines including Koopa, iTransformer, and PatchTST. The prediction horizon (H) varied from [48, 96, 144, 192] and lookback window was set to $2*H$. The summarized results are presented below along with the link to full results. Notably, DeepEDM ranks 1st in MAE and MSE winning in 39 out of 48 cases. The results thus provide more direct support to the statement of “a single model that generalizes to new time series."

https://anonymous.4open.science/r/icml_rebuttal-B440/generalization_exp_results.md

2. Lookback Window Reporting

Since historical data is typically accessible, any reasonable lookback window can be assumed and the lookback length is often treated as a hyperparameter [Bergmeir, 2023]. Therefore, we tuned lookback length separately for each model and forecasting horizon, and only reported best results without a detailed breakdown. This protocol was also considered in prior works [TimeMixer, PatchTST, DLinear]. The effects of lookback length in forecasting performance was indeed discussed in Appendix A.5.2 (Figure 4). If needed, we can add a detailed breakdown to the Appendix.

3. Ablation Study Reporting and Results of Linear Models

Our ablation study is designed to concisely highlight the method design choices. These detailed experiments span 9 datasets, 4 lookback settings, and 4 configurations with multiple runs. To maintain clarity and conciseness, we presented aggregated results describing key trends. Per reviewer’s request, we have included the full table below, which we will add to the Appendix.

https://anonymous.4open.science/r/icml_rebuttal-B440/ablation_full_results.md

Linear models: As prior work [DLinear, RLinear] has noted, linear models are effective for forecasting, particularly for simpler datasets. However, they struggle with more complex datasets. As shown in the above table, DeepEDM significantly outperforms the linear model on benchmarks like ILI, Weather, ECL, and Traffic by a large margin, e.g., on ECL the relative improvement in MSE from the linear model to our method is ~10%.

3. Transfer learning / domain adaptation

This is an exciting future direction! Our work focuses on single-domain forecasting but shows transfer within-domain (see Q1). Cross-domain transfer learning requires further study and we leave it as future work.