We thank the referee for the careful reading and very supportive feedback on our paper, and for raising a couple of great points which we now have addressed with further results (provided in tabular rather than graphical form, as new figures are not allowed this time).

Weaknesses

W1 (further analyses/ other RNNs): Fair question. We had initially chosen the AL-RNN as a recent SOTA model proposed specifically for DSR, which combines a couple of favorable features from a dynamical systems perspective, like direct translation into symbolic dynamics and providing topologically minimally forms [1] (which we are happy to comment on in our revision). However, we have now also tested other RNNs, like different variants of PLRNNs [2], LSTMs as suggested, vanilla RNNs, and reservoir computers which are also a popular choice in the DSR context [3]. As can be seen in Table R3 below, while AL-RNNs are on par with other PLRNN variants, other types of RNNs (vanilla, LSTM, RC) perform significantly worse, further supporting our specific choice. On the side, we discovered that we can improve DynaMix’ performance even further by including noise into the AL-RNN experts during training (see line “Expert: probabilistic ALRNN”).

More generally, we also performed a series of other ablation studies to clarify the role of different model components, also included in Table R3 below. While the STF-based training itself (see also Fig. 24 in Appx.), the attention mechanism, a sufficient number of experts ($\geq 5$), and a sufficient diversity of the training set with different chaos typologies, are all crucial components, the precise form of the context mapping (whether CNN or linear layer) and the exact expert model within the class of PLRNNs has less of an influence on the overall architecture’s success.

Table R3: Ablation experiments & different expert models: Mean across 5 different training runs of the median performance across the 54 DS test sets.

W2 (delay embedding): Some form of embedding is indeed necessary to achieve strong DSR results (cf. sect. 3.3), but this does not necessarily need to be a delay embedding, but could also be a positional encoding (eq. 6) as similarly employed in Transformer-based models like the different versions of Chronos. Indeed, for all empirical examples in Fig. 8 we had used positional encoding for DynaMix, i.e. DynaMix outperformed the other FMs in terms of DSR (and often forecasting) even though all models received the same types of inputs (scalar time series and positional embedding). However, we have also tested TTM with delay embedding now (the only one of the FMs which actually allows for this type of multivariate embedding), hardly improving the results as provided below.

Table R6: Reconstruction results for TTM with delay embedding

W3 (non-stationary TS): Thank you. Non-stationary time series are indeed still an open issue, with which other models struggle as well. As a proof of concept, however, we have now designed a pre-processing pipeline, which (similar as in FEDformers [4]) first estimates trend components from the context signal and adds them back on for inference. We have tested this idea on the non-stationary Air Passengers TS benchmark (cf. [5]), and compare performance of DynaMix with this enhancement to the best competitor (Chronos-t5-base; see Fig. 4c & Table 1) in Table R7 below. While we are not allowed to provide figures in the rebuttal which would illustrate the approach and results more clearly, we will include these in our revision. This is just a proof-of-concept at this stage, but it illustrates the approach in principle works and could be extended and potentially optimized.

Table R7: Forecasting results for non-stationary Air Passenger TS benchmark

References

[1] Brenner, M., et al. (2024). Almost-linear RNNs yield highly interpretable symbolic codes in dynamical systems reconstruction. Advances in Neural Information Processing Systems, 37, 36829–36868.

[2] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR.

[3] Patel, D., & Ott, E. (2023). Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(2).

[4] Zhou et al. (2022) FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. Proceedings of the 39th International Conference on Machine Learning, PMLR 162:27268-27286

[5] Ansari, A. F., et al. (2024). Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815.

We thank the referee for their thorough assessment of our work, and the very supportive and encouraging feedback!

Weaknesses

W1 (ablations): Thank you for bringing this up, we have now performed a series of ablation experiments, summarized in Table R3 below. While the STF-based training itself (see also Fig. 24 in Appx.), the attention mechanism, a sufficient number of experts ($\geq 5$), and a sufficient diversity of the training set with different chaos typologies, are all crucial components, the precise form of the context mapping (whether CNN or linear layer) and the exact expert model within the class of PLRNNs has less of an influence on the overall architecture’s success. Using as experts LSTMs, vanilla RNNs or reservoir computers (RCs) instead of any of the PLRNN-variants, however, led to decay in performance. On the side, we discovered that we can improve DynaMix’ performance even further by including noise into the AL-RNN experts during training (see line “Expert: probabilistic ALRNN”).

Table R3: Ablation experiments & different expert models: Mean across 5 different training runs of the median performance across the 54 DS test sets.

W2 (comparison to non-FMs): Very good point. To put our model’s zero-shot performance into perspective, we have now trained various DSR models (AL-RNN trained by STF [1], reservoir computers for DSR [2], and neural ODEs for DSR [3,4]) on the same data used in the paper, with results shown below in Table R2. As evident from this table, DynaMix’ zero-shot generalization performance is completely on-par with that of SOTA DSR models specifically trained on these data, even without any fine-tuning or retraining (performance across the 54 DS test sets was statistically largely indistinguishable according to t-tests, while performance on the empirical TS was on average even slightly better).

Table R2: Performance for custom trained DSR (ALRNN, RC, Neural ODE) models across all 54 test set DS (as in Fig.4; median+/-MAD) and for the empirical time series (as in Fig. 8). Best in bold.

Minor

M1) Agreed, this sect. is rather short, and we will move more details from Appx. A.2 back to the main text in the revision.

M2) Yes, fair point. One solution here is to only show a subset of the empirical data in the main text (moving the others to the Appx.), and using the space instead to show predictions from each foundation model in separate panels (or, alternatively, to focus only on the comparison between DynaMix and its strongest competitor in each panel).

Questions

Q1) Yes, this is possible. First, please note that the empirical examples (incl. human EEG or the air pressure data) are inherently likely of much higher dimensionality, yet DynaMix is still able to reconstruct and forecast these systems well. Second, we have now included another, 6d DSR benchmark (Lorenz-96), see results in Table R5 below.

Table R5: Reconstruction & forecast results on 6d Lorenz-96 across all foundation models

Q2) As it currently stands, when observations arrive at non-equal temporal intervals, this would need proper preprocessing (like interpolation) for all the models reported in the paper. So this is currently a limitation that we will add to our Limitations sect..

References

[1] Brenner, M., et al. (2024). Almost-linear RNNs yield highly interpretable symbolic codes in dynamical systems reconstruction. Advances in Neural Information Processing Systems, 37, 36829–36868.

[2] Patel, D., & Ott, E. (2023). Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(2).

[3] Chen, R. T., et al. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31.

[4] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR.

We thank the referee for taking the time to thoroughly scrutinize our work, and for pointing out various omissions in terms of model evaluation, which we have now all addressed and provided in the form of result tables below (as figures are not allowed for the rebuttal this time).

Weaknesses

W1 (further comparisons): Good point. Our comparison group here were other foundation models (to which we now have further added TimesFM [5]: Dstsp: 12.50 ± 0.93, DH: 0.36 ± 0.11, MASE: 1.11 ± 0.63), as ‘standard’ DSR models cannot perform zero-shot reconstruction and forecasting but need training on at least segments of the new data. However, we fully agree that such a comparison would provide practical context, and have now compared DynaMix’ zero-shot performance with that of three SOTA DSR models explicitly trained on the data, namely the AL-RNN trained with STF as in [1], reservoir computers (RC) for DSR [2], and neural ODEs (NODE) for DSR as in [3,4]. As shown in Table R2 below, the zero-shot generalization performance of DynaMix in terms of both long-term statistics and forecasting is completely within the ballpark of custom-trained SOTA DSR models, even without any fine-tuning or retraining (statistically largely indistinguishable across the 54 DS test set according to t-tests, and on average even slightly better on the empirical TS).

Table R2: Performance for custom trained DSR models (ALRNN, RC, Neural ODE) across all 54 test set DS (as in Fig.4; median+/-MAD) and for the empirical time series (as in Fig. 8). Best in bold.

W2 (ablation studies): Yes, the MoE concept itself is not novel of course, but what is very novel in our minds is the whole model design and training for DSR, making it to our knowledge the first foundation model with zero-shot DSR capabilities, incl. long-term properties. We, however, fully agree with the referee that some more insight into which model components are crucial for its performance is useful, and have now performed a series of ablation experiments as summarized in Table R3 below. While the STF-based training itself (see also Fig. 24 in Appx.), the attention mechanism, a sufficient number of experts, and a sufficiently diverse training set with different chaos typologies, are all crucial components, the precise form of the context mapping (whether CNN or linear layer) and the exact expert model within the class of PLRNNs [4] has less of an influence on the overall architecture’s success (using as experts LSTMs, vanilla RNNs or reservoir computers instead of any of the PLRNN-variants, however, led to decay in performance). Hence, it is really the overall design and training procedure which make the framework so strong. On the side, we discovered that we can improve DynaMix’ performance even further by including noise into the ALRNN experts during training (see line “Expert: probabilistic ALRNN”).

Table R3: Ablation experiments & different expert models: Mean across 5 different training runs of the median performance across the 54 test sets.

W3 (scalability): We now explicitly tested model scaling with the number of AL-RNN experts. As can be appreciated from Table R4 below, a minimum number of experts ($\geq5$) is necessary to achieve optimal performance, while after that performance rather plateaus. At the same time, the computational load (training times) scales only linearly with the number of experts $J$ (tested for $J=10:10:80, R^2=0.97$; note we are not allowed to provide figures this time, from which this may have been clearer; regarding inference times, these are all in the split-second range anyway).

Regarding diversity, we would like to point out that in sect. 4.4 we did apply DynaMix to a very diverse range of empirical settings with dynamics fundamentally different from the training set. This, in our minds, demonstrates its generalization capabilities with regards to fundamentally different dynamical systems.

Table R4: Dependence of model performance on # of experts ($J$): Mean across 5 different training runs of the median performance across the 54 DS test sets.

Questions

Q1) We have now done so, see response to W1 and Table R2 above.

Q2) Yes, now done, please see our response to W2 and Table R3 above.

Q3) We think that the applications of our model to the diverse set of empirical data in Fig. 8 and Table 1 of the paper already constitute test cases with highly novel and diverse dynamics, since no such data were part of the training set! We now further included a high-d hyper-chaotic example, see response to Q5 below. Finally, as our response to W3 and Table R4 above indicates, our system is very well behaved and highly robust with regards to scaling up model size.

Q4) Yes, fair question. The original motivation was that the AL-RNN is a recent SOTA model proposed specifically for DSR, which combines a couple of favorable features from a dynamical systems perspective, like direct translation into symbolic dynamics and providing topologically minimally forms [1] (on which we will comment in our revision). As furthermore shown in Table R3 (see reply to W2 above), however, AL-RNNs also perform substantially better in this context than other typical RNN choices like LSTMs or reservoir computers often used for DSR.

Q5) In our minds the empirical test case examples (sect. 4.4) already cover systems with very complicated and much higher dimensional underlying dynamics, human EEG for instance, or the air pressure and cloud request data. We now have further included a 6d Lorenz-96 system in its hyper-chaotic regime and compared the DSR and forecasting performance of all models in Table R5 below, verifying that the strong performance of DynaMix also holds for complex hyper-chaotic systems.

Table R5: Reconstruction & forecast results on 6d Lorenz-96 across all foundation models

References

[1] Brenner, M., et al. (2024). Almost-linear RNNs yield highly interpretable symbolic codes in dynamical systems reconstruction. Advances in Neural Information Processing Systems, 37, 36829–36868.

[2] Patel, D., & Ott, E. (2023). Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(2).

[3] Chen, R. T., et al. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31.

[4] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR.

[5] Das, A., et al. (2024, July). A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning.

Thank you so much for further clarification and experiments! Based on your responses, I have updated my recommendation to accept.

We thank the referee for the thorough reading, and the strong and very encouraging support of our work!

Weaknesses

W1 (MASE instead of MAE): Thanks for raising this point. We had standardized the time series for these comparisons, but in any case also computed the MASE now as suggested, with results reported in Table R1 below. As can be seen, using MASE, DynaMix even comes out top of the list, surpassing the best Chronos model even w.r.t. forecasting.

Table R1: Forecast performance, same as Fig. 4C in paper, but for MASE (sorted from best to worst)

W2 (complexity of training): Yes the training is non-standard, but on the one hand, training techniques like STF are indeed important for achieving good dynamical systems reconstruction results, see [1,2] and Fig. 24. And on the other, our overall architecture is instead somewhat simpler we would argue than typical time series FMs.

Questions

Q1) Yes, all time series were standardized dimension-wise, see sect. 3.2 and A.2, for all models. We will make this more explicit in the revision!

Q2) We are not 100% sure we understood the question correctly, but please note that Fig. 3b already demonstrates the capability of the model to forecast into regions of state space not even covered by the context information (in this sense non-local). We confirmed this now with two further examples (Van-der-Pol oscillator and Rayleigh oscillator) which we will include in the revision. We also confirmed on one example (cyclic Rössler DS) that DynaMix can, for instance, correctly forecast a limit cycle with half or twice the period of that experienced in training (Dstsp={0.48, 1.72}, DH={0.008, 0.026}, MASE={0.15, 0.36}).

[1] Hess, F., et al. (2023). Generalized Teacher Forcing for Learning Chaotic Dynamics. Proceedings of the 40th International Conference on Machine Learning, 13017–13049. PMLR.

[2] Mikhaeil, J., et al. (2022). On the difficulty of learning chaotic dynamics with RNNs. Advances in Neural Information Processing Systems, 35, 11297–11312.