Forecasting Needles in a Time Series Haystack

Though the benchmark is only designed for time series forecasting, and the model performance is only evaluated in terms of the zero-shot forecasting capabilities, evaluation of other tasks might be required, e.g., classification.

Also, it would be interesting to discuss other stochastic processes for the time series generation and include an additional experiment for the ablation study.

It is hard to read in some figures in the paper, e.g., Figure 4, Figure 6-8 and Figure 14-35; I believe the fontsize should be increased.

W1. Very strong constraints are applied to the time series setting (>8 spikes, >512 segments, etc) in order to establish the synthetic spike benchmarks. It is not clear from this work how helpful these constraints are when applying the benchmark to other real-world settings and under what settings they could be relaxed. Better establishing the importance of the ‘spike’ setting could improve this weakness.

W2. The NITH-Real dataset seems likely to have some overlap with the foundation model training sets, particularly through its use of the UCR Anomaly and NAB benchmarks. Separating out the results by benchmark and analyzing the overlap where possible could help reduce this risk.

W3. The continuous pre-training approach can lead to loss of generalization. Could the authors evaluate how the models trained on the NITH-Synth benchmark fare on other standard forecasting benchmarks? There are many existing methods for improving continual pre-training which can be applied in this setting as well to mitigate this performance difference if it exists.

1. Clarity: The paper introduces a lot of "novel" things, and mathematical instruments to model and evaluate time series models on spiky data. There's a lot of notation which is hard to follow ($\alpha, \beta, \theta, \omega$, to name a few) as sometimes symbols apply prior to their usage. For example, it is particularly hard to follow 219 -- 225, where it's unclear what $\alpha$ and $\beta$ are until it is introduced in Algorithm 1.
2. Too many "novel" things, but their : The authors propose too many new things, that it is hard to keep track. Moreover, it seems that the authors miss a lot of basic things and instead over-complicate their proposed solutions. For example, [1] proposes a simple way to inject spikes in time series which was not considered. For the "haystack", the authors use Gaussian Processes, when simpler time series can be generated using polynomial or linear trends, periodicity (sine waves), and some noise (Gaussian or Random walks). Similarly, there has been a lot of work in time series anomaly detection on better metrics to evaluate time series of similar natures, for example [2]. Moreover, simpler metrics such as $\alpha^{T}(Y - \hat{Y})$, where $\alpha$ is a vector which gives more weight to needles rather than the haystack.
3. Baselines and related work: The authors only consider time series foundation models. They do not consider any statistical models such as ARIMA, or models which are designed to predict spiky time series. Moreover, the authors do not consider any prior work on modeling or evaluating spiky time series, e.g. [3, 4, 5, 6]. Also, the authors continually pre-train pretty large models on NITH, perhaps smaller models can be fine-tuned more effectively on spiky time series.
4. Normalization: All time series models including foundation models use some form of normalization (e.g. Chronos uses mean scaling while TimesFM uses RevIN), and I believe that this influences the forecasting performance. This is perhaps the most component of the design space of these models.
5. Lags and lack of connection to a real-world application: The authors mention (random) lag multiple times in the paper, but it is unclear to me why is this an important consideration for spiky time series? I feel this is also where connection to a real-world problem (e.g. energy load prediction) would be helpful. For example, in these cases it may be more important for to predict whether there will be spikes, or the number of spikes and the amplitude of the spike, each of which is a different problem (classification or regression), and can be evaluated with different metrics.
6. Loss function: Most TSFMs are trained to predict some notion of conditional mean (MSE) or median (MAE) of a time series. These loss functions are not appropriate for training models that can be expected to do well on spiky data, thus the fact that TSFMs don't perform too well is not surprising. On the other hand, I feel that the authors could have spent more time on this important aspect.
7. Memorization: The authors claim that the benchmark tests a model's ability to memorize spikes. Could the authors provide more information and evidence on why this is the case?

Minor

1. In line 380, I am not sure what "struffle" means
2. I think the results in Figure 7 may not be comparable because the encoder-only, decoder-only and encoder-decoders models have different number of parameters, so it is unclear if the differences are due to the difference in # of parameters or due to the inductive biases of these models.
3. It is not immediately clear to me what is a good value of the scaled DTW.
4. In line 316, I suspect that the authors have mistakenly annotated the wrong predicted time series (they have interchanged Pred 2 and 3).

References

1. Goswami, M., Challu, C. I., Callot, L., Minorics, L., & Kan, A. Unsupervised Model Selection for Time Series Anomaly Detection. In The Eleventh International Conference on Learning Representations.
2. Paparrizos, J., Boniol, P., Palpanas, T., Tsay, R. S., Elmore, A., & Franklin, M. J. (2022). Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. Proceedings of the VLDB Endowment, 15(11), 2774-2787.
3. Fadlallah, B., Chen, B., Keil, A., & Príncipe, J. (2013). Weighted-permutation entropy: A complexity measure for time series incorporating amplitude information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 87(2), 022911.
4. López, Manuel Zamudio, Hamidreza Zareipour, and Mike Quashie. "Forecasting the Occurrence of Electricity Price Spikes: A Statistical-Economic Investigation Study." Forecasting 6.1 (2024): 115.
5. Fatone, L., Mariani, F., Recchioni, M. C., & Zirilli, F. (2012). The analysis of real data using a stochastic dynamical system able to model spiky prices.
6. Zhang, X., Jin, X., Gopalswamy, K., Gupta, G., Park, Y., Shi, X., ... & Wang, Y. (2022). First de-trend then attend: Rethinking attention for time-series forecasting. arXiv preprint arXiv:2212.08151.

The major weakness, in my opinion, is the difficulty in clearly understanding the manuscript's goals by reading the title and abstract. When the authors say that shocks and sudden spikes are common characteristics in real-world time series, I suppose there is substantial data available to address this problem. So, from the real-world perspective, is the authors' main contribution just the compilation of those data? On the other hand, the authors also proposed a function to create "spiky" time series. It is an interesting proposal, but their focus is essentially on modifying the stochastic component. It is not clear whether it is also possible to modify the deterministic part of the time series, thereby varying the general behavior of the time series. Moreover, I understand that foundational time series models are currently prominent, making it natural to use them to attract readers' attention. However, it is unclear why classical approaches were not considered in their investigation. Another important point is the authors' choice to select MAE, MSE, and DTW as evaluation "metrics." Why were these selected? I understand they are important when comparing predicted observations, but the application presented by the authors involves not only predicting observations, but also predicting spikes. There are more appropriate "metrics" for this task that are not influenced by non-spike predictions.