Large Pre-trained time series models for cross-domain Time series analysis tasks

We thank the reviewer for their comments. We will address them as follows:

Domain-generalization settings

We wish to emphasize that we generalize to a wide range of domains but we require to learn segmentation module for each of the domains. Generalizing to unseen domains is an important research question to tackle for general pre-trained model

Title

We chose LPTM as the title since our innovation using adaptive segmentation for cross-domain pre-training enables state-of-art performance across multiple datasets with heterogenous patterns. We are happy to add to adaptive segmentation to the title.

Moreover, other pre-trained baselines also have generic names such as TimesFM (Time-series foundational model).

Eq 2

The intuition for Eq 2 is that since the GRU captures the information at timesteps i and j in context of entire time-series, the segment score function uses this information encoded in z^(i) and z^(j) to derive the segment score. The exact form of the equation was chosen empirically and is similar in form to Bahdanau attention function. All the parameters (W_1, W_2 and b) are learnable.

Scaling and number of tokens

Actually, we found that adaptive segmentation doesn’t provide any significant contribution to overall overhead of the model since the transformer backbone consumes the bulk of the compute time. While the entire adaptive segmentation module’s runtime is quadratic in length of the time-series, Empirically we found that the average size of the segments across datasets is around 5.2 with some domains such as ETT having larger average lengths (13.4) and some domains such as behavioral datasets with lower average lengths (3.1).

Regarding the heuristic, solving for the optimal set of segments that provide the maximum score as well as cover the entire length of the input is a non-trivial optimization problem whose complexity could be upto exponential in number of time-steps. Therefore, we use a simple heuristic of eliminating the lowest valued segments till we cannot cover the entire time-series. (l 12-14).

Choice of SSL tasks

Randon masking and lask token predictions are popular SSL pre-training methods that are widely adopted in LLMs and some time-series methods like PatchTST. We also point out that using these simple tasks already provides state-of-art results. While we can use more sophisticated methods, the focus of our work is on using adaptive segmentation to enable multi-domain pre-trained time-series model. Indeed LPTM is very flexible to adapt other SSL loss functions to improve performance could be a interesting extension of our work.

Stability of training and mode collapse

We did not observe instability or mode collapse during pre-training. We found that the training of the model was stable across different domains and datasets. This was observed across multiple runs of the model as well. The frequency of updating the segmentation loss relative to SSL loss is key in ensuring the stability of training.

The overall objective of self-supervised pre-training involves a discrete objective of choosing an optimal segmentation strategy along with the prediction of masked segments. Similar to other works in graph learning[1] and reinforcement learning on discrete action space [2] which involve learning over discrete structures that cannot be trivially integrated we use a bilevel optimization approach, and periodically optimize the segmentation by learning on SSL tasks.

Improving the efficiency and stability of this approach and more tightly coupling the two objectives could be an important research direction to improve model performance.

Convergence of scorer and encoder

It is not straightforward to compare the SSL loss with the loss for score function since they indicate different objectives. We found that the SSL loss may slightly perturb when there is an update to score function loss but generally follows a relatively smooth downward trend to convergence. The scoring function loss, while trending lower over time, doesn't converge to zero and hence is not a predictor of SSL loss, Eq 5 only forces the scoring function to be trained to update in the direction of decreasing SSL loss.

Model Sizes

LPTM model size is around 100M which is 2x smaller than than TimesFM, the smallest of the pre-trained baselines and is around 10x smaller than Chronos which has around 1B parameters.

Classification tasks

While we focused on forecasting for most of the experiments similar to other general pre-trained time-series models, we also showcased the flexibility of LPTM to perform classification due to our architectural choice. Table 4 contains only 32 datasets and this is a typo. We added the other 3 datasets in the additional pdf.

Mean rank of Models

We post the mean ranks in the additional PDF. LPTM has the best average rank of 2.5 and is significantly better than other models.

Significance test

We used the standard t-test to determine statistical significance. We have added the standard deviations across 10 runs in the additional pdf.

Regarding Limitations

We thank the reviewer for the suggestions. We will add the discussion on multivariate datasets and issues regarding data privacy to discussions.

Code link

Sorry. We didn’t realize the link was expired during the review process. The code can be found in the LPTM.zip file at https://osf.io/vy3sc/?view_only=c10508d340ba468287c984496cf57be1

We thank the reviewer for comments and suggestions. We address them as follows:

Aggregate rank metric

We thank the reviewer for the suggestion. We will add the average rank of each model as:

LPTM has the best average rank of 2.5 with significant lead over second best model.

Code link

Sorry. We didn’t realize the link was expired during the review process. The code can be found in the LPTM.zip file at https://osf.io/vy3sc/?view_only=c10508d340ba468287c984496cf57be1

Long-term forecasting benchmarks

We have benchmarked on ETT and M4, two popular long-term forecasting benchmarks in the paper. We have added additional benchmarks on ILI, Electricity and Exchange datasets (similar setup to [1]):

Error bars

We have added the std. Dev across 10 runs in the additional PDF.

References

[1] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." arXiv preprint arXiv:2211.14730 (2022).

We thank the reviewer for their comments and we address them as below:

Intuition on segmentation

We note that even for vision models such as ConvNets and ViT, the images are ingested as fixed sized patches (such as 16 x 16 patches) which have some similarity to segments in time-series. Moreover, we argue that there is much more diverse heterogeneity in time-series compared to images since even images of different domains have important properties such as edges, shapes that are useful to capture by the model.

However, for time-series, each domain can have a wide range of patterns specific to the domain along with cross-domain information. Further, these patterns could differ across time. Therefore, segmentation that adapts to these patterns across time can better capture these semantic structures and handle cross-domain downstream tasks.

Stability of adaptive segmentation

First, we did not observe instability in training. We evaluated our model over multiple runs with different random initializations and we didn't experience large variation in stability or performance

Since the overall objective of finding optimal segments as well as pre-train weights of the backbone model on SSL tasks has discrete components we use a bi-level optimization to simultaneously learn both objectives. We update the loss for segmentation every 10 epochs to make the training stable for SSL. This kind of training strategy is used in other bi-level optimizations encountered in discrete tasks such as graph structure learning and reinforcement learning on discrete action space[1,2].

Fine-tuning adaptive segmentation

We fine-tuned adaptive segmentation to provide optimal segmentation for each domain dataset since the patterns observed across different domains is very heterogenous. For example, while we need to capture monthly or yearly patterns for demand datasets, behavioral datasets need to encode different kinds of patterns occurring in order of seconds or minutes.

Training a single segmentation strategy across all domains therefore would be highly suboptimal. In fact, empirically, we observed that training a single segmentation strategy across all domains lead to unstable training due to which we didn’t explicitly include it as a baseline.

Interpretability of segmentation

We also show some visualizations of the segmentations for different domains in Appendix Section D. We can get some intuitive understanding from these visualizations such as LPTM capturing smaller segments near epidemic peak to encode fine-grained variations as well as segment lengths correlating with variance of time-series in general.

References

[1] Hu, Minyang, et al. "Learning Continuous Graph Structure with Bilevel Programming for Graph Neural Networks." IJCAI. 2022.

[2] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

We thank the reviewer for the comments. We address them as follows:

The proposed framework is not differentiable..

As stated in the paper we did not observe any instability in the training. Yes, the segmentation of time-series is a discrete operation. LPTM tackles this challenge via a novel differentiable scoring module which learns the best segmentation strategy based on pre-trained dataset.

We note that the frequency of training every 10 batches is key to providing stable training since changing the segmentation module every iteration will change the SSL objective function too frequently leading to instability. Similar strategies are used in many papers that encounter bilevel optimization such as for discrete operations such as graph learning [1] and reinforcement learning over discrete action spaces [2].

Handling missing values

Handling missing values is not in the scope of our current work. However, LPTM could be extended to handle missing values using a similar method to random masking task by masking the segments associated with missing values. One issue to tackle would be to selectively mask part of the segment.

Regarding pre-trained data

For pre-trainied models such as Lag-LLAMA and TimesFM, we start with the weights of the models publicly available and pre-train on the same datasets we use to pre-train LPTM. Therefore, all baselines are pre-trained with at least as much data as LPTM is to provide a fair comparison. In fact, the other pre-trained models such as TimesFM and Chronos have already been pre-trained on much larger pre-trained datasets. In spite of this, LPTM achieves SOTA performance. For fine-tuning, all models are fine-tined on the same train-test splits.

Further, we use all the datasets described in lines 222-244 for pre-training. Only the unseen test split was used for evaluation on Tables 1 and 2.

is the GRU applied entry-wise to the time series?

Yes, for each $y^{i}$ we get an embedding $z^{(i)}$ from the GRU.

what is z_i?

This is a typo, we meant $z^{(i)}$.

Value of s(i,j)

Yes, larger is the value of $s(i,j)$ more important the segmentation module believes the segment between i and j to be. The intuition for Eq 2 is that since the GRU captures the information at timesteps i and j in context of entire time-series, the segment score function uses this information to derive the segment score.

Index k in Eq 3

k is just the indexing variable to aggregate the output embeddings of the self-attention for time-stamps from i to j. We will clarify this.

Regarding Eq 5

Due to the discrete problem of finding the optimal set of segments, the actual objective function during SSL is not continuous. However, we overcome this by using a bi-level optimization approach and periodically optimize the segmentation by learning on SSL tasks. This provides a more stable approach to training when we encounter discrete objectives similar to such strategies used in many papers that encounter bilevel optimization such as for graph learning [1] and reinforcement learning [2].

Missing values removed (line 225)

Many time-series in Project Tycho are very sparse with lots of missing values. We remove these time-series since LPTM and other baselines cannot handle them. However, as noted earlier, extending LPTM to time-series with missing values is an interesting and important challenge for future research.

References

[1] Hu, Minyang, et al. "Learning Continuous Graph Structure with Bilevel Programming for Graph Neural Networks." IJCAI. 2022.

[2] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.