Large Pre-trained time series models for cross-domain Time series analysis tasks

We thank the reviewer for their comments. We address their questions as follows:

In natural language processing, doesn't a single token also sometimes fail to convey the full semantic information of a sentence?

We specifically refer to semantic information of each individual token. While single tokenized word/subword does not impart full semantic meaning, they have inherent semantic meaning of varying importance. In contrast, for time-series, each individual time-step is a real number that doesn’t provide good semantic interpretation of local temporal dynamics in isolation in most cases.

** how is the optimal combination of data samples from different domains determined?**

Our main aim is to show that LPTM can adapt to pre-train and learn from wide range of domains. Therefore, we do not optimize on the right combination of datasets to optimize downstream performance. We chose widely used datasets from time-series literature that covers diverse range of applications.

However, finding or optimizing for the right combination of datasets is an interesting research direction which actively explored even in LLM pre-training and can be explored for time-series as well using LPTMs.

How can Eq.2 effectively handle "out-of-distribution" samples from domains that were not encountered during pre-training?

The segmentation module identifies good segmentation strategies for each of the domains encountered in the pre-training. Therefore, they cannot be directly used on downstream tasks of new domains. However, they can handle unseen datasets and tasks of domains trained pre-training.

I'm not sure about what the authors intend by "how good" in the paragraph following Eq.2.

The scoring function is trained to predict the “goodness” of segments by helping find the subset of segments that decreases the SSL losses during pre-training. This is later explained in Section 3.4 (Eq. 4) where scoring function s(i,j) is trained to be similar to negative log of SSL losses which determined how good the chosen set of segments are in performing the SSL tasks.

In section 3.3, what are the fundamental differences between random masking and last token masking in time series pre-training? Is there any deeper analysis or extended discussion available?

Random masking masks a small fraction of tokens chosen at random for each batch. Last masking masks the last few tokens only and aims to predict it. Random masking helps the model to learn to generate missing parts of the time-series at any time-period and helps learn contextual temporal dynamics is relation to observed time-series. Last masking is more similar to forecast that determines on the parts of the time-series in the future. Both the losses help model understand and generate time-series from different perspectives and help improve efficacy of LPTM for most benchmarks.

Regarding Fig 1

To model the score of segments we need to capture the temporal dynamics of the segment which is done by GRU. This is then also used to embed the segments for the same reason. Moreover, a linear projection cannot captured segments of varying length and their dynamics. The patch scoring is done using the scoring function (Eq 2) and the high scoring subset of segments is chosen and pruned by heuristic mentioned in Sec 3.2 after Eq. 2. We also define h(i) in the paragraph as the the maximum scoring segment that start from time-step i. The full model in Figure 1 is pre-trained on the SSL tasks (Sec 3.3) to learn the parameters.

Regarding Writing

We thank the reviewer for their comments. We would be grateful if the reviewer can point to specific instances where the writing is unclear.

Out abstract has multiple paragraphs to first introduce the novel challenges of pre-trained time-series models that are not yet well explored and then introduce the technical novelties and advantages of LPTM in subsequent paragraphs. The abstract is less than 200 words and clearly introduces the scope, importance and novelty of our work.

To the best of our knowledge we have clearly defined all the mathematical notation in the paper. The only source of notational confusion, as other reviewers pointed out is that we have one type in Eq 4 where $g(i,j)$ should be $s(i,j)$.

Regarding zero-shot forecasting

Pre-trained LPTM models are observed to perform similarly or better than SOTA baselines using significantly smaller training data and training time. However, we need some amount of fine-tuning to specific task since each task is different to pre-training SSL tasks and require the model to adapt to it.

However, designing zero-shot learning on LPTM pre-trained models is indeed a very interesting future research problem with non-trivial challenges we do not tackle in our current work.

Codebase

We thank the reviewer for the comments. We will improve the documentation of the code when we release it publicly.

Rationale behind Equation 4

Empirically we found that adding the log to SSL losses provided for much stabler training, without which pre-training was not possible. While ideally, the SSL losses converge towards zero, in practice we found that they were generally much larger throughout the pre-training process. Using adaptive loss functions across different ranges of SSL loss values could further lead to improved stability.

Yes, Equation 4 does indeed have a type where $g(i,j)$ should be replaced by $s(i,j)$. We thank the reviewer for pointing this out and hope this should clarify the confusion.

Why does the score rely on z_i and z_j but not the full sequence of between indices and ...

Yes, GRU may fail to capture important long-term dependencies. However, the alternative of combining all the indices in the sequences may be challenging since we need to still present the ordering of the sequence of hidden representations to make sure the model can view the time-steps i to j in order. Further addition of multiple segments may make the segmentation module inefficient in finding scores for all possible segments. However, improving on this drawback of capturing all segments' patterns effectively and efficiently is an interesting problem for future work.

Implication of score function

The score function $s(i,j)$ is trained to predict the logarithm of SSL losses (Eq 4) which enables the model to in turn use it to find better segmentation strategies. This is because a higher $s(i,j)$ using the trained segmentation module implies the segment is useful for the model to undestand important patterns and properties of the time-series (via the SSL tasks).

h(i) in the Fig 1

The scoring function first determines the indices $h(i)$ for each time-step $i$ and we use it to prune the set of segments to input to the transformer model. We agree that that not all the segments caan be chosen and some will be pruned. We will update the figure to reflect this.

which aggregation function is used, why?

The aggregation of the time-step representations $z_i, \dots, z_j$ refers to simple summation after passing through a self-attention layer. We will clarify that. We used summation since it is a standard aggregation function used to aggregate along the temporal dimension to represent a time-series, or a segment (in our case) using a fixed length vector embedding.

Comparison with ROCKET and other baselines

The TARNet baseline we compared against provided significantly better performance that these methods as shown their paper. Therefore, we only compred against TARNet which is the state-of-art classification model.

About the combinatorial problem

The combinatorial problem referred to is to find the optimal set of segments that cover the entire time-series that have the maximum sum of scores. The set of satifiable soultions is exponential in the number of time-steps. We used a simpler heuristic that guarantees the coverage of the segments but doesn't guarantee optimality. A more thorough theoretical analysis of the problem would be an interesting problem to improve the performance of LPTM.

We thank the reviewer for the comments. We would address some of them as follows:

The model architecture used is quite straightforward - just a transformer encoder. Exploring more sophisticated temporal modeling architectures could be beneficial.

Our important contribution is the importance of learning good segmentation strategies for enabling effective representation for multi-domain time-series pre-training. The main challenge for performing pre-training on heterogeneous datasets is not the model architecture but the segmentation module. We show that by using the vanilla transformer architecture we generally outperform state-of-art models designed for specific downstream tasks. Our segmentation module can be applied to any transformer-based architecture to improve and enable multi-domain pre-training

Ablations only evaluate the removal of components, could also analyze additions like other SSL tasks.

Many SSL tasks are usually designed for specific applications or for pre-training on a single data source. Moreover, they may not automatically be retrofitted with our framework of using segments of time series. Moreover, we showed significant performance improvement using simple SSL tasks of random masking and last token prediction. Identifying better SSL tasks that can be performed on segment tokens is an interesting future research direction.

Though diverse, the pre-training datasets are still limited to a few domains. Expanding the data diversity could help.

We leverage datasets from seven very diverse domains ranging from traffic and finance to epidemiology and behavior tracking. Moreover, these datasets are popular benchmarks widely used in literature. Therefore, we showed the efficacy of LPTM in pre-training on diverse datasets. If the reviewer can suggest any other specific domains we would gladly consider adding them as well.

Theoretical analysis on how pre-training and adaptive segmentation provide benefits is lacking

While we empirically show the efficacy of the segmentation module in handling diverse datasets, generating a convincing theoretical justification is non-trivial. This is due to the complexity of the model and dataset distributions from different domains. Our main motivation is based on various patterns of smoothness and local dynamics of the time series that cannot be captured by tokenizing single time stamps or uniform length segments. This is also analogous to various tokenization strategies used in large language models where the strategies are not theoretically justified but are motivated by empirical and linguistic arguments.

Comparisons to more sophisticated domain-specific models like those using additional covariates would be informative.

This is an interesting question. LPTM architecture is designed to be a general time-series forecasting model without additional sources of features similar to other top-performing state-of-art baselines. Adapting our methods to multiple covariates is an important challenge for future work.

Analysis of computational requirements for pre-training is needed, especially regarding scaling up.

We provided the computing environment and time for pre-training and training in Appendix Section B. We also measure the compute time required for training in Table 3 and show that LPTM requires a significantly smaller time to reach the performance of best baselines.

Lack of analysis of any negative societal impacts or limitations of the approach.

We comment on the potential negative societal impacts as discussed on page 9 Section 6 such as the requirement of testing for fairness of predictions, data quality issues, and ethical impacts of the decisions taken by the forecasts. While a detailed analysis of ethical impacts is not the scope of our work, we agree that understanding such impacts is an important problem.

The word of "multi domain" is overused: the reviewer don't see the specific module for multi domain setting. However, the "a separate segmentation module for each dataset domains to capture varied sizes of segments that differ across datasets" in page 3 Section 3.1 limits the ability of generalization on this model.

The segmentation module's main goal and main contribution of our work is to identify segmentation strategies for datasets from each of the different domains. This allows LPTM to effectively pre-train and deploy on datasets of these domains. Therefore, we aptly emphasize the ability of LPTM to handle multi-domain data sets and tasks.

We thank the reviewer for their comments.

Reproducibility

We provide additional details on important hyperparameters in Appendix Section B.

For RANDMASK, we found the optimal gamma = 0.4, and for LASTMASK gamma = 0.2 was optimal

For Flu-US we pre-trained and trained on the dataset from 2004-2012 and evaluated for the 2013-2020 seasons. We used the standard recommended train-val-test split for ETT2, PEM-Bays, NY-Bike, Nasdaq, and M3 as done in other works. The ratio is usually around 70-20-10. The test and training data splits are also provided for classification datasets from the source UCI repository.

We also mention the computing environment in Appendix Section B as using Nvidia Tesla v100 GPU with 32 GB VRAM. The CPU used is an Intel Xeon with 64 cores and 128 GB of memory. The code running time is benchmarked using wall time or execution of the entire pipeline of training.

Clarity

The score function $s(i,j)$ as defined in Equation 2 provides a score for each of the time-series segments that is used to pick the appropriate segments of the model. This is optimized in Equation 4 by making the score function reflect the negative of SSL training loss. Therefore, set of segments with higher scores will provide lower SSL loss.

Equation 4 does indeed have a type where $g(i,j)$ should be replaced by $s(i,j)$. We thank the reviewer for pointing this out and hope this should clarify the confusion.

On requirement of fine-tuning

Even pre-trained language models or vision models required some fine-tuning or prompting to adapt to a downstream task. Therefore, it is not clear why a pre-trained model does not need fine-tuning. Indeed, the SSL tasks differ significantly from the downstream tasks such as classification and forecasting. While LASTMASK has similarities with forecasting, the horizon length required for forecasting differs for each downstream task the the last segment length of the segmentation module.

Smaller models may outperform LPTM

We do not believe that smaller models trained from scratch can outperform LPTM. Indeed some of the baselines like EpiFNP, MICN, and ColaGNN which are not transformer-based have a smaller number of parameters but in most cases underperform LPTM significantly and only slightly outperform LPTM is specific benchmarks in few instances.

Differences with PEM work

This work and the submission "PEM: Pre-trained Epidemic Time-series Models" are very different and do not have any overlap:

1. PEM paper focuses on generating pre-trained epidemic time-series models that are exclusively used for epidemic analysis tasks such as forecasting, epidemic onset and peak detection, etc. This work tackles a much more general problem of pre-training models for generic time-series from different domains.
2. The novel contributions in PEM paper are mostly on novel SSL methods that are specifically designed for epidemic time-series. This work does not focus on any novel SSL task but instead introduces a novel segmentation module that can enable models to pre-train from datasets of multiple domains including and beyond epidemic time series.

While both works tackle challenges for pre-training time-series models, their approaches and scope are vastly different.