Towards Generalisable Time Series Understanding Across Domains

(Q1) Do models pre-trained on a specific domain outperform those pre-trained across domains?

We evaluate our model against several domain-specific baselines that are either i) fully supervised or ii) pre-trained and fine-tuned exclusively on the target dataset. These include N-BEATS [15], TimesNet [16], Autoformer [20], DLinear [18], MAE [21], ViT [21], iTransformer [22], CM-AE [19], MMCL [21], and PatchTST [10]. The experiments show that OTiS outperforms such domain-specific approaches in 10 out of 15 benchmarks, with inferior performance observed in only 2 out of 15 benchmarks. We have conducted additional ablation studies to investigate different pre-training strategies for OTiS in the context of EEG event type classification. The results show that domain-specific pre-training does not provide improved downstream performance compared to pre-training across domains. We have added these observations to Appendix G.2.

[10] Nie, Y. et al. “A time series is worth 64 words: Long-term forecasting with transformers.” International Conference on Learning Representations (ICLR). 2023.

[15] Oreshkin, B. et al. "N-BEATS: Neural basis expansion analysis for interpretable time series forecasting." International Conference on Learning Representations (ICLR). 2019.

[16] Wu, H. et al. "TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis." International Conference on Learning Representations (ICLR). 2022.

[18] Zeng, A. et al. "Are transformers effective for time series forecasting?" AAAI Conference on Artificial Intelligence (AAAI). 2023.

[19] Radhakrishnan, A. et al. "Cross-modal autoencoder framework learns holistic representations of cardiovascular state." Nature Communications. 2023.

[20] Wu, H. et al. “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.” Advances in Neural Information Processing Systems (NeurIPS). 2021.

[21] Turgut, O. et al. "Unlocking the diagnostic potential of ecg through knowledge transfer from cardiac mri." arXiv preprint arXiv:2308.05764. 2023.

[22] Liu, Y. et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." International Conference on Learning Representations (ICLR). 2023.

---

(Q2) Does padding a large pre-training corpus to a fixed temporal/variate dimension waste computational resources?

We would like to clarify that we do not pad our pre-training corpus offline, as doing so would waste memory and limit scalability. Instead, we pad the variate dimension to the maximum number of variates within each batch. Furthermore, we use attention masking to ignore the padded tokens during gradient calculation, thus preventing a waste of computational resources.

(4) Additional baselines and interpretation of the results

We have added TimesNet [16] and iTransformer [22] as baselines for the classification and regression tasks, respectively. Moreover, we thank the reviewer for pointing out the performance differences between the ETT*1 and ETT*2 (both Electricity Transformer Temperature) datasets, which we also noticed during our study. The experiments indicate that the prediction of ETT*2 is generally easier than ETT*1 across all baselines. For both datasets, we have analysed the distribution shapes and the frequency components. Our findings reveal that ETT*1 exhibits long-tailed distributions and consistently includes large spikes, which may contribute to the increased difficulty in forecasting. Since the ETT*1 and ETT*2 were collected from two distinct regions in China [25], the external influences on the two transformers may greatly differ. For instance, one transformer may be positioned outside a steam vent or in a sunny spot, making its temperature harder to predict due to the influence of undocumented external signals.

[16] Wu, H. et al. "TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis." International Conference on Learning Representations (ICLR). 2022.

[22] Liu, Y. et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." International Conference on Learning Representations (ICLR). 2023.

[25] Zhou, H. et al. "Informer: Beyond efficient transformer for long sequence time-series forecasting." AAAI Conference on Artificial Intelligence (AAAI). 2021.

---

(5) Additional ablation study on the composition of the dual masking strategy

The composition of the masking schemes is empirically set to 75% random masking and 25% post-fix masking. We have included an ablation study on the composition of the masking schemes in Appendix G.1.

---

(6) Comparison with training from random initialisation and additional experiments in zero-shot settings

We have reworked the experiments section to include a randomly initialised OTiS that is trained fully supervised. The results confirm the widely reported advantages of pre-training [4][5][6][7][8][9]. Additionally, we have conducted an ablation study to investigate different pre-training strategies for OTiS on EEG event type classification, as detailed in Appendix G.2, which further stress these findings. Moreover, we have conducted experiments under zero-shot conditions. The zero-shot results in unseen domains, such as EMG, reveal that OTiS outperforms baseline models even without domain-specific training, underscoring the generalisability of its extracted time series features. We have included these observations in Appendix F and reworked the experiments section to present the zero-shot results.

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[5] Zhou, T. et al. "One fits all: Power general time series analysis by pretrained lm." Advances in Neural Information Processing Systems (NeurIPS). 2024.

[6] Goswami, M. et al. "MOMENT: A Family of Open Time-series Foundation Models." International Conference on Machine Learning (ICML). 2024.

[7] Woo, G. et al. "Unified Training of Universal Time Series Forecasting Transformers." International Conference on Machine Learning (ICML). 2024.

[8] Yang, C. et al. “Biot: Biosignal transformer for cross-data learning in the wild.” Advances in Neural Information Processing Systems (NeurIPS). 2024.

[9] Jiang, W. et al. “Large brain model for learning generic representations with tremendous EEG data in BCI.” International Conference on Learning Representations (ICLR). 2024.

Thank you for your extensive evaluation and constructive feedback on our work. We hope the following clarifications and additional experiments adequately address the points raised.

---

(1) Adaptation of the masked data modelling for time series analysis (e.g. regarding the masking ratio)

We would like to clarify that our contributions include the domain-specific tokenisation, the dual masking strategy, and the normalised cross-correlation loss, all of which are specifically designed for time series analysis. Additionally, we would like to emphasise that masked data modelling (MDM) is a widely adopted pre-training strategy in time series [6][7][8][9][10][12][21][24], primarily because it does not rely on heavy data augmentations difficult to design for sequential data [23]. Time series variates often exhibit high correlations, making higher masking ratios beneficial compared to the imaging modality, as they help eliminate redundancies in the learned representations. In our pre-training, we empirically set the masking ratio to 75%. Prior studies on MDM for time series, such as Ti-MAE [24], have explored optimal masking ratio for this modality. Their findings suggest that, similar to MDM in imaging, a masking ratio of 75% translates to best downstream performance.

[6] Goswami, M. et al. "MOMENT: A Family of Open Time-series Foundation Models." International Conference on Machine Learning (ICML). 2024.

[7] Woo, G. et al. "Unified Training of Universal Time Series Forecasting Transformers." International Conference on Machine Learning (ICML). 2024.

[8] Yang, C. et al. “Biot: Biosignal transformer for cross-data learning in the wild.” Advances in Neural Information Processing Systems (NeurIPS). 2024.

[9] Jiang, W. et al. “Large brain model for learning generic representations with tremendous EEG data in BCI.” International Conference on Learning Representations (ICLR). 2024.

[10] Nie, Y. et al. “A time series is worth 64 words: Long-term forecasting with transformers.” International Conference on Learning Representations (ICLR). 2023.

[12] Dong, J. et al. “SimMTM: A simple pre-training framework for masked time-series modeling.” Advances in Neural Information Processing Systems (NeurIPS). 2024.

[21] Turgut, O. et al. "Unlocking the diagnostic potential of ecg through knowledge transfer from cardiac mri." arXiv preprint arXiv:2308.05764. 2023.

[23] Assran, M. et al. "Self-supervised learning from images with a joint-embedding predictive architecture." Conference on Computer Vision and Pattern Recognition (CVPR). 2023.

[24] Li, Z. et al. "Ti-mae: Self-supervised masked time series autoencoders." arXiv preprint arXiv:2301.08871. 2023.

---

(2) Can a shared patch projector reflect different semantics among variates and domains? Could using different patch sizes for different frequencies, as in MOIRAI [7], lead to further improvements?

The authors of MOIRAI [7] (2024) presented a subsequent study [26] (2024) in which they eliminate the dependency on multiple projection layers for different frequencies. Instead, they employ a shared projection layer with a unified patch size across all frequencies (i.e. domains). They argue that frequencies are not a reliable indicator of the underlying patterns in time series, and that human-imposed inductive biases may hinder model generalisability. We agree with the authors and believe that projection layers should be viewed as general feature extractors, independent of frequency, variate, or domain. The extracted features serve as a learned vocabulary, which can then be slightly modulated to the domain and variate through specific positional embeddings, as implemented in OTiS.

[26] Liu, X. et al. "Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts." arXiv preprint arXiv:2410.10469. 2024.

---

(3) How can domain-specific variate embeddings capture inter-variate relationships? These learned embeddings may limit generalisability, as they do not translate to unseen variates/domains during inference.

To investigate whether adaptation to unseen domains is required for competitive performance, we have conducted additional experiments under zero-shot conditions, as detailed in Appendix F. The zero-shot results in unseen domains, such as EMG, reveal that OTiS outperforms baseline models even without domain-specific fine-tuning, underscoring the generalisability of its extracted time series features. We have included these observations in Appendix F and reworked the experiments section to present the zero-shot results.

Thank you for the replies. I have read the rebuttal and the revision, which addressed my concerns regarding the performance and method design. But there are a few unsolved concerns:

Regarding W2: Although the authors provide another work to support it, I would be interested to know if the authors have done some specific empirical evaluations to draw this conclusion.

Regarding W3: I don't think the author answered my question. I agree that learnable embeddings can help the model distinguish the heterogeneity of data in different domains (which may lead to what the rebuttal has mentioned: OTiS can outperform baseline models without domain-specific fine-tuning). However, I still cannot be convinced that the "learnable embeddings can explicitly capture inter-variate relationships" mentioned in this work.

Regarding Q1: I read Appendix G.2 carefully: the authors provide further experiments to prove that the pre-trained model can outperform the models supervised-trained (or self-supervised trained + fine-tuned) on one dataset. The results are not convincing to me because of (1) the lack of state-of-the-art baseline models and self-supervised training methods, such as PatchTST, and (2) The results cannot solve the concern of OTiS in data scaling. Concretely, training OTiS on (1) a set of datasets (not one) related to the target dataset and (2) a domain-universal dataset (which may include the former set and some less related datasets). Is it possible that the first model that is pre-trained on a smaller scale works better?

(3) ctd, Does OTiS pre-trained on domain-specific datasets outperform OTiS pre-trained across domains?

We have updated Table 12 accordingly with the results of PatchTST [10], as outlined in the following.

$*$ Model was randomly initialized and trained fully supervised.

$^\dagger$ Model was pre-trained only with the EEG data of our pre-training corpus (i.e. TDBrain [33] and SEED [34]).

The experiments reveal that general models (i.e., OTiS-Base$_\text{EEG}$), trained on a smaller scale, do not perform better than foundation models (i.e., OTiS-Base and LaBraM), trained on a large scale.

We have revised Appendix G.2 to more carefully highlight these observations and hope that our discussion and clarifications adequately address the points you raised. If there are any remaining questions or concerns, we are happy to discuss them further.

(3) Does OTiS pre-trained on domain-specific datasets outperform OTiS pre-trained across domains?

Thank you for pointing this out out; we believe there may be a misunderstanding here. Note that OTiS-Base$_\text{EEG}$ in Table 12 refers to OTiS pre-trained on TDBrain [33] and SEED [34], i.e. a set of two EEG datasets related to the target TUEV dataset [35]. In contrast, OTiS-Base refers to OTiS pre-trained on the full pre-training corpus detailed in Table 1 of our manuscript.

The additional experiments on the TUEV [35] data provided in Appendix G.2 show that both OTiS-Base$_\text{EEG}$ and OTiS-Base outperform i) domain-specific baselines (either fully supervised or pre-trained and fine-tuned on the target dataset, i.e. one dataset), ii) general baselines (pre-trained on few external source datasets and fine-tuned on the target dataset), and even iii) foundation models (pre-trained on multiple external source datasets and fine-tuned on the target dataset). The domain-specific baselines include ST-Transformer [36], CNN-Transformer [37], FFCL [38], and SPaRCNet [39]. The general methods include ContraWR [40]. The foundation methods include BIOT [8] and LaBraM [9]. We would like to clarify that other than stated by the reviewer, these latter models represent state-of-the-art baselines that are trained using self-supervised learning. Additionally, we have introduced PatchTST [10] as a new baseline. We have reworked Appendix G.2 to summarise the baselines, similar as in the following Table.

$*$ Pre-trained on 6 EEG datasets (totalling 13,000 recording hours), including the target dataset (i.e. TUEV [35])

$^+$ Pre-trained on 16 EEG datasets (totalling 2,500 recording hours), including TUAR [41], TUEP [42], TUSZ [43], and TUSL [44], which are subsets of the TUH [35]. As TUEV [35] is a subset of TUH [35], too, there may be potential information leakage through overlapping subjects between subsets.

(4) Clarification of the baseline models

We have reworked the experiments section to clarify the categorisation of the baselines. Additionally, we have included a summary of all baselines, detailing their architectures, pre-training strategies, and domain adaptation techniques, in Appendix B.

---

(5) Comparison with traditional baselines trained on a single, specific dataset

In extensive benchmarking, we compare our model against multiple domain-specific baselines that are either i) fully supervised or ii) pre-trained and fine-tuned exclusively on the target dataset. These include N-BEATS [15], TimesNet [16], Autoformer [20], DLinear [18], MAE [21], ViT [21], iTransformer [22], CM-AE [19], MMCL [21], and PatchTST [10]. These baselines span all key use cases in time series analysis, providing a comprehensive comparison. The experiments show that OTiS outperforms such domain-specific approaches in 10 out of 15 benchmarks, with inferior performance to such approaches in only 2 out of 15 benchmarks. We have conducted an additional ablation study to investigate different pre-training strategies for OTiS, as detailed in Appendix G.2, which further stress the widely reported advantages of general pre-training across domains [4][5][6][7][8][9].

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[5] Zhou, T. et al. "One fits all: Power general time series analysis by pretrained lm." Advances in Neural Information Processing Systems (NeurIPS). 2024.

[6] Goswami, M. et al. "MOMENT: A Family of Open Time-series Foundation Models." International Conference on Machine Learning (ICML). 2024.

[7] Woo, G. et al. "Unified Training of Universal Time Series Forecasting Transformers." International Conference on Machine Learning (ICML). 2024.

[8] Yang, C. et al. “Biot: Biosignal transformer for cross-data learning in the wild.” Advances in Neural Information Processing Systems (NeurIPS). 2024.

[9] Jiang, W. et al. “Large brain model for learning generic representations with tremendous EEG data in BCI.” International Conference on Learning Representations (ICLR). 2024.

[10] Nie, Y. et al. “A time series is worth 64 words: Long-term forecasting with transformers.” International Conference on Learning Representations (ICLR). 2023.

[15] Oreshkin, B. et al. "N-BEATS: Neural basis expansion analysis for interpretable time series forecasting." International Conference on Learning Representations (ICLR). 2019.

[16] Wu, H. et al. "TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis." International Conference on Learning Representations (ICLR). 2022.

[18] Zeng, A. et al. "Are transformers effective for time series forecasting?" AAAI Conference on Artificial Intelligence (AAAI). 2023.

[21] Turgut, O. et al. "Unlocking the diagnostic potential of ecg through knowledge transfer from cardiac mri." arXiv preprint arXiv:2308.05764. 2023.

[22] Liu, Y. et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." International Conference on Learning Representations (ICLR). 2023.

Thank you for your engaging comments and the thorough evaluation. We hope the following clarifications and additional experiments adequately address the points raised.

---

(1) A patch projector with a unified patch size, shared across all domains, fails to account for unique sampling frequencies and does not adequately address differences in temporal dynamics

Recent state-of-the-art foundation models, such as MOIRAI [7] (2024), introduce multiple projection layers with different patch sizes to handle distinct frequencies. However, the authors of MOIRAI presented a subsequent study [26] (2024) in which they eliminate the dependency on multiple projection layers for different frequencies. Instead, they employ a shared projection layer with a unified patch size across all frequencies (i.e. domains). They argue that frequencies are not a reliable indicator of the underlying patterns in time series, and that human-imposed inductive biases may hinder model generalisability. We agree with the authors and believe that projection layers should be viewed as general feature extractors, independent of frequency, variate, or domain. The extracted features serve as a learned vocabulary, which can then be slightly modulated to the domain and variate through specific positional embeddings, as implemented in OTiS.

[7] Woo, G. et al. "Unified Training of Universal Time Series Forecasting Transformers." International Conference on Machine Learning (ICML). 2024.

[26] Liu, X. et al. "Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts." arXiv preprint arXiv:2410.10469. 2024.

---

(2) Considering variate heterogeneity through learned embeddings is not uncommon

Prior to the stated works [2][3], learnable 2D positional embeddings were extensively studied by Dosovitskiy et al. [29], and we do not claim this as a novel aspect of our study. Instead, we introduce a unique approach by employing non-learnable temporal embeddings (shared across domains) and learnable variate embeddings (specific to each domain). This special composition of the “2D” positional embeddings represents a novel contribution of our study, which to the best of our knowledge has not been explored in time series analysis before.

[2] Heterogeneity-Informed Meta-Parameter Learning for Spatiotemporal Time Series Forecasting, KDD, 2024

[3] Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting, NeurIPS, 2020

[29] Dosovitskiy, A. et al. "An image is worth 16x16 words: Transformers for image recognition at scale." Advances in Neural Information Processing Systems (NeurIPS). 2020.

---

(3) Comparison with channel-independent baselines

We have conducted extensive benchmarking to compare our model against several state-of-the-art baselines employing channel-independent strategies, similar to the proposed UniTime [1]. These include N-BEATS [15], TimesNet [16], TF-C [17], DLinear [18], PatchTST[10], CM-AE [19], Time-LLM [4], GPT4TS [5], and MOMENT [6]. Covering all key use cases in time series analysis, such as classification, regression, and forecasting, these baselines provide a comprehensive comparison. Our experiments reveal that OTiS outperforms such channel-independent approaches in 10 out of 15 benchmarks, with inferior performance to such approaches in only 2 out of 15 benchmarks. These results validate the effectiveness of OTiS’ channel-mixing strategy.

[1] UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting, WWW, 2024

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[5] Zhou, T. et al. "One fits all: Power general time series analysis by pretrained lm." Advances in Neural Information Processing Systems (NeurIPS). 2024.

[6] Goswami, M. et al. "MOMENT: A Family of Open Time-series Foundation Models." International Conference on Machine Learning (ICML). 2024.

[10] Nie, Y. et al. “A time series is worth 64 words: Long-term forecasting with transformers.” International Conference on Learning Representations (ICLR). 2023.

[15] Oreshkin, B. et al. "N-BEATS: Neural basis expansion analysis for interpretable time series forecasting." International Conference on Learning Representations (ICLR). 2019.

[16] Wu, H. et al. "TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis." International Conference on Learning Representations (ICLR). 2022.

[17] Zhang, X. et al. "Self-supervised contrastive pre-training for time series via time-frequency consistency." Advances in Neural Information Processing Systems (NeurIPS). 2022.

[18] Zeng, A. et al. "Are transformers effective for time series forecasting?" AAAI Conference on Artificial Intelligence (AAAI). 2023.

[19] Radhakrishnan, A. et al. "Cross-modal autoencoder framework learns holistic representations of cardiovascular state." Nature Communications. 2023.

Thanks a lot for the quick response and the suggestion! We would like to clarify that the suggested UniTS [30] does not utilise any text representations. Instead, it employs learnable domain-agnostic variate embeddings (i.e., learnable embeddings shared across all domains) to implicitly accommodate distinct domains.

To empirically evaluate the effectiveness of our learnable domain-specific variate embeddings, we have conducted an ablation study, as described in the experiments section. In this study, we replaced the learnable domain-specific embeddings with learnable domain-agnostic variate embeddings, similar to the approach in UniTS [30]. The results, presented in Figure 5, highlight two key advantages of domain-specific variate embeddings: (i) enhanced robustness, evidenced by a smaller interquartile range, and (ii) improved downstream performance.

Additionally, in extensive benchmarking experiments we compare our model against several state-of-the-art baselines, including those that leverage text representations to distinguish between domains. For instance, in Time-LLM [4], the authors encode explicit descriptions of both the dataset and the domain as text representations, which are then used with the time series representations to perform forecasting tasks. Our experiments demonstrate that OTiS outperforms such approaches in 4 out of 6 forecasting benchmarks, effectively validating the utility of learnable domain-specific variate embeddings.

We have updated the related works section to include a discussion of UniTS [30] and hope that these experiments and comparisons adequately address the point you raised. If there are any remaining questions or concerns, we would be happy to discuss them further.

---

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[30] Gao, S. et al. "UniTS: A unified multi-task time series model." Advances in Neural Information Processing Systems (NeurIPS). 2024.

Dear Reviewer yUJh,

Thank you once again for engaging in a discussion.

In your previous response, you raised the concern whether the domain-specific variate embeddings used in our study are effective in distnguishing different domains. You suggested comparing our model against the UniTS [30] model, which uses domain-agnostic embeddings, and baselines that use textual representations to differentiate between domains. To address this concern, we have (1) conducted an ablation study to investigate domain-agnostic embeddings, similar to UniTS [30], and (2) compared our model against the Time-LLM [4] model, which uses textual representations for domain differentiation. These experiments demonstrate that our domain-specific variate embeddings are most effective in distnguishing different domains, yielding to robust improvements in downstream performance.

We hope these efforts adequately address the remaining point you raised. If you believe so, we would greatly appreciate a final adjustment of your scores to reflect this.

Authors

---

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[30] Gao, S. et al. "UniTS: A unified multi-task time series model." Advances in Neural Information Processing Systems (NeurIPS). 2024.

Thank you for your thoughtful comments on our work. We hope the following clarifications and additional analyses adequately address the points raised.

---

(1) Claims regarding the generalisation across domains may be overstated

See (Q1)

---

(2) Analysis of variate embeddings in domains with complex relationships (i.e. where variates are not as straightforward as their spatial arrangement)

We have analysed further domain-specific variate embeddings in Appendix E.2, showcasing that OTiS is capable of capturing complex inter-variate relationships. We acknowledge the concerns of the reviewer that high correlations between spatially proximate variates (e.g. EEG electrodes) might facilitate learning these relationships. The reviewer has suggested investigating the relationship between voltage $U$ [V] and current $I$ [A], described by $U = R * I$, where $R$ [Ohm] denotes resistance. However, the linear relationship between these two variates may represent a trivial case, while scenarios involving more complex (i.e. non-linear) inter-variate relationships would offer deeper insight into OTiS’ modelling capabilities. To this end, the Weather dataset [14] provides a more suitable test case, spanning diverse climatological categories such as temperature, humidity, wind, radiation, pressure, and precipitation, which exhibit non-linear relationships. As detailed in Appendix E.2, our exploration of Weather-specific variate embeddings learned during fine-tuning demonstrates that OTiS effectively models such complex relationships.

[14] Max Planck Institute for Biogeochemistry. “Weather station.” 2024. https://www.bgc-jena.mpg.de/wetter/.

---

(Q1) How does the domain-specific tokeniser adapt to unseen domains?

Let $S$ denote a previously unseen domain with $V_S$ variates and $D$ denote the embedding dimension of our model. We randomly initialise variate embedding $E_S^V \in R^{V_S \times D}$ and fine-tune them along with the encoder and, if required, the decoder, for the specific application in $S$. To investigate whether adaptation to unseen domains is even necessary for competitive performance, we have conducted additional experiments under zero-shot conditions, as detailed in Appendix F. The zero-shot results in unseen domains, such as EMG, reveal that OTiS outperforms baseline models even without domain-specific fine-tuning, underscoring the generalisability of its extracted time series features. We have included these observations in Appendix F and reworked the experiments section to present the zero-shot results.

---

(Q2) How does the domain-specific tokeniser generalise across different systems within the same domain (e.g. electrical transformers and power generators in an imaginary "Energy" domain)?

Similar to how we separate the EEG and ECG domains in our pre-training corpus, rather than combining them under a broader “Medicine” domain, one could similarly define distinct domains for electrical transformers and power generators. See (Q3) for a more detailed discussion on the definition of domains.

---

(Q3) At what level of granularity should domains be defined?

In general, the level of granularity at which to define a domain depends on the underlying characteristics of the data. We believe that a domain should be defined at a level where the data shares meaningful patterns, particularly with respect to inter-variate relationships and temporal dynamics. For example, we define the NN5 dataset [27] (daily cash withdrawals) as ‘Banking’ domain and the FRED-MD dataset [28] (macro-economic indicators) as ‘Economics’ domain, even though both could broadly fall under a ‘Finance’ domain. However, the Banking domain is characterised by high periodicity and little long-term trends, whereas the Economics domain exhibits the opposite. The key is to balance between a too broad definition, which may obscure important patterns, and too narrow definition, which may limit generalisation. As discussed in our limitations section, automated pipelines that leverage embedding similarities to compare datasets could aid in defining domains, reducing reliance on human-imposed inductive biases.

[27] Taieb, S. et al. "A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition." Expert systems with applications. 2012.

[28] McCracken, M. W. et al. "FRED-MD: A monthly database for macroeconomic research." Journal of Business & Economic Statistics. 2016.

---

(Ethical concerns) Authors use a GitHub link to share the code, which could lead to personal information leakage and may require further investigation.

Regarding the ethics concerns, we have carefully set up the GitHub repository upon submission, excluding any identifying metadata, commit histories, or personal information. We thus strictly adhere to anonymity guidelines while maintaining reproducibility.

(Q4) Is there any intuition behind what information could be shared across such diverse domains to make pre-training on them useful?

For OTiS, we employ a shared projection layer across all frequencies, variates, and domains, as we view this layer as a general feature extractor. We believe that low-level logic in time series, e.g. periodicity (or more simply, the pattern that a “low” is often followed by a “high”), can be learned across domains. We have analysed the time series of our diverse pre-training at the scale of a single patch (size 24) and found that, visually, they are indistinguishable at this scale: they all exhibit periodicity, regardless of the domain. We hypothesise that OTiS effectively captures such patterns across domains, which can then be leveraged during fine-tuning. This is particularly beneficial for domains with limited data, where the available data is often insufficient for learning such patterns with a randomly initialised model.

(Q1) How long does it take to fine-tune on new tasks?

We fine-tune our model on all tasks using a single NVIDIA RTX A6000-48GB GPU and 32 CPUs. With this setup, the training times for the three example tasks are as follows.

---

(Q2) How does the fine-tuned model perform compared to task-specific models? Are the test datasets representative of cases that require pre-trained models?

We evaluate our model against current state-of-the-art baselines using the established benchmark datasets in time series analysis. Our experiments confirm the widely reported advantages of pre-training for these very benchmarks [4][5][6][7][8][9][10][11][12], demonstrating that (i) general models (pre-trained on external source data and fine-tuned on the target data) and (ii) foundational models (pre-trained on large corpora and fine-tuned on the target data) outperform (iii) domain-specific models (either fully supervised or pre-trained and fine-tuned exclusively on the target data). The baselines in our experiments include 11 domain-specific models, 6 general models, and 4 foundation models. We have conducted an additional ablation study (further 4 domain-specific models, 1 general model, and 2 foundation models) to investigate different pre-training strategies for OTiS, as detailed in Appendix G.2, which further stress these findings. In conclusion, we have reworked the experiments sections to provide details on the baselines and to highlight the advantages of pre-training.

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[5] Zhou, T. et al. "One fits all: Power general time series analysis by pretrained lm." Advances in Neural Information Processing Systems (NeurIPS). 2024.

[6] Goswami, M. et al. "MOMENT: A Family of Open Time-series Foundation Models." International Conference on Machine Learning (ICML). 2024.

[7] Woo, G. et al. "Unified Training of Universal Time Series Forecasting Transformers." International Conference on Machine Learning (ICML). 2024.

[8] Yang, C. et al. “Biot: Biosignal transformer for cross-data learning in the wild.” Advances in Neural Information Processing Systems (NeurIPS). 2024.

[9] Jiang, W. et al. “Large brain model for learning generic representations with tremendous EEG data in BCI.” International Conference on Learning Representations (ICLR). 2024.

[10] Nie, Y. et al. “A time series is worth 64 words: Long-term forecasting with transformers.” International Conference on Learning Representations (ICLR). 2023.

[11] Zhang, X. et al. “Self-supervised contrastive pre-training for time series via time-frequency consistency.” Advances in Neural Information Processing Systems (NeurIPS). 2022.

[12] Dong, J. et al. “SimMTM: A simple pre-training framework for masked time-series modeling.” Advances in Neural Information Processing Systems (NeurIPS). 2024.

---

(Q3) How is the ground truth obtained in Figure 3?

We have added a detailed explanation of the alignment of the learned EEG-specific variate embeddings with the true electrode layout to Appendix E.2. Note that the electrode placement of all EEG datasets used in our study follows the international 10-20 system for EEG recordings [13]. However, we would like to clarify that the 3D electrode coordinates of the 10-20 EEG system are not used for training. Instead, our model implicitly learns to model the spatial structure solely from the EEG recordings seen during training. The term “ground truth” could thus be irritating, which is why we would consider the electrode coordinates only as reference points. To determine how well the learned EEG-specific variate embeddings reflect the true electrode layout of the 10-20 EEG system, we perform the following steps. Assume the 3D electrode coordinates of the 10-20 EEG system to be defined in Euclidean space $\mathbb{E}_Y^3$. We first project the EEG-specific variate embeddings into Euclidean space $\mathbb{E}_X^3$, then align them with the 3D electrode coordinates of the 10-20 EEG system in $\mathbb{E}_Y^3$ through multivariate linear regression, and eventually quantify their correlation by determining the coefficient of determination $R^2$.

[13] Homan, R. et al. "Cerebral location of international 10–20 system electrode placement." Electroencephalography and clinical neurophysiology. 1987.

Thank you for your careful and thorough evaluation of our work. We hope the following clarifications and additional experiments adequately address the points raised.

---

(1) Explanation and interpretation of the results concerning generalisability and performance gains

We have reworked the experiments section and added discussions to Appendix F and G, explaining the results of our study in more detail. To summarise, we observe that domain-specific models (either fully supervised or pre-trained and fine-tuned exclusively on the target data) are inferior to general models (pre-trained on external source data and fine-tuned on the target data) and foundational models (pre-trained on large corpora and fine-tuned on the target data), as discussed in the experiments section and Appendix G.2. Moreover, we have conducted additional zero-shot experiments, demonstrating that our model is able to extract distinct representations for different inputs, as discussed in Appendix F. These distinct representations can be observed across domains and tasks, suggesting that the time series features extracted by OTiS are generalisable. Our experiments further show that adaptation to the specific task through fine-tuning generally boosts downstream performance.

---

(2) Clarification of the principal component analysis

We have reworked the results section to clarify the principal component analysis. In particular, we have added a detailed explanation on the alignment of the EEG-specific variate embeddings with the 3D electrode coordinates of the international 10-20 system for EEG recordings to Appendix E.2. See (Q3).

---

(3) Analysis of the overlap between training and testing variables

We are not entirely sure what the reviewer means by variables, but we have extensively analysed the effects of domain-specific variate embeddings in full training and zero-shot settings in the experiments section and the Appendix. We believe this analysis shows why our method outperforms the baselines, but if the reviewer has a specific aspect that they would like to discuss further in detail, we would happily engage.

Dear Reviewer pz7i,

We would like to follow up on the discussion period to ensure that all of your points have been addressed by our rebuttal. In particular, we have worked to provide detailed responses to your questions regarding the (1) training time during fine-tuning, (2) downstream performance of models trained exclusively on the target data, (3) principal component analysis of the EEG-specific variate embeddings, and the (4) intuition behind the information shared across domains that makes pre-training on them beneficial.

We hope our rebuttal, along with the discussion involving Reviewers UJog and tXLU, has adequately addressed the points you raised. If you believe this to be the case, we would greatly appreciate a final adjustment of your scores to reflect this.

Thanks again for your evaluation of our work and your valuable feedback.

Authors

Thank you for your constructive comments and thorough evaluation. We hope the following clarifications and additional experiments adequately address the points raised.

---

(1) Clarification of the baseline models

We have added a subsection to the experiments section, discussing the categories of the baselines. Moreover, we have included a summary of all baselines, detailing their architectures, pre-training strategies, and domain adaptation techniques, in Appendix B.

---

(2) Comparison with state-of-the-art foundation models for time series analysis

We have compared our approach against six state-of-the-art foundation models, including Time-LLM [4], GPT4TS [5], MOMENT [6], MOIRAI [7], BIOT [8], and LaBraM [9], across classification and forecasting tasks, as discussed in the experiments section and Appendix G.2.

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[5] Zhou, T. et al. "One fits all: Power general time series analysis by pretrained lm." Advances in Neural Information Processing Systems (NeurIPS). 2024.

[6] Goswami, M. et al. "MOMENT: A Family of Open Time-series Foundation Models." International Conference on Machine Learning (ICML). 2024.

[7] Woo, G. et al. "Unified Training of Universal Time Series Forecasting Transformers." International Conference on Machine Learning (ICML). 2024.

[8] Yang, C. et al. “Biot: Biosignal transformer for cross-data learning in the wild.” Advances in Neural Information Processing Systems (NeurIPS). 2024.

[9] Jiang, W. et al. “Large brain model for learning generic representations with tremendous EEG data in BCI.” International Conference on Learning Representations (ICLR). 2024.

---

(3) Evaluation of the prompting (i.e. zero-shot performance)

We have conducted further experiments to investigate the quality of the time series features extracted by our frozen model. These include zero-shot experiments for classification, linear probing for regression, and minimal tuning (< 1k trainable parameters) for forecasting, as detailed in the experiments section and Appendix F.

Dear Reviewer Aff6,

Thank you once again for your positive feedback on our study.

We would like to follow up on the discussion period to ensure that all your points have been addressed in our rebuttal. Specifically, we have (1) added a subsection to categorise all baseline models, (2) provided comparisons with state-of-the-art time series foundation models [45], including Time-LLM [4] and GPT4TS [5], and (3) included a new subsection to evaluate our model in zero-shot settings.

We hope these updates to the manuscript have adequately addressed the points you raised. If you agree, we would greatly appreciate a final adjustment of your scores to further support our study.

Authors

---

[4] Jin, M. et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." International Conference on Learning Representations (ICLR). 2023.

[5] Zhou, T. et al. "One fits all: Power general time series analysis by pretrained lm." Advances in Neural Information Processing Systems (NeurIPS). 2024.

[45] Ye, J. et al. "A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Model." arXiv preprint arXiv:2405.02358. 2024.