United We Train, Divided We Fail! Representation Learning for Time Series by Pretraining from 75 Datasets at Once

We thank you for your very thoughtful review and for pointing out positive aspects as well as open opportunities, which we try to address in the following.

We agree that we provided a very extensive overview of current approaches and their comparability to supervised learning.

Your raised points have greatly helped us to improve our manuscript. We address all your concerns below.

XIT only provides minor novelty over TS-TCC.

XIT's core encoder and TC module are derived from TS-TCC. We substitute its previous NT-Xent loss, integrating multiple runs of their TC loss. This configuration solely employs the cross-prediction task, presenting entirely novel inputs and outputs. Dismissing its novelty would be akin to disallowing the use of a Convolutional ResNet as a backbone in a new model. The underlying concept emphasizes the explicit need for both intra- and inter-dataset structuring tasks simultaneously.

The application of mixup on time series by Wickstrøm et al. (2022), on the other hand, is mainly a data augmentation method with the goal of learning a representation by predicting the mixing lambda. This is challenging for intra-dataset pretext tasks. For inter-dataset tasks, the model tends to learn to differentiate between datasets, which is not the goal of this task. We tackle this by ensuring that the lambda is not simply predicted but that the latent space corresponds to the time domain in terms of mixing distance. This ensures that we can really leverage cross-dataset information. Our experimental evaluation shows that this intuition carries over into practice. As a side note: In initial probings, we did not see any meaningful effect by simply predicting the lambda as proposed by Wickstrøm et al. (2022).

The significance of the SICC loss is not made clear in the ablations

Thank you for pointing that out. We updated the paper to make this more clear, see Table 3 (Pg. 8). The evaluation indeed shows that we need all of our proposed methods to improve the performance. More details are in the Global Response 3.

The TS-TCC framework is itself not intuitive to begin with - what exactly is the need for such data augmentations and contrived losses derived from these?

The need for weak and strong augmentations is already extensively discussed in the TS-TCC paper (see section 3.1 and ablations in section 5.4). The general motivation is that for a model to learn strong representations, one needs to pose a real challenge by means of the loss formulation. This is exactly what TS-TCC does, as predicting the weak from the strong augmented variant (and vice versa) can indeed be challenging. We, however, agree that simpler models should also be studied in further work.

Is it possible to predict z_t from z_t-1 for every t?

While it is possible, it would also increase the required memory tremendously. The SICC loss uses the predicted context of all elements in the batch. Building the required distance matrix of all pairs is fairly expensive, which one likely would not want to repeat for each time window. Furthermore, it is sometimes likely not an expressive task for time series with lots of leading padding.

Is it possible to replace this pipeline with a much more simple one?

Please see the second part of our answer to question one. However, in essence, simple mixup was not enough, and this is why we wanted to induce specific latent structuring methods for this specific challenge. However, making it easier is an interesting course of action and will be on our list of future work.

How is the context vector formed?

We utilize a sequence transformer (summarization model $\mathcal{S}$) taking the previous $z_k$, where $k \in [0, t-1]$ as input and producing a “forecast” of the context vector (see section 2.2). To get a more visual explanation, please see Figure 2 of our paper. For implementation details, please refer to “Encoder & Summarization Model” in appendix A.2.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

The authors thank the reviewer for the very extensive review, highlighting strengths and raising opportunities for improving our manuscript. This really helped us to improve upon our previous manuscript and helped to provide an even clearer scientific story.

We address all your concerns below.

Could the authors experimentally demonstrate why it is infeasible to use TS-TCC (and TF-C) for pre-training on multiple datasets?

We provided an intuition in Table 1, but in addition to that, we made a much more extensive experimental evaluation in Table 2. Next to even other baselines, it illustrates the superiority of XIT compared to the other methods.

Multiple standard baselines are missing.

We thank you for pointing that out. We provided 3 more baselines beyond TS-TCC and TF-C, as shown in the newly added Table 2. Namely, we evaluate TS2Vec (Yue et al., 2022), TNC (Tonekaboni et al., 2020), and T-Loss (Franceschi et al., 2019):

The results strengthen the empirical effectiveness of our method.

The experiment framework differs from some prior work, where the representations are used to train a downstream classifier such as SVM for classification.

While TS2vec indeed follows a slightly different scheme, we adhere to the standard linear benchmarking evaluation approach by Oord et al. (2018) and Chen et al. (2020), consistent with the evaluation methodology of TS-TCC and TF-C. In particular, we train a linear classifier (single MLP layer) on top of a frozen self-supervised pretrained encoder model.

Why didn’t you include accuracy in your evaluation?

We provided exemplary accuracy scores in the appendix for Table 1. However, especially with unbalanced datasets (UEA + low data sampling), accuracy may not be a reliable metric. Therefore, in line with TS-TCC, we also presented AUROC and Macro-F1 scores.

What was the structure of the model?

Refer to our response in question 3. Additionally, we have adjusted the manuscript for improved clarity.

The ablation results seem insignificant.

Thank you for pointing that out. The ablation results were indeed not presented in full clarity. To improve, we revamped our results in the new Table 3 by applying ranking after Demšar et al. (2006).

How are the models fine-tuned?

For the experiments in Table 1, we fine-tuned on 2.5% of the data. For the subsequent experiments, we fine-tuned on 100% of the hold-out dataset since many UCR datasets are already challengingly small.

We trained for 2000 steps with early stopping. For details, we provide a paragraph on “Implementation details” in Appendix A.2.

For the best possible outcome, we run a thorough hyperparameter search for the optimal classifier config. Finally, we ran the experiments on multiple seeds.

How sample efficient is the fine-tuning procedure?

This is the main challenge that we want to tackle with this paper. We particularly leverage the structures of multiple datasets in situations when large amounts of labeled data are not available. Therefore, the finetuning procedure is very sample efficient. Namely, the results in Table 1. were obtained from fine-share tuning on as little as 147 time series (HAR).

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

Dear Reviewer,

We appreciate the time and effort that you have taken to provide us with your insightful review. We would like to ask if the reviewer has any further concerns or is satisfied by our responses to the original review.

We are looking forward to any further discussion with the reviewer and would like to thank the reviewer again for helping make our paper better.

Regards, The Authors

Thank you for reading our paper and pointing out its strengths and opportunities for improvement.

We are pleased to see you agree that the topic is highly relevant and exciting, that the paper is written clearly, and that you find the ablation studies sound. We further improved the presentation of the ablation study to make the results clearer to interpret and even more convincing (see Global Response 3). We answer your concerns next.

More recent baselines

Thank you for bringing this up. We have now added many more recent baselines in the revised manuscript for a fair comparison.

In particular, we follow your suggestion to compare with TS2Vec (Yue et al., 2022). However, we use different models than CoST (Woo et al., 2022a) and OFA (Zhou et al., 2023). The first is exclusively introduced as a forecasting model, whereas we focus on classification. The latter was released after the paper submission, and we therefore exclude it and will happily investigate it in future work. Instead, we first evaluated the already included TF-C (Zhang et al., 2022) and TS-TCC (Eldele et al., 2021) models on the UCR dataset, each across three seeds and with their respective ranks.

Going even further, we added three other self-supervised time series pretraining models that are typically used for classification. Namely, we compared to TS2Vec (Yue et al., 2022), TNC (Tonekaboni et al., 2020), and T-Loss (Franceschi et al., 2019). Please see the Global Response (2. part) for the table of results of XIT and the in total five baselines.

We can see that in light of this significantly extended comparison, our method performs favorably and further supports the necessity of our method.

Novelty

We discussed this in general in the Global Response (1.). Specifically regarding your assessment, we want to point out that Sohn (2016) and Chen et al. (2020) laid the groundwork for a large family of different pertaining methods, of which XIT is indeed a member. This, however, makes it no less novel, especially given that soft alignments, as in SICC, are not typically used there.

Regarding the novelty of XD-MixUp, we want to point out that we reference both the original mixup idea by (Zhang et al., 2022) and the work of Wickstrøm et al. (2022) in section 2.1. To contextualize it further: XD-MixUp serves a very different purpose than the variant of MixUp in both publications. In the original one, it was introduced as a data augmentation and for soft labels in classification tasks. We, however, employ it to structure the (intermediate) latent space and effectively connect separated datasets into a coherent shared representation. Wickstrøm et al. (2022) proposed a completely separate learning approach with a different loss that promotes correctly predicting the mixing factor $\lambda$. However, this does not explicitly structure the latent space and instead relies on its implicit effects. In the initial probings of our research, we did not find this approach to be working in the face of the more challenging multi-dataset setting we tackle.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

Dear Reviewer,

We appreciate the time and effort that you have taken to provide us with your insightful review. We would like to ask if the reviewer has any further concerns or is satisfied by our responses to the original review.

We are looking forward to any further discussion with the reviewer and would like to thank the reviewer again for helping make our paper better.

Regards, The Authors

Thank you for the extensive review and depicting the strong points as well as pointing out opportunities for improvement.

Your raised points have greatly helped us to improve our manuscript. We address your concerns below.

XIT only provides minor novelty over TS-TCC and MixUp.

XIT's core encoder and TC module are derived from TS-TCC. We substitute its previous NT-Xent loss, integrating multiple runs of their TC loss. This configuration solely employs the cross-prediction task, presenting entirely novel inputs and outputs. Dismissing its novelty would be akin to disallowing the use of a Convolutional ResNet as a backbone in a new model. The underlying concept emphasizes the explicit need for both intra- and inter-dataset structuring tasks simultaneously.

Application of mixup on time series by Wickstrøm et al. (2022) is mainly a data augmentation method with the goal of learning a representation by predicting the mixing $\lambda$. This is challenging for intra-dataset pretext tasks. For inter-dataset tasks, the model tends to learn to differentiate between datasets. We tackle this by ensuring that $\lambda$ is not simply predicted but the latent space corresponds to the time domain in terms of mixing distance. This ensures that we can really leverage cross-dataset information. Our experimental evaluation shows that this intuition carries over into practice.

In initial probings, we did not see any meaningful effect by simply predicting the lambda as proposed by Wickstrøm et al. (2022).

The paper follows the same experimental setting as TF-C, however, reports a significantly lower performance and different metric than the reported one in the previous work.

Yes, in favor of greater comparability, we followed the experimental setup of TF-C as an initial experiment in Tab. 1. Furthermore, this experiment illustrates that TF-C exhibits unstable training, prone to worsening or erratic behavior with each new dataset, leading to unpredictable performance. In contrast, XIT demonstrates robustness, with minimal decreases and significant increases across all cases. TS-TCC performs adequately with 5 datasets but lacks robustness, exhibiting similar instability to TF-C when more data is leveraged. In datasets like ECG, consistent increases are observed across all models, yet XIT benefits the most. This motivates our new scaling experiment, shown in Tab. 2 of the paper. A small excerpt is shown in the global response 2.

We ran our experiments on the provided open-source code of TS-TCC as well as TF-C. However, the scores provided in the paper were not just reproducible. Also, note that we padded the time series to equal lengths, which was not performed in the respective initial publications. Similar difficulties were observed in the very extensive survey by Ma et al. (2023), see Table 13. There, the claimed superiority of TS-TCC could not be confirmed. Sadly, our attempts to contact the authors of TF-C were not responded to.

Table 1 reports an AUROC significantly lower than 0.5 for Epilepsy, which is a binary classification problem.

This is indeed surprising, but not impossible. The model performs way worse than random guessing, which is the case when the model overfits on a single class, as an example. This is especially the case for imbalanced datasets.

The standard deviation of Table 2. is very high, which makes it hard to reason about the result.

For this, we provided novel experiments (See answer 2).

Moreover, there is no evidence provided that this would not apply to TS-TCC and it even seems to improve with more datasets.

We provided more insights into the results of Tab. 1 in global answer 2. Furthermore, we ran additional experiments to show that TS-TCC does not scale to multiple datasets.

Standard deviations in ablation

Thank you for pointing this out. We improved the presentation of the results to make it easier to interpret them.See Global Response 3.

What is the performance of XIT for Table 1 using the same setting as TS-TCC for their table 2 and fig 3?

Our method makes inherent use of multiple datasets (specifically, through XD-MixUp and SICC). In contrast, Table 2 in Eldele et al. (2021) is a homogenous one-to-one experiment, e.g., HAR to HAR, which is not applicable in our case. Furthermore, their experiments assume an abundance of data, where supervised learning excels. We therefore evaluate a more challenging task, namely many-to-one training, e.g., in Tab. 1 and 2.

We also report on the experiments of scaling dataset amounts (in our case, in the many-to-one setting) in the appendix Fig. 5. We went as low as forming a single batch was still possible. We can hereby confirm the intuitively convincing findings of TS-TCC that in the face of high finetuning data amounts, supervised learning becomes increasingly feasible.

We hope that we have clarified your concerns and that you can reconsider our rating. Do let us know if you have any further questions or comments.

Dear Reviewer,

We appreciate the time and effort that you have taken to provide us with your insightful review. We would like to ask if the reviewer has any further concerns or is satisfied by our responses to the original review.

We are looking forward to any further discussion with the reviewer and would like to thank the reviewer again for helping make our paper better.

Regards, The Authors