Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations

We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.

Clarification on the differentiability of the sorting process (calculation of σ from P)

While the calculation of σ from P is not differentiable, it is not used directly for the permutation of the segments. Instead, we use σ to populate the intermediate zero matrix Ω with elements from P. Matrix Ω is then turned into a binary permutation matrix which, when multiplied with the list of segments, permutes them in the correct order. Crucially, this matrix multiplication operation is differentiable, and Ω, which was built with elements from P, builds a bridge for the gradient to flow from the final output back to the elements of P. So even though the creation of σ itself is not differentiable, gradients can still flow back from the final output, through the permutation matrix, to P through the elements that were selected while creating σ.

Adding more recent forecasting baselines

We have now added S3 to PatchTST and CoST [1] for forecasting on the ETT (univariate and multivariate), Weather, and Electricity datasets. The results are shown in Table R1 of the enclosed document. In these additional experiments, we observe a similar trend where the addition of S3 improves performance.

Case study.

We now present 3 sample time-series before and after the S3 layer in Figure R1 of the accompanying doc (we will add more to the final paper). Here, we observe that S3 is optimized differently to learn a unique shuffling pattern that suits the baseline model and task. Given that the learnable parameters of S3 are optimized through training of the network and with the goal of maximizing the performance on that specific task, the final learned shuffling pattern serves the purpose of shuffling the time-series such that adjacent learning of originally non-adjacent segments results in more effective representations, as evidenced by the consistent improvement in results. The fact that the shuffling pattern in S3 is optimized with the learnable network and for each specific task is in fact an advantage as different models and different tasks may benefit from different re-structuring of the data.

Univariate vs. multivariate results.

Thank you for this interesting observation. One possible explanation for more improvements on multivariate time-series can be that multivariate datasets inherently contain more complex dynamics and interdependencies (both temporally and between variables), which can be more challenging for models to understand. Accordingly, S3 can further help with learning of the more complex temporal dependencies, further improving performance. Empirically, we can observe that models tend to struggle more with multivariate datasets from Tables 2 and 3 in the original paper. For instance, TS2Vec has an average MSE value of 0.1184 on ETTh1 univariate forecasting, and 0.7612 on ETTh1 multivariate forecasting, which is significantly larger than the former. While we acknowledge that comparing metric values for a model across different datasets or tasks can be misleading due to potential inconsistencies in data scales or nature of the task, in this case, both the dataset and the task are more or less the same. Accordingly, the key difference between univariate and multivariate settings here would be the complexity, i.e., the dataset and tasks are more complex in multivariate due to the higher number of dimensions.

Clarification on number of segments in Figure 6 and A3

The “number of segments” in Figure 6 and A3 from the original paper is the maximum number of segments over all S3 layers in the model. For instance, let us assume 3 layers of S3 with $n_0$=4, $n_1$=8, $n_2$=16 (where $n_i$ is the number of segments for layer $i$). Here, the maximum number of segments is 16. Similarly, if $n_0$=24, $n_1$=12, $n_2$=6, then the maximum number of segments is 24. We will make sure to clarify this in the final paper.

Inference process

During inference, we integrate S3 as the first layer of the model. The input sequences go through S3 first where they are segmented into several segments (the number of segments was optimized during training). These segments are then shuffled using the set of shuffling parameters that were optimized and fixed at the end of the training phase, and the learned weighted average yields the final sequence. This sequence is then fed into the subsequent stages of the baseline model for further processing and output generation. So essentially, S3 acts like any other learnable layer in a network where it is optimized during training and used with the fixed parameters during inference.

Computational overhead

S3 adds only a few hundred parameters, the exact count of which depends on the specific hyperparameters selected. We show the number of added parameters in Table 8 of the paper, where we observe that in comparison to hundreds of thousands to millions of parameters in the baselines, S3 adds very few parameters, which can be considered negligible. In terms of inference time, given the ratio of added parameters vs. the original number of parameters in the baseline models, the added time is also negligible.

Experiments on Anomaly detection

To further expand the tasks beyond univariate and multivariate classification and forecasting, we have now performed additional experiments for the task of anomaly detection on the KPI and Yahoo datasets. The results for this are shown in Table R4 of the enclosed document, where we observe that the addition of S3 results in considerable performance gains.

References

[1] Woo, et al., CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR, 2022.

We would like to thank the reviewer for their comments and further engaging with us on this matter. We think we now have a better understanding about the scope of the question.

To analyze the scenarios in which S3 is most effective, we have now performed additional experiments, and considered 3 factors: (1) length of time-series sequences, (2) non-linearity, and (3) long-term temporal dependency. For (1), we considered the %improvement vs. sequence length. For (2) we considered the %improvement vs. the mean of squared residuals w.r.t. a linear model. And finally for (3), we considered the %improvement vs. Hurst exponent [1]. We used PatchTST (transformer-based) as the baseline model given that it is the most recent work in the area, and used the ETT, Weather, and Electricity datasets. For sequence length, however, PatchTST uses equal input lengths for everything, so we used Informer (another transformer-based model) instead for which the input length is varied depending on the dataset and forecasting horizon. The outcome of this analysis is shown in the table below, where m is the slope of a linear relationship and R is Pearson's correlation coefficient. We observe that there are direct positive relationships between improvement by S3 versus sequence length, and also long-term temporal dependency, while the non-linearity in the time-series shows no relationship. For sequence length, the moderate correlation suggests that while the impact is not very strong, it is reliable, implying that sequence length is likely a relevant factor for %improvement. Long-term temporal dependency has a substantial impact on %improvement, as indicated by the large slope. Although, the lower correlation suggests that this relationship could be influenced by other factors in the data. We hypothesize that should these long-term temporal dependencies be simple to learn or highly repetitive, the impact of S3 will be less significant as re-ordering will not be required to learn such simple dynamics while for more complex long-term dependencies, S3 has a much stronger impact (as per the high slope).

To answer your second question, we have analyzed how w1 and w2 change over training of ETTh1 multivariate dataset with PatchTST. While unfortunately we cannot provide you with a new figure at this stage (due to NeurIPS rules), we have tried to provide a discretized version in the table below. We observe that the values steadily converge to the final results without too much fluctuation (some small fluctuations in the beginning are generally observed).

References

[1] Tong, et al., “Learning fractional white noises in neural stochastic differential equations", NeurIPS, 2022.

We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.

Comparison with data augmentation methods.

We acknowledge that at first glance S3 seems to share similarities to data augmentation. However, S3 has meaningful learnable parameters that train along with the rest of the model to enable a segmentation and mixup that is (a) variable throughout the training process, and (b) customized to that model and task, setting it highly apart. To further test this, we have now performed new experiments where we apply shuffling augmentation before training, shuffling augmentation plus mixup, noise augmentation, and noise augmentation plus mixup. In this experiment, we have also compared S3 to the augmentation method presented in the paper cited in your comment [1]. Please see Table R2 in the accompanying doc where we observe that S3 outperforms such data augmentation strategies. Moreover, a common characteristic of data augmentation techniques is that they tend to improve performance well for smaller datasets (often due to the lack of variations and diversity) but not so much for larger datasets [2]. To evaluate the impact of S3 based on the size of the training set, we took the LSST dataset from UEA and created 10 different subsets by dropping different amounts of data ranging from 20% to 99%. We then retrained SoftCLT [3] with and without S3 on these subsets as well as the original dataset. The results (averaged over 3 runs) are presented in Figure R2 of the accompanying doc. The results demonstrate that there is no evident trend in the performance gain with respect to the dataset size, indicating that S3 does not only benefit smaller datasets.

Experiments on a more recent transformer model

We have now added S3 to PatchTST and CoST [4] for forecasting on the ETT (univariate and multivariate), Weather, and Electricity datasets. The results are shown in Table R1 of the enclosed document. In these additional experiments, we observe a similar trend where the addition of S3 considerably improves performance.

Experiments on additional dataset for forecasting

We have now performed additional experiments for forecasting electricity and weather datasets for several baselines. The results are shown in Table R3 of the enclosed document.

Experiments on anomaly detection

To further expand the tasks beyond univariate and multivariate classification and forecasting, we have now performed additional experiments on two popular baselines with and without S3 for anomaly detection on the KPI and Yahoo datasets to further evaluate the improvements in representation learning that S3 brings. The results for this are shown in Table R2 of the enclosed document.

References

[1] Xiyuan Zhang, Ranak Roy Chowdhury, Jingbo Shang, Rajesh Gupta, and Dezhi Hong. Towards diverse and coherent augmentation for time-series forecasting. ICASSP, 2023.

[2] Iwana BK, Uchida S. An empirical survey of data augmentation for time series classification with neural networks. Plos one. 2021

[3] Seunghan Lee, Taeyoung Park, and Kibok Lee. Soft contrastive learning for time series. ICLR, 2024.

[4] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR, 2022.

We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.

Visualisation of S3

We have now included several visualizations in Figure R1 in the enclosed document that demonstrate how the segments are rearranged according to what the model thinks is optimal for the task. The figures allow for a visual comparison between the original sequence, the shuffled sequence, and the final output.

Additional baselines for forecasting, and fully supervised baseline for classification

We have implemented S3 with PatchTST, CoST [1] on the ETT (univariate and multivariate), Weather, and Electricity datasets for forecasting, and the results are shown in Table R1 of the enclosed document. We observe that S3 is able to further improve the performance of the additional baselines. For a fully supervised method, our original submission (Table 1) included the baseline DSN which was fully supervised.

S3 with pre-trained foundation models

Thank you for the very interesting suggestion! Given the very limited time during the rebuttal period and the resources needed to explore large foundation models, we were unable to perform these experiments at this time. We do agree that this area would be very interesting to explore for S3 and is indeed something we had discussed as an exciting future direction to take this work. We hope to follow up our current work with a follow up study on time-series foundation models.

Experiments on transfer learning

We have now performed an additional experiment in this regard: following the protocol used in TS2Vec [2], we trained the TS2Vec+S3 encoder on FordA for classification. We then froze the encoder and used the model along with fine-tuning of the classification head and S3 to classify the sequences for the other 127 UCR datasets. The average accuracy score over all 127 datasets for TS2Vec without S3 is 0.8037, and with S3 the accuracy score is 0.8160. Based on these results, it appears that S3 is indeed effective toward both in-dataset (in-distribution) and cross-dataset (out-of-distribution) settings. Please note that since the goal of S3 is to customize the reordering of the time-series specific to each dataset and task, the S3 layers would need to be fine-tuned as well (when the classification head is being re-trained), while the rest of the model stays frozen. We will add the full table of individual results on all the datasets in the final paper.

References

[1] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR, 2022.

[2] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. AAAI, 2022.

Dear Authors,

I really appreciate the time and effort that you have put into the study. I really like it, and I would like to maintain my current score to reflect my very positive assessment of this paper. I really appreciate figure R1, and I think it should find its way into the paper. Also the transfer learning experiments while preliminary, are very promising, and I would encourage the authors to add an element of this to their revised manuscript.

Best,

Reviewer 5XMz

We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.

Visualizations and qualitative analysis.

We have now included several visualizations in Figure R1 of the enclosed document. The figure shows how the segments are rearranged according to what the model thinks is optimal for the task.

Performance on transformer-based models.

We have now added S3 to PatchTST on the ETT (univariate and multivariate), weather, and electricity datasets and obtained the results presented in Table R1 of the accompanying document. We observe that S3 is able to further improve the performance of the additional baselines. Also, please note that in the initial set of experiments originally presented in our paper, Informer [1] is a transformer-based model, where we observed significant improvements when S3 was added (please see Tables 2 and 3 in our paper).

Does S3 need to be trained for each task?

We naturally retrain S3 for each task since the backbone model itself requires to be retrained as well. The S3 layer is simply added to the backbone and retrained just like all the other layers of the model. Having said this, we performed an experiment where we selected the exact same hyperparameters for all the tasks to observe how well a common hyperparameter could perform. We present this result in Table 6 of the original paper, where we observe that although the results are expectedly lower than the optimum hyperparameters, we still see meaningful gains.

References

[1] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai hang. Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI, 2021.

We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.

Other forecasting datasets.

As per your comment, we have now performed additional experiments on the popular Electricity and Weather datasets, and present the results in Table R2 in the accompanying doc. We observe that adding S3 results in overall improvements over all the baselines for both datasets.

CoST and other transformer baselines.

As per your comment, we have now added S3 to CoST and PatchTST [1] on the ETT (univariate and multivariate), Weather, and Electricity datasets, and present the results in Table R3 of the accompanying doc. We observe improvements for all the datasets.

Error bars.

The sources of variation in the performance of S3 are $\mathbf{P}$, $\mathbf{w}_1$, and $\mathbf{w}_2$ as well as the random seed for the baseline model to which S3 is added. We have performed an experiment where we present the standard deviations of several classification and forecasting models in Table 7 of the original paper over 5 random initializations. We observe that (a) the standard deviations are generally very small, and (b) any possible variance stems from the baseline model as opposed to the S3 module.

Ablation and comparison against augmentation.

(1) We performed a detailed ablation of each component of S3 in Table 5 of the original paper where we observed a drop in performance when each one is ablated.

(2) In regards to the question on augmentation, we acknowledge that at first glance S3 seems to share similarities to data augmentation. However, S3 has meaningful learnable parameters that train along with the rest of the model to enable a segmentation and mixup that is (a) variable throughout the training process, and (b) customized to that model and task, setting it highly apart. To further test this, we have now performed a new experiment where we apply shuffling augmentation (with different number of segments) before training, shuffling augmentation plus mixup, noise augmentation (with a mean of zero and a variance of 1, with a magnifying coefficient of 0.01), and noise augmentation plus mixup, on the ETTm1 multivariate dataset. Please see Table R2 in the accompanying doc where we observe that S3 outperforms such data augmentation strategies.

(3) Lastly, a common characteristic of data augmentation techniques is that they tend to improve performance well for smaller datasets (often due to the lack of variations and diversity) but not so much for larger datasets [2]. To evaluate this, we took the LSST dataset from UEA and created 10 different subsets by dropping different amounts of data ranging from 20% to 99%. We then retrained SoftCLT [3] with and without S3 on these subsets as well as the original dataset. The results (averaged over 3 runs) are presented in Figure R2 of the accompanying doc. The results demonstrate that there is no evident trend in the performance gain with respect to the dataset size, indicating that S3 does not only benefit smaller datasets.

References

[1] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.

[2] Iwana BK, Uchida S. An empirical survey of data augmentation for time series classification with neural networks. Plos one. 2021

[3] Seunghan Lee, Taeyoung Park, and Kibok Lee. Soft contrastive learning for time series. ICLR, 2024.