Self-Supervised Contrastive Learning for Long-term Forecasting

Second, the following R-Table 4 demonstrates the change in performance based on the complexity of the long-term branch in our decomposition architecture. Without Autocon, our model may be slightly better or comparable to the second-best model. The highest performance is achieved only when the AutoCon loss is employed. This further underscores the necessity of the AutoCon loss we proposed in order to accurately capture long-term variation.

R-Table 4: Increasing complexity of long-term branch of our model over three datasets in output-720 univariate setting. Other setting is the same to experiments as shown in M-Table 1.

W3: Applying our AutoCon with high-order and non-linear correlation

As reviewer hiP7 mentioned, thoroughly calculating high-order correlation across a very long time series can be computationally challenging. What we have claimed and demonstrated in this work, however, is the necessity of leveraging global autocorrelation for long-term predictions. These two are independent topics, and we find it difficult to understand why the former is pointed out as a "weakness" of the latter. Even if one decides to combine the two somehow in the future (e.g. extend our work to incorporate high-order global correlation), there are many possibilities to explore. For example, one could try to reduce the computational complexity by leveraging domain knowledge of the specific dataset or the task (e.g. ignore the correlation between certain variables or certain time durations). Alternatively, one could propose to capture the global high-order correlation in a completely different manner than how we capture the global autocorrelation. Therefore, we find it unconvincing that the reviewer would use an independent problem as a weakness for the current submission.

W4: Typos

Thank the reviewer for pointing out the typos. We have corrected them in the revised manuscript.

References

[1] Efficient tests for an autoregressive unit root, 1992
[2] Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting, NeurIPS'22
[3] Are Transformers Effective for Time Series Forecasting?, AAAI'23

W1: Collapse of our contrastive framework with stationary time-series

We thank reviewer hiP7 for the insightful comment regarding the potential challenges posed by nearly stationary input sequences in our contrastive learning framework. We understand your concern that "the nearly stationary sequences" might lead to stationary autocorrelation results, which could, in turn, affect the contrastive learning mechanism.

However, even in the nearly stationary sequences, the autocorrelation does not produce stationary results since stationary results in autocorrelation are theoretically possible only in the case of infinitely long time series. In practice, we deal with finite samples from time series. In such cases, autocorrelation converges to zero, decreasing from 1 to 0 as the lag $\tau$ increases even in "the nearly stationary time series". In other words, autocorrelation exhibits a non-stationary trend and shows a decreasing pattern. This is because the covariance with lagged series $Cov(S_{t+\tau}, S_t)$ decreases as $\tau$ increases when numerically calculating the autocorrelation. For example, as shown in M-Figure 5, the autocorrelation results of four datasets converge regardless of their stationarity.

Additionally, the Weather time series, which is considered the most stationary among the tested time series according to the Augmented Dick-Fuller (ADF) test [1] as shown in statistics [2], also converges to 0 in a similar manner. Naturally, such stationarity implies small (or absence of) long-term variation. Therefore, even if our method is applied, as described in Dataset Analysis of Section 4.1, performance improvement could be marginal (3% reduced MSE). However, we believe there is an unequivocal difference between gaining only a slight performance improvement and "collapse of contrastive learning". (If reviewer hiP7 was trying to convey a different meaning than "collapse", we would be happy to further discuss this matter.)

W2: Increasing complexity of long-term branch in the decomposition architecture

Increasing the complexity of the long-term branch is essential for learning long-term representations, but it is not the only reason for the superior performance of our methodology. In other words, even with the increased complexity, capturing long-term variation is not an easy task if the current framework only uses the forecasting loss. It is essential to use the AutoCon loss to effectively learn long-term variation that leads to performance improvement. We have already demonstrated the need for AutoCon loss through ablations in M-Figure 6 and M-Table 3. However, to fully address the reviewer's concerns, we conducted two additional ablation studies: increasing complexity in DLinear [3] and in our model.

First, DLinear utilizes only a single linear layer for both long-term and short-term branches. We increase the complexity of the long-term branch by stacking linear layers with an activation function. However, as evident in R-Table 3 below, even when stacking layers in the long-term branch, performance tends to decrease or remain similar. This demonstrates that merely increasing the complexity of the long-term branch is not effective in the existing decomposition architecture.

R-Table 3: Increasing Complexity of Long-term Branch in DLinear in input-336-output-720 univariate setting

• In the case of the 2 layers, there exist linear layers along with the temporal axis only (i.e., the outputs shapes as follows: (B, I, C) > (B, D, C) > (B, O, C) where B is batch size, I is input length, D is hidden dimension, and O is output length)

W2:Lack of performance comparisons of different self-supervised objectives

Since we used self-supervised learning (SSL) loss in addition to the forecasting loss, it seems reasonable to compare our methodology with approaches utilizing different self-supervised objectives. We fully agree with this point and have already presented comparisons with three recent methodologies that employ various self-supervised losses in Section 4.3. However, to comprehensively address reviewer MDze's concern, we have designed and provided results for two possible self-supervised objectives based on HierCon [2] and SupCon [3] that can be incorporated into our two-stream model structure. HierCon induces the representations of two partially overlapped windows to be close to each other, while SupCon encourages the encoder to learn close representations for the windows with the same month label.

The two SSL objectives were tested with our model architecture, replacing only the AutoCon loss. As shown in R-Table 2 (we also added this in Appendix C.3), compared to the one without any SSL loss, HierCon shows a slight improvement in performance on the ETTh1 and ETTh2 for short-term predictions with a length of 96, but it performs worse in the long-term prediction, as it emphasizes only temporal closeness. SupCon leveraged monthly information beyond the window length, leading to performance improvements even in the long-term prediction. However, SupCon can only learn a single predefined periodicity, unlike AutoCon. Consequently, SupCon shows lower performance than AutoCon in learning the periodicities existing in the time series through autocorrelation. We additionally demonstrated the performance of well-known SSL baselines that are compatible with our model architecture to address the concerns of reviewers as much as possible. However, if there are specific SSL methods that reviewer MDze wishes to see, we would appreciate any suggestions or proposals from reviewer MDze.

R-Table 2: Comparison with different self-supervised objectives in our redesigned architecture. Another setting is the same to the experiments as shown in M-Table 1

W3:Experiment setting and results

• 1: 14 baselines, but seven are found
  In our study, we analyzed 14 distinct baseline models, which are distributed across different sections of our results due to the breadth of our analysis. Specifically, we evaluated seven models in the context of univariate forecasting, as detailed in M-Table 1. Additionally, four models were examined in the multivariate forecasting domain, as shown in M-Table 2, which includes models such as Crossformer and N-HITS that are specifically tailored to learning inter-variable dependencies. Furthermore, we included three time-series representation methods in our analysis, the results of which are presented in M-Table 4. We apologize for causing confusion by stating the aggregate number as 14 in the abstract. In the revised version, we added the clue ``in multiple experiments'' to the abstract to prevent confusion.

• 2: marginal performance improvement
  We understand that the reviewer's concerns about marginal performance improvement primarily stem from short-term prediction (Output-96) and the Weather dataset. However, our method contributes to capturing long-term variations beyond the window scope by leveraging global autocorrelation, thereby enhancing the performance of long-term predictions. Therefore, marginal performance improvement in the short-term setting is reasonably anticipated. In relation to the Weather dataset, we already elaborated in Dataset Analysis of Section 4.1 on how performance improvement varies based on the degree of long-term variation in the time series. In that analysis, we have already demonstrated that weather exhibits less long-term variation compared to other datasets, and we have explained that this could inevitably limit the performance improvement of our model. This analysis, rather, serves as significant evidence that our model is indeed operating as intended, validating our intuition that focused on the importance of long-term autocorrelation. In other words, it confirms that our model has behaved according to our intentions, affirming the accuracy of our emphasis on global autocorrelation. Most importantly, we believe stating "Performance improvement seems marginal" based on only a few numbers does not accurately capture the overall statistics. As shown in M-Table 1, our method achieved the top performance 42 times across all benchmarks. On average, it achieved a 12% reduction in MSE and a maximum reduction of 34% in MSE.

• 3: practical meaning of extending window
  Contemporary time-series communities continue to propose numerous models and methods aimed at predicting the more distant future, with a focus on improving long-term performance. For instance, Informer [4], in response to the real-world need for long-term forecasting, extended the forecasting length beyond the conventional 48 output length—primarily evaluated in M4 [5]; further expanded it to 720. To achieve both performance and computational efficiency in predicting a sequence of length 720, a huge number of models and methods have been proposed [6, 7, 8, 9, 10,]. At the same time, ReVIN [11] has successfully extended the prediction length up to 960, demonstrating that longer predictions become feasible when addressing and resolving the issue of distribution shift. Most recently, a new concurrent effort [12] has emerged with the aim of further increasing the output length up to 2880. Given the collective efforts of numerous researchers, extending our input to 2160 is not merely a setting to demonstrate superiority in performance but is rooted in genuine practical significance and necessity. If reviewer MDze, however, can present a concrete argument as to why all these efforts have little practical meaning, we are more than happy to further discuss this matter. Long-term prediction is our core research interest, and if there is any convincing counter-argument we were unaware of, we would like to take it into account in defining our future research.

W1: The forecasting module is a simple dual-head forecaster and already achieves good performance

In response to the reviewer's concern that "The forecasting module is a simple dual-head forecaster and already achieves good performance", we have already demonstrated through an ablation study that the performance decreases when AutoCon is not used as shown in M-Table 3 and (b) of M-Figure 6. Therefore, without our contribution, the AutoCon loss, the dual-head forecaster fails to achieve good performance in long-term forecasting.

And the reviewer claimed, "Capturing long-term variation via the contrastive loss part is separated from the forecasting mechanism", but our long-term encoder is jointly optimized by both forecasting and the AutoCon loss. This learning approach aligns with our initial proposal of the two-stream architecture, where the linear structure empowers short-term (motivated by recent linear models [1]), and the contrastive loss to capture the long-term variations. Empirically, considering the performance decline observed in the ablation studies when excluding the AutoCon loss, we find it unconvincing that reviewer MDze claims the contrastive loss operates independently from the forecasting mechanism.

Finally, our deliberate model design has indeed demonstrated superior long-term prediction performance compared to baseline models. Therefore, as desired by the reviewer, we can incorporate even more complex architectures based on these findings in future work.

Q1:Implementation details

• 1:Training protocols
  Our training protocol is identical to conventional training methods, except for the inclusion of AutoCon as an additional loss, in addition to the forecasting loss. To clarify this further, we also present the algorithm and added it to the Appendix A.1 of revision.

• 2:Weight hyperparameter of AutoCon
  We already have shown the analysis of the optimal values and sensitivity for the contrastive loss weight hyperparameter for each dataset. However, due to the page limit, we included it in M-Figure 9 of Appendix A.6. We would appreciate it if the reviewer could refer to the appendix once again.

References

[1] Are Transformers Effective for Time Series Forecasting?, AAAI'23
[2] TS2Vec: Towards Universal Representation of Time Series, AAAI'22
[3] Supervised Contrastive Learning, NeurIPS'20
[4] Informer: Beyond efficient transformer for long sequence time-series forecasting, AAAI'21
[5] The M4 Competition: Results, findings, conclusion and way forward, International Journal of Forecasting'18
[6] Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, NeurIPS'21
[7] FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting, ICML'22
[8] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis, ICLR'23
[9] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, ICLR'23
[10] MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting, ICLR'23
[11] Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift, ICLR22
[12] WITRAN: Water-wave Information Transmission and Recurrent Acceleration Network for Long-range Time Series Forecasting, NeurIPS'23

Q2:Additional M-Figure 6 results over other datasets

We thank reviewer eBaK for pointing out and addressing our oversight. M-Figure 6 presents the experimental results for the ETTh2 dataset. We explicitly identified it in the description of M-Figure 6 of our revised paper. Additionally, we provide results for ETTh1 (R-Figure3) and Electricity (R-Figure4), which show the long-term variations. Although there are some differences in magnitude, the overall trends are similar across the three datasets. We have also added these additional results to the Appendix B.2. We sincerely appreciate the reviewer`s constructive and helpful comments once again.

References

[1] Informer: Beyond efficient transformer for long sequence time-series forecasting, AAAI'21
[2] Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, NeurIPS'21
[3] FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting, ICML'22
[4] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis, ICLR'23

W1:Details of the encoder to understand representation similarity

We sincerely apologize for any ambiguity caused by an insufficient explanation. We mainly focused on experiments and visualization results to enhance understanding, which unfortunately led us to neglect the encoder details that should have been included for clarity. Following the reviewer eBaK's helpful guidance, we first clarify how the timestamp-based features are derived and incorporated into the encoder (Question 1).

• Q1: How are the timestamp-based features derived and incorporated into the encoder?
  The timestamp signifies the moment at which data is observed, and this moment is expressed in units of 'year,' 'month,' 'day,' and 'hour.' The timestamp provides a forecasting model with periodicity on a yearly, monthly, and daily basis as input, aiding the model in learning various business cycles. Most models use timestamps for long-term forecasting, commonly adding numerical values of periodic functions aligned with each unit's cycle from the timestamp as features to the input sequence. Specifically, if we use four units (i.e., month, day, weekday, and hour), we get a timestamp-based feature $f_{time} \in \mathbb{R}^{I \times 4}$, which is indexed by $\mathcal{T}$ and is aligned with input sequence $\mathcal{X} \in \mathbb{R}^{I \times c}$ where $I$ denote the input sequence length and $c$ is the number of variables. Then, with a projection layer $W_{time}$ and $b_{time}$, the time embedding $e_{time} = W_{time} \cdot f_{time} + b_{time} \in \mathbb{R}^{I \times d}$ is added to value embedding $e_{value} = W_{value}\cdot \mathcal{X}+b_{value}$. Finally, the temporal encoder (e.g., RNN, TCN, and Attention) takes the embedding sequence $e = e_{value} + e_{time} \in \mathbb{R}^{I \times d}$ to learn temporal dependency. This approach of utilizing the timestamp information was initially introduced in Informer [1], and our model, along with recent models [2, 3, 4], adopts the same approach.

Now, we clarify the rationale for our AutoCon objective in relation to this timestamp information. In the baselines used in M-Figure 2, TimesNet, FEDformer, and our model all obtained representations using these timestamps, incorporating them into the input sequences, while PatchTST does not utilize the timestamps. An important observation is that both TimesNet and FEDformer do not effectively capture the annual cyclic patterns, despite utilizing timestamps as the same as our model. These failures are noteworthy, particularly considering the Electricity time series, which displays a yearly-long periodicity. These results show that it is challenging for the model to learn yearly patterns even when given input sequences and timestamps, solely relying on the existing forecasting loss. Therefore, this result demonstrates the necessity of our AutoCon loss. Furthermore, to better demonstrate the importance of our loss in capturing long-term periodicities, we conducted an additional experiment where we compared two models: Ours with and without AutoCon loss. As shown in R-Figure1, compared to the one with AutoCon loss, the one without AutoCon loss demonstrates only a hint of periodicity (not too different from other baselines).

W2:Constant similarity of baseline models even with various patterns

As pointed out by reviewer eBaK, M-Figure 2 indeed seems to suggest that the baselines failed to learn even the short-term correlations, not to mention long-term correlations. This is actually due to the smoothing we applied to the representation similarity, in order to help the readers better understand that none of the baselines are able to capture long-term correlations. We apologize for any confusion caused by not explicitly mentioning this process and have denoted it in the caption of M-Figure 2 in the revision. We also provide original representation results, which are enlarged for each baseline, without smoothing out the short-term fluctuations as shown in R-Figure2. As anticipated by reviewer eBaK, the three baseline models are indeed learning short-term correlations within the window, although they are not learning long-term correlations. We hope that additional visualizations clarify the ambiguity caused by the smoothed representation similarity plots.

W3: Results on other evaluation metrics

We concur with reviewer pLfj's recommendation to incorporate additional evaluation metrics, as this will further substantiate the superior performance of our method in comparison to the baselines. While existing metrics (i.e., MSE and MAE) are standard metrics for long-term forecasting evaluation, they have limitations. Specifically, they may not adequately capture aspects such as the shape and temporal alignment of the time series, which are crucial for a comprehensive evaluation of a forecasting model's performance.

To address these limitations, we introduced two additional metrics based on Dynamic Time Warping (DTW) [1]: Shape DTW and Temporal DTW [2]. Shape DTW focuses on the similarity of the pattern or shape of the predicted sequence to the actual sequence, providing insight into the model's ability to capture the underlying pattern of the time series. Temporal DTW evaluates the alignment of the predicted sequence with the actual sequence, highlighting the model's accuracy in forecasting the timing of events.

These additional metrics offer a more nuanced assessment of our model's performance, particularly in areas where MSE and MAE may fall short. Lower values in both Shape DTW and Temporal DTW indicate better performance, signifying lesser distortion between the predicted and actual sequences. As shown in R-Table 1, our method demonstrates superior performance for three datasets, which show obvious long-term variations, not only in MSE and MAE but also in these shape and temporal alignment-focused metrics. Although it is difficult to measure Accuracy or AUROC due to the absence of labels in the forecasting task, our comprehensive evaluation of the newly used metrics underscores the superiority of our method. Also, these experiments were added to Appendix B.6 in the revised version of the paper.

R-Table 1: Univariate forecasting results, evaluated using Shape and Temporal DTW in a 720-output setting across three datasets

W4: Computational Cost

In Section 4.4, we have briefly provided an analysis of the computational costs for each model. However, due to space limits, detailed information can be found in Appendix B.4. We measured the computational cost of various baselines for four different output lengths, ranging from 96 to 2160. In summary, our model recorded significantly lower inference times compared to the baselines except for linear models in a single inference process. Additionally, it demonstrated much faster inference speed compared to CNN-based, TimesNet, and Transformer-based models even as the prediction length increased. We would sincerely appreciate it if the reviewer could refer to it once more.

Q1: Open-sourced code

We have already attached our code as supplementary material in the open review system. If this is not made public after the review process, we are also willing to publicly open our code on Github and include the GitHub repository URL in the main paper as either an abstract or a footnote.

References

[1] Dynamic programming algorithm optimization for spoken word recognition, IEEE transactions on acoustics, speech, and signal processing'1978
[2] Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models, NeurIPS'19

W1: Novelty of the work

We appreciate reviewer pLfj's thoughtful feedback regarding the novelty. We would like to respectfully clarify our perspective on this matter.

Our primary novelty lies in the introduction of global autocorrelation as a foundational element for long-term prediction. Using global autocorrelation in long-term forecasting, to the best of our knowledge, has not been previously explored. While we acknowledge the potential for future refinement and the introduction of more intricate mathematical models, we believe the core novelty of our work is in emphasizing the significance of autocorrelation-based contrastive learning, AutoCon, within the sliding window-based long-term forecasting.

To incorporate AutoCon, we have renovated a decomposition architecture that is suited for representation learning. While our model can be viewed as a combination of existing components, its architecture is specifically designed to emphasize the significance of the long-term branch. As we mentioned in Section 3.2, this aspect has often been overlooked by even complex existing models, yet our approach maintains simplicity while being effectively structured for long-term representation.

We understand that perspectives on novelty may vary, and we respect alternative opinions. However, it is our strong belief that the incorporation of autocorrelation-based contrastive learning into the long-term forecasting framework, as demonstrated in our work, constitutes a meaningful advancement in the field.

W2: Dataset Statistics and Extension to imbalanced data

• 1: Dataset Statistics
  For reviewer pLfj's valuable suggestions, we added M-Table 5 summarizing dataset statistics in Appendix A.3. The dataset statistics will provide readers with a quick reference to understand the diverse nature of the datasets used in our experiments.

• 2: Extension to imbalanced data
  Regarding reviewer pLfj's query on the performance of our methodology in imbalanced data scenarios, we appreciate the opportunity to clarify this aspect. As our focus is on time series forecasting, which is typically treated as a regression task, the concept of data imbalance manifests differently compared to classification tasks. In time series forecasting, data imbalance might refer to scenarios where certain patterns or trends are underrepresented. While our current study does not explicitly address this form of imbalance, our methodology's ability to learn long-term dependencies can be indirectly beneficial in scenarios where certain long-term trends and patterns are less frequent compared to repeated short-term fluctuations within the window. Specifically, AutoCon captures the overall tendency of the time series data across its entire length using the global autocorrelation. By considering the global structure of the data, it helps the model to identify and learn from long-term patterns, including those that may not occur frequently. Thus, we believe that in imbalanced scenarios, the model can recognize and adapt to these infrequent but potentially significant patterns, leading to more accurate forecasting. We acknowledge the importance of this aspect and will consider investigating this in future work to further enhance the robustness of our forecasting approach. Once again, we thank the reviewer for the constructive feedback.