Navigating Concept Drift and Temporal Shift: Distribution Shift Generalized Time-Series Forecasting

Response to Literature:

We believe we have covered most state-of-the-art literature in both time-series forecasting and distribution shift research, including recent works such as iTransformer (ICLR 2024) and FOIL (ICML 2024). Fedformer is already discussed in the current version, and we are willing to including FRNet in the discussion for completeness.

Response to Novelty:

First and most importantly, this work aims to show how to mitigate concept drift (SAM) and how to integrate concept drift method together with temporal shift (ShifTS). Finding better temporal shift methods is not the main scope of this work. We choose to use vanilla RevIN in ShifTS as it is simple yet effective. We admit there are advanced temporal shift methods, such as SAN or NS-Tansformer. However, these methods require either forecasting model modification or pre-training.

Second, we want to clarify that instance normalization is necessary for ShifTS. To mitigate concept drift where the conditional distribution $P(Y^H|X^L)$ is unstable, we need to first mitigate temporal shift and ensure the marginal distribution is stable, and the most effective way is by doing instance normalization, such that all marginal distributions ~P(0,1).

We have made a modification to Section 4.3 for a more clear presentation.

Response to Experiment:

(Also to Q3) We showcased an example of concept drift and temporal shift in Figures 4, 5, and 6(a), in Appendix A. Also, the selected datasets are known for irregular patterns and have been used for previous distribution shift studies [1].

(Also to Q2) The proposed ShifTS is model agnostic. Therefore, we compare two types of baselines: First, we show that with ShifTS, existing time-series forecasting models, such as Informer, can gain consistent improvements, as shown in Table 1. Second, we show ShifTS better handles distribution shift issues compared to distribution shift methods, including concept drift methods for general purposes (e.g., classification) such as IRM temporal shift methods for time series, such as RevIN, and combined methods, as shown in Table 2.

For forecasting models, we are using up to iTransformer and TimeMixer of ICLR 2024. For distribution shift methods, we are using up to FOIL of ICML 2024. We believe the baselines we used are new enough, we would appreciate it if the reviewer could point out the wanted baselines.

Response to Conclusion:

Thanks for pointing this out. We have added a limitation discussion in Appendix D in the revised version.

Response to Q1:

$d_{x}$ and $d_{y}$ refer to the dimension of exogenous features and target features.

[1] Time-Series Forecasting for Out-of-Distribution Generalization Using Invariant Learning. ICML 2024

Response to W1:

Thanks for pointing this out.

First and most importantly, this work aims to show how to mitigate concept drift (SAM) and how to integrate concept drift method together with temporal shift (ShifTS). Finding better temporal shift methods is not the main scope of this work. We choose to use vanilla RevIN in ShifTS as it is simple yet effective. We admit there are advanced temporal shift methods, such as SAN or NS-Tansformer. However, these methods require either forecasting model modification or pre-training.

Second, we want to clarify that instance normalization is necessary for ShifTS. To mitigate concept drift where the conditional distribution $P(Y^H|X^L)$ is unstable, we need to first mitigate temporal shift and ensure the marginal distribution is stable, and the most effective way is by doing instance normalization, such that all marginal distributions ~P(0,1).

We agree with the reviewer’s comment on Section 4.3. We have modified Section 4.3 with the aforementioned intuitions for a clearer presentation.

Response to W2:

First, while we agree that forecasting the whole horizon window of the exogenous features can be as hard as the original problem, this work does not aim to forecast $X^H$ using $X^L$. Instead, it uses $X^L$ to forecast $X^{SUR}$, which is a simpler task as follows: First, forecasting $X^{SUR}$ does not need to estimate the values of the whole horizon window, but only some key points showing invariant patterns (as shown by the purple circles in the add-on case study at:https://anonymous.4open.science/r/shifts_iclr-56A0/case_study.pdf), this is generally easier; Second, invariant patterns can exist in both look back and horizon windows, with similar representations, finding these invariant patterns and using representations in the loopback window to forecast similar representations in the horizon is easier. This misunderstanding may come from an unclear presentation in Figure 1, where we have uploaded a clearer version to avoid this misinformation.

Second (Also to Q4), we want to clarify two points in the reviewer’s comment: (1) Concept Drift vs. Temporal Shift: It seems there may be some confusion between the definitions of concept drift and temporal shift. In this work, concept drift specifically refers to changes in the correlation between exogenous features and targets. Therefore, we only consider temporal shifts within a single exogenous feature. (2) Temporal Shift in the Dataset: The dataset used in this study does exhibit temporal shift, as illustrated in Figures 4 and 6(a). Datasets such as ETT are known for their irregular patterns, which effectively demonstrate the effectiveness of our methods in handling distribution shifts. Furthermore, the use of these datasets is consistent with prior studies on distribution shift [1].

Third, we are willing to add synthetic experiments with controlled concept drift and temporal shift. The results still show the benefits of ShifTS. Detailed setup and results are shown at: https://anonymous.4open.science/r/shifts_iclr-56A0/synthetic_exp.pdf

Response to W3 & Q1:

We run the experiments on three different random seeds and take the average, following a typical experiment and report manner as common in previous studies. For Table 2, we additionally average the forecasting results across four horizon windows. We believe the results are statistically meaningful.

Additionally, we are able to show the results of Table 2 before averaging at: https://anonymous.4open.science/r/shifts_iclr-56A0/expand_table2.pdf. The results and benefits of ShifTS are consistent with the averaged results in Table 2.

Response to W4:

The motivation for sparsity regularization corresponds to lines 202~205. We believe higher weights refer to the invariant patterns, while the lower weights refer to the variant patterns. We use sparsity regularization to filter the variant patterns whose correlations to target are more likely to change over time.

Response to W5:

Figure 3 aims to indicate when to use ShifTS, and when ShifTS is more beneficial. We believe the p-value test is strong enough to show a positive correlation between MI and performance gains. Maybe the reviewer can point out other statistical tests that are wanted.

Response to Weakness:

We respectfully disagree with the comments from the reviewer regarding the novelty.

First, as the reviewer him/herself mentioned in the weakness, the proposed framework is interesting and is understandable for why it works well, we believe this work’s idea is intuitive, simple, and effective.

Second, while general concept drift problems are widely studied, studying concept drift issues in time-series forecasting is largely overlooked with existing methods showing certain limitations, as mentioned in our paper lines 54~59.

Response to Question:

Thanks for pointing out the definition questions. We wanted to clarify - there is just some confusion in the terminology here. Gama et al paper does use Equation 2 as a definition of ‘concept drift’. However, this is too general.

Indeed in the same paper (Gama et al) immediately after Equation [2], they elaborate that there are two types of concept drift:

(1) “Real concept drift refers to changes in p(y|X)”, which we refer to simply as ‘concept drift’ in our work - this terminology is also used in e.g. in [2]; and

(2) “Virtual drift happens if the distribution of the incoming data changes without affecting p(y|X)” (change in p(X)), which we refer to as ‘temporal shift’ in our work, and this terminology is also used in [3].

Hence this is just a matter of using slightly different terminology than Gama et al. - we have arguably used terms in our paper that are more popular in the time-series domain [2, 3]. Therefore, we respectfully disagree with the reviewer’s comment that the definitions in our work are not rigorous. Also, to meet the reviewer's expectations, we have used a more rigorous definition in the updated version.

[1] A survey on concept drift adaptation

[2] FEDD: Feature Extraction for Explicit Concept Drift Detection in Time Series

[3] AdaRNN: Adaptive Learning and Forecasting of Time Series

Response to W1:

We would like to provide the intuition of ShifTS as follows: Given our assumption and the fact that both marginal and conditional distributions can be unstable, in the context of time-series forecasting, we first do normalization and denormalization to ensure both marginal distributions follow 0-1 distribution, e.g. both $P(Y^H)$ and $P(Y^L)$ or $P(X^L)$ follow standard distribution after normalization. This intuition is given in the updated Section 4.3. With this process, we assume the marginal distribution becomes invariant and then learn the stable conditional distribution through SAM. In SAM, we search and aggregate over both the lookback and horizon window to find the invariant patterns over time ($X^{SUR}$). For the patterns in the horizon window ($X^H$), we use conventional time-series forecasting models to forecast them using $X^L$, and then use these invariant patterns to learn accurate targets.

Response to W2 and Q1:

First, we clarify that the datasets used in this study are inherently irregular and noisy, as illustrated in Figures 4, 5, and 6(a). These datasets have also been utilized in previous studies on distribution shifts [1].

Second, to address the concern about the effectiveness of ShiftTS on noisy data, we are willing to include additional experiments conducted on synthetic datasets with additive noise. The results consistently demonstrate the advantages of ShiftTS. The detailed experimental setup and results are provided at: https://anonymous.4open.science/r/shifts_iclr-56A0/synthetic_exp.pdf

Response to Q2:

Thank you for pointing out the real-time forecasting setup. Sure, ShiftTS is applicable to online settings. To demonstrate this, we provide an initial evaluation of the ETTh1 and ETTh2 datasets, using a lookback window of 96, a horizon of 96, and retraining every 96 timesteps (with training data increasing accordingly). The results presented in the following table still show the benefits of ShiftTS in this context.

Response to W1:

Thanks for pointing this out and sorry it is a typo. It should be $P(Y^H)=P(Y^H|Y^L)P(Y^L)+P(Y^H|X^L)P(X^L)$, then the equation follows a very fundamental statistical law (law of total probability, https://en.wikipedia.org/wiki/Law_of_total_probability). This formulation also follows Equation 1 in [1] and definitions in [2]. We correct this on the updated version.

Response to W2:

We would like to expand the caption of Figure 2 (Please refer to the updated version) and clarify the process as follows: First, ShifTS consists of normalization (a) in the beginning and denormalization (c) at the end to mitigate the temporal shift as RevIN [3] does. Second, for mitigating concept drift part, ShifT processes two-stage forecasting as (b): It first forecasts the surrogate exogenous features, determined by the SAM, which are stable and invariant patterns towards forecasting the target. It then uses both forecasted surrogate exogenous features and original $Y^L$ to predict $Y^H$.

The module ‘Agg’ is a simple MLP that aggregates weighing the exogenous features to predict the target, as also explained in the updated version.

Response to W3:

First, $M$ is initialized uniformly. During training, $M$ is utilized to compute the softmax weights, and features with higher scores are selected as invariant patterns. $M$ is updated through standard backpropagation. The implementation code is provided.

Second, to clarify, we do not state that $M$ is independent of $Y^L$ and $X^L$. Instead, we say ‘the mask high attention weights are recognized as invariant patterns, which remain unchanged during test time steps’, which means we use invariant correlations between $X$ and $Y$ to make accurate predictions during the test.

Response to W4:

Following the clarification of W2, given our assumption that both marginal and conditional distributions can be unstable in time-series forecasting, we first do normalization and denormalization to ensure both marginal distributions follow 0-1 distribution, e.g. both $P(Y^H)$ and $P(Y^L)$ or $P(X^L)$ follow standard distribution after normalization. With this process, we assume the marginal distribution becomes stable and then learn the invariant conditional distribution through SAM. We give detailed and deeper intuitions in the updated Section 4.3.

Response to W5:

First, we provide an intuition for how $M$ effectively mitigates concept drift. $M$ is trained to select features across lookback and horizon windows. Features assigned higher weights by $M$, when averaged over the training data, are likely to represent invariant patterns. These invariant patterns are used to construct the surrogate feature $X^{SUR}$, which is less likely to suffer from concept drift during testing.

Second, we are willing to provide a case study to further demonstrate the features selected by $M$ tend to be invariant between training and testing. The case study is shown at: https://anonymous.4open.science/r/shifts_iclr-56A0/case_study.pdf

[1] A Survey on Concept Drift Adaptation

[2] Adarnn: Adaptive learning and forecasting of time series

[3] Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift