Learning to Extrapolate and Adjust: Two-Stage Meta-Learning for Concept Drift in Online Time Series Forecasting

Thanks for your constructive suggestions! We address your concerns in the following.

{Q1} Issues about efficiency of LEAF and adjustment stage. Does adjusment for each sample need to load a checkpoint for every sample?

{R1} In LEAF, we do not require loading a checkpoint for each individual sample. Instead, we utilize a low-dimensional latent embedding to represent the model parameters. During the inference stage, the adjustment module generates sample-specific latent embeddings, which are subsequently decoded into specific model parameters. This process can be seamlessly integrated into the model's forward propagation and does not require the loading of a checkpoint for each individual sample. Furthermore, it is important to note that we do not generate the complete set of parameters for the entire model. Only a subset of parameters, typically from a few layers, is generated, thereby reducing the computational complexity of LEAF.

To verify our statements, we have added a comparison of the running time for LEAF, the baselines, and a variant of LEAF to investigate the efficiency of our model. The table below presents the running time for a single period while varying the number of samples in that period. One variant of LEAF, called EN, does not utilize the adjustment module. The results indicate the following findings:

(1) LEAF is not significantly more computationally expensive than Fine-tune, despite its superior performance compared to Fine-tune.

(2) Comparing LEAF with the EN variant, the adjustment for each individual sample is not a computational burden, as it is performed in the low-dimensional latent space.

(3) The cost time is sub-linear with respect to the number of samples, thanks to the parallel computing capability of GPUs.

Table 1: Running time comparison for a single period with varying number of samples.

{Q2} The framework is over-complicated.

R2 While LEAF integrates various ideas to address concept drift in online time series forecasting, its core motivation is straightforward. The key idea behind LEAF is to leverage an extrapolation module to predict the appropriate model parameters for future periods, taking into account the macro-drift. Additionally, LEAF learns a surrogate loss to adjust the model parameters for each individual sample, considering the micro-drift within each sample. The following procedures are performed in the low-dimensional latent space to reduce the computational complexity and alleviate overfitting issues.

{Q3} What is $L_{surrogate}$? How the loss is computed?

{R3} $L^{surrogate}$is parametrized by a neural netowrk $s$. In our implementation, $s$ is a MLP with 2 layers. The output of MLP is a positive real number which is treated as the "unsupervised loss" or "self-supervised loss" value. In traditional test time training setting, there is a specific self-supervised loss or unsupervised loss, so during inferencing the parameters can be adjusted according to the loss. Nevertheless, the identification of an appropriate self-supervised task or unsupervised loss function that effectively addresses the issue of concept drift in time series data remains a non-trivial challenge. The reason is that a specific test-time objective may not contribute to the prediction performance. As a result, we introduce a network to learn to generate surrogate loss for each individual sample. Specifically, during the meta-training phase, the surrogate loss is learned to capture and compensate for different types of deviations. The rationale is that micro-drift (drift within an individual sample) is often caused by external events, such as weather conditions or holidays. Although data affected by micro-drift may exhibit different patterns, they tend to deviate from their context in a similar manner. For example, the electricity consumption pattern during the summer typically differs from that in the winter, while during holidays it tends to decrease regardless of the season. During the meta-testing phase, the model utilizes the learned surrogate loss to adjust the latent embedding based on the specific characteristics of each individual sample. Consequently, we propose the learning of an unsupervised or self-supervised loss function via meta-learning as a strategy to manage micro-drift during test time.

Thanks again for you suggestions! We would like clarify that both the extrapolation and adjustment module in LEAF are significant.

We summarize the results of the ablation studies in the following tables. We have additionally added the ablation studies with TCN as target predicion model to further verify our claim .The variants include:

• Latent finetune: no extrapolation and adjustment module.
• EN: no adjustment module.
• A: no extrapolation module.
• EN + A: LEAF.

The results indicate the following findings:

• Comparing EN and A with Latent finetune indicates that both the extrapolation and adjustment module works.
• Comparing EN and A with EN + A indicates that using only extrapolation or the adjustment module alone does not perform as well as the complete LEAF framework. Only utilizing the adjustment module is not good enough.

Table 1: Ablation study results on Load-1 and ETTm1 with prediction models of DLinear.

Table 2: Ablation study results on Load-1 and ETTm1 with prediction models of PatchTST.

Table 3: Ablation study results on Load-1 and ETTm1 with prediction models of TCN.

Thanks for your constructive suggestions! We address your concerns in the following.

{Q1} What is the main difference between LEAF and online metric leaning methods?

{R1} LEAF introduces a novel and distinct contribution by addressing concept drift in online time series forecasting in two stages. LEAF differs from existing metric learning methods in several important ways. The primary goal of [1] is to learn a suitable latent feature space where samples within same class are close together compared to samples from different classes with a focus on noval class problem. [2] aims to mitigate the issue of confidence drift within a hybrid architecture comprising a static offline model and an online adaptive model. Conversely, the objective of LEAF is to capture the dynamic evolution of the distribution $p_t(X, Y)$ by compressing the distribution information into a low-dimensional latent embedding and subsequently modeling the evolution of the latent embedding. Additionally, the approaches of the first and second studies adhere to a conventional stream learning paradigm, wherein drift detection is followed by adaptation. However, our approach posits that concept drift may occur at any moment for any sample, and as such, the detection and adaptation of drift is implicitly integrated into the extrapolation and adjustment phase.

[1] Metric learning based framework for streaming classification with concept evolution. IJCNN 2019.

[2] Online metric learning for an adaptation to confidence drift. IJCNN 2016.

{Q2} Detailed implementation of network architecture.

{R2} Actually, in our implementation, the parameter decoder, relation network (R), embedding function (g), and loss network (s) are implemented as 2-layer MLPs. Our primary objective in this research is to address concept drift in online time series forecasting using a general framework. To achieve this, we have simplified the design of the network architecture to ensure that any improvements are solely achieved through the learning framework, rather than relying on a complex network structure. However, it is possible to substitute them with more sophisticated networks to achieve better performance in practical applications. Nevertheless, in this research work, our focus is solely on investigating the framework itself.

{Q3} Issues about computational efficiency.

R3 We have added a comparison of the running time for LEAF, the baselines, and a variant of LEAF to investigate the efficiency of our model. The table below presents the running time for a single period while varying the number of samples in that period. One variant of LEAF, called EN, does not utilize the adjustment module. The results indicate the following findings:

(1) LEAF is not significantly more computationally expensive than Fine-tune, despite its superior performance compared to Fine-tune.

(2) Comparing LEAF with the EN variant, the adjustment for each individual sample is not a computational burden, as it is performed in the low-dimensional latent space.

(3) The cost time is sub-linear with respect to the number of samples, thanks to the parallel computing capability of GPUs.

Table 1: Running time comparison for a single period with varying number of samples.

{Q3} Issues of The Naïve setting.

{R3} It is important to add a baseline that use warm-up and meta-train data as training set. Following your suggestion, we have added a Naïve baseline (namely Naïve+) by treat both warm-up and meta-train data as training data. The results are shown in the following table. LEAF still outperforms this baseline.

Table 4: Performance comparison with Naïve+.

{Q4} Issues about "model agnostic".

{R4} We do not generate all parameters of a model, and parameters of few layers are generated. Each kind of layer can be applied, not only linear layer. For example, when DLinear is used as the target model, we generate parameters of a linear layer. When PatchTST is used as the target model, We generate parameter of the last transformer block (including Q/K/V projection networks and a feed-forward network). To apply to different types of layers, we need to know the parameter shape of the network layer in advance, and we calculate the total number of the parameters need to be generated to determine the width of the decoder in LEAF. The generated parameters are reshaped to appropriate shapes and loaded into corresponding layers of the target prediction model.

Thanks for your constructive suggestions! We address your concerns in the following.

{Q1} Issues about model efficiency.

{R1} We have added a comparison of the running time for LEAF, the baselines, and a variant of LEAF to investigate the efficiency of our model. The table below presents the running time for a single period while varying the number of samples in that period. One variant of LEAF, called EN, does not utilize the adjustment module. The results indicate the following findings:

(1) LEAF is not significantly more computationally expensive than Fine-tune, despite its superior performance compared to Fine-tune.

(2) Comparing LEAF with the EN variant, the adjustment for each individual sample is not a computational burden, as it is performed in the low-dimensional latent space.

(3) The cost time is sub-linear with respect to the number of samples, thanks to the parallel computing capability of GPUs.

Table 1: Running time comparison for a single period with varying number of samples.

{Q2} More baselines including model retraining, Non-stationary Transformer and more target prediction model, e.g. TimesNet.

{R2} Thank you for your suggestion. In response to your feedback, we have incorporated the Retrain and Non-stationary Transformer baselines, as well as the target model TimesNet, into the experiments in the revised version. However, due to time constraints during the rebuttal phase, only the Load-1 and Ettm1 datasets were used. We will report the results on all six datasets in the final version of the paper. By including these baselines, our intention is to provide a comprehensive evaluation and comparative analysis of the performance and effectiveness of our proposed approach. The results are represented in the following tables. Table 2 verifies that LEAF outperforms retraining baseline and Non-stationary Transformer. Table 3 verifies that LEAF works better than other baselines on the state-of-the-art CNN-based prediction model, TimesNet.

Table 2: Performance comparison with Retrain and Non-stationary Transformer. Retrain uses all historical samples as training data in each period. Non-stationary Transformer is a transformer model trained on a warm-up and meta-train set and tested on a meta-test set.

Table 3: Performance comparison with TimesNet as target prediction model.

Thanks again for you suggestions!

We have uploaded the revised version of our paper. The modified and newly added parts are indicated with blue highlights. The major modifications made include:

(1) We have included the numeric results of the Retrain and Naive+ baselines. This addition provides a comprehensive evaluation of the effectiveness and advantages of our proposed method. Furthermore, we clarify that the original Naive baseline is not incorrect. In real-world scenarios, the warm-up set represents historical data, based on which we train the initial model. The meta-train and meta-test sets, on the other hand, represent streaming data. The Naive baseline is the most naive strategy, implying that the model will no longer be updated during data stream comes. However, we acknowledge the importance of Naive+ baseline.

(2) We have conducted ablation studies on a new model variant called "A" that solely utilizes the adjustment module. The results indicate that the adjustment module works; however, using only extrapolation or the adjustment module alone does not perform as well as the complete LEAF framework.

(3) We have added remarks about the "model agnostic" claim in Section 3.2.3. This addition helps readers better understand how to apply LEAF to different types of prediction models.

Additionally, Table 3 in our previous response confirms that LEAF outperforms other baselines on the state-of-the-art CNN-based prediction model, TimesNet, using three datasets: Etth2, Load-1, and ECL, each with different properties. Since we haven't finnished TimesNet experiments on all six datasets due to limmited time, we promise that we will conduct these experiments on all six datasets and present the results in the final version of the paper.

Thanks for your constructive suggestions! We address your concerns in the following.

{Q1} Literature review of surrogate gradient.

{R1} Althogh exsisting works have studied meta-learning explicit [1] and implicit gradients [2] for few-shot learning. LEAF introduces a novel and distinct contribution by introducing a meta-learned surrogate loss for sample-specific adjustment. The surrogate loss in LEAF differs from previous methods in several important ways. Firstly, the surrogate loss in LEAF is sample-wise, allowing it to leverage information within each individual sample. This enables the model to adapt and update itself for each sample, capturing micro-drift that may occur over time. Furthermore, the surrogate loss in LEAF is specifically designed to address the challenges of online time series scenarios. It takes into consideration comprehensive information about the test sample itself, as well as its contextual and relational information with the training data. This design ensures that the surrogate loss effectively incorporates relevant information to capture and handle concept drift in the time series data. In summary, LEAF stands out from existing works by introducing a sample-wise surrogate loss that is carefully designed for online time series scenarios, effectively leveraging information within each individual sample and considering its context and relationship to the training data.

[1] Semi-Supervised Learning with Meta-Gradient. AISTATS 2021.

[2] Meta-Learning with Implicit Gradients. NeurIPS 2019.

{Q2} Issues about visual resolution.

{R2} We apologize for any confusion caused. We have used vector diagrams for all our figures. We kindly recommend using a recent PDF reader, such as the Adobe Acrobat Reader DC version 2018.009.20044 that we utilized, for optimal viewing and clarity.

{Q3} The MAPE metric was missing of the time series prediction study.

{R3} In the experiments, we calculate metrics based on normalized data. However, MAPE is not suitable for normalized data, as many data points have values of zero or close to zero which will cause anomalies in MAPE. Besides, many works [1-4] of time series forecasting do not consider this metric because of this reason.

[1] Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. AAAI 2021.

[2] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. NeurIPS 2021.

[3] PatchTST: A Time Series is Worth 64 Words: Long-term Forecasting with Transformers.. ICLR 2023.

[4] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. ICLR 2023.

{Q4} Financial futures datasets (such as NASDAQ 100 dataset).

{R4} Following your suggestion, We have added experiments to evaluate LEAF compared with other baselines on NASDAQ 100 dataset. The results are in the following table, which demonstrate that LEAF outperforms other baselines with different target prediction model in such a stock trading scenario with frequent abrupt drifts.

Table 1: The results on NASDAQ100-small dataset. The look back window is 96, the forecast horizon is 24, and the model update interval is 672.

Thanks for your constructive suggestions! We address your concerns in the following.

{Q1} Issues about theoretical foundation.

{R1} Our proposed method leverages the well-established theories of model-based and optimization-based meta-learning. These theories have been extensively researched and provide a solid foundation for our approach. By deeply adapting and applying these approaches, we aim to address the challenge of concept drift in online time series forecasting effectively. In order to ensure clarity and readability for the readers, we have omitted the detailed theoretical explanations of these methods and instead focused on presenting a concise and clear explanation of our motivation and approach.

{Q2} More ablation studies are needed.

{R2} In the ablation studies conducted in our paper, we examined the performance of the model as we gradually introduced the designed modules, namely latent optimization, extrapolation module, and adjustment module. As we did not elaborate on the individual performance of the adjustment module, we supplement an additional ablation study to investigate its effectiveness. The results are presented in the following table. It is evident that the adjustment module plays a crucial role in enhancing the forecasting performance by capturing the micro-drift. However, it falls short of surpassing the complete version of LEAF, which emphasizes the significance of the extrapolation module. These findings underscore the importance of both modules in achieving improved forecasting accuracy.

Table 1: Ablation study results on Load-1 and ETTm1 with prediction models of DLinear. "A" is the newly added vairant that only utilize the adjustment module.

Table 2: Ablation study results on Load-1 and ETTm1 with prediction models of PatchTST. "A" is the newly added vairant that only utilize the adjustment module.

Thanks for your constructive suggestions! We address your concerns in the following.

{Q1} The novelty of using an LSTM to model the evolution of latent space, e.g., DeepState; explore more aggressive approaches for extrapolation than only using a vanilla LSTM for predicting the evolution of parameters

{R1} While there exist various works, such as DeepState, that model the evolution of time series with hidden states, LEAF takes an innovative approach by representing the model parameters with evolving hidden states. This addresses the challenge of the model changing as data experiences concept drift. Our primary objective in this research is to address concept drift in online time series forecasting using a general framework. To achieve this, we have simplified the design of the network architecture to ensure that any improvements are solely achieved through the learning framework, rather than relying on a complex network structure. Similarly, other networks in LEAF, such as the loss network $s(\cdot)$, embedding network $g(\cdot)$ and relation network ${R}(\cdot)$, are also kept simple, with just two layers of Multilayer Perceptron (MLP). However, it is possible to substitute them with more sophisticated networks to achieve better performance in practical applications. Nevertheless, in this research work, our focus is solely on investigating the framework itself.

{Q2} Detailed description of the relation network (R) and embedding function (g) and the alternative ways to implement them and why proposed implementations work the best should also be addressed; Overfitting problem of off-the-shelf MLP.

{R2} Actually, in our implementation, both the relation network (R) and embedding function (g) are implemented as 2-layer MLPs. The reason for this design choice is explained in {R1}. Adjusting the model for each individual sample may potentially lead to overfitting problems. To mitigate this issue, we have introduced several strategies. Firstly, we optimize the sample-specific model in a low-dimensional latent space, which limits the degree of freedom for adjusting the model. This can be seen as an advanced and more general version of adjusting the last linear layer with a few parameters for each individual sample [1]. Additionally, we only perform one iteration of SGD with a learning rate of 1e-3 to adjust the sample-specific latent embeddings. As a result, the adjusted embedding will not deviate significantly from the base embedding, thereby avoiding overfitting.

[1] Gerald Woo, et al. Learning Deep Time-index Models for Time Series Forecasting. ICML 2023.

{Q3} Which data is shown in Fig.1(a)?

{R3} The data shown in Fig.1(a) is actually synthetic data. We chose to use synthetic data because when illustrating the macro-drift of real-world data over a significant period of time, the micro-drift that occurs over a shorter duration may not be easily discernible. However, we acknowledge that using real-world data would be more representative and plan to replace the synthetic data with real data in the final version of our work.

{Q4} Are there any external features and time-related features used in the models? If so, how does their presence affect the overall meta-learning process?

{R3} We have introduced the time-related features (e.g., hour of day, day of week, and month of year) in LEAF and other baselines. Besides, these covariates are also used in the target prediction model. This step, which is a standard practice in time series forecasting, has been omitted in the paper for brevity. These time-related features are embedded with embedding layers which are then added to the time series representations. These features provide information of global time stamp, helping the model understand the overall evolving pattern of time series.

Dear Reviewer 9YY8,

Since the End of author/reviewer rebuttal is approaching, may we know if our response addresses your main concerns? If so, we kindly ask for your reconsideration of the score. Should you have any further advice on the paper and/or our rebuttal, please let us know and we will be more than happy to engage in more discussion and paper improvements.

Thank you so much for devoting time to improving our work!