Retrieval Augmented Time Series Forecasting

6. Concerns on performance comparison with win matrix

Thank you for pointing out this issue. We agree with the reviewer’s concern and have moved the average result table (which was in the Appendix of the previous manuscript) to the main manuscript to provide a clearer comparison of forecasting performance across models. Table 1 in the revised manuscript presents the average MSE performance evaluated across different forecasting horizon lengths. We observe that our model consistently outperforms other contemporary baselines, highlighting the effectiveness of retrieval in time-series forecasting.

7. Comparison with PatchTST

Our work has the same setup and similar set of datasets as in the PatchTST paper. As responded in the previous question, we experimented with all baselines in the same environmental setting, including the learning rate and batch size, and they may be different from the original paper. Thus, we re-evaluated all models again with their optimal parameters searched over the validation set and reported the updated results in the experiment section (Table 1).

8. The meaning of inductive bias and Its relationship with retrieval

By incorporating retrieval to provide relevant inputs, we eliminated the need for the forecasting model to learn every short-term pattern during inference. We believe this process introduces a bias or assumption—an inductive bias—that helps prevent overfitting by constraining the model's parameter space search. This bias is particularly beneficial in scenarios with short-term patterns, where temporal correlation is minimal, such as random walk, and the model would otherwise need to memorize all changes for accurate predictions. Instead, retrieval allows the model to reference relevant information, freeing up learning capacity to focus on other features. This is empirically demonstrated through our analyses over the synthetic datasets in Section 5.2-5.3.

Thank you for your thoughtful comments, which are valuable for enhancing the quality of our paper. Below, we address your concerns in detail.

1. Difference between RAFT and transformer variants

Thank you for raising this question. Regarding the conceptual difference between transformer variants and ours, transformers learn relationships within a fixed lookback window through attention mechanisms. Even though increasing the lookback window size of a Transformer allows for the use of more past information, using long sequences as input has significant limitations, including a reduction in available data points and a substantial increase in task and computational complexity. In contrast, our model extends beyond the lookback window by retrieving relevant data points from the entire time-series and incorporating them into the input. This also allows for efficient reference to more past information without increasing the input length, distinguishing it from traditional transformer variants. We also showed that our model outperforms other transformer variants across various datasets (see Figure 4). We have added this discussion to the revised manuscript (Line 161).

2. Comparison of related work on few-shot forecasting

Thank you for your recommendation. As noted in our general response, our work assumes a single time-series possibly with multiple channels. We propose performing retrieval directly on the training data of a single time-series for forecasting. The model suggested by the reviewer assumes the existence of multiple types of time-series datasets and retrieves additional information from external data to improve the prediction of the target time-series. These approaches differ in experimental setup, underlying motivation, and the effects achieved through retrieval. For instance, our model aims to reduce the learning burden through training data retrieval, enabling the discovery of more generalizable features in a supervised setting. In contrast, the suggested literature focuses on improving prediction performance in label-scarce scenarios by leveraging external data. Note that a more recent work in a similar setting to the suggested model [1] is already discussed in our related work section (Line 140). As suggested, we have added the recommended literature alongside [1] in the related work section (Line 142).

[1] Mq-retcnn: Multi-horizon time series forecasting with retrieval-augmentation (2022)

3. Comparison of related work on pretrained/foundational time series models

Since our model performs forecasting on a given time-series in a supervised setting, we did not include comparisons with large foundation models specialized for zero-shot or few-shot scenarios. However, we agree with the reviewer that these models have brought advancements to the time-series forecasting field, and we have added a discussion about them in the related work section (Lines 115-117). The key contribution of our paper is demonstrating that retrieval from the training dataset reduces the learning burden during forecasting. We would like to clarify that this is orthogonal to approaches that retrieve additional information from external datasets, as our focus is on leveraging the training data itself for generalizable training.

4. Limited datasets

Thank you for the comment. We conducted our experiments following the evaluation settings and benchmark datasets used in the latest supervised time-series forecasting studies [2,3,4]. To further demonstrate our model's performance, as suggested, we additionally evaluated it on two benchmark datasets discussed in [2,3,4] and the recommended papers that were not included in our original paper. Accordingly, we updated our results in the experiments section. Our findings confirm that our model continues to exhibit strong performance on these newly added datasets.

[2] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR 2024)
[3] SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters (ICML 2024)
[4] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (ICLR 2023)

5. The choice and setup of the baselines

Thank you for raising this critical point to help improve our paper. We also observed that baselines often vary in performance and settings across different papers. Instead of tailoring settings for each dataset and baseline, we aimed to ensure a fair comparison by evaluating all methods in the same environment. However, we agree with your perspective and have conducted an additional evaluation by comparing our model's best results against the best results reported in the original papers for the baselines, including PatchTST. We updated the results accordingly in the experiment section (Table 1). In most settings, we confirmed that our model outperforms PatchTST.

I would like to thank the authors for their updated paper. I will respond to the individual points below and only touch on points that require further discussion.

1. Difference between RAFT and transformer variants I still have the concern is that the proposed method is conceptually similar to transformer variants (and CVZn shares this concern). A transformer uses learned attention weights to attend to learned patterns (or even memorized examples) using the lockback window is a query. Through this mechanism, a transformer would attend to the training set, and maybe even to an individual time series. I agree with the authors that the lookback window is an important hyperparameter to tune but I do not agree the with the argument that it would require the increase of the lockback window to have give the transformer the ability to attend on the training set, but rather having enough capacity for the model to retrieve learned patterns at inference time (encoded in the learned weights). Thus, I keep this concern.

2. Comparison of related work on few-shot forecasting I understand that this is a different setup (several datasets vs single time series). Still, the ideas are conceptually similar, even if applied to a different setting. This needs to be clearly highlighted in the paper.

4. Limited datasets Thank you for including additional datasets. I agree that these datasets have been used in many several papers in the field, but I would still highlight that recent work raised the bar by evaluating on much more and larger datasets, which provides more empirical evidence to model performance. Compared to these recent studies, the number of datasets/observations evaluated in this study are still limited.

5. The choice and setup of the baselines Thank you for revising the approach of tuning baselines and including additional ones. I have looked into the results for PatchTST specifically. It appears that the results in the updated manuscript deviate from the original work (https://arxiv.org/abs/2211.14730). For example, for ETTh1 and ETTh2, the averaged MSE for PatchTST is 0.413 (ETTh1) and 0.330 (ETTh2) in the original paper but 0.516 (ETTh1) and 0.391 (ETTh2) in this work. The authors mention that they are " comparing our model's best results against the best results reported in the original papers for the baselines, including PatchTST". From what I understand, the PatchTST results here stem from the tuning of learning rate/batch size that the authors have employed. Thus, the performance improvements that RAFT shows over PatchTST are only valid for the specific setup in this work. I would still argue that the best performing setting identified in earlier work should be the appropriate reference point for comparing baselines.

I would like to thank again the authors for their revision. I will maintain my score because the points mentioned above are still not sufficiently addressed.

Thank you for your follow-up questions. We further provide the responses to your questions below:

1. Difference between RAFT and transformer variants

We want to clarify that Transformers can only apply attention to time-series frames within the lookback window provided as input. In other words, the retrieval set they use to best explain the given input is restricted to the lookback window and does not encompass the entire training set (i.e., query, key, value can be only chosen from the lookback window.) As a result, Transformers cannot directly attend to the entire training set during inference; their attention mechanism is limited to patterns within the provided lookback window. In contrast, our model extends beyond the lookback window by traversing the training set to include additional relevant frames, thereby broadening the scope of attention. The reason we referenced lookback window size in the previous response is that for a Transformer to mimic our model’s ability to attend to the entire training set, its lookback window would need to be set to the same length as the training set, which we mentioned as an illustrative example.

2. Comparison of related work on few-shot forecasting

We have included a discussion about related works utilizing the retrieval concept for few-shot forecasting, which is highlighted in Lines 140–142 of the revised manuscript.

4. Limited datasets

Thank you for your comment. We acknowledge that some recent works on developing the foundation model for time-series data have evaluated their models on a wide range of datasets, although our approach aligns with the conventions commonly followed in the mainstream time series forecasting domain [1,2,3,4,5,6,7]. Unfortunately, it is not feasible for us to complete evaluations on new datasets within the discussion period due to time constraints. However, we will include evaluations on additional datasets in the camera-ready version of our paper.

[1] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR 2024)
[2] SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters (ICML 2024)
[3] PatchTST: A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (ICLR 2023)
[4] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (ICLR 2023)
[5] MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (ICLR 2023)
[6] Are Transformers Effective for Time Series Forecasting? (AAAI 2022)
[7] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (ICML 2022)

5. The choice and setup of the baselines

We excerpted the baseline results from the original TimeMixer paper. According to the TimeMixer paper, a unified hyper-parameter search setting was applied for each baseline, and we followed the same approach for tuning in our work. The difference in reported performance is likely due to the broader parameter search range used for PatchTST in their experiments. We will run experiments within the same search space as PatchTST and include the results in the revised manuscript if possible within the limited time available during the discussion period. Thank you for pointing this out.

7. Comparison with other retrieval-based methods.

We want to emphasize that our method is the first to utilize a retrieval process within the training dataset to reduce the learning burden of the forecasting model, as clarified in the general response. The other retrieval-based algorithms referenced in our related work aim to improve inference performance by transferring knowledge across different types of time-series data within a dataset containing diverse time-series. Specifically, in [4] from our related work, an additional process is employed to identify relevant time-series among various time-series, and only frames from other time-series data with the same timestamp as input are used as the target for retrieval. This differs from our setting, which focuses on forecasting for a single time-series and retrieves patches with different timestamps from the input. Therefore, we did not conduct direct performance comparisons but instead discussed these methods in the related work section.

8. Visualization of retrieved patterns.

Our Appendix Figures 8–10 provide visualizations of the retrieved patterns and their corresponding future values. We included examples from both cases where our model performs well (e.g., ETTh1, Traffic) and cases where it performs less effectively (e.g., Exchange Rate) under the univariate setting. If there are any specific dataset and setting combinations that should be visualized, please let us know, and we would be happy to provide additional results.

Thank you for your thoughtful reviews and for recognizing the technical novelty of our proposed retrieval approach. Below, we address your concerns in detail.

1. Choice of similarity measure.

Thank you for this important question. We provided an analysis of different similarity measure choices in Appendix C, including learnable similarity functions. Among the candidates, Pearson’s correlation demonstrated the best performance (see Table 5). We did not include cross-correlation or dynamic time warping (DTW) which consider offset differences between two segments, as our sliding window approach inherently addresses the offset differences.

2. Computational efficiency.

As clarified in the general response, we assume a single time-series possibly with multiple channels, and the retrieval process is restricted to the training set of the same time-series. Thus, in our setting, the length of the time-series matters, not the size of the dataset. Our model incorporates a retrieval process to find similar patches in the given data. For efficient training, the retrieval process is pre-computed for the training and validation data, requiring computation only once during training. The pre-computation speed for retrieval of our model is O($N^2$), where N denotes the length of the time-series in the training data. To reduce this retrieval time, one approach is to increase the stride of the sliding window beyond 1, speeding up the search process. We confirmed the trade-off between the stride of the sliding window and performance in Appendix E. Additionally, methods such as clustering time-series segments and using centroids for search or applying tree-based techniques and approximate nearest neighbor (ANN) algorithms can also be considered. Thank you for the suggestions!

3. Sensitivity to hyperparameters.

For the set of periods ($P$), we followed the approach proposed in the TimeMixer paper ($P=\{1,2,4\}$), which also introduced the concept of multi-periodicity. We determined other hyper-parameters through grid search on a held-out validation set with the data split into train, validation, and test sets chronologically in a 7:2:1 ratio. The range of hyper-parameters used for grid search is now provided in Appendix B. As suggested, we added analysis on the number of retrievals ($m$) and the temperature ($\tau$) in Appendix C.

4. Limited ablation studies.

As suggested, we conducted an additional ablation study (see Appendix C.2). We observed that each component in our retrieval module contributes to performance improvements.

5. Evaluation on more diverse datasets.

We conducted our experiments following the evaluation settings and benchmark datasets used in the latest supervised time-series forecasting studies [1,2,3]. To further demonstrate our model's performance, as suggested, we additionally evaluated it on two benchmark datasets discussed in [1,2,3] that were not included in our original paper. Accordingly, we updated our results in the experiments section. Our findings confirm that our model continues to exhibit strong performance on these newly added datasets.

[1] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR 2024)
[2] SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters (ICML 2024)
[3] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (ICLR 2023)

6. Handling multiple retrieved patterns.

Since retrieval is a widely used method across various domains, its design can take various forms. To demonstrate the effectiveness of retrieving relevant frames from the training set in time-series data, we employed the simplest form of retrieval. As you expected, there is still room for improvement in retrieval techniques specifically tailored for time-series data, including exploring when, where, and how to apply retrieval. We discussed this interesting direction for future work in the conclusion section.

Thank you for your thoughtful comments. Below, we address your concerns.

1. Clarification on the retrieval process

Thank you for the clarification question. In our experiment, we assume a single time-series possibly with multiple channels, which is split into train, validation, and test sets chronologically. The retrieval process is restricted to the training set of the same time-series, as shown in Figure 1. Our model has no limitations on the length of the training data that can be searched. For other settings, we used the problem setting and the benchmark datasets of prior works in the forecasting domain [1, 2, 3].

[1] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR 2024)
[2] SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters (ICML 2024)
[3] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (ICLR 2023)

2. Difference between RAFT and transformer-variants. Retrieval seems to only make sense in the zero-shot setting.

As clarified in the first question’s response, our problem setting only assumes a single time-series, thus retrieval to other external time-series data is not available. We also want to clarify that our retrieval process is not limited to the lookback window but instead operates over the entire training dataset of the time series. Transformers, on the other hand, learn relationships only within a fixed lookback window through attention mechanisms. Our model extends beyond this limitation by retrieving relevant data points from the entire time-series and incorporating them into the input. While increasing the lookback window size of a Transformer can provide access to more past information, this approach comes with significant drawbacks, including a reduction in the number of available data points and a substantial increase in task and computational complexity.

Our retrieval enables the model to directly reference relevant timestamps in the training set, eliminating the need for the model to memorize every pattern in the training data. By doing so, the retrieval module reduces the learning burden of the forecasting model of memorizing all patterns in the training set and helps the model learn more generalizable features. This would be particularly beneficial for long-term time-series forecasting tasks involving multiple channels. In the discussion section (Sec 5.2~5.3), we also showed that when the time series has rare or less-correlated patterns, the retrieval-augmented model can handle these cases more effectively than a model without retrieval, which would need to memorize all such patterns to make accurate predictions.

3. Ablation study is limited. The current experiments simply take NLinear as "without retrieval", but the design space is much larger and this is an important set of experiments to better understand the effects of retrieval.

Our model becomes identical in structure to NLinear if the retrieval module is removed, leaving only the linear predictor. Therefore, we used NLinear as an ablation baseline. Additionally, by assuming only a single period, we minimized the influence of factors like multi-periodicity, simplifying the setup to isolate and evaluate the sole effect of the retrieval module.

To further demonstrate our retrieval module design, we conducted additional ablation studies (e.g., the number of retrievals or retrieval without attention) and reported results in Appendix C. If the reviewer's suggestion regarding the "design space" refers to testing the retrieval module in an ad-hoc manner across other model structures beyond NLinear, such as transformers or other baselines, we acknowledge that there are numerous design choices for integrating retrieval into different architectures. Instead of exploring all these possibilities, we focused on demonstrating its effect in the simplest form. That said, we also agree that our model can enhance performance when integrated into structures beyond simple models. To verify this, we added the retrieval module to the Transformer-variant model (e.g., AutoFormer) and observed the performance improvements. These results have been included in the Appendix D.

4. Conclusion of the research question - “Do current models possess the necessary inductive biases and learning capacity to extract generalizable patterns from training data and achieve high accuracy?”

Thank you for the question. We believe that our answer is “no” for current models, and the proposed retrieval module serves as an effective approach to achieve the goal in the research question. Our retrieval module allows traditional time-series models to overcome the limitation of having to learn every specific pattern in the training data by replacing memorization with retrieval. This enables the forecasting model to focus on generalizable patterns across the entire time series, effectively reducing the learning burden and optimizing the model’s required capacity.