Rethinking the Power of Timestamps for Robust Time Series Forecasting: A Global-Local Fusion Perspective

Thanks for your positive comments and insightful suggestions. Please find our response below.

Q1: Some typos in the paper.

We apologize for the oversight that led to grammatical errors in the paper, causing issues with your reading. These typos will be corrected in the final version.

Q2: Reasons for using quantile.

As demonstrated in the Electricity dataset in Figure 3, history windows may contain anomalous noise, such as spikes, due to various complex factors in the real world. The mean and variance are highly sensitive to such spiky noise and tend to misnormalize the output of the mapper to an incorrect distribution. In contrast, quantiles are more robust to such noise and align better with our initial objective of providing robust global information. This is also confirmed by the ablation experiments in Section 4.4.

Q3: Challenges in capturing traffic spikes.

We visualize the full traffic flow in Figure 2 of the Global Response Attachment PDF. The test set has demonstrated drift, with peaks significantly higher in the latter half compared to the training set. Unless the history window incorporates relevant information (e.g., sharper peaks) or the model undergoes additional adaptive training, resolving this issue is challenging.

Q4: Additional results for ablation experiments.

We supplement the ablation experiments based on the iTransformer backbone across two representative datasets. The history window is set to 96, and the prediction window to 192. The results about MSE are presented in the table below. The results underscore the necessity of denormalization and the superiority of employing quartiles for robust denormalization.

Thanks for your positive comments and insightful suggestions. Please find our response below.

Q1: Explanation of the motivation for the component.

We apologize for the confusion. We will further explain the role of the denormalizer in mitigating data drift. The statistical characteristics (e.g., mean or variance) of time series data collected from the real world typically change over time as the system evolves dynamically. For instance, an increase in the popularity of a service can cause customer metrics (e.g., request counts) to rise over time. Previous studies [1, 2] have shown that models trained on the training set may underperform on the test set when the distributional difference between the training and test sets is significant. We assume that the difference between the distributions of the history and prediction windows is negligible within a small sliding window. As illustrated in Equation 3, the inverse denormalizer mitigates data drift by aligning the output of the mapper to the distribution of the local history window, thereby smoothing the difference between the training and test sets.

Q2: Additional results for more backbones.

The principle for selecting the backbone in our study is to balance model architecture, prediction accuracy, and timestamp treatment within a constrained space. Based on the library you provided, the current top three rankings for long-term forecasting are iTransformer, TimeMixer, and TimesNet. iTransformer and TimesNet have already been discussed in our paper, and we now supplement experimental results for TimeMixer. The history window is set to 96 and the prediction window to 192 for all datasets except ILI, where the history and prediction windows are both set to 36. The MSE are presented in the table below. The results demonstrate that our proposed method enhances performance across various mainstream backbones.

References

[1] 2022, Reversible Instance Normalization for Accurate Time-Series Forecasting Against Distribution Shift

[2] 2023, Adaptive Normalization for Non-stationary Time Series Forecasting: a Temporal Slice Perspective

Dear Reviewer Y7JY,

We greatly appreciate the time and effort you have invested in reviewing our paper and providing insightful feedback. As a gentle reminder, it has been more than 5 days since we submitted our rebuttal. As the discussion period is drawing to a close, we wish to ensure that our rebuttal has comprehensively addressed your concerns. We are keen to receive any further feedback you might have and are prepared to make additional clarifications or modifications as needed. Thank you once again for your valuable insights. We look forward to your final thoughts.

Thanks for your valuable comments. We will answer the questions one by one.

Q1: Comparison with additional baselines.

Thank you very much for your supplement. All three articles have significant contributions to our field. However, there seems to be some misunderstanding regarding our work. The timestamps mentioned in our paper denote external assistive information, such as "2024-08-01 12:00:00," along with their embeddings. Upon reviewing their papers and codes, it is evident that none of these three works incorporate this type of information, thereby classifying them in the same category as the baseline DLinear. The term "timestamp" used in their papers is more accurately replaced with "timestep", which indicates a concept of temporal position. Generally, these three works do not utilize assistive information represented by timestamps and do not belong to "existing methods that focus on better utilizing timestamps". Furthermore, the superiority of the baselines adopted in our paper (updated to 2024) compared to these three articles (published in 2021) has been extensively validated by previous work[1, 2]. Due to spatial limitations in the response, we will refrain from repeating it here.

Q2: Theoretical analysis of the proposed method.

iTransformer does not utilize the timestamp information of the prediction window, which may contribute to its underutilization of timestamps. Therefore, we focus solely on the comparison between the summation scheme represented by Informer and our proposed method, which can be abstracted as early fusion (feature-level fusion) and late fusion (decision-level fusion). Early fusion (Informer) integrates modalities into a single representation at the input level and processes the fused representation through the model. Late fusion (GLAFF) allows each modality to run independently through its own model and fuses the outputs of each modality. Compared to early fusion, late fusion maximizes the processing effectiveness of each modality and is less susceptible to the noise of a single modality, resulting in greater robustness and reliability. This has been validated by extensive previous work [3, 4]. To mitigate the effect of noise and fully exploit the robustness of global information represented by timestamps, our proposed method adopts late fusion. Furthermore, we supplement the ablation results about timestamp across the nine datasets. The MSE results in Table 1 of the Global Response attachment PDF clearly indicate that simple fusion methods, such as summation or concatenation, are ineffective with timestamps.

Q3: Explanation of the Adaptive Combiner.

Firstly, according to the ablation results on the nine datasets in Appendix B.2, the effect of the Adaptive Combiner is not "minimal", second only to the complete removal of the prediction backbone. Secondly, the combined weight $\mathbf{W}$ of the global mapping $\hat{\mathbf{Y}}$ and local prediction $\bar{\mathbf{Y}}$ in the Adaptive Combiner is derived from the MLP weight generation network. We do not specifically set the initial model weights of the MLP layer. They are entirely adapted autonomously based on the MSE loss using gradient descent from the randomly initialized values. It is important to note that the combination weights $\mathbf{W}$, which cannot be directly initialized, are not the model weights of the MLP.

Q4: Effects of timestamp granularity and noise.

Firstly, it must be noted that the timestamp information we integrate has been extensively utilized but ineffectively by various baselines. We do not introduce any supplementary information or assumptions. In other words, our approach does not impose more stringent requirements on timestamps (e.g., higher precision sampling granularity) compared to the Informer (2021) backbone. Secondly, maintaining the stability of prediction models in noisy environments is another extensive area where numerous dedicated works, such as RobustTSF [5], have provided excellent solutions. Nonetheless, to assess the robustness of our proposed method, we present the MSE results of the iTransformer backbone on two representative datasets. The results in the table below clearly demonstrate that our proposed method is robust against varying proportions of Gaussian noise, owing to components such as Robust Denormalizer.

Q5: Results across varying lookback window lengths.

We supplement the MSE results on two representative datasets with fixed prediction windows by varying the history windows. The prediction window is fixed at 192. From the experimental results in Table 2 of the Global Response attachment PDF, iTransformer and DLinear, which derive their final predictions from a linear layer, demonstrate a general trend of enhanced prediction accuracy with longer history windows. Furthermore, our proposed methods consistently enhance prediction performance.

Q6: Limitations of the proposed method.

We have discussed the limitations of our study in Appendix D. The primary emphasis is on the computational cost imposed by the attention mechanism, and a potential solution has been proposed.

References

[1] 2024, Self-Supervised Contrastive Learning for Long-Term Forecasting

[2] 2024, Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning

[3] 2024, Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

[4] 2023, A Comparative Analysis of Early and Late Fusion for the Multimodal Two-Class Problem

[5] 2024, RobustTSF: Towards Theory and Design of Ro-Bust Time Series Forecasting With Anomalies

Dear Reviewer XBcx,

We greatly appreciate the time and effort you have invested in reviewing our paper and providing insightful feedback. As a gentle reminder, it has been more than 5 days since we submitted our rebuttal. As the discussion period is drawing to a close, we wish to ensure that our rebuttal has comprehensively addressed your concerns. We are keen to receive any further feedback you might have and are prepared to make additional clarifications or modifications as needed. Thank you once again for your valuable insights. We look forward to your final thoughts.

Thanks for your positive comments and insightful suggestions. Please find our response below.

Q1: Usefulness of the proposed method in non-regular forecasting.

The initial point to state is that the timestamp information we integrate has been extensively utilized, though ineffectively, by various existing methods. We introduce no additional information or assumptions. Regarding the non-regular forecasting scenario you describe, we consider it a novel situation distinct from the current task. It appears to utilize transaction counts instead of time intervals to denote timing changes. Due to the differing domains, our proposed approach cannot natively support this scenario. As mainstream methods like iTransformer cannot address time series forecasting with missing values, we cannot expect a framework to solve all forecasting challenges. To my knowledge, neither the datasets nor the baselines examined in our paper are relevant to the new scenario you mentioned. Nevertheless, the idea of information fusion that we have proposed may be beneficial. Furthermore, the non-regular forecasting appears intriguing, and we look forward to further discussion after the review period concludes.

Q2: Confusion caused by the notations.

We use $\mathbf{S}$ and $\mathbf{T}$ for timestamps, where the last dimension is 6, and $\mathbf{X}$ and $\mathbf{Y}$ for observations, where the last dimension represents the number of channels in the multivariate time series. We apologize for the confusion and will incorporate your suggestion to enhance the readability of our article. Regarding the output in row 197, a marker on $\mathbf{Y}$ is indeed necessary to differentiate it from the actual labels, and we will address this in the final version.

Q3: Explanation of parsing statistics and Equation 3.

The median and quantile in Equation 3 are derived from the historical actual observation $\mathbf{X}$ and the historical initial mapping $\tilde{\mathbf{X}}$ within a local sliding window to accommodate the dynamics of the time series. We apply the same statistics, as indicated in the parentheses in Equation 3, to both the historical mapping $\tilde{\mathbf{X}}$ and future mapping $\tilde{\mathbf{Y}}$. We apologize for any confusion, but we do not advise concatenating $\tilde{\mathbf{X}}$ and $\tilde{\mathbf{Y}}$, as the following Adaptive Combiner will require separate historical mapping $\hat{\mathbf{X}}$ and future mapping $\hat{\mathbf{Y}}$ .

Q4: Performance of the proposed method on aperiodic data.

Robust global information represented by timestamps indeed offers less assistance for predicting non-periodic data than periodic domains such as transportation and electricity. In fact, this paper primarily focuses on effectively integrating available assistive information. Accurate prediction of dynamic and non-periodic time series is more challenging and requires the inclusion of more assistive information. The information fusion approach we have proposed is also valuable for integrating other supplementary information.

Q5: Modeling approaches for other side information.

We should design the modeling method for each type of assistive information based on its characteristics and practical requirements. According to the findings of our paper, simple fusion techniques such as summation or concatenation often fail to fully utilize the assistive information. However, more nuanced modeling generally entails greater resource consumption. In practical applications, it is essential to carefully balance prediction accuracy with computational cost.

Q6: Enhancing the engagement level of presentations.

It is true that bar charts provide a more visually engaging presentation of backbone performance improvements. However, arranging various combinations of backbones, metrics, datasets, and prediction lengths within a limited space remains a considerable challenge. We will persist in seeking more effective presentation methods.

Q7: Impact of timestamps on baseline methods.

We supplement the ablation results about timestamp across the nine datasets. The history window is set to 96 and the prediction window to 192 for all datasets except ILI, where the history and prediction windows are both set to 36. The MSE results in Table 1 of the Global Response attachment PDF clearly indicate that simple fusion methods, such as summation or concatenation, are ineffective with timestamps.

Q8: Results across varying lookback window lengths.

We supplement the MSE results on two representative datasets with fixed prediction windows by varying the history windows. The prediction window is fixed at 192. From the experimental results in Table 2 of the Global Response attachment PDF, iTransformer and DLinear, which derive their final predictions from a linear layer, demonstrate a general trend of enhanced prediction accuracy with longer history windows. Furthermore, our proposed methods consistently enhance prediction performance.

Thanks for your positive comments and insightful suggestions. Please find our response below.

Q1: Explanation of computation time and memory consumption.

As stated in Appendix D, higher prediction accuracy is typically accompanied by a higher computational cost. Balancing these needs requires consideration within the context of actual production demands. This study primarily focuses on enhancing prediction accuracy. In future work, we will consider model reduction to broaden the applicability of the proposed method. Additionally, we have not provided the percentage increase in computation time and memory consumption due to insufficient statistical significance. In the case of memory consumption, our proposed method increases consumption by 25MB for both DLinear (from 0.1MB) and TimesNet (from 150MB), representing percentage increases of 25000% and 17%, respectively. This statistical method may cause confusion and misunderstanding for readers.

Q2: Explanation of the misunderstanding caused by Figure 2.

We apologize for the confusion. As you noted, the statistics used in the inverse normalization for the historical mapping $\tilde{\mathbf{X}}$ and future mapping $\tilde{\mathbf{Y}}$ are indeed identical, as indicated in the parentheses in Equation 3. The two "Stats" are plotted in Figure 2 primarily for a more organized framework diagram.

Q3: Theoretical proof of the proposed method.

iTransformer does not utilize the timestamp information of the prediction window, which may contribute to its underutilization of timestamps. Therefore, we focus solely on the comparison between our proposed method and the summation scheme represented by Informer. The method proposed by Informer and our approach can be abstracted as early fusion (feature-level fusion) and late fusion (decision-level fusion). Early fusion (Informer) integrates modalities into a single representation at the input level and processes the fused representation through the model. Late fusion (GLAFF) allows each modality to run independently through its own model and fuses the outputs of each modality. Compared to early fusion, late fusion maximizes the processing effectiveness of each modality and is less susceptible to the noise of a single modality, resulting in greater robustness and reliability. This has been validated by extensive previous work [1, 2]. To mitigate the effect of noise and fully exploit the robustness of global information represented by timestamps, our proposed method adopts late fusion.

Q4: Applicability of the proposed method in the absence of timestamps.

Thanks to the timestep-level embedding and attention mechanism, our proposed approach can tolerate missing timestamps. Specifically, it only requires adding the corresponding mask matrix for the Attention-based Mapper to exclude the missing timestamps, allowing the entire framework to function seamlessly. We present the MSE results of the iTransformer backbone on two representative datasets. The history window is set to 96, and the prediction window to 192. The results in the table below clearly demonstrate the robustness of our proposed method to varying percentages of missing timestamps.

Q5: Availability of the datasets.

The timestamp information we integrate has been extensively utilized, though ineffectively, by various existing methods. We introduce no additional information or assumptions. As detailed in Appendix A.1, all datasets employed (including timestamp information) are publicly accessible. We will also release the code and datasets upon completion of the paper review.

Q6: Visualization of the output from the mapper.

We visualize the Mapper and Denormalizer outputs for the two prediction cases in Section 4.3 of the paper in Figure 1 of the Global Response attachment PDF. From the experimental results, the Attention-based Mapper captures the majority of the shape information, while the Robust Denormalizer aligns the distribution of mapping values. Together, they provide comprehensive and robust assistance for accurate prediction of the backbone model.

References

[1] 2024, Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

[2] 2023, A Comparative Analysis of Early and Late Fusion for the Multimodal Two-Class Problem