Response to Reviewer N2A5

We would like to sincerely thank Reviewer N2A5 for providing a detailed review and insightful suggestions.

W1: There is no comparison between native models that can be used for forecasting with Exogenous Variables.

In $\underline{\text{Table 7 of the Appendix}}$, we have compared the performance of TimeXer with previous approaches, NBEATSx and TFT, both of which are natively designed for forecasting with exogenous variables. To make this clearer, we present the results here and TimeXer surpasses these two native models.

Q1: How is the memory consumption of the model measured? It seems that there is no implementation in the provided code.

We assess memory consumption by monitoring GPU memory usage during the model training phase.

Q2: The prediction task of TimeXer and previous prediction task such as iTransformer and DLinear are essentially different types of prediction targets. More details need to be disclosed on how to transfer previous prediction tasks in the text.

Sorry for the missing details. As stated in $\underline{\text{Line 193 of main text}}$, we discover that the forecasting with exogenous variables task can be generalized into multivariate forecasting tasks through the channel independence mechanism. Technologically, we consider the variables in the dataset as mutually independent endogenous variables, with each variable considering all other variables except itself as exogenous. We also have provided the code for multivariate forecasting in the $\underline{\text{supplementary materials}}$.

Dear reviewer,

The author-reviewer discussion ends soon. If you need additional clarifications from the authors, please respond to the authors asap. Thank you very much.

Best,

AC

I will complete my response promptly. Thank you for the reminder.

Response to Reviewer 1zKu

Many thanks to Reviewer 1zKu for providing a detailed review and questions.

W1: There is a potentially slight inconsistency between the description of the method and the scale of the data used in the experiments.

Thank you for your careful reading. Since most of the baseline forecasters can only support cases where the lengths of the exogenous and endogenous variables are the same, we set the lengths of the exogenous and endogenous variables to be the same to align with previous benchmarks for a fair comparison.

We have experimented with the case where the exogenous and endogenous variables are of unequal length in $\underline{\text{Section 4.3 of main text}}$, encompassing two common real-world scenarios: (1) unequal look-back length (2) different sampling frequencies. As presented in $\underline{\text{Figures 3,4 of main text}}$, TimeXer exhibits superior generality over various practical cases.

W2: Some highly relevant literature is missing.

Thank you for the recommendation. We will include them as baselines in the camera-ready version. Considering these two models are proposed for multivariate data, we compare the TimeXer performance in multivariate forecasting with those given in their original papers. Note that the experimental setup in FourierGNN is not the same as TimeXer, we use the results reported by ForecastGrapher[1].

[1] "ForecastGrapher: Redefining Multivariate Time Series Forecasting with Graph Neural Networks." arXiv preprint arXiv:2405.18036 (2024).

W3: Essentially, TimeXer and iTransformer share a similar model structure, yet TimeXer consumes less memory. What do you believe accounts for TimeXer being more memory-efficient than iTransformer?

Many thanks for your insightful question. We have included a theoretical efficiency analysis w.r.t. iTransformer in $\underline{\text{Appendix E}}$. Here is an intuitive understanding.

The key design of iTransformer is embedding each variate series into one token. Afterwards, the embedding variate tokens will be fed to Transformer, which will apply a self-attention mechanism among variate tokens. Although this design can keep refining the learned variate token in multiple layers, it does cause more complexity.

As for TimeXer, as presented in $\underline{\text{Figure 2(b) of main text}}$, exogenous series will be embedded to variate tokens at the beginning, which will be shared in all layers and interact with the endogenous global token by cross attention. Thus, TimeXer omits the interaction among learned exogenous variate tokens, which makes TimeXer more efficient than iTransformer.

W4: I would like to see the complete removal of the exogenous variable enhancement module, for instance, by setting the exogenous variables to zero or random numbers, to explore TimeXer's adaptability to exogenous variables.

We have provided the experimental results on TimeXer's generality under missing exogenous values in $\underline{\text{Appendix C}}$. As per your request, we have conducted more ablation studies on the quality of the exogenous variables.

Technologically, we use two kinds of exogenous variables: (1) set the series into zeros (TimeXer-zerosEX) and (2) set the series to random numbers with mean 0 and variance 1 (TimeXer-randomEX). As demonstrated in the $\underline{\text{Table 1 in the Author Rebuttal PDF}}$, the inclusion of exogenous variables leads to an improvement in model performance, while the performance decreases when the exogenous variables are meaningless noise.

Q1: Have you considered exploiting large language models (LLM) and leveraging rich textual knowledge, to improve the model's ability to understand exogenous variables at multiple granularities?

Following the reviewer's suggestion, we have endeavored to leverage large language models for forecasting with exogenous. Technologically, we design a multi-grained prompt and use a T5 tokenizer to extract textual information, which is then integrated into TimeXer by adding it to the corresponding variate tokens. The prompt consists of the name and sampling frequency of each variable, for example, "This exogenous variable is load generation, with a sampling frequency of 1h. We use it to enhance the prediction of electricity price in the Nord Pool market".

Our experiments indicate that there was no improvement in model performance, likely due to insufficient textual information. We believe that a more sophisticated design for prompts is necessary to capture exogenous variables at multiple levels of granularity, which we will address in future work.

Q2: In most forecasting scenarios, do you think that the historical information of the target sequence itself is more useful, or is the information from exogenous variables more valuable?

Thanks for the reviewer's valuable suggestion. As per your request, we set the exogenous and endogenous series to zeros respectively (TimeXer-zerosEX and TimeXer-zerosEN), and the results are listed in $\underline{\text{Table 1 in the Author Rebuttal PDF}}$. We find that TimeXer benefits more from the historical information of endogenous series.

Therefore, in most forecasting scenarios, the historical information of the target sequence itself is more useful. However, there are specific cases where the exogenous variable has a time-lagged influence on the non-stationary endogenous variable, in which case the exogenous series may be more useful.

Response to Reviewer Mh7v

Many thanks to Reviewer Mh7v for providing a detailed review and insightful questions.

W1: The implementation details on how to extend TimeXer to multivariate time series prediction are not sufficiently detailed.

Sorry for the missing description. The success of the channel independence mechanism has demonstrated that multivariate forecasting can be viewed as multiple independent univariate forecasting with a shared backbone. Inspired by this, we find that TimeXer can be generalized to multivariate forecasting by employing the channel independence mechanism.

Specifically, For multivariate time series [B, T, C], we obtain endogenous embedded vector [B*C, N, D] and exogenous embedded vector [B, C, D], where N is the number of patches and D is the hidden dimension. Note that the endogenous embedded vector is already channel-independent and further fed into the shared Endogenous Self-Attention Module to capture temporal dependencies within each variable. Further, we select the global token of each variable (which is then reshaped from [B*C, 1, D] to [B, C, D]) to perform cross-attention with the exogenous embedded vector to capture cross-variate dependencies. We have provided the relevant code in $\underline{\text{supplementary materials}}$.

W2: The article does not provide a specific explanation for why the non-overlapping patch method is used instead of the overlapping method.

Following your suggestion, we have completed the ablation study on the effects of patching in TimeXer. We adopt the patching method used in PatchTST (ICLR 2023), setting the patch length to 24, consistent with TimeXer, and the stride to 12, to generate a sequence of overlapped patches. Compared to the overlapping method, TimeXer has the lowest complexity while having the optimal performance.

It is also notable that not only in NLP and CV, contemporary time series approaches ([41 of our paper] and [1]) also use non-overlapping patches. This preference might stem from the limited redundancy present in time series data, as excessive overlap can result in a smoothed representation for each patch, consequently failing to capture correct temporal dependencies.

[1] "Timer: Generative Pre-trained Transformers Are Large Time Series Models." ICML 2024.

W3: Figure 3 shows the prediction performance corresponding to different step lengths, but it lacks experimental results for some extreme cases, such as reducing the step length of exogenous/endogenous variables to 1.

As per your request, we conduct experiments in some extreme cases. Concretely, we set the look-back length of exogenous/endogenous variables to 1 respectively and list the results as follows, where EN and EX denote the endogenous and exogenous series respectively.

We can find that TimeXer’s performance is relevant to the look-back length of the endogenous variable, deteriorating when historical information is severely limited. Additionally, the length of the exogenous variable also influences forecasting performance, with better results observed when more external information is available.

W4: It is recommended to explain more clearly in the methods and introduction sections why the global token is applied to the time series prediction task.

Thanks for your suggestion, we will provide a more detailed description in $\underline{\text{Line 146-148 of main text}}$ to better explain the role of the global token in the revision: "Given the distinct roles of endogenous and exogenous variables in the prediction, TimeXer embeds them at different granularity. Directly combining patch-level endogenous tokens and variate-level exogenous tokens results in information misalignment. To address this, we introduce a learnable global token for each endogenous variable that serves as the macroscopic representation to interact with exogenous variables. This design helps bridge the causal information from the exogenous series to the endogenous temporal patches."

Q1: The influence of exogenous variables on endogenous variables is multi-grained, including aspects like periodicity, trend, noise, and stability. Does the Transformer have the capability to capture such multi-grained influences, or does the author think that more promising methods (such as LLMs) could be exploited to capture these associations?

Thank you for your insightful question. Previous works, e.g. Autoformer, have attempted to use the seasonal-trend decomposition method to reveal the entangled temporal patterns in time series data and better capture underlying temporal dependencies. We think it will be reasonable to apply more elaborative designs (e.g. decomposition or LLMs) to the endogenous and exogenous series and leverage our proposed architecture to capture these multi-grained influences.

Since we attempt to "unleash the potential of the canonical Transformer without modifying any component" ($\underline{\text{Lines 60-61 of Introduction}}$), we incorporate external information into TimeXer predictions through series-level dependencies, which works well. We would like to leave the exploration of multi-granularity effects in future work. As for the exploitation of LLMs, Reviewer 1zKu has a similar question, you can refer to our rebuttal to 1zKu Q1 for experiments on leveraging LLM.

Response to Reviewer r8UR

We would like to sincerely thank Reviewer r8UR for providing a detailed review and insightful suggestions.

W1: Methodological Gaps in Handling Practical Situations

As the reviewer mentioned, we focus on forecasting with exogenous variables and have provided a comprehensive analysis for practical forecasting scenarios in $\underline{\text{Lines 31-39 of main text}}$. Although the real-world scenarios are multifarious, our proposed TimeXer provides a simple, flexible, and effective method to tackle practical challenges, such as missing value or mismatch information.

With our specially designed variate embedding layer (learned embedding and cross-attention module), TimeXer can naturally handle the mismatch series length and diverse information, which has been comprehensively verified in our experiments:

(1) Mismatch Length: In $\underline{\text{Lines 255-256 and Figure 3 of main text}}$, we enlarge the length of exogenous and endogenous series respectively, resulting in unequal input lengths.

(2) Mismatch Frequency and Temporal Misaligned: We conduct experiments on large-scale time series data ($\underline{\text{Figure 4 of main text}}$). As we stated in $\underline{\text{Lines 275-276 of main text}}$, the sampling frequencies of the endogenous and exogenous variables are 1h and 3h respectively. Also, the temperature, pressure, and wind meteorological indicators exist inherent temporally misaligned relation (temperature will change earlier, then pressure and finally wind).

(3) Missing Values: In $\underline{\text{Section C of the Appendix}}$, we randomly mask the exogenous series to obtain cases with missing values.

The above results verify that our variate embedding design provides a simple and neat way to bridge Transformer to practical situations, whose flexibility and effectiveness should not be overlooked.

W2&Q1: Lack of Intuitive Examples for Exogenous Variable Utilization

Following your suggestion, we have included more showcases in $\underline{\text{Figure 1 of the Author Rebuttal PDF}}$.

• We visually present the prediction results in two cases, with and without exogenous variables, to validate the role of exogenous variables.
• As per your request in Q1, we add an extreme case where there is no historical information on endogenous series to explore whether TimeXer can learn from exogenous variables in scenarios where only external information is available.
• In addition, we also add a special case where the prediction of exogenous variables is available to the model. This is a practical scenario for the $\underline{\text{EPF datasets}}$ where the exogenous variables are the day-ahead predictions of the source generation.

We can observe that the inclusion of exogenous variables successfully enhances model performance, and when further exogenous variable predictions are utilized, the performance achieves the best.

W3&Q2: Lack of Causality Between Variable Similarity and Prediction Accuracy

In our paper, we adopt $\underline{\text{Figure 5 of main text}}$ to demonstrate that the cross-attention module empowers TimeXer with special interpretability. We would like to highlight that the similarity we mentioned is not directly equivalent to "shape similarity"; rather, it is derived from the attention mechanism, which captures the intrinsic temporal patterns present within time series data. Intuitively, time series with similar shapes may exhibit shared temporal features, resulting in a higher similarity score. Thus, the most similar exogenous series learned by the attention mechanism may intuitively resemble the endogenous variable, but it cannot be said that all similar series will definitely contribute to the prediction.

As per the reviewer's request, we conduct analysis studies on the causality between variable similarity and model performance. Technologically, we select the most similar and least similar variable to the endogenous variables through the learned attention map and further conduct experiments by removing this exogenous variable. The results are listed as follows, which can observe that series with larger similarity (learned by TimeXer) is more beneficial to the prediction.

Q3: In the experiments, univariate forecasting with exogenous variables and multivariate forecasting were not distinctly separated. Why did you choose not to clearly differentiate between these two types of forecasting scenarios in your descriptions?

As stated in $\underline{\text{Line 193 of main text}}$, we find that forecasting with exogenous variables can be a unified forecasting paradigm that generalizes straightforwardly to multivariate forecasting by taking each variable as endogenous, the other variables are exogenous.

To verify the effectiveness and generality of TimeXer, we conduct experiments under two different forecasting paradigms, namely short-term forecasting with exogenous variables on EPF benchmarks and long-term multivariate forecasting on well-established benchmarks. The experimental results for these two distinct forecasting tasks are given in $\underline{\text{Tables 2, 3 of main text}}$, respectively.