Q1 and Q2: The author flattens all patches from different variates into a unified sequence. I wonder whether the sequence is (1) normalized, (2) univariate or multivariate. Furthermore, if the input dimension is high, the unified sequence may be too long. What is the difference of constructing a unified sequence and simple concatenation? It seems very similar to me.

The sequence is normalized and multivariate. We conduct the instance normalization before patching and de-normalization before output (adding back the mean and standard deviation back to the prediction), which is also done in PatchTST [1].

We agree that when the input dimension is high, the unified sequence can be long. That is also the purpose of using dispatchers to reduce the memory consumption when the length is large. The flattened patch sequence is the same as doing concatenation with multiple variates of time series, where we allow the model to capture dependencies on different variates and different time.

Q3: How does the author determine the order of raw input series to construct the unified sequence? If the order is random then I suspect that the encoder may not be time-aware, making the output less reliable without additional input of time.

We use the original order of the multivariate, i.e., we do not change the order. We apply a learnable additive position encoding $W_{pos}$ as in [1] to capture the order information.

Reference:

[1] A TIME SERIES IS WORTH 64 WORDS: LONG-TERM FORECASTING WITH TRANSFORMERS. ICLR 2023.

Dear Review uAEX,

We appreciate your valuable feedback and suggestions. We would like to clarify and answer questions below:

W1: The paper claims that 'previous transformer models lack ability to simultaneously capture both inter-variate and intra-variate dependencies', which is not accurate. In fact, there is some literature focusing on modeling inter- and intra- series information in a Transformer structure, for example, CATS: Enhancing Multivariate Time Series Forecasting by Constructing Auxiliary Time Series as Exogenous Variables (ICML 2024) adopts a very similar idea by constructing an auxiliary series to capture inter- and intra- series. The author should revise their literature review accordingly and explain their difference with CATS.

Thanks for pointing out this work. We woud like to clarify that CATS and our work both aim to capture inter- and intra- series dependencies, however, CATS construct auxiliary series and capture inter-series dependencies from auxiliary series. In contrast, our method is applied directly on the original series with considering all multivariate as a unified sequence. We will revise the literature review accordingly in our pdf.

W2: Time series forecast using LLM has been a trand in recent years. The author is encouraged to include more LLM-based methods as benchmarks, such as LLM4TS or GPT4TS.

Thanks for the suggestions. We provide the table to compare our model UniTST with GPT4TS below. As mentioned in the manuscript, we are using sequence length as 96.

We found that UniTST usually outperform GPT4TS on most of the cases(especially on Electricity, Traffic, and Weather) while GPT4TS slightly outperforms UniTST on ETT datasets with a few prediction lengths. Interestingly, we can also see that GPT4TS favors the relatively short prediction lengths (i.e., with the long prediction length, it is generally worse than UniTST).

In this work, we mainly focus on investigating fundmental building blocks of Transformer-based multivariate methods. Therefore, we leave the discussion and investigation for LLM-based methods as future works.

The response to other questions is posted in the next comment.

Q1: What is the value of t' corresponding to the correlations in Figure 3?

The x-axis means t in variate 10 and the y-axis indicates t' in variate 0. t and t' are represented as the patch indices (basically aggregate several time stamps to a patch).

Q2: Since the router operation, that is, the Dispatcher proposed in the article, is used, it may not be possible to directly visualize the attention weights between variables learned by the model to correspond to the correlation calculated by Definition 1. Then, can you provide the correlation value calculated for the model's prediction results and compare it with the correlation based on real data given in Figure 3? Does the model truly capture the dependencies across different variates and different times that you proposed or other fitting results?

As Figure 3 we showed in the manuscript is the input data on the training set, we are not able to plot the correlation of our model's prediction and compare them. But in the revised manuscript, we provide two case visualization on the correlation map of multivariate relationship in the predicted time series from Solar-Engery in Appendix D (Figure 10). Compared with iTransformer, we can see that the correlation map of UniTST is more aligned with the ground truth time series.

W3 and Q3: It is recommended to provide code for reproducibility.

Thanks for the suggestions. We added our code as the supplementary materials.

Dear URks,

We appreciate your valuable feedback and suggestions. We have revised our manuscript based on your comments. Also we would like to clarify and answer questions below:

W1: For the current research situation, this contribution is relatively ordinary. There are many almost identical practices in the past. For example, Different sEnsors at Different Timestamps (DEDT) in https://arxiv.org/abs/2309.05305 is completely the same as "across different variates and different time" in the second line of the third paragraph of the Introduction. Cross-correlation coefficient in https://arxiv.org/abs/2401.17548 is almost same with the Definition 1, just with different names. I think the explanation of the problem and the drawings are not as clear as those in past articles. At the end of the second paragraph of the Introduction, the problems in the two stages are mutually influential. I think there is a lack of experimental proof for the problem. Moreover, the two-stage method in the past https://arxiv.org/pdf/2402.19072 is not inferior to UNITST in terms of effect.

We respectfully disagree on this point. Althrough "Different sEnsors at Different Timestamps (DEDT[1]) in https://arxiv.org/abs/2309.05305" is somewhat similar to ours "dependencies across different variates and different time" from the concept perspective. We are designing a different method to capture these: our method is a simple transformer-based method while theirs is a GNN-based model.

Compare with LIFT[2] (https://arxiv.org/abs/2401.17548), we provided evidence to demonstrate that the cross-correlation coefficient commonly exist in the real-world datasets. In terms of the methodology, we design a method to directly capture cross-variate cross-time dependencies instead of explicitly calculating the leading indicators.

For the two-stage method TimeXer [3] (https://arxiv.org/pdf/2402.19072) you mentioned, we provide the comparison between UniTST and TimeXer as follows:

We can see that, for MAE, UniTST is better than TimeXer on 5 out of 8 datasets. For MSE, UniTST outperforms TimeXer on 4 out of 8 datasets. We would like to point out that TimeXer still falls into the category where the models sequentially capture cross-time and cross-varaite dependencies. In contrast, we aim to propose a simple unified module to simultaneously capture intra-variate and inter-variate dependencies.

References:

[1] Fully-Connected Spatial-Temporal Graph for Multivariate Time-Series Data. AAAI 2024

[2] Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators. ICLR 2024

[3] TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables. NeurIPS 2024.

W2: The Dispatcher is also the same as the router in Crossformer . There is no innovation.

The design of dispatchers is similar. However, we would like to point out the whole architecture and the way we are capturing cross-time and cross-variate dependencies are different. Crossformer is basically a two-stage method to sequentially capture these dependencies, while our method is an one-stage method to simulanteously capture cross-time and cross-variate dependencies.

W4: Incidentally, the article uses the template of ICLR 2024. When reviewing, it is not convenient to locate the exact line.

Thanks for pointing out this issue. We have submitted a revised manuscripts with the correct template. More discussions on related work and additional case studies are provided (highlighted in blue).

Dear Reviewer URks,

First, we appreciate your feedback and suggestions again. We would like to kindly remind you that the deadline for pdf updates is in just a few hours. We are looking forward to your reply, so that we can clarify further if needed. Following your suggestions, we have provided some comparisons with other methods and explained the difference. Additionally, we have also provided the code for reproducibility and revised the manuscript with more literature reviews.

we would like to explain/re-emphasize that our contributions are not mainly on the methodology itself. It is more on how we find the right problem to solve and come out with a simple method: first identify the important limitation of previous works by showing empirical evidence in real-world data (i.e., failing to capture inter-series and intra-series dependencies as pointed out in Line 44 to 82), and propose a simple and effective method to directly and effectively solve this limitation.

Hope you are satisfied with our response. If you still have any concerns, we are also willing to discuss/clarify further. We are looking forward to your reply.

Dear all,

I am the AC of this paper and I have noticed all the messages, as well as the "mistyped score".

I kindly remind you all please be polite and respectful to the authors and reviews. The discussion is still open and please be free to continue on the results/contribution discussions in a thoughtful and respectful way. Between November 26th and December 3rd, authors can reply to the messages.

I will also initiate an AC-Reviewer discussion soon.

Thank you all for your submission/review/contribution to ICLR-2025.

Your AC

We are aware that you made a comment here: https://openreview.net/forum?id=cuFnNExmdq&noteId=9nj2gK3Gq8. We are quoting your full comments as below to better respond:

I share your concerns, and at the same time, the author's initial response was also delayed, failing to genuinely address the doubts regarding motivation.

We would like to ask for clarification on why you are saying "the author's initial response was also delayed, failing to genuinely address the doubts regarding motivation." We did not really get the logic between "the author's initial response was also delayed" and "failing to genuinely address the doubts regarding motivation."

Meanwhile, we would like to also point out that our initial response to you was posted on 24 Nov 2024, 16:04, which is 3+ days before the original deadline. You may consider it is a bit late (really up to different people). But we want to point out that we were also asked to add more experimental results by reviewers. And we all know experiments take time. Additionally, we faced computational resources shortage within our organization, which is out of our control.

We tried our best during the discussion phase and replied all comments politely. Therefore, we would like to kindly ask for some respect for our efforts and hope you can understand our situation, if possible.

Dear Reviewer 3TG2,

We appreciate the valuable feedback and suggestions. We would like to clarify and answer the questions as below:

W1: The motivation behind the study is not clearly articulated, which makes it difficult to fully understand the underlying rationale and significance of the research problem. ... Strengthening the motivation and highlighting the unique contributions of the work would enhance its originality and impact within the field.

Thanks for pointing out the issue. We would like to highlight again that our main contribution is first pointing out the importance of simultaneously capturing inter- and intra-series dependencies with evidence in real-world data (see Figure 2 and 3) and previous works lack this ability. We believe this is also valuable as no work provides evidence to explain why we need to simultaneously capture inter- and intra-series dependencies, to our best knowledge. With this motivation, we propose a simple and effective method to directly and effectively solve the limitation of previous methods.

W2: It is also essential to assess the whole model's memory impact relative to other models. A more rigorous evaluation would involve additional comparisons with state-of-the-art (SOTA) models, focusing on computational cost and model parameter size both before and after integrating the dispatcher. This broader comparison would offer a clearer understanding of the dispatcher’s role and efficiency within the model.

Thanks for the suggestions. We provide more evaluation results on computational cost and model parameter size for UniTST w/ and w/o dispatchers, PatchTST, and iTransformer. The memory cost is obtained on Traffic datasets with batch size 64 for all models.

We can see that UniTST w/ dispatchers has slightly more parameters than w/o dispatchers while w/ dispatchers have lower memory cost by avoiding the huge attention matrices. In addition, we also found that iTransformer has much more model parameters than UniTST and it requires slightly more memory compared with UniTST w/ dispatchers.

Q1: The ablation study does not specify the batch size used, also there are questions about the choice of memory usage as a comparative metric.. A detailed explanation is needed to clarify why memory usage was selected as a benchmark for this experiment and why one out of four ablation experiments resulted in an out-of-memory (OOM) error.

In the ablation study, as pointed out on the top of page 8, we ensure that these hyperparameters (the number of layers, hidden dimensions, batch sizes) are set as the same for both w/ and w/o routers. The specification are: batch size 64, hidden dimensions as 128 and 4 layers.

Q2: Although the results from the experiment in Figure 3 are cited as the primary source of motivation, the rationale behind conducting the experiment in Figure 3 itself is unclear. Most existing models that aim to capture inter-variable and intra-variable dependencies in multivariate data do not employ patch operations. Thus, the introduction of patching in the experiment warrants further explanation.

Thanks for raising this issue on the motivation figure and the patching operations. We would like to provide further explanations on why we use patches. Patching essentially aggregate several adjacent time stamps as a sub-series and map to an embedding. In fact, patching has been used in several previous work of different types, e.g., patchTST (channel-independent) and Crossformer (aiming to capture inter-variable and intra-variable dependencies). The models work on the whole series (such as iTransformer) can be also considered as setting patch size as series size.

In our humber opinion, patching avoid the noisy information at individual time stamp level and enhances the local semantic meaning. With patching, our proposed attention module can capture the dependencies between patches (sub-series). In contrast, if not using patching, the attention is applied on individual time stamp level, which has no semantic meaning. Additionally, without patching, we cannot investigate the correlation between two scalar values of different time stamp. In language domain, each word has real semantic meaning, while in time series, each value at one time stamp has no semantic meaning. This difference created the difficulty of time series modeling. Based on these reason, we believe patching can help modeling ability. Therefore, we incorporate patch operations in our method.

Q1: Could the authors further expand the search range for hidden dimensions to evaluate all models' performance fairly? Also, Please publicly disclose the optimal parameters for all models across various tasks as determined through validation sets (batch size, hidden dimensions, etc.). This is crucial to validate the performance of UniTST and mitigate any impact from selective parameter choices.

For the results of previous models (including iTransformer), we reuse the results from iTransformer paper [1] as we are using the same experimental setting, which we believe should be the best results of these models (or at least for iTransformer which is generally the second best model as shown in our evaluation result tables).

Here, we disclose the hyper-parameters of UniTST used on all datasets:

Q2: I noticed the experimental results in Table 6, which seem to show minimal improvement compared to iTransformer in many cases. Could the authors, after addressing question 1, provide new experimental results, analyze the reasons behind them, and conduct a comparative analysis of the efficiency (time/memory) of these two methods?

Training Time Comparison : (second/iteration)

Memory and model parameter size comparison:

The memory cost is calculated on Traffic dataset with batch size 64 for both models.

On the above results, we can see that UniTST can be trained faster on Exchange and Weather datasets. For model parameter size, UniTST uses significantly fewer parameters than iTransformer. Additionally, UniTST also has slightly lower memory cost.

Q3: Can the authors provide a case study of a variable prediction curve?

Sorry we are not sure what it refers to. Could you clarify the meaning of the variable prediction curve? Varying prediction length and see how the performance changes? Or the plots directly on the predictions generated by different models?

Reference:

[1] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

Dear Reviewer wpST,

We appreciate the valuable feedback and suggestions. We would like to clarify and answer the questions as below:

W1: The authors' model design lacks innovation. While flattening patches makes sense, there is a lack of innovation in the model design, as the Transformer architecture used does not appear to offer any significant novelty. The approach to Attention with dispatchers and the setup resembling ETC[1] and Crossformer[2] are nearly identical.

We would like to point out that we provided the discussiong between our model and Crossformer in Section 2 and 4.2. Despite the model design on attention with dispatchers may be similar, we are using this different purposes: our method uses dispatchers to reduce the memory complexity for allowing the attention to simultaneously capture inter- and intra- series dependencies on the unified sequence. Crossformer uses a two-stage method to sequentially capture intra-series dependencies and then inter-series dependencies, which is relatively limited as we discussed in Section 3.

W2 and W3: The authors' lack of thorough research on past methods is concerning. While authors took note of iTransformer from ICLR 2024, they failed to consider contemporary state-of-the-art methods such as TimeMixer[3], FITS[4], and ModernTCN[5]. Furthermore, this still lacks a comparison with GNN-based methods like CrossGNN[6] and FourierGNN[7] from NeurIPS 2023. The experimental comparisons are not enough insufficient. The methods mentioned in W2 were also not compared by the authors, therefore, it cannot be concluded that UniTST achieves SOTA performance.

Thanks for the suggestions. We added several methods, ModernTCN [1] (TCN-based), TimeMixer[2] (MLP-based), CrossGNN[3] (GNN-based), and TSLANet[4] (CNN-based) to be compared with our method. Following iTransformer, we use sequence length as 96 for all models. We show the average results over 4 prediction lengths (i.e., 96, 192, 336, 720) on the following table.

In the table, we can see that, for MSE, UniTST achieves the best performance on ETTm1, ECL, Traffic (3 out of 8 datasets). TimeMixer and ModernTCN are the best for MSE on 2 out of 8 datasets. For MAE, UniTST are the best model on ETTm1, ECL, Traffic, and Weather (4 out of 8 datasets).

W4: The lack of details in reproducibility.

Thanks for the suggestions. We added the code to the supplementary materials.

References:

[1] Luo, Donghao, and Xue Wang. Moderntcn: A modern pure convolution structure for general time series analysis. ICLR 2024.

[2] Wang, Shiyu, et al. Timemixer: Decomposable multiscale mixing for time series forecasting. ICLR 2024

[3] Huang, Qihe, et al. Crossgnn: Confronting noisy multivariate time series via cross interaction refinement. NeurIPS 2023.

[4] Eldele, Emadeldeen, et al. Tslanet: Rethinking transformers for time series representation learning. ICML 2024

I have contacted with the program committee on the template misuse issue. We agree that the template deviation of this submission is ok since 1) there are only tiny differences between the deviations and 2) this submission still have half a page leftover, and the deviation doesn’t give them more space.

Best,

AC