Many thanks to Reviewer b5RW for providing thorough detailed comments.

Q1: Robustness and completeness of the experiments.

Thanks for your scientific rigor. We provide the additional experiments in our revised paper:

• Report the standard deviations of iTransformer performance in $\underline{\text{Table 5 of the revised paper}}$ with five random seeds, exhibiting stable performance.
• Update the main results in $\underline{\text{Figure 1 and Table 1 of the revised paper}}$. We report the averaged results from four subsets of ETT and PEMS. We also add the Exchange dataset. The detailed results are updated in $\underline{\text{Table 9 and 10 of the revised paper}}$.
• Full results of the inverted Transformers on more datasets in $\underline{\text{Appendix E and F of the revised paper}}$ to further support the conclusions of our model analysis.

Q2: About the different PatchTST results from its original paper.

We carefully read through the forecasting settings of the PatchTST paper and its official repo. There's a lot of difference here:

• Enlarged lookback window: PatchTST paper adopts tunable lookback length in $\{336, 512\}$, while ours adopts the unified length of $96$, following the unified long-term forecasting protocol of TimesNet.
• More epochs to train: PatchTST paper trains the PatchTST model with $100$ epochs, while we train all models with $10$ epochs.
• Learning rate: PatchTST paper adapts a carefully designed learning rate strategy.

Q3: About the writing issues.

Thanks for your valuable suggestions. We conduct the following revisions of our paper:

• Abstract: "Transformer is challenged" -> "Transformers are challenged".
• Introduction: "Concerning irrationality of embedding multivariate points" -> "Concerning the potential risks of embedding multivariate points".

Q6: Discussions on the Channal-Independence/Dependence, Instance Normalization, and the linear models.

Thanks for your valuable suggestion. We newly added RLinear to our baseline, which is very competitive on several datasets, such as ETT and Weather.

As the reviewer insightfully mentioned the CI/CD and RevIN[1], it is important to discuss on these techiques:

• Previous works[2][3] have discussed the CI/CD tradeoff on model capacity and robustness. For example, CI can greatly increase the performance when data is scarce, while CD benefits from a higher capacity ideally. In the context of our work, we think it is essential to make the variates independent, especially when we know there are potential risks of delayed series and loosely organized measurements. Still, no matter CI/CD, multivariate correlations can not be explicitly utilized. Perpendicular to these works, iTransformer repurposes an architecture with the native Transformer modules to cope with it.
• RevIN or Series Stationarization[4] has been widely applied for the distribution shift (or non-stationarity) for real-world time series. They can boost the performance of both linear models and Transformers. There are Transformers in our baseline already equipped with this technique (PatchTST and Stationary), and we newly include the linear one, RLinear. These works strive for modeling temporal dependencies. In iTransformer, it is modeled naturally by the FFN and layernorm, and still leaves further improvement for us to tackle the distribution shift.
• Compared with linear forecasters, iTransformer is good at forecasting high-dimensional time series (that is, numerous variates, and complicated correlations). Under the univariate forecasting scenarios, iTransformer becomes in fact a stackable linear forecaster (FFN with the laynorm). For variate correlating, iTransformer keeps the variate independent and the attention module is on duty.

[1] RevIN: Reversible Instance Normalization For Accurate Time-series Forecasting Against Distribution Shift.

[2] The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting.

[3] Is Channel Independent strategy optimal for Time Series Forecasting?

[4] Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting.

Q7: Complete experiments to enhance the robustness of the results and reproducibility.

We extensively include the following experiments to our paper:

• To support $\underline{\text{Table 2}}$: promotion of the inverted Transformer on all datasets in $\underline{\text{Appendix F.1 of the revised paper}}$.

• To support $\underline{\text{Figure 5}}$: additional results of PEMS to show iTransformer can generalize on unseen variates in $\underline{\text{Appendix F.3 of the revised paper}}$.

Additional results of increasing lookback on Solar-Energy: increasing the lookback length lead to an U-curve of performance (change point in $336$ and $720$) on this dataset. We also provide the explanations in $\underline{\text{Q4 of Reviewer yopU}}$.

Q8: Clarify the protocol for the experiments trained with partial variates.

Different from the efficiency training strategy, which randomly chooses part of the variates in each batch, there is no randomness in variate selection in $\underline{\text{Figure 5}}$. We conventionally take the first $20\%$ variate of each dataset for training and use all for inference. So CI-Transformer and iTransformer use the same set of variates during the whole process of training. We have revised the part with a more rigorous description.

We also elaborate more on the the efficiency training strategy with more datasets added, and comprehensively test the model efficiency and performance in $\underline{\text{Appendix D of the revised paper}}$.

We would like to sincerely thank Reviewer HTy3 for providing a detailed review and insightful suggestions.

Q1: About the writing and figure revisions.

Thanks for your valuable suggestions. We conduct the following revisions of our paper:

• Misspelling: hardy -> hardly; Soloar-Energy -> Solar Energy.
• Abstract: "inverts the duties of the attention mechanism and the feed-forward network" -> "applies the attention and feed-forward network on the inverted dimensions". The detailed duties are elaborated in $\underline{\text{Section 3.2}}$.
• Figure 8: add the label of the x-axis.
• Abbreviate the Electricity dataset as ECL.

As for the fairness of $\underline{\text{Figure 1}}$, the results corresponds strictly to $\underline{\text{Table 1}}$. We set the maximum value of the radar as the best result of the benchmark for aesthetic reasons.

Q2: Avoid dataset confusion and be precise on ETT and PEMS datasets.

Thanks for your scientific rigor. To avoid confusion, we extensively test the performance of all four ETT subsets and the same four PEMS subsets as the SCINet. We report the averaged results of these subsets in $\underline{\text{Figure 1 and Table 1 of the revised paper}}$ to The detailed dataset descriptions are updated in $\underline{\text{Appendix A.1 of the revised paper}}$.

To be more precise and reproducible, we also report the standard deviations of iTransformer performance in five runs, indicating stable results, which are updated in $\underline{\text{Table 5 of the revised paper}}$.

Q3: The proof supports the potential risks of unaligned timestamps and measurements on specific data sets.

We provide a detailed analysis of the Traffic dataset in $\underline{\text{Appendix E.3 of the revised paper}}$. We find there are systematical delays in the road occupancy that each series describes, because the sensors are installed in different areas (imagine how a traffic jam influences the road occupancy of different areas).

Delay detection is the essential work that the MTSF model should cope with. However, it is because the vanilla Transformer embeds the localized time points that the obvious decay will be fused with noise.

Another support comes from the declined Traffic performance of the second and third rows in $\underline{\text{Table 3}}$, where attention is applied to the tokens of simultaneous time points. Since they do not reflect the same event, the performance can degrade a lot because of meaningless attention maps, unless the model has an enlarged respective field to capture the decay.

As for the unaligned measurements, it is common in time series forecasting, such as organizing together the different meteorological indicators (temperature and rainfall in Weather), or the quantity and proportion of the same observation (ILI dataset).

Q4: More detailed explanations about $\underline{\text{Figure 7(b)}}$ about the multivariate correlations.

We newly add detailed explanations about multivariate correlations in $\underline{\text{Appendix E.1 of the revised paper}}$. In a nutshell, the x- and y-axis place the variates, and the attention map can thus exhibit the Pearson Correlations among variates. We want to show that the attention maps can reflect the correlations of raw series (comparing the subplots of the same column), benefiting Transformer with enhanced interpretability. Besides, the correlations, which vary from the lookback window to forecast window, will be reflected by the attention maps that change accordingly (comparing the subplots of the same row).

Q5: The proof supports the fluctuation of PEMS.

We provide PEMS prediction visualization in $\underline{\text{Appendix E.2 of the revised paper}}$. It shows that PEMS has more fluctuating series variations compared with others. We highlight "PatchTST may lose focus on specific locality to handle rapid fluctuation" for that the patch length should be adaptively tuned for the series (such as frequency). Instead, iTransformer embeds the whole series as a token to describe the underlying process globally.

Thank you for the revision of the paper and replies to my review. As I already said in my review, the paper has enough material to be accepted and Authors applied some of my comment as well as other reviewers' comments, which is, in my humble opinion, "bullet-proofing" the paper even more.

But here are few comments based on the new version:

Data unaligned

I would just notify Authors that in the paper they mentioned dataset with "unaligned timestamps". However, in all the dataset Authors are using, timestamps are not unaligned. It is just the effect of specific events that is delayed, and it is only due to spatial nature of some dataset. Therefore, they should not say there are dataset with unaligned timestamp (because aligning variates for these specific events will results in misalignment for other timestamps) but rather talk about delayed or unaligned event.

Market Dataset

FYI, Authors did not provide the results of RLinear.

Protocol for experiments with partial variates

We conventionally take the first 20% variates of each dataset [...]

Then, my comment still stands. What would have been the results if Authors had taken the last 20% variates? or any other set of 20% variates. There is a risk that taking the first 20% variates is a specific scenario for which the transfer works correctly. Therefore, to be rigorous it would have been better to try various configurations and check if the performances vary a lot or if it is stable.

Regarding CD/CI/DistributionShift discussion

Discussions on the Channel-Independence/Dependence, Instance Normalization, and the linear models.

In my opinion, the comments of the Authors (and their opinion on this important topic) should be in the paper, even a condensed version. However, right now I can't see it in the revised version. But I do understand that the paper "format" (and page limitation) prevents to fully discuss and further investigate this topic.

Anyway, I'm convinced that the proposal of iTransformer will be an important milestone for Multivariate Time Series Forecasting, therefore no problem on my side for the new version of the paper.

Many thanks to Reviewer hg6h for providing thorough insightful comments.

Q1: The standard deviation of the results and the significance tests.

Thanks for your scientific rigor. We repeat each experiment five times with different random seeds. The standard deviations of iTransformer performance are updated in $\underline{\text{Table 5 of the revised paper}}$, which shows the performance is stable.

We also conducted the significance tests to compare iTransformer and the mentioned previous SOTA. We include all subsets of ETT and PEMS. The performance is also averaged from four prediction lengths with each containing five runs of random seeds. The standard deviations and test statistics are listed below, showing that the performance is on par with the previous SOTA PatchTST and SCINet within the margin of error.

To our best knowledge, the benchmark of time series forecasting has been excavated for a long time. And PatchTST and SCINet as the SOTA on specific datasets has surpassed concurrent works by a large margin, iTransformer can be on par with on their advantaged datasets while further go ahead on other chanllenging datasets (such as ECL and Traffic), and thus achieves the comprehensive SOTA.

Q2: Compare the model efficiency with linear models.

We newly measure the model efficiency in $\underline{\text{Figure 10 of the revised paper}}$, which comprehensively compares the training speed, memory footprint, and performance of the following models: iTransformer, iTransformer with our efficient training strategy and iTransformer with an efficient attention module; linear forecasters: DLinear and TiDE; Transformers: Transformer, PatchTST, and Crossformer. We have the following observations:

• The efficiency of iTransformer exceeds other Transformers in datasets with a small number of variates (such as Weather with $21$ variates). In datasets with numerous variates (such as Traffic with $862$ variates), the memory footprints are basically the same, but iTransformer can be trained faster.
• iTransformer achieves particularly better performance on the dataset with numerous variates, since the multivariate correlations can be explicitly utilized in our model.
• By adopting an efficient attention module or our proposed efficient training strategy on partial variates, iTransformer can enjoy the same level of speed and memory footprint as linear forecasters.

Q3: About the different TiDE results from its original paper.

We carefully read through the forecasting settings of the TiDE paper and its official repo. The differences are listed here:

• Enlarged lookback window: TiDE paper adopts the lookback length of $720$, while ours uses the length of $96$, following the unified long-term forecasting protocol of TimesNet.

• More epochs to train: TiDE paper trains the TiDE model with $100$ Epochs, while we reproduce the results of all models with $10$ epochs.

We try our best to keep hyperparameters comparable for the fairness of experiments, such as adopting the same hidden dimension and the number of layers. We also compare their performance under the same lookback length of TiDE, where iTransformer still achieves better results, since our model can also benefit from an enlarged lookback window.

Dear Reviewer hg6h,

Thanks for your valuable and rigorous review, which has inspired us to improve our paper further substantially. Following your suggestions, we have enhanced the robustness of experiments with the standard deviations of all datasets reported, newly tested the model efficiency comprehensively, and compared the training protocol with TiDE.

During the rebuttal, we received insightful reviews and valuable comments to improve our paper further. $\underline{\text{Reviewer HTy3}}$ highlighted our contributions and gave high comments. And $\underline{\text{Reviewers yopU}}$ raised the score as we have addressed the concerns. We hope that this new version can address your concerns to your satisfaction and notice our contributions. We eagerly await your reply and are happy to answer any further questions. We kindly remind you that the reviewer-author discussion phase will end soon. After that, we may not have a chance to respond to your comments.

Thanks again for your valuable review. Looking forward to your reply.

Authors

Many thanks to Reviewer yopU for providing a detailed review and insightful questions.

Q1: Describe the train-validation-test split in experiments.

We adopt the same train-validation-test split protocol as previous works, where the train, validation, and test datasets are strictly divided according to chronological order to make sure there are no data leakage issues. The data processing and split protocols are also included in $\underline{\text{Appendix A.1 of the revised paper}}$.

Q2: Do we implicitly assume that variate heterogeneity is more important than temporal dependency for forecasting?

We think they are both important to achieve good MTSF performance. However, the problem is that the heterogeneity of variates can hardly be considered in the vanilla Transformer. After embedding, the variates are projected into the channels of embedding. It ignores the problem of inconsistent physical measurements and can fail to maintain the independence of variates, let alone capture and utilize the multivariate correlation, which is essential for forecasting with numerous variates, and complicated systems driven by the latent physical process (such as meteorological systems).

In addition, even if the FFN and layernorm seem simpler than attention blocks, they are efficient and competent in learning the temporal dependency of a series, which can be traced back to statistical forecasters such as ARIMA and Holt-Winter. They also have no problems with inconsistent measurements since they work on the time points of the same variate, and have an enlarged respective field as the whole lookback series can be embedded as the variate token.

Q3: The reason for the not-so-good performance when applying attention to both temporal and variate dimensions.

As we observed the same declined performance of Traffic in the third row, the reason can be that the designs both apply the attention module to the temporal dimension. In our opinion, capturing temporal dependencies by attention is not a big problem. But it is based on the fact that the time points of each timestamp are essentially aligned.

We analyze the Traffic dataset in $\underline{\text{Appendix E.3 of the revised paper}}$, and we find there are potential risks of systematical delay among the road occupancy that each series describes, since the sensors are installed in different areas of the highway.

Consequently, applying attention to the time points of the same timestamp on such a dataset can make the model learn meaningless attention maps, unless the model has an enlarged respective field to capture the decay or causal process. It is one of the reasons that we advocate embedding the whole series as the variate token.

Q4: Explanations about the error jump when the input length increases from $336$ to $720$ in $\underline{\text{Figure 6}}$.

We have also noticed the problem that our approach embedding a large lookback series simply by MLP can be simple and coarse-grained. As we increase the lookback length by a large step, the input series inevitably contains more nonbeneficial information for future prediction, and the model capacity should also increase accordingly. It is instructive for us to develop fine-grained variate tokenization and well-designed embedding mechanisms.

Q5: Explanations about the effectiveness of our approach.

We add new sections to analyze the effects of the inverting:

• Analysis of multivariate correlations in $\underline{\text{Appendix E.1 of the revised paper}}$.
• Potential risks of the vanilla Transformer embedding in $\underline{\text{Appendix E.3 of the revised paper}}$.
• Full results of the iTransformers in $\underline{\text{Appendix F of the revised paper}}$ to support that our method can boost the Transformers' performance, and generalize on unseen variates.