TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting

Many thanks to Reviewer ixvB for providing the insightful review and comments.

Q1: Add Scaleformer, MTS-Mixers, TSMixer in comparison.

In the original submission, we compared TimeMixer with 15 baselines. Following the reviewer's request, we have included Scaleformer, MTS-Mixers, and TSMixer into comparison.

As we stated in the $\underline{\text{Section 4 of main text}}$, we experiment with these baselines with their official code but align the hyperparameters to ensure a fair comparison. All the results have been included in the $\underline{\text{Appendix H of revised paper}}$. Here are some key experiments.

In addition to the quantitive results, showcase comparisons have also been added in $\underline{\text{Appendix J of revised paper}}$ for an intuitive comparison.

Q2: Complete the ablation studies to prove the efficacy of asymmetric design.

In the original submission, we conducted experiments on M4 and ETTh1, which are two representative datasets in long- and short-term forecasting. Thanks for the reviewer's suggestion, we have completed the ablations in all the benchmarks. Besides, we also add a new ablation setting, where we adopt the bottom-up mixing for trend and top-down mixing for the seasonal part.

Overall, we have conducted 10 types of ablations in all 18 benchmarks. Here are part of the main results. The complete results have been included in $\underline{\text{Table 5 and Table 15,16,17 of revised paper}}$. In all benchmarks, our official design performs best consistently, which can verify the efficacy of our asymmetric design.

It is also notable that completely reversing mixing directions for seasonal and trend parts leads to a serious performance drop. This may come from that the essential microscopic information in finer-scale seasons and macroscopic information in coarser-scale trends are ruined by unsuitable mixing approaches.

Dear Reviewer,

We kindly remind you that it has been 3 days since we posted our rebuttal. Please let us know if our response has addressed your concerns.

Following your suggestion, we have answered your concerns and improved the paper in the following aspects:

• We have added 3 new baselines (Scaleformer, MTSMixer, TSMixer) in all 18 benchmarks to demonstrate the advancement of TimeMixer. Both quantitive results and showcases are provided in the revised paper.
• We can complete 10 types of ablations in all 18 benchmarks, which verifies the effectiveness of our design in TimeMixer. The implementation details of each ablation have also been included.
• We have also provided the implementation details for the visualization experiments.

In total, we have added more than 100 new experiments. All of these results have been included in the $\underline{\text{revised paper}}$.

Thanks again for your valuable review. We are looking forward to your reply and are happy to answer any future questions.

Q2: Concerns about evaluations in Figure 4.

The original motivation of Figure 4 is to "provide an intuitive understanding of the forecasting skills of multiscale series", which means that our experiment subject is the learned multiscale features, not predictors. Thus, we think our previous fix-PDM-retrain-FMM design can verify this point.

But as per your request, we have also provided the case of directly visualizing the results of each predictor in $\underline{\text{Appendix A (Figure 7) of revised paper}}$ as a supplement, which is also intuitive in verifying the different forecasting capabilities of series in different scales.

But since we use the sum ensemble, the scale of each predictor output is around $\frac{1}{M+1}$ of real series. More discussions are included in the Q3.

Q3: Are there any intuition, theoretical reasons, or empirical results for this design?

Firstly, we would like to highlight that in our design, the loss is calculated based on the ensembled results, not each predictor, that is $MSE(\mathbf{x},\hat{\mathbf{x}})$=$MSE(\mathbf{x},\sum_{m=0}^{M}\hat{\mathbf{x}}_{m})$.

Thus, if you change the ensemble strategy as "average", the loss will be $MSE(\mathbf{x},\hat{\mathbf{x}})$=$MSE(\mathbf{x},\frac{1}{M+1}\sum_{m=0}^{M}\hat{\mathbf{x}}_{m})$. Obviously, the difference between "average" and "mean" is only a constant multiple.

Theoretical intuition: It is common sense in the deep learning community that deep models can easily fit constant multiple. For example, if we replace the "sum" with "average", under the same supervision, the deep model can easily fit this change by learning the parameters of each predictor equal to the $\frac{1}{M+1}$ of the "sum" case, which means these two designs are equivalent in learning final prediction under the deep model aspect. And we do expect the reviewer can think about this design from the deep learning perspective.

Empirical results: Following your suggestion, we also provide the empirical results here. We can find that the performances of these two strategies are very close.

These results have also been included in $\underline{\text{G.4 of revised paper}}$.

Thanks again for your detailed and prompt response, which is very important to us. All the results for Q2 and Q3 are inclued in $\underline{\text{revised paper highlighted in purple}}$. The new results for Q1 will come soon.

Q1-Part 2: "Ablation studies (Table 5) in long-term forecasting have to include the results with a large search space of the input length, such as in Table 14."

In Table 5, we provide the long-term forecasting results of ETTm1 in the input-96-predict-336 setting. Following the reviewer's suggestion, we also conducted the ablations on input-336 (searched input length). Here are the results. We can find that under the input-336 configuration, the relative promotion is more significant than the smaller input setting.

As for the "noise in finer-scale seasons" that the reviewer mentioned, we would like to recall that the bottom-up mixing layer will downsample the seasonal information, and the downsample operation can play a denoising role in principle. However since we attempt to propose a practical model, the theoretical proof is left as future work. In fact, it is hard to find strict proof for deep models. We can only provide the intuitive understanding and empirical results (extensive ablations that we have provided).

Q1-Part 3: "Also, TSMixer, MTSMixer, and Scaleformer have to be compared to TimeMixer in Table 14."

Following the reviewer's request, we reproduce these baselines with their official code and conduct a comprehensive hyperparameter search, where TimeMixer still performs best. Here are the results. (See $\underline{\text{Table 23 of revised paper}}$ for full results)

During the rebuttal period, following the instruction from the reviewer, we have done our best to provide double experiment results for both settings for scientific rigor. We hope our effort in rebuttal (more than 200 new results in total, 9 new pages, 7 days experiment with 32 A100 GPUs) can resolve the reviewer's concerns.

Many thanks for your response.

Inconsistency with TSMixer offical results comes from the training epochs.

As for the comparison with TSMixer, we would like to highlight that the difference is from the training epoch, where we only employ 10 training epochs due to the time limitation in rebuttal, while TSMixer employs 100 epochs. Note that training 10 epochs is a convention in this area.

Beisdes, as we always emphasized, we have made a great effort to ensure fair comparison. The unified hyperparameter setting results are also important, given many works have followed this setting, such as Scaleformer, MTSMixer, TimesNet.

About ablations.

Since many preivous papers only provide the input 96 settings and we have provided ablations in both input settings strictly following your suggestions, we do hope the reviewer can take the double workload into consideration.

Overall, we have made our best to ensure fair compairson. Hope the reviewer can reconsider the score.

Dear Reviewer,

Many thanks for your valuable review. With the deadline for the author-reviewer discussion phase drawing near (less than 1 hour), we wish to ensure that our response sufficiently addressed your concerns. After this hour, we may not have a chance to answer your question.

For clearness, we summary our revision that we made towards your concerns here.

(1) New baselines: we have added 3 baselines that you mentioned in all 18 benchmarks under both unified and search hyperparameter settings.

(2) New ablations: we have completed 10 types of ablations in all 18 benchmarks under the unified hyperparameter setting and provided the requested ablations in two specific settings faithfully following the reviewer's request.

(3) New visualizations: we have provided visualization for FMM in two types of configurations and clarified that ensemble strategies that the reviewer mentioned are equivalent in the deep model perspective.

In total, we have provided extensive experiments (more than 200 new results, 9 new pages) to address every concern that you proposed. Hope our effort can address your concerns to your stastification and looking forward to your reply.

Dear Reviewer,

Thanks again for your valuable and constructive review, which helped us improve the paper in completing ablations and comparisons. All the updates have been included in the $\underline{\text{revised paper highlighted in purple}}$.

We kindly remind you that the reviewer-author discussion phase will end in 3 hours. May we know if our response addresses your main concerns? After that, we will not have a chance to respond to your comments.

Sincere thanks for your dedication! We eagerly await your reply.

First of all, I want to express my gratitude for your effort.

I read all of the authors' responses and the discussions with other reviewers.

However, there are three points where I am reluctant to raise the point.

1. Actually, I found that the performance scores of TSMixer in your paper are different from those of TSMixer officially reported in [1] whereas PachTST is the same in both cases. This inconsistency is quite critical in believing the scores in the paper.

2. Furthermore, when comparing the scores of TimeMixer to those of TSMixer reported in [1], I don't think there is a large margin between them. At this point, I don't know that the module the authors proposed is truly required to boost forecasting performance when considering that TimeMixer is similar to TSMixer without some methods.

3. Finally, in this case where performance improvement is mediocre, I believe that more ablation studies are required to emphasize the effectiveness of asymmetric design.

Because of these points, I decide to uphold the score as before. (Still, I respect the different opinions of other reviewers.)

[1] Chen et al., TSMixer: An All-MLP Architecture for Time Series Forecasting, 2023

We would like to sincerely thank Reviewer ansv for providing the valuable feedback.

Q1: "A deeper dive into this comparative analysis would have been enlightening. Introducing spectral analysis or similar methodologies might offer theoretical insights that bridge this understanding gap."

Thanks for the reviewer's valuable suggestion. We have provided the spectral analysis in the $\underline{\text{Appendix I of revised paper}}$. It is observed that TimeMixer outperforms other baselines significantly in capturing different frequencies.

Q2-part 1: "How does TimeMixer's Past Decomposable Mixing fundamentally differentiate itself from frequency-based methods like the Discrete Fourier Transformer (DFT)?"

(1) Ablations on decomposition methods

In this paper, we adopt the moving-average-based season-trend decomposition, which is widely used in previous work, such as Autoformer, FEDformer and DLinear. Following the reviewer's suggestion, we also try the DFT-based decomposition as a substitute. Here we present two types of experiments.

• DFT-based high- and low-frequency decomposition: We treat the high-frequency part like the seasonal part in TimeMixer and the low-frequency part like the trend part.
• DFT-based season-trend decomposition: We replace the moving average with DFT-based seasonal part extraction, which the most significant frequencies are extracted as season. Then the rest is trend.

The results are shown as follows, which has also been included in $\underline{\text{Appendix G.2 of revised paper}}$.

We can have the following observations:

• Replacing the season-trend decomposition with high-low-frequency decomposition does not bring performance gain. Since we only explore the proper mixing approach for the former decomposition in the paper, the bottom-up and top-down mixing strategies may be not suitable for high- and low-frequency parts. New visualizations like $\underline{\text{Figure 3 and 4 of original submission}}$ are expected to provide insights to the model design. Thus, we would like to leave the exploration as the future work.

• Enhancing season-trend decomposition with DFT perfroms better than moving average. However, since moving average is quite simple and easy to implement with PyTorch, we eventually chose the moving-average-based season-trend decomposition in TimeMixer, which can also achieve a favorable balance between performance and efficiency.

(2) Discussion about season-trend decomposition and DFT-based methods

Firstly, we would like to highlight that we propose TimeMixer towards a simple but effective method, which we have already emphasized in the Introduction section. In the spirit of building a practical model, proving that seasonal-trend decomposition is optimal or fundamentally different from other decomposition methods is out of the scope of our paper.

But thanks for the reviewer's suggestion. We would like to leave this discussion of the optimality and completeness of our model as future work, which has been added to the $\underline{\text{Appendix F of revised paper}}$.

Q2-part 2: "Could there be alternatives to average pooling that might perform better? How can the authors demonstrate that average pooling is indeed the optimal choice?"

As per the reviewer's request, we replaced the average pooling with 1D convolutions with stride as 2. Here are the results. we can find that the complicated 1D-convolution-based outperforms average pooling slightly. But considering both performance and efficiency, we eventually use average pooling in TimeMixer.

Again, we just want to build a practical model. Although average pooling may be not optimal, it obviously an effective and easy-to-implement choice. The above ablation has also been included in $\underline{\text{Appendix G.3 of revised paper}}$.

We would like to sincerely thank Reviewer P2SD for providing a detailed review and insightful suggestions.

Q1: "The descriptions 'Up-Mixing' and 'Down-Mixing' in Figure 2 may not be appropriate and should perhaps be reversed."

Sorry for these two inaccurate formalizations. Following your suggestion, we have rephrased 'Up-Mixing' and 'Down-Mixing' to 'Bottom-Up-Mixing' and 'Top-Down-Mixing' in both figures and text of the $\underline{\text{reviserd paper}}$.

Q2-part1: "The paper lacks significant innovation. In my view, the author's main contribution seems to be the integration of these two modules from MICN and Autoformer."

(1) TimeMixer is clearly distinct from MICN and Autoformer.

Firstly, TimeMixer is a MLP-based model, while MICN is convolution-based and Autoformer is Transformer-based.

Secondly, although multiscale analysis and decomposition have been used in previous models, we would like to highlight that their usage in TimeMixer is distinct from MICN and Autoformer in both motivation and technical design, which is summarized in the following tables:

We do appreciate that MICN and Autoformer introduce multiscale analysis and decomposition into deep time series forecasting. But obviously, these three models are quite different in both model architecture and usage of the aforementioned two modules.

(2) The design in adopting different mixing directions for decomposed parts should not be overlooked.

In the paper, we propose to utilize top-down and bottom-up mixing directions for seasonal and trend parts respectively, which is a key contribution in our model. This design is well supported by:

• Motivation analysis in $\underline{\text{3rd paragraph in Introduction}}$ and $\underline{\text{Section 3.2}}$.
• Ablations in $\underline{\text{Table 5}}$, which verifies that our proposed mixing approach performs best.
• Visualizations in $\underline{\text{Figure 3}}$, which demonstrates that seasonal and trend parts should utilize different mixing directions.

Thus, TimeMixer is far beyond simple integration of multiscale analysis and decomposition. To highlight this point, we have rephrased the listed contributions in the $\underline{\text{Introduction of revised paper}}$.

Q2-part 2: "The so-called FMM module appears to be essentially a linear layer mapping the concatenated multiscale features back to the prediction length in the time dimension."

(1) There might be some misunderstandings of FMM (Future-Multipredictor-Mixing).

As presented in $\underline{\text{Equation 6 of main text}}$, we adopt different predictors to regress the future from past features in different scales separately.

Thus, we think the review might be misconceived in the following two points:

• We do not concatenate multiscale features, but keep the past information in their original length and directly regress the future separately. (This is also different from the multiscale design in MICN.)
• FMM is not a single linear layer mapping, but an ensemble of multiple predictors.

For clearness, we have rephrased $\underline{\text{Section 3.3 in the revised paper}}$ to clarify the misleading parts.

(2) FMM can utilize complementary forecasting capabilities in multiscale series, not just align the time dimension.

As we presented in $\underline{\text{Figure 4 of original submission}}$, different scales present complementary forecasting capabilities, which is why we adopt multiple different predictors in FMM. To our best knowledge, this point is firstly explored by TimeMixer, which is also an important design in our paper.

Q3: "The improvement in experimental performance is marginal, especially in three larger datasets, 'Traffic', 'Weather,' and 'Electricity'."

We believe that to measure the value of a deep model, we should consider $\underline{\text{performance, hyperparameter tuning cost and efficiency}}$ simultaneously.

(1) In this unified hyperparameter setting, TimeMixer surpasses other baselines significantly.

As shown in $\underline{\text{Table 2,3,4 of original submission}}$, TimeMixer is clearly better than the previous state-of-the-art PatchTST, SCINet, and TimesNet in both performance and efficiency. This advantage is meaningful since sometimes we do not have enough time or sources for hyperparameter searching in real-world applications.

(2) In the hyperparameter-search setting, TimeMixer is much more efficient than the second-best model PatchTST.

As pointed out by the reviewer, after a comprehensive hyperparameter-search, the relative improvement of TimeMixer against PatchTST is smaller than the unified hyperparameter setting. But we have to highlight that, TimeMixer still holds a significant advantage in efficiency, making it a valuable model to this community and applications.

To make this clearer, we list some results from $\underline{\text{Table 8 and Table 14 of original submission}}$ as follows. As presented in $\underline{\text{Table 6 of original submission}}$, Solar-Energy is the largest dataset and Weather is the second. We can find that TimeMixer surpasses PatchTST with 25% and 7.8% relative promotion in these two large benchmarks even after the hyperparameter searching. Besides, TimeMixer is over 2 times faster than PatchTST. In addition, we also present the results of DLinear, which is comparable to TimeMixer in efficiency. But TimeMixer beats it with more than 10% performance gain.

These analyses have also been included in the $\underline{\text{Appendix E of revised paper}}$.

Dear Reviewer,

Thanks for your valuable and constructive review, which has inspired us to improve our paper further substantially. This is a kind reminder that it has been 3 days since we posted our rebuttal. Please let us know if our response has addressed your concerns.

Following your suggestions, we have provided the following revisions to our paper:

• Elaborate the difference between TimeMixer and MICN, Autoformer in using decomposition and multiscale analysis.
• Revise Figure 2 and text of the main text to resolve inaccurate formalizations.
• Revise the listed contributions in the $\underline{\text{Introduction section}}$ to highlight our contributions in proposing new mixing directions for decomposed components and utilizing multiple predictors for multiscale series.
• Revise the $\underline{\text{Future-Mixing section}}$ to clarify our design in Future-Multipredictor-Mixing block.
• Highlight that although under hyperparameter searching, the relative promotion of TimeMixer is smaller than the unified hyperparameter setting, TimeMixer is 2x faster than the second-best model PatchTST.

In this paper, we propose the TimeMixer as a simple but effective model and provide extensive experiments, visualization, and ablations to support our insight. All the revisions are included in the $\underline{\text{revised paper}}$.

Sincere thanks for your dedication! We are looking forward to your reply.

Dear Reviewer,

Thanks again for your valuable and constructive review, which helps us revise our contribution and advantages to a clearer stage.

We kindly remind you that the reviewer-author discussion phase will end in 24 hours. After that, we may not have a chance to respond to your comments.

Besides, during the rebuttal period, we also completed the ablations in all 18 benchmarks (more than 100 new experiments), which may be helpful to you in further justifying our contribution in PDM and FMM. All of these results have been included in the $\underline{\text{revised paper}}$.

Sincere thanks for your dedication! We are looking forward to your feedback.