Learned Data Transformation: A Data-centric Plugin for Enhancing Time Series Forecasting

[W3-It would be helpful to include Mean Squared Error (MSE) results in Table 4.]

We have included the full data transfer results, including MSE, in Table 14 in Appendix A.6. Across various datasets (e.g., Traffic, ETTh1/ETTh2 when using longer inputs), MSE is less stable than MAE and can even degrade in some cases, such as Traffic. This may stem from MSE's sensitivity to smaller values, particularly after standard scaling by mean and variance. We hope this addition clarifies the challenges of using MSE in these scenarios.

[W4-It seems inappropriate to claim that MoE is used without a gating network. Additionally, the method for selecting an appropriate number of PTNs, as well as the specific values used in the paper, is unclear.]

In Section 4.2, we described a design that decomposes time series into up to four sub-series, processed by individual "expert" models. These models transform their respective sub-series, and the outputs are concatenated to restore the original length, producing the transformed results.

Our approach shares two similarities with traditional MoE frameworks:

1. Traditional MoE uses hard-gating or soft-gating [4], with soft-gating assigning weights to expert outputs. Our concatenation of outputs is loosely analogous to a uniform weighting strategy in soft-gating.
2. MoE typically balances input distribution across experts, often via auxiliary loss [5]. We achieve a similar balance by manually controlling routing.

We acknowledge that the term "MoE" might cause confusion. A more accurate description would be "parallel version of PTN", reflecting its role in scaling PTN parameters. We will update the terminology in the final version of the paper.

Regarding the selection of the number of experts, we briefly addressed this in Appendix A.5, where we also discussed how MoE applied to data transformation could resemble a decomposition process. To clarify, the MoE-based variation was not employed in the main experiments but rather introduced as an optional acceleration strategy (not listed in our contributions). While not a primary focus, it offers an interesting direction for future research.

[4] From Sparse to Soft Mixtures of Experts

[5] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

[W5-There is no complexity analysis when adding the proposed model to the base models. ]

We have added a "Complexity Analysis" section in Appendix A.4.3 to discuss the complexity of PTN. By analyzing the core components of PTN, intra-patch and channel-wise attentions, we identified batch size as a previously overlooked factor in complexity analysis (see P1 in Appendix A.4.3). This led us to present more efficient implementations (see P2 and P3 in Appendix A.4.3) that limit parallel computations within a batch. Experimental results (Table 8, Table 9, Figure 10, Figure 11 in the revised paper) demonstrate these implementations reduce PTN's complexity to match that of linear models, enhancing its practicality.

These efficient implementations and the corresponding empirical studies are summarized as follows:

• Mask Fusion (P2 in Appendix A.4.3): We introduced a method to accelerate attention computations by merging intra-patch and channel-wise attention masks. This reduced complexity from $\mathcal{O}((l_p + C)LC)$ to $\mathcal{O}(l_p^2C^2)$, where $l_p$ (patch length) is as small as 4 in our experiments. Details and results supporting this method are in Appendix A.4.3, Figure 8, and Figure 9, showing its effectiveness in improving PTN's efficiency. A brief view of the results is shown in table T1 in "Comments to All" and in Figure 9 in Appendix 4.3.
• Variate Sampling (P3 in Appendix A.4.3): According to our new implementation, adopting the variate sampling technique proposed in iTransformer [1] can also lead to improvement in terms of both training and inference efficiency. The relative increase in time and memory costs can be reduced from several times to a factor of less than one. The results are reported in table T2 in "Comments to All" and in Tables 8 and 9 in Appendix A.4.3.

In addition to performance-neutral optimizations introduced above, there are also lossy acceleration methods. As discussed in Section 5.3, a data transfer method can improve training efficiency by avoiding the direct training of a complex backbone. As inspired by your question if raw is used in inference, although the answer is negative, we reconsider the possibility of using raw data directly as inputs. For inference, we, therefore, propose a method based on train-time distillation that generates a student model without PTN, incurring no additional inference overhead but requiring only the training of an extra backbone. Details are provided in Appendix A.4.4, along with experimental results shown below. However, this approach is not universally applicable and may fail in certain cases, such as the iTransformer on ETT datasets, for reasons outlined in response to W2. The brief results are shown in T3 in "Comments to All" and full results are shown in Table 10 in Appendix 4.4.

[1] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

[W6-There are also many unclear points and typos in the paper, such as:...]

Thank you for your feedback. Here are our clarifications and updates:

• Figure 1: The "7/8 part" refers to the weights within a window when computing normalization. When calculating $X-\bar{X}$ in a given window, the $i$-th normalized value is derived as $(-\frac{1}{n}, -\frac{1}{n}, \ldots, \frac{n -1}{n}, \ldots, -\frac{1}{n}) * (x_1, x_2, \ldots, x_i, \ldots, x_n)$ (* denotes element-wise product). This approach is unified in a Moving Kernel form with other methods. We have revised Figure 1 and its caption for improved clarity.
• Attention Outputs: The outputs of the two attentions are added together. Additionally, we have added Figure 8 in Appendix A.4.3 to present a different implementation discussed earlier.
• Input Data: For the standard version, we use transformed data as input. Inspired by your advice, we are also exploring the possibility of directly using raw data as input, which would incur no additional cost during inference (see Appendix A.4.4 "Efficient Student Model By Train-Time Distillation").

Finally, all other typos and errors have been corrected. Thank you for helping us improve the paper.

[W1-The comparison with baselines in Table 9 for look-back length 512 appears to be unfair. For instance, iTransformer should also use a look-back length of 512, and the results of PatchTST in this table are much worse than those reported in the PatchTST paper (PatchTST/64 in Table 3 of its paper).]

Thank you for raising this concern. We have updated the iTransformer results with a look-back length of 512 in Table 12 (previously Table 9). Notably, we observed a significant performance drop on the ETT datasets, likely due to overfitting. This could explain why the original iTransformer paper focuses on using a smaller look-back length of 96, as, perhaps, the architecture was not designed for it.

For clarity, here is a brief overview of the updates made in Table 12:

The results of PatchTST were impacted by a bug reported in [1, 2, 3], which is thoroughly discussed at https://github.com/VEWOXIC/FITS. After addressing and fixing this issue, we can confirm that our reported results are credible and reliable.

[1] FITS: Modeling Time Series with 10k Parameters

[2] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

[3] SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters

[W2-The proposed model does not seem to perform well on complex datasets, such as Traffic. It would be beneficial to provide results on more complex datasets, such as the PEMS datasets used in the iTransformer paper.]

We have now included the full results on the PeMS datasets in Table 14, located in Appendix A.6. For clarity, the output lengths differ from those reported in the iTransformer paper. Specifically, we use {12, 24, 36, 48} for our experiments, whereas the original paper lists {12, 24, 48, 96}. This discrepancy arises due to an error in their reporting, as detailed in https://github.com/thuml/iTransformer/issues/91 along with other discussions on reproducing the PeMS results (https://github.com/thuml/iTransformer/issues?q=is%3Aissue+pems). To ensure fair and accurate comparisons, we reran experiments for other backbone models to validate the results.

As briefly shown in table T3 in "Comments to All", our method shows notable improvement on PeMS datasets. iTransformer does not have consistent improvement compared to the other two backbones, which we attribute to its overfitting problem, as also explained in [6]. Additionally, as explained in Section 4.1 in the paragraph starting with "Training loss", our method may struggle in scenarios where the backbone's convergence is not guaranteed. This limitation helps explain iTransformer's occasional degraded performance, such as on PeMS04 and certain ETT datasets, as observed in the main experiments (now in Table 11 and Table 13).

[W2-Based on the results in Table 10, the PTN module appears to enhance performance primarily for simple linear models, while its effectiveness on more complex models, such as iTransformer and PatchTST, varies against the dataset. Considering that the main objective of this paper is to propose a general plugin, it is essential to select a sufficient range of predictors for experimentation.]

The results are now shown in Table 13 in the revised paper. We have also observed the differences in performance boosts from diverse angles. We are going to organize our analysis from aspects of backbones and datasets.

Varying performances on different backbones

As the reviewer pointed out, the proposed PTN module is more effective for linear models while having varying boosts for complex models. This is because of some of the intrinsic drawbacks in transformer-based models. As the paper [2] suggests, the transformers (both temporal and channel-wise) suffer from overfitting problems, especially on small datasets. In Section 4.1, the paragraph starting with "Training loss" explains that we do not use a strict constraint on gradient norms for consideration of convergence. Thus the revised loss does not guarantee reaching Pareto frontier and results in unstable performance for more complex models. For better generalization on more models and more stable performance, we plan to explore it in future work.

[2] SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

Varying performances on different datasets

Another observation is that the proposed models do not constantly improve performances over different datasets, we believe some of the cases are occasional and we add additional experiments on PeMS datasets as reviewer gMsX suggested. The detailed experiments on PeMS datasets are shown in Table 14 in the Appendix A.6, which shows the effectiveness of our methods on complex datasets. We also want to point out that the results in the iTransformer [1] paper are mistakenly described. The Github issue (https://github.com/thuml/iTransformer/issues/91) serves as a proof as well as other discussions that can not reproduce the PeMS results (https://github.com/thuml/iTransformer/issues?q=is%3Aissue+pems). We have rerun the baseline for more fair comparisons.

For the current work, we believe the instability is acceptable because this does not interfere with the improvement of SOTA methods on each dataset. For ETT datasets where linear models are better, the improvement is stable, and for the rest three datasets where transformer-based methods excel, we also have fair improvements.

We will add these explanations to our paper.

[W3-The article contains several errors that require careful proofreading. For example, "Encoder" in line 65 should be corrected to "Decoder," and the shape of the matrix in line 129 needs clarification. Additionally, there are concerns regarding Figure 2(b), where lraw decreases as lpred increases, which seems counterintuitive and requires further explanation.]

Thank you for pointing out these issues. We have carefully proofread the article and corrected the mentioned errors.

Regarding the concerns about Figure 2(b) and the relationship between $l_{raw}$ and $l_{pred}$, we appreciate the opportunity to clarify. Figure 2(b) illustrates that $l_{raw}$ represents prediction results measured on raw data, which are important but not directly used in training due to the transformation. On the other hand, $l_{pred}$ reflects prediction errors on the transformed data, and these two metrics do not necessarily exhibit a positive correlation. In fact, $l_{raw}$ is positively correlated with the sum of $l_{pred}$ and $l_{prox}$. As shown in Figure 2(b), when this sum cannot be further optimized, there exists an optimal allocation of the two losses that minimizes $l_{raw}$. We will revise the explanation in Figure 2 to ensure clarity and avoid any potential misunderstanding. Thank you again for the feedback.

[W1-When the predictor is a linear model, the complexity of PTN seems to far exceed that of the predictor itself. It is suggested to provide the time and memory complexity analysis of the proposed module. Additionally, for predictors that include modules capturing channel-wise and patch-wise correlations, the proposed PTN appears redundant, which may affect its generalizability.]

Below, we address the concerns regarding complexity and redundancy through comprehensive theoretical analysis and empirical validation.

Complexity

We have added a "Complexity Analysis" section in Appendix A.4.3 to discuss the complexity of PTN. By analyzing the core components of PTN, intra-patch and channel-wise attentions, we identified batch size as a previously overlooked factor in complexity analysis (see P1 in Appendix A.4.3). This led us to present more efficient implementations (see P2 and P3 in Appendix A.4.3) that limit parallel computations within a batch. Experimental results (Table 8, Table 9, Figure 10, Figure 11 in the revised paper) demonstrate these implementations reduce PTN's complexity to match that of linear models, enhancing its practicality.

These efficient implementations and the corresponding empirical studies are summarized as follows:

• Mask Fusion (P2 in Appendix A.4.3): We introduced a method to accelerate attention computations by merging intra-patch and channel-wise attention masks. This reduced complexity from $\mathcal{O}((l_p + C)LC)$ to $\mathcal{O}(l_p^2C^2)$, where $l_p$ (patch length) is as small as 4 in our experiments. Details and results supporting this method are in Appendix A.4.3, Figure 8, and Figure 9, showing its effectiveness in improving PTN's efficiency. A brief view of the results is shown in table T1 in "Comments to All" and in Figure 9 in Appendix 4.3.
• Variate Sampling (P3 in Appendix A.4.3): According to our new implementation, adopting the variate sampling technique proposed in iTransformer [1] can also lead to improvement in terms of both training and inference efficiency. The relative increase in time and memory costs can be reduced from several times to a factor of less than one. The results are reported in table T2 in "Comments to All" and in Tables 8 and 9 in Appendix A.4.3.

In addition to performance-neutral optimizations introduced above, there are also lossy acceleration methods. As discussed in Section 5.3, a data transfer method can improve training efficiency by avoiding the direct training of a complex backbone. For inference, we propose a method based on train-time distillation that generates a student model without PTN, incurring no additional inference overhead but requiring only the training of an extra backbone. Details are provided in Appendix A.4.4, along with experimental results shown below. However, this approach is not universally applicable and may fail in certain cases, such as the iTransformer on ETT datasets, for reasons outlined in response to W2. The brief results are shown in T3 in "Comments to All" and full results are shown in Table 10 in Appendix 4.4.

[1] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting Redundancy

In brief, we argue that the designs in PTN and the predictor are not redundant, as they optimize parameters using different losses: $l_{prox} + l_{pred}$ for PTN and $l_{pred}$ for the predictor, making their search space distinct. To address redundancy concerns, we have explored two questions in Section 5.4, "Ablation Study and the PTN Design":

Q1: Are the same "redundant" modules effective when used as a predictor?

A1: No, they are not. When we adapt these modules into a predictor without transforming input $X$ (merging transformations on $Y$ and predictions), the performance significantly drops, as shown in Table 5 ("ConvPred").

Q2: How does performance change when removing the "redundant" designs?

A2: As shown in Figure 6 in the revised paper (note: mislabeled figures have been corrected), the impact of attention is minimal, and the choice of attention depends more on datasets than backbones.

While there may be side effects, such as reduced generalizability due to similar architectures between PTN and backbones, we cannot conclude the redundancy based on the evidence provided. We will further explore this issue in the future.

[W1-The paper's abstract and introduction do not clearly convey the overall research idea and process, making it difficult to understand the framework.]

To put it briefly, the core idea of our paper is to learn an effective transformation of raw data to enhance a forecasting model's performance compared to using the original data. Unlike traditional methods that rely on handcrafted data transformation techniques such as instance normalization, data augmentation, and preprocessing like down-sampling and patching, we propose the use of a learnable neural network. This network, trained in an end-to-end manner, is termed the Proximal Transformation Network (PTN). As the name suggests, PTN aims to generate transformed data closely resembling the original while enabling improved model performance. The proximity loss acts as a regularization term to reinforce the generalization on the raw data. Additionally, predictability is assessed using a standard loss, like Mean Squared Error (MSE), computed between the label and the model's predictions, both derived from the PTN-generated transformed data. The motivation of PTN is intuitive, but how to learn such an effective transformation is non-trivial.

In our revised paper, we have now made clearer explanations on these aspects in the 2nd and 3rd paragraphs of the Introduction. Moreover, we've revised Figure 3 with workflow markers to aid in understanding the framework. Please also see our detailed response to W3 below.

[W2-The authors mention that PTN shows potential to make time series forecasting more interpretable. However, the enhanced embedding is latent, produced by deep learning. It is unclear how this actually enhances interpretability.]

We had discussions on the interpretability of our PTN model in Section 5.2 "Interpretation of Proximal Transformation", focusing on the following two aspects.

1. In Figure 4, we demonstrated that the transformed data generated by PTN show distinct clustering patterns in the loss space, which can be correlated to the predictability of the tested time series. We would like to clarify such visual analysis is performed on the raw and transformed data, without involving the latent embedding space.
2. In Figure 5, we depicted how all transformed time series evolve to resemble the raw time series during training (see 'raw' and 'transformed' in Figure 5 (a) and (c)). Again, the interpretability here is discussed based on raw and transformed data, not embeddings. The only instance involving "the latent embedding" is in Figure 5 (d), where we examine the efficacy of the employed Convolution Encoder. By using the same "Point-wise Linear Head" to decode the Encoder's output for each variate, we confirmed that the Convolution Encoder manages to align the distribution of different variates, in comparison with RevIN shown in Figure 5 (b). This analysis examines the utility of the intermediate component of our model, specifically the Encoder.

Overall, our interpretability analysis primarily focuses on the data before and after transformation. We will make this point clearer in the revision.

[W3-The authors provide a caption for the framework, but it does not help clarify the framework's procedure. It's unclear how the transformed embedding is involved in enhancing the prediction task. I suggest that the authors include a framework overview to illustrate the entire process.]

Following the suggestion, we have revised the framework in Figure 3. To be specific, we've rearranged the module positions and incorporated arrows with different colors and numbered labels to clearly denote the two sequential steps: proximal transformation and prediction. The caption now includes a detailed description of the entire pipeline to improve clarity. Additionally, we have included a new Figure 7 in Appendix 4.2, which provides a flow chart illustrating the operation of PTN and the predictor during both training and inference processes.

Essentially, the PTN framework functions as an end-to-end data transformation system, generating transformed time series for any backbone predictor model to train on. Importantly, no embeddings are used directly for prediction with the backbone model.

[W3-Why the performance is not stable compared with baselines?]

The results are now shown in Table 13. We have also observed the differences in performance boosts from diverse angles. We are going to organize our analysis from aspects of backbones and datasets.

Varying performances on different backbones

As the reviewer pointed out, the proposed PTN module is more effective for linear models while having varying boosts for complex models. This is because of some of the intrinsic drawbacks in transformer-based models. As the paper [2] suggests, the transformers (both temporal and channel-wise) suffer from overfitting problems, especially on small datasets. In Section 4.1, the paragraph starting with "Training loss" explains that we do not use a strict constraint on gradient norms for consideration of convergence. Thus the revised loss does not guarantee reaching Pareto frontier and results in unstable performance for more complex models. For better generalization on more models and more stable performance, we plan to explore it in future work.

[2] SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

Varying performances on different datasets

Another observation is that the proposed models do not constantly improve performances over different datasets, we believe some of the cases are occasional and we add additional experiments on PeMS datasets as reviewer gMsX suggested. The detailed experiments on PeMS datasets are shown in Table 14 in the Appendix A.6, which shows the effectiveness of our methods on complex datasets. We also want to point out that the results in the iTransformer [1] paper are mistakenly described. The Github issue (https://github.com/thuml/iTransformer/issues/91) serves as a proof as well as other discussions that can not reproduce the PeMS results (https://github.com/thuml/iTransformer/issues?q=is%3Aissue+pems). We have rerun the baseline for more fair comparisons.

For the current work, we believe the instability is acceptable because this does not interfere with the improvement of SOTA methods on each dataset. For ETT datasets where linear models are better, the improvement is stable, and for the rest three datasets where transformer-based methods excel, we also have fair improvements.

We will add these explanations to our paper.

[W1-What is the motivation of losses in section 3.2? ]

In short, we question if the raw data are "good" enough to train a TSF model. If not, how can we find more data with higher chances of being "good" for a model? Specifically, we use $l_{pred}$ to measure how good the data are and $l_{prox}$ to guide the search for improved data.

Noise in time series data is common and often unidentifiable. When a model encounters high loss on certain samples, it is unclear whether the issue lies in the model’s inability to learn patterns or the presence of noise. If the noise could be removed, the model would learn better. Therefore, we want to shift the original time series to see if the model can fit the data better and hopefully predict better. As shown in Figure 2, we begin with arbitrary data transformations and gradually approach the original data, effectively moving along $l_{prox}$ from larger to smaller values. During the process of "moving along the axis", we have different sets of data (in theory we have infinite sets). We can train a predictor to see if it can fit the data well on the current position on this axis by measuring their performance with $l_{pred}$, which indicates how they predict on the transformed data. This way, we gradually approach the transformed series and simultaneously train our predictor. Figure 2 (b) validates our assumption that a predictor can learn better with transformed data. In vanilla training, with fixed $l_{prox}$ as 0, the predictor can only learn suboptimal results. Whereas our method relaxes the constraints on the objective (from a sample $y$ only to a set of $\tilde{y}$ proximal to $y$) to train better and more robust predictors.

We will emphasize the motivation and make relevant presentations clearer.

[W2-Please study the time and space complexity of the proposed model. ]

We have added a "Complexity Analysis" section in Appendix A.4.3 to discuss the complexity of PTN. By analyzing the core components of PTN, intra-patch and channel-wise attentions, we identified batch size as a previously overlooked factor in complexity analysis (see P1 in Appendix A.4.3). This led us to present more efficient implementations (see P2 and P3 in Appendix A.4.3) that limit parallel computations within a batch. Experimental results (Table 8, Table 9, Figure 10, Figure 11 in the revised paper) demonstrate these implementations reduce PTN's complexity to match that of linear models, enhancing its practicality.

These efficient implementations and the corresponding empirical studies are summarized as follows:

• Mask Fusion (P2 in Appendix A.4.3): We introduced a method to accelerate attention computations by merging intra-patch and channel-wise attention masks. This reduced complexity from $\mathcal{O}((l_p + C)LC)$ to $\mathcal{O}(l_p^2C^2)$, where $l_p$ (patch length) is as small as 4 in our experiments. Details and results supporting this method are in Appendix A.4.3, Figure 8, and Figure 9, showing its effectiveness in improving PTN's efficiency. A brief view of the results is shown in table T1 in "Comments to All" and in Figure 9 in Appendix 4.3.
• Variate Sampling (P3 in Appendix A.4.3): According to our new implementation, adopting the variate sampling technique proposed in iTransformer [1] can also lead to improvement in terms of both training and inference efficiency. The relative increase in time and memory costs can be reduced from several times to a factor of less than one. The results are reported in table T2 in "Comments to All" and in Tables 8 and 9 in Appendix A.4.3.

In addition to performance-neutral optimizations introduced above, there are also lossy acceleration methods. As discussed in Section 5.3, a data transfer method can improve training efficiency by avoiding the direct training of a complex backbone. For inference, we propose a method based on train-time distillation that generates a student model without PTN, incurring no additional inference overhead but requiring only the training of an extra backbone. Details are provided in Appendix A.4.4, along with experimental results shown below. However, this approach is not universally applicable and may fail in certain cases, such as the iTransformer on ETT datasets, for reasons outlined in response to W2. The brief results are shown in T3 in "Comments to All" and full results are shown in Table 10 in Appendix 4.4.

[1] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting