CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting

We would like to express our sincere gratitude to Reviewer fhNE for their positive assessment of our work, specifically acknowledging the comprehensiveness of our experiments and the proposed structure. Your recognition of the significance we attributed to the appendix experiments is truly gratifying. We want to assure you that we are fully committed to addressing and resolving any concerns you have raised.

Q1:Model complexities/Running time

As we use the patching trick, the order of the total complexity would be the same as PatchTST and Crossformer, which is $\mathcal{O}(L^2/S^2)$. Some Transformer type models (e.g., FEDformer and Autoformer) may even break the quadratic dependent in $L$ and reach linear or nearly linear complexity in $L$. Other CNN and RNN type models (e.g., TimesNet and FilM) by nature maintain the $O(L)$ complexity. When $S$ is set as a not-very-small number (e.g. $S\approx O(\sqrt{L})$), our model's complexity can also be nearly linear. The condition $S\approx O(\sqrt{L})$ is not very restrictive. Take $L = 900$ as an example, the length of $S=30$ would be enough. For the case $L = 96$, the corresponding $S$ would be around $10$.

The results of the detailed experiment on running time are reported in the following table. We use the TimesNet's codebase. The input/forecasting lengths are set as 96/96 and we keep the batch size the same for all benchmarks and run the experiments on a single A100/80G GPU. Our proposed model yields comparable running time to transformer baselines as well as linear complexity baselines except Dlinear, which implies in practice model could also behave like a linear time model and won't introduce overhead computational cost.

Q2:Inconsistent value

We sincerely apologize for the typographical error. The correct value for wo.hidden weather(720) should be 0.351, instead of 0.341, and the average should be 0.242. This mistake occurred due to inadvertently typing a 4 instead of a 5. We will diligently verify all the numbers to ensure accuracy and regret any confusion caused by this error.

Q3:Model performance without Robust loss

As depicted in the subsequent table, the model itself is capable of achieving state-of-the-art (SOTA) performance using the original MSE loss. The introduction of the robust loss proposed in this paper further enhances the performance. Moreover, it is noteworthy that for the traffic dataset, the original MSE loss yields significantly better results. Therefore, it is important to highlight that the primary driver behind the observed improvements is the proposed model itself, rather than relying solely on the choice of the loss function.

Q4:Theoretical analysis for mitigating overfitting issues

Thanks for your suggestion. In this paper, we mainly introduce the exponential smoothing (EMA) module and weighted robust loss to mitigate the overfitting.

We believe the EMA part would use a weighted mean to average out some noise. We may use the following simple example to illustrate it. Let $\{x_1,....,x_{t}\}$ be sequences of i.i.d $\sigma^2$ sub-Gaussian variables and Let $f(t) = (1-\beta)\sum_{i=1}^{t}\beta^{t-i}x_i$ with some $\beta\in (0,1)$. One may verify $f(t) = (1-\beta)x_{t} + \beta f(t-1)$, which implies $\{f(t)\}$ is the sequence of $\{x_t\}$ after EMA. Via the Hoeffiding's inequality, there exist some constant $C>0$ such that with $1-\delta$, we have:

$

|f(t)-f(t-1)| \le \sigma\sqrt{(1-\beta)^2\left(1+\beta^2\right)C^{-1}\log\left(\frac{1}{\delta}\right)}.

$

If we set $\beta \to 0$, $f(t)\to x_t$ and above inequality shows the divergence between $x_t$ and $x_{t-1}$ would be $O(\sigma)$. When change $\beta$ toward $1$, the divergence decreased on the order of $O(\sigma^2\sqrt{1-\beta)^2(1+\beta^2)})$, which is faster than $O(\sigma)$. It implies that if the sequence $\{x_t\}$ contains high-frequency noise, the smoothed sequence $f(t)$ will have a higher chance of wiping it out and it eventually stabilizes the attention module. We will add those discussions into the final version.

Another design to tackle the overfitting is the weighted loss. In Appendix H of the paper, we use toy case with input length 2 to show the proposed loss may give better estimation accuracy.

However,we have to admit that performing end-to-end theoretical analysis for the proposed deep model can be very complicated and beyond the scope of this paper.

We would like to express our sincere gratitude to Reviewer eTmJ for providing a positive evaluation of our work's intriguing and challenging contributions. We are truly encouraged by the thorough and well-summarized strengths section that you have provided. We greatly appreciate your detailed and insightful comments, as they have been instrumental in helping us improve our work. Rest assured, we are fully committed to addressing any concerns you have raised.

Q1: Learnabe decay weight VS Fixed decay weight

The results of the learnable decay weight experiment are presented in the following table. The numerical outcomes are found to be very similar to those obtained with the fixed decay weight. However, it is important to note that the training speed could be significantly slower when using the learnable decay weight requiring up to 50% extra training time. Due to this time-consuming aspect, we have chosen to utilize a fixed decay weight in our approach.

Q2: About the short-term forecasting M4 benchmark

Following the discussion with reviewer ajLB on short-term forecasting using the M4 dataset, we have decided to relocate this section to the supplementary information as an initial exploration. In exchange, we will incorporate relevant ablation study findings from the supplementary information into the main draft. We concur with reviewer eTmJ's assessment that relying solely on the M4 dataset for short-term forecasting is inadequate, especially considering the issue of different input lengths for baseline models. We acknowledge that our incorporation of the settings proposed in the recent Timesnet paper lacked a deeper analysis. It is crucial to recognize that as the M4 dataset is a competition dataset, most participants leverage ensemble methods to enhance their performance. However, introducing ensemble methods would significantly complicate the problem and deviate from our main research focus. Additionally, we acknowledge that certain baseline models in the M4 dataset employ different input lengths due to the competition's lack of requirement for uniformity. To strengthen the soundness of our findings, we intend to explore alternative datasets for the short-term forecasting task, an aspect we plan to address in our future work.

Q3: Dynamic projection with Large channel datasets like Traffic and ECL

Thanks for your suggestion. We've added the results of Weather, ECL, and Traffic datasets under the same setting in Appendix N.2. The projecteddimension varies in 1,8,16. The results are summarized in the following table. We have observed similar performance behavior, suggesting that altering the dynamic projection dimension does not significantly impact the forecasting accuracy.

Q4: Ablation for structures to verify the difference between CARD and PatchTST.

From the model structure perspective, our model has two major differences: attention over hidden and channel-aligned modules, which are trying to address the PatchTST potential two drawbacks: ignorance of the cross-channel information and lack of usage within patch information. In Table 18, we examine the performance changes when removing those two structures and verify the effectiveness of those two parts. Moreover, we have also included our ablation analysis for architecture variants in the appendix, specifically in Figure N.1, which visually demonstrates the differences. If they are more engaging for the readers, we can certainly move this analysis to the main draft. The only reason we did not include it here initially was due to limited space constraints.

Q5: Data size for transformer

The purpose of this experiment is to challenge the claim made in Dlinear that the transformer model cannot benefit from more data. We believe this claim to be an overstatement, as we demonstrate that even a slight increase in dataset size, from 70% to 100%, can result in performance improvement. Although we could show even larger improvements with an even smaller dataset size, we deliberately avoid getting into a debate about whether a 10% data size is too small and leads to underfitting the model. We acknowledge that these findings are preliminary. To truly test whether the transformer model follows a power law accuracy curve, we would need a much larger dataset. Our goal is simply to highlight that the claim made in Dlinear is underexplored at the very least, and we hope to encourage further investigation within the research community.

We extend our sincere appreciation to Reviewer gMyA for providing a positive evaluation of our methodology's design and the clarity of our visualized explanation. We are particularly grateful for the opportunity you have given us to elaborate on the two experimental settings and address the related question, as we have invested significant time in investigating this particular issue. Please rest assured that we are fully committed to addressing and resolving any concerns you have raised.

Q1: Some of the main design ideas are presented and verified by the previous work, such as channel independence and etc.

In our paper, we did not utilize channel independence. Instead, our primary objective is to develop a transformer model that effectively harnesses channel-dependent (CD) information. This focus is stated in the abstract, where we introduce the CARD model's channel-aligned attention structure that aims to capture dependencies among multiple variables over time, and other parts of the paper also echo this statement (e.g., summary of contribution, Figure 1, and Section 3.3).

The core innovation of our paper lies in addressing an important research question: how to construct an efficient transformer for CD settings. In existing literature, CD transformers, such as Autoformer and FEDformer, generally underperform compared to channel-independent (CI) transformers like PatchTST. Various experiments by Han et al., 2023 suggest that enhancing the robustness of CD transformers is crucial. We therefore introduced the EMA module to improve the model's stability and considered a robust loss design. Additionally, to maintain reasonable computational cost and align the information within patch, dynamic projection and dual attention are introduced respectively. While we agree with the reviewer that similar structures may have been explored in deep learning literature, we believe such a combination is new in CD transformer design in the time series forecasting domain. Our focus is not on claiming the invention of these structures but on their effective combination to improve CD transformer performance, which we consider a significant novelty of our research.

Moreover, we want to highlight our robust loss. In recent literature (e.g., Autoformer, FEDformer, and PatchTST), the major point is trying to design new model structures but not focusing on training procedures. In this paper, we analyze the variance-varying issue in multi-step forecasting and propose a weighted loss. This relatively simple modification not only enhances the performance of our model but also improves other models, as demonstrated in Table 4. To our best knowledge, this loss design is also new in recent transformer time series forecasting literature.

Q2: Comparision setting: fixed short input or better performance

Perhaps we didn't clarify it enough, which may have led to confusion for reviewer gMyA. We want to highlight that there are two distinct experimental settings for long-term forecasting benchmarks in our study. The first approach, used by Autoformer, FEDformer, Timesnet, etc., involves a fixed input length of 96 and forecasting for all horizons. These models compare accuracy across different algorithms, and we refer to this approach as "fixed short input". The complete table of results for this particular setting can be found in Appendix E, specifically in Table 8.

On the other hand, models like PatchTST, Dlinear, Crossformer, and some other baselines focus more on the final result by using a much longer input length. Some of them even tune the input length to achieve better performance. As the reviewer correctly noted, PatchTST reports results for input lengths of both 336 and 512 in their main table. However, it is worth mentioning that PatchTST also presents results for an input length of 720 in its paper appendix table 9, although it is not as good as the 336 and 512 cases.

We refer to them as the "better performance" approach. The complete table of results for this particular setting can be found in Appendix F, specifically in Table 10.

We believe that both approaches are valid and contribute to testing different aspects of the algorithms. One approach evaluates the forecasting ability with a limited input length, while the other tests the ability to achieve better performance using longer inputs. Therefore, we have included results for both settings. The reason for using an input length of 720 is that we achieved better performance in this hard setting length, whereas PatchTST's performance slightly deteriorated. However, we didn't want to lower the reported performance of the baseline models, so we included their best performance setting which was for an input length of 512, to align with the "better performance" approach.

We hope that this clarification makes our approach clearer. If reviewer gMyA still prefers the results on 336/512 input length, we are also more than happy to include them in the final version.

Thanks for your response, I think the method is more complicated than the current SOTA method, and the comparison between the main experiment and SOTA method uses unofficial code and does not align the experimental settings. At this stage, I will modify my score to 5.

Dear Reviewer gMyA,

Thanks for your valuable feedback. While we believe that our analysis using two different settings offers a more comprehensive understanding, we are more than happy to provide more results that align with the settings used in PatchTST. We hope these results will address your concerns regarding the comparison to PatchTST.

As mentioned in the Appendix, our 720 experiments use the codebase in Nie et al., 2023, i.e., the official PatchTST code. We have modified the input length from 720 to 336/512 to further align with the settings presented in Table 3 of the PatchTST paper. Due to resource limitations, we currently only completed the experiments on 5 out of the 7 datasets, and the results are summarized in the following tables:

In both 336 and 512 settings, CARD gives very promising performance. Additionally, the average results across four different forecasting lengths show that CARD outperforms PatchTST in all experiments with 512 input length and only underperforms in one metric with 336 input length.

Q3: Adding ILI and Exchange datasets

We have included their results in the following table. Although we did run these two datasets and found that the card's performance was good, we were hesitant to consider them as suitable benchmarks. The ILI dataset is both small and highly diverse, with normalized predicted MSE values close to 2. We are uncertain whether the relatively good performance on such a dataset can be extrapolated to larger and more practical datasets in real-world scenarios. Similarly, when it comes to the Exchange dataset, its nature as a financial dataset governed by free-market principles makes it extremely challenging to make accurate predictions. The signal-to-noise ratio in this dataset is significantly low, posing challenges for short-term forecasting tasks, and rendering long-term forecasting even more arduous, if not seemingly impossible. For example, all baselines in TimesNet paper fail to beat even the simple repeat forecasting reported in Zeng et al. 2023. In the final version, we will include them in the main table.

We express our sincere gratitude to Reviewer ajLB for providing a positive assessment of the architecture and empirical contributions of our work. Your belief in the significance of our findings, especially in comparison to the claims made by previous work, serves as great encouragement for us. We deeply appreciate the detailed and insightful feedback you have provided, and we are fully committed to addressing your concerns.

Q1: Including some early baselines

We highly concur with the reviewer's suggestion that certain early baselines have not been thoroughly examined. Notably, algorithms like nbeats and even persistent models exhibit excellent performance across certain datasets. The reason for excluding them in this paper lies solely in the limitation of available space, as we were apprehensive that some reviewers may raise concerns regarding the omission of recent works. We have run the experiment on Nlinear, Linear, and Repeat models in our experimental settings and summarized the results in the following table. Our model consistently outperforms those baselines. We assure you that we will incorporate these baselines in a comprehensive table, presented in smaller font size, in the refined version of our manuscript.

Q2: About the short-term forecasting M4 benchmark

After careful consideration, we align with the reviewer's evaluation. In order to test the M4 dataset, we have incorporated the setting proposed in the recent Timesnet paper. However, it is worth noting that as M4 is a competition dataset, most participants employ ensemble methods to enhance performance. Notably, the Nbeats algorithm has exhibited exceptional performance on this dataset, because of its 180 models ensembling trick. The original paper introduces structural and parameter variations to diversify the base model, optimizing it for ensembling. The success of ensembling depends on two crucial factors: the ensembling method itself and the choice of baseline algorithms. It is important to note that different ensembling methods can be better suited for different datasets and may facilitate varying baseline algorithms. Although incorporating such a comparison would undoubtedly introduce complexity, we acknowledge the need to rerun and construct an ensemble framework for all baseline models, including the card method proposed in our study. However, it is important to note that even with this comparison, ensuring absolute fairness may not be guaranteed, as a particular ensemble method might inherently favor a specific baseline model. We recognize that this may deviate from our original narrative, as a model that excels in ensembling may not necessarily outperform others on its own. Ensembling relies heavily on the diversity among baseline models. Additionally, as raised by another reviewer, eTmJ, it is important to acknowledge that certain baseline models in the M4 dataset utilize different input lengths due to the competition's lack of requirement for uniform input lengths. Consequently, we believe your suggestion is highly fitting and thoughtful. We propose relocating this portion to the supplementary information, treating it as a preliminary exploration, and highlighting the significantly improved performance of algorithms like Nbeats when utilized within an ensemble framework.