CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns

Thank you for your valuable comment!

W1: The notations are not strictly consistent. The notations for instance normalization (in Section 3.2 and Algorithm 2) are independent from the whole framework and not consistent with the previous problem definition.

Thanks for pointing this out. We will correct it to make the notations in the paper more consistent. In the original version of our paper, we used simpler notations (e.g., $x_{in}$, $y_{out}$) in Section 3.2 and Algorithm 2 to help readers better understand the model's input-output workflow. However, this indeed led to inconsistency in the notations. Therefore, we will properly revise these in the revised paper.

Q1: Since Figure 2 depicts the entire CycleNet architecture, instance normalization should also be included.

Thanks, and we will include instance normalization in Figure 2 to present the complete workflow of CycleNet.

Q2: The description of alignment and repetition of Q (Lines 152-157) can be further improved. The link to Algorithm 1 should be added, and a schematic figure (even in Appendix) would make it clearer to understand.

Thanks for your suggestion. In our submission, due to space limitations in the main text, we adopted a concise writing style. We will further elaborate on the alignment and repetition of $Q$ in the revised paper, including more textual explanations in the main text, adding the link to Algorithm 1, and most importantly, supplementing with an intuitive schematic figure to help readers better understand this process (as shown in Figure 1 of the attached pdf).

Q3: How extreme points affect RCF in the Traffic dataset should be further explained (Line 255). Extreme points can also affect other methods. It is unclear why this would specifically deteriorate RCF.

The reason behind this is indeed complex. Firstly, the basic fact is that there are indeed some outliers with extremely high values in the Traffic dataset. For instance, the 12630 value of the 857th channel in the Traffic dataset, after global standard normalization, still exceeds 25, while the normal range of values for surrounding points is [-2, 2]. Secondly, the fundamental working principle of RCF is to learn the historical average cycles in the dataset.

In such cases, the average cycles learned in RCF can be affected by these significant outliers, such as the mean of a certain point in the cycle being exaggerated. Consequently, in each prediction process, the original sequence subtracts a locally exaggerated average cycle, resulting in an inaccurate residual component, thereby affecting the local point predictions within each cycle. The more inaccurate these local point predictions are, the larger the discrepancy between MSE and MAE, as MSE significantly amplifies the impact of a few large errors. This explains why in Table 4, combining iTransformer with RCF decreases MAE but increases MSE, indicating overall prediction accuracy improvement but anomalies in local point predictions.

Therefore, models like iTransformer and GNN, which accurately model inter-channel relationships, are more suitable for scenarios with extreme points and temporal lag characteristics. For example, when a sudden traffic surge occurs at a certain junction, these models, having correctly modeled the spatiotemporal relationships, can accurately predict possible traffic surges at other junctions. In contrast, the current CycleNet only considers single-channel relationship modeling, thus being somewhat limited in this scenario.

However, the core of CycleNet still lies in exploring a more effective periodic modeling approach and proposing a model that balances performance and efficiency. We believe that further addressing the issue of RCF being affected by extreme points and investigating how to incorporate channel relationship modeling within CycleNet in future work would be highly promising. We will include these discussions in the revised paper.

Thank you again for your careful review, and we hope our responses address your concerns.

Thank you for your kind and careful review!

W1: In the introduction, the author's statement establishes a close relationship between long-term prediction and periodic information. Etc.

In fact, periodic information is indeed one of the most important factors for achieving long-term forecasting. Recent works have demonstrated that by effectively utilizing periodic information, it is possible to achieve near state-of-the-art prediction accuracy with fewer than 1,000 parameters [1]. This strongly underscores the importance of periodic information in long-term forecasting tasks. Additionally, the popular work DLinear has shown that a single-layer linear model can outperform many well-designed models in long-term forecasting tasks [2]. This is because a simple linear layer can robustly extract periodic information from the historical data of a single channel (evident from the clear periodic patterns in the weight distribution of these linear models) [3].

Without periodicity, long-term forecasting becomes very challenging. For instance, DLinear shows that on financial datasets, even the most advanced deep learning models cannot outperform simply copying the most recent data point [2].

Thus, existing evidence highlights that the presence of periodicity in data is crucial for accurate long-term predictions. Our paper builds on this premise, exploring a simple yet effective method of leveraging periodicity through RCF. We acknowledge that periodic information is not the sole factor for accurate time series predictions. Other factors like short-term trends, multivariate dependencies, and seasonality are also critical, especially for short-term forecasting tasks. However, the core of this paper focuses on improving long-term forecasting performance by better utilizing periodicity in data. Therefore, this motivation and contribution here may be not in conflict with scenarios where other factors are more influential.

W2: The authors select data sets with different periodicity to illustrate the validity of the model. However, the authors only demonstrate the validity, and experiments can be added to show how the proposed method performs differently on periodically different data sets and discuss the underlying rules. At the same time, it is also worth showing how CycleNet performs on data sets where there is no obvious periodicity.

Sorry if we misunderstood your first point here. We have already shown the performance of our model on datasets with different cycle lengths in Table 5 and Figure 3 of our submission, indicating that when the model's hyperparameter $W$ is set correctly, the RCF technique significantly improves prediction accuracy, and the trainable recurrent cycle $Q$ can accurately learn the corresponding cycle patterns.

We assume that your question is about how the proposed method performs differently with different cycle lengths and we have supplemented Figure 2 (m-p) in the attached PDF to visualize the learned recurrent cycle $Q$ when setting different cycle lengths $W$. We found that when $W$ is correctly set to 144 (the weekly cycle length of the Electricity dataset), $Q$ learns the complete cycle pattern, including weekly and daily cycles. When $W$ is set to 24 (the daily cycle length), $Q$ captures only the daily cycle. With $W$ set to 96 (four times the daily cycle), $Q$ learns four repeated daily cycles. However, with $W$ set to 23 (with no matching semantic cycle length), $Q$ learns nothing, resulting in a flat line. For datasets without obvious periodicity (e.g., Exchange-Rate dataset), the behavior of $Q$ is similar, learning a flat line. We will further include these results in the revised paper.

W3: Combined with the results in Table 2 and Table 4, the results predicted with Linear alone are even better than some well-designed methods, whether this is due to the particularity of the data set or some other reason.

The results in Table 2 and Table 4 cannot be directly compared because Table 2 reports results averaged across all prediction horizons $H ∈ {96, 192, 336, 720}$, whereas Table 4 presents results for specific horizons individually.

However, if we average the results from Table 4, the Linear model does indeed outperform some well-designed models, such as Autoformer. This phenomenon can be traced back to the DLinear paper [2], which demonstrated that a purely linear layer can outperform many well-designed Transformer models at the time. DLinear's success is attributed to its channel-independent approach for multivariate forecasting, where each channel is modeled with a shared linear layer. This approach allows the model to focus on extracting historical information from individual channels, leading to more robust periodic information for long-term forecasting (as evidenced by the striped patterns in the learned weights of its linear layers). Previous models like Autoformer mixed information across multiple channels, making it difficult to extract robust periodic information. Since DLinear, many models have adopted the channel-independent approach for long-term forecasting, including PatchTST, FITS, SparseTSF, and our CycleNet.

Therefore, it is reasonable that a simple linear model can outperform some well-designed models in this context.

[1] Lin, Shengsheng, et al. "SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters." In International Conference on Machine Learning, 2024.

[2] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023.

[3] Toner, William, and Luke Darlow. "An Analysis of Linear Time Series Forecasting Models." In International Conference on Machine Learning, 2024.

Thank you again for your kind review, and we hope our response can address your concerns.

Thank you for your valuable comment!

W1: It seems that the proposal (CycleNet) does not work well for complex datasets, e.g., Traffic. It is better to show more results on the same kind of datasets like the PEMS datasets used in the iTransformer paper.

The current CycleNet did not achieve SOTA on the Traffic dataset because it is a simple single-channel modeling method that uses the proposed RCF technique to achieve a balance of performance and efficiency. Its backbone for prediction is only a single Linear layer or a dual-layer MLP. For scenarios like Traffic, which exhibit spatiotemporal characteristics and temporal lag characteristics, more complex multi-channel relationship modeling is indeed required.

As we know, traffic scenarios may experience sudden surges in flow (i.e., extreme points), and the surge at one intersection can affect the flow at other intersections over time. The table below shows the average number of extreme points per channel, counted with Z-Score > 6:

We can see that the Traffic dataset has significantly more extreme points than other datasets. In this case, models that fully model inter-channel relationships, like iTransformer, can accurately predict the flow at other intersections after an extreme point appears. In contrast, univariate models or over-parameterized multivariate models are less capable in such situations. This is why in Table 2, iTransformer significantly outperforms other models as it can accurately predict the flow at other intersections after an extreme event.

However, it is also notable that, aside from iTransformer, CycleNet still significantly outperforms other models. Especially, this result is achieved with CycleNet's backbone being simple linear and MLP layers. Therefore, returning to CycleNet's core contribution, which is exploring a simpler yet effective use of periodicity and establishing a method that balances performance and efficiency, it is undoubtedly successful.

Additionally, based on your suggestion, we have added results of CycleNet on the PEMS datasets. Here, it is essential to clarify that although PEMS and Traffic are both transportation datasets, PEMS has significantly fewer extreme points than Traffic:

Thus, in this case, CycleNet is expected to perform better on PEMS than on Traffic. Below are the MSE comparison results of CycleNet and other models in the scenario where the lookback length is 96 and the prediction horizon is 12:

In this case, CycleNet/MLP performs on par with iTransformer, which models multi-channel relationships. Even the CycleNet/Linear, with a single linear layer as the backbone, outperforms other deep nonlinear models. Therefore, these results further validate the effectiveness of the proposed RCF technique and demonstrate that the CycleNet model achieves a balance of performance and efficiency. We will include these analyses and full experimental results in the revised paper.

W2: The Section 3 is not well written and lacks a lot of details, especially about the Learnable Recurrent Cycles. The authors may consider reorganizing Section 3 and Appendix A.1 to make Section 3 clearer.

Thank you for pointing out this issue. We will carefully optimize Section 3 in the revised paper to make it clearer for readers. Additionally, we have supplemented it with a schematic figure in Figure 1 of the attached pdf, which more intuitively describes the workflow of the recurrent cycle $Q$. We will add this schematic figure to Section 3 of the revised paper.

Thank you again for your valuable review, and we hope our response can address your concerns.

Thank you for the responses. Could you give the full results on the PEMS dataset where the lookback length is 96 and the prediction horizon is {12, 24, 48, 96} (only the results of CycleNet, iTransformer and SCINet are OK).

*Sorry, the prediction horizon should be {12, 24, 48, 96} instead of {12, 24, 36, 48} according to Table 9 in iTransformer's paper.

Thank you for your detailed and thoughtful review!

W1: Lack of clarity of the source of results.

We will clarify this in the main text of the revised paper. Previously, we clarified this in Appendix A.4.

W2 & W3 & Q1: Baseline: Add more appropriate baselines to correctly position the RCF technique.

Thank you for your reminder.

(i) Leddam (LD) and SparseTSF are both accepted papers at ICML 2024. We timely noted the latter because it emphasizes the importance of periodicity, similar to our work. LD upgrades the Moving Average kernel (MOV) technique in STD to a weights-learnable module, while SparseTSF uses sparse techniques to decompose sequences. We will compare these two techniques with our proposed RCF technique.

(ii) TFDNet extracts features in the Time-Frequency domain after using STD techniques to decompose sequences. Since it fundamentally uses regular MOV-based STD techniques, it may not be necessary to compare it directly here.

(iii) RLinear: We will add a comprehensive comparison with it in the main text. In fact, the Linear and MLP in Table 4 are combined with RevIN, so they represent RLinear and RMLP, and CycleNet significantly outperforms them. Additionally, we thoroughly evaluated the impact of RevIN on CycleNet in Appendix Table 8.

In summary, we compared CycleNet/Linear (RCF+Linear), LDLinear (LD+Linear), DLinear (MOV+Linear), SparseTSF (Sparse+Linear), and Linear in Table 1 of the attached pdf. To ensure fairness, we did not use RevIN since DLinear originally did not use it.

It can be observed that:

(i) CycleNet significantly outperforms other methods, proving the superiority of RCF as a new STD technique.

(ii) As a sparse prediction method, SparseTSF may rely more on longer lookback length and RevIN, thus performing poorly here.

(iii) Most surprisingly, the performance of LDLinear and DLinear is almost identical in our setup. This result differs from the results reported in the LD paper (Table 4). Other researchers have also noted this phenomenon on the LD project's GitHub (Issue #1), but have not yet received a response from the authors. To some extent, LD is essentially a weight-trainable MOV, so it is hard to achieve high performance gains compared to the original MOV. We are not sure if there is any mistake here, so we will further confirm with the authors after the double-blind review phase.

Q2: Visualization: Add more plots under different configurations.

We have added these visualizations in Figure 2 of the attached pdf. The basic configuration is Electricity-Lookback=96-Horizon=96-Backbone=Linear-W=168. It can be observed that:

(i) Horizon: The learned patterns remain almost unchanged as the horizon changes. This indicates that the horizon length does not affect the learned pattern results.

(ii) Lookback: The overall pattern remains unchanged as lookback changes. However, upon closer observation, it can be seen that the learned pattern becomes smoother with increased lookback. This is because a longer lookback provides the backbone with richer periodic information, thereby reducing the importance of the learned pattern component.

(iii) Backbone: The patterns change somewhat with different backbones. When DLinear is the backbone, the learned patterns are smoother, as DLinear's decomposition technique itself extracts certain periodic features. When iTransformer is the backbone, the learned patterns differ more, as it additionally models multichannel relationships, so the learned periodic patterns may consider multichannel feature interactions. PatchTST's performance is more similar to Linear, as it is also a regular single-channel modeling method, only with stronger nonlinear learning capabilities compared to the Linear model.

(iv) Cycle length $W$: When $W$ is 144 (the weekly cycle length for the Electricity dataset), the recurrent cycle $Q$ learns the complete periodic pattern, including weekly and daily cycles. When $W$ is set to 24 (the daily cycle length), the recurrent cycle $Q$ only learns the daily cycle pattern. When $W$ is set to 96 (four times the daily cycle length), the recurrent cycle $Q$ learns four repeated daily cycle models. However, when $W$ is set to 23 (without matching semantic meaning), the recurrent cycle $Q$ learns nothing, i.e., a straight line.

Q3: Discussion and other points.

Very sorry that the response here cannot cover every point you have mentioned due to the text length limitation. We will carefully incorporate your suggestions into the revision (e.g., providing more results of other datasets on Table 4 and Figure 4), and we can further discuss any uncovered points during the discussion phase.

Q4: Additional points: The computation cost of RCF on other Backbones.

In fact, the cost of RCF is fixed and independent of the Backbone. The primary additional fixed overhead is the parameter quantity of $D×W$, and the CPU time required for alignments and repetitions of the recurrent cycles (~10 seconds per epoch in our experimental environment). Overall, this additional overhead is very small compared to the other deep backbones. We will supplement this quantitative analysis in Table 3 (i.e., the additional overhead introduced by RCF). By combining these overheads and the ablation results of other backbones in Table 4, it can be inferred whether the gain in accuracy justifies the increase in complexity and associated cost.

Q5: Proof-read.

Thank you, and we will revise these points in the revised paper.

Again, thank you for your thorough review, and we hope our responses address your concerns.

Other researchers have also noted this phenomenon on the LD project's GitHub (Issue #1), but have not yet received a response from the authors.

I was not aware of the existing issue with LD. It is good to include these results in your final revision and mention the existing reproducibility issue with LD. Such information will be helpful for future readers.