Parsimony or Capability? Decomposition Delivers Both in Long-term Time Series Forecasting

Thank you for your thoughtful review and constructive comments.

---

W1: We chose the ECL and Traffic datasets for our experiments primarily because they present more challenging tasks due to their larger data scale. Specifically, these datasets include a mass of variables and cover an extensive data collection period. This increased data scale results in a less biased estimate of the impact of each module, providing a more comprehensive evaluation of our model's performance. To address the concern regarding the exclusion of the ETT dataset, we have conducted an additional ablation study using the ETTh1 dataset. The results from this study are provided below for your reference. This inclusion further validates the robustness and generalizability of our approach across different dataset scales and complexities. We will improve this study in the revision.

---

W2: We will include the model size in Table 1 in the final version of the paper.

---

W3: We have conducted a comparative analysis between SSCNN and MTSF models based on LLM. Due to the time constraint, we directly copy the results obtained by baselines from an ICML 2024 publication [1]. The results, which are reported at the end of the reply box, demonstrate SSCNN's advantages in both performance and efficiency. This analysis strengthens our claim that SSCNN is a highly competitive model in the field of time series forecasting. We will include this comparative analysis in the final version of the paper to provide a more comprehensive evaluation of SSCNN's capabilities.

---

W4: We appreciate your concern, which prompts us to emphasize the key contributions of SSCNN that might otherwise be underestimated. As discussed in the last paragraph of Section 2, many lightweight models achieve a reduction in parameters, however, at the cost of sacrificing prediction accuracy. In contrast, SSCNN manages to maintain strong competitive performance with comparably reduced parameters. To substantiate this claim, we offer an additional comparison between SSCNN and some representative lightweight models, including FITS, highlighting the balance SSCNN strikes between model complexity and prediction accuracy. Due to the time constraint, we directly borrow the results obtained by baselines from an ICML 2024 publication [2]. This comparison will be included in the supplementary materials of the final version, showcasing SSCNN's advantages in terms of both efficiency and effectiveness in time series forecasting.

[1] Bian, Yuxuan, et al. "Multi-patch prediction: Adapting llms for time series representation learning." arXiv preprint arXiv:2402.04852 (2024).

[2] Lin, Shengsheng, et al. "SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters." arXiv preprint arXiv:2405.00946 (2024).

Thank you for your thoughtful review and constructive comments.

---

W1: We acknowledge the importance of including 720-step predictions. In our experiments, we observed that the ranking of models for 720 output steps differed only slightly from the ranking for 336 output steps. This suggests that the models maintain consistent performance over long-term horizons. To support this observation, we conducted additional experiments with SSCNN and selected baselines on the ECL, Traffic, and ETTh1 datasets for output steps within 96, 192, 336 and 720.

---

W2: To provide a clearer understanding of the SSCNN architecture, we will enhance Figure 1 with more explanatory information. Here's an overview: (a) Embedding: Raw data points are independently mapped to a high-dimensional representation space using a shared 1x1 convolution operator across all variables and horizons. (b) In-sample Inference: The historical time series representations are processed by three temporal attention-based normalization (T-AttnNorm) layers with different instantiations of the attention map ($\mathcal{I}$), followed by a spatial attention-based normalization (S-AttnNorm) layer. This process yields estimations of four structured components ($\mu$) along with the residual ($R$). (c) Out-of-sample extrapolation: The decomposed components corresponding to each variable are individually extrapolated to future horizons using the matrix multiplication (MatMul) operator, with distinct instantiations of the attention map ($\mathcal{E}$). (d) Component fusion: Finally, all four components, combined with the residuals, are input into a polynomial regression layer to capture their complex interrelations.

Additionally, we will ensure that the font styles in Figure 1 are consistent with the main text in the final version.

---

W3: We appreciate your attention to these issues. To address them:

(a) We will include a clear definition and explanation of the operator $\lfloor . \rfloor$, which denotes the floor function, in the paper.

(b) Regarding Figure 3(a), we will correct the labeling error by replacing “HDformer” with “SSCNN”.

(c) We acknowledge the issue with the text size of the legends in the figures. We will adjust the text size to ensure consistency with the main text, thereby enhancing the readability and overall presentation quality of the paper.

Thank you for your helpful feedback.

Thank you for your thoughtful review and constructive comments.

---

W1: We have addressed the concerns raised in your questions. Please kindly refer to the response to Q1 and Q2 below.

---

W2: We have conducted a comparative analysis between SSCNN and MTSF models based on LLM [1] [2]. Due to the time constraint, we directly copy the results obtained by baselines from an ICML 2024 publication [1]. The results, which are reported at the end of the reply box, demonstrate SSCNN's advantages in both performance and efficiency. This analysis strengthens our claim that SSCNN is a highly competitive model in the field of time series forecasting. We will include this comparative analysis in the final version of the paper to provide a more comprehensive evaluation of SSCNN's capabilities.

[1] Bian, Yuxuan, et al. "Multi-patch prediction: Adapting llms for time series representation learning." arXiv preprint arXiv:2402.04852 (2024).

[2] Jin, Ming, et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." The Twelfth International Conference on Learning Representations.

---

Q1: The Polynomial Regression layer depicted in Figure 1 is inspired by the work of [3], where we extend the original module to include both additive and multiplicative relations. This extension allows us to model more complex interactions between the decomposed components. We will provide a more detailed explanation of the Polynomial Regression layer in the methodology section, including its structure and rationale. Additionally, we will cite relevant references to enhance the clarity and contextual understanding of this approach.

[3] Deng, Jinliang, et al. "Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting." IEEE Transactions on Knowledge and Data Engineering (2024).

---

Q2: Thank you for highlighting this point. The decision to exclude an attention mechanism from the long-term component was made after careful consideration. Although it can be incorporated, as we have done with the seasonal, short-term, and spatial components, we found that it brought little to no gain in forecasting accuracy across the datasets we analyzed. To clarify, the attention mechanism is beneficial when a component exhibits a significantly evolving distribution over time. It reduces estimation bias by assigning higher weights to data points with higher correlation. However, for the long-term component in our case, the distribution tends to be stable throughout the input period. In such scenarios, the attention mechanism does not contribute additional value, rendering it redundant and unnecessary. We will include this explanation in the main text of the final version to enhance the coherence of our methodology and improve reader comprehension.

---

Q3: The 4x4 blocks in Figure 1 are used to exemplify the selection maps $\mathcal{I}^*$ and $\mathcal{E}^*$ as defined in the main text, with $T_{in}$ and $T_{out}$ both instantiated as 4. To enhance clarity, we will explicitly label these blocks in the figure to indicate that they are examples of the associated selection maps. Additionally, we will address the inconsistent text formatting, such as the use of E, to ensure consistency between the figure and the main text.

Thank you for your thoughtful review and constructive comments.

---

W1: Thank you for highlighting this important point. We apologize for not providing sufficient context on the four components used in our model. Disentangling these components has been shown to be effective for time series forecasting in numerous studies [1] [2] [3] [4]. Our approach builds on these established methods with a novel selection mechanism, which provably reduces estimation bias for the considered components. Our method advances the performance of decomposition-based approaches to the SOTA level, which can be achieved by only large-scale models before our work. The fact that our model achieves this with only four components suggests that, regardless of model complexity, the representation power of these large-scale models does not exceed the scope covered by these four components. As you pointed out, this raises a critical question for future research: are there additional components that could be modeled to further improve forecasting accuracy, especially for longer sequences? Currently, this question remains open and unanswered. We will discuss this issue and provide relevant references at the beginning of Section 3 in the revised version, ensuring a more comprehensive and informative explanation.

[1] Cleveland, Robert B., et al. "STL: A seasonal-trend decomposition." J. off. Stat 6.1 (1990): 3-73.

[2] Wen, Qingsong, et al. "RobustSTL: A robust seasonal-trend decomposition algorithm for long time series." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.

[3] Deng, Jinliang, et al. "Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting." IEEE Transactions on Knowledge and Data Engineering (2024).

[4] Wang, Shiyu, et al. "TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting." The Twelfth International Conference on Learning Representations (2024).

---

W2: Thank you for your insightful suggestion. Our primary aim is to demonstrate the pioneering predictive capability of a purely decomposition-based model with an exceptionally low parameter count—less than 1% of what large-scale models require. The current architecture leverages the statistical characteristics of the dataset of interest, ensuring that the design is not arbitrary. In particular, our analysis of correlation, autocorrelation, conditional correlation, and conditional autocorrelation—visualized in Figure 7—guided our decisions regarding the architecture's configuration and critical hyper-parameters, such as cycle length and the connection between the variables. While these decisions are based on our judgment, they can be automated through a Python script, a straightforward implementation. Despite its simplicity, this method has already resulted in a sensible architecture that outperforms baselines, thereby validating our contributions. We acknowledge that the current method may not achieve the absolute upper limit of performance possible with all potential decomposition-based model configurations. This limitation is discussed in the "Conclusions, Limitations, and Social Impacts" section. We agree that learning-based methods, such as neural architecture search (NAS), offer a promising avenue for further improvement and optimization. This remains an exciting area for future work.

---

W3: Thank you for your valuable suggestion. We will enhance the layout and formatting in the final version of the paper. This will involve resizing and rearranging the figures in the methodology section to ensure they are clear and informative while providing additional space for valuable content in the main text. Our goal is to make the main body of the paper more self-contained and comprehensive.

Thank you for your thoughtful review and constructive comments.

---

W1: We apologize for the lack of captions for Figure 2 and Figure 3. We also include the caption for Figure 1, as requested by Reviewer aS9R. The captions for these figures are shown below

Figure 2: Examination of parameter scale and computation scale against the forward window size and the backward window size on the ECL dataset. (a) (c) For short forward window size or backward window size, SSCNN consumes more parameters than DLinear only. As the window size grows, the SSCNN scales up at a much slower inflation rate than DLinear. (b) (d) The consumption of FLOPS by SSCNN ranks in the middle place among SOTAs, less efficient than iTransformer and Autoformer yet more efficient than PatchTST and Crossformer.

Figure 3: Impacts of backward window size. While the performance of other SOTA forecasters does not necessarily benefit from the increased lookback length, SSCNN improved performance on the enlarged lookback window.

Figure1: (a) Embedding: Raw data points are independently mapped to a high-dimensional representation space using a shared 1x1 convolution operator across all variables and horizons. (b) In-sample Inference: The historical time series representations are processed by three temporal attention-based normalization (T-AttnNorm) layers with different instantiations of the attention map ($\mathcal{I}$), followed by a spatial attention-based normalization (S-AttnNorm) layer. This process yields estimations of four structured components ($\mu$) along with the residual ($R$). (c) Out-of-sample extrapolation: The decomposed components corresponding to each variable are individually extrapolated to future horizons using the matrix multiplication (MatMul) operator, with distinct instantiations of the attention map ($\mathcal{E}$). (d) Component fusion: Finally, all four components, combined with the residuals, are input into a polynomial regression layer.

---

W2: We apologize for not highlighting the limitations of the model prominently enough. Due to space constraints, we discussed the limitations in the last section together with broader impacts and conclusions. We will discuss the limitations in detail and put them into a new separate section in the revision

Briefly, the identified limitations include: (a) While our approach effectively reduces model size, its computational complexity remains comparable to Transformer-based models, which is more demanding than linear models. This suggests potential for further optimization in terms of computational efficiency. (b) Currently, the combination of the proposed decomposition modules is determined with statistical insight into the data, which may not be optimal. Future work could explore methods for efficiently searching for the optimal combination of these modules. These points outline practical areas for improvement and future exploration. We will ensure these limitations are clearly presented in the final version of the paper.

---

W3: We apologize for the confusion between the second and third contributions. The second contribution relates to Section 4, where we theoretically explore the equivalence in capability between the patching operator and the decomposition operator. We also highlight the advantage of decomposition in terms of parameter efficiency, suggesting that decomposition can serve as a parameter-efficient alternative to patching, which is often considered essential in many state-of-the-art (SOTA) models. The third contribution pertains to the empirical evaluation of SSCNN against SOTA baselines, detailed in Section 5.

To clarify, we will revise the second contribution as follows:

"We conduct an in-depth theoretical comparison between decomposition and patching, analyzing both capability and parsimony. This analysis challenges the necessity of patching as an essential component in modern time series forecasting models, proposing decomposition as a more parameter-efficient alternative."

---

Q1: Please check the response to W3.

---

Q2: Please check the response to W1.

---

Q3: We will rectify this issue in the final version.

Thank you for your thoughtful review and constructive comments.

---

W1: We apologize if the current symbolic formulas present challenges to comprehension. To enhance understanding, we have tried to make the selection mechanisms formulated by Equations (3)-(7) more accessible with the example attention maps provided in Figure 1. To further illustrate the computational flow, we present a toy example focusing on the inference of long-term, seasonal, and short-term components, with the scaling operation and the selection mechanism being disabled for simplicity:

Given a historical sequence of observations: 1.1, 3.1, 1.0, 3.0, 0.9, 2.9:

Long-term Component: Derived as 2, 2, 2, 2, 2, 2, resulting in the sequence of residuals: -0.9, 1.1, -1, 1, -1.1, 0.9.

Seasonal Component: Computed as -1, 1, -1, 1, -1, 1, leading to the residual series: 0.1, 0.1, 0, 0, -0.1, -0.1.

Short-term Component: Derived as 0, 0.1, 0.05, 0, -0.05, -0.1 with a window size of 2, obtaining the residual series: 0.1, 0, -0.05, 0, -0.05, 0.

---

W2: We apologize for any confusion caused by the perceived disconnect between the experimental results and the theoretical claims of the paper. The experimental results presented in Table 1 demonstrate SSCNN's improved efficacy compared to state-of-the-art methods, including the patching-based methods PatchTST and iTransformer. These results align with the theoretical claims made in Section 4.1, suggesting that SSCNN's decomposition-based approach either matches or exceeds the capabilities of patching-based methods. Additionally, Figure 2 illustrates the optimal parsimony of SSCNN, supporting the analysis in Section 4.2. Furthermore, Figure 3 highlights SSCNN's superior scalability with an increasing lookback window size. Unlike other baselines, SSCNN consistently benefits from more historical data points, regardless of the dataset's dynamic nature. This scalability is crucial for time series forecasting, as more data generally leads to more accurate predictions following our expectation.

---

W3: (1). Thank you for catching this error. The label "HDformer" in Figure 3(a) is indeed a typo and should be replaced with "SSCNN." We will correct this in the revised version.

(2). In Figure 2, for subfigures (a) and (b), $T_{in}$ is set to 96, while $T_{out}$ is represented by the x-axis. For subfigures (c) and (d), $T_{out}$ is set to 96, while $T_{in}$ is represented by the x-axis. We will clarify this information in the revised figure caption to improve understanding.

(3). The forward window size in Figure 3 is indicated by $T_{out}$, which is noted in each subfigure. We will make sure this information is more clearly presented in the revised version for better clarity.

---

W4: We will revise the third contribution as follows in the final version: We conducted comprehensive experiments to evaluate SSCNN across various dimensions. SSCNN demonstrates improved effectiveness, with increases ranging from 4% to 20%, and a reduction in model size ranging from 70% to 99% on average across all datasets, compared to state-of-the-art baselines.

---

W5: We will also address the other concerns you raise about the layout and formating in the final version.

---

Q1: The concept of compositionality in time series forecasting has roots in classical statistical methods, most notably STL [1]. Our method builds on this foundational idea by integrating it with deep learning techniques, enabling nonlinear decomposition and re-composition of the time series data. Additionally, we also extend STL to consider the short-term and spatial components. In this way, our method achieves a more nuanced and effective modeling of complex time series patterns.

[1] Cleveland, Robert B., et al. "STL: A seasonal-trend decomposition." J. off. Stat 6.1 (1990): 3-73.

---

Q2: The novelty of our approach lies in the reinvention of these layers that are inspired from the existing literature. As mentioned in the introduction and related work sections, some existing works have also explored the idea of modeling different heterogeneous components separately. These works typically employed techniques such as plain moving averages or MLPs. However, they often struggled to outperform advanced baselines like PatchTST across various benchmark datasets, as shown in Tables 1 and 2. Moreover, these approaches lacked scalability, requiring exponentially more parameters than SSCNN for large forward and backward window sizes, as demonstrated in Figures 2(a) and 2(c). Our method significantly advances these prior approaches by integrating customized selection mechanisms for the components of different behaviors within both backward and forward estimation processes. This extension enhances our model's capability and parsimony, resulting in superior performance.

---

Q3: We have preliminarily reported the measurements of training and inference times for SSCNN compared to two representative baselines, PatchTST and iTransformer, on the Electricity and ETTh1 datasets. We will improve this evaluation study in the revision. Our preliminary findings indicate that iTransformer is the most efficient model among the three, requiring the least time for both training and inference. The relatively higher running time of SSCNN can be in part attributed to its sequential handling of the considered components. In contrast, iTransformer handles these components in parallel using a unified MLP, contributing to its efficiency advantage. We have mentioned this limitation of SSCNN in the section of "Conclusions, Limitations and Broader Impacts".

---

Q4, Q5, Q6: Please check the above response to W3.