CRAFT: Time Series Forecasting with Cross-Future Behavior Awareness

Summary

We thank the reviewer for reviewing our manuscript and giving a lot of significiant suggestions. We have responded to the reviewer's comments below and have addressed them accordingly in the uploaded revision of the paper. Given the improvement suggestion by all reviewers, we have re-modified our manuscript and uploaded the revised version. We look forward to the reviewer's reassessment of our manuscript.

Response to Weakness 1: about the citation problem

Thanks for your careful suggestion. We have checked the original DLinear paper: "DLinear is a combination of a Decomposition scheme used in Autoformer and FEDformer with linear layers. It first decomposes a raw data input into a trend component by a moving average kernel and a remainder (seasonal) component." In the original paper of DLinear, the auhtors do use the expression of "remainder component".

Response to Weakness 2 and Weakness 3: some parts of the writing lack clarity and the inconsistent notations

Thank you for pointing out the writing and expression problems in this article. We have rechecked the article and sorted out the confusing content and corresponding corrections as follows :

1. Original text : Table 2: Comparative forecasting results with the look-back window length of L $\in$ {30, 90, 180} and prediction window length of P $\in$ {7, 14, 30} respectively. The best results are highlighted in bold and the second best results are highlighted with a underline.

   Reason for correction: lack clarity, the look-back window length L is not displayed in the table. Corrected text: Table 2: Comparative forecasting results with the look-back window length of L $\in$ {30, 90, 180} and prediction window length of P ∈ {7, 14, 30} respectively, correspond one-to-one with the look back window L ∈ {30, 90, 180}. The unit of length is days. The best results are highlighted in bold and the second best results are highlighted with a underline.

   In addition, column name “Length” in Table 2 will be changed to “Length P”

2. Original text :

a) etc. $C_t=\{C_{t-L+1:t},C_{t+1:t+p}\}$, where $C_{t-L+1:t}$ is CFB in the look-back window and $C_{t+1:t+p}$ is CFB in the prediction window. (Near line 152)

b) Time series forecasting (TSF) with Cross-Future Behavior (CFB) can be defined as $Y_{t+1:t+p}=H(Y_{t-L+1:t}, X_{t}, C_{t})$. (Near line 148)

c) $X_t$ is covariate features, $C_t$ is the CFB feature and H is the prediction function to be learned. (Near line 150)

d) KPM module aims to transfer the future trend information from CFB feature $C_{t}=\{C_{L}, C_{P}\}$ to labels $Y_{P}$ . (Near line 200)

Reason for correction : reuse the same symbol $C_t$ in set symbol and elements within a set.

Corrected text :

a) etc. $C_{𝕋}=\{C_{t-L+1:t},C_{t+1:t+p}\}$, where $C_{t-L+1:t}$ is CFB in the look-back window and $C_{t+1:t+p}$ is CFB in the prediction window.

b) Time series forecasting (TSF) with Cross-Future Behavior (CFB) can be defined as $Y_{t+1:t+p}=H(Y_{t-L+1:t}, X_{𝕋}, C_{𝕋})$.

c) $X_𝕋$ is covariate features, $C_𝕋$ is the CFB feature and H is the prediction function to be learned.

d) The same applies to the value of $C_t$ in the following text. KPM module aims to transfer the future trend information from CFB feature $C_{𝕋}=\{C_{L}, C_{P}\}$ to labels $Y_{P}$ .

3. Original text :

a) which is designed as a linear network: (Near line 244)

$${ZC}_P^T=Complete\\_network(ZC^T) \ \ \ \ \ \ \ \ \ \ \ \ \ Eq(5)$$

b) Finally, the latent vector $\hat{ZC}_P^T$ and $\hat{ZY}_P^T$ are converted into target predicted values with decoder: $\hat{C}_P^T=Decoder(\hat{ZC}_P^T),\hat{Y}_P^T=Decoder(\hat{ZY}_P^T)$. (Near line 256)

c) The final result is calculated by adding these two values with the reminder predicted values (acquired with the linear mapping of $Y_L^R$, $C_L^R$ in Figure 3): (Near line 258)

$$\hat{C}_P=\hat{C}_P^T + \hat{C}_P^R, \hat{Y}_P=\hat{Y}_P^T + \hat{Y}_P^R \ \ \ \ \ \ \ \ \ \ \ \ \ Eq(7)$$

d) The intact matrix expression of prediction window of cross-future behavior $\hat{ZC}_P$ (Near line 340)

e) which corresponding to the recovery loss of $\hat{C}_{KP}$ in the KPM module, the prediction error of $\hat{C}_P$ in the ITM module, the reconstruction drift of $\hat{Y}_P$ at the ETG module and the forecast deviation of label $\hat{Y}_P$ respectively. (Near line 347)

Reason for correction: Confused $\hat{ZC}\_P^T$ and $\hat{ZC}\_{TP}^T$, $\hat{C}\_P$ and $\hat{C}\_{TP}$

Corrected text:

a) which is designed as a linear network: (Near line 244) $\hat{ZC}\_{TP}^T=Complete\\_network(ZC^T) \ \ \ \ \ \ \ \ \ \ \ \ \ Eq(5)$ b) Finally, the latent vector $\hat{ZC}\_{TP}^T$ and $\hat{ZY}\_P^T$ are converted into target predicted values with decoder: $\hat{C}\_{TP}^T=Decoder(\hat{ZC}\_{TP}^T),\hat{Y}\_P^T=Decoder(\hat{ZY}\_P^T).$

c) The final result is calculated by adding these two values with the reminder predicted values (acquired with the linear mapping of $Y\_L^R$, $C\_L^R$ in Figure 3): (Near line 258) $\hat{C}\_{TP}=\hat{C}\_{TP}^T + \hat{C}\_{TP}^R, \hat{Y}\_P=\hat{Y}\_P^T + \hat{Y}\_P^R \ \ \ \ \ \ \ \ \ \ \ \ \ Eq(7)$ d) The intact matrix expression of prediction window of cross-future behavior $\hat{C}\_{TP}$. (Near line 340)

e) which corresponding to the recovery loss of $\hat{C}\_{P}$ in the KPM module, the prediction error of $\hat{C}\_{TP}$ in the ITM module, the reconstruction drift of $\hat{Y}\_P$ at the ETG module and the forecast deviation of label $\hat{Y}\_P$ respectively. (Near line 347)

Synchronously, the symbols in the figures of the paper have also been modified accordingly.

Response to Weakness 2: about the citation

Thanks for your suggestions. According to the summary in first paragraph in the Introdcution Section and the summary in the Related Work. Backbone for time series forecasting Section, you can know that we also obey the consistent idea with you about the classification of existing time series forecasting method, i.e., the CNNs, RNNs, and Transformer-based methods.

Recently, more and more research focues on time series forecasting problem. We use "time series forecasting" as the keyword to search in Web of Science and DBLP. According to our statistics, the numbers of targeted paper in Web of Science are 4252 (2024), 4839 (2023), 4824 (2022), 4279 (2021), 3692 (2020), and 3220 (2019). The numbers of targeted paper in DBLP are 2024 (688), 2023 (586), 2022 (436), 2021 (366), 2020 (300), and 2019 (196). The numbers of targeted paper published in machine learning related conference are 27 (ICLR), 30 (AAAI), 20 (IJCAI), 20 (ICML), and 32 (NeurIPS).

In any paper, there is impossible to cite all related papers totally. What we can do is to obey the correct classification criteria and list the latest and most relevant works. In addition, the recommended eight papers, especially the [3-4] paper, do not have strong authority and relevance.

More importanly, the reviewer points out that "no attention was given to any time series forecasting methods from ICLR 2024", but we have cited five time series forecasting methods methods from ICLR 2024:

[1] Rasul K, Bennett A, Vicente P, et al. VQ-TR: Vector Quantized Attention for Time Series Forecasting[C]//The Twelfth International Conference on Learning Representations. 2024.

[2] Zhao L, Shen Y. Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators[C]//The Twelfth International Conference on Learning Representations. 2024.

[3] Li Y, Chen W, Hu X, et al. Transformer-Modulated Diffusion Models for Probabilistic Multivariate Time Series Forecasting[C]//The Twelfth International Conference on Learning Representations. 2024.

[4] Wang X, Zhou T, Wen Q, et al. CARD: Channel aligned robust blend transformer for time series forecasting[C]//The Twelfth International Conference on Learning Representations. 2024.

[5] Nie Y, Nguyen N H, Sinthong P, et al. A time series is worth 64 words: Long-term forecasting with transformers[C]//The Twelfth International Conference on Learning Representations. 2024.

Summary

We thank reviewer U8vN for reviewing our manuscript and pointing out a set of significiant inquiries. We have responded to the reviewer's comments below and have addressed them accordingly in the uploaded revision of the paper. Given the improvement suggestion by all reviewers, we have re-modified our manuscript and uploaded the revised version. We look forward to the reviewer's reassessment of our manuscript.

Response to Weakness 1: about the data leakage problem

Thanks for your concerns. However, we have to declare that CFB are a normal prior features rather than labels. Features in machine learning refers to individual measurable properties or characteristics of a phenomenon being observed, using as input variables for model prediction. Typically, features should satisfy the measurable, independent, and varied types condition. Labels are the outputs that the models are trained to predict, and the labels should depend on the input features. The depend property between features and labeles is the theoretical foundation for reasonable machine learning model construction.

On the other hand, in machine learning, there is a important research area, named feature engineering. The aim of feature engineering is to discovery significiant features to improve the performance of machine learning model. Our research to discovery CFB feature belongs to this area.

Returning to the specific field of time series forecasting, many existing works are also attempting to introduce more features to improve the performance of time series forecasting. Wen et al. [1] classified covariate features in TSF into dynamic historical, known future, and static variables. Chen et al. [2] used auxiliary information like promotions and vacations in M5 dataset for prediction. These additional featues are also similar to CFB features, and they do not involve data leakage problem.

Thus, we can not agree with your doubts about data leakage. CFB features we introduce in our paper are prior features that we can acquire and use before the current prediction timepoint. Table 2 predicts the labels before 7, 14 and 30 days at most. CFB features is related to the forecasting results. The relathionship between CFB features and forecasting results is the theoretical foundation for our research.

[1] Wen R, Torkkola K, Narayanaswamy B, et al. A multi-horizon quantile recurrent forecaster[J]. arXiv preprint arXiv:1711.11053, 2017. [2] Chen S A, Li C L, Yoder N, et al. Tsmixer: An all-mlp architecture for time series forecasting[J]. arXiv preprint arXiv:2303.06053, 2023.

Summary

We thank reviewer for reviewing our manuscript and pointing out a set of significiant suggestions. These suggestions improve the quality of our manuscript significantly. We have responded to the reviewer's comments below and have addressed them accordingly in the uploaded revision of the paper. Given the improvement suggestion by all reviewers, we have re-modified our manuscript and uploaded the revised version. We look forward to the reviewer's reassessment of our manuscript.

Response to Weakness 1: broader exploration of its implications and applications

Thanks for your significiant suggestion. Our proposed CFB features and CRAFT method is indeed applicable to forecasting problems with advance operation. Currently, we only collect suitable experimental dataset in e-commerce area. Therefore, we organize our paper around e-commerce. In the origial manuscript, we have pointed this limitation in the conclusion sections as: "As current public available dataset (e.g., ETT, ECL, Weather) does not contain the CFB feature or similar feature, we only verify CRAFT on our dataset. In the future, we will further collect related data to form series benchmark dataset. In addition, we will explore the application of CRAFT on more scenarios."

In the revised pdf, we have discussed the application of CFB features and CRAFT method in other area and supplemented the content to conclusion section: "This paper only explores the application of CFB and CRAFT on e-commerce area. Electricity demand, stock price, and disease spread forecasting typically do not have clear lead-up operation events, making it difficult to apply our method. In addition, current public available dataset (e.g., ETT, ECL, Weather) does not contain the CFB feature or similar feature, we only verify CRAFT on our dataset limited by this actual situation. n the future, we will further collect related data to form series benchmark dataset. Moreover, we will continue to generalize the definition of CFB to enable CRAFT method can be applied more broadly."

Response to Weakness 2: conduct longitudinal studies and studies under different market conditions

Thanks for your suggestion. Actually, we have already conducted longitudinal studies to evaluate how CRAFT performs over extended periods and under different market conditions:

Longitudinal studies: Section 5.2.1 (Comparison with Baselines) is the longitudinal studies. We conduct experiment with prediction window length of P 7, 14, 30 respectively, correspond one-to-one with the look back window L 30, 90, 180. As for why the window is not long enough, such as a full year, it is because predictions made 7-30 days in advance are sufficient for sellers to adjust prices and inventory. Predictions made too far in advance do not have significant meaning for actual business operations.

Under different market conditions: Section 5.2.3 (Cases Studies) verifies that our proposed method is effective in capturing future trends, under different trends for validation. Section 5.3 (Online A/B Test) is the experiment under different market conditions, including 2023 Mid-autumn, 2023 National Day, 2024 New Year's Day, and 2024 Spring Festival. We are sorry that our introduction in the first draft of the paper is not clear enough.

Response to Weakness 3: sensitivity analysis and the how hyperparameters are selected?

Thanks for your sincere suggestion. The response regarding sensitivity analysis and hyperparameter selection is as follows:

Sensitivity analysis: Actually, we have already presented the sensitivity analysis studies in Section E.2 Hyper Parameter Analysis. We offered the sensitivity analysis of the loss coefficient and the parameter in solving koopman matrix that has a significant impact on CRAFT. In addition, we will provide the search space and optimal values for other model parameters in Appendix F Experimental Configuration.

Guidelines for selecting hyperparameters: We use the loss function of train set and the prediction accuracy of validation set to guide the selection of hyperparameters, which is a common method in machine learning. Firstly, the time-series data is divided into training set, validation set, and testing set along the time dimension, which can avoid the problem of data traversal. The model is trained on the training set and hyperparameters are selected based on the descent of the loss function in the training set and the prediction error in the validation set. Cheng et al. [1] used the similar method. We will also supplement the guidelines on hyperparameter selection in Appendix F Experimental Configuration.

[1] Cheng H, Wen Q, Liu Y, et al. RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies[C]//The Twelfth International Conference on Learning Representations.2024.

Summary

We thank reviewer for reviewing our manuscript and pointing out a set of significiant questions. We have responded to the reviewer's comments below and have addressed them accordingly in the uploaded revision of the paper. Given the improvement suggestion by all reviewers, we have re-modified our manuscript and uploaded the revised version. We look forward to the reviewer's reassessment of our manuscript.

Response to Weaknesses:

Thank you for your insightful and detailed comments. We appreciate the opportunity to address the limitations of our work. Here are the responses to each point:

1. Real-World Data Complexity and Variability: We acknowledge that real-world data often contains noise, outliers, and non-stationarity, which can indeed pose challenges for any forecasting model. To verify CRAFT's ability to handle such complexities, we conduct comprehensive experiments on different data. In the comparative experiment (Section 5.2.1), we conduct experiment on data with different window length. In the case study (Section 5.2.3), we conduct experiment on different data with different trends for validation. In the online A/B test (Section 5.3), we conduct experiment during different holiday, including 2023 Mid-autumn, 2023 National Day, 2024 New Year's Day, and 2024 Spring Festival. These diverse experiments on various data sets, along with the rich experimental results, have all validated the superiority of our approach and the diversity of our dataset.
2. Data Redundancy: The introduction of CFB features indeed increases the dimensionality of the features, which is also the main reason why the direct use of CFB features in other existing methods shown in Table 2 is not effective enough. Therefore, to address the issue of fully utilizing CFB features, we propose the CRAFT method. The core idea of CRAFT is to utilize the trend of CFB to mine the trend of time series data to be predicted.
3. Interpretability: We recognize the importance of interpretability, especially for stakeholders who need to understand the decision-making process. CRAFT is composed of three main parts: KPM module, ITM module, and ETG module. Each module has a clear function and well-defined input and output results. KPM can extract the key trends of the label and CFB, predicting the label in the prediction window. ITM supplements the unknown area of CFB, making the final prediction of the label in the prediction window. ETG, with a hierarchical structure, can acquire more representative trends from higher levels. Figure 12 visualizes the experiemtal results for every modules. According to Figure 12, we can not only observe the experiemental results after every modules, but also know the consistency of our model's prediction results with the actual labels.
4. Data Transferability: Our proposed CFB features and CRAFT method is indeed applicable to forecasting problems with advance operation. Currently, we only collect suitable experimental dataset in e-commerce area. We will discuss the application of CFB features and CRAFT method in other area and supplement the content to conclusion section.
5. Generalizability: Currently, we only collect suitable experimental dataset in e-commerce area. Therefore, we organize our paper around e-commerce. If we can acquire more suitable dataset in other area, we will conduct experiment on other dataset.
6. Scalability: Due to the using area limitation, we do not conduct experiment on other large-scale datasets. In the revised version of our manuscript, we will discuss the application of CFB features and CRAFT method in other area and supplement the content to conclusion section.
7. Robustness to Changing Conditions: Actually, our paper has discussed some content related to robustness to changing condition. In the comparative experiment (Section 5.2.1), we conduct experiment on data with different window length. In the case study (Section 5.2.3), we conduct experiment on different data with different trends for validation. In the online A/B test (Section 5.3), we conduct experiment during different holiday, including 2023 Mid-autumn, 2023 National Day, 2024 New Year's Day, and 2024 Spring Festival.
8. Dependency on High-Quality Data: We understand that the quality of input data is crucial for the effectiveness of CRAFT. We are sorry that we only conduct experiment on our dataset limited by the data condition.