Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspective

Reply to W1

Thanks for the question. In time series forecasting, all the patch-based models obey the calculation chain: Patching —> Embedding (preliminary representations) —> Modeling (senior representations) —> Forecasting (Dimensionality Reduction). Specifically, Patching technique does not directly produce representations, but it determines where the representations come from (the candidate patches). Therefore, we think that the Patching technique actually determines the structure of the representational space, thus affecting the representations. In terms of representations, we think the Embeddings are the preliminary representations directly produced from the patches, and the hidden representations across model layers are the senior ones.

Reply to W2

Thanks for the suggestion. To provide more empircal evidence, we conduct additional experiments on another 6 datasets from TFB. Their statistical information is as follows:

To comprehensively evaluate the capability of SRS, we make plugin experiments with all 5 models in our paper on the above 6 datasets, the mean results of 4 forecasting horizons are reported as follows:

Based on the above results, we have two main observations:

1. The SRS excels at datasets with high shifting values (NASDAQ) and low seasonality values (AQWan), because there exists more special samples about changeable periods, anomalies and shifting in these cases. The SRS can improve all patch-based models significantly by 17% — 28% on these datasets.
2. When the datasets are closer to normal cases with higher seasonality values and lower shifting values, the performance improved by the SRS gradually decreases. On NYSE and ZafNoo, the performance is improved by 12% — 18%. Note that even on AQShunyi and PEMS08, the improvement still exists and is about 4% — 10%. This provides evidence that the SRS can handle both the normal and special cases well.

Reply to W3

Thanks for the suggestion. To provide more details regarding efficiency, we conduct additional experiments. Our experiments consist of two parts:

(1) The efficiency comparison between SRSNet and other baselines:

(2) The complexity introduced by the SRS module:

Based on the above tables, we have two main observations:

1. The SRSNet makes accuracy and efficiency meet, though not enough efficient as Linear-based models, but is more efficient than most transformer-based baselines and achieves stronger performance.
2. It is observed that the overhead introduced by the SRS is relatively trivial. The Memory, MACs, and Params increase little. And the Training Time and Inference Time are also controllable under the end-to-end supervised cases. And the performance can be consistently improved.

Reply to W4

Thanks for the question. The motivation we devise the mechanism is to keep the propagation of gradients in the Argmax operation in the equation (2) of the paper:

$\mathcal{S} ^s = \text{Scorer} ^s(\mathcal{P} ^\prime), \mathcal{I} ^s = \text{Argmax}(\mathcal{S} ^s)$

Since the Argmax operation finds the patches with the max scores and return their indices, it disrupts the gradients. To avoid this phenomenon, we need to attach the gradients back to the selected patches. Note that the gradients are on the scores generated by the scorer, so we need to first retrieve the corresponding scores of the selected patches through equation (3):

$\mathcal{S} ^s _{max} = \mathcal{S} ^s[\mathcal{I} ^s], \mathcal{S} ^s _{inv} = \text{detach}(1/\mathcal{S} _{max} ^s)$

We then process the scores and normalize them to 1 values, while keeping the gradients on them. The detail is that we detach the reciprocal $\mathcal{S} ^s _{inv}$ to preserve the gradients.

$\mathcal{P} _{max} ^s = \mathcal{P} ^\prime [\mathcal{I} ^s], E ^s = \mathcal{S} ^s _{max} \odot \mathcal{S} ^s _{inv}$ Finally, we attach the normalized scores to the patches through Hadamard product: $\tilde{\mathcal{P}} ^s _{\text{max}} = \mathcal{P} _{max} ^s \odot E ^s$

Thanks for your sincere suggestions again! We will release the full results when the discussion phase starts. If you have any additional question, we can make further discussion!

Thanks for your in-depth insights and suggestions. Through Adaptive Fusion, the SRS can integrate both the representations from Adjacent Patching and Selective Patching with learnable weights, thus can combine the advantages of both methods.

As you mentioned, the weight parameter in Eq. 13 can actually reflect which scenarios the two techniques separately excel at. We manage to study this by devising experiments on specific datasets and report the weight values. Intuitively, the Selective Patching may excel at datasets with shifting phenomenon and the Adjacent Patching may excel at datasets with strong seasonality. So that we conduct additional experiments on datasets with varying values of shifting and seasonality from TFB, a well-recognized time series forecasting benchmark. Specifically, we choose NASDAQ, NYSE, and AQShunyi with descending shifting values in TFB, and choose PEMS08, ZafNoo, and AQWan with descending seasonality values in TFB. The statistical details of them are as follows:

We conduct comprehensive plugin experiments on them, reporting the average MSE and MAE metrics, and how the weight values converged.

Based on the above results, we have two main observations:

1. The SRS excels at datasets with high shifting values (NASDAQ) and low seasonality values (AQWan), because there exists more special samples about changeable periods, anomalies and shifting in these cases. The SRS can improve all patch-based models significantly by 17% — 28% on these datasets.
2. When the datasets are closer to normal cases with higher seasonality values and lower shifting values, the performance improved by the SRS gradually decreases. On NYSE and ZafNoo, the performance is improved by 12% — 18%. Note that even on AQShunyi and PEMS08, the improvement still exists and is about 4% — 10%. This provides evidence that the SRS can handle both the normal and special cases well.

We then report the converged mean weight values $(1-\alpha) \in [0, 1]$ on them. (higher values indicate more dependence on representations from Selective Patching, lower values indicate more dependence on representations from Adjacent Patching).

We also have two main observations:

1. When the seasonality values are higher and the shifting values are lower, models have less dependence on representations from the Selective Patching. This implies that the Adjacent Patching may well handle these common cases or the Selective Patching is gradually converged to the Adjacent Patching.
2. When the seasonality values are lower and the shifting values are higher, models have more dependence on representations from the Selective Patching. It is because Adjacent Patching may not well process these special cases, while our proposed Selective Patching can adaptively select the patches with more useful information for forecasting.

Thanks for your sincere suggestions again! We will release the full results when the discussion phase starts. If you have any additional question, we can make further discussion!

Dear Reviewer 7m7L,

Please help to go through the rebuttal and participate in discussions with authors. Thank you!

Best regards, AC

Reply to W1:

Thanks for the question. There are two main reasons we retain the representations generated by the Conventional Adjacent Patching:

(1) Since the SRS is a trainable plugin, the scorers may have some randomness in the early stage of training, so it may not accurately find suitable patches. Therefore, retaining the representations generated by the Conventional Adjacent Patching and performing adaptive fusion can effectively enhance the stability of the training. As the training continues, the ability of the scorer gradually improves, enabling it to select better patches and increase the weight during fusion, thereby improving the end-to-end prediction performance.

(2) The SRS is proposed for some special cases such as changeable periods, anomalies, and shifting, where the Selective Patching and Dynamic Reassembly can help retrieve useful information in the contextual time series. In common scenarios where the sample is relatively normal, the Conventional Adjacent Patching can also well handle so that we retain the representations generated by it. Through adaptively fusing the representations, SRS can handle both the common and special cases.

Reply to W2:

Thanks for the question. We first explain the details of the matrix of Selective Patching. The shape of the matrix is $\mathbb{R}^{K\times n}$, where $K$ denotes the number of all candidate patches (generated by stride=1), and $n$ denotes the number of patches we need to pick up from all $K$ ones. Since the Selective Patching supports selection with replacement, which means one patch can be selected multiple times, so we make $\text{Scorer}^s$ score $n$ scores for each patch, thus forming the matrix with the shape of $\mathbb{R}^{K\times n}$. To select the $n$ appropriate patches, we only need to check the scores of all $K$ patches each time and pick the highest ones.

We then shed light to the main differences between Selective Patching and Dynamic Reassembly. Different from Selective Patching, the nature of Dynamic Reassembly is to sort the selected patches and then reassemble them in the determined order, so that the scores are not formed like a list instead of a matrix.

Reply to W3:

Thanks for your remind, we will fix these citations.

Thanks for your sincere suggestions again! If you have any additional question, we can make further discussion!

Reply to W1

Thanks for your in-depth insights and suggestions. We admit that some expressions are not quite appropriate, and we will modify them later.

We first provide some intuitions. Assuming the contextual time series is a continuous interval [1, T] covering T points, the useful informative intervals in normal cases (like strong seasonality) are also the [1, T]. Adjacent Patching can well handle these normal cases. In our work, we mainly consider the cases that not all of the contextual time series is useful for forecasting, such as anomalies, changeable periods, and shifting. In these cases, the contextual time series are Interrupted as $[1, T_1] \cup [T_1, T_2], \cdots, \cup [T_{k-1}, T]$, containing $k$ informative sub-intervals. If the $k$ sub-intervals cover x% of the contextual time series, only x% of representations created by Adjacent Patching are useful for forecasting. In contrast, our proposed Selective Patching can choose the useful patches in the $k$ sub-intervals, and support re-selection to amplify useful patches. The Dynamic Reassembly then introduces positional information for better representational learning. We also fuse the representations from both the Selective Patching and Adjacent Patching, to ensure the ratio of useful information is greater than x%, thus handling both the normal and special cases.

To provide more empircal evidence about when the SRS outperforms the Adjacent Patching, we conduct additional experiments on datasets from TFB, a well-recognized time series forecasting benchmark. Since datasets with strong seasonality values are close to normal cases while those with strong shifting values are special ones, we choose two groups of datasets separately with descending seasonality values and shifting values. The higher the values, the stronger seasonality/shifting the time series data have. We list the statistical information:

We conduct plugin experiments on these datasets and report the mean results:

We have two main observations:

1. The SRS excels at datasets with high shifting values (NASDAQ) and low seasonality values (AQWan), because there exists more special samples about changeable periods, anomalies and shifting in these cases. The SRS improves all models significantly by 17% — 28% on these datasets.
2. When the datasets are closer to normal cases with higher seasonality values and lower shifting values, the performance improved by the SRS gradually decreases. On NYSE and ZafNoo, the performance is improved by 12% — 18%. Note that even on AQShunyi and PEMS08, the improvement still exists and is about 4% — 10%. This demonstrates that the SRS excels at both the normal and special cases.

Reply to W2

Since the SRS is proposed for special cases like changeable periods, anomalies, and shifting, we think the proportion of special samples in the dataset will affect the performance. In the Reply to W1, we make additional experiments on datasets having more special samples (NASDAQ, AQWan, NYSE, ZafNoo), and find the improvement of SRS can achieve 12% — 28% (quite non-trivial). We hope this can be considered as additional evidence.

Reply to Q3

As you mentioned, on datasets with strong seasonality, the Adjacent Patching can well handle the information in contextual time series and form useful representations. In the reply to W1, our experiments reveal the phenomenon that the improvement of SRS decreases as the seasonality increases, while the improvement still exists and is about 4% — 10%. It is because the Dynamic Reassembly still works and reassembles the patches for better positional information, and the Adaptive Fusion can adaptively integrate the representations, which ensures the improvement even in normal cases.

Reply to W3 & Q2

We make an analysis on the overhead introduced by the SRS on above 6 datasets. Due to the space limitation, we report the results on PEMS08 (biggest) and NYSE (smallest) here, and will release full results when the discussion phase starts. Specifically, we report the Number of Parameters, Max GPU Memory, MACs, Training Time, and Inference Time:

It is observed that the overhead introduced by the SRS is relatively trivial. The Memory, MACs, and Params increase little. And the Training Time and Inference Time are also controllable.

Reply to Q1

We further study on the initialization strategies, including Constant, Xaiver, Kaiming, and Randn. We also report the standard deviation of 5 rounds:

The results show that random-based initialization strategies such like Xaiver, Kaiming, and Randn (ours) are relatively stable. The Constant initialization strategy causes unstable training in some cases.

Reply to Q4

We have provided some showcases in Appendix A.2 of the paper. For clarity, we label the sub-figures with (a)-(f) in the order from left to right and from top to bottom.

Figure 5 (d), (e), (f), and Figure 7 (c) show the selected patches under changeable periods, which accurately picks up the correct periods most related to the forecasting horizons.

Figure 6 (a), (b), and Figure 7 (a), (e), (f) show the selected patches when anomalies occur, which avoid including the abnormal patches.

Figure 5 (b), (c), and Figure 6 (c), (e), (f) show the selected patches when shifting occurs, which selects the in-distribution patches for forecasting.

We also showcase some normal cases such as Figure 5 (a), Figure 6 (d), and Figure 7 (d), where the selected patches are similar to those produced by adjacent patching, demonstrating the flexibility of the Selective Patching.

Thanks for your valuable suggestions again! We will release the full results when the discussion phase starts. If you have any additional question, we can make further discussion!

Dear Reviewer xNzv,

Please help to go through the rebuttal and participate in discussions with authors. Thank you!

Best regards, AC