Response to Reviewer ECSx

Thank you for your detailed review and questions. Please find our answers below.

Q1: Were the results of DLinear and PatchTST obtained with a lookback of 96?

We have reviewed the results for DLinear and PatchTST in Table 12 and confirmed that you are correct—their look-back windows are not 96. We apologize for this mistake. Since most of Table 12's content was referenced from Table 14 of the GPT4TS paper, we assumed that the same look-back window was used across comparison methods.

We also apologize for the incorrect assertion in line 236. Upon review, DLinear, PatchTST, and FITS indeed do not use the default look-back window length of 96. We have adjusted the experiments and used longer look-back windows (512 and 720) for these methods to ensure fairer comparisons. We have included the updated experiments in the second part of the global reply.

Q2: Regarding the Anomaly Detection experiment, did you use point adjustment techniques and manual thresholds?

You are correct that our brief statement about following the TimesNet approach was problematic. To address your question first, we did use point adjustment techniques with manual thresholding, and this framework was consistently applied across all comparison methods. Here is a detailed explanation of our anomaly detection strategy:

Training Phase: During training, we applied a simple reconstruction loss to help the model learn the distribution of normal data.

Testing Phase: For testing, we used the following code (showing only the main parts):

def test(self, setting):

        attens_energy = []
        self.anomaly_criterion = nn.MSELoss(reduce=False)

        # (1) stastic on the train set
        with torch.no_grad():
            for i, (batch_x, batch_y) in enumerate(train_loader):
                outputs = self.model(batch_x, None, None, None)
                score = torch.mean(self.anomaly_criterion(batch_x, outputs), dim=-1)
                attens_energy.append(score)

        attens_energy = np.concatenate(attens_energy, axis=0).reshape(-1)
        train_energy = np.array(attens_energy)

        # (2) find the threshold
        attens_energy = []
        test_labels = []
        for i, (batch_x, batch_y) in enumerate(test_loader):
            batch_x = batch_x.float().to(self.device)
            outputs = self.model(batch_x, None, None, None)
            score = torch.mean(self.anomaly_criterion(batch_x, outputs), dim=-1)
            attens_energy.append(score)
            test_labels.append(batch_y)

        attens_energy = np.concatenate(attens_energy, axis=0).reshape(-1)
        test_energy = np.array(attens_energy)
        combined_energy = np.concatenate([train_energy, test_energy], axis=0)
        threshold = np.percentile(combined_energy, 100 - self.args.anomaly_ratio)

        # (3) evaluation on the test set
        pred = (test_energy > threshold).astype(int)
        test_labels = np.concatenate(test_labels, axis=0).reshape(-1)
        gt = test_labels.astype(int)

        # (4) detection adjustment
        gt, pred = adjustment(gt, pred)

        return

The adjustment function is defined as:

def adjustment(gt, pred):
    anomaly_state = False
    for i in range(len(gt)):
        if gt[i] == 1 and pred[i] == 1 and not anomaly_state:
            anomaly_state = True
            for j in range(i, 0, -1):
                if gt[j] == 0:
                    break
                else:
                    if pred[j] == 0:
                        pred[j] = 1
            for j in range(i, len(gt)):
                if gt[j] == 0:
                    break
                else:
                    if pred[j] == 0:
                        pred[j] = 1
        elif gt[i] == 0:
            anomaly_state = False
        if anomaly_state:
            pred[i] = 1
    return gt, pred

We applied point adjustments using gt, pred = adjustment(gt, pred) to correct some false positives and false negatives.

Additionally, we set a manual threshold with threshold = np.percentile(combined_energy, 100 - self.args.anomaly_ratio), where combined_energy is the combined energy score of the training and testing sets, and it is a hyperparameter. The anomaly_ratio used for the different datasets are shown in the table below:

As shown in the table, we did not apply excessive manual intervention for specific datasets.

Of course, this information should have been included in the original paper, which is a shortcoming of our work. We will add this detailed explanation in the revised version.

Q3: Was the ablation study in Table 5 conducted only on the ETTh2 dataset?

Following your suggestions, we have added ablation experiments on several additional datasets, and we have included the results in the third part of the global response.

Q4: The Exchange dataset belongs to the financial domain and typically exhibits weak periodicity.

Our method does indeed maintain strong performance on some datasets with weaker periodicity, which may seem inconsistent with our focus. However, our approach involves more than just the periodic pyramid; it also incorporates time series decomposition. The success of our method on trend-dominant datasets like Exchange is largely due to this approach. Also as you mentioned the DLinear method, which also employs the operation of temporal decomposition, so in our experiments DLinear achieved very good performance on the Exchange dataset. We have provided a detailed response in the global reply, specifically in the fourth section.

We had overlooked this phenomenon in the original paper, and your insight has helped us recognize it.

Thank you once again for your valuable feedback. If you have any further questions, please feel free to ask.

Response to Reviewer jAZw

Thanks for your valuable comments. We will explain your concerns point by point.

Q1: Some important related works are missing

Thank you for your reminder. We have carefully read the paper you provided and found them very helpful for improving the background of our paper. We will include these works in the related work section of the revised version.

Q2: Experiments could be improved

Thank you very much for your valuable feedback. Here are our additions or modifications:

1. The authors should clearly state whether they reproduce the results for comparison methods or they copy the numbers from the paper.

   We apologize for the oversight in the original paper. We have reproduced some of the experimental results of the comparison methods, but due to time constraints, we also included results from the papers of these methods. Specifically, for the long-term forecasting task, we reproduced results for TSLANet and FITS, while other results were taken from previous papers:

   (1) Time-LLM and iTransformer are from Table 13 of the TSLANet paper.

   (2) GPT4TS, Dlinear, PatchTST, TimesNet, FEDformer, Autoformer, Stationary, ETSformer, LightTS, Informer, and Reformer are from Table 14 of the GPT4TS paper.

   We will include these details in the revised version.

2. The authors are suggested to tune this parameter for all comparison methods and provide the best results.

   We apologize for this oversight. To ensure fair comparison, we have adjusted the look-back window for other comparison methods as per your suggestion and re-conducted the experiments. We have included this information in the second part of the global reply.

3. The authors should provide the script for hyper-parameter tuning to improve the reproducibility of the paper.

   Thank you for your valuable feedback. We will explain our hyperparameter tuning script with the following example and will release the script along with all the code to improve reproducibility:

   For the long-term forecasting task on the Electricity dataset with a prediction length of 96:

   for d_model in 64 128 256 512 768
   do
   for layers in {1..5}
   do
   for top_k in {2..5}
   do
   for batch_size in 4 8 16 32 64
   do
   for learning_rate in 0.0001 0.0002 0.0005 0.001 0.002
   do
   
   python -u run.py \
   --layers $layers \
   --d_model $d_model \
   --top_k $top_k \
   --learning_rate $learning_rate \
   --batch_size $batch_size \
   --......
   
   done
   done
   done
   done
   done
   

   Based on this script, we recorded results from multiple experiments and selected the best-performing hyperparameters.

4. The ablation study is only shown on one dataset (ETTh2)

   Based on your suggestion, we have added more ablation experiments, and we have included this information in the third part of the global reply.

5. The training and inference complexity and actual time could be analyzed, as well as memory efficiency.

   Thank you for your suggestion. We have added the experiments as you recommended,and show it in the first part of the global reply section.

Q3: There is no periodic in the 'Yearly' dataset, the proposed method still performs quite well

Thank you for your insightful comments. Our method does indeed maintain good performance on datasets with weaker periodicity, which might seem mismatched with the focus of our approach. However, due to our use of time series decomposition, our method can effectively capture future trends by predicting the trend part even in datasets like Yearly that do not have obvious periodicity but exhibit significant trends. This is illustrated by the "trend part" in Figure 7 of the original paper. We have provided a detailed response in the global reply, specifically in the fourth section.

Q4: It is not clear why the proposed methods have a smaller number of parameters than PatchTST

Thank you for your insightful comments. The main reasons for this phenomenon are:

1. Number of patches: If we consider each periodic component in our method as a patch, our total number of patches is smaller than that of PatchTST. Specifically, PatchTST uses a fixed number of 64 patches (there are two versions, 64 and 42, and the 64 version was used in the experiments), while the number of patches in our method is variable. In the experiments shown in Figure 16 of the original paper, we set $k$ to 3 for both tasks, resulting in 40 and 32 patches after periodic decomposition, which is less than the 64 patches in PatchTST.

2. Model Size Variance: Not all comparison methods use the same model size. We refer to what was done in the TimeaNet experiments and retain the hyperparameters of the models of the original method, thus maximizing the best possible performance of the individual models. In the experiments shown in Figure 16, the PatchTST model has a dimension of 512 and 3 layers, while our model has a dimension of 16 and 2 layers. This difference in model size is another reason for the smaller parameter count in our method compared to PatchTST.

Q5: It seems that the trend part is not used for classification tasks.

Thank you for your insightful comments. As you noted, the trend part is not used in classification tasks (without time series decomposition) but follows the path indicated by the red arrow. In classification tasks, reconstructing the original data is unnecessary, so the trend part is not extracted and added back. Additionally, the trend part is a crucial discriminative feature for classification data, so it should not be separated from the original data before feature extraction. We acknowledge that this aspect was not explained in detail in the original paper, and we will provide a more thorough explanation in the revised version.

Thank you again for your valuable time and constructive feedback. If you have any further questions, please feel free to ask.

Response to Reviewer GBMh

Thanks for your valuable comments. We will explain your concerns point by point.

Q1: (cr. W1). Add in the main body of the paper a discussion about the training and inference time, as well as some critical results and main conclusions.

Thank you very much for your valuable feedback. Concerns about computational complexity and scalability were also raised by other reviewers. Therefore, we have supplemented our experiments in this area, including the complexity, time, and memory efficiency of training and inference. Please refer to the first part of the global reply for details. We will include this content in the revised version to make the main text more complete.

Q2: (cr. W2). Add in the main body of the paper a discussion about the significance of the improvements or about the time-accuracy tradeoff for Peri-midFormer against Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks.

Thank you very much for your valuable feedback. Our method does indeed perform worse than Time-LLM and GPT4TS on some tasks. However, due to page limitations, we placed the complexity validation in the appendix, which may affect readers' understanding of the paper. Additionally, we did lack validation and discussion on the time-accuracy trade-off in the original paper. We have addressed this in the first part of the global reply section and will include it in the revised version.

In addition, due to the limitation of the amount of content that can be displayed, we currently only supplemented our experiments on the long-term forecasting task, and if necessary, we will also supplement more experiments on the imputation and anomaly detection tasks.

Q3: L1. It is not clear how the proposed periodic pyramid can be effectively and efficiently integrated into multi-dimensional time series.

We apologize for not clearly explaining the operational mechanism of our method in the original paper. The figures only showed the operation for a single channel, and although we mentioned retaining the original channels in line 132, this did not effectively convey our approach. We should have emphasized our use of a channel-independent strategy and clarified that the figures only illustrate operations for one channel. This will be addressed in the revised version.

Thank you again for your valuable feedback, which has helped make our work more complete. If you have any further questions, please feel free to ask.

Response to Reviewer W65V

Thank you for your detailed review and questions. Please find our answers below.

Q1: Complexity and Scalability

We have adopted the suggestions from you and two other reviewers to include additional experiments, covering the complexity, time, and memory efficiency of both training and inference, to further validate the method's complexity and scalability. Please refer to the first part of the global reply section for details.

Q2: State-of-the-Art Comparison

In the original paper, we stated that our method achieved SOTA results (see line 77). Additionally, we compared the performance of various methods in the explanation of each experiment's results, highlighting the outstanding performance of our method. However, this may still be insufficient and could require more thorough discussion of the comparative results. If possible, we will provide a more comprehensive analysis of the comparative results in the revised version.

Q3: Interpretability

The abstract indeed lacks an explanation of the periodic pyramid structure in time series, which affects the understanding of the method. We will include this content in the revised version.

Additionally, the lack of a clear explanation of the fundamental principles of the periodic pyramid makes it seem unsupported. Therefore, we have re-examined our method and analyzed its fundamental principles as follows:

To demonstrate the essence of attention computation among multi-level periodic components, we need to analyze how the interactions between periodic components at different levels affect the final feature extraction.

In time series analysis, different periodic components correspond to different time scales. This means that through decomposition, we can capture components of various frequencies within the time series. The essence of the periodic pyramid is to capture these different frequency components through its hierarchical structure.

Using single-channel data as an example, and given that we adopt an independent channel strategy, this can be easily extended to all channels. Assume the time series $x(t)$ can be decomposed into multiple periodic components ${x_n}(t)$ :

x(t) = \sum\limits_{n = 1}^N {{x_n}} (t) \tag{1}

Taking two different periodic components as examples:

{x_i}(t) = {A_i}\sin \left( {\frac{{2\pi t}}{{{T_i}}} + {\phi _i}} \right),{x_j}(t) = {A_j}\cos \left( {\frac{{2\pi t}}{{{T_j}}} + {\phi _j}} \right) \tag{2}

where $A$ is amplitude, $T$ is period, and $\phi$ is phase. Due to the overlap and inclusion relationships between different periodic components, we employ an attention mechanism in the periodic pyramid to capture the similarities between different periodic components, focusing on important periodic features. When applying the attention mechanism, we have:

{Q_i} = {W_Q}{x_i}(t),\quad {K_j} = {W_K}{x_j}(t),\quad {V_j} = {W_V}{x_j}(t) \tag{3}

where ${W_Q}$ 、${W_K}$ and ${W_V}$ are learnable weight matrices. From equations (2) and (3):

{Q_i} = {W_Q}{A_i}\sin \left( {\frac{{2\pi t}}{{{T_i}}} + {\phi _i}} \right),\quad {K_j} = {W_K}{A_j}\cos \left( {\frac{{2\pi t}}{{{T_j}}} + {\phi _j}} \right) \tag{4}

Further, the dot-product attention can be expressed as:

{Q_i}K_j^T = {A_i}{A_j}\left( {{W_Q}\sin \left( {\frac{{2\pi t}}{{{T_i}}} + {\phi _i}} \right)} \right){\left( {{W_K}\cos \left( {\frac{{2\pi t}}{{{T_j}}} + {\phi _j}} \right)} \right)^T} \tag{5}

Using the trigonometric identity $\sin (a)\cos (b) = \frac{1}{2}[\sin (a + b) + \sin (a - b)]$, the dot-product ${Q_i}K_j^T$ can be further expressed as:

{Q_i}K_j^T = \frac{1}{2}{A _i}{A _j}\left\\{ {{W _Q}\left[ {\sin \left( {\frac{{2\pi t}}{{{T _i}}} + {\phi _i} + \frac{{2\pi t}}{{{T _j}}} + {\phi _j}} \right) + \sin \left( {\frac{{2\pi t}}{{{T _i}}} + {\phi _i} - \frac{{2\pi t}}{{{T_j}}} - {\phi _j}} \right)} \right]} \right\\}{\left( {{W_K}} \right)^T} \tag{6}

Based on this, considering the periodicity and symmetry of $\sin (a + b)$ and $\sin (a - b)$, when the periods of two time series components are close / same (intra-level attention in the pyramid, see the right side of Figure 3 in the original paper) or have overlapping / inclusive parts (inter-level attention in the pyramid, see the right side of Figure 3 in the original paper), the values of these two sine functions will be highly correlated, resulting in a large ${Q_i}K_j^T$ value. This indicates that the periodic pyramid model can effectively capture similar periodic patterns across different time scales.

Next, incorporating this into the calculation of the attention score:

{s _{ij}} = \frac{{\exp \left( {\frac{{{Q _i}K _j^T}}{{\sqrt {{d _k}} }}} \right)}}{{\sum\limits _{m} {\exp } \left( {\frac{{{Q _i}K _{m}^T}}{{\sqrt {{d _k}} }}} \right)}} \tag{9} where $m$ denotes the index of all key values, including $j$. It can be seen that the attention scores between highly correlated periodic components will be higher, which we have already validated in Figures 13 and 14 of the original paper.

From the above derivation, it can be seen that the attention mechanism can measure the similarity between different periodic components. This similarity reflects the alignment between different periodic components in the time series, allowing the model to capture important periodic patterns. By capturing these periodic patterns, the periodic pyramid can extract key features of the time series, resulting in a comprehensive and accurate time series representation. This representation not only includes information across different time scales but also enhances the representation of important periodic patterns.

Due to character limitations, there are parts of the derivation will be explained inside the official comment.

The above proof partially explains the effectiveness of the periodic pyramid feature extraction method. We hope this addresses your concerns. If you have any further questions, please feel free to ask.

Additions to the answer to the third question (Q3: Interpretability):

1. Qualitative analysis of the essence of the periodic pyramid:

Qualitatively analyzing the essence of feature extraction through the periodic pyramid lies in the combination of its multi-level structure and attention mechanism:

(1) Multi-level Structure: By decomposing the time series into multiple periodic components, the periodic pyramid can capture features at different time scales. This decomposition allows the model to handle both short-term and long-term dependencies at various levels.

(2) Attention Mechanism: The attention mechanism can adaptively focus on the most relevant parts of the periodic pyramid, enhancing the model's focus on important features.

(3) Feature Aggregation: By aggregating features from different levels, the periodic pyramid can generate a comprehensive representation of the time series that includes information from all periodic components. This aggregation ensures that the model can fully capture the complex dynamic patterns in the time series.

2. Additional proof of continuation of Equation (9):

Further, the attention vector ${{\bf{a}}_i}$ of ${x_i}(t)$ can be obtained as:

{{\bf{a}}_i} = \sum\limits_m {{s _{im}}} {V_m} \tag{10}

where ${V_m} = {W_V}{x_m}(t) = {W_V}{A_m}\cos \left( {\frac{{2\pi t}}{{{T_m}}} + {\phi _m}} \right)$, therefore ${{\bf{a}}_i}$ can be expressed as:

{{\bf{a}} _i} = \sum\limits_m {\frac{{\exp \left( {\left\\{ {{W _Q}\left[ {\sin \left( {\frac{{2\pi t}}{{{T _i}}} + \frac{{2\pi t}}{{{T _m}}} + {\phi _i} + {\phi _m}} \right) + \sin \left( {\frac{{2\pi t}}{{{T _i}}} - \frac{{2\pi t}}{{{T _m}}} + {\phi _i} - {\phi _m}} \right)} \right]} \right\\}{{\left( {{W _K}} \right)}^T}/\sqrt {{d _k}} } \right)}}{{\sum\limits _{m} {\exp } \left( {\left\\{ {{W_Q}\left[ {\sin \left( {\frac{{2\pi t}}{{{T _i}}} + \frac{{2\pi t}}{{{T _{m}}}} + {\phi _i} + {\phi _{m}}} \right) + \sin \left( {\frac{{2\pi t}}{{{T _i}}} - \frac{{2\pi t}}{{{T _{m}}}} + {\phi _i} - {\phi _{m}}} \right)} \right]} \right\\}{{\left( {{W _K}} \right)}^T}/\sqrt {{d _k}} } \right)}}} {W _V}{A _m}\cos \left( {\frac{{2\pi t}}{{{T _m}}} + {\phi _m}} \right) \tag{11} where $m$ is the same as in Equation (7) in the original paper and is used for selecting components that have interconnected relationships with ${x_i}(t)$. Equation (11) is the expanded form of Equation (7) in the original paper, which leads to an explanation for the good performance of the Periodic Pyramid Attention Mechanism in capturing the periodic properties of the different levels in the time series.