Revisiting PCA for Time Series Reduction in Temporal Dimension

We deeply appreciate the insightful review and will do our utmost to address the questions and weaknesses.

W1&W3: Theoretical underpinnings of PCA.
Thank you for your comments. When we initially discovered that PCA could effectively reduce the temporal dimensionality of time series data, we found it intriguing. We searched for theoretical support in existing literature but did not find any relevant references, possibly due to the counterintuitive nature of our findings. Therefore, we explained this phenomenon through a combination of PCA mechanism analysis and visualization techniques. Specifically, PCA can effectively denoise and retain the primary statistical characteristics of the data. This is supported by the principles of PCA, its applications in other domains, and the visualizations we provide. Regarding why PCA can maintain model performance despite disrupting trends and periodicities, we speculate that the presence of specific trends or periodicities in historical series is not necessarily essential for the learning process of TSA models, Instead, the presence of consistent and coherent patterns is sufficient for models to provide accurate predictions. We offer the following detailed explanation: If we assume that all historical windows in the training set exhibit an increasing trend, and we simultaneously change them to a decreasing trend while keeping the trend of the target series unchanged (also assumed to be an increasing trend), this would not significantly affect the model's learning. Essentially, the model would learn that a decreasing trend in historical series can lead to an increasing trend in future series, rather than an increasing trend leading to an increasing trend. Similarly, applying the same transformation or scaling to the periodic information in all historical windows in the training set would not significantly impact the model's learning. Therefore, although PCA may alter the trend or periodicity, it introduces new coherent patterns—such as the main directions of variation, denoised low-dimensional representations, and latent features. These new consistent features in the training set enable the model to learn effectively.

W1&W2: Comparation with other frequency-based dimensional reduction techniques.
A2: Thanks you for your useful suggestions, and we have compared PCA with FFT and DWT as your suggestion. In the experiments, the original series is first transformed from the time domain to the frequency domain using either FFT or DWT. The top k frequency components (where k is 48, the same as the number of principle components) are then selected and input these into the TSA models. The results are shown in Table A. It is evident that the top k frequency components obtained using FFT or DWT fail to accurately capture the key information in the original series and effectively compress the series, leading to a significant decrease in model performance. We have also included these contents in Section G of the Supplemental Materials.

Table A: Comparison of PCA with FFT and DWT as series reduction methods. Lower MSE/MAE indicates better performance. Bold font represents the superior result.

W4: Explanations of the metrics.
A4: Thank you for your valuable feedback. We appreciate your suggestion to enhance the clarity of our tables by providing explanations of the metrics used and clarifying what constitutes a "good" or "bad" result. For TSF and TSER tasks, the metrics MSE, MAE, and RMSE are all better when lower. For TSC tasks, the metric Accuracy is better when higher. In the revised paper, we have indicated this in the captions of tables.

W5: Detailed visualizations.
A5: Thanks for your comments. The detailed effects of PCA are illustrated in Figure 7 of the Supplemental Materials. We previously misunderstood and thought that the Supplementary Material needed to be submitted separately from the main text, so we placed it in a zip file. In the updated version, we have integrated the Supplementary Material with the main text. From Figure 7 we can see that PCA series include the primary information of the original series with a small subset of initial values (principal components), while the remaining values exhibit minimal fluctuations.

Q1: PCA’s effectiveness.
A6: Thank you for your valuable review. PCA transforms the original series into a different space by projecting them onto a new set of axes defined by the principal components. It retains only the most significant principal components while discarding the less important ones, which serves as a noise filtering mechanism. Additionally, since PCA only performs a spatial transformation, many of the statistical characteristics of the original series, such as mean, peak values, and higher-order moments, are preserved. This ensures that the transformed data retains key properties of the original data, which can be crucial for time series analysis. Moreover, while PCA can filter noise and retain statistical information, it also disrupts the periodicity and trends of the original series. However, we discuss from another perspective that periodicity and trends are not necessarily essential for time series analysis. Instead, the presence of consistent and coherent patterns is sufficient for models to provide accurate predictions.

Q2: Impact of the number of dimensions on model’s performance.
A7: Thanks for your comments. The Impact of the number of dimensions on model’s performance is illustrated in Figure 6 of the Supplemental Materials. In the updated version, we have integrated the Supplementary Material with the main text. From Figure 6 we can see that as the number of principal components increases, the importance of the selected features also increases, but the rate of increase diminishes. However, after the number of principal components reaching to 48 (the number chosen in our experiment), further increasing the number of principal components results in minimal change in feature importance.

Q3: Eigenvectors.
A8: When selecting eigenvectors, we choose the largest n eigenvectors because they correspond to the directions of maximum variance in the data. By retaining these eigenvectors, we ensure that the transformed data captures the most significant features and information, thereby effectively reducing the dimensionality while minimizing information loss.

Dear Reviewer ohhk,

We would appreciate it if you could let us know whether our responses have adequately addressed your concerns. We are happy to address any further questions or concerns you may have. Thank you!

Best wishes,
Paper6918 Authors

Dear Reviewer ohhk,

The above comment to Reviewer KQnv also applies to you, as the authors have provided an extensive response to your review, including additional experimental results.

The discussion period is almost over, so please read the responses the authors of submission 6918 have provided to your review. Please specify which of your concerns were addressed and explain your decision to update or not update your score.

All the best,

The AC

Q6: Figure 3 and more datasets.
A13: Thanks for your suggestions. We conducted tests on more datasets and found that PCA has similar effects in denoising and retaining the statistical information of the original series. Specifically, we have added the experiments on multiple UCR datasets for TSC tasks. The results in Table A show that PCA preprocessing retains the principal information of the series on the UCR datasets, matches the TSC performance of the original series, and enables faster training/inference. And we also applied PCA to the commonly used TSF datasets, Electricity and Traffic. The results in Table B show that PCA preprocessing retains series principal information on Electricity and Traffic datasets, matching TSF performance with original series, and enabling training/inference acceleration.

Table A: TSC experiments on UCR datasets. The * symbols after models indicate the application of PCA before inputting the series into the models. Accuracy metric is adopted. Bold font represents the superior result.

Table B: TSF experiments on the Electricity and Traffic datasets. Bold font represents the superior result.

Q7: Explanation of trends and periodic patterns in historical series.
A7: Thanks for your comments. Applying PCA to time series disrupts the original periodicity and trends. However, through experiments, we found that despite this disruption, the model still achieves similar performance as before. We have provided our explanations of this phenomenon: if we assume all historical windows in the training set exhibit an increasing trend, and we simultaneously change them to a decreasing trend, while keeping the trend of the target series unchanged (also assumed to be an increasing trend), this would not affect the model's learning. Essentially, the model would learn that a decreasing trend in historical series can lead to an increasing trend in future series, rather than an increasing trend leading to an increasing trend. Similarly, applying the same transformation or scaling to the periodic information in all historical windows in the training set would not greatly impact the model's learning. Through the observations, we want to show that the presence of specific trends/periodicities in historical series is not necessary for the learning process of TSA models. Instead, the presence of consistent and coherent patterns is sufficient for models to provide accurate predictions. Therefore, although PCA may alter the trend or periodicity, it introduces new coherent patterns (equivalent to applying the same transformations to all historical windows)—such as the main directions of variation, denoised low-dimensional representations, and latent features. These new consistent features in the training set enable the model to learn effectively.

We deeply appreciate the insightful review and will do our best to address the questions and weaknesses.

W1&Q1: Dimensionality reduction along the time dimension.
A1: We apologize for any confusion. We would like to clarify that throughout the paper, PCA is specifically applied for dimensionality reduction along the time dimension in time series data. All our experiments involve using PCA to reduce the dimensionality of the original time series (compressing the time steps) before feeding the reduced series into the models. For classification tasks, the original series lengths (time steps) range from 152 to 1751, which we reduce to 16, 48, or 96. For forecasting tasks, the original series length is 336, which we reduce to 48. For extrinsic regression tasks, the original series lengths are 84 or 266, which we reduce to 16 and 48, respectively.

W2&Q2: PCA in classification tasks.
A2: We apologize for any inconvenience in understanding. For the three TSA tasks, PCA is initially fitted on the training data. During inference, the pre-fitted PCA model is applied to the test data without re-fitting, resulting in a significant reduction in inference time, as demonstrated in Figure 4 and Table 9.

W3&Q3: Description of the related work.
A3: Thank you for your comments. We have re-read the paper carefully, but we did not find errors in our understanding or description of the paper. First, the experiments in the paper use multiple variables to predict a single target variable. Second, the method in that paper does not process the target variable series; instead, it reduces the dimensionality of the M covariate series to P series, without shortening the length of each series. Third, we merely speculate that other covariates series may have limited utility for predicting the target variable, and there are many studies [1,2] show that channel-independent is more effective for time series forecasting. However, we did not claim that the covariate series are definitely useless for predicting the target series. We would greatly appreciate it if you could provide detailed feedback on any inaccuracies in our descriptions and offer specific reasons.

[1] Zeng, Ailing, et al. "Are Transformers Effective for Time Series Forecasting?." arxiv preprint arxiv:2205.13504 (2022).
[2] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." arxiv preprint arxiv:2211.14730 (2022).

Q4: Figure 3 and Dimensionality reduction along the time dimension.
A4: We apologize for any confusion. We would like to respectfully clarify that in our paper, PCA is applied to the time dimension rather than the feature dimension. In our study, each time step is considered a 'feature' for PCA process. Additionally, Figure 3(a) shows the effect of applying PCA to the series and then inversely transforming them back to the original series. The PCA-inversed series is significantly smoother than the original series, indicating that PCA effectively filters out noise while preserving essential features. Figure 3(b) is intended to demonstrate that the distribution of statistical characteristics of the PCA-inversed series are similar to those of the original series. Neither of these subplots aims to demonstrate the effectiveness of PCA in reducing dimensionality along the feature dimension or the time dimension.

Q5: Figure 3 and impact of the number of dimensions on model’s performance.
A5: Thanks for your comments. Figure 3 is a schematic diagram intended to illustrate the denoising effect of PCA and its ability to retain the statistical information of the original series, rather than to present experimental results. The Impact of the number of dimensions k on model’s performance is illustrated in Figure 6 of the Supplemental Materials. In the updated version, we have integrated the Supplementary Material with the main text. From Figure 6 we can see that as the number of principal components k increases, the importance of the selected features also increases, but the rate of increase diminishes. However, after k reaching to 48 (the number chosen in our experiment), further increasing k results in minimal change in feature importance.

Dear Reviewer KQnv,

The discussion period is almost over, so please read the responses the authors of submission 6918 have provided to your review.

Please specify which of your concerns were addressed and explain your decision to update or not update your score.

All the best,

The AC

We sincerely appreciate the valuable review and will do our best to address the questions and weaknesses.

W1&Q1: Comparation with other representation learning methods.
A1: Thank you for your insightful comments and references. We have carefully read the methods you mentioned. While these representation learning methods are effective for feature extraction and series compression in specific scenarios, they are not general-purpose approaches and cannot be easily integrated with arbitrary time series models or tasks, and obtaining time series representations (compressing time series) is just the initial part of these methods. For example, in the TS2Vec framework, an SVM classifier is required for TSC tasks, while a ridge regression model is needed for TSF tasks. Therefore, these methods are more specialized models rather than pluggable, general-purpose frameworks or modules, which is inconsistent with our goal for PCA. Additionally, the primary objective of these representation learning methods is to learn better representations rather than to accelerate training/inference. As a result, they typically do not optimize for training efficiency or memory usage as extensively as PCA does. As your suggestions, we compared PCA with these representation learning-based methods on TSC tasks. As shown in Table A, the Linear + PCA model achieved the best performance in most settings. Additionally, PCA requires less training/inference overhead and less GPU memory compared to representation learning methods, as shown in Table B. We have also included these contents in Section I of the Supplemental Materials.

Table A: Comparation of PCA with T-Loss, TS2Vec, and TimeVQVAE on TSC experiments. The accuracy metric is adopted. Bold font is the superior result.

Table B: Computational efficiency, and memory usage comparation of PCA with T-Loss, TS2Vec, and TimeVQVAE. Bold font is the superior result.

W2&Q2: Comparation with other classification baselines.
A2: Thanks for your comments. In the top-tier study TimesNet[1], models such as Linear, FEDformer, and Informer are also used for TSC tasks, we also followed their experimental settings and tested the same models and datasets. As your suggestions, we applied PCA to SOTA classification models, InceptionTime[2] and ResNet[3]. The results are shown in Table C. It is evident that PCA is model-agnostic and remains effective even when applied to these SOTA classification models. We have also included these contents in Section J of the Supplemental Materials in the revised paper.

Table C: TSC experiments of Inception and ResNet. The accuracy metric is adopted. The * symbols after models indicate the application of PCA before inputting the series into the models. Bold font is the superior result. PCA preprocessing retains series principal information, matching TSC performance with original series, and enabling training/inference acceleration.

[1] Wu, Haixu, et al. "Timesnet: Temporal 2d-variation modeling for general time series analysis." arxiv preprint arxiv:2210.02186 (2022).
[2] https://github.com/TheMrGhostman/InceptionTime-Pytorch/blob/master/inception.py
[3] https://github.com/hsd1503/resnet1d/blob/master/resnet1d.py

Q1: Title of section 3.2.
A3: Thanks for your helpful suggestion. Using "intuitional justification" as title would be more appropriate and we have updated the title in the revised paper.

Q2: More TSC datasets.
A4: Thanks for your comments. We adopted the TSC datasets used in the TimesNet[1] paper and selected series with long length for our experiments (series with short lengths have limited benefit from PCA dimensionality reduction). We have added the experiments on multiple UCR datasets based on your suggestions. The results in Table B show that the application of PCA doesn't decrease accuracy; Instead, it slightly improves performance in some cases. More importantly, the computational cost is greatly reduced, showing the effectiveness of PCA compression processing. This PCA processing aligns with the concept of Pareto optimization, improving or maintaining accuracy while greatly reducing computational resources. We have also included these contents in Section H of the Supplemental Materials.

Table B: TSC experiments on the UCR datasets. The * symbols after models indicate the application of PCA before inputting the series into the models. The accuracy metric is adopted. Bold font represents the superior result.

[1] Wu, Haixu, et al. "Timesnet: Temporal 2d-variation modeling for general time series analysis." arXiv preprint arXiv:2210.02186 (2022).

Q3: Accucary of Table 2.
A5: Thanks for your careful observations.The first two datasets are challenging for TSC. We retested them and obtained the same results. We also verified the results reported in the TimesNet[1] and found that their method also struggled with classifying these datasets. We speculate that the following reasons contribute to this phenomenon: First, these datasets have fewer samples, which introduces some randomness in the results. Additionally, different models have varying difficulties in capturing the features of different datasets. For a specific model, PCA preprocessing may make some datasets easier to learn while making others more difficult. However, from the overall results of TSC, TSF, and TSER, PCA preprocessing does not degrade model performance and can accelerate training and inference while reducing memory usage.

Q4: Backbone in Table 5.
A6: We apologize for any inconvenience. The backbone model for the results in Table 5 is the Linear model.

Q5: Explanation of positive trends.
A7: We apologize for any inconvenience in understanding. Here is our detailed explanations: if we assume all historical windows in the training set exhibit an increasing trend, and we simultaneously change them to a decreasing trend, while keeping the trend of the target series unchanged (also assumed to be an increasing trend), this would not significantly affect the model's learning. Essentially, the model would learn that a decreasing trend in historical series can lead to an increasing trend in future series, rather than an increasing trend leading to an increasing trend. Similarly, applying the same transformation or scaling the periodic information in all historical windows in the training set would not significantly impact the model's learning.
Through these observations, we want to show that the presence of specific trends or periodicities in historical series is not necessarily essential for the learning process of TSA models. Instead, the presence of consistent and coherent patterns is sufficient for models to provide accurate predictions. Therefore, although PCA may alter the trend or periodicity, it introduces new coherent patterns (equivalent to applying the same transformations to all historical windows)—such as the main directions of variation, denoised low-dimensional representations, and latent features. These new consistent features in the training set enable the model to learn effectively.

We deeply thank the valuable review and do our best to address questions and weaknesses.

W1: The purpose of PCA.
A1: Thanks for your comments. In our study, PCA is employed as a pluggable, general-purpose preprocessing technique for time series analysis (TSA), which can be integrated with various TSA models and applied to different downstream TSA tasks. While the early layers of neural networks may have similar effects, it's necessary to design different early layers for different neural networks to ensure that these layers can effectively extract series features and reduce dimensionality. Additionally, adding a dimensionality reduction layer at the beginning of the existing neural network might increase the training/inference burden and raise the risk of overfitting. Furthermore, we also experimented with adding a linear/1D-CNN dimension reduction layer before the original neural network to achieve dimensionality reduction. However, the experimental results shown in Table 6 indicate that their performance is inferior to that of PCA-based dimensionality reduction.

W2: Comparation with other frequency-based dimensional reduction techniques.
A2: Thanks you for your useful suggestions, and we have compared PCA with FFT and DWT as your suggestion. In the experiments, the original series is first transformed from the time domain to the frequency domain using either FFT or DWT. The top k frequency components (where k is 48, the same as the number of principle components) are then selected and input into the TSA models. The results are shown in Table A. It is evident that the top k frequency components obtained using FFT or DWT fail to accurately capture the key information in the original series and effectively compress the series, leading to a significant decrease in model performance. We have also included these contents in Section G of the Supplemental Materials.

Table A: Comparison of PCA with FFT and DWT as series reduction methods. Bold font represents the superior result.

Thanks to the authors for your detailed feedback with new empirical results. After very careful consideration, although I appreciate very much the new empirical evidence and agree that it has greatly improved the quality of the paper, I decided to maintain my original score.

The major concern is that I am still not fully convinced by the main motivation, i.e., PCA should be taken as a versatile plug-in before feeding time series into neural networks. The main reason is that usually one would like to have all information fed into the neural network, which is usually carefully designed and will be carefully trained under proper constraints. Performing dimension reduction before feeding the input data into the neural network, is sort of implicitly suggesting that the neural network lacks the ability to identify the necessary information through the noise (which, at the same time, can be removed by the simple PCA method).

I would suggest the authors to set up a stronger motivation for the adoption of PCA. One possible direction is considering the unique property of time series datasets, for example, the task-specific datasets are mostly pretty small but still with noise, while training neural networks is also notoriously hard.