GENERATIVE TIME SERIES LEARNING WITH TIME-FREQUENCY FUSED ENERGY-BASED MODEL

We thank the reviewer for offering the valuable feedback. We have addressed each of the concerns raised by the reviewer as below.

W1.They explained their model well, but the placement of Figures and Tables is not appropriate.

● We rearranged some tables and figures in the updated paper .

W2.Figure 1 is not intuitive. I don't know what the "+" mark next to the Time-Freq Block part means. Also, the inference part in Figure 1 is more difficult to understand.

● We updated both panels of Figure 1 to make them clearer in the revised paper.

● The "+" mark means we add the time and freq feature as indicated in Equation (7).

● The Inference part at the bottom panel in Figure 1 is just a standard Langevin MCMC process, which shows how to sample one time step from $y_{1:T}$to $y_{T+1:T+\tau}$with 59 steps of Langevin sampling.

W3.The table mentioned in the ablation study is difficult to see because it is not in the main paper.

● This is unfortunate due to the space limit of the main paper.

W4.The overall structure of the paper is not friendly to readers.

● We rearranged some tables and figures to make the updated paper more reader friendly. However, we were unable to include all tables and figures to the main paper due to the space limit imposed by ICLR.

Q1.Isn't there a process in Equation 6 to invert from the fequency domain back to the time domain? If so, I don't understand how the fequency domain feature and time domain feature are added in Equation 7.

● No, we do not invert the frequency domain back to the time domain. As the goal of the training the energy based model is to get well estimation of the unnormalized probatlity density for temoral path, we extract the time domain and freqency domain feature for each layer, then add the time domain and frequency domain features, which yields better results in our experiments that concating them together.

Q2.The residual dilated convolutional network part in the 4.4 ablation study does not seem to have any meaning. There is no doubt that performance increases as the number of layers increases because the capacity of the model increases. Rather, it seems meaningful to use the same number of layers but subtract the residual part.

● We try to show in this ablation study that the multi-layer rather than a single-layer dilated convolutional neural network is necessary to filter out the noise in the time series. It is well-known that some high frequency modes could be noise instead of valuable signal. Since our TF-EBM model keeps all the frequency modes out of the FFT, which is different from some other works (FedFormer, Fast Fourier Convolution, etc. ) that keeps a portion of them, we reply on such multi-layer dilated CNN to automatically extract useful modes.

We thank the reviewer for offering the valuable feedback. We have addressed each of the concerns raised by the reviewer as below.

Q1: It's unclear how significant the resulting improvements are, the numbers. Can the authors quantify if the improvements are significant? (instead of just reporting means, report the errors too). Also, this might be a typo but Table 3 first row FEDformer MSE is the lowest.

● Our TF-EBM model, being a probabilistic time series model, is evaluated using the CRPS metric to assess its distribution modeling capabilities. To strengthen the comparison, we have incorporated the TimeGrad model, another prominent EBM-based time series model, into Table 1. Additionally, we have included extended CRPS results for longer forecast horizons of 336 and 720 time steps in Table 8 of the Appendix. The improvements over state-of-the-art probabilistic models can be substantial, with a 30% reduction in CRPS for traffic data. Furthermore, we present both MAE and MSE metrics based on the error between the mean value of the forecasted distribution and the ground truth. Employing the mean value as the point estimate allows for a fair comparison with recent transformer-based models.

● The typo in the MSE field inTable 3 for FEDformer is fixed in the updated paper.

Q2: It could be worth discussing why certain differences in results arise, e.g. Table 2 DeepAR seems to perform basically identical to the proposed method for exhange rate and electricity datasets, but performs noticeably worse on the other two datasets. What's the reason? This could provide insight into where the proposed method can provide maximum improvement (and why).

● The exchange rate and electricity datasets contain 8 and 370 time series, respectively, while the traffic and Wikipedia datasets have 963 and 2000 series, significantly larger than the other two datasets. The exchange rate and electricity datasets have relatively high autocorrelations (>90% as shown in Table 6) which could make them suitable for autoregressive models such as DeepAR, especially in short-horizon forecasting.

● We performed additional experiments by extending the forecast horizon to 336 and 720 time steps, using the CRPS metric to evaluate performance. As shown in Table 8 in the Appendix of the updated paper, our TF-EBM outperforms DeepAR by a substantial margin for the electricity and exchange rate datasets, except for the electricity dataset with a 336 forecast horizon. This demonstrates TF-EBM's superior long-term forecasting capability, while DeepAR struggles with error accumulation

Q3: The paper mentions that deterministic methods generally don't help with uncertainty quantification, that sets the reader up to seeing uncertainty quantification being addressed by the paper, but it doesn't seem like it has been addressed in text or in experiments.

● Our TF-EBM model's uncertainty quantification capability stems from its probabilistic nature. As a probabilistic time series model, TF-EBM generates probability distributions for future values, allowing us to sample from these distributions to obtain the empirical quantiles of the uncertainty of our predictions. This probabilistic approach enables us to readily quantify the uncertainty associated with our predictions. For example, we can output the 70% confidence interval along with the mean value as shown in Figure 3 in the appendix. In contrast, point forecast models like TimesNet and AutoFormer only provide single-valued predictions for each time step, lacking the ability to capture the inherent uncertainty in time series data.

● To further demonstrate the uncertainty quantification capability, we performed an experiment in Section G in the appendix of the updated paper. The synthetic ground truth is injected with different levels of noise. We output the 99% confidence interval along with the mean forecasts. One can observe that the model forecasts in the time steps with larger noise variance show wider confidence interval than those in the low variance time stpe. The uncertainty of the forecasts is consisitent with the level of the noise.

We thank the reviewer for offering the valuable feedback. We have addressed each of the concerns raised by the reviewer as below.

W1.The main weakness I see in the paper is that clear procedures for separating model development from the experiments evaluating model skill were not described.

● The datasets we used to evaluate our model performance are all public datasets that are widely available and heavily tested in other time series forecasting literature. We used the same dataset splitting ratio as in other published papers for training and testing data.

● For the CRPS resutls in Table 2 and 8, we followed the same splitting ratio as in [1].

● For all the MAE/MSE results in Table 3, 4, 5, 6, and 9, we used the same data preparation code released by the authors of the TimesNet paper: https://github.com/thuml/Time-Series-Library/blob/main/data_provider/data_loader.py

● In order to evaluate model's performan on new datasets, we included the transfer learning experiments in the Appendix of the revised paper. To be specific, we pre-train our model on the electricity dataset, and fine tune it on traffic and four ETT datasets using just three epochs. The results are summarized in Table 9. The fine-tuning performance is still better than or comparable to other models.

[1]Richard Kurle, Syama Sundar Rangapuram, Emmanuel de Bézenac, Stephan Günnemann, and Jan Gasthaus. Deep rao-blackwellised particle filters for time series forecasting.Advances in Neural Information Processing Systems, 33:15371–15382, 2020.

Q1.I found equations 6 and 7 to be helpful, but they did not seem to align precisely with Figure 1 and did not completely fill out all details of the model. A few specific questions are below:

● a)The Dildated Conv and Conv layers shown inside the Time-Freq Block (TFB) in Figure 1 do not represent the g operator. Note that we named it as DSC (DilatedConv -> SiLU -> Conv1d) network in the updated paper to make it clearer. The g operator (g_time and g_freq) is only applied in the Time Feat Moudule and Frequency Feat Module in Figure 1 which is formalized in the Eq 6. The Time-Freq Block shown in Figure 1 comprises two compents: one is the DSC network, and the other is the time-frequency (TF) domain features extracting network. The Figure 1, Eq.6 and Section 3.5 are all updated in the revised paper.

● b)The inputs to the DSC network shown in the sub-blocks within TFB on the left side of Figure 1 are not the time feats or frequency feats. The inputs to the DSC network of the L-th layer are the sum of the inputs and outputs of DSC of the (L-1)th layer (defined by Eq 5 in the updated paper). The inputs to the time and frequency feats are the outputs of the DSC network (red arrow in Figure 1). The outputs of the time and frequency feats are added together to produce the ultimate energy scalar defined by Eq 7.

● c) We have named DilatedConv -> SiLU -> Conv1d as DSC in updated paper, so Eq 5 has been updated as $y^l=y^{l-1}+DSC(y^{l-1})$ to clarify the statement.

Q2.How were missing data values (e.g., as in imputation settings) handled in computation of the FFT?

● For both imputation and forecasting tasks, missing values are replaced with random Gaussian noise prior to the FFT transform, followed by Langevin sampling to generate model-derived samples.

Q3.The abstract states that long-term probabilistic forecasting is an important problem, but we do not see CRPS results for the long-term forecasting example. Could a more complete suite of results be added?

● We included CRPS results for the long-term (forecast horizon equals 336 and 720) forecasting in Table 8 in the Appendix. We compared against three models: DeepAR, DeepState, and TImeGrad, but not the RSGLS-ISSM or ARSGLS because their code implementations were not publicly available. We did not test the Wiki dataset as it only has 792 data points that are not enough for the long-term forecast.

Q4.The ablation study looking at the contribution of time and frequency extraction only looks at a subset of the data sets. I think it would be interesting to provide results for all data sets used in the paper.

● We performed the same ablation study on the exchange rate and traffic datasets as well. The results are included in Table 6 in the Appendix, which also includes the average of the lag-1 autocorrelation, standard deviation, and four dominant power spectrums. However, such characteristics of the timer series does not seem to have strong correlation with the relative value of using both feature groups. Actually,the model equipped with time and frequency feature extracting module performs the best on almost all datasets and forecasting ranges compared with only equipped with either time or frequency extracting module. The three exceptions on the ETTm1 and traffic datasets do not seem to have strong statistical significance.

W1.The motivation for using energy-based learning specifically is not clearly articulated. References connecting energy-based models to time series properties could help.

● Energy based model (EBM) has been widely applied in the image generation because EBM models high-dimentional joint distribution via a flexible unnormalized probability density benefitting deep model design. As for the time series, forecasting task, especailly for long-term forecast needs to model the long-term temporal coherence, and EBM can naturally serve as a powerful tool to model the joint distribution of the long-term correlation. There are limited number of papers applying EBM on time series forecasting works to the best of our knowledge. Previous EBM models used in time series forecasting including Time Grad/Score grad cited in the Related work section focus on modeling the join distribution across time series at each time step, while we utilize the EBM to model the joint distribution along the temporal path of the time series, which can guarantee the maximum likelihood of the whole sequence.

W2.More analysis on why the time-frequency modeling outperforms just time or just frequency features could strengthen this contribution.

● We consider the particularity of time series as seqence data which can be descirbed in the time and freqency demension. As we laid out in Section 3.5 and references therein, the time and frequency dimension mostly characterize the local and global patterns of the time series, respectivley, that are both crucial for long-term time series forecasting. Therefore, it is a natural choice to combine both patters to enhance the model performance.

● We added additinoal ablation study on the exchange rate and traffic datasets with results included in Table 6 in the Appendix. The results on the full datasets show that combing both time and frequency features contribute better forecasting ability in our model design.

W3.The pre-training evaluation is limited. More exhaustive experiments on the transfer learning capabilities could be done.

● We evaluated the transfer learning capabilities of our model by pre-training it on the electricity dataset and fine-tuning it on the traffic and four ETT datasets. The results, summarized in Table 9, demonstrate that the fine-tuned model outperforms or matches the performance of other models.

W4.Only univariate time series are evaluated. Comparing with PatchTST is needed in this setting.

● As we mentioned in both the Introduction and the model framework section, unlike PatchTST, TF-EBM is an univaraite time-series model, and we will extend it to the multivaraite case as we mentioned in the Future work section. We wil compare with PathTST by that time.

W5.The comparison to autoregressive models like LSTMs is missing. This could reveal advantages over common recurrent approaches.

● We showed DeepAR model results in Table 1 for the CRPS metric. DeepAR is a strong autoregressive model based on LSTM. Additionally, in both Table 1 and 8, we added results from the TimeGrad model, which is a strong autoregressive EBM model.

● We appended additional column in Table 3 for the MAE/MSE metric obtained using LSTM model. They are also taken directly from TimesNet paper.

● TF-EBM outperforms both DeepAR and TimeGrad on most of the datasets.

We thank the reviewer for offering the valuable feedback. We have addressed each of the concerns raised by the reviewer as below.

W1.The paper got good results in several benchmarks but lack of novelty in two ways. If the author claims to improve EBM based learning method in time series, the author should discuss more results about MCMC sampling results, e.g. the chain mixing problem. MCMC chain mixing is a well-known issue in non-sequential signals, it could be worse in time series. If the author claim the novel parameterization of EBM and specially designed fusion block (i.e. the important inductive bias for time series), I would suggest add more insightful ablation studies. There are many design choices for fusion. The author should clarify the benefits of current design choice.

● As explained in Section 3.2, we opted for the short-run MCMC technique due to its efficiency in accelerating the Langevin sampling process. Through our experimentation, we observed that even non-mixing chains have the capability to produce favorable outcomes, as demonstrated by the experiment results.

● Our main novelty is choosing EBM as the backbone to model the long-term temporal dependence of time series. The existing energy-based models applied in time series fields like Time Grad/Score Grad model the joint distribution of multiple time series at each time step, and use an autoregressive structure like RNN to model the temporal dependence which can suffer from error accumulation. However, we use EBM to model the joint distribution of the whole temporal path of each time series ( $y_{1:T+\tau}$) innovatively. During training and predicting process, we do not perform gradient update on the context$y_{1:T}$, but we only update the predicted values $y_{T+1:T+\tau}$following the Langevin dynamics with calculated gradient. Therefore, the forecasts $y_{T+1:T+\tau}$can be sampled at once benefiting the long-term forecasting, which is demonstrated in the extensive experiments.

● The main goal to applying EBM to time series modeling is to get a accurate estimator for the probability density of the entire whole temporal path, in which the time and frequency fusion architecture plays an important role. In the image generation work, the density estimation network is directly applied on the raw pixel, however, it is different in the time series field because time series can be described in both time and frequency dimension. Our novelty here is to use the DSC network defined in Section 3.4 of the updated paper to extract different levels of information of the time series. Specifically, we apply a time and frequency feat extracting network to extract local and global features, respectively, which are then added together to generate the ultimate energy scalar. Additionally, the ablation study conducted in Section 4.4 and Appendix C further demonstrates the effectiveness of our chosen modeling approach. It is worth noting that alternative fusion methods like concatenation or gating may potentially yield even better results, and therefore, in our future work, we plan to thoroughly investigate and explore these alternatives.

● In summary, our paper introduces a novel EBM-based encoder-only generative time series model that leverages fused time and frequency features.

○ Our main novelty is to choose EBM as the backbone due to its flexibility as a general distribution modeling tool, as the energy function can be any scalar-returning function and does not need to integrate to 1, making it particularly well-suited for neural networks. Moreover, to our knowledge, there is a limited body of work applying EBM to time series analysis tasks.

○ The model generates forecasts in one forward run unlike autoregressive models which suffers from inefficiency and error accumulation especially for long-term forecasts.

○ The proposed versatile model demonstrates its effectiveness in various tasks, including short-term and long-term forecasting, imputation, pre-training, transfer learning (Appendix F), and uncertainty quantification (Appendix G). These extensive experiments validate the model's performance using the current model design.

○ The model's pre-training capability makes TF-EBM a promising candidate for a time series foundation model to scale-up time series training.

W2.Meanwhile, the author missed some pioneering literature about basic EBM

● We have added the pioneering literature mentioned about basic EBM in the updated paper.