Frequency-aware Generative Models for Multivariate Time Series Imputation

Thank you for the thoughtful comments and valuable suggestions. We have tried our best to incorporate the suggestions in the revised version. Below, we provide our response to the questions and concerns.

1. (W1) The formulation and writing of this paper should be improved

We appreciate your feedback on improving clarity and grammatical accuracy. We have meticulously revised the manuscript, correcting errors such as "Recently, researchers attempt to utilize large language models" to "Recently, researchers have attempted to utilize large language models" (line 62), and ensuring all sentences are structurally sound and clearly presented.

2. （W2）Explain how multiple scales of seasonality and trend features in time series affect the residual’s usefulness, which may be too noisy

Thank you for your comments. Since our method does not explicitly perform STL decomposition, it is not affected by the multiple scales of trends and seasonality. Our method leverages Fourier transform to analyze both dominant-frequency and high-frequency information, which inherently addresses multiple scales of trend and seasonality. By our dominant-frequency filter, it is possible to capture multiple scales of trend and seasonal feature information. In addition, even when the high-frequency information that mainly contributes to the residual is noisy, we can still use the dominant-frequency information and adjust the weights of the high-frequency condition by attention layers for accurate imputation. As shown in Table 1, our method always outperforms other methods in different scenarios.

3. （W3）Straightforward and may limit its contribution

Our core insight is that existing methods cannot accurately imputation the residual term, therefore introducing the corresponding frequency-domain information. The insight of modelling high-frequency signals for time series imputation has not yet been recognized by existing studies. Our proposed filters and representation learning modules are simple and effective for imputation, and can shed light on further works. In addition, our proposed filters and representation learning modules can also be flexibly ported to other generative models and are not limited to applying diffusion models.

4. （Q1）Explain why the absence of the High-frequency filter impacts performance differently on KDD, Guangzhou, and PhysioNet datasets

Thank you for your question. The difference in impact of dominant-frequency and high-frequency information depends on the characteristics of datasets. For the PhysioNet dataset, the obtained dominant-frequency information may be biased due to it contains at least 79.71% missing values. For the KDD and Guangzhou datasets, the dominant-frequency information will play a larger role because of the lower missing rates. In these cases, our method can adaptively adjust the weight of high-frequency information and dominant-frequency information to help the imputation model.

5. （Q2）Hyper-parameters for the experiments chosen

For the two critical hyperparameters of the high-frequency filter and the dominant-frequency filter, we conduct empirical search to identify the optimal hyperparameters, as shown in Fig. 9 and Fig. 10 in Appendix A.4.2. For other settings related to diffusion model, we adopt hyperparameters recommended by the existing well-established models [23,43]. These models have demonstrated strong performance in similar tasks, and their hyperparameters have been extensively validated in those literature. To improve the presentation of the hyperparameters, we will add the above description of the parameters for the diffusion model to Appendix A.4.2.

6. （Q3）Purpose of Proposition

The purpose of Proposition 3.1 is to illustrate the mechanism behind introducing the high-frequency condition and the dominant frequency condition for facilitating generative models to impute missing values. This Proposition shows that with the introduction of the two conditions, the entropy of the target distribution will be reduced, the randomness of the target distribution will be narrowed, so that the generative models will more accurately impute missing values. To add empirical evidence for this proposition, we also add experiments using CRPS to evaluate the gap between the learned and ground truth distributions for different generative baselines in the following table:

As shown in the above table and Table 1, our method outperforms other methods for all metrics due to the introduction of frequency domain conditions, thus echoing the proposition.

We really appreciate your thoughtful comments and valuable suggestions. We have incorporated the suggestions in the revised paper. Below, we provide our response to the concerns.

1. (W1) Clarify if temporal dependency modelling is sufficient

We would really appreciate your feedback. We apologize for the inadequate illustrations and explanations of the time-frequency and attribute-frequency representation learning modules In line 144, we state that the Encoder(.) is implemented by the transformer backbone. Since a position encoding layer exists in this encoder, we can add the timestamp information for the corresponding Query and Key in the two representation learning modules. To improve the presentation, we add a figure illustrating the detailed architecture of the denoising network, which is uploaded in the supplementary pdf file at the top of the page. And we will add this figure to Section 3.3. In addition, the setup where modelling temporal dependencies than modelling attribute dependencies by two attention layers is common in time series imputation methods [26,43]. Considering that our method outperforms other imputation methods for different missing rates and different missing mechanisms in Table 1, Fig. 4, Fig. 7 and Fig.8, the modelling of the time dependence is sufficient.

2. (W2) Describe the model architecture and specifications used for Projectors in the denoising network

Thank you for your suggestions. We apologize for the inadequate description and explanation of the detailed structure of the denoising network. As shown in Fig.2, which is uploaded in the supplemental pdf, the input projector is an MLP layer, and the output projector is a 2-layer MLP with a ReLU activation function. This setup is consistent with the other SOTA imputation methods [26,43], which have been extensively validated in those literature.

3. (W3) Clarify how time-frequency and attribute-frequency representations are used in the diffusion model

Thank you for your valuable suggestions for boosting our manuscript. We apologise for not having a very clear representation of the denoising network architecture of the diffusion model. The Fig.2 in supplemental pdf illustrating the detailed architecture of the denoising network will add in the Section 3.3. To achieve the imputation of generative models that are helped by the guidance of frequency-domain information, we should incorporate the representation learning modules into the generative model architecture. As shown in the figure, $\mathbf{R}^{a}$ is a hidden vector in the denoising neural network of the diffusion model.

2. (W2,Q1) Clarify why your method outperforms conventional ones, considering trend, seasonality, and missing patterns in datasets

Thank you for your valuable suggestions for boosting our manuscript. We conduct a case study of FGTI for trend, seasonal, and residual terms over the pre-decomposed KDD dataset. We first perform STL decomposition of the KDD dataset into Trend, Seasonal and Residual terms. Then we select 10% observations of the original KDD dataset as the mask positions by MCAR, and then mask the corresponding positions of the three terms. To study the role of high-frequency information, dominant-frequency information, and frequency-domain information, we consider the three ablation scenarios: (1) w/o Dominant-frequency filter (2) w/o High-frequency filter and (3) w/o Frequency condition in Section 4.3. We report the imputation results of different scenarios over different terms in following table:

We can find that for the Trend term, retaining the dominant-frequency information gives the best results, while the-high frequency information may interfere the imputation. For the Seasonal term, the results are similar to the Trend term, but the dominant-frequency information contributes less to the imputation for the Seasonal term than for the Trend term. This suggests that the Seasonl term mainly corresponds to the dominant-frequency information, but also contains some of the high-frequency information. In contrast, the results of the experiments on the Residual term show that it mainly corresponds to high-frequency information. Since we choose Transformer as the encoder in Cross-domain Representation Learning and utilize cross-attention as the fusion mechanism of the two frequency domain information in Time-frequency Representation Learning and Attribute-frequency Representation Learning, our method can self-adaptively adjust the weights of the high-frequency information and the dominant-frequency information for different timestamps. Thus our method can outperform conventional methods for datasets with any circumstances in most cases, as illustrated in Table 1.

In addition, our method's pipeline does not perform STL decomposition of the time series, but extracts the frequency domain informations to guide imputation, so we do not need to know the trend and seasonality of the dataset. Thanks to the encoder structure and cross-attention based fusion mechanism in Cross-domain Representation Learning, we can adaptively adjust the weights of the two frequency domain information on datasets dominated by different terms.

Furthermore, we consider three missing mechanisms, MCAR, MAR, and MNAR, in our comparative experiments. MCAR implies that each attribute could be missing at any frequency, while MAR potentially implies that all attributes' missing status depends on the frequency of a particular attribute (e.g., high temperatures could cause sensor failure). For MNAR, the missing status of all attributes depends on the frequency of that attribute. As shown in Fig 4, Fig 7 and Fig 8 in the manuscript, our method still outperforms conventional methods with different missing mechanisms (patterns).

We thank the constructive comments and suggestions for further experiments. We will try our best to incorporate the suggestions in the revised version. Below, we provide our response to the questions and concerns.

1. (W1) Compared with the most recent baselines

Thank you for the valuable suggestions. TimeCIB [a] is a very competitive VAE-based imputation method, so we incorporate TimeCIB into the baselines for comparative experiments. In addition, we have already considered TIDER [27] in our experiments.

We first report the part of updated Table 1 for the added baseline:

We then report the imputation results for varying missing mechanism with 10% missing values in the following table, and update Fig.7, Fig.8 and Fig.9:

After this, we report the updated resource consumption over the KDD dataset with 10% missing values in the following table, and update Fig.5:

Finally, we report on the updated downstream application experiments in the following table, and update Fig.6:

We can find that FGTI still outperforms all baselines due to the guidance of high-frequency information and dominant-frequency information, as well as the generating ability of the diffusion model.

3. (W3) Provided a breakdown (seasonality and trend) of the assessments

Thank you for your suggestions. The pipeline of our method does not perform a direct STL decomposition, but instead uses the high-frequency and dominant-frequency information to potentially guide the imputation of the three terms. Due to the effect of missing values, we will get different decomposition results before and after imputation, so for each term the ground truth will be lacking for the evaluation. Therefore, we first perform STL decomposition of the original KDD dataset, then inject the missing values for the three terms respectively to verify the imputation results, and the results are shown in Fig. 1 of Section 1. In addition, we also add a Case study about the trend, seasonal and residual terms using the pre-decomposed KDD dataset, and the results are shown in the above table. The results show that dominant-frequency information can mainly help the imptuation of the Trend and Seasonal terms, and the high-frequency information is mainly help to the Residual term.

4. (C1) Replace RMSE or MAE with examples, and clarify Figure 1 reporting details

Thank you for your valuable suggestions for boosting our manuscript. Regarding the setting for the study of Fig. 1, we first perform STL decomposition of the KDD dataset into Trend, Seasonal and Residual terms. Then we select 10% observations of the original KDD dataset as the mask positions by MCAR, and mask the corresponding positions of the three terms. Finally, we impute the missing values of the three terms by different methods separately. Following your valuable suggestion, we add the examples of imputations on trend, seasonal and residual terms in Fig. 1, which is uploaded in the supplementary pdf file at the top of the page. and add the setup for this survey study. The results also indicate that the SOTA time-series imputation methods are inaccurate in imputing the residual term.

5. (Q2) Explain how masking ratio/patterns affect model performance

Thank you for your comments. We agree that mask ratio and mask pattern are critical for the learning process. We randomly mask different ratios of observations as the imputation target at each training step, instead of using a fixed mask ratio, following CSDI [43]. The performance of FGTI with different mask ratios is shown in the following table:

We can find that since the random mask strategy can increase the learning complexity and enhance the modelling ability of the diffusion model, thus our masking strategy achieves optimal or sub-optimal performance in most cases.

Then, we explore the performance when using different mask patterns. Following CSDI [43] and PriSTI [26], we consider (1) Block missing (2) Mix missing strategy (3) Random missing mask pattern，the results is shown in the following table:

It can be found that Block missing or Mix missing is not comparable to Random missing in most cases due to the possibility that the mask pattern may not correspond to the actual missing scenario. So, we use the Random missing mask pattern by default.

Reference:

[a] Conditional Information Bottleneck Approach for Time Series Imputation

5. (W5) What cause lower MAR and MNAR improvements compared to MCAR for Guangzhou and PhysioNet

Thank you for your question. MCAR, MAR and MNAR reflect different missing scenarios in reality. For MCAR, the missing value is not dependent on other attributes. For MAR, the probability of missingness depends only on available information (e.g. missing records of traffic flow data may be related to rush hour, activities in public places). For MNAR, the missing observation depends on othe the attribute itself (e.g. reliability of recording devices). The effect of the models on different mechanisms depends on the temporal and attribute relations they capured in the dataset. Under the MAR and MNAR mechanisms, the critical temporal or attribute dependencies are more likely to be missing compared to MCAR, and therefore models may perform not well for MAR and MNAR. This also reflects the importance of introducing frequency domain information to guide the imputation.

6. (L1) The contribution of the proposed method might be incremental

Thank you for your comments. As far as we know, this is the first work to recognize the insight of paying special attention to residual term when impute time series, and introduce frequency-domain information to guide generative models. We choose to implement FGTI through the diffusion model because diffusion models are currently demonstrating excellent performance in several fields [17,23]. The frequency domain filters and the cross-domain representation learning module can be flexibly ported to other generative models. We think this insight can shed light into potential future directions in time series imputation. Therefore, we argue that our FGTI is not an incremental work.

References:

[a] MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns

[b] A wavelet-based approach for imputation in nonstationary multivariate time series

[c] Rethinking general time series analysis from a frequency domain perspective

[d] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

[e] Frequency-domain MLPs are More Effective Learners in Time Series Forecasting

[f] Learning Latent Seasonal-Trend Representations for Time Series Forecasting

4. (W4) Compared with the frequency domain method, and the method address the three terms

Thank you for the valuable suggestions. We add LaST [f] and FreTS [e] as baselines for comparison. Since they focus on the time series forecasting task, we adapt them to the imputation task based on the TimesNet [48] setting. Furthermore, we have considered the TIDER model which takes into account trend, seasonal and residual terms in the MF process. We first report the part of updated Table 1 for the added baselines:

We then report the results for varying missing mechanisms with 10% missing values in the following table, and update Fig.7, Fig.8 and Fig.9:

After this, we report the resource consumption over KDD dataset with 10% missing values in the following table, and update Fig.5:

Finally we report the updated downstream application experiments in the following table, and update Fig.6:

We can find that our FGTI still outperforms all baselines due to the guidance of high-frequency information and dominant-frequency information, as well as the generating ability of the diffusion model.

3. (W3) Include additional metrics such as CRPS

Thank you for the thoughtful comments and valuable suggestions. Since the CRPS is used to evaluate probabilistic imputation baselines, we report the CRPS for all probabilistic imputation baselines according to the experimental settings in CSDI [43]and PriSTI [26]. First, we report the CRPS performance with different missing rates in the following table：

Then we report the CRPS by varying the missing mechanism with 10% missing values in the following table：

Combining the results in Table 1 and Fig.4, Fig.7 and Fig.8, it can be found that the variations of CRPS are basically the same as RMSE and MAE for a specific model with different settings. Thanks again to the authors for their valuable suggestions. We will add this part of the experiment in the Appendix.

Thank you for the thoughtful comments and valuable suggestions. Below, we provide our response to the questions and concerns.

1. (W1) Why large amplitude frequency components guide trend and seasonal term imputation

Thank you for your comments. We recognise that our current expression is insufficient. In STL decomposition, the dominant frequency components(with large amlitude) typically correspond to the trend and seasonal terms, while the high-frequency information is often associated with the residual term [a]. This is because trend captures long-term movements, and seasonal captures major period patterns. We also add an empirical evidence on the pre-decomposed KDD dataset in following table:

This suggests that the Trend term mainly corresponds to the dominant-frequency information, the Seasonl term mainly corresponds to the dominant-frequency information, but also contains some of the high-frequency information, the Residual term mainly corresponds to high-frequency information. Since we choose transformer as the encoder and utilize cross-attention as the fusion mechanism of the two frequency domain information, our method can self-adaptively adjust the weights of the high-frequency information and the dominant-frequency information for the characteristics of the three terms in different datasets.

And we will add the content of the above expressions in Section.1 and Section 3.1.2.

2. (W2) Discuss existing frequency domain research for time series tasks

Thank you for your valuable suggestions for improving the quality of the manuscript. Our current Related work section mainly focuses on the currently state-of-the-art time series imputaiton methods. For the time series imputation methods in frequency domain, mvLSWimpute [b] utilizes wavelet transforms to guide imputation, APDNet [c] uses the Fourier Temporal and Fourier Variable Interaction modules to model dependencies. In addition, the frequency domain time series forecasting methods FEDformer [d], FreTS [e] can also be applied to imputation task. However, they did not consider how to use frequency domain information to accurately model the residual terms of the missing data, which is critical for boosting the overall imputation performance. In contrast, our FGTI captures high-frequency information and dominant-frequency information to get a more accurate modeling of the residual term, while assisting in describing trend and seasonal terms.

We will add to the discussion on imputation methods in frequency domain to the related work section.