TimeAutoDiff: Generation of Heterogeneous Time Series Data via Latent Diffusion Model

Thanks for your comments on our work! We made significant revisions inspired by your feedback. Please take a look at our revised manuscript and followings are the answers to your questions and comments on weakness and questions:

Q) One concern is its current inability to generate interpretable results. In many practical applications, especially in high-stakes fields like finance and healthcare, stakeholders must clearly understand how models make decisions. The lack of interpretability can lead to skepticism and reluctance to adopt the model, as users may hesitate to trust a system whose inner workings are opaque. This limitation could hinder the model's applicability in scenarios where understanding the rationale behind generated data is crucial for decision-making;

A) This comment is exactly consistent with what we have mentioned in Appendix A, discussing the future directions of our work. Specifically, as mentioned in the “Interpretability” part, the interpretability can be incorporated by simply adopting the decoder in TimeVAE in our framework for continuous only signals. Interpretability of time series data has a long history, and it is deeply rooted in the assumption that the signal is decomposable into three parts: trend, seasonality, and noise. But the main bottleneck is “it is not clear on how to define the decomposability of categorical data”, which requires further research and notions. Given this, this topic is out of the scope of our current work and defers it to future research. We would like the reviewer to view our work as a trial for modeling time series tabular data with heterogeneity and show how the model can be used in time varying metadata conditional generation. (Please see our section 4.3 in the newly revised manuscript.)

Q) Another point is the pure focus on continuous data: While the method demonstrates good performance with continuous data, its methods are primarily tailored for this data type. This focus raises concerns about the model's effectiveness when dealing with heterogeneous datasets that include categorical features. The inability to seamlessly integrate and process diverse data types could restrict the model's usability in real-world applications where such heterogeneity is common.

A) Thanks for the comment, but we want to emphasize our model TimeAutoDiff is primarily designed for taking care of heterogeneous features, which both include continuous and categorical features. The key idea is to combine VAE and diffusion model for dealing with this heterogeneity, and we design our own VAE and diffusion to encode the temporal dependencies of rows in the data by using the inductive bias of RNNs. The dataset used in the experimental parts have heterogeneous features, and by the suggested four metrics in our paper, our model gives the superior performance when compared with other time series data synthesizers, which didn’t consider modeling the heterogeneity of data.

Q) Another point is the dependence on latent space reduction: The paper notes a performance drop when the feature sizes increase, which necessitated a reduction in the latent space dimension. This reliance on dimensionality reduction to maintain performance raises concerns about the model's scalability and robustness. If the model's effectiveness is sensitive to the dimensionality of the latent space, it may struggle to perform well in high-dimensional settings without careful tuning.

A) Latent space dimensionality reduction should not be viewed as a disadvantage of the model; rather, it is a distinct merit that provides opportunities to enhance performance and efficiency in the inference step of diffusion model, and it is an aspect we aim to leverage effectively in our problem setting, just in case we have features in high-dimensional space. The primary purpose of latent diffusion model in vision-related tasks is to speed up the sampling time. For instance, see Vadhat’s paper. They project the image data in a high-dimensional original data space to low-dimensional latent space and let the diffusion model learn the projected data in low-dimensional space. Since, diffusion model is trained on low-dimensional space, the sampling time in diffusion becomes much shorter, while maintaining the good quality of generated images. (Here, we assume the decoder in VAE is expressive enough.)

• Limitations and fairness: Limitations and fairness are not discussed enough. In particular can generated data be biased with respect to conditional metadata?

Generated data can indeed be biased with respect to conditional metadata, arising from various factors. Bias in the training data, such as inherent associations between metadata and outputs, may lead the model to replicate these biases, for instance, generating disproportionately high traffic volumes for "Clear" weather even when the true relationship is less deterministic. Imbalanced metadata distributions further exacerbate this issue, as underrepresented conditions in the training set often result in less reliable outputs for those conditions, such as biased outcomes for minority demographic groups in healthcare datasets. Simplified assumptions in the model, such as assuming linear relationships between metadata and outputs, can overlook complex dependencies, producing data that fails to reflect the true conditional distribution. Noise injection, a feature of models like diffusion models and VAEs, can introduce additional bias if the noise interacts with metadata in unexpected ways, particularly for rare metadata values. Furthermore, limitations in conditional architectures, such as inadequate metadata encoding, can prevent the model from capturing nuanced dependencies, leading to misaligned outputs. To mitigate such biases, ensuring balanced training data, employing robust metadata encoding techniques, applying regularization or fairness constraints, performing post-generation bias audits, and designing disentangled latent spaces are crucial steps. While conditional generative models aim to align generated data with metadata, addressing these biases is essential to ensure fairness and reliability. We include this paragraph in the Appendix A of our revised manuscript.

• Design choices: It would be great to have more details on the design choices of the decoder and ϵθ. Do other architectures significantly affect the performance of the method?

(A). We add detailed descriptions on our decoder in Appendix E.
(B). We add several ablation tests inspired by your comments. We replaced MLP with the RNN structure in the decoder. (C). Inspired by [Bilos et al., 2023], we explore injecting continuous noise from a stochastic process (Gaussian process) into the DDPM. The experimental results show that none of the ablated models outperformed the original configuration significantly. Specifically, the second configuration demonstrates the benefits of modeling temporal relations twice in the VAE and DDPM due to the following reasons:

$**Hierarchical Temporal Dependency Modeling:**$ The VAE encoder captures compact latent representations with temporal dependencies, providing a structured input for the diffusion model. This allows the diffusion process to refine finer-grained patterns without redundantly encoding high-level temporal structures, resulting in more realistic outputs.

$**Noise-Tolerant Latent Representation:**$ Encoding temporal dependencies early in the VAE encoder ensures that the latent variable $\mathbf{z}$ is robust to noise. This noise resilience helps maintain critical temporal structures during the diffusion process, enhancing the fidelity of the generated data.

Bilos et al., 2023 ICML: Modeling temporal data as continuous functions with stochastic process diffusion

$*Questions:*$

Q. The sampling time column ranks models by their speed, with lower numbers indicating faster sampling. Please add details.

A. The detailed sampling time (in second) is in the last row of Table 1. Additionally, we move the table to Appendix E.

Q. We set the dimensions of output features in this way as we used the mean-squared (MSE), binary cross entropy (BCE), and cross-entropy (CE) in the Pytorch package. It looks disconnected from the main text. Could you clarify?

A. We mentioned this because the BCE and CE functions in the PyTorch package take specific arguments for input and target dimensions. Specifically:

• BCE takes one argument: the logit (unnormalized score) for each class when the label is equal to 1. This requires the input to have the same dimensions as the target (e.g., [batch_size, num_features]), where values represent logits for each feature.

• CE expects the input to be the logits (unnormalized scores) with shape [batch_size, num_classes] and the target to be class indices with shape [batch_size] (not one-hot encoded).

• MSE requires both the input and target tensors to have the same shape (e.g., [batch_size, num_features]), where values represent continuous outputs.

To ensure compatibility with these loss functions, we carefully set the dimensions of the output features to align with the expected input shapes for MSE, BCE, and CE during training. We revised the paragraph in our newly revised paragraph. [Line 240~247.]. We provide the figure of our decoder in the Appendix L, Figure 12. in our revised manuscript.

Thanks for your comments on our work! We made significant revisions inspired by your feedback. Please take a look at our revised manuscript and followings are the answers to your questions and comments on weakness and questions:

• Comparison to prior methods : It is worth discussing more the key differences to other diffusion-based models such as Diffusion-TS (https://github.com/Y-debug-sys/Diffusion-TS), and TimeDiff (https://openreview.net/pdf?id=ESSqkWnApz). Additionally, it is worth expanding metrics, such as consistency and diversity, as discussed in the TSGM framework (https://github.com/AlexanderVNikitin/tsgm).

A.) Diffusion-TS’s main purpose is to generate time series data with interpretability. They employ the Autoencoder + DDPM framework, employing transformers as encoder and decoder for obtaining the disentangled representations of time series.

The main difference between Diffusion-TS and ours is on the problem setting that their assumption on the signal is only restricted to continuous time series, whereas ours is focused on the heterogeneous features. Diffusion-TS lies on the assumption that the signal is decomposable into three main parts: trend, seasonality, and noise. However, the decomposition of heterogeneous features, specifically discrete variables is not well defined in the literature, it is beyond the scope of our work, requiring further research.

TimeDiff integrates two types of diffusion models to handle heterogeneous features in EHR datasets, employing DDPM for continuous variables and multinomial diffusion (Hoogeboom et al., 2021) for discrete variables. In contrast, our approach leverages a VAE to project time series data into a latent space and utilizes DDPM exclusively for modeling the time series within this latent representation, which is continuous. We include this comparison in the Appendix E.

B.) Appreciate for the comments. After carefully reading the TSGM paper, we realized the consistency metric they proposed is for measuring the downstream performance of classification tasks. Nonetheless, to be computed, this metric is restrictive requiring several conditions that is not compatible with our original experimental setting. First, the data should be the classification time series dataset; that is, for each multivariate time series data, they should have the label. But the dataset we consider in this paper is not designed for classification problems. Second, the metric requires the data to be generated from conditional model. But for comparison purposes, the baseline models that we employed cannot afford the conditional generation, except for our model. Instead, we measure the predictive score which is the most widely used metric for measuring the performance of synthetic data in downstream tasks in the literature (See Table 1 in our paper.).

Nonetheless, we used two metrics suggested by TSGM: MMD and Entropy. Maximum Mean Discrepancy (MMD) is for measuring the similarities between synthetic and real time series data, and Entropy measures the diversity of the synthetic data. The results are summarized in the Table 11 in the newly revised manuscript. (Appendix O)

We used two metrics proposed by TSGM [nikitin2023 et al]: Maximum Mean Discrepancy (MMD) and Entropy. MMD measures the similarity (or fidelity) between synthetic and real time series data, while Entropy assesses the diversity of the synthetic data. The results are summarized in Table 11 and are consistent with those in Table 1.

TimeAutoDiff achieves the lowest MMD scores across all four datasets, aligning with the discriminative scores reported in Table1. This indicates that TimeAutoDiff effectively generates synthetic data that closely resembles real data. For diversity, higher Entropy values indicate a dataset with more diverse samples. However, as noted in TSGM, Entropy should be considered alongside other metrics, as random noise can also result in high Entropy values. TimeAutoDiff produces synthetic data with higher Entropy values than the real data, though not as excessively as other baseline models. This suggests that our model generates synthetic data that preserves the statistical properties of the original data, maintaining diversity without introducing excessive deviation.

Thanks for your comments on our work! We made significant revisions inspired by your feedback. Please take a look at our revised manuscript and followings are the answers to your questions and comments on weakness and questions:

Originality. DDPM or VAE for time series generation is not new. Most techniques used are also from prior works.

A) Both models are widely used in generative modeling, and we believe it is effective to leverage the strengths of established methods to address certain problems, rather than introducing complex new approaches. We hope the reviewer recognizes our work as an effort to tackle tabular time series modeling with heterogeneous features by effectively applying two powerful existing models. Furthermore, latent diffusion model (VAE + Diffusion) is already a widely used framework in various domains other than tabular domain: Dalle-2 (text-to-image model from OpenAI), GeoLDM (3D-Moleculer generation), VideoLDM (video generation). And we borrowed this idea in the tabular domain (which is rarely studied) to tackle the heterogeneity issue and showed it can indeed solve this issue in time series modeling. (Please see the “Motivation of TimeAutoDiff” section in Subsection 1.1. in our newly revised manuscript.) We believe this is a useful / helpful idea to the community and believe this can be further expanded to solve more practically involved real-world problems as we mentioned in the conclusion part.

Soundness. It’s well-known that VAE is considered 1-step DDPM. I find the motivation and reasoning in line 144-154 very unconvincing. It’s still unclear why DDPM is good at handling time series data, why VAE, as a special case is DDPM, can bring more to the table. Also, I don’t think citing unpublished and prior venue’s rejections as SOTA method is convincing, e.g., TSGM in line 156.

A) VAE is a useful tool for handling heterogeneity of tabular data. In regular tabular data setting (i.i.d row setting), (for instance, TabSyn, AutoDiff) VAE or autoencoder framework have already been widely used. We employ this idea into time series tabular setting with our own encoder & decoder in VAE. The handling of heterogenous feature is easy through VAE because VAE’s objective function is involved with the reconstruction error, whereas diffusion model is primarily designed to minimize the negative log-likelihood function of continuous random variables. There’s no way we can deal with heterogeneity of data so far with only one diffusion model, other than combining two different types of diffusion models for continuous and discrete variables (for instance, in tabular modeling literature: TabDDPM, TimeDiff). But this method is known for not being able to capture the correlations among the continuous and discrete random variables.

Regarding your comments on diffusion model, you are right. Diffusion model can be thought of as VAE with T (diffusion steps) hidden layers in both encoder and decoder. (Using score-based framework, it becomes infinite layers.) These multiple diffusion steps (hidden layers) enable iterative refinements of distributional modelling. In the forward process, the prior distribution gets close to standard Gaussian, nonetheless, VAE (arguably with one diffusion step diffusion model) lacks this expressive power. You can recall the tradeoff between reconstruction error and KL-divergence in VAE, and if we harshly penalize the divergence, restricting the shape of latent space to follow Gaussian, the reconstruction will be very high, and vice versa. But diffusion model can easily avoid this problem, enabling the distributional modeling for high-dimensional problems. (In our case, the data dimension is $T\times F$).

Why diffusion model is good for time series data modeling? Diffusion models are particularly well-suited for time series modeling due to their ability to handle the complexities of sequential data while capturing the inherent dependencies and uncertainties. The iterative denoising process of diffusion models allows them to effectively learn temporal dependencies across time points, making them ideal for modeling the sequential dynamics in time series. Their flexibility enables the integration of specialized architectures, such as RNNs or Transformers, within the denoising network to better capture temporal relationships. Furthermore, diffusion models excel at handling heterogeneous data, which is common in time series, by leveraging tailored output heads and loss functions (e.g., MSE, BCE, CE) for mixed feature spaces. The noise injection and removal mechanism of diffusion models enhances robustness, allowing them to manage stochastic elements, outliers, and irregularities in the data. Additionally, diffusion models are designed to scale well with high-dimensional inputs, making them suitable for complex multivariate time series data with long horizons.

$**Questions:**$

1. I would like to inquire about the rationale behind employing RNN as the encoder for time-series data and capturing temporal dependencies between features across various timestamps. The temporal dependencies between features can also be captured by the denoising network of the diffusion model. Therefore, is it necessary to utilize an RNN to capture temporal dependencies when encoding the data?

A) Nice catch. Followings are answers to your questions:

$**Hierarchical Temporal Dependency Modeling:**$ The VAE encoder is designed to capture a compact representation of the input data, including temporal dependencies, which serves as the latent variable for the subsequent diffusion process. Encoding temporal dependencies explicitly in the latent space helps the diffusion model operate on a representation that already integrates key temporal patterns. This hierarchical approach allows the diffusion model to focus on modeling finer-grained dependencies and generating realistic outputs, rather than redundantly encoding high-level temporal structures.

$**Noise-Tolerant Latent Representation:**$ By capturing temporal dependencies early in the VAE encoder, the latent variable $\mathbf{z}$ becomes a noise-tolerant representation of the time-series data. This benefits the diffusion process, as the added noise during diffusion is less likely to obscure important temporal structures, improving the overall quality and fidelity of generated data.

Inspired by your comments, we performed additional ablation tests. We replaced the RNN architecture in our encoder with fully connected layers and observed the performance of the ablated model at the sub-par level when compared with the original configuration. Please check the table 7 in the Appendix L.

2. I would encourage the author to provide a more comprehensive discussion and emphasize the values or practical significance of the tasks the model is evaluated on, specifically unconditional generation and time-variant meta-data conditioned generation.

A) Please see our answer to your second comments on weakness.

1. The work lacks innovative methodology. Similar ideas for tabular data, which arguably encompass time-series data, have been explored in existing works [1]. Despite some crucial technical differences between the two works, including the design of the denoising network ϵθ and the approach to the latent space, the high-level framework of both works is both a latent diffusion model that projects structured data to a latent space and uses a diffusion model to model the distribution of latent codes.

A) Thanks for your comment. This is somehow duplicate answer to the second reviewer’s first comment. But we don’t think adopting a Latent diffusion model framework is a significant drawback in terms of novelty of our work. LDM (VAE + Diffusion) is already a widely used framework in various domains other than tabular domain: DALLE-2 (text-to-image model from OpenAI), GeoLDM (3D-Moleculer generation), VideoLDM (video generation). And we borrowed this idea in the tabular domain (which is rarely studied) to tackle the heterogeneity issue and showed it can indeed solve this issue in time series modeling. (Please see the “Motivation of TimeAutoDiff” section in Subsection 1.1. in our newly revised manuscript.) We believe this is a useful / helpful idea to the community and believe this can be further expanded to solve more practically involved real-world problems as we mentioned in the conclusion part.

B) As mentioned in our paper, TimeAutoDiff is inspired by TabSyn [1] and AutoDiff [2], employing LDM framework. But we argue that TabSyn [1] cannot encompass the time-series data. The time series data has 2-dimensional form (i.e., 2D data): one dimension for time-axis and another dimension for feature axis, whereas the data format TabSyn can afford is in 1-dimensional space (only feature dimension). In their modeling scheme, they take each row of the table as one data point, and think the rows are i.i.d., and this means they didn’t model the temporal dependencies of rows in the table, which is an obvious difference between our work and theirs.

2. Most of the experiments are limited to unconditional generation of time-series data, and it is unclear how the proposed model can be readily adapted to tasks with more practical applications, such as forecasting and imputation.

A) Thanks for the comment. We take your comments seriously, and felt indeed the original version lacks the part how our model can be used in practice. Both imputation and forecasting tasks can be framed as conditional generation, and they can be tackled through masking techniques. But we are planning these tasks as separate projects to deal with these topics in a more in-depth manner. (We also give the detailed discussion section for this topic in Appendix J.)

B) Instead, we work on meta-data conditioned generation to show how our model can be used in practical applications. Taking your comments seriously, we moved the scalability and ablation tests parts to the Appendix and move the time varying metadata part that was originally in the Appendix M to the main body of the revised manuscript. In this section, under synthetic and real-data setting, how our model can be used for modeling P(Continuous | Discrete) and P(Discrete | Continuous) distribution. We believe this functionality is useful for counterfactual scenario explorations in many fields, given the prevalence of heterogeneous time series data in the real-world. References

[1] Zhang, Hengrui, et al. “Mixed-type tabular data synthesis with score-based diffusion in latent space.” arXiv preprint arXiv:2310.09656 (2023).
[2] Namjoon, Suh et al. "AutoDiff: combining Auto-encoder and Diffusion model for tabular data synthesizing." NeuRIPS 2023, SyntheticData4ML workshop.