A Study of Posterior Stability for Time-Series Latent Diffusion

We thank the reviewer for such kind and professional feedback.

Part-1: More datasets and evaluation metrics.

Due to the limited space in the main text, we previously placed the experiment results with two extra time-series datasets (i.e., Retail and Energy) and another evaluation metric (i.e., MMD) in Table 4 of our Appendix F.3. We also explored other data modalities (e.g., texts) in Table 5 and Table 6 of our Appendix F.4.

This part aims to address your concerns in Weakness-1. We welcome any further questions you might have.

Part-2: Same-dimensional latent variables.

In the case where latent variable $z$ has the same dimension as $X$, the issue of posterior collapse might indeed be mitigated to some extent, though such a high-dimensional latent variable will make the latent diffusion as computationally costly as the vanilla diffusion model.

As you suggested, we conducted an experiment that ran latent diffusion with a latent variable of the same dimension (i.e., 360) as time series. The results are as below.

This table shows that the performance gain achieved by a high-dimensional latent variable is not as significant as that achieved by our framework. One possible explanation for these results is that the high dimensionality of latent variables makes the diffusion model harder to learn [1, 2], though it can mitigate the issue of posterior collapse.

This part aims to address your concerns in Question-1. We welcome any further questions you might have.

Reference

[1] High-Resolution Image Synthesis with Latent Diffusion Models, CVPR-2022

[2] Score-based Generative Modeling in Latent Space, NeurlPS-2021

Part-2: More recent baselines and other data modalities.

As you recommended, we have newly adopted two up-to-date baselines that aimed to address posterior collapse:

• One is Inverse Lipschitz Constraint [7] from ICML-2023;
• The other is Mutual Information Constraint [8] from JMLR-2022.

The experiment results are shown in the below table.

We can see the two new baselines can indeed mitigate the problem of posterior collapse, though their performance gains are much smaller than our framework. We will include the above new experiment results in the final version.

On the other hand,*we had indeed considered other data modalities (e.g., images). In Appendix F.4 of our paper, we compared our models with the baselines on text and image datasets, with experiment results shown in Table 5 and Table 6.

This part aims to address your concerns in Weakness-3, 5 and Question-2. We welcome any further questions you might have.

Part-3: Other concerns in Weakness-6 and Question-2.

We have introduced 2 new baselines that addressed posterior collapse in our Part-2 answer to you, with other 2 up-to-date time-series generative baselines in our Part-2 answer to Reviewer NiiK. These 4 new baselines are all from accepted papers in 2023 or 2024. We will also delete repeatedly cited papers in the final version.

The key component in our framework that makes the decoder sensitive is the collapse simulation loss as defined in Eq. (14). The core idea is to apply the diffusion process to simulate a posterior-collapsed latent variable $z$: $P(z|X) \approx P(z)$, and penalize the decoder if it yields high conditional probability $P(X|z)$ in this case. Our experiments in Fig. 4 and Table 1 both verified the effectiveness of this method in practice.

References

[1] Generating Sentences from a Continuous Space, ACL-2016

[2] Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing, NAACL-2019

[3] Lagging Inference Networks and Posterior Collapse in Variational Autoencoders, ICLR-2019

[4] Axiomatic Attribution for Deep Networks, ICML-2017

[5] A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions, ICML-2022

[6] Guided Integrated Gradients: An Adaptive Path Method for Removing Noise, CVPR-2021

[7] Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network, ICML-2023

[8] Mutual Information Constraints for Monte-Carlo Objectives to Prevent Posterior Collapse Especially in Language Modelling, JMLR-2022

Many thanks for your comprehensive and constructive feedback.

Part-1: A recap of dependency measures.

We would like to provide a step-by-step review of the key technique introduced in our paper: dependency measures, which might help your understanding and answer related concerns.

1, Motivation of this technique

A serious consequence of posterior collapse is that the time-series decoder tends to ignore the input latent variable for conditional generation. While this fact is widely recognized in the literature [1,2,3], there remains a lack of a principled method to quantify how much a decoder might neglect the latent variable. The dependency measure was developed in this background, and it will be indispensable for practitioners to diagnose the problem of posterior collapse.

2, Underlying principle (related to your concerns)

The dependency measure is a typical gradient-based attribution method [4], with a specific design for time-series generative models, considering the variable length, autoregressive nature, and discrete structure of time series. The core idea is to measure the sensitivity of a temporal model to its input variable through first-order gradients. The gradient-based attribution itself is principled, with many previous theoretical and empirical works [5,6] that verified its effectiveness.

3, Notations and key properties

Given a time-series decoder $f$, the global dependency $m_{t,0}$ is a signed measure estimating the dependency of predicting variable $x_t$ on latent variable $z$, while local dependency $m_{t, i}, 0 < i < t$ quantifies such dependency on variable $x_i$.

The measure $m_{t, i}, 0 \le i < t$ is always bounded between $-1$ and $1$, satisfying that $\sum_{0 \le i < t} m_{t, i} = 1$. In typical cases where time series exhibit structural dependencies among variables, $m_{t, i}, 0 \le i < t$ is mostly non-negative, so we can infer:

• $m_{t, i} \approx 1$ means that the variable $x_i, i > 0$ or latent variable $z, i = 0$ dominates the attention of decoder $f$;
• $m_{t, i} \approx 0$ suggests quite the opposite: the decoder $f$ ignores the variable.

4, Use cases (related to your concerns)

As mentioned in the beginning, the most significant symptom of posterior collapse is that the time-series decoder tends to ignore the input latent variable for conditional generation [1,2,3]. In that situation, based on the properties of dependency measures, the global dependency $m_{t, 0}$ should be close to $0$ for every time step $t$, or soon vanishes with increasing time $t$. In light of this diagnostic logic, posterior collapse can be asserted when vanishing global dependencies (i.e., $m_{t, 0} \approx 0$ for at least large $t$) are observed.

On the other hand, based on the property that all measures sum to $1$, one can claim that the decoder heavily depends on input observations under the impact of posterior collapse: high local dependencies $\sum_{1 \le i < t} m_{t, i} = 1 - m_{t, 0} \approx 1$. This is exactly the case when we explained the upper right subfigure of Fig. 1.

6, For other data modalities (related to your concerns)

The dependency measures are specifically designed for time series but can naturally be extended to other types of sequential data (e.g., text). As mentioned in our experiments in Appendix F.4, text Latent Diffusion also seriously suffers from posterior collapse, which only has a minor impact on image Latent Diffusion. Therefore, dependency measures will yield similar outcomes for text and time-series data, while exhibiting different behavior for image data.

This part aims to address your related concerns in Weakness-1, 2, 4 and Question-1. We welcome any further questions you might have.

Part-2: A recap of dependency measures

We would like to provide a brief review of a key technique introduced in our paper: dependency measures, which might help your understanding and answer related concerns.

1, Motivation of this technique

A serious consequence of posterior collapse is that the time-series decoder tends to ignore the input latent variable for conditional generation. While this fact is widely recognized in the literature [1,2,3], there remains a lack of a principled method to quantify how much a decoder might neglect the latent variable. The dependency measure was developed in this background, and it will be indispensable for practitioners to diagnose the problem of posterior collapse.

2, How it works, and notations

The dependency measure is a type of gradient-based attribution [7], with many previous theoretical and empirical works [8,9] that verified its effectiveness. The core idea is to measure the sensitivity of a temporal model to its input variable through first-order gradients. Given a time-series decoder $f$, the global dependency $m_{t,0}$ is a signed measure estimating the dependency of predicting variable $x_t$ on latent variable $z$, while local dependency $m_{t, i}, 0 < i < t$ quantifies such dependency on variable $x_i$.

3, Key properties, especially about the negative dependencies

The measure $m_{t, i}, 0 \le i < t$ is always bounded between $-1$ and $1$, satisfying that $\sum_{0 \le i < t} m_{t, i} = 1$.

About the negative measures, please note:

• Both positive and negative measures are valid, though positivity is more common to see because typical time series exhibit structural dependencies among variables.
• There are cases where time series are shuffled or get noisy, such that non-positive measures might be observed.

In summary, negative measures do not mean bad, and they are not directly related to posterior collapse except for observing negative global dependency $m_{t, 0}$.

4, Use cases, and relation to less expressivity

Regarding the use cases of dependency measure and its relation to less expressivity, please note the following facts from our paper:

• Proposition 3.1 of our paper only considered the cases of a fully collapsed posterior: $P(z | X) = P(z)$, where Latent Diffusion will be as inexpressive as a simple VAE.
• The dependency measure is a diagnostic tool for the cases of either fully or partially collapsed posterior [3]: $P(z | X) \approx P(z)$, where the global dependency $m_{t, 0}$ will be close to $0$ for every time step $t$, or soon vanishes with increasing time $t$.

Therefore, The dependency measure extends the applicability of Proposition 3.1 to more scenarios.

This part aims to address your related concerns in Weakness-3 and Question-1. We welcome any further questions you might have.

Part-3: Other concerns in Question-2

As mentioned in our Part-1 answer, the decoder of vanilla Latent Diffusion only has access to the last variable of the backward diffusion model: $z^0 = z$. Similarly, in our framework, the latent variable fed into the decoder is sampled from a very small number of variables in the backward process. Appendix F.1 of our paper showed experiment results (i.e., Table 2) about the effect of that number. For jointly training the autoencoder and diffusion model, we found it quite stable in practice.

References

[1] Generating Sentences from a Continuous Space, ACL-2016

[2] Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing, NAACL-2019

[3] Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network, ICML-2023

[4] Latent Diffusion for Language Generation, NeurlPS-2023

[5] High-Resolution Image Synthesis with Latent Diffusion Models, CVPR-2022

[6] A Variational Perspective on Diffusion-Based Generative Models and Score Matching, NeurlPS-2021

[7] Axiomatic Attribution for Deep Networks, ICML-2017

[8] A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions, ICML-2022

[9] Guided Integrated Gradients: An Adaptive Path Method for Removing Noise, CVPR-2021

We thank the reviewer for his or her comprehensive and constructive feedback.

Part-1: A review of Latent Diffusion, posterior collapse, and our framework

We would like to provide a step-by-step review of the posterior-collapsed latent diffusion and our framework, which might help your understanding and answer related concerns.

1, How does Latent Diffusion samples?

The sampling process of Latent Diffusion takes two steps:

• The backward process of a diffusion model incrementally denoises a Gaussian noise $z^L$ into a latent variable $z = z^0$;
• The decoder $f$ of an autoencoder renders the sampled variable $z$ into time series $X$.

Therefore, as a kind reminder, it is the last variable $z^0$ of the backward process that controls the time-series decoding, while all its previous variables $z^i, 1 \le i \le L$, are not involved.

2, Definition of posterior collapse

Based on the above fact, the posterior collapse in our paper was only defined for the last variable $z^0$ in the backward process, rather than for others $z^i, 1 \le i \le L$. We believe this problem setup is quite appropriate, and defining "time-dependent and time-independent" sub-problems, as you recommended, might not align with the architecture of Latent Diffusion, unless the decoder $f$ has access to more than just the last variable of the backward process.

On the other hand, the definition of posterior collapse (i.e., "Problem Formulation” paragraph in Sec. 3.1) in our paper is consistent with many previous works in the literature [1,2,3]. We would appreciate any references with the suggested problem setup, though we believe their applicability to our paper might be limited.

3, The significance of studying posterior collapse in Latent Diffusion

Latent Diffusion represents one of the state-of-the-art generative models, which is well-known in both academia (e.g., LD4LG [4]) and industry (e.g., Stable Diffusion [5]). Therefore, it is very important to study the potential risks of this advanced architecture, such as posterior collapse.

On the other hand, please note some key findings that are first presented in our paper:

• A fully posterior-collapsed Latent Diffusion is equivalent to a simple VAE, making it a much less capable generative model than even a vanilla diffusion model [6];
• In cases of partial posterior collapse [3], we introduced a principled method: dependency measure, to quantify the severity of the problem, identifying a previously unknown phenomenon: dependency illusion;
• As shown in Sec. 4.2 of our paper, the posterior collapse can be perfectly addressed with the diffusion process.

These points clearly indicate that the problem of posterior-collapsed Latent Diffusion goes far beyond previous research focusing on a simple autoencoder, calling for a more systematic study. Our work was inspired by this background.

4, How does our framework address the problem?

The key component in our framework to address posterior collapse is the collapse simulation loss as defined in Eq. (14). The core idea is to apply the diffusion process to simulate a posterior-collapsed latent variable $z$: $P(z|X) \approx P(z)$, and penalize the decoder if it yields high conditional probability $P(X|z)$ with non-informative variable $z$. In other words, the defined loss forced the decoder $f$ to condition on latent variable $z$ for generation, making it informative about time series $X$.

Besides the promising main experiment results measured by Wasserstein distance, Fig. 1 and Fig. 4 of our paper (i.e., dependency measures over time) showed that the latent variable $z$ in our models maintained a stable control over the decoder$f$ with increasing time $t$, indicating a non-collapsed posterior $P(z | X)$.

This part aims to address your concerns in Weakness-1, 2, 4. We welcome any further questions you might have.

We thank the reply from the reviewer. Below is a point-by-point summary highlighting how our rebuttal tried to address your concerns:

• "1, How does Latent Diffusion samples?" of Part-1 clarified that time-series decoding is only controlled by the last backward latent variable $z = z^0$ (instead of any process), so your concern about "There are potentially time series which are driven by a static latent process" does not apply to the architecture of Latent Diffusion. In other words, the virtual time steps in diffusion models have nothing to do with the time steps in time-series data;
• "2, Definition of posterior collapse" of Part- clarified that our definition of posterior collapse is consistent with many previous works, and we believe that your concern about "time-dependent and time-independent mode collapses" does not fit the problem setting of our paper. We would also appreciate any references you could provide;
• ”3, The significance of studying …" of Part-1 aimed to address your concern about "how the introduced technique solved the problem of mode collapse”;
• "4, How does our framework address…" of Part-1 aimed to address your concern about “how the introduced technique solved the problem of mode collapse”;
• ”2, How it works ..." and "3, Key properties … negative dependencies" of Part-2 aimed to address your concerns about "there is little theoretical analysis exploring its properties" and "negative local dependency”. We reviewed the key properties of dependency measures and clarified about negative dependencies;
• "4, … relation to less expressivity" of Part-2 aimed to address your concern about “its relationship to the reduction to a VAE model”.

A kind reminder is that your review repeatedly referred to "mode collapse", instead of the posterior collapse studied by our paper. These two terminologies have very distinct definitions in the context of generative models. We are concerned that there might be some misunderstandings that we could further address.

Thank you again for your reply, and we are looking forward to the opportunity to further clarify any of your concerns that have been not addressed well.

Part-3: Explanations of some terminologies.

We prepared the below table that re-explains and connects some key terminologies appearing in our paper.

This table aims to answer your Question-2, and we will include it in the final version for better clarity. We welcome any further questions you might have.

References

[1] Time-series Generative Adversarial Networks, NeurlPS-2019

[2] On Measuring Fairness in Generative Models, NeurlPS-2023

[3] On Memorization in Probabilistic Deep Generative Models, NeurlPS-2021

[4] Anonymization Through Data Synthesis Using Generative Adversarial Networks, IEEE-2020

[5] Data Synthesis based on Generative Adversarial Networks, VLDB-2018

[6] Equivariant Diffusion for Molecule Generation in 3D, ICML-2022

[7] TimeLDM: Latent Diffusion Model for Unconditional Time Series Generation, arXiv-2024

[8] Latent Diffusion for Language Generation, NeurlPS-2023

[9] AudioLDM: Text-to-Audio Generation with Latent Diffusion Models, ICML-2023

[10] Time Series Diffusion in the Frequency Domain, ICML-2024

[11] Automatic Integration for Spatiotemporal Neural Point Processes, NeurlPS-2023

We thank the reviewer for providing such comprehensive and constructive feedback.

Part-1: Time-series Generative Models vs Forecasting Models.

Our paper focuses on Latent Diffusion, a type of Generative Models different from your recommended models (e.g., ARIMA and TCNs), which belong to time-series Forecasting Models and are indeed free from posterior collapse. The following table compares the two classes of models, showing the use cases of our paper.

Key points from the table: Time-series Forecasting Models (e.g., ARIMA) are without a key component of Generative Models (e.g., Latent Diffusion): latent variable z, which might incur posterior collapse. For this reason, well-performing Forecasting Models do not suffer from posterior collapse, though we can see that Generative Models also have their unique values in real-world applications (e.g., Data Synthesis and Drug Discovery).

Advanced time-series architectures in our models: On the other hand, we built the time-series decoder of our Generative Models with either modern Transformer or LSTM (see Table 1 of our paper), which are both your recommendations. As indicated above, while those architectures are without posterior collapse, the latent variable z that initializes them is the root cause incurring the problem.

This part aims to address your concerns in Weakness-1, 3 and Question-1, 3. We welcome any further questions you might have.

Part-2: Latent Diffusion is the state-of-the-art, with more recent baselines adopted.

Our paper is based on Latent Diffusion that first appeared in CVPR-2022, representing one of the most advanced architectures of Generative Models. There are many very recent papers that focused on time-series Latent Diffusion, or even more broadly: sequence data with Latent Diffusion. For example, TimeLDM [7], LD4LG [8], and AudioLDM [9] that appear in NeurlPS-2023, ICML-2023, and the arXiv months ago. Therefore, from the recent literature, we believe that Latent Diffusion stands as a state-of-the-art time-series Generative Model.

As you recommended, we have additionally adopted two up-to-date time-series baselines for comparison:

1. One is Frequency Diffusion [10], a (not latent) diffusion-based Generative Model appearing in ICML-2024;
2. The other is Neural STPP [11], a flow-based Generative Model appearing in NeurlPS-2023.

The experiment results are shown in the below table.

We can see that vanilla Latent Diffusion achieved competitive performances to the up-to-date baselines (e.g. Neural STPP), showing that it is still very advanced. In particular, our framework significantly improves Latent Diffusion, with performances notably outperforming the baselines, indicating that posterior collapse is indeed a performance bottleneck of Time-series Latent Diffusion. The above empirical results further confirmed that Latent Diffusion is a state-of-the-art time-series Generative Model, and verified the importance of addressing posterior collapse.

This part aims to address your concerns in Weakness-1, 2 and Question-1. We welcome any further questions you might have.

We thank the reviewer for such kind and constructive feedback.

Part-1: Time-series Generative Models vs Forecasting Models.

Our paper focuses on Latent Diffusion, a type of Generative Models, which are very different from time-series Forecasting Models. The following table compares the Generative and Forecasting Models, showing their significant difference in evaluation metrics.

Key point from the table: Generative Models (e.g., Latent Diffusion) are evaluated by distribution-level metrics (e.g., Wasserstein distance): comparing the generated and real sample distributions, which largely differ from Forecasting Models that are evaluated by observation-level accuracy-like metrics (e.g., MSE). Such a distinction stems from their different model definitions.

Diverse metrics adopted in our paper: Previously, We had considered the diversity of evaluation metrics. In Appendix F.3 of our paper, Table 4 compared our models with the baselines in terms of another widely used metric: maximum mean discrepancy (MMD) [1].

This part aims to address your concern in Weakness-2. We welcome any further questions you might have.

Part-2: Other advanced baselines, and ablation studies.

As you suggested, we have additionally adopted two up-to-date time-series baselines for comparison. One is Frequency Diffusion [2], a (not latent) diffusion-based Generative Model appearing in ICML-2024; The other is Neural STPP [3], a flow-based Generative Model appearing in NeurlPS-2023. The experiment results are shown in the below table.

We can see that Latent Diffusion is competitive with up-to-date time-series generative baselines, and it can significantly outperform the baselines with our framework, showing the significance of addressing posterior collapse. Previously, we had conducted ablation experiments as shown in Table 2 of our Appendix F.1, verifying the effectiveness of different components (e.g., collapse simulation) in our framework and interpreting the effects of their hyper-parameters.

This part aims to address your concern in Weakness-1, 3. We welcome any further questions you might have.

References

[1] Training generative neural networks via maximum mean discrepancy optimization, UAI-2015

[2] Time Series Diffusion in the Frequency Domain, ICML-2024

[3] Automatic Integration for Spatiotemporal Neural Point Processes, NeurlPS-2023

Dear Reviewer NiiK,

We would like to thank you again for your kind and insightful review! As the discussion stage is approaching its extended deadline, we noticed that we have not yet heard from you. We look forward to your feedback for our rebuttal, and we would appreciate any improvement in the rating if your concerns were adequately addressed!

Best Regards,

The Authors

We thank the reviewer for his or her kind and comprehensive feedback.

Part-1: More baselines, and other data modalities.

As you recommended, we have additionally adopted two up-to-date baselines for comparison:

1. One is Frequency Diffusion [1], a (not latent) diffusion-based Generative Model appearing in ICML-2024;
2. The other is Neural STPP, a flow-based Generative Model appearing in NeurlPS-2023.

The experiment results are shown in the below table.

We can see that Latent Diffusion is competitive with up-to-date time-series generative baselines, and it can significantly outperform the baselines with our framework, showing the significance of addressing posterior collapse.

On the other hand, we had indeed considered other data modalities (e.g., images). In Appendix F.4 of our paper, we compared our models with the baselines on text and image datasets, with experiment results shown in Table 5 and Table 6.

This part aims to address your concerns in Weakness-1 and Question-2, 3. We welcome any further questions you might have.

Part-2: Scalability of dependency measures.

From Eq. (10) of our paper, we can see that the computational complexity of a dependency measure is $O(NM)$, where $N$ is the number of Monte Carlo samples and $M$ is the length of time series. Therefore, the computational cost of the dependency measure linearly grows with the increasing sentence length, and this linear complexity can be further optimized with parallel GPU computation. In practice, a set of 10000 time series with an average length of 30 cost us only about 5min on a single GPU device. In summary, the dependency measure scales to large and long time-series datasets well.

This part aims to address your concern in Question-1. We welcome any further questions you might have.

Part-3: Other concerns about experiments.

As mentioned at the beginning of our Experiments section (i.e., Sec. 6), we move the Experiment Details section to Appendix E of our paper, due to the limited space in the main text. We will relocate that section and improve the graph quality as you suggested in the final version.

References

[1] Time Series Diffusion in the Frequency Domain, ICML-2024

[2] Automatic Integration for Spatiotemporal Neural Point Processes, NeurlPS-2023