Constrained Posterior Sampling: Time Series Generation with Hard Constraints

We thank the reviewer for the detailed and thoughtful feedback on our work and for acknowledging the effectiveness of CPS as it does not require any backpropagation through the denoiser.

On extension to non-linear constraints: We have compared CPS against other baseline methods for a general class of constraints. Specifically, we imposed the Autocorrelation Function (ACF) value at a particular lag as an equality constraint for the stocks dataset.

We observe that, even though the projection step does not result in the global optimum, the iterating projection and denoising process pushes the generated sample to the constraint set. Additionally, the sequential denoising process removes the adversarial artifacts caused by the projection step. This provides very low constraint violation metrics for CPS, when compared to other baselines, while maintaining high sample quality (Appendix B.3, Table 4). We request the reviewer to refer On extension to non-linear constraints section of our response to Reviewer gJxX for a detailed explanation.

On the empirical evidence for the claim made in the paper:

Claim - Projecting the noisy intermediate latent (PDM and PRODIGY) has adversarial effects on sample quality.

Evidence - Note that all approaches use the same denoiser. The only conceptual difference between these approaches and CPS is that CPS projects the posterior mean estimate. Consequently, this results in the noisy latent, $z_{t-1}$, being pushed off the manifold for the diffusion step $t-1$, resulting in imperfect denoising. As a result, imperfect denoising causes sample quality degradation which is empirically shown in Table 2.

Section 3 (lines 229–240) outlines two key reasons for the same.

On performing imputation with CPS: The reviewer is correct. CPS can be extended to perform forecasting and imputation by imposing the value at constraint to all the timestamps where the true values are available.

Clarification regarding the likelihood maximization claim: In lines 162-166, we state that the proposed algorithm satisfies the desired constraints while maximizing the likelihood of the generated samples.

In general, diffusion models are trained to maximize the log-likelihood of observing samples from the dataset, i.e., log p(x). Therefore, diffusion models allow for sampling from p(x).

However, in constrained time series generation, the objective is to sample from p(x) subject to certain constraints.

To sample from p(x), CPS uses a pre-trained diffusion model that was trained to maximize log-likelihood. This is what we refer to as maximizing the likelihood of generated samples.

Diffusion models sample from p(x) through sequential denoising. Now, to ensure constraint satisfaction, in CPS, we perturb the denoised sample, after every denoising update, using a projection step towards the constraint set.

This ensures that the generated samples have high likelihood while belonging to the constraint set.

Performance of Guidance-Based approaches on fewer constraints: In Figure 5, we compare all approaches that can provide satisfaction guarantees for convex constraints. Below, we extend Figure 5 with guidance-based approaches for the Stocks dataset.

From the table, CPS comfortably outperforms guidance-based approaches in sample quality, even for a smaller number of constraints. Additionally, due to the absence of any principled projection step, these approaches do not provide constraint satisfaction even for convex constraints. We observe similar trends for all real-world datasets.

Clarification regarding the qualitative results in Figures 4, 9, and 10: In Figures 4 and 9, we observe almost perfect overlap for a few datasets, specifically for the conditional variants and stocks. This is primarily due to conditional inputs and a large number of constraints.

For the conditional variants (Air Quality Conditional and Traffic Conditional), the generated samples are already condition or metadata-specific and therefore track the real sample closely. In addition, when we impose constraints, the generated samples almost perfectly match the real samples. This does not happen in the unconditional cases and is empirically demonstrated through the differences in the DTW metrics between the unconditional and conditional cases (2.35 vs 1.83 for Air Quality and 3.41 vs 0.84 for Traffic, lower is better).

For the stocks dataset, the number of constraints is too high (450) that the feasible set is very small and therefore the generated samples closely track the real sample from the constraint set.

Consequently, in both these figures, note that the least similarity between the real and generated samples can be observed for the unconditional traffic and air quality datasets which do not have a large number of constraints as the stocks dataset.

The generated samples only roughly follow the trend of the real sample, but do not match as perfectly as compared to the conditional variants.

Therefore, in Figure 10, while depicting the distribution of samples for a given constraint set, we observe higher overall variance for the unconditional traffic and air quality setups when compared to the Stocks dataset which has a larger number of constraints.

Additionally, as we impose the value at constraints, note that the standard deviation is 0 at 1, 24, 48, 72, and 96 timestamps.

Finally, the mismatch with the real samples do not translate to poor quality samples. Our objective with the qualitative results is to mainly showcase the matching trends with least distortions due to projections. This is confirmed by the high sample quality metrics for CPS.

Clarification regarding the constraints used in Figures 5 and 7: The reviewer is correct. The x-axis label for both figures 5 and 7 should be “Number of constraint categories”. We will update figure 5 accordingly. In both figures, we use the following convention:

1 constraint - argmax

3 constraints - argmax, value at argmax, value at 96

5 constraints - argmin, argmax, value at argmax, value at 72 and 96

7 constraints - argmin, argmax, value at argmin, value at argmax, value at 48, 72 and 96

9 constraints - argmin, argmax, value at argmin, value at argmax, mean, value at 24, 48, 72 and 96

11 constraints - argmin, argmax, value at argmin, value at argmax, mean, mean change, value at 1, 24, 48, 72 and 96

We uniformly distributed the value at constraints over these 6 experimental settings. For the stocks dataset, we always impose the OHLC constraint (lines 291 and 292).

Yes, the constraints are applied to every channel in multivariate datasets like Air Quality and Stocks.

For the air quality dataset (6 channels), there are 66 constraints (6 mean + 6 mean change + 6 argmax + 6 argmin + 6 value at argmax + 6 value at argmin + 5x6 value at constraints). The same extends to the conditional variants as well.

For the traffic dataset (1 channel), there are 11 constraints (similar to the air quality dataset). The same extends to the conditional variants as well.

For the stocks dataset (6 channels), there are 450 constraints (66 (similar to air quality) + 4x96 OHLC constraints). Note that the OHLC constraints are on an individual time stamp level.

For the waveforms dataset (1 channel), there are 106 constraints (11 (similar to traffic) + 95 trend constraints between successive time stamps).

Modifications to the manuscript

1. We will better describe likelihood maximization in line 162 to 166.

2. We will include more details regarding non-linear constraints in the experiments section.

3. We will update the manuscript to use uniform notation for the sample quality metric based on the Frechet Distance.

4. We will include prior works such as [1], [2], [3], [4], [5], etc., and more.

5. We will address the issue with the hyperlinks.

6. We will update the x-axis label for Figure 5.

References

[1] The Rise of Diffusion Models in Time-Series Forecasting.

[2] A Versatile Diffusion Model for Audio Synthesis.

[3] Non-autoregressive Conditional Diffusion Models for Time Series Prediction.

[4] Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement.

[5] Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting.

We thank the reviewer for the detailed feedback on our work and for acknowledging the effectiveness of projecting the posterior mean estimate over the existing approaches like PDM and PRODIGY.

Regarding the update step in Algorithm 1: Combining lines 7-9 in Algorithm 1, our update step is exactly the same as the DDIM update step with $\hat{z_0}$ being replaced by $\hat{z_{0, pr}}$.

The original DDIM update step results in sampling from an unconditional distribution, whereas, updating $\hat{z_0}$ with $\hat{z_{0, pr}}$ can be viewed as sampling conditioned on the specified constraints. [1] has employed similar algorithms for image personalization.

Further, we observed that using $\hat{\epsilon}$ is better for sample quality metrics. Below, we list the FD and the DTW metrics for CPS and CPS with the noise term updated for consistency.

Updating both posterior mean and noise results in overdependency on the projection step and pushes $z_{t-1}$ off the noise manifold for $t-1$. This takes $z_{t-1}$ far away from the training domain of the denoiser, resulting in poor sample quality as shown by the numbers above.

On the mismatch in the penalty schedule/coefficients between theory and experiments: Firstly, the exact penalty coefficients as shown in the theorem cannot be computed for real-world applications as constraints need not satisfy Assumption 4.3.

Therefore, our choice of penalty coefficients in Algorithm 1 is similar in essence to the penalty coefficients used in Theorem 4.4. The key takeaway from Theorem 4.4 is that, for convergence, the penalty coefficients should be decreasing with respect to $t$ (diffusion step) and can be chosen to increase from a small value during the initial denoising steps (t=T) to a very large value ($\gamma(1) \rightarrow \infty$) during the final denoising steps.

We ablated with linearly, quadratically, and exponentially (lines 183 -187) decreasing function choices (Table 5 of Appendix B.3 (page 21)). Note that the linear functional form, where the penalty coefficient linearly decreases from 10000 to 0 (increases during denoising), is proportional to the penalty coefficients obtained from Theorem 4.4 at all diffusion steps.

Interestingly, on all the real-world datasets, our key observation is that the function choice does not matter, and we obtain very similar sample quality metrics with linear, quadratic, and exponential choices (difference in third decimal place, check Table 5 of Appendix B.3 (page 21)). The results in the main table 1 are obtained using the exponential choice described in Algorithm 1.

On the mismatch between penalty functions in Algorithm 1 and in Theorem 4.2: The smoothness assumption on the penalty function for Theorem 4.2 followed by non-smooth penalty functions in the experimental setup and Algorithm 1 follow prior works from posterior sampling for inverse problems [2,3]. Prior works draw theoretical insights from linear and noise-free observation settings and extend them to non-linear and even non-differentiable cases.

Theorem 4.2's main focus is expressing the projection step as a series of gradient updates with fixed step sizes to convert $\hat{z_0}$ into $\hat{z_{0, pr}}$. Consequently, the projection step can be viewed as sampling from a Dirac delta distribution centered at $\hat{z_{0, pr}}$.

Here, the smoothness assumption only aids in ease of choosing the step size and the number of gradient updates, as described in Eq. 7 (page 21, Appendix C.1). This reasoning can also be extended to optimizing non-smooth penalty/objective functions in line 5 of Algorithm 1 as $\hat{z_{0, pr}}$ can be expressed as a sequence of updates from $\hat{z_0}$.

We also note that our experiments include both equality (mean, value at, etc.,) and inequality constraints (OHLC).

On the usefulness of theoretical analysis:

Theorem 4.2: The theorem draws equivalence between the projection step and sampling from a Dirac delta distribution centered at $\hat{z_{0, pr}}$. Essentially, the theorem provides the modified $p(z_{t-1} | z_t)$ under smoothness assumptions on the penalty function. As described earlier, these insights can be extended to non-smooth cases.

Theorem 4.4: Theorem 4.4 investigates the efficacy of CPS on the well-studied linear problem with unique solution and shows that we can do away with costly constrained optimization after each denoising step (baseline approaches - PDM and PRODIGY) and perform cheaper unconstrained optimization with steadily increasing penalty for constraint violation as we denoise.

Secondly, the theorem provides a framework for the choice of penalty coefficients (small during initial denoising steps and very large during the final denoising steps). This works well with our intuition that constraint satisfaction need not be heavily prioritized during the initial steps when the signal-to-noise ratio in the denoised sample is very low.

We also note that CPS, designed based this intuition, outperforms state-of-the-art approaches on sample quality.

Regarding the hyperparameter search for guidance-based methods:

Below, we provide a detailed analysis of how the constraint violation magnitude varies with the guidance weights.

Note that, for Guided DiffTime, as the guidance weight increases, the violation magnitude also increases drastically.

Whereas, for DiffTS, the violation magnitude largely remains the same with minor fluctuations.

For fair comparison, the guidance weight corresponding to the smallest violation magnitude from this table is used in our experiments.

On the inference latency for large number of constraints (~100): We refer the reviewer to Appendix D (pages 34 and 35) for a detailed analysis on inference latency. Our key observation is that the inference latency linearly increases with the number of constraint categories for all the real-world datasets.

In Figures 5 (page 9) and 7 (page 34), and Table 6 (page 34), we increase the number of constraint categories and observe the sample quality and inference latency.

We request the reviewer to check the Clarification regarding the constraints used in Figures 5 and 7 in our response to Reviewer iu3N for the list of constraints used in each experimental setup. In the caption for Figure 7, we also provide details regarding the hardware setup used for our experiments.

From Table 6, note that the average latency for the univariate traffic dataset is small, when compared to the other multivariate datasets (Air Quality and Stocks), for any number of constraint categories.

Additionally, note that the Stocks dataset has higher latency for any number of constraint categories because of the large number of OHLC constraints being imposed.

We also provide convergence analysis for Algorithm 1 under both Assumptions 4.1 and 4.3 in lines 1197 to 1201.

On the systematic evaluation of individual constraint types:

Below, we provide the FTSD (sample quality) comparison for individual constraint types evaluated on the traffic dataset.

From the table, we note that even for individual constraint types, CPS outperforms the baselines on sample quality for a large number of constraint types. Further, while combining multiple constraints together, from Figure 5 (page 9), we clearly observe that CPS provides the best sample quality for any number of constraints.

Modification to the manuscript

1. We will add the tables for systematic verification of constraints and hyper parameter search to the paper.

2. In the experiments and the limitations sections, we will address the differences in the penalty functions and coefficients between the theoretical justification section and Algorithm 1.

3. We will include a discussion and the table regarding consistency in the update step.

4. We will include a paragraph at the end of the theoretical justification section to explain the usefulness of the theorems.

References

[1] RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control

[2] Pseudoinverse-Guided Diffusion Models for Inverse Problems

[3] Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion

I appreciate the authors' detailed response and the additional empirical evidence provided. However, after careful consideration, I am maintaining my original rating. While the authors have addressed several technical questions and provided valuable empirical analysis, the core methodological concerns that motivated my original assessment remain unresolved. My main issue is not the absence of perfect theory, but rather the presentation of theoretical contributions that don't apply to the actual method.

We thank the reviewer for the feedback on our work. The reviewer acknowledges the depth of the provided empirical evaluations along with the simplicity and the theoretical guarantees for CPS.

On the range of constraints used in experiments: We note that the prior work [1] in constrained time series generation primarily focused only on linear or convex constraints individually (argmin, fixed value, and bounds). The approaches in [1] have not been tested for various combinations of constraints on multiple real-world datasets.

However, our experiments involve detailed evaluation on multiple combinations of constraints using a range of metrics that capture the sample quality and diversity.

Additionally, the number of constraints imposed on a generated sample is much higher (~100 and upto 450 for the stocks dataset) in our work than in [1]. We request the reviewer to check the Clarification regarding the constraints used in Figures 5 and 7 in our response to Reviewer iu3N.

On extension to non-linear constraints: We have compared CPS against other baseline methods for a general class of constraints. Specifically, we imposed the Autocorrelation Function (ACF) value at a particular lag as an equality constraint for the stocks dataset.

Firstly note that even CPS has constraint violation in this case as the constraint is non-convex. Secondly, the projection step, after every denoising step, does not provide the global minimum for $\hat{z}_{0,\mathrm{pr}}$, which was the case for linear and convex constraints used in the main table 1.

Even in such cases where the projection operation does not yield a $\hat{z}_{0,\mathrm{pr}}$ which is the global minimum, the iterative projection and denoising steps in CPS ensure that the adverse effects of projection are minimized and provide high sample quality.

We observe that CPS provides the best sample quality metrics while maintaining the least constraint violation magnitude (Appendix B.3, Table 4).

On limited novelty: Though the key difference between CPS and other approaches such as PDM and PRODIGY is the projection of the posterior mean estimate, we note that CPS applies the constraints on the estimate of a clean sample and not on the noisy latents, in accordance with the fact that the constraints are defined for clean samples. This improves the sample diversity (lines 215-218).

Additionally, this key difference ensures the modified latents ($z_{t-1}$) lie on the manifold corresponding to the noise level $t-1$, thereby ensuring accurate denoising with respect to PDM and PRODIGY. This results in improved sample quality (lines 219-222).

Regarding the qualitative results: In Figures 4 and 9, we observe almost perfect overlap between the real and the generated time series samples for a few datasets. Specifically, this can be observed for the conditional variants of the Air Quality and Traffic datasets along with the Stocks dataset.

This is primarily due to conditional inputs and a large number of constraints.

For the conditional variants (Air Quality Conditional and Traffic Conditional), the generated samples are already condition or metadata-specific and therefore track the real sample closely. In addition, when we impose constraints, the generated samples almost perfectly match the real samples. This does not happen in the unconditional cases and is empirically demonstrated through the differences in the DTW metrics between the unconditional and conditional cases (2.35 vs 1.83 for Air Quality and 3.41 vs 0.84 for Traffic, lower is better).

For the stocks dataset, the number of constraints is too high (450) that the feasible set is very small and therefore the generated samples closely track the real sample from the constraint set.

Consequently, in both these figures, note that the least similarity between the real and generated samples can be observed for the unconditional Traffic and Air Quality datasets which do not have a large number of constraints as the Stocks dataset. The generated samples only roughly follow the trend of the real sample, but do not match as perfectly as compared to the conditional variants.

Accordingly, as diffusion models generate stochastic outputs, Figure 10 depicts the distribution of samples generated for the same constraint set. We observe higher overall variance for unconditional Traffic and Air Quality datasets when compared to the Stocks dataset which has a larger number of constraints.

Additionally, as we impose the value at constraints, note that the standard deviation is 0 at 1, 24, 48, 72, and 96 timestamps.

Our objective with the qualitative results is to mainly showcase the matching trends with least distortions due to projections. This is also confirmed by the high sample quality metrics for CPS.

Modifications to the manuscript

1. We will include explanation regarding non-linear constraints in the experiments section.

2. We will add a paragraph explaining the qualitative results and the perfect overlap on selected datasets.

References

[1] On the Constrained Time-Series Generation Problem