On the extension to non-linear constraints (Question 3)

We agree with the reviewer that the objective function in the projection step (line 5, Algorithm 1) is non-convex for arbitrary constraint classes. Consequently, the projection step may not result in the global optimum. Additionally, it is not possible to guarantee constraint satisfaction for non-convex constraints.

We request the reviewer to check Sec. H of the Appendix, where we have added new experiments with a general class of constraints. Specifically, we imposed the Autocorrelation Function (ACF) value at a particular lag as an equality constraint for the stocks dataset.

We observe that, even though the projection step does not result in the global optimum, the iterating projection and denoising process pushes the generated sample to the constraint set. Additionally, the sequential denoising process removes the adversarial artifacts caused by the projection step. This provides very low constraint violation metrics for CPS, when compared to other baselines, while maintaining high sample quality (check Table 6 in the Appendix, page 31).

Clarification regarding the design parameter $k$ in the convergence analysis (Question 4)

In Theorem 2, for ease of analysis, we consider $k$ as a design parameter that affects the penalty coefficient $\gamma(t)$ in a linear fashion $\left(\gamma(t) = \frac{2k(T-t+1)}{\lambda\_\mathrm{min}(A^TA)}\right)$. Note that we have considered $\gamma(t)$ as a linearly decreasing function of $t$.

Here, we observe that for a finite number of denoising steps $T$ and a finite value of $k$, CPS does not converge to the true solution $x^*$, and the upper bound on the terminal error is given in Eq. 26 (Sec A.2 of the Appendix, lines 951-956). However, as $T$ and $k$ tend to infinity, CPS converges to $x^*$. More specifically, our proof in Sec A.2 of the Appendix implies that the way to decrease the terminal error is to have large values of $T$ with $\gamma(1) = \infty$. This coincides with our choice of $\gamma(t)$ for the final denoising steps, where $\gamma(t)$ tends to infinity.

In short, we only consider $\left(\gamma(t) = \frac{2k(T-t+1)}{\lambda\_\mathrm{min}(A^TA)}\right)$ as a linearly decreasing function of $t$ with an adjustable design parameter $k$ for ease of theoretical analysis. $k$ is not a part of Algorithm 1.

We have highlighted all the changes and additions in red in the manuscript.

[1] PSA-GAN: Progressive Self Attention GANs for Synthetic Time Series, Jeha et al. [2] Time Weaver: A Conditional Time Series Generation Model, Narasimhan et al. [3] Time-series Generative Adversarial Networks, Yoon et al. [4] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation, Tashiro et al.

We thank the reviewer for taking the time to provide a thoughtful and thorough review of the paper. The reviewer recognizes the generality of the approach as it can be applied to any domain. Additionally, the reviewer acknowledges the extensive comparisons with correct baselines.

Addressing the weaknesses pointed out by the reviewer.

On the extension to a general class of constraints (Weakness 1)

CPS can be extended to non-convex constraint sets as well. As pointed out by the reviewer, the projection step will not result in the global optimum. Additionally, as hinted by the reviewer, our projection step can be swapped with a constrained optimization step for a general class of constraints without any modification to the rest of the algorithm.

To emphasize the ability to extend to a general class of constraints, we showcase CPS’s performance for non-convex constraints, specifically the equality constraints based on the Autocorrelation function (ACF) value for a specified lag (check Table 6 and Sec. H in the Appendix, page 31).

Given a finite number of optimization steps for each projection operation, it is not possible to guarantee constraint satisfaction for all the baselines, including CPS. However, we observed that CPS provides the best sample quality and the lowest constraint violation metrics.

This is attributed to the iterating projection and denoising process, which pushes the generated sample very close to the required constraint set while simultaneously minimizing the adverse effects of projection through sequential denoising.

On the evaluation of fidelity (Weakness 2)

We relied on the same fidelity metrics used in prior works [1,2]. However, we agree with the reviewer’s proposed evaluation metric. In [3], this is referred to as the discriminative score (DS), where a 2-layer recurrent neural network (RNN) is trained to perform binary classification between the real and the generated time series. Consequently, the classification error on a heldout test set is reported as DS. Lower values of DS represent poor classification accuracy, indicating high sample fidelity. The DS values for all the experimental setups in Table 1 are provided in Table 2 (Sec. E, page 28) in the Appendix. Overall, CPS conveniently outperforms the baseline approaches even with respect to DS.

We thank the reviewer for revising the score. We are glad that your questions have been addressed. Below, we address your remaining question regarding how CPS is different from Projected Diffusion Models (PDM) [A].

We note 3 crucial differences between our CPS approach and PDM. (1) We first highlight the key difference between the sampling procedures in CPS and PDM.

PDM Approach - PDM projects all the intermediate latents $z_T, \dots, z_1$ to the required constraint set along with $z_0$. Therefore, for every denoising step $t$, PDM operates by obtaining $z_{t-1}$ from $z_t$ and projects $z_{t-1}$ to the constraint set using a constrained optimization step.

Our Approach (CPS) - CPS does not project the intermediate latents. Instead, the posterior mean estimate $\hat{z}_0$ (or the expectation of the clean sample given the noisy latent $z_t$, $E[z_0 | z_t]$) is projected to the constraint set. Therefore, for every denoising step $t$, CPS does the following operations:

• Computes the posterior mean estimate $\hat{z}_0(z_t)$ from $z_t$.
• Projects the posterior mean estimate $\hat{z}_0(z_t)$ to the constraint set using an unconstrained optimization step (line 5, Algorithm 1). The reviewer is correct; $\gamma(t) \Pi(z)$ is the penalty for constraint violation. This turns into a hard constraint when $\gamma(t)$ becomes infinity at the final denoising step.
• Finally, the projected posterior mean estimate $\hat{z}\_{0, \mathrm{pr}}(z_t)$ is transformed to $z_{t-1}$ using a non-markovian forward noising process (Eq. 2, lines 165-167).

Next, we discuss two major issues of PDM that degrade its sample quality and how our method circumvents these issues.

(2) The constraint set is defined for the clean samples and not the intermediate noisy latents of a denoising process. As the goal is to generate constrained time series samples, it is sufficient if the generated sample $z_0$ belongs to the constraint set (i.e., terminal constraint). However, PDM assumes the constraint set for clean samples to be the same for noisy intermediate latents. By forcing the latents to satisfy the same terminal constraint, PDM eliminates most sample paths ($z_T, \dots, z_0$) that may eventually satisfy the terminal constraint. This results in poor sample diversity.

How does CPS handle this problem? CPS eliminates this problem by projecting the posterior mean estimate $\hat{z}_0$ and not the noisy intermediate latents. Recall that $\hat{z}\_0$ is the expected clean sample which has a similar noise level as the constraint set. Furthermore, the projected posterior mean estimate $\hat{z}\_{0,\mathrm{pr}}(z_t)$ is transformed to $z_{t-1}$ using a non-markovian forward noising process. This effectively allows for sample paths where the generated sample $\hat{z}_0$ alone satisfies the terminal constraint and the intermediate noisy latents can be flexible.

(3) PDM projection step pushes $z_{t-1}$ off the noise manifold for $t-1$. In PDM, the projection step, when applied directly to the noisy latent $z_{t-1}$ pushes it out of the noise manifold corresponding to the diffusion step $t-1$. Consequently, a pre-trained denoiser struggles to accurately denoise the projected $z_{t-1}$ as it would be out of the training domain of the denoiser.

How does CPS handle this problem? This effect is significantly reduced in CPS because CPS does not project $z_{t-1}$. Instead, CPS projects the expected clean sample $\hat{z}\_0$. Consequently, the projected posterior mean estimate $\hat{z}\_{0, \mathrm{pr}}(z_t)$ is transformed into $z_{t-1}$ by using a non-markovian forward noising process (Eq. 2). This ensures that $z_{t-1}$ stays very close to the noise manifold corresponding to the diffusion step $t-1$. Therefore, a pre-trained denoiser can denoise $z_{t-1}$ more accurately in our approach. This ultimately preserves the generated sample quality.

Finally, we have highlighted the main differences in the sampling process between the CPS and PDM, and how this difference affects the generated sample quality. Empirically, we show that CPS performs much better than PDM on the real-world datasets, providing 7x reduction in the Frechet Time Series Distance metric and 4x reduction in the Discriminative Score metric overall, as given in Table 3 (page 30).

We hope our response has clarified the differences between our algorithm and PDM [A]. We are happy to address any further questions during the discussion phase.

References

[A] Constrained Synthesis with Projected Diffusion Models, Christopher et al.

We thank the reviewer for taking the time to provide a thoughtful and thorough review of the paper. The reviewer acknowledges the simplicity of the proposed approach and recognizes the effectiveness of CPS with respect to generating high-quality samples that adhere to hard constraints.

Addressing the weaknesses pointed out by the reviewer

On the validity of the similarity metrics. (Weakness 1)

From our experiments, we note that higher similarity scores between the synthesized and the real test samples indicate better sample quality. From Table 1, observe that the highest SSIM and the lowest DTW scores (best similarity metrics) correspond to the lowest TSTR and the lowest Frechet Distance metrics (best sample quality metrics). We agree with the reviewer’s point that there could be multiple generated samples that belong to the constraint set. However, we specifically design the constraint set with a large number of constraints such that only one real test sample exists per constraint set (check lines 427-429). Therefore, any generated sample with high sample quality is generally expected to be similar to the real test sample. As a result, we report the similarity metrics.

Therefore, even though COP variants have perfect constraint satisfaction, their sample quality metrics are poor, and this is also reflected in the poor similarity metrics.

On the comparison with Loss DiffTime. (Weakness 2)

Loss DiffTime provides better sample quality but fails to adhere to the constraint set due to the absence of principled projection steps. We agree with the reviewer that comparing against Loss DiffTime will showcase the relative performance gap for training-free approaches like CPS. To this end, we trained the Time Weaver-CSDI models with constraints as conditional input. The results are provided in Table 5 (Sec. G, page 30) in the Appendix. Overall, Loss DiffTime fails to generate samples that belong to the required constraint set because there is no explicit way to force the generated sample to adhere to the required constraints. Constraint-specific training only aids in generating high-quality samples.

Thank you for the response. You have addressed most of my questions and concerns well. I still have a comment on the similarity metrics, though I don't consider it a major issue.

However, we specifically design the constraint set with a large number of constraints such that only one real test sample exists per constraint set (check lines 427-429). Therefore, any generated sample with high sample quality is generally expected to be similar to the real test sample.

While there may only be one real sample satisfying the constraints, there may be other possible samples in the real data distribution that also satisfy the constraints, and have non-negligible probability to appear in a dataset of the size of the synthetic dataset. Because of this, it is possible that the perfect synthetic data generator that exactly samples from the distribution of real samples satisfying the constraint would not get the best possible score.

Regardless, I think this is a fairly minor issue, since your method works very well according to the other metrics which do not have this issue.

On the comparison with Diffusion-TS. (Question 4)

We modified the Diffusion-TS baseline for constrained time series generation. However, since it is a guidance-based approach, it fails to generate samples that adhere to hard constraints. Although the guidance approach provided in Diffusion-TS is not for constrained sample generation, we agree with the reviewer that this approach can be modified for our problem setting. We provide the comparison against Diffusion-TS in Sec. G of the Appendix (check Table 4, page 30).

However, similar to the Guided-DiffTime approach, there is no principled projection step for constraint satisfaction. This is shown through the high values for the constraint violation metrics.

We provide an additional comparison against a very recent constrained sample generation approach, Projected Diffusion Models (PDM) [3] (check Table 3 in Sec. G of the Appendix, page 30). Overall, we observe that CPS outperforms all the training-free methods in generating high-quality samples that adhere to the required constraints.

Clarification regarding the convergence analysis. (Question 5)

$k$ is independent of $T$. In our convergence analysis, $k$ is a design parameter that affects the penalty coefficient $\gamma(t)$ in a linear fashion $\left(\gamma(t) = \frac{2k(T-t+1)}{\lambda_{\mathrm{min}}(A^TA)}\right)$, and $T$ is the total number of denoising steps. $T$ and $k$ are independently chosen and the choice of $T$ does not affect the value of $k$. Theorem 2 claims that, under Assumption 2, CPS arrives at the unique solution $x^*$ when $T$ and $k$ tend to infinity. For any finite $k$, the terminal error in the generated sample is upper bounded, as shown in Theorem 2.

For both finite $T$ and finite $k$, the upper bound on the terminal error is shown in Eq. 26 (lines 951-956) in Sec A.2 of the Appendix.

We have corrected the citation errors and updated the manuscript accordingly. We have highlighted all the changes and additions in red in the manuscript.

[1] On the Constrained Time-Series Generation Problem, Coletta et al. [2] Diffusion-TS: Interpretable Diffusion for General Time Series Generation, Yuan et al. [3] Constrained Synthesis with Projected Diffusion Models, Christopher et al.

We thank the reviewer for taking the time to provide a thoughtful and thorough review of the paper. The reviewer acknowledges the practical importance of addressing the constrained time series generation problem. Additionally, the reviewer recognizes that the effectiveness of CPS is shown on multiple real-world datasets.

Response to the questions from the reviewer

On the novel contributions with respect to Coletta et al. ([1]). (Question 1)

Our main novelty, which leads to significant improvements in both sample quality and constraint satisfaction metrics, is the iterated projection and denoising process. Conceptually, the closest approach to CPS is Guided-DiffTime. The main difference between the two approaches is how the noisy latent for the step $t-1$, $z_{t-1}$, is obtained from the noisy latent for the step $t$, $z_t$.

Guided-DiffTime approach - Guided-DiffTime obtains the noise estimate in $z_t$ and modifies the noise estimate using a gradient update to minimize constraint violation. This is indicated by $\hat{\epsilon} \leftarrow \hat{\epsilon} - \rho\nabla_{z_t}\Pi(\hat{z}_0(z_t))$, where $\Pi(\hat{z}_0(z_t))$ is the constraint violation. This requires a backpropagation step through the denoiser. As explained in [1], $\rho$ is chosen through trial and error.

Our approach - However, in CPS, we first compute the posterior mean estimate from $z_t$ (line 4, Algorithm 1), project it to the constraint set (line 5, Algorithm 1), and transform the projected posterior mean estimate to $z_{t-1}$ (lines 7-9, Algorithm 1). In CPS, there is no backpropagation through the denoiser.

This key modification provides the following advantages over Guided-DiffTime:

1. CPS provides constraint satisfaction guarantees for convex constraints, whereas Guided-DiffTime cannot. In CPS, the projection operation after each denoising step (including the final step) provides constraint satisfaction guarantees for convex constraints for an appropriate step size and number of update steps in the projection operation.
2. As a consequence of the projection operation, CPS can handle a large number of constraints much more effectively than Guided-DiffTime. Guidance from different constraint violations needs to be treated accurately, resulting in a tedious hyperparameter tuning process (check lines 495-497). However, the projection step in CPS circumvents this problem, and the subsequent denoising steps remove the unnatural artifacts caused by the projection step, resulting in high sample quality.

Coletta et al. also propose Constrained Optimization Problem (COP). Below, we list the conceptual and empirical advantages of CPS over COP.

1. CPS does not require any external module. Whereas, COP uses a discriminator network to enforce realism during the projection process. This requires additional training (apart from the diffusion model) and leads to complex sampling procedures.
2. COP is very sensitive to the initial seed of the optimization process. The sample quality is poor if the initial seed is far away from the constraint set. In this case, the projection operation destroys the sample quality to achieve constraint satisfaction (check lines 476-482). Overall, CPS achieves much better sample quality metrics than Guided-DiffTime and COP.

On handling high-dimensional constraints. (Weakness 3)

CPS is equivalent to [A] in terms of computation, as both approaches require a projection step after every denoising step. We evaluated our approach for time series samples up to 576 dimensions (e.g., the air quality and the stocks dataset). We have provided the inference time taken to generate samples with up to 66 and 450 constraints for the air quality and the stocks datasets, respectively (check Table 8 in Sec. J of the Appendix). In comparison to the baseline approaches, CPS is only slightly more in terms of inference time while providing high sample quality and perfect constraint satisfaction.

Furthermore, we note that there are multiple ways to reduce the inference time, such as:

1. Reducing the number of update steps in each projection operation (line 5 of Algorithm 1) during the initial denoising steps where the signal-to-noise ratio is very low.
2. The projection operation (line 5 of Algorithm 1) need not be performed for every denoising step. Consequently, we can develop principled methods to identify the denoising steps where projection is required based on constraint violation.

We have highlighted all the changes and additions in red in the manuscript.

[A] Constrained Synthesis with Projected Diffusion Models, Christopher et al.

We thank the reviewer for taking the time to provide a thoughtful and thorough review of the paper. The reviewer acknowledges our approach’s ability to scale to a large number of constraints without any constraint-specific training.

Addressing the weaknesses indicated by the reviewer:

On the comparison of CPS with constrained diffusion models from other domains. (Weaknesses 1 and 2)

CPS is scalable to new constraints without any training. However, the prior works ([B],[C], and [D]) are not. [B], [C], and [D] focus on learning data distributions, where the domain of the samples is constrained to some manifold. These works introduce constraint-specific training modifications.

Additionally, CPS tackles a different problem, where the sample domain is not restricted to any manifold, but the goal is to generate samples that adhere to arbitrary constraints. Any constraint-specific training solution hinders the ability to adapt to arbitrary constraints.

We elaborate on this point further by explaining our evaluation process, which involves generating N samples where each sample adheres to a different constraint set. For example, considering the class of constraints to include mean and argmax, our evaluation process would be the following:

1. Generate sample 1 with the following constraints: mean = 4.5, argmax = 10,
2. Generate sample 2 with the following constraints: mean = 1.2, argmax = 30, and so on.

Adopting the approaches in [B], [C], and [D] would involve training a model for each of the N samples.

On the comparison with Projected Diffusion Models [A].

[A] addresses the same problem as ours. The major difference between [A] and our approach is the projection step. In [A], the noisy latents ($z_T, \dots, z_1$) are projected using a constrained optimization approach, whereas CPS projects the posterior mean estimate ($\hat{z}_0$, check line 4, Algorithm 1) using an unconstrained optimization approach.

We compared our approach (CPS) against [A], and CPS outperforms [A] on sample quality metrics for all the real-world datasets used in our experiments. We request the reviewer to check Table 3 in Sec. G of the Appendix for the results.

We hypothesize that the projected version of the noisy latent $z_{t-1}$ has a very low likelihood when compared to the training samples that the pre-trained diffusion model has seen, and therefore, the overall denoising process results in samples with poor fidelity.

On the contrary, during the sample generation process, CPS projects the posterior mean estimate (line 5, Algorithm 1) and subsequently transforms the projected posterior mean estimate to obtain $z_{t-1}$, using a non-markovian forward diffusion process (Eq. 2). Our key intuition is that the projection step suitably modifies $z_{t-1}$ to steer the denoising process towards the constraint set and the forward process nullifies the adversarial effects of the projection step.

We have added a new section to the Appendix - Sec. F - Extended Related Works, where we explained the difference between CPS and prior works [B], [C], and [D].

Extension to a general class of constraints.

Additionally, similar to [A], we show that our approach is not restricted to convex constraints and works for a general class of constraints. Specifically, we experimented with the stocks dataset to generate samples with a constraint on the Autocorrelation Function (ACF) value at a specific lag (check Sec. H of the Appendix). Even for such a general class of constraints, our approach outperforms the baselines in terms of sample quality and constraint violation metrics (check Table 6).

Further, generating time series samples with such arbitrary constraints has significant practical applications in terms of stress-testing and synthesizing samples for novel what-if scenarios. The prior state-of-the-art work in this domain, [E], only considers simple bounds, argmax, and fixed value constraints. The evaluation was performed on a single dataset. More importantly, [E] never considers the combination of multiple constraint classes, a key area where all the baselines fail to perform well. In our work, we have significantly improved the comparison in terms of the combination of multiple constraint classes, as well as datasets (6 datasets).

[A] Constrained Synthesis with Projected Diffusion Models, Christopher et al. [B] Homogeneous linear inequality constraints for neural network activations, Frerix et al. [C] Mirror diffusion models for constrained and watermarked generation, Liu et al. [D] Metropolis sampling for constrained diffusion models, Fishman et al. [E] On the Constrained Time-Series Generation Problem, Coletta et al.