Training-Free Constrained Generation With Stable Diffusion Models

Thank you for the time invested in reviewing our work and for praising the innovative aspects of the proposed method. We will address the point raised below and will integrate the suggestions in our final version.

Q1: Is the speed of the methods depending on the complexity of the constraint?

Yes. The performance of the proposed method depends largely on the complexity of the constraints enforced and the ease with which their violations can be measured. In the paper, we presented three cases with varying levels of complexity (and execution times) and reported runtimes in Appendix F.

Q2: How to evaluate the complexity of the constraint? Is there some numerical value to measure that？

Generally, one may want to categorize them in convex and non-convex, but in this paper, we go well beyond non-convexity as we include non-differentiable black box simulators that are used to constrain our model. To provide some details:

• Section 6.1: This is a knapsack constraint. While in general knapsack problems are NP-complete, the version adopted (which uses integers as weight) is known to be weakly NP-complete and admits a fully polynomial approximation. It is solved efficiently in $O(nm)$ where $n$ is the number of pixels and $m$ is the number of values to compute for the dynamic program.

• Sections 6.2 and 6.3: The inclusion of a blackbox simulator and a nonconvex neural surrogate characterizes these problems as NP-complete. This makes these settings complex but also very compelling.

Overall, it is evident that evaluating a constraint through a simple differentiable operation on the sample (e.g., microstructure porosity) is straightforward and leads to a fast and efficient correction algorithm. In contrast, evaluating a constraint using external numerical simulators or, more generally, black-box models (e.g., as in the case of our metamaterials experiment) involves higher computational costs and greater implementation complexity.

Thank you again for your review and your praise for our work. We believe that our responses have addressed each concern presented in your assessment but we are more than happy to provide additional details as needed. Many thanks!

Thank you for the time you dedicated to this detailed and constructive review! We appreciate that the strengths of our work were identified and highlighted, and we find all the comments helpful for further improving it. We address each of the points in detail below.

W1 & C3: Some of the figures are a bit unclear - also, because the captions are very brief and do not explain the figures in detail.

Thank you for your comment. We agree this as an area for improvement and will revise the captions accordingly:

• Figure 4: Successive steps of DPO. The sample is iteratively improved and the stress-strain curve aligns with the target. Structural analysis shows the progressive deformation under controlled compression.

• Figure 5: Left: Denoising process of Cond vs. Latent (Ours). Out method drives the denoising toward a copyright-safe image. Top-right: showing projection from original (O) to projected (P) in the PCA-2 space. Bottom-right: Constraint satisfaction and FID scores.

W2: It could be beneficial to include additional baselines from related work, or, for example, a baseline that enforces the constraints during training.

During development we indeed evaluated several additional baselines (which are discussed in Appendix E) but, due to space constraints, we included those that either represent the state-of-the-art in specific domains (Christopher et al. (NeurIPS 2024) and Bastek & Kochmann (Nature 2023) for microstructure generation and metamaterial design, respectively) or that we considered to be the best performing among existing alternatives (text-conditional model for copyright-safe generation).

We also highlight that in the cases of metamaterials and copyright, the selected baselines indeed incorporate constraints during the training phase. For example, Bastek & Kochmann impose periodic structure constraints during training. Similarly, constraints are encoded in Section 6.3 using a text-conditional model, which closely resembles existing work for training-time latent diffusion methods for constrained inverse-design settings [1]. Additionally, given the complexity of the constraints modeled in this paper (e.g., in Sections 6.2 we use a finite element analysis solver is used in the loop, and in Section 6.3 the constraints are highly nonconvex) very few existing methods are applicable. To the best of our knowledge, we compare to the strongest methods that currently exist for the settings explored. We will, however, empahsize in the paper, the additional baselines discussed in Appendix E.

W3: The computational complexity, efficiency and runtime of the method is not discuss in detail.

We are happy to expand on this point. Appendix F provides the requested runtimes, but we will make sure to give them greater visibility in the final version of the paper. In terms of constraint complexity:

• Section 6.1: This is a knapsack constraint. While in general knapsack problems are NP-complete, the version adopted (which uses integers as weight) is known to be weakly NP-complete and admits a fully polynomial approximation. It is solved efficiently in $O(nm)$ where $n$ is the number of pixels and $m$ is the number of values to compute for the dynamic program.

• Sections 6.2 and 6.3: The inclusion of a blackbox simulator and a nonconvex neural surrogate characterizes these problems as NP-complete, but also quite compelling applications.

We will report these details in the final version of the paper. Thanks for the suggestion.

C1: What is the difference between "physical laws" and "first principles"?

First principles are fundamental concepts; physical laws are their mathematical consequences or empirical summaries. We favor the law form when writing constraints. In practice, our usage partially overlaps; we will clarify as needed.

C2: Although the authors later define how stable diffusion models operate in a learned latent space, at this point in the text it might not clear to a reader who is not familiar with latent diffusion models what a "latent diffusion model" is.

This is a good observation, thank you. We will clarify.

C4: It seems a bit strange that there is no division of the equation by the (direction) of perturbation or something similar needed. Why is the case?

We will add a short derivation to show that, for the chosen step size, the direction scaling is absorbed into the proximal update, eliminating an explicit division.

C5: A more formal heading could be "Supplementary Proofs" or something similar.

Agreed. Thank you!

Q1: How would training-time enforcement of the constraints compare to the proposed approach

Thank you for the question, since this is one of our contributions. Besides requiring heavier training, we see two key limitations of training-time enforcement methods: (1) they cannot guarantee constraint satisfaction even for the case of convex constraints, and (2) It does not support generalization to unseen constraints, since the model only learns to ``satisfy'' (we quote the word satisfy here because this is only true at a distribution-level, not at the level of each sample) those present during training. These are two key strengths of our proposed approach. Interestingly, we also find that this allows us to fine-tune models using very sparse datasets, a property we do not observe with training-time methods. This raises a broader and open question: why does sampling-time enforcement enable generalization under limited supervision? We are actively investigating this.

Additionally, let us point out that some of the baselines we compare against do use training-time constraint enforcement, so the empirical comparisons reflect this distinction.

Q2: For example in Equation (6), the authors introduce the trade-off between constraint satisfaction and maintaining similarity to the original learned distribution. Could instead of this soft constraint also a hard constraint be used?

Yes, in principle, and we do so when the constraint is treated as an indicator function. However, the hard version requires solving a projection exactly at every step and requires a closed form solution. This is impractical for our black‑box constraints settings (Sections 6.2 and 6.3). The soft penalty gives a practical balance.

Q3: Where is the smoothing function defined? Is it just the sum operator?

Thanks for this question, since this is a key contribution of our work but, we agree its description would deserve additional space to make full justice to it.

The smoothing function is the map

where $\phi$ is the non‑differentiable simulator, $\varepsilon \sim \mathcal{N}(0,I)$ (or any full‑support noise, really) and $\nu>0$ is a temperature that scales the perturbation. Averaging the simulator's output over these random local perturbations smooths away the simulator’s discontinuities: for any fixed $\nu$, $\Phi_\nu$ is continuously differentiable even though $\phi$ may not be. As $\nu \to 0$ the bias vanishes and $\Phi_\nu$ approaches the raw simulator; as $\nu$ grows, the function becomes smoother but drifts toward the mean behavior of the simulator in a neighborhood of $\mathbf{x}$.

To make this practical, we estimate the expectation with $M$ Monte‑Carlo samples,

and treat the sample average as a differentiable proxy.

Now, because the noise is injected outside the simulator, gradients flow through the re‑parameterization $\nabla_{\mathbf{x}} \Phi_\nu(\mathbf{x}) = \frac{1}{\nu} \mathbb{E}\left[ \phi(\mathbf{x} + \nu\varepsilon) \varepsilon\right]$. The same formula with the finite‑sample average gives an unbiased gradient estimator that we use into our proximal Langevin update.

We will make sure to extend this discussion.

Q4: Figure 8: Does the performance get even better when you perform more iterations/steps?

Yes. By proceeding with more steps, it is in principle possible to achieve even smaller errors, as we have empirically observed. Since the computation time increases, however, it is also important to strike a good trade-off between these two desiderata.

Thank you again for your review and for championing our work. We believe our response has clarified all the outstanding questions, but we are happy to provide additional details as needed.

[1] Dontas, Michail, et al. "Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion." arXiv preprint arXiv:2412.00557 (2024).

Thank you for reviewing our work carefully and for highlighting its strengths and novel contributions. We answer each question below. Please let us know if we can provide any further clarification.

Q1: Do all intermediate samples, $x_t$, have to obey the constraints, or just the final sample at $t = 0$?

This is also the question we asked ourselves at the beginning of this investigation. In practice, only the final sample must satisfy the constraints, but constraining the intermediates matters. Indeed, already earlier work showed that enforcing post-sampling correction at $x_0$ results in substantial quality degradation because a late projection can push away the point from the target probability density region [1,2]. Our Theorem 4.1 formalizes the gain: keeping $x_t$ within the constrained manifold steers the chain toward the constraint set while still converging to $p_\text{data}$ (Theorem 4.2).

As you noted, early samples could be too noisy in some domains to be evaluated. We follow this general principle: apply projections as early as possible, while ensuring that the constraint is being properly evaluated. This is when the highest sample quality is achieved according to our observations.

Q2: I’m confused by the dimensionality of $z$ and the mapping $g$ in Section 4.1, and 4.2.

Thank you for noting this! The notation was overloaded. In Equation (6), we meant a sample and penalty function operating in the same space (latent or ambient). We will replace $z_t$ with $x_t$ there and keep Equation (7) as the latent‑space generalization. This makes the ambient‑space case clear while preserving generality. Again, we appreciate this observation and this has been addressed on our end already.

Q3: Do you mean every diffusion step, $t$, the projection in (7) needs to be run, and then additionally, the equation after line 178 needs to be run for the inner Langevin sampling steps w.r.t. $i$? Or, is equation 7 implicitly being satisfied by the corrected Langevin sampling steps in the eqn after line 178?

At each diffusion step (the outer loop in Algorithm 1), Equation (7) is satisfied by the correction (the inner loop in Algorithm 1). The proximal mapping defined in this equation (and implemented in the inner loop) is facilitated by the "inner minimizer" -- essentially, the equation presented for the inner minimizer can be viewed as a single unrolled step of the iterative process used to solve the optimization defined by Equation (7). We will add verbiage to better clarify this.

Q4: “Choosing $g$ as the indicator of a set, reproduces the familiar projection step introduced earlier”

To clarify, here we are referring to the case where $g$ is defined as an indicator function. In proximal operator, an indicator function has a very specific definition:

Hence, the constraint violation term completely dominates the optimization whenever $y \notin C$, and the proximal map becomes:

exactly the projection onto $C$. Additionally, provided this definition of an indicator function, there is no need for a complement. We hope this clarifies your question.

Q5: In Section 6.3 — do you need a projection path in, e.g. latent space like PCA?

Not strictly, but it can significantly enhance the efficacy of the method. For instance, a scalar classifier output could potentially be used as a score to guide the correction, but, as we had observed in our early experiment, it can be too lossy (the information captured by the model is compressed into a single value in the final layer). We thus project the penultimate activations to PCA‑2 so the guidance uses richer features without running into high‑dimensional clustering issues. Higher‑dimensional PCA might also help, but exploring that trade‑off is left as a possible future direction.

Thank you agin for your feedback! We hope these clarifications strengthen the paper and are happy to discuss any remaining concerns.

[1] Christopher, Jacob K., Stephen Baek, and Nando Fioretto. "Constrained synthesis with projected diffusion models." Advances in Neural Information Processing Systems 37 (2024): 89307-89333.

[2] Yuan, Ye, et al. "Physdiff: Physics-guided human motion diffusion model." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

Thank you for your review and for highlighting the novelty points in our work!. We found the aspect you pointed out to be very constructive, and we address them below. Please let us know if any pending aspect remains to be clarified; we are very happy to engage further.

For black-box constraints, the approach estimates subgradients via finite differences, which may be noisy or inefficient in high-dimensional settings.

The point is well taken: numerical differentiation can be costly and unstable as the problem size or simulation cost grows. Our goal, however, is to give a general framework that can be applied to any black‑box model, even when classical differentiability fails. Crucially, the differentiable sensitivity analysis tool provided makes it possible to insert previously incompatible simulators into the inner loop of diffusion models! We believe this is an important and exciting contribution, as illustrated by our results. Let us also remark that, as the reviewer may be aware, in this generative‑model context an approximate gradient is often sufficient; the stochastic nature of diffusion already tolerates noise, and even coarse gradient information still drives the sampler toward high‑quality outputs.

In terms of efficiency, this can be improved on a case‑by‑case basis, e.g., batching model calls, parallel execution, GPU acceleration, but we consider these as engineering steps that are orthogonal to the main message and contribution of the paper and were not the focus of this initial study. While we have currently addressed this point in the limitation section, we will be happy to further emphasize it, should the reviewer deem it essential.

Authors compare their work with too few baselines.

Throughout our work, we took special care in identifying what we found to be the the strongest available state-of-the-art baselines for each task: Christopher et al., for microstructure generation, and Bastek & Kochmann for metamaterial design. In the case of copyright-safe generation, since it has not been addressed in prior work, the best available state-of-the-art approach we found was conditional diffusion (Ho et al.). During development, we also considered other baselines (which are discussed in Appendix E), but we omitted them from the main text to keep the exposition focused on meaningful targets for improvement.

Strong theoretical convergence guarantees are provided only for convex constraints.

This is not entirely accurate. Our convergence guarantees are provided under a $\beta$-prox-regularity assumption (in the sense of [A] Definition 13.27). This is a relaxation of typical convexity assumptions, which implies that for each viable point and normal direction, small perturbations still project uniquely and smoothly back to $\mathbf{C}$. Hence, the theoretical convergence guarantees extend beyond solely convex constraint sets.

However, as the reviewer will surely know, guarantees for truly general non‑convex constraints remain out of reach for the optimization field at large, so proving results under prox‑regularity should be considered a strength rather than a limitation.

Thank you again for your positive assessment of our work! We hope these clarifications address your concerns, and would appreciate your championing our work further. We are happy to follow-up if other questions arise. Many thanks!

[A] Rockafellar, R. Tyrrell, and Roger JB Wets. Variational analysis. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998.

Thank you! We remain available in case there may be additional points; Since the reviewers' questions have been answered, we also hope could see this reflected in the originality, quality, and novelty scores of this review. Again, thank you for your time and help.