Fast constrained sampling in pre-trained diffusion models

We thank the reviewer for their time and constructive feedback. We respond to the weaknesses raised and questions below:

Limited evaluation tasks:

Our evaluation is performed with both linear (free-form inpainting, x8 super-resolution, box inpainting) and non-linear (style guidance) constraints. This is consistent with previous papers on training-free diffusion sampling algorithms, some of which are often limited to just linear constraints (e.g. PSLD [25], P2L [4]). Additionally, in Appendix B.1, we included another non-linear task, mask-guided generation, which we omitted from the main text due to space constraints. There, we utilize a separately-trained face segmentation network to guide image generation with a target face mask. Any readers interested in mask-guided generation were pointed to it in line 60. Overall, we demonstrated our algorithm on a variety of tasks, showcasing its widespread applicability and consistent advantages over existing baselines.

Performance trade-offs:

We argue that we can perform as good as or better than existing training-free sampling algorithms while only requiring a fraction of the inference time. We would like to emphasize that there is no notable performance trade-off when using our algorithm, as we discuss below.

• Regarding Table 1, although our algorithm performs considerably worse than P2L in 8x super-resolution, we highlight two key disadvantages of P2L.

  1. P2L requires 30 minutes to run on a single image, whereas our method only takes one minute.
  2. P2L uses a VLM (PaLI) to caption the low-resolution images before the diffusion, which introduces additional steps to the inference.

  As mentioned in lines 207-208, we believe that the main advantage of P2L in super-resolution comes from introducing text conditioning. All previous algorithms relied solely on the unconditional model to perform super-resolution, which may not be optimal when the model is required to synthesize specific high-resolution textures in the images.

  To verify this assumption, we ran the x8 super-resolution with text prompts generated from a Qwen-2.5 VLM [Qwen]. We provided the downsampled images and asked the VLM to generate a text caption for each, which we then used as conditioning when running our algorithm. The results below indicate that we can close the performance gap and even improve upon the evaluation metrics when introducing text captions to the super-resolution process. Therefore, there is no trade-off in super-resolution performance when using our algorithm if we also introduce a VLM to caption the low-res images.

  Method	PSNR ↑	LPIPS ↓	FID ↓
  P2L (with VLM captions)	23.38	0.386	51.81
  Ours	24.26	0.471	60.99
  Ours + VLM captions	24.95	0.405	44.74

• In Table 2, which presents a more difficult task by inpainting 25% of an image, our method significantly outperforms all baselines and is only outmatched by the fully fine-tuned SD-Inpaint model, which required 100k additional steps of training. Similarly, in Table 3, our method consistently achieves lower style scores compared to the baselines.

To summarize, our method is substantially faster than all baselines and outperforms them on the more challenging tasks. To showcase this better, we will add the text-guided super-resolution result and the relevant discussion to the paper to complement Table 1.

Limited necessity of training-free sampling:

We added the SD-Inpaint result in Table 2 as a theoretical upper limit of training-free approaches. We expected that training-free methods would produce worse results than the fine-tuned model, which required considerable additional compute to train. However, this comparison does not take away from the necessity for developing efficient and accurate training-free sampling algorithms:

1. Training-free sampling algorithms can help understand the inner workings of these massive generative models. In the box inpainting task, we show that, using our method, Stable Diffusion accurately continues the textures from the given parts of an image in the inpainted regions, retaining the overall image structure (Figure 4). These findings can provide valuable insights into how these models understand and process images.

2. For the linear tasks considered in this (and previous) training-free sampling papers, data can be easily obtained. However, there are cases where there is not enough data to fine-tune a model for conditional sampling. Recent works have applied the training-free sampling algorithms we compare to in our paper, in settings where no trained conditional model exists:

   • [Reb1] Training-free image generation with image layouts and motion vectors as conditions.
   • [Reb2] Training-free motion generation with text prompts, keyframes, and obstacles in 3D space as conditions.
   • [Reb3] Training free molecule generation with geometric constraints.

Inpainting and super-resolution provide a good testbed for benchmarking training-free sampling algorithms. We show that our algorithm improves upon the performance and efficiency of existing training-free sampling algorithms, and future works can utilize our method as a drop-in replacement for the training-free conditioning guidance.

Visualizations:

The artifact mentioned in Figure 3 is a watermark texture hallucinated by the model. Stable Diffusion was trained on web images, which were often watermarked, and can thus generate such patterns. We would like to point out that the hallucinated watermark is adhering fully to the imposed low-resolution image constraint. The mean squared error between the downsampled watermark image and the constraint is around 0.0008, which is also the mean squared error we get when we compare the constraint to another generated lighthouse image that has no hallucinated watermark. We agree that this image may cause unnecessary confusion to the reader and will replace it with another sample that has no watermark hallucination. We will move this example to Appendix B.7 More Qualitative Results.

Regarding Figure 5, the baselines seem to copy the color palette, but do not exhibit the same brush strokes and textures visible in the reference style image. For instance, the tree texture from Van Gogh's painting (bottom row) has been correctly replicated in the cat's body in our generated sample, but is nowhere visible in the baselines. The baseline methods seem to only synthesize a cat fur texture with colors from the reference style.

This qualitative observation translates to the results of Table 3, where the proposed algorithm achieves a lower style score (lower is better), representing the style of the reference image more accurately. If imposing the constraint too strictly generates undesired images, a user can easily relax the constraint by reducing the learning rate $\lambda$ or optimization steps $K$; it is an advantage that our algorithm provides the option to enforce the constraint as strictly as necessary.

Incorporate highlighted, intuitive summaries to better guide readers through the methodological details and key insights:

The main intuition behind our method is that in diffusion models, we can approximate the Newton optimization step $J^{-1}e$ with several smaller steps in the direction of $\lambda Je$, which is cheap to compute. This approximation is possible in denoising diffusion because $J$ and $J^{-1}$ exhibit similar structures, as discussed in lines 133-134.

To better highlight this intuition, as also suggested by Reviewer NFw9, we performed a smaller-scale experiment, where it is possible to exactly compute the Jacobian and compare $J$ and $J^{-1}$. For a diffusion model trained on MNIST, we showcased how the two matrices encode similar structures by comparing the similarity of their top 20 eigenvectors across different timesteps. Please refer to our response to Reviewer NFw9 for further details on this experiment.

We will include this toy experiment in the main paper, highlighting how we exploit the similar structures of $J$ and $J^{-1}$ to formulate our method. Moving beyond a toy dataset, e.g. a Stable Diffusion-scale model, is computationally infeasible, as a single Jacobian computation would require ~16000 backward passes through the model. Nevertheless, we trust that this toy example can help readers better understand our approach.

Adding a framework figure:

Instead of a framework figure, in which it would be difficult to visualize the difference between the various gradient updates in high-dimensional space, we will add examples of the eigenvectors of $J$ and $J^{-1}$ from the toy experiment above. The eigenvectors of $J^{-1}$ are images that showcase how the model encodes correlations between image pixels, i.e., how the model input should change for a constraint applied to its output. By showing that our proposed $Je$ update captures similar correlations to what the optimal inverse update $J^{-1}e$ would, we provide a visual explanation of the intuition behind our method.

Additional references:

[Qwen] Bai, Shuai, et al. "Qwen2. 5-VL technical report." arXiv preprint 2025

[Reb1] Wang, Zixuan, et al. "Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesis." CVPR 2025

[Reb2] Karunratanakul, Korrawe, et al. "Guided motion diffusion for controllable human motion synthesis." ICCV 2023

[Reb3] Ayadi, Sirine, et al. "Unified guidance for geometry-conditioned molecular generation." NeurIPS 2024

We thank the reviewer for their thorough comments. We respond to the questions below:

The assumption made in Eq. (11) that $Je$ is similar to $J^{-1}e$ needs further empirical/theoretical justification:

We understand that the intuition provided in line 133 may not be satisfying. The main difficulty of empirically demonstrating the similarities between $J$ and $J^{-1}$ lies in computing the Jacobian $J$ itself. Computing the Jacobian locally for one input $x_t$ could require ~16000 autodifferentiation passes, one for every row of the Jacobian. Understandably, for a model at the scale of Stable Diffusion (800M parameters), running an experiment where we compute the Jacobian at multiple input points is infeasible within the timeframe of this rebuttal.

As an alternative, we discuss a smaller-scale experiment with a small diffusion model (25M parameters) trained on the MNIST dataset. We padded the MNIST digit images with zeros to 32x32 pixel images, making the input and output of the model 1024-dimensional. Here, $J \in \mathbb{R}^{1024\times1024}$ can be readily computed with automatic differentiation (~10s on our machine), allowing us to also compute $J^{-1}$ assuming $J$ is not singular.

Measuring the similarity between the two matrices is challenging; we find that they differ significantly in scale, with $J^{-1}$ having much larger values than $J$, which makes measuring some norm of their difference (e.g. Frobenius) unreliable. What we are interested in is whether the two matrices exhibit similar structures, i.e., given an input, whether they transform it in similar ways. To measure this, we can compute the eigendecomposition of the two matrices and compare the top-K eigenvectors. If the transformation defined by $J$ is similar to $J^{-1}$, we expect the most important eigenvectors to be similar, which we can easily measure with cosine similarity.

We perform the following experiment: Given a noisy image $x_t$ at timestep $t$, we compute the exact Jacobian $J$ using automatic differentiation (torch.autograd.functional.jacobian), its inverse $J^{-1}$, and the eigenvectors and eigenvalues of both matrices. We then select the top 20 eigenvectors, excluding their imaginary parts, and ordering them by the magnitude of the corresponding eigenvalue. To account for misalignment between the eigenvalues, we pair each eigenvector of $J$ with its closest eigenvector from $J^{-1}$ and measure the average cosine similarity across all eigenvector pairs.

We run this experiment for different timesteps $t$, sampling 10 random images at each timestep from the MNIST dataset to compute $J$ and $J^{-1}$. As a baseline, we choose random $J$ and $J^{-1}$ matrices computed at the same timestep and repeat the experiment to show that the similarities measured are not spurious.

By showing the high similarity between the top 20 eigenvectors of the two matrices, we empirically demonstrate the intuition of line 133; the two transformations $J$ and $J^{-1}$ can be used interchangeably to compute the gradient direction for the constraint. A particularly surprising finding is that in some cases, we found almost the same eigenvector (cosine similarity $\approx 1$) from the two matrix decompositions.

Although we demonstrate this in a smaller-scale experiment, we believe it supports our argument in the paper for using an inexact Newton method that approximates the steps of $J^{-1}e$ with $Je$. We will include the above experiment and discussions in the paper to strengthen the intuitive explanation given.

Eqs. (10)-(13) and introduction of the learning rate:

Eq. (10) tells us that we must compute an approximate solution $e^\*_t$ that strictly reduces the residual error $r$ if we want the method to converge. The solution we choose in this paper is $e_t^* = \lambda Je$. We agree that it is better to include the learning rate $\lambda$ in Eq. (11) to avoid confusion.

Regarding the role of the learning rate, the sole reason we introduce it is to ensure the convergence of the proposed update. Although in our experiments we tuned the learning rate empirically, in Appendix A we provide an in-depth discussion of how $\lambda$ is related to the spectra for $J$ and affects the convergence. We will point readers to that in the main paper.

Why is this considered Newton's method?:

Newton's method involves solving for the inverse of the Jacobian $J^{-1}$ and performing the update $J^{-1}e$. In cases where computing the inverse is infeasible, we can approximate Newton's method with updates $Me$, where $M$ is chosen such that it is as close (in structure) to $J^{-1}$ as possible, but cheap to compute. These methods are called Inexact Newton methods and they have been studied in numerical optimization before (see Inexact Newton Methods [5]).

Our algorithm utilizes an Inexact Newton method that approximates $J^{-1}e$ with $Je$. The reason this approximation works well is specific to denoising diffusion models and is discussed above.

Finite-difference approximation and exact gradient computation:

To compute the $Je$, we use the finite difference approximation $Je \approx (f(x + \delta e) - f(x))/\delta$. The automatic differentiation libraries implement the backward gradient computation, which gives the gradient descent direction $J^Te$. This is what e.backward() computes in PyTorch.

Some libraries also offer forward-mode auto-differentiation, which computes the Jacobian-vector product $Je$. However, forward differentiation is not always implemented or optimized as well as the backward propagation of gradients. In the case of PyTorch, which is what we use for running Stable Diffusion, forward mode differentiation is not directly implemented for many of the custom Stable Diffusion layers.

To perform a comparison between our finite difference approximation and the exact forward computation, we resorted to the double backward trick, which computes the forward-mode gradient with two backward calls. This is, of course, slower and memory-intensive, but we use it as a baseline to verify the finite-difference approximation employed in the paper. For completeness, we also ablated the choice of $\delta$

We repeated the box inpainting task of Table 2 and present the results below:

$K$ number of steps, $\lambda$ learning rate, $\delta$ finite difference step size

The finite-difference approximation is (i) robust to the choice of $\delta$, and only fails when using too small (0.0005) and too large (0.5) $\delta$ values, and (ii) performs as well as the exact forward mode auto-differentiation.

Comparing the performance against the baseline with the same computation budget:

In Appendix B.6, we briefly discuss what happens when the number of constraint optimization steps per denoising diffusion step is increased from 5 to 20. What we observed is that once the constraint optimization converges to a local optimum, the additional steps do not improve the final generated image and may also, in some cases, introduce unwanted artifacts. This is in line with the observations made by previous works (DPS [3]).

We believe that the empirical values for the number of steps $K$ and learning rate $\lambda$ we chose in the paper are adequate for the optimization to converge sufficiently, and thus increasing the computation would not impact the result significantly. Since the results we obtained were already better than the baselines (see our response to Reviewer uiuT for even better super-res results), and our main argument is that training-free inference can be practical (<1min), we did not experiment much with increasing inference time.

We would like to thank the reviewer for taking the time to read our paper and provide valuable feedback. We respond to each of the weaknesses mentioned below:

Quantitative performance is not uniformly state-of-the-art:

In Table 1, we showed that the P2L baseline performs better on x8 super-resolution compared to our method. However, we point out that:

1. P2L requires 30 minutes to run on a single image, severely limiting its usability. Our method only requires 1 minute.
2. P2L uses a VLM (PaLI) to caption the low-resolution image before running the image sampling algorithm. We believe that for super-resolution, the text becomes integral in synthesizing the correct textures in the given image, which we hypothesized was the main advantage of P2L as mentioned in lines 207-208. Since all other previous methods only utilized the unconditional model, we also resorted to unconditional diffusion sampling in the main paper.

To offer a complete picture, we verified the hypothesis that the advantage of P2L comes from using VLM-generated text prompts. We ran our 8x super-resolution algorithm with captions generated from an open-source VLM (Qwen-2.5 [Qwen]). For each low-res image, we first captioned it with the VLM using the instruction "Describe this image" to obtain a text prompt, and then applied our algorithm, conditioning the diffusion model on the VLM-generated prompt.

The results presented below show that we improve upon the super-resolution PSNR and FID metrics of P2L when we use text captions with our proposed algorithm. When the text is correct, it accurately guides the generation of high-resolution details in the image. We observed that most current VLMs (GPT-4o, Qwen-2.5) are fully capable of providing meaningful captions even with low-resolution images.

When it comes to comparing the various training-free sampling algorithms for super-resolution, adding a VLM to the pipeline introduces uncertainty, as the evaluations also need to consider the quality of the VLM and the specific prompt(s) used. Thus, we will include both results (super-res with and without text prompts) in the main paper and Table 1 for completeness.

Artifacts in generated images:

The artifact in Figure 3 row 2, is a watermark hallucinated by the model. This is solely an artifact of the image prior of Stable Diffusion, which was trained on (often watermarked) web images. We decided to use this image for Figure 3 because we found it interesting that our algorithm not only follows the conditions closely but also hallucinates novel high-frequency details under the constraint.

Here, the hallucinated watermark is adhering to the imposed constraint. The mean squared error between the downsampled watermark image and the constraint is around 0.0008, which is also the mean squared error we get when we compare the constraint to another generated lighthouse image that has no hallucinated watermark. In contrast, previous methods often produce blurry results that lack hallucinated high-frequency textures, which the image prior should induce in the generated samples. For instance, all other methods in Figure 3 row 2 produce blurry textures for the rocks.

To avoid confusion, we will replace this image with another sample that has no watermark hallucination, and move this example along with the discussion to Appendix B.7 More Qualitative Results.

[Qwen] Bai, Shuai, et al. "Qwen2. 5-VL technical report." arXiv preprint 2025

We want to thank the reviewer for their comments and address their concerns below:

Sacrificing performance for efficiency:

We would like to point out that there is no significant drop in performance when using our method, while it requires just a fraction of the time to run. In Tables 2 and 3, we show that our algorithm is consistently superior to existing training-free approaches, especially in the more challenging box inpainting task, where it is only outperformed by the fine-tuned SD-Inpaint model, which required an additional 100k iterations of training.

Regarding Table 1, although our algorithm performs worse than the P2L baseline in 8x super-resolution, P2L requires 30 minutes to run on a single image (vs 1min for ours) and also uses the PaLI VLM to caption the low-resolution images before the diffusion inference. As briefly mentioned in lines 207-208, we believe that the main advantage of P2L comes from introducing text conditioning, whereas all previous methods relied on the unconditional model. To verify this assumption, we ran the x8 super-resolution with text prompts generated from the downsampled images using Qwen-2.5 [Qwen] as the VLM. The results we present below show that we can indeed bridge the performance gap and even improve upon the PSNR and FID metrics when introducing text captions to the super-resolution process. We will add this result and relevant discussion to the paper to complement Table 1.

To summarize, our method is both substantially faster and performs as well as or better than the baselines. There is no notable trade-off between efficiency and performance.

Approximate Newton vs gradient descent:

The goal of training-free algorithms is to compute and apply a gradient during the diffusion sampling that 'pushes' the generated image towards some desired condition. The previous methods we compare to in the paper (P2L, LDPS, PSLD, FreeDoM, MPGD) all use gradient descent to compute the update direction. Gradient descent is expressed as $J^Te$, where $J = \nabla f(x)$ is the denoiser Jacobian and $e$ the constraint error.

We propose an alternative way to compute the update direction, based on approximating the steps a Newton method would make. The exact Newton method would use $J^{-1}e$ as the update direction, but this requires computing the inverse of the Jacobian, which is infeasible for these large diffusion models. We show that we can adequately approximate the Newton step with smaller updates in the direction of $Je$. This has been applied before in numerical optimization, as discussed in Inexact Newton Methods [5].

Therefore, the comparison between our approximation to Newton's method and the exact gradient descent is shown in the experiments of the main paper. The previous methods using gradient descent are slower and produce worse results than the proposed approximate Newton method.

Approximate Newton vs exact Newton:

If the reviewer was referring to the accuracy approximating Newton's step $J^{-1}e$ with multiple inexact steps $\lambda Je$, unfortunately, there is no practical way to quantify it, as computing $J^{-1}$ for models at the scale of Stable Diffusion is prohibitive. In Equations (10)-(13), we discussed why our approximation converges to the correct result, but we are unable to directly compare the two update directions for the tasks shown in the paper.

Regardless, we ran a smaller-scale experiment on MNIST where we can easily compute $J$ and $J^{-1}$ and show that both exhibit very similar structures. This aligns with the intuition we provided in lines 133-134 for using the proposed approximation. In the interest of space, we refer the reviewer to our response to Reviewer NFw9 for more details on this additional experiment.

Finite-difference approximation vs exact gradient:

To avoid any confusion regarding the term approximation, it's important to note here that to compute the proposed direction $Je$, we employ finite differences for the Jacobian-vector product $Je \approx (f(x + \delta e) - f(x))/\delta$. This is also an approximation, but for the Jacobian-vector product computation and not the constraint gradient update direction.

The baselines that utilize gradient descent use automatic differentiation/backpropagation through the network. Our update can also be computed using automatic differentiation (forward-mode automatic differentiation). However, forward differentiation is not always implemented or optimized as well as the backward propagation of gradients.

For completeness, we ran an ablation to study the choice of $\delta$ for the finite difference approximation and compare it to the exact gradient computation. For the forward gradient computation, we used the double backward trick since the forward gradient is not implemented for all Stable Diffusion layers in PyTorch. We ran the box inpainting task of Table 2 and found that our method (i) is robust to the choice of $\delta$, only breaking when using very small (0.0005) and very large (0.5) values, and (ii) performs as well as the exact Jacobian-vector product computation while requiring 1/5th of the inference time.

$K$ number of steps, $\lambda$ learning rate, $\delta$ finite difference step size

Thus, using finite differences to compute the Jacobian-vector product is much faster and does not incur a drop in performance.

Using Stable Diffusion 1.4:

Stable Diffusion 1.4 has been widely used to benchmark training-free inference algorithms. Therefore, we also chose this model to showcase the advantages of our proposed method against the existing baselines. The proposed algorithm is not constrained to Stable Diffusion 1.4; we showcase qualitative results on a rectified flow model in the Appendix and provide an implementation for Stable Diffusion 2.1 in the supplementary code -- the results we get with SD-2.1 are similar to SD-1.4, but we did not perform a full-scale evaluation since all the baselines are evaluated only on SD-1.4.

[Qwen] Bai, Shuai, et al. "Qwen2. 5-VL technical report." arXiv preprint 2025