Linearly Constrained Diffusion Implicit Models

We thank the reviewer for their feedback and comments. We appreciate that the reviewer finds that our method produces good results and that the runtime improvements are notable and significant. We have addressed all the questions and hope the reviewer considers the additional evidence towards a score improvement.

We appreciate the two additional papers that are closely related. We have added these to the related works with a discussion. Specifically, the below is a comparison between our method and the two that are mentioned:

Constrained Diffusion with Trust Sampling: This paper follows a similar motivation by noting that a single or fixed number of optimization steps at each denoising step is sub-optimal. They take multiple optimization steps based on a trust region that considers the variance of $x\_t$ under the forward noising process. In contrast, our “trust region” is based on the analytical distribution of $||Ax\_t - y||^2$ under the forward noising process. Furthermore they use a constant step size while we perform a line search for the optimal step size. As a result, their method requires 1000 NFEs (200 DDIM steps) while ours requires ~50 or ~100 NFEs for high quality results.

Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models: COPAINT does Bayesian posterior shaping (jointly optimizing revealed and unrevealed regions with approximated likelihoods) whereas CDIM does projection-based constraint enforcement (optimize a simple quadratic residual after each diffusion step). COPAINT is heavier per step but builds coherence gradually; CDIM is simpler and faster, enforcing constraints deterministically at every iteration.

“This approach doesn't seem to extend to latent diffusion models or general non-linear inverse problems” We note that, like DPS, our method can be extended to non-linear problems but without the theoretical guarantees of exact constraint satisfaction and with using significantly more denoising steps. In this paper, we specifically wanted to focus on exact constraint matching and fast sampling, which are only possible for linear problems. We will add a deeper discussion of nonlinear problems of this in the paper and explain how they can be solved with more steps via a linearized approximation of a non-linear operator $A$, which converges on the constraint more slowly.

“I don't fully understand the paragraph from 145-150. Should the equation on line 147 be $\hat{x}_0$” Yes! Thanks so much for pointing out this typo. The observed residual on line 147 should indeed by $||A\\hat{x}\_0 - y||^2$. Hopefully this clears up any confusion.

“Why is this the best way to define the plausible region” We draw from several works (cited on line 149) showing that denoising sampling should mirror forward dynamics as closely as possible. Because the forward noising model adds zero mean Gaussian noise, the conditional mean $\\mathbb{E}\_{\varepsilon_t}\\left[ ||A \\hat{x}\_0 - y||^{2} \\mid y \\right]$ is exactly the most probable (maximum-likelihood) value that this residual can take. Defining the ``plausible'' band as a narrow neighborhood around that mean therefore keeps our reconstructions statistically typical, anything much smaller or larger would be exponentially unlikely under the model and would signal over or under projection.

“The notation for $\\hat{x}\_0$ seems inconsistent” We appreciate the suggestion on the notation of $E[x\_0 | x\_t]$, which has been mentioned by other reviewers. We have changed the notation $\\hat{x}\_0$ to $\\hat{x}\_0(x\_t)$ throughout the paper to clarify this point

We thank the reviewer for their positive comments and helpful suggestions. We are glad that the reviewer appreciated the novelty of the method and our comprehensive experiments, and also gave us a good score for all areas: quality, clarity, significance, and originality.

We appreciate the additional comparisons suggested. DAPs is a very recent state-of-the-art method that we did not initially compare against, so we report comparison numbers below. These will also be present in the paper revision both in the table and in figure 1.

DAPs exhibits strong performance, and the results in the main table of DAPs are for DAPs-1k (1000 NFEs). To facilitate a fair comparison, we used the official code and ran the configuration for DAPs-50 (50 NFEs) which has a comparable runtime as CDIM with T’ = 25 denoising steps. Below are the results on 100 images from the FFHQ test set reporting LPIPS (net=”vgg”). Lower is better.

Random inpainting:
DAPs-50 : 0.306 LPIPS
CDIM (T’=25): 0.271 LPIPS

Gaussian blur:
DAPs-50: 0.339 LPIPS
CDIM (T’=25): 0.283 LPIPS

4x Super Resolution:
DAPs-50: 0.358
CDIM (T’=25): 0.290

As you can see, CDIM outperforms DAPs when both are restricted to the same fast sampling schedule (even though DAPs-1k exhibits stronger performance with 10x the runtime as CDIM).

We also have numerous qualitative comparisons that we will include in the revision and would like to share with the reviewer when possible. The DAPs-50 images are slightly noisy due to the accelerated schedule, and the quality improvement is visibly apparent when using CDIM.

We can therefore see that when restricted to a fast sampling environment (50 NFEs or 4 seconds), our method outperforms DAPS, a very recent and state-of-the-art method.

We also thank the reviewer for the other references. We will make sure to include them all in the related work.

Regarding non-linear problems, we can solve them by approximating the stopping criteria $\mu_t(y)$ and $\sigma_t(y)$ via a linearized approximation of the non-linear operator A. Although this method works to comparable quality as baselines when using a large number of denoising steps, we lose both the theoretical guarantees of constraint convergence and the high-quality results during accelerated schedules. This is why we didn't want to focus on nonlinear problems, as we wanted to focus on speed and constraint convergence. Nevertheless we will add this discussion into the paper to show how non-linear problems could be tackled with more denoising steps. (The need for more denoising steps when using nonlinear operators has been discussed in prior works, with DAPs using 4000 NFEs for their latent/non-linear results and only 1000 for linear results.)

We thank the reviewer for their comments. We are slightly confused by the reviewer’s summary of the paper, as they mention how our method uses KL divergence minimization, something that is not mentioned once in the paper. KL divergence and the L2 residual minimization that the reviewer mentions are from a non-anonymized arxiv preprint of the paper from several months ago. The version submitted for review has a substantially different method that does not incorporate KL divergence minimization or discuss L2 residual minimization. We would like to make sure that the reviewer has engaged with the method submitted for review, given that the primary weakness stated is the claim that our method is not novel.

“The primary reason for the speedup is the use of DDIM instead of DDPM”

We have shown experimental results that simply using DPS with DDIM does not yield good results (see Figure 5). Furthermore, methods like DPMC use DDIM and still require over 200 NFEs to get good results. The speedup comes from our conditional step (DDIM is only for the unconditional steps), which enables fast convergence to the constraint in fewer overall NFEs than previous methods.

“Why are variational methods, such as RedDiff, and recent flow-based baselines omitted from the comparisons?” Based on feedback from reviewers we have added a comparison with DAPs, a recent state of the art work from CVPR 2025 that outperforms RedDiff.

To facilitate a fair comparison, we used the official code and ran the configuration for DAPs-50 (50 NFEs) which has a comparable runtime as CDIM with T’ = 25 denoising steps. Below are the results on 100 images from the FFHQ test set reporting LPIPS (net=”vgg”). Lower is better.

Random inpainting:
DAPs-50 : 0.306 LPIPS
CDIM (T’=25): 0.271 LPIPS

Gaussian blur:
DAPs-50: 0.339 LPIPS
CDIM (T’=25): 0.283 LPIPS

4x Super Resolution:
DAPs-50: 0.358
CDIM (T’=25): 0.290

As you can see, CDIM outperforms DAPs when both are restricted to the same fast sampling schedule (even though DAPs-1k exhibits stronger performance with 10x the runtime as CDIM).

We also have numerous qualitative comparisons that we will include in the revision and would like to share with the reviewer. The DAPs-50 images are slightly noisy due to the accelerated schedule, and the quality difference is visibly apparent.

We can therefore see that when restricted to a fast sampling environment (50 NFEs or 4 seconds), our method outperforms DAPS, a very recent and state-of-the-art method.

“How does the method perform on more challenging settings like 8x SR?”

Below we include PSNR and LPIPS on 8x superresolution with $sigma_y = 0.05$ Gaussian measurement noise with T’=50 steps (approx 10 second inference) .Given that we cannot share images with the reviewer, we cannot send the qualitative 8x super resolution results, but will include some examples in the revised paper. Regarding more challenging tasks, we also show several qualitative examples in the paper of 50% inpainting, a significantly harder task than the 25% box inpainting task used in the experiment section.

8x superres results
FFHQ
LPIPS: 0.353
PSNR: 23.63

ImageNet
LPIPS: 0.457
PSNR: 19.67

“Can the approach be extended to latent diffusion models or nonlinear measurement settings, or is it inherently limited to pixel-space and linear operators?”

Just like DPS, the method can work on latent and non-linear operators, but the theoretical guarantees fail since we rely on linearity of expectation of the Tweedie’s estimate: A(E(x)) = E(A(x)). Therefore the quality improvements in highly accelerated sampling schedules is not as evident for non-linear operators. We have run phase retrieval and non-linear deblurring and plan to include those results in the revision.

“The paper does not discuss any limitations despite the fact that the method cannot be applied to nonlinear forward operators”

Note that in the Conclusion we discuss how we do not handle non-linear constraints because of the linearity of expectation.

I thank the reviewers for addressing my concerns and will raise my voting to 4.

Confusion of summary: Thank you for pointing this out. The confusion likely arose due to substantial conceptual overlap with the earlier version. I want to assure the authors that I have reviewed the current version of the submission.

We thank the reviewer for the detailed and helpful comments. We’re glad they found the results strong and the contribution relevant, and appreciate the careful review of the proofs. Below we address all comments and will fix the issues noted. We hope the author will consider the below for a score improvement.

We would like to start by addressing the main concern regarding the theoretical justification for constraint satisfaction. The reviewer asks about the proof in Appendix C. We note that this proof specifically describes the rate of convergence at which $\lVert A\\hat{x}\_0(x\_{t-1}) - y\rVert^2 \\rightarrow 0$ (which we also demonstrate empirically). However the fact that the constraint can be satisfied follows from the following, much simpler argument: As $t \\rightarrow 0$ the Tweedie's estimate becomes the identity function $\\hat{x}\_0(x_t) \\rightarrow x_t$ and therefore the optimization problem $min\_{x\_{t-1}} ||A\\hat{x}\_0(x\_{t-1}) - y||^2$ becomes the concave quadratic $min\_{x\_{t-1}} ||Ax\_{t-1} - y||^2$ . Specifically, at $t-1 = 0$, we can write this out as $x\_0 \gets min\_{x\_0} ||Ax\_0 - y||^2$ and this quadratic can be optimized to arbitrary precision by following the gradients $\\nabla x\_{t-1}$ of our current sample at $t-1 = 0$. We are performing exactly this minimization at $t-1 = 0$ because our stopping criteria is $||Ax\_{t-1} - y||^2 \\leq \\mu_{t-1}(y) \\pm \\sigma\_{t-1}$ where $\\mu\_{t-1}(y) \\rightarrow 0$, $\\sigma\_{t-1} \\rightarrow 0$ as $t-1 \\rightarrow 0$. By using the optimal step size $\\eta$ via line search, the constraint is met to arbitrary precision (assuming that $y$ is in the range of $A$). This is in contrast to methods like DPS that make a single update to the constraint for each denoising update, and use heuristic step sizes that may not decrease the constraint error to zero. Later in this rebuttal we show quantitatively that the mean absolute pixel error for an observed region in inpainting goes to 0 for our method while it does not for DPS.

“The paper seems to pursue two distinct goals: (1) enforcing hard constraints and (2) fast diffusion-based inverse problem solving”

We thank the reviewer for this comment and plan to strengthen our discussion of the connection between these two goals in the revision. These goals are closely related: accelerated unconditional sampling methods like DDIM struggle with constrained sampling because the constraint enforcement steps typically do not converge as fast as the unconditional steps. By proposing a new principled method that quickly decreases the constraints (enforcing the hard constraints even in accelerated sampling), we are able to produce high quality results for fast diffusion-based inverse problem solving. That fact that we guarantee hard constraints is a result of the proposed step size mechanism which is designed for fast posterior sampling by finding the optimal number of steps and step size at each iteration.

“Do you have an explanation for why DPS performs poorly when combined with DDIM? Apart from the use of DDIM and the adaptive step size, is there any fundamental difference between your method and DPS?”

DPS takes one constraint step per denoising step with a heuristic step size (footnote 5, p.6 of DPS) and doesn't take into account the additive noise distribution. In contrast, our method proposes a theoretically grounded method for taking optimal steps on the constraint, showing it guarantees constraint recovery and that the optimal step size can be analytically computed without a network pass, using only A’s functional form.

“DPS doesn’t match the constraint” is not immediately obvious. You might consider reporting the numerical error on the constraint”

We thank the reviewer for this suggestion. Based on your suggestion we have run the following experiment to report in this rebuttal and for the paper revision. We performed box inpainting with no measurement noise, meaning we should exactly match the constraint, $Ax = y$ on the observed region. We ran both CDIM and DPS with DDIM, each with 50 total NFEs (25 denoising steps). Below we report the per-pixel mean absoluate error $\\frac{1}{m}\\Sigma|Ax\_t - y|$ at the end of select timesteps: This is the measure of how far off the average pixel deviates from ground truth in the known region. (Pixel values between -1 and 1).

T = 960
CDIM: 0.72
DPS: 0.66

T = 480
CDIM: 0.61
DPS: 0.62

T = 40
CDIM: 0.0058
DPS: 0.072

T = 0
CDIM: 0.00045 (In theory would be 0 but we have $\\overline{\\alpha}\_0 = 0.9999$ not $1$ for numerical stability)
DPS: 0.069

CDIM matches the constraint to numerical precision, while DPS differs from observed pixels by an average of almost 7%. Any attempt to make DPS more closely match the constraint, for example increasing the gradient step size, results in divergence for accelerated sampling schedules.

“It would also be interesting to highlight your method on box inpainting task”

We include several examples in the paper of a 50% inpainting task (harder than box inpainting) where we exactly match the constraint. As an example, consider Figure 3 where sub-panel b where we can force the observed region to match the noisy pixels exactly even when they are out of distribution.

“a. Once the constrained optimization problem is replaced with a penalized one, there is little to no guarantee that the constraints will be exactly satisfied.”

We note that as $t \\rightarrow 0$, the penalized optimization turns into the hard constraint problem given by equation (10). This is because our stopping criteria $\\sigma$ goes to 0 as $t \\rightarrow 0$, so there is no early stopping. We continue running optimization steps on $||A\\hat{x}\_0(x\_{t-1}) - y||^2$ until we reach the desired level of convergence. Because this is concave quadratic, it can always be optimized to sufficiently meet the hard constraint. Therefore we have guarantee of exact constraint satisfaction at $t=0$ (and only at $t=0$ because higher noise levels do have the early stopping criteria).

“b. The justification for the penalized formulation (footline on p.4) is questionable. If I understand correctly, the constraint $A\\hat{x}\_0(x\_{t-1})$ should hold at t = T. So the argument seems invalid as stated.”
We thank the reviewer for pointing out this subtle fact. It is true that the constraint in question is $\\hat{x}\_0(x\_{t-1})$, so even at the first optimization step, we are computing the Tweedie’s estimate $E[x\_0 | x\_{T-1}]$ and not $E[x\_0 | x\_{T}]$. However, the intuition holds that even at $t=T-1$, $x\_t$ is largely uninformative, so finding $x\_t$ where $A\\hat{x}\_0(x\_{t-1})=y$ is effectively impossible. The tweedie’s estimate is the expectation over all denoising trajectories, which would still be a blurry image at high noise values, even if say we set $x\_{T-1}$ to our GT image.

“c. Referring to Eq. (11) as a Lagrangian is incorrect.”

We thank the reviewer for pointing out this correction. We will change the description of the equation.

“d. Proof in Appendix C”
We thank the reviewer for the careful reading and agree that Appendix C requires clearer exposition.

Boundedness of $x_t$
Since the forward process is a convex combination of a bounded datum $x_0$ where $||x\_0|| \\le R$ and Gaussian noise with coefficient $\\sqrt{\\beta_t}\\le 1$, the resulting sample is almost surely bounded: $||x\_t|| \\le R+O(\\sqrt{\\beta_t})$. We will add this explicit bound.

Role of $x\_t$ versus $\\hat x\_0(x\_t)$
Throughout Appendix C, $x\_t$ is the optimization variable selected by the projection step, while $\\hat x\_0(x\_t)$ is its Tweedie estimate. Equation 37 is saying that the Tweedie’s estimate of our current optimization iterate converges to the identity function. We will make this distinction explicit to avoid confusion. We don't intend to say that $||A\\hat{x}\_0 - x\_t|| \\rightarrow 0$ but that $||\\hat{x}\_0(x\_t) - x\_t|| \\rightarrow 0$, meaning that optimizing the tweedie estimate of the iterate is the same as optimizing the iterate itself at low noise values. We also show this empirically.

Convexity of the objective as $t\\to 0$
Tweedie convergence gives $||\\hat x\_0(x\_t) - x\_t|| = O(\\beta\_t)$; expanding Eq.~(37) shows that $||A\\hat x\_0(x\_t) - y||^2 = ||A x\_t - y||^2 + O(\\beta\_t)$. Hence, for sufficiently small $t$, the landscape is an arbitrarily small perturbation of the convex quadratic $||A x_t - y||^2$, so standard gradient descent reaches any desired accuracy. These clarifications will be incorporated, but they leave the main conclusion unchanged: as $t-1\to 0$, gradient descent on $||A\hat x_0(x_{t-1}) - y||^2$ can satisfy the constraint to arbitrary precision.

“It would be useful to include a numerical comparison with Diff-PIR [1]”
We thank the reviewer for pointing out Diff-PIR. Below are some direct comparisons reporting LPIPS (lower is better), which are also added to the paper

4x Super Resolution Imagenet:
Diff-Pir (100 NFEs): 0.371 LPIPS
Ours (100 NFEs): 0.339 LPIPS

Gaussian Deblur Imagenet:
Diff-Pir (100 NFEs): 0.355 LPIPS
Ours (100 NFEs): 0.347 LPIPS

With the same number of NFEs, our method performs better.

“I couldn't fully understand Proposition 1...relies on notation introduced later,”

Thank you for pointing this out, indeed the notation was introduced after the proposition making it hard to follow. The key idea of Proposition 1 is that we can choose our optimal step size based on the value of $||Ax\_t - y||$ (calculable via line search without a network pass) and that this also guarantees that our true target, $||A\\hat{x}\_0 - y||$ decreases towards its desired value by a commensurate amount.

“To improve readability, I suggest consistently using the notation $\\hat{x}\_0(x\_t)$”
We thank the reviewer for this suggestion. It has also been suggested by other reviewers and we will change the notation in the revised paper.

I thank the authors for addressing my questions.

• I appreciate the 2 news experiments, on the constraint satisfaction and comparison to DiffPir. Thanks for that.

• Thanks for pointing to Figure 3, I had missed that one, apologies.

• I appreciate the efforts put into trying to clarify the mathematical arguments in the proof that the constraint is satisfied at time $t = 0$, but I have to say I am still not convinced. The arguments are not rigorous; in particular, saying that it becomes a concave quadratic (did you mean convex?) as $t \to 0$ is incorrect. First of all, I find the current notation $x_t \to \lVert A\hat{x}_0(x_t) - y \rVert^2$ not very easy to manipulate from an optimization perspective. Therefore, I will rather denote $f_t(x) = \lVert A \hat{x}_0^t(x) - y \rVert^2$, where I included a dependency of $\hat{x}_0$ on $t$, and changed the name of the global variable on which you are optimizing to $x \in \mathbb{R}^d$. Then $f_t$ converges pointwise to $f(x) = \lVert Ax - y \rVert^2$, but this does not imply that $f_t$ is convex for some $t$ close to 0. I kindly ask the authors not to claim they provide a mathematical proof; to me, this is a heuristic (and heuristics are fine), and it should be clearly stated as such.

We would like to thank the reviewer for their positive feedback and thoughtful comments. We are glad that they appreciate the novelty and strength of the results. Below we address the specific concerns/questions raised. We plan to upload a revision with all these specific fixes.

“In a couple of cases, CDIM shows performance that is close, but not exceeding the best slower baseline.”

We acknowledge that this is true, however the baselines mentioned use far more compute than us. When restricting those methods to a same amount of compute (such as the DPS + DDIM image in figure 5), our method performs significantly better.

“Would it be possible to extend this method for latent-based models”

We note that, like DPS, our method can be extended to latent-based models with some amount of success. However, because Tweedie’s formula for the expected posterior mean does not pass through a non-linear function, $A(E[\hat{x}_0]) \neq E[A(\hat{x}_0)]$, we cannot get the same theoretical guarantees on convergence of the constraint, or the same speedup. In addition, other heuristics may be required to get satisfactory convergence in latent models, which is why we have left this for a follow up work.

“How sensitive are final results to the choice of $\pm \sigma$ as the stopping criterion? What if you choose a dynamic tolerance (e.g. varying with t) improve performance?”

In Figure 4 we have shown qualitative results when different stopping criteria than $\pm 1 \sigma$ are used. Additionally, in Table 2 we have given quantitative results both on the resulting runtime and the generation quality for a chosen task when using these different stopping criteria. Although it would be possible to choose a dynamic tolerance, we note that the standard deviation $\sigma$ is itself highly informative on the plausible region, decreasing dramatically as t->0, so a constant factor is sufficient.

“Section D of appendix looks incomplete.. typos and inconsistencies”

We thank the reviewer for pointing this out. We will address the typos and inconsistency in the revised version. Indeed subsection D.4 should be referring to table 3 and 4, and we will revise the inconsistencies in experimental notation.