Step-wise Triple-Consistent Diffusion Sampling for Inverse Problems

[C3] (i) Comparison Results. (ii) Asymptotic exact posterior sampling in [2,3,4,7]. (iii) Backward consistency in DPS. (iv) Choosing DCDP and DAPS as baselines.

We thank the reviewer for this comment.

(i) Please see our global response. For the suggested latent solvers, please see our response to Comment 3 of Reviewer iQjf.

(ii) We agree with the reviewer that [2,3,7] provide asymptotic exact posterior sampling. However, this exact posterior sampling is achieved based on assumptions that may not be fully realistic/possible, and consider other sampling techniques to estimate the intractable term. For example, FPS [2] is built based on the assumption that the pre-trained score model is a perfect estimator of the log likelihood (see Assumption 4.1 in [2]). In MCGdiff [3], the asymptotic posterior is obtained through Sequential Monte Carlo (SMC) sampling which are also estimators used when the posterior is infeasible due to complexity. Furthermore, the methods in [2,3] are only applicable to linear imaging IPs. Similar to MCGdiff, the authors in [7] introduced Twisted Diffusion Sampling (TDS) which is also based on MCS sampling. TDS is a general conditional diffusion sampler where the authors considered two problems: Image in-painting on MNIST (not a high dimensional dataset), and motif-scaffolding (a task in protein design) which is not an imaging inverse problem. In PnP-DM [4], the authors provide insights on the non-asymptotic behavior of the proposed algorithm for the which the method uses Split Gibbs Sampling. Therefore, we believe that trying to approximate exact posterior sampling under some assumption and/or using other estimators may not be sufficient or the best way to obtain improved reconstructions as supported by the results.

(iii) Please see our response to the previous comment for the backward consistency in DPS.

(iv) We agree with the reviewer that two key baselines in our paper, DCDP and DAPS, are unpublished yet. However, since we ran their codes and verified their results—and given the rapid pace of this field—we believe they remain valid baselines.

Generally, to select baselines, we focused on methods addressing various linear and non-linear IPs, using diverse datasets and pre-trained models, accounting for measurement noise, and demonstrating superior reconstruction and/or run-time performance.

Notably, DAPS achieves SOTA PSNR results, while DCDP offers competitive PSNRs with reduced run-time. Additionally, we compare to DMPlug (NeurIPS24) in the global response which also achieved very competitive results.

[C4/Q1] Sensitivity to choice of $\delta$ and how it is determined in practice.

Thank you for comment/Question. In SITCOM, we proposed to use a stopping criterion to prevent the noise overfitting, determined by $\sigma_y$. In Appendix F.4, we show the impact of using the stopping criterion vs. not using it. In this experiment, as suggested by the reviewers, we run SITCOM using different values of $\delta$ and report the average PSNR values and run-time (in seconds).

We use $N=K=20$ and $\sigma_y = 0.05$ and set $\delta$ to values above and below $\sigma_y$. Results for BIP (linear IP) and NDB(non-linear IP) are given in the following table where we use 20 FFHQ test images with $m=256\times 256 \times 3$. The first row shows the values of $a$ for which $\delta = a\sqrt{m}$.

As observed, the PSNR values vary between 29.02 to 30.37 (resp. 29.80 to 30.20) only for BIP (resp. NDB) with stopping criterion between 0.025 to 0.055 which indicates that even if values lower (or slightly higher) than the measurement noise level are selected, SITCOM can still perform reasonably well.

This means that if we use classical methos to approximate/estimate the noise, we can achieve good results. In practice, we can use classical methods (such as Liu et al., 2006 and Chen et al., 2015) to estimate the noise present in the measurements which we discuss in the Limitations and Future Work (Appendix B).

We note that other works, such as DAPS and PGDM [B], also use $\sigma_{y}$ but not to estimate the stopping criterion. In these papers, $\sigma_{y}$ is encoded in the updates of their algorithms (see Eq. (7) in PGDM and Eq. (9) in DAPS).

[B] Pseudoinverse-guided diffusion models for inverse problems. ICLR, 2023.

We thank the reviewer for their comments.

[C1] Guarantees for the output of SITCOM. Assurances on correct sampling from the posterior distribution like in [2,3,4,7].

We agree that having a theoretical guarantee is indeed preferable. Previous studies [2,3,4,7] have provided such guarantees for sampling from the posterior distribution, but their performance is not optimal. In contrast, other methods, such as ReSample, DAPS, and DCDP, which leverage step-wise data consistency, achieve better results. However, establishing a performance guarantee for these methods is challenging. There seems to be some trade-off. Our approach falls into the latter category with the major distinction of incorporating the step-wise backward consistency through the proposed network regularization. To address this limitation, we conducted extensive numerical experiments and ablation studies to evaluate our method. Please see our global response.

[C2] Resemblance to DPS and ReSample and scope of technical novelty. [Continued]

We thank the reviewer for their comment. Many DM-based IP samplers/solvers share similarities, as summarized in the following table. Previous works do indeed contain data and forward consistencies. However, we argue that these alone are insufficient to achieve an improved performance. We believe that incorporating backward consistency can further enhance performance (as supported quantitatively and computationally).

The technical novelty of SITCOM can be highlighted as:

• Formulating the three consistencies: We present a systematic formulation of the conditions where the goal is to improve the accuracy of each intermediate reconstruction, thereby reducing the number of sampling steps required.
• Introducing the step-wise Backward consistency: We introduce step-wise network regularization for the backward consistency by leveraging the implicit bias of the network at each iteration. In this rebuttal, we present additional numerical results demonstrating the benefits of enforcing the proposed backward consistency (see the third experiment of the second part of the global response).
• Integrating all consistencies: The formulation, initialization, and regularization of the optimization problem in Eq.(S1) along with the preceding two steps ensure that all consistencies are enforced without violation. Note that the initialization (and regularization) in Eq.(S1) is partially used to ensure closeness such that the forward consistency is satisfied.

It is worth noting that while some previous works, such as RED-diff and DMPlug, also utilize the implicit bias of the network, they adopt the full diffusion process as a regularizer, applied only once. In contrast, our method uses the neural network as the regularizer at each iteration and focuses specifically on reducing the number of sampling steps for a given level of accuracy. We compared our approach with these methods to highlight this advantage (see the first part of the global response). Please refer to the highlighted changes on pages 5 and 6 of the revised manuscript.

We thank the reviewer for their comments.

[C1/C2] The impact of each individual component in SITCOM.

Thank you for your suggestion. Please see the first part of the global response.

[C3] The possibility of adding SITCOM components to DPS or DDNM.

We would like to thank the reviewer for their question. In the first study of our global response, we show the impact of the iterative application of measurement consistency. In this experiment, we compared SITCOM (i.e., $K>1$) with $K=1$ which is DPS plus the resampling formula (for forward consistency). Here, $K$ is the number of gradient updates per sampling step.

The results indicate that DPS, as is already established, requires a relatively very large $N$ to obtain descent results. We note that including the resampling in DPS's formula violate their theoretical results.

For the case of DDNM, which is only applicable to linear IPs, the algorithm initially estimate an image at every reverse step using Tweedie's formula without measurement consistency. Subsequently, DDNM enforces data consistency by refining this estimate using the pseudoinverse of the forward operator. In order to remap the image back to $t-1$, the authors use the standard sampling (Eq.(14) of their paper and Eq.(7) in our paper) instead of the resampling formula for partially applying the forward consistency (Eq.(8) or Eq.(S3) of our paper).

While we conjecture that using the resampling for remapping would improve the performance in DDNM, we don't believe that DDNM can apply our proposed backward consistency (with the step-wise network regularization) as they apply measurement consistency differently.

Thank you for addressing the reviewers’ concerns. While the authors have successfully addressed my specific concern about the work, the discussion with another reviewer remains unresolved. Therefore, I maintain my previous score.

[C2] (i) Technical Differences between DCDP and SITCOM. (ii) Comparing SITCOM and DCDP with the same $N$ and $K$. We would like to thank the reviewer for their comment/suggestion.

(i) From a higher level perspective, the motivation of SITCOM is different from DCDP. In DCDP, the DM is used as a diffusion purifier [A], where the forward and reverse processes are used at inference. As the DM is trained on clean images, the intuition in DCDP is that when the reverse sampling steps (without measurement consistency) are applied on a noisy version of some input image, the DM-purified one will be closer to the learned/clean image distribution (see Theorem 3.1 in [A]). Then, the measurement consistency is applied like in Eq.(5) on the DM-purified version (denoted as $v_k$ in Algorithm 1 in DCDP). In SITCOM, the motivation is to find an efficient modification of the reverse sampling steps such that we can achieve accelerated reconstructions by proposing a new method for enforcing the backward consistency through the proposed network regularization.

The formulation/encoding of the DM network is also significantly different as the output of the DM in DCDP is only used as initialization for their optimization problem (the data fidelity function in the DCDP algorithm). In SITCOM, the optimization variable is the input of the DM network that is formulated to enforce measurement- and backward-consistency (through the proposed step-wise network regularization).

In SITCOM, the gradients need to be computed w.r.t. the input of the network, not parameters. The reviewer is correct, the computation of these gradients requires a backward pass through the network parameters which is not needed in DCDP. However, the proposed backward consistency allowed us to reduce the sampling and optimization steps and still achieve leading or competitive PSNRs as we discuss next.

(ii) The authors in DCDP propose two versions of DCDP in the pixel space. Version 1 simply uses the SDE steps to apply purification (using a linear decaying schedule), whereas Version 2 uses Tweedies formula to approximate this step (the $`DPUR`$ function in DCDP). In our experiments, we only consider version 1 based on the authors' observation that version 2 results in lower perceptual quality because of the "regression to the mean" effects. To handle measurement noise, DCDP modifies the purification schedule, $T_k$ (per sampling step $k$ ), such that the decay does not go to $T_k=0$, whereas in SITCOM, we use a stopping criterion.

Following the reviewer's suggestion, we compare SITCOM with DCDP using the same $N$ and $K$, denoted as $K$ and $\tau$ in DCDP. We set $\sigma_y = 0$ because SITCOM and DCDP handles measurement noise differently.

NDB:

For non-linear deblurring, DCDP requires a large number of $N$ and $K$ to achieve comparable PSNRs (the last row) which results in extended run-times.

GDB:

The reviewer is correct. SITCOM is slower than DCDP when the number of iterations is fixed due to the need to backpropagate. However, SITCOM allows to reduce $N$ and $K$ while achieving competitive PSNRs. We can see from the table that the PSNR of SITCOM with $N=10$ matches DCDP with 20, therefore we claimed that SITCOM reduced the total run time.

For any fixed $N$ and $K$, on both tables, SITCOM always returns higher PSNR.

[A] Diffusion Models for Adversarial Purification. ICML22.

[C1] (i) How the proposed consistencies relate to other works. (ii) Validity of the comparisons. (iii) Suggested study to show the impact of each consistency in SITCOM.

We thank the reviewer for their comment.

(i) How the proposed consistencies relate to other works: We agree with the reviewer that measurement consistency, in some form, is necessary for (and used in) all IP solvers. The forward consistency, as described/stated in Section 3.1 (Issue 2 and the paragraph before Eq.(8)), is adopted from (Lugmayr et al., 2022) based on their work's observation/intuition.

The technical novelty of SITCOM can be highlighted as:

• Formulating the three consistencies: We present a systematic formulation of the conditions where the goal is to improve the accuracy of each intermediate reconstruction, thereby reducing the number of sampling steps required.
• Introducing the step-wise Backward consistency: We introduce step-wise network regularization for the backward consistency by leveraging the implicit bias of the network at each iteration. In this rebuttal, we present new numerical results demonstrating the benefits of enforcing the proposed backward consistency (see the third experiment of the second part of the global response).
• Integrating all consistencies: The formulation, initialization, and regularization of the optimization problem in Eq.(S1) along with the preceding two steps ensure that all consistencies are enforced without violation. Note that the initialization (and regularization) in Eq.(S1) is partially used to ensure closeness such that the forward consistency is satisfied.

It is worth noting that while some previous works, such as RED-diff and DMPlug, also utilize the implicit bias of the network, they adopt the full diffusion process as a regularizer, applied only once. In contrast, our method uses the neural network as the regularizer at each iteration and focuses specifically on reducing the number of sampling steps for a given level of accuracy. We compared our approach with these methods to highlight this advantage (see the first part of the global response). We have also further clarified this novel aspect. Please refer to the highlighted changes on pages 5 and 6 of the revised manuscript.

In addition to the empirical comparisons with DPS [1], we refer the reviewer to the first study in our global response to highlight the impact of the iterative application of measurement consistency.

In what follows, we discuss the differences between SITCOM and two other papers.

For PGDM [2], the authors assumes that $p(x_0|x_t)$ follows a Gaussian distribution which is then used to approximate the intractable term $\nabla_x \log p_t(y|x_t)$. This approximation is used to construct a guidance vector (with the measurement and forward operator) used in the remapping. To this end, there are several differences between SITCOM and PGDM.

• We use forward consistency (i.e., Eq.(S3)) for the remapping whereas PGDM uses a DDIM step to obtain $x_{t-1}$ (i.e., Eq.(7) with guidance in their paper). We refer the reviewer to the second study in the global response to show the impact of the resampling formula in SITCOM.
• Measurement consistency in SITCOM is applied through an optimization problem that also integrates backward consistency given Definition 1, whereas PGDM integrates the measurements vector into the guidance vector.
• In SITCOM, the input to the Tweedie's formula is our optimization variable in (i.e., Eq.(S3)) whereas in PGDM, Tweedie's is used to obtain an estimate (without measurement consistency) that will later be used to construct the guidance vector for the remapping. See Algorithm 1 in Appendix A of PGDM. F

or LGD [3], the authors used the same assumption in PGDM with the difference of using Monte Carlo sampling. The authors proposed to use the average of multiple samples to approximate the intractable term. In general, all these methods do not use a step-wise network regularization for the backward consistency.

(ii) Validity of the comparisons: In terms of comparisons, we are using the same tasks, test images, pre-trained models, and measurement noise levels. If the reviewer is referring to the hyper-parameters in the baselines, we would like to clarify that we are using the configuration set by the authors in their codes and recommended in their papers. Additionally, we only use tasks evaluated by the authors of the papers. For example, we don't include phase retrieval results for DMPlug as this task was not considered. It would be great if the reviewer could elaborate more on this point.

(iii) Suggested study to show the impact of each consistency in SITCOM.: We thank the reviewer for the suggested experiment about highlighting the impact of every step in SITCOM. Please see the second part our global response.

[C3] Minor: Comparison with [5,6,8].

Thank you for your comment. For DMPlug [5] and FPS [8], we include comparison results in the global response. For DPnP [6], the results are very similar to DPS and due to the time limit of the rebuttal, we opted not to compare with them for now. However, we promise to include them in a subsequent version of the paper.

[C4] Code link.

We would like to thank the reviewer for pointing this out. Once the reviews were released, we immediately fixed this issue.

[C5] Minor: Formulation of SITCOM in the latent space and comparison results.

We thank the reviewer for their comment. Latent-DMs were originally introduced to speed up the sampling procedure as the forward and reverse processes are applied in a lower dimensional space. This was the motivation of methods like ReSample and PSLD. However, in lower dimensional spaces, it is required to have two pre-trained models: The pre-trained DM and the pre-trained encoder and decoder, used to convert between the pixel and latent spaces. For solving IPs with latent DMs, a backward pass through the decoder (and possible forward passes through the encoder as is done in Latent-DCDP) is needed, which is an additional computational cost that may not result in a much reduced computational time (see the run-time results in Table 10 of ReSample). This is demonstrated in ReSample where the measurement consistency optimization problem was only applied for sampling time indices in set $\mathcal{C}$ of their algorithm. This is the main reason we opted not to consider SITCOM with latent space DMs.

While the formulation for extending SITCOM to the latent space may seem straightforward (as we can simply apply the decoder to the argument of $\mathcal{A}(\cdot)$ in Eq.(S1)), we still need to verify it and check whether other techniques (such as the ReEncode function in latent-DCDP) are needed.

We promise to further investigate this either in a subsequent version of the paper or for future work.

To fully address the reviewer's comment, in the following table, we include results of pixel-space SITCOM and latent space methods.

Dataset: FFHQ. Metric: PSNR. Noise level: 0.05. Source: Table 1 and Table 2 of DAPS.

It is important to note that latent-DM IP methods are not fully comparable to pixel space SITCOM as different pre-trained DMs were used. However, similar to ReSample (where they compared their latent-DM IP solver with pixel-space ones), we included the above results.

[C4/Q4] Deterministic Samplers for SITCOM such as DDIM and EDM.

This is a great question. We thank the reviewer for their suggestion. In SITCOM, we don't use the standard DDIM or DDPM sampling; instead, we only use the stochastic resample formula in Eq.(S3) to obtain $x_{t-1}$. This is a stochastical sampler with randomness coming from the $\eta_t$ in Eq.(S3). We are not aware of a deterministic version of Eq.(S3).

However, your question has prompted us to recognize that the stochastic resampling formula can indeed be adapted into a deterministic form, akin to the modification from DDPM to DDIM. We conducted an experiment where we compare SITCOM to the deterministic version of SITCOM for which we set $\eta_i$ deterministically via $\eta_i = x_N$ for all $i\in \\{N,\dots,1\\}$, where $x_N \sim \mathcal{N}(0,I)$ is the initial input of the diffusion model. The following table includes the results of MDB with fixing $K=20$ (number of gradient updates per sampling step).

We observe that at larger values of $N$, the stochasticity in Eq.($`S`_3$) returns better results. Interestingly, with $N\in \\{2,3,4,5\\}$, deterministic SITCOM returns slightly improved results. We will for sure include this discussion in a subsequent version of the paper.

In the case of EDM, we think it is possible to extend SITCOM to EDM which would involve replacing the Tweedie’s formula in SITCOM with the second-order ODE solver from EDM. This requires investigating the best and most efficient way of incorporating the proposed backward consistency within steps of EDM.

We thank the reviewer for this suggestion and promise to consider this direction for future investigation of our work.

[Q5] How does SITCOM handle noise characteristics that deviate significantly from those tested in the paper?

We thank the reviewer for this question. Here, we evaluate SITCOM with additive measurement noise vector sampled from the Poisson distribution. We use $N=K=20$ and $\lambda_{y} = 5$ (which determines the noise level as in DPS). We set $\delta$ (the stopping criterion) to different values (top row).

Results for SR (linear IP) and NDB(non-linear IP) are given in the following table where we use 20 FFHQ test images with $m=256\times 256 \times 3$. The first row shows the values of $a$ for which $\delta = a\sqrt{m}$.

As observed, the results of setting $\delta$ to values between $0.035\sqrt{m}$ to $0.08\sqrt{m}$ return PSNR values within approximately 1 dB. This indicates that SITCOM can perform reasonably well on noise sources other than Gaussian with different values of the stopping criterion (even if it is set for the additive Gaussian measurement noise case). We will consider more noise settings in a subsequent version of the paper.

[C2/Q2/Q6/Q3] (i) Formal mathematical justifications for the choices of $N$, $K$, and $\lambda$. (ii) Necessity of Triple Consistency. (iii) Convergence properties. (iv) Error bounds. [Continued]

(ii) For the necessity of the triple consistency, we refer the reviewer to studies in the second part of the global response.

(iii) Due to the use of the neural network in our objective function, where the optimization variable is the input of the network, analyzing the proposed steps theoretically is a very challenging task including the convergence analysis of SITCOM. This is due the non-linearity of the network and the non-convexity of the loss landscape.

In terms of the convergence properties of the optimization in Eq.(S1), we use the ADAM optimizer for at most $K$ steps. This means that if we look at the optimization at each sampling step, the theoretical convergence results of ADAM applies, such as the ones in [B].

For the empirical convergence of SITCOM, we provide PSNR curves of SITCOM using different values of $N$ and $K$ to show the empirical convergence results: https://anonymous.4open.science/r/SITCOM-7539/convergence_plots.md.

As observed, with $NK=400$, $(N,K)=(10,40)$ yields the best results. Notably, after completing the optimization steps at each sampling step, we observe a drop in PSNR, attributed to the application of the resampling formula in Eq.(S3).

(iv) We agree that deriving mathematical error bounds will benefit the understanding of the theoretical performance of SITCOM. However, this is particularly challenging when network regularization is involved. As noted in [C] in the context of unconditional diffusion: the input error $\epsilon_{t+1}^{\text{cumu}}$ to the module $p_{\theta}(x_{t-1}|x_t)$ is parameterized by a non-linear network, it is not certain whether the input error is magnified or reduced in the error contained in its output $\epsilon_{t}^{\text{cumu}}$. The added data consistency in SITCOM further complicates this analysis.

To address this limitation, we added new numerical experiments and ablation studies as suggested to evaluate our method. Please see the second part of the global response.

[C3] (i) Improvements in ImageNet for PR and GDB tasks. (ii) NFEs.

We thank the reviewer for this comment/suggestion. As recommended, we will re-organize this part for further clarification.

(i) We conjecture that the complexity is related to the complexity of the task (ill-posedness) and the diversity of the dataset itself instead of our method. ImageNet is a classification dataset with several labels that are unrelated semantically, whereas FFHQ is a dataset of mostly faces.

For GDB, We are slightly lower than the SOTA in ImageNet. For PR, there is an interesting phenomena that still requires further investigation which we discuss in Appendix D. In short, we observe that we have slightly more failure cases (i.e., very low PSNRs) than DAPS. However, once we don't fail, we obtain higher PSNRs. To support this statement, we non-cherry pick a test image from ImageNet and run DAPS and SITCOM for 10 times. We report the PSNR for each. The results are given in (https://anonymous.4open.science/r/SITCOM-7539/PR_ImageNet_exp_vs_DAPS.md). The red results indicate failure cases. In SITCOM (resp. DAPS), we see 3/10 (resp. 2/10) cases. However, when we don't fail, our PSNRs are higher (the two bold results in SITCOM that is higher than the best results in DAPS).

(ii) In terms of NFEs, given the proposed stopping criterion, this results in at most $NK$ NFEs of the pre-trained model (forward pass), $NK$ backward passes through the pre-trained model, and $NK$ applications each for the forward operator and its adjoint to solve the optimization problem in Eq.(S1). With early stopping, the computational cost is lower. We discuss this in the paragraph before Section 3.6. In Tables 1 and 2 (Tables 1 and 4 in the revised paper), we opted to report the run-time instead of NFEs as different methods may apply the DM network differently and the computational cost could increase from other steps that don't use the DM network (for example, the optimization problem in DCDP).

That said, we promise the reviewer to include NFEs in a subsequent version of our paper.

[B] Convergence of Adam Under Relaxed Assumptions. NeuriPS23. [C] On Error Propagation of Diffusion Models. ICLR24.

We would like to thank the reviewer for their comments.

[C1] Further clarifications/intuitions of the proposed backward consistency for a boarder audience.

We would like to thank the reviewer for this comment. We have incorporated a discussion on the motivation for the proposed backward consistency, drawing from the well-known Deep Image Prior (DIP) [A] regularization, on page 5. Additionally, we have revised Definition 1 to enhance clarity.

In SITCOM, we apply the network regularization during each sampling step such that we fully exploit the implicit regularization of the network to reduce the number of sampling steps and consequently the total running time. We emphasize that the step-size backward consistency in SITCOM is the reason behind the reduced number of sampling steps.

[C2/Q2/Q6/Q3] (i) Formal mathematical justifications for the choices of $N$, $K$, and $\lambda$. (ii) Necessity of Triple Consistency. (iii) Convergence properties. (iv) Error bounds.

(i) We would like to thank the reviewer for their comment. Finding the optimal number of optimization steps and the optimal regularization parameter in SITCOM is challenging due to the presence of the neural network in the objective function in Eq.(S1). However, we conduct extensive results to shed more light into SITCOM's performance w.r.t. $N$, $K$, and $\lambda$:

In Appendix~F.1, we report the results of pairs from $N\in \\{10,20,30\\}$ and $K\in \\{20, 30, 40\\}$. To fully address the reviewer's comment and infer if there is any trade-offs between quality and computational cost, we conduct extended ablation studies with $N \in \\{4,8,12,\dots,40\\}$ and $K\in \\{5, 10, 15,\dots,40\\}$. The results of GDB is given below (PSNR,run-time in seconds). We note that the results of the 8 tasks are given in (https://anonymous.4open.science/r/SITCOM-7539/Extended_Ablation.md).

Gaussian Deblurring.

We observe that with $N=12$ and $K=15$, we obtain 28.3dB (with only 18.3 seconds) whereas at $N=K=40$, we obtain 29 dB which required nearly 110 seconds. This indicates that there is indeed a trade-ff between computational cost and PSNR values. Furthermore, we observe that 61 entries (out of 80) achieve a PSNR values of more than or equal to 28 dB. This indicates that SITCOM is not very sensitive to the choice of $N$ and $K$. We note that we observe similar patterns for all tasks.

[A] Deep Image Prior. CVPR18.

We would like to thank the reviewer for their comments.

[C1] Modeling of Backward Consistency & the projection onto an intersection of data- and prior-consistent solutions.

The reviewer is right that if we want to find a point in the intersection of two sets $C_1\cap C_2$, a common approach is using alternating projection. However, due to the specific structure of the backward consistent set, we can avoid doing the alternating projection and instead solve a single optimization problem that includes the violation of both constraints as penalty terms. Essentially, we are minimizing a smoother version of the objective $1_{C_1}(x)+1_{C_2}(x)$, where $1_{C}(\cdot)$ is the indicator function that equals $\infty$ when $x$ is outside $C$ and 0 when $x$ is inside $C$. More specifically, $x$ is defined to be backward-consistent if it can be expressed as $x= f_{\theta}(z)$ with some $z$. It is data-consistent if $\mathcal{A}(x) =y$ (assuming no noise). Combining these, we then obtain the following direct optimization problem: $\min_{x,z} \\{ \||\mathcal{A}(x)-y\||^2 \text{ s.t. } x= f_\theta(z) \\}$, solving which enforces both consistencies. Since $x$ is a function of $z$, we can eliminate it and arrive at $\min_{z} \{ \||\mathcal{A}(f_\theta(z))-y\||^2\}$, which is the optimization problem in our proposed algorithm without the regularization term.

[C2] The denoiser fixed-point condition for backward consistency, and the clarity of the Tweedie's consistency.

The triple consistency discussed in the paper is step-wise, meaning that they are enforced at each sampling step. For instance, consider data consistency: DPS enforces global data consistency, i.e., the $\hat{x}_0$ is only consistent with the data at the last step when $t=0$, while ReSample enforces it in many steps (indices in set $C$ of their algorithm), from $t=T$ down to $t=0$. This stepwise data consistency enforcement is one of the main factors driving the improved results of ReSample. Similarly, our backward consistency (as defined in Definition 1) is also step-wise with the major distinction that we require the reconstruction $\hat{x}_0$ at each step to be consistent with the network regularization and Tweedie's formula.

This contrasts with RED-diff and PGDM, which enforce global network regularization which uses the whole diffusion process as a regularizer. The global regularization approach, however, inevitably is not as effective for the improvement of the reconstruction quality (see the additional comparison results that we include in the global response).

Indeed, while Tweedie's formula is one way to incorporate step-wise network regularization, it may not be the only (or the best) approach in terms of restoration quality, but it is indeed very efficient in terms of reducing the run-time while maintaining competitive PSNRs. We plan to explore alternative methods in future work.

[C3] Additional Comparison Results. Please see the global response.

[C4] Hyper-parameters of existing baselines. For the comparisons in our paper (and rebuttal), we would like to point out that we have utilized the same datasets, pre-trained DMs, and the recommended (fine-tuned) hyperparameters as specified in the baseline papers. Furthermore, we downloaded and executed their code, verifying that the recommended hyperparameters align with the results reported in their respective papers.

[C5] Sampling efficiency and Run-time results. We agree with the reviewer that the main motivation of introducing SITCOM is to obtain an efficient sampler for solving IPs using DMs while maintaining competitive or leading PSNRs. In our paper, we report the runtime (average and standard deviation in minutes) in the 6th and 10th columns of Tables 1 and 2 in the original submission and Tables 1 and 4 in the revised paper.

[Q1] SITCOM's sensitivity to the choice of $\lambda$. In Table 5 of Appendix F.2 (which is now Table 7 in the revised paper), we report PSNR results of SITCOM with different values of $\lambda$ using 3 linear tasks and one non-linear task. Here, we include PSNR and run-time results of Gaussian Deblurring. The remaining tasks are in (https://anonymous.4open.science/r/SITCOM-7539/Extended_Ablation.md).

As observed, the PSNR values do not significantly change, indicating that SITCOM is not sensitive to the choice of $\lambda$. We conjecture that initializing the optimization variable in Eq.(S1) as $x_t$ (Step 2 of our Algorithm) plays a bigger role in ensuring the closeness stated in Condition 3.

1: Impact of the iterative application of measurement consistency in the first step of SITCOM:

This experiment examines the impact of multiple gradient steps for measurement consistency (Condition 1). We run SITCOM with $K=1$ vs. $K>1$ using different sampling steps ($N$) for two tasks (linear and non-linear), fixing steps 2 and 3 while varying step 1. The study uses 20 FFHQ images with $\sigma_{y} = 0.05$.

For $K=1$, only one gradient update is applied for data consistency (as in DPS), while $K=20$ (converged) strictly enforces the data consistency term.

Average PSNR and run-time (in seconds) are given in the following table.

The case with $K=1$ can be thought of as DPS plus the resampling formula.

Using multiple gradient steps yields the best results. For $K=1$, achieving decent results requires $N=1000$ with a learning rate of 10, leading to longer runtimes. Extended ablation studies for $N$ and $K$ are included in our response to [R. qJJQ;Q3].

2: Impact of the adopted resampling formula for the Forward Consistency in SITCOM (i.e., Step 3):

This experiment examines the impact of the resampling formula for enforcing forward consistency in step 3 of SITCOM. Specifically, we compare SITCOM using Eq.(S3) vs. Eq.(7) for remapping to time $t-1$ in step 3. Using Eq.(7) corresponds to standard DM sampling but replaces $\hat{x}_0$ with $\hat{x}'_0$ (from Eq.(S2)).

We note that the forward consistency in SITCOM is enforced via the regularization term in step 1 (with the initialization of Eq.(S1)) and resampling in Eq.(S3), similar to ReSample.

We use 20 FFHQ test images with $\sigma_{y} = 0.05$, setting $N=K=20$. The table below shows average PSNR/runtime (in seconds) for one linear and one non-linear task.

As observed, utilizing Eq.(S3) and the regularization/initialization in Eq.(S1), indicating that this is the best approach for the forward consistency.

3: Impact of the Backward Consistency:

In SITCOM, we apply a step-wise network regularization for the backward consistency such that we fully exploit the implicit regularization of the network. Removing the step-wise network regularization is equivalent to removing the requirement for $\hat{x}_0 = f(v_t;\theta,t)$, which makes $x_0$ a free variable. This reduces Eq.(S1) to the optimization in Eq.(5).

To this end, we compare optimizing over the input of the DM (as in SITCOM) with optimizing over the output of the DM (i.e., Eq.(5)) at sampling steps $t' \in \\{200, 400, 600, 800\\}$. Specifically, we run SITCOM from $t=T$ to $t=t'+1$. Then, at $t=t'$, we perform two separate optimizations with initializing the optimization variable as $x_{t'}$ : One over the input and another over the output using the same number of optimization steps, $K=20$, with the same noise setting as above.

SR:

NDB:

As observed, given the same run-time (determined by the number of gradient updates), optimizing over the input consistently achieves better results, highlighting the impact of the proposed step-wise network regularization for the backward consistency. We note that the lower PSNRs is due not running the algorithm until convergence.

It is worth noting that while some previous works, such as RED-diff and DMPlug, also utilize the implicit bias of the network, they adopt the full diffusion process as a regularizer, applied only once. In contrast, our method uses the neural network as the regularizer at each iteration and focuses specifically on reducing the number of sampling steps for a given level of accuracy.