Hybrid Regularization Improves Diffusion-based Inverse Problem Solving

Dear Reviewer 86ej，

Thank you for your time and constructive comments on our work. Below are the responses we have prepared to your questions.

---

Q: Comparisons with recent works such as DDNM [1] and GDP [2].

A: Thank you for your advice. We have incorporated the methods you mentioned into our experiments. The implementations of DDNM and GDP are from official repositories. We have performed inpainting, super-resolution, and compressed sensing experiments on ImageNet and FFHQ. The results are shown below, while Tables 1 and 2 (Page 8) of the manuscript have been updated.

As can be seen from the results, HRDIS is consistently superior to these advanced methods.

---

Q: The performance on real-world samples of blind inverse problems 

A: A known limitation of our approach is the requirement to access the degradation model, which is a common challenge for all training-free diffusion-based inverse problem solvers. We have frankly recognized this limitation in Appendix A.2 (Page 14) of the manuscript.  Be that as it may, we conducted additional experiments for blind inverse problems. One potential way to mitigate the issue is to assume a simple, learnable degradation model and jointly optimize it. Following the approach in [2], we consider the degradation model as:
 $y=fx+\mathcal { M }=:A_{f,\mathcal { M }}(x)$ 

where $f$ is a scalar and $\mathcal { M }$ is a mask, both of which are initially unknown. We then alternately optimize these parameters and the image using HRDIS through the following steps:


1. Update $f$ and $\mathcal { M }$: Optimize them using the gradient $\nabla_{f,\mathcal { M }}\\|y-A_{f,\mathcal { M }}(\mu)\\|^2$.
2. Update $\mu$: Optimize the image using the gradient $\nabla_{\mu}\\|y-A_{f,\mathcal { M }}(\mu)\\|^2+\mathbb{E}[\omega_t (\epsilon_{\theta}- \epsilon_{hybrid})]$.
    We performed preliminary experiments on real-world low-light data (LOL dataset). Our results demonstrate that HRDIS achieves performance superior to GDP (Table 12, Page 24), with the added advantage of being faster. Specifically, HRDIS requires about 300 NFE, compared to GDP's 1000 NFE. Figure 19 (Page 24) in the revised manuscript provides qualitative results for the reviewer's reference.

---

Q: Extention to other inversion problems, such as image editing.

A: To be honest we have limited knowledge of recent advances in image editing. It seems to us that our proposed approach has the potential to be extended to this area. For instance, our inpainting function could be adapted to modify specific regions of an image with a mask. Additionally, it is possible to combine our hybrid regularization with techniques like Delta denoising score [3]. We acknowledge this as an exciting avenue for future exploration and plan to investigate it further in our subsequent work.

---

[1] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model, ICLR 2023.

[2] Generative Diffusion Prior for Unified Image Restoration and Enhancement, CVPR 2023.

[3] Hertz A, Aberman K, Cohen-Or D. Delta denoising score. ICCV 2023.

Dear HwgS，

Thank you for your constructive feedback. Below is our detailed response to your concerns.

---

Q: Clarify for eq. 10.

A: Thank you for your suggestion, we have clarified this in the revised manuscript.

---

Q: Discussion on 'Consistency Models Improve Diffusion Inverse Solvers'[1].

A: The work in [1] focuses on leveraging consistency models (CMs) to enhance diffusion posterior sampling (DPS) [2]. In DPS, the single-step denoising step represents the mean of clean samples, while the single-step estimation in CMs ensures the prediction of clean samples. However, a limitation of this approach is that it depends on pre-trained CMs, which are not widely available. In contrast, our method does not face this limitation. We utilize a diffusion model, which has the advantage of richer pre-training checkpoints that are more generally accessible.

---

Q: Use CMs directly in the original RED-diff framework.

A: Unfortunately the repository for CMs (https://github.com/openai/consistency_models) does not provide checkpoints for the benchmark datasets ImageNet256 or FFHQ256. We decided to experiment with a checkpoint on the LSUN bedroom. Figure 18 (Page 23) of the revised manuscript shows some qualitative results on box inpainting. We find that using a CM slightly improves the RED-diff of high-frequency features, which is similar to the role of CR to a certain extent.

---

Q: Report on reconstruction metrics.

A: In the revised manuscript, we have included additional reporting on PSNR and SSIM metrics on Tables 9-11 (Page 23). Our findings indicate that HRDIS performs strongly on compressed sensing and three nonlinear problems. For scenarios involving free-form masks or large-scale downsampling, uncertainty increases, resulting in a trade-off between PSNR and FID. Methods such as DDRM and RED-diff, which often produce feature-averaged reconstructions, tend to achieve higher PSNR scores. HRDIS prioritizes generating perceptually consistent reconstructions.

---

Q: $\beta$=0.2 is a common sweet spot for different tasks.

A: In fact, we performed the ablation on $\beta$ over a relatively wide range, evaluating the values 0.0, 0.1,0.2,0.5,1.0 within the interval [0,1]. We believe that if finer intervals are taken, the optimal $\beta$ should be slightly different for each task. Nevertheless, a uniform setting of $\beta=0.2$ is enough for our framework to compete with the SOTA method.

---

[1] Xu, Tongda, et al. Consistency Models Improve Diffusion Inverse Solvers. arXiv preprint arXiv:2403.12063 (2024).

[2] Chung, Hyungjin, et al. Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR 2023.

Dear Reviewer tzUN,

Thank you for your valuable feedback and thoughtful comments on our manuscript. We will address your questions point by point below.

---

Q: Further insights for over-sharpness or noise in Consistency Regularization (CR)

A: We hypothesize that two aspects cause the results of CR. On the one hand, CR directly optimizes image pixels, and consistency distillation (CD) is equivalent to optimizing a neural representation of the image. It is well known that neural representation naturally has implicit regularization [1,2], whose spectral bias is resistance to over-sharpness and noise. On the other hand, CR optimization may be more challenging because it also involves measurement loss compared to CD.

---

Q: Less diversity in the inversed image.

A: Experimentally we have not observed a reduction in diversity so far. In fact, HRDIS mitigates the mode collapse of the original RED-diff, allowing the use of different seeds to produce diverse results.

---

Q: The use of EMA.

A: CD did initially use the EMA (Exponential Moving Average) technique. However, this trick was abandoned in subsequent improvements by Song et al. [3]. As a result, we do not use EMA in our method either.

---

Q: 2x network inference.

A: In practice, due to the use of the descending timestep strategy proposed in RED-diff, we suggest retaining the predicted noise from the previous step to approximate the difference in CR. This results in almost no increase in computational cost for each optimization step compared to RED-diff. At the same time, HRDIS usually reduces the number of iterations because the gradient is more stable.

---

Q: The results in Figures 6 and 7.

A: The degradation setups used for our experiments mainly align with those adopted in prior works. 

Phase retrieval involves recovering the phase information in the Fourier domain [4], which is inherently unstable. We followed the setup of [4] and took the best one of four independent runs for each method.

Nonlinear deblurring simulates the blurring kernel using a pre-trained UNet model to replicate realistic degradation scenarios [4].

For compressed sensing, we utilize block-based sampling, where the signal is decomposed using Singular Value Decomposition (SVD) to extract its primary sparse features. This operation is performed in blocks, each of size 32 × 32 [5]. 

HRDIS usually has superior performance on these tasks, consistently recovering higher-quality results compared to the baseline methods.

---

Q: In line 421-422, "FFHD" should be "FFHQ"

A: Thanks for your careful reading. We have corrected the clerical error.

---

[1] Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. CVPR 2018.

[2] Rahaman, Nasim, et al. On the spectral bias of neural networks. ICML 2019.

[3] Song, Yang, and Prafulla Dhariwal. Improved techniques for training consistency models. ICLR 2024.

[4] Chung H, Kim J, Mccann M T, et al. Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR 2023.

[4] Wang Y, Yu J, Zhang J. Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. ICLR 2023.

Dear Reviewer Yt8A,

Thank you for your time and feedback on our manuscript, We have carefully addressed your questions and concerns point by point below.

---

Q: Ablation study for DR formulation combined with a hybrid noise model.

A: Following your suggestion, we combine DR's formulation with the DDIM framework of hybrid noise to perform ablation experiments. The results are shown as follows:

ImageNet-Inpainting | Method | LPIPS ↓ | FID ↓ | CA ↑ | |--------------|-----------|----------|---------| | DR | 0.117 | 24.6 | 69.5 | | DR+DDIM | 0.113 | 23.5 | 70.4 | | HRDIS (Ours) | 0.096 | 20.1 | 71.3 |

ImageNet-Super Resolution

FFHQ-Nonlinear Deblurring

We find that DR+DDIM improves the RED-diff. It actually aligns with the conclusions of Sec.3.1 in the manuscript. The random perturbation of the RED-diff creates uncertainty, but the DDIM solver may alleviate this issue.  However, DR+DDIM still underperforms compared to HRDIS. HRDIS provides smooth interpolation of DR with CR and is a more flexible framework capable of forming synergies between the two. We have added this discussion on Page 21 of the revised manuscript and Figure 15 (Page 21) illustrates the qualitative comparison.

---

Q: Difference with one-step DDS and comparison with DDS.

A: DDS [1]: DDS can be considered as approximate posterior sampling, which is realized by gradually guiding the unconditional DDIM sampling. DDS involves three steps:

Step1: predict $x_{0|t}$ with $x_t$. Step2: solve $\min_{{x}^\prime_{0|t}}1/2 \|\|y-A{x}^\prime_{0|t} \|\|^2 + \gamma/2\|\|{x}^\prime_{0|t}-x _ {0|t}\|\|^2$ (Eq.1) with conjugate gradient (CG). Step3: calculate $x_{t-1}$ with DDIM.

DDS iteratively makes modifications for sampling trajectories $x_t, x_{t-1},...,x_0$ by Eq.1.

HRDIS: Our HRDIS is rooted in the RED-diff framework and treats sampling as a stochastic optimization. A key distinction is that HRDIS maintains an additional variable $\mu$ (initialized as $A^Ty$) throughout the optimization process, which is not present in DDS and has a distinct role from ${x}^\prime_{0|t}$. This variable is updated using gradients $\nabla_{\mu}L_{HR}$ with an off-the-shelf optimizer (Adam).

These fundamental differences result in distinct sampling behaviors. We show the evolution of the sampling process on Figure 16 (Page 22) of the revised manuscript. DDS exhibits a gradual change from noise to the reconstruction. HRDIS exhibits a evolution bridging the degraded image and the reconstruction.

Comparison with DDS: Since DDS (https://github.com/HJ-harry/DDS/) only considers linear measurement. We conduct experiments on inpainting, super-resolution (SR) and compressed sensing (CS). The results are summarized below:

HRDIS outperforms DDS on most tasks. Figure 17 (Page 22) of the manuscript shows some qualitative results. We observe that DDS may produce samples that are inconsistent with context.

---

Q: Assertion about inverse problem solver.

A: We fully agree with the reviewer's statement. Approximate posterior sampling is currently the mainstay of diffusion model-based solvers (in terms of image quality and generality). The work in our paper demonstrates for the first time that RED-diff-like stochastic optimization methods can compete with posterior sampling, or even better, in both of these aspects. We expect that our work can bring new algorithmic opportunities into this field.

---

[1] Chung et al. Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems. ICLR 2024

Thank you for addressing some of my concerns by providing an ablation study. Therefore, I will increase the rating if the following remaining concerns are addressed.

• Given that DR + DDIM improves performance, the remaining difference and advantage of the proposed method stem from the use of a different time step for the distillation ($t + \epsilon$). However, the authors have not provided any ablation study or theoretical analysis to confirm the significance of this step. To address this, the authors should conduct an ablation study by varying the $\epsilon$ value, as DR + DDIM can be considered a special case of the proposed method.

• Step 2 of DDS is equivalent to Equation (11), as the last regularization term can be represented by Equation (12), except for the difference in the time step. The DDS notation $x_{0t}$ in the authors' response is, in fact, equivalent to $\hat \mu_{0|t}$ in the authors' notation, and DDS cost function is also minimized with respect to $\mu$. Therefore, the visualization in Fig. 16 is not a fair comparison. The noisy sample is displayed for DDS, while Tweedie's posterior mean is presented for the proposed method. This inconsistency may misrepresents the similarity of DDS relative to the proposed method.

Dear Reviewer Yt8A, 

Below, we provide further clarifications regarding your concerns:


---

 The intuition for the addition of $\varepsilon$. We hypothesize that it is because HRDIS operates with the stochastic optimizer Adam. The original $\varepsilon$ scheduling in DDIM may not directly align with our framework. 

---

Differences in sampling behavior. We would like to address what seems to be a little misunderstanding about our approach and its distinction from DDS. To clarify, HRDIS always maintains an additional optimization variable, $\mu$, throughout the optimization process. In contrast, DDS discards $x_{0|t}^{\prime}$ after computing $x_{t-1}$ at next step. This fundamental difference leads to one of the key implications: 

1. Ancestral Sampling in DDS: DDS must always start from Gaussian noise due to its reliance on ancestral sampling of diffusion model, limiting its flexibility.
   
2. Flexible Initialization in HRDIS: HRDIS supports flexible initialization of $\mu$, enabling it to potentially achieve better results.  As far as we are aware, DDS does not offer an equivalent capability for flexible initialization similar to our HRDIS. However, if we have misunderstood or overlooked any aspects of DDS's behavior, we would greatly appreciate further clarification.

Dear Reviewer Yt8A,

Thanks for your insightful feedback and suggestions.

---

Regarding your first concern: We conducted an ablation study on $\varepsilon$ in the CR item, focusing on inpainting and super-resolution tasks using the FFHQ dataset. The quantitative results are presented in Figure 20 (Page 25) of the revised manuscript. Assuming the diffusion model operates over the time interval $[0,1]$, we found that $\varepsilon$ performs well when it is around the order of $1e-2$.

• When $\varepsilon$ is too small ($1e-3$), the effect of CR diminishes, leading to blurred results.

• When $\varepsilon$ is too large ($1e-1$), the resulting image quality degrades due to the discretization error.

Additionally, Figure 21 (Page 25) includes qualitative visualizations that illustrate these effects, providing a clearer understanding of how $\varepsilon$ influences the final output.

---

Regarding your second concern: As per your suggestion, we visualized the evolution of Tweedie's mean in DDS and included the modified visualization in Figure 16 (Page 22) of the revised manuscript. Our findings indicate that DDS’s sampling process still differs fundamentally from our proposed method. Specifically, the evolution in DDS does not exhibit the same smooth transition from degraded to reconstructed images as seen in our approach. This difference highlights the unique characteristics and advantages of our method.

Dear Reviewer Yt8A,

Thank you for your time and improved score! The constructive discussions with you have significantly strengthened our manuscript. 

Based on your suggestions, we will incorporate the comparisons with DDS and the role of $\varepsilon$ in the appendix into the main text. Additionally, as recommended, we will make sure that the discussion of HRDIS convergence is included in the limitations section.

Best,

Authors of Submission 9721

Dear Reviewer Yt8A,

Thank you for your comments. Here are the responses we have prepared for you. We would like to clarify your third question first because it is important for understanding our approach.

---

The use of $\mu$: To further illustrate the differences, we provide a comparison of graph models (Figure 22, page 26) in the revised manuscript. We refer the reviewer to it.

Taken as a whole, DDS corrects the ancestral sampling trajectory by solving a sub-optimization problem at each step. The sampling process of our HRDIS/RED-diff, on the other hand, is considered as a single stochastic optimization problem. The optimization result is the reconstructed sample.

Locally, in DDS, $x_{0|t}^{\prime}$ only related to $x_{0|t}$ and $y$, which is easy to followed, since $\min_{{x}^\prime_{0|t}}1/2 \|\|y-A{x}^\prime_{0|t} \|\|^2 + \gamma/2\|\|{x}^\prime_{0|t}-x _ {0|t}\|\|^2$. $x_{0|t}^{\prime}$ has no explicit dependency on $x_{0|t+1}^{\prime}$

We use the superscript $(n)$ to denote the optimization step for brevity. In HRDIS/RED-diff, $u^{(n+1)} = u^{(n)}-\nabla L_{HR/DR}$. It can be seen that $u^{(n+1)}$ and $u^{(n)}$ have explicit dependency. If we do not maintain the $u^{(n)}$, $u^{(n+1)}$ can not be readily computed only from the noisy sample. $x_{0|t}^{\prime}$ of DDS is not, it can be computed directly by noisy sample $x_{t}$ as you mentioned.

The above difference leads to a more continuous and smooth transformation of $u$. $x_{0 | t}^\prime$ does not have this guarantee. Such difference could gradually increase with the number of iterations, leading to a completely distinct performance of HRDIS and DDS. Regarding quantitative results, our HRDIS is usually much higher than the DDS. We often observe that DDS fails to harmonize with context $y$.

---

Initialization for the conjugate gradient of DDS: We can certainly use $A^\top y$ to initialize as a starting point for each sub-optimization problem in DDS. However, it is important to note that $\min_{{x}^\prime_{0|t}}1/2 \|\|y-A{x}^\prime_{0|t} \|\|^2 + \gamma/2\|\|{x}^\prime_{0|t}-x _ {0|t}\|\|^2$ is a convex problem (when A is a linear degradation), and its local minimum is the global minimum (such characteristic is significantly different from the HRDIS/RED-diff's stochastic optimization problem). Without considering numerical accuracy, the initial point may have little effect on the solution results. We included an experiment as you mentioned, and we found that DDS still performs worse than HRDIS.

---

The use of stochastic Adam optimization: We believe that $\varepsilon$ in HRDIS should also be re-selected based on empirical results, as Adam has momentum, and direct DDIM scheduling can work but may not be optimal.