Masked, Regularized Fidelity With Diffusion Models For Highly Ill-posed Inverse Problems

We would like to thank you for your constructive comments. See below for our point-by-point replies.

[W1]: "In real-world applications (see Appendix G8), the restored images exhibit a smoother appearance compared to the ground truth. This suggests that there is still room for improvement in the number of steps used by the Dilack algorithm."

While the synthetic lensless imaging cases use the exact forward model, the real lensless imaging cases may often use the mismatched forward model, which makes the inverse problem much more challenging.

[W2]: "The computational complexity of the algorithm is quite high."

Appendix F.1 already discussed the computation cost, which is modestly increasing from 340 seconds to 390 seconds per image. However, note that our Dilack is practically the only solution that works for lensless imaging cases, so this modest increase can be justified.

[Q1]: "How effective is the algorithm when applied to synthetic images and other tasks, such as super-resolution and denoising?"

For super-resolution (SR), $A$ matrix consists of a combination of a blur kernel and a downsampling operation. As the degree of downsampling increases, the condition number becomes larger, presenting a challenging scenario where the strengths of our proposed method are expected to be effective. However, highly ill-posed SR has not been well investigated in the field yet, so we believe that we need to carefully validate one by one. Especially, the impact of downsampling operator is worth investigating. Thus, at this moment, we can say that our Dilack can work for SR as compared to other prior works, but there is still room for improvement due to the reasons mentioned (see Appendix G.15 and Fig. S14 (highlighted in blue), which are the results of our additional experiments with SR x4, x8).

For denoising, it may be difficult to expect the applicability of our Dilack since highly ill-posed cases with high noise levels have different ill-posedness from other tasks with kernels. Full investigation will be needed for these cases. Nonetheless, under Gaussian noise levels ($\sigma$ = 0.05 to 0.1), our method performs comparable to DPS, though a more comprehensive investigation is required for these cases as a future work (see Appendix G.15 and Fig. S15 (highlighted in blue), which are the results of our additional experiments).

[Q2]: "In the TV-regularized optimization component of the Dilack algorithm, is it feasible to perform just a single optimization step instead of seeking the minimum value? What would be the anticipated impact on overall performance?"

Thank you for interesting suggestion. In our original Dilack, we ran 1000 ADMM iterations in line 8 of Alg 1 and this guidance anchor update occured G = 1 times. As suggested, we have conducted the cases with (1) 1 ADMM iteration (with the initial vector $x'_{t-1}$ from line 6) with 1000 guidance anchor updates (i.e., G=1000) and (2) 1 ADMM iteration with 1 guidance anchor update and the results are reported in the below table, demonstrating the superiority of our Dilack over other settings. It seems that initial guidance was very important especially for solving highly ill-posed inverse problems.

As promised, we have updated our corrected results for Fig. S14 as described below (Figures will be updated in the final version). Here are the average PSNR and SSIM for 4 images (data indices 0–3) using five different random seeds for FFHQ super-resolution task (x4 and x8) with Gaussian noise ($\sigma$ = 0.05), following the same experimental settings as DPS. Our proposed Dilack yielded comparable performance to the prior arts (DPS).

Table. Super resolution x4 and x8 experiments

We would like to thank you for your constructive comments. See below for our point-by-point replies.

[W1]: "The central concept of the proposed approach is not clearly, and the overall description of the methodology lacks clarity."

We have revised the submission with a particular focus on improving clarity in Abstract, Introduction, Analysis, and Methodology sections. See the revised manuscript, with all modifications highlighted in blue text.

[W2]: "The proposed algorithm incorporates many hyperparameters, which may complicate the implementation and optimization processes. This abundance of hyperparameters can also raise concerns regarding the stability and generalizability of the algorithm, as tuning them effectively may be challenging for practitioners."

Newly introduced hyperparameters are related to ADMM solver, which has been well-studied in the field, or ROI mask, which is relatively easy to optimize. Thus, determining them relied on prior works or simple tuning. Specifically, in lensless imaging, ADMM automatically adjusts hyperparameters through residual balancing [1,2], while fixing the $\lambda$ and $\rho$ during hardware calibration. For ROI masks, only two parameters — patch size and mask threshold — were introduced, which turned out to be not relatively sensitive to different tasks / data. Moreover, unlike comparison methods (e.g., DPS, DiffPIR) that require task-specific tuning (e.g., step-sizes), our method consistently uses a fixed guidance $\alpha$ value.

[1] Wohlberg et al., "ADMM penalty parameter selection by residual balancing," arXiv, 2017. [2] Galatsanos et al., "Methods for choosing the regularization parameter and estimating the noise variance in image restoration and their relation," IEEE T-IP, 1992.

[Q1-first part]: "The experimental results presented in the manuscript are unconvincing. For example, in the table above Fig. 4 regarding Large Motion Deblurring, the proposed method only outperforms other methods in 1 out of 8 cases."

Firstly, note that our proposed Dilack significantly outperformed all compared methods for two lensless imaging cases as shown in Table 1, which are already convincing experimental results. Secondly, as discussed in Section 3 (Challenge 1: Large motion deblurring), it was expected that DiffIR performs as well as the proposed method at least for large motion deblurring cases that have less complicated kernels than lensless imaging and the results were reported in Table 1, demonstrating that our propose Dilack yielded comparable performance to DiffIR and still outperformed most prior works. DDPG is a recent SOTA method that yielded great PSNR performance for large motion deblurring. However, the output image in DDPG often yielded artifacts near edges or details like characters on ImageNet as illustrated in Fig. 4 so that our Dilack outperformed DDPG in SSIM as well as FID, LPIPS. For FFHQ, faces have less details (e.g., no text, less sharp edges) over ImageNet, so this disadvantage of DDPG has been minimized, yielding SOTA performance in all metrix except for FID.

[Q1-second part]: "The authors should also discuss the computational cost associated with their method, as this is crucial for evaluating its practicality."

Appendix F.1 already discussed the computation cost, which is modestly increasing from 340 seconds to 390 seconds per image. Note that our Dilack is practically the only solution that works for lensless imaging cases, so this modest increase can be justified.

[Q2]: "The parameters $L_{LS}$ and $L_{Pi}$ are defined in formulas (9) and (10), respectively, but are used in Figure 1 without reference to their definitions. It is essential for the authors to mention these definitions in the caption of Figure 1 to ensure clarity for readers."

We have added citations to Fig. 1(a) and included references to the corresponding equations for $L_{LS}$ (Eq. 9) and $L_{Pi}$ (Eq. 10) in the caption as suggested.

[Q1]: "Consistency in Comparisons: Why do Tables 2 and 3 compare against different methods? Consistently using the same methods across both tables would allow for a more direct comparison of results."

Note that the compared prior works were developed based on either pixel-space diffusion model or latent-space diffusion model. Thus, we grouped prior works into two and compared with one group or the other by fixing the diffusion model, reporting the results in Tables 2 and 3, respectively. This separation was our effort not to alter the original prior works and to demonstrate our Dilack working with both pixel-space and latent-space diffusion models without training or tuning.

[Q2]: "Code Release: Is there a plan to release the code for this method?"

Yes, we plan to release. In fact, as noted in Appendix F.1, we have already submitted the Dilack code as part of the supplementary materials.

[Q3]: "Clarification on Appendix Statement about ADMM and Regularization: The sentence in the Appendix, “Incorporating Total Variation (TV) regularization into the Alternating Direction Method of Multipliers (ADMM) enhances the algorithm’s performance across various applications,” is somewhat misleading. It is not the use of TV specifically but rather the addition of a regularization term that enables the solution of the inverse problem. Additionally, any proximal splitting algorithm—not just ADMM—could be used for this purpose. Could you clarify this?"

Thank you for raising this. It seems that the sentences in lines 899-905 may be misleading and are not essential, so we removed them in the revision. ADMM and TV are somewhat independent components, one is for optimization and the other is for a regularizer in the cost function. Thus, TV should be incorporated into the cost function for inverse problem and ADMM can solve that inverse problem.

However, it is still true that the proposed combination of TV and ADMM, or $ADMM\_\{TV\}$, is quite effective for highly ill-posed inverse problems over other combinations. To demonstrate it, we have conducted additional experiments with different optimizations / regularizers in PiAC guidance for Dilack for the lensless (turing kernel) imaging on the FFHQ dataset (sampled 100 images) as summarized in the below table. This table demonstrated that our proposed Dilack with $ADMM\_\{TV\}$ consistently outperformed Dilack with other combinaions such as ADMM-$L_1$, ADMM-$L_2$, FISTA (Fast Iterative Soft-Thresholding Algorithm) with $L_1$ and HQS (Half-Quadratic Splitting) with TV.

• Quantitative Comparison of Dilack and Its Variants with Different Regularizers & Optimization Algorithms

[Q4]: "Applicability of Data-Fidelity Term with Other Priors: If the primary contribution is to modify the data-fidelity term, could this approach work with priors other than diffusion models? This generalization could broaden the method’s applicability."

The primary contribution of this work is the methods to leverage pre-trained diffusion models for solving highly ill-posed inverse problems from real lensless imaging or large motion blurring. While one of our core contributions is to propose a data-fidelity term, note that proposing good approximate measurement matching terms has been important in a number of recent works - see Table 1 / Figure 1 of a recent survey paper [2]. All fidelty designs were taylored to working with diffusion models. Thus, it is unclear if our proposed fidelity term is indeed generalizable.

However, to alleviate your concern, we have applied our guidance setting using a CNN denoiser (following DPIR [1]), but it did not yield poor performance as expected. While this result was not in the revision, we can incorporate that into the next revision if needed.

[1] Zhang et al., "Plug-and-play image restoration with deep denoiser prior," IEEE T-PAMI 2021. [2] Daras et al., A Survey on Diffusion Models for Inverse Problems, arXiv 2024.

[W5]: "Comparison with DPS Method: The comparison with DPS may be unfair, as DPS’s performance could likely be improved with step-size optimization. This omission may explain DPS’s lower performance in the experiments."

To relieve your concern, we have conducted additional experiments using our two tasks on 100 sample images from the FFHQ dataset as shown in the below table, confirming that varying the step-size did not signifcantly improve the performance of DPS. Quantitative results and further details on this experiment have been included in Appendix G.10 of the revised manuscript (highlighted in blue).

• Ablation study on step-size effect on DPS

[W6]: "Mismatch Between Equation (17) and Algorithm 1: Equation (17) does not match the steps in Algorithm 1 due to the addition of noise in the algorithm."

Thank you for indicating this mismatch. We will revise it as below to address this concern.

Revised Equation (17):

$\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t|\mathbf{y}) \simeq \mathbf{s}_{\theta^}(\mathbf{x}_t, t) - \rho, \mathcal{M}_{\text{t}} \left[ \nabla_{\mathbf{x}_t} || \tilde{\mathbf{x}}^{} - \hat{\mathbf{x}}_{0|t}~||_2^2 \right].$

See the response to [W3] for the justification on replacing $x_t$ with $x_{0|t}$.

For your information, the exact equation for $\mathbf{x}_{t-1}$ with respect to $\mathbf{x}_t$, considering the noise addition pointed out by the reviewer, is given as follows:

$\mathbf{x}_{t-1} \simeq \frac{1}{\sqrt{\alpha_t}} \mathbf{x}_t + \frac{1 - \alpha_t}{\sqrt{\alpha_t}} \mathbf{s}_{\theta^}(\mathbf{x}_t, t) + \tilde{\sigma}_t \mathbf{z} - \rho \mathcal{M}_{\text{t}} \left[ \nabla_{\mathbf{x}_t} \left| \tilde{\mathbf{x}}^{} - \hat{\mathbf{x}}_{0|t} \right|_2^2 \right],$

where:

• $\alpha_t$ is the Noise Scaling Coefficient,
• $\tilde{\sigma}_t$ is the Diffusion Adjustment Coefficient,
• $\mathbf{z}$ is Gaussian noise with mean 0 and variance equal to the identity matrix.

[W7]: "PSNR Comparison in Posterior Sampling Context: Since the goal is to sample from the posterior distribution, comparing PSNR values may not be meaningful unless comparing the posterior means. Are PSNR values in the paper calculated based on a single sample from each method?"

We would like to point out that all prior works including DPS, DiffRIR, DDPG, ReSample and LDPS were consistently evaluated in PSNR with fixed-seed sampling (i.e., a single sample per measurement) and we also stick to this setting. While a single sample may not provide a reliable value for one image, single samples for the entire test dataset seem to provide reasonable results by aggregating them.

However, to relieve your concern, we have included additional results with varied seed values in Appendix G.15 of the revised manuscript.

[W8]: "Algorithm Structure for Clarity: Since (G = 1) is used in practice, restructuring the algorithm into two main steps—1) ADMM-TV and 2) Diffusion—could improve clarity and make the methodology easier to understand."

Thank you for the suggestion. We have revised our manuscript in a simpler way as suggested by benchmarking ReSample [Song et al., 2024] that enforces hard consistency in the beginning of diffusion process.

We would like to thank you for your constructive comments. See below for our point-by-point replies.

[W1]: "Readability Issues: The paper is sometimes challenging to read due to long, complex sentences, particularly in the abstract and Remark 1. Simplifying these sections could improve readability and flow."

As suggested, we revised the abstract and Remark 1, highlighted all modifications in blue text.

[W2]: "Sampling vs. Maximization: The method proposed here, like DPS and DiffPIR, is a sampling method aimed at sampling from the posterior distribution. However, this crucial point seems to be overlooked or misunderstood by the authors, as the paper appears to focus on solving equation (2) for posterior maximization rather than posterior sampling."

We also exploited the posterior sampling framework like DPS and DiffPIR, but for the highly ill-posed problems like modern lensless imaging and large motion deblurring. However, these challenging inverse problems have quite large null space, so that naive sampling usually provide images far from GT. Our Dilack implicitly narrowed down this null space by using TV so that more faithful images to the measurement can be generated. Our Dilack has a similar aspect with [Chung et al., 2022b] (regularized fidelity in our Dilack vs. constrained fidelity in [Chung et al., 2022b]), but with a number of differences in our fidelity term design.

Note that DPS and DiffPIR also generated the experimental results with fixed-seed sampling like ours, so the comparisons in this work is fair. However, to relieve your concern, we have conducted additional experiments on sampling diversity, which are detailed in Appendix G.15 of the revised manuscript (highlighted in blue).

[W3]: "Explanation of $x$ Replacement in Equation (14): It is unclear why $x$ in line 298 is replaced by $x_{0|t}$ in equation (14). Why is the data-fidelity term $||x_{TV} - x||^2$ not used instead? The new data-fidelity term appears significantly different from $L_{Pi}$, making it unclear whether a comparison between the two terms is justified. On what basis can Equation (15) be considered approximately true? Has it been verified, for example, for simpler inverse problems that $L_{Pi}$ approximates $L_{PiAC}$?"

If the term in line 298 (of the initial manuscript) is used in the context of posterior sampling, $x$ becomes $x_t$. Then, it is straightforward (e.g., consider $\tilde{x}^*$ as a measurement vector and the identity operator as a measurement operator) to apply Theorem 1 of DPS [Chung et al., 2022a], leading to the term $|| \tilde{x}^* - \hat{x}_{0|t}||^2$ as described. Note that the Theorem 1 of DPS does not have to have a mildly ill-posed inverse problems - even for highly ill-posed problems, the forward model does not yield drastic change in the measurement, so the theorem works.

The relationship between $L_{Pi}$ and $L_{PiAC}$ looks interesting. There are two components to consider. One is $A^\dagger y$ where $L_{Pi}$ uses a minimum norm solution and $L_{PiAC}$ uses a TV solution, leading different maximum a posteriori (MAP) estimations. The other is $A^\dagger A$ where $L_{Pi}$ uses it as it is and $L_{PiAC}$ approximates it as an identity. The former only compares the values in the projected spaces while the latter compares the values in the whole spaces including the null space. Our Dilack sacrifies the aspect of measuring in the projected spaces for highly ill-posed inverse problems, but instead we approximate it by comparing the values in the null space that was filled by TV regularization.

[W4]: "Comparison with SOTA Methods: It would be beneficial to compare the proposed approach with other state-of-the-art methods in image inverse problems, such as Plug-and-Play (PnP) methods like DPIR [1]."

As suggested, we have conducted additional experiments using DPIR pre-trained model, have evaluated it on lensless imaging (Turing PSF) trained with ImageNet and confirmed that our Dilack ourperformed DPIR by large margin. The quantitative results are summarized in the below table.

[1] Zhang et al., "Plug-and-play image restoration with deep denoiser prior," IEEE T-PAMI 2021.

Thanks for the detailed answers.

The authors answered most points. After careful reading, I still think that the following points where not properly answered.

• W2 : the method is presented as a posterior maximization method with (2) when it is a posterior sampling method.
• W3 : This is somewhat not consistent to introduce the proposed data-fildelity term from the DPS one. The proposed data-fidelity is more similar to the standard L2 data-fidelity (9). Indeed A^\dagger A is very different from Identity for highly ill-posed inverse problems. The discussion you provide to compare L_{Pi} and L_{Piac} should be incorporated in the manuscript.

We would like to thank you for your constructive comments. See below for our point-by-point replies.

[W1]: "The other compared methods address the original (highly ill-posed) input, whereas the proposed method basically restores a more well-posed input (the TV-regularized solution). This raises the question of how fair the comparison is. It may make more sense to view the proposed method as a framework that can be integrated with existing methods rather than as a standalone inverse problem solution."

If the "input" you mentioned is the initial image, all methods including ours use a zero mean Gaussian noise as an initial image (see line 1 of Algorithm 1 in the paper). If the "input" you mentioned is the measurement, then all methods including ours use a measurement vector y. Thus, in terms of "input", the comparisons were fair in our paper.

The key difference between ours and other prior works is the design of guidance (i.e., data fidelity term). This difference was motivated by studying highly ill-posed modern lensless imaging systems where prior diffusion-based works can not yield good images. Our PiAC guidance is the result of a number of advances in data fidelity term designs such as pseudo-inverse incorporation, TV regularization and mask operation so that the diffusion model can sample while ensuring (regularized) measurement consistency. So our proposed method is standalone.

[W2]: "The setup presented in the paper is questionable; for example, how realistic is the kernel blur used in the study for real-world scenarios where the signal is almost completely lost?"

Kernel blurs for lensless imaging are in fact "real" kernels used in our lensless imaging systems (hardware) we have built. Kernel blurs for large motions, which are less ill-posed than the lensless imaging cases, may not occur as often as small motion blurs or lensless imaging, but they are realistic for imaging with zoom lens (small motion in camera will result in large motion blur).

[W3]: "The paper focuses on the non-blind case, assuming that the degradation forward model is known. It would be more interesting to investigate the more general, blind case instead, where the degradation model is unknown."

This work is a demonstration that solving these highly ill-posed inverse problems is extremely challenging even though they are non-blind. Note that no prior arts were able to solve these challenges. Thus, it will be even more challenging to solve blind problems, so we believe that one may be able to try to solve them based on our work in the future. We believe that this work was an important stepping stone towards the blind, highly ill-posed inverse solvers. We will discuss potential directions for this important future works in the revision such as the potential work for handling model mismatch in lensless imaging.

[Q1]: "In Figure 1, it appears that the diffusion model used is an unconditional model trained on ImageNet, which is known to be relatively weak due to the limited amount of training data compared to the large number of classes. One potential reason for the failure of existing IP methods may be the use of this weak pre-trained model. It would be interesting if the authors could reproduce the same analysis on the FFHQ dataset, where the model is much stronger, to determine whether the failures of existing methods are due to the weak pre-trained model or intrinsic issues."

We did not clearly indicate the pre-trained diffusion model used in Fig. 1. To clarify, the results in Fig. 1 are based on sampling outputs from existing methods (DPS, DiffPIR) and Dilack (ours) using an FFHQ pre-trained diffusion model, not ImageNet. Even with this stronger pre-trained model, existing zero-shot diffusion-based approaches still struggle significantly with tasks like lensless imaging and large motion blur, underscoring the intrinsic challenges of these modern practical imaging systems.