Blind Inversion using Latent Diffusion Priors

We thank the reviewer for their constructive suggestions. We are pleased that the reviewer recognizes the "good experimental results in blind deblurring and sparse view 3D reconstruction." Below, we address the raised concerns in detail:

1. Mathematical Formula in EM Framework

We agree with the reviewer that kernel estimation (M-step) should utilize posterior samples from $p(x|y)$ instead of a point estimate. This is exactly what we implement in this work. Leveraging Tweedie’s formula[1], $\hat{x}_0$, approximates posterior samples based on intermediate reverse diffusion states $(x_t)$:

$x_0 \sim p(x|y) \rightarrow \hat{x}_0 \sim p(x_0|x_t)p(x_t|y)$,

where $p(x_0|x_t)$ is defined by $\frac{1}{\sqrt{\bar{\alpha}(t)}}[x_t+(1-\bar{\alpha}(t))s_\theta(x_t,t)]$. Note that since $x_t$ is drawn from a posterior distribution conditioned on the observation $y$, $\hat{x}_0$ also represents a set of posterior samples, allowing us to retain the core principle of the expectation-maximization.

This approximation is utilized because the forward models are inaccurate in the initial stages of the blind inversion problem. Executing the entire reverse diffusion process (~1000 steps) is computationally intensive and, as shown in Appendix E and Table 6 of the paper, often worsens reconstruction by incorporating excessive model errors. Instead, leveraging approximate samples from intermediate states for fast model correction yields better performance.

Additionally, $q(x|y)$ in Equation 5 is an auxiliary distribution that we introduced to help the derivation of the EM algorithm, as done in many prior works [2,3]. The EM algorithm optimizes L(q, $\phi$) over this auxiliary distribution q(x|y) and the model parameter $\phi$ . According to Jensen’s inequality, the upper bound of this objective function is, $\log p_\phi (y)$, which is achieved only when $q(x|y)$ matches the true posterior $p_\phi(x∣y)$. Hence, the E-step maximizes the objective function by conducting posterior sampling of $p_\phi(x∣y)$. We have revised Equation 5 and accompanying explanations to make this clearer.

2. Details about the Experimental Setup

   1. Denoiser. The denoiser is a DnCNN [4] with 5 blocks and 32 channels. We used 60,000 motion blur kernels [5] for training, each combined with an additional channel for the noise level map. The network was trained for 5,000 epochs using MSE loss with a learning rate of 1e-5. Training was conducted on an NVIDIA A100 GPU, taking approximately seven hours. The information is already provided in Appendix C.2.

   2. Training kernels and test kernels. All blur kernels were generated via stochastic walks in 2D space [5], following the BlindDPS setting. Training and test kernels are non-overlapping. We will explicitly state this in the revision.

   3. Circular padding. We acknowledge that circular padding may seem unrealistic for image processing. However, it was used in both baselines, BlindDPS and FastEM, so we leverage the same settings to ensure fair comparisons. We will clarify this in the revised manuscript.

   4） Stable Diffusion. Stable Diffusion is trained on the LAION 5B dataset, while the testing datasets are ImageNet and FFHQ. Although all datasets are collected from the Internet, prior work [6] has shown that they belong to different data distributions. Therefore, the computational imaging community typically considers them as non-overlapping and widely uses ImageNet and FFHQ for evaluation.

3. Presentation Thank you for highlighting these inconsistencies. We apologize for the oversight and appreciate your attention to detail. We have corrected these issues and made the manuscript more consistent:

   1. Citation styles: Citation styles have been corrected in the updated manuscript.
   2. Equation 11: You are correct that it should not contain F^2. The correct HQS algorithm has been updated in our revision.

We have also addressed other typos and missing citations to improve clarity and consistency throughout the paper.

We thank the reviewer once again for the insightful feedback and hope our responses address all concerns. We are happy to provide further clarifications in additional discussions. We kindly hope the reviewer could consider raising the score if satisfied with our responses. Thank you!

Reference

[1] Bradley Efron. Tweedies Formula and Selection Bias. 2011.

[2] Jaemo Sung, et al. Latent-Space Variational Bayes, PAMI 2008.

[3] MATTHEW, et al. The Variational Bayesian EM Algorithm for Incomplete Data: with Application to Scoring Graphical Model Structures. Bayesian Statistics 2000.

[4] Kai Zhang, et al. Plug-and-play Image Restoration With Deep Denoiser Prior, TPAMI 2021.

[5] Fabien Gavant, et al. A physiological camera shake model for image stabilization systems, SENSORS 2011.

[6] Litu Rout, et al. Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models, NeurIPS 2023.

We thank the reviewer for the constructive suggestions and feedback. We are pleased that the reviewer finds the paper “well-organized” and acknowledges our “extensive experimental results” and comprehensive “ablation study.” Below, we provide detailed responses to the concerns raised:

1. Novelty and contribution

The integration of latent diffusion models with the Expectation-Maximization (EM) framework for inverse problems is far from trivial, contrary to the reviewer’s assumption. Prior work on blind inversion has demonstrated that jointly optimizing model parameters and underlying images introduces severe non-convexity, making image reconstruction inherently unstable.

The introduction of latent diffusion models further amplifies this challenge, because their encoder-decoder structure adds fragility and computational inefficiency due to the nonlinear mapping between pixel and latent spaces. To address these challenges, we introduced novel techniques such as: 1) "annealed consistency" to mitigate model errors and 2) "skip gradients" to accelerate computation.

Our method not only outperforms all existing diffusion-based blind deblurring baselines but also, for the first time, demonstrates capability in solving challenging pose-free 3D reconstruction tasks. We will explicitly highlight these key contributions in the revised manuscript.

2. Runtime compared to BlindDPS

We agree that when runtime is heavily constrained (as shown in Table 2), our method must trade reconstruction quality for inference speed. However, for scenarios that prioritize quality over speed—such as scientific imaging tasks like microscopy denoising or super-resolution—our method offers an alternative solution with better reconstruction accuracy.

Additionally, BlindDPS relies on training task-specific priors, whereas our approach eliminates this requirement by utilizing a pre-trained Stable Diffusion model, significantly reducing the computational overhead in the training phase.

3. Performance regarding FastEM and BlindDPS

The performance differences between our results and those reported in the FastEM paper are primarily due to the different motion blur kernel sizes. Larger kernels define harder blind inversion tasks, degrading reconstruction quality. Since FastEM does not specify the kernel size used in the paper, we tested a range of sizes and found that FastEM only performs well with small kernels but struggles significantly with larger, more complex ones.

We present experimental results of FastEM with small kernels in Appendix F of the revision. We successfully replicated FastEM’s reported performance for small kernels but observed significant performance drops with larger kernels in Figure 3.

Another evidence supporting our findings comes from the BlindDPS paper, where the reported PSNR of 22.24 aligns closely with the 22.58 reported in our paper but is significantly lower than the 24.05 reported by FastEM. This suggests that FastEM defined an easier task than the original BlindDPS paper but hides this critical information in the paper.

Additional experiments supporting these claims have been included in the updated manuscript (Appendix. F). Our comparisons in the paper are fair and accurate.

4. Non-converging patterns from tunable parameters

1. Annealing parameters: We did not observe non-converging patterns with annealing. On the contrary, annealing consistently enhances performance by preventing reconstructions from overfitting to model errors during diffusion posterior sampling. As shown in Fig. 5, removing annealing parameters results in oscillatory artifacts but does not lead to divergence, annealing at least performs better than in this case. We also included additional results for further validation in the updated manuscript (Appendix F).

2. Optimization Hyperparameters: Equations 11 and 12 define iterative optimization procedures for model parameter estimation. If learning rates are too large or noise parameters deviate from true noise levels, the optimization can become unstable, producing non-converging patterns. This is a common challenge across all optimization algorithms but can be resolved with careful hyperparameter tuning.

We thank the reviewer again for their insightful suggestions and hope our responses address all concerns. We are happy to provide further clarifications in additional discussions. We kindly hope the reviewer could consider raising the score if satisfied with our responses. Thank you!

We thank the reviewer for their constructive suggestions. We are pleased that the reviewer recognizes our work as "the first to use a pre-trained LDM for blind restoration." Below, we provide detailed explanations and clarifications for the concerns raised:

1. Novelty and contribution

The integration of latent diffusion models (LDMs) with the Expectation-Maximization (EM) framework for inverse problems is far from trivial, contrary to the reviewer’s assumption. Prior work on blind inversion has shown that jointly optimizing model parameters and underlying images introduces severe non-convexity, making image reconstruction highly unstable.

The inclusion of LDMs amplifies these challenges due to the encoder-decoder structure, which introduces additional fragility and computational inefficiency through the nonlinear mapping between pixel and latent spaces. To address these issues, we developed two techniques: 1) "annealed consistency" to mitigate model errors and 2) "skip gradients" to accelerate computation.

For imaging applications, our method not only outperforms all existing diffusion-based blind deblurring baselines but also demonstrates, for the first time, capabilities in solving challenging non-linear, pose-free 3D reconstruction tasks. We will explicitly highlight these contributions in the revised manuscript.

2. Expectation-Maximization algorithm and the derivation of our method

Our method does employ an EM framework, not an alternating optimization approach. In this framework, the true images ($x$) are the latent variables, and the goal is to estimate model parameters ($\phi$) from observations ($y$) without knowing $x$.

In the E-step, we generate multiple posterior samples using diffusion posterior sampling. For simplicity and robustness, we approximate the posterior via Tweedie’s formula [1]:

$x_0 \sim p(x|y) \rightarrow \hat{x}_0 \sim p(\hat{x}_0|x_t)p(x_t|y)$,

where $p(x_0|x_t)$ is defined by $\frac{1}{\sqrt{\bar{\alpha}(t)}}[x_t+(1-\bar{\alpha}(t))s_\theta(x_t,t)]$, based on the intermediate state $(x_t)$. Thanks to the property of the EM algorithm, it has a theoretical guarantee of local convergence.

The reviewer’s misunderstanding may stem from how we introduced $q(x|y)$ in Equation 5. $q(x|y)$ is an auxiliary distribution to aid the EM derivation, not a variational distribution. The EM algorithm optimizes $L(q, \phi)$ over q(x|y) and $\phi$. By Jensen’s inequality, the upper bound, $L(q, \phi) = \log p_\phi (y)$, is achieved only when $q(x|y)$ matches the true posterior $p_\phi(x∣y)$. Thus, our E-step conducts posterior sampling of $p_\phi(x∣y)$ (via stable diffusion), while the M-step maximizes the expected likelihood. We have revised Equation 5 and its accompanying explanation to clarify this point.

3. Details about the Experimental Setup

1. Imaging Task Parameters: For the blind deblurring task, the blur kernels are $64 \times 64$ pixels, generated using random walks. The training and testing kernels are non-overlapping. Additionally, the images are assumed to have additive Gaussian noise with $\sigma=0.02$.

2. Algorithm Hyperparameters: The hyperparameters differ between the 2D and 3D tasks. For clarity, these differences are outlined in the table below.

Reference [1] Bradley Efron. Tweedies Formula and Selection Bias. 2011.

4. Performance regarding FastEM and BlindDPS

   1. Difference from FastEM paper:

      The performance differences stem primarily from the size of the motion blur kernels. Larger kernels pose harder blind inversion tasks, degrading reconstruction quality. Since the FastEM paper does not specify kernel sizes, we tested a range of sizes and found that FastEM only performs well on small kernels but struggles significantly with larger, more complex ones.

      Experimental results of FastEM with both small kernels are included in Appendix F of the revision. We successfully replicated FastEM’s reported performance on small kernels but observed significant performance drops with larger ones in Figure 3 of the original paper.

   2. Difference from BlindDPS paper:

      The original BlindDPS paper reports blind inversion experiments only on the FFHQ dataset, not on ImageNet. FFHQ, which primarily contains human faces, has a simpler and more consistent distribution compared to the diverse and complex data in ImageNet. This difference makes blur kernel estimation significantly easier on FFHQ. As demonstrated in Figure 3 of our paper, we successfully replicated BlindDPS’s strong performance on FFHQ. However, poor blur kernel estimation is observed on ImageNet samples.

      Additional experiments supporting these claims have been included in the updated manuscript (Appendix F). Our comparisons are fair, accurate, and more comprehensive than prior work.

5. Additional experiments on GibbsDDRM and Non-Blind Posterior Sampling

   1. GibbsDDRM:

      As suggested, we conducted additional experiments with GibbsDDRM, detailed in the updated manuscript (Appendix F). GibbsDDRM and BlindDPS share a similar framework, both utilizing pixel-space diffusion and dual sampling processes for blind inversion. The primary difference is that GibbsDDRM employs an analytic prior, while BlindDPS uses a score-based prior. Consequently, their reconstruction quality is comparable, as confirmed by our experiments and consistent with the results reported in Table 1 of the original GibbsDDRM paper. These findings further reaffirm that our approach remains superior.

   2. Non-Blind Posterior Sampling:

      We computed PSNR metrics for non-blind posterior sampling. It defines an upper bound on reconstruction quality as expected.

We thank the reviewer once again for their insightful suggestions. We hope our responses address all concerns and are happy to provide further clarifications in additional discussions. We kindly hope the reviewer could consider raising the score if satisfied with our responses. Thank you!

I thank the authors for modifying the text on EM above Eq 5, which makes it clear now (and seems somewhat different than the original version, e.g. the KL divergence there lacked justification and its second argument was the posterior and not the complete likelihood).

I still argue that the proposed method cannot be termed EM.

Specifically, in the E-step in considered EM approach, replacing q by posterior sampling requires sampling multiple x0 given the current phi, which means full reversed flow akin to the non-blind case (this is obviously computational heavy -- but follows from EM). The M-step would require minimizing the likelihood given those x0 samples (after summing their per sample likelihoods). Importantly, the EM iterations are associated with the same likelihood p(y;phi).

Your method, on the other hand, seems to diverge significantly from EM. There is no full E-step. No full posterior sampling flow given phi, and furthermore, all the iterations that do not enter line 282 in Alg 1 appear to be iterations of prior sampling (no guidance) and not posterior sampling. Furthermore, in EM, the iterations are applied to the same likelihood p(y;phi), but in your scheme the likelihood p_t(y;phi) changes at each iteration as the index t changes.

Therefore, the proposed method is at most an EM-inspired technique rather than EM or approximate EM.

To conclude, due to some improvement compared to the first version I sightly increased the score. But I am still concerned about the presentation, the heuristic nature of the method (it does not inherit EM properties and lacks theoretical analysis) and its novelty compared to exiting works.

Dear Reviewer E53D,

We sincerely thank you for your valuable feedback, time, and efforts. During the rebuttal period, we are happy that most concerns have been addressed, which not only strengthened the paper but also led to positive reevaluation and score improvements from other reviewers. We hope our responses have fully addressed your concerns as well. If not, we are eager to address any remaining concerns, and provide further clarifications or additional experiments as needed. A prompt response would be greatly appreciated to help us make timely improvements. We sincerely look forward to receiving your updated feedback.

Best,
Authors

We thank the reviewer for their constructive suggestions and thoughtful feedback. We are pleased that the reviewer recognizes the broad interest of our research problem across the machine learning and signal processing communities. Below, we provide detailed explanations and clarifications for the concerns raised:

1. Annealing Consistency and the noiseless condition

Empirical evidence and theoretical analysis of the annealing consistency technique are presented in Figure 5 and Appendix B of the original paper. This technique significantly enhances imaging quality by adaptively balancing the weights between data fidelity and generative priors based on model accuracy. Regarding the noiseless condition ($\sigma=0$), we use a small virtual uncertainty (e.g., $\sigma=1e-2$) to ensure sampling stability. This approach is consistent with prior work, including the classical DPS and LatentDPS algorithms.

2. Role of “Gluing” regularization vs. Annealing Consistency

The effectiveness of the proposed annealing consistency technique is experimentally demonstrated in Figure 5 of the original paper. The second column (gluing without annealing) is compared to the third column (both gluing and annealing), showing that annealing consistency clearly improves performance. We agree that an isolation study would better quantify the individual contributions of “gluing” regularization and “annealing consistency.” According to your suggestion, we included additional experiments in the updated manuscript (Appendix F), removing the gluing term while retaining annealing consistency. Results show that both techniques independently improve reconstruction, but their combination achieves the best performance.

3. More discussions on baseline performance

   1. Performance Difference on FFHQ vs. ImageNet: The FFHQ dataset, dominated by human faces, has a simpler and more consistent distribution compared to the diverse and complex ImageNet dataset. This complexity makes it harder for BlindDPS to estimate motion blur kernels on ImageNet, consequently influencing the imaging quality. Notably, BlindDPS only reports results on FFHQ in their original paper, likely due to these challenges.

   2. FID Metric Comparison: As requested, we computed the FID score for our method and compared it with FastEM and BlindDPS (below table). The results confirm our method's superiority.

   	FastEM	BlindDPS	Ours
   FID Score (ImageNet)	58.91	53.35	49.21

   3. Performance of FastEM: The original FastEM paper does not specify the size of the blurring kernels used. To address this, we tested FastEM across various settings. Our experiments reveal that FastEM is unstable when dealing with large or complex kernels and does not improve a lot from the initial observations, as the reviewer pointed out. The results reported in the original FastEM paper primarily focus on simple kernels, whereas our experiments use larger kernels, which is consistent with BlindDPS’s settings. Additional results with small kernels are included in the revised manuscript (Appendix F, Fig. 15), and these findings do not influence our conclusions.

4. Parameter $\phi$ Across Applications

Parameter $\phi$ represents any parameter that defines the forward model, such as blurring kernels in deblurring, masks in inpainting, frequency masks in compressed sensing, or scanning angles in computed tomography. We will add further explanations in the revised manuscript.

5. Generalization of the PnP Denoiser

The PnP denoiser trained on Gaussian noise can serve as a general regularizer for inverse imaging problems, as theoretically and experimentally proved in prior works [1, 2]. Grounded in the theory of denoising score matching [3], Gaussian denoisers capture a wide range of image priors, from low-level properties (e.g., sparsity, smoothness) to high-level features (e.g., semantic and structural coherence). Due to the page limit, we could not provide detailed explanations in the original paper. We are happy to include them in the revised version.

6. Comprehensive list of hyperparameters

We have added a comprehensive table of tunable hyperparameters for our method, as well as those used by FastEM and BlindDPS, in the updated manuscript. Key differences are summarized below:

7. Network architecture and experimental setup for Table 1

The E-step used the Stable Diffusion v-1.5 model, as described in Sec. 5.1 of the original paper. The M-step employs a DnCNN model with 5 blocks and 32 channels[4], as explained in Appendix C.2.

Additional experimental details for Table 1 are included in Appendix C, and we followed official implementations for BlindDPS and FastEM. We will add further clarifications to the revised manuscript.

8. Other minor points

We have updated citations, initialization details, and typographical errors as per your suggestions. There is no difference between Equation (11) in our paper and Equation (24)/Equation (B.9) in FastEM. Additionally, we have incorporated GibbsDDRM [5] into the literature review and performed additional experiments using it as a new baseline (Appendix F).

We thank the reviewer once again for their insightful suggestions. We hope our responses address all concerns and are happy to provide further clarifications in additional discussions. Thank you!

Reference

[1] S. V. Venkatakrishnan, et al. Plug-and-play priors for model-based reconstruction. IEEE GlobalSIP, 2013.

[2] Yu Sun, et al. An online plug-and-play algorithm for regularized image reconstruction. IEEE TCI, 2019.

[3] Vincent P. A connection between score matching and denoising autoencoders. Neural computation, 2011.

[4] Kai Zhang, et al. Plug-and-play Image Restoration With Deep Denoiser Prior, TPAMI 2021.

[5] Murata, et al. Gibbsddrm: A partially collapsed Gibbs sampler for solving blind inverse problems with denoising diffusion restoration. ICML, 2023.

Dear Reviewer 6ZdJ,

We sincerely thank you for your valuable feedback, time, and efforts. During the rebuttal period, we are happy that most concerns have been addressed, which not only strengthened the paper but also led to positive reevaluation and score improvements from other reviewers. We hope our responses have fully addressed your concerns as well. If not, we are eager to address any remaining concerns, and provide further clarifications or additional experiments as needed. A prompt response would be greatly appreciated to help us make timely improvements. We sincerely look forward to receiving your updated feedback.

Best,
Authors

Thanks for your kind response! We look forward to your final evaluation.