InverseBench: Benchmarking Plug-and-Play Diffusion Priors for Inverse Problems in Physical Sciences

We thank the reviewer for the positive feedback and constructive suggestions. We address the question below and incorporate the reviewer's suggestions below.

W1: We have updated Figure 2 with zoom-in regions and error maps for MRI reconstructions to highlight the differences between them.

W2: The typos are corrected in the revised version.

Q1: For most algorithms, the reported results, including those in Table 4, are based on a single posterior sample for each test instance. Specifically, we obtain one sample from the algorithm per test instance and compute the mean and standard deviation of the evaluation metrics overall test samples. The only exceptions are ensemble-based methods such as EnKG and EKI, which operate with an ensemble of particles. For these methods, we compute the average error across the ensemble members for each test instance and then report the aggregate statistics over the test set.

W1: We thank the reviewer for the feedback. In response, we highlight two examples from our benchmarks, DAPS and PnP-DM, that illustrate the nuances of performance differences between natural image restoration tasks and the scientific inverse problems.

Both DAPS and PnP-DM achieve near state-of-the-art results on natural image restoration tasks such as inpainting and super-resolution. However, their performance significantly deteriorates in problems like Full Waveform Inversion (FWI) in InverseBench. This issue stems from their inability to account for the stability conditions required to query the forward model. For example, the numerical PDE solvers used in FWI and the Navier-Stokes equations impose strict stability constraints, such as the Courant–Friedrichs–Lewy (CFL) condition. The Langevin Monte Carlo (LMC) subroutine used by DAPS and PnP-DM introduces noisy perturbations in the input. While such noise is tolerable in natural image restoration tasks, it violates the stability conditions in these scientific inverse problems, resulting in numerical instability and unreliable gradient estimates. We discuss this in detail in lines 435–451 of the paper, where we also provide examples of these instabilities.

Q1: Thank you for pointing this out; your understanding is correct. We employ a separate model, initialized with different random seeds, trained on a dataset of 50,000 non-publicly available images. This model is used to unconditionally generate 5 images for the validation dataset and 100 images for the test dataset. These generated datasets are utilized for hyperparameter tuning and evaluation.

Q2: We certainly agree that investigating robustness is very important. As alluded to by your question, there are two main types of robustness that are important to study: prior mismatch and forward model G mismatch. We have added a discussion with some of our preliminary results on this research direction in Appendix B.

Q3: This is a good question. We have experimented with FFHQ and ImageNet pretrained priors on black hole imaging and inverse scattering tasks. Our findings are summarized as follows: the more ill-posed a problem is, the more critical the role of the prior becomes.

Because black hole imaging is highly ill-posed, combining the FFHQ pretrained prior with PnP diffusion methods produces an image resembling a human face while still satisfying the observational black hole data with a $\chi^2$ value between 1.1 and 1.5. However, these images significantly diverge from the true black hole images, with a substantial PSNR drop of about 5–8 dB compared to in-distribution data priors. However, for the less ill-posed inverse scattering task, comparable results can be achieved between the baseline and the PnP diffusion with in-distribution priors. The table below presents the evaluation results for inverse scattering on 100 test samples. We can observe that both the FFHQ and ImageNet priors improve the results compared to the baseline, which uses only total variation (TV) as the prior.

An important implementation detail to highlight is the mismatch between the dimensions of the priors and the dimensions of the tasks. For example, the typical dimension of an ImageNet pretrained prior is (3, 256, 256), while the dimension for inverse scattering is (1, 128, 128). To resolve this mismatch, we introduced a wrapper before the forward function of each task to map the prior dimensions to the task dimensions. Specifically, we downsampled the images and converted them to grayscale, aligning the (3, 256, 256) prior dimensions with the (1, 128, 128) task dimensions used in inverse scattering.

We appreciate the constructive feedback and references. We address the reviewer's questions and concerns below.

W1 & Q1: To ensure sufficient tuning of the hyperparameters for each algorithm, we employ a two-stage strategy combining grid search with Bayesian optimization and an early termination technique, using a small validation dataset. Specifically, we first perform a coarse grid search to narrow down the search space and then apply Bayesian optimization. For problems where the forward model is fast, such as linear inverse scattering, MRI, and black hole imaging, we conduct 50-100 iterations of Bayesian optimization to select the best hyperparameters. For computationally intensive problems such as full waveform inversion and Navier-Stokes equation, we use 10-30 iterations of Bayesian optimization combined with an early termination technique (Hyperband stopping algorithm [1]). We have added a discussion on hyperparameters and initialization to Appendix A.4.2 in the revised version.

W2: In general, we observe three key aspects that can influence out-of-distribution performance: (1) the ill-posedness of the task, (2) the relevance of the trained prior, and (3) the balance between the prior and observation in the sampling algorithm. Tasks that are more ill-posed, use less relevant priors, or involve fewer data gradient steps typically exhibit worse generalization performance. Additional discussions on this topic are provided in Appendix B’s robustness section.

W3: We agree that the naming can be confusing since one of the methods has “Plug-and-Play” in its title. Nonetheless, this general class of methods is commonly known as plug-and-play methods [2].To avoid confusion, we use different fonts when referring to a specific algorithm like PnP-DM in the main text throughout the paper (e.g. line 442). We have added a clarification on the difference between PnPDP and PnP-DM in lines 155-156 to avoid confusion.

Q2: We appreciate the reviewer’s observation regarding the relationship between hyperparameter space and the relative ranking of algorithms. As noted in our response to W1, we employ a tuning strategy to ensure fair comparisons and sufficient optimization for all methods.

The stronger performance of RED-diff, DiffPIR, DAPS, and PnP-DM can likely be attributed to their algorithmic design, which leverages more sophisticated mechanisms that inherently require more parameters to tune. However, a larger hyperparameter space does not always guarantee better performance. For example, RED-diff, PnP-DM, and DAPS perform poorly on full waveform inversion as shown in Figure 1. This suggests that the relationship between the size of the hyperparameter space and performance is task-dependent and not straightforward. While larger parameter spaces can offer greater flexibility, performance ultimately depends on how well the algorithm aligns with the structural requirements of each specific inverse problem.

Q3: Thanks for bringing these works to our attention. We have included these papers in the discussion on related works for MRI in Appendix A.2.2.

[1] Li, Lisha, et al. "Hyperband: A novel bandit-based approach to hyperparameter optimization." Journal of Machine Learning Research 18.185 (2018): 1-52.

[2]: Venkatakrishnan, Singanallur V., Charles A. Bouman, and Brendt Wohlberg. "Plug-and-play priors for model based reconstruction." 2013 IEEE global conference on signal and information processing. IEEE, 2013.

I would like to thank the authors for their efforts in the rebuttal. Your answers and further explanations have effectively addressed my initial questions. However, one additional point slipped my mind when I was drafting my earlier reviews: How do you ensure that the priors used for comparison across all methods have the same capability? I am sorry for the delay in raising this question.

W1 & Q1: Thank you for your constructive feedback. We agree that metrics like PSNR and SSIM do not fully reflect the quality of MRI reconstructions, especially for diagnostic purposes. Therefore, we propose to quantify the degree of hallucination by employing a pathology detector and calculating the mAP50 metric over reconstructions. The results are summarized in the table below. For each method, we report the Precision, Recall, and mAP50 metrics for detection, and PSNR, SSIM, and Data Misfit for reconstruction. We also provide the rankings based on mAP50 and PSNR. Overall, the two rankings are correlated, which means that better pixel-wise accuracy indeed leads to a more accurate diagnosis. However, there are a few algorithms for which the two rankings disagree: Residual UNet, Score MRI, and RED-diff. The best methods are rE2E-VarNet and PnP-DM.

Details: 1) The detector is finetuned based on a medium-sized YOLOv11 model on a training set of fully sampled images with the fastMRI+ pathology annotations (22 classes in total) [2]. 2)The detection metrics are calculated based on 14 selected volumes with severe knee pathologies, which lead to 171 test images in total.

W2: We appreciate the reviewer’s feedback on the scope of our work. While we acknowledge that InverseBench focuses on a subset of scientific inverse problems, we note that the selected problems span very diverse scientific domains, including black hole imaging, seismology, optical tomography, medical imaging (MRI), and fluid dynamics. Importantly, these tasks were chosen to represent a broad spectrum of problem characteristics encountered in scientific inverse problems, as outlined in Table 2.

Regarding the reviewer’s question about “physics-heavy”, we note that most scientific inverse problems inherently involve physical modeling, as they often arise from real-world systems governed by physical laws. Even in the case of MRI, the forward model is based on the Bloch equations, which are also from physics principles.

If the reviewer’s concern relates to the emphasis on problems with explicit forward models for measurements, we could clarify this scope in the title by replacing “Scientific Inverse Problems” with “Model-based Scientific Inverse Problems.” We welcome the reviewer’s feedback on this proposed revision.