PnP-Flow: Plug-and-Play Image Restoration with Flow Matching

Q1. Indeed, our images tend to be slightly oversmoothed, but we do not see it as a weakness, but rather inevitable due to the distortion perception tradeoff. Our method seems to be more on the “distortion” side of the perception/distortion tradeoff (implying high PSNRs and smoother images). As shown in [4], a method cannot consistently yield high perception and low distortion images, therefore one should generally not expect one method to beat all the others in both perceptual and distortion metrics.

Q2. Indeed, our method is the cheapest among the benchmarked flow matching methods. This is because we only need (few) evaluations of the velocity field. D-Flow needs to backpropagate through the ODE solution (very memory intensive) and flow priors needs to compute the trace of the flow’s jacobian. OT-ODE requires to solve a linear inverse problem at each iteration.

[4] Blau, Michaeli, The perception-distortion tradeoff, CVPR 2018

Q1. We computed the LPIPS metric, see Appendix A.10. We have competitive performance, with no method consistently leading the LPIPS metric. Our method seems to be more on the “distortion” side of the perception/distortion tradeoff. As shown in [4], a method cannot consistently yield high perception and low distortion images, therefore one should generally not expect one method to beat all the others in both perceptual and distortion metrics.

Q2. We think this is an interesting suggestion, however in this paper we want to focus on pretrained flow matching/diffusion/denoising models since they do not rely on paired data and are not task-specific. If we understand your suggestion correctly, a U-Net would need to be retrained for changes in the forward operator or degradation noise model.

Q3. We appreciate the suggestion. A new visualization can be found in Appendix A.9, Figure 13.

[4] Blau, Michaeli, The perception-distortion tradeoff, CVPR 2018

Q1. We added comparisons with DPS and DDRM in Appendix A.13 (Table 10&11). DDNM seems like an very interesting approach for future research: indeed, we could use a similar null space decomposition and only perform our proposed PnP Flow within this null space. We add a discussion on DDNM and related diffusion methods in the appendix, see Appendix A.12.

Q2. Yes, our method is adaptable to the blind case and we thank the reviewer for this interesting suggestion. We demonstrate this on a preliminary experiment on blind deconvolution and blind inpainting where we only assume to have access to the observation and the pre-trained model, without additional data. Then we iterate (over time) between our algorithm steps and update steps on a randomly initialized kernel/mask. We see that the reconstruction is of higher quality than the blurry measurement. See the promising results in Appendix A.14.

Thank you for your review. We will address your comments on the numerical section more thoroughly in our next response. It seems that you found certain parts of the paper unclear, while the other reviewers considered the presentation to be good. Could you kindly provide specific points where the explanation was unclear, and indicate what additional details you believe are necessary? For reference, the function F (which is the data-fidelity term) is defined on line 45, and the steps of the algorithm are detailed in Section 3.2. We look forward to making the necessary revisions to enhance the clarity of the paper.

Regarding Weakness: W1 and W2. Since all other reviewers found the presentation good (rated 3), we would appreciate a clarification on what parts you find need more explanation. Without your feedback, we cannot improve the presentation.

Regarding the questions:

Q1. We implemented a super-resolution task with a Laplacian noise degradation model (which is heavy tailed and therefore challenging). Some of the Flow-Matching based methods were not applicable as they are designed for Gaussian noise (D-Flow, OT-ODE), but we benchmarked against PnP Diffusion, PnP GS, Flow priors. Evaluation of the methods on 100 test images are reported in the Table below and in Appendix A.11 (Table 9) with more details.

Q2. We added DDRM and DPS to our benchmark. For the results, see Appendix A.13 (Table 10&11).

Q3. While we agree that this would be a very interesting experiment, we are unfortunately unaware of any publicly available unconditional pretrained flow matching model for ImageNet. Training one from scratch would require resources beyond our current capacity.

Q4. Zooming in the images of Fig. 3 and 4 in the paper or Fig. 7 in the appendix, one can actually see important visual differences, which are discussed in Section 5.3. Note that none of the competing methods performs as consistently across all tasks as our method.

Q5. In the revised version of the paper, we conduct new blind-restoration experiments. We adapt our method PnP-Flow to cases of unknown blur or unknown mask without any additional data, yielding excellent visual results. Details and results have been added in Appendix A.14.

Q6. We did not observe failure cases for our method. As opposed to the benchmarking methods, our method is not prone to artifacts and other kinds of failures, although it tends to produce averaged results (slightly over-smoothed), consistent with its interpretation as a minimum-mean-square-error (MMSE) estimator. Interestingly, a competing method like OT-ODE may generate more realistic textures but often produces highly artifacted and distorted images, as shown in Figure 7 in the appendix.

Q7. Benchmarks were conducted exclusively for Flow-Matching-based methods, as they share the same network architecture. A comparison with diffusion methods would not be relevant here.

For all the interested readers we list our contributions below:

1. We proposed a Plug and Play version of Flow Matching, which introduces a time dependent denoiser based on the velocity from flow matching.

2. We compared extensively against other flow matching (and diffusion) methods, which did not have available code. We make the comparisons public.

3. We show that our proposed method has theoretical advantages over diffusion (straight paths), and practical advantages (non Gaussian latent).

4. During the rebuttal, we also added more diffusion baselines, and showed that our method can be applied to blind inverse problems and non-Gaussian noise.