FIG: Flow with Interpolant Guidance for Linear Inverse Problems

We thank reviewer iHSi for the valuable feedback. We address your concerns and questions below.

QD1: Novelty

We proposed a new measurement interpolant guided scheme for solving linear inverse problems using diffusion and flow based prior models. Different from the existing gradient-based algorithms such as DPS, DDNM, OT-ODE, and $\Pi$GDM, our approach avoids the time-consuming backpropagation through the score or flow function. We also would like to point out that the linear inverse problems with extremely noisy measurements is still very challenging for existing algorithms. For example, in Section 4, we included experiments with

• 4x super resolution with high-noise $\sigma_n = 1$, comparing to the commonly used $\sigma_n = 0.05$
• 16x super resolution with noise $\sigma_n = 0.2$.

Our algorithm with flow matching prior model outperforms the baselines on those challenging tasks (see Table 2, Figure 4, and Figure 5).

QD2: Experiments

We thank the reviewer for pointing out the paper GDP to us. However, we choose not to include it as our benchmark due to two reasons. First, GDP has slightly worse results than DDNM in terms of evaluation metrics on super resolution$\times$4 and colorization on ImageNet. As we have already used DDNM for our benchmark, we don’t think it is necessary to compare it with GDP. Second, GDP focuses on noiseless measurements. As our paper investigates noisy cases, GDP does not serve as an appropriate benchmark.

We added a discussion on GDP in Section 2.3. GDP considered blind image reconstruction tasks, which is an important future extension of FIG. We highlighted this in Section 5 and cited the paper.

QD3: Effectiveness of the measurement interpolation

We believe that the effectiveness of our method compared to various benchmarks is fully demonstrated in Table 1 and 2. We show that FIG achieves SOTA among different tasks and noise levels. As the measurement interpolants are incorporated via gradient descent, its effectiveness is revealed in an ablation study on gradient descent steps $K$ in Appendix C.1. Please let us know if there are other ablation studies we can carry out to better address your concerns.

QD4: Blind scenario

Thanks for pointing out this interesting research direction. Consider the task of blind motion deblurring, where we have diffusion or flow matching priors on both the image and the blurring kernel. When fixing either the kernel or the image, the noise model is linear, thus measurement interpolants can be constructed. We believe using FIG’s gradient descent updating scheme consecutively on the image and kernel would lead to solving the problem. However, we think that this extension is not trivial and is beyond the scope of this paper, as the title indicates our focus on linear inverse problems. We highlight this as an intriguing direction for future research in Section 5.

QD5: Other base models

Thanks for the suggestion. Currently, we have not conducted experiments on latent models, as base models are not the primary concern of our paper. In fact, to the best of our knowledge, none of the related works could be directly applied to latent diffusion or flow matching models. They all need to be redesigned specifically to accommodate the encoder-decoder structure and latent space. We agree that latent models present a promising direction, and we are eager to explore the application of our algorithm in this area in future works.

Thanks a lot for your time and valuable feedback. We hope that our response and the updated paper answered your questions. Please let us know if you have any further questions or comments. Thank you once again, and we look forward to your response.

We thank reviewer 9aCd for the valuable feedback. We address your concerns and questions below.

QC1: Potential extensions

Thanks for the suggestions. Our motivation of FIG comes from the observation that, although various algorithms are proposed to solve non-linear inverse problems with potential extension to latent diffusion or flow matching models, none of them could efficiently handle linear image restoration tasks, especially under high noise. We deem the noisy linear case an important problem to be solved before considering more complicated scenarios. To this end we propose FIG that focuses on solving linear inverse problems.

As we mention in Section 5, our algorithm does not directly extend to non-linear measurements or latent encoding. The reason is that we require the likelihood of the measurement interpolant $y_t$ given the diffusion or flow matching prior $x_t$, which is intractable for general non-linear measurements and latent encoding. However, there is a class of nonlinear inverse problems that FIG could potentially be applied to, namely the bilinear problems such as blind deconvolution. Consider the task of blind motion deblurring [1], where we have diffusion or flow matching priors on both the image and the blurring kernel. We believe using FIG’s gradient descent updating scheme consecutively on the image and kernel would lead to solving the problem. However, we think that this extension is not trivial and is beyond the scope of this paper, as the title indicates our focus on linear inverse problems. We highlight this as an intriguing direction for future research in the updated Section 5.

Reference:
[1] Chung, Hyungjin, et al. “Parallel diffusion models of operator and image for blind inverse problems. IEEE.” In CVPR, 2023.

QC2: Additional ablation studies on rescaling

We have added Figure 9 for a visualization of the effect of $w$ in Appendix C. Fig. 9 gives a visual illustration of effects on different $w$ for super resolution on CelebA with flow matching prior. It’s clear that $w$ affects the performance of our algorithm and there exists an optimal $w$.

QC3: Additional comparison

We have added a new baseline C-$\Pi$GDM under the recommendation of Reviewer V1uU. Please see the response to QB for more details. In our implementation, we successfully reproduced results in the C-$\Pi$GDM paper for noiseless measurements. The noisy measurement results indicate that C-$\Pi$GDM, although being highly efficient, struggles to deal with measurement noise.

QC4: The paper mentions the possibility of overfitting when applying FIG. Could the authors elaborate on this aspect, providing insights into how to mitigate this risk?

Thanks for pointing this out. By overfitting, we refer to situations when the algorithm learns the pattern in the noisy measurement, see Fig. 7d. We find that by tuning the gradient descent steps $K$ and the learning rate $c$, overfitting can be easily avoided. Please see Appendix C.1 for intuitive figure illustrations.

Thanks a lot for your time and valuable feedback. We hope that our response and the updated paper answered your questions. Please let us know if you have any further questions or comments. Thank you once again, and we look forward to your response.

We thank reviewer V1uU for the valuable feedback. We address your concerns and questions below.

QB1: Novelty and Difference from DPS.

We kindly disagree with the reviewer’s assessment that “this work adheres to the scheme outlined in the DPS paper.” We would like to emphasize that our method FIG is completely different from DPS. At $x_t$, DPS uses Tweedie’s formula to estimate $\hat{x}_0(x_t)$, and differentiate the loss $\Vert y - A\hat{x}_0(x_t)\Vert_2$ to update $x_t$. This involves differentiating through the score or velocity network due to the Tweedie term $\hat{x}_0(x_t)$. In FIG, we use the forward process to interpolate the measurement $y$ to measurement interpolants $y_t$, which is detailed in Section 3.1. We then use the loss $\Vert y_t - Ax_t\Vert_2$ to update $x_t$, which does not require differentiating through the score/velocity network. Equation (16) makes our point self-evident. Furthermore, FIG and DPS yield different reconstruction images, evaluation scores, and runtime efficiency as shown in Section 4. This is clear evidence that FIG is different from DPS.

In FIG, we proposed a new measurement interpolant guided scheme for solving linear inverse problems using diffusion and flow based prior models. We also would like to point out that the linear inverse problems with extremely noisy measurements is still very challenging for existing algorithms. For example, in Section 4, we included experiments with 4x super resolution with high-noise $\sigma_n = 1$, comparing to the commonly used $\sigma_n = 0.05$ 16x super resolution with noise $\sigma_n = 0.2$. Our algorithm with flow matching prior model outperforms the existing algorithm on those challenging tasks (see Table 2, Figure 4, and Figure 5).

QB2: This paper seems primarily focused on adapting DPS to flow matching models.

As is explained in QB1, FIG is different from DPS. Moreover, we implemented DPS with flow matching models as a baseline for comparison. Appendix E details how we implement DPS-Flow.

QB3: Theorem 1 has been discussed in previous work.

We have acknowledged previous works that provide similar proofs in Appendix A.1 in the original paper. However, since FIG’s conditioning is different from previous methods, we believe that showing the dynamics (Theorem 1) of FIG in the main paper is important. As mentioned in Section 3.3, the FIG updating scheme is a numerical discretization of Eq. (20) that corresponds to the dynamical property in Eq. (17), thus suggesting the importance of its justification in Theorem 1. We believe that delaying Theorem 1 to the Appendix will affect the completeness of the main paper.

QB4: The paper omits mention of highly related work, such as C-PG(D/F)M.

Thanks for pointing out. We have added the C-$\Pi$GDM baseline to the experiments (Section 4) and related works (Section 2.3). We are aware of the fact that C-$\Pi$GDM includes a flow matching version. Yet due to the fact that only the diffusion version is released in the corresponding Git repo, we added the diffusion version to our baseline. In our implementation, we successfully reproduced results in the C-$\Pi$GDM paper for noiseless measurements. The noisy measurement results indicate that C-$\Pi$GDM, although being highly efficient, struggles to deal with measurement noise.

QB5: Efficiency

As is explained in the answer of QB1, FIG is efficient as it does not need to differentiate through the pre-trained neural net. Table 3 empirically demonstrates FIG’s efficiency. We acknowledge that C-$\Pi$GDM is, to the best of our knowledge, the most efficient algorithm for image inverse problems with diffusion or flow matching prior. They cleverly adopt a strategy to warm start from $t=0.4T$ with specific initializations, substantially reducing the NFEs. Empirically, we observe that although C-$\Pi$GDM is very fast, their reconstruction qualities are worse than other baseline algorithms for Gaussian deblurring and Inpainting with noisy measurements (see the updated Table 1).

QB6: Evaluation Metrics.

We reported the FID and KIDs and discussed the recovery-perceptual trade-off in the newly added Appendix D. In noisy image reconstruction tasks, dismissing PSNR and SSIM undervalues their critical role in assessing reconstruction fidelity. These metrics ensure pixel-level accuracy, essential for recovering precise details from degraded inputs. While FID and KID capture perceptual realism, they may miss structural details in high-noise scenarios or misrepresent results if reconstructed images deviate from the ground truth since they measure distance in feature space. Also, from the reported perceptual metrics in the newly added Table 10, we observe that our algorithm delivers the best recovery metrics and also achieves comparable perceptual scores.

We thank reviewer BWWP for the valuable feedback. We address your concerns and questions below.

QA1: What limitations exist in how other methods condition on measurements?

We find methods of high reconstruction quality are often time consuming, and efficient methods fail to consistently obtain high quality reconstruction throughout different tasks. Furthermore, none of the existing methods are able to handle challenging scenarios like high measurement noise. Thus we see FIG, an algorithm that could efficiently handle both easy and hard image restoration tasks, as a significant improvement for linear image restoration with diffusion/flow matching prior. Please refer to the updated Section 2.3 for more detailed discussions on the limitations of existing methods.

QA2: How measurement interpolants address those limitations? Why is conditioning on interpolants theoretically or empirically advantageous?

By incorporating measurement interpolants defined in Eq. (13), FIG’s conditional updating scheme only involves simple gradient descent on an L2 loss of a linear function of the signal (Algorithm 1 step 7-9), which substantially improves the efficiency. The experiments show that the simple updating scheme of FIG is not only effective for low noise tasks, but robust to high noise as well.

Theoretically, we show in Eq. (20) that the updating scheme is a numerical discretization of a probability flow ODE, whose corresponding dynamics is shown in Theorem 1, justifying the consistency of our algorithm. Corollary 1 demonstrated that the FIG updating direction is equivalent to maximizing a regularized log-likelihood, providing further intuition of the algorithm.

QA3: I would like to know the motivation for the measurement interpolants. I have an intuition of why they are used, but I could not find that in the paper and I would like the authors to explain; More precisely, I am asking what was wrong with the conditioning of other methods and how this new approach makes that right.

The intuitive idea of measurement interpolants comes from filtering [1], such that the measurement $y$ is interpolated to mimic an observation process {$y_t$}$_{t\in[0,T]}$. However, traditional filtering methods such as Sequential Monte Carlo are extremely slow, and corresponding empirical performances fail to match SOTA baselines. In view of the drawbacks, we consider using gradient descent instead. We also want to point out that the conditioning of other methods is not wrong, they are just different ways of incorporating the measurements into the prior model. See the last paragraph in Section 2.3 for a comprehensive review of existing methods. Comparatively, FIG’s way of incorporating measurement information is empirically shown to be efficient and robust to measurement noise, demonstrating an advantage on existing benchmarks.

QA4: I would like to have an explanation for what each of the theorems contributes. Possibly this is connected to the previous point. I suggest to 1) add a brief paragraph after each theorem explaining its practical implications; 2) include a discussion subsection that ties together how the theorems collectively justify or enable the proposed method; 3) provide intuitive explanations or examples to complement the formal mathematical statements.

Thanks again for the suggestions. We’ve updated the paper accordingly in Section 3.3.

Reference:
[1] Dou, Zehao, and Yang Song. "Diffusion posterior sampling for linear inverse problem solving: A filtering perspective." The Twelfth International Conference on Learning Representations. 2024.

Thanks a lot for your time and valuable feedback. We hope that our response and the updated paper answered your questions. Please let us know if you have any further questions or comments. Thank you once again, and we look forward to your response.