Solving Inverse Problems with FLAIR

We thank the reviewer for acknowledging the novelty of our variational formulation, the relevance of our work in light of recent developments in large-scale latent diffusion models, the high quality of our results and the comprehensiveness of our experiments . Your feedback is much appreciated and we we incorporated it into our manuscript. If our answers are satisfying, we kindly ask to consider raising your score.

1. Similarity to RED-diff

The cited equations are part of our background section. We do not claim them as contribution, but they are important to understand the method. One of our contributions lies in the derivations of the variational objective for flow matching models stated in Propostion 1. The derivations can be found in the appendix A.1-A.3. We also compare to Repulsive Score Distillation (RSD), which adapts and extends RED-diff to latent diffusion models. We outperform RSD on all relevant metrics, which shows how valuable our contributions are. We will emphasize our contributions to the variational objective for flow matching models more clearly in the final version.

2. Zero variance of variational distribution

One can think of the new variational distribution as particle approximation of the posterior. In our experiments we only used one particle. With the particle approximation the gradients of the regularizer can be approximated using Wasserstein Gradient Flow and equation (7) is not required. However, eq. (7) motivates the weighting of the regularizer with the conditional flow matching loss for the CRW.

3. Source of Posterior Variance

While the variational objective remains fixed, noise vectors are drawn randomly during the optimization which leads to a different optimization trajectory. This is analogous to how different random seeds with the same text prompt yield diverse outputs in generative models like Stable Diffusion. In our case, these different trajectories correspond to different posterior modes.

4. Non linear forward operators

As you said the decoder, which is part of the forward operator for latent models makes all the tested tasks non-linear forward operators. We added the task of non-linear deblurring. The non-linearity lies in the non-linear conversion from sRGB to linear color space and back. The blur opoerator is applied in the linear color space. Additionally, values are clipped after blurring simulating saturation effects.

5. Stochasticity of HDC Thank you for spotting this error. The optimization of the data term is not stochastic. The optimization of the variational objective however is due to the random sampling of noise vectors.

6. Overfitting to noise in HDC

The optimization of the data term can be stopped early for noisy observations as it is the case for our experiments. As mentioned in A.4 we stop at an error of less than $1\times 10^{-4} \cdot \text{len}(y)$. However, for tasks like inpainting, absence of measurement noise is realistic.

7. Claim that FLAIR is able to recover atypical modes

This is a conclusion we draw from the observation that FLAIR reconstructs diverse samples (see Fig. 3) which also look realistic. As shown in Fig. 3 baseline approaches collapse to a single solution which is close to the mean, but does not look realistic. Under the unconditional prior all modes are unlikely. One has to condition the model very strictly to increase the likelihood for certain modes e.g. with very precise text prompts which is not feasible for degraded images. Our solution is to compute “special noise vectors” $\hat{x}$ to condition the model and increase the likelihood of modes which match the measurement.

8. Expectation over time t

You are correct. Thank you very much for pointing out that error. We will corrected it in the manuscript.

9. Why does the model perform better without HDC, DTA, CRW?

At this point we also want to emphasize that PSNR and SSIM are not ideal metrics to access the quality of the reconstructions, because they favor a MMSE prediction which is overly smooth and unrealistic. Consequently, any plausible texture shift, while perceptually correct, incurs a large $\ell_2$ penalty and reduces PSNR/SSIM. LPIPS on the other hand is able to measure distances between images in a way that correlate with the human perception

When deactivating our contributions the method severely degrades terms of LPIPS. The explanations is that the results collapse to a posterior mean similar to the baseline methods. This mean-prediction achieves a higher PSNR, but is perceptually farther from the ground truth and achieves worse LPIPS, as the respective qualitative results in the ablation study show.

To highlight that our model can also be used to obtain good MMSE estimates, we also conducted ensemble predictions by running posterior sampling 8 times and averaging the result. The results show that ensembling increases PSNR values, but reduces LPIPS and confirms that our samples are indeed distributed around the posterior mean and that results very close to the posterior mean like the baseline methods are perceptually further away (LPIPS) from the ground truth compared to our samples. We included the ensembling experiments in our supplementary material.

80 samples of DIV2K

100 samples of FFHQ

10. HDC ablations settings

Without HDC we perform only 1 step on the data term similar to some of the baselines. We will clarify this in the manuscript.

11. What is $\nu=0.01$ in Table 1? In Line 204 you write that you use an observation noise of 0.5%

Many methods specify the noise in absolute terms without mentioning the data range. We use 0.5% of the data range as noise which corresponds to 0.01 on a range from -1 to 1. We will be consistent in the final version.

12. Reduction of NFEs

Our method can also be deployed with a reduced number of data term steps (e.g., maximum of 5), if a faster runtime is required. Furthermore, one can switch to a "tinyVAE" version of SD3.

To validate that the usage of the tinyVAE or less steps does not degrades the performance noticeably we also provide a metrics for x12 Super Resolution on 100 samples of FFHQ:

In our manuscript we set the number of NFEs to a fixed value for all methods and baselines for an easier comparison. For different tasks the required number might be lower or higher. While runtime optimization is out of scope for this project we want to highlight that the number of NFEs needed does not only depend on the posterior sampling method used but more so on the stochastic interpolant. For example RED-diff used 1000 NFEs for most of their experiments on a different model.

We thank the reviewer for acknowledging the novelty of our unified inverse-problem solver, the strong generalization of our flow prior across tasks, and the high quality of our results. We also appreciate your recognition of the importance of our ablations. Your feedback is much appreciated and has been incorporated into the manuscript.

If our answers are satisfying, we kindly ask to consider raising your score.

1. DTA resembles classifier-free guidance; CRW is a heuristic weighting fitted offline hence incremental novelty

We want to thoroughly highlight, that the proposed deterministic trajectory adjustment does not share any resemblance to CFG. However, CFG can be used to increase the prompt following and therefore condition the model more strongly on the given text prompt. In the inverse problem setting we usually do not have good text prompts or no text prompts at all because the input is degraded and potentially we do not know what the original image depicts. With the deterministic trajectory adjustment we propose to use a "special noise vector" $\hat{x}_1$, which additionaly conditions the generative model on the current optimization estimate $\mu$. This ensures that the optimization does not jump between possible modes and instead stabilizes the regularization term. Other variational approaches like RED-diff or RSD also have to chose a weighting for the regularizer. We are the first to propose one which is motivated by the accuracy of the regularizer.

2. Runtime Concerns

This is not a new drawback of our method but one that all training free posterior sampling methods share. We decided to run all inverse problems using the same amount of neural functions evaluations. For certain problems the required number could be higher or lower. The amount of NFEs depends also on the generative model used.

See a runtime and memory analysis below. Our hard data consistency can strongly influence the runtime and we also provide measurements with number of data term steps<=5 and a fast version using a "tinyVAE" of SD3.

To validate that the usage of the tinyVAE or less steps does not degrades the performance noticeably we also provide a metrics for x12 Super Resolution on 100 samples of FFHQ:

3. Pixel Space experiments

We thank the reviewer for the suggestion. Our focus was on high-resolution tasks and arbitrary scenes, where latent models such as SD3 offer clear benefits in terms of dataset scale and fidelity. Therefore, we did not include pixel-space experiments in the manuscript.

However, we agree that such experiments are valuable for completeness, and we now provide them in the table below. s shown, our method outperforms previous works also in pixel space, demonstrating its broader applicability.

4. Statistical Relevance

We want to highlight that our method is training free and that variance in the reconstructed images is intentional. We test all methods with the same random seeds and evaluate over 1000 samples for FFHQ and 800 for DIV2K. pFID is evaluated on patches of 256x256 which results in 9000 samples and 7200 samples for evaluation. In table 5 in A.8 shows the means and standard deviations over different samples.

For the rebuttal we also conducted experiments on 100 samples of FFHQ and 80 of DIV2K where we sampled 3 reconstructions per input for each method and computed the mean for each metric over all samples and the std of the means.

5. User studies

A user study is the gold standard for image quality assessment, but was out of scope for this project. However, in a training free approach there are limited parameters to tune and metric overfitting is usually not an issue. Specifically, we conducted a small grid for FLAIR and all baselines methods to select optimal learning rates. All other parameters were kept fixed. Furthermore, LPIPS is already a distance metric which aligns very well with the human perception. The "realness" of the images on the other hand is usually evaluated with the FID metric. At this point we also want to emphasize that PSNR and SSIM are not ideal metrics to access the quality of the reconstructions, because they favor a MMSE prediction which is overly smooth and unrealistic. Consequently, any plausible texture shift, while perceptually correct, incurs a large $\ell_2$ penalty and reduces PSNR/SSIM.

To highlight that our model can also be used to obtain good MMSE estimates, we also conducted ensemble predictions by running posterior sampling 8 times and averaging the result. The results show that ensembling increases PSNR values, but reduces LPIPS and confirms that our samples are indeed distributed around the posterior mean and that results very close to the posterior mean like the baseline methods are perceptually further away (LPIPS) from the ground truth compared to our samples. We included the ensembling experiments in our supplementary material.

100 samples for FFHQ and 80 for DIV2K

Furthermore a high PSNR does not mean that the results accurately follow the measurement. For x12 super resolution on DIV2K with FLAIR achieves 22.83 dB vs RSD with 23.45 dB. However, if we evaluate the PNSR in the measurement space to access how well the prediction follows the measurement y: PSNR(y, A($\hat{x}$)). FLAIR achieves 45.45dB in PSNR and RSD 40.28 dB. That means our prediction follows the measurement more closely even though the PSNR of the prediction compared to the ground truth is lower.

6. Non Gaussian noise and non-linear forward operators

The data term can also be adapted to non-Gaussian noise sources. E.g. Chung et al. Diffusion posterior sampling for general noisy inverse problems, ICLR 2023 elaborate on how to adapt the data term for Poisson distributed nose. We adapted the manuscript to clarify that our approach is not limited to only Gaussian noise models.

Because SD3 is a latent model, the non-linear decoder of the SD3-VAE is part of our forward operator. Therefore, all tasks we evaluated on are non-linear. We also conducted experiments on non-linear motion deblurring (non-linearity through saturation effects and gamma correction) see tables below:

Non-linear Motion Deblurring:

6. CRW compared to a linear $t$ weighting or a constant $\lambda$

Our CRW ablation is a comparison to $t$ weighting.

We thank the reviewer for acknowledging the strong perceptual performance of our method. Your feedback is much appreciated and we incorporated it into our manuscript. If our answers are satisfying, we kindly ask to consider raising your score.

Our work is positioned within the field of training-free posterior sampling using generative priors. Given that flow matching models represent the current state-of-the-art in image generation, we believe it is both timely and important to develop posterior sampling methods that are specifically compatible with these powerful generative priors.

1. Highlight insights and innovations

We have revised the manuscript to put more emphasis on clearly highlighting the core innovation of our approach.

• Variational posterior sampling for flow matching:

  To the best of our knowledge, we are the first to introduce a variational approach for posterior sampling with flow matching models. The derivation of this method is non-trivial and is provided in detail in Appendix A.1–A.3. We also compare to Repulsive Score Distillation (RSD), which adapts and extends RED-diff to latent diffusion models. We outperform RSD on all relevant metrics, which shows how valuable our contributions are.

• Deterministic trajectory adjustments:

  Our proposed deterministic trajectory adjustment is a crucial and novel component of the method, and our ablation studies show it yields the largest improvements in LPIPS. This adjustment is theoretically motivated and, to our knowledge, has not been previously explored in the literature.

• Model accuracy-weighted regularization (CRW):

  We are the first to propose using model accuracy as a weight for the regularization term (CRW), which is feasible only for variational posterior sampling approaches. This is not possible for non-variational methods.

• Hard data consistency in a variational framework:

  While hard data consistency has been used in other works, our framework (FLAIR) is, to our knowledge, the first to apply it within a variational setting.

2. Challenges from the abstract

Our method addresses non-linear mappings by operating in the latent space while explicitly backpropagating through the decoder to ensure consistency with the observed data in pixel space. This allows us to handle the non-linear encoder–decoder structure effectively, while still being able to initialze with the pseudo inverse. We observed that FLAIR reconstructs diverse samples (see Fig. 3) which also look realistic. As shown in Fig. 3 baseline approaches collapse to a single solution which is close to the mean, but does not look realistic. Under the unconditional prior all modes are unlikely. One has to condition the model very strictly to increase the likelihood for certain modes e.g. with very precise text prompts which is not feasible for degraded images. Our solution is to compute “special noise vectors” $\hat{x}$ to condition the model and increase the likelihood of modes which match the measurement.

3. CRW weighting

The CRW weighting is determined through a one-time accuracycalibration procedure, independent of the task or any hyperparameter. Specifically, we compute the expected model error using a small set of arbitrary calibration images. This error is derived from the conditional flow-matching loss, which directly corresponds to the regularization term in our framework. Additional details can be found in the Appendix A.6.

4. PSNR/SSIM metrics

We want to highly emphasize that our goal is to obtain diverse and visually convincing reconstructions that remain consistent with the observed data (i.e., sampling from the posterior modes), rather than pixel-wise accurate ones (i.e., estimating the posterior mean). This inherently leads to lower PSNR and SSIM scores, since there are many possible samples that can explain the observed data. However, our method achieves significantly closer reconstruction in perceptual distance as shown by the LPIPS evaluations. Furthermore, the strong FID scores show that our results are highly realistic while baselines fail to reconstruct images which look realistic.

Due to the perception-distortion tradeoff, it is fundamentally not possible to achieve both high perceptual quality and high pixel-level fidelity simultaneously, especially under severe degradations. Reconstructions that appear realistic often deviate from the pixel-wise mean and therefore yield lower PSNR and SSIM, despite being perceptually closer to a human observer.

To highlight that our model can also be used to obtain good MMSE estimates, we also conducted ensemble predictions by running posterior sampling 8 times and averaging the result. The results show that ensembling increases PSNR values, but reduces LPIPS and confirms that our samples are indeed distributed around the posterior mean and that results very close to the posterior mean like the baseline methods are perceptually further away (LPIPS) from the ground truth compared to our samples. We included the ensembling experiments in our supplementary material.

100 samples for FFHQ and 80 for DIV2K

DIV2K Results

FFHQ Results

We therefore want the reviewers to be aware that our method is explicitly designed for posterior sampling with a generative prior, not for producing a single point estimate such as the MMSE solution. Consequently, PSNR is not an appropriate metric for evaluating such generative approaches (especially under heavily ill posed problems), as it penalizes diversity and favors averaging, which is contrary to the goal of realistic posterior sampling.

5. Real world non Gaussian noise

This goes beyond the scope of the paper. However, we want to emphasize that our method as well as other methods in the field are not limited to Gaussian noise. The data term can. also be adapted to other noise distributions (e.g. Poisson noise).

6. Training and Inference Time

We want to emphasize that our model is completely training free. We simply use the SD3 weights as they are. A runtime and memory consumption analysis for x12 super resolution can be seen below. Our hard data consistency can strongly influence the runtime and we also provide measurements with number of data term steps<=5 and a fast version using a "tinyVAE" of SD3.

To validate that the usage of the tinyVAE or less steps does not degrades the performance noticeably we also provide a metrics for x12 Super Resolution on 100 samples of FFHQ:

7. Novelty of Hard Data Consistency (HDC)

The decoupling of the data term is a novelty for variational posterior sampling approaches based on stochastic interpolants such as diffusion or flow matching models. A major drawback of most posterior sampling approach with stochastic interpolants is the need to approximate the likelihood term, because it is not tractable. Applying hard data consistency to the approximation does not necessarily bring an advantage and could event be harmful. However, in the variational formulation no approximation for the likelihood term has to be applied and therefore decoupling the data term and applying hard consistency can bring a large advantage as is shown in our ablations. We are the first to analyze and leverage this advantage.

I have read the authors' rebuttals. The authors have addressed my concerns, including clarifying the innovation of this research relative to existing work and adding discussions on quantitative evaluation results and on non-Gaussian cases. Taking into account the innovation and contribution of this study, I have updated rating in the system.

We thank the reviewer for acknowledging the novelty of integrating flow matching into a variational framework, the high quality of our results, and the comprehensiveness of our experiments. Your feedback is much appreciated and we we incorporated it into our manuscript.

If our answers are satisfying, we kindly ask to consider raising your score.

1. Core components are an assembly of known ideas

We would like to clarify that our work introduces several novel contributions beyond assembling existing techniques:

• Variational posterior sampling for flow matching:

  We are the first to introduce a variational approach for posterior sampling with flow matching models. The derivation of this method is non-trivial and is provided in detail in Appendix A.1–A.3.

• Deterministic trajectory adjustments:

  Our proposed deterministic trajectory adjustment is a crucial and completley novel component of the method, and our ablation studies show it yields the largest improvements in LPIPS. This adjustment is theoretically motivated and has not been previously explored in the literature.

• Model accuracy-weighted regularization (CRW):

  We are the first to propose using model accuracy as a weight for the regularization term (CRW), which is feasible only for variational posterior sampling approaches. This is not possible for non-variational methods.

• Hard data consistency in a variational framework:

  While hard data consistency has been used in other works, our framework (FLAIR) is, to our knowledge, the first to apply it within a variational setting.

Our work is positioned within the field of training-free posterior sampling using generative priors. Given that flow matching models represent the current state-of-the-art in image generation, we believe it is both timely and important to develop posterior sampling methods that are specifically compatible with these powerful generative priors.

2. Non-trivial hyperparameters

The hyperparameters we introduce are $\lambda_{\mathcal{R}(t)}$ and $\alpha$, both of which are fixed across all our experiments and datasets. The gradient descent learning rate for the data term must be selected depending on the chosen task, which is also the case for all baseline methods.

Regarding the CRW, we explore alternative weightings in Appendix A.6. Additionally, our ablation study without CRW corresponds to setting $\lambda_{\mathcal{R}(t)} = t$.

We also note that, in variational approaches, a function for $\lambda_{\mathcal{R}(t)}$ must always be specified (e.g., RSD and RED-diff both use $\lambda_{\mathcal{R}(t)} = t$). Our contribution is to make this choice dependent on the model error.

3. Runtime Memory Analysys

Runtime optimization was not the scope of our project. Our baseline methods are evaluated with the same number of neural function evaluations. The methods may differ in the number of gradient steps taken on the data term, which depends on the early stopping constant chosen.

Our method can also be deployed with a fixed number of data term steps (e.g., 5), if a deterministic runtime is required. If runtime or memory footprint are of importance one can switch to a "tinyVAE" version of SD3.

To validate that the usage of the tinyVAE or less steps does not degrades the performance noticeably we also provide a metrics for x12 Super Resolution on 100 samples of FFHQ: