InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems

Thank you for your comprehensive review and constructive feedback. We address each concern below:

• Generalization Capabilities We acknowledge this limitation and appreciate the opportunity to clarify our contribution's scope. The reviewer correctly notes that our method operates within a trained distribution of degradations, which differs from the unlimited flexibility of zero-shot methods. However, our approach occupies a valuable middle ground with specific practical advantages.
  Within the trained degradation family, our method demonstrates strong generalization - for instance, in our random matrix experiments reported in Tables 1 and 2, the degradation space is so vast that test-time degradations are virtually guaranteed to be unseen during training, yet our method significantly outperforms alternatives. Additionally, Section 4.1 demonstrates generalization to tasks slightly outside the training distribution (outpainting, which shares structural similarities with inpainting but involves different spatial patterns).
  The practical value lies in scenarios where: (1) a known family of degradations is encountered frequently enough to justify training, and (2) computational efficiency and quality gains are important for deployment. This represents a meaningful trade-off between task-specific methods and zero-shot approaches.

• Alternative Approaches We acknowledge that our current training-based comparisons are limited. Following the suggestion of reviewers 8Xy4 we include a comparison to InDI [1], a flow-based restoration method, in our multi-degradation setup. Preliminary results corresponding to Table 1 can be found below.

• Ablations We will include comprehensive ablation studies in the appendix. Specifically, we address the suggested experiment of removing the forward operator $\boldsymbol{H}$ and providing only $\mathbf{y}$ as input to $T_\psi$, which we have implemented for the FFHQ64 experiment and include below. This will help isolate the contribution of our mechanism.

[1] Delbracio et al., "Inversion by direct iteration: An alternative to denoising diffusion for image restoration." arXiv preprint arXiv:2303.11435 (2023).

Thank you for your comprehensive review and constructive feedback. We address each concern below:

• Additional Method We acknowledge that adding more comparisons to these recent posterior samplers would strengthen our paper. Following the reviewer’s suggestion, we have evaluated MGPS [2] on our experimental setting. The results, corresponding to Tables 1 and 2 in our paper, are presented below:

• Non-linear Operators The reviewer raises an important point about pseudo-inverse tractability. Due to the exposure in training, we believe the model would be able to learn to use the operator even with a roughly approximated pseudo-inverse. The joint-attention scheme can also be applied directly with the transformed degraded measurements, without casting them back to the image space (that is, skipping the application of $\boldsymbol{H}$). We hope to explore many non-linear operators in future work.

• Latent Diffusion The reviewer correctly identifies the main computational bottlenecks for extending our approach to latent diffusion models. We will clarify these limitations in our revised manuscript. However, recent work such as [3] demonstrates promising approaches for handling operators in latent space that could potentially be adapted to our framework. We note that latent diffusion essentially introduces a non-linear encoding/decoding operator that could be handled using similar principles to those outlined above.

• Noise Level We have not observed that the method is limited to certain levels or types of noise, we have simply used a single moderate value of $\sigma$ in our experiment. We expect InvFussion to be more beneficial as the noise level increases, as the blind restoration problem becomes harder. We will add an experiment with a high rate of Poisson noise to the final version to assess our hypothesis.

[3] Raphaeli, Ron, Sean Man, and Michael Elad. "Silo: Solving inverse problems with latent operators." arXiv preprint arXiv:2501.11746 (2025).

Thank you for your comprehensive review and constructive feedback. We address each concern below:

• Resolution We appreciate this important concern about scalability. Kindly note that Appendix C2 reports experiments on 256x256 images. We will highlight these experiments in the main text.

• NFE Comparison We have included NFE counts for all compared methods in Tables 1 and 2. Our method achieves superior performance while requiring significantly fewer NFEs than most competing approaches—a key practical advantage for real-world deployment. We will enhance the main text to better emphasize this computational efficiency.

• Additional Methods We highlight that the cited works [1-3] train separate models for individual degradation types, whereas our approach trains a single unified model capable of handling multiple degradations. This represents a fundamentally different problem setting with distinct practical advantages. That said, we recognize the value of these comparisons, and we have trained and evaluated [2] on the multi-degradation task demonstrated in Table 1 (results below). Additionally, some approaches like [2] could potentially be combined with our architecture, which we see drastically improves the adaptation to the multi-degradation setting.

• Degradation Aware We appreciate this conceptual point and the reviewer's distinction. Any method attempting to restore from a zero-filled degradation $\boldsymbol{H}^\dagger\mathbf{y}$ is essentially operating in a blind setting, as no additional information is provided beyond the degraded sample (corresponding to Eq. (2)). In contrast, we use "degradation-aware" to denote methods that incorporate some—even partial—information from the known degradation, enabling a non-blind solution.
  We agree with the reviewer that conditioning on the operator's action is not equivalent to conditioning on the operator itself. Nevertheless, our architecture gets to "probe" $\boldsymbol{H}$ several times along the network, each time applying it to different feature vectors. Therefore, theoretically, the network can deduce a lot about $\boldsymbol{H}$. Both our results and those of zero-shot methods (which also implicitly condition on the action of the operator) demonstrate that incorporating this partial information about the operator's action substantially improves results, as expected from the theory in Eq. (2).
  While we maintain that our approach provides meaningful degradation conditioning beyond standard pseudo-inverse reconstruction, we will refine our positioning in the final version to more precisely describe our contribution within the existing framework of degradation-conditioned methods.

• LPIPS We will include LPIPS scores for all experiments as requested. This will provide additional perspective on perceptual quality alongside our existing metrics. Here are the results of LPIPS on the experiments corresponding to Tables 1 and 2:

• Cross-Attention Thank you for this clarification. The reviewer is correct that "cross-attention" is the more standard terminology. Our "joint-attention" (also referred to as "extended-attention" in some literature) processes queries against both self-branch and cross-branch keys/values simultaneously, differing from the sequential "self-then-cross attention" approach. We will clarify this point in the final version. This architectural choice is based on our early experiments; we expect a “self-then-cross attention” architecture to perform nearly as well in all experiments.

• Typing Errors We will address all noted typographical errors and consider updating the method name.

• Comparison to End-to-End Non-Diffusion Methods We anticipate that diffusion-based approaches will demonstrate superior image quality compared to methods like Restormer, which employs transformer architecture (similar to ours) but is trained to minimize L1 loss. For scenarios requiring low distortion with reduced computational cost, our single NFE MMSE estimation method provides a viable alternative.

• Scaling With Respect to NFEs We are not sure we understand the reviewer’s question. As a diffusion model, we expect image quality to worsen the fewer NFEs are used for sampling. Given the computational speedup compared to zero-shot methods, we have kept the NFE count as recommended by [28].

Thank you for your comprehensive review and constructive feedback. We address each concern below:

• Motivation Rather than enforcing physical consistency through explicit constraints like unrolling methods, our approach is motivated by the principle of learning-based adaptation to measurement operators. Even though consistency with the measurements is not enforced, we find that our architecture manages to learn to satisfy the consistency constraint quite well.

• Deep Feature Dimensions This architectural constraint is readily accommodated in modern diffusion architectures, particularly DiT variants, through straightforward hyperparameter adjustments that do not compromise performance. The implementation involves rounding the hidden dimension in specific layers to ensure the total number of hidden feature elements is divisible by the total image dimensions. Importantly, this constraint need not apply to all layers, enabling architectural flexibility. We will clarify the scope and implementation details of this limitation in the final version.

• Computation of $\boldsymbol{H}$ While we acknowledge this computational consideration, it is important to emphasize that this limitation is inherent to many inverse problem solving methods, including established zero-shot approaches. Moreover, the practical applications we target—and many others in the field—benefit from efficient implementations of both $\boldsymbol{H}$ and $\boldsymbol{H}^\dagger$. We will explicitly discuss this computational consideration in the final version.

• Comparison to Unrolling While we agree that comparisons to supervised unrolled networks would provide valuable context, direct comparison presents significant practical challenges for our evaluation framework. Methods like Deep Image Prior operate at the single-image level, making a single 10K FID evaluation computationally prohibitive. Additionally, these methods typically optimize for different objectives—such as MAP estimation rather than posterior sampling quality—and employ fundamentally different architectures, limiting the conclusiveness of direct comparisons. To provide meaningful context nonetheless, we evaluated PSNR and LPIPS metrics for approximately 250 images restored using ASeq-DIP [4], a recent DIP method.
  Despite ASeq-DIP's significantly higher per-sample computational cost, InvFussion achieves superior performance across all metrics. We will include this analysis in the final manuscript.

• Table 3 We acknowledge the reviewer's concern regarding the PSNR values. To address this, we repeat the experiment with a larger constant out-painting mask, yielding the following results below.
  These results maintain the same performance trends observed in Table 3. We emphasize that our primary objective is robust performance on degradations within the training distribution, which encompasses a wide and diverse range of conditions.

• PSNR Comparison The PSNRs reflect the inherent Perception Distortion Tradeoff [1] in restoration methods. Our approach prioritizes generating perceptually realistic samples, as evidenced by consistently superior FID scores, while maintaining competitive PSNR values among methods with comparable perceptual quality. The MMSE variant demonstrates our model's flexibility to achieve higher PSNR when distortion minimization is the primary objective
• Other Methods In our work, we have included comparisons with a representative selection of well-established zero-shot methods that remain dominant in the field. Following suggestions from reviewers 8Xy4 and XFfT, we will incorporate additional comparisons with InDI [2] (training-based) and MGPS [3] (zero-shot), both representing more recent developments. These comparisons demonstrate competitive performance for our proposed method.

• Typos We will conduct thorough proofreading to address all typographical and grammatical errors in the final manuscript.

• PSNR of examples For the facial image example shown in Figure 3: DDRM: 8.98 dB, DDNM: 11.06 dB, DPS: 13.14 dB, PGDM: 14.85 dB, DAPS: 9.44 dB, Palette: 13.65 dB, InvFussion: 16.06 dB. These values align with the trends observed in Table 3, demonstrating consistent performance across different test scenarios.

• Unconditional and MMSE We will improve our explanations of these concepts in the final version. For unconditional generation, we simply omit conditioning on any measurement, allowing the model to generate samples from the prior distribution rather than the posterior distribution given a degraded image. For MMSE estimation, we demonstrate how training-based inverse problem solvers can be adapted for MMSE estimation by leveraging the near-equivalence between the high-noise regime in diffusion training and linear regression problems.

• Projection for Noisy Degradations The $\epsilon$-ball projection suggested by the reviewer ($||\mathbf{y}-\boldsymbol{H}\mathbf{x}|| < \epsilon$) can indeed be used as a projection. The CFID values with this projection, corresponding to Table 8, are as follows:

[1] Blau, Yochai, and Tomer Michaeli. "The perception-distortion tradeoff." Proceedings of the IEEE conference on computer vision and pattern recognition (2018).

[2] Delbracio et al., "Inversion by direct iteration: An alternative to denoising diffusion for image restoration." arXiv preprint arXiv:2303.11435 (2023).

[3] Moufad, Badr, et al. "Variational Diffusion Posterior Sampling with Midpoint Guidance." The Thirteenth International Conference on Learning Representations (2025).

[4] Alkhouri, Ismail, et al. "Image reconstruction via autoencoding sequential deep image prior." Advances in Neural Information Processing Systems 37 (2024).

We thank the reviewer for considering our rebuttal and for their continued engagement.

Consistency Zero-shot methods — and indeed data-free DIP methods — rely on explicit closed-form mappings between measurements $\mathbf{y}$ and the operator $\boldsymbol{H}$. In contrast, InvFussion employs supervised training to learn this mapping implicitly through the training loss, rather than enforcing consistency explicitly at test time.

From a theoretical perspective, this implicit consistency can be understood through our network's optimal denoising solution, $\mathbb{E}[\mathbf{x}_0 | \mathbf{x}_t, t, \mathbf{y}, \boldsymbol{H}]$. This expectation, which is used in the generation process, is inherently consistent with the measurements because it conditions on both $\mathbf{y}$ and $\boldsymbol{H}$. Crucially, without the explicit conditioning on $\boldsymbol{H}$ enabled by InvFussion, the standard optimal solution $\mathbb{E}[\mathbf{x}_0 | \mathbf{x}_t, t, \mathbf{y}]$ lacks this consistency guarantee. This is because without accounting for the specific degradation process, the expectation also averages all possible degradations, similar to Eq. (2) in our paper.

This theoretical consistency is validated by our results: InvFusion's CFID scores are nearly identical to the corresponding FID values, indicating that projecting our model's outputs onto the measurement space has minimal impact. This suggests that our network has indeed learned to produce outputs that are already highly consistent with the measurements. To provide more direct evidence of this consistency, we include quantitative measurements of MSE in the degradation's range-space below, corresponding to the experiment in Table 2. MSE is used to accommodate degradations of differing rank and range, and therefore has a different scale for each degradation class. The MSE values confirm that InvFussion achieves consistency comparable to methods that employ explicit projections, such as DDNM or DDRM. These results offer a complementary metric to the FID/CFID analysis.

Regarding ablation studies, we propose to include an ablation study examining the effect of varying the number of layers that utilize the InvFussion block. This will help quantify how repeated exposure to the degradation operator influences the final output consistency and provide deeper insights into our architectural choices.

Comparison to Unrolling and DIP The challenge raised by supervised unrolling methods in our setting is the adaptation of the training scheme of such approaches to a multi-degradation framework. While we cannot complete this within the scope of the current rebuttal time frame, we commit to including results on supervised unrolling methods in the final version of our manuscript.

Regarding hyperparameter settings for ASeq-DIP, we employed the default parameters from the official GitHub implementation; UNet architecture with two levels, Adam optimizer with a learning rate of 1e-4, 5000 outer training iterations and 10 inner ones. We have not been able to explore hyperparameter settings within the rebuttal time frame, because a separate network must be optimized for each sample, leading to a high computational cost and an extremely slow runtime. We will include a more thorough comparison in the final revision where we will test different hyperparameters for ASeq-DIP. At the same time, to ensure a fair comparison we must take into account the tradeoff between costly hyperparameters and total algorithm runtime, as our method specifically aims to alleviate the longer run-times of zero-shot methods.

We are happy to respond to additional questions the reviewer may have.

Dear reviewers,

The reviewers-authors discussion phase will end in less than 24 hours.

If not done already, make sure to submit the "Mandatory Acknowledgement" that confirms that you have read the reviews, participated in the discussion, and provided final feedback in the "Final justification" text box.

Be mindful of the time and efforts the authors have invested in answering your questions and at least acknowledge their responses. Make sure to provide a fair and scientifically grounded review and score. If you have changed your mind about the paper, please update your review and score accordingly. If you have not changed your mind, please provide a clear and sound justification for your final review and score.

Best regards, The AC