C1 Overfitting robustness for the task of denoising: Since our method is an unsupervised approach optimizing a single image, our observations suggest that repeatedly setting the network input to the output, along with autoencoding regularization, enables convergence to autoencoding-specific images instead of the identity.

The attached PDF in the global response presents the average PSNR for 20 images from the CBSD68 dataset for denoising. We observe two key points. First, aSeqDIP shows higher robustness against noise overfitting compared to other DIP-based methods, consistent with MRI and CT findings. Second, unlike MRI and CT, the onset of noise overfitting occurs earlier, but the subsequent decay is very small. Please refer to our response to the following comment for the proposition.

C2 Usefulness of Prop. 3.1: The main point of this proposition is to highlight that the aSeqDIP algorithm minimizes a different optimization problem compared to Vanilla DIP. Proposition 3.1 is not intended to indicate solution quality or analyze RMSE during optimization but to state that, under strong convergence assumptions, our algorithm minimizes the optimization problem in (5). Solving (5) explicitly is challenging due to the equality constraint and the network's non-linearity. In other words, the proposition illustrates the types of solutions our algorithm converges to. Our empirical results show that our algorithm converges quickly and remains stable.

C3 Performance of DM-based methods and their training data: Regarding the difference in PSNR values between our reported results and those in Score-MRI and MCG, we'd like to clarify that for our MRI experiments, we used Cartesian sampling, a more common scheme, while Score-MRI used Uniform 1D, Gaussian 1D, Gaussian 2D, and VD Poisson disk sampling. For CT, our aSeqDIP results (and other baselines except MCG) were obtained using the full 512x512 pixels in the AAPM dataset, whereas MCG downsized the images to 256x256 pixels for faster training and sampling (see the caption of Table 3 in the MCG paper). Therefore, we compared the ground truth after resizing the MCG results back to 512x512. We will include this discussion in the revised paper. In the following table, we compare aSeqDIP (a dataless method) versus MCG (a data-centric method) at their downsized pixel space, as reported in Table 3 of their paper. As observed, we achieve very competitive results.

We use the testing images from the fastMRI and AAPM datasets. The pre-trained models for Score-MRI and MCG were trained on the training sets of these datasets.

For our experiments with natural images (denoising, in-painting, and non-uniform deblurring), we used the CBSD68 dataset. For DPS (a DM-based method), we utilized a pre-trained model from ImageNet (128x128, 256x256, and 512x512), known for its high generalizability according to "The Emergence of Reproducibility and Consistency in Diffusion Models." This model is more generalizable than the alternative trained on FFHQ (faces dataset). In response to Reviewer MAra Comment 4, we experimented with the FFHQ testing set, using pre-trained models of DPS and DDNM trained on the FFHQ training set.

For the DM-based baselines, we used the default hyper-parameters provided by DPS (natural images), Score-MRI, and MCG (sparse view CT).

Note that we are not the first to observe unwanted artifacts in DPS (see Figure 3 in "Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency"). We believe these artifacts arise because DM-based approaches, which learn $p(x)$, require modifications to sample from $p(x | y)$. These modifications, often approximations, may not be accurate or suitable for all tasks and images.

A major point of our comparison is to show that our method achieves competitive or superior results compared to DM-based methods, without needing pre-trained models or training data.

End-to-end (E2E) supervised models outperform many DIP-based methods like VarNet because the best DIP optimization steps can vary significantly. This motivates our aSeqDIP approach. The table below shows that our method achieves higher average PSNR than VarNet for 15 MRI scans (4x). While E2E models are faster at inference with a single forward pass, our method's advantage is that it is fully data-independent.

C4 Code: Thank you for bringing this to our attention. We mistakenly uploaded an older version of our code. We have fixed the link and uploaded .py files for all the tasks.

Q1 Including measurements: See our global response.

Q2 Comment on using "sequential optimization of multiple network architectures": In our algorithm, each network is initialized by the optimized parameters of the previous network, and then optimized using (3) such that the input of the network is the output of the previously optimized network (see line 157). No re-initialization or architecture change are needed. In the revised manuscript, we will re-word it for further clarification.

Q3 Curves of Figure 2: Larger variance in the standard Gaussian distribution corresponds to larger additive perturbations even for the case of $\mathbf{x}^*=\mathbf{0}$ (the red curve). We conjecture that this still leads to larger distances from the GT and hence worst performance. In regard to the case where the input is the all-zero vector, we ran Vanilla DIP with $\mathbf{z}=\mathbf{0}$, and the reconstruction quality is very low. We conjecture that this is due to (i) the impact of the first convolutional layer, which is only its bias, and (ii) the output of the first layer is concatenated to later layers through the skip connection.

Limitation Comment: See our global response.

We would like to thank the reviewer for their prompt response to our rebuttal. We hope that the following will address the reviewer's remaining concerns with baselines.

We would like to emphasize that the main point of comparing our method with data-intensive approaches (VarNet, Score-MRI, DPS, DDNM, and MCG) is to demonstrate that we achieve competitive results, all without the need for training data or pre-trained models by appropriately setting up the optimization and regularization of deep image prior. In what follows, we address your concerns in three parts.

If Score-MRI is so unstable that when trained on the entire fastMRI brain training set is still outperformed by a data-free method on the fastMRI brain validation set, then maybe it is not a good DM baseline.

Score-MRI DM was originally trained on natural images and then fine-tuned on the entire training set of fastMRI using the single-coil setting. The reason for this pre-training + fine-tuning is that DMs require a significant amount of training data to enter the generalization regime (see Figure 2b in "The Emergence of Reproducibility and Consistency in Diffusion Models"), which may not be available for tasks such as MRI and CT. During testing, they used the multicoil real setting DM on both the real and imaginary parts. In their paper, they mention: "Our model requires magnitude images only for training, and yet is able to reconstruct complex-valued data, and even extends to parallel imaging."

To the best of our knowledge, Score-MRI and CCDF (which we came across after submitting the paper) are the two best DM-based MRI baselines. In CCDF, they demonstrated slightly improved PSNR scores while requiring fewer sampling steps, making it faster. Refer to Table 5 in the CCDF paper, "Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction." In that paper, Score-MRI is referred to as "Score-POCS." For the 4x Gaussian 1D sampling, CCDF reports 32.51 dB, whereas Score-MRI achieved 31.45 dB.

The type of undersampling mask should not make up for the difference in PSNR as Uniform 1D and Gaussian 1D are also Cartesian masks.

Thank you for your comment. Note that in the Score-MRI paper (multicoil setting in Table 2), the PSNR values vary by nearly 4 dB depending on the mask, ranging from 29.17 dB (Gaussian 2D) to 34.25 dB (Gaussian 1D).

We acknowledge the reviewer's comment that the masks they used are indeed Cartesian. Thank you for the correction. We needed to review the details of their code implementation to confirm this.

The first column of Figure 4 (and its caption) in their paper describes the exact masks they used. Our sampling mask is the 1D Poisson Disk, which the Score-MRI authors did not use. The Poisson Disk mask in their results is 2D Poisson Disk.

To fully address the reviewer's concern, we ran aSeqDIP with the 1D Uniform setting, the sampling mask used in the first row of Table 2 in Score-MRI. The results are averaged over 20 MRI knee scans (with 4x undersampling). As observed, we achieve competitive PSNR results with Score-MRI in this setting as well.

Further, I find it extremely strange that the VarNet is outperformed by a data-free method as the VarNet provides stable SOTA results (see fastMRI challenge) if trained and tested on the same type of data. So what was the training and testing setup for this experiment with the VarNet?

The reviewer is correct; VarNet indeed achieves very competitive results. Since VarNet does not provide a pre-trained model, we initially trained their architecture from scratch using 3,000 fastMRI multicoil datapoints and tested it with the fastMRI testing dataset. Please note that due to time constraints and the additional experiments conducted during the rebuttal, in our initial response, we used 3,000 knee images for training instead of the available 8,000 datapoints.

To fully address the reviewer's concern, we trained VarNet with the full training set and their PSNR results do indeed improve when compared to the results of VarNet trained with only 3K points. See the following table. Our method is still quite competitive and could prove beneficial in limited training data regimes.

Note that Score-MRI paper also reported VarNet results (7th column of Table 2), and the results slightly varied when different masks were used.

We are happy to address any more concerns.

Thanks,

Authors

We would like to thank the reviewer for their response and raising their score. We are glad that the reviewer is satisfied with our rebuttal and the additional experiments.

Following the reviewer's recommendation, we will add the additional experiments and more details about the baselines configurations in the revised manuscript.

C1 WHY this approach achieves better results? Strength of the DM-based baselines. Inductive bias discussion: We thank the reviewer for their constructive comment. The answers to these questions will definitely strengthen our paper. We divided our response to this comment into the following three parts.

• Answer to why we achieve better results when compared to diffusion models: Our approach and DM-based methods are conceptually different. Provided a set of training images, DMs are trained to approximate the underlying distribution $p(x)$. Subsequently, when employed to solve imaging inverse problems (IPs), the problem becomes sampling from the conditional distribution $p(x\mid y)$ where $y$ denotes the measurements. To this end, DM-based IP solvers attempt to approximate this conditional distribution (which may not be accurate and/or suitable for some tasks) using different approaches for which the reverse sampling steps are modified to achieve this target. aSeqDIP is different in the sense that it is an optimization-based method that depends solely on an input-adaptive Unet architecture and the generative power of the network, and a loss function that is designed to mitigate irrelevant noise overfitting (which is the major obstacle with DIP-based methods). We argue that the reverse sampling modifications needed to sample from the conditional distribution in DM-based methods present a more challenging design choice when compared to our approach.

• Answer to how strong the data-dependent baselines are: We believe that the DM-based methods that we used as baselines are very strong on the tasks they were considered for. For MRI and CT, we used the following criteria: Select a method that perform strongly for every task, and use the test dataset with the same training dataset distribution for which these DMs were trained on. For example, for MRI, we used Score-MRI as baseline. This method utilizes a pre-trained DM that was originally trained on natural images and then fine-tuned with the training set of the fastMRI dataset. In our MRI experiments, we used the testing set of fastMRI. Similar approach was used in CT where we used the MCG method (with an AAPM dataset-trained DM) and the AAPM testing dataset.

For our experiments with natural images (denoising, in-painting, and non-uniform deblurring which is added in this rebuttal), we used the CBSD68 dataset. As such, for DPS (the DM-based method), we utilized a pre-trained model that was trained on a very large and diverse dataset which is ImageNet 128X128, 256X256, and 512X512. This pre-trained model is much more generalizable when compared to the other option which was trained on FFHQ (a dataset of faces). According to `The Emergence of Reproducibility and Consistency in Diffusion Models', the ImageNet pre-trained model has high generalizability. In our response to Reviewer MAra Comment 4, we experimented with the testing set of the FFHQ dataset where the pre-trained models of DPS and DDNM (the DM-based baselines) were trained on the training set of FFHQ.

The main message of our comparison with data-centric methods is to demonstrate that our method can achieve competitive or superior results compared to DM-based methods, all without requiring pre-trained models and training data.

• Answer to why we generally achieve better results: We think that our method, similar to all DIP-based methods, benefits from the implicit bias inherited in the Unet architecture. The structure of a randomly initialized CNN is used as a prior as it was shown in the original DIP paper. The architecture of a generator network alone is capable of capturing a significant amount of low-level image statistics even before any learning takes place. However, the number of optimization steps required in DIP represents a challenge. Therefore, we believe that our method, exploiting autoencoding regularization and input-updating, keenly taps into the generative and denoising nature of CNNs for more explicit regularization to alleviate overfitting.

C2 Text is not too dense but a bit too repetitive and wordy. Including compromised target images: We do agree that some points are sort of repeated in the Introduction and other places in the paper. In the revised manuscript, we will improve the writing and readability of the figures. In the PDF attached in the global response of this rebuttal, we included the degraded images in our visualizations of Figure 5. We will do the ones in the Appendix in the revised manuscript.

C3 Figure 1 location and explanation: We acknowledge the reviewer's comment. In the revised manuscript, we will either elaborate more in the caption of this figure (or add more context to the diagram itself) or re-locate it until after we present our method.

C4 Figures captions: Thank you for your comment. We will improve the captions in the revised manuscript.

Q1 Sufficiency of the used baseline methods: To address your question about how strong the DM-based baselines are, please refer to our response to your first comment. See also the second part of our response to comment 3 of Reviewer Q1ff where we experimented with a leading end-2-end supervised reconstruction model. Regarding the DIP-based baselines, we would like to highlight that we considered the recent self-guided DIP work which has demonstrated highly competitive performance in terms of reconstruction quality and robustness to noise overfitting across multiple tasks.

Q2 known limitations of aSeqDIP: Thank you for your question. Please see our global response. Additionally, we evaluate our approach using different additional tasks, baselines, and settings. We hope that these results will shed more light into the capabilities and limitations of our method. In particular, for run-time and practical convergence, see our response to comment 3 of Reviewer qnNG. For testing with a non-linear task, see our response to Reviewer MAra comments 2 and 4.

We would like to express our sincere gratitude to the reviewer once again for their insightful comments.

As the open discussion period is drawing to a close, we would be deeply grateful if the reviewer could kindly respond to our rebuttal. This would provide us with the opportunity to address or clarify any remaining concerns thoroughly.

C1 Novelty and Differences with the original DIP: We appreciate the reviewer's comment. We would like to emphasize that our proposed method differs significantly from other DIP-based methods, including Vanilla DIP. These distinctions, which we believe set our work apart, are highlighted below.

• Formulation and Algorithmic Perspective: In addition to the input-adaptive nature of Algorithm 1 (which is motivated by the discussion in Section 3.1), we propose the use of the auto-encoding term (Section 3.2.1) which is not used in Vanilla DIP. Furthermore, the original DIP does not require the input to be an image, whereas in our case, it is an image-to-image mapping.
• Noise Overfitting Perspective: The proposed input-adaptive procedure along with the auto-encoding term result in the major key benefit of significantly delaying the noise overfitting decay which is the main challenge in all of the DIP-based approaches. See Figure 4, Figure 10, and the first figure in the PDF attached in the global response of this rebuttal.
• Applicability Perspective: In our paper, we presented experimental results using multiple reconstruction and restoration tasks. Most other DIP-based methods consider at most one to three tasks. For example, ref-guided DIP only considered MRI, whereas TV-DIP considered denoising and in-painting. Furthermore, in addition to the four tasks in the paper, in this rebuttal, we included a non-linear inverse imaging task which is non-uniform deblurring (see our response to the following comment).

Based on the highlighted remarks, we respectfully ask the reviewer to reconsider their opinion regarding novelty.

C2 It would be interesting to see how this model works for non-linear image restoration tasks such as image deblurring: We thank the reviewer for their comment. Here, we include results of the non-linear non-uniform image deblurring task using the setting in the DPS code. In particular, we use the ''blur-kernel-space-exploring'' setting. In what follows, we report the achieved PSNR (averaged over 25 images form the CBSD68 dataset) of aSeqDIP when compared to DPS, Self-guided DIP, and SGLD-DIP. As observed, our method significantly outperforms all DIP-based methods while reporting improved results when compared to DPS.

C3 Comparison with "Rethinking DIP for denoising": Thank you for your comment. In what follows, we compare aSeqDIP with the suggested paper. We use the task of denoising and report the average PSNR over 25 images from the CBSD68 dataset. As observed, our method reports higher PSNR. We believe that our approach, exploiting autoencoding regularization, keenly taps into the generative and denoising nature of CNNs for more explicit regularization to alleviate overfitting.

C4 Comparison with SOTA DM-based method, DDNM: Thank you for your question. In what follows, we present average PSNR results (averaged over 20 images) for the tasks of denoising (with $\sigma_d=25$), random in-painting (97% missing pixels), box-in-painting (with HIAR of 25), and non-uniform deblurring of our method versus DDNM (the suggested paper) and DPS on the FFHQ testing dataset. For DPS and DDNM, we used a pre-trained model that was trained on the training set of FFHQ. As observed, our training-data-free method achieves competitive or slightly improved results when compared to data-intensive methods on all tasks other than box-inpainting (for which we slightly under-perform), all without requiring a pre-trained model.

C5 The current results (Table2), the improvement is very small: Thank you for your comment. While the PSNR and SSIM improvements compared to other baselines are not very significant, we would like to highlight the following points. First, compared to DIP-based methods, our approach not only achieves higher reconstruction quality but also significantly improves robustness to noise overfitting. See the PSNR curves in Figures 4 and 10, as well as in the figure in the attached PDF in the global response. Second, when compared to DM-based methods, our approach not only achieves comparable or slightly improved PSNR and SSIM scores but also has the significant advantage of being independent of any training data and pre-trained models. We hope that emphasizing these points will highlight the additional advantages offered by aSeqDIP.

Regarding running the experiments for many rounds, do you mean for our method or the other baselines? Or do you mean running our method with different initializations of $\phi_1$?

We would like to sincerely thank the reviewer once again for their insightful comments.

As the open discussion period is drawing to a close, we would be deeply grateful if the reviewer could kindly respond to our rebuttal. This would provide us with the opportunity to address or clarify any remaining concerns thoroughly.

We would also like to respectfully highlight that other reviewers have acknowledged the novelty and strengths of aSeqDIP, with remarks such as "The idea is fantastic and thought-provoking", "The proposed approach for enforcing DIP to find its fixed point is interesting and novel", "The introduction of the regularization term (autoencoding term) during iterative training and showing that it is useful and that tuning it is important", and "Robustness towards overfitting demonstrating the benefits of the proposed approach".

C1+Q1: Comparison with TV-DIP: We thank the reviewer for their comment. In what follows, we include a comparison with TV-DIP in terms of the average PSNR over 15 scans in MRI (with 4x and the fastMRI test dataset) and 20 images from the CBSD68 dataset for denoising and in-painting. As observed, our method outperforms TV-DIP.

C2: The current introduction does not convey the message that the key benefit of aSeqDIP is to address the problem of overfitting: We acknowledge the reviewer's comment. In the revised Introduction Section of the paper, we will emphasize that the main problem of DIP-based methods is noise overfitting as the selection of the number of optimization steps in DIP can differ not only from task to task but also from image to image within the same task. We will add a discussion that the main goal of introducing aSeqDIP is to mitigate the issue of noise over-fitting through the input-adaptive procedure and the use of the auto-encoding term.

C3 Discussing the limitations such as runtime and practical convergence: Please see our global response.

Q2 Runtime of aSeqDIP: We thank the reviewer for their comment. In the last column of Table 2, we report the average run-time (minutes) of all the methods. For the second best PSNR and SSIM (self-guided DIP), our method is 2X faster for MRI and CT reconstruction and requires 1 minute less than self-Guided DIP for denoising and in-painting. When compared to DM-based methods, aSeqDIP requires a slightly less run time while achieving an improvement in terms of the reconstruction quality (PSNR and SSIM). While DM-based methods only require function evaluations and our method is an optimization-based approach, the generally larger run-time reported for DM-based methods is due to the necessity of running a large number of reverse sampling steps.

Q3 [Subjective] The convergence assumption in the Proposition: We agree with the reviewer that the convergence assumption in Proposition 3.1 is strong. The main point we'd like to convey in this proposition is that aSeqDIP is trying to solve the optimization problem in (5), which is different that Vanilla DIP. In Remark 3.2, we discuss this point by comparing aSeqDIP and Vanilla DIP. In the revised manuscript, we will present the point of the proposition as a remark for further clarification.

First, we would like to thank the reviewer for their response.

For TV-DIP, during the rebuttal, we used "https://arxiv.org/pdf/1810.12864" following the reviewer's suggestion. We coded the TV regularization term in Equation (3) and used Equation (5) to tune the parameters $\Theta$. According to Section 4.1 of the TV-DIP paper, we used 5000 optimization steps. For the architecture, we used the U-Net architecture (with skip connections) from the original DIP paper, which is the same as the one the authors describe in Figure 3.

In regard to the hyper-parameters (including the regularization parameter), in Section 4.1, the authors of TV-DIP mentions: "All algorithmic hyperparameters were optimized in each experiment for the best signal-to-noise ratio (SNR) performance with respect to the ground truth test image".

As the authors do not provide the regularization parameter, in the results we provided in the rebuttal, we used $\lambda = 1$, which is similar to aSeqDIP.

To fully address the reviewer's concern and ensure that the choice of $\lambda$ we used (in our rebuttal) in TV-DIP is sufficient, we run TV-DIP with additional values of the regularization parameter for the task of denoising and report the average PSNR results for 20 images from the CBSD68 dataset. As observed, the TV-DIP results with $\lambda$ near 1 is better than other values we considered here. In all cases, aSeqDIP achieves better PSNR.

Following the reviewer's suggestion, we will add this discussion and the TV-DIP results in the revised paper. We are happy to address any other concerns the reviewer might have.

We hope that, in light of our responses, the reviewer might consider raising their score.

Thanks,

Authors