UGoDIT: Unsupervised Group Deep Image Prior Via Transferable Weights

We'd like to thank the reviewer for their comments. We are glad that the reviewer acknowledges our use of a multi-decoder UNet and the improvements over baselines.

(W1) Setup value. Run-time. Comparison with test-time training methods.

Value of the setup is in applications where only a few degraded measurements are available (e.g., in some scientific and medical applications). In these low-data regimes, data-centric methods requiring large clean datasets are not applicable, and our approach provides a strong case under such scenario.

Our Run time when compared to standalone DIP-based shows that UGoDIT achieves significantly faster convergence because the learned encoder provides a strong initialization at test time. See our response to Reviewer usw1 (D2) for the number of iterations needed to achieve 30dB at inference across the three tasks.

Comparison with test-time training methods: See our response to Reviewer eMg2 (W4+W5) for comparison with a TTT method from ICML 2022 in addition to 2 other baselines. We would also appreciate any specific TTT papers that the reviewer considers relevant.

(W2) Why the shared encoder works in the limited-data regime? The "more accurate" nature of the encoder is only evaluated by reconstruction. Why the encoder trained on 5-6 images is not overfitted?

We thank the reviewer for raising these important questions.

Shared encoder in the limited-data regime: UGoDIT jointly optimizes the encoder with multiple decoders across a small set of degraded images, which encourages it to capture task-common features. This inductive bias, supported by multi-task learning (see our response to Reviewer 2CSh (W1)), enables the encoder to generalize even when trained with as few as 5–6 images.

Evaluation of "encoder accuracy": We use reconstruction quality as the primary evaluation metric because the encoder is trained solely to produce features that aid reconstruction. Thus, improved reconstruction accuracy on unseen test images directly supports the claim that the encoder has learned meaningful representations. We would welcome suggestions from the reviewer on alternative evaluation criteria/metrics they would find more convincing.

Encoder's features not over-fitting: The shared encoder saves parameters and hence prevents over-fitting. In Appendix D, we provide a feature analysis showing that the encoder trained with multiple decoders learns smoother and more structured intermediate representations compared to a single-decoder baseline. These features transfer well to new test data, which explains the strong generalization despite the small training set.

(W3+Q2) Evaluation on $>20$ test images.

Please see our response to Reviewer 2CSh (W2+W4+Q2), where we conducted experiments using 100 test images.

We report the results of an FFHQ-trained-UGoDIT-6 using 100 ImageNet test images and 20 CBSD65 test images for the SR task in response to Reviewer usw1 (Q3).

(W4+Q3) UGoDIT MRI with uniform sampling.

In our experiments, we used variable density random sampling masks commonly used in compressed sensing MRI [A].

In response to the reviewer's suggestion, we report the performance of UGoDIT-6 using uniform (equispaced) undersampling patterns on fastMRI brain data. For each acceleration factor (AF), one-third of the k-space lines are fixed at the center, while the remaining lines were uniformly distributed across the high-frequency regions.

As observed, UGoDIT results are still competitive using a different undersampling patterns even when compared to SITCOIM which was trained on thousands of fully sampled images.

[A] Sparse MRI: The application of compressed sensing for rapid MR imaging, MRM 2007

(W5+Q4) Selection of $M$ images. Unfair advantage (compared to high-data methods that are trained across varying conditions) if all MRI data are chosen from similar cases.

We thank the reviewer for this comment. The training and testing images were selected at random including MRI, i.e., they are not selected to be close to the test images.

In our evaluation, DMs are trained on fully sampled measurements, whereas supervised methods are trained separately for each setting. For example, in Table 1, we have two pre-trained VarNets, one for 4x and one for 8x, where 8000 pairs of sub-sampled and full-sampled images are used. In UGoDIT's main results, at most, we train on six sub-sampled images.

To show the impact of the selected training images, see to our response to Reviewer 2CSh (W2+W4+Q2).

(W6) VarNet (being outdated) selection, results, run-time, and visualizations.

We thank the reviewer for this comment.

On VarNet being "outdated": We respectfully disagree with the characterization of VarNet as outdated. VarNet [17] is a paper from MICCAI 2020 (a well-known venue for MRI) that remains a widely used baseline in MRI reconstruction. However, to address the reviewer's concern, we compare with Recurrent VarNet (CVPR 2022) [B].

Our VarNet implementation follows the official codebase but was trained on 8,000 images (as mentioned in Table 1), which is fewer than what typically used to report SOTA results. This difference likely explains the lower metrics we observed. However, the performance of our approach trained in very low data regime is still competitive with methods trained on orders of magnitude more images indicating the advantages of expertly exploiting shared architectures, regularization, and implicit biases of neural networks to direct data-consistent reconstructions.

The ~5 seconds reflects end-to-end inference including BART-based sensitivity map generation and data loading. This also depends on the compute power.

We will include VarNet and Recurrent VarNet visuals with error maps in the revised paper. The key message for our comparison with supervised methods is to show that UGoDIT can achieve competitive results without needing large number of paired training images.

[B] Recurrent Variational Network: A Deep Learning Inverse Problem Solver applied to the task of Accelerated MRI Reconstruction, CVPR 2022

(W7) STRAINER and aSeqDIP are better than DPS but noisier.

The reviewer is indeed correct. DPS is known to produce artifact-free but more blurred images that could often violate measurement consistency, leading to lower PSNR/SSIM despite perceptually cleaner results (see also the visualizations in Figure 5 of aSeqDIP). At every reverse step of DPS, one gradient is taken w.r.t. $y$ and $A$. In contrast, STRAINER and aSeqDIP enforce data consistency more explicitly.

(W8) Ablations: (1) Impact of the NN architecture choice beyond Appendix E; (2) How much of the encoder arm needs to be trained over the $M$ images.

Since our method builds on DIP, we use a standard U-Net with residual connections. If the reviewer refers to non-CNN architectures, we note that Transformers and MLPs, unlike CNNs, lack spectral bias [15]. We kindly ask the reviewer to indicate if they refer to other CNNs such as U-Nets that don't use residual connections.

To address the second part, we report the results of UGoDIT-6 (MRI task) with freezing different number of encoder layers.

As observed, the default setting (5 frozen layers) return the best results. We thank the reviewer for this suggestion. We will include this study in the revised paper.

(W9) Works that train DMs from scratch without adapting from natural images; Unsupervised methods fall short of generative models need citation.

We say that DM-based methods often use fine-tuning in MRI/CT which is the case in several recent methods [12,13,17]. Our sentence does not rule out the existence of other methods that train MRI/CT DMs from scratch. We would greatly appreciate it if the reviewer could point us to such works.

Thanks for suggesting to cite paper(s) that supports the claim: unsupervised methods often fall behind DM methods. This claim is supported in works such as [C] as well as the results of baseline SITCOM in Tables 1 and 2, where a DM method is shown to be better than our proposed unsupervised method.

[C] Sequential Diffusion-Guided Deep Image Prior for Medical Image Reconstruction, ICASSP 2025

(W10) Description of UNet in Sec. 3.1.

We will re-write according to the reviewer's suggestion.

(W11) Impact of using $M$ decoders is shown with (5), why not (1)?

Since Eq. (1) corresponds to vanilla DIP, overfitting is expected. Eq. (5) describes the single-encoder/single-decoder case, which—similar to UGoDIT—uses the autoencoding term with input updates that help mitigate overfitting.

Moreover, there is no distinction between training and testing for Eq. (1), as vanilla DIP is a training-data-free method and does not involve $M$ training images.

(Q1) Advantage over dataless methods if UGoDIT does not generalize well on OOD.

In Appendix A.1, our method with MRI does indeed require a large number of iterations to converge. However, SR results in Appendix A.2 show that UGoDIT generalizes very well to OOD data in terms convergence speed and PSNR. Therefore, for MRI, if faced with unseen data that is from a different anatomy, using UGoDIT or aSeqDIP would return similar PSNRs (see Figure 5), but for fast convergence, it would be better to use a dataless method, such as aSeqDIP. We thank the reviewer for their comment. See our response to Reviewer 2CSh (W1) for generalization. For an analysis on the learned features, see Appendix D.

We are glad that the reviewer finds our paper ''easy to follow'' and ''presented in a conceptually simple way'', and acknowledges the several imaging tasks, OOD experiments, and the several ablation studies.

(W1) Main contribution is the shared-encoder with $M$ decoders. Self-consistency was proposed by aSeqDIP.

Indeed, our main contribution is the shared-encoder architecture jointly trained with $M$ decoders on degraded images. This design learns transferable encoder weights that significantly accelerate and improve DIP-based reconstruction at test time.

While we extend the “auto-encoding self-consistency” formulation in aSeqDIP [24] to our multi-decoder setting, UGoDIT differs in three key ways:

• aSeqDIP operates on per-image and learns no transferable prior; UGoDIT learns a shared encoder across $M$ degraded inputs.
• UGoDIT’s encoder is fixed at test time, enabling faster reconstructions and better generalization (Figures 2-3, and Appendix B). See also our response to Reviewer usw1 (D2).
• The multi-decoder structure enables task-specific learning (where each task is an inverse problem), accelerating convergence at test time.

Thus, we think that extending self-consistency to a transferable, low-data regime with shared encoder weights is non-trivial and impactful, especially when clean data is unavailable, and it has led to significant improvements in PSNR and convergence.

(W2) Clarity on Eq. (2) minimizer and early stopping.

This is a two-part concern, and we clarify both points below:

Regarding Eq. (2): The objective in Eq. (2) is jointly minimized over encoder $\phi$, the decoders $\psi_i$, and the inputs $z_i$. Eq. (2) is the same as Eq. (1) but with the autoencoding hard constraint over the training set that serves to mitigate overfitting. Since CNNs (like a U-Net) are known to denoise or autoencode images, they serve as an explicit constraint to restrict the solution space. The reviewer’s suggestion that $z_i = A^T y_i$ is the minimizer assumes a fixed network or identity mapping. In contrast, $g_{\psi_i} \circ h_\phi$ is a deep network trained to balance measurement consistency and autoencoding regularization. The input $z_i$ is updated based on the network’s current parameters, and its convergence is determined by the evolving encoder–decoder mapping.

Regarding early stopping: The need for early stopping in DIP arises from the risk of overfitting to noise or undesired points in the null space of $A$ when optimizing over all network weights per image. In UGoDIT, the autoencoding term $\|\|g_{\psi_i} \circ h_\phi(z_i) - z_i\|\|_2^2$ helps enforce input–output consistency, acting as a regularizer that stabilizes PSNR curves over longer iterations. As shown in Appendix C (Figure 11), this reduces the need for early stopping.

(W3) [MAIN] (1) Training with $M$ clean images; (2) Including test image as the (M+1)'th sample at training time.

We appreciate the reviewer's important suggestions. These will help improve our work.

UGoDIT is designed specifically for the low-data regime without clean images. However, we agree that training the shared encoder on clean images is a valid comparison and may represent an upper bound as we show next.

If clean images are available, we first need to modify the training objective since there would no longer be forward operators $A_i$ or inverse problems during training. The optimization reduces to a pure multi-decoder autoencoder, where Eq.(3a) becomes:

$\phi, \psi_i \leftarrow \arg \min_{\phi,\psi_i} [\sum_{i\in[M]} \|\|g_{\psi_i}\circ h_\phi(z_i) - z_i\|\|^2_2 ],$ ... (*)

where $z_i = x_i$ are the clean ground truth images.

To test this idea, we compare UGoDIT trained on degraded images vs. a multi-decoder autoencoder trained on clean versions of UGoDIT's degraded images for the SR task.

As observed, test-time PSNR for the auto-encoder with clean data is slightly better that UGoDIT. This means that training on $M$ clean images could also serve as a prior for test-time reconstructions. The decoder for clean training is initialized at random and then optimized to reconstruct the test image.

Although this represents a different setup from UGoDIT, we will include this ablation study in the revised paper.

As for the second part of the question, we hypothesize that including the test image as the ($M+1$)-th sample during training will not improve performance compared to our approach of learning a prior and testing on the image separately. This is supported by the results of Figure 1, where the average training curve is lower than the testing curve. The reason is that during training, the network parameters are optimized to solve multiple inverse problems, while at test time the decoder parameters are optimized (with the help of the frozen pre-trained encoder) to solve a single image-specific problem. Furthermore, including the test image as the ($M+1$)-th changes the problem from generalizing to unseen measurements to directly fitting all available data, which—as the reviewer correctly notes—comes at a higher computational cost. UGoDIT, in contrast, is designed for inference on unseen data using a fixed encoder. That said, we believe that the reviewer’s suggestion is insightful, for which we leave a deeper investigation to future work.

(W4+W5) [MAIN] Comparison with measurement splitting and test-time training.

We thank the reviewer for pointing us to these unsupervised learning baselines. Please see below comparison for the task of MRI.

As observed, our method returns the best results as compared to the suggested baselines. We did not have sufficient time to compare to [D], but we promise to include it in a revised version.

[A] (SSDU) Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data, MRM 2020

[B] (ZS-SSL) Zero-Shot Self-Supervised Learning for MRI Reconstruction, ICLR 2022

[C] (TTT) Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing, ICML 2022

[D] Noise2Self: Blind Denoising by Self-Supervision, ICML 2019

(W6) Why does UGoDIT outperform high-data methods?

We thank the reviewer for this question. We believe there are two factors:

Strong inductive bias of DIP-like architectures:
Untrained convolutional networks already capture useful low-frequency image structures (specifically when overfitting is mitigated), UGoDIT leverages this property while learning a transferable encoder that further enhances these representations. For example, see the visualizations in Figure 5 of aSeqDIP [24].

Regularization through shared encoder and multi-decoder training: Our training encourages the encoder to learn features that generalize across tasks, while the multi-decoder setup reduces gradient interference and overfitting, resulting in representations that are robust even without large datasets.

We note that we do not claim that UGoDIT will always outperform high-data methods. In settings where large-scale clean datasets are available and well-matched to the test domain, supervised or generative models can achieve higher performance (such as SITCOM vs. our method in Table 2).

Our key message is that UGoDIT offers competitive performance in the low and degraded data regime, where such large clean datasets are not available.

(W7) [MAIN] Selection of the $M$ training images.

We'd like to thank the reviewer for this comment. The training and testing images were selected at random including for the MRI experiments. We refer the reviewer to our response to Reviewer 2CSh (W2+Q3) for more details and experiments about the performance of UGoDIT using different training sets over the same 100 test images.

In our response to Reviewer usw1 (Q3), we report the results of an FFHQ-trained-UGoDIT-6 using 100 ImageNet test images and 20 CBSD65 test images for the SR task (additional OOD evaluation).

(W8) Figure 13 results for $M\in\\{1,2,3\\}$.

We extend the results of Figure 13 to the following table with $M\in \\{1,2,3\\}$. We note that $M=1$ trains a one-decoder autoencoder and then fixes the weights, so the performance might highly depend on the selected training image.

As observed, the performance slightly degrades when $M$ becomes smaller. We will include this result in the revised version.

(W9) Training/testing code and setup. Evaluation with $>20$ test images.

Thank you for the comment. We added more details in the anonymous readme file and included the MRI code. See our response to Reviewer 2CSh (W2+Q3) for evaluation over 100 test images.

In our response to Reviewer usw1 (Q3), we report the results of an FFHQ-trained-UGoDIT-6 using 100 ImageNet test images and 20 CBSD65 test images for the SR task (additional OOD evaluation).

(W10) Visualizations in the main text. ROIs. MRI orientation.

We will include visual examples in the main text and fix the orientation of the MRI images. We will also enlarge the ROI (central region) crops and the error maps.

We'd like to first thank the reviewer for their feedback. We are glad that the reviewer acknowledges the limited-data setting of our work and its fully unsupervised nature.

(D1) OOD performance varies across tasks: Lower OOD performance for MRI compared to methods like aSeqDIP.

We thank the reviewer for this comment. We agree that, as shown in Appendix A, UGoDIT performs well on some OOD tasks (e.g., super-resolution) but shows reduced OOD performance for MRI.

We believe this variability stems from two factors:

1. Encoder specialization: UGoDIT learns a shared encoder that captures features from the training degradations. When the test distribution deviates substantially (e.g., training on knees, testing on brains), the learned encoder may not fully generalize.

2. Role of autoencoding regularization: The autoencoding term stabilizes training but also biases the encoder toward the statistics of the training images. This can limit adaptability under severe distribution shifts, whereas per-image methods like aSeqDIP can adjust their weights to the new domain.

While this is a limitation, we see it as an avenue for future work — for example, exploring encoder fine-tuning at test time or hybrid strategies that combine UGoDIT’s transferable features with adaptive refinement. We will discuss this point explicitly in the revised manuscript.

(D2) While UGoDIT performs well on certain tasks like super-resolution (SR), it does not consistently outperform other methods like aSeqDIP on all tasks.

We agree with the reviewer that UGoDIT’s performance varies across tasks. However, compared to aSeqDIP and other dataless methods, Figure 3 shows that UGoDIT achieves significant improvements in convergence speed while also providing higher reconstruction quality: approximately +3 dB PSNR for MRI, +1 dB for SR, and +2 dB or more for NDB.

To support this claim, we report the number of optimization iterations needed to achieve 30dB across the three task, comparing UGoDIT-6 and aSeqDIP. We note that, at inference, the number of parameters to optimize in UGoDIT (with $M=6$) is lower than aSeqDIP as the encoder in our case is fixed.

When compared to data-centric baselines such as SITCOM, UGoDIT returns slightly lower PSNR but does so without requiring any pre-trained model or access to large datasets of clean images, making it more applicable in low-data settings.

(D3) The experimental results of the paper show that when the number of images exceeds six, there is no significant improvement in model performance. This indicates that the enhancement from this training approach is limited.

We agree that our experiments show that performance tends to saturate when the number of training images exceeds six. This suggests that UGoDIT is particularly well-suited for the very low-data regime, where only a handful of degraded images are available — a setting where many existing methods struggle.

We hypothesize that the saturation effect arises because the encoder quickly captures the shared structure present in the training images, and adding more examples beyond this point yields diminishing returns. This is consistent with our goal of designing a method that is effective even under severe data scarcity.

This property also makes UGoDIT appealing for domains where only a few measurements are accessible (e.g., specialized scientific imaging tasks such as black hole observations). We plan to explore such applications in future work.

(Q1) As a single-image reconstruction method, it is essential to compare the required FLOPs and training time.

We thank the reviewer for this comment. At test-time, UGoDIT is indeed a single-image reconstruction method.

We report the training time and FLOPs at training for the SR task, and compare it with aSeqDIP. All methods were run for 4000 iterations.

As observed, the more training images, the higher run-time and FLOPS for UGoDIT as we have more decoders to train when $M$ increases. We are at most less than four times the run-time when compared to aSeqDIP. As for testing, we only require less than half the run-time as aSeqDIP since the optimization in UGoDIT takes place w.r.t. only the decoder, whereas aSeqDIP optimizes the entire network.

Moreover, our run-time and FLOPs are significantly lower than training diffusion models or supervised methods. For example, as per Figure 1 of [A], diffusion models require more than 17 GFLOPs.

[A] Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture, CVPR 2024

(Q2) Comparison and discussion with latest INR methods such as [B].

We thank the reviewer for pointing us to this baseline. We note that the INR-based method we compare with (i.e., STRAINER [37]) is from NeurIPS 2024.

Additionally, in this rebuttal, we include comparisons with three more baselines (see our response to Reviewer eMg2 (W4+W5)).

We could not find a code base for [B], and due to the limited time of the rebuttal, we couldn't code it from scratch. However, we promise the reviewer to include discussion (and/or results) of [B] in our revised paper.

[B] Conv-INR: Convolutional Implicit Neural Representation for Multimodal Visual Signals, ArXiv 2024

(Q3) Generalization validation on larger datasets.

We thank the reviewer for this suggestion.

For more diverse OOD evaluation, we report the results of an FFHQ-trained-UGoDIT-6 using 100 ImageNet test images and 20 CBSD65 test images for the SR task.

We would like to clarify that we include an evaluation in Appendix A.2, where we test an FFHQ-trained-UGoDIT-6 on 20 natural test images from ImageNet for the super-resolution task in Figure 7 (with visual examples in Figure 10).

The results demonstrate that UGoDIT maintains strong performance across different datasets. We will include these in the revised paper.

Thank you for your response. I still believe that the lack of discussion and comparison with relevant INR-based approaches is a key limitation of this work. Such as

[1] Single Image Super-Resolution via a Dual Interactive Implicit Neural Network (WACV 2023)

[2] Implicit Neural Networks with Fourier-Feature Inputs for Free-breathing Cardiac MRI Reconstruction (2023)

are directly related and should been considered for comparison or discussion.

Second, if the authors attribute the saturation effect to the encoder quickly capturing shared structures in the training images, as stated — “We hypothesize that the saturation effect arises because the encoder quickly captures the shared structure present in the training images, and adding more examples beyond this point yields diminishing returns.” — then a more meaningful experimental design would be to train the encoder on a broader and more diverse set of natural images beyond just faces and mri, with increasing dataset sizes, rather than only evaluating on natural images.

We thank the reviewer for their comments and feedback. We are glad that the reviewer finds that our paper addresses overfitting to noise and high computational cost at inference for DIP-based methods, and acknowledges that we provide a thorough and extensive evaluations. As suggested by the reviewer, we will include the the limitations in the main body of the paper.

(W1) Theoretical justification to why sharing the initial encoder layers would learn better feature learning beyond what's currently mentioned in section 3.2.

We thank the reviewer for this valuable comment. While our discussion in Section 3.2 provided intuitive motivation, we now offer a justification from the literature of multi-task representation learning. In our setting, each degraded image corresponds to an image-specific inverse problem, which can be interpreted as a separate task.

Our setup aligns with the multi-task learning framework studied in [A, B], where a shared representation (encoder $h_\phi$) is learned jointly across multiple tasks, each with its own function (i.e., task-specific decoder $g_{\psi_i}$ for $i \in [M]$). Specifically, Theorem 3 in [A] and Theorem 2 in [B] demonstrate that this structure (i) reduces generalization error across tasks and (ii) promotes robust and transferable feature learning in the shared encoder. Importantly, these theoretical results rely on the structural assumption of a shared encoder with task-specific heads.

In contrast, the single-encoder, single-decoder setup (as in Eq. (5)) violates this assumption, since the decoder receives gradients from multiple tasks simultaneously. This prevents specialization and introduces gradient interference — a known challenge in multi-task learning [C]. Consequently, the theoretical benefits shown in [A, B] do not extend to the single-decoder case which motivates our proposed multi-decoder method. We will include this discussion in the revised paper.

[A] A Model of Inductive Bias Learning, JAIR 2000
[B] The Benefit of Multitask Representation Learning, JMLR 2016
[C] Gradient Surgery for Multi-Task Learning, NeurIPS 2020

(W2+W4+Q2) Is there bias in training set? Impact of the selection of training set. Evaluation on more than 20 test images. Performance of UGoDIT when the training set differs substantially from the test set.

We would like to emphasize that we selected the training and testing images completely at random. This includes the set of images in Figure 15, our in-distribution (Section 4), and out-of-distribution (Appendix A) evaluations. This means that there is no intentional, skin-tone, or favorable bias.

We note that the selection of images in our OOD experiments is done such that the training and testing sets are substantially different. For example, the faces training images are the degraded versions of the ones in the first row of Figure 15 while the testing images are the ones in Figures 8 (a bird) and 9 (a front view of a cinema building).

To fully address the comment, we evaluate six models of UGoDIT-6, each trained with a distinct set of randomly selected training images $Y_i$, and baselines using the same 100 test images for the SR task. Distinct training sets mean that for any $i$ and $j$, $Y_i \cap Y_j = \emptyset$.

As observed, the averaged PSNR of our method ranges between 30.58 dB to 30.95 dB, i.e., the variability using different small training sets is very small (less than 0.4 dB). This indicates that the in-distribution results of UGoDIT is not highly sensitive to the selection of training images.

In our response to Reviewer usw1 (Q3), we report the results of an FFHQ-trained-UGoDIT-6 using 100 ImageNet test images and 20 CBSD65 test images for the SR task (additional OOD evaluation).

For MRI, we refer the reviewer to our additional results using additional sampling patterns in our response to Reviewer oaHC (W4+Q3).

We will include both results to the revised paper.

(W3) The paper is verbose; equations are repeated, and the description of UGoDIT and its intuition should be streamlined.

We thank the reviewer for this helpful comment. In the revision, we will refine the writing near the optimization formulation of UGoDIT and the alternating updates of Eq.(3) by consolidating related derivations, and move secondary details to the appendix.

We agree with the reviewer that equations (3) to (5) have similarities, especially in terms of using a measurement consistency term and an autoencoding term. We will update the writing to reduce redundancy. That said, we respectfully clarify that the equations serve distinct purposes: Eq. (3) introduces the alternating updates to solve Eq. (2). Eq. (4) defines the test-time optimization. Eq. (5) introduces the baseline with the shared decoder.

(Q1) Can the authors provide more analysis on the OOD capabilities of UGoDIT?

We thank the reviewer for this question. We include a detailed out-of-distribution (OOD) evaluation in Appendix A, where we assess UGoDIT’s performance when tested on distributions that differ from the training set. Specifically:

• In Appendix A.1, we train UGoDIT on knee MRI scans and test it on brain scans. We observe that performance degrades compared to in-distribution cases. See Figure 6.
• In Appendix A.2, for the super-resolution task, we train on FFHQ (dataset of faces) and test on ImageNet (see Figures 8 and 9). UGoDIT shows better convergence and improved final PSNR over standalone DIP (aSeqDIP), demonstrating robustness under natural image distribution shifts.

We will clarify and emphasize these findings more clearly in the revised version.

(Q2) Can the authors provide visualizations or insights on the quality of features learned in the shared encoder?

We thank the reviewer for this question. It is indeed important to shed more light on the learned features.

In Appendix D, we include a study analyzing the features of a UGoDIT encoder trained on MRI data. We analyze the encoder outputs at various layers and compare them with those from a single-decoder baseline.

Our analysis shows that the shared encoder in UGoDIT tends to learn smoother and more structured intermediate representations that are more robust to input variations.

As for visualizations, we note that features at the output of the trained encoder are of much lower dimension and do not typically hold semantic meanings. We will add more context to further clarify this point.

(Q4) Can the authors comment on the stability to overfitting noise over a long number of iterations of UGoDIT?

We thank the reviewer for this question. We partially address this in Appendix C, where we evaluate UGoDIT’s robustness to noise overfitting across different values of the regularization parameter $\lambda$, using 10000 iterations. Additionally, in Appendix A.1, we run UGoDIT (and other variants) for 30000 iterations in the OOD setting.

We would like to clarify that the autoencoding term $\|\|g_{\psi_i} \circ h_\phi(z_i) - z_i\|\|_2^2$ acts as an implicit constraint that encourages the output of the NN (i.e., the reconstruction) to remain close to the input. This promotes stability during optimization and mitigates overfitting to noise in the measurements. For this reason, we hypothesize that even with substantially more iterations, PSNR curves will plateau rather than degrade, as the network converges to a solution that satisfies both the data fidelity and autoencoding terms.

Due to NeurIPS restrictions on including external links, we were not able to share plots for running UGoDIT for more than 30000 iterations. However, we promise the reviewer that we will include experiments with more iterations in the revised version to further empirically support this claim.