Meta-Prior: Meta learning for Adaptive Inverse Problem Solvers

We thank the reviewer for taking the time to review our work and for their constructive feedback. Please find our response below:

W1. Unpromising experimental results. We respectfully disagree with the reviewer. Our experiments show that the meta-learning framework can be successfully applied to the inverse imaging problem, and that it can even be leveraged to effectively learn to solve inverse imaging problems in a fully unsupervised setting.

Q1. Difference between learned and analytic solution. We appreciate the reviewer's keen observation regarding the dissimilarity between the learned and analytic solutions in Fig. 2. The noted differences are indeed present, and we acknowledge the stochastic nature of the task as a contributing factor. Unfortunately, due to space constraints, we are unable to include an additional figure displaying the absolute difference map. However, we have added a mention to this observation in the text, emphasizing the inherent stochasticity of the task and its impact on the learned solution.

Q2. Worse performance of meta-model. In Figure 3, we evaluate the performance of the meta-model on each finetuning task. Note that the meta-model is the model with “barycenter weights” that is meant to perform averagely well on the sum tasks (this corresponds to the outer loss function). As a consequence, it is expected that the inner model (blue curve) performs worst than the meta model (orange curve).

W2 & Q3. Comparison with additional methods. We agree with the reviewer that the submitted paper was missing comparisons. We have added a comparison with several methods, namely standard regularization-based approaches, which are widely used in real imaging settings due to their versatility, as well as with the DPIR method, which reaches state-of-the-art performance in synthetic imaging problems. Note that these methods can be seen as unsupervised techniques. up to a single scalar regularization parameter requiring to be fine-tuned. The proposed method performs on par with these different algorithms. Furthermore, we also compare the results obtained by our backbone architecture when trained in a classical supervised fashion on the test task. As expected, this latter approach outperforms the proposed method.

We thank the reviewer for taking the time to read our work and for their constructive feedback. Please find our response below.

1. Precisions on datasets & baselines. We thank the reviewer for raising this point, which is in line with the comment of reviewer jubY. We have added the missing information regarding the datasets to the paper. Our experiments in the super-resolution setting confirm that the MAML framework can be leveraged to learn image reconstruction networks while enforcing data consistency only at fine-tuning time. However, we do not make the same observation on the MRI problem, which may be explained by the strong distribution shift between the training and testing tasks and datasets.

2. Comparison with additional methods. We agree with the reviewer that the submitted paper was missing comparisons. We have added a comparison with several methods, namely standard regularization-based approaches, which are widely used in real imaging settings due to their versatility, as well as with the DPIR method, which reaches state-of-the-art performance in synthetic imaging problems. Note that these methods can be seen as unsupervised techniques, up to a single scalar regularization parameter requiring to be fine-tuned. The proposed method performs on par with these different algorithms. Furthermore, we also compare the results obtained by our backbone architecture when trained in a classical supervised fashion on the test task. As expected, this latter approach outperforms the proposed method.

We thank the reviewer for taking the time to read our work and providing constructive feedback. Please find our response below.

Unclear motivation. We have amended the abstract and the introduction to clarify the paper’s goal and the main messages. The main motivation of our paper is learning in contexts where no ground truth is available for a given reconstruction task. This context typically arises in MRI, where only partial measurements for a given k-space sampling scheme $A$ are available. When changing the sampling scheme $A$, one does not have access to the true data $x$ but only to the measurement $y$ and the sensing operator $A$. Thus supervised approaches cannot be used to directly train a network in this setting [1]. However, in many cases one has access to pairs $(x, y)$ for different sensing operators $A$. The motivation of our paper is to propose a principled method to leverage these extra data points. Existing methods either rely on handcrafted priors, that have limited performances, or deep implicit prior, that struggle to generalize to non-toy imaging settings. We propose to frame the problem using meta learning. We show theoretically that this framework allows to learn a prior that is used to complete missing information in $\operatorname{Ker}(A)$ and show that this framework allows to give satisfactory reconstruction. Our motivations are thus three-fold: designing a principled framework for explicit prior learning, pulling information from multiple sources (tasks) and accelerating sensing operator adaptation.

No baseline methods and novelty with regards to MAML. We have added comparisons with baseline methods for image reconstruction (see common answer). The proposed method is an extension of the MAML approach. The novelty in our work is to show how this framework can be adapted in this context, with two main settings (supervised and unsupervised). We further give insights on what this model can learn, and show with a wide variety of experiments that the proposed approach works in practical cases.

Comparison with other methods. We agree that the initial submission was missing comparisons, which have now been added to the paper. In essence, the proposed method performs on par with the SOTA unsupervised image restoration DPIR algorithm (Zhang et al, 2021) after only 50 steps of finetuning on the test task. However, the proposed model performs less well than its counterpart trained in a supervised setting specifically on the task of interest.

Clarification of Figure 5. Following the reviewer’s suggestion, we have replaced Figure 5 with a table (see Table 1) including a comparison with various baselines. We believe that this change improves the clarity of the paper.

References

[1] Shimron, Efrat, et al. "Implicit data crimes: Machine learning bias arising from misuse of public data." Proceedings of the National Academy of Sciences 119.13 (2022)

We thank the reviewer for taking the time to read our work and for providing constructive feedback. Please find our response below.

1. Methodology

1.a Restrictions on the measurement operator A. Our current implementation has only been trained on image-to-image tasks of the same shape; however, this could be waived in a future implementation. Moreover, there is however no restriction on the dimension and nature of the data used at the fine-tuning stage due to the versatile backbone primal-dual network. For instance, in the MRI experiment, the data of interest lives in $\mathbb{C}$ while the network was trained on variables in $\mathbb{R}$. So there are no direct requirements on $A$ to apply our framework. However, as exposed in Theorem 3.1, the larger the kernel of $A$, the more diverse the task needs to be for the model to learn to complete the missing information and give good performances.

1.b MAML training cost. The full model is being fine-tuned at the level of the inner optimization. We acknowledge that MAML is a heavy training procedure; however, PDNet is a light model with only 361 trainable parameters per layer, i.e. a maximum of 38k parameters in its deepest configuration. We however agree that for larger models, such as UNets, the approach may face memory issues, which were not a problem in the current setting. We added a comment on this point, thanks for pointing it out.

2. Experiments

2.a Clarification on the training data. We thank the reviewer for pointing this out; we have amended the paper to clarify this matter (see “Training data” as well as experiments’ specific paragraphs). Our network is trained on the BSD500 training and test set combined, containing 400 natural images, and on which all of the 4 inner problems are solved; the outer loss is computed on the test set of the BSD500 images, containing 100 natural images. Finetuning: (i) for SR the model is finetuned on the same dataset used at the inner-level of the MAML task, i.e the 400 natural images from BSD500. The test task is evaluated on the Set3C dataset. (ii) for MRI the model is finetuned on 10 slices from the fastMRI validation dataset, on the same MRI mask that will be used at test time. At test time, the model is evaluated on (different) 10 slices from the fastMRI validation set.

2.b/c Comparison with the task-specific model. We thank the reviewer for the suggestion; we have added in Table 1 a comparison with the PDNet architecture trained on the training set (inner level problem) for the test task of the BSD400 dataset for the SR task, and on the fine-tuning set of the MRI problem. This provides a fair upper bound on the best performance achievable by a model trained specifically on the task of interest. Our experiments show that the MAML model does not perform as well, but performs on par with other baseline unsupervised approaches that were also added to the paper.

2.d Unsupervised fine-tuning Yes, in the unsupervised setting, the inner optimization is also unsupervised. This is stated in the paragraph after equation (2) in the paper, stating: ``It can be either the supervised loss $\mathcal{L}\_{\text{sup}}$ or the unsupervised loss $\mathcal{L}\_{\text{uns}}$, with an extra regularization term controlled by $\lambda>0$ to ensure that the model $f_{\theta_i}$ is close to the meta-model $f_{\theta^*}$. When $\mathcal{L}\_{\text{inner}}$ uses $\mathcal{L}\_{\text{sup}}$ (resp. $\mathcal{L}\_{\text{uns}}$), we call this problem the supervised meta-learning (resp. unsupervised meta-learning). Finally, the outer training loss $\mathcal{L}\_{\text{outer}}$ is used to evaluate how well the model $f_{\theta_i}$ generalizes on the task $\mathcal{T}\_i$, using $\mathcal{L}\_{\text{sup}}$.”

2.e Benefits of the proposed approach We believe that the main advantage of our approach lies not so much in the model itself, which is small size and applied to simple imaging tasks, but rather in enabling us to train a model in a situation where no ground truth is available for the specific task considered. A notable instance of this problem is MRI, where the physical acquisition procedure prevents direct access to true data $x$. To learn in such a setting, we leverage tasks for which pairs $(x, y)$ for various sensing operators $A$ are available to propose a principled meta-learning method. Our approach addresses the limitations of handcrafted and implicit priors, and we show theoretically that this framework allows us to learn a prior that is used to complete the missing information in $\operatorname{Ker}(A)$. We have restructured the introduction of the paper to better highlight this motivation.