Self-supervision Meets Bootstrap Estimation: New Paradigm for Unsupervised Reconstruction with Uncertainty Quantification

We are really grateful for your recognition of our work and contributions. Below, we address your suggestions and comments.

Reply to Weaknesses

• While we would love to add more comparison with other advanced MRI reconstruction methods, unfortunately, it requires substantial time and access to their original codes (some are not shared though) for an informed and detailed comparison. We add citations to these referred papers, and will further explore them in our future work.
• The differentiability of the aggregation function becomes a main concern in our derivation but we found it operable for backpropagation and thus it won’t influence the optimization. For example:
  • With or without the aggregation function, the gradients of the reconstruction model parameterized by $\theta$ are always: $\frac{\partial{\mathcal{L}}}{\partial{\theta}}=\frac{\partial{\mathcal{L}}}{\partial{f_{AF}}}\frac{\partial{f_{AF}}}{\partial{f}}\frac{\partial{f}}{\partial{\theta}}=\frac{\partial{\mathcal{L}}}{\partial{f}}\frac{\partial{f}}{\partial{\theta}}$
  • As for the gradients of $y$ and $U$, we add an explanation after Equation 7, that aggregation function results in a re-undersampling mask which may not influence the differentiability to $y$. The gradients of $U$, on the other hand, may be influenced, but they are not optimized for MRI reconstruction, thus it may not a problem.

Reply to Questions

• In the original version of our paper, uncertainty has been limited to the randomness of undersampling, while the Bootstrap resampling actually “simulates” it. It’s true that for the reconstruction of re-undersampled images, Bootstrapping incorporates uncertainty, but in inference, no re-undersampling will be conducted.
• We’re extending our work as we discussed in Conclusion. There would be a trivial extension that applies virtual sample and pseudo resampling in similar tasks related to Compressed Sensing. Other image-to-image translation tasks like super-resolution and deblurring could also benefit from our work by defining virtual sample sets in particular domains. We’re open to further comments.
• At this stage, Mote Carlo method would not be appliable as it incorporates sequential sampling and requires extra control for the re-undersampling purpose, etc. We will further explore this idea.

Thank you very much and please let us know if you have any further comments.

Thanks for your detailed comments. In response, we provide explanations below.

Response to Weakness

• Thank you for pointing out the missing reference. In this revision, we add the discussions about it.
• Figure 1 presents the pipeline, and we add math equations to Section 2. We further add brief introduction to some SSR methods to Appendix E.
• The paper is substantially reorganized responding to the related comments, including enhancing clarity and explanations. For take-home information, we argue and evidence that self-supervised MRI reconstruction can be modeled as error-oriented parameter estimation, accordingly we introduce a novel method - Bootstrap estimation for SSR (BootRec).
• We polish the paper per your related comments.

About the contribution of our work:

• We emphasize the key attributes of our methodology in a new section - Section 4.3.

• As mentioned in the introduction and related work parts, the existing methods are mostly heuristic in their design. Our method makes a meaningful trial to answer the following questions with reasons beyond "it just works":

• Why use this re-undersampling mask instead of others?

• Why use this loss function instead of others?

• The re-undersampling methods, though not the only methods in SSR, it outperforms other models in similar settings [1,2], and our methods are also compared with other types of unsupervised approaches like Deep Imaging Prior.

• Thank you for providing useful information on MRI UQ. The uncertainty quantification part in our work is not the focused claim, we instead focus on the estimation of errors, so the notions become different from [3]. We instead compare our methods with error-oriented UQ in Appendix G, which shares the idea of training a separated network or module to predict uncertainty. Besides, although our methods do not include epistemic uncertainty in the design, it can cooperate with any such methods, including the Bayesian Neural Network in [3], and it is training-free. A detailed comparison and assessment in terms of UQ goes beyond this paper, thus we focus on reconstruction.

Response to Questions

• It’s a very good question to challenge the distribution of U. While the assumption of Bernoulli is a common deed in MRI communities, like the diagonal covariance matrix assumption in UQ and other tasks, it’s not satisfactory. Also, a 1D Cartesian mask may be ideal in the dimension of sampling, and in other cases, different points do have correlations. However, even if the sampling points are not independent, we may always find some “nearly independent elements” in the acquisition, such as the radial tracings in radial sampling or to conduct methods like whitening to reduce the correlation. As for the re-undersampling pattern, we believe we can guarantee independence.
• The paper is reorganized according to the related comments.

[1] Zero-shot self-supervised learning for mri reconstruction. In International Conference on Learning Representations, 2022.

[2] Parcel: Physics-based unsupervised contrastive representation learning for multi-coil mr imaging. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 14(8):1–12, oct 2022b. ISSN 1557-9964. doi: 10.1109/TCBB.2022.3213669.

[3] Bayesian Deep Learning for Accelerated MR Image Reconstruction. MLMIR 2018

[4] Uncertainty Quantification in Deep MRI Reconstruction

Thank you for your kind response.

• The interpretation is among the key aspects of our work. Indeed, we never claim the contribution of proposing re-undersampling or introducing self-supervision in MRI reconstruction.
• The field of uncertainty quantification is not very familiar to us so I may not be qualified to judge different uncertainty notions in this paper. In our observation, the errors seem to be more explicit and easier to judge since we can get the GT values. Further studies will be beneficial and thanks to your valuable perspective.
• Please be aware that our quantification leverages re-undersampling but is not limited to SSR tasks (Section 5.2 and Appendix G conduct estimations on arbitrary models). In fact, we observe minimal constraints associated with applying our methodology and find broader applications (see our discussion with reviewer FBWg) in self-supervised learning of image-regression tasks, though with limited space to extend the discussion.
• We'd make a final revision of our paper if possible.

We appreciate your comments. We made major revisions to address your comments and enhance the paper's readability. Here we provide the point-to-point responses.

Reply to Weaknesses

• We add a note to the Introduction section directing readers to the table of notations in the appendix. The equations in the main content are largely reduced as well (from 25 to 16).
• Do you mean the Us in Equation(5)? The two $$U$s refer to the “random variable” and “vector value” of the mask$U$ with display styles provided by the ICLR formatting instructions. Anyway, we remove the equations.
• We add explanations to the figures. Specifically, for Figure 4 in the original submission (i.e., Figure 5 in this revision), it shows the correspondence of the estimated results v.s. the actual values.
• We add a new figure - Figure 1 - to summarize the framework of MRI reconstruction, which may help explain our work better. The detailed settings of the previous methods are provided in Appendix F.

Further replies to the concerns:

• We agree and notice the inclusion of variance estimation becomes redundant and compromises the integrity of the article. Such discussion may be pertinent to another independent research.
• We will work on more theoretical analysis.
• The abstract is revised.

We would greatly appreciate if you could kindly revisit the entire manuscript at your convenience, in particular, check our significant revision. Please let us know if you have any further comments.

Thank you for your recognition of the novelty of our work, as well as to your helpful suggestions.

As we clarify in the summary of changes to all reviewers, our focus of this work is not on the Uncertainty Quantification (UQ). In the revision, we reorganize the paper for more focus on proposing a novel reconstruction method. Below, we address your specific comments.

The evaluation of UQ

• We add more discussion on UQ in the section on Related Work while focusing our quantification on MSE estimation in UQ. We also discuss the work you referred to. The UQ in our work is specifically referred to error estimation.
• We compare the estimation of MSE with residual magnitude regression of the recommended work [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging] (which should cover the family of similar methods) and SURE estimation of [Uncertainty Quantification in Deep MRI Reconstruction] in Appendix F. Unfortunately we cannot compare our work with the method in [Rigorous Uncertainty Estimation for MRI Reconstruction] because we do not have access to that paper's materials. We are investigating the implementation of more methods and further analysis may also be added later.
• For evaluation metrics, we find the correlation between the estimated and actual MSE serving as a good indicator for MSE-based UQ, since the ground truth is clear. Other existing evaluation measures mainly focus on variance and quantiles, so we are not able to use them.

Visual Presentation of Reconstruction Results

• Visual examples including the reconstruction results and error maps are added to Figure 8 in the revised paper.

The application of UQ

• Specifically for our work, the purpose of estimating MSE has a rather simple and special answer: that is to optimize it. That’s the core idea of our proposed self-supervised MRI reconstruction method, please refer to Section 5.3. Please be noted, this may not be generalized to other UQ notions or methods since they may either need training itself or be not suitable for optimization.
• The application of general UQ is worthy of discussion. We add a brief discussion at the beginning of Section 2.3a.