Robustness of Deep Learning for Accelerated MRI: Benefits of Diverse Training Data

Thanks for noting that the problem we consider has 'significant real-world relevance' and that 'the experiments are performed on a large range of MRI reconstruction datasets with multiple real-world types of distributional shifts'.

The main concern of the reviewer is whether our results go beyond those in [1-6]. Here is how our contribution are different from [1-6], and how they go significantly beyond those existing works:

• F. Knoll et al. [1] and J. Huang et al. [6] characterise the severity of specific distribution-shifts, and suggest the use of transfer learning to avoid distribution shifts. Contrary, we investigate the role of training data diversity in influencing robustness and find that diverse training data can significantly improve robustness.

• C. Ouyang et al. [2] propose an approach that modifies natural images for training MRI reconstruction models. Contrary, we consider models trained in the conventional way, i.e., training on raw k-space data, and analyse systematically how combining different data-distributions affects robustness, and we find that it can significantly improve out-of-distribution robustness.

• Similar to F. Knoll et al. [1] and C. Ouyang et al. [2], SUH Dar et al. [3] propose transfer-learning to train a reconstruction model on natural images and then fine-tune it on MR images. The goal of their work is to increase training data efficiency. X. Liu et al. [5] propose a special network architecture to improve the performance of training on multiple anatomies simultaneously for. Their results are based on retrospectively under-sampled DICOM images and the method only works for emulated single-coil data. Both works focus on improving in-distribution performance and do not study the robustness issue. Our objective is to study how performance and robustness of models is affected by data diversity of raw k-space data.

• F. Caliva et al. [4] consider the problem of adversarial robustness, whereas we consider the problem of robustness under natural distribution-shifts.

Additionally, the reviewer is concerned that the choice of studying U-net is over-simplistic as better models exist. We choose the Unet model since it is a well known baseline and our results are agnostic to the architecture. However, we agree it is important to explicitly show this, and have carried out a significant number of additional simulations after submitting the paper. Specifically, we have expanded our results in Section 3 and 7 with results obtained from the End-to-end VarNet (mentioned by the reviewer), a state-of-the-art model for MRI reconstruction, and also with results from ViT to demonstrate that our results continue to hold true even for non-convolutional architectures.

The reviewer also raises ethics concerns regarding the datasets used in our work. We have checked the terms and conditions of each dataset carefully, and confirm that they are safe to use for research, and that no conditions are violated. Most of the dataset are specifically released for image reconstruction research, which is what we do in this paper. We hope that our response clarified the reviewer's concern and if yes would be happy if the reviewer would consider raising their score.

References:

[1] F. Knoll et al. “Assessment of the generalization of learned image reconstruction and the potential for transfer learning.” Magnetic Resonance in Medicine, 2019.

[2] C. Ouyang et al. “Generalising Deep Learning MRI Reconstruction Across Different Domains.” arXiv:1902.10815, 2023.

[3] SUH Dar et al. “A Transfer-Learning Approach for Accelerated MRI Using Deep Neural Networks.” Magnetic Resonance in Medicine, 2020.

[4] F. Caliva et al. “Adversarial Robust Training of Deep Learning MRI Reconstruction Models.” arXiv:2011.00070, 2021.

[5] X. Liu et al. “Universal Undersampled MRI Reconstruction.” Medical Image Computing and Computer Assisted Intervention, 2021.

[6] J. Huang et al. “Evaluation on the generalization of a learned convolutional neural network for MRI reconstruction.” Magnetic Resonance in Medicine, 2022.

[7] R. Taori et al. “Measuring Robustness to Natural Distribution Shifts in Image Classification.” In Advances in Neural Information Processing Systems, 2020.

I would like to thank the authors for the clarification, and I am pleased to see that the authors have experimented with Variational networks and ViT. These results would consolidate the conclusions in the manuscript.

I would also like to thank the authors for clarifying their conclusions: 1. similar to those with image classification, deep networks also suffer from distributional shifts (in the manuscript, termed as robustness, although for most of readers of ICLR, the term robustness more often refers to adversarial robustness which the manuscript fails to discuss); 2. diversity of training data enhances distributional robustness. I am also pleased to see the authors to have read through the suggested existing works. With that being said, I would still kindly remind that these conclusions/contributions still may not go beyond those by [1-6]:

1. Deep MRI reconstructions suffer from robustness issue: This has been the main focus of [1,2,6] (robustness against distributional shifts) and [4] (robustness against adversarial perturbations). I appreciated the increased scales of datasets and experiments in the manuscript. However, readers may argue there were no significant breakthroughs over those by [1,2,4,6].

2. Incorporating training datasets with more diversities enhances the robustness: I agree this is an interesting empirical result. However, this conclusion is neither novel, nor rigorous:

a) Novelty: The benefit of diversity of training data has been the focus of [2,5]. [2] finds that training with MS-COCO, which bears more diversity (at the patch scale) than application-specific medical datasets, yields the best overall distributional robustness (evaluating the model on datasets that are different from those for training). [5] finds that aggregating multiple different training datasets leads to the best overall performance compared with dataset-specific training (with their new workflow). Despite the increased scale of experiments in the manuscript, it is difficult to identify major breakthroughs beyond those reached by [2,5].

b) Rigor: The term diversity is quite a sloppy umbrella term. Theoretically, the data diversity would only make sense when the diverse training data could better cover the distribution of the test data (at least better covers its support). However, this has been ignored by the authors, and adding further in-depth analysis/experiment requires substantial amount work that falls beyond the scope of a revision. Also, [5] finds that ... simply mixing all the datasets does not lead to an ideal model... (Table 1, Independent versus Shared), which contradicts a bit to the claims of your manuscript. This result in [5] suggests that diverse training data does not necessarily lead to benefits, while the authors did not provide an in-depth analysis to the theoretical ground behind the scene.

In summary, while the manuscript presents a practical problem setting in together with extensive empirical results, the relatively modest level of novelty and theoretical depth may not align well with the standards and scope of ICLR.

Many thanks for the review and for noting that this work is 'one of the largest in terms of experiments on the effects of data for MRI reconstruction', and that the findings can be useful for practitioners.

• The reviewer raises concerns about (i) the lack of confidence intervals in Section 3, where we show that training a model on two distributions gives similar performance to training models on the distributions separately, and (ii) that global SSIM scores for evaluating pathology reconstructions (Section 5), could be problematic due to small pathology sizes.

• Regarding concern (i), we changed Figure 3 to include the mean and standard deviations over 5 independent runs and added Figure 14 which also reports the mean and standard deviations over 5 independent runs for the same experiment as in Figure 2 but on smaller datasets. It can be seen from Figure 14 that the confidence intervals are extremely small (almost not visible) for models trained on more than 3k images, which concerns the vast majority of experiments in our paper, and thus we did not add confidence intervals throughout, since this wouldn't be justified given the cost of thousands of GPU hours and the fact that the confidence intervals would be very small.

• As a response to concern (ii), we have now re-evaluated the models only on the region containing the pathology (instead of computing metrics on the entire image), see the revised Figure 5. We further distinguished between small pathologies and relatively large pathologies.

• We also note that we do not claim that models trained only on healthy images can reconstruct pathologies. Rather, our results indicate that models trained on data without pathologies can reconstruct pathologies as accurately as models trained on data with pathologies. Regarding our reason to submit to the 'datasets and benchmark category': One category mentioned regarding the scope of the datasets and benchmarks tracks is often (e.g., at NeurIPS) 'Data-centric AI methods and tools, e.g. studies in data-centric AI that bring important new insight', we believe our work falls into this category.

In the following we address the questions from the reviewer:

• Answer to Q1: We have expanded our results in Section 3 and 7 with results obtained from the End-to-End VarNet and also with results from ViT to showcase that our results hold true even for non-convolutional architectures. We initially chose the U-net since we expect our results to be relatively agnostic to the architecture of the end-to-end network, and our results indicate that they are, and the U-net is easy and fast to experiment with.

• Answer to Q2: The 7T database paper does not say much about a data release, and it only mentions 24 volunteers. Noting this, it might be good to list the number of subjects for each dataset in Table 1. We made a mistake with the citation and have now corrected it. We thank the reviewer for noticing this error. We also included the number of subjects in Table 1.

• Answer to Q3: While transfer learning can definitely boost the performance on the target distribution, we do not advocate for it as it deteriorates robustness (see the new Appendix E.3 in the revised manuscript).

• Answer to Q4: In Section 7, the models are substantially larger (except for VarNet) than in the previous sections and the results show that the conclusions from the previous sections still apply (i.e., diverse training sets give the same in-distribution performance but better out-of-distribution performance; hence, increased effective robustness).

• Answer to Q5: While generative models offer an interesting path for investigation, we focus exclusively on models trained end-to-end, since those models (in particular the end-to-end VarNet) are widely recognized in the community and currently achieve state-of-the-art results in accelerated MRI.

If our responses and the new experiments and changes in the paper have alleviated the reviewers' concerns we would be happy if the reviewer would consider raising their score.

References:

F. Knoll et al. “Assessment of the generalization of learned image reconstruction and the potential for transfer learning.” Magnetic Resonance in Medicine, 2019.

R. Taori et al. “Measuring Robustness to Natural Distribution Shifts in Image Classification.” In Advances in Neural Information Processing Systems, 2020.

T. Nguyen et al. “Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP.” In Advances in Neural Information Processing Systems, 2022.

Many thanks for the review and the positive feedback.

• The reviewer has raised a valid and important concern regarding the absence of confidence intervals in Section 3, where we demonstrate that training a model on two distributions yields comparable performance to training on individual distributions. In the revised version, we incorporated mean and standard deviations over 5 independent runs over different data and starting from a different initialization in Figure 3 and we introduced a new Figure 14. From Figure 14, it can be seen that if the dataset size is beyond 3k, the confidence interval is vanishingly small (not visible) and thus we did not add confidence intervals throughout, since this wouldn't be justified given the cost of thousands of GPU hours and the fact that the confidence intervals would be very small.

• The reviewer emphasized the need for a discussion on model capacity and hyperparameter tuning. To address this, we have conducted experiments akin to those in Section 3 but utilizing a much smaller U-net, see the new Appendix E.1. The results align with those obtained with the larger U-net. Furthermore, we describe the process of determining hyperparameters in the new Appendix D. We thank the reviewer for suggesting this.

• We confirm that the reviewer has correctly observed that we track the in-distribution loss as an attempt to mitigate distributional overfitting. This decision is based on the results in Section 6 which suggest that distributional overfitting begins when training is close to convergence.

• Concerning the reviewer's remarks on visualization, we appreciate the interesting suggestion. The current choice of visualization method aligns with the conventions established in the effective robustness literature (see R. Taori et al., 2020, and T. Nguyen et al., 2022). This decision was made to maintain consistency and facilitate comprehension for readers who are already familiar with the effective robustness framework.

References:

R. Taori et al. “Measuring Robustness to Natural Distribution Shifts in Image Classification.” In Advances in Neural Information Processing Systems, 2020.

T. Nguyen et al. “Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP.” In Advances in Neural Information Processing Systems, 2022.

Many thanks for the review, for acknowledging the careful design of our experiments, and for noting that our findings are useful for researchers in medical imaging machine learning.

• The reviewer suggests that extending the current scope of our work to include other medical applications like medical image segmentation would be beneficial. We agree that a data-centric study on improving robustness is also important in other medical applications like detection and segmentation, but since those problems and datasets are quite different from image reconstruction, they are unfortunately beyond the scope of this paper. We did, however, add more experiments on other image reconstruction networks like VarNet and ViT to further strengthen our claims.

• The reviewer would like to see experiments regarding distribution-shifts with respect to vendors and anatomic views. In Section 7 of the original manuscript we have the mentioned distribution-shifts, however they were hidden since we did not point out that there are those shifts. For example, the distribution shift from fastMRI knee to NYU dataset is a distribution shift in anatomic views, and from fastMRI to Stanford 2D or CC-359 is a distribution shift related to vendors. We have updated the corresponding paragraph in Section 7 of the revised manuscript to emphasize that those shifts are in terms of vendors and anatomic views.

• The reviewer would like to see experiments beyond 4-fold acceleration. While it could be interesting to see what happens at higher acceleration rates, we chose the 4-fold acceleration setup to provide more practical insights, as exceeding 4-fold acceleration tends to diminish clinical relevance (M. Muckley et al., 2021; A. Radmanesh et al., 2022). We added the explanation to our revised manuscript in Section 2.

References:

M. Muckley et al. “Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction.” IEEE Transactions on Medical Imaging, 2021.

A. Radmanesh, et al. "Exploring the Acceleration Limits of Deep Learning Variational Network–based Two-dimensional Brain MRI." Radiology: Artificial Intelligence, 2022.

Many thanks for the feedback and for noting that the study is methodologically designed, well written, and that our main finding that training on diverse datasets improves out-of-distribution generalization rests on a large variety of experiments.

• Regarding that we do not 'explicitly study the impact of dataset size on model performance with out-of-distribution data': Section 4 and 5 contain experiments with different dataset sizes, but we indeed do not consider different training set sizes throughout for the following reason: Training on larger datasets leads to better models, but not necessarily more robust models. Specifically, in and out-of-distribution performance is correlated for the distribution shifts we consider, see [3] and Figure 4 in our paper, and see [1,2] for correlations of in-and out-of-distribution relations in classification and beyond. In Figure 4, increasing the training set size improves a model in that it improves both in and out-of-distribution performance (it moves on the line) but it is not more robust than expected of a model with a given in-distribution-performance. Contrary, training on more diverse data, leads to an effective robustness gain in that it gives better out-of-distribution performance relative to the in-distribution performance.

• Regarding results for vendor-related distribution shifts: In Section 7 of the original manuscript we consider vendor shifts. Specifically, in Section 7 we introduce distribution-shifts from the fastMRI dataset, which are obtained by Siemens scanners, to the Stanford 2D or CC-359 dataset, which are obtained by GE scanners. In both cases we find that training on a diverse dataset improves robustness. We have updated the corresponding paragraph in Section 7 in the revised manuscript to emphasize this distribution-shift.

• The reviewer states that we do not adequately discuss the limitations of our results and further suggests that we discuss the training times of our models: In the revised manuscript we have expanded the discussion on limitations by adding that in practice it might be difficult or expensive to collect diverse and large datasets. We believe that the training time is not a significant concern, given that the current bottleneck in medical imaging is data availability rather than computational resources. Nevertheless, we agree that stating the training times is important, and we have included this information in Appendix D of the revised manuscript.

• Regarding distribution-shifts related to visual aspects of MRI images vs. how these shifts manifest in k-space data: Since k-space and image are related through the Fourier transform, a distribution shift in the image space is also a distribution shift in the k-space.

• The reviewer asks if mixing data from two distributions, P and Q, in a disproportionate manner like ‘P+0.005Q is sufficient for good model training’: In terms of in-distribution performance, we find that skewed data up to 90%P and 10%Q do not pose a problem (see Figure 3 in the revised manuscript). In case of distribution-shifts from distribution P to distribution Q, we do not have data from distribution Q available. If data from distribution Q was available, then including it into the training set would eliminate the distribution shift.

References:

[1] B. Recht et al. “Do ImageNet Classifiers Generalize to ImageNet?” In International Conference on Machine Learning, 2019.

[2] R. Taori et al. “Measuring Robustness to Natural Distribution Shifts in Image Classification.” In Advances in Neural Information Processing Systems, 2020.

[3] M. Darestani et al. “Measuring Robustness in Deep Learning Based Compressive Sensing.” In International Conference on Machine Learning, 2021.