Deep Learning in Medical Image Registration: Magic or Mirage?

We thank the reviewer for their positive and helpful feedback, we are glad to find they found the paper having a clear motivation, detailed observations, clear illustrations and easy-to-read paper. We believe we have addressed all the remaining concerns and hope the reviewer increases their score and advocates for the acceptance of our paper.

”New solutions or deeper insights are missed.”, “Findings in Section 4,5,6 are somewhat valuable, but the relevancy between different phenomena is not very clear”

Our work serves as a bedrock for contextualizing the performance of current state-of-the-art deep learning and classical registration methods, and identifying the critical confounding variables responsible for the performance gaps observed in DLIR and classical methods. We show empirically the two confounding variables – instrumentation bias of classical methods, and label supervision are majorly responsible for the superior DLIR performance of in-distribution data.

Our work provides an independent, fair and unbiased evaluation of both DLIR and classical methods, and show that in the unsupervised case, classical baselines outperform DLIR methods without having any in-distribution data – making it useful in the low-data regime. Furthermore, since anatomical labels are an intrinsic property of the anatomy (hence being domain-invariant), one may expect label supervision to add some robustness to domain shift. However, this is not the case (Sec 6).

These experiments naturally imply a recipe to use for a given instance of registration (mentioned in Sec 7). Moreover, these results also allow us to reconsider deploying expensive data collection pipelines, especially when domain shift may be expected at test time (as a consequence of the finding in Sec 6). Moreover, observations in Sec.4. and Sec.6. serve as a problem statement for DLIR-based methods to alleviate these limitations. We urge the reviewer to also read more details about this in the common rebuttal.

What are the potential directions of future work based on this paper?

We appreciate the reviewer's interest in future research directions. Our limitations section outlines several avenues for further investigation. This study can serve as a foundational basis for examining and disentangling the variables responsible for robust performance in various contexts, such as: Multimodal Registration: Extending our analysis to multimodal registration scenarios to explore the applicability of our findings across different imaging modalities. Non-Brain Anatomy: Investigating the generalizability of our results to other anatomical regions beyond brain MRI. Diverse Registration Settings: As suggested by Reviewer 23gn, conducting experiments on inter-subject, intra-subject, and atlas-based registration to provide a comprehensive understanding of performance across different registration paradigms. We have also incorporated these future work suggestions more explicitly in the revised paper to guide readers on how our findings can be extended and applied in broader contexts.

We thank the reviewer for their insightful and helpful feedback, which has significantly contributed to improving the quality of our paper. We believe we have addressed all the remaining concerns and hope the reviewer will consider increasing their score and advocating for the acceptance of our paper.

The training set of deep learning models may not be sufficient

All methods are trained on the modified OASIS training split of 364 images, yielding approximately 132k inter-subject training pairs (364 * 363). This is a substantial number of image pairs, and in practice, most DLIR methods converge within 2-3 epochs. Furthermore, most baselines like LapIRN train on an even smaller subset of OASIS, approximately 62k training pairs (250 * 249) (although we train on 364 images for consistency). Thus, the dataset size used is sufficient for a fair comparison with classical models.

The authors need to implement more experiments on images of more organs and tissues with monomodal and multimodal registration tasks

We have already acknowledged this in the Limitations section in our paper, which forms the basis for future research. We chose the inter-subject MRI brain registration problem due to its widespread adoption by the biomedical ML community, ensuring adequate reproducibility and unbiased evaluation. While we recognize the importance of exploring other organs and tissues, none of our analysis, baselines, or evaluation protocols use domain-specific knowledge, suggesting that our results are not strictly limited to brain MRI. Expanding to more variations is a priority for future work.

The subtitle “DLIR methods do not generalize across datasets” is almost common sense

We agree, this is almost trivial in hindsight! However, seeing the results in Sec.4. and Sec.5., one might expect the domain-shift performance to fall between the results of in-distribution unsupervised and supervised training, with supervised DLIR methods outperforming classical methods to some extent. However, this is not the case (please see common rebuttal for more details), showing that large amounts of labeled data would not necessarily help on datasets with even the same modality and resolution but slightly different intensity statistics. This raises a question for large-scale, annotated data collection efforts, and addressing current limitations of DLIR methods to be robust to domain shift.

Please explicitly clarify the advantages compared to the well-known lean2reg challenge.

Learn2reg challenge which focuses on methods to maximize registration performance (measured by label overlap, landmark distance, determinant of Jacobian) on a particular dataset with data/modality specific challenges. In contrast, we perform a meta-analysis and unbiased evaluation on the factors that lead to better performance of aforementioned registration algorithms. We find that DLIR methods attribute the performance improvement to architecture, loss functions, and training specifications. However, we show that most of the improvement is attributed to being able to learn from supervised label map objectives during training. Moreover, we show that training with label maps (which are intrinsic to modality and therefore invariant to the downstream modality – see more in common rebuttal) does not necessarily imply robustness in performance to domain shift. We have modified the “Contributions” subsection to emphasize these distinctions.

Does the conclusion still hold on large datasets? AMOS dataset provides lots of CT and MR images can be used for model training

The AMOS dataset, with 500 CT and 100 MR images, is comparable to the OASIS dataset in size (414 images). We selected the OASIS dataset for reproducibility and fairness considerations, as numerous DLIR methods provide pretrained models and training recipes for OASIS. The active community around OASIS and its long-standing presence has led to detailed analyses of dataset-specific challenges (for example, registration of smaller subcortical structures). Although the AMOS dataset is relatively new, it holds significant potential for future work, and we plan to explore it in subsequent studies.

Why not submit this work to the dataset and benchmark track?

Our work goes beyond merely reproducing or benchmarking existing methods. We provide a theoretical and empirical analysis of the factors responsible for improved performance in registration tasks, particularly the access to label maps during training, and their implications for robustness across similar datasets. This paper is a meta-analysis and unbiased evaluation, similar in spirit to [7].

We thank the reviewer for their time and insightful feedback, and are pleased to note the recognition of the originality of our work, the well-motivated design of our experiments, and the challenge posed to existing claims in the DLIR literature. We address some of their concerns below, and hope that their review and discussion can help convince all reviewers of the importance of our work, and champion our paper.

Two of the three claims seem trivial.

We agree that the claims seem straightforward in hindsight. However, we contextualize this conclusion from the claims made by existing DLIR literature, i.e. a carefully selected intensity loss function, network architecture, pretraining schema and large datasets lead to SOTA performance on alignment of anatomical regions. A variety of studies cited in our paper report results with and without label supervision, which we show is the confounding variable that leads to a significant improvement in performance. Therefore, performance of DLIR methods with and without Dice score must be compared to justify the factors involved in improving performance of DLIR methods. Our analysis in Section 3 elucidates that unsupervised methods do not benefit from advancements in network architecture, a point further verified in Section 4. Section 5 is included to underscore the fact that DLIR methods’ superior performance indeed comes from incorporating label supervision during training, a capability absent in classical methods.

Section 6 seems trivial in the context of our paper, but for the biomedical community at large, pretrained models and large datasets are built to train models that generalize outside the training domain (see common rebuttal). Our findings in Section 6 suggest a need to reassess huge labeled data collection efforts and to formulate DLIR advancements that exhibit robustness against minor domain shifts instead. This reevaluation is crucial for leveraging large amounts of annotated data effectively. We have added this both in the Motivation and Discussion sections.

“I feel that the abstract and intro set up a high expectation..”

We acknowledge that the phrasing of our contribution may have set an overly ambitious expectation. We have revised the statement in the paper to: “We reassess and recalibrate performance expectations from classical and DLIR methods under access to label supervision and training time, and its generalization capabilities under minor domain shifts.” This modification more accurately reflects the scope and implications of our study.

“shouldn't there also be a correlation within each individual dataset”

We find that since the labeled regions are fixed within a dataset, the variation between MI and registration score is dominated by other intra-dataset factors. The correlation between MI and registration performance occurs at the granularity of the dataset, which is shown in Fig.3. This granularity highlights the broader applicability of our findings across different datasets rather than within a single dataset.

“are the hypothesis/conclusion specific to brain MRI”:

We assert that the methods and evaluation metrics employed in our study do not incorporate domain-specific information, making our observations broadly applicable. Brain MRI was chosen due to the extensive availability of solutions and open-access resources provided by the neuroimaging and machine learning communities. This choice ensures that our analysis is unhindered by fairness or reproducibility concerns.

We thank the reviewer for their highly detailed and insightful feedback, this has immensely helped in improving the quality of our work. We believe we have addressed all major concerns and incorporated changes in the paper.

“I would argue that this is typically not always true considering the reported findings in the learning-based papers, e.g., VoxelMorph (VM), TransMorph (TM), etc”

We appreciate this perspective and have addressed this concern in Sec4 (Instrumentation Bias) and Fig3. VM reports a Dice score of 0.749 for ANTs using suboptimal parameters ($\sigma$ of 9px followed by 0.4px), while we observe a Dice score of 0.787 with default parameters, highlighting the need for unbiased evaluation. Our evaluation of VM uses their pretrained model and scripts, yielding consistent results w/ their paper. Other studies on lung [5] and histology [6] registration also show that classical DIR methods far outperform DLIR methods. An unbiased and unified evaluation of algorithms is necessary to fairly compare the performance gaps of said algorithms. Our paper finds that TM is the only DL method beating classical methods (p<0.01) on the OASIS dataset with and without Dice loss (Fig 2 and 4). TM also demonstrates good generalization among DLIR methods competing with SynthMorph, although still underperforming compared to classical methods. We urge the reviewer to review these figures and results. We are happy to clarify any remaining discrepancies in the discussion.

“Performance is different for inter-subject and atlas-based registration, image pairs with large age variations won't perform well in the learning-based registration tasks”

In general, a vast majority of methods focus on inter-subject registration, and atlas-building and atlas-based registration are considered separate problems altogether. Therefore, we keep inter-subject registration datasets a consistent theme in the paper to corroborate existing findings and avoid mixing up results from different problem statements.

Re large age variations: A good registration model in general should perform well across large variations in age, subject demographics, and acquisition configurations. Classical methods performing well for age variations further strengthen our findings in Sec.4 and Sec.6, i.e. labeled data becomes necessary in these settings for DLIR methods to perform better. We have added these observations into the paper further motivating our study.

“consider experimenting and reporting performances for within-patient, inter-patient, and atlas-to-patient registration results”

These are interesting experiments indeed and form the scope for future work, along with multimodal registration, and registration of different anatomy like chest CT or abdomen. Due to space constraints, fairness and reproducibility considerations, we choose OASIS dataset for training since a large number of SOTA DLIR methods provide training recipes for this dataset, and IBSR/CUMC/MGH/LPBA datasets that have been extensively used in unbiased community-standard neuroimaging evaluations [7,9].

“model performance is very inconsistent with the reported version in the original papers, LKU-Net is achieving 0.77 average Dice… same with LapIRN model's performance”

We’re not sure where these numbers come from. In Fig.3 we show that the LKU paper reports a 0.88 Dice score, and our evaluation shows a 0.904 Dice score – Fig.4. (top) shows that LKU-Net is the top performer with Dice supervision. Similarly, we show in Fig2. that LapIRN reports 0.808 Dice but we report 0.788 – a reasonable deviation. In fact, LapIRN performance improves significantly when Dice supervision is added (Fig2 v/s Fig4), the latter of which the original paper did not implement or report.

Our work takes great care to accurately reproduce and not misrepresent the performance of DLIR and classical methods. We ask the reviewer to review the subsection on Instrumentation bias (Sec 4) – we’re confident they will find DLIR methods represented accurately. DLIR methods may misrepresent classical baselines by choosing suboptimal hyperparameters; we work towards a more fair, unbiased and unified evaluation of classical and DLIR methods under the same umbrella.

“hypothesis presented in Sec 6 missing some important information”

For all datasets, we follow the preprocessing steps followed by Klein et.al. [7]. We have added this detail in the paper.

“W2. Missing large deformation diffeomorphic and related baselines”

Classical methods like ANTs, Greedy, FireANTs build on the LDDMM framework to avoid explicitly storing the full 4D velocity fields by integrating the infinitesimal velocities using gradient descent, & are shown to model very large deformations by independent evaluations [7,8]. Moreover, these are widely used classical baselines used by the broader biomedical community, and should not be seen as a reason to invalidate the results in the paper.

“W3. Overclaimed hypothesis”

We have changed L332-335 to reflect the conclusions for inter-subject registration, which is a defacto standard for registration. Since none of the analysis, baselines, and evaluation make any domain/subject-specific modeling assumptions, and the datasets are community-standard benchmarks, the results are valuable, both within the neuroimaging and the biomedical communities at large.

“W4. (Minor) Technical writing”

The contribution of the paper is to recalibrate claims about DLIR methods' performance by isolating label supervision as the primary contributor to their superiority. This motivates the question if label supervision can be robust to domain shift and finds it is not, prompting a reconsideration of data collection pipelines to improve registration performance, and rethinking DLIR design decisions to be robust to domain shift. We have made these changes in the contributions and discussion sections.