Efficient Adaptive Filtering for Deformable Image registration

[W3] Reply: As stated in our response to Q1, AdaWarp is a plug-and-play module rather than a full backbone network, allowing flexibility in selecting the most suitable backbone based on dataset specifics to achieve higher accuracy. Contrary to the concern that using different network structures for different datasets might lead to inconsistencies or generalizability issues, we believe this demonstrates AdaWarp’s adaptability across various architectures and datasets with differing modalities and input constraints.

Network Architectures:

• Both Ada-Res and Ada-Cost perform well on the ACDC dataset, as it is relatively simpler compared to the Abdomen dataset. While ACDC has local discontinuities and sliding motions, displacements are smaller (up to 15 voxels) than in the Abdomen dataset (up to 60 voxels).
• On the Abdomen dataset, Ada-Res achieves suboptimal performance due to the larger displacements. However, AdaWarp can be easily adapted and integrated into a more suitable backbone to optimize task-specific performance such as a combination of image pyramid and discrete optimization cocnept, highlighting its generalizability rather than inconsistency.

Input Constraints and Modalities:

• The ACDC dataset is used for unsupervised learning, where segmentation masks are not used during training or testing.
• The Abdomen dataset is used for semi-supervised learning, where segmentation masks are provided only during training as auxiliary loss supervision and are not used during testing.

The unsupervised and semi-supervised settings remain consistent across all methods. Rather than reflecting inconsistency, this setup demonstrates AdaWarp’s generalizability across modalities (MRI in ACDC, CT in Abdomen) and input constraints (unsupervised in ACDC, semi-supervised in Abdomen).

For the ablation studies, the key variables in AdaWarp are the spatial sampling rate ($s_s$) and the range sampling rate ($s_r$), which we have discussed in Section 5.1 of the discussion. Additionally, we analyzed the impact of varying $\lambda$ in Figure 6 and thoroughly examined the accuracy-efficiency and accuracy-smoothness trade-offs. Please let us know if there are specific components or parameters you would like us to explore further.

[Q2] Reply: Thank you for pointing this out. We acknowledge that the description was unclear, and we clarify it here. The segmentation network used is a pretrained network with weights from [1], and the pre-softmax feature maps from its output are used as feature maps for SAMConvex [2]. These feature maps are specifically used to compute the dissimilarity loss in SAMConvex. For details, please refer to their respective papers. Briefly, ConvexAdam [3] and SAMConvex are instance-optimization-based methods, requiring iterative optimization for each input pair rather than amortized optimization. ConvexAdam uses MIND [4] as feature maps, while SAMConvex relies on a contrastively pretrained network (which we lack), so we substitute the segmentation feature maps.

• How we use the segmentation feature maps: We use the feature maps of moving and fixed images as input to Ada-Cost but continue to use raw images for dissimilarity loss computation. Thus, the feature map does not replace the guidance generator. For more details, refer to our reply to [C] for Reviewer UMDT.
• Semi-supervised vs. Unsupervised: Yes, this makes Ada-Cost a semi-supervised method on the abdomen dataset. However, other methods on this dataset also use auxiliary segmentation loss for supervision, so we believe this is fair. For clarity, the Ada-Cost used on the ACDC dataset relies solely on raw images as input, without any segmentation network or segmentation loss.

[1] Liu, J., Zhang, Y., Chen, J.N., Xiao, J., Lu, Y., A Landman, B., Yuan, Y., Yuille, A., Tang, Y. and Zhou, Z., 2023. Clip-driven universal model for organ segmentation and tumor detection. ICCV 2023.

[2] Li, Z., Tian, L., Mok, T.C., Bai, X., Wang, P., Ge, J., Zhou, J., Lu, L., Ye, X., Yan, K. and Jin, D., 2023, October. Samconvex: Fast discrete optimization for ct registration using self-supervised anatomical embedding and correlation pyramid. MICCAI 2023.

[3] Siebert, H., Großbröhmer, C., Hansen, L. and Heinrich, M.P., 2024. ConvexAdam: Self-Configuring Dual-Optimisation-Based 3D Multitask Medical Image Registration. TMI 2024.

[4] Heinrich, M.P., Jenkinson, M., Papież, B.W., Brady, S.M. and Schnabel, J.A., 2013. Towards realtime multimodal fusion for image-guided interventions using self-similarities. MICCAI 2013.

Dear Reviewer RMNZ,

We sincerely appreciate your comments and suggestions. Below, we address your concerns listed in the weakness section as [W1], [W2], and [W3], along with your questions. Note that some questions overlap with the weaknesses, so we have combined them in our responses.

[W1] Reply: Thank you for raising this point. We have added more references to the literature review in the revised manuscript and included two recent multi-scale approaches: CorrMLP [1], also mentioned by Reviewer MgEu, and RDP [2]. Both methods demonstrate improved performance compared to existing baselines in Tables 1 and 2. Additionally, we note that several existing baselines, such as MemWarp, LapIRN, and discrete optimization-based methods, also employ image pyramids for multi-scale processing. Below are the results of the added methods compared to ours for reference:

Cardiac Dataset:

Abdomen Dataset:

[Q1] Reply: Thank you for raising this point. Initially, we developed Ada-Res for the ACDC dataset to achieve strong performance. However, applying Ada-Res to the abdomen dataset revealed suboptimal performance due to inherent challenges like the aperture and large displacement problems. To address these, we researched solutions tailored to abdomen datasets and found that incorporating image pyramids and discrete optimization was more effective. Leveraging the flexibility of AdaWarp, we integrated these approaches into Ada-Cost. To verify generalizability, we tested Ada-Cost on the ACDC dataset and observed improved Dice scores but slightly degraded HD95 compared to Ada-Res. We have updated the manuscript to reflect Ada-Cost results on ACDC and ensure consistency in model structure across datasets. Below are the results of the Ada-Cost applied to ACDC dataset in comparison to initial Ada-Res:

Cardiac Dataset:

In addition, we would like to clarify that while many studies use the same backbone network across datasets to demonstrate generalizability, AdaWarp is a flexible network module rather than a full backbone that can be easily integrated into existing frameworks.

• For instance, we incorporated AdaWarp into ConvexAdam as a bilateral filter (Table 2). Although the improvement is limited due to the non-learnable nature of ConvexAdam, this demonstrates AdaWarp's flexibility.

• As discussed in the manuscript, AdaWarp can also be integrated into other registration frameworks, such as zero-shot lung CT registration based on keypoints and image intensities, with preliminary results shown in Table 3. Furthermore, AdaWarp can be applied to medical image segmentation tasks, with preliminary results provided in Table 4.

[1] Meng, M., Feng, D., Bi, L. and Kim, J., 2024. Correlation-aware Coarse-to-fine MLPs for Deformable Medical Image Registration. CVPR 2024.

[2] Wang, H., Ni, D. and Wang, Y., 2024. Recursive deformable pyramid network for unsupervised medical image registration. TMI 2024.

[W2&Q3] Reply:
We have added more references to image registration baselines in the related work and included comparisons with two recent multi-scale approaches, CorrMLP and RDP, both quantitatively and qualitatively. For qualitative results, please refer to Figure 4 and Figure 10 in the revised manuscript. A brief summary of key observations is provided in the third point of the Author Response Summary.

[Q4] Reply:
We apologize for not recording detailed timing information in the original manuscript. While timing depends on the model's parallel or sequential structure, inference and training times are generally correlated with the Multi-Adds (G) used. Since all models were trained under the same configuration, Multi-Adds (G) serves as a proxy for timing comparison. Additionally, we note that Ada-Cost converges faster than other methods. For example, on the abdomen dataset, Ada-Cost typically converges within 20–30 epochs, whereas other methods may require 50–80 epochs.

[Q5] Reply:
Thank you for raising this question, which ties closely to the qualitative results. The interpretability of AdaWarp can be summarized as follows:

• Physical Prior: AdaWarp is motivated by the piece-wise smooth assumption, prevalent in medical imaging. Unlike other learning-based methods that treat the model as a black box, AdaWarp introduces interpretability by constructing cost volumes after feature extraction and applying edge-preserving filtering to reflect local discontinuities. The qualitative results confirm that these discontinuities are effectively preserved.
• Simpler Architecture: Unlike methods relying on complex modules like self-attention or recurrent networks, AdaWarp uses only a handful of convolutional layers. These simpler mechanisms are more interpretable and easier to understand compared to advanced neural network components.

Thank you for your understanding.
We acknowledge that it was our oversight to pre-assume that readers would share our consensus that discussions about image registration discontinuities commonly refer to the displacement field. This should have been made clearer in the manuscript.

In addition, we would like to emphasize that a better ability to preserve local discontinuities is only one of the benefits of incorporating the P-S prior.
As the name "piece-wise smooth" suggests, it involves two key aspects:

1. The differences between pieces, reflected by the boundaries.
2. The smoothness within each piece.

As demonstrated in Figure 10 of the revised manuscript on the cardiac dataset, Ada-Cost produces more realistic displacement fields. While all methods generate smooth fields within the left ventricle blood pool and left ventricle myocardium, only Ada-Cost produces a smooth field in the right ventricle by having a single shrinking center, matching realistic cardiac motion. This is further validated by the increase in the Dice score for the right ventricle achieved by both Ada-Cost and DeBG, which also incorporates the P-S prior.

Dear Reviewer RMNZ,

Thank you for your thoughtful consideration of our detailed responses. We greatly appreciate the time and effort you have dedicated to reviewing our work and revisiting your assessment. While we are grateful for the score increase, we would like to further emphasize how our work contributes to and benefits both the medical imaging and learning representation community, in the hope that you might consider an even higher score.

1. Advancing the Field of Medical Image Registration:

   • Addressing a Critical Need: Our work fills a significant gap in medical image registration by seamlessly incorporating the piece-wise smooth (P-S) prior into an end-to-end trainable framework. This advancement not only enriches the theoretical foundations of registration algorithms but also provides a practical tool that the community can leverage for more accurate and reliable image alignment.
   • Methodological Innovation with Broad Applicability: Beyond registration, our method builds up connetctions of learnable adaptive filtering with self-attention and gated attention mechanism. We believe this approach has the potential to serve as a building block for next-generation neural networks. Its adaptability and efficiency make it suitable for various image processing tasks, potentially inspiring new research directions and applications beyond the medical imaging community.

2. Empowering the Community through Empirical Validation:

   • Setting a New Performance Benchmark: Our results demonstrate that Ada-Cost achieves state-of-the-art performance, both in quantitative metrics and in producing realistic displacement fields. By aligning with realistic cardiac motion, as shown in Figure 10, our method provides a reliable benchmark that others in the community can build upon.
   • Simplicity Enhancing Accessibility: Despite using a very simple architecture consisting of only a handful of convolutional layers, Ada-Cost surpasses much more complex models. Its computational efficiency and ease of implementation make it accessible for researchers and practitioners, facilitating wider adoption and fostering further innovation.
   • Handling Complex Clinical Scenarios: The ability of Ada-Cost to preserve local discontinuities, critical in datasets with sharp boundaries and sliding motions like the abdomen dataset, addresses significant challenges in medical imaging. This capability can improve the accuracy of diagnoses and interventions, directly benefiting clinical outcomes.

3. Catalyzing Future Research and Clinical Applications:

   • Foundation for Advanced Models: By establishing a stronger baseline for learning-based registration models, Ada-Cost serves as a foundational building block for developing more powerful and advanced image registration frameworks.
   • Extending Impact Beyond Registration: We have demonstrated preliminary results extending Ada-Cost to other challenging tasks, such as keypoint-based lung motion estimation and medical image segmentation. Additionally, our preliminary derivation on connections of adaptive filtering with self-attention showcases the method's potential to contribute to the development of next-generation neural networks. These extensions highlight the method's adaptability and its potential to address diverse challenges in medical imaging, ultimately benefiting patient care.

4. Collaborative Engagement and Community Benefit:

   • Open Dialogue and Transparency: Throughout this discussion, we have engaged thoroughly and openly with your concerns, fostering a constructive dialogue that we believe strengthens the work and its presentation.
   • Enhancing Clarity for Community Adoption: We have updated the manuscript to improve clarity and ensure that all key claims are well-supported by evidence, both qualitatively and quantitatively. By doing so, we aim to make our work more accessible and beneficial to the community, encouraging others to build upon our findings.

We sincerely thank you for already increasing your score, which reflects your recognition of the novelty, simplicity, and practical impact of our work. Given its potential to serve as a foundation for further advancements and its broader applicability to critical tasks in medical imaging, we kindly ask you to consider these points for further raising your score, as we believe our contributions make a substantial impact on the community.

Thank you once again for your thoughtful feedback and engagement with our work.

[F] Reply: Thank you for pointing this out. The choice of baselines differs due to the following considerations:

1. Iterative methods (ANTs, Demons, Bspline): These were included for the cardiac dataset but excluded for the abdomen dataset because our parameter tuning failed to yield reasonable performance on the latter.
2. MemWarp and TextSCF: MemWarp is included for the cardiac dataset as it was originally developed for this type, whereas TextSCF is included for the abdomen dataset for similar reasons.
3. ConvexAdam and SAMConvex: These methods are included for the abdomen dataset due to its characteristic large deformations. While they may also perform well on cardiac data, their primary design focuses on datasets with large deformations, making the baselines fair for the comparisons in this paper.
4. DeBG: As described in the original paper (lines 113–121 and 369–377), this method uses shuffled channels instead of real splatting to represent range dimensions, inaccurately preserving the image manifold in higher-dimensional space. As a result, this shuffling cannot handle the image pyramid-based approach used in the abdomen dataset, where each pyramid level shares the same feature map and cost volume across different scales.

[H] Reply: Thank you for your question and the reference. We believe this question is crucial to the image registration community. The claim here is solely based on the results in Table 1, where learning-based methods generally outperform traditional iterative methods (ANTs, Demons, Bspline) in registration accuracy for the cardiac dataset used in this paper, though with slightly higher SDlogJ. However, this trend may not hold universally. The relative advantages of learning-based methods over iterative methods in purely unsupervised settings (without label supervision) remain an active research topic. Iterative methods, particularly discrete optimization-based ones, often achieve comparable accuracy to unsupervised learning methods on datasets with low-complexity deformations and higher accuracy on datasets with large deformations. This discrepancy stems from two main factors:

1. Dissimilarity function: Unsupervised learning methods share the same dissimilarity function as iterative methods, and amortized optimization provides no additional advantage for label matching performance in low-complexity scenarios. [2]
2. Regularization: In unsupervised learning, the burden of smoothness regularization is enforced entirely on the network weights, whereas iterative methods directly compute gradients backflow on the deformation field. This indirect optimization in unsupervised learning can lead to inferior performance compared to iterative methods, particularly on datasets with large deformations. Although some works [3] integrate smoothness regularization explicitly as part of the network, they still require segmentation masks for effective implementation.

However, we have observed two cases where iterative methods may fall behind learning-based methods in image registration:

1. Unsupervised settings with large-scale datasets: In large-scale datasets with low-complexity deformations (e.g., the 4000-subject LUMIR dataset [4] from the Learn2Reg Challenge 2024 [5]), from our empirical experience, iterative methods, whether continuous, discrete, or instance optimization with neural networks, consistently underperform compared to pure learning-based methods, despite extensive parameter tuning. We attribute this gap to the regularization term. With sufficient data, while the dissimilarity metric offers no clear advantage for amortized optimization [2], the regularization term leverages the neural network’s ability to incorporate contextual information [3], enabling superior flow propagation and smoothness regularization.
2. Semi-supervised settings with label supervision: When label supervision (e.g., segmentation or keypoints) is introduced, learning-based methods outperform iterative methods by effectively utilizing surrounding contextual information from labels, resulting in higher registration accuracy. However, additional losses like segmentation loss may cause less smooth deformations at object boundaries, leading to implausible fields and higher SDlogJ values.

[1] Hansen, L. and Heinrich, M.P., 2021. Revisiting iterative highly efficient optimization schemes in medical image registration. MICCAI 2021.

[2] Jena, R., Sethi, D., Chaudhari, P. and Gee, J.C., 2024. Deep Learning in Medical Image Registration: Magic or Mirage?. arXiv preprint 2024.

[3] Heinrich, M.P., 2019. Closing the gap between deep and conventional image registration using probabilistic dense displacement networks. MICCAI 2019

[4] Liu, Y., Chen, J., Wei, S., Carass, A. and Prince, J., 2024. On finite difference jacobian computation in deformable image registration. IJCV 2024.

[5] https://learn2reg.grand-challenge.org

Dear Reviewer UMDT,

We sincerely thank you for your valuable comments. Below, we address your points [A], [B], [C].

[A] Reply: Thank you for pointing this out. We agree that the statement could be clarified. Specifically, traditional iterative methods can be categorized into two main types: continuous optimization-based and discrete optimization-based, each with distinct characteristics.

• Continuous optimization: These methods typically linearize the dissimilarity term, requiring a series of small iterative updates. Each step accounts for only a minor deformation, which leads to slow convergence and difficulty in handling large deformations.
• Discrete optimization: These methods are better suited for handling large deformations and require much fewer iterations to converge, making them favorable for datasets with large displacements. However, the primary drawback is their high memory consumption, which increases exponentially with the disparity volume size.

By "the ability to incorporate contextual information," we refer to the advantage of learning-based methods in integrating label supervision, such as segmentation masks or keypoints. This integration is less straightforward in both continuous and discrete optimization-based approaches. While methods like ConvexAdam can incorporate segmentation masks generated by models like nnU-Net, such masks are unavailable without neural network-based learning. This limitation highlights the lack of contextual information typically inherent to iterative methods.

We hope this explanation clarifies our statement.

[B] Reply: We appreciate the reviewer’s attention to this detail. The proposed AdaWarp method addresses deformable image registration by incorporating the Piece-wise Smooth (P-S) Assumption. In this context, adding one additional range dimension is sufficient to achieve the desired effect. Since most medical image registration problems involve single-channel inputs, a single range dimension can effectively represent voxel-wise intensity differences. With a higher sampling rate $s_r$, this approach enhances edge distinction and ensures adequate sensitivity to object boundaries. Expanding to more dimensions could indeed be interesting, allowing the representation of contextual differences beyond raw intensity differences. Such an extension could generalize self-attention from adaptive Gaussian filtering to a broader learnable adaptive filtering framework. However, this direction is outside the scope of the current paper.

[C] Reply: Thank you for highlighting this. A visual explanation of the bilateral grid process for a one-dimensional signal is shown in Fig. 2. In this example, the original signal is represented as $f: x \rightarrow \mathbb{R}$, where $f(x)$ corresponds to the image intensity at position $x$. When projecting to a higher-dimensional space (from 1D to 2D here) with an additional range dimension (image intensity), the image is represented as $f: (x, r) \rightarrow \mathbb{R}$, where $r$ denotes the intensity at position $x$. Here, $f(x, r)$ represents the corresponding intensity before blurring ($f(x, r) = r$). The guidance map serves as the additional coordinate to access elements in the bilateral grid. Instead of using raw image intensities directly as this coordinate, we employ a trainable guidance map generator. This design allows the network to adaptively learn a more effective coordinate representation, improving the flexibility and performance of the model.

[D&E] Reply: Thank you for pointing out the lack of descriptions for $\lambda$ values. We did not intentionally tune these hyperparameters but performed a grid search with $\lambda = 0.01, 0.1, 1.0$, and $5.0$. We found $\lambda = 0.01$ to be optimal for the ACDC dataset and $\lambda = 1.0$ for the abdomen dataset (note: the $\lambda = 5.0$ in the original manuscript was a typo). Most baseline methods use the same parameters and training settings as AdaWarp. Below, we provide more details.

ACDC Dataset:
All learning-based methods adopt the same hyperparameters as AdaWarp, with $\lambda = 0.01$, MSE as the dissimilarity loss, and scaling-and-squaring with 7 steps for the diffeomorphic transformation model. For Figure 4, while keeping other hyperparameters the same, we vary computational complexity by adjusting the starting channel count in FourierNet, LKU-Net, and Ada-Res, and by modifying the backbone of TransMorph (tiny, small, and normal).

Abdomen Dataset:
The abdomen dataset presents more challenges due to the large displacement problem. To clarify:

• FourierNet, VoxelMorph, TransMorph, and Ada-Cost use local NCC as the dissimilarity loss with $\lambda = 1.0$. ConvexAdam and SAMConvex also use $\lambda = 1.0$, employing MIND and segmentation feature maps, respectively, to compute the dissimilarity. All these methods adopt scaling-and-squaring with 7 steps for the diffeomorphic transformation model.
• For TextSCF, we follow its original implementation with $\lambda = 0.1$ and without the diffeomorphic transformation model. The $\lambda = 0.1$ version with integration is also presented in Figure 6.
• Both LKUNet and LapIRN results in Table 2 use $\lambda = 1.0$ without the diffeomorphic transformation model. Additionally, we ran counterparts using scaling-and-squaring with 7 steps for the diffeomorphic transformation, and the results are as follows:

Abdomen Dataset:

Both methods show performance degradation in anatomical alignment, as measured by Dice, when required to produce smoother and more plausible deformation fields.

[I] Reply: Thank you for pointing this out. This question is similar to the first one raised by Reviewer RMNZ, and we address the essentials here.

Why: We do not intentionally use different architectures; the choice is application-driven. As also mentioned in our reply to Reviewer RMNZ, the core of this paper is the AdaWarp module, which generalizes to different architectures as needed. Initially, we developed Ada-Res for the ACDC dataset to achieve a reasonably good performance. However, applying Ada-Res to the abdomen dataset revealed suboptimal performance due to challenges like the aperture and large displacement problems. To address this, we tailored a solution for the abdomen dataset by incorporating image pyramids and discrete optimization, inspired by prior multi-scale approaches and ConvexAdam, leading to the development of Ada-Cost. Leveraging AdaWarp’s flexibility, we integrated these approaches effectively. While we hadn’t previously tested Ada-Cost on the ACDC dataset, we have now conducted these experiments as part of the rebuttal. The results are as follows:

Cardiac Dataset:

This is interesting, as methods that perform well on ACDC may struggle on more challenging datasets like the abdomen dataset, whereas methods designed for challenging datasets like the abdomen dataset can be easily adapted to simpler datasets like ACDC. Note that for Ada-Cost results on ACDC, raw images were used as input, without feature maps from a segmentation network.

[G] Reply: We have included t-tests for both Dice and HD95 metrics in the revised manuscript.

[J] Reply: As noted in the first point of the Author Response Summary, we will release the code immediately upon the paper's acceptance.

[K] Reply:
For qualitative results, please refer to Figure 4 and Figure 10 in the revised manuscript. A brief summary of key observations is provided in the third point of the Author Response Summary.

[A] Further Clarification: Thank you for your feedback on our response to your comment [A]. We have updated the manuscript to reflect the following discussion:

• Segmentation Accuracy:
  While there is ongoing debate on whether learning-based registration methods outperform traditional iterative methods, deep learning has undeniably dominated segmentation. Thus, existing iterative methods still rely on segmentation maps generated by deep learning, which provide superior contextual information compared to traditional segmentation algorithms.

• Amortized Optimization:
  Traditional iterative methods can incorporate contextual information (e.g., segmentation maps) but require instance-wise optimization, meaning new segmentation masks and energy function optimization are needed for every unseen image pair. In contrast, learning-based methods utilize amortized optimization, requiring only a single network trained on a cohort of image pairs, making them more efficient and scalable.

Dear Reviewer MgEu,

We sincerely thank you for your valuable comments. Below, we address your concerns listed in the weakness section as [W1] and [W2], and answer your questions [Q1] and [Q2].

[W1] Reply: We would like to emphasize that the focus of this study is not solely on improving registration efficiency, but on introducing a novel neural network architecture that achieves the best overall performance in image registration. This overall performance cannot be evaluated using a single surrogate metric, such as Dice. Instead, it requires a multidimensional comparison, taking into consideration of factors like accuracy-efficiency and accuracy-smoothness tradeoffs. We believe these aspects have been overlooked in previous studies, and we elaborate on them below.

1. Accuracy-efficiency: Evaluating accuracy or efficiency in isolation is insufficient. Our model demonstrates that with similar accuracy, it achieves lower computational complexity, and with similar computational complexity, it achieves higher accuracy. This balance is essential to claim better overall performance.
2. Accuracy-smoothness: A key advantage of learning-based methods is their ability to integrate label supervision. However, without adequately handling the smoothness of the deformation field, they can produce implausible or unrealistic deformations, even with high Dice scores. For example, using a pretrained nnU-Net for segmentation achieves over 90% Dice on ACDC and multi-organ abdomen segmentation tasks. If we then perform label matching [1] directly on the predicted masks, the Dice score improves further, but the resulting deformation field is often unrealistic due to the lack of smoothness and consideration of image textures.

We hope this explanation addresses your concerns.

[W2] Reply: Thank you for pointing this out. We would like to clarify that the cardiac MRI and abdomen CT datasets are indeed suitable testbeds for the proposed piece-wise smooth assumption. However:

1. Brain MRI datasets and the piece-wise smooth assumption: Brain MRI datasets do not violate the piece-wise smooth assumption. For adjacent regions with smooth transitions (e.g., left and right thalamus), these regions are treated as a single "piece" under the assumption, similar to existing models. For regions with clear boundaries (e.g., cerebral white matter and cortex), our model can perform better by explicitly handling such transitions.
2. Relative difficulty of brain MRI datasets: Brain MRI datasets are generally easier compared to cardiac and abdomen datasets. As noted in the second paragraph of the introduction in [2], abdominal scans are more complex than brain scans. Cardiac datasets exhibit local discontinuities and sliding motions, which are absent in brain MRI. Multi-organ abdomen datasets pose additional challenges:
   • The aperture problem [3], arising in homogeneous or textureless regions, where the limited local evidence within a small window (defined by the network's effective receptive field (ERF) [4]) restricts accurate displacement estimation.
   • The large displacement problem, where the displacement of a small structure between image pairs exceeds its own size, making accurate alignment more challenging.

We hope this clarifies our rationale.

[1] Durrleman, S., Prastawa, M., Charon, N., Korenberg, J.R., Joshi, S., Gerig, G. and Trouvé, A., 2014. Morphometry of anatomical shape complexes with dense deformations and sparse parameters. NeuroImage 2014.

[2] Heinrich, M.P., 2019. Closing the gap between deep and conventional image registration using probabilistic dense displacement networks. MICCAI 2019.

[3] Horn, B.K. and Schunck, B.G., 1981. Determining optical flow. Artificial intelligence 1981.

[4] Luo, W., Li, Y., Urtasun, R. and Zemel, R., 2016. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS 2016.

[Q1] Reply: We agree that VoxelMorph is highly efficient with a low parameter count, and we did not exclude it intentionally from Figs. 5 and 6. The main reason for its exclusion is that VoxelMorph is not as competitive as other methods, and adding it would further crowd the figures. Here are its relevant statistics:

1. Figure 5: VoxelMorph would be positioned within the polygon formed by the lines of FourierNet and TransMorph, with Dice: 76.35%, Multi-Adds: 19.5 G, and Parameter Size: 0.32 MB.
2. Figure 6: VoxelMorph would appear near TransMorph, slightly to its upper left, with Dice: 47.05%, Multi-Adds: 73.30 G, and Parameter Size: 0.32 MB.

[Q2] Reply: Thank you for mentioning CorrMLP. While CorrMLP has an interesting approach, we would like to clarify that AdaWarp shares the same motivation as CorrMLP. Although not explicitly stated in the manuscript, the low-resolution feature maps from deeper layers of neural networks inherently possess larger effective receptive fields (ERFs) than shallower layers. Please refer to the ERF visualization via this link, also included in the revised manuscript appendix. Darker and more widely spread regions indicate larger ERFs. The details of ERF computation can be found in [1]. Key observations:

• Leveraging Large Receptive Fields: Subfigures (a), (b), and (c) illustrate feature maps from different encoder levels (L1: full resolution, L2: 1/2 downsampled, L3: 1/4 downsampled) from VoxelMorph. Deeper layers (e.g., L3) have larger ERFs, confirming that low-resolution features from deeper layers capture broader context. AdaWarp leverages the deepest encoder layer for the largest possible ERF. As shown in Table 4 (first row vs. last row), maintaining object boundaries (via AdaWarp) is essential for accuracy, as large ERFs alone are insufficient.
• Effectiveness of AdaWarp Over Swin-Unet: We compared ERF heatmaps of Swin-Unet and Ada-Swin (pre-softmax feature maps of models used in Table 4). Both share identical encoders, differing only in the decoder (Swin-Unet uses a U-Net structure, while Ada-Swin uses AdaWarp). Ada-Swin shows larger ERF regions and achieves 1.89% higher accuracy than Swin-Unet.

As for the results of CorrMLP, we used the source code from their public repositories and ran experiments on the cardiac and abdominal datasets. The training settings were identical to those used for AdaWarp, including the same regularization strength and squaring-and-scaling integration with 7 steps. We have updated Tables 1 and 2 in the revised manuscript to reflect these results, and we briefly list them here for comparison.

Cardiac Dataset:

Abdomen Dataset:

From the results, CorrMLP proves to be a highly competitive technique, outperforming all other baselines in registration accuracy while maintaining competitive smoothness of the deformation field. However, we observed a discrepancy between our reproduced results and those reported in Table 2 of the CorrMLP manuscript [2]. While we used the same data split for training and testing, they reported an Avg. Dice of 81.0%, whereas ours was 79.2%.

Upon further investigation, we found this discrepancy may stem from differences in image preprocessing. CorrMLP resampled images to a voxel size of 1.5x1.5x3.15 mm³ and cropped to 128x128x32, while we resampled to 1.8x1.8x10 mm³ and cropped to 128x128x16. The ACDC dataset protocol [3] specifies slice thicknesses ranging from 5 mm to 10 mm and spatial voxel sizes between 1.34x1.34 and 1.68x1.68 mm².

We are uncertain if CorrMLP's preprocessing aligns with standard practices, as upsampling from 5–10 mm to 3.15 mm in the axial direction introduces redundant information, potentially inflating the Dice score.

[1] Luo, W., et al. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS 2016.

[2] Meng, M., et al. Correlation-aware Coarse-to-fine MLPs for Deformable Medical Image Registration. CVPR 2024.

[3] Bernard, O., et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?. TMI 2018.

Thanks for the authors' effort in addressing my concerns. Overall, my concerns have been largely addressed, although I am still not convinced that brain MRI registration is an easier task than cardiac and abdomen registration. Anyway, the evaluation in the cardiac and abdomen datasets is fine for this study. Nevertheless, after reading the comments from other reviewers, I think it's hard to increase my original score, so I decided to maintain it.

Dear Reviewer MgEu,

Thank you for your thoughtful review and for acknowledging that our responses have largely addressed your concerns. Since the primary issues have been resolved, we would like to highlight the broader impact and contributions of our work, which we hope might encourage you to increase the score.

1. Interpretability and Simplicity:

   • Our method seamlessly incorporates a physical prior (piece-wise smoothness) into a neural network, enhancing interpretability while achieving strong performance.
   • The architecture is extremely simple, making it easy to use and adapt for other researchers, while still setting a new baseline in learning-based registration.

2. Connecting Attention Mechanism and Adaptive Filtering:

   • By introducing learnable adaptive filtering, we have drawn initial connections between self-attention and learnable adaptive filtering, as well as between gated attention and learnable adaptive filtering. This potentially generalizes modern neural network architectures and positions learnable adaptive filtering as a foundational block for the next generation of neural networks.

3. Connecting Traditional Algorithms and Modern Neural Networks:

   • In addition, our method bridges the gap between traditional image processing methods, such as bilateral filtering, and modern neural network mechanisms. This fusion not only enriches the medical imaging field but also offers a pathway for integrating classical techniques into contemporary deep learning frameworks.

4. Broader Applicability:

   • Beyond the cardiac and abdomen datasets, we have demonstrated AdaWarp’s versatility with preliminary results on keypoint-based lung motion estimation and medical image segmentation. These results showcase its potential to address diverse and critical tasks for the community.

Given that our responses have addressed your concerns and our work contributes to the community through its interpretability, simplicity, and broader impact, we kindly request that you reconsider and further increase your score. We believe our study offers meaningful insights for researchers and practitioners alike.

Thank you again for your engagement and valuable feedback.

Dear Reviewer TEi6,

Thank you for your critiques regarding the novelty of the manuscript. Please refer to the revised manuscript and the 2nd point in the Author Response Summary for detailed explanations of our claims of technical novelty. Specifically addressing your concerns:

1. We agree that latent feature representations are a common technique used in most neural networks, and we did not claim this as novel. However, the major novelty of our work lies in addressing a significant gap in the registration literature by seamlessly incorporating the physical prior, i.e., the piece-wise smooth assumption, into neural networks in an end-to-end trainable manner. To the best of our knowledge, no existing literature has achieved this.

2. We understand that simple incremental modifications may lack sufficient novelty. While a differentiable bilateral grid may appear straightforward at first glance, there is no fully functional and trainable bilateral grid for learnable adaptive filtering in either the registration or natural image processing literature. As discussed in the revised manuscript and the 2nd point in the Author Response Summary, the closest methods, such as deep bilateral grids [1][2], use channel shuffling to inadequately represent the range dimension, resulting in suboptimal performance. Our AdaWarp addresses this limitation and holds broader implications for future applications in image processing.

[1] Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W. and Durand, F., 2017. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics (TOG), 36(4), pp.1-12.

[2] Xu, B., Xu, Y., Yang, X., Jia, W. and Guo, Y., 2021. Bilateral grid learning for stereo matching networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12497-12506).

Dear Reviewer TEi6,

We hope this message finds you well. As the deadline for the discussion period approaches, we would like to kindly remind you that we have not yet received any follow-up feedback from you. To make progress, we would appreciate it if you could clarify a few points from your previous review.

"While the method leverages an encoder to extract a latent representation that approximates the deformation field at a low resolution, this approach mainly contributes to the model's efficiency but is not unique."

To make this critique more actionable, could you suggest specific approaches that improve efficiency in image registration and provide a comparison of their pros and cons relative to the proposed method? This would help us better contextualize and address your concern.

"The use of latent feature representations for similar tasks has already become common in the field."

We would like to clarify that our paper does not claim the use of latent feature representations as a novelty or contribution. This point is merely a common component in modern neural network architectures.

"The core of AdaWarp is a differentiable bilateral grid, which naturally incorporates the P-S prior. In implementation, the guidance map aids in processes like splatting, blurring, and slicing. This incremental modification lacks sufficient novelty."

Could you elaborate on why this is considered an incremental modification? To substantiate your claim, we kindly ask you to address the following questions:

• Are there any existing methods that perform the same process as a fully differentiable bilateral grid? If so, can you list them and show how they compare to ours in terms of novelty and functionality?
• Which part specifically do you consider incremental? Could you point out the subtle modifications we have made that lead to this characterization?

Your insights would be invaluable in helping us refine our work and better address the concerns raised. We sincerely appreciate your time and look forward to your response.

Best,
The Authors