MotionTTT: 2D Test-Time-Training Motion Estimation for 3D Motion Corrected MRI

We thank the reviewer for the feedback and the positive evaluation of our work. In the following we address the weaknesses in the order as pointed out by the reviewer.

• Weakness 1, various types of rigid motion: We agree that in practice motion can be categorized into different types. However, the range of different motions is large varying with patient population and condition. Also we are not aware of good available models of the different types of motion based on real measurements in the literature. Hence, it is common to evaluate with randomly simulated motion (see line 202 for examples). Note that for the experiments in Figure 3 we simulated a total of 100 different motion trajectories with up to 10 randomly generated motion events per trajectory covering a wide range of possible motions. In general, we expect our method to work well for different types of motion as different to previous deep learning based approaches it was not trained on a particular type of simulated motion.
• Weakness 2, compared methods: [1] proposes a domain adaptation method for reconstructing radial MRI in a limited data regime, but does not discuss motion correction. [2] proposes a novel network architecture for end-to-end motion artifact reduction but only in 2D and in general it has been found that end-to-end methods, while being significantly faster, result in inferior reconstruction quality compared to reconstruction based on previously estimated motion parameters like ours. See e.g. Fig. 3 in [4] and Fig. 7 in [5].
  While diffusion model based motion correction is an interesting direction, the reason why we do not compare to [3] is that the proposed method can only operate 2D motion, while we investigate motion estimation in 3D. Directly extending the method to 3D requires training a diffusion model on entire 3D volumes, which is computational difficult and would require much larger 3D datasets. At the same time it is unclear how to use a 2D diffusion model to estimate motion in 3D.
• Weakness 3, some typos: Thanks, we fixed it.

[4] Haskell et al. “Network Accelerated Motion Estimation and Reduction (NAMER): Convolutional Neural Network Guided Retrospective Motion Correction Using a Separable Motion Model”. In: Magnetic Resonance in Medicine (2019).
[5] Hossbach et al. “Deep Learning-Based Motion Quantification from k-Space for Fast Model-Based Magnetic Resonance Imaging Motion Correction”. In: Medical Physics (2023).

Thanks for the feedback. In the following we address the concerns and questions in the order as pointed out by the reviewer.

• Weakness 1, experiments and alternative techniques seem to be presented throughout the result section: Thanks for the feedback. We will make sure to describe all methods that we compare in the main Figure 3 at the end of the setup Section 5.1 before the results section.
• Weakness 2, a proper ablation study seems required, in order to assess the added-value of each step: In fact, Figure 3 already shows the performances during all steps of MotionTTT. L1 indicates the performance without any motion correction (before applying MotionTTT), MotionTTT-L1 the performance after motion estimation and MotionTTT-L1+Th after motion estimation plus data consistency loss thresholding.
• Weakness 3, baselines regarding the reconstruction step or the removal of motion artifacted lines: Regarding the reconstruction step, our method is agnostic to the method used for reconstruction based on the estimated motion parameters. Hence we do not expect any new insights from trying additional reconstruction methods as a better reconstruction method would lead to a better performance for any motion estimation method.
   
  Regarding baselines for the removal of motion artifacted lines, we are not sure to which method the reviewer would like to see a comparison. Our method estimates motion parameters with a high accuracy and hence will outperform any method that only detects and then removes motion corrupted lines. Only in the presence of severe motion we remove lines that exhibit a large data consistency loss indicating inaccurate estimation of motion parameters, a feature that comes for free with the proposed method. However, also in the most severe cases typically not more than 5% have to be discarded.
• Weakness 4, effect of the low reconstruction performance of the U-net on motion estimation: It is not clear if a better reconstruction performance of the U-net due to an increased amount of training data would further improve the ability to estimate motion parameters especially as our experimental results show, the current reconstruction performance of the U-net already achieves highly accurate motion parameter estimation across a wide range of simulated motion and significantly improved image quality in case of prospectively acquired real motion-corrupted data. Nevertheless, we will investigate if even further improvements can be achieved by training on additional data sources like e.g. 2D brain data.
• Weakness 5, why did the authors not try to directly reconstruct from the 3D k-space domain, in order to estimate motion parameters in 3D: Training 3D models on entire 3D volumes is infeasible in terms of memory and compute requirements. However, we agree that instead of chunking the volume slice-by-slice as we do, one could also train a 3D model on 3D chunks and perform chunk-by-chunk reconstruction.
  We decided to build on the well-established 2D convolutional U-net as it enabled highly accurate motion estimation combined with fast reconstruction and a low network parameter count, which is important as the entire volume has to be reconstructed in every iteration. In the future, this also enables enlarging the training set with 2D data in order to overcome the limited availability of 3D dataset.
• Weakness 6, regarding switching between 2D and 3D: As the reviewer pointed out correctly, slicing the k-space in arbitrary directions is not appropriate. Hence, we only slice the zero-filled reconstruction in the image domain at the network input and add slices together at the network output before applying the 3D fourier transform to obtain the reconstructed k-space data. We will emphasize this more clearly in the Section 4, step 2: Test-time-training for motion estimation.
• Question 1, existence of another database of 3D data to train a 3D reconstruction UNet directly: To the best of our knowledge the used dataset is the largest 3D brain dataset containing the original k-space measurements publicly available. While other smaller ones exist, we do not believe that even if compute would not be a problem the amount of training examples would suffice to train a 3D model on entire volumes.
• Question 2, training a reconstruction technique with higher acceleration factor to compensate for thresholding: We only threshold a relatively small number of lines, typically not more than 5% of the acquired lines even under the most severe motion. Nevertheless, there is a discrepancy between the undersampling mask the U-net was trained on in the absence of motion and the undersampling of the motion-corrected network inputs due to rotations in the k-space as discussed in the paper in Section 5.2. However, both changes in the mask due to thresholding and rotations are motion specific and hence difficult to train on without simulating motion.
  Hence, in the paper we use L1-minimization for reconstruction, which is mostly independent of changes in the mask.
• Question 3, application to other 3D sampling strategies (stack of stars): See author rebuttal point 1.
• Question 4, missing L1 reconstruction for the real-motion data: We do show the results for MotionTTT+Th-L1 in Figure 6.
• Limitation, risks of hallucinations and bias: As currently the final reconstruction results of our method are based on L1-minimization no learning based hallucinations can occur. If the motion parameter estimation is off, the result will be a reconstruction with motion artifacts similar to when no motion correction is applied. We are not sure what type of bias the reviewer is referring to.

We hope those responses address the reviewer's concerns and the reviewer considered raising their score.

We thank the reviewer for the feedback. In the following we address the concerns in the order as pointed out by the reviewer.

• Weakness 1, introduction of existing modules and lack of novelty: We would like to point out that combining a 2D reconstruction network trained on motion-free data with test-time-training for motion estimation in 3D has not been proposed before, neither a closely related method. Our work is the first that enables efficient motion estimation in 3D based on a 2D network trained on motion-free measurements. As discussed in our related work section, previous work on deep learning based motion estimation has exclusively focused on the 2D case and methods that are required to simulate motion during training and hence are specific to the type of motion used during training. Thus the work contains significant technical novelty.
• Weakness 2 part 1, lack of further baselines in particular motion corrupted image reconstruction networks: While we are not aware of any deep learning based baselines for 3D motion estimation, there are indeed methods for 3D end-to-end reconstruction from motion corrupted volumes as discussed in our related work section, second paragraph. As we note there, this class of methods is well known to perform poorly relative to methods that perform motion estimation and then reconstruct, like our method. See Fig. 3 in [1] and Fig. 7 in [2].
• Weakness 2 part 2, average quantitative results over more test examples: For a given method each result in Fig. 3 in the paper is averaged over 5 test examples with each 2 independent motion trajectories. As we consider 10 different levels of motion, we evaluate in total over 100 different motion trajectories each consisting of up to 10 randomly generated motion events. Regarding the required number of test examples, for image reconstructions tasks this number is typically lower than in other ML domains like computer vision as one test image contains already many different structures and details that the method needs to recover in order to achieve a high image quality score. Especially, in 3D the amount of data per test volume is large and comparatively small test sets are often used (e.g. 7 volumes used in [3]).

We hope this addresses the reviewer's concerns and if yes, that the reviewer considers raising their score.

[1] Haskell et al. “Network Accelerated Motion Estimation and Reduction (NAMER): Convolutional Neural Network Guided Retrospective Motion Correction Using a Separable Motion Model”. In: Magnetic Resonance in Medicine (2019).
[2] Hossbach et al. “Deep Learning-Based Motion Quantification from k-Space for Fast Model-Based Magnetic Resonance Imaging Motion Correction”. In: Medical Physics (2023).
[3] Johnson and Drangova. “Conditional Generative Adversarial Network for 3D Rigid-Body Motion Correction in MRI”. In: Magnetic Resonance in Medicine (2019).

Thanks for the feedback and for acknowledging the novelty of our work. In the following we address the weaknesses (W), questions (Q) and limitations in the order as raised by the reviewer.

• W 1, lack of theoretical investigation of why the optimization works:
  To get an understanding on why the proposed optimization works, we developed the following theory for a (very) simplified setup that models our approach. This setup also helps to understand the necessity of defining a learning rate schedule that in the presence of motion explores the loss landscape with an initially large learning rate before gradually decaying it. We will include this result and discussion in the revised version of our paper.
   
  We consider the signal $\mathbf{x} \in \mathbb{R}^n$ that lies in a $d$ dimensional subspace, i.e., the signal is generated as $\mathbf{x} = \mathbf{U} \mathbf{c}$ with orthonormal $\mathbf{U} \in \mathbb{R}^{n \times d}$ and Gaussian vector $\mathbf{c}$.
   
  Let $\mathbf{F}_\mathcal{T}$ be the Fourier matrix with rows chosen in the set $\mathcal{T}$. We assume a measurement model, where the signal $\mathbf{x}$ takes on $N_s$ motion states defined by the unknown translations $t _1^\ast,\ldots,t _ {N _ s}^\ast$, and for each translated version of the signal, a set of measurements is collected according to
  \mathbf{y _ s} = \mathbf{D} _ {t _ s^\ast, \mathcal{T} _ s} \mathbf{F} _ {\mathcal{T} _ s} \mathbf{x}, \tag{1} where $\mathbf{D} _ {t _ s^\ast, \mathcal{T} _ s}$ is a diagonal matrix with $e^{i 2 \pi t_s^\ast l / n}, l\in \mathcal{T} _ s$ on its diagonal. Here, we use that a circular shift by $t_s^\ast$ in the spatial domain is a multiplication with a complex exponential in the frequency domain.
  We write the acquisition of the entire measurements $\mathbf{y} \in \mathbb{R}^k$ from all motion states as $\mathbf{y} = \mathbf{D} _ \mathbf{t^\ast} \mathbf{F} _ \mathcal{T} \mathbf{x}.$ For simplicity we assume the fully sampled case, i.e., $k=n$.
   
  Next, we define a network $f( \mathbf{x}^\dagger) = \mathbf{U} \mathbf{U}^T \mathbf{x}^\dagger$, for which it is straightforward to see that if all motion parameters $\mathbf{t}^\ast$ are known we have that f( (\mathbf{D} _ {\mathbf{t}^\ast} \mathbf{F})^\dagger \mathbf{y} ) = \mathbf{x}, \tag{2} i.e., the network reconstructs the signal perfectly. The loss function used in the paper for this setting is \mathcal{L} _ {\text{TTT}} (\mathbf{t}) = || \mathbf{D} _ {\mathbf{t}} \mathbf{F} _ {\mathcal{T}} f( (\mathbf{D} _ {\mathbf{t}} \mathbf{F} _ \mathcal{T})^\dagger \mathbf{y} ) - \mathbf{y} || _ 2^2, \tag{3} on which we can perform test-time-training with gradient descent with respect to the motion parameters $\mathbf{t}$.
   
  Assuming that $\mathbf{U}$ is a random subspace and that $\mathbf{c}$ is drawn from a Gaussian with identity covariance matrix, it can be shown that the objective function concentrates around \mathcal{\tilde L} (\mathbf{t}) = || \mathbf{D} _ \mathbf{t} - \mathbf{D} _ {\mathbf{t}^\ast} \tag{4} ||^2_F. This expected objective function has a unique minimizer at $\mathbf{t} = \mathbf{t}^\ast$ and is convex in a small region around $\mathbf{t}^\ast$, but not globally.
   
  For setting the number of motion states $N_s=1$ we obtain a simple one-dimensional optimization problem and can inspect the behavior of the loss function graphically. Without loss of generality we set $t^\ast = 0$ and plot the loss $\mathcal{L} _ {\text{TTT}} (t)$ in PDF Fig. 4. As we can see the loss exhibits a global minimum at $t=t^\ast$, but also local minima for $t \neq t^\ast$.
   
  In order to minimize such a function in practice, we defining a learning rate schedule (see Appendix B.2 in the paper), where from the start we explore the loss landscape with a large learning rate if the initial loss is large as this indicates the presence of strong motion and initializing all motion parameters with zero might constitute to a large distance to the true motion parameters. Then, the learning rate is reduced gradually. Also once estimated motion parameters approach the true parameters, we did not observe significant deviations from this solution indicating the existence of a global minimum (or a good local one) in practice.
• W 2, experimental investigation under what conditions the method is most effective in terms of undersampling factor and sampling trajectory: See author rebuttal.
• Q 1, why and under what conditions does the method work: See Weakness 1 and 2.
• Q 2, is the linear trajectory the optimal undersampling trajectory: See author rebuttal. In short, a linear trajectory is not the best choice for motion estimation with our method, an interleaved or random order is better.
• Q 3, do you need the fully sampled k-space data to learn $f_\theta$: We do not, as discussed in Section 6, our method can leverage recent advances in self-supervised training of MRI reconstruction networks. This is a strength of our method, other proposed deep learning based method for (2D) motion correction in MRI rely on simulated motion artifacts during training which require fully sampled k-space data.
• Limitations, the method’s effectiveness might vary significantly with different sampling patterns and motion artifacts: Regarding the role of the sampling pattern see author rebuttal. Regarding the role of motion artifacts we evaluated our method on motion events occurring at random time points with different amplitudes and frequencies resulting in different motion artifacts achieving good performance for all of them. In general, we expect our method to work well for different types of motion as it was not trained on a particular type of simulated motion. Remaining limitations have been discussed above.

We hope the provided theory as well as the additional simulations and clarifications address the concerns of the reviewer and if yes, that the reviewer considers raising their score.