Bidirectional Recurrence for Cardiac Motion Tracking with Gaussian Process Latent Coding

Dear reviewer Tej6: Thank you for your kind comments and suggestions, which help us improve our paper's quality. Here are our responses to weaknesses and questions. New experiments are included in the uploaded PDF file to better illustrate your questions.

Q1. Method applied to complete cardiac cycle

The video frames of the complete cardiac cycle can be continuously fed into our model like the recurrent neural networks (RNNs). The outputs are the deformation between two adjacent frames. Specifically, motion tracking can be regarded as a task of continuous registration and computing the diffeomorphism between every adjacent frame of a video. The motion tracking task is we give the video with $t$ frames ($x_0, x_1, \cdots, x_{t-1}, x_t$) of the complete cardiac cycle, and we output the diffeomorphism ($\phi_1, \cdots, \phi_{t-1}, \phi_t$), where $\phi_i$ indicates the movements from $(i-1)$-th frame to $i$-th frame. In this paper, we strictly follow the previous research [15, 16] and ensure that all the input and output are consistent in experiments.

Q2. The difference between... Is it a fair comparison?

As the response in Q1, the input of our approach and the traditional DL-based registration network remains the same, i.e., $\phi_1=\text{net}(x_0, x_1), \phi_2=\text{net}(x_1, x_2), \cdots, \phi_t=\text{net}(x_{t-1}, x_t)$. The traditional DL-based registration network computes the diffeomorphism between only two adjacent frames. In comparison, by employing the recursive manner, our approach can aggregate and maintain the historical temporal information of cardiac motion by hidden state. The introduction of the Gaussian Process can also promote temporal consistency and regional variability in compact latent space. As shown in experimental results, our method reaches higher tracking accuracy while maintaining lightweight inference speed, computational consumption and model parameters. We ensure that the higher performance can be conducted by our approach under the same computational resource compared to those traditional DL-based registration network.

Q3. A pair of images or a complete cycle of images as the input?

Please see our posed comment below.

Q4. Comparison to the latest registration methods.

We have added three state-of-the-art registration/tracking methods [56, 57, 58] in our experiments on the ACDC dataset. Please see Table R.1.

Q5. Significantly increases the computational cost and parameters...

As the response in Q2, our experiments illustrate that our approach will not significantly increase computational cost and model parameters. This is because our method can recursively accept the pair-wise frame as input step by step and each input shares the same network as the RNNs. This indicated that our method has a similar inference speed, computational consumption and model parameters as those other methods. All result showcases the significant improvement in tracking and registration accuracy.

Q6. Potential applications

As we discussed in the section Appendix A2. Broader Impacts, the cardiac motion tracking can help estimate and quantify myocardial motion within a cardiac cycle. It is a cost-efficient and effective approach for assessing cardiac function. [59] has utilized cardiac motion to efficiently predict human survival. [60] apply motion tracking for physiological motion compensation systems. Motion tracking has also been used for the identification of dilated cardiomyopathy patients in [61]. Our approach can improve the performance of tracking algorithms, and improve the precision of the myocardial motion tracking in model evaluation. Also, we provide motion information to radiologists and sonographers to facilitate human cardiac assessment.

Q7. How do authors process the dataset and train the proposed method?

Initially, we select 32 frames from each video, crop and resize all frames to $256\times 256$ in weight and height. For an input video, the input size is $256\times 256\times 32$. During training, the batch size will be set as 1, and our network will take the first frame $x_0$ and initial hidden state $h_0$ (zero vector) as inputs, generating the feature $f_0$ and hidden state $h_1$. For the next time step, the second frame $x_1$ and $h_1$ will be input to the network and generate the $f_2$ and $h_2$. With such a recursive manner, the feature $f_t$ is always generated by $x_{t}$ and $h_{t}$, where the $h_{t}$ help maintain and aggregate the temporal feature information of cardiac motion from $0$ to $t$ moment. The bidirectional process shares the same pipeline as the above description. For the optimization of the training, we use the loss in section 3.5.

Q8. Why separate the original images into several patches? ...

1). Based on our observations of cardiac motion, we first found that cardiac motion has only a small displacement in Echocardiography/MRI images. The position of cardiac main structures remains roughly unchanged throughout images of the same person (See Appendix Figures B1 and B2). 2). We do not simply decompose images into patches, we adapt multiple convolutional layers to convert the images into small patches. Through the receptive field brought by the convolution kernel, we can build the relationship within patches and all patches are not isolated.

Q9. Are the improvements significant...

Our method aims to provide a new approach by integrating temporal information via the recursive manner and Gaussian Process to make more accurate velocity field estimation. Compared to other methods, [36] reports a 1%-1.5% increase compared to [10], and [16] demonstrates only 1% improvement compared to the best method. Our method surpasses the SOTA method with 1%-3% accuracy in 3D and 4D datasets, which illustrates our improvements are significant. Furthermore, these improvements may also contribute to other tasks that may related to cardiac function assessment as illustrated in Q6.

Thank you for your review and valuable feedback, we will address your concerns in the following. Our new experiments are included in the uploaded PDF file to better illustrate your questions. All the references and citations are included in the content of the paper's rebuttal at the beginning.

We thank you very much for all your efforts and valuable time!

Q3. A pair of images or a complete cycle of images as the input?

The input of our model is a sequence of images (frames). However, those frames are not inputted all at once, instead, they are inputted one by one like RNNs and our model is shared for different input frames. The number of frames depends on the task requirements. For example, if we want to estimate the motion of the entire cardiac cycle, the input sequence includes the video frames of the complete cardiac cycle. But if we just want to estimate the motion between two frames, it is ok to only input them in order. In our experiments, motion tracking requires the estimation of the entire cardiac cycle that follows previous works [15, 16] settings. Hence, we unsupervised train network with the entire cardiac cycle as [15, 16], and inference the result between every two adjacent frames in each time step.

Dear Reviewer Tej6:

Thank you again for your valuable comments and your efforts. As the author-reviewer discussion period is coming to a close, we know that there exist some concerns that need to be addressed and clarified. In our rebuttal, we have provided a comprehensive response to your concerns through detailed explanations and thorough experimental validations. May we know if the responses have addressed your concerns?

Thank you for your time and consideration.

Best regards

Dear Reviewer Tej6:

We thank you very much that you can review our paper and provide valuable feedback that helps improve our work. As the author-reviewer discussion period is coming to a close, we would like to know whether our responses have addressed your concerns. We know that the discussion may take up your time. However, your feedback is really important to us, and we would like to address your question in our discussion further.

Thank you for your time and consideration, we look forward to hearing from you.

Best regards

Dear Reviewer Tej6:

The deadline for the author-reviewer discussion period is now approaching. We understand your time is very valuable and we appreciate all your efforts in writing reviews. During the remaining discussion period, we are still waiting for your response and hope our comprehensive response can address your concerns. We will also provide detailed answers to your other questions or concerns.

We all thank you for your time and consideration, and we are looking forward to hearing from you.

Best regards

Dear Reviewer Tej6:

The deadline for the author-reviewer discussion period is almost close. Thank you for your efforts and we are all waiting for your responses. We are here always ready to answer and address your questions or concerns.

Thank you for your time and consideration, and we are looking forward to hearing from you.

Best regards

Dear Reviewer bQtH: Thank you for your valuable feedback, we would like to address your questions point by point in the following.

Q1. Line 26, Can optical flow ...

The optical flow (OF) is also able to capture the temporal coherence. In section Appendix A1, we discussed the difference between OF-based and Diffeomorphism Mapping methods. We consider that OF-based methods do not necessarily preserve topology, non-globally one-to-one (objective) smooth and continuous mapping with invertible derivatives. These attributes do not match the description of human cardiac which is considered as an incompressible material. We also add a new experiment compared to optical flow-based methods in Table R.3 of our rebuttal pdf.

Q4. No experimental comparison with DDPM ...

In our experiments, we compared two state-of-the-art DDPM methods [11, 12]. As shown in Tables R.1 and R.2. Our GPTrack can reach more accurate results compared to DDPM methods with faster inference speed and less computational consumption.

Q5. The relationship between the $\phi_t$...

The diffeomorphism $\phi_t$ is the $t$-th tracking output of our method, which is differentiable and invertible mappings generated by the $t$-th and ($t$-1)-th frames.

Q6. Comparison with method [36]...

We have added the corresponding experiment in Tables R.1 and R.2.

Q7 . Clarify of "long-term."...

As illustrated in Figure 2 of our manuscript, the "long-term" indicates that the historical information of the cardiac motion will be maintained and encoded as features by our framework. Previous conventional methods only consider the relationship between two adjacent frames, while ours can aggregate the features across long intervals bidirectionally.

Q9. Line 135, the meaning of "motion consistency", define and assess consistency

Sorry that our description of "motion consistency" makes you confused. Motion consistency is the consistency of movements between two adjacent state spaces across different individuals. The state space is the set of all possible states of the human cardiac motion. We consider that cardiac motion usually has a certain principle. For example, as shown in Figure 2 in our manuscript, the Diastole and Systole always occur in each heartbeat cycle of every person. The movement of each organ (Ventricle, Atrium, Myocardium, et al.) always has its fixed direction, speed and range. Those motions can be considered as "consistency" and formulated as temporal patterns for learning. In our approach, we utilize the recursive manner to aggregate the long-term temporal feature without increasing the time complexity. Also, we employ the Gaussian Process to capture the temporal correlation and heterogeneity in cardiac motion, and provide predictions and interpolation. These designs help promote temporal consistency and regional variability in compact latent space.

Q11. Figure 3, what is meant by "elu(x)"...

As described in line 166, the $elu(\cdot)$ is the exponential linear unit [42]. We will make it clearer in our final version.

Q12. Line 206, the formula below this line is not numbered...

Thanks for your kind reminder. We have corrected this typo.

Q13. How is $z_{t}$ generated ...

Firstly, as mentioned in Line 203, we assign independent GP priors to all values in the latent coding of $\mathbb{R}^{P \times C}$ dimension, indicating $R \times C$ scalar Gaussian processes are independently assigned across $T$ time stamps. Due to our independent modelling, we take one scalar Gaussian process $\\{z_t\\} _{t=1}^T$ for illustration, whose corresponding observation is $\\{f_t\\} _{t=1}^T$. Then for each scalar sequential Gaussian process, within the framework of representing sequential Gaussian process with Kalman filtering, we use the initial condition $**z**_0$ calculated in Line 220 and mean-variance calculated by Equation 10 to update $**z**_t := \boldsymbol{\mu}_t,~t=1,...,T$. To sum up, the multidimensional $z_t$ comes from aggregating totally $R \times C$ scalar Gaussian process together due to our independent modelling, and each individual is sequentially updated by the Kalman filtering of Equation 10.

Q14. Line 221, sometimes $\mu_{t}$ is a scalar, and sometimes it is a vector.

Thanks for your kind reminder. Here $\boldsymbol{\mu}_{t}$ together with Equation 10 below are all vectors, and we have corrected this typo.

Q15. Conflict in Line 258 and Line 272...

The $368 \times 368$ in Line 272 is a typo, which should be the same size $384 \times 384$ as Line 258. We will correct them in the final version.

Q16. The error decreases and then increases in Figure 6...

In Figure 6, the error is calculated between the $t$-th tracking result tracked from the initial (reference) frame and the ground truth of the $t$-th frame. Since the human heartbeat is a reciprocating motion (refer to Figure 1), while the heartbeat state of the current frame is close to the initial frame, the registration between these two frames is easier, and therefore the smaller the error (the higher the Dice value).

Q17. Comparison with OF-based methods...

Please see the Table R.3 for more details.

Q18. FSDiffReg is only included in Table B3.

Sorry that we only report FSDiffReg in Table B3 and Table 3 since FSDiffReg [11] only provides a 3D network for MRI images and reports the experiment in 4D MRI images. As shown in Table R2, we further the FSDiffReg in 3D echocardiogram video and report its motion tracking results.

Q2,3,8,10,19,20 and 21. Typos / Citation not formatted correctly / Redundant word / Acronyms repeated unnecessarily / Superficial discussion of results / Inconsistencies terminology and structures.

Thank you very much for pointing out the drawbacks and helping further improve our work. We will carefully check the paper to avoid typos and reorganize the connection and details to make it more coherent and easy to read.

We appreciate your review and valuable feedback, we will address your concerns in the rebuttal. Our new experiments are included in the uploaded PDF file to better illustrate your questions. All the references and citations are included in the content of the paper's rebuttal at the beginning.

Thank you for your efforts and valuable time!

The rebuttal addressed my concerns

Dear reviewer ssxr: Thank you for your efforts and valuable feedback, we would like to address your questions point by point in the following.

Q1.Are there any results showing the impact of varying the number of frames?

Table R.4 shows the ablation study of varying frame numbers on the CardiacUDA dataset. Methods that do not consider the temporal information such as VM-DIF [9] and FSDiffReg [11] would not be affected by the frame length significantly. The DeepTag employs the Lagrangian displacements affected by the frame length, whose performance decreases when reducing the number of frames. Our GPTrack also sees degradation when reducing the number of frames, especially when the input only contains two frames, which leads the motion-tracking task to a pair-wise registration task. However, with the contribution of our designed framework and Gaussian Process, the result of GPTrack can also outperform other methods.

Q2. The myocardium is the tissue of interest for motion tracking, as the blood pool of the left/right ventricle involves flow filling. Could there be more focus on comparing the myocardium?

Thank you for your kind suggestion, we highlight the myocardium in Tables R.1 and R.2, in our final version and following research, we will also emphasize the importance of myocardial in cardiac motion tracking.

Q3. The authors add some evaluation comparing the physiological plausibility.

Thank you for your helpful suggestions on how to strengthen our paper. We have reported the related metric in Tables R.1 and R.2 and highlighted the results with blue colour (Please see Rebuttal pdf). Related research mentioned by the reviewer [54, 55] for motion tracking with biomechanics-informed prior uses supervised learning and our method employs unsupervised learning for motion-tracking tasks. Hence, in our rebuttal, we may not be able to compare these two works in our experiments. However, we also consider that physiological plausibility is an important index that can help evaluate the LV myocardial strain estimates and the learnt biomechanical properties. Following the research [54, 55], we compute the mean absolute difference between the Jacobian determinant and 1 ($||J|-1|$) over the tracking areas. In our final version, we will also add these metrics and cite related works in our final revision.

Q4. It appears that the authors misunderstood the cardiac MRI in Figure 2. The short-axis CINE image shows the myocardium of the left ventricle (green) and right ventricle (red) rather than the left atrium.

Thank you for pointing out our mistake and misunderstanding, we will be really carefully revising all Figures, Tables and Descriptions. Make sure all content in our paper is presented correctly in our final version.

Thank you very much for your review and valuable feedback. Your suggestions and comments are important to us and we will address all your concerns in the rebuttal. Our new experiments are included in the uploaded PDF file to better illustrate your questions. All the references and citations are included in the content of the paper's rebuttal at the beginning.

Thank you for your efforts and valuable time!

Dear Reviewer ssxr:

Thank you again for your valuable comments and your efforts, your kind suggestions help further our paper's quality. We know that there may still be some concerns that need to be addressed and clarified. In our rebuttal, we have provided a comprehensive response to your concerns through detailed explanations and more experimental validations. As the author-reviewer discussion period is about to close, may we kindly know if the responses have addressed your concerns?

Thank you for your time and consideration.

Best regards