Med-Real2Sim: Non-Invasive Medical Digital Twins using Physics-Informed Self-Supervised Learning

We thank the reviewer for the very thoughtful comments and feedback!

Weaknesses:

Thank you for highlighting these issues. Please see below a point-by-point response to your concerns.

• While the Windkessel model does not fully capture the complexity of cardiac dynamics, this simplicity is actually a strength of our approach. The heart is an incredibly complex organ system, with millions of parameters describing its electrophysiology, hemodynamics, and biomechanics. However, it is not always necessary to simulate the entire cardiovascular system to make specific diagnostic or treatment decisions. Our model focuses on predicting pressure-volume (PV) loops relevant to diagnostic and treatment procedures for end-stage heart disease. In this context, the Windkessel model is the most parsimonious in-silico simulator for the relevant physiological processes. Additionally, we want to stress that our main contribution is a general two-stage physics-informed self-supervised learning approach and not a specific in-silico model—this approach is general enough to incorporate any cardiovascular system simulator, including more complex lumped circuit models of the arterial system.
• Please note that non-invasive data is not necessarily of low quality; it simply means the data was acquired without invasive procedures, such as catheterization. Echocardiograms, which we use in this paper, are a widely-used, gold-standard non-invasive method for diagnosing cardiac function. Our paper proposes a new method that extracts additional valuable information from echocardiograms, which previously required invasive procedures and were impractical in most primary care settings. This method enables new use cases and does not compromise the accuracy of any existing procedures.
• This dataset includes a reasonably diverse demographic, comprising 68% White, 14% Black, 7% Asian patients, and individuals from other ethnicities. Additionally, 55% of the patients are male. We will provide a detailed demographic performance breakdown in the Appendix of the final paper.

Responses to questions:

Question 1: We selected learnable versus fixed parameters based on the intended use case of predicting cardiac left ventricular pressure-volume (PV) loops in combination with a sensitivity analysis to understand how they affect the model. Ultimately, parameters directly related to the circuit analog of blood flow in the left ventricle were allowed to be variable, while others were fixed to isolate the prediction of these dynamics. We first defined intervals of validity for each of the 14 independent parameters, based on how variations of each parameter affects the PV loops (realistic shape and magnitude of pressures of volumes), while ensuring that the volume ranges and EF ranges of the two echo datasets were covered. These were centered around reference values from the literature. For the 14 independent parameters, we did perform a sensitivity analysis based on the state-space matrices in Eqs. 21. We selected the final 7 learnable parameters, based on how changes in these parameters affect the volume space, their physical meanings, their resulting PV loop shapes, and their ability to encompass the volume ranges of the two echo datasets. We include a table in the global rebuttal PDF that summarizes the sensitivity analysis results and modeling choices.

Question 2: It is definitely true that, in general, patients with similar echocardiograms may be assigned identical digital twins and it is also true that this may not capture sufficient differences between patients. This is a general issue encountered in all inverse modeling problems, and is not peculiar to our setup. In our setting, we use a lumped parameter model with a small set of parameters that we have reasons to believe will be able to be distinguished between patients because they have a visual representation in an echocardiogram. This means that there are correlations between visual features and values of the in-silico model parameters that allow us to uncover meaningful differences in the digital twins that can distinguish, for example, mitral stenosis (Fig. 3(b)). We believe that studying the identifiability of specific models, developing general conditions for identifiability of physics-based models and developing new learning procedures that improve the chances of model identification are all very interesting directions for theoretical research that complements our applied method. One potential direction to improve identification would be incorporating patient level meta-data or other modalities as is done in other digital twin settings to enhance the model’s ability to distinguish between patients to make a measurement model more nearly injective.

Question 3: It would be really interesting to study the use of electrocardiograms in this setting and even more interesting to combine these data assets. Echocardiograms were the natural choice for our model because they are a standard tool for encoding information about left ventricular pressure in addition to left ventricular volume, which they are routinely used to assess [Popescu et al 2022]. However, we recognize that incorporating additional non-invasive modalities such as electrocardiograms could potentially enhance the model’s performance. We are curious to explore the possibility in future studies.

Thank you for catching the references and minor typos, we will happily update those. Figure 7 is intended to show shorting the circuit to calculate an equivalent circuit.

Bogdan A Popescu, Carmen C Beladan, Sherif F Nagueh, Otto A Smiseth, How to assess left ventricular filling pressures by echocardiography in clinical practice, European Heart Journal - Cardiovascular Imaging, Volume 23, Issue 9, September 2022, Pages 1127–1129, https://doi.org/10.1093/ehjci/jeac123

We thank the reviewer for your very thoughtful comments! Below, we provide a detailed response to the weaknesses and questions.

Weaknesses:

Please see below a point-by-point response to your concerns.

• The limited baseline comparisons stem from the novelty of our problem setup. While there are many variants of Physics-Informed Neural Networks (PINNs) in the literature, not all are suitable for our specific problem. Our setup is unique because it involves a partially known model that combines both known and unknown components. As explained in lines 721-725, this setup can naturally be approached using a generative inverse problem approach to model $\mathcal{F}^{-1}$. We compare this model to other generative and PINN solutions in Table 4, using two taxonomies: the known versus unknown forward model (rows) and paired versus unpaired input-output data (columns). Table 4 highlights that the various PINN variants make different assumptions about the problem setup, making them unsuitable as baselines for our case. Therefore, we chose to compare our approach only with baseline PINN and Neural ODE methods. Our goal was to demonstrate that our surrogate model achieves a different goal but still has a comparable understanding of the underlying physics as these standard approaches, rather than showing superior performance in learning physics. We are open to implementing additional baselines if the reviewer suggests specific PINN variants that are applicable to our setup.
• We did conduct an ablation by removing the physics-based model to evaluate its impact on ejection fraction (EF) predictions (Table 1). Additionally, we explored other feature extraction architectures, such as LSTM and transformer models, but these did not achieve the same level of loss minimization as our 3D-CNN approach. For the two-stage training strategy, we also experimented with alternative differentiable ODE solvers. However, these solvers were not compatible with our architecture and resulted in similar parameters across different patients or introduced additional numerical error. Consequently, we opted for our two-stage approach, which includes a physics-informed pretext task and physics-guided fine-tuning with 3D-CNN. We are very open to suggestions from the reviewer for additional ablation studies that could further validate our approach and their implications for our set up.

Responses to questions:

Question 1: First, we want to clarify the difference between the patient's cardiac states and cardiovascular parameters. Our model is dynamic: it analyzes an ultrasound video to extract a cardiac state that varies over time, characterized by changes in pressure and volume throughout the video, which lasts for several minutes. The underlying parameters, however, are fixed because they depend on the heart's anatomy, deformities, and tissue properties, which change over days, months or years, not within the short duration of a single echocardiogram acquisition. While modeling the long-term evolution of cardiac health is a fascinating problem, our model can still address this setup by applying our two-stage procedure to new echocardiograms acquired over different years.

Question 2: The analogy between cardiac blood flow and electric circuits is a classical result in hydrology and fluid mechanics. In this analogy, blood flow is analogous to electrical current as it is driven by pressure differences and encounters resistance like electricity is driven by voltage and material resistance. Valves act as diodes, capacitors emulate the elastance behavior of heart chambers, and heart tissues induce resistance analogous to that in electric circuits. This analogy is often referred to as the “drain-pipe theory” and is widely used in fluid mechanics beyond cardiovascular modeling. The class of models studied in our paper, called Windkessel models, with varying complexity have been used to model cardiac dynamics for many years, though other classes of models do exist. The model we take to be fixed for our paper was validated in the Simaan et al paper (which we specifically used for its incorporation of an LVAD). The authors synthetically perturbed the model to demonstrate the behavior was as clinically expected and compared to real human measurements to demonstrate accuracy.

Kirchhoff's laws are adhered to in the model simply because they are adhered to in any electric circuit. In this way, Kirchhoff's Voltage Law parallels the conservation of energy in the circulatory system, where the sum of pressure drops around a closed loop equals the pressure driving the flow. Kirchhoff's Current Law reflects the conservation of mass in blood flow, ensuring that the total blood flow entering a junction equals the total flow exiting.

Finally, we want to address your overall comment and clarify that we do not make strong assumptions about PDE dynamics. Instead, we use a low-fidelity model that exactly characterizes cardiac hemodynamics in the heart chambers relevant to cardiovascular diseases, while ignoring the details of blood flow in other arterial peripherals. We strongly believe that our approach has broad applications for individualized treatment of end-stage heart failure. This includes not only the use of LVAD, as demonstrated in Section 4.4, but also any other medical device with a corresponding in silico model. Additionally, our method can be used to diagnose diseases not typically identified using echocardiograms alone, such as mitral stenosis.

Thank you for your review of our paper. We greatly appreciate your and other reviewers feedback from our ICML submission, and we have implemented changes to address these comments in our new submission. We are happy to share that we have added additional validation and comparison of the physics based surrogate model in response to your feedback. Specifically, we compared this approach to a baseline PINN approach and a Neural ODE approach, which demonstrate similar performance but fundamental differences in set up (Table 2). We also took into account your suggestion of further validating the surrogate model and its generalizability by adding predictions on an additional out-of-sample synthetic dataset in Section 4.5 and Appendix Figure 9.

In response to your specific concerns, we restate or add the following in a point-by-point response.

Weakness 1: Thank you for your observation regarding the comparison with supervised 3DCNN and other segmentation models in predicting EF from echocardiogram data. While we acknowledge the potential for supervised segmentation models to yield superior results in terms of mean absolute error (MAE) for EF prediction, it's important to clarify that our study's primary objective differs from that of these supervised algorithms. In our research, the primary focus is utilizing the volume states that compose EF (volume at ES and ED) as a clinically significant label to facilitate the training of our model for latent parameters and PV loops. We incorporate physics as fixed layers for this purpose, a process which may marginally compromise the model's EF prediction accuracy. In Table 1, we showcase that the sacrifice in EF prediction accuracy is minimal and acceptable within the context of our objectives. We acknowledge that certain state-of-the-art Unet models designed for segmentation tasks may indeed achieve superior MAE in EF prediction. However, it's crucial to note that these models are specifically optimized for segmentation purposes, whereas our methodology integrates physics-informed learning to achieve broader objectives beyond segmentation alone.

Weakness 2: Thank you for bringing up the limitation on the validation of our latent parameters. Evaluating solely comparing EF derived from physical parameters to observed EF for validation is clearly insufficient as you point out. Validating these individual parameters is of course challenging given that they are parameters of a physical model rather than exact patient measurements. In our particular experimental setting, our parameters certainly have physiological meaning, which we describe qualitatively, but they are not directly measurable. For example, Rm, mitral valve resistance, in our model is an electric circuit analogy parameter that is not equal to measures of mitral valve resistance in clinical practice. We sought to perform qualitative validation, i.e. describing qualitative relationships between Rm and actual mitral valve resistance and in using predicted PV loops to predict mitral valve stenosis disease labels. In future work, we would seek to find correlations between model parameters and physiological effects or conditions. For example, [Lamberti et al 2024 below] show correlation between pulmonary vascular compliance, a parameter similar to those in our model, and right heart decompensation with LVAD implantation.

Kimberly K. Lamberti et al, Dynamic load modulation predicts right heart tolerance of left ventricular cardiovascular assist in a porcine model of cardiogenic shock. (2024).DOI:10.1126/scitranslmed.adk4266

Weakness 3: The surrogate model/pretext task is definitely intended to be domain dependent and would need to be trained on synthetic data for any new in silico mathematical model. The data will always be synthetic for pretraining the physical dynamics, so it would not be dependent on the image data’s collection. In our case, the surrogate model is trained on a synthetic multidimensional grid filling a space of the form [a_1, b_1] x .. x [a_n, b_n], where [a_i, b_i] is the validity interval for the parameter θ_i, and θ=[θ_1,...,θ_n] and is chosen accordingly to the problem addressed. From this synthetic dataset, covering all possible states (and thus all possible patients), the surrogate model M(θ) is domain independent, and capable of predicting M(θ) even for extreme or shifted situations (since these are included in the synthetic dataset). We add an out-of-sample prediction on 1000 new points and show a uniform distribution of errors and comparison of this method to other physics informed approaches to demonstrate that it learns just as well and is better suited to our set up.

Thank you for the feedback on our paper!

The proposed method is indeed general and not limited to the cardiovascular system example. Any physical or biological system that can be described through ordinary differential equations can utilize our approach. The problem setup in Section 2.1 and the training/loss functions in Section 2.2 are designed to be very general, capturing any mapping from non-invasive observations to a physical system defined by differential equations. We chose to focus on cardiovascular hemodynamics as our example because our contribution is conceptual rather than empirical. By studying a specific disease area in depth, we were able to illustrate a wide range of use cases for digital twins, from diagnosis to simulation of counterfactual treatment outcomes. These use cases were previously not possible, and our goal was to demonstrate the potential rather than validate a new model for an existing use case at scale. We believe that our in-depth focus on a disease area enhances rather than undermines our contribution.

To give another example for a problem setup in oncology where the exact same framework can be applicable, consider the in-silico model proposed in [Baldock et al 2013], which describes the use of an ordinary differential equation for modeling proliferation and invasion of tumor cells. The two parameters of this model: dispersion D and proliferation p, have physiological meaning and impact on cancer prognosis [Baldock et al 2013]. They have been proposed to be measured using a mathematical formula from diffusion weighted MRI [Ellingson et al 2010 below]. Our approach could learn the dynamics of the ODE and build a non-invasive direct approach for these parameters to be extracted from images with a 3DCNN, provided data is available. By demonstrating a method for creating digital twins that can be generalized, our work could encourage more public sharing of relevant healthcare datasets, fostering the development of new digital twins across various medical fields. This is a crucial step towards broader applications and greater impact in medical research.

In response to your questions regarding the choice of architecture and ablation studies, we want to stress that our framework is not specific to this architecture. We found the 3D-CNN architecture to be the most empirically performant architecture in predicting ejection fraction using the PINN layers compared to other architectures we tried, such as LSTM and transformer models. This is in line with previous results that showed the effectiveness of 3D-CNNs in modeling echocardiography [Ouyang et al 2021]. Regarding ablation studies, we conducted an ablation by removing the physics-based model to assess its impact in EF predictions. We are very open to suggestions from the reviewer for additional specific ablation studies that could further validate our approach and their implications for our set up.

Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C.P., Heidenreich, P.A., Harrington, R.A., Liang, D.H., Ashley, E.A. and Zou, J.Y., 2020. Video-based AI for beat-to-beat assessment of cardiac function. Nature, 580(7802), pp.252-256.

Ellingson, B.M., LaViolette, P.S., Rand, S.D., Malkin, M.G., Connelly, J.M., Mueller, W.M., Prost, R.W. and Schmainda, K.M. (2011), Spatially quantifying microscopic tumor invasion and proliferation using a voxel-wise solution to a glioma growth model and serial diffusion MRI. Magn. Reson. Med., 65: 1131-1143. https://doi.org/10.1002/mrm.22688