Teach Multimodal LLMs to Comprehend Electrocardiographic Images

Thanks to the reviewer for the positive feedback on the first work converting ECG signals into visual input and establishing a comprehensive benchmark for evaluating ECG, opening a new domain for ECG analysis.

Reliance on Accuracy and LLM Scoring

We conducted a human evaluation on 50 sampled reports from the PTB-XL Report and 30 questions from the ECG Arena. Human evaluation scores and their Pearson correlation with LLM-based scores are presented in Table 1. The results indicate a strong correlation between the two sets of scores. In the final version, we will extend this evaluation to the full dataset and include comprehensive human evaluation results.

ViT Model Limitations

Thanks for pointing this out! We conducted an ablation study by unfreezing the ViT module parameters during training and reported the model performance in Table 2 below. The results show a performance improvement (i.e., average score from 71.8 to 75.0) compared to the original model with frozen ViT parameters. We have included these new findings in the revision.

Translation Issues in Report Generation

We acknowledge the importance of maintaining accurate translations, particularly in evaluation tasks. During the evaluation data curation process, we validated the translated reports for 500 test samples through two methods: 1) manual verification by human experts and 2) automated validation, where the translated reports are compared against SCP-ECG (standardized protocols for ECG analysis including rhythms, waveforms, and diagnoses) provided along with each ECG in the dataset [Wagner et al.].

Wagner, Patrick, et al. "PTB-XL, a large publicly available electrocardiography dataset." Scientific data 7.1 (2020): 1-15.

Evaluation Metrics in Report Generation

We include clinical efficacy metrics to evaluate the model performance in report generation. Specifically, we label the generated reports and compare them with ground truths across 23 categories (i.e., diagnostic subclass in PTB-XL). Metrics including precision, recall, and F1-score are utilized for evaluation. Since no established labeling tool exists for ECG reports, we leverage a language model (GPT-4o) to extract labels based on the 23 categories. We also experimented with a rule-based labeling approach, but it performed poorly in this context. As shown in Table 3 below, our model significantly outperforms representative proprietary and open-source MLLMs in terms of clinical efficacy metrics, highlighting its robustness across diverse evaluation criteria.

Thanks to the reviewer for acknowledging our contributions, including the instruction-tuning dataset covering a wide range of ECG-related tasks and a new evaluation benchmark for ECG image interpretation, both of which advance the field!

The need for AI models on ECG printouts

The need for AI models capable of analyzing ECG printouts is highlighted by the potential to overcome limitations of signal-based models, expand access to advanced diagnostic tools in resource-constrained settings, seamlessly integrate with existing clinical workflows, and support diverse applications such as retrospective studies, remote healthcare, and AI-assisted cardiology. See our detailed illustrations below.

• Limitation of current signal-based models. In many clinical settings, ECG data are frequently stored only as printed images without corresponding signal repositories, particularly in resource-constrained, and remote settings [Siontis et al.,]. These environments often face limited access to specialized medical expertise, while the increasingly accumulated paper ECGs surpass the capacity of human experts to analyze them effectively [Schopfer et al.]. However, these widely used paper ECGs or scanned ECG images are incompatible with existing ECG models, which are trained and tested exclusively on time-series physiological signals. Furthermore, variations in data formats and system architectures among ECG device vendors [Cuevas-González et al.] pose additional challenges to the interoperability and practical application of signal-based models in clinical practice [Chung et al.,].
• Motivation for developing image-based models. Image-based models can address the aforementioned limitations by enabling AI inference directly from ECG images, which are the primary formats used by clinicians [Cuevas-González et al.,]. This approach accommodates rural, low-resource, or remote clinic settings, where signals may be unavailable, making advanced diagnostic tools accessible to broader populations.
• Clinical implementation for image-based models. In many healthcare facilities, access to inexpensive portable scanners (or even smartphone cameras) is far more common than access to advanced ECG signal management systems. AI tools for ECG images can be implemented on lightweight hardware, such as laptops or mobile devices, with minimal computational requirements. Clinicians already rely heavily on printed ECGs for diagnosis, making image-based AI solutions a natural extension of current practices without requiring systemic changes.
• Multiple real-world use cases. Image-based models offer diverse real-world applications, including 1) supporting remote clinics and low-resource healthcare where ECGs are printed for interpretation and long-term storage lacks infrastructure for signal storage or processing [Chung et al.,]; 2) enabling retrospective studies by analyzing archived ECG printouts; 3) AI-assistant for cardiology. With the ability to comprehend and respond to complex natural language queries, the models can be used as AI assistants for cardiology, enhancing human-in-the-loop clinical decision-making, education, and research.

Siontis, Konstantinos C., et al. "Artificial intelligence-enhanced electrocardiography in cardiovascular disease management." Nature Reviews Cardiology 18.7 (2021): 465-478.

Schopfer, David W. "Rural health disparities in chronic heart disease." Preventive Medicine 152 (2021): 106782.

Cuevas-González, Daniel, et al. "ECG standards and formats for interoperability between mHealth and healthcare information systems: a scoping review." International Journal of Environmental Research and Public Health 19.19 (2022): 11941.

Chung, Cheuk To, et al. "Clinical significance, challenges and limitations in using artificial intelligence for electrocardiography-based diagnosis." International journal of arrhythmia 23.1 (2022): 24.

Additional comparison with existing signal-based ECG models

We evaluate our model against MMCL and MOMENT under three different settings: zero-shot, linear probing, and full fine-tuning. In the zero-shot setting, MMCL and MOMENT extract ECG signal features and compute the cosine similarity with the text features of class names, which serves as the classification probability. The text encoders used are all-MiniLM-L6-v2 and bert-base-uncased, chosen to match the feature dimensions of the respective models. We also attach a linear classification head to the three models, evaluating them in other two settings: freezing the encoder and fine-tuning only the classification head (Linear Prob) and jointly training both the encoder and the classification head (Full Finetune).

Thanks to the reviewer for positive feedback on constructing comprehensive ECG image instruction tuning dataset from diverse data sources and outstanding model performance against proprietary and open-source MLLMs by a large range in diverse tasks.

Further improvement in instruction-following and multistep reasoning capabilities

To improve the model’s capabilities of instruction-following and multistep reason, future work can focus on two main areas: (1) incorporating a more diverse set of instruction-following data to enhance the model's generalizability, and (2) scaling up high-quality chain-of-thought (CoT) and multi-turn training data informed by clinicians' expertise, established knowledge databases (e.g., SNOMEDCT [Stearns et al.,]), literature or textbooks. This curated data would include intermediate reasoning steps such as identifying key features, relating these features to diagnoses, and providing well-grounded rationales to enhance multistep reasoning.

We believe that scaling up and diversifying training data will improve instruction-following and multistep reasoning performance. This is also supported by our data ablation studies presented in Tables 5 and 6, which indicate the potential for improving model performance with additional training resources. We aim to explore these directions in future research to address the gaps noted in our current work. We have included this discussion in the revision.

Stearns, Michael Q., et al. "SNOMED clinical terms: overview of the development process and project status." Proceedings of the AMIA Symposium. American Medical Informatics Association, 2001.

Thanks to the reviewer for recognizing our contributions, including the curated training dataset ECGInstruct and the comprehensive benchmarking suite ECGBench, to the field!

Technical novelty

We present the first comprehensive study on using MLLMs for interpreting ECG images, addressing critical limitations in existing methodologies and introducing a standardized benchmark for model evaluation in this domain.

• The use of MLLMs for ECG Image Interpretation. The problem we studied is novel that, to the best of our knowledge, we are the first work to investigate the MLLMs in ECG image interpretation (also mentioned by reviewer FYMk). Compared to existing ECG models that are trained and tested on time-series ECG for specific tasks, our model can directly read printed or scanned ECG images and provide interpretations given complex user queries for a variety of ECG-related tasks.
• Addressing the Gap in Current MLLMs for ECG Interpretation. Despite the importance of interpreting ECG images (e.g., application in resource-constrained or remote settings, complexities in clinical implementations, etc.), even state-of-the-art MLLMs (e.g., GPT-4o, Gemini-1.5 Pro, or Claude 3.5 Sonnet) fail to provide accurate and comprehensive ECG interpretations, as demonstrated in Figure 1. Our work identifies this critical gap between the capabilities of current methodologies and real-world clinical needs. We aim to bridge this gap by teaching MLLMs to comprehend ECG images in a manner akin to human interpretation.
• Methodology for instruction-tuning data curation. Training such models is challenging due to the lack of large-scale, high-quality, and diverse datasets comprising ECG images and associated text instructions. Our data curation process stands out through 1) realistic image synthesis resembling the real-world artifacts (e.g., distortions, varying background colors, injected textual meta-information, etc.) in paper ECGs to enhance the model robustness against “imperfect” images commonly encountered in practice; 2) incorporating clinician’s insights to identify key ECG-related tasks for more effective understanding; 3) diverse instruction generation: generate comprehensive and diverse instruction data based on original diagnosis and/or clinician’s reports using language models in a scalable approach; 4) quality checking: filtering out low-quality data to maintain high standards for training.
• Comprehensive evaluation for benchmarking and future directions. Evaluation is as crucial as model development, especially in the context of LLMs. A robust evaluation suite like ECGBench not only benchmarks model performance but also highlights current limitations, guiding future advancements. As discussed in the paper, ECGBench provides a concrete assessment framework for identifying gaps (e.g., enhancing multistep and complex reasoning capabilities) and informing the next steps in improving model capabilities.

Technical details

We follow the model architecture of LLaVA, which includes three core components: a vision encoder, a large language model, and a projector to align image and text modalities. Table 1 below summarizes all the model parameters. Specifically, for the LLM, we utilize Vicuna-1.5-7B, while the vision encoder is based on CLIP-ViT-Large-Patch14-336. We employ a 2-layer MLP as a projector to map the visual features from the CLIP encoder onto the tokens used by the LLM. These features are mapped onto predefined image tokens, which encapsulate the features of ECG images. The tokens representing ECG features are then concatenated as an image context preceding the dialogue.

We format all datasets into a chatbot-style multi-turn dialogue format (same as Vicuna-1.5-7B) and use the special token <image> to represent image features within the text data. For example, a sample data instance is: “Human: <image> Describe this ECG image.\nAssistant: This image …”. To enhance the model’s ability to handle ECG images of various sizes encountered in real-world scenarios, we employ Anyres. Anyres divides high-resolution images into multiple sub-images of size 336x336. The features of these sub-images are then concatenated with the global features of the original image to form the final image representation.

We fine-tune all parameters of the vision encoder (see newly added vision encoder unfrozen experiments in response to the reviewer FYMk <Table 2>), projector, and the LLM. The training process uses a learning rate of 2e-5, a batch size of 128, and a cosine scheduler with a 5% warm-up period over three epochs. The loss is calculated using the cross-entropy loss function, focusing on the response portion of the dialogue. We have included the technical details and Table 1 in the revision.