Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model

Q2: 12-lead ECGs were considered as data where some leads can be directly deduced from lead II and III including I, aVR, aVL, and aVF - such channel redundancies may have impact on ECG vocabulary generation. The framework does not show generalisability for less number of channels or even a single channel ECG where there is no spatial items. Experiments with reduced lead sets or single-channel ECGs would be beneficial that would demonstrate the framework's generalizability.

A2: Although these channels may seem redundant, as mentioned in the response to Question 1, larger ECG words enhance semantic expressions, thereby improving both pretraining and downstream task performance. Therefore, we chose to train the model using the standard 12-lead ECG configuration, which is also commonly used in clinical diagnostics. Regarding the adaptability to fewer leads or single-lead ECGs, we conducted additional experiments based on the configuration in [2], and the results are presented in the table below.

The results demonstrate that downstream task performance generally improves with an increasing number of leads, particularly in the Superclass and Subclass subsets for disease diagnosis. Remarkably, even under the single-lead condition, HeartLang outperforms most baseline methods reported in Table 1. This highlights exceptional adaptability of HeartLang to single-lead configurations and underscores its robust generalization capability across different lead configurations. The related results and discussions have been supplemented in the appendix.

Q3: Data split lacks clarity if the train, test and validation splits combine segments across recordings or subject wise splits were considered. A clarification of data splitting strategy in the methodology seems necessary. This is because, if subject wise splits were not considered, then there will be data leakage.

A3: For the division of datasets, we strictly follow the method recommended by MERL (ICML 2024) to ensure fairness in comparing downstream validation results. Specifically, for the PTB-XL series datasets, we use the official dataset processing code for partitioning (https://github.com/helme/ecg_ptbxl_benchmarking). As stated in the PTB-XL original paper, the data division adheres to the principle of patient independence, ensuring that ECG data from the same patient does not appear in different folds, thereby avoiding data leakage. For the CPSC2018 and CSN datasets, we utilize the CSV files provided by MERL for partitioning (https://github.com/cheliu-computation/MERL-ICML2024/tree/main/finetune/data_split). Although the MERL paper does not explicitly illustrate the issue of patient independence, to ensure fairness in comparisons with the baseline methods, we have adopted this approach for dataset partitioning.

We are extremely grateful for your review of the manuscript. You have raised a number of important issues. We agree with your comments and have modified our manuscript accordingly. Below we give a point-by-point response to your concerns and suggestions.

Q1: The ECG vocabulary was learned which shows similar semantic information, however, its consistency, as well as its randomness needs to be rigorously explored. It's necessity seems questionable due to the fact that normal heartbeats from a single recording may have turned into separate ECG words in the vocabulary which does not make much sense. Although the methodological process can generate it, its utility needs to be justified first. This thus undermines one of the claimed novelty. An analysis of vocabulary stability across different recordings or subjects seems necessary.

A1: We would first like to clarify why similar normal heartbeats within the same recording may be mapped to different ECG words. For similar normal heartbeats within a single ECG recording, due to the effects of Spatio-temporal Embedding and Position Embedding, normal heartbeats at different lead positions, different time points, and different sequence positions generate distinct embeddings. These different embeddings are then mapped to the nearest vector in the ECG vocabulary (i.e., different collective ECG words). The entire vector-quantized heartbeat reconstruction process dynamically guides the construction of the ECG vocabulary through both quantization loss and reconstruction loss. In this process, Spatio-temporal Embedding and Position Embedding provide contextual information, influencing the embedding of normal heartbeats before mapping, ultimately resulting in mappings to different collective ECG words.

In previous natural language processing research, dynamic word embeddings that incorporate contextual information have been shown to be superior to static word embedding methods [1]. Dynamic embeddings are now the standard processing paradigm in NLP (e.g., BERT, T5, GPT).

Our study adopts a similar approach, which may seem counterintuitive. However, we believe that context-rich ECG words are meaningful, as they lead to more semantically rich representations. In self-supervised pretraining, where there is no additional supervision, the model learns directly from the data itself. Richer semantic expressions increase the complexity of the pretraining process, allowing the model to learn more generalized representations. We conducted additional experiments on PTB-XL datasets with an ECG vocabulary size of 64, which limits the semantic expressions of ECG words and is similar to the vocabulary size used in previous ECG language processing studies. The downstream validation results are presented in the table below.

The results show that as the vocabulary size increases, the performance of downstream tasks improves significantly. This indicates that enhanced semantic expressions lead to better representations learned during the pretraining stage. Due to the influence of contextual information, even similar normal heartbeats are mapped to different collective ECG words, enriching the semantic expressions. This, in turn, enables the model to learn more general representations during pretraining, ultimately boosting the performance of downstream tasks. In summary, a more diverse collection of ECG words will result in richer semantic expressions, which in turn will enhance the performance of both pre-training and downstream tasks. The relevant results and discussion have been added to the appendix.

Q4: The proposed framework is a variant of EEG based framework, however, the contribution is apparently for ECG modality and the data slicing philosophy. Authors should clarify the claim of framework based novelty accordingly.

A4: Here are the architectural differences between our approach and LaBraM:

• QRS-Tokenizer for Implementing the New Perspective: The QRS-Tokenizer is a key component for realizing the concept of "Heartbeats as Words and Rhythms as Sentences," which significantly sets our work apart from previous studies on physiological signals. Unlike the Neural Tokenizer in LaBraM, which essentially follows the traditional slicing approach with fixed-size and fixed-step time windows and lacks the semantic concept, our approach introduces a novel perspective. As highlighted in Section 5.2, "Evaluation on Signal Slicing Perspective," the "Heartbeats as Words and Rhythms as Sentences" perspective achieves an average macro AUC improvement of 5.36 compared to the "Fixed-size and Fixed-step Time Windows" approach. Notably, for superclass and subclass subsets, our perspective yields an average macro AUC increase of 8.38. These improvements underscore the innovation of our proposed perspective and the QRS-Tokenizer.
• Reconstruction objectives during the vocabulary construction stage: In LaBraM, the reconstruction objective for its VQ-NSP process is the Fourier Spectrum of EEG signal patches. In contrast, in our paper, the reconstruction objective for HeartLang's VQ-HBR process is the original heartbeat. The entire goal of the VQ-HBR process is to construct an ECG vocabulary where morphologically similar and contextually consistent heartbeats are mapped to the same collective ECG word. Fundamentally, LaBraM remains a traditional time-frequency modeling approach, while our method focuses on a morphology-semantic expression level of modeling.
• Effectiveness of QRS-based Temporal Embedding: In our analysis of the HeartLang architecture, ablation experiments reveal the simplicity and effectiveness of the QRS-based Temporal Embedding. The indices for HeartLang's Temporal Embedding primarily originate from a byproduct of the QRSTokenizer—the QRS complex index. In Section 5.4 of the ablation studies, we find that the framework with only Temporal Embedding consistently achieves the second-highest or even the highest performance. This simple and effective model design offers inspiration for future research in the ECG domain, particularly by providing a feasible solution for injecting temporal information into subsequent ELP studies.
• Differences in Model Architecture Details: The backbone network of HeartLang does not employ a Temporal Encoder to capture dynamic information. Instead, we use a simple and efficient QRS-based Temporal Embedding, whose effectiveness has been demonstrated in ablation experiments. Additionally, HeartLang does not utilize Symmetric Masking during the pretraining phase to improve computational efficiency. We aim for the performance improvement of our method to stem from the proposed ECG language modeling perspective rather than auxiliary techniques.

We acknowledge being inspired by the LaBraM paper you mentioned and have appropriately cited it in the main text. As you pointed out, our work is more focused on innovating the concept of data slicing for ECG signals. We treat ECG signals as a language and model them using approaches from natural language processing. Given that most tasks in the ECG research are classification tasks, we adopt a BERT-like architecture. There have already been some successful efforts to apply the BERT architecture to other domains, such as BEiT v2 [3] (computer vision) and LaBraM [4] (EEG signals). Our aim is to apply the BERT architecture to the ECG research field, which has resulted in a similar model architecture. However, there will still be differences, such as the aforementioned points.

Thank you again for your valuable feedback and insights.

[1] Kawin Ethayarajh. 2019. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. EMNLP 2019.

[2] Oh, Jungwoo, et al. "Lead-agnostic self-supervised learning for local and global representations of electrocardiogram." Conference on Health, Inference, and Learning. PMLR, 2022.

[3] Peng, Zhiliang, et al. "Beit v2: Masked image modeling with vector-quantized visual tokenizers." arXiv preprint arXiv:2208.06366 (2022).

[4] Jiang, Wei-Bang, Li-Ming Zhao, and Bao-Liang Lu. "Large brain model for learning generic representations with tremendous EEG data in BCI."ICLR 2024.

Q2: What are the novel findings?

A2: Our novel findings primarily focus on the analysis of the ECG vocabulary and the HeartLang architecture.

• For the ECG vocabulary, a novel finding in the current manuscript is that in our constructed vocabulary, even similar heartbeat morphologies can yield different semantic representations based on contextual information. In natural language processing, context plays a crucial role in shaping the semantic representation of specific words. For example, the word "run" can be either a noun or a verb, with its meaning dependent on surrounding context. In our work, the vector-quantized heartbeat reconstruction training dynamically constructs the vocabulary, with spatio-temporal embedding and position embedding allowing our vocabulary to incorporate contextual information. In contrast, previous ECG Language Processing (ELP) research has commonly used K-means clustering for vocabulary construction, which captures only heartbeat morphology without contextual information. This discovery significantly enriches the semantic depth of the vocabulary in ELP, contributing to advancements in the field.
• For the HeartLang architecture, we find that the Temporal Embedding based on QRS waves is both simple and effective in our ablation studies. The Temporal Embedding in HeartLang is indexed primarily from the by-product of the QRS-Tokenizer—the QRS complex index. Specifically, we divide each 10-second ECG signal into 10 intervals, and for each individual ECG word, we assign the temporal embedding corresponding to the interval where its QRS complex index is located. In the ablation study in Section 5.4, we observe that when the framework includes only Temporal Embedding, it often achieves the second-highest or even the highest performance. This simple and effective model architecture could inspire future research in the ECG field, especially by offering a feasible solution for incorporating temporal information in subsequent ELP studies.

Thank you again for your valuable feedback and insights.

We are extremely grateful for your review of the manuscript. To begin, the manuscript you referenced is a workshop paper. These two manuscripts differ significantly in their experimental setups, content, and findings.

Q1: What extent does the current manuscript differ to the manuscript accepted at KDD-AIDSH 2024?

A1: The current manuscript differs significantly from the manuscript you mentioned in terms of experimental setup, content, and findings.

• In terms of experimental setup, the current manuscript uses the MIMIC-IV-ECG dataset for pretraining and the PTBXL-Superclass, PTBXL-Subclass, PTBXL-Form, PTBXL-Rhythm, CPSC2018, and Chapman-Shaoxing-Ningbo (CSN) datasets for downstream task validation. In contrast, the manuscript you mentioned only used the PTB-XL series datasets for both pretraining and downstream validation. The MIMIC-IV-ECG dataset comprises 800,035 12-lead ECGs collected from 161,352 subjects, making it one of the largest publicly available ECG datasets to date. Meanwhile, the PTB-XL dataset includes only 21,837 12-lead ECGs collected from 18,885 subjects. For downstream task validation, we utilized six datasets—PTBXL-Superclass, PTBXL-Subclass, PTBXL-Form, PTBXL-Rhythm, CPSC2018, and CSN—covering over 100 cardiac conditions. The current manuscript follows the recommendations from MERL (ICML 2024) regarding upstream and downstream datasets to ensure fair comparisons with 10 other self-supervised learning methods.
• In terms of experimental content, the current manuscript differs by including validation on additional downstream datasets such as the CPSC2018 and CSN datasets (Section 5.1), an evaluation of the proposed perspective on signal segmentation (Section 5.2), a detailed discussion on ECG Vocabulary (Section 5.3), and comprehensive ablation studies (Section 5.4). In Section 5.2, we validate the advantages of the "Heartbeat as Words and Rhythms as Sentences" perspective. Compared to traditional fixed-time window segmentation methods, our approach demonstrates an average improvement of 5.36 in macro AUC across downstream tasks. Notably, in disease superclass and subclass datasets, this perspective increases the macro AUC by an average of 8.38. In Section 5.3, the current manuscript provides a detailed discussion of the ECG Vocabulary, revealing that even similar heartbeat morphologies can yield different semantic representations based on context. This finding marks a significant advancement over previous ECG Language Processing (ELP) studies, where vocabulary construction typically relied on pre-clustering, often lacking contextual information and resulting in a limited vocabulary with less rich semantic representation. In Section 5.4, a comprehensive ablation study on the HeartLang structure demonstrates the importance of each component within HeartLang.
• The current manuscript thus differs significantly from the manuscript you mentioned in all these areas, which you can refer to for review based on the content outlined above.

We truly appreciate your encouragement, careful review, and valuable suggestion. You have raised an important issue. We agree with your comments and have modified our manuscript accordingly.

Q1: The approach of this research, which treats each heartbeat as a word and the rhythm of the time series as a sentence, is a very good idea for bringing the success of large-scale language models in natural language to electrocardiograms, and as described in this paper, it has also been successfully implemented. Continuing from the above weakness section, it would be even better if there was a clear discussion of the limitations in the main text.

A1: Your concerns are valid, and HeartLang maybe exhibit some performance decline in classifying certain heart conditions with less pronounced QRS waves. We will clarify this limitation in our discussion. However, as our current manuscript has already reached the 10-page limit imposed by ICLR for the main text, we have placed this discussion in the appendix to facilitate reference for future research.

Thank you again for your valuable feedback and insights.

We are extremely grateful for your review of the manuscript. You have raised an important issue. We agree with your comments and have modified our manuscript accordingly.

Q1: It is mentioned that the QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences but as per my understanding the QRS tokenizer should be used to segment into each ECG beats aka words?

A1: Yes, your understanding is correct. Thank you for your careful observation and we apologize for any confusion caused by our wording.

Indeed, the process described in Section 3.1, “Generating ECG Sentences Using the QRS-Tokenizer,” consists of two main steps: QRS Detection and Generating ECG Sentences. During the QRS Detection step, the raw ECG signal undergoes QRS wave detection to determine the position of each heartbeat and segment individual ECG words (heartbeats). In the subsequent Generating ECG Sentences step, we concatenate these words to form an ECG sentence by rules.

Following your suggestion, we will revise the manuscript to adjust statements like “QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences” to “QRS-Tokenizer is used to generate the ECG sentences from the raw ECG signals.”

Additionally, regarding the potential limitation you mentioned about the broader application of HeartLang, please refer to Section 5.1, "Evaluation on Linear Probing," and Table 1. Our study follows the downstream task validation approach recommended by MERL (ICML 2024), utilizing six downstream datasets—PTBXL-Superclass, PTBXL-Subclass, PTBXL-Form, PTBXL-Rhythm, CPSC2018, and CSN—to cover over 100 distinct cardiac conditions. HeartLang has shown strong competitiveness compared to 10 other self-supervised methods. The extensive downstream validation and strong competitiveness demonstrate relatively broad applicability of HeartLang.

Thank you again for your valuable feedback and insights.

We are truly grateful for your professional and meticulous review. Your observations and suggestions have significantly contributed to the improvement of our manuscript.