Deep ECG-Report Interaction Framework for Cross-Modal Representation Learning

Q4.1: About the novelty and discussion with MRM.

A4.1: We highlight our main innovations in A3.3 (rebuttal to Reviewer 3). In addition, MEM is proposed to learn knowledge-enhanced semantic representations of radiograph with a multi-task scheme including radiograph self-completion and report completion. Although MEM introduces image features into the task of report completion, it is still essentially a report mask generation task. The role of image features is to inject some visual features to assist mask prediction of reports. This task can not be regarded as a cross-modal reconstruction task because it mainly completes the report by covering the text representation. Our DERI architecture aims to reconstruct another modal representation based on ECG representation and report representation, respectively. This process of cross-modal mutual reconstruction in our DERI is designed to guide the model towards deeper modal interactions, allowing the ECG representation to effectively learn high-level semantics in the report. In terms of motivation and implementation method design, our approach is very different from MEM, not similar. On the other hand, MEM is a mask reconstruction task, so the original representation of the modes is obscured. However, the refactoring of MEM is actually done using the representation of the same mode, so this does not mean that such a refactoring process is more difficult than our method. Our method is to achieve cross-modal reconstruction, and the model needs to overcome the direct distribution gap of different modes, which is often more difficult than the reconstruction of the original mode. To solve this problem, we put cross-modal refactoring behind the cross-modal alignment phase to achieve better refactoring. This effective combination of multimodal alignment and reconstruction helps us dig deeper into the relationship between ECG and report to learn effective representations with more clinical semantics.

Q4.2: A lack of baseline and comparison in the report generation task, comparison with MEIT, and discussion about GPT-2.

A4.2: We conducted more experiments in the report generation tasks, including comparison with MEIT, and the results are shown in the Table below.

Although GPT-2 was proposed in 2019, it is now still widely used in clinical report generation as we discussed in our paper. Ref. [A] (2023) and Ref. [B] (2024) have verified the great performance of GPT-2 on medical report generation. In addition, compared to LLM such as LLaMA, GPT-2 needs much fewer computing resources. And our experimental results on PTB-XL compared with new baselines with LLMs showed that our method with GPT-2 obtained the best performance. Therefore, using GPT-2 as the text decoder is not outdated.

Following to A3.3: c. The RME-module with random masking designed in our DERI is used to enhance the global feature of the ECG signals. Compared with random dropout used in MERL, which removes a fraction of the neurons (units) or edges in a network layer during training, our RME-module uses two different random masks to mask the local representation of part of the ECG and introduces an attentional mechanism to extract global features from the masked ECG representation. By aligning these two independent masked global representations, our model can more efficiently extract the global features of the ECG for further cross-modal alignment and reconstruction.

Comprehensive experiments on various downstream datasets are conducted to evaluate the proposed DERI method and experimental results have verified the great performance of our method.

Q3.4: For instance, two of the losses used in the paper are almost identical to the ones used in MERL (CLIP loss and mask loss); the paper merely describes them in a different way. The report generation method is also quite similar to many multimodal approaches, such as the BLIP method, and does not represent true innovation. Moreover, these papers have not been cited.

A3.4: On the one hand, CLIP loss has shown its great performance in multi-modal learning, such as Ref. [A], Ref. [B], and Ref. [C]. It is widely used for cross-modal representation learning, and this is the reason why we use this loss in our DERI. On the other hand, the main innovations of our method, which can be seen in A3.3, do not focus on the innovation of the loss function. BLIP also provides an effective method to generate text based on visual encoding, which is also widely used in clinical report generation such as R2Gen [D] and HERGen [E], we have cited these two great work in our articles.

[A] Hafner M, Katsantoni M, Köster T, et al. CLIP and complementary methods[J]. Nature Reviews Methods Primers, 2021, 1(1): 1-23.

[B] Zhang, R., Guo, Z., Zhang, W., Li, K., Miao, X., Cui, B., ... & Li, H. (2022). Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8552-8562).

[C] Fan, L., Krishnan, D., Isola, P., Katabi, D., & Tian, Y. (2024). Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36.

[D] Chen, Z., Song, Y., Chang, T. H., & Wan, X. (2020). Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16056.

[F] Wang, F., Du, S., & Yu, L. (2024). HERGen: Elevating Radiology Report Generation with Longitudinal Data. arXiv preprint arXiv:2407.15158.

We hope our response has addressed all concerns. We would greatly appreciate any further constructive comments or discussions.

We appreciate your review and the positive comments regarding our paper. We would like to respond to your comments as follows.

Q3.1: The paper shows a lack of understanding of related work, with many previous related articles not cited or discussed. The following articles [1-4] need to be added and discussed in the paper (compare their advantages, disadvantages, and innovations in the paper).

A3.1: Thanks for your valuable comment. We have discussed these related works below:

Ref [1] aims to use an instruction prompt to generate reports based on the ECG signal input with LLMs. However, in our work, our DERI framework is not designed only for report generation, but also for ECG classification. It is used to learn effective representation with more clinical cardiac information from ECG signals. Although we also conduct report generation, it is just one of the ways to verify the representation of ECG learned. In addition, the experiments are conducted on 4090 GPUs with less computing resources. To better compare our method and the MEIT framework in Ref [1], we add report generations on the PTB-XL dataset, which are not used in our pre-training process. The compared results are shown in the following table below with Ref [4].

Ref [2] proposes a method called METS to simply calculate the cosine similarity between the ECG embedding and the report embedding for multi-modal ECG-Text SSL pre-training. However, this is too simple to learn the cross-modal representation of ECG and reports. Actually, it’s the previous works of the MERL with simpler methods with lower results. MERL is the work further improved by the authors of Ref [2], so we did not focus on this paper when we selected the baseline method.

Ref [3] is proposed to conduct ECG zero-shot classification based on LLMs and RAG and then obtain a great performance on ECG classification. However, this method needs to construct a vector database first for retrieval and the performance depends on the quality of the database. This RAG method has high storage requirements and high computing costs. In addition, although this method is also a multimodal method, it does not use the diagnostic report corresponding to the ECG signal, and cannot extract the high-level semantics in the report as efficiently as our DERI method. However, the use of some baseline feature engineering for ECG signals in this method also gives us inspiration for further work. To better compare our method with this method, we compared the classification performance on PTB-XL, as shown in the table below:

Ref [4] proposes ECG-Chat, which is a great work for multi-modal ECG learning. We didn’t discuss it since it was released on Arxiv less than 2 months before our work was submitted to ICLR. It combines the ECG encoder and classification results to construct instructions for LLM. In addition to the ECG signal, it also uses the electrical health record and other information for DSPy and GraphRAG. Compared to the original report, ECG-Chat can generate structured reports that include medical history diagnoses and recommendations. Therefore, to compare our DERI with ECG-Chat, we compared the result of the generated report task on the PTB-XL dataset with ECG-Chat. The additional experimental results with these baselines on ECG report generation. It should be noted that the results of baselines are referred to in the original paper.

We appreciate your review and the positive comments regarding our paper. We would like to respond to your comments as follows.

Q1.1: The paper needs rephrasing to improve clarity and readability, especially in the methodology section.

A1.1: We will improve the clarity and readability of our paper, check and correct all grammatical errors, and improve where the description is vague. We use the expression "provide clinical semantics visually" because clinical reporting information can directly provide clinical semantics for ECG interpretation. "visually" is used to show the meaning of "directly", and we will revise it in our revised manuscript.

Details and references for the text encoder and masking/reconstruction will also be provided. Specifically, for the encoders used for ECG and reports, we adopt a random initialized 1D ResNet18 and the pre-trained MedCPT Query Encoder. MedCPT Query Encoder can generate embeddings of biomedical texts for semantic search (dense retrieval), which is suitable for our work. For cross-modal mutual reconstruction, we introduce two transformer decoders to reconstruct the representation of one modality with the other modality (use ECG embedding to reconstruct report embedding and use report embedding to generate ECG embedding). Detailed parameter settings for the decoder are provided in Section 4.2 lines 312-314. For random masking in RME-module, given a representation $x \in R^{b,n,c}$, we first generate a random Gaussian noise $m \in R^{b,n}$. The noise of each sample is sorted in ascending order to get the index of each sample. We choose the $k$ (decided by masking ratio) element indexes with the highest noise and the least noise as the masked feature indexes respectively, to ensure that the two masked representations are different.

The loss function provided in equations 1 to 5 in our method adopts the CLIP loss (I), which has been widely used in multi-modal contrastive learning. These equations are used to clearly illustrate how we conduct cross-modal representation alignment.

For Page 4 sec 3.2, cross-modal decoders are used to reconstruct the representation of each modal with the other modal, in other words, we reconstruct ECG encoding and text encoding as shown in Figure 1. Linear projectors $P_{e}$ and $P_{t}$ are used to map them into an alignment space of the same dimension for obtaining mixed-modal representation.

Page 7 lines (360-362) are rephrased as “The experimental results of linear probing are provided in Table 1. The ‘Random Init’ in Table 1 represents using the model structure of our proposed DERI to obtain the ECG-specific mix encoding without pre-training. The model is trained on the downstream dataset in a fully supervised manner for ECG classification.”

We also improve the captions of figures and tables in our paper for better clarity.

Q1.2: The training approach does not apply to unlabeled ECG in the usual context and necessitates the availability of accurate diagnostic reports by a cardiologist.

A1.2: Diagnostic reports are used in the pre-training stage of our method. We proposed a novel framework containing ECG-Report Multiple Alignment and Cross-Modal Mutual Reconstruction to enable the model to learn a more effective representation of the ECG signal. The reports provide a clinical description of the ECG signal, which contains direct high-level semantics. After pre-training, our model can effectively learn the representation of ECG signals which contain high-level semantic insights without diagnostic reports and labels. This is also why DERI has obtained significant improvement over the baselines on public datasets for ECG classification. Experiments on report generation also show that our DERI can learn more high-level semantic information than MERL, which only aligns ECG features with report features in a relatively shallow way.

Q1.3: Does incorporating textual reports in pre-training have an element of supervision?

A1.3: We believe that in a sense the inclusion of text reports in pre-training does have some element of supervision. The text reports provide additional semantic information to the ECG signal. The clinical information contained in each report actually guides the ECG signal as it represents the professional interpretation of the physician. This means that the model captures specific semantics by aligning the ECG signals with the corresponding reports, and this alignment process is equivalent to some form of supervised learning as the textual content helps the model to identify key features in the signals. However, the doctor's diagnostic report is not exactly equivalent to the category labeling of the ECG signal but is more of an interpretation of the ECG waveform. However specific diseases tend to have multiple waveform states and different diseases can have the same waveform. Therefore we do not consider the introduction of diagnostic reports for representation learning to be supervised learning.

We appreciate your review and the positive comments regarding our paper. We would like to respond to your comments as follows.

Q2.1: In contrast to other modalities such as chest X-rays and pathological diagnoses, electrocardiogram reports have been mainly produced mechanically by diagnostic equipment for many years. Therefore, this study is more likely to be learning of waveform data and its correct labels, rather than two-modal learning of waveform data and its interpretation using natural language. The scope of this study may be narrower than the general interest of the ICLR main conference.

A2.1: Our work proposes to learn an effective representation of ECG signals with the help of diagnostic reports. During the pre-training process, we conducted Deep ECG-report Interaction to train the model to learn effective representation with more clinical cardiac information. After that, the model can be used to learn a representation of ECG signals without report. ECG classification and report generation are the downstream tasks we conduct to verify the ability of our DERI to learn ECG representation. Therefore, our method is a total representation learning model that fits perfectly with the scope of the International Conference on Learning Representations.

Q2.2: Is it possible to disclose what the electrocardiogram reports used as training data in this study are specifically like? Was it verified how diverse the content of these reports is in terms of natural language?

A2.2: Actually, we have provided some example samples of the diagnostic reports used as training data in the supplement, including short reports, medium reports, and long reports, which are also shown below:

Short report: sinus bradycardia. prolonged qt interval. borderline ecg.\

Medium report: sinus rhythm. poor r wave progression - probable normal variant. inferior st-t changes may be due to myocardial ischemia.

Long report: sinus bradycardia. prolonged qt interval. possible anterior infarct - age undetermined. lateral t wave changes may be due to myocardial ischemia. abnormal ecg.

To verify how diverse the content of these reports is in terms of natural language, we conduct a statistical analysis on the reports of MIMIC-ECG, which calculated the counts of each word on all reports, and rare word count, TTR and Herdan’s C of each report. Since there are 771,693 reports and the average sentence has 14 words, we consider rare words that appear no more than 0.1% of the total number of reports (771). Based on the rare word count, we calculate the TTR and Herdan’s C of each report as below:

$TTR = \frac{Rare Word Count}{Report Word Count}$

$TTR = \frac{log(Rare Word Count)}{log(Report Word Count)}$

Here, TTR is the ratio of the number of rare words (types) to the total number of words (order), reflecting the diversity of the text. A higher TTR indicates higher lexical diversity and less lexical repetition. Herdan's C reflects the lexical diversity of a text by calculating the logarithmic ratio of the number of rare words in the text to the total number of words. The results are shown below:

We also find that there are 818 different words used in these reports while 486 words are regarded as rare words. Moreover, there are 40 words that appear only once in all reports while 110 words that appear less than 10 times in all reports. However, our distribution shift experiments show that our DERI method will not be limited by the scope and the bias of the report used in pre-training since it performs well for samples without reports.

We hope our response has addressed all concerns. We would greatly appreciate any further constructive comments or discussions.