Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs

This paper has two main shortcomings. The first is that the handling of ECG-entity alignment appears somewhat rough. The masking of the ECG signals is done randomly, which means that it is possible for the part of the signal corresponding to an entity was coincidentally masked. This could negatively affect the performance of knowledge alignment. I believe this is an obvious potential issue, but it seems that the authors have not addressed it in any way. The second issue is that the formatting of the paper is somewhat disorganized, and it appears that in an effort to compress the paper to within 10 pages, the authors removed a lot of content. This makes the technical section seem incomplete, and I had to rely on the figures to guess some implementation details. If this paper is accepted, I hope the authors can restore this content and place some of the experiments in the supplementary materials.

• It is unclear why lead masking is clinically useful in the way it is performed in the paper since not all combinations of leads are measured.
• The method is an improvement over MERL but it the results over MERL (especially in Table 1) needs discussion since they seems incremental.
• It is unclear why alignment with the entities helps the model since this information is practically already in the ECG report. This seems to be the main novelty of the method.

1. The rationale and advantages of the "lead-aware ECG encoder" should be further elaborated. The basic idea is to treat signals from different leads as separate tokens and to apply different position embeddings to these leads. While this approach can help learn and distinguish information from various leads, it may also complicate network training due to the increased introduction of raw but noisy signals as input. Therefore, it is essential to further clarify and validate the rationale and advantages of this approach.

2. One of the main contributions is the claim regarding the ability to handle missing leads in deployment. Although the proposed lead random masking strategy is reasonable, its effectiveness and advantages should be further validated. The baseline compared (MERL with zero mean filling) is suboptimal, and it is expected that this baseline would demonstrate inferior performance. It is challenging to assert that the proposed framework achieves its goals based solely on the limited experimental results presented. A better comparison might be MERL with random masking during training.

3. The proposed framework utilizes a fixed classifier for classification. Therefore, it remains unclear whether the claim of "zero-shot classification" is accurate. Moreover, the comparison with the previous state-of-the-art (SOTA) MERL may also be questionable, as MERL adopts a CLIP-based approach for final classification.

4. The entire framework appears to be an ad-hoc combination of several existing components, making the advancements in this field unclear. The authors should further clarify their technical contributions or insights, in addition to presenting the positive experimental results.

The primary difference between K-MERL and the previous MERL (Liu et al., 2024) approach is the extraction of entities from cardiac pre-text; however, this alone does not merit a high score in terms of originality and novelty.

Additionally, the experiments used to validate the hypotheses largely replicate those conducted in the MERL (Liu et al., 2024) paper, lacking significant variation or new insights. For instance, since text reports are harder to obtain compared to raw ECG data, it would be meaningful to conduct experiments to determine how many ECG-text pairs are necessary for this approach to be effective. Additionally, if utilizing diagnoses based on ICD codes yields better results than extracting cardiac entities from the ECG text reports, it would reduce the significance of using text reports. Instead, it would suggest a novel approach that combines ECG data with diagnostic history as a new modality.

Furthermore, while the authors claim that K-MERL enables learning of multimodal ECG representations, the text reports in the MIMIC-ECG dataset are essentially rule-based text diagnoses initially generated by the ECG equipment provider and subsequently reviewed by medical professionals. Thus, these ECG text reports are more a structured representation of ECG features than an entirely new modality (e.g., blood tests, vital signs, or medical images) distinct from the ECG modality itself.

• My biggest concern is that IMO the paper is not properly contextualized according to the existing literature. Methods like MedKLIP, KAD, or MAVL are briefly mentioned in line 106 and no further explanation is provided about what makes K-MERL stand out compared with them. I am finding the proposed framework mirrors KAD's framework without adding any novelty to the Multi-Modal framework.

• Authors claim that they design a novel "lead-specific tokenization". (Line 63) I do not see any differences between the way they embed lead information compared with other studies such as ST-MEM paper (Also used as a benchmark), which also includes this kind of lead embedding in its framework.

• Even the amount of baselines used during the evaluation is significant, most of them (all except one) are just trained on single-modality data. In addition, Most of those are not ECG-specific methods but image-processing ones. I am missing some relevant baselines such as PCLR [1], MAEFE [2], or DEAPS [3]. (See references).

[1] Nathaniel Diamant, Erik Reinertsen, Steven Song, Aaron D. Aguirre, Collin M. Stultz, and Puneet Batra. Patient contrastive learning: A performant, expressive, and practical approach to electro- cardiogram modeling. PLOS Computational Biology, 18(2):1–16, 02 2022. doi: 10.1371/journal. pcbi.1009862. URL https://doi.org/10.1371/journal.pcbi.1009862.

[2] Huaicheng Zhang, Wenhan Liu, Jiguang Shi, Sheng Chang, Hao Wang, Jin He, and Qijun Huang. Maefe: Masked autoencoders family of electrocardiogram for self-supervised pretraining and transfer learning. IEEE Transactions on Instrumentation and Measurement, 72:1–15, 2023. doi: 10.1109/TIM.2022.3228267.

[3] Adrian Atienza, Jakob Bardram, and Sadasivan Puthusserypady. Contrastive learning is not optimal for quasiperiodic time series. In Kate Larson (ed.), Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 3661–3668. International Joint Con- ferences on Artificial Intelligence Organization, 8 2024. doi: 10.24963/ijcai.2024/405. URL https://doi.org/10.24963/ijcai.2024/405. Main Track.

1. Contribution Clarity and Experimental Validation:

   • The paper highlights improvements over MERL, specifically in alignment between ECGs and free-text reports and a method considering spatio-temporal aspect of ECGs. However, there are no experiments demonstrating superior alignment in K-MERL compared to MERL, nor is there an ablation study showing that using LAMA for cardiac entity extraction results in less noise.
   • The spatio-temporal aspect is not convincingly validated. While partial lead input experiments are presented, they are insufficient. Comparisons with other spatio-temporal methods are missing.

2. Presentation Issues:

   • The token size $p$ is not mentioned in the tokenization section.
   • Figure 7 lacks clear labeling between parts (a) and (b).
   • Figures are poorly aligned with the text, placed too closely together, and not self-contained, particularly Table 2.
   • The inclusion of baseline performance in Figure 7 for ablation results is unnecessary and could have been presented more effectively in a table.
   • The notation for loss calculation is problematic, as $L$ is the mini-batch size and $N$ the total dataset size, which seems inappropriate.
   • If I have understood correctly, the formula in Section 3.2 under Lead-specific Spatial Positional Embedding, [$lead_{1}$ + W$e_{i}^{l}$[$p_{1}$], ..., $lead_{1}$ + W$e_{i}^{l}$[$p_{M}$], ..., $lead_{12}$ + W$e_{i}^{l}$[$p_{1}$], ... , $lead_{12}$ + W$e_{i}^{l}$[$p_{M}$]] should be updated to: [$lead_{1}$ + W$e_{i}^{1}$[$p_{1}$], ..., $lead_{1}$ + W$e_{i}^{1}$[$p_{M}$], ..., $lead_{12}$ + W$e_{i}^{12}$[$p_{1}$], ... , $lead_{12}$ + W$e_{i}^{12}$[$p_{M}$]]

3. Prompt statement:

   • The prompt statement selection lacks justification, unlike the detailed description provided in MERL.

4. Missing Performance Metrics:

   • The paper does not show results for full fine-tuning or partial fine-tuning.
   • Training and inference times compared to MERL are not reported, which is essential for understanding the practical implications of the proposed method.