Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners

[R4-1]: Lack of originality / Limited originality: The framework combines multiple loss functions (ETS, ETM, MLM, MEM), none of which are original contributions by the authors. Furthermore, the coefficients used to balance these losses are not systematically explored or justified. This makes the work appear more like an engineering effort rather than a novel machine learning contribution, suggesting it may be better suited for a clinical application journal rather than ICML … The novelty of the contributions raises questions about the framework's suitability for a high-impact venue like ICML.

We appreciate your feedback and would like to kindly refer to the key contributions of our works as in R2-1. While the individual loss components are inspired by prior works, our contribution lies in how they are specifically adapted, integrated for the underexplored masked ECG-text multimodal autoencoder setting. We acknowledge that we did not systematically explore loss weighting coefficients, which can be a valuable direction for future work. However, given the context of investigating ECG-text multimodal pre-training by proposing a novel generalized contrastive masked autoencoder framework that has already demonstrated to surpass existing benchmarks, we kindly believe your conclusion about “an engineering effort” only based on the lack loss coefficients may overlook our approach.

We also emphasize the growing importance of ECG-based modeling in both machine learning and clinical applications. As you pointed, “ECG analysis is an important task for medical application”, ECGs are non-invasive, widely used physiological signals, and clinical reports provide contextual information that complements the waveform, but the combination between them remains underexplored in dynamic SSLs. Therefore, we hopefully aim for ICML (Primary Area: Applications->Health / Medicine) as a suitable venue to foster this impactful direction.

[R4-2]: Heavy reliance on CLIP-style ETS loss and Flan-T5: In Table 6, removing the ETS loss (which essentially reduces the model without using vanilla CLIP loss) results in a significant drop in performance. This indicates that the model heavily depends on the CLIP loss, while other components contribute relatively little. Similarly, replacing T5 with Med-CPT in Table 7 also leads to a performance decline. These observations raise concerns about whether the improvements stem primarily from leveraging more powerful off-the-shelf text encoders rather than from the proposed framework itself.

We would like to respectfully correct a potential misinterpretation in your examples. First, in our attempt to conduct ablation studies, it is ordinary when our proposed components have better performance effects than when not using them. In Table 6, while removing the ETS loss leads to a performance drop, removing other components (e.g., Flan-T5 or N3S) also results in comparable degradation. In Table 7, Flan-T5 is the encoder we proposed in our framework, and is slightly better than Med-CPT (the best in MERL). We can also see that even with a weaker encoder (e.g., BERT), the remaining architecture still outperforms the last row in Table 6 (same BERT usage). Therefore, we sincerely believe the improvements reflect the synergistic design of the overall framework, rather than over-dependence on a specific component.

[R4-3]: Inconsistent ablation studies on individual losses In Appendix Tables 12 and 13, the authors ablate ETM and MLM losses but fail to provide a consistent and systematic comparison of each individual loss component. This lack of clarity makes it difficult to discern which parts of the framework actually contribute to performance improvements. Additionally, the absence of detailed ablation studies undermines the ability to assess the incremental value of each loss function.

We kindly note that our Table 12 evaluates the effect of ETM, while Table 13 evaluates the joint effect of MLM and MEM purposely, which together form our overall generative reconstruction objective.

Furthermore, rather than the need of detailed individual loss effect, we believe that focusing more on our main ablation studies which examine the incorporation of core components are more important in our framework, with the effects of N3S, ETS, Flan-T5 (Table 6), variations of text encoders (Table 7), the scalability in the ECG encoder (Table 8).

---

We hope these clarifications address your concerns. We thank you for your review and hope that you will consider raising our score, as we believe our work offers valuable contributions to the ICML community.

[R3-1]: "Were baselines pretrained from scratch on the same data as D-BETA, were their model weights taken as is and used for fine-tuning, or were results taken directly from their respective papers?"

Regarding baseline comparison, we used the results reported in the original baseline papers. They can use different datasets for the pre-training stage but we keep the fair comparisons by strictly following the baselines’ released data splits, preprocessing and downstream configurations.

[R3-2]: "What is the authors’ justification for using a language model pretrained on natural text (that is then fine-tuned)? Could the approach potentially be improved by using a text encoder pretrained on cardiovascular reports specifically, perhaps negating the need to fine-tune the text encoder?"

Thank you for your thoughtful question. Flan-T5 is pre-trained on large data of natural language that possibly captures rich medical context from domain-specific website (e.g., cardiovascular disease related forums), which indicates it is helpful. Additionally, trained with huge corpus, Flan-T5 shows strong adaptability across unseen tasks. If further fine-tuning Flan-T5 in D-BETA, it could perform better on specific cardiovascular reports. As shown in Table 7, we can see that Flan-T5 after fine-tuned during D-BETA pre-training stage is nearly 1% better than fine-tuned Med-CPT, a biomedical-pretrained model used in MERL (as a SOTA approach). That said, we agree that using a Flan-T5 text encoder first pre-trained specifically on cardiovascular reports might also offer additional benefits.

[R3-3]: "Can the authors provide an ablation study on the negative sampling? I see a discussion of why N3S might be helpful in Section A.2, but do not see numerical results backing this up."

We kindly note that the effectiveness of N3S is already reported in Table 6, where removing N3S causes a 2% drop in zero-shot performance. Additionally, in Appendix A.2, we report that the ETM accuracy without N3S stagnates at ~75%, while with N3S it exceeds 96%. We believe these results support N3S’s impact of semantically-aware negative sampling.

[R3-4]: "Do the authors intend to release code and model weights? This will be important to ensure reproducibility."

Yes, as noted in the Impact Statement part (Lines 442-451), we will publicly release the pretrained models and code upon acceptance. We are happily committed to ensuring reproducibility and enabling future research.

[R3-5]: "Technical novelty is limited (though the additions are clearly helpful): the sigmoid loss is borrowed from SigLIP – per the authors’ admission - and the combination of contrastive and generative approaches has been seen in C-MELT [1]."

We respectfully understand the reviewer’s perspective and would like to refer to R2-3 and R2-1 for further discussions on our related justifications and contributions. We do not directly follow the presentation in their paper but make suitable adjustments in our modeling context and our core contributions go beyond using ETS loss. We kindly note that [1] is not discussed here due to the sensitive policy of the conference.

[R3-6]: “A few additional citations to ECG-text foundation models could be included [2-4]. I wouldn’t say these are “essential” in that their omission/inclusion changes my stance on the paper, but it is helpful to inform the reader that this is a growing space with many relevant approaches.”

Thank you for your kind suggestions. [2] is a comprehensive review paper that synthesizes various aspects of ECG foundation including background, existing datasets, common modeling approaches, and various practical applications. [3] also focuses on the ECG foundation model but proposes to use LLMs to enrich ECG report text with medical knowledge then perform a signal-text-label contrastive learning. Meanwhile, [4] does not pretrain ECG-text directly but interestingly views ECG signals as a language, with QRS complexes as words and rhythms as sentences. Alongside the discussed works in R1-2, we believe these reflect the growing momentum in this space as you kindly noted.

---

We appreciate your time to review our work. At this end, we have addressed your comments and we would be grateful if you are satisfied with our responses and could acknowledge our rebuttal.

[R2-1]: "The overall task design seems to largely build upon existing multimodal joint modeling methods and appears more like an aggregation of several tasks, lacking sufficient novelty."

While our framework builds upon several established tasks, to the best of our knowledge, we are the first to design and investigate the contrastive masked autoencoder ECG-text pre-training. As novelty is subjective, we focus on the contribution and significance of our work to the ICML (especially with the primary area of Applications->Health/Medicine) community:

• We propose D-BETA that specially uses a transformer-based ECG encoder and the Flan-T5 model (overlooked in ECG-clinical community), together with attention-based fusion modules and decoders.
• We propose and investigate the insight of discriminative ETS loss into masked ECG-Text autoencoder implementations (with ETM, MLM, MEM and arbitrary lead augmentation), enabling robust multimodal representation learning (unexplored in the literature).
• We are the first to introduce the N3S technique to address data unexplored redundancy in the MIMIC-IV ECG dataset, improving the quality of negative samples and boosting model performance.
• We conduct extensive experiments across zero-shot, linear probing, and fully fine-tuned settings, demonstrating that our approach consistently outperforms strong baselines across more than 100 cardiac conditions.

We also verify the effectiveness of proposed components by conducting the diverse ablation studies and necessary appendix, as as acknowledged by the reviewers NfLP ("Experimental designs are generally sound"), ffzJ ("Experiments are thorough and show clear improvement over existing state-of-the-art") and TuyF (“providing a thorough analysis of its capabilities in practical applications”). Finally, we emphasize the practical implication of the ECG research field and sincerely believe that our work reflects a principled design and empirical effort rather than simple aggregation.

[R2-2]: "Although the paper cites a substantial number of related works, it might benefit from discussing more recent advances in multimodal fusion or self-supervised learning in the context of ECG data."

Thank you for the suggestion. This is closely related to R1-2 in which we discuss more recent works regarding multimodal ECG-Text fusion.

[R2-3]: "The paper does not include an experiment comparing ETS directly against a conventional softmax-based loss … can the authors further demonstrate the efficiency and superiority of the ETS loss over traditional methods?"

Thank you for the thoughtful point. Our ETS loss is inspired by the Sigmoid contrastive loss proposed in SigLIP[1], which has already demonstrated strong theoretical and empirical advantages over softmax-based losses. These “Sigmoid” benefits remain efficiency and superiority in our implementation since we kindly introduce the unaffected ETS heads with Dense layer and Tanh activation while simply setting t=1 and b=0 (as we design positive-negative balanced batches during training). Building on this foundation, we focus our main ablations on showing ETS’s benefit in our specific multimodal ECG-text setup.

[1] Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

[R2-4]: "In the loss module, the authors emphasize the efficiency advantages of the proposed approach. Could the authors elaborate on how the N3S strategy impacts the overall computational burden? Does it affect the model’s efficiency and scalability?"

We appreciate this point and acknowledge the importance of discussing the computational aspect of using N3S. First, the FAISS index is constructed once before training using precomputed text embeddings in the small Flan-T5 space. During training, we only perform efficient nearest-neighbor retrieval using this index and this action will introduce some tradoff. Importantly, FAISS is known for its impressive speed and scalability in large-scale vector search tasks, and in our case, produces comparable runtime overhead and still doesn’t affect the efficiency and scalability much. Empirically, on 1× NVIDIA A100-40GB, the one-time loading of the model and FAISS index takes ~1.2 seconds while the average (over 1000 samples) time using FAISS was approximately 0.00219 seconds (compared to 0.00002 seconds without it).

[R2-5]: "Have the authors reproduced and fairly compared related methods under a unified experimental setting to ensure a fair comparison of performance?"

Regarding baseline comparison, we used the results reported in the original baseline papers. In D-BETA, we always aim for the fair comparisons by strictly following the baselines’ released data splits, preprocessing and downstream configurations.

---

We sincerely appreciate your valuable feedback and have responded to your reviews and hopefully these meet your expectations and acknowledgement.

We thank the reviewer for their constructive feedback on our submission. We would like to address your comments below:

[R1-1]: “In some scenarios, the text info, especially from doctors, may be seen as a ground truth or label, rather than training data. Some discussion on this would be helpful, too.” … “In evaluating and comparing the proposed method to other baselines, whether all other baselines incorporate the text information in training … if some of the other methods do not have the text info available, the comparison may not be a fair one.”

Firstly, our model is not explicitly exposed to ground-truth labels during pre-training. The textual reports in the MIMIC-IV-ECG dataset are machine-generated and serve as various descriptive inputs, not as discrete predefined label annotations.

Furthermore, additional text description can be beneficial, but importantly, we presented the efficient way to leverage them in an underexplored multimodal setting, optimizing performance in the same downstream experiments with the baselines: (1) Comparing with strong ECG-only SSL methods to show multimodal modeling power. (2) Comparing MERL (ICML publication) in the same multimodal aspect.

Moreover, evaluated downstream tasks such as fine-tuned classification and identification use unseen ECG recordings, no text is involved; or complete zero-shot tasks (consistent with the latest close work like MERL). Therefore, we believe these comparisons are both fair and meaningful.

[R1-2]: “It would be helpful to expand the literature review here, as this is the key contribution of this paper.”

We acknowledge that discussing more details of recent related ECG-text works would be more helpful to our work. Therefore, we kindly synthesize more recent works available, and by this, we also want to highlight the rapidly growing developments and attention to this medical domain:

Firstly, [1] pretrains on a large private dataset using a standard contrastive SSL with ResNet-like ECG and BERT-based text encoders. Their evaluation is also limited to a few downstream tasks and diseases. [2] build on a similar contrastive modeling design but introduces prompt-based zero-shot inference. While insightful, their evaluation remains relatively narrow. ESI [3] opens another interesting angle when using a RAG pipeline to produce auto-generated detailed report data for the pretraining stage using multiple datasets (including MIMIC IV, PTB-XL, and Chapman). However, the contrastive modeling approach is still relatively similar (Bert-like text encoder, fully convolutional Convnext model, no MEM, ETM, or lead augmentation), and evaluation is relatively modest.

Recently, as ECG representation learning continues to evolve, a few works have begun to explore more practical, user-end applications. For example, [4,5] leverage instructed multimodal LLMs to deal with ECG report generation and clinical question answering. These methods largely benefit from a well-pretrained ECG encoder, which D-BETA also can be potentially adapted.

[1] Lalam, Sravan Kumar, et al. "Ecg representation learning with multi-modal ehr data." TMLR (2023).

[2] Liu, Che, et al. "Etp: Learning transferable ecg representations via ecg-text pre-training." ICASSP (2024).

[3] Yu, Han , et al. "Ecg semantic integrator (esi): A foundation ecg model pretrained with llm-enhanced cardiological text." (2024).

[4] Zhao, Yubao, et al. "ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis." (2024).

[5] Yang, Kai, et al. "ECG-LM: Understanding Electrocardiogram with a Large Language Model." HDS (2025).

[R1-3]: “Please explain when applying MAE to the ECG (multi-channel), how are the different channels masked”

As described in Section 3.1 (Lines 139-143), we apply random lead masking, where entire ECG channels are independently masked with a probability of 0.5. This lead-wise masking encourages the model to learn robust representations across varying input configurations.

[R1-4]: “Please explain explicitly the formation of the latent representation z as a fusion of the outputs from the ECG-specific and text-specific encoders. Overall, the paper would benefit from some theoretical analysis/heuristic on how cross-learning and fusion improve model performance.”

Our fusion module is designed as stacked cross-attention blocks, which allows interaction between multi-lead ECG signals and semantic text embeddings (Lines 175-180) before decoding. This produces essential joint embeddings for our learning objectives: for example, the ETM task requires a unified representation to determine whether an ECG-text pair is matched, which naturally depends on information from both modalities. Similarly, MLM and MEM benefit as one modality provides useful context to reconstruct the other (e.g., ECG noise or text ambiguity compensation). This fusion also facilitates future downstream tasks such as ECG report generation, where signal-based text generation is critical.