Large-scale Training of Foundation Models for Wearable Biosignals

We thank the reviewer for their kind mention of our manuscript's strengths and pointing out a set of important modifications. We have responded to the reviewer’s comment below, and have addressed them accordingly in the paper. We believe the reviewer’s comments have positively improved our manuscript.

It would be great to include a figure representing the model showing encoder, MLP head and other implementation details.

We thank the reviewer for pointing this out. We have now added a high-level visualization of our pre-training framework in the new Appendix Fig. 3.

What was the reason for choosing the InfoNCE loss vs. e.g., the normalized temperature-scaled cross entropy loss (NT-Xent) from SimCLR?

This is a good point and we apologize for the confusion. As far as we understand, the term InfoNCE loss and NT-Xent are often used interchangeably for a similar loss function, for example, the SimCLR paper (“A Simple Framework for Contrastive Learning of Visual Representations”, Chen et al,., 2020) says “This loss has been used in previous work (Sohn, 2016; Wu et al., 2018; Oord et al., 2018); for convenience, we term it NT-Xent (the normalized temperature-scaled cross entropy loss)” and in the CPC paper (“Representation Learning with Contrastive Predictive Coding”, van den Oord et al., 2018), the same loss was termed as InfoNCE. To further clarify this, we have now added NT-Xent term as well to where we define the loss, “We use InfoNCE (also known as NT-Xent) to maximize the mutual information”, and we would like to clarify that the contrastive component of our loss is similar to SimCLR.

Usually large batch sizes and more learning steps are beneficial in SSL, have you experimented with even bigger batch sizes than 256?

The reviewer raises a very good point. We agree with the reviewer that larger batch sizes have been shown to be beneficial in SSL, for instance in the original SimCLR paper (Chen et al,., 2020). However, several follow up works have shown that adding momentum training and/or regularization and/or different objective functions could alleviate the need for larger batch sizes: examples are BYOL (“Bootstrap your own latent: A new approach to self-supervised Learning”, Grill et al., 2020) and Barlow Twins (“Barlow Twins: Self-Supervised Learning via Redundancy Reduction”, Zbontar et al., 2021), to name two. More importantly, a recent work has shown that it is possible to even train SimCLR on ImageNet with smaller batch sizes without an important drop in performance (“Towards Democratizing Joint-Embedding Self-Supervised Learning”, Bordes et al., 2023). To address this we have now added this to the new Appendix A.1: “Regarding batch size in our pre-training, while early contrastive self-supervised learning works have shown that larger batch size is necessary to improve performance (Chen et al., 2020; 2021), follow up works removed this dependency by changing the pre-training (Grill et al., 2020; Zbontar et al., 2021), and a recent work has shown that it is possible to train SimCLR on ImageNet with smaller batch sizes without an important drop in performance (Bordes et al., 2023). We also did not see meaningful change in performance with larger batch sizes in our early experiments.”

Could you clarify the reason for selecting this specific encoder and specific embedding sizes, have you experimented with other models?

We appreciate the reviewer for their careful comment. We picked the EfficientNet model architecture given its efficacy for model performance w.r.t number of parameters. However, to address the reviewer’s comment, we did a new Ablation 5.2.4 and a new Table 4 by comparing the performance of our default 1D-EfficientNet model with our variation of 1D-ResNet and 1D-ViT to enable comparison with alternative model architectures and model sizes: “For comparisons, we used smooth effective rank as a crude proxy of general downstream performance, and observed that different encoder architectures, with different model sizes, can achieve relatively similar performance, demonstrating that the performance is not unique to the 1D-EfficientNet encoder architecture (Table 4). We observed that 1D-EfficientNet model yielded similar performance to these alternative encoders with significantly smaller number of parameters, which was one of the main reasons we picked 1D-EfficientNet as our default backbone encoder given less memory footprint when potentially running on a wearable device. An interesting future direction is scaling up the model size further, particularly with transformer-based models (e.g., 1D-ViT) given their scalability (Kaplan et al., 2020), and larger datasets to study its effect on downstream performance". Also regarding the embedding size, in our early experiments, we investigated larger and smaller embedding sizes and 256 offered a reasonable sweet point and we did not tune it further afterwards.

We appreciate the reviewer for supporting our manuscript, carefully detailing its strengths, and their thoughtful suggestions. We have responded to the reviewer’s comment below, and have addressed them accordingly in the paper. We believe the reviewer’s comments have positively improved our manuscript.

An area of potential exploration is the specific impact of the KoLeo regularization on the model's performance. Ablation studies that incrementally remove or vary the strength of KoLeo regularization could provide clarity on its role and efficacy. Such an analysis would help in understanding whether the regularization is crucial for the model's success, and to what extent it contributes to the overall performance. This is particularly important as the unique characteristics of biosignals may interact with regularization techniques differently compared to other domains.

We thank the reviewer for pointing this out. We have now performed an experiment where we dropped KoLeo regularization from our pre-training framework to study its importance in isolation. As detailed in the updated Table 3 and updated Appendix Table 12, we observed that removing KoLeo regularization from our pre-training resulted in a drop in performance, evaluated with smooth effective rank and downstream demographic variables’ prediction, for both PPG and ECG. This demonstrates the specific impact of KoLeo regularization and we have now added to the paper: “We observed that for both PPG and ECG modalities, removing KoLeo regularization from our objective function resulted in reduced evaluation metrics (Table 3 and Appendix Table 12), demonstrating its impact on the model’s performance”.

Transparency and completeness in research reporting are essential for reproducibility and peer review. Therefore, it is recommended that the authors include all results, particularly those referenced as "results not shown" within the main body of the paper or the appendix. Having access to these results would allow the scientific community to fully evaluate the findings, methodologies, and claims made within the paper. It would also enhance the credibility and utility of the work for those looking to build upon it.

This is an important point. We have now added the following to the manuscript, providing further information and evidence: 1) new Appendix Table 10, including downstream performance evaluation of PPG and ECG embeddings in predicting demographic variables when the number of pre-training segments in PPG is similar to that for ECG in Table 1, 2) new Appendix Table 11, including demographic variables’ performance evaluation calculated at segment-level granularity, where each segment contributes one and only one sample in the downstream training/evaluation.

The paper would benefit from a deeper dive into the effects of various augmentation techniques on the performance of the biosignal models. Unlike image data, where the impact of different augmentations is well-studied, the domain of biosignals remains relatively unexplored in this aspect. An appendix providing detailed results and analysis of how different augmentations influence the learning process would be invaluable. This could include which augmentations contribute most to model robustness or performance, and any augmentation-specific phenomena observed with biosignals. Given the novelty of the field, such insights could be highly influential for future research in biosignal analysis.

We appreciate the reviewer for this thoughtful comment. We have now added details about the selection of our augmentation functions/probabilities in the new Appendix section A.1 and have done a new Ablation 5.2.5 with the new Appendix Fig. 7 regarding the effect of single augmentation functions in our pre-training framework: “We studied the effect of single augmentation functions for each modality. To do so, for both PPG and ECG, we kept each of the augmentation functions in isolation separately (with probability 1), and repeated our pre-training framework from scratch. We observed that PPG pre-training was more sensitive to the choice of single augmentation functions (more variance in performance), and channel permute and cut out were the most effective augmentation functions in isolation for PPG and ECG, respectively (Appendix Fig. 7). Future work can study more optimal ways to design modality-specific augmentation modules”.

We appreciate the reviewer’s feedback, their kind words about the strengths of our work, and their careful examination of our manuscript. We have responded to the reviewer’s comment below, and have addressed them accordingly in the paper. We believe the reviewer’s comments have positively improved our manuscript.

There are potential concerns on the technical methodology front. While this study is the product of extensive training and evaluation, much of the methodology draws from pre-existing studies. Although there are several SSL studies tailored for time-series data, particularly in the realm of biosignals in healthcare, the authors did not extensively compare different model architectures. Readers might be keen to discern whether biosignal SSL performance is more contingent upon scale or the model architecture itself.

We thank the reviewer for their thoughtful comment. First, we agree with the reviewer that our study is the product of "extensive training and evaluation". We would like to emphasize that “biosignal SSL” is largely under-explored at this scale compared to other domains of deep learning and there are no universally-agreed-upon model for wearable biosignals (PPG/ECG) that we can really compare our models to. However, to address the reviewer’s comment, we did a new Ablation 5.2.4 and new Table 4 by comparing the performance of our default 1D-EfficientNet model with our variation of 1D-ResNet and 1D-ViT to enable comparison with alternative model architectures and model sizes. As we mentioned in text with different model sizes, different model architectures can reach to reasonable accuracies addressing reviewer’s point about “SSL performance is more contingent upon scale or the model architecture itself”. We now also mention that: “For comparisons, we used smooth effective rank as a crude proxy of general downstream performance, and observed that different encoder architectures, with different model sizes, can achieve relatively similar performance, demonstrating that the performance is not unique to the 1D-EfficientNet encoder architecture (Table 4). We observed that 1D-EfficientNet model yielded similar performance to these alternative encoders with significantly smaller number of parameters, which was one of the main reasons we picked 1D-EfficientNet as our default backbone encoder given less memory footprint when potentially running on a wearable device. An interesting future direction is scaling up the model size further, particularly with transformer-based models (e.g., 1D-ViT) given their scalability (Kaplan et al., 2020), and larger datasets to study its effect on downstream performance".

While this study encompasses two modalities, it seems that the authors have considered them in isolation. Pretrained models have shown effectiveness across varied modalities like images and language. It would be beneficial for the authors to delve deeper into this aspect.

This is a great point. We respectfully believe that multi-modal pre-training, similar to CLIP (“Learning Transferable Visual Models From Natural Language Supervision”, Radford & Kim et al., 2021) is beyond the scope of our current work, and merits a separate detailed investigation on its own. However, we agree with the reviewer that this is an interesting future direction and we have now added this as a potential future direction in the updated Discussion section: "Another future area of investigation of investigation is multi-modal pre-training by: 1) using a multi-modal encoder that takes multiple modalities (e.g., PPG and ECG) as input, 2) or CLIP-style multi-modal pre-training by supervising one modality with another one (Radford et al., 2021), or training multiple encoders for multiple modalities simultaneously (Girdhar et al., 2023), using a contrastive loss".

We appreciate the reviewer’s feedback, their kind words about the strengths of our work, and their careful examination of our manuscript. We have responded to the reviewer’s comments below, and have addressed them accordingly in the paper. We believe the reviewer’s comments have positively improved our manuscript.

The paper could better address previous research. For example, it claims that previous works employed biosignals recorded in clinical or controlled experimental settings, however, both [1] and [2] used large-scale free-living datasets in the wild.

We thank the reviewer for pointing out an improvement regarding this. We would like to clarify that throughout the manuscript, we had meant to claim that our manuscript is the first study on “large-scale PPG and ECG data”, however, we agree with the reviewer that some of the mentioned work could have been better discussed in our paper, and we have made improvements to the Introduction and Related work sections to address the reviewer’s comment about the prior work and to include them. The changes for this comment are:

• We modified the Introduction section: “Self-supervised learning often does not require explicit labels, making it suitable to pre-train foundation models on unlabeled biosignals, and has been recently proven successful for health applications (Hallgrimsson et al., 2018; Cheng et al., 2020; Kostas et al., 2021; Sarkar & Etemad, 2022; Mohsenvand et al., 2020; Gopal et al., 2021; Kiyasseh et al., 2021; Mehari & Strodthoff, 2022; Wu et al., 2020; Spathis et al., 2021; Tang et al., 2021; Yuan et al., 2023; Lai et al., 2023). While there have been recent efforts to apply SSL on wearable accelerometer data in free-living conditions (Spathis et al., 2021; Yuan et al., 2023), other SSL work have mostly used biosignal modalities such as PPG, ECG, and electroencephalogram (EEG) collected in clinical or controlled experimental settings (Cheng et al., 2020; Kostas et al., 2021; Sarkar & Etemad, 2022; Mohsenvand et al., 2020; Gopal et al., 2021; Kiyasseh et al., 2021; Mehari & Strodthoff, 2022; Lai et al., 2023)“.

• We modified the Related work section: “Recent work have used self-supervised learning for wearable accelerometer signals in free-living conditions and large dataset (Spathis et al., 2021; Yuan et al., 2023)”.

• We would like to respectfully clarify that we still believe our claim in Abstract “To the best of our knowledge, this is the first study that builds foundation models using large-scale PPG and ECG data collected via wearable consumer devices — prior works have commonly used smaller-size datasets collected in clinical and experimental settings.” is still accurate in consideration of the works the reviewer has mentioned as they use accelerometer data, which is a different modality compared to PPG/ECG.

Regarding the augmentations, it is not clear whether assigned probabilities are found through hyper-parameter tuning or heuristics. I point the authors to this paper for further experimental decisions about the order and impact of these augmentations [3].

We thank the reviewer for this careful comment. We have now added details about the selection of our augmentation functions/probabilities in the new Appendix section A.1 and have done a new Ablation 5.2.5 with new Appendix Fig. 7 regarding the effect of single augmentation functions in our pre-training framework. We now mention that “we did not comprehensively tune our augmentation module probabilities, we started with a preliminary set of probabilities in our early experiments based on prior work (Cheng et al., 2020; Tang et al., 2021), for instance, cut out was shown to be the most effective augmentation in (Cheng et al., 2020) and we assigned a higher probability to it. We did not tune these probabilities afterwards and the only change made was to make the ECG augmentation probabilities stronger (2x probability) to allow for more mismatch between positive pair representations” and that “we studied the effect of single augmentation functions for each modality. To do so, for both PPG and ECG, we kept each of the augmentation functions in isolation separately (with probability 1), and repeated our pre-training framework from scratch. We observed that PPG pre-training was more sensitive to the choice of single augmentation functions (more variance in performance), and channel permute and cut out were the most effective augmentation functions in isolation for PPG and ECG, respectively (Appendix Fig. 7). Future work can study more optimal ways to design modality-specific augmentation modules”.