Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals

Dear reviewer hfvK, we appreciate your time and insightful feedback. Especially, thank you for evaluating our work as “novel and effective”, and “well written”. We greatly appreciate your positive comments!

Reproducibility and randomness control

We totally agree that reproducibility is often a concern in the physiological signal field, and plan to publicize all related code soon with the exact settings. Thank you for your suggestion about the randomness control. While we have provided the model performance different across different random seeds under the same dataset splits in Appendix Table 5, we agree that the data splits are another important source of randomness. However, because we performed many experiments across a diverse range of datasets, we believe the diversity in experiments could compensate for the data splits; and thus did not conduct additional experiments due to time constraints. We will make sure to note this in the limitation of the work.

Architecture capacity analysis and additional ablations

Thank you for your question regarding the architecture capacity and additional ablations!

To understand the size of the model, we compare the total number of parameters (Params) and FLOPs of our approach and baselines.

We can see that, bioFAME is very parameter-efficient due to its fix-size frequency filter design. With the same depth (4), heads (8), and dimensionality (64), bioFAME contains only ~40% parameters of the transformer baseline PatchTST. The parameter size of bioFAME also stands competitive than many CNN-based architectures, with only TS-TCC smaller than it.

To gain a better understanding of the throughput of the model, we used the fvcore package to compute the FLOPs of the models over a batch of SleepEDF data (batch size = 64). We found that the FLOPs of bioFAME is significantly lower than the transformer baseline PatchTST (<30%); yet greater than CNN-based architectures.

To understand the sensitivity of our architecture to hyperparameters, we also conducted additional ablation experiments to examine how the performance would change with respect to the changes in numbers of layers and the latent dimensionality. We observed that increasing the latent dimensionality could further improve the performance; while increasing the network depth gives little performance gains.

Dear reviewer,

Thank you once again for your insightful feedback. We have carefully revised our manuscript in line with your suggestions and believe we have comprehensively addressed your concerns. Could you kindly confirm whether the revisions align with your expectations? We are available to engage in further discussion or answer any additional questions you may have.

Thanks!

Other multi-modal contrastive frameworks as baselines

We thank the reviewer for their suggestion. To the best of our knowledge, no existing multimodal contrastive framework demonstrated effective performance on multimodal physiological signals. Existing work [1] even suggests that training on multiple different physiological signals gives worse performance, due to the large variation across different signals. We believe that (i) the lack of such baselines highlights the contribution of our work, as we are the first work that could effectively use many modalities (EEG, EMG, EOG, Resp) for pretraining; (ii) it might be out of our scope to implement such baselines.

[1] Zhang, Xiang, et al. "Self-supervised contrastive pre-training for time series via time-frequency consistency." Advances in Neural Information Processing Systems 35 (2022): 3988-4003.

The pros and cons of channel-independent learning

We are grateful for your insightful comments on the computational efficiency and global information representation in channel-independent learning. We agree that the usage of channel independence is a balance between transferability and capacity. To address the capacity concern, bioFAME incorporates a lightweight encoder (Encoder 2) in its architecture. We still selected channel independence to ensure transferability, but the lightweight encoder during pretrained helped the information across different channels and modalities mix and integrate. We note that, without channel independence, many of our experiments cannot be conducted due to the shifts and mismatch of the total amount of channels.

Details of experimental settings during fine-tuning stage

Thank you for your question about the detailed experimental settings, we will make sure to revise the text to clarify such experimental details! For the amount of labels used during fine-tuning stage, we presented the information in Appendix Table 7. We conducted full-scale fine-tuning instead of linear probing, and applied such fine-tuning pipeline to all baselines, because we believe it is a more common practice for transformer backbones [1], and is also more effective for contrastive learning methods [2].

[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[2] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

Dear reviewer BHTD, we appreciate your time and thoughtful feedback. Especially, thank you for evaluating our work as “interesting and practical”, having “extensive evaluation”, and recognizing our “robustness tests”. We greatly appreciate your positive comments!

Novelty of our approach, and why we emphasize pretraining on biosignals

Thank you very much for pointing us to works in the sensing community! We will make sure to include and discuss them in the revised text. Also, thank you for recognizing the novelty of our transformer encoder, MAE paradigm, and attention-based fusion mechanism.

Although the channel-independence technique itself isn’t novel, we would like to emphasize that our work aims to present a multimodal pretraining strategy that could (i) effectively combine information across different modalities including Electroencephalogram (EEG), Electrooculography (EOG), Electromyography (EMG), and Respiration rates (Resp) at the same time during pretraining, and (ii) could be flexibly applied on many other (unseen) modalities (e.g. Electromechanics). To the best of our knowledge, we are the first work that demonstrates that using information from different physiological modalities could improve the performance on downstream tasks.

We agree that the proposed method is generally applicable to various time-series sensing modalities, and consider it as an advantage of our work. However, we think the challenges, and the enabled applications of our approach are unique in the biosignal field:

1. Firstly, different physiological signals are recorded using different types of sensors placed at various locations on the body. The diversity in sensor technology and placement differentiates them as unique modalities. The signals also differ significantly in terms of their signal characteristics, sampling rate, and effective frequency bands.

2. Secondly, each signal type is affected differently by various types of noise and artifacts, causing large distributional shifts within each modality. In practice, the available combination of those signals also varies, causing large distributional shifts across modalities. The challenges in mapping, aligning, and analyzing data under such distributional shifts are central to research in physiological signal processing.

For the novelty of our approach, aside from the motivation that we clarified above, we would like to emphasize that our approach is technically distinguished from other works from two perspectives:

1. The most innovative aspect of bioFAME is its use of a fixed-size frequency filter. Inspired by FNO, our approach is specifically designed for biosignals and is significantly different from methods used in vision. The approaches in other works rely on frequency kernel of size $C^{N \times D}$, where N is the number of patches and D is the dimensionality of each patch. Thus, it cannot be directly transferred on data of different sizes, while our method, which learns queryable frequency filters of size $C^{K \times D}$, is adept at handling biosignals of varying lengths (e.g. 178 for Epilepsy, 5120 for FD-B) without performance degradation.

2. Our model is also different from Fourier-based approaches that were previously used in physiological signals and the sensing community. Our approach does not rely on explicit frequency-space encoder or frequency-space decoder, and does not use reconstruction loss in the frequency space. We found our method more flexible, parameter-efficient, and also easier to implement than other approaches.

We hope the novelty of the proposed approach isn’t shadowed due to the channel-independence technique we used to improve transferability. Such techniques are carefully selected and combined together to ensure that the final architecture is transferable and flexible!

Dear reviewer,

Thank you once again for your insightful feedback. We have carefully revised our manuscript in line with your suggestions and believe we have comprehensively addressed your concerns. Could you kindly confirm whether the revisions align with your expectations? We are available to engage in further discussion or answer any additional questions you may have.

Thanks!

Dear reviewer YFSy, we appreciate your time and insightful feedback. Especially, thank you for evaluating our work as “novel”, “well-motivated and evaluated”, and has “substantial improvements that are impressive”. We greatly appreciate your positive comments!

Additional experiments to further support the claims

We believe your comment “models are only pretrained on sleep data, … might reflect the nature of sleep data” is very insightful and spot on. We do believe that the performance of our model would be greatly affected by its pretraining dataset, similar as foundation models in many fields [1, 2, 3].

However, we believe that this would not dim the effectiveness of our approach. To validate that our pretraining strategy outperforms other strategies under diverse pretraining datasets, we conducted two additional experiments: we pretrain bioFAME, and a competitive baseline PatchTST on (i) FDA dataset; (ii) TUH EEG dataset, and transfer the learned representation on new datasets.

The below table shows the collected results, where bioFAME (S) represents the SleepEDF pretrained model for benchmarking; bioFAME (FDA) represents the FDA pretrained model; and bioFAME (TUH) represents the TUH EEG pretrained model.

There are two major observations: (i) bioFAME, as a robust pretraining strategy, outperforms the PatchTST baseline given the same pretraining data in diverse situations. (ii) The representation quality of bioFAME highly correlates with the quality of pretraining dataset, and our choice of multimodal sleep dataset gives high-quality representation. Interestingly, pretraining on FDA dataset gives large performance gain on FDB downstream task; while pretraining on the TUH EEG corpus in general gives worse performance, probably due to limited training time (see details below).

Below are the more specific training & dataset details:

1. FDA dataset is another corpus of single-channel Electromechanics dataset. We divide the signal into chunks of length 5120, creating a training dataset of sample size 8,000, and pretrain the network on it for 200 epochs with learning rate 1e-3 using an Adam optimizer for both models.

2. TUH EEG is a large-scale abnormal EEG corpus. We preprocess the signal following a public GitHub repo https://github.com/AITRICS/EEG_real_time_seizure_detection, and then chunk the signal into samples of length 4000, creating a 20-channel unimodal (EEG) dataset of sample size 102,903. Due to the limited amount of time, we pretrain the network for 10 epochs with learning rate 1e-3 using an Adam optimizer for both models.

We identify that large amount of high-quality dataset for pretraining is one of the major challenges of the biosignal field, and hypothesize that sleep data, with its diverse modalities and long period of time available, is an appropriate testbed for our proposed method. We hope our approach, by combining information across modalities for pretraining, can further advocate and encourage the production of such datasets, and benefit the physiological research field.

[1] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[2] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

Dear reviewer GTFm, we appreciate your time and valuable comments. Especially, thank you for evaluating our experimental analysis as “extensive”, and recognizing the effectiveness of our proposed method.

In response to the major criticism about multimodal experimental settings and the confusion regarding “how is the multimodal approach applied” to “single-channel EMG recordings”, we must clarify that the goal of the paper is to present a multimodal pretraining strategy that could (i) effectively combine information across different modalities including Electroencephalogram (EEG), Electrooculography (EOG), Electromyography (EMG), and Respiration rates (Resp) at the same time during pretraining, and (ii) could be flexibly applied on many other (unseen) modalities (e.g. Electromechanics). This is fundamentally different from works like neuro2vec [1], which represents a robust strategy that pretraining and testing on the same physiological modality, and does not evaluate how much adding additional information during pretraining could further help with building robust representation. We hope our approach can fundamentally address the challenge and unify learning on many different physiological signals, which is a problem that has not been systematically tackled before in this community.

[1] Wu, Di, et al. "neuro2vec: Masked Fourier spectrum prediction for neurophysiological representation learning." arXiv preprint arXiv:2204.12440 (2022).

W1 - Novelty of our approach and additional discussion with existing methods

For the novelty of our approach, aside from the motivation that we clarified above, we would like to emphasize that our approach is technically distinguished from other works from two perspectives:

1. The most innovative aspect of bioFAME is its use of a fixed-size frequency filter. Inspired by FNO [1], our approach is specifically designed for biosignals and is significantly different from methods used in vision [2, 3, 4]. The approaches in vision rely on frequency kernel of size $C^{N \times D}$, where N is the number of patches and D is the dimensionality of each patch. Thus, it cannot be directly transferred on data of different sizes, while our method, which learns queryable frequency filters of size $C^{K \times D}$, is adept at handling biosignals of varying lengths (e.g. 178 for Epilepsy, 5120 for FD-B) without performance degradation.

2. Our model is also different from Fourier-based approaches that were previously used in physiological signals [5]. Our approach does not rely on explicit frequency-space encoder or frequency-space decoder, and does not use reconstruction loss in the frequency space. We found our method more flexible, parameter-efficient, and also easier to implement than other approaches.

We thank the reviewer for the comment one more time. We will revise the manuscript to extend the discussion of our approach and its differences with existing methods. We will also make sure to cite and discuss papers [3, 5], as [1, 2, 4] are already discussed in the related works and method preliminaries.

[1] Li et al. Fourier neural operator for parametric partial differential equations. In ICLR, 2021.

[2] Liu et al. The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training. In AAAI, 2023.

[3] Li et al. Architecture-Agnostic Masked Image Modeling - From ViT back to CNN. In ICML, 2023.

[4] Xie et al. Masked Frequency Modeling for Self-Supervised Visual Pre-Training. In ICLR, 2023.

[5] Wu et al. neuro2vec: Masked Fourier Spectrum Prediction for Neurophysiological Representation Learning. In arXiv, 2022.

Dear reviewer,

Thank you once again for your insightful feedback. We have carefully revised our manuscript in line with your suggestions and believe we have comprehensively addressed your concerns. Could you kindly confirm whether the revisions align with your expectations? We are available to engage in further discussion or answer any additional questions you may have.

Thanks!

Thanks for the detailed and comprehensive responses, and sorry for the late reply at the end of the rebuttal period! I appreciate the technical contributions and well-organized writing of this work. However, the authors' responses have not addressed all my concerns, and I list them as follows:

• (W1, W4, Q1) The novelty is my main concern. Firstly, I knew that these works (e.g., works [1-4] mentioned in the authors' responses) I mentioned were proposed in different fields, and this work aims to tackle the pre-training problem of bio-signals in multi-modalities. However, as a new method for the AI community, it is necessary to provide strong motivations and reasons if the new method directly utilizes existing techniques to tackle special problems in downstream applications. Otherwise, Otherwise, I can arrange and combine some related techniques to make another new paper. I suggest the authors clarify these points in the main text (better to use empirical analysis and comparison). Secondly, the relationship and difference between bioFAME and neuro2vec [5] mentioned in the response are not convincing to me. And, I suggest the authors add this discussion in the related works (or some sections in the appendix).

• (W2, W5) These issues were well addressed and added to the revised version. A minor issue is that the FLOPs of bioFAME are relatively high, which will be a limitation of the proposed network architecture.

• (W3, W5, Q2) These issues were almost addressed. A minor issue is the performance gains of bioFAME are not significant on Epilepsy and SleepEOG. Is there any explanation? Since the performance gain of bioFAME comes from both the network architecture and the pre-training, it is better to clarify whether bioFAME can consistently achieve superior performances on different scenarios, or the limitations should be discussed and explained.

Overall, I decided to keep my current rating based on the above issues.

bioFAME is effective with diversified pretraining datasets

Reviewer YFSy pointed out that the contribution of our approach might be limited by the specific selection of pretraining dataset. To address this concern, we conducted two new experiments evaluating bioFAME’s performance under diversified pretraining datasets. Additional empirical results suggest that our approach, outperforming robust transformer baseline, is a powerful methodology for extracting biosignal representation regardless of the type of pretraining datasets.

More details about experimental setup

Additionally, we will go through the paper to further improve clarity in response to the reviewers’ comments and questions. Specifically, we will ensure all experimental details and setup are included to help the readers reproduce our results. We will also public our codebase soon for others to explore.