BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals

Thank you for your detailed feedback. We respond to your comments below:

Regarding downstream MEG datasets

We further introduce an additional publicly available MEG dataset, MEG-MMI $1$ . The results (in balanced accuracy) are listed below:

It can be seen that BrainOmni still outperforms the baselines on the additional MEG dataset.

$1$ Liuzzi, Lucrezia, et al. "Magnetoencephalographic correlates of mood and reward dynamics in human adolescents." Cerebral cortex 32.15 (2022): 3318-3330.

Regarding MEG baselines

We would like to first clarify that all of our chosen baselines are not specifically designed for EEG but rather general-purpose architectures for multi-channel time-series data -- spanning CNN+Transformer (CNNTransformer), pure CNN (ContraWR and SPaRCNet), and separately spatial and temporal Transformer (STTransformer). Training them directly on MEG, therefore, provides a fair indication of how those paradigms perform in the MEG domain. We further include an additional model, FAMED $2$ , primarily designed for MEG signals. The results (in balanced accuracy) are shown in the following table:

Results show the superior performance of BrainOmni.

$2$ Hirano, Ryoji, et al. "Deep learning based automatic detection and dipole estimation of epileptic discharges in MEG: a multi-center study." Scientific Reports 14.1 (2024): 24574.

Regarding cross-subject 3-fold validation setup

In the downstream part, we evaluated a variety models on a variety of datasets. Since most datasets lack standard train/test splits (only TUAB and TUEV have) and some compared models report results using different or undisclosed splits (e.g., AD65 dataset used in $3$ ), we built a unified evaluation pipeline across all datasets and models to guarantee fully comparable results. The cross-subject 3-fold validation setup, where every individual appears once in test, can capture inter-subject variability better than a single fixed split, achieving a more comprehensive and robust assessment.

To verify that our performance is not an artifact of this protocol, we followed CbraMod’s original PhysioNet-MI dataset filtering (only use motor imagery sessions) and splitting (subject (1-70, 71-89, 90-109) for (train, valid, test)) procedure, repeated experiments with 5 random seeds, and report the averaged metrics of BrainOmni. Results are shown in the following table:

Comparing row 1 and 2, BrainOmni still outperform competing methods under the same splitting, demonstrating that the gains are not due to differences in data assignment.

Furthermore, we generated an alternative subject split, by reshuffling the subject list (with random seed 0) while maintaining the train/val/test ratio as $4$ . Then we reproduced CBraMod's downstream experiments strictly using their codebase and following their hyperparameters listed in $4$ . The results are listed in row 3 of the above table.

It can be observed that the baseline performance dropped significantly, demonstrating the sensitivity caused by inter-subject variability, and highlighting the necessity and robustness of our cross-subject 3-fold setup.

$3$ Yuan, Zhizhang, et al. "Brainwave: A brain signal foundation model for clinical applications." arXiv preprint arXiv:2402.10251 (2024).

$4$ Wang, Jiquan, et al. "CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding." The Thirteenth International Conference on Learning Representations (2025).

Regarding the number of latent source variables

We added an ablation study on the number of latent source variables $Q \in \{8, 16, 32\}$, with results listed below:

Overall, Q=8 performs worse than Q=16 and Q=32, while Q=16 and Q=32 show similar performance across three datasets. Combined with the trend in Figure 3$c$, this shows that too few source variables lead to poor reconstruction and limited downstream performance, whereas increasing Q beyond 16 brings diminishing improvements.

Regarding the channel dropout rate

The channel dropout serves as a structural regulariser, enabling BrainTokenizer to adapt to varying channel layouts and quantities, as well as equipping it with the capability to predict target position electrodes based on existing ones, which improves robustness to channel corruption or occlusion. The 25% value was chosen based on experience. We did not perform a full hyperparameter search on this value due to computational constraints, but we expect that tuning this parameter could further improve results.

Regarding the loss functions in BrainTokenizer

The BrainTokenizer training loss contains four components: time-domain loss, frequency-domain loss, Pearson correlation loss, and RVQ commitment loss. Given that the time-domain and commitment losses are essential for VQ-AE training, we add an ablation study for how frequency loss and PCC loss influence the reconstruction performance, with results listed in the following table.

With only time-domain and commitment loss, the decoder outputs collapsed into near‑constant signals. Introducing the frequency‑domain loss greatly improved spectral detail (lower MAE/AMP/PHASE), but led to spurious high‑frequency impulses. The PCC term mitigates these impulses by aligning overall waveform trends, yet on its own provides weaker spectral reconstruction. Finally, we chose to employ both loss functions simultaneously, suppressing the impulse while preserving as much frequency-domain detail as possible.

We sincerely appreciate your feedback and hope that our responses adequately address your concerns.

Thank you for your positive comments and thoughtful feedback. We response to your comments below:

Regarding the sequence length in channel compression step

The sequence length fed into the cross-attention layer is only C (the number of channels). The channel compression module does a cross-attention operation to project heterogeneous inputs into a unified latent source variable. The learnable query is shared across all windows, ensuring that channel compression remains consistent between different temporal windows. Local temporal feature is captured by the SEANet Encoder, and long-range spatiotemporal dependencies are subsequently modelled in Stage 2 by the Criss‑Cross Transformer.

Regarding the 3-fold CV strategy

Since most datasets lack standard train/test splits and some compared models report results using different or undisclosed splits, we chose to use the 3-fold cross-validation to ensure a fair comparison while also taking computational cost into account. Specifically, for datasets without an official split (i.e., datasets except TUAB and TUEV), we first split all subjects into three equally sized subsets A, B and C. Then three experiments are performed in which (train, validation, test) are assigned in turn to (A, B, C), (B, C, A) and (C, A, B). For TUAB and TUEV, which each provide an official train/eval split, we keep the eval set for all testing, split the official training subjects equally into three folds, and in each of the three runs use two folds for training and the remaining fold for validation.

Regarding the finetuning strategy

In downstream fine-tuning, the pretrained BrainOmni produce embedding in shape [B, Q, W, D] (B - batch size, Q - number of compressed channels, W - temporal window length, and D - embedding dimension). The embedding is firstly average-pooled over W, and then flattened to [B, Q*D]. The classification head is a two-layer MLP. The BrainTokenizer has 5M parameters.

As for the number of training epochs, we would like to clarify that rather than using the 30‑epoch checkpoint for testing, we evaluated model performance on the held‑out validation set at the end of each epoch and used the checkpoint that achieved the highest balanced accuracy on the validation set for final test evaluation. This strategy prevents smaller models from overfitting due to excessive training epochs.

Regarding the effect of joint training

Firstly, we would like to highlight that a key goal of BrainOmni is to scale across EEG and MEG modalities, especially given MEG's complexity and limited data availability. Although joint training on both EEG and MEG data naturally increases the total amount of pretraining data, it is not guaranteed to help. In fact, unifying these two modalities in a single framework is non-trivial. Due to their distinct signal characteristics, spatial-temporal resolutions, and data formats, naïvely combining EEG and MEG may even hurt performance, as the model could divert capacity to modality-specific patterns, compromising optimal learning for either modality. Despite these challenges, BrainOmni successfully supports joint training across EEG and MEG: results in Table 5 show that EMEG pretraining improves downstream performance on both modalities.

For further verification, we ran an ablation experiment in which we subsampled the combined EMEG pretraining set to match the size of the EEG‑only and MEG‑only corpora. The results are listed below:

Results above show that (i) given roughly the same amount of pretraining data, joint training using EEG and MEG helps improve the performance (comparing rows 1&2 to rows 3-5); (ii) an increase in the total amount of pretraining data does not necessarily yield improved performance (rows 3-5).

Regarding the preprocessing

The power-spectral-density-based bad-channel detection is implemented as follows:

Input: raw_data, threshold=10
1. Compute per‑channel PSD over the full recording:
     PSD ← compute_psd(raw_data).data
2. Stabilise and log‑transform:
     L ← log(PSD + 1e‑16)
3. Compute pairwise distances between channel spectra:
     for i in 1…C, j in 1…C:
         dist[i,j] ← L[i] – L[j]
4. Mean distance per channel:
     m[i] ← mean(j, dist[i,j])
5. Identify outliers via IQR:
     Q1 ← 25th_percentile(m), Q3 ← 75th_percentile(m)
     IQR ← Q3 – Q1
     upper ← Q3 + threshold·IQR
     lower ← Q1 – threshold·IQR
     bad_channels ← { ch_i | m[i] > upper or m[i] < lower }

The reference inconsistency is caused by the fact that different data recording setups may use different physical reference electrodes. To address this, we first compute a global average reference by taking the mean across all channels, and then subtract this channel-average signal from each channel waveform, effectively aligning all channels to the same virtual reference. This protocol is applied regardless of whether a reference channel is unavailable or the signals have already been referenced, and it is performed both at the recording level and separately for each signal type.

Thank you again for your positive comments and constructive suggestions. We will further clarify the above points in our manuscript. We hope our response resolves your concerns.

Thank you for your insightful feedback and for acknowledging our research direction and experimental validation. We response to your comments below:

Regarding the performance compared to specialised models

We appreciate you for raising this important point, and agree on the importance of explicitly addressing these points. Paper $1$ designs channel combinations and feature extraction methods based on the distribution of features across frequency bands and brain regions, demonstrating the outstanding performance of machine learning approaches in this task. That said, we would like to highlight a key difference in experimental setup and data splitting strategy, which makes direct comparison of results between BrainOmni and the cited classical methods inappropriate for our context.

The referenced baseline ( $1$ ) employs a random split approach on the PD31 dataset, which may result in the same subjects appearing in both training and testing sets, thereby compromising subject independence. Instead, our experiments utilise a cross-subject split, ensuring no subject-level information leakage, which is a more challenging and practically relevant evaluation setting. Indeed, paper $1$ explicitly acknowledges concerns regarding data leakage associated with random splits in the "UNM‑based results" section.

Since the implementation of [1] is not open-sourced and the specific data split used is not disclosed, a direct comparison under a unified setup is not feasible. To estimate the impact of the split strategy, we conducted an additional experiment using a random split on PD31, following a similar protocol. This resulted in a substantially higher accuracy of 92% for BrainOmni, highlighting that the choice of data split can significantly affect performance.

We fully agree with your suggestion that it is important to provide a balanced discussion of BrainOmni’s strengths and limitations in relation to specialised baselines. This also reflects a broader challenge in current AI research: general-purpose models often trade off some task-specific performance in exchange for scalability, flexibility, and cross-domain transferability.

BrainOmni is designed to serve as a unified foundation model across heterogeneous non-invasive neurophysiological electrophysiological modalities (specifically EEG and MEG). It supports cross‑device and cross‑subject generalisation, making it suitable for practical applications where domain shifts are inevitable. While it may not always outperform domain-specific baselines tailored to one dataset or one task, it offers broader coverage and generalisability across datasets, tasks, and modalities — a key advantage for scalable and reusable neuro-AI systems. We will add this discussion to our manuscript.

$1$ Aljalal, Majid, et al. "Detection of Parkinson’s disease from EEG signals using discrete wavelet transform, different entropy measures, and machine learning techniques." Scientific Reports 12.1 (2022): 22547.

Regarding the questions

Answer to Q1: We visualise the reconstruction results of unseen devices without finetuning.

Answer to Q2: We didn't separately finetune the BrainTokenizer because it relies on a learned VQ codebook whose discrete embeddings encode meaningful spatiotemporal tokens, and updating it on limited downstream data risks corrupting those codebook semantics. Instead, following standard practice, we freeze the entire Tokeniser (including the VQ quantiser) during downstream adaptation and finetune only the Transformer layers.

Thank you again for your positive comments and constructive suggestions. We hope our response resolves your concerns.

Thank you for the comments and for acknowledging that BrainOmni is a step towards a general-purpose model. We would like to first clarify several important misunderstandings presented in the summary:

• BrainOmni is designed for EEG and MEG signals, not "functional MRI", as stated in the title.
• All datasets used in our study are listed in Appendices D and E, without including "NSD, HCP, or Movie10K fMRI datasets".
• As shown in Table 1, our downstream tasks include clinical (e.g., Alzheimer’s, Parkinson’s), abnormal (e.g., TUAB), event (e.g., TUEV), emotion (FACED), and motor imagery tasks. None of them fall under the category of "cognitive decoding", and no "visual-language fMRI tasks" were involved.

We then respond to each comments below:

Weakness1: No clear discussion of training efficiency, hardware requirements, or inference speed, which is crucial for scaling.

Thank you for pointing this out. Our pre-training was performed on 2*8 NVIDIA A100 GPUs. Stage 1 (BrainTokenizer) took approximately 11 hours. Stage 2 required about 14 hours for the tiny model and 18 hours for the base model. Time cost for downstream fine‑tuning depends on dataset size. On a single A100, it took roughly 60 samples/sec for training and 90 samples/sec for inference. We will add this information to our manuscript.

Weakness2: Attention map visualisations are provided, but no in-depth explanation of specific learned cognitive representations or their neuroscientific implications.

We discussed the neuroscience background and how BrainTokenizer parallels the classical source estimation process in Appendix B. Specifically, the cross‑attention between raw EMEG signals and the sensor layout implements the backwards (sensor -> source) and forward (source -> sensor) mappings, so that each latent "source" functionally mirrors a cortical current source. We presented the topographic attention map in Appendix J. It shows that each source variable in our model has automatically learned a hierarchical structure, indicating that each source variable focuses on extracting features from specific scalp regions. This supports our hypothesis that BrainOmni learns meaningful, brain‑relevant representations rather than generic signal features.

Weakness3: While diverse in modality, most benchmarks are still based on visual-language fMRI tasks.

As stated in the clarification above, this study focuses on EEG and MEG signals and does not involve visual-language fMRI tasks.

Weakness4: It’s unclear how well BrainOmni generalizes across individuals, or whether subject-specific adaptation is needed.

We would like to highlight that all downstream evaluations are conducted in a strict cross‑subject setup, with no overlap of individuals between train, validation, and test splits. BrainOmni shows superior performance under this setting (Table 2), demonstrating its robust generalisation to unseen subjects.

Question1: What are the computational costs for training and inference? Is BrainOmni feasible for clinical use or only research labs?

Please refer to the point 1 for the computational device and time costs. As for applicability, it's worth noting that BrainOmni-tiny has only ~10 million parameters, which is lightweight and suitable for clinical settings as well.

Question2: How does the model generalize across subjects with different brain anatomy or signal quality?

Firstly, as stated in Section 2.2, we designed a Sensor Encoder which encodes the physical coordinates of each channel, enabling BrainOmni to cope with different device layouts across datasets and subjects. Secondly, we pretrained BrainOmni on over 2600 hours of EMEG data from diverse sources, improving generalisation across differing anatomies and noise levels. Preprocessing steps (bad channel detection, channel re‑referencing, normalisation, etc.) also help to improve the robustness regarding signal quality.

As stated in response to weakness 4, BrainOmni shows superior cross‑subject generalisation performance to unseen subjects. As discussed in Section 4.2, BrainOmni also shows robust cross-device generalisation to unseen devices.

Question3: Can you further interpret the learned representations or link them to known cognitive networks?

Please refer to the response to Weakness 2 for a detailed explanation for the neuroscientific implications of learned representations.

Question4: Can BrainOmni be extended to handle EEG, MEG, or structural MRI for truly multi-modal brain modelling?

As stated in the clarification, BrainOmni is designed as a foundation model for unified EEG and MEG signals. Table 5 demonstrates the effectiveness of this unified pretraining: joint E/MEG pretraining consistently outperforms single-modality pretraining across all datasets. This serves as a crucial first step toward broader multi-modal modelling, which is inherently challenging due to the structural and functional differences across neural signal types. We look forward to extending BrainOmni to include other neural modalities, as discussed in Appendix A.

We thank you again for the comments and we hope our rebuttal fully resolves your concerns.

We thank Reviewer 73i4 for the comments. We believe we have addressed your concerns in detail in the rebuttal and hope our responses fully resolve the issues raised.