Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI

We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.

[W1] While the authors acknowledge the challenge of varying configurations in EEG data collection, particularly concerning electrode variations across different datasets, the manuscript does not provide a comprehensive solution to this challenge. Specifically, the Temporal & Spatial embedding section offers an ambiguous explanation regarding spatial embedding (SE). The method appears to encode channels based merely on their sequential order, which may not accurately represent the channel's functional significance or location in the brain. Given the heterogeneity in electrode configurations and positions across various datasets, it's imperative for the authors to elucidate how their approach effectively manages this inconsistency, and use qualitative results to demonstrate the model has learned different electrode configurations.

• Thanks for your valuable suggestion. First of all, our spatial embeddings are initialized according to the international 10-20 system (https://en.wikipedia.org/wiki/10%E2%80%9320_system_(EEG)) which was commonly followed by most existing EEG datasets. Though different datasets may have different configurations, they all utilize a subset of electrodes of the international 10-20 system, providing the possibility to solve this problem. Specifically, our solution is adding the corresponding spatial embedding according to the international 10-20 system. For example, both dataset A and dataset B have the electrode FP1, so the FP1 electrodes in the two datasets are added by the same spatial embedding. By this means, the spatial embeddings incorporate the spatial information into the model training, which is unremovable during pre-training. If we try to discard the spatial embeddings, the pre-training loss cannot converge because it does not have the ability to identify which masked electrodes to reconstruct. To demonstrate the effectiveness of the spatial embeddings, we provide the ablation study where the spatial embeddings are directly discarded during fine-tuning on downstream datasets as follows:

The results clearly demonstrate the importance of spatial embeddings in capturing spatial information. We have updated this experiment to the revised paper in Appendix L.

[W2] The paper draws parallels to another LEM, BIOT, developed by Yang et al., and even follows some experimental settings from the same. Given the apparent similarities, it remains unclear as to what drives the performance improvements of the proposed model — is it solely attributed to the increased model size, enhanced training data volume, or specific architectural designs? A more in-depth comparative discussion and analysis between the two models would be beneficial to ascertain the genuine contributions and innovations of the current work.

• Thanks for your valuable feedback. Our conclusion to your concern is that the well-designed large-scale self-supervised pre-training contributes most to our superior performance compared to BIOT. In Appendix J, we have conducted ablation to see whether the pre-training really works. It can be seen that without pre-training, the performance decreases steeply on the two datasets especially on TUEV.

[Q1] Will the pre-trained checkpoints be released for open-source development as well?

• Yes, we will release the pre-trained models as well as the codes.

We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.

[W1] Some methodological points require clarification, e.g. the impact and choice of windowing/tokenization hyperparameters (Q1), the learning and reuse of spatial embeddings (Q3), the potential overlap between pretraining and downstream datasets (Q5) and the sampling of examples during pretraining (Q8).

• We appreciate your concerns. In the following Questions section, we will address each aspect of your inquiry separately, providing a comprehensive response to ensure clarity and understanding.

[W2] I don’t think the results are clear enough to deduce what is claimed in the analysis of Figure 5, i.e. that the performance saturates for the base model at 500 h and for the large model at 2000 h. For instance, the Large models seem to continue learning over 2000 h on TUEV. Similarly, the Base model might not be saturated yet; the performance curve seems pretty noisy. Maybe repeating this analysis with a log-scale would be clearer (1h, 10h, 100h, 1000h, 2500h).

• Thanks for your valuable suggestion. We have updated Figure 5 with additional experiments on 1h, 10h, and 100h. Please refer to Figure 5 in the revised paper. As can be seen from the trend in Figure 5, the performance of the Base model consistently increases with the amount of data at 500 hours or less. Therefore, 500 hours of EEG data is enough for pre-training the Base model. For the Large model, the performance tends to saturation and achieve its optimum as the data size reaches 2,000 hours.

[W3] The use of the term “BCI” (e.g. in the title) is confusing as this typically refers to a subset of the tasks/datasets considered in this work. For instance, the term BCI is usually used to describe cases where there is an interface between brain activity and a computer that bypasses normal communication pathways. Under this definition, tasks such as pathology detection (TUAB) or event detection (TUEV) are not BCI tasks. I would recommend adapting the language of the manuscript to make this clearer.

• Thanks for your valuable opinion. A Brain-Computer Interface (BCI) can be defined as "a system that measures central nervous system (CNS) activity and converts it into artificial output that replaces, restores, enhances, supplements, or improves natural CNS output, thereby altering the ongoing interactions between the CNS and its external or internal environment" [1]. Based on this definition, we propose that tasks like TUAB and TUEV contribute to the first part of the BCI system's measurement process. This involves measuring neural activity and utilizing it as input for subsequent processes, including control, intervention therapy, and neural circuit remodeling. Therefore, we contend that tasks such as TUAB and TUEV fall within the scope of BCI tasks.

[Q1] Some of the hyperparameter choices for the windowing and tokenization steps are not clear to me. First, what were the window strides (in first paragraph of Section 2.1) used for the different datasets? Tables 3 and 4 report a “Data stride” value but I’m not sure whether that’s the same thing. Second, what is the impact of the selected patch size and patch stride on performance, i.e. are these choices important? Related to the point about how symmetric masking might be providing regularization that is useful for larger models, would a smaller window and/or stride help create more pretraining examples?

• We apologize for not clarifying some hyperparameters. For the window stride, it is exactly the same thing as the data stride in Table 3 and Table 4. As for the setting of patch size and patch stride, we set them to 1 second because 1 second is the smallest unit for many EEG tasks. The common time length for most EEG tasks is between 1 second and 10 seconds. For example, the time length of a sample for datasets we use in this paper is 10 seconds for TUAB, 5 seconds for TUEV, 1 second for SEED-V, and 2 seconds for MoBI. On the other hand, this setting is consistent with some baselines such as BIOT where the authors conduct experiments with different settings of patch size and overlapping length, and the results show that 1 second is a relatively good choice for EEG segments. For the setting of data stride, we set it to 4 seconds for pre-training based on the following considerations: (1) covering all training data. As the maximum sequence length is 256, the sample for 64-channel EEG signals is 4 seconds. All data can be visited if we set the data stride to 4 seconds (no overlapping). (2) boosting the training speed (reducing the number of samples). If we set the data stride to 1 second or less, the number of samples could be so large that it will take several times as long to complete the training.

We are deeply grateful to you for the thorough review and constructive comments, which have greatly assisted us in improving the quality and presentation of our manuscript. Please see below for the point-by-point responses to your comments.

[W1] Some papers have not been mentioned that work on heterogeneous datasets

• Thank you for your insightful suggestions. We appreciate your encouragement to broaden our study's scope. In response, we have integrated these recommendations into our revised paper, specifically in the related work section.

[W2] The choice to use mean squared error on phase values is strange to me. Due to their cyclical nature, very nearby phases, e.g., -pi+eps,pi-eps would get a large squared error that would also depend on whether one uses phases from 0 to 2pi or -pi to pi etc. So would make more sense tome to either always only use the minimum distance, so if one would put the phases on a unit circle the minimum distance on the circle, or maybe even regress fourier coefficients instead of amplitude/phase. I even wonder what would happen if one just does not put any loss on the phase prediction at all, only on the amplitudes that would be interesting to check as well.

• Thank you for your insightful comments on the selection of mean squared error for phase values. We acknowledge your concerns about the cyclical nature of phases and the potential biases associated with different ranges. We are currently making efforts to conduct additional experiments based on this suggestion. However, due to limited computational resources and time constraints, we may not be able to complete these experiments in time. Rerunning the neural tokenizer training and LaBraM pre-training (especially with a total of 2,500 hours of data) can take several weeks. If the experiments are completed, we will update the results as promptly as possible. We believe that our comprehensive analysis will help researchers better understand the merits of each method and their applicability in different scenarios. We sincerely appreciate your constructive feedback, which has led to a more in-depth exploration of our research. Your insights have enriched our paper and improved its overall quality.

[W3] Why bold lowest std in table 1 and 2, better remove that, is rather confusing

• Thank you for your suggestion, and we have made the necessary changes accordingly.

[W4] Font could be a bit bigger in Figure 1 and also parts of Girue 2 (e.g., channel names on bottom) Fig 2 the amplitude phase plot on top right is confusing to me. Symmetric in Figure 2 not symmetric

• Thank you for your valuable feedback. We have taken your suggestions into account and made the necessary adjustments to improve the clarity and readability of the figures.

[Q1] I assume Table 1/2 only compares to other self-supervised models? That should be written a bit more explicitly otherwise there are other papers worth citing

• Thanks for your valuable suggestion. We appreciate your attention to the details of the baselines presented in Tables 1 and 2. To provide a clearer understanding of these baselines, we present the following detailed descriptions:

1. SPaRCNet: This is a 1D-CNN-based model equipped with dense residual connections.

2. ContraWR: It converts biosignals into multi-channel spectrograms and then employs a 2D-CNN-based ResNet framework accompanied by contrastive learning.

3. CNN-Transformer: This hybrid model combines the strengths of CNNs and Transformers for enhanced performance.

4. FFCL: The Fine-tuning Framework for Classification (FFCL) integrates embeddings from both CNN and LSTM encoders for effective feature fusion.

5. ST-Transformer: This multi-level EEG transformer concurrently learns spatial and temporal features for improved performance.

6. BIOT: A framework based on the Transformer architecture, it takes spectral energy vectors extracted by FFT on the original signals as inputs. It can be pre-trained using either supervised or self-supervised learning techniques.

It is worth noting that these baselines represent state-of-the-art supervised (SPaRCNet, CNN-Transformer, FFCL, and ST-Transformer) and self-supervised (ContraWR and BIOT) methods in the recent literature. We have sincerely cited the mentioned paper in our revised manuscript, specifically in the introduction section. We hope this provides clarification and addresses your concerns.

[Q2] Are TUAB and TUEV disjoint recordingwise from TUAR TUEP TUSZ and TUSL?

• Yes, the TUAB and TUEV are indeed disjoint from TUAR, TUEP, TUSZ, and TUSL. Detailed descriptions of these datasets can be found in the provided URL link (https://isip.piconepress.com/projects/tuh_eeg/html/downloads.shtml).

Thanks for your comments. It seems you report per-10-second-sample balanced accuracy on TUAB and that is where the discrepancy comes from to other results, e.g., in https://www.sciencedirect.com/science/article/pii/S1053811920305073 (best recordingwise accuracy is around 86%, if one computes balanced accuracy for TCN in Figure 7, it is 85.3%). Please additionally report recordingwise accuracy by averaging the predictions of all 10 second samples that belong to the same recording.

Thank you for your prompt response. We appreciate your concern and are pleased to offer the recording-wise accuracy of TUAB by averaging the predictions of all 10-second samples belonging to the same recording: 84.08% (for the Base model), which is 2.68% higher than the balanced accuracy reported in our paper. Regarding the reason for not conducting a recording-wise comparison at the outset, it is crucial to recognize that the outcomes presented in our paper are derived under identical experimental configurations, facilitating horizontal comparisons. As you are aware, various dataset divisions and experimental arrangements can result in non-comparable findings. For instance, in the paper you mentioned [1], their basic settings differ a lot from ours:

1. The data division is very different. [1] performed 5-fold cross-validation without test set while we splitted the dataset with training, validation, and test set. It is noted that the test set is provided by the dataset itself, so we just divided the training patients into training and validation groups by 80% and 20%.

2. [1] discarded the first 60 seconds of every recording while we did not. This would bring a higher accuracy because they observed a large number of recording artifacts in this period.

3. [1] used a maximum of 20 min of every recording to avoid considerable feature generation and resampling times for exceptionally long recordings while we did not.

4. [1] resampled the EEG signals to 100 Hz while we resampled them to 200 Hz.

5. The length of a sample in [1] was 6 seconds while we utilized 10-second samples.

Although discrepancies may exist between recording-wise accuracy and balanced accuracy, we believe that both metrics offer a direct reflection of the model's performance and share a consistent change trend. Nevertheless, we appreciate your insight as it can enrich and enhance the credibility of our paper's conclusions.

Reference

[1] Gemein L A W, Schirrmeister R T, Chrabąszcz P, et al. Machine-learning-based diagnostics of EEG pathology[J]. NeuroImage, 2020, 220: 117021.