CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding

W5: Furthermore, when critically analyzing the model's usefulness based on the metrics—and considering literature from the field, such as O. Alkob et al. (2018) [1] and Combrisson and Jerbi (2015)—the metric values fall short of demonstrating real usefulness for BCI applications.

R: Thank you for your detailed and thoughtful feedback. We acknowledge the importance of contextualizing our work in light of existing literature, particularly the studies by O. Alkob et al. (2018) [1] and Combrisson and Jerbi (2015) [2]. Below, we address your concerns regarding the metrics and their implications for BCI applications:

1. Learning Generic Representation: As highlighted by O. Alkob et al. (2018) [1], the efficacy of neurofeedback treatments and broader BCI systems can vary significantly among individuals, emphasizing the importance of developing models that generalize well across subjects. Our work focuses on EEG representation learning, aiming to create a foundation model that captures generalized EEG features. This approach is distinct from building application-specific BCI systems but is highly relevant to improving downstream task performance. A robust foundation model has significant potential to enhance real-world BCI systems, aligning well with the goal of enabling adaptive and personalized BCI solutions.

2. Addressing Real-World Challenges: We fully recognize that real-world BCI systems face unique challenges, including signal noise, individual variability, and the need for real-time adaptability. These challenges were also noted by O. Alkob et al. (2018) [1] in their discussion of neurofeedback inefficacy. In terms of data preprocessing, we only applied the most basic bandpass filtering and notch filtering, minimizing preprocessing steps in the hope that our model can learn useful representations even in noisy environments. To address individual differences, we employed a strict subject-independent train/validation/test split across most of the downstream datasets to rigorously evaluate generalization. These approaches are designed to ensure that our work holds practical value for real-world BCI systems.

3. Bridging EEG Representation Learning and Practical Applications: While our primary focus is on EEG representation learning, we believe that this is a critical step towards enabling effective BCI systems. Our work explores the way for a One-For-All solution, holding the promise of addressing multiple tasks with a unified model, positioning them as a transformative tool for diverse EEG applications. This goal aligns closely with the overarching theme of ICLR, which emphasizes the importance of advancing learning representations to tackle real-world challenges effectively.

4. Metric Interpretation and Comparability: As noted by Combrisson and Jerbi (2015) [2], evaluating classification performance in small datasets or under subject-independent settings requires careful consideration of chance levels and statistical significance. In our study, we utilized metrics such as balanced accuracy, Cohen’s kappa, and weighted F1-score for multi-class classification, which provide a more comprehensive evaluation than raw accuracy, particularly under conditions of class imbalance and inter-subject variability. While raw accuracy may not meet traditional expectations in certain tasks, the inclusion of more nuanced metrics ensures a fair and reliable assessment of our model’s capabilities.

In summary, while our current results may not directly demonstrate the final utility of our model in real-world BCI applications, they underscore the potential of representation learning to enhance downstream BCI tasks and address the variability highlighted in neurofeedback and BCI research. We appreciate the reviewer’s insights.

[1] O. Alkoby, A. Abu-Rmileh, O. Shriki and D. Todder, "Can we predict who will respond to neurofeedback? A review of the inefficacy problem and existing predictors for successful EEG neurofeedback learning", Neuroscience, vol. 378, pp. 155-164, May 2018.

[2] Combrisson E, Jerbi K. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. Journal of neuroscience methods. 2015 Jul 30;250:126-36.

W7: Using the TEUV and TUAB datasets for abnormal detection and event-type classification while employing TEUG for pre-training leads to data leakage. This is indicated on the Temple University dataset's own webpage: "The TUH EEG Events Corpus (TUEV: v2.0.1): This corpus is a subset of TUEG."

R: Thank you for raising the important concern regarding data leakage between the TUEV, TUAB, and TUEG datasets. We would like to clarify the following points:

1. Purpose of Using TEUV and TUAB for Evaluation: The primary reason for using the TEUV and TUAB datasets for evaluation is to conduct a more comprehensive comparison with the state-of-the-art methods like BIOT and LaBraM. Both of these methods have discussed scenarios where the evaluation datasets overlap with the pre-training data. In our case, the pre-training process is completely unsupervised and does not involve any label information, which we believe effectively prevents full data leakage.

2. Additional Experiments to Address Data Leakage Concern: We fully understand your concern about potential data leakage. To further eliminate this doubt, we conducted an additional experiment in which we removed the TUAB and TUEV subsets from the TEUG dataset before pre-training CBraMod. After this step, we used the pre-trained model to validate its performance on the TUAB and TUEV subsets. This experiment ensures that no data overlap exists between the pre-training and evaluation phases and provides further confidence in the integrity of our results.

   	TUEV, 6-class
   Methods	Balanced Acc.	Coken’s Kappa	Weighted F1
   CBraMod (excluding TUEV)	0.6659 $\pm$ 0.0124	0.6744 $\pm$ 0.0121	0.8331 $\pm$ 0.0071
   CBraMod	0.6671 $\pm$ 0.0107	0.6772 $\pm$ 0.0096	0.8342 $\pm$ 0.0064

   	TUAB, 2-class
   Methods	Balanced Acc.	AUC-PR	AUROC
   CBraMod (excluding TUAB)	0.8249 $\pm$ 0.0025	0.9221 $\pm$ 0.0015	0.9156 $\pm$ 0.0017
   CBraMod	0.8289 $\pm$ 0.0022	0.9258 $\pm$ 0.0008	0.9227 $\pm$ 0.0011

   The experimental results show that, after thoroughly eliminating data leakage, the performance of our model only experiences fluctuations, but remains competitive. This indicates that our method successfully learns a generic EEG representation through pre-training, thereby enhancing the performance on datasets unseen during pre-training. The experiments are added in the revised version of our paper (Line 1269, Page 24), (Line 1310, Page 25).

We hope these clarifications and the additional experiment help address your concerns regarding data leakage. Thank you for your careful review, and we look forward to your feedback.

We thank the reviewer for the helpful analysis and constructive feedback.

W1: While the particular combination of methods is novel, it primarily combines existing ideas from other deep learning subfields. This limits the overall novelty of the work, especially given the lack of sufficient ablations to discern which of the numerous components is truly important. To be very clear, there is no problem using components from other fields; in my opinion, this is highly encouraged, but the study needs to go in-depth to understand what leads to compatibility and the "bridge construction."

R: Thank you for your valuable comment. Our core contributions, Criss-Cross Attention and Asymmetric Conditional Positional Encoding (ACPE), are specifically designed to leverage the unique structure of EEG data and address limitations positional encoding of traditional EEG foundation models.

1. Criss-Cross Attention:

   • Existing foundation models mostly adopt the full EEG modeling strategy, flattening EEG patches to a sequence and using general transformer to model the dependencies among all EEG patches together, similar to ViT[1] in the field of computer vision. It ignores the unique structural characteristics of EEG signals. EEG signals have different structural characteristics compared to image. It contain heterogeneous spatial and temporal dependencies, but images contain only spatial dependencies. And the dependencies among EEG patches within the same channel or time interval could be stronger than those among patches from different channels and time intervals, but such a prior assumption may not be valid in image.
   • The criss-cross attention mechanism was developed to better capture and utilize the structural relationships inherent in EEG data. It models heterogeneous spatial and temporal dependencies separately through two parallel attention mechanisms, spatial and temporal attentions. It is not just a general attention mechanism but one that is tailored to the specific characteristics of EEG signals.

2. Asymmetric Conditional Positional Encoding:

   • Current EEG foundation models use absolute position encoding based on electrode numbers as channel embeddings, as seen in methods like LaBraM[2]. However, EEG channels are not simply equivalent to electrode positions, as they are also influenced by factors such as reference schemes (e.g., ear reference, average reference, REST, and bipolar montages). Therefore, we believe that this traditional position encoding approach limits the adaptability of EEG foundation models to different downstream tasks.
   • To address this challenge, we proposed an alternative solution with our ACPE (Asymmetric Conditional Positional Encoding) inspired by CPE[3]. This method leverages a convolutional network to dynamically learn the spatial relationships between each patch and other patches, allowing the model to better adapt to varying channel configurations in downstream tasks. By doing so, we aim to enhance the flexibility and applicability of EEG foundation models across diverse EEG data formats.

In our revised manuscript, we have enhanced the discussion of these methods, particularly with respect to their motivation and design. Furthermore, in the original version, we already provided compelling evidence of the effectiveness of these innovations through experiments in the "Attention Mechanism Comparison" and "Positional Encoding Comparison" sections.

To further address concerns regarding the interpretability, we have added new interpretability analysis in the revised version (Line 1638-1750, Page 31-33), including (1) Topography Visualization, (2) Representation Visualizations on Downstream Datasets and (3) Visualization of Patch Relationships from each Criss-cross Transformer Layer.

• (1) Topography Visualization demonstrates that CBraMod effectively captures symmetric patterns between the left and right hemispheres from motor imagery data.
• (2) Representation Visualizations on Downstream Datasets demonstrates that our pre-trained model can learn representations with clustering effects tailored to downstream tasks.
• (3) Visualization of Patch Relationships from each Criss-cross Transformer Layer confirms that our criss-cross attention effectively models criss-cross dependencies.

We hope these clarifications, along with the new interpretability analysis, provide a clearer understanding of the novelty and significance of our work. Thank you once again for your feedback.

[1] An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.

[2] Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI//The Twelfth International Conference on Learning Representations.

[3] Conditional Positional Encodings for Vision Transformers//The Eleventh International Conference on Learning Representations.

W3: The choice of datasets for motor imagery, emotion, and sleep stage tasks seems arbitrary. For instance, in the case of motor imagery, the most widely used dataset is BNCI 2014 version 004, and the largest is from Stieger et al. (2021) to the best of my knowledge. However, the authors chose Physionet-MI.

R: Thank you for your comment. We appreciate your feedback on our dataset selection. Here are our responses:

1. Comprehensive Dataset Selection: Our evaluation involves up to 10 downstream tasks across 12 publicly available datasets, which, to the best of our knowledge, is unprecedented. Covering every well-known dataset for each downstream task is not feasible, and the datasets we selected are highly recognized within their respective fields. From the perspective of the authors, the choice of datasets should not be considered arbitrary simply because they do not align with the reviewer’s preferred datasets.

2. Additional Experiment for Validation: To further address the reviewer's concern, we conducted additional experiments on the BCIC-IV-2a dataset, as suggested by the reviewer. BCIC-IV-2a consists of 9 subject with two sessions. We use a strict subject-indenpendent train/validation/test strategy as other MI datasets in our paper. Subject 1-5, 6-7, 8-9 are used for training, validation, and test, respectively. The data preprocessing procedure is identical for all the methods. The results are shown in the follows:

   	BCIC-IV-2a, 4-class
   Methods	Balanced Acc.	Coken’s Kappa	Weighted F1
   EEGNet	0.4482 $\pm$ 0.0094	0.2693 $\pm$ 0.0121	0.4226 $\pm$ 0.0108
   EEGConformer	0.4696 $\pm$ 0.0106	0.2924 $\pm$ 0.0141	0.4533 $\pm$ 0.0128
   SPaRCNet	0.4635 $\pm$ 0.0117	0.2847 $\pm$ 0.0147	0.4432 $\pm$ 0.0126
   ContraWR	0.4678 $\pm$ 0.0125	0.2905 $\pm$ 0.0160	0.4413 $\pm$ 0.0142
   CNN-Transformer	0.4600 $\pm$ 0.0108	0.2800 $\pm$ 0.0148	0.4460 $\pm$ 0.0114
   FFCL	0.4470 $\pm$ 0.0143	0.2627 $\pm$ 0.0176	0.4238 $\pm$ 0.0139
   ST-Transformer	0.4575 $\pm$ 0.0145	0.2733 $\pm$ 0.0198	0.4471 $\pm$ 0.0142
   BIOT	0.4748 $\pm$ 0.0093	0.2997 $\pm$ 0.0139	0.4607 $\pm$ 0.0125
   LaBraM	0.4869 $\pm$ 0.0085	0.3159 $\pm$ 0.0154	0.4758 $\pm$ 0.0103
   CBraMod	0.5138 $\pm$ 0.0066	0.3518 $\pm$ 0.0094	0.4984 $\pm$ 0.0085

   It demonstrates that our approach continues to outperform existing methods on this dataset, further reinforcing the generalizability of our method. In the revised version, we added this experiment (Line 1329-1355, Page 25-26).

We hope this clarifies our approach and demonstrates the robustness of our methodology. Thank you again for your valuable insights.

We appreciate your continued engagement and detailed feedback. However, we would like to address some misunderstandings and provide additional context regarding the concerns raised:

1. Comparison with EEG-SimpleConv:
   We carefully reviewed the EEG-SimpleConv[1] paper and its associated code and found that the reported performance of 72.1 ± 7.3 accuracy under the leave-one-subject-out (LOSO) protocol relies heavily on several advanced preprocessing and training techniques referred to as "Key Ingredients," including Euclidean Alignment (EA), session statistics, Mixup, and subject-wise regularization. When these techniques are not employed, EEG-SimpleConv achieves significantly lower performance: 56.4 ± 9.0 accuracy (as reported in the original paper).

   In our work, we deliberately adopt minimal preprocessing to ensure a fair and standardized evaluation, which makes direct comparison to methods employing extensive preprocessing techniques, such as EEG-SimpleConv with Key Ingredients, inappropriate.

2. Fair Comparison with LOSO Protocol:
   To provide a more rigorous and fair evaluation, we conducted experiments under the same LOSO protocol used by EEG-SimpleConv, both with and without incorporating the "Key Ingredients." The results are summarized as follows:

   	BCIC-IV-2a, LOSO
   Methods	Balanced Acc.	Cohen’s Kappa	Weighted F1
   EEG-SimpleConv (w/o Key Ingredients)	0.5650 ± 0.0989	0.4201 ± 0.1319	0.5484 ± 0.1047
   CBraMod (w/o Key Ingredients)	0.5968 ± 0.0816	0.4558 ± 0.1088	0.5889 ± 0.0875
   EEG-SimpleConv (w/ Key Ingredients)	0.7221 ± 0.0768	0.5765 ± 0.1069	0.6977 ± 0.0918
   CBraMod (w/ Key Ingredients)	0.7405 ± 0.0635	0.5997 ± 0.0833	0.7195 ± 0.0682

   These results demonstrate that:

   • Our reproduction of EEG-SimpleConv matches the original paper’s findings.
   • CBraMod consistently outperforms EEG-SimpleConv under both settings, achieving higher balanced accuracy, Cohen’s Kappa, and weighted F1 scores while also demonstrating lower inter-subject variance, highlighting its robustness and effectiveness.

3. Motivation of EEG foundation models: The significance of our EEG foundation models lies in their ability to learn generalized representations from large-scale unlabeled data, enabling robust performance across a wide range of downstream tasks, not only motor imagery or emotion recognition. In fact, learning generalizable representations is one of the key pursuits emphasized by the ICLR conference. Unlike traditional methods that rely on task-specific feature engineering, prior knowledge, or extensive preprocessing techniques, these models provide a unified and stable framework, streamlining EEG analysis and reducing dependency on handcrafted solutions.

We hope this clarification adequately addresses your concerns and provides a clearer context for the experimental design and results presented in our work. We remain open to further discussion and thank you again for your thoughtful feedback. Regardless of whether this paper is accepted, we sincerely appreciate the reviewer's detailed and valuable feedback, particularly the insights on the motor imagery task. We hope our responses can help bridge the gap between the reviewer's and the authors' academic perspectives, working together to advance the field of EEG decoding.

[1] El Ouahidi Y, Gripon V, Pasdeloup B, et al. A Strong and Simple Deep Learning Baseline for BCI Motor Imagery decoding[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2024.

We are grateful for the reviewer's attentive analysis and helpful feedback.

W1: This work acknowledges that current EEG foundation models still face challenges in handling EEG data of varying formats. However, the solution proposed by CBraMod during the pretraining phase, which involves the selection of 19 common EEG channels, appears to be a simplistic approach that does not effectively address the aforementioned issues.

R: Thank you for your insightful comment regarding the challenge of handling EEG data with varying formats. We acknowledge that this remains a significant issue for EEG foundation models.

1. Limitation of existing EEG foundation model on adaptation: Current EEG foundation models, such as those employing Transformers, are indeed effective in handling the varying length of EEG signals due to their strong adaptive capabilities. However, one limitation of the Transformer architecture is its inability to learn the spatial relationships between EEG channels, which is crucial for handling variations in the EEG channel format across different downstream tasks. Typically, existing models address this limitation by using absolute position encoding based on electrode numbers as channel embeddings, as seen in methods like LaBraM. However, EEG channels are not simply equivalent to electrode positions, as they are also influenced by factors such as reference schemes (e.g., ear reference, average reference, REST, and bipolar montages). Therefore, we believe that this traditional position encoding approach limits the adaptability of EEG foundation models to different downstream tasks.

2. Our Proposed Method: To address this challenge, we proposed an alternative solution with our ACPE (Asymmetric Conditional Positional Encoding). This method leverages a convolutional network to dynamically learn the spatial relationships between each patch and other patches, allowing the model to better adapt to varying channel configurations in downstream tasks. By doing so, we aim to enhance the flexibility and applicability of EEG foundation models across diverse EEG data formats.

In our revised manuscript, we have enhanced the discussion of these methods, particularly with respect to their motivation and design (Line 81-93, Page 2). We believe our approach offers a novel and valuable contribution to the design of positional encoding in EEG foundation models, and we hope it provides useful insights for overcoming the challenges of EEG data variability.

Thank you again for your valuable feedback.

W2: To learn generic representations from both time-domain and frequency-domain EEG signals, this study conducted pretraining on the TUEG dataset, which primarily focuses on medical data related to conditions such as epilepsy. However, as a foundation model, CBraMod exhibits significant discrepancies between the pretraining data and the data used for downstream tasks. This raises questions regarding its theoretical validity.

R: Thank you for your comment. We address your concerns regarding the use of the TUEG dataset for pretraining with the following points:

1. Large, Diverse, and Long-duration Dataset: Although TUEG primarily focuses on medical data related to conditions such as epilepsy, it includes a large and diverse participant pool, along with long-duration EEG recordings. This diversity allows the model to learn generalizable features that are not specific to any one condition, while the long-duration recordings provide rich temporal context for capturing complex dependencies within EEG signals.

2. Broad Applicability Across Tasks: By pretraining on this extensive dataset, CBraMod learns generic EEG representations that improve performance on downstream tasks. Despite the medical focus of the pretraining data, the learned features are not overly specialized to epilepsy-related signals. Instead, they capture essential EEG characteristics that can be adapted to a wide range of tasks. The experimental results show that CBraMod achieves the state-of-the-art performance across up to 10 downstream BCI tasks (12 public datasets), empirically proving its strong capability and generalizability.

We hope these points clarify the rationale behind our choice of pretraining data. Thank you again for your valuable feedback.

W3: The ablation studies related to pretraining, being a crucial experiment, should be included in the main text.

R: Thank you for your suggestion. We acknowledge that the ablation studies related to pretraining are a crucial part of our experiments. In the original version of the paper, these ablation studies were included in the appendix. As the reviewer suggested, we have moved this content to the main text to highlight its importance and ensure clearer visibility in the revised version (Line 507-527, Page 10). We believe this will provide a more comprehensive understanding of the impact of pretraining on our model's performance.

Q3: Could the authors elucidate their perspective on the motivation or value of foundation models within the EEG domain? Additionally, how does CBraMod balance the relationship between model size, computational cost, and performance?

R: Thank you for your insightful question. Below is a structured response regarding the motivation behind EEG foundation models and the trade-off between model size, computational cost, and performance in CBraMod:

1. Motivation and value for EEG Foundation Models: The primary goal of EEG foundation models is to learn generalized representations from large-scale unlabeled EEG data with self-supervised learning, thereby enhancing the performance of diverse downstream tasks without relying on handcrafted features, prior knowledge specific to individual tasks or large amounts of labeled data. Unlike traditional methods that rely on task-specific feature engineering or supervised learning requiring large amounts of labeled data, a foundation model provides a unified framework to streamline EEG analysis and enable applications such as brain-computer interfaces (BCIs).

2. Long-term Vision: In the long run, EEG foundation models aim to revolutionize the existing paradigms of EEG research and application, similar to how large language models (LLMs) and vision models have done for their respective fields. These models have the potential to inspire more paradigm-agnostic or universal brain-computer interface (BCI) designs. In the revised version of our paper, we detailedly discussed the impilcations of our work (Line 1754-1781, Page 33).

3. Model Size, Computational Cost, and Performance Trade-offs: In our original paper, we provide comparisons of model size and computational cost between our approach, supervised algorithms, and other EEG foundation models (Line 1607-1636, Page 30-31). While our method is computationally more expensive than traditional supervised methods, it is still more efficient compared to existing EEG foundation models.

4. Performance Benefits: While it’s challenging for EEG foundation models to be more efficient than traditional supervised algorithms, they offer better performance. By exploring the way for a One-For-All solution, EEG foundation models hold the promise of addressing multiple tasks with a unified model, positioning them as a transformative tool for diverse EEG applications.

This summarizes the motivation behind EEG foundation models and the trade-offs in terms of model size, computational cost, and performance in CBraMod. Thank you for your insightful question!

Q4: During the pretraining phase, 19 common electrodes were utilized. How does the model handle discrepancies in the number of electrodes when applied to downstream tasks?

R: Please see the Response to W1.

Dear Reviewer m8H6,

First and foremost, we would like to sincerely thank you for your careful review and valuable feedback on our work. We have carefully considered each of the concerns you raised and would like to provide more detailed responses to address these concerns.

Concern 1: First, the authors have not provided a clear and effective explanation of how CBraMod addresses the challenge posed by the diverse formats of EEG data. This leads me to believe that the CBraMod model can only process inputs of the same format (i.e., the 19 selected common channels) during the pretraining phase. This limitation contradicts the authors' claim that CBraMod can handle EEG data of varying formats as a foundational model.

Response 1: Thank you for your valuable comment. We would like to clarify the concerns raised regarding the diversity of EEG formats and CBraMod's ability to handle them.

1. CBraMod’s Capability to Handle EEG data with varying numbers of channels and temporal segments: The backbone of CBraMod is the criss-cross transformer, which is an improved version of the traditional transformer architecture for EEG signals. The transformer backbone allows CBraMod to process EEG data with varying numbers of channels and temporal segments. The criss-cross transformer is designed to handle such diversity at a technical level, making it highly adaptable to different EEG formats. Therefore, CBraMod is not limited to processing data with the 19 selected channels during pretraining.

2. Improvement in positional encoding with ACPE: To address the challenges of positional encoding in EEG data, we introduced Asymmetric Conditional Positional Encoding (ACPE). Traditional EEG foundation models rely on electrode-wise absolute positional encodings, which often fail to generalize across different channel configurations, particularly when various referencing schemes (e.g., earlobe, average, REST, or bipolar references) lead to non-equivalent channel-electrode mappings. ACPE employs a convolutional network to dynamically learn the relative positional relationships among channels, allowing CBraMod to adapt flexibly to diverse channel layouts in downstream tasks on the fine-tuning process. This capability ensures that CBraMod can generalize beyond the specific format used during pretraining, helping mitigate the challenge of heterogeneous EEG formats. In fact, the strong performance of CBraMod across all downstream datasets has already empirically demonstrated its ability to handle diverse EEG data.

3. Large EEG Corpus TUEG: We selected the TUEG dataset because it includes a large and diverse participant pool, along with long-duration EEG recordings. The dataset is sufficiently large to facilitate the learning of generic EEG representations that can be transferred to various downstream tasks. Importantly, this choice does not imply that CBraMod is limited to training on 19-channel EEG data. CBraMod is inherently capable of handling EEG data with varying numbers of channels, and we plan to extend the model to encompass more diverse pretraining datasets in the future work.

We hope our contributions can provide insights for EEG foundation model research.

[1] Yang C, Westover M B, Sun J. BIOT: biosignal transformer for cross-data learning in the wild[C]//Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023: 78240-78260.

[2] Jiang W, Zhao L, Lu B. Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI[C]//The Twelfth International Conference on Learning Representations.

Dear Reviewer m8H6,

Thank you very much for your follow-up comments! We appreciate your continued engagement with our work and would like to address the concerns you raised.

In this work, our core contribution lies in the architectural improvements, such as the criss-cross transformer and ACPE, aiming to learn a generalizable EEG representation. The diversity of channel configurations during the pretraining phase is not so important consideration of this work. For pretraining, we selected the TUEG dateset with a fixed set of 19 channels which can provide comprehensive coverage of different brain regions, ensuring sufficient representativeness. This strategy that selecting a fixed number of channels also aligns with the strategies adopted by existing methods such as BENDR [1] and Neuro-GPT [2]. It is worth noting that, although we selected a fixed set of channels for pretraining, this does not imply that CBraMod is inherently constrained to processing data with only 19 channels during pretraining.

To further address your concerns, we conducted additional experiments where CBraMod was pretrained on datasets with varying scales and channel configurations. We then evaluated the pretrained model on entirely new downstream datasets to validate its ability to generalize beyond the specific formats seen during pretraining.
Given the limited time, we selected a few not-large datasets for pretraining. The pre-training datasets are as follows:

We pre-trained CBraMod on these datasets with different channels and fine-tuned CBraMod in FACED, then compared the performance with CBraMod without pre-training. The experimental results are shown as follows:

The results clearly show that CBraMod pretrained on such datasets with different configurations outperforms an untrained (randomly initialized) CBraMod. These findings demonstrate that CBraMod can address varying channel configurations during pre-training and effectively improve the performance of downstream tasks.

We hope these explanations and additional results address your concerns more comprehensively. Thank you once again for your valuable feedback, which has greatly contributed to improving our work.

Best regards,

The authors.

[1] Kostas D, Aroca-Ouellette S, Rudzicz F. BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data[J]. Frontiers in Human Neuroscience, 2021, 15: 653659.

[2] Cui W, Jeong W, Thölke P, et al. Neuro-GPT: developing a foundation model for EEG[J]. arXiv preprint arXiv:2311.03764, 2023, 107.

Dear Reviewer m8H6,

We sincerely appreciate your thoughtful engagement with our work throughout the review process and for raising your score! Your feedback has been instrumental in helping us refine our manuscript, and we truly value the time and effort you have dedicated to evaluating our submission!

We fully understand your concerns regarding the pretraining setup. Although the discussion period was limited, we have still proved that CBraMod can effectively handle EEG data with varying channel configurations during the pretraining phase with new experiment. Of course, we recognize that there is still room for further exploration in pretraining data construction, but it is not the key contribution of our current work. Moving forward, we will actively investigate how the choice of pretraining datasets and configurations may impact model performance, ensuring a more comprehensive understanding in future work.

Once again, thank you for recognizing the contributions of CBraMod and for your constructive feedback, which will guide us as we continue to advance our research!

Best regards,

The authors

We thank the reviewer for the positive review and insightful analysis.

W1: Limited Comparative Analysis with Recent Models. While the paper compares CBraMod with several non-foundation and foundation model baselines, the inclusion of more recent EEG foundation models, e.g. BrainWave (https://arxiv.org/abs/2402.10251) could provide a more comprehensive evaluation of CBraMod’s relative performance.

R: Thank you for your valuable suggestion regarding the inclusion of recent EEG foundation models such as BrainWave. While we acknowledge the potential relevance of these models to our study, there are two main challenges in incorporating them into our current analysis:

1. Availability of Code and Pretrained Weights: Many of the recent EEG foundation models, have not publicly released their source code (e.g. BrainWave) or pretrained model weights (e.g. EEG2Rep). This significantly limits our ability to directly compare their performance with CBraMod. As a result, we are unable to include them in our experimental evaluation at this time.

2. Strength of Current Baselines: The baseline models included in our study are among the most representative and competitive models currently available in the field of EEG foundation models. These models have demonstrated the state-of-the-art performance on EEG-related tasks, and we believe they provide a robust comparison for assessing the performance of CBraMod.

We hope this clarifies our rationale for the choice of baselines. We are committed to extending our analysis in the future work. Once the code and pretrained weights for newer models like BrainWave become available, we would be eager to incorporate them into our evaluations.

W2: Lack of interpretability. The authors did not provide any kind of interpretation of the obtained models. First of all, it would be very interesting to see what pieces of EEG (source topographies, frequency bands) contribute to the "powerful embeddings" learnt by the foundation model. Secondly, for the majority of the downstream tasks there is well defined defined hypothesis regarding the way the decoded information is encoded in the brain. For example motor imagery classification would require information in the 18-14 Hz and 15-25 Hz range on the sources located on the sensory-motor cortex with very distinct topographies. Addition of these would make the paper more convincing.

R: Thank you for your insightful comments on the interpretability of our model. We would like to clarify our approach and address your suggestions as follows:

1. Powerful Representations Learned from Data: To address the concern regarding the potential benefits of interpretable features like frequency bands, we have conducted new experiments to compare CBraMod on weak processed (only 60hz notch filter and 0.3 hz high-pass filter) with filter-bank-based FBCNet [1] (4 bands: 48Hz, 814Hz, 15-25Hz, 25-40Hz) on motor imagery tasks (PhysioNet-MI).

   	PhysioNet-MI, 4-class
   Methods	Balanced Acc.	Coken’s Kappa	Weighted F1
   FBCNet	0.6164 $\pm$ 0.0103	0.4965 $\pm$ 0.0159	0.6212 $\pm$ 0.0113
   CBraMod	0.6417 $\pm$ 0.0091	0.5222 $\pm$ 0.0169	0.6427 $\pm$ 0.0100

   The results show that our model on weak processed achieves competitive or superior performance, demonstrating that it effectively learns relevant information even without explicit frequency-based preprocessing. We believe these findings underscore the strength of our model's data-driven design and its applicability across EEG tasks. More details are presented in the revised version of our paper.

2. Interpretability: To further address concerns regarding the interpretability, we have added new interpretability analysis in the revised version (Line 1638-1750, Page 31-33), including (1) Topography Visualization, (2) Representation Visualizations on Downstream Datasets and (3) Visualization of Patch Relationships from each Criss-cross Transformer Layer.

   • (1) Topography Visualization demonstrates that CBraMod effectively captures symmetric patterns between the left and right hemispheres from motor imagery data.
   • (2) Representation Visualizations on Downstream Datasets demonstrates that our pre-trained model can learn representations with clustering effects tailored to downstream tasks.
   • (3) Visualization of Patch Relationships from each Criss-cross Transformer Layer confirms that our criss-cross attention effectively models criss-cross dependencies.

We hope these clarifications and additional experiments address your concerns about interpretability and model performance. Thank you again for your valuable feedback.

[1] Mane R, Chew E, Chua K, et al. FBCNet: A multi-view convolutional neural network for brain-computer interface[J]. arXiv preprint arXiv:2104.01233, 2021.

Q4: Could the authors provide additional details on whether the entire CBraMod model or only specific layers are fine-tuned for downstream tasks?

R: Thank you for your question regarding the fine-tuning process. Unless otherwise specified, we fine-tune the entire CBraMod model during the adaptation to downstream tasks. This approach allows the model to adjust all of its parameters to the specific requirements of each task, ensuring optimal performance. By fine-tuning the entire model, CBraMod is able to leverage the generalizable embeddings learned during pretraining while adapting to the nuances of the target task, leading to better task-specific performance.

Q5: Related to (4). Could the authors be more explicit regarding the strategies to prevent overfitting during fine-tuning on smaller downstream datasets (e.g., dropout rates, weight decay specifics beyond pre-training)?

R: Thank you for your question regarding strategies to prevent overfitting during fine-tuning. To mitigate overfitting on smaller downstream datasets, we use a dropout rate of 0.1 and a weight decay of 0.05 during the fine-tuning phase. These regularization techniques help prevent the model from overfitting to the limited data available for each specific task. For further details on these hyperparameters and additional settings used during fine-tuning, please refer to the paper (Line 1055-1069, Page 20), where we provide a comprehensive description of the training procedure and hyperparameter choices.

Furthermore, we have added a new experiment on low-resources settings, where we fine-tuned CBraMod only using 30% data of the training set in each downstream dataset. The details are presented in the revised version of our paper (Line 1494-1524, Page 28-29). Overall, our model outperforms existing SOTA methods in low-resource scenarios. This empirically demonstrates that our model can effectively mitigate the challenges posed by limited downstream task training data to some extent.

Dear Reviewer k8cX,

Thank you for your encouraging feedback and for maintaining a positive score for our work. We sincerely appreciate your thoughtful evaluation and recognition of our efforts in addressing the reviewers' comments. Your support motivates us to continue improving and advancing EEG research.

Best regards,

The authors

We are grateful to the reviewer for the encouraging comments and thoughtful questions.

W1: The current version shows a good architecture for feature extraction instead of a real foundation model. It would be beneficial to give a clearer description of the motivation for proposing such a ‘foundation model’ and how this model facilitates the application of EEG analysis and even brain-computer interfaces. Could we use only a few data for finetuning or tuning specific model faster?

R: Thank you for your comments and suggestions regarding the motivation for our proposed EEG foundation model and its potential for fine-tuning under limited data conditions. We address your points as follows:

1. Motivation for EEG Foundation Models: The primary goal of EEG foundation models is to learn generalized representations from large-scale unlabeled EEG data with self-supervised learning, thereby enhancing the performance of diverse downstream tasks without relying on handcrafted features, prior knowledge specific to individual tasks or large amounts of labeled data. Unlike traditional methods that rely on task-specific feature engineering or supervised learning requiring large amounts of labeled data, a foundation model provides a unified framework to streamline EEG analysis and enable applications such as brain-computer interfaces (BCIs).

   We acknowledge that EEG foundation models are still in their early stages compared to large-scale models in NLP and computer vision. This highlights the significant room for exploration and innovation in designing robust, scalable EEG foundation models.

2. Architectural Contributions: To advance this exploration, our architecture incorporates two key contributions:

   • Criss-Cross Attention Mechanism: Designed to effectively leverage the structural properties of raw EEG data, enabling the model to capture complex dependencies across channels and time.
   • Asymmetric Channel Positional Encoding (ACPE): A dynamic positional encoding scheme that overcomes the limitations of traditional absolute position encoding tied to electrode numbering, improving the model’s adaptability to diverse referencing schemes and spatial patterns.

   We have refined the revised manuscript to provide a clearer description of these motivations and their implications for EEG analysis. (Line 81-93, Page 2)

3. Low-Resource Fine-Tuning Experiment: To address your concern regarding the model’s efficiency in scenarios with limited data, we conducted an additional experiment under low-resource conditions. The results demonstrate that our method significantly outperforms existing approaches when fine-tuned with only a small amount of data. These findings underscore the practical utility of our proposed architecture in real-world applications where data availability is often constrained. (Line 1494-1524, Page 28-29)

We hope these clarifications and additional experiments address your concerns and highlight the potential of our model as a foundation for EEG analysis and BCI development. Thank you again for your valuable feedback.

Q1: What features do the time-domain and frequency-domain branches capture? It would be beneficial to illustrate the specific frequency components learned by the frequency-domain branch within the patch encoder.

R: Thank you for your insightful question regarding the features captured by the time-domain and frequency-domain branches. Below, we provide further clarification:

1. Time-Domain Branch: As described in the original manuscript, the time-domain branch utilizes a convolutional neural network (CNN) to directly process raw signals within each patch. This enables the model to extract temporal features such as signal patterns, trends, and temporal dependencies that are critical for EEG analysis.

2. Frequency-Domain Branch: For the frequency-domain branch, we first apply a Fourier Transform to convert raw signals into the frequency domain. The transformed frequency components are then projected into frequency-domain embeddings through a linear layer. This branch captures spectral characteristics, such as power distribution across frequency bands, which are essential for tasks like abnormal detection and event classification.

3. Experimental Validation: In the appendix section "Ablation Study on Time-Domain and Frequency-Domain Signals" (Line 1440-1463, Page 27-28), We provided experimental evidence demonstrating the complementary contributions of these two branches. The results show that Time-Domain and Frequency-Domain Branch both contribute to the performance gains.

We hope this response clarifies the distinct roles and contributions of the time-domain and frequency-domain branches in our model. Thank you again for your thoughtful feedback.

Q5: Many layers are employed, yet the feature maps are not large. Did each layer contribute significantly to the final results?

R: Thank you for your question regarding the contribution of multiple layers in our model. To address this concern, we have included a performance comparison for varying numbers of Transformer layers with the fixed 200 embedding dimension. The experimental results are as shown as follows:

It is evident that increasing the number of model layers leads to a noticeable improvement in performance. We believe that increasing the number of transformer layers could potentially further improve performance. However, due to constraints on time and computational resources, we are currently unable to evaluate whether additional transformer layers would lead to further performance gains.

We hope this additional experiment clarifies the role of the layers in the model's overall performance. Thank you again for your thoughtful feedback.

Q6: The paper mentions that only 19 channels were used during pre-training, whereas additional channels were introduced in the downstream task. Could you clarify how the balance between channels was managed?

R: Thank you for raising this point regarding the balance between channels during pretraining and downstream tasks. As mentioned in the Response to Q2, we leverage the dynamic characteristics of our ACPE (Asymmetric Conditional Positional Encoding) to adapt to the varying number of channels in different downstream tasks. This approach allows our model to effectively handle changes in the number of channels by dynamically learning spatial relationships between the channels, rather than relying on fixed positional encoding schemes. Consequently, the model can seamlessly adjust to the introduction of additional channels in downstream tasks without compromising performance.

We hope this explanation clarifies how the balance between channels is managed. Thank you again for your thoughtful feedback.

Dear Reviewer 1DYx,

Thank you for your feedback and for maintaining a positive score. We are glad to have addressed most of your concerns. In the revised version of our paper, we have added interpretability analysis for the generic EEG representations of our model (Line 1638-1750, Page 31-33). We appreciate your suggestion to further explore the interpretation of generalizable features.

Thank you again for your valuable comments.

Best regards,

The authors

Dear Reviewers,

We hope this message finds you well. We greatly appreciate your insightful feedback on our paper. We have made significant revisions to address your concerns, including:

• Motivation of CBraMod: Clarified the motivation behind our approach and its unique contributions.
• Interpretability: Added new analyses to visualize model representations and dependencies.
• Low-Resource Performance: Demonstrated the model's effectiveness in low-data scenarios.
• Dataset Choice: Conducted additional experiments on new datasets to ensure comprehensive and fair comparison.
• Baseline Comparisons: Included well-established models like EEGNet and EEGConformer.
• Value to EEG/BCI Field: Expanded on the potential impact and long-term value of our work.

We kindly request your prompt review of our rebuttal to finalize the decision process. Your timely response is crucial.

Thank you for your time and consideration.

Best regards,

The authors.

Thanks! Very interesting! The reason I asked to test with a different band is that I was interested in how much EMG activity contributes to decoding. Seems like not much! However your model seems to have the largest drop in performance as compared to the other models which may suggest its greater reliance on the relatively high frequency features to squeeze the high scores. Most of the power in the regular EEG situated above 30-40 Hz comes from the muscles. See the works by Witham et al. (or smth like this). Also, in your response the depression dataset posits a 2-class problem and not a 9-class as you have copy-pasted) .

Dear Reviewer k8cX,

Thank you for your insightful and thought-provoking feedback, which has greatly contributed to refining and contextualizing our work. Below, we provide responses to your new questions.

Q1: Section E.4 where you classify depressed vs. normal subjects. What will happen to the performance of all (or some of) the models when you pre-filter the data in the 0.3-30 Hz range rather than 0.3-75 Hz as you (and perhaps other authors) did in the mental disorder classification task?

R1: We conducted additional experiments as suggested, comparing the performance of EEGConformer, LaBraM, and CBraMod under the 0.3-30 Hz pre-filtering condition on the Mumtaz2016 dataset. The results are summarized below:

As shown, filtering the data to 0.3-30 Hz generally leads to a performance degradation (expecially on balanced accuracy) across all models. This is likely due to the loss of high-frequency information (30-75 Hz), which may still carry relevant features for mental disorder classification tasks.

Q2: While I do appreciate you willingness to demonstrate the superiority of your model over other solutions I feel a bit perplexed. Just purely from the statistical reasons, having performed so MANY additional comparisons you demonstrate that your model beats all the other solutions, don't you think that there should be some tests where some other models perform better? May be I missed smth.

R2: We are delighted that the reviewer has raised such a question, as it suggests that our model’s performance appears to have exceeded expectations and sparked significant interest. However, we must clarify that CBraMod does not consistently outperform all existing methods under all conditions. On a few datasets, our model performs on par with other models.

For example, in the performance evaluation on the TUEV and TUAB datasets excluding potential data leakage (shown in Response to Reviewer iYYg (7/n)), our CBraMod achieves competitive performance with state-of-the-art (SOTA) methods but does not always significantly surpass them. Specifically:

In these cases, CBraMod performs similarly to LaBraM, with LaBraM even showing slightly better balanced accuracy on TUAB.

Additionally, as noted in the manuscript (Line 1470-1479, Page 28), our CBraMod does not consistently outperform existing EEG foundation model when the pre-trained weights of all the foundation models are completely frozen during fine-tuning on some downstream datasets.

Finally, prompted by your thoughtful question, we wish to acknowledge a specific limitation of CBraMod: as a patch-based EEG foundation model with a patch size of 1 second, it may not be suitable for EEG signals shorter than 1 second in duration. While we do not consider this a significant limitation for an EEG foundation model (this limitation also exists on LaBraM), we will include this clarification in the limitations section of the manuscript to ensure full transparency.

We greatly appreciate your feedback, which has helped us refine our understanding of CBraMod’s performance and limitations.

Best regards,

The authors