NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals

Thanks for your wisdom and valuable comments. We sincerely appreciate your recognition of our work. Here are our responses to your questions and concerns point-by-point.

[Q1] The effectiveness of EEG-Text Embedding Space Alignment

1. The visualizations of EEG and text embeddings are presented in Appendix F.2, which you may have missed. Without alignment, the EEG embeddings expand outside the text space, but with alignment training, the EEG embeddings are brought closer to the text space, achieving partial overlap.
2. We apologize for not clarifying the source of the text embeddings. For alignment, we use embeddings from GPT-2’s text vocabulary, and each batch is randomly sampled from the vocabulary. We have added this information in Section 3.2.
3. In our ablation study, we found that without EEG-text embedding alignment, the model fails to produce meaningful predictions in multi-task instruction tuning. Specifically, it outputs random words instead of the expected options (e.g., (A), (B), or (C) in choice questions), resulting in near-zero scores across most downstream task metrics. This misalignment likely stems from disordered attention scores, as EEG and text embeddings remain in separate spaces. We discussed this phenomenon in Appendix F.2.

[Q2] The theoretical foundation of multi-channel autoregressive pre-training

• Thank you for this insightful question. We agree that further theoretical grounding could enhance the understanding of how NeuroLM models dependencies between EEG channels and captures causal relationships in an autoregressive framework. In response, we have modified the Theory Analysis section in 2.2 to provide a more detailed formulation: $-\sum_{i=1}^N(E_{z_i\sim q_\phi(z|x_i)}[-\log p_\psi(y_i|z_i)]-\sum_{j=2}^T\log p_\theta(z_{i,j}|x_{i,<j}))$. Specifically, we revised the second term in the Equation (7) to explicitly reflect causal relationships by using a negative log-likelihood loss that accounts for latent variables across all channels at each time step. $z_{i,j}$ denotes latent variables of the i-th sample of all channels at time step j and $z_i=[z_{i,1},...,z_{i,T}]$, while $x_{i,<j}$ denotes the input of the i-th sample of all channels prior to time step j. This adjustment clarifies how the model learns dependencies over time and across channels, capturing the causal relationships inherent in EEG data. By conditioning each latent variable on past time steps, our approach allows NeuroLM to model multi-channel autoregressive relationships, which are crucial for capturing sequential dependencies in EEG signals. We hope this revision strengthens the theoretical foundation of our multi-channel autoregressive approach and provides a clearer mathematical basis for how NeuroLM models EEG data dependencies.

[Q3] The lack of discussion on instruction design's impact on downstream tasks

• Thank you for this insightful question. We agree that the design of instructions and variations in prompt formulations may affect task performance. In this work, we incorporated some prompt-design tricks, such as randomizing the order of options in multi-choice tasks, as discussed in Section 3.4. This helped us ensure that NeuroLM could generalize to various option orders, thus increasing robustness across tasks. We also considered more complex prompt variations, such as introducing a chain-of-thought approach for EEG tasks. However, applying such advanced prompting techniques to EEG data is currently challenging due to the inherent difference in processing language and neural signals. Developing effective ways to incorporate these techniques would add significant complexity, and we see this as a promising direction for future research.

[Q4] The absence of an analysis on data curriculum or input order.

• Thank you for this thoughtful question. In this study, we applied the simplest approach to data ordering by randomly shuffling the inputs across tasks. Our results show that even with this straightforward strategy, NeuroLM achieves promising performance across diverse datasets with varied sizes and complexities. This suggests that NeuroLM is robust to input order, but we agree that a structured data curriculum could further optimize performance. Exploring data sequencing strategies, such as introducing tasks in increasing complexity, could be a valuable avenue for future work. Given the scope of this study, however, including and exploring this intriguing idea would add excessive and overly varied content to the paper. Curriculum learning is indeed an exciting and worthwhile direction for future research, as it may offer additional performance gains beyond our initial findings.

Thanks for your wisdom and valuable comments. Here are our responses to your questions and concerns point-by-point.

[W1 & Q3] Absence of Fine-Tuning Examples

• The primary contribution of NeuroLM lies in its unified multi-task instruction-tuning approach, which differentiates it from fine-tuning-focused models like LaBraM. Our goal was to establish a framework that generalizes across diverse EEG tasks without task-specific fine-tuning, showcasing NeuroLM’s flexibility and potential for handling new tasks or prompts directly through instruction tuning. That said, we recognize that exploring fine-tuning capabilities could potentially enhance NeuroLM’s task-specific performance and further demonstrate its utility in practical applications. Achieving this would likely involve several future improvements: 1) Stronger Language Models: NeuroLM currently utilizes GPT-2, a relatively small and older LLM. Incorporating more advanced LLMs, such as LLaMA 3 series could significantly improve both the model's generalization capabilities and fine-tuned task-specific performance. 2) Fine-Grained Contrastive Learning-Based Alignment: Enhancing the EEG-text alignment process through contrastive learning could enable the model to better align EEG signals with textual descriptions, leading to more robust representations and performance gains. 3) Expanding Data Resources: Pretraining on a larger and more diverse dataset of EEG signals could improve the model’s capacity to generalize across tasks and handle fine-tuning more effectively. Additionally, using high-quality annotated datasets could further refine task-specific outcomes. These directions represent opportunities for future work that can build on NeuroLM’s foundation. In this paper, our primary aim is to demonstrate the feasibility and advantages of the instruction-tuning paradigm for EEG tasks, and we look forward to exploring these enhancements to further unlock the potential of NeuroLM in practical applications.

[W2 & Q1] Dependency on GPT-2

• Thank you for this feedback. We appreciate the interest in seeing NeuroLM evaluated with more advanced LLMs. In this work, we used GPT-2, a smaller and older LLM, to establish the feasibility of our approach. The fact that NeuroLM performs well with GPT-2 gives us confidence that it would achieve even stronger results with more powerful models, such as LLaMA series or other recent LLMs designed for enhanced performance and multimodal capabilities. Due to limited computational resources and the high demands of training larger LLMs, we apologize that we are unable to add comparisons with other models in this rebuttal stage. However, exploring NeuroLM’s performance with advanced LLMs is an exciting avenue for future work, and we believe that our method’s adaptability across models will offer even broader potential as it scales with more sophisticated LLM architectures. Anyway, we hope to present our preliminary results in this paper to inspire future exploration.

[W3] EEG-Text Alignment

• To assess the impact of EEG-text alignment, we have conducted ablation studies in which we removed the alignment loss, and the results showed that training fails entirely without it. Specifically, without the alignment, the model struggled to produce meaningful predictions, outputting arbitrary codes instead of expected responses like (A), (B), or (C), resulting in near-zero scores across most metrics. This appears to stem from disordered attention scores, as EEG and text embeddings remain unaligned in separate spaces. We have detailed these observations in Appendix F.2, and we appreciate your suggestion, as it highlights the importance of EEG-text alignment in NeuroLM's overall performance and robustness.

[W4] Complexity and Reproducibility

• We promise to make our codes and model weights publicly available based on publication for the community to further explore this new paradigm.

[Q2] What role does adversarial training play...

• In this work, we used adversarial training for EEG-text alignment to demonstrate the feasibility of our new multi-task instruction-tuning paradigm with LLMs. This approach provides a straightforward way to align EEG signals with text embeddings, enabling NeuroLM to process multiple EEG tasks within an LLM framework. While adversarial training was effective for this initial demonstration, we recognize that contrastive learning-based alignment could further enhance model performance by achieving more detailed alignment between EEG and text as mentioned in Appendix H. However, incorporating contrastive learning would introduce significant complexity, potentially detracting from our focus on establishing the feasibility of this multi-task framework. We see contrastive alignment as a promising direction for future work, allowing us to refine and advance the alignment mechanism in NeuroLM.

Thanks for your wisdom and valuable comments. We sincerely appreciate your recognition of the contribution of our work. Your tips are really helpful for improving the quality of our paper. Here are our responses to your questions and concerns point-by-point.

[W1] Even though NeuroLM's performance is currently lower compared to LaBraM...

• We are deeply grateful for your suggestion that NeuroLM's key strengths lies in its unified instruction-tuning approach, which indeed has the potential to enable generalization to novel tasks or prompts without the need for extensive task-specific fine-tuning. We have expanded on this in the revised manuscript in Section 3.3 by highlighting that NeuroLM’s instruction-tuned design can facilitate zero-shot or few-shot generalization to unseen EEG tasks, as it has been optimized to handle a broad range of instructions and queries.

[W2] The section on methodology...

• Thank you for highlighting the need for additional clarification regarding the VQ-VAE codebook and its role in NeuroLM. The VQ-VAE framework is essential for discretizing continuous EEG signals into neural tokens, which we then align with LLM text embeddings. To initialize the codebook, we use k-means clustering on the initial batch of embeddings from the VQ encoder, creating 8192 cluster centers that form the basis of the 8192 codes in the codebook.
• For codebook learning, we follow Equation (3) to iteratively update the codes. Since the lookup and replacement operations are undifferentiable, we apply a stop-gradient operation to transfer gradients from the codebook directly to the VQ encoder, effectively stabilizing the training. The second term in Equation (3) aligns the codes with embeddings from the VQ encoder, while the third term brings these embeddings closer to the codes in the codebook. After training the neural tokenizer, the codebook is merged with the LLM’s text vocabulary, enabling multi-channel autoregression.
• Since this tokenizer training strategy aligns closely with LaBraM, we simplified its explanation in the initial draft. We apologize for any inconvenience and will include these detailed explanations in our revision.

[W3] For the EEG-text alignment...

• Thanks for your suggestion. We have conducted experiments removing the alignment loss, and the results confirm that training fails entirely without it. Without alignment, the model struggles to produce meaningful predictions, often outputting arbitrary codes instead of expected options like (A), (B), or (C), leading to near-zero scores across most metrics on downstream tasks. This outcome appears to stem from disordered attention scores, as EEG and text embeddings remain in separate spaces. We have detailed this observation in Appendix F.2.

[W4] Change the colormap for Figure 1...

• We have revised the colormap accordingly to ensure a clear distinction between the models, enhancing readability.

[Q1] It would be highly beneficial...

• Definitely, we will make our codes and model weights publicly available based on publication.

[Q2] The expected benefits of large-scale pretraining...

• Thank you for highlighting this important point. Frankly, the choice of multi-channel autoregressive pre-training in NeuroLM was indeed a compromise. It is widely acknowledged that masked signal modeling tends to be a more efficient representation learning approach than next-token prediction, which contributes to LaBraM’s stronger performance. Given the causal nature of current LLMs, we aimed to find a feasible pre-training strategy that works within these constraints, with autoregressive modeling being the most compatible option for EEG tokens. However, NeuroLM does have potential pathways for performance improvements. As noted in Appendix H, one promising direction involves leveraging contrastive learning to align EEG signals with textual descriptions within the VQ encoder, drawing inspiration from approaches like CLIP [1]. This would enable more detailed alignment with text and likely bring significant performance gains to NeuroLM. Nonetheless, our current work focuses on establishing the feasibility of incorporating EEG into LLMs through the multi-task instruction-tuning framework. Introducing contrastive learning for pre-training, while promising, would add considerable complexity to the current study, and we see this as an exciting avenue for future work.

[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision. ICML 2021

Thanks for your wisdom and valuable comments. Here are our responses to your questions and concerns point-by-point.

[W1] While NeuroLM performs well in multi-task learning...

• Our target: The primary objective of this work is to enable off-the-shelf LLMs to perform EEG tasks through a unified instruction-tuning paradigm. NeuroLM’s key strengths lies in its unified instruction-tuning approach, which may have the potential to enable generalization to novel tasks or prompts without the need for extensive task-specific fine-tuning. This work is the first to demonstrate that such a multi-task instruction-tuning approach can be effective for EEG tasks within an LLM framework.
• Reasons for not achieving SOTA: NeuroLM's multi-channel autoregressive pre-training approach was selected to ensure compatibility with the causal structure of current LLMs, even though it may not reach the efficiency of masked signal modeling, which is known to produce strong results in single-task models like LaBraM. The autoregressive approach allows us to adapt EEG data to a causal language model framework, enabling multi-task learning within the constraints of current LLM architectures.
• Potential improvements: We do recognize opportunities to further enhance NeuroLM’s performance. As mentioned in Appendix H, future improvements could involve contrastive learning to align EEG signals with textual descriptions in the VQ encoder, similar to the CLIP framework. This alignment would allow a more nuanced integration of EEG and text data and could yield significant gains in task performance. In this work, our focus was on establishing the feasibility of integrating EEG with LLMs in a unified, multi-task framework. Adding contrastive learning to our pre-training process would introduce substantial complexity, so we view this as a promising direction to pursue in future work. That said, we aim to present our preliminary results in the paper to encourage further exploration in this area.

[W2] Model performance fluctuates...

• Thank you for noting this point. We observed that performance variability across tasks and datasets is influenced by several factors, including dataset size, task complexity, and variability in EEG data quality. For example, tasks with more extensive and balanced datasets allow NeuroLM to capture more representative features, resulting in stable performance. In contrast, smaller or highly specialized datasets tend to introduce variability, as the model’s generalization capacity is challenged. It’s worth noting that, apart from LaBraM, NeuroLM achieves comparable or even superior performance relative to other single-task baselines across most downstream datasets, underscoring the effectiveness of its unified multi-task learning approach. To further stabilize performance across datasets, one promising direction for future work involves exploring adaptive weighting strategies. This approach could help balance learning across tasks with different data distributions and complexities.

[W3] Although the model reduces the need...

• Thank you for raising this point. While NeuroLM’s pre-training and multi-task learning approaches help reduce dependence on extensive labeled data, the model’s performance and generalization are indeed influenced by the quality of its training data. High-quality EEG data is crucial for capturing robust and generalizable representations, as it helps mitigate noise and variability that can affect model effectiveness—a common challenge across EEG processing methods. Many datasets contain some lower-quality samples, such as those with interference due to subject movement (EMG, EOG). Such noise can detract from model performance. For a fair comparison, we included all data across methods without filtering, but removing low-quality data presents a promising opportunity to further enhance NeuroLM’s performance.

[Q1] The paper mentions that NeuroLM...

• Thank you for this question. NeuroLM’s unified instruction-tuning approach enables it to handle multiple EEG tasks across diverse datasets without requiring task-specific fine-tuning. Notably, we did not include the specific downstream datasets during neural tokenizer training and multi-task instruction tuning. In our experiments, we observe that NeuroLM maintains competitive performance across six distinct datasets, suggesting robust generalization. However, we acknowledge that a systematic evaluation of cross-dataset generalization could provide additional insights. In future work, we plan to evaluate NeuroLM’s transferability explicitly by testing on new datasets, further assessing its adaptability and robustness across varied EEG datasets.

Paper Revisions and Additions

In response to reviewer feedback, we have revised several areas of the paper to enhance clarity and address specific points:

1. We clarified visualizations of EEG-text embedding alignment in Appendix F.2, illustrating the embeddings before and after alignment.
2. We highlight NeuroLM’s key contribution when comparing to other single-task baselines in Section 3.3.
3. We added details regarding the specific text embeddings used during EEG-text alignment in Section 3.2.
4. We adjusted the color for LaBraM in Figure 1 to improve visualization clarity.
5. We modified the Theory Analysis section in 2.2 to provide a more detailed formulation.

Long-Term Contribution

With this paper, we hope to make a lasting contribution to the academic community by demonstrating the feasibility of using LLMs for multi-task EEG instruction tuning. We see NeuroLM as the first step in incorporating EEG into LLMs for multi-task question-answering paradigms, and we hope this work will serve as a foundation for future research to further enhance EEG-LLM integration.

Once again, thank you for your valuable feedback, which has greatly strengthened our work.