Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG

We sincerely appreciate your thoughtful review and the valuable feedback you've provided on our manuscript. Below, we address your concerns in detail:

[Q1]: "The authors have chosen an LSTM-based approach for aligning neural signals with text embeddings... "

Response: We chose an LSTM-based approach for aligning neural signals with text embeddings because our dataset is limited in size. Transformers require significantly more data to avoid overfitting and achieve robust performance. Empirically, we found that a simple 2-layer LSTM provided the best trade-off between performance and generalization.

[Q2]: "The authors used the Vec2text corrector module to reconstruct sentences from neural embeddings ..."

Response: Thank you for asking this question. A simple MSE loss does not distinguish between matching and non-matching neural and text pairs, treating all distances equally. In our task, it is crucial that each neural segment is as close as possible to its corresponding decoded text while remaining dissimilar to other texts. The combination of contrastive and triplet losses ensures this by enforcing both global alignment (contrastive loss) and local margin constraints (triplet loss).

[Q3]: "This paper achieves remarkable performance with as little as 30 minutes of neural data. It raises concerns about the robustness and overfitting of the model when applied to different subjects, which suggests that more rigorous validation methods ... "

Response: We acknowledge your concern about overfitting. We combined all subjects primarily to enhance brain coverage in our experiments. Additionally, our out-of-domain experiments demonstrate the model's ability to generalize to unseen semantic contexts, providing some evidence of its robustness. However, we agree that leave-one-subject-out or similar cross-validation methods would provide a more rigorous evaluation of generalizability, and we plan to explore these approaches in future work as we collect more data for this task.

[Q4]: "The authors propose the use of a pre-trained T5 Corrector to reconstruct sentences from text embeddings..."

Response: Thank you for raising this question. To clarify, the Vec2Text Corrector module in our approach operates auto-regressively, generating text one token at a time, similar to standard generative models. Its primary distinction lies in its iterative refinement mechanism. This approach complements traditional auto-regressive generation by enabling fine-grained corrections, which enhances alignment with the underlying embeddings.

[Q5]: "The presentation of decoding results in Figure 2(B) could be clearer. There is no discussion of whether important named entities or key phrases were successfully decoded...."

Response: We appreciate your suggestion to improve the clarity of decoding results in Figure 2(B) by emphasizing key phrases and named entities. We will incorporate this feedback in the camera-ready version by bolding successfully decoded words and italicizing words with similar meanings to better highlight the model's semantic reconstruction strengths and weaknesses.

[Q6]: "The dataset description lacks important details. Key demographic information, such as the subjects' age, gender, and other relevant characteristics, is missing..."

Response: We appreciate your thoughtful suggestion. We acknowledge that including detailed demographic information about the subjects and a clearer description of the audio stimuli would provide a more comprehensive understanding of the experimental setup. We will incorporate these details in the camera-ready version.

We sincerely appreciate your thoughtful review and the valuable feedback you've provided on our manuscript. Below, we address your concerns in detail:

[W1]: "Generally, the semantic meaning of the reconstructed sentences in both Neuro2Semantic and the baseline method appears to diverge significantly from the original text"

Response: Thank you for your comment. Semantic decoding from neural data, regardless of modality (e.g., EEG, iEEG, MEG, fMRI), remains a challenging task, that no existing methods have fully resolved. Our work demonstrates substantial progress in this area by leveraging transfer learning to achieve results comparable to or better than existing methods in similar conditions.

[W2]: "it would be beneficial to include a related work section that compares the proposed method with previous approaches focused on decoding continuous sentences from neural signals"

Response: We appreciate this valuable suggestion. While our current manuscript discusses related works relevant to our methodology, we acknowledge that further expanding the discussion to include studies focused on the alignment of modalities and related advancements would strengthen our presentation. We appreciate this suggestion and will expand our related work section in the camera-ready version.

[Q1]: "Did authors try different architectures for phase one, instead of using LSTM? "

Response: Yes, we have experimented with both Transformers and GRUs. Transformers performed worse than LSTMs due to the limited size of our dataset, as they require more data to effectively capture long-term dependencies without overfitting. GRUs did not offer substantial improvements, so we opted for a 2-layer LSTM to keep the aligning model simple and better suited to our data constraints.

[Q2]: "How many sentences/trials are used evaluated to obtain current results? "

Response: Our dataset consisted of 30 trials and 419 unique sentences. We used the last trial of each story for evaluation (6 trials = 89 sentences)

[Q3]: "What could be the upper bound of the performance, especially when more data is provided? "

Response: While it's challenging to estimate a precise upper bound, our scalability experiments (Section 4.4) suggest that performance improves significantly with more training data, as larger datasets enable better alignment between neural and text embeddings. The point you raised suggests a promising path for future research.

[Q4]: "Why not include model from Wang et al., 2024 as baseline comparison? "

Response: The approach used in (Wang et al., 2024), is identical to that of (Tang et al, 2023)--our current baseline.

[Q5]: "What is the intuition behind using both Triplet loss and the contrastive loss since they are serving the same function? "

Response: While both Triplet and Contrastive losses aim to align neural embeddings with text embeddings, they complement each other by optimizing different aspects of the alignment. Contrastive loss ensures global separation of matching and non-matching pairs, while Triplet loss enforces a margin between positive and negative pairs, improving local embedding consistency. Empirically as we showed in the appendix combination of the two results in better alignment.

[Q6]: "It is interesting that, using phase 2 alone, without any alignment between text and neural signal, the model can still generate some outputs with comparable BLUE score to the full model, could author describe more on this? Also how the random control baseline is performed? "

Response: While the BLEU score for Phase 2 alone is approximately half of the full model, it is relatively high because fine-tuning the corrector on the training stories allows it to adapt stylistically, leading to text that resembles the story context regardless of the neural embeddings. However, Phase 1 is critical for aligning neural signals with text embeddings, ensuring the decoded sentences correspond to the actual neural data. For the random control baseline, we used untrained models in both phases, which produced irrelevant text, confirming that both alignment and fine-tuning are necessary for meaningful semantic decoding.

We sincerely appreciate your thoughtful review and the valuable feedback you've provided on our manuscript. Below, we address your concerns in detail:

[W1] "The experiment is not adequate to claim that this is a solid SOTA for brain-to-text translation ..."

Response: Thank you for pointing out these important considerations. We do not claim SOTA but aim to demonstrate the potential of our transfer learning paradigm, particularly in low-data settings like iEEG. To address concerns about the role of the language model (corrector), we have conducted experiments to ensure the performance is not solely due to its fluency. As shown in Table 1, random trial baselines (with models untrained in Phases 1 and 2) generate irrelevant sentences, and Phase 2 alone achieves only half the BERT and BLEU scores of the full model, underscoring the importance of the alignment phase. While our training and test examples in Section 3.2 included contextually similar sentences (e.g., about moving out for college), Section 3.3 demonstrates that the model generalizes well even to entirely unseen stories. This highlights the robustness and promise of our approach.

[W2] "The experimental part lacks the cross-session and the cross-subject analysis. Only 30 minutes of data may lead to a severe over-fitting problem."

Response: We agree that our study does not include cross-session or cross-subject analyses. However, to enhance the coverage of the brain in our experiments, we combined the electrode responses from multiple subjects to form a "super-subject." This approach allowed us to leverage a broader range of neural signals, which is particularly valuable in iEEG studies where electrode coverage is often sparse. While cross-subject validation remains an important direction for future work, this methodology was essential to demonstrate the potential of our framework with limited data.

[W3] "The novelty is very limited ..."

Response: We acknowledge the release of BrainECHO on October 19th, 2024, just 11 days before the ICLR deadline, and consider it concurrent work. While BrainECHO focuses on non-invasive EEG/MEG decoding via audio spectrogram alignment, our work addresses semantic text reconstruction from invasive iEEG recordings, emphasizing alignment of brain responses to semantic representations. The concurrent timing underscores the growing interest in this field and does not diminish the novelty of our transfer learning-based approach, which is tailored for low-data iEEG settings. We view these works as complementary advancements in brain-to-text decoding.

[Q1] "What is the key difference between doing the iEEG-to-text translation rather than EEG-to-text translation?"

Response: The key difference lies in the nature of the data and its applications. iEEG signals have a higher signal-to-noise ratio and quality compared to EEG, making them more suitable for precise decoding tasks, particularly as future BCI devices like Neuralink are expected to be invasive. While iEEG data availability is limited due to its invasive nature, our contribution lies not in introducing a new modality but in leveraging a transfer learning paradigm combined with a direct text embedding-to-text approach (Vec2Text). This highlights the significant potential of transfer learning in low-data settings for advancing BCI applications.

We sincerely appreciate your thoughtful review and the valuable feedback you've provided on our manuscript. Below, we address your concerns in detail:

[W1] "This paper only compares one baseline, and this baseline is very poor."

Response: As discussed in the Discussion section of our manuscript, we compared our approach primarily with Tang et al. (2023) due to its status as a recent state-of-the-art in semantic decoding and its methodological alignment with our objectives, it's important to note that their model was trained on an extensive dataset of 17 hours of fMRI data. In contrast, our Neuro2Semantic framework leverages transfer learning to achieve superior performance with as little as 30 minutes of iEEG data. This demonstrates the effectiveness of our approach in low-data settings like ECoG studies, where collecting extensive neural data is inherently challenging.

[W2] "Model performance is highly dependent on the quality and relevance of pre-trained text embeddings ..."

Response: We acknowledge this limitation in our manuscript, attributing it partly to the limited size of our dataset. Nonetheless, our out-of-domain performance results show that even with these constraints, the model captures semantic content effectively, despite not always fully reconstructing syntactic structures. Additionally, our scalability experiments indicate that with larger datasets and more diverse vocabularies, the model's generalization capabilities would improve significantly, enhancing its effectiveness in domain-specific applications.

[Q1] "What is the reason for the poor performance of baseline, especially much worse than the reported values?"

Response: Thank you for raising this question. To clarify, could you specify which "reported values" you are referring to? Are you comparing the baseline's performance in our paper to the results reported in its original paper

[Q2] "Have the authors explored other more efficient sequence processing models (such as Transformers or GRUs) to improve the capture and computational efficiency of long-term dependencies?"

Response: Yes, we have experimented with both Transformers and GRUs. Transformers performed worse than LSTMs due to the limited size of our dataset, as they require more data to effectively capture long-term dependencies without overfitting. GRUs did not offer substantial improvements, so we opted for a 2-layer LSTM to keep the aligning model simple and better suited to our data constraints.