Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models

We thank the reviewer for their insightful and valuable feedback. We appreciate the reviewer's comments that our proposed method is well-motivated and delivers superior performance over strong baselines with thorough experiments. We addressed all the concerns of the reviewer below and revised the manuscript accordingly.

W1: re: novelty of our methods

We address the first and second weakness below by clarifying the novelty of our work, which includes the technical contributions of the proposed methods and valuable insights for the neuroscience domain.

1. We would like to emphasize that DCoND is not simply an application of the diphone concept from the ASR community. Unlike existing ASR models, DCoND does not simply decode diphones - it uses diphone decoding as an intermediate step to decode monophones. The main task of modeling monophone probabilities is broken down into subtasks of modeling diphone probabilities, after which the marginalization over diphones is done to obtain the monophone probabilities. This newly introduced marginalization step, which integrates diphone and monophone decoding in a unified framework, represents the primary novelty that sets DCoND apart from traditional ASR methods.

2. Additional novelty in our work is in the non-trivial application of context-dependent representations in the neuroscience domain. To the best of our knowledge, no previous studies have demonstrated that neural activity underlying attempted speech contains and can be decoded into context-dependent representations, let alone leveraging them to enhance performance of brain-to-text decoding models. DCoND-L with WER of 8.06 outperforms NPTL with an 14.80% improvement, proving the significance of using context-dependent representations for neural signals. Our novelty on this front provides both scientific insights and algorithmic solutions to the brain-to-text decoding problem, bringing significant values for the neuroscience community at ICLR.

We will summarize and outline these novelties in the revised version of the paper.

W2: re: novelty regarding our use of LLMs for error correction

Although LLMs have been used for ASR correction, decoded phonemes are hardly used as references for correction. In our paper we demonstrate that decoded phonemes can also contribute to the accuracy of transcription error correction, where DCoND-LIFT with phoneme inputs reaches an average WER of 5.9 vs. 6.13 without phoneme inputs. We would like to re-emphasize that our contributions on the transcription error correction include enhancements particularly geared towards the data-limited regime of neuroscience applications via the efficient use of existing available resources:

• (i) incorporating decoded phonemes as an informative source of reference for language models to reduce the error accumulation induced during the use of LLMs for error correction.
• (ii) combining multiple LLMs working as a multi-agent system, each specialized for distinct steps in the word decoding pipeline, to improve the overall word decoding accuracy.
• (iii) demonstrating the efficiency of in-context learning paradigm for rapid inference in resource-constrained settings. These enhancements together efficiently and significantly improve WER of the brain-to-text pipeline, with 18.4% and 35.4% improvements over the best baseline for DCoND-LI and DCoND-LIFT, respectively.

Q1&Q2: re: benefits of the marginalization step

We appreciate the reviewer’s suggestion for additional analyses regarding the benefits of the marginalization step. Below we provide additional findings to demonstrate the effectiveness of collaboratively decoding diphones and monophones through marginalization (as in DCoND) vs. decoding diphones alone (as in ASR methods). Indeed, empirical results demonstrate that decoding monophone with diphone marginalization (DCoND) resulted in a better PER than decoding using diphones alone. When coupled with a 3-gram language model that transcribes monophone sequences to word sequences, DCoND also yields a better WER than a diphone model followed by a 3-gram language model designed for diphone-to-word transcription.

The superior performance of DCoND can be attributed to its marginalization formulation, which allows for more imperfect diphone predictions - as long as the marginalized probability attains the highest value at the correct monophone, the model does not need to assign the highest probability at the correct diphone. When the amount of paired neural-text data is insufficient to train a good diphone decoding model - as is the case in the neuroscience domain - the marginalization step in DCoND proves to be an effective strategy.

W3: re: elaboration of the Divide-and-Conquer method

We thank the reviewer for the suggestion to provide further details regarding our Divide-and-Conquer strategy. We included the empirical results (please see further description in response to W1) and additional intuitive explanation on why DCoND is better than a model that decodes diphones only. We will revise the manuscript further to include a primer on the Divide-and-Conquer method upon preparing the final version of the manuscript.

Q1: re: technical differences between brain-to-text and speech-to-text

Our technical differences lie in the newly developed DCoND algorithm for brain-to-phonemes neural decoder as well as the multi-agent LLM system for phoneme-to-text transcription. These components are unique and critical for brain-to-text decoding and have not appeared in speech-to-text context. We provided more information on these in our response to W1 and the original manuscript (please see ‘Brain-to-Text Decoding vs. Speech-to-Text Decoding’ in the Related Work section). We will make sure to include further discussion in related work upon preparing the final version of the manuscript.

We thank the reviewer for their valuable feedback. We appreciate the reviewer comments that our proposed method is grounded in neuroscientific insights and achieves state-of-the-art performance in the Brain-to-Text decoding benchmark. We have addressed all the raised concerns below and revised the manuscript accordingly.

W1: re: novelty of our methods

1. We would like to emphasize that DCoND is not simply an application of the diphone idea from the ASR community. Unlike existing ASR models, DCoND does not simply decode diphones - it uses diphone decoding as an intermediate step to decode monophones. The main task of modeling monophone probabilities is broken down into subtasks of modeling diphone probabilities, after which the marginalization over diphones is done to obtain the monophone probabilities. This newly introduced marginalization step, which integrates diphone and monophone decoding in a unified framework, represents the primary novelty that sets DCoND apart from traditional ASR methods.

   We provide additional experiments to demonstrate the effectiveness of collaboratively decoding diphones and monophones through marginalization (as in DCoND) vs. decoding diphones alone (as in ASR methods). Indeed, empirical results below demonstrate that decoding monophone with diphone marginalization (DCoND) resulted in a better PER than decoding using diphones alone. When coupled with a 3-gram language model that transcribes monophone sequences to word sequences, DCoND also yields a better WER than a diphone model followed by a 3-gram language model designed for diphone-to-word transcription.

   	Diphone	DCoND
   PER	19.14 $\pm$0.08	15.44$\pm$0.46
   WER	12.73$\pm$0.22	11.79 $\pm$0.30

   The superior performance of DCoND can be attributed to its marginalization formulation, which allows for more imperfect diphone predictions - as long as the marginalized probability attains the highest value at the correct monophone, the model does not need to assign the highest probability at the correct diphone. When the amount of paired neural-text data is insufficient to train a good diphone decoding model - as is the case in the neuroscience domain - the marginalization step in DCoND proves to be an effective strategy.

2. Additional novelty in our work is in the non-trivial application of context-dependent representations in the neuroscience domain. To the best of our knowledge, no previous studies have demonstrated that neural activity underlying attempted speech contains and can be decoded into context-dependent representations, let alone leveraging them to enhance performance of brain-to-text decoding models. DCoND-L with WER of 8.06 outperforms NPTL with an 14.80% improvement, proving the significance of using context-dependent representations for neural signals. Our novelty on this front provides both scientific insights and algorithmic solutions to the brain-to-text decoding problem, bringing significant values for the neuroscience community at ICLR.

3. Our contributions in the phonemes-to-words pipeline involve enhancements particularly geared towards the data-limited regime of neuroscience applications via the efficient use of existing available resources:

   (i) combining multiple LLMs working as a multi-agent system, each specialized for distinct steps in the word decoding pipeline, to improve the overall word decoding accuracy.

   (ii) incorporating decoded phonemes as an informative source of reference for language models to reduce the error accumulation in the multi-agent system.

   (iii) demonstrating the efficiency of in-context learning paradigm for rapid inference in resource-constrained settings.

   These enhancements together efficiently and significantly improve WER of the brain-to-text pipeline, with 18.4% and 35.4% improvements over the best baseline for DCoND-LI and DCoND-LIFT, respectively.

We will summarize and outline these novelties in the revised version of the paper.

W2: re: relevance to the ICLR community

As we elaborated in our response above, our paper offers unique technical novelties and scientific insights, not simply a trivial application of existing technologies. We submitted our paper under the “Applications to neuroscience & cognitive science” subject area of ICLR, as we believe our contributions offer technical novelties that bridge neuroscience and machine learning. These values align closely with the ICLR community’s interest in advancing interdisciplinary research.

We thank the reviewer for their insightful and valuable feedback. We appreciate the reviewer’s comments that our proposed method may be beneficial for other fields and includes interesting analyses. We have addressed all the raised concerns below and revised the manuscript accordingly, including updates to the notations and technical descriptions.

W1: re: notation for Z

We thank the reviewer for pointing out the issue in the current definition of Z. We have revised the definition to $Z \in \mathbb{Z}^{T’}$, to indicate that Z takes discrete values indicating phoneme classes.

W2: re: earlier introduction of CTC loss in the text

We appreciate the reviewer’s suggestion of mentioning the CTC first in the method description to improve the clarity of the text. We have revised the paper to mention the CTC loss when introducing the GRU model.

W3: re: notations for dimensions of X and Z

We thank the reviewer for their suggestion to clarify the dimensions of the variables. X is a two dimensional matrix ($X \in R ^{T \times D}$), $Z$ is a one dimensional vector with length $T’$ as introduced in the Problem Formulation. For simplicity, equation 1 only shows the prediction for the monophone $Z$ at a single timestep, i.e. $T’ = 1$. We have added clarification for the dimensionality in the revised manuscript as the reviewer suggested.

W4: re: how our method handles sparsity issues of diphones and triphones

The sparsity (percentage of diphone/triphone classes that don’t exist in the training set) is 36.97% for diphones and 88.72% for triphones. The sparsity is addressed for both diphones and triphones by our proposed marginalization method. For the diphone case, the 1600 diphone classes are reduced to 40 monophone classes by the marginalization process, where the monophone loss on these 40 classes are computed. This incorporation of monophone loss beside the diphone loss helps the DCoND model tolerate some errors on the diphone predictions, making DCoND less susceptible to the sparsity issue brought about by the use of diphones.

To deal with sparsity issues in triphones, in Table 3 we have shown the performance of a clustering method which shares a similar idea to the phonetic decision tree clustering method the reviewer suggested. We clustered 40 phonemes into 14 classes based on articulation similarity, as proposed by Herff et al. [1]. Using these groupings, we constructed triphones with the structure: $cluster\_j \rightarrow phoneme\_i \rightarrow cluster_q$, where there are 14, 40, and 14 classes for $cluster_j$, $phoneme_i$, $cluster_q$, respectively. We included more details in Appendix A.2. We showed in Table 3 that this grouping approach significantly underperformed DCoND, with PER and WER being 28.55 and 13.98, respectively, significantly worse than 15.34 PER and 8.06 WER of DCoND.

W5: re: the presence of SIL between words

The dataset used in our study was collected from a patient with Amyotrophic Lateral Sclerosis (ALS). The patient, due to their medical condition, is unable to speak swiftly. Therefore the produced speech contains noticeable SIL between words. Prior works have demonstrated that including SIL as a decoding target improves overall decoding performance compared to models that do not explicitly account for SIL [2]. These findings provide a baseline for our approach to support the inclusion of SIL in generating the representation of output.

W6: re: typo in section 4.3

We thank the reviewer for pointing out the typo. We have changed the reference to Table 1 in the revised manuscript.

[1] Christian Herff, Dominic Heger, Adriana De Pesters, Dominic Telaar, Peter Brunner, Gerwin Schalk, and Tanja Schultz. Brain-to-text: decoding spoken phrases from phone representations in the brain. Frontiers in neuroscience, 9:217, 2015.

[2] Willett, Francis R., Erin M. Kunz, Chaofei Fan, Donald T. Avansino, Guy H. Wilson, Eun Young Choi, Foram Kamdar et al. "A high-performance speech neuroprosthesis." Nature 620, no. 7976 (2023): 1031-1036.

W2: re: efficiency and effectiveness of our LLM pipeline

Indeed, the current phoneme-to-word pipeline, LIFT, involves multiple steps. However, each integrated language model (LM) plays distinct roles in the LIFT pipeline (transcription generation, re-scoring, and error correction) which collectively contributes to the overall performance. In our Ablation Study (‘Contribution of LLMs’ in Section 4.5 and Appendix A.3) we demonstrated with extensive experiments how removing one of these steps would degrade the overall WER.

In particular, when only 5-gram LM is used, DCoND-5gram achieves WER of 10.05 vs. 5.77 if using the full pipeline (Figure 5); when GPT3.5 is removed (DCoND-L), the WER is 8.06 (Table 1). Additionally, we carried out an experiment where the re-scoring step was removed, and observed that WER increased from 5.77 to 7.15. These results indicate that each LM has its own distinct contribution in the overall decoding performance. A more efficient way using a single LM/LLM to achieve the current performance can be challenging, but could be an interesting direction for future research.

Q1: re: architecture details

Yes, the reported results in the paper are obtained with a 5-layer RNN (GRU) as the model architecture for DCoND. In addition to GRU, we also conducted experiments with Transformer and LSTM architectures and included detailed results in Table 6 (Appendix A.5). We would like to emphasize that the novelty of DCoND is not on the neural decoder architecture, but rather on the algorithmic level (decoding monophones through marginalization of diphones) which is applicable to various neural decoders. Indeed Table 6 in Appendix A.5 shows that DCoND marginalization method is model-agnostic and consistently outperforms the monophone decoding baseline across all architecture designs, demonstrating its robustness and versatility in accommodating diverse architectural choices.

We appreciate the reviewer’s suggestion to move the implementation details to the experimental section. We have included the implementation details in Appendix A.6 and will also mention the architecture in the experimental section to enhance the clarity of the paper.

Q2: re: generalizability of the hyperparameter balancing diphone and monophone losses

The optimal value of the hyperparameter $\alpha$ which represents the tradeoff between diphone loss and monophone loss can vary across different datasets.

We thank the reviewer for their insightful and valuable feedback. We appreciate the reviewer’s comments that our paper is well-organized, providing strong results and can make a difference. We address the reviewer's concerns below regarding novelties and technical contributions.

W1: re: novelty of our methods

1. We would like to emphasize that DCoND is not simply an application of the diphone idea from the ASR community. Unlike existing ASR models, DCoND does not simply decode diphones - it uses diphone decoding as an intermediate step to decode monophones. The main task of modeling monophone probabilities is broken down into subtasks of modeling diphone probabilities, after which the marginalization over diphones is done to obtain the monophone probabilities. **This newly introduced marginalization step, which integrates diphone and monophone decoding in a unified framework, represents the primary novelty that sets DCoND apart from traditional ASR methods.

   We provide additional experiments to demonstrate the effectiveness of collaboratively decoding diphones and monophones through marginalization (as in DCoND) vs. decoding diphones alone (as in ASR methods). Indeed, empirical results below demonstrate that decoding monophone with diphone marginalization (DCoND) resulted in a better PER than decoding using diphones alone. When coupled with a 3-gram language model that transcribes monophone sequences to word sequences, DCoND also yields a better WER than a diphone model followed by a 3-gram language model designed for diphone-to-word transcription.

   	Diphone	DCoND
   PER	19.14$\pm$0.08	15.44$\pm$0.46
   WER	12.73$\pm$0.22	11.79$\pm$0.30

   The superior performance of DCoND can be attributed to its marginalization formulation, which allows for more imperfect diphone predictions - as long as the marginalized probability attains the highest value at the correct monophone, the model does not need to assign the highest probability at the correct diphone. When the amount of paired neural-text data is insufficient to train a good diphone decoding model - as is the case in the neuroscience domain - the marginalization step in DCoND proves to be an effective strategy.

2. Additional novelty in our work is in the non-trivial application of context-dependent representations in the neuroscience domain. To the best of our knowledge, no previous studies have demonstrated that neural activity underlying attempted speech contains and can be decoded into context-dependent representations, let alone leveraging them to enhance performance of brain-to-text decoding models. DCoND-L with WER of 8.06 outperforms NPTL with an 14.80% improvement, proving the significance of using context-dependent representations for neural signals. Our novelty on this front provides both scientific insights and algorithmic solutions to the brain-to-text decoding problem, bringing significant values for the neuroscience community at ICLR.

3. Our contributions in the phonemes-to-words pipeline involve enhancements particularly geared towards the data-limited regime of neuroscience applications via the efficient use of existing available resources:

   (i) combining multiple LLMs working as a multi-agent system, each specialized for distinct steps in the word decoding pipeline, to improve the overall word decoding accuracy.

   (ii) incorporating decoded phonemes as an informative source of reference for language models to reduce the error accumulation in the multi-agent system.

   (iii) demonstrating the efficiency of in-context learning paradigm for rapid inference in resource-constrained settings.

   These enhancements together efficiently and significantly improve WER of the brain-to-text pipeline, with 18.4% and 35.4% improvements over the best baseline for DCoND-LI and DCoND-LIFT, respectively.

We will summarize and outline these novelties in the revised version of the paper.