Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding

Thank you for the feedback. We appreciate that the Reviewer finds our paper easy to follow and finds the use of time masking to reduce overfitting interesting. We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text.

General Note: We realized after submission that all the entries in Willett et al. 2025 [b] used a bidirectional GRU. A bidirectional GRU can only decode speech once the entire trial has finished, and roughly doubles memory usage, training times, and calibration times relative to the unidirectional GRU. All our models are either unidirectional GRUs or causal Transformers which only rely on present and past neural activity to generate outputs.

Training time takes longer when using time masking: Our time-masked Transformer showed no drop in performance when trained for the same time as the entries which used GRUs.

• Additional Details: We find that a $5$-layer time-masked Transformer trained for $250$ epochs (~1 hour of training time) exhibits slightly better validation WER than our original $7$-layer time-masked Transformer trained for either $600$ epochs or $250$ epochs. This may be because the smaller Transformer trained for less epochs has a lower validation CTC loss than the larger variant. We had previously only focused on validation phoneme error rate (PER), as done in Willett et al., 2023 [a]. This Transformer requires roughly the same amount of training time as the baseline bidirectional GRU and about half the training time as the Linderman Lab GRU, which was the best performing entry with only modifications to the GRU in Willett et al., 2025 [b] (Table 1).

Evaluate whether time-masked GRU can process non-overlapping inputs, and if this yields efficiency gains: A time-masked GRU trained with non-overlapping inputs and a smaller hidden state size reduces parameter count by $35$ M relative to the original GRU architecture while maintaining competitive performance with our original time-masked GRU.

• Additional details: Our original hypothesis was that the GRU would not perform optimally with non-overlapping inputs due to its lossy hidden state. Similar to Willett et al., 2023 [a], we found this was the case with the original GRU size. However, we found that performance with non-overlapping inputs improved when reducing the hidden size, suggesting that models with non-overlapping inputs generalize better when smaller. We will continue optimizing the GRU with non-overlapping inputs for efficiency gains, and update our manuscript to discuss how processing non-overlapping inputs may be a good heuristic to follow when designing lightweight architectures.

Results are not displayed with the 5-gram LM setup: We have included results using the updated models with the 5-gram LM setup, which show that our models outperform strong baselines in a real-time decoding setting, and even match the performance of strong baselines which rely on bidirectional GRUs (Table 1).

• Additional details: We have added results with the offline language modeling setup used by the entries in the competition (Willett et al., 2025 [b]). Specifically, the offline language modeling setup involved using a 5-gram LM, with a second pass using the 5-gram LM, and rescoring with OPT 6.7B. We refer to the setup where only the 3-gram LM is used with no second pass as the online LM setup. Results are displayed in Table 1.

Is the difference between CORT and No CORT similar with WER?: We have revised our test-time adaptation method to a computationally lightweight pseudo-label based approach, which was inspired by Fan et al., 2023 [c], and there is a substantial reduction in WER when applying our improved TTA method relative to no TTA (Table 2 and Table 3).

• Additional details: While we did find a small reduction in WER when applying CORT relative to No CORT across days (average WER improvement across days shown in Figure 4, left), this improvement was small. We thus sought to examine if applying time masking and Transformers could provide efficiency gains to the pseudo-label based TTA method proposed by Fan et al., 2023 [c]. The method used by Fan et al., 2023, CORP, adapts a model at test-time by treating the language model (LM) refined model outputs as pseudo-labels, and then training the model with CTC loss using these pseudo-labels. Since the LM refined outputs are already being generated for real-world use, there is no additional cost to generate these pseudo-labels. However, CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. Here, we present "DietCORP", a lightweight version of CORP which only uses one adaptation step per trial and no previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table 2), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 3). Only the patch embedding module was adapted. DietCORP leads to very stable WER across days relative to when not applying it with minimal hyperparameter tuning.

Reductions in memory usage and calibration times are marginal: With our updated TTA method, DietCORP, we reduce peak GPU memory usage by $81$% and calibration time by $82$% relative to the bidirectional GRU.

• Additional details: DietCORP only requires $1.04$ GiB of peak allocated GPU memory usage and $29.3$ ms to calibrate per trial using the Transformer. When applying our TTA method to the unidirectional GRU, it requires $2.8$ GiB of peak allocated GPU memory usage and $76.9$ ms to calibrate per trial. These values further increase to $5.6$ GiB and $163.0$ ms when using the bidirectional GRU. We will also include comparisons when using CORP in the updated manuscript, where DietCORP applied to the Transformer exhibits even larger computational gains. We accept that the reductions in memory usage and training times are perhaps marginal for offline training, where models are often trained using large servers while the neuroprosthesis is not in use. However, we believe these efficiency gains are important in the context of test-time adaptation, where calibration is performed on-device in the background while the patient is using the neuroprostheses.

Tables

Table 1: We show results with the $5$-layer time-masked Transformer trained for $250$ epochs (Transformer - Ours) as well as comparisons with the baseline provided by Willett et al., 2023 [a] and the Linderman Lab GRU discussed in Willett et al., 2025 [b]. We run the baseline and Linderman Lab GRU in unidirectional mode, and show metrics in bidirectional mode in parenthesis when available.

Table 2: DietCORP is our computationally lightweight version of CORP, a test-time adaptation method proposed by Fan et al., 2023 [c]. Values are word error rate (mean ± SEM across $N=4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days.

Table 3: Values are word error rate (mean ± SEM across $N=4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days.

[a] Willett et al., 2023, A high-performance speech neuroprosthesis

[b] Willett et. al, 2025, Brain-to-Text Benchmark '24: Lessons Learned

[c] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication

Thank you for the feedback. We appreciate that the Reviewer noted our focus on optimizing performance in a real-time streaming setting. We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text.

General Note: We realized after submission that all the entries in Willett et al. 2025 [b] used a bidirectional GRU. A bidirectional GRU can only decode speech once the entire trial has finished, and roughly doubles memory usage, training times, and calibration times relative to the unidirectional GRU. All our models are either unidirectional GRUs or causal Transformers which only rely on present and past neural activity to generate outputs.

Low performance on benchmark: When using the 5-gram language modeling setup used by the entries in the Brain-to-Text Benchmark ‘24, the time-masked Transformer achieves a mean WER of 8.01%, which ties the 2nd place entry (Linderman Lab GRU) despite using ~$15$ times less parameters and decoding in real-time. (Table 1).

• Additional Details: We have added results (Table 1) with the offline language modeling (LM) setup used by the entries in Willett et al., 2025 [b]. Specifically, the offline language modeling setup involved using a 5-gram LM, with a second pass using the 5-gram LM, and rescoring with OPT 6.7B. We refer to the setup where only the 3-gram LM is used with no second pass as the online LM setup.

DConD-LIFT achieves 5.77% WER: The primary performance gain in the first place entry, DConD-LIFT, stems from applying GPT 3.5, while their main improvement to the GRU itself (decoding diphones) achieves a WER of 8.06%.

• Additional Details: DConD-LIFT (Li et al., 2023) shows remarkable accuracy gains by applying GPT 3.5 to an ensemble of bidirectional GRUs. However, since their submission requires access to cloud resources and is not streaming compatible, we believe a comparison with our method strictly based on accuracy is not entirely fair.

Benchmark time masking against other regularization strategies: We have included a comparison with the Linderman Lab GRU which incorporates speckled masking (i.e. input dropout) in Table 1, and our original study also included comparisons with other regularization strategies (channel masking, additive white noise, baseline shifts).

• Additional Details: We ran the Linderman Lab GRU, which uses specked masking (i.e. input dropout), in unidirectional mode. The time-masked Transformer performs substantially better than the Linderman Lab GRU in a real-time decoding setting (Table 1), and matches the performance of the bidirectional Linderman Lab GRU (Table 1). Furthermore, the baseline GRU from Willett et al., 2023 [a] used high amounts of additive white noise and baseline shift augmentations, providing a strong comparison with our proposed time masking augmentation. We also incorporated speckled masking (denoted as input dropout in our manuscript), additive white noise, and baseline shifts when training models with time masking. We mentioned these details in the Methods and Appendix, and will state them more clearly in the updated manuscript. Regarding SpecAugment, we note that SpecAugment consists of three parts: time masking, channel masking, and time warping on an audio spectrogram. Time masking was a major focus of our study, and we also provided preliminary using channel masking in Appendix Section 7.3. We did not implement time warping since it had a minor effect in the original SpecAugment study (Park et al., 2023 [e]), and we did not implement speed perturbation since we are working with data from one patient who speaks at a relatively constant rate.

Comparison with existing test-time adaptation (TTA) methods: The main TTA method in the field of communication neuroprostheses is continual online calibration with pseudo-labels (CORP) (Fan et al., 2023 [c]), and in our updated manuscript we present a computationally lightweight version of CORP.

• Additional Details: In CORP, the model is adapted at test-time by treating the language model refined model outputs as pseudo-labels. CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. We propose "DietCORP", a lightweight version of CORP which only requires one adaptation step per test trial and does not require maintaining previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table $2$), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 3). DietCORP leads to stable WER across days relative to when not applying it. DietCORP only requires $1.04$ GiB of peak allocated GPU memory usage and $29.3$ ms to calibrate per trial using the Transformer, versus $5.6$ GiB and $163$ ms to calibrate when using the bidirectional GRU. We will provide direct comparisons with CORP in the updated manuscript.

Only the patch embedding module is updated for TTA: We only updated the patch embedding module because it requires less resources, which is important for edge devices, and we will show additional results when adapting a larger fraction of the weights in the updated manuscript.

Mathematical Formulation for TTA: We will provide a detailed mathematical formulation of our updated TTA method in the manuscript.

Theoretical investigation into why Transformers are more efficient than GRUs: We hypothesize that Transformers are more efficient because they were optimized to process non-overlapping inputs, and show efficiency gains with a GRU that processes non-overlapping inputs.

• Additional Details: When optimizing the time-masked GRU to process non-overlapping inputs, we obtain competitive performance with a GRU that uses less than half the parameters as the baseline GRU ($21$ M versus $56$ M). We will provide a discussion of this in the updated manuscript.

Exploratory analysis of day-to-day shifts in neural activity: We will discuss previous work (Willett et al., 2021 [f]) which shows that day-to-day shifts in neural activity can be represented as rotations from the original space into a new space in our updated manuscript.

Optimal hyperparameters for time masking: We show results with other time masking hyperparameters in Appendix Section 7.3, and will reference this more clearly in the updated manuscript.

Dorsal 6v: The time-masked Transformer performs better than the baseline GRU when using only Dorsal 6v, which we will include in the updated manuscript.

Tables

Table 1: We show results with the time-masked Transformer trained for 250 epochs (Transformer - Ours) as well as comparisons with the baseline provided by [a] and the Linderman Lab GRU discussed in [b]. We run the baseline and Linderman Lab GRU in unidirectional mode, and show metrics in bidirectional mode in parenthesis when available.

Table 2: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days.

Table 3: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days.

[a] Willett et al., 2024, A high-performance speech neuroprosthesis

[b] Willett et. al, 2025, Brain-to-Text Benchmark '24: Lessons Learned

[c] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication

[d] Li et al., 2024, Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models

[e] Park et al., 2019, SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

[f] Willett et al., 2021, High-performance brain-to-text communication via handwriting

We thank the reviewer for the response and are very appreciative of the increase in score. We would like to make several clarifications regarding the points the reviewer brought up, which we detail below. Reviewer comments are in quotes.

Note: In our original response, we used the term offline LM setup when using the 5-gram LM, second pass with the 5-gram LM, and rescoring with OPT 6.7B, and the term online LM setup to refer to only using the 3-gram LM. We used these terms because they were used by Willett et al., 2023. However, as pointed out by Reviewer YQY9, later work has used the “offline” setup for online experiments. We thus replace the term offline LM setup with 5-gram + 2nd pass + OPT setup and the term online language modeling setup with 3-gram setup in future responses.

"In terms of clarity and significance, while the method achieves performance comparable to the Linderman Lab GRU and offline language model setups, the paper does not clearly articulate the benefits in the online LM setting."

We would like to clarify that the causal, real-time decoding time-masked Transformer achieves performance comparable to the non-causal, bidirectional Linderman Lab GRU. When comparing both models in a real-time decoding setting, our model performs roughly $25$% better than the Linderman Lab GRU when using the 5-gram + 2nd pass + OPT setup (originally referred to as offline LM setup, see note above) and the 3-gram setup (originally referred to as online LM setup). These benefits were stated for both of these setups in Table 1 in our previous response.

"The focus remains mainly on the model architecture rather than the practical implications of the online formulation. This limits the clarity and may lead to an overstatement of the method’s contributions."

The improvement on model architecture, specifically replacing the Transformer with the GRU, is only out of three of our core contributions. We also show that time masking is a highly effective augmentation on this benchmark and propose a computationally lightweight while effective test-time adaptation method by leveraging our two previous contributions. These contributions together offer a neural speech decoding algorithm that improves accuracy while reducing computational costs associated with test-time adaptation. We thus believe our contributions offer clear practical implications that are well-supported by the empirical data presented.

"Furthermore, the claim that Transformers outperform GRUs is not convincingly supported by the presented evidence."

In the manuscript we submitted to NeurIPS we acknowledged that switching from Transformers to GRUs only leads to a small accuracy gain. Therefore, our focus is not on the fact that Transformers outperform GRUs in terms of accuracy. The primary benefit of Transformers over GRUs is that they require less memory to train and have faster calibration speeds, which makes them ideal for test-time adaptation. As noted in our previous response, applying DietCORP to Transformers leads to an $81$% reduction in peak GPU memory usage and an $82$% reduction in calibration time relative to applying DietCORP to bidirectional GRU. When comparing to a unidirectional GRU, the Transformer still reduces peak memory usage by $63$% and calibration times by $62$%. We therefore believe that our claims about the benefit of the Transformer, specifically its suitability for test-time adaptation, are well supported by the present evidence.

"Regarding quality, questions remain about the use of Dorsal 6v and the role of bidirectional GRUs.".

We ran analyses with Dorsal 6v only and found that time-masked Transformer maintains its performance advantage over the baseline GRU, as noted in our previous response. We are happy to perform additional analyses regarding Dorsal 6v if the reviewer has additional questions. As we stated in our General Note, bidirectional GRUs were used by the entries in Willett et al., 2025, and we make this explicit because our proposed decoding algorithm operates in real-time whereas bidirectional GRUs must wait for the entire trial to finish. Therefore, the role of bidirectional GRUs is mainly to compare our performance in the more challenging, real-time (streaming) setting against previous entries that operated in the non-real-time (non-streaming) setting. We agree that in the ideal case there would be a separate benchmark track for streaming decoding algorithms and non-streaming algorithms.

"Additionally, while time masking is central to the proposed approach, the paper lacks a hyperparameter sensitivity analysis. Since such methods are known to be sensitive to masking parameters, this omission weakens the empirical grounding of the results."

As we stated in our previous response under the title Optimal hyperparameters for time masking, we included this information in our original submission in Appendix Section $7.3$.

We will show an increased hyperparameter tuning range in the updated manuscript. We address each point below.

"However, the explored range appears to be distributed on both sides of the spectrum, making it difficult to tune effectively."

We are unsure what both sides of the spectrum means. If it means that we tuned the hyperparameters both up and down, we are unsure why this is an issue. We are happy to follow a procedure suggested by the reviewer to tune the hyperparameters.

"Moreover, the range is too limited to observe clear trends, and it remains challenging to identify optimal values."

In our original submission we showed performance across 6 time masking hyperparameters. We found validation PER was stable for a range of time masking choices except when lowering the masking percentage to $2.5\%$ or when increasing the number of masks to $30$. We believe the relative robustness of our method to a variety of time masking hyperparameter is an advantage, rather than the lack of a clear trend being a disadvantage.

"This [lack of hyperparameter tuning for time masking] remains a limitation of the current work."

Increasing our time masking hyperparameter tuning range would only likely increase the accuracy of the time-masked Transformer. Therefore, while we definitely agree with the reviewer that more extensive hyperparameter tuning is beneficial, we believe it does not limit our current claims.

I thanks the authors for their additional response. As the authors pointed out, the paper does include experiments related to hyperparameters. However, the explored range appears to be distributed on both sides of the spectrum, making it difficult to tune effectively. Moreover, the range is too limited to observe clear trends, and it remains challenging to identify optimal values. This remains a limitation of the current work.

Thank you for the feedback. We appreciate that the Reviewer finds our improvement relative to the NPTL 3-gram baseline noteworthy. We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text.

General Note: We realized after submission that all the entries in Willett et al. 2025 [b] used a bidirectional GRU. A bidirectional GRU can only decode speech once the entire trial has finished, and roughly doubles memory usage, training times, and calibration times relative to the unidirectional GRU. All our models are either unidirectional GRUs or causal Transformers which only rely on present and past neural activity to generate outputs.

Low performance on benchmark + Results are not shown with 5-gram LM setup: When using the 5-gram language modeling setup used by the entries in the Brain-to-Text Benchmark ‘24, the time-masked Transformer achieves a mean WER of 8.01%, which ties the 2nd place entry (Linderman Lab GRU) despite using ~$15$ times less parameters and decoding in real-time. (Table 1).

• Additional Details: We have added results (Table 1) with the offline language modeling (LM) setup used by the entries in Willett et al., 2025 [b]. Specifically, the offline language modeling setup involved using a 5-gram LM, with a second pass using the 5-gram LM, and rescoring with OPT 6.7B. We refer to the setup where only the 3-gram LM is used with no second pass as the online LM setup. The only entry that performs substantially better than our entry is DConD-LIFT (Li et al., 2024 [d]) (WER = 5.77%). However, the primary performance gain from DConD-LIFT stems from applying GPT 3.5 to an ensemble of bidirectional GRUs, which requires cloud resources and is not streaming compatible. When focusing on the improvements made to the GRU itself by Li et al., 2024 [d], specifically diphone decoding, their bidirectional GRU achieves a WER of 8.06% which is slightly worse than our much smaller, causal time-masked Transformer.

How would the model perform with the additional advances described in Willett et al., 2025 [b]?: Our preliminary data suggest that incorporating the post GRU block from the Linderman Lab to the Transformer does not improve performance, and our causal time-masked Transformer slightly outperforms the bidirectional GRU which decodes diphones from Li et al., 2024 [c].

Transformer performs with lower accuracy than SOTA: We would like to clarify that the time-masked Transformer outperforms competing GRU-based entries in the real-time decoding setting (Table 1), and that a major point of our study is that time-masked Transformers can decode speech with high accuracy and are well-suited for test-time adaptation.

Relation of phoneme error rate to word error rate is not clear: We will expand our existing discussion of this (in the Limitations) for the updated manuscript.

• Additional details: We used PER to select the best model during validation because that is the procedure used by Willett et al., 2023 [a] and because computing WER during training is computationally expensive. However, we agree that PER is not an ideal metric to optimize for, and we will include cases that highlight this in the updated manuscript. We believe exploring ways to directly optimize WER require a dedicated study.

Technical contributions are not new: Time masking and Transformers were not used by any other entry in Willett et al., 2025 [b], and we further combine these contributions to provide an updated version of our previous TTA method (CORT).

• Additional details: Time masking has been applied in three other neural speech decoding studies to our knowledge ([e,f,g]), however these studies used closed source ECoG datasets, used much smaller amounts of time masking, and did not quantify the impact of time masking relative to other augmentations. We will cite these studies, as well the reference the reviewer provided (Chang et al., 2022), in our updated related works section.

• Previously we combined our two contributions, time masking and Transformers, to propose CORT. However, the improvement in WER from CORT is relatively small. We thus sought to examine if applying time masking and Transformers could provide efficiency gains to the pseudo-label based TTA method proposed by Fan et al., 2023 [c]. The method used by Fan et al., 2023, CORP, adapts a model at test-time by treating the language model (LM) refined model outputs as pseudo-labels, and then training the model with CTC loss using these pseudo-labels. Since the LM refined outputs are already being generated for real-world use, there is no additional cost to generate these pseudo-labels. However, CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. Here, we present "DietCORP", a lightweight version of CORP which only uses one adaptation step per trial and no previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table 2), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 3). Only the patch embedding module was adapted. DietCORP leads to very stable WER across days relative to when not applying it with minimal hyperparameter tuning.

• DietCORP only requires $1.04$ GiB of peak allocated GPU memory usage and $29.3$ ms to calibrate per trial using the Transformer. When applying our TTA method to the unidirectional GRU, it requires $2.8$ GiB of peak allocated GPU memory usage and $76.9$ ms to calibrate per trial. These values further increase to $5.6$ GiB and $163.0$ ms when using the bidirectional GRU. We will also include comparisons when using CORP in the updated manuscript, where DietCORP applied to the Transformer exhibits even larger computational gains.

• In summary, while we agree our three contributions are not entirely novel in the context of machine learning as a whole, we believe they represent a unique application on the first open source neural speech decoding dataset, and have been combined in novel ways. Furthermore, we would like to point out that the NeurIPs guidelines state that applications of existing methods to improve performance are equally as valuable as proposing novel methods.

Tables

Table 1: We show results with a $5$ layer time-masked Transformer trained for $250$ epochs (Transformer - Ours) as well as comparisons with the baseline provided by Willett et al., 2023 [a] and the Linderman Lab GRU discussed in [b]. We run the baseline and Linderman Lab GRU in unidirectional mode, and show metrics in bidirectional mode in parenthesis when available.

Table 2: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days.

Table 3: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days.

[a] Willett et al., 2023, A high-performance speech neuroprosthesis

[b] Willett et. al, 2025, Brain-to-Text Benchmark '24: Lessons Learned

[c] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication

[d] Li et al., 2024, Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models

[e] Metzger et al., 2023, A high-performance neuroprosthesis for speech decoding and avatar control

[f] Metzger et al., 2022, Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis

[g] Littlejohn et al., 2025, A streaming brain-to-voice neuroprosthesis to restore naturalistic communication

After posting the previous comment, we realized we made a slight error in passing the prompts to Llama. When correcting for this error, we achieve a word error rate of 5.98% using the time-masked Transformer + Llama 3.1 8B. This is competitive with the performance DConD-LIFT reports when using bidirectional GRUs and GPT 3.5 of WER = 5.90% +/- 0.08% (mean +/- standard deviation, Table 4) across 5 seeds.

Thank you so much for the positive review! We appreciate that the reviewer found our focus on improving the neural network that maps to phonemes useful.

Thank you for the feedback. We appreciate that the Reviewer finds our test-time adaptation method useful, and we have since made additional improvements to it. We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text.

Motivation for replacing the GRU with the Transformer: Our primary motivation for replacing the GRU with the Transformer was to propose an architecture that was more suitable for test-time adaptation (TTA), where computational efficiency is very important since TTA is typically performed on-device while the participant is using the neuroprosthesis.

• Additional details: A $5$-layer version of our time-masked Transformer, which performs slightly better than our original $7$ layer model, contains less then $10$ M parameters. By comparison, the bidirectional GRU used by the entries in Willett et al., 2025 [b] contain roughly $130$ M parameters. When applying our updated test-time adaptation method (see Updated Test-Time Adaptation Section), we find that the Transformer requires only $1.04$ GiB of peak GPU memory usage and $29.3$ ms to calibrate per trial, whereas the bidirectional GRU requires $5.6$ GiB of peak GPU memory usage and $163$ ms to calibrate per trial (computed on an Nvidia GeForce RTX 3090 GPU). These gains are important for applying TTA on local, resource-constrained devices.

Shallow fusion violates streaming requirements: Shallow fusion is compatible with streaming systems, and was used in a streaming setting in Supplementary Video 1 from the Willett et al., 2023 [a] study.

• Additional details: While we applied the shallow fusion step after the model logits were acquired for convenience, the shallow fusion stage itself does not require access to future logits. During shallow fusion, each beam is constructed one timestep at a time, and the most probable beam for that timestep is selected to be shown on the screen. Therefore, shallow fusion is streaming compatible because text is outputted in real-time, and the algorithm does not need to wait until the entire trial has completed.

How does online recalibration with time masking calibrate in the beginning for a new subject or in other settings: We have included TTA results in a more challenging setting (Table 2).

• Additional details: While we agree that online recalibration to new participants is a very interesting question, there was no other data from other participants at the time we wrote this study. We agree an interesting direction for future work is to calibrate a decoder to a new participant with minimal supervised trials, and then use online recalibration to maintain the performance of this decoder on the new participant. We will include this point in our discussion. We have included a more challenging setting where the model is evaluated on $8$ held-out days in our updated test-time adaptation method section. Under this setting, our updated method is able to maintain stable performance, and the effect of recalibration is quite large (Table 2).

Updated Test-Time Adaptation Method: Using time masking and Transformers, we designed a computationally lightweight version of CORP (Li et al., 2023 [b]) which is a significant improvement over our old TTA method, CORT.

• Additional details: While we did find a small reduction in WER when applying CORT relative to No CORT across days (average WER improvement across days shown in Figure 4, left), this improvement was small. We thus sought to examine if applying time masking and Transformers could provide efficiency gains to the pseudo-label based TTA method proposed by Fan et al., 2023 [c]. The method used by Fan et al., 2023, CORP, adapts a model at test-time by treating the language model (LM) refined model outputs as pseudo-labels, and then training the model with CTC loss using these pseudo-labels. Since the LM refined outputs are already being generated for real-world use, there is no additional cost to generate these pseudo-labels. However, CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. Here, we present "DietCORP", a lightweight version of CORP which only uses one adaptation step per trial and no previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table 1), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 2). Only the patch embedding module was adapted. DietCORP leads to very stable WER across days relative to when not applying it with minimal hyperparameter tuning. We provide additional comparisons with CORP in the updated manuscript.

Tables

Table 1: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days.

Table 2: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days.

[a] Willett et al., 2023, A high-performance speech neuroprosthesis

[b] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication