Thank you for the positive assessment. We respond to your points below.

• Re: brain region information. Nearly all devices in our data are placed in hand/arm regions in the motor cortex. Within this scope, even the coarse anatomy varies across humans (and between species), so it is not clear how to use a general coordinate system or discriminative region label. Such labels may be non-critical, thankfully, as [Azabou 25] shows that the neural activity itself is predictive of the brain region they are sourced from.
• Fixing out of distribution in Fig 4: We hope so, but do not have a clear path forward. By thought experiment, we infer that POYO should be more robust to the synthetic shuffles, but likely only with the a priori foresight to tune the unit embeddings first.
• Error decomposition / traces: Frequency-band error decomposition is an interesting suggestion. In constrained monkey experiments, decoding R2s are quite high, so sensor noise is likely not dominant. We didn’t observe salient qualitative changes in predictions beyond previous decoding models, but will add in an example trace prediction for some of our evaluation tasks in the appendix.

Azabou et al, 2025. Multi-session, multi-task neural decoding from distinct cell-types and brain regions

---

We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:

4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field.

The reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).

---

Apicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354

Banville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322

Kaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w

Aristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1

We thank the reviewer for their careful reading of our manuscript (e.g. the Fig 11 typo catch), and we agree with your summary. We address your concerns individually below.

Weaknesses

• POYO-scaling: Due to evaluation noise, we do not believe that any model is significantly different in the 100% M1 point or M2 50%/100% points. Nonetheless, we do also see that POYO started worse than NDT3 but is at parity by the larger data scales. You raise an interesting interpretation that this might imply that POYO will scale faster with more data. We had interpreted this trend as meaning that our POYO reproduction trained poorly at lower scales. Overall, Fig 19 is open to interpretation. For example, NDT2 sees similar slopes as POYO in the right half of each of these plots, so by similar argument we could have aimed to scale NDT2. Since we could not afford to scale pretraining on multiple methods, we judged it best to scale NDT3 after designing it for scaled training and verifying it had highest overall performance in scratch settings. We look forward to future work that provides more rigorous comparison. We believe we have made it relatively easy to demonstrate better scaling than NDT3 with clear goalposts (90 minutes, tying with baselines at 100% FALCON scale, Sec 3.2 generalization analyses).
• NDT2 objectives: You have identified a subtle implementation detail! In the NDT2 codebase, most models are indeed prepared with a neural-only SSL objective, and probed for behavior labels. However, some of NDT2’s experiments (e.g. Fig 5) simplify this 2-step training into a single model that trains with both neural and behavioral objectives. For fairness (as you say), we have used this joint training objective through our evaluation of NDT2 in this work. We mentioned this briefly in Line 1317-1318. To clarify, we now modify Line 1317 to read:

“The only departure we take from NDT2’s default design is that we train NDT2 with both a neural reconstruction loss (25% masking) and a supervised decoding loss. This joint objective was introduced late in the NDT2 paper (Fig 5) and performs comparably to multi-stage training while simplifying large scale evaluation.”

Questions

• NDT3-Neural against NDT2: See above.
• H2: In conversation with researchers familiar with the H2 task, we speculate that our irregular training is just a result of overparameterized Transformers with insufficient regularization. H2 only contains about 600 data points in total. Most works in the field use small RNNs, except Card 25, which adopted the Transformer after collecting much larger volumes of data. The pretrained model loss is higher likely because NDT3 pretrains on continuous regression tasks while H2 is a Seq2Seq task. We do not see a reason NDT3 could not perform reasonably on H2 if its pretraining mixed in a larger dataset of Seq2Seq spiking decoding tasks.
• Spike-sorting: We agree with your intuition that spike-sorted inputs could enrich the representations learned in pretraining. Practically, this choice is often already made for us, as few datasets are released with broadband data or sorted spikes, and some BCI labs do not save the broadband data due to its large storage footprint. We are broadly interested in the development of models that support richer neural data features, but the field needs convincing evidence that the additional overhead of these features are justified.

Limitations: We have initially omitted an explicit Limitations section as our results have a section (3.2) analyzing methodological limitations. We acknowledge, though, that there are other limitations in our work unaddressed by 3.2, and will include some of your suggestions in a new explicit Limitations section in camera-ready. Similarly, we will restore our impact statement from section A in the appendix to the main text.

By contextualizing why we chose to scale NDT3 over POYO, clarifying NDT2’s joint objective, and lengthening the discussion, we hope we have addressed your concerns on the work’s weaknesses, and ask that you might consider a non-borderline score. Thanks for your time!

Card et al, biorXiv 2025. Long-term independent use of an intracortical brain-computer interface for speech and cursor control

---

We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:

4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field.

The reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).

---

Apicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354

Banville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322

Kaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w

Aristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1

We thank you for your time and detailed review of our paper. We agree with your summary with one small clarification: although 90 minutes of downstream data is far from trivial. As evidence, prior studies (POYO, NDT2) only showed improvements when tuning datasets of single experimental sessions lasting ~1-10 minutes, and intracortical neural data releases overall remain short; monkey studies often publish with 100-200 minutes total. In this framing, the fact that NDT3’s benefit saturates at 90 minutes is a non-issue. We stated that the benefit is of limited practical utility, because in recent years BCI research has begun adopting multi-session modeling as a norm (Line 132), and in this regime 90 minutes of calibration data could be achieved after say, 3-5 BCI sessions. In this more realistic regime, unaddressed by prior neural pretraining work, we envision NDT3 as still being useful when onboarding new BCI users (in the lab and the real world), or when testing new behavioral paradigms with monkeys.

With this context, we next address the main body of your critiques and questions, which center on the issue of cross-subject generalization.

Our view is that our work identifies for the first clear time that there are significant cross-subject transfer issues when scaling models on spiking activity. Thus NDT3’s contribution is not to improve cross-subject transfer in this regime, because there is no baseline to improve over at our scale of evaluation, but to definitively establish the challenge of cross-subject generalization in this domain. We agree that the former interpretation would justify much of your critique (i.e. weakness 1), and we believe our miscommunication may have caused much of your subsequent review. Specifically, addressing weakness 3 as to whether our evaluation addresses cross-subject generalization, we state at the start of 2.3, (line 122):

Our main evaluation uses four human and four monkey datasets [...] All downstream monkeys are held-out of pretraining, and all humans are held-out of < 2 khr models. That is, we evaluate scaling of cross-subject generalization.

That is, only the 2khr model contained data from some evaluation subjects (Addressing Q1). Of course, the 2khr setting is our largest scale, and we realize now that we asserted (Line 125) that the work nonetheless evaluated cross-subject generalization without further justification. The following explanation was omitted:

We did not hold humans out of our full 2khr model since we have so few human datasets to begin with, and pre-determined evaluation datasets overlapped with subjects with significant data. To alleviate concerns on subject leakage, we verified that the 2khr model, with subject leakage, did not outperform other models on the test datasets with leaked subjects (Fig 11, top row), and did outperform other models on a held-out human evaluation (Fig 11, top right). This observation (Line 1157), supports our claim that NDT3’s demonstrated benefits are all in a setting where no evaluation subject data was seen in pretraining (contrasting with the critique in weakness 2), and that our evaluation reflects scaling cross-subject gains overall. To summarize, our pretraining data rarely included data from evaluation subjects, and where it did, models did not benefit from this data.

If you may now agree that we did evaluate cross-subject generalization, we can address Q2, on scaling with the number of subjects. Our main results (Fig 3C/D) show that performance increases with (cross-subject) pretraining data hours. Individual task results varied greatly, so we did not attempt more precise scaling laws. More precise quantification of the scaling benefit of cross-subject hours is measured in NDT2 Fig 3, 4 [Ye 23], for one downstream evaluation. Unfortunately, your strongest request of identifying scaling with respect to the number of training subjects is challenging because the data available in different subjects is heterogeneous in volume and quality (Fig 2A) and more simply, because the number of subjects in intracortical neural data is low overall. We hope that our work motivates future neural data collection efforts with hundreds of subjects, to analyze precisely your requested analysis.

Finally, your Q3 is interesting, but unfortunately we have not run analyses on how probes might scale with pretraining. We agree probes would provide a stricter view of pre-trained features. We chose to evaluate E2E fine-tuning to best reflect downstream usage. Downstream BCI deployment most likely will value E2E tuning’s greater or equal performance relative to probes because tuning cost is not significant at NDT3’s size. It is our understanding that linear probes are most frequently used to evaluate representation learning when model pretraining differs from downstream application (e.g. ImageNet/CLIP to generic vision tasks). This is also of interest for NDT3, so probe-based evaluations are of interest for future work.

We believe your main concern with this work has been around pretraining leakage and our claim of cross-subject generalization. We will add a summary of our response here around Line 125. Thank you for advocating for more clarity on this point. We hope that this addition and our response here have alleviated your concerns, and respectfully request a reconsideration of your score if so.

Ye et al, 2023. Neural Data Transformer 2. https://openreview.net/pdf?id=CBBtMnlTGq

---

We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:

4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field.

The reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).

---

Apicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354

Banville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322

Kaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w

Aristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1

We thank the reviewer for your time and thoughtful questions. We generally agree with your summary. We would like to clarify that NDT3 is trained with both supervised behavior and self-supervised neural data prediction objectives, in pretraining and tuning. The related analyses include pretraining curves for each objective in Fig 14, an ablation of neural objectives in pretraining in Fig 9, and an ablation in fine-tuning in Fig 16 / C.11.

Weaknesses

NDT design. We agree there is room for innovation beyond NDT3’s design. This work is meant to establish a rigorous baseline that provides clear goalposts for future work (whether by beating the 90 minute threshold, superior generalization in Fig 4’s analyses, or higher FALCON scores). We do not think these innovations will come easily, though (see global response below). Our thoughts on your suggested culprits:

• Patch tokenisation: We were optimistic that NDT3’s patching would not be limiting because the patching scaled well in NDT2. In retrospect, this is only because those evaluation datasets were small. POYO’s tokenization is promising, but we were not successful with it.
• Spike timing: This is a keen observation, and we have since observed that spike timing improves NDT, but we do not expect scaling to qualitatively improve from using spike timing over binned spikes, since binning effectively just lightly noises the input. Practically, binned spike times are necessary as many datasets are only packaged with binned spike data over precise spike timing.
• Autoregressive causality: We do not see this design as a likely culprit since other work has demonstrated scaling in domains with no sense of causality/time with random token dropout (e.g. Vision MAE).
• Lack of post-training: We don’t believe other domains have required post-training to observe good scaling. We are hopeful post-training can improve NDT3’s utility, however, and would be interested in evaluating this in future efforts.

H2 task: We did evaluate NDT3 on H2 (C.4, Fig 10). While length generalization and memory were not issues for us (thanks to FlashAttention), we did not find transfer from NDT3’s continuous regression pretraining to H2’s seq2seq classification task. The use of Transformers is likely not fundamentally limiting for either discrete or continuous speech tasks, though, as both Wairagkar 25 and Card 25 use Transformers.

POYO comparison: This topic is important to detail. NDT-CrossEnc is so named because NDT’s patching is its most suspect design flaw (Section 3.2), but we tried to compare with POYO more completely than this, and did consider your three raised differences:

1. Supervised training: NDT2 and NDT3 are both trained with supervised objectives in this work (see the aforementioned Fig 9/14/16 on its importance in NDT3).
2. Unit embedding vs patch embedding: NDT-CrossEnc reduces patch size to 1, and each patch receives identifying embeddings, so NDT-CrossEnc also essentially receives unit embeddings. One remaining difference is that unit embeddings are distinguished across datasets. In NDT2, dataset-specific parameters were accounted for with a separate session embedding token. We dropped this in NDT3 due to its low benefit (according to NDT2’s ablations) and high cost (45M parameter NDT3 would incur about 20M new params from ~20K datasets x 1K params/dataset).
3. Binned spikes vs spike timing: As mentioned, we cannot scale with spike timing as this data is frequently missing in BCI datasets.

In summary, we believe NDT-CrossEnc fairly translates POYO to the scaled BCI setting. We have confirmed our design with the released codebase and in correspondence with a researcher familiar with the POYO model. We did not pursue an exact reproduction as POYO has not shown cross-subject transfer that is qualitatively different from NDT2 (Both were evaluated in low-data downstream setting). We certainly agree that our documentation of NDT3-CrossEnc reproduction (Line 1354-1362) could be improved; we will restructure based on your feedback, similar to the above bullets, to improve clarity.

One remaining option would be to directly use the public POYO codebase, by adapting POYO model/dataloading/evaluations to the BCI use case (causal / binned spikes). Our confidence in doing this fairly is low, since there are open issues regarding reproducing reported results at the original POYO code, and the new codebase also lacks evaluation scripts or end-to-end examples. At this point, we believe it would be most expedient to seek comparison through external benchmarks like FALCON.

NDT-MTM: We agree NDT-MTM may not have been exhaustively optimized; NDT3 also did not use Mu-parameterization. To clarify, we omitted comparison with NDT-MTM not because of poor performance, but because its main distinction from NDT was its use of new masking strategies to improve neural data reconstruction (as opposed to behavior decoding).

Clarity: Thank you for this note. We had compressed such summaries to fit the content limit, but will add these summaries in the camera-ready.

Questions

294-306: Yes, we meant 4D. Thanks for catching this typo.

Post-training: Thanks for asking! Post-training has not been explored in the neural FM space, but we included the analysis in 4C to specifically highlight a target for post-training, to prioritize a decoder that will generalize to more behaviors when the neural activity would support it. We allude to this in Line 257. One pertinent question will be how to preserve this post-training through the final fine-tuning that all neural models currently require.

Synthetic data: To attempt to improve generalization to unseen channel orderings, we piloted the use of channel shuffling augmentations in pretraining. This helped at small scales but began to harm performance by 200 hours of pretraining data. We did not report this as we think this warrants a more thorough investigation that was beyond the scope and content limits of this paper. Other synthetic augmentations from the BCI literature are definitely worth testing for neural foundation models, but have mostly been used to address small signal nonstationarities over time, so we do not expect it to change cross-subject scaling.

Cross-species: We did not explicitly mention cross-species transfer in this work, but you raise a good point, so we will clarify the following in the Discussion: The 200 hour model does show cross-species transfer, as it only pretrains on monkey data and positively benefits human evaluations. We do not think this is due to differences between NDT2/NDT3, but rather a result of training and evaluating at a larger scale.

Latency: With KV cache enabled and at 1 second context length, we see mean inference times of about 4ms and 9ms for the 45M and 350M models on an NVIDIA 4090. On a 4070, the 350M model slows to ~15-18ms. We add these timings to Line 1412.

Thank you for all these suggestions, and we hope that our clarifications and comments (justification of NDT3 and our POYO reproduction, resolution of other weaknesses and questions) address your main critiques and respectfully request reconsideration of the borderline score.

---

We attach below a global rebuttal summary if you are interested in how your feedback relates to other reviewer’s comments:

4GhD/iNS9/cDht sought more understanding of how NDT’s design might cause the reported limitations. We share this desire but believe addressing it clearly will require scaling alternative models, and we exhausted our resources scaling NDT3 (20K A100 hours). We do believe that NDT3’s design was appropriate as the first choice to scale to 100M+ parameters as 1) NDT2 showed the main choice of patchwise tokenization worked well in the previous order of magnitude, 2) NDT3 outperformed the alternate candidate (POYO) in our hands (Fig 19), and 3) NDT3 closely adheres to the vanilla Transformer that successfully scales in other fields. We are hopeful for future improvements but note that cross-subject transfer is broadly challenging beyond our design and neural modality (Banville 25, Apicella 24). The most promising path forward appears to be scaling to thousands of subjects (Kaifosh 25, Aristimunha 25), which is not yet feasible for implanted BCIs. Instead, we personally are actively exploring richer neural data features and alternate tokenization, as some reviewers have suggested. More generally, we hope that NDT3’s 90 minute saturation mark and limitation analyses have provided clear goalposts for the field.

The reviewers requested various improvements to clarity. We agree the work is dense and unclear at times, which likely caused some reviewers to request analyses or question claims that were explored in the appendix. We will add 1) summary boxes to our main text sections (4GhD) and 2) a table of contents in the appendix. We will also 3) extend the discussion with an explicit limitations section and restore the impact statement to the main text (iNS9), and 4) tweak figures to improve readability (cDht). Finally, we 5) will add a reference to the supplementary analysis that supports the rigor of our cross-subject evaluation (HeMw).

---

Apicella 24, Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods. https://doi.org/10.1016/j.neucom.2024.128354

Banville 25, Scaling laws for decoding images from brain activity. https://arxiv.org/pdf/2501.15322

Kaifosh et al, 2025, A generic non-invasive neuromotor interface for human-computer interaction. https://www.nature.com/articles/s41586-025-09255-w

Aristimunha 25, EEG Foundation Challenge. https://arxiv.org/abs/2506.19141v1

Card et al, biorXiv 2025. Long-term independent use of an intracortical brain-computer interface for speech and cursor control

To clarify, I was wondering if you'd be able to state something like a percentage improvement (maybe just on certain tasks, e.g., 4D bimanual) in the text along with your mention of showing cross-species transfer. Upon looking at Fig. 11 again, I agree that these can get quite messy depending on the task, where in some tasks all models seem to be quite close. The figure was enough for me to get a sense of the result, would just hope that the framing in the discussion is crystal clear so the reader knows exactly what to expect.

Thanks for engaging, can't overstate how much I appreciate all the analyses in the paper!