Generalizable, real-time neural decoding with hybrid state-space models

We sincerely thank you for your positive assessment of our paper and insightful review! Below, we respond to your concerns and questions:

W1. The results in Table 2 are not well presented, because from the numbers except from PCA-KNN all the other result do not look to differ significantly. given that the plus-minus is the std (where if i am not wrong the $\pm$ is not reported).

We apologize for the lack of clarity, we reported the arithmetic mean of mean performance across sessions $\pm$ the geometric mean of standard deviations across sessions, aggregated across seeds. We will update the caption to state this clearly. We also conducted a paired t-test between session-wise performances across seeds, between the best NHP-pretrained + finetuned o-POSSM and the best non-pretrained POSSM, where we found the results to be significant with p < 0.05 (p = 0.0152). Furthermore, comparing the mean performances across sessions for different seeds between all NHP-pretrained and non-pretrained models yielded p < 0.05 for 7/9 valid comparisons, p <= 0.1 for 2/9 cases (lower performing o-POSSM variants vs. non-pretrained POGRU). Thus, we show that the transfer result is statistically significant overall. We will update our paper to reflect this.

W2. In section 4.3, if I understood correctly you first transform the spikes with some preprocessing and then in the first step you retrieve again spikes, which looks odd.

Apologies that this was not clear. We do not transform or preprocess the spikes; the data from the original source is already preprocessed through 20ms binning [1]. Therefore, we do not have access to raw spike timings. From preliminary experiments on other datasets, we found that having access to these raw spike timings can significantly improve performance (see table below). Additionally, given that the dataset is small, we found that our two-stage training curriculum yielded the best performance. Given access to more data and raw spikes, we believe this scheme would become unnecessary.

Q1. In the first cross attention layer, why you don't use again a SSM so you don't need to process spikes by 50ms windows. You could also do that with a sliding window attention architecture. Furthermore, you could use a sliding window attention instead of an SSM, which could be an interesting extra comparison (I mention that because it seems that they perform very similarly as it was shown in MesaNet).

If we understand you correctly, using an SSM instead of the first cross-attention (CA) layer would mean that each token is processed by the model as each spike is recorded, i.e., one forward pass per token. This could perhaps allow for more fine-grained prediction timescales, but might be wasteful in terms of computational efficiency and is overall a harder problem because: (1) the sequences will become far longer and (2) the timing between spike tokens is highly irregular. Observing a window of activity for the whole population is important for the model to predict behaviour in dynamics tasks, which is why we use CA to process chunks of activity. Please let us know if we have misunderstood something or if you have an alternate implementation in mind, we would be happy to explore this.

Sliding window attention (SWA) seems better suited to our tasks, and we have experimented with giving the model overlapping time chunks of a certain width and stride (e.g., 50ms and 20ms respectively) – performance is comparable to the contiguous non-overlapping window case.

Replacing the recurrent backbone with SWA makes for an interesting additional comparison. We provide some results in the table below with one SWA layer in single session models. We would like to note that von Oswald et al. [2] show that smaller sliding windows do not perform very well. In our case, the performance of SWA does not match the best recurrent backbone – and we interestingly observe that larger window sizes perform slightly worse than smaller ones. This would have to be studied further, with hyperparameter tuning of the number of attention layers or heads, their dimensionality, dropout, etc. – which we will explore in the future.

Q2. In line 154, in the equation it would be nice to have the dimenssionality of the $W_k$ and $W_v$ matrices, I assume they are MxN.

Yes, thank you for the suggestion and apologies for not including this earlier – we have updated the paper to state the dimensionality explicitly.

Q3. Something that was a bit unclear to me is whether you have feature mixers (a small MLP, or a multiplication with a gate like it happens in Mamba) after the attention or SSM layer. Feature mixers although a bit computationally expensive, they can improve the performance considerably.

We did not use mixer MLPs after the attention or SSM layers, mostly due to efficiency and strong performance even without them. While mixer MLPs provided marginal improvements in POSSM-SS models, they may be necessary at greater pretraining scale, so we will explore these in more detail in future work. A comparison with POSSM-Mamba-SS on the Perich et al. dataset [3] is given below, showing a slight increase in performance with mixer MLPs. We will add these results to our paper.

Q4. From Table 1, some models trained on single sessions perform better than pretrained models with UI (like POSSM-GRU) for the new dataset, do i read it, right? It's also interesting POSSM-SS models perform so much better than POYO-SS, do you have any idea why that is the case? It seems that the UI finetuning is not that useful for the SSM models. Maybe a possibility could be to do a LORA fine-tuning, which could be more efficient?

Thank you for this question, this was the case before. However, we now pretrained models on more data, including all 10 sessions from the Churchland et al. dataset [4] as opposed to just 2 earlier. With this, we significantly improved our UI results, and now they perform on par with or sometimes even better than POSSM-SS models (see table) on data from Perich et al. [3]. With these, we are able to finetune very few parameters overall for far less time and still achieve good performance.

LoRA is a great suggestion: we are exploring this in ongoing work involving SSL pretraining. For the sake of further compute efficiency and few-shot benefits on top of LoRA, it might be worth bootstrapping unit embeddings using existing pretrained ones (rather than starting from random for a new session or animal) – this direction requires further exploration. We believe this kind of LoRA-based UI pipeline is best suited to adaptation across days for the same animal, and especially when embedding multiunits (channels) rather than single neurons.

We present some preliminary results below, where we assign existing trained unit embeddings for new units (in a new animal heldout from training), and finetune a LoRA for the embedding layer. We anticipate improvements with further hyperparameter tuning.

It is unclear exactly why POYO-SS slightly underperforms POSSM-SS, although potential explanations include:

• Prior work [5] finds that recurrent architectures are better-suited to neural decoding than transformers (caveat: smaller dataset scale).
• Other work [6] shows that the inductive biases of SSMs allow them to learn delay embeddings better than transformers, thus enabling better performance on dynamics tasks (especially with less data and fewer parameters).

We do find that POYO performs on par with or marginally better than o-POSSM on some multi-session experiments (e.g., Flint et al. dataset [7]). Deeper investigation into the latent representations learnt by these models is required to understand these differences in performance, and we are keen on exploring this in future work. For a preliminary study, please refer to our response to Reviewer eRCh, where we interpret the SSM hidden states.

References:

1. Willett et al. "A high-performance speech neuroprosthesis." Nature (2023).
2. von Oswald et al. "MesaNet: Sequence Modeling by Locally Optimal Test-Time Training." arXiv:2506.05233 (2025).
3. Perich et al. "A Neural Population Mechanism for Rapid Learning." Neuron (2018).
4. Churchland et al. "Neural population dynamics during reaching." Nature (2012).
5. Willett et al. "Brain-to-Text Benchmark'24: Lessons Learned." arXiv:2412.17227 (2024).
6. Ostrow et al. "Delay embedding theory of neural sequence models." arXiv:2406.11993 (2024).
7. Flint et al. "Accurate decoding of reaching movements from field potentials in the absence of spikes." Journal of Neural Engineering (2012).

Once again, thank you for your time, insightful suggestions, and kind assessment of our paper. We will endavour to include all these additional results in our paper and supplementary material. Overall, we hope this rebuttal addresses your concerns, and would be grateful if you would consider supporting our paper more strongly. We would be happy to answer any other questions during the discussion phase.

Thank you for your detailed response and I am looking forward to see your final manuscript.

We thank you sincerely for your positive review of our paper and for outlining important research directions that we are actively investigating for future work. Below we respond to your questions and concerns:

W1. Limited Evaluated Scope to Motor Areas

We acknowledge that in this work, we have mostly considered datasets with motor-cortical neurons and some motor-related behaviour. However, these datasets also contain neurons from Broca's area (speech task [1]) and S1 (primary somatosensory cortex; random target task [2]) – demonstrating that the method can process neurons from a broader set of neural regions.

An investigation of POSSM's performance on data from other brain regions is an interesting direction for future work, and we have good reason to believe that it would work well beyond the motor domain. Notably, POYO+ [3], which adapts the POYO tokenization for regular time series data such as Calcium recordings (similarly to our tokenization for binned spike counts in the speech and handwriting experiments), demonstrates strong behavioural decoding performance on visual cortex recordings from the Allen Brain Observatory dataset [4]. As POSSM shares the same multitask readout framework and tokenization scheme as POYO while exhibiting similar performance, we expect that it will generalize well to other brain regions and tasks.

To support this, we provide preliminary results on 5 random sessions from the Allen Visual Behavior Neuropixels dataset [4], where we decode running speed from multi-region neural activity (visual cortex + several subcortical areas including visual thalamic areas, hippocampus, and midbrain).

This result provides early evidence that POSSM is effective across a variety of brain regions. We also anticipate that even better performance can be obtained with a mixture of more careful experimentation, training on multiple tasks [3], backbone changes, and hyperparameter tuning. Note that the sequence lengths at evaluation are 10x the training sequence lengths, again demonstrating strong length generalization.

W2. Reliance on Supervised Training

We agree with the reviewer on the limitations of using a purely supervised learning paradigm. Indeed, requiring labeled data restricts both the amount and diversity of training data, potentially resulting in models that are biased towards certain tasks with denser or easier-to-curate lables. Moving away from supervised training and towards self-supervised learning (SSL) objectives is a key direction that we are investigating in ongoing work. With SSL, we aim to leverage unlabeled neural data to further improve downstream decoding performance, especially when finetuning to new sessions or subjects. We are also investigating the interpretability of the learned parameters, such as the unit embeddings [3], trained through SSL – specifically, whether we can identify finer details about the neural population (e.g., cell types, brain regions, etc.). Finally, we are also interested in how best to train an SSM using SSL objectives, compared to Transformer-based masked autoencoders [5]. As this is an ongoing effort, we leave discussion of our results for future work.

W3. Lack of Intrinsic Multimodality

Thank you for mentioning multimodality, which is another important direction in our ongoing research. While we have only considered neural spiking data in this paper, we would like to point out that several of these datasets are recorded with multiple distinct "modalities", such as threshold crossings, sorted or unsorted spikes (a unit could be either a single neuron or a "multiunit" or "channel" depending on the dataset), and binned spike counts from public datasets [1,6]. To this end, we have recently trained POSSM on the speech task [1] on both the binned spike counts and spike-band powers. Excitingly, we observe strong performance when integrating both modalities (see table below).

Finally, in ongoing work, we are training our models on both spikes and local field potentials (LFPs) in order to investigate whether the same cross-attention layer can process both spikes and LFPs (with an additional modality embedding), or if other forms of early/late modality fusion work better. The tokenization scheme we use is identical in both these cases (LFPs are tokenized similarly to binned spike counts and Calcium imaging in POYO+ [3]), so we would argue that our approach is quite general and involves minimal preprocessing. We also thank you for acknowledging in your review that our architecture is "flexible enough to accommodate different neural data modalities through proven tokenization schemes".

Q1. Could the authors comment on how POSSM's applicability might be extended beyond motor areas? For example, how could it be adapted to cognitive or sensory domains where clear time-to-time correspondence is lacking? Addressing this would help establish its broader utility as a generalizable neural interface.

This is an important question, and we have provided promising evidence (running speed decoding from Neuropixels data [4]) and clarifications (having neurons from S1 and Broca's area in our data mix) that POSSM extends beyond just motor areas in the response to W1. Furthermore, we already explore domains where time-to-time correspondence is lacking – the speech decoding task [1] does not have dense labels for each time chunk, so we use connectionist temporal classification (CTC) [7] to predict the best sentence response at the whole-trial level for the given neural activity sequence. We have also carried out an experiment on predicting cursor velocity 50ms or 100ms in the future, thus querying the output module with timestamps beyond the current time chunk. Results are provided in the table below. We believe that such flexibility will enable the use of POSSM in tasks such as those you have outlined.

Q2. Could the authors provide further insights into the interpretability of POSSM and how it may facilitates understanding of neural dynamics?

Thank you for the question. While the our work is focused on strong predictive performance and fast, real-time inference for neural decoding, we agree that interpreting how deep learning models arrive at their predictions is an important line of work.

First, simple methods such as vanilla RNN/SSM models already help with understanding neural dynamics very well through dynamical systems analysis techniques, so we do not think POSSM can provide much additional benefit here. The spike tokenization scheme, on the other hand, can enable classification of brain regions and cell types [3] – the similarity of learned unit embeddings for neurons could potentially correlate with functional similarity of neurons across sessions or animals.

As for the interpretability of POSSM itself, we looked at the evolution of hidden states from the POSSM-Mamba and POSSM-GRU models over test trials. Unfortunately, NeurIPS does not allow us to share figures or links for the rebuttal – nevertheless, we aim to describe our findings here. We found that the top 2 principal components (PCs) of the hidden states explained 68% and 51% of the variance for the Mamba and GRU respectively. For both models, the first PC resembled the magnitude of the cursor speed in upcoming time chunks, while the second PC seemed to represent change in cursor velocity (~acceleration, direction-sensitive). We leave more detailed analysis of the hidden states and various architectural components for future work. We would like to note, however, that enforcing learned latents to be human-interpretable rarely leads to improved downstream performance [8], while the objective of our work was to obtain strong performance under real-time inference constraints.

References:

1. Willett et al. "A high-performance speech neuroprosthesis." Nature (2023).
2. O’Doherty et al. "Nonhuman primate reaching with multichannel sensorimotor cortex electrophysiology." Zenodo (2017).
3. Azabou et al. "Multi-session, multi-task neural decoding from distinct cell-types and brain regions." ICLR (2025).
4. de Vries et al. "Sharing neurophysiology data from the Allen Brain Observatory." Elife 12 (2023).
5. Ye et al. "Neural data transformer 2: multi-context pretraining for neural spiking activity." NeurIPS (2023).
6. Willett et al. "High-performance brain-to-text communication via handwriting." Nature (2021).
7. Graves et al. "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks." ICML 2006.
8. Mittal et al. "Does learning the right latent variables necessarily improve in-context learning?" ICML (2025).

Once again, we thank you for your time and positive review, and hope that our response has addressed your concerns. We will endeavour to include all these additional results in our paper and supplementary material. We would be grateful if you would consider supporting our paper more strongly as a result. We are also happy to answer any other questions you may have and look forward to discussing further!

We sincerely thank you for your detailed review. We address your comments below:

W1. The overall evaluation suite report limited comparisons ...

In Table 1, we provide performance metrics for several non-POYO-style methods, including MLP, S4D, Mamba, GRU, and NDT-2, which can be used in a causal manner. These are popular, SOTA models for neural decoding from spikes [1], and unlike previous works, we have endeavoured to make these baseline models as performant as they can get. Notably, we have improved the results of the NDT-2 baseline through hyperparameter tuning and pretraining on all our data (see table). However, we demonstrate in Table 1 that POSSM handily outperforms all non-POYO baselines. If you have any additional suggestions on baseline methods, we would be happy to compare against them.

W2. The claim that full fine-tuning yields better results ...

We would like to clarify that this statement should not be seen as a new claim, but as an interpretation of our results. We are emphasizing that - especially on new subjects - full finetuning (FT) has better performance overall, although it remains notable that unit identification (UI) is close to or better than single session performance (see next response). We will rephrase this statement to clearly indicate this. Again, we would like to kindly point out that this is not our main claim or contribution.

W3. This claim seems to clash with what reported in either ...

The initial results at training step 0 are random because unit and session embeddings are randomly initialized. Thus, the variance in performance at the initial steps is expected to be high – we do have runs where o-POSSM starts higher than POSSM-SS. We would like to emphasize that the key takeaway of Fig. 3a is that UI leads to much faster convergence and higher performance than single session models (see table below), in less than 1/3rd of the training steps and by finetuning far fewer parameters (~10K vs ~600K).

These improved metrics are a result of pretraining on more data, now including all sessions from the Churchland et al. dataset [2]. With this, we have improved the performance of our few-shot finetuning as well:

The performance improvements from pretraining are now more apparent both for fewer and more trials. Furthermore, we would argue that in the "real world", calibration should be as data- and compute-efficient as possible, so the fewer trials needed to calibrate the BCI, the better (64 trials = >5 minutes of active task performance, which can get cumbersome if done everyday).

W4. Unclear what data displayed in Figure 3c ... training on Monkey C-CO and testing on T-CO ...

Yes, you are correct – we will clarify this in the caption. Thank you for pointing out this clarity issue. While NeurIPS does not allow us to provide figures with the rebuttal, we have updated the caption to be clearer and modified all relevant figures to indicate the subjects and sessions clearly in our draft manuscript.

W5. The author evaluated their new family of models on the human speech decoding task ...

In our paper, we initially considered the problem of decoding speech from neural spikes alone. Importantly, the GRU you are referring to is achieved with spike-band powers included (we reproduced PER = 21.74%). This is a multimodal approach that is outside the scope of this paper, but we are actively exploring such approaches in ongoing work. As a preliminary experiment, we provide performance comparisons between the baseline and POSSM. We achieve competitive performance, even before full exploration of the design space. We believe with an improved tokenization scheme and access to more data, a significant bump in performance can be attained.

Our work focuses on PER, not downstream language modeling, which many prior works optimize via ensembling and bidirectional GRUs (most other methods such as transformers and SSMs do not outperform the GRU, and most top teams use biGRU decoders with training tricks [1]). These methods also lack public code and do not report PER scores [1], making direct comparisons difficult. In contrast, our unidirectional model does not have these speech-specific optimizations and is designed to be general-purpose, yet still attains competitive performance. While the current dataset is small, we expect our latent representations to become more expressive when trained with larger datasets, such as the upcoming Brain2Text-2025 benchmark. We are also exploring finetuning a model pre-trained on motor tasks, similar to what we did for handwriting.

W6. Little to no details ...

Apologies that this was not clear. Currently, these details are provided in the Appendix (Section S.5.2, Table 5), but we have now updated our draft to include these details in the main paper. Scaling is primarily determined by the number of SSM/RNN layers, and also by an increase in the dimensionality of the input and RNN hidden state.

Q1. Is the gain of pre-training meaningful ...

This proposed study is made difficult by the fact that sessions in existing datasets have limited amounts of data. However, we show that single-session training is far more expensive, as it requires training ~600K params while UI only needs to tune ~10K. While being parameter-efficient, we also know that UI achieves higher performance more quickly than training from scratch. These benefits are especially felt when there are multiple downstream subjects to whom the model must be finetuned, as the reduced training times and parameter counts are compounded. Finally, UI outperforms single-session especially in low data regimes (c.f. finetuning results above), which are reflective of BCI use cases, indicating that UI is a viable and useful approach in real-world settings.

Q2. Val $R^2$ performance seems to be close between ...

In our updated results shared above, we show gains from additional pretraining for o-POSSM. We decided to scale up the size of the model only if there was a performance benefit from doing so, and we believe that with even more pretraining data, the gains would be higher. We would also like to note that the variance in sizes for o-POSSM variants is quite high (S4D 4.5M, GRU 7.9M, Mamba 9M).

Q3. The o-POSSM has the same inference time ...

The o-POSSM models are slightly slower on average, especially on CPU. The similarity in inference times is simply the result of PyTorch having efficient, parallelizable implementations of various architectural components (attention and GRU), we do nothing in addition to ensure the closeness in inference times. We are, however, working on using Flash Attention for our initial cross-attention layer to further speed up computations, especially during training.

Q4. How does the particular tokenization scheme ...

The ViT-like patching scheme used in NDT-2 groups together arbitrary neurons within a patch, does not have a mechanism to identify separate units, and is not permutation-invariant like our scheme. Meanwhile, we explicitly learn embeddings for each unit, and we believe that this scheme both generalizes better than and enables a more straightforward and efficient finetuning procedure than NDT-2 by relearning just the unit/session embeddings.

Furthermore, we believe that using spike timing information could be instrumental in improving decoding performance. In the table below, we show results with and without access to raw spike times for each spike, where the latter would be identical to binning-based approaches. We see a considerable performance drop in the latter case, thus further motivating our choice of tokenization scheme.

We believe that these differences in tokenization could be the key factor behind the gap in performance between POSSM/POYO-style models and NDT-2. In ongoing work, we are also investigating how well the unit embeddings encode neuronal properties when trained with SSL objectives.

References:

1. Willett et al. "Brain-to-Text Benchmark '24: Lessons Learned." arXiv:2412.17227 (2024)
2. Churchland et al. "Neural population dynamics during reaching." Nature (2012).

We thank you once again for your review and attention to detail, and believe your comments have helped improve our work. We hope we have addressed your concerns, and if so, we kindly ask if you would be willing to reconsider your score. We look forward to discussing further with you!

We thank you for recognizing the several significant contributions of our work, including our fast, real-time inference, generalization capabilities, and successful transfer of dynamics from NHP datasets to a human handwriting task. Below, we address the points from your review:

However, the improvements are marginal.

We would like to respectfully note that this point appears to be in contradiction with other points in the review, e.g., "POSSM matches or exceeds performance of state-of-the-art models across three domains...", "Of particular note is the successful transfer of dynamics from pretraining on NHP datasets to a human subject handwriting task—an important result for neuroprosthetic applications where human data is limited", and "The architecture also achieves 5–9× faster inference than transformer baselines...".

First, we would like to emphasize that the primary goal of our work is not just to improve decoding performance, but to do so in a way that maintains strong performance while generalizing easily to new subjects and being efficient for real-time use. We have also endeavoured to make our baselines as competitive as possible. Nevertheless, we have demonstrated that our method consistently matches or outperforms several popular and state-of-the-art baselines. In new experiments, we also pretrained our large-scale, multi-session o-POSSM models on an expanded dataset that now includes all 10 sessions from the Churchland et al. [1] dataset. With this, we significantly improved the performance of our unit identification finetuning to be comparable to or better than single-session performance, while maintaining or improving full finetuning performance. We provide these results in the table below.

With these new models, we have further improved our results on data efficiency (few-shot finetuning). We provide these results in the tables below, on a CO session:

and an RT session:

Finally, we have also integrated an additional modality – spike-band powers – in our speech decoding model, allowing us to further improve the performance over our previous results and the highly performant GRU baseline [2].

Overall, we believe that our results not only perform comparably with or advance the state-of-the-art on several tasks, but also maintain efficiency both in terms of finetuning data and inference times. If you have any other questions or concerns about our results, we would be happy to address them during the discussion phase!

One minor suggestion would be to include a small schematic highlighting differences in causal vs. non-causal evaluation modes.

Thank you for the suggestion! While NeurIPS does not allow us to add figures or links in the rebuttal, we will include these in our final paper.

References:

1. Churchland et al. "Neural population dynamics during reaching." Nature (2012).
2. Willett et al. "Brain-to-Text Benchmark'24: Lessons Learned." arXiv:2412.17227 (2024).

Overall, while we are thankful for your positive comments, we are quite confused by the "borderline reject" rating of our paper as the review does not list specific weaknesses or questions for us to address. We would be grateful for any clarifications you might be willing to offer regarding your assessment, as this would help us better understand the recommendation. If our responses here and to other reviewers have addressed your concerns, we kindly ask if you would be willing to reconsider your score. We are also happy to address any concerns and questions you might have during the discussion phase, and eagerly look forward to your response.