SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding

We thank the reviewer for the insightful review, valuable suggestions and constructive feedbacks. We provide clarification to the reviewer questions in our point-by-point responses below:

Q1. re: clarification on whether IDEncoder is fine-tuned in the new sessions.

We thank the reviewer for the questions. The IDEncoder is trained with behavior labels during held-in sessions and its weights are frozen during inference on held-out sessions. IDEncoder’s role is to identify units from their temporal activations in calibration trials. We assume this identity is largely preserved for each individual unit across sessions, only their order is changed due to shifts in electrodes between sessions. Therefore, with sufficient training during held-in sessions, IDEncoder can infer this preserved identity out-of-the-box on the held-out sessions without any fine-tuning.

Q2. re: clarification on gradient-free, zero-shot, and few-shot terminologies

We thank the reviewer for the comments. By SPINT being a few-shot unsupervised method we mean SPINT would need a few unlabeled calibration trials from the held-out sessions for adaptation. This adaptation does not require any parameter updates in either the IDEncoder or the rest of the model as we elaborate in the response above, therefore SPINT is a gradient-free framework. The reviewer is correct that SPINT is not a zero-shot method, as it would still need a few calibration trials at test time. However, unlike existing few-shot baselines where model weights are adapted with gradients at test time (NoMAD and CycleGAN), SPINT can be adapted in the forward-only manner. Therefore we created a new class of method called gradient-free few-shot unsupervised to properly distinguish SPINT with the existing baselines.

One model can belong to different model categories (ZS, FSU, FSS, OR), depending on whether they have access to the held-out calibration neural activity and behavior labels. In Table 1 of our paper we followed the reporting format in the FALCON paper [1] to mention each model’s class in the column next to their names.

Q3: re: intuition how SPINT can work with only one calibration trial on M1

We agree with the reviewer that this is a surprising and interesting result. The ability to generalize even with one calibration trial can be attributed to the consistency of the animal behavior and the relatively large size of the M1 dataset (5 and 6 times larger than the H1 and M2 datasets, respectively). After being sufficiently trained on a sufficient amount of structured data, when presented with even a single trial at test time, the IDEncoder can extract distinct features from the temporal responses of the units, and the cross-attention mechanism can selectively aggregate these information for behavior decoding.

Q4: re: robustness of SPINT against test populations with larger size than the training population size

We appreciate the reviewer’s thoughtful question. We conducted additional experiments on to gauge the robustness of SPINT when the population is larger than the size seen during training. Specifically, we artificially duplicated a random subset of units in the population and evaluated SPINT’s cross-session performance on this larger population. We found that SPINT is robust to the increased population size, even up to twice the size of original population (Table 1). Since the EvalAI platform was down during the rebuttal period, we show here the performance on the public calibration portion of the held-out sessions on M1 dataset.

Table 1: SPINT robustness to increased size of population. Results are reported as mean $\pm$ standard deviation across held-out calibration sessions.

Q5. re: robustness of SPINT against noise in spike assignment

We thank the reviewer for raising an interesting question. We agree with the reviewer that noise introduced by the spike sorting pipeline could negatively impact SPINT performance. We conducted additional experiments on the M1 dataset where we artificially shuffle spikes among a subset of random units and evaluated SPINT cross-session performance:

Table 2: SPINT robustness to noise in spike assignment. Results are reported as mean $\pm$ standard deviation across held-out calibration sessions.

This is somewhat expected, as SPINT relies on consistent spiking features in a unit’s responses to accurately identify the unit. With spikes being assigned to incorrect units, this ability to extract unit identity would be impacted, leading to a degraded decoding performance as a result. Our initial attempts to introduce this noise corruption during training did not result in improved performance; however, we agree with the reviewer that this is an interesting avenue to explore in future extensions of the method.

[1] Brianna Karpowicz, Joel Ye, Chaofei Fan, Pablo Tostado-Marcos, Fabio Rizzoglio, Clayton Washington, Thiago Scodeler, Diogo de Lucena, Samuel Nason-Tomaszewski, Matthew Mender, et al. Few-shot algorithms for consistent neural decoding (falcon) benchmark. Advances in Neural Information Processing Systems, 37:76578–76615, 2024.

I thank the reviewers for answering my questions.

I had only minor comments and did not see any major weaknesses when reviewing the paper.

I understand Reviewer MJ6k's response, but believe that the task of "cross-session nonstationarity within a subject" is a very important task. However, when comparing to other methods, I agree that looking at cross-subject generalization is very important as competing methods may be designed for this task, and it's important to properly state the conceptual differences with POYO, NDT-3 and other competing tasks. I would recommend making this crystal clear in the manuscript.

For now, I wish to keep my score and recommend acceptance.

We thank the reviewer for providing valuable suggestions and feedbacks on our manuscript. We address the reviewer's concerns and questions in our point-by-point responses below:

W1. re: failure cases

We thank the reviewer for the comments. We show in the paper that SPINT can handle well the signal drift across held-in and held-out sessions, which can be three weeks to one month apart in the extreme cases of all three datasets. We also showed in Figure 3 our analyses on how SPINT performed with very limited calibration data. Except for the M1 dataset where SPINT can perform well even with one calibration trial, we observed that with one calibration trial in M2 and H1, SPINT performance degraded with held-out $R^2$ around zero in these extreme cases.

W2. re: sensitivity to hyperparameter choices

We appreciate the reviewer’s request for more details on the hyperparameter choices of fixed dropout rates and attention head counts. While we employed dynamic dropout instead of the conventional fixed dropout, we provide below the sweeping analysis on different choices for fixed dropout rate together with different range of dynamic dropout rates for reference. We also include sweeping experiments on attention head counts as the reviewer suggested. SPINT’s performance on held-out sessions remains stable across different choices of dropout rates and attention head counts. The results demonstrate SPINT’s robustness to the choices of hyperparameters.

Table 1: SPINT's cross-session performance against dynamic dropout and different choices of fixed dropout rates. Results are reported as mean $\pm$ standard deviation across held-out calibration sessions.

Table 2: SPINT's cross-session performance against different choices of range for dynamic dropout. Results are reported as mean $\pm$ standard deviation across held-out calibration sessions.

Table 3: SPINT's cross-session performance against different choices of attention head counts. Results are reported as mean $\pm$ standard deviation across held-out calibration sessions.

W3. re: evaluation limited to motor decoding

We appreciate the reviewer’s concern. While this is driven mainly by the availability of standardized datasets and evaluation platforms, we will mention this perspective in the Limitations section of the revised manuscript.

Q1. re: SPINT handling of large shift between sessions

Please see our response in W1

Q2. re: whether SPINT can adapt with one or two trials

Yes SPINT can adapt with as few as one or two calibration trials. We showed in Figure 3 of our main paper SPINT’s performance on M1 and H1 datasets when the number of calibration trials are restricted to these values.

Q3. re: whether SPINT can work on non-motor tasks

In the absence of actual experiments, we speculatively think that SPINT’s lightweight and neuronal identity based approach will be useful in non-motor tasks as well. Our algorithm can be extended to speech and text coding with the following adjustments. In general, for speech/text coding, the data is of variable length and has sparse labels. To address the above difficulties, we can 1) apply zero-padding or learnable padding for the variable-length data so they can have the same maximum length. This padding batch trick is widely used in LLM that can be easily incorporated in our data processing pipelines. 2) The issue of no dense label in text/speech decoding can be solved via the connectionist temporal classification (CTC) loss during model training. In the model inference phase, one can use either greedy or beam search approaches for the final character/phoneme decoding. This will serve as an exciting future extension of our work.

Q4. re: privacy and ethical concerns of long-term neural decoding

We work on public datasets within the framework of the FALCON Benchmark and focused more on algorithm development to realize the goal of consistent motor decoding. Thus, our in-silico experiments did not involve any human or animal experiments. In terms of data privacy and ethics protocol in the collection of neural data recording, the FALCON Benchmark has properly addressed these points in their publication.

L1. re: broader impacts

We thank the reviewer for suggesting this point. We added them in the discussion section by considering 1) the ethical concerns in iBCI data collection; 2) privacy of iBCI data usage and distribution; 3) the safety of iBCI technology.

Broader Impacts. iBCI technologies raise significant ethical and societal considerations that require proactive mitigation strategies. First, we advocate for consistent consent protocols and enhanced IRB oversight to address challenges of long-term neural data recording and collection, particularly for vulnerable populations with potential neurological and neuropsychiatric disorders. Second, neural data's inherently identifying nature necessitates privacy-preserving techniques beyond traditional anonymization, including different privacy regularizations to prevent risky inference of subject information. Third, we acknowledge dual-use risks including potential neuro-surveillance applications and cognitive manipulation, alongside equity concerns where high costs may exacerbate healthcare disparities. To mitigate these risks, we recommend developing open-source frameworks, establishing international governance standards similar to those proposed for AI-related algorithms and systems, implementing technical safeguards against unauthorized access, and conducting regular ethical impact assessments. To this end, we hope the above considerations can underscore the need for interdisciplinary collaboration between technologists, ethicists, and policymakers to ensure responsible development and deployment of future iBCI technologies.

We thank the reviewer for an insightful review, thoughtful questions, and valuable feedbacks. We address the reviewer's concerns and questions below:

W1: We thank the reviewer for raising the question about SPINT’s ability to transfer across subjects. Our focus in this paper was to isolate the challenge of cross-session nonstationarity within a subject, which remains a central issue in iBCI systems. When we said SPINT contributed an initial step toward a plug-and-play decoding framework for long-term iBCI, we referred specifically to the out-of-the-box cross-session adaptation within a subject, where we demonstrated gradient-free generalization without test-time labels. While we designed SPINT specifically for this practical usage, we agree with the reviewer that cross-subject evaluation would bring further values to the method. We therefore conducted additional experiments on a dataset of an extra monkey performing the M1 task (M1-B) (https://dandiarchive.org/dandiset/001209). While this extra dataset is not part of the official FALCON Benchmark and does not have a private held-out test set on EvalAI, the subject in this dataset (Monkey X) performs the same reach-to-grasp task as Monkey L in the official M1-A dataset, and the FALCON organizers curated its data into held-in and held-out sessions following the same standardized format of the benchmark. We did not observe straight-out transferability when evaluating one subject’s checkpoint on another subject’s held-out sessions. However, with full model weight supervised fine-tuning on the new subject’s held-in sessions, the pretrained SPINT quickly adapts to the new subject, maintaining a gap over the model trained from scratch on the new subject in terms of cross-session generalization (Tables 1 & 2). The EvalAI platform was down during the rebuttal period, therefore we show the performance on the public calibration portion of the held-out sessions.

Table 1: Cross-session performance of SPINT on subject X when finetuned from Monkey L vs. trained from scratch. Results are reported as mean $\pm$ standard deviation across held-out calibration sessions.

Table 2: Cross-session performance of SPINT on subject L when finetuned from Monkey X vs. trained from scratch.

These results highlight the subject transferability of SPINT, where cross-session generalization ability on a new subject is maintained and efficiently achieved within a few epochs of fine-tuning.

W2 & W3: We chose the FALCON Benchmark for our analyses because it contains standardized splits with clear constraints on the number of test-time samples, label availability, and latency requirements in real-world iBCI systems that we aspire for SPINT. We also appreciate the reviewer highlighting the similarities and differences between SPINT and POYO. We hope the appreciation we have for POYO and other baselines was clear in our paper. To provide further clarity, we will include a discussion on the unique aspects of POYO and SPINT, and will cite POYO+ and POSSM papers in the final version of our paper.

Direct comparison of SPINT and POYO on the FALCON Benchmark is not trivial, as the benchmark’s continuous decoding evaluation paradigm strictly requires participating models to operate on binned spikes, which POYO does not natively support with its unit tokenization scheme. Following the reviewer’s recommendation, we conducted extra experiments to evaluate SPINT on the suggested Perich et al., 2018 dataset [1] that POYO was also trained on. Due to resource constraints and the lack of standardized few-shot evaluation splits on this dataset as we have in FALCON, we selected 10 and 2 sessions of Monkey C performing the Center-Out (CO) reaching task as held-in and held-out sessions, respectively. We reserved 32 trials on each held-out session as the few-shot calibration set. We binned the spike trains with a 20-ms bin width and followed FALCON’s continuous decoding paradigm on this dataset. SPINT demonstrated good cross-session generalizability on this dataset, achieving $R^2$ of 0.83 $\pm$ 0.01 on the heldout sessions, without fine-tuning or using labels in the held-out sessions. For comparison, POYO-mp trained on 100 sessions (that is, POYO-mp was trained on more data) achieved $R^2$ of 0.9675 $\pm$ 0.0079 by fine-tuning unit ID and $R^2$ of 0.9708 $\pm$ 0.0116 by fine-tuning the entire model on the held-out sessions. When fine-tuned on the held-out sessions, SPINT could further achieve held-out $R^2$ of 0.88 $\pm$ 0.01. These results show the promise of SPINT on datasets beyond the FALCON Benchmark.

[1] Perich, Matthew G., Juan A. Gallego, and Lee E. Miller. "A neural population mechanism for rapid learning." Neuron 100.4 (2018): 964-976. (https://dandiarchive.org/dandiset/000688?search=000688&pos=1)

W4: Due to the character limits, we kindly ask the reviewer to check our Q3 response to Reviewer CKq1.

W5: We appreciate the reviewer’s comment on the cross-task trainability. As the reviewer suggested, a modification to the SPINT architecture to employ a readout scheme as in POYO+ will facilitate SPINT’s training across behavior tasks. SPINT could be extended to have this multi-dataset pretrainability in future works. We will add this perspective to the Limitations section in the revised manuscript.

W6: We thank the reviewer for bringing up similar dropout schemes employed in previous works that we missed. We will cite the references the reviewer suggested, and revise our wording so as to not overclaim dynamic dropout as our main contribution. One relatively minor difference between SPINT’s dropout scheme and that of earlier implementations is that dynamic dropout uses the full valid interval of probabilities in SPINT (i.e., [0,1]), instead of a limited low-probability range (i.e., [0, 0.2]). In Figure 4B, by “without dynamic dropout” we meant with no dropout at all. Due to the character limits, we kindly refer the reviewer to the dynamic dropout table in W2 response to Reviewer CKq1 for further dropout analyses.

Q1: Please see the above comments in W6.

Q2: Although we designed SPINT for few-shot unsupervised adaptation, SPINT could also be used for few-shot supervised learning, as the reviewer correctly suggested. We performed experiments where we either fine-tune the IDEncoder or the whole model using calibration data on the few labeled samples of held-out sessions, and observed that this led to improved cross-session performance. Fine-tuning the IDEncoder obtained held-out $R^2$ of 0.69 $\pm$ 0.10, and fine-tuning all model weights obtained held-out $R^2$ of 0.70 $\pm$ 0.10, as compared to the gradient-free unsupervised SPINT with held-out $R^2$ of 0.64 $\pm$ 0.10. We will report the new results in the revised manuscript.

Q3: Intended to be lightweight, our SPINT model does not have self-attention layers and only has 1 cross-attention layer for simple information aggregation. To analyze the scaling effect of model size, we provide the results for two sets of experiments: 1) similar to POYO, we added self-attention layers before the cross-attention layers, with the number of self-attention layers ranging from 1 to 4. 2) We scaled up the cross-attention layers from 1 to 5. Our results are listed below.

Our results indicate that scaling up model size by adding self-attention and cross-attention layers does not significantly improve the performance. This might be due to the small scale of the FALCON datasets, as the reviewer noted. We will incorporate these results and the associated interpretation into the revised manuscript.

Q4: Our single behavior query was initialized at random and learned during training. The behavior query Q has dimension BxW, where B is the number of behavior covariates and W is the neural window length. For multi-task extension, there can be multiple queries for decoding behavior in the corresponding task.

Q5: We agree with the reviewer that the individual spike tokenization in POYO could potentially work better since spike timings contain helpful information that can be lost in the binning process. However, using POYO spike tokenization in the context of the FALCON continuous decoding paradigm is difficult, as the benchmark strictly requires participating models to adopt the binned spike format for evaluation. Please also see our additional comments in W2 and W3.

Q6: We performed a hyperparameter search to find the reported optimal sliding window size. Heterogeneity in the task structure leads to different optimal sizes for each task, as a trial in H1 typically lasts for ~20s, much longer than trials in M1 and M2. Due to the character limits, we refer to the window size table in Q2 from Reviewer fpRR.

We thank the reviewer for a thoughtful review and valuable feedbacks. We provide point by point clarifications and answer questions below.

W1. re: shallow architecture limiting long range modeling, neural ID Encoder forgetting the within-trial structure.

We understand the reviewer’s concern regarding the long range modeling with shallow architecture. We would like to clarify a point that might have been a source of confusion: the IDEncoder uses the whole trial as input, as opposed to the sliding context window that the cross-attention module uses. Therefore the IDEncoder would be able to learn the within-trial structures.

Secondly, the shallow architecture of SPINT does not necessarily limit long range performance. The opposite might be true for CNNs where each layer has limited receptive field; however, in SPINT, fully connected layers and attention layers are used to model both temporal and spatial population responses, and each temporal step (or spatial unit) is accessible to all other timesteps (or spatial units) regardless of how far apart they are in the context. As long as the sliding window is long enough to include sufficient context for behavior decoding, the model should perform well as we showed in our analyses.

W2. re: restricted evaluation to 1 human and 2 monkeys, behavior limited to arm/hand tasks.

We appreciate the reviewer's feedback regarding the number of subjects and diversity of the tasks. We provided additional results demonstrating the benefits of using SPINT in cross-subject scenarios (please see table 1 & 2 in response to reviewer MJ6k). The focus of the hand/arm tasks are mostly based on the availability of the human/monkey experiments. Due to the limitation of recording devices, most neural recordings are collected while monkeys are sitting, leading to the popularity of arm/hand tasks in the literature.

W3. re: gradient-free adaptation still needs session specific calibration data.

Indeed, our proposed SPINT framework still requires few session-specific calibration trials although it does not need labels for those trials (thus appreciably improving its practicality). Compared to current state-of-the-art (SOTA) few-shot decoding approaches (NoMAD and CycleGAN) which rely on fine-tuning to achieve desirable motor decoding performance, SPINT is a gradient-free approach that removes the need for test-time fine-tuning.

More broadly, an ideal BCI decoder needs to work under the zero-shot operation regime, meaning that no calibration data and no model fine-tuning are needed. Our approach represents a practicable and effective step towards this goal via a minimal use of calibration trials (in M1 dataset we could achieve 0.65 held-out $R^2$ even with 1 calibration trial, outperforming other SOTA methods).

Q1. re: whether contrastive or predictive objectives could remove the need for labelled data for the neural ID encoder

We thank the reviewer for bringing up this insightful idea. We did briefly consider using unsupervised learning objectives such as contrastive or predictive loss to learn the representation for each neural unit. We found that these self-supervised objectives have difficulty in capturing the behavior-relevant aspects of neuronal identity, resulting in performance loss. Additionally, contrastive and predictive objectives require pretraining which necessitates additional training steps on downstream behavior decoding tasks. Thus we opted for the current SPINT design instead of using the contrastive or predictive objectives. We will add this discussion to the revised version of the paper.

Q2. re: whether a lightweight temporal transformer on top of the cross-unit attention would benefit decoding behaviors with longer latencies

We thank the reviewer for this suggestion. The ability to model long-range information of SPINT depends on the length of the sliding context window. We performed sweeping experiments to find the best window length that can yield the best performance and found that longer context does not always translate to better performance.

Table 1: SPINT held-out performance across different window lengths. Results are reported as mean $\pm$ standard deviation across held-out M1 sessions.

We will incorporate these results and the discussion in the revised version of the paper.