Neural Encoding and Decoding at Scale

The use of mask-based pre-training for Transformers is quite common. I am curious whether the three embeddings—Modality Embedding, Temporal Embedding, and Session Embedding—are truly effective...

While masked modeling is a common objective for training transformers, it is not yet clear which masking schemes work well for modeling neural activity and behavior. We aimed to address this question in our paper and with our proposed method.

We appreciate the suggestion to ablate the different embeddings for NEDS. To do this, we performed an ablation study on single-session data (10 test sessions) to evaluate the importance of the temporal and modality embeddings. Session embeddings were not ablated, as they are essential for distinguishing data from different sessions. The ablation results, presented in the following table, demonstrate that the embeddings generally improve performance across the 5 tasks we evaluate (with a few exceptions). We want to note that these are single-session ablations and will be more sensitive to hyperparameters.

The paper only compares two methods, POYO+ and NDT2, on a single dataset. Although some performance improvements were achieved, the overall framework does not differ fundamentally from prior work. The experimental section is also limited to evaluating performance in terms of encoding and decoding, lacking insightful scientific findings.

Prior work in modeling spiking neural activity and behavior focuses on modeling a single direction of the relationship (i.e., decoding or encoding). NEDS introduces a new framework for modeling neural activity and behavior by jointly modeling the two modalities, allowing for simultaneous encoding and decoding at test time (POYO+ and NDT2 solely focus on decoding). We believe this multimodal approach is novel for this domain.

We agree that more experiments are needed to extract scientific insights. However, encoding and decoding remain key tools for understanding what and how information is represented in the brain. For example, the International Brain-wide Map [1] used these methods to characterize how visual, sensory, and motor information are distributed across the mouse brain. We are excited to extend this with NEDS, leveraging large-scale, multi-animal data to capture richer information and reveal how neuronal functional profiles relate to brain anatomy, as shown in Figure 4.

My main concern lies in the potential value of the contribution of the neuro-behavioral conversion presented in this article. Compared to previous works, this article seems more like an incremental project and does not address fundamental issues in the field, such as whole-brain alignment for neuro-behavioral conversion or conversion specific to the human brain...

We thank the author for the suggested citation which we would be happy to include. We want to emphasize that we strongly believe that our contribution is not incremental. Currently, there are very few foundation models trained on spiking data, contributing to the novelty of our approach. Also, current foundation modeling approaches for spiking activity have primarily focused on decoding, whereas, to the best of our knowledge, no prior work has unified neural encoding and decoding.

We note key differences from NeuroBOLT [1]: NeuroBOLT translates between neural modalities (fMRI and EEG) without modeling behavior, and uses separate encoders per modality, unlike NEDS, which uses a shared transformer to unify representations.

We also feel that the critique related to whole-brain alignment does not apply to spiking recordings which are spatially restricted. Whole-brain alignment is a more tractable problem in fMRI research due to its broader spatial coverage, but is not typically addressed in invasive electrophysiology.

[1] International Brain Laboratory, et al. "A Brain-Wide Map of Neural Activity during Complex Behaviour." bioRxiv (2024).

[2] Li Y, Lou A, Xu Z, et al. NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping[J]. Advances in Neural Information Processing Systems, 2025, 37: 23378-23405.

It could be better to extend the results to other recordings, or you may discuss the heterogeneity of the neuropixel data to show the capability of the model.

We agree with the reviewer's suggestion to extend NEDS to other datasets. To address this, we are currently training NEDS on a primate motor task dataset to show its generalizability across different recording setups, species, and behavioral tasks. In future work, we are excited to extend our approach to larger and more diverse datasets.

Just one hold-out test is a bit thin, and it would be beneficial to do at least one or two more validations to make sure the results are stable, albeit at a cost.

We agree that additional validation would strengthen the evaluation. Our results rely on a fixed hold-out set due to computational and experimental constraints, and while more rigorous cross-validation would provide a more accurate assessment of model differences, it is currently challenging given the scale of our datasets and models. The development of shared benchmarks will help enable more robust comparisons across different approaches; we plan to participate in such benchmarks in future work.

Have to test the impact of the token length? How do you prepare the continuous and discrete data to cover an overall behavior?

This is a great question and something we have not explored. We utilize the tokenization scheme from [1] for our model where the neural data is binned at a specific resolution (20ms) and then each time step is passed into the transformer as a token. We fix the context length to 2 seconds, aligned to movement onset, for all trials and animals. The model performance is more likely influenced by the time bin size rather than token length, as suggested in previous work [1]. We are currently training NEDS on two different bin sizes on a monkey dataset. We are excited to further explore this in future work. For more details on how we tokenized continuous and discrete behaviors, please refer to Section 3.3, “Modality-Specific Tokenization,” in the main paper. At a high-level, the discrete data is transformed into tokens that are repeated multiple times to match the resolution of the continuous data.

Is there any difference between the data from different labs (total 10, mentioned in the section 3.1)?

This is an interesting question! Part of the goal of the International Brain Laboratory (IBL) was to reproduce the same experiment and data across multiple labs. To do this, the IBL utilized a standardized experimental protocol and quality metrics to ensure consistent data quality across labs [2]. As can be seen in Figure 7 of [2], the authors conducted an experiment using all neurons in the IBL repeated site datasets to predict their brain region and lab identity. The findings indicate that while brain region identity can be reliably decoded from single-neuron profiles, lab identity cannot be inferred from the data.

It would be beneficial to give some details about the four kinds of behaviors, or give some references.

We appreciate the feedback and will incorporate more descriptions of the behaviors into the final camera-ready.

Have you introduced the design of the objective functions and some model details, such as the position embeddings.

Yes, details about the objective function and position embeddings can be found in Section 3.3, “Architecture,” of the main paper.

Is it possible to test the performance in the animal-independent condition, which means test the performance with fine-tuning on the test 10 mice?

Currently, fine-tuning is required for evaluating these models to align the session and neuron-specific weights into a shared representation. Zero-shot performance of these models is an exciting future direction.

Statistical analysis would make the results better.

We agree that statistical analysis would improve our results. To address this in the limited time window during the rebuttal, we are currently re-running our single-session analysis with different random seeds to determine whether model performance was significantly influenced by randomness. We plan to include these results in our response once the experiments are complete.

[1] Pei, Felix, et al. "Neural Latents Benchmark'21: Evaluating latent variable models of neural population activity." arXiv preprint arXiv:2109.04463 (2021).

[2] International Brain Laboratory, et al. "Reproducibility of in vivo electrophysiological measurements in mice." bioRxiv (2024): 2024-12.

We appreciate that the reviewer found our paper well-written and our evaluation convincing. We agree that evaluating NEDS on additional datasets would make our paper stronger. In response, we are currently applying NEDS to a primate motor task dataset (MC-RTT [1]) to demonstrate its generalizability across different recording modalities, species, behavioral tasks, and data structures. We plan to include these results in our response once the experiments are complete.

[1] Pei, Felix, et al. "Neural Latents Benchmark'21: Evaluating latent variable models of neural population activity." arXiv preprint arXiv:2109.04463 (2021).

We appreciate the reviewer’s suggestion to evaluate the generalizability of our model on additional datasets, tasks, and unaligned data. Given the complexity of the neural recordings we analyze in our paper, spanning multiple brain regions and animals, we intentionally focused on a small set of well-defined, trial-aligned behaviors (e.g., choice) for our evaluation. We agree with the reviewer that it would be interesting to test NEDS across more complex datasets and tasks. To address this, we are currently training multiple baselines models and NEDS on the MC-RTT primate motor task dataset [1], which differs significantly from the IBL visual decision-making task and is also unaligned. We plan to include these results in our response once the experiments are complete.

[1] Pei, Felix, et al. "Neural Latents Benchmark'21: Evaluating latent variable models of neural population activity." arXiv preprint arXiv:2109.04463 (2021).