Cross-Modal Representational Knowledge Distillation for Enhanced Spike-informed LFP Modeling

We thank the reviewer for their thoughtful and constructive evaluation of our work. We appreciate their recognition of our contributions and address their comments below.

Substantial improvement over MS-Spike models

As the reviewer noted, distilled LFP models do not yield substantial gains in behavior decoding compared to their teacher MS-Spike counterparts. This observation is expected and aligns with prior findings that spike signals alone outperform LFP signals in decoding, which is precisely why we chose the spike model as the teacher for distillation. However, we emphasize that our goal is to enable a high-performance unimodal LFP model, which does not need to use any spiking activity for decoding. As such, the value of our approach lies not in surpassing spike-based decoding but in enabling improved decoding just from LFP signals—a more practical, more robust, and less invasive modality—while narrowing the gap to spike-based performance. Further, please see item 3 in our response to reviewer KUPu, where we show that the distilled LFP models can still achieve a competitive decoding performance on a new session without any further finetuning or distillation.

Multimodal modeling

We thank the reviewer for this thoughtful comment. We agree that our multi-modal baseline—simple concatenation of spike and LFP signals at the input level—is indeed a naive approach compared to more principled multi-modal learning strategies. We acknowledge this limitation and note that more sophisticated multi-modal approaches could be explored as a complementary direction, particularly when both spike and LFP data are available at inference time (L967–968).

However, we would also like to highlight an important challenge in this setting. Multi-modal models can degrade significantly when the higher-quality modality (spikes) is missing at inference, sometimes performing worse than unimodal baselines (Ma et al., 2022). This issue becomes even more pronounced in unsupervised multi-modal modeling, which is crucial for training generalizable models, where the lower-quality modality can dominate training, making it harder to learn robust and generalizable representations. In contrast, our cross-modal distillation framework is explicitly designed to enable high-quality inference from LFP signals alone, without requiring spikes at test time. This is critical given the practical advantages of LFPs and that spike recordings degrade over time. Please see the important applications of an LFP-alone model in the introduction (L30-38).

Taking these considerations together, we believe our cross-modal distillation offers a simpler, more scalable, and more robust approach for the specific problem of improving LFP decoding when spike data is unavailable at inference, which is a critical scenario in neuroscience and neurotechnology applications. This argument is further supported by our new analyses.

Unfreezing the teacher network

We thank the reviewer for their question! Per the reviewer’s request, we performed an experiment on unfreezing the teacher network at different stages of the distillation training. When the teacher was updated from the start of distillation, the model suffered from representational collapse, achieving only $0.09~R^2$ on session 20160622_01 of Monkey I. We then tested a delayed unfreezing schedule by unfreezing the teacher network at 10th or 75th epochs of the distillation. While these prevented collapse, they resulted in slightly inferior performance: unfreezing the teacher network at 10th and 75th epochs achieved $0.668 \pm 0.072$ and $0.674 \pm 0.066$ average $R^2$ over 8 sessions (5 finetuning sessions of Monkey I and 3 finetuning sessions of Monkey C as in Fig. 3), where freezing the teacher network completely (default setting) achieved $0.681\pm0.067$ $R^2$.

We would also like to point the reviewer to prior work showing that freezing the teacher network during distillation leads to improved student performance when the teacher is fine-tuned (Abbaspourazad et al., 2025; Li et al., 2025). Despite this prior evidence and our experience, we acknowledge that an alternative scheduling algorithm outperforming the completely frozen scenario can be developed, which we will leave as an interesting future direction.

References:

• Ma et al. (2022). Are Multimodal Transformers Robust to Missing Modality? IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Abbaspourazad et al. (2025). Wearable Accelerometer Foundation Models for Health via Knowledge Distillation. arXiv.
• Li et al. (2025). Distilling Knowledge from Large-Scale Image Models for Object Detection. ECCV.

We thank the reviewer for their acknowledgement of our work’s strengths and its importance for the computational neuroscience community. Below, we answer the reviewer’s questions:

Number of different sessions across MS-LFP and MS-Spike

We appreciate this insightful observation and agree that the smaller number of sessions in the MS-LFP dataset (34 vs. 226 for MS-Spike) can indeed be a contributing factor to the performance gap between the LFP-only and spike-only models. However, this imbalance stems not from our experimental design but from the limitations of publicly available datasets—a key motivation for our cross-modal knowledge distillation from spikes to LFPs. As noted in L26–28, most motor neural datasets have focused on spike recordings, likely due to theirr higher signal-to-noise ratio (SNR) as well as more memory-efficient data sharing. In contrast, LFP signals are typically derived from broadband recordings sampled at high frequencies (e.g., 30 kHz), which require much larger storage (often 1–8 GB per session in datasets like Makin et al.), whereas spike timing data for the same session can be shared in a much smaller storage (100 MB–1 GB). Given this data availability difference in the neuroscience domain, recent modeling approaches largely utilized spike signals to train models on larger datasets and provide more rigorous evaluations.

Given these constraints, we did our best to include as many publicly available datasets containing both spike and LFP signals as possible. While the current availability of LFP data is limited, we believe that with larger-scale LFP datasets and more sophisticated pretraining strategies, it would be possible to develop more accurate MS-LFP models in the future. However, even in a future large-scale setting, we believe that the proposed cross-modal distillation would offer advantages due to the SNRs of spikes and LFP signals. As supporting evidence, we trained an MS-Spike model using the same 34 sessions as MS-LFP, which achieved $0.544\pm0.083$ $R^2$, outperforming MS-LFP ($0.325\pm0.042$ $R^2$) but underperforming the MS-Spike model trained on 226 sessions ($0.650\pm0.065$ $R^2$), likely due to the reduced training set. This aligns with prior work showing spikes often carry more behaviorally relevant information, as reflected in higher decoding performance (Bansal et al., 2011). Therefore, using spike models to guide LFP models can help denoise the LFP signal and improve its behavioral relevance.

LFP vs. LFP power features

We understand and agree with the intuition that raw LFP signals should, in principle, contain at least as much information as the LFP power features, since the latter are computed from the former. As expanded below, the reason for using LFP power in some datasets was that in those datasets, raw LFP signals were not available.

We would like to highlight that the LFP baselines trained on raw LFP signals and those trained on LFP power features are not directly comparable, as they originate from different datasets collected during different experiments. Specifically, recordings from monkeys Ch, H, La, and M were obtained from the dataset by Gallego-Carracedo et al., where only preprocessed LFP power features are publicly available. Consequently, separate models had to be trained on these sessions. Thus, these results also reflect performance differences across distinct datasets and preprocessing pipelines rather than a strict comparison of LFP signals versus LFP power features.

Generalizability

The distillation is performed on a single-session basis by using the LFP signals of that session, after fine-tuning the pretrained MS-Spike model on the spike signals of that session, which is then used as the teacher network during distillation. Thus, it is correct that during the test time of the distilled LFP models, there are no ‘truly’ unseen subjects or sessions; when we say ‘unseen’, we refer to the being ‘unseen’ in the pretraining of MS-Spike models. We would like to highlight that this is the case for other multi-session and multi-task models of neural activity, such as NDT2 or POYO, i.e., before testing the pretrained model on a new subject/session, some degree of fine-tuning is required for all of these models as they contain subject/session-specific parameters that need to be learned. In this manner, the reviewer’s understanding is correct, and our work does not differentiate from the prior work in this regard, but follows a similar pretraining/fine-tuning strategy.

Reviewer’s understanding of results in Section 4.3 is also correct. The spike signals of sessions on the x-axis of Fig. 5 are indeed used during pretraining of the teacher MS-Spike model. However, during the distillation objective, we use the LFP signals of a single session (20160622_01 of Monkey I), after fine-tuning the pretrained MS-Spike model on that same session’s spike signals to use it as a teacher network. Thus, the LFP signals of the sessions on the x-axis of Fig. 5 are never seen during parameter updates/training; they are just used for inference. In Fig. 5, we show that the distillation objective learns a spike-LFP alignment that generalizes beyond the recording session on which the distillation is performed.

Nevertheless, we understand the reviewer’s concern, and we sincerely thank them for bringing it up. Now, we pretrained a new MS-Spike model where we first held out the recording sessions shown in the tables below (in addition to the sessions on which we performed distillation). Then, we performed the following:

1. Pretrain a new MS-Spike model where all sessions shown in tables below are held out, in addition to session 20160622_01 of Monkey I, and all sessions of Monkey C.
2. Fine-tune the MS-Spike model on spike signals of session 20160622_01 of Monkey I and session e1_1 of Monkey C separately
3. Use fine-tuned MS-Spike models from step 2 as teacher and perform distillation with LFP signals of session 20160622_01 of Monkey I and session e1_1 of Monkey C separately (so distillation is only performed on single sessions)
4. Perform inference on the LFP signals of sessions below (held-out).

In this setting, we obtained the following results. Note that ‘Distilled LFP’ and ‘MS-LFP’ refer to results in Fig. 5, whereas ‘New Distilled LFP’ refers to the new model trained as described above.

Monkey I:

Monkey C:

In this setting, the generalization performance of the new distilled LFP of Monkey I model slightly decreased compared to the one in Fig. 5; but, it still significantly outperformed the MS-LFP model. Unlike Monkey I, the distilled LFP model of Monkey C outperformed its counterpart in Fig. 5 and MS-LFP. These results indicate that the our distillation approach can generalize to recording sessions even when they are completely unseen, i.e., neither of the spike signals nor the LFP signals are used during training or even fine-tuning. This result also shows that even for new sessions that do not contain paired spikes, the distilled LFP models can still offer an advantage.

We believe that this result is a great addition to our manuscript, and we sincerely thank the reviewer for their great insight on recommending that.

About LFP features that allow superior performance for LFP signals

Prior work in neuroscience shows that LFP and spike signal dynamics have common dynamical modes that are dominantly predictive of downstream behavior (Abbaspourazad et al., 2021). Given this evidence, aligning the LFP representation to spike representation through the distillation objective likely allows the learning of the behaviorally relevant dynamical features in LFP time-series data.

Number of tokens

The number of tokens processed during pretraining and finetuning of MS-Spike is 120.1M and 4.8M, respectively. We will include these details in our manuscript.

Overall, we thank the reviewer for their careful evaluation and for highlighting important points, which we addressed in detail with new analyses and clarifications. We hope that these responses and the new evidence provided will help the reviewer to reassess their overall evaluation of our work.

References:

• Stavisky et al. (2015). A high performing brain–machine interface driven by low-frequency local field potentials alone and together with spikes. JNE.
• Abbaspourazad, et al. (2021). Multiscale low-dimensional motor cortical state dynamics predict naturalistic reach-and-grasp behavior. Nature Comm.
• Ahmadipour et al. (2024). Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity. JNE.
• Bansal et al. (2011). Relationships among low-frequency local field potentials, spiking activity, and three-dimensional reach and grasp kinematics in primary motor and ventral premotor cortices. Journal of Neurophysiology.

Additional baselines and expanding the distillation approach to other models

We thank the reviewer for thoughtful suggestions. As requested, we incorporated new baselines for further validations.

First, we trained MSID (Ahmadipour et al., 2024)—a multimodal model of neural activity from suggested groups—on 30 sessions from Monkey I. MSID achieved a mean $R^2$ of $0.48\pm0.09$, substantially lower than our unsupervised distilled LFP models, which achieved $0.66\pm0.05$. The underperformance of MSID is consistent with its reliance on linear latent dynamics.

Second, we attempted to fine-tune the BRANT model, using the publicly available checkpoint. Despite extensive attempts—both freezing and updating the BRANT backbone—the model failed to converge on the Monkey I dataset. We hypothesize this is due to:

• Domain shift between the human SEEG data used for pretraining and the NHP LFP data used in our setting, and
• Model scale—BRANT’s ~505M parameters likely require larger datasets to fine-tune effectively.

Additionally, for LFP data without distillation, we compared our rotary transformer backbone, 10-layer LSTMs, and 10-layer 1D CNNs (all producing 256-dimensional embeddings), trained in the supervised setting. Across 30 sessions of Monkey I:

While these results are comparable, our supervised distilled LFP models still achieved significantly higher performance ($0.76\pm 0.04~R^2$) than all of them. We selected the transformer backbone primarily for its compatibility with masked autoencoding objectives and its strong performance across diverse deep learning tasks.

While performance is important, we emphasize that the core contribution of our work lies in its conceptual advance—enabling high-performance unimodal LFP decoding through cross-modal distillation, rather than simply aiming to outperform existing LFP models by changing the model architecture.

Importantly, our framework is architecture-agnostic: the distillation approach could be applied to architectures such as BRANT, MtM, NDT2, or POYO in the future.

Clarity of methods, adding supervision, freezing

We thank the reviewer for helpful suggestions on the write-up, and we apologize for the confusion caused. In Section 3.2, we originally referred to Appendix A.1 for MS-Spike fine-tuning details. However, we will now also include a clear summary of these details directly in Section 3.1, where they are most relevant. We will also add a step-by-step guideline for training/finetuning details, summarized as follows:

1. Pretraining an MS-Spike model on pretraining sessions
2. Finetuning the MS-Spike model on one of the finetuning tasks, depending on the level of supervision. For supervised fine-tunings, we add a regression head to the pretrained MS-Spike model, as the pretraining was unsupervised. During all fine-tunings, all model parameters are updated.
3. Initializing an LFP model, then training it via the cross-modal knowledge distillation objective where the teacher model is the fine-tuned MS-Spike model from step 2. During this step, the teacher MS-Spike model is fully frozen (as described in L208-209).

Learning schedules

We chose to keep the teacher MS-Spike model fully frozen during distillation. The decision was based on simplicity and prior evidence (Abbaspourazad et al., 2025; Li et al., 2025).

As described in response to reviewer ZtJZ item 3, we tested the impact of unfreezing the teacher model. When the teacher was updated from the start of distillation, the model suffered from representational collapse, achieving only $0.09~R^2$ on one session of Monkey I. We then tested a delayed unfreezing schedule, where the teacher network remained frozen for the 10 or 75 epochs and was updated thereafter. While these prevented collapse, they achieved slightly worse performance.

Multi-modal modeling

As highlighted in L100–101, our goal is distinct from multi-modal modeling: rather than leveraging both LFP and spike signals at inference time, we aim to build high-performing LFP-only models (the important applications for LFP-only models are provided in the introduction L30-38).

As also discussed in L24–28, our motivation to use a pretrained spike-based teacher is twofold:

• Spike models (e.g., NDT2, MS-Spike) are well-established as SOTA for motor decoding (L39–41),
• Public spike datasets are significantly more abundant than multimodal datasets, enabling more generalizable pretraining. (L27-28).

Another advantage of cross-modal knowledge distillation over multi-modal learning is its simplicity, i.e., not requiring complex alignment approaches as in many multi-modal works (Peng et al., 2025; Wang et al., 2019). This is particularly important when one modality is of lower quality than another (i.e., LFP vs. spikes) (Huang et al., 2021).

We agree that comparing against multi-modal baselines is important, and thus, we provided comparisons against 2 multimodal baselines: SS-MM and SS-MM-ZS (see L231-237). As shown in our results, distilled LFP models consistently outperform the SS-MM-ZS baseline (Figs. 3,4,6,10,11,12,13). Further, we now show that our distillation approach outperforms a multi-modal baseline, MSID.

Denoising & discarding information of LFP

The reviewer is correct that the distillation objective contributes to denoising the LFP signals. However, we do not believe this process completely discards LFP-specific information that may be complementary. We highlight 3 reasons supporting this:

• We include an autoencoding objective with the distillation loss, encouraging the model to retain intrinsic LFP information.
• The distillation loss does not converge to zero (e.g., 0.8 cosine similarity), suggesting that the model retains some degree of LFP-specific encoding.
• As shown in Fig. 3, distilled LFP models can even outperform their teacher MS-Spike models on certain sessions. This would be unlikely if important LFP-specific information were discarded.

Generalization

We do demonstrate generalization to a new subject in Section 4.5, where we evaluate multi-session (MS) distillation approach on a held-out animal (Monkey M). The procedure is as follows:

1. Pretrain an MS-Spike model by including some sessions of Monkeys Ch, H, and La (in addition to the 226 sessions), excluding any data from Monkey M.
2. Using the MS-Spike model from step 1 as the teacher, pretrain an MS distilled LFP-power model through the distillation objective by excluding Monkey M
3. Fine-tune the MS-Spike model from step 1 on Monkey M’s spikes
4. Fine-tune the MS distilled LFP model from step 2 on Monkey M’s LFPs, where the fine-tuned MS-Spike model from step 3 is used as the teacher.

This pipeline transfers the spike-LFP alignment learned across animals to a new subject, and the resulting distilled LFP models generalize well to Monkey M. We will make this clearer in the manuscript.

We would also like to ask the reviewer to kindly see item 3 in response to reviewer KUPu, showing generalizability to fully unseen recording sessions with no training/finetuning, and distillation.

Patching, data details, evaluation

We agree that the ordering of spiking neurons does not inherently carry semantic meaning. While the neuron index is arbitrary, the patched groups still represent neurons recorded from the same cortical region (M1) with shared and complementary dynamics. The learnable patch embeddings and the self‑attention mechanism allow the model to flexibly capture these correlations without assuming any fixed spatial structure. A similar patching mechanism can also be found at NDT-2.

We provided patch sizes for spike and LFP signals across different settings in Appendix A.2 (L645-646, L649-651), where we used a patch size of 64 for spikes, and a size of 32 and 288 for LFP and LFP power signals, respectively. These choices are made for computation efficiency, as small patch sizes increase the input sequence lengths.

For data splits, we perform a random 80/10/10% train/validation/test split. Models are evaluated based on downstream behavior decoding: we train a linear regression head on the embeddings produced by the encoder, detailed in Appendix A.4. During evaluation, we combine validation & test sets to improve the robustness of performance statistics, particularly for sessions with limited data.

Teacher model size

Per reviewer’s request, we now performed distillation from a 4M parameter MS-Spike model, where the student distilled LFP models had 10M parameters. Mean performance in Fig. 9a, where both teacher MS-Spike and student distilled LFP models had 10M parameters, was $0.681$ $R^2$. With a 4M parameter MS-Spike model, we obtained a slightly worse mean performance of $0.649~R^2$, and a larger drop compared to student model scaling as expected, but distilled LFP models again outperformed other LFP-only baselines.

Overall, we hope the further evidence above on our model’s performance and generalization answers the reviewer’s concerns and encourages the reviewer to reconsider our work.

References:

• Ahmadipour et al. (2024). Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity. Journal of Neural Engineering.
• Abbaspourazad et al. (2025). Wearable Accelerometer Foundation Models for Health via Knowledge Distillation. arXiv.
• Li et al. (2025). Distilling Knowledge from Large-Scale Image Models for Object Detection. ECCV.
• Peng et al. (2025). Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing. ICML.
• Wang et al. (2020). What Makes Training Multi-Modal Classification Networks Hard?. arXiv.
• Huang et al. (2021). What Makes Multi-modal Learning Better than Single (Provably). arXiv.