In vivo cell-type and brain region classification via multimodal contrastive learning

We thank the reviewer for the detailed and useful review. We hope to address any concerns and questions below.

Weaknesses

• We thank the reviewer for the suggestion to do significance testing for our results (especially for NEMO’s smaller improvements). We ran a one-sided t-test (with p<.05 significant) for both brain region and cell-type classification. We found that NEMO was significantly better than all baselines (including SimCLR) on brain region classification and also significantly better than all baselines (except SimCLR) on cell-type classification. For cell-type classification, NEMO was significantly better than SimCLR on balanced accuracy using the fine-tuned MLP and the frozen MLP, but not the linear classifier. We also found that NEMO significantly outperforms SimCLR on the f1 score using the frozen MLP, but not the fine-tuned MLP or linear classifier (although the p-value is 0.0505 for the fine-tuned MLP; p < .05 is significant). Please see Supplementary Figures 17-23 with additional details in Supplementary M. Overall, this analysis suggests that NEMO is the strongest model for both brain region and cell-type classification. We hope this alleviates some of the reviewer’s concerns.
• While we agree that there may not be a perfect 1-to-1 relationship between spike waveform and firing patterns, there is evidence that cell-types can be roughly separated by these two modalities in vivo [1]. Our paper provides further evidence that there is a relatively strong relationship between these two modalities. In light of this, finding the best self-supervised multimodal algorithm will be essential for accurate cell-type classification given the limited opto-tagged labels available.The analogy of the word “puppy” and the image of a “puppy” is interesting, but the relationship between these two are also not strictly 1-to-1 as there are many different types of puppies. Also, multimodal contrastive learning (e.g, CLIP) has a nice property that if the signal is correlated across the two modalities and the noise is independent, then the extracted features can be more informative and generalizable than unimodal contrastive learning where the data augmentations are the only way of learning invariance to the noise (which relies on making assumptions) [2]. We believe this is part of why NEMO does significantly better than SimCLR on the IBL whole-brain dataset which is noisier than the more curated UHD cell-type classification dataset (where performance is more similar).
• We agree the citation to CEED was inappropriate. We apologize for this oversight and have adjusted the citation to now correctly mention that CEED was utilized to perform cell-type classification. A significant difference in our results is that CEED was never applied to optotagged data and, therefore, all of their cell-typing results are on data with no ground-truth (e.g., their comparison to WaveMAP). Thank you for this correction!

Questions

Figures feedback

• We have adjusted the figures and text with the feedback provided by the reviewer. We hope this improves the clarity and correctness of the work.

“In the PhysMAP paper that you benchmarked, they applied three public datasets from S1, A1, and V1/Hippocampus. Have you tested these datasets?”

• We have not tested NEMO the S1, A1, and V1/Hippocampus datasets used in the PhysMAP paper. We appreciate the suggestion by the reviewer and plan to look into these datasets for our future applications. We would also be interested in testing NEMO on the cerebellum dataset from [3] once this becomes publicly available. We believe that the development of a cell-type classification benchmark would also be an exciting future direction given the challenge of preprocessing and utilizing new datasets.

“Please specify how you computed the two primary metrics: macro-averaged F1 score and balanced accuracy.”

• We have added some clarification for the metrics used in our manuscript. The macro-averaged F1 score is calculated as the unweighted mean of F1 scores for each class (i.e., cell-type or brain region). The balanced accuracy measures the average accuracy per class.

[1] Petersen, Peter C., et al. "CellExplorer: A framework for visualizing and characterizing single neurons." Neuron 109.22 (2021): 3594-3608.

[2] Huang, Wei, et al. "On the Comparison between Multi-modal and Single-modal Contrastive Learning." arXiv preprint arXiv:2411.02837 (2024).

[3] Beau, Maxime, et al. "A deep-learning strategy to identify cell types across species from high-density extracellular recordings." bioRxiv (2024).

We thank the reviewer for the useful comments and suggestions. We appreciate that the reviewer found our experiments carefully chosen and sound!

Weaknesses

• As suggested, we will add additional details for the baselines and clustering analyses in the supplement to improve the clarity of the paper. To this end, we have added a schematic of the encoder architectures in Supplement Figure 2.
• To test the impact of our different augmentations, we ran two new analyses on the UHD cell-type classification dataset: (1) remove one augmentation and (2) add one augmentation. We have added this result to Supplementary Figure 14 with additional details in Supplementary L.2. Please see the overall response for more details.
• We agree with the reviewer that multi-unit is an overloaded term for neuroscientists. We have switched single-unit and multi-unit to single-neuron and multi-neuron everywhere in the paper.
• We have updated figure 3 and the blurry figure in the supplement (Supplementary Figure 7). We hope this fixes the problem!

Questions

“The VAE training is unclear to me. Do you jointly embed waveforms and autocorrelograms, or do you use two separate encoders/decoders with a shared latent space?”

• For the VAE baseline, we follow the procedure of [1] which trains a VAE separately on each modality and then concatenates the embeddings before classification.

“How important are the data augmentations? Can you provide an ablation experiment for that?”

• As mentioned in the above response, we now include an experiment ablating the data augmentations in the overall response.

“Waveforms and autocorrelograms are two reasonable choices for input modalities. However they are not the only conceivable choices. Have you thought/tried other choices or thought about learning an encoding of spiking activity directly?”

• This is a great question! A couple of ideas we are thinking about is to utilize the peristimulus time histogram (PSTH) or correlations of the neuron with other neurons in the population. We plan to explore these features in future analyses. Encoding the spiking activity directly is another interesting idea although this would require a much more complicated neural network architecture.

“What are the results when you cluster on the latent embeddings directly instead of running UMAP first? How stable is the clustering?”

• To quantify the stability of the NEMO clustering result, we re-ran our clustering pipeline with multiple random seeds. Although there is shared structure between clusterings, we found that there was variability with different random seeds (see Supplementary Figure 16). We expect this to be the case because clustering the full brain dataset leads to a very coarse clustering when, in reality, there is much more variability per region. We should be cautious over interpreting these clusters as this is more of an exploratory analysis. We will add this caveat to the paper.

“Can you provide a definition for the modularity index?”

• The modularity index is a hyperparameter in the Louvain clustering that quantifies the relative density of edges within communities compared to the edges between communities. It ranges from −0.5 (indicating non-modular clustering) to 1 (indicating fully modular clustering). Tuning this parameter will affect how the different nodes are grouped together in the network.

[1] Beau, Maxime, et al. "A deep-learning strategy to identify cell types across species from high-density extracellular recordings." bioRxiv (2024).

We thank the reviewer for their feedback and we hope to address some concerns below

Weaknesses

• Novelty: While we make good use of previous work, we also introduce novel data augmentations and brain classification approaches in our paper. We are also, to our knowledge, the first application of multimodal contrastive learning for either brain region or cell-type classification. Overall, we think NEMO will be a valuable machine learning tool for systems neuroscientists.

Questions

“The authors emphasized “in vivo”, however, are there any results about the computational efficiency of the NEMO model that can support it? Does it support real-time processing?”

• To clarify, by “in vivo” we mean that the recordings we analyze are from the brain in its awake and behaving state. We clarify this so that the reader understands that we are not doing any “in vitro” analysis (i.e., outside the brain). This is similar terminology as used in other papers including [1]. We do not make any claims about real-time processing and we are happy to clarify this in the final version of the paper.

“... The authors are encouraged to validate the data augmentation strategy adopted in the study, augmentations directly for the ACG images, is reasonable by showing a couple examples.”

• To validate the impact of our different augmentations, we ran two new analyses on the UHD cell-type classification dataset: (1) remove one augmentation and (2) add one augmentation. We have added this result to Supplementary Figure 14 with additional details in Supplementary L.2. We also show example augmentations in Supplementary Figure 3. Please see the overall response for more details. We thank the reviewer for this suggestion.

“The authors compared NEMO with the VAE-based method. However, in my opinion, it seems that a critical comparison is missing: the comparison between naive models”

• We agree that a comparison of naive models is important. We actually do quantify the performance of the naive models in our paper. We have three evaluation schemes: linear classification using the embeddings (i.e., linear classifier), MLP classification using the embeddings (i.e. frozen MLP), and full end-to-end fine-tuning (fine-tuned MLP). Both the linear classifier and frozen MLP evaluate the performance of each model without any additional fine-tuning of the embedding method (e..g, NEMO, VAEs, etc.). We hope this clarification addresses the reviewer’s concern.

[1] International Brain Laboratory, et al. "Reproducibility of in-vivo electrophysiological measurements in mice." bioRxiv (2022): 2022-05.

We thank the reviewer for the thoughtful comments and suggestions. We appreciate that the reviewer believes that NEMO will be useful for systems neuroscientists; This is our hope as well!

Additional experiments and figures

• We agree that the description of the architectures was too limited so we have added a schematic of the encoder architectures in Supplement Figure 2.
• We chose 5 neurons per depth as an initial starting point to evaluate the contribution of multi-neuron ensembling. We agree with the reviewer that this analysis could be improved so we ran an additional experiment where we vary the number of neurons from 3 - 25 for brain region classification. Please see the overall response for more details.
• To understand the sensitivity of NEMO and the VAE baseline to hyperparameters, we are currently running a grid search over the learning rate, dropout, and embedding dimension for the UHD cell-type classification dataset. We plan to add this to the supplement once it finishes running and we will update the rebuttal accordingly.
• To test the impact of our different augmentations, we ran two new analyses on the UHD cell-type classification dataset: (1) remove one augmentation and (2) add one augmentation. We have added this result to Supplementary Figure 14 with additional details in Supplementary L.2. Please see the overall response for more details.

Questions

“What exactly is the definition of a channel in this context?”

• The definition of a channel in this context is a single electrode. So when we restrict the template to one channel, we are restricting the template to be the waveform recorded on the single largest amplitude electrode.

“Why was additive Gaussian noise chosen as the sole data augmentation?”

• We used gaussian noise for single-channel templates to help compensate for low spike count templates being noisier. We could not think of any additional nuisance variables for single-channel templates. For multi-channel templates, we introduced electrode dropout and per-channel amplitude jitter which lead to improvements for NEMO on both brain region and cell-type classification (please see Supplement E). We will add this explanation to the main text.

“How do other data augmentation types impact the performance and results?”

• To test the impact of our different augmentations, we ran two new analyses on the UHD cell-type classification dataset: (1) remove one augmentation and (2) add one augmentation. We thank the reviewer for the suggestion. Please see the overall response for more details.

We have finished running the hyperparameter sensitivity analysis. The results for this analysis can be found in Supplement L, Supplementary Figure 13, and Supplementary Table 4. As can be seen, NEMO is robust to the range of hyperparameters we tested: the average performance over the tested hyperparameters is the same as the default NEMO model reported throughout the paper. The VAE is also robust to the range of hyperparameters we tested. We also found that the learning rate we used for cell-type classification for the default VAE was lower than the original learning rate from [1] (we adjusted this to improve reconstruction) and that a larger learning rate improves downstream performance. We have adjusted this parameter and updated Figure 2 and Table 1. NEMO still substantially outperforms the VAE baselines and the linear NEMO model is still significantly better than the end-to-end fine-tuned VAE. We thank the reviewers for this suggested analysis.

[1] Beau, Maxime, et al. "A deep-learning strategy to identify cell types across species from high-density extracellular recordings." bioRxiv (2024).