Neuron Platonic Intrinsic Representation From Dynamics Using Contrastive Learning

How exactly are the experiments on the Steinmetz dataset supporting the claims of neuron intrinsic representation learning?

-The intrinsic representation of a neuron is, in fact, a time-invariant representation. We obtain a time-invariant representation for each neuron through a self-supervised approach. The question then arises: how can we validate the effectiveness of this time-invariant property? To address this, we take into account the time-invariant attributes of neurons that are already known from prior knowledge, such as neuron type and the brain region in which the neuron is located.

For the Bugeon Dataset: why was only data of mice A used?

-Thanks for the suggestion, the cross animal results are more meaningful for neuroscience, so we changed the evaluations from one animal to multiple animals. Specifically, we have four mice, so 4-fold was used, with the folds based on the identity of the mice. We also divided a validation set and used a random search strategy to tune the hyperparameters of methods being compared, employing the best models on the test set. We have revised the result in the paper.

For the Steinmetz Dataset; why wan’t also e.g. a 10-fold crossvalidation used (folds along the mice identity)?

-Following your suggestion, we have reconducted the experiments and used 10-fold cross-validation (with folds based on the identity of the mice). We also replaced the original bar charts with a table to present more detailed information, including the precision, recall, and F1 scores details.

Would you expect you method to also perform well if the number of Izhikevich neuron “types” was greatly increased (e.g. 10,20,30 categories)?

-Strictly speaking, the performance of the model depends on the similarity between different neuron types, rather than the number of types. If the added categories are very similar to each other, the model's performance is likely to decrease. However, if the added categories still exhibit distinguishable differences, the performance should remain unchanged.

Will/are the exact labels used for three experiments, and in particular for Bugeon be available for others to be able to reproduce the experiments exactly?

-Of course, I am willing to provide a jupyternotebook[1] to show how to preprocess Bugeon dataset[3], download dataset from [2].

Would you expect different results if max-pooling was used instead of mean-pooling?

-I believe that max pooling will not perform better than average pooling, because before the neuronal data is input, it has already undergone binning of the firing rate, which is already similar to a max pooling operation. If we were to use max pooling again, it would lead to a significant loss of information.

Is there any more literature / papers exactly doing single neuron characterization based on neuron population activity (possibly on other datasets)? It seems to be challenging to find more related literature.

-I came up with this idea when I saw that there's work now using deep learning to learn the inherent properties of different singers voices, and you can refer to the papers that inspired my idea[4].

[1]https://drive.google.com/file/d/1Gf9RS49K2W0npLE3kY77GD--s0oHgXRA/view?usp=drive_link [2]https://figshare.com/articles/dataset/A_transcriptomic_axis_predicts_state_modulation_of_cortical_interneurons/19448531 [3]https://www.nature.com/articles/s41586-022-04915-7 [4]https://dl.acm.org/doi/10.1109/TASLP.2022.3169627

Thank you once again for the clarifications and addressing the weaknesses. I will adjust my soundness score and overall score accordingly.

The distinction between “intrinsic” vs “internal” is a valid point. However, then an explanation is required as to how an “external property” such as location of an object can be an “intrinsic property” of said object, which is not obvious and therefore must be included e.g. with the introduction of the dataset. I suppose that indeed for neurons the location of a specific neuron is not independent of its other properties and inherently linked (see developmental neurobiology). This would then explain the validity of the Steinmetz dataset experiments.

Thank you for the revised abstract and included limitations. I will adjust my presentation score and overall score accordingly.

1. Regarding the abstract: it is much more understandable and straightforward to me. I especially appreciate addressing each previous point carefully. Some remaining ambiguities, and related potential suggestion e.g.:

• “different segments of the same neuron” -> “different recording segment from the same neuron”? (line 16)
• “brain regions” -> “location within brain regions”? (line 18)
• now the method name NeuPIR or NeurPIR is not mentioned until page 4 or 7 -> probably it should be included already in the abstract and it should definitely be used in a consistent manner; one xor the other.

2. Regarding Figures and Tables: I still believe Figure 1 could benefit from more labels such as X, H, Z, P, F or at least CEBRA, VICReg. Tables should not go over the margins (e.g. Table 3), and Titles of subplots should be appropriately sized (e.g. Figure 2). Consider also using PDF figures for quality improvement instead. Finally, aggregate metrics across firing types / neuron types / brain regions would still help to compare the methods at a glance (at least in appendix).

I will assume in good faith that a Steinmetz dataset explanation will be included and the minor complaints (in so far the authors agree) will be addressed in the final versions of the manuscript. The new soundness and presentation score is 3 each, and the overall score an 8.

The abstract says "PRH posits that representations of different activity segments of the same neuron converge, while segments from inherently dissimilar neurons diverge" (lines 19-21). What representations this is referring to is not clear in context.

• PRH inspired us to treat each neuron as a system and learn the time-invariant intrinsic properties of each neuron's system representation. Our representation indeed differs from the Platonic representation, as you pointed out. Using "Neuron Platonic Intrinsic Representation" is not appropriate, and we have revised the title accordingly. Additionally, we have clarified the relationship between PRH and Neuron Intrinsic Representation in both the abstract and the introduction.

The introduction should be more specific about what the method actually is, what the contrastive objective is, etc.

• In this paper, we set the optimization objective for obtaining the intrinsic representation of neurons as follows: clips from the same neuron should have a higher average similarity than clips from different neurons. We employ a contrastive learning approach to achieve this optimization goal, treating different segments from the same neuron as positive pairs and segments from different neurons as negative pairs. During the implementation of the contrastive learning method, we realized that directly optimizing for the separation of samples from different neurons might be too rigid (since different neurons do not necessarily mean dissimilarity). Therefore, we chose the VICReg approach, which only uses positive pairs for optimization but indirectly separates dissimilar samples through regularization terms. Thank you for your suggestion. We have revised the introduction to make the reasoning clearer and to provide more detailed key points.

Identifying cell type seems like a paradigmatic task where compressing neural activity into binned firing rate loses important information which may be contained in the spike train.

• Identifying cell types is indeed a paradigmatic task, but our work is not focused on cell type identification. Rather, we use this as a downstream task to demonstrate that the learned time-invariant intrinsic representations contain meaningful information, of which cell type is one component. Although binning the firing rate may lose some detailed information, it still retains key features of neuronal activity. For example, binning the firing rate allows us to observe the overall response pattern of neurons under specific stimuli. What we are learning is the time-invariant intrinsic representation of individual neurons, which is also a overall representation. Binned firing rate is a standard preprocessing step in neuroscience data analysis when processing large-scale neural data. This approach is primarily aimed at reducing computational complexity and data storage requirements, making some level of information compression necessary. On the other hand, it aligns with the fact that neurons encode information according to frequency patterns.

The method section should provide more information about what CEBRA is.

• CEBRA serves as an encoder for the dynamic activity information of neuronal populations. Its advantage lies in its ability to encode both neuronal activity data (NT) and corresponding auxiliary variables, such as behavior and external stimuli (MT), into a low-dim (D*T). In CEBRA, D does not correspond to individual neurons, aiming to uncover the relationships between neuronal activity and these variables. Our work focuses on obtaining intrinsic representations for individual neurons. We cleverly utilize CEBRA from a different perspective as a preprocessing step for our input data. From each neuron, we use CEBRA to integrate the single neuronal peripheral information (such as activity of neighboring neuronal populations and behavioral data) for each segment of a single neuron. This process encodes the peripheral information associated with each segment of an individual neuron. Our work primarily involves contrastive learning to different segments of the same neuron. Thank you for your suggestion. I have added a detailed description of CEBRA in the Methods section, along with an explanation of how we use it.

The cross-animal generalization experiment (5.3) is interesting but as predicting cell type is the more relevant problem I would be interested to see a similar generalization experiment with a cell type dataset as in 5.2.

• Thanks for the suggestion, the cross animal results are more meaningful for neuroscience, so we changed the evaluations from one animal to multiple animals. Specifically, we have four mice, so 4-fold was used, with the folds based on the identity of the mice. We also divided a validation set and used a random search strategy to tune the hyperparameters of methods being compared, employing the best models on the test set. We have revised the result in the paper.

Line 54 and 55:

• Although VICReg does not explicitly use negative pairs like other contrastive methods, it still indirectly promotes the separation of different samples through the variance and covariance regularization terms. The variance regularization ensures that the representations have sufficient spread, meaning that for dissimilar samples, their feature representations are less likely to collapse into a small region of the embedding space. The covariance regularization helps ensure that features are not highly correlated, promoting more distinct and diverse representations. VICReg relies on a more subtle approach, achieving the desired separation through variance and covariance regularization.[1]

Figure 1:

• In this paper, we set the optimization objective for obtaining time-invariant intrinsic representations of neurons as follows: clips(segments) from the same neuron should have a higher average similarity than clips from different neurons. Figure 1 illustrates how this objective can be achieved using a contrastive learning approach, where different clips from the same neuron are treated as positive pairs and clips from different neurons are treated as negative pairs. This is the main idea conveyed by Figure 1. However, when selecting a specific contrastive learning method, we realized that directly optimizing for the separation of clips from different neurons may be too rigid (as different neurons do not necessarily represent dissimilarity). Therefore, we chose to implement VICReg, which optimizes using only positive pairs, but indirectly separates dissimilar samples through regularization terms. We have revised the caption of the figure to explain more detail.

Line 134:

• In this paper, we are not learning representations of neuron populations, nor are we performing dimensionality reduction on neuronal population activity. Instead, we aim to learn a time-invariant intrinsic representation for each individual neuron. Each neuron is considered separately, and the activity information from the remaining neurons in the population is treated as peripheral information (equation 1: X_st) for that specific neuron, which acts as auxiliary variables. The peripheral information for each neuron is processed and encoded using CEBRA.

Line 149:

• If a neuron has data from multiple sessions, we will select segments of 512 time points from different sessions as positive pairs for that neuron. If a neuron only has data from a single session, we will randomly select non-overlapping segments, with lengths randomly ranging from 200 to 512 time points, as positive pairs for that neuron. During implementation, we have ensured the use of an algorithm that prevents overlap between these segments.

Line 155:

• A session refers to a single stage or individual experiment in a neuroscience study where data is collected. If you want to know more about session, please read this link[2]. The input dimension of the model is the same as the length of the segment. For example, if there are 300 time points, the first session's Xse would be represented as [1, for i in range(300)].

Line 161:

• Based on the previous question, if a neuron only has data from a single session, we will randomly select non-overlapping segments of lengths ranging from 200 to 512 time points as positive pairs for that neuron. For these segments of varying lengths, we ultimately need to map them to the same dimensionality. Adaptive average pooling is a suitable tool for this task, as it does not require explicitly specifying a fixed pooling window size. Instead, we can set a consistent output size, and the pooling operation will automatically adjust to accommodate the varying segment lengths. Adaptive average pooling is capable of handling inputs of different lengths and extracting important feature information, mapping them to a consistent dimensionality.

Line 202:

• It is set to a fixed value of 1, as we referenced in the original VICReg paper and other works that utilize VICReg, which all adopt this setting. The choice of the target value is an interesting topic worth further exploration, especially in terms of its impact on negative pairs. However, this would become the focus of another paper dedicated to contrastive learning.

Figure 4:

• We replaced the original bar charts with a table to present more detailed information, including the precision, recall, and F1 scores details.

[1] Bardes, Adrien, Jean Ponce, and Yann LeCun. "Vicreg: Variance-invariance-covariance regularization for self-supervised learning." arXiv preprint arXiv:2105.04906 (2021). [2]https://allensdk.readthedocs.io/en/latest/visual_behavior_optical_physiology.html#session-structure

• Our model is a two-stage design: the first step: the process of obtaining neuronal representations is a self-supervised process, sharing a single model, and any new mouse data can be added to the old data for training. Step 2: The training set and test set are presented in the second step, and K-Fold is used for downstream tasks on the neuronal representations obtained in the first step, and folds along the mice identity.
• We enriched the content of Figure1 and wrote a detailed caption. The revised pdf has been uploaded.
• The role of CEBRA is to integrate the surrounding information of a single neuron, and if the surrounding information is not integrated, the model cannot converge if it is directly input, which is what we did at the beginning, and then we found CEBRA to solve it. Other contrastive learning methods require the design of negative sample pairs, which is relatively complex for neuroscience data, and the design of negative sample pairs has a greater impact, while VICReg does not need to design negative sample pairs
• We performed a k-fold experiment with three different random numbers for each fold, and the final results were reported as mean and standard deviations, and the results table has been updated.

Thanks for your advice. I hope you can consider it again!

Thank you for your response. That resolved most of my questions. I'm still a bit unclear on how NeurPIR incorporates activity from other neurons. $X_{st}$ in Equation 1 represents the visual stimuli, so it seems actual activity of other neurons in the population is not included as inputs to NeurPIR. Could you clarify on this point and also provide the dimensionality of each component $X_{st}$, $X_{be}$, $X_{se}$, $X_{si}$?

Minor points: the revised manuscript looks better than the original; however, some typos and formatting issues still remain in this version:

• Some of the terms are used inconsistently, such as NeuPRINT (sometimes written as NeuPrint, Neuprint, NeurPrint), Lamp5 (sometimes written as LAMP5), Vis, Thal, Hipp, Mid (written as vis, thal, hip, mid on line 425), $X_{st}$, $X_{be}$, $X_{se}$, $X_{si}$ in Figure 1 should be properly subscripted.
• Rows/columns in tables should be in increasing order of average performance.
• Typos on line 424 (should be "LOLCAT and NeuPRINT"?) and 428 (sentence missing subject)

Time complexity of NeuPIR compared to other baselines?

• If we disregard the time spent on data preprocessing, the time consumed by all methods would indeed increase proportionally with the multiple of the sample size. You might have doubts about this statement, especially when we employ contrastive learning, as it should not exhibit proportional growth. However, it is important to note that we are using VICReg, which only requires the training of positive sample pairs. About time cost, Neuprint takes the representation of neurons as an input embedding of a transformer and updates it using a backpropagation algorithm, which is difficult to train, takes 25 hours, takes up 60 gigabytes of memory on top of the A100(80G). LOLCAT is a supervised training, it is the easiest to converge, the same A100(80G) above, takes 1 hour, uses 10G memory. Our method is a self-supervised contrast learning process, which is the same as A100(80G), takes 15G memory and takes 4h to converge.

Would it be possible to evaluate on more fine-grained information such as spatial location of individual units?

• We can conduct more fine-grained brain area classification experiments, but if the brain areas are divided more finely, the number of samples in each brain area may not be sufficient, and the neurons in some brain areas are very similar, making it difficult to distinguish. These are all worth exploring in the future, thank you for the new inspiration

How many data samples are required to learn effective embedding?

• each session we use is 5000400 (timepointsneurons), each neuron we will sample 8 positive pairs, effective embedding needs at least two session like this.

We have revised the paper to do K-fold cross-validation, and each fold is experimented with with a different random number and reports the mean and variance. We tried to delete any part of VICReg for experiments, but no convergence could be achieved if any part was deleted.