Scaling Properties For Artificial Neural Network Models of the $\textit{C. elegans}$ Nervous System

Dear Reviewer 3D8y,

Thank you for your comprehensive review and valuable feedback on our manuscript. We appreciate the opportunity to address your concerns and questions.

1. Self-Supervised Learning:
We acknowledge your observation regarding the term "self-supervised." Our approach aligns with its usage in natural language processing (NLP), focusing on tasks deriving supervision from the input data itself, such as sequence-to-sequence, one-step future prediction tasks. We recognize the broader scope of SSL and will clarify in our revised manuscript that our application pertains to the sequence prediction paradigm in NLP, where the model learns to predict future states based on past and present data.

2. Causality in Model Predictions:
Regarding causality, our models are designed for sequence-to-sequence prediction, with the target sequence being a one-time-step forward shift of the input sequence. This setup implies that for each time point in the sequence, except the last one, the model has access to future states. However, the LSTM and Transformer models (set with is_causal=True) inherently enforce causal prediction. Our Feedforward model core is very similar to the input-to-hidden and hidden-to-output nonlinear transformations that you commented on: “The formulation in step 1 on page 4 is not clear if the non-linear projection operates at each time point independently or with a causal memory.” The nonlinear projection operates at each time point independently. To put it another way, the transformation is applied along the feature axis, not the temporal axis. Thus, there is no mixing or breaking of temporal order, respecting causality. These clarifications will be detailed in the revised manuscript.

3. Relation to Neural Population Dynamics:
We understand your concerns regarding the relation to neural population dynamics. It's important to note that almost all of C. elegans' neurons exhibit graded potentials (Jiang et al. 2022), not action potentials/spikes, which influences our modeling approach. Our methods are tailored to capture these unique dynamics, rather than traditional spiking activities. This distinction will be more clearly highlighted in the revised manuscript.

4. Amount of Data:
We appreciate your constructive criticism on the "Amount of Data" section. We plan to revise this section to clarify our methodology. We create a combined pool of worms from all sources and sample subsets of varying sizes. So a dataset in this case is a particular subset of $n$ worms. For each such dataset, we create the training set by taking the first half of the neural activity recording of each of the $n$ worms. Since each worm has some $v < 302$ of neurons measured, we calculate for the training set a “time steps per neuron” quantity as the length of the neural recording divided by $v$. Hence for each dataset of $n$ worms, we have a distribution of “time steps per neuron” (hence the mean and horizontal error bars in Figure 3). The creation and calculations for the validation set are similar, with the neural activity being the second half of the neural activity recording of each worm. The revised manuscript will provide a more coherent explanation of this approach.

5. Sampling and Filtering:
We will replace FFT smoothing with a simple Gaussian filter in our preprocessing method, using a standard deviation hyperparameter optimized through cross-validation to suit the dynamics of C. elegans neural activity. This will be clarified in our revised manuscript.

Responses to Your Questions:

• Causal Predictions: Our LSTM and Transformer models are inherently causal. We will modify our Feedforward model to ensure causality.
• Maximal Time Delay: We plan to run experiments to answer your question about the maximal time delay for predictive models. We understand this as the offset of the target sequence from the input sequence, currently at a delay of 1. If we correctly understood, this question might be rephrased as, "How many future timesteps can models predict before deviation from true neural activity becomes substantial?"
• Auto-regressive Prediction Capability: Our models are capable of auto-regressive prediction, particularly the LSTM and Transformer models.
• Rationale for Lowest Frequencies: The choice of retaining only the lowest frequencies was initially based on an attempt to remove as much noise from the neural activity traces while still maintaining high cosine similarity to the original (non-smooth) signals. However, we now realize that the amount of smoothing was excessive and unnecessary. Our improved models can still achieve better-than-baseline performance with minimal or no smoothing of the neural activity data. We will revise this in our updated manuscript.

We are committed to addressing these concerns and enhancing the clarity and quality of our paper. Thank you for your valuable feedback.

Sincerely,
Submission 3075

Dear Reviewer 3D8y,

Clarifications on Model Predictions and Causality: We value your further insights and agree that considering the length of the context window is crucial for capturing short-time calcium dynamics. Our Feedforward model, serving as one type of baseline, probes predictability based solely on the feature dimension, an approach intended to benchmark against more complex temporal models.

Linear Core Without Memory: We've designed the Feedforward model to interrogate the predictive capacity inherent in neural activity patterns without temporal history—akin to a feature regression.

Feedforward Model as Baseline: It is designed to act as a baseline feature regressor to assess how much predictivity is present without temporal memory. This approach helps us to understand the significance of temporal history in predictive performance.

Rationale for Baseline Loss: The baseline loss using the naive predictor, which assumes the next timestep will be identical to the current one, serves as the best predictor for time series resembling a random walk. It establishes a performance floor, especially given the temporal correlations often observed in neural activities.

Temporal Structure Analysis and Request for Guidance: While we acknowledge the potential insights that could be derived from studying the auto-correlation and cross-correlation structure of different neurons, we face certain challenges with our dataset that make the execution of this analysis complex.

Variability Across Subjects: The measured neurons vary from worm to worm, which means standardizing a correlation analysis across multiple datasets is not straightforward. If in a particular worm there are $k$ measured/labelled neurons (out of the possible 302), we would be conducting $k$ autocorrelations and $k(k−1)$ cross-correlations for that worm, but the comparison across different worms and datasets becomes less clear.

Aggregation of Analysis: Aggregating this data meaningfully poses a significant challenge. We are considering focusing on common neurons measured across worms and applying statistical methods to manage missing data.

Seeking Reviewer's Expertise: We genuinely seek your input on how to approach this large-scale correlation analysis. How might we compare neurons across animals in a scalable way? What insights do you believe such an analysis could provide, given the diversity of neuron subsets measured? We are earnestly interested in your thoughts and suggestions on how we can tackle these challenges to uncover meaningful structures within the neural activity data of C. elegans.

Ensuring Causality in Data Preprocessing: In response to your advice on filtering methods, we have changed our preprocessing to utilize an exponential kernel smoothing method.

Exponential Kernel Smoothing: This method is inherently causal, using only current and past data points to compute the smoothed value, thereby maintaining the causality of our data.

Confidence Score Clarification: We noted your lowered confidence score and would like to confirm if this update will be reflected post-rebuttal, as it has not yet been modified in the review system.

Future Directions and Request for Suggestions: We invite further suggestions on how to best approach the large-scale analysis of correlation structures and are particularly interested in your perspective on how to handle the variability in neuron measurements across different worms and datasets.

Thank you for your feedback, which has been instrumental in refining our approach. We aim to address the critical points you have raised in our ongoing and future research.

Sincerely, Submission 3075

Dear Reviewer KaWC,

Thank you for your detailed review and constructive feedback on our manuscript. Your insights are invaluable to improving our work.

Code Availability: We understand the importance of code availability for reproducibility. In accordance with ICLR's double-blind review guidelines, we have refrained from linking to code repositories during the review period to preserve anonymity. We plan to release our code publicly in the final version of our manuscript, post-review.

Model Architecture: Your observation about exploring models with more than a single layer in the core is appreciated. Our decision to use single-layer core models was intentional, aiming to maintain simplicity while ensuring sufficient expressiveness. Deeper models can obscure interpretability, a key consideration since we plan in future work to analyze the learned weights and their relation to the relatively simple neural network of C. elegans. We believe that for modeling a biological neural network like that of C. elegans, wide, shallow network cores offer a balance of interpretability and expressiveness. We will elaborate on this decision and its implications in the revised manuscript.

Teacher Forcing in Inference: We agree that our description of teacher forcing was unclear. We use two strategies during inference: Autoregressive Generation and Ground Truth Feeding. We acknowledge the need for a clearer distinction between these two inference strategies. Autoregressive Generation relies on the model's own outputs for subsequent predictions, while Ground Truth Feeding uses actual data to guide predictions. The latter was incorrectly referred to as teacher forcing, which we will correct in our revised manuscript to avoid any confusion. We will rectify this in the revised manuscript, clarifying these strategies and their roles in model inference.

Neuron Coverage: The coverage of neurons across datasets is important. If certain neurons are present across many datasets, we believe this makes the models prediction that neuron better. We have included a figure that showcases the coverage of neurons when the worms from all our seven datasets are pooled together, which we had prepared as part of our initial analysis. This figure will be available in the appendices of our revised manuscript and will illustrate the distribution of recordings for each neuron within our meta-dataset. Here is a (hopefully anonymous) link to such a figure showing the neuron distribution in the train and test splits; note that distribution is the same since currently we are just splitting temporally for train (1st half) and test (2nd half): https://www.reddit.com/user/Successful_Carob3111/comments/17v9vcl/neuron_distribution/?utm_source=share&utm_medium=web2x&context=3

Choice of Frequency Retention: The decision to retain only the lowest 10% of frequencies was initially an attempt to reduce noise while maintaining high cosine similarity with the original signals. The initial choice to retain the lowest 10% of frequencies was to reduce noise. However, further investigation revealed that our models perform adequately with less or no data smoothing. We will update our preprocessing to use instead a simple Gaussian smoothing with a cross-validated standard deviation hyperparameter.

Linear vs. Log Scale in Figures: The x-axis in Fig. 3 uses a log-scale, which was not explicitly stated. We will update the figure to display more x-tick values and clarify this in the revised manuscript.

Experiments Without Learning Rate Scheduler: We initially ran experiments without a learning rate scheduler, and the results were consistent with those reported. Additional optimizations like the LR scheduler and early-stopping improved efficiency.

Metrics and Test/Train Split: Yes, All metrics reported in Section 3 derive from evaluations on validation sets. We will update the manuscript to clarify the one time-step offset in the formula. We will include these details in our revised manuscript. Swapping and restructuring the train/test splits did not significantly affect the results, affirming the robustness of our findings. Our manuscript will discuss these tests, underscoring our straightforward but effective splitting strategy and its implications for model evaluation.

Your feedback has been instrumental in helping us refine our paper. We look forward to incorporating these changes and providing an updated manuscript.

Sincerely, Submission 3075

We greatly value your thoughtful critique and aim to address the points you've raised with as much detail as possible.

Motivation for Using C. elegans:
We realize the necessity to strengthen the rationale for employing C. elegans datasets. Our motivation is predicated on its simplicity and the high degree of experimental control it offers, allowing for the exploration of neural dynamics in a well-defined biological system. We intend to enhance the introductory section to better encapsulate the novelty of our findings, namely the specific scaling properties of neural predictors within this biological context, which have not been extensively documented before.

Preprocessing Concerns:
Your concerns about our preprocessing steps resonate with similar concerns raised by Reviewer 3D8y. In response, we have modified our preprocessing approach. Specifically, we have changed our smoothing method from FFT smoothing to an exponential kernel smoothing method. This new method is a causal filter, addressing Reviewer 3D8y's concerns about causality in preprocessing.

Additionally, it's crucial to highlight that our preprocessing aims to abstract away from the specifics of any particular dataset, striving for methods general enough to be applicable to C. elegans neural data recorded under various behavioral and experimental paradigms. The slow dynamics of calcium indicators, which are a known issue, influence our decision to adopt a unified preprocessing approach. Different labs may use various versions of calcium indicators with differing dynamics, and our methods are designed to bring all datasets to an "equal footing," allowing us to utilize them in a unified manner. This approach focuses on extracting shared neural structures inherent in the stereotyped nervous system of C. elegans, without getting bogged down in the intricacies of individual datasets.

Preprocessing and Sampling Rate Clarification:
Regarding your question on our choice of a uniform timestep ($\Delta t = 1.0$ second), we understand there might be a misunderstanding stemming from our explanation in the manuscript. To clarify, our objective was to ensure a consistent sampling rate across all datasets, which necessitated adjusting all datasets to a common slower rate. This common rate was chosen to be at 1 Hz (1-second intervals), which is slower than the lowest original sampling rate of approximately 1.67 Hz (as per the dataset with a 0.6-second interval, shown in subplot F of the attached figure). Hence, our resampling was indeed a form of downsampling for every dataset, as each original dataset had a faster sampling rate than 1 Hz.

The intent behind this approach was to avoid the introduction of artificial data points through interpolation, which can occur during upsampling. By resampling to a 1-second interval, we are effectively reducing the data points to a rate that is equal to or less frequent than the original data, thereby ensuring no upsampling occurs.

To provide further clarity, we have included a plot that demonstrates the mean sampling intervals across the datasets we have used, highlighting that none of the original sampling rates were slower than our chosen uniform rate of 1 Hz. [https://www.reddit.com/user/Successful_Carob3111/comments/180s79j/sampling_interval_datasets/?utm_source=share&utm_medium=web2x&context=3].

Response to Specific Queries:

• Analytical Accessibility: We will clarify that by "increased analytical accessibility," we refer to the progress in making ANNs more interpretable, akin to the manipulability and transparency of the C. elegans system.
• Aligning C. elegans with ANNs: We acknowledge the unique attributes of the C. elegans neural network. Our comparison with ANNs is in the context of their use as predictive models, not a direct equivalence. We will adjust the language to reflect the distinct features of C. elegans while discussing its utility in modeling.
• Emergent Dynamics: We concur that "emergent" dynamics could more accurately describe the complex patterns arising from the neural activity in C. elegans. The revision will incorporate this term for precision.
• Ignoring Behavior in Neural Dynamics: Our study indeed focuses on neural activity alone to ascertain the predictability inherent to the neural data. We hypothesize that certain predictive capabilities can be derived from the neural activity patterns without behavioral context. This hypothesis will be further expounded in our revised manuscript.
• Data Sampling and MSE Justification: We will provide a more robust justification for our choice of a 50:50 train:test split and the use of MSE. The manuscript will elaborate on how these decisions align with our objective of assessing the ANN's ability to predict future neural states and the relevance of this metric to our analysis.

Train-Test Split and Data Utilization:

• Choice of a 50:50 Train-Test Split: Our decision to use a 50:50 train-test split was an arbitrary choice but roughly aimed at ensuring that the data distribution for both training and validation is similarly rich. This split challenges the models to generalize well over a diverse range of data. We avoided larger train-test ratios to prevent the validation set from being a simpler subset of the training set, which could occur if the training set were disproportionately large. By employing a balanced split, we challenge our models to generalize effectively across a diverse range of neural activity sequences.

• Implementation of the Train-Test Split: To clarify the implementation, we uniformly sample sequences of length $L$ from the first and second halves of each worm's full calcium recording to create the training and validation sets, respectively. For the training sets, we sample 32 sequences per worm from the first half, and for the validation sets, we sample 16 sequences per worm from the second half. Consequently, a dataset with $n$ worms produces a training set with $n \times 32$ length-$L$ sequences and a validation set with $n \times 16$ length-$L$ sequences. We use a batch size of 64 for both train and test data loaders.

• Repetitions of Experiments: We conduct repetitions of our experiments with different random seeds. This approach introduces variability in aspects such as the samples drawn from the train and test halves and the selection of worms to create datasets of a given size. These repetitions allow us to obtain standard deviations and confidence intervals for our results, addressing your comment on the assessment of variability in outcomes.

Baseline Loss and Metric Choice:

• Rationale for Baseline Loss: The baseline loss using the naive predictor, which assumes the next timestep will be identical to the current one, serves as the best predictor for time series resembling a random walk. It establishes a performance floor, especially given the temporal correlations often observed in neural activities.

• Using MSE for Neural Data: We opted for Mean Squared Error (MSE) as our metric because it provides a straightforward measure of the difference between vectors of predicted and actual neural activity. Although other losses like mean-absolute scaled error (MASE) have been explored, we found MSE to be more universally understood and sufficient for our purposes.

• Neurally-Salient Approach: The choice of MSE is a starting point for our analysis. We recognize that if behavior were included, different metrics might be more appropriate. However, given the focus of our study on neural activity prediction, MSE serves as a suitable metric. We are open to exploring other metrics in future work, especially as we expand our models to incorporate behavior prediction.

• Control Methods: All models are trained with a maximum of 500 epochs using the AdamW optimizer, an initial learning rate of 0.001, and a learning rate scheduler triggered by validation loss plateau. Early stopping is implemented with a patience of 50 epochs. The training objective is sequence-to-one-timestep shifted sequence prediction, with MSE computed between the model's output sequence (masked along the neuron dimension) and the ground-truth one-timestep shifted sequence. This methodology ensures that our models are rigorously evaluated for their predictive capabilities.

We hope these clarifications address your concerns and provide a deeper understanding of our methodological choices. Our aim is to develop a robust and generalizable approach to neural activity prediction in C. elegans, and your feedback is invaluable in guiding our efforts.

Sincerely, Submission 3075