ZAPBench: A Benchmark for Whole-Brain Activity Prediction in Zebrafish

Thank you for your positive review and the constructive feedback, which we address below.

Do any other whole-brain, cellular-resolution activity datasets exist publicly for zebrafish?

We are not aware of any datasets with comparable size or degree of preprocessing that are public. For the ZAPBench dataset, a significant amount of work went into acquiring ground truth cell-level annotations (multiple person-weeks) and achieving reliable motion stabilization and cell segmentation results.

The authors could do more to describe which cellular-resolution calcium imaging datasets already exist. While the authors argue that their contribution is to the forecasting community, I think some publicly available datasets could be trivially processed for this goal as well. Specifically, Nguyen et al., 2016 (which they reference), and the MICrONS dataset arguably serve offer similar information as the authors' dataset, in other organisms.

We added a citation to the MICrONS collaboration to the introduction. Our goal with this dataset was to model whole-brain activity in a vertebrate with a high degree of coverage, and, in the future, be able to combine this data with structural information from the same specimen.

Are there any preliminary results in the EM analysis? e.g. Can you show preliminary results of registering cell bodies across EM and calcium imaging? Can you show the reconstructed morphology of one of the neurons in the calcium data? I think some results like this are necessary to justify inclusion in this paper.

We added a short note about the current state of the EM reconstruction in Appendix B.8, including Fig. S9 illustrating the quality and extent of the registration and sample neuron reconstructions.

Why do you choose MAE over MSE, which might have advantages in optimization? Zeng et al., 2023 includes both MAE and MSE.

For simplicity we focused on a single metric. MAE is straightforward to interpret and anticipating possible future inclusion of probabilistic methods, we favored MAE over MSE as CRPS reduces to MAE for the deterministic case. We now include MSE results in addition to MAE results (supplementary figures S5 and S6).

Doesn't the output of the stimulus model also depend on past covariates and past activity? The equation only shows future covariates as an input.

The equation shows how the output of the stimulus model is generated at test time; we clarified the text to better reflect this.

As someone new to calcium signal analysis, an MAE of < 0.03 seems surprisingly good for the naive models - why do you think this is? I think computing the global variance of the data, or the variances of the individual neurons could help contextualize whether a 3% average error is good or not. This is related to my earlier comment about colorbars on the trace heatmaps.

We added colorbars to the heatmaps. Errors are small since neural activity is sparse and we average over all ~70k cells. To provide a better intuition for these numbers we developed an interactive viewer for all predictions, a demo video of which is uploaded as supplementary data. Links are not yet included to ensure double-blind review.

"Variability due to seeding" - does this mean initialization of the model parameters? The term "seed" is also used in FFN inference, which potentially overloads the term.

We clarified this in the text, thanks!

How does the stimulus baseline work for the TAXIS dataset, where there is no training dataset available for lookup?

Thank you very much for pointing this out! We adjusted figure 5, supplementary figure S2, and the text accordingly.

The flood-filling network approach was designed for tracking neuronal processes, but the segmentation task here is cell body instance segmentation

FFN is an instance segmentation method, which is optimized for tracking objects larger than the field of view of the network through space thanks to its recurrent nature. We hypothesized that segmenting somas which fully fit within a single field of view would not require recurrent processing, and simplified the model accordingly. In early experiments, we found neither simple heuristic methods nor pre-trained segmentation approaches to work well, and so we focused further efforts on the FFN.

It is hard to see much detail in several of panels with whole-brain views. Consider enlarging panels or adding more insets to, especially, Fig 1B, Fig 2C

Thanks for raising this. We have interactive versions of both of these figures and plan to include links to neuroglancer/fluroglancer for zoomable views in the final version of the manuscript.

It is hard to interpret Fig 2B, I encourage alternative registration visualizations e.g. showing deformed gridlines, overlaid images, and/or differences between overlaid images.

We will add an interactive side-by-side view in the final version of the manuscript. We also added an inset to Fig 2B showing the same inset as in Fig 2A.

Thank you for the positive review acknowledging key strengths of our work. We address your feedback below.

Why does the paper report data only from one zebrafish brain? What are the key challenges, and given the work you have done so far, can you collect data from more brains?

The specimen we used for the benchmark is the result of extensive screening for data quality, covering both animal behavior and imaging. For instance, specimens might not show robust responses to standard stimuli or fail to display movement characteristics expected from prior work. Similarly, the acquired images might suffer from artifacts resulting from unexpected minor movements of the microscope, insufficient action of the paralytic applied to the fish, or instabilities of the laser illumination.

Additional volumes could be collected with some experimental effort. While we expect our preprocessing pipeline to work for similar volumes, its various stages will require tuning for any specific volume used (in particular, the segmentation model might require the collection of more annotations). The use of additional volumes would also require careful cross-specimen spatial registration at cellular resolution, which is a challenging problem.

We added a discussion of some of these issues to Appendix B.7.

Why does the paper not report estimates of action potential? I am not an expert in the properties of GCaMP. Can you clarify the limits of accuracy with which action potential be estimated from GCaMP activity?

Because of throughput limitations of the imaging setup (~1 Hz volumetric), we used a nuclear-targeted GCaMP calcium indicator. The fluorescence kinetics, as well as the kinetics of calcium itself within the nucleus are slower than those in the neurites (this is also mentioned in section B.7). The signals we record do not allow the detection of individual action potentials and can be considered a low-pass filtered image of the underlying high-frequency electrical activity of the neurons.

JAX's adoption is growing, but PyTorch is still more widely used. What can you do so that the community can reuse your code, e.g., wrap it with CLI and containerize the code?

Our input pipelines and utilities are usable with both JAX and PyTorch. Reproducing JAX runs will indeed be possible through a CLI command. Since JAX can be easily installed through pip we currently do not plan to provide a container. We will however provide a set of tutorials to ensure that it is as easy as possible for others to build on our work.

The MAE (mean absolute error) metric is insufficient to develop probabilistic models and models of underlying biology: An important class of models is the ones that predict the probability of underlying biology activity, e.g., the probability of action potential from GCaMP activity or estimating functional connection between pairs of neurons from synchrony of activity. The mean absolute error metric per neuron is unlikely useful for ranking such models. The authors should introduce a metric that enables modeling of the statistical distribution of activity per neuron or pair of neurons.

We agree that probabilistic models are an important class to explore in the future and now explicitly refer to Continuous Ranked Probability Score (CRPS) as a metric to consider for evaluating them in the conclusions of our revised manuscript. Also note that CRPS reduces to MAE in the deterministic case, and that predictions of all models will be made available in case researchers want to compute additional metrics.

Thank you for the review and for your positive comments regarding originality, quality, clarity and significance of our work. We address your feedback below.

It would be helpful to understand the rationale behind the chosen models. Are they meant to represent a range of prediction approaches, or were they selected based on prior performance in related studies? This clarification can provide insight into how comprehensive the benchmark is in capturing predictive capabilities across model types and whether the authors plan to expand the set of models in future work.

For our selection, we did an extensive literature review and selected state-of-the-art models. We preferred simple over more complex architectures and sought to explore different types of input-output mappings.

In particular, we selected both TSMixer (Chen et al., 2023) and TiDE (Das et al., 2023) based on their state-of-the-art performance with comparatively simple architectures on a range of common benchmark problems. TSMixer can model multivariate dependencies whereas TiDE is a global univariate model with support for various types of covariates which we explore through ablations. We consider the simple linear model an important baseline, as it was shown to outperform sophisticated transformer-based architectures in Zeng et al. (2022). In addition, we wanted to contrast time-series forecasting with end-to-end forecasting from video, and include naive baselines to calibrate performance.

Note that our selection is not meant to be exhaustive. We aimed to include baselines that are simple while setting a reasonable performance target to hopefully outperform in the future. As discussed in our conclusions, we see a number of directions worth exploring and hope that ZAPBench can be a catalyst for the development of increasingly accurate models of brain activity. We look forward to others engaging with the dataset and benchmark, and also plan to explore additional models ourselves—future availability of the proofread connectome will open exciting opportunities for follow-up work, e.g., using graph-based approaches.

Lack of Real-World Validation: Although ZAPBench provides a controlled dataset for benchmarking, validation in diverse environments or settings would strengthen the applicability of its findings. For instance, if the framework could be tested or at least hypothetically mapped to real-time or less controlled environments, this would add value by showing the benchmark’s robustness and adaptability.

We agree that ultimately it would be interesting to close the loop from predictions back to experiments. Unfortunately, this is far beyond the scope of the current submission.

Additionally, introducing comparisons with other neural prediction benchmarks, if available, could highlight the strengths and limitations of ZAPBench in relation to existing datasets and benchmarks, increasing confidence in its practical utility.

We discuss related neural prediction benchmarks in the introduction and highlight that ZAPBench is unique in its coverage. To the best of our knowledge, existing benchmarks focus on a small fraction of the brain rather than targeting single-neuron resolution for a whole vertebrate brain.

Limited Analysis on Predictability Boundaries: Limited Analysis on Predictability Boundaries: While the paper raises the important question of the fundamental limits of predictability, it could benefit from a deeper analysis or even a dedicated section on the boundaries of predictability within neural systems. Specifically, the authors could include additional metrics or scenarios that highlight instances where predictability fails or reaches theoretical limitations. This would provide researchers with a clearer understanding of where model improvements might be focused, enhancing the practical impact of the benchmark.

Thank you for raising this. As discussed in our conclusions, the “performance ceiling” for this task is unknown, so hill-climbing against progressively stronger baselines seems like the best practical strategy. Theoretical insights into this issue would of course be very interesting -- however, we currently do not see a clear path towards this.

Regarding the question what future improvements might focus on, we outline a number of directions we consider worth exploring in our results and conclusion sections. For example, our analyses suggest that time series models may so far underutilize multivariate information. We also identified particular brain areas that are less well predicted than others, and mention that we currently do not include probabilistic approaches. Future availability of the connectome will open exciting opportunities for grey-box approaches. We will make all model predictions available upon publication, so that researchers have the possibility to compute additional metrics or systematically screen for scenarios in which predicability is currently low.

Thank you for the positive review acknowledging the value and quality of our work. We address your feedback below.

Regarding time-series predictions, there are no sota methods employed in this study, e.g. temporal fusion transformer, informer, n-beats, and deepar to name a few.

The publication introducing TSMixer (Chen et al., 2023) found that it outperformed Temporal Fusion Transformer, Informer, and DeepAR. The authors of TiDE (Das et al., 2023) found that it outperforms N-HiTS (Challu et al., 2023), an improved version of N-BEATS, among other state-of-the-art methods. Therefore, having evaluated TSMixer and TiDE on ZAPBench, we respectfully argue that we do employ SOTA methods.

The authors are encouraged to discuss how likely the results abtained from a single test zebrafish to other live zebrafishes.

Thank you for raising this issue. We expect forecasting models to lose predictive power when they are applied to recordings from a specimen different from the one the models were trained on. The updated paper now includes these remarks in our discussion of this limitation in Appendix B.7.

For the 2000 neurons manually annotated, how accurate they are? Are there any metrics to measure the segmentation accuracy?

Thank you for these questions. We think that our training set of manually annotated neurons is highly accurate. We worked with a team of annotation specialists, who developed a custom protocol for this data, and went through several rounds of iterative feedback and refinement with them. The annotation protocol was published for reproducibility of the workflow—we currently omit the reference for double-blind review.

Following submission, additional ~1,300 neurons not included in the training set were labelled by the same annotators and protocol. Using these new annotations, we report metrics measuring the accuracy of our automatic segmentation in supplementary table S1 in the revised manuscript (counts and variation of information statistics).