Forecasting Whole-Brain Neural Activity from Volumetric Video

Thank you for the review.

It does not offer additional insights or novel perspectives to advance the field of artificial intelligence. Therefore, this manuscript would be more appropriately submitted to conferences or journals focused on neuroscience or medical image processing, as it does not align with the thematic scope of ICLR.

We respectfully disagree with this statement and refer the reviewer to the ICLR 2025 call for papers that lists applications to neuroscience as one of the relevant topics. Accordingly, we selected “Applications to neuroscience & cognitive science” as the primary area for this submission, as can be seen on top. We further emphasize that our work does indeed advance the field of AI by extending existing methods to a new, complex, and data-rich application domain, as well as by technical advances needed to scale them to volumetric videos.

The authors fail to elucidate the significance of this study anywhere within the main text, which raises questions about the necessity of forecasting whole-brain neuronal activity. The authors are encouraged to provide additional context in the abstract or introduction section.

Thank you for this suggestion. We extended the introduction section of the paper to clarify the relevance of brain activity forecasting.

The paper presents a limited comparison with only one baseline, namely the "trace-based model," as shown in Figure 6. It raises the question whether ZAPBench, as a benchmark, evaluated only this single model type. The authors are encouraged to include additional baselines for comparison to substantiate the superiority of the proposed method.

We compare to the best trace-based model on ZAPBench (c.f. line 383 in the revised manuscript), selected from a set of four state-of-the-art time series models. Also note that we are the first to propose a video-based model for such data and therefore unfortunately no existing video baselines to compare against exist. It is highly non-trivial to scale existing video-based models to the volumetric case while maintaining reasonable computational performance.

On line 308, the authors refer to a "segmentation mask," yet on line 68, they claim that their method can directly process 4D data. The authors are requested to clarify the apparent contradiction in their narrative.

Our method performs video-to-video forecasting of neural recordings. However, to make a fair comparison and quantify the benefit of our method, we compared it to trace-based models. We do this by applying the segmentation mask to the voxel-level video forecast (i.e., model output).

In the fourth contribution (line 108), the authors state that their proposed method is the "only approach that consistently benefits from multivariate information." However, I did not encounter any experimental justification related to multivariate information in the main text. If such experiments were conducted, please direct me to the relevant sections within the paper.

The best time series-based method on ZAPBench is a univariate model, which we also compare to. The multivariate time-series methods did not show a performance improvement. However, the video-based model performs best and relies on multivariate information (the model processes information from multiple cells simultaneously). Sec. 3.2 and Figure 5 in the paper clearly show and quantify the benefit of increasing the receptive field of the UNet, which corresponds to utilizing more multivariate information. Furthermore, based on a suggestion by reviewer 5NDV, we now also include an ablation evaluating the importance of the information contained outside of the cell masks. This additional experiment reveals that unsegmented voxels do not have a significant impact on the video model's performance.

The description of temporal dimension processing on line 200 is unclear. I would like to confirm whether the authors' approach involves merging the temporal dimension with the batch dimension, such as transforming the data shape as follows: (batch, 2048, 1152, 72, T) --> (batch*T, 2048, 1152, 72). If not, please provide clarification on their methodology.

We do not merge the batch and temporal dimension but, as pointed out in lines 215 and following, use the temporal dimension as channels/features. Since we have a grayscale image, we would normally have a single channel at the input level. Here, we instead use the frames as input channels, which helps significantly with model scalability.

We respectfully ask the reviewer to reconsider their evaluation in light of these clarifications.

Thank you for the review and for acknowledging the novelty of directly modeling volumetric video for forecasting brain activity and the extensive experimental evidence we presented.

“it is unclear whether or not this is due to the additional information that exists in the raw data and that the video-based model is indeed taking advantage of such information.”... “I suggest applying the segmentation masks to the video, i.e. set regions outside of the identified cells as background, and train the video-based model on the masked videos.”

Thank you for this suggestion. We have now performed this experiment and added the results in the updated revision of the paper, finding that: “The grand average test MAE for that model (0.0266±0.0046) was not significantly different from that of the video model processing the complete volume (0.0267±0.0042). This indicates that the unsegmented voxels are unlikely to contain information that could improve forecasts and that any gains relative to the trace-based models can be attributed to the utilization of the spatial distribution of the underlying calcium signals within and across the segmented cells.”

“MAE might not be the most intuitive metric for getting a sense of performance” … “I suggest the authors include metrics that are commonly used in neural response prediction”.

Thank you for the suggested metrics. Unfortunately, we do not have sufficient repetitions within the trials to reliably estimate the variances required by these metrics. Nonetheless, we updated the paper to refer to the mentioned prior work as potential future evaluation metrics for cases where more trials are available.

While an improved MAE on the test set clearly indicates a better generalizing model, we agree that the absolute numbers are hard to interpret. To provide an intuition of what an MAE difference of e.g. 0.005 as reported in our results can look like, we included a new supplementary figure (Figure 10 in the revised manuscript).

“Can the authors share the time it took to train the final (best) video-based and trace-based models?”

We have added this information to Appendix A.4. We acknowledge the significantly increased computational cost of this approach, but also note that the reduced extent to which the raw data needs to be preprocessed when forecasting directly in the video domain.

Minor questions

• Minor Q1: The frame rate of the video is roughly 1 Hz.
• Minor Q2: We use $C=4$ and $C=256$ as two extreme cases to assess the relevance of temporal context and quantify it in Figure 5. Indeed, it shows the difficulty of predicting $H=32$ frames from only 4 conditioning frames. However, generative natural video models can generate many more frames starting from only a few and we therefore argue that it does make sense to at least assess performance in this regime.
• Minor Q3: We indeed optimize the trace-based MAE for the video models so that a fair comparison with the trace-based models can be made. If we optimize the voxel-wise MAE, the models perform relatively worse when evaluated with trace-MAE, as it corresponds to a different weighting of neurons by their size.
• Minor Q4: The hyperparameters (learning rate, optimizer, weight decay, dropout) were optimized on the validation set using small grids of 3-4 values each, starting from commonly used values.

Thank you for the positive review acknowledging the quality of the work and the significance of advancing neuronal forecasting. In the following, we address your concerns and questions:

• We edited the abstract and introduction to improve the clarity issues you pointed out regarding the task, segmentation mask, and receptive fields.
• We extended the caption of Figure 2 as suggested. Indeed, the colored blobs denote the segmentation masks of individual cells.
• In the paper we follow the benchmark setup of ZAPBench, which poses the task of predicting $H=32$ future timesteps. Longer time horizons currently seem to be less interesting as performance of all existing methods significantly deteriorates to the level of naive baselines beyond 32 timesteps. We note that shorter time horizons are automatically part of the current evaluation.
• We completely agree with your assessment that these techniques should not be expected to work (out of the box) in all domains. This is what we wanted to express by calling them "counterintuitive" -- to call the attention of the reader to the fact that different strategies might be required in the video forecasting domain. We have now softened the statement and just say "contrary to our expectations" instead.

Could "scaling the field of view of the network so the size remains constant while increasing to full resolution" (line 349) make the full-resolution model handicapped in terms of field of view?

The higher resolution models use one and two more downsampling blocks for the 2x and full resolution models, respectively. Therefore, we make sure they are not handicapped in terms of their capacity and field of view. Details can be found in Appendix 1.3 and specifically from line 644 starting with “To ensure equitable comparison, the architectures of these models were kept broadly consistent, with necessary adjustments to accommodate the differing input resolution”.