realSEUDO for real-time calcium imaging analysis

Weaknesses:

1. The main improvement come from engineering the implementation of SEUDO such as to run in close to real time (is that a weakness or strength?).

Our response: Thank you, and we believe that our improvements represent a strength of our approach as we are able to successfully improve and implement a previously slow and unyieldly algorithm in real-time.

2. Lines 228: The merging procedure seems quite ad-hoc and heuristic. The logic is only laid out at the end (lines 247), maybe it would help to discuss this earlier.

Our response: We appreciate this suggestion and agree that this section can be improved. In the revision we will begin this section with the logic of our merging procedure to clarify the steps we take.

3. Only two algorithms are chosen for comparison.

Our response: In our work we selected the most relevant algorithms. As a baseline we selected CNMF as a standard, SOTA, well-maintained method representative of off-line methods. Comparing to other off-line methods we did not believe to be relevant given that we expect them all to perform at or below the level of CNMF (see, e.g., Song et al. 2021 and Charles et al. 2022 for comparisons between offline methods). OnACID currently represented the only CPU-based online method in existence and so our comparisons had to be to it.

4. Fig. 2B is very hard to read and understand. Does this mean, OnACID produces no false alarms and no ambiguous units?

Our response: Yes, that is correct. In the NAOMi data, under the definition from Song et al. 2021 of a found cell (spatial overlap of over 50% and temporal correlation >0.5) OnACID performed very conservatively in the best parameter set we could find for it. This meant that it actually reduced quite a bit of noise as well as true transients. Thus the correlations are overall higher, however the total number of cells was reduced. Additionally, OnACID is still very slow (<10 Hz) meaning that it is not truly "real-time" for this data.

Questions:

1. A lot of engineering work has gone into the fast implementation of the online algorithm using C++ and various bells and whistles. Have the authors considered a GPU based implementation e.g. in jax and how would that compare in performance?

Our response: We had considered both GPU and FPGAs as alternative approaches that could have been viable. While potentially beneficial in some ways, our ultimate goal was integration of our online cell-finding into a larger closed-loop pipeline. In this case bringing the pipeline closer to the imaging is necessary, and most imaging groups don’t have high performance GPUs or the expertise to install and make use of FPGAs. We thus decided that a purely CPU approach would reduce the hardware complexity in the long term as we work with experimental groups to implement this method in closed loop systems.

2. Where does equation 7 come from? Min operations are quite sensitive to outliers. The details won’t matter, but in the spike sorting literature, a threshold of n*0.6745(median (abs(x))) is often used for detecting spikes, which is indeed a multiple of a robust estimate of the noise sd without the spikes.

Our response: Equation (7) simply comes from the desire to assess the width of the pixel histogram mode (the lower half of the histogram as there is usually an asymmetric tail due to active pixels). It is true that the min is usually sensitive to noise, however we actually compute this min on the denoised data which provides a more stable minimum value. We agree that this is unclear in the paper and will clarify this detail in the text.

3. Why was the tool not compared to the other online algorithm FIOLA?

Our response: FIOLA is an online algorithm for demixing, and not cell identification. Specifically it is more of an optimized pipeline that relies on pre-computed ROIs to extract time-traces from online. The authors mention this in the FIOLA paper (page 1420). Thus we did not feel it is an appropriate comparison point. In fact we can envision parts of the realSEUDO concept being integrated into their very excellently engineered pipeline that solves other important problems, such as motion correction and spike extraction.

4. Why was the plain CNMF algorithm used and not one of the more advanced implementations/methods or an additional algorithm?

Our response: We specified CNMF, however used the most recent version available in the CaImAn package provided by the Simons institute. We felt that it represented one of the state-of-the-art methods and was related to OnACID which gave a good complementary point of reference of what the same algorithm could look like on-line and off-line.

5. Doesn’t Fig. 2C speak very clearly for the results of the OnACID algorithm?

Our response: OnACID had varying results, performing better in the NAOMi simulated data, for example, than the CA1 real dataset. Specifically we found that OnACID performed quite well in the simulated NOAMi data under conservative parameters. This is primarily due to the high thresholding value that both reduced noise in the estimated time-traces. However this higher threshold also removed smaller transients (see e.g., Profile 2 in Fig. 2D). Moreover OnACID was considerably slower and unable to run at real-time speeds (<10 Hz as opposed to realSEUDO's >65 Hz, see Fig. 2C).

6. Lines 348: Why do the authors say that a c++ implementation makes the code open source?

Our response: We simply wanted to make sure the community understood that the code was not written completely in MATLAB, which is not technically open source even when released. The C++ code has a wrapper in MATLAB, and we are working on similar wrappers for Python. We will clarify this point in the paper.

Weaknesses: While the performance with a small number of neurons is impressive there are two drawbacks. First, the testing datasets only include a very limited number of neurons and the scalability with much larger datasets are not presented. For example, current methods such as with a mesoscope (Sofroniew et al. 2016), allow for much larger recording samples in the of thousands of neurons. From the figure 3c we can see that the algorithm drops in performance quite fast and when much larger samples are included, it could potentially perform worse than existing methods, something that remains to be tested. Lastly, calcium sensors already limit our ability to extract with fine temporal precision the neural activity thus the usefulness of a method that far exceeds that temporal resolution remains limited. Another potential issue is to validate the model with more typical cortical recordings that have higher level of background fluorescence, and how it deals with overlapping sources of fluorescent signal.

Our response: We thank the reviewer for noticing the positive aspects of our work and the novelty of our contribution. We would like to address the three main weaknesses that concern this reviewer: 1) scaling to very large imaging, 2) speed of calcium sensors and the need for even faster processing, and 3) the potential for overlapping sources of fluorescence.

1. We agree that newer methods have been developed that aim to improve the number of sources that can be detected simultaneously. It is also the case that 500um-by-500um fields of view are still the standard for most labs and our patching approach easily extends to these sizes. We of course are interested in testing the boundaries of the capabilities of our method, and we will mention this limitation presented by the reviewer in the discussion and limitations section.
2. The reviewer is right in mentioning the speeds of most calcium sensors. However we believe that two main points justify even faster processing. For one, new indicators such as GCaMP8f are even faster and may require more than 30Hz imaging to properly capture (and thus even faster processing). Second, the cell finding step is only one part of a closed loop system. The idea is to enable on-line experimental perturbations, which will also require online model fitting and the computation of the right perturbation (e.g., stimulation or experimental change) after each cell’s activity is demixed by realSEUDO. Thus if realSEUDO takes even less time, there is more time for later analysis steps that will enable true closed-loop experiments
3. In terms of overlapping sources of fluorescence, the reviewer is correct that this is a critical issue, especially in on-line demixing where sources of fluorescence are unknown. Thus, a key design choice in selecting SEUDO as the core of our approach was its ability to be robust to overlapping sources of fluorescence even when one source was unknown. The original SEUDO paper, Gauthier et al. 2022, demonstrated this ability, which we further demonstrate remains present in realSEUDO by counting the number of ``false transients’’ (i.e., bursts of activity due to bleedthrough from other sources rather than the cell itself) in Section 4, bottom of Page 7. Specifically, realSEUDO can continue to remove false bleedthrough activity despite the many overlapping sources of fluorescence in the CA1 tissue that we tested realSEUDO on.

Questions: By facilitating the real-time processing of CI data, the paper shows the potential to improve the accuracy and efficiency of closed-loop studies. The claims of broad applicability would be strengthened by more extensive testing under various experimental situations and with diverse kinds of brain data. Additionally, it would be great if they could characterize and show the performance with different patch overlaps. Finally, while the paper discusses the integration with motion correction algorithms from OnACID, integration of all preprocessing steps such as motion correction within the realSEUDO pipeline would further widen its potential usability.

Our response: Again we appreciate the reviewer’s positive outlook on this work. We aimed to focus our validation on datasets with rigorous ground truth, which included real data from Hippocampus and Visual cortex, along with the realistic simulations. As the Hippocampus data had much richer validation data (i.e., it was previously annotated by the authors of the study not just in terms of cells but also time-traces), we focused on this dataset for our work. We will mention these aspects of our study in more detail in the discussion to clarify. Moreover we will make mention of the need to integrate out advance, which focused on one critical step of the pipeline, with other steps such as motion correction that have been addressed by other researchers.

Weaknesses:

1. The work largely relies on previously described algorithms like FISTA and SEUDO. Since a significant portion of the method describes these algorithms in detail, it is unclear what the key contributions of this work are.

Our response: While we leveraged prior developments in the field, we would like to note that the ability to create a method that met the criteria for real-time demixing was not a trivial combination of these tools. Several significant changes, from efficient computation, implementation, and model modification had to work together in conjunction just to have SEUDO run in realtime. Moreover we developed a feedback loop for generating new ROIs (note that SEUDO only identifies time-traces given a-priori known ROIs) in an online manner.

2. The evaluation was conducted on synthetic data (Naomi) and a real dataset (36 videos). Including additional sources like 2p movies from the Allen Brain Observatory would have enhanced the credibility of the results.

Our response: While more results are always better, we carefully selected datasets that had reliable ground truth, either through data generation (e.g., the NAOMi simulation) or through rigorous prior assessment (e.g., the data from Gauthier et al. 2022). Other datasets do not have very reliable ground truth and thus would not serve as well to validate our approach. Moreover data from the Allen Brain Observatory is only disseminated as traces and ROIs, not as raw data. This is especially problematic as the SEUDO paper (Gauthier et al. 2022) previously showed that these are not reliable decompositions of the data.

3. The given metrics should be improved. While the images are good, providing tables with quantitative metrics can enhance the overall presentation and support the claims made. In addition to correlation, metrics like IoU could be evaluated on neuron profiles.

Our response: We appreciate the suggestion. As the IoU implies that the full set of pixels identified is the important quantity, we instead compute the “unique neurons found”. This is since in functional imaging we instead typically need to know that the time-traces 1) correspond to real neurons in the data and 2) accurately reflect the temporal activity that will be used to study neural activity with respect to stimuli and behavior. The Unique Neurons Found requires both that ROIs well align with known ROIs spatially without the strict pixel counting that might not accurately reflect what a neuroscientist deems “success” in regards to cell finding. We will clarify our criteria for selecting these metrics and how the metrics used reflect these key requirements for validation in the results and discussion. We also selected to place the comparison as bar plots to situate them visually next to the images of the identified components and example time-traces, compartmentalizing that example.

4. The experimental setting lacks an ablation study on the main components of the work, such as automatic cell recognition (e.g. with and w/o X_temp) and patching.

Our response: We agree with the importance of ablation studies in algorithmic development, however the overall algorithm is quite simple. In fact directly adding ROIs to the dictionary instead of using X_temp is, as the reviewer saw, really the only section that can be ablated without effectively breaking the ROI selection loop. We have previously had ROIs added directly to the dictionary, which has resulted in a lot of noise being treated as signal. We did directly test an ablation study focused on the speed of computation, slowly adding every improvement. We noted the increasing framerate on the last paragraph before Section 3.2 on Page 5, which shows the necessity of each improvement to the overall performance.

Questions:

1. What are the key methodological differences between SEUDO and realSEUDO?

Our response: There are differences in SEUDO and realSEUDO on both the computation and algorithmic levels. On the algorithmic level we introduced a full feedback loop that identifies ROIs. SEUDO, as derived and presented in Gauthier et al. (2022) is a time-trace estimation method only. It relies on pre-defined ROIs and focuses on robust estimation of the activity for each ROI. We add the ability to use the residuals from SEUDO to detect in an online manner new ROIs to iteratively build a dictionary of ROIs from nothing.

2. How did you choose the benchmark datasets? We chose the benchmark datasets as datasets with extensive information (either through simulation in NAOMi data or manual expert annotation in the Gauthier et al. dataset) on both the ROIs and also the time-traces (e.g., our false positive comparisons). These properties gave us the best targets to validate our data quantitatively.

3. Why correlation was selected as the main (and only) metric?

Our response: Correlation as a metric was also predicated on ROI overlap. Essentially our “Unique cells detected” number combined both temporal correlation and spatial information to claim that we detected a cell well. We isolated the correlations into a separate histogram to highlight the success in identifying the time-traces; a quantity that is vital to the fitting of computational neuroscience models.

4. How do the authors justify the lack of an ablation study on the main components of the work? Can the authors describe the quantitative improvement of their method compared to existing methods they rely on?

In terms of completely removing parts of our algorithm, the main parts are SEUDO and the feedback for specifying ROIs. Removing either would break the loop and not allow for the algorithm to be run at all. In terms of speed, we do show how each change we made improves speed (a type of ablation) showing that all different improvements we made are necessary to achieve the full speed-up. These numbers are detailed on the last paragraph before Section 3.2 on Page 5.

Weaknesses: While the paper presents a novel and efficient approach with realSEUDO, several limitations are apparent. The reliance on specific hardware configurations, such as Linux CPU performance modes, could limit accessibility and generalizability. Additionally, the implementation primarily in C++ with MATLAB wrappers, while effective, could benefit from broader integration into Python pipelines to enhance usability. The paper also assumes access to external motion correction algorithms and does not address the full pipeline holistically, which may impact its applicability in scenarios without these additional tools.

Our response: We appreciate the reviewer’s positive comments and feedback. We agree that the additional limitations of the current infrastructure should be more explicit. Some drawbacks, such as additional wrappers in Python (as well as pure C++ integration into imaging pipelines) are a primary focus of future development in this project. We are also testing the exact limitations of the hardware impact on performance. We do believe that in the long-run CPU integration will allow for more generalizability than, e.g., GPU or FPGA approaches, and will add more text in the discussion to discuss the pros and cons of each. Finally, as online motion correction was well addressed by others (e.g., in the OnACID work) we instead focused on the less-considered cell finding. It is of course important, as the reviewer indicates, that full pipelines work well together and so integration of prior methods are a primary consideration as incorporate these pipelines into full closed-loop systems.

Limitations: The paper presents notable advancements with realSEUDO but has some limitations. Firstly, the performance of the system is highly dependent on specific hardware configurations and the proper setting of Linux CPU manager modes, which may not be universally applicable. Additionally, the reliance on MATLAB for prototyping, combined with limited Python integration, may restrict its accessibility and ease of use for a broader audience. Moreover, the paper focuses on cell detection without fully integrating or addressing the motion correction and complete processing pipeline, which could limit its applicability in real-world scenarios where such components are not readily available.

Our response: We thank the reviewer for noting our advancements and seeing the potential in our work. As the reviewer mentioned, the primary goal of this work is to address a critical step in the pipeline for real-time cell detection. We agree that with every advance there will be limitations and that these limitations should be clearly delineated. Moreover we agree that the stage of the pipeline we address does not live in a vacuum and should be mentioned in the context of the other stages of the pipeline to best inform readers of when/how they can use the method. We will thus discuss in more detail the limitations raised by the reviewer and the steps we are actively taking to expand the ease of others to adopt and implement realSEUDO. These include additional integration tools into Python on the user’s end, as well as building in prior motion correction algorithms that will be a part of the complete pipeline development.