ActSort: An active-learning accelerated cell sorting algorithm for large-scale calcium imaging datasets

Thank you for your time and effort in reviewing our manuscript. Your feedback was extremely helpful for increasing the clarity of our manuscript, and most importantly in distinguishing ourselves from published material.

Summary Respectfully, we do not agree with your summary of our work in several key domains: First, we are not designing software for two photon imaging analysis, our dataset consists mainly of 1p movies (4 out of 5 movies). Second, we are not aware of any published work with deep classifiers that is used to sort cells in 1p movies. Next, there is no work prior to us designing an active learning routine for cell sorting, so we are not sure which many other works are referred to here. Finally, we believe we have several other contributions left out in your summary, please kindly see our general response above.

Anonymity: ActSort is a standalone quality control pipeline, not part of any previous publication. It is compatible with not only EXTRACT, but also all other cell extraction algorithms. Moreover, ActSort is not affiliated with EXTRACT and does not endorse it. We simply used EXTRACT, since other state of the art alternatives to process 1p movies (CAIMAN, or earlier studies used ICA), were less efficient and time consuming. To mitigate this confusion, we added a new experiment on using ActSort on ICA extracted cells as shown in Fig N7 in the PDF. We are happy to address our contributions accordingly if the reviewer can let us know which specific previously published paper you are referring to.

We were carefully following the guidelines to maintain anonymity. For instance, we had not shared the user manual or links to our lecture videos for this very reason. In the references, we did our best to cover a broad community, including multiple calcium imaging techniques and cell extraction algorithms from multiple groups in the field, using published datasets with permissions from authors. Fortunately, your assumption is simply not correct.

Comments on figures and use of appendices Respectfully, we disagree with the reviewer that the extensive additional benchmarking and the detailed explanation of the algorithm in the appendices and the figures are a weakness. This is, in our opinion, a strength of our work. Moreover, as evident from other reviewer reports, our work is self-contained in the main text, and only uses appendices to refer to methodological details and additional benchmarking studies to strengthen our claims. If you can let us know which figures you find confusing, we would love to work on them.

Regarding discussion of software and framework The software, the benchmark, and the active learning framework together form our novel conceptual contributions in addition to the specific active learning query algorithms. All three of them are equally spaced in the main text. We are the first to introduce this framework to experimental neuroscience, so it is reasonable that we spend time explaining them carefully, as a big fraction of our readership will be divided between active learning researchers and experimental neuroscientists.

Deep classifiers Amazing point! We added a new experiment to address this (Fig. N8). Please refer to the “New experiments” section in the general response. In short, we tried ResNet-50 for feature extraction directly from movie frames and cell extraction outputs. Not only were the extracted features suboptimal to engineered features, the time to run cell data over the network was beyond our design.

Line 113f: spikes Good question! We use the peakseek built-in function in MATLAB to obtain the spikes count from the extracted cell traces by the cell extraction algorithm (ICA or EXTRACT).

Clarification on Fig. 2. Great point! We apologize for the confusion here. The d-prime is between the distribution of $p(X|c = \text{cell})$and $p(X|c = \text{not cell})$ where $X$ represents the target feature. The positive examples represent the candidate being an actual cell and vice versa. To clarify, we added an illustration to explain the definition of absolute discriminability index, a metric for quantifying the distance between these two probability distributions (Fig. N1). Additionally, as requested, we also added a statistical quantification on the effect size of the traditional features and the new features (Fig. N2).

Fig 2: Traditional features: Great question, one we should have made clear! The traditional features represent the features that CAIMAN and CLEAN used for cell classification on 1p imaging movies. Please see our general response to reviewers.

Detailed math for appendix and typos Thank you for pointing these out. We have updated the math for all features accordingly. We changed L140 to “In this work, we designed an active learning query algorithm, Discriminative Confidence based Active Learning query algorithm, in short DCAL, and compared its performance with traditional random selection and the two other query strategies. ” The Fig 3A legend is placed under Figure B-D. We moved the legend above to make it more accessible for the reader as you suggested.

Final clarifications As we are concluding our response, we wish to address the weaknesses 1 and 5 from your report directly. We want to make it super clear that we have NOT copied any part of ActSort from any published work, nor did any part of ActSort was published before anywhere. We apologize if there was any confusion, and hope the rebuttal, the general response, all additional experiments we shown in the PDF, and modifications, make it clear that ActSort is NOT part of a large framework previously described in other papers.

We are looking forward to your additional comments if you have any on how we can improve clarity. Now that we have addressed all your raised weaknesses and corrected the misconceptions about our contributions - for which we thank you -, we hope you will consider increasing your score.

Thank you for your response and careful reading of our rebuttal. We answer your questions below.

Point 1: First, the field has had no real success in generalization across modalities in this direction before. Please see discussions above regarding CAIMAN’s deep classifiers being suboptimal for 2p movies and not recommended for 1p movies, and Cascade requiring a decade of public benchmark accumilation to achieve this on spike extraction. Also, please note that the imaging and experimental conditions between two 1p movies may be as diverse as those between a 1p and 2p movie.

Second, after consulting with the experimental neuroscientists, we realized that this path was also inconsistent with a major concern of theirs: Reproducibility. A deep network is a non-convex approach, and often requires retraining with new data (See Cascade here). Unless experimental groups also publish their retrained networks, the annotation would not be reproducible. This is not true for the model in our work. Here, as long as the group of annotated neurons are provided, a third party can always validate the reproducibility of the annotation.

Finally, how the input to such a deep classifier should look like is an open question, as we discussed in details above. CAIMAN, for example, only used extracted cell profiles, but we believe (and our Fig. 2 shows) that spatiotemporal information should somehow be incorporated. We already have several other contributions, so we do not wish to open this direction as well. So instead, we did the next best thing to address it, which is considering whether a simple pre-trained network can become a solution.

Overall, building a specialized deep network is out of scope. We believe we properly discussed this now in the paper and above.

Point 2: Agreed. We will do so, and we also think this is more appropriate. This was an oversight.

Point 3: ‘Bad’ events are the ones that had low calcium amplitudes. Mathematically, we computed the 90th trace quantile and divided it by half. Any "event" that has lower amplitude than this was considered a "bad" event. We now realize that ‘bad’ is not a correct term for this, instead we should call them "weak" events. For the rest of the features, we can share more details for whichever feature you desire to know about.

To address your comment about our tone

In our response, we made it clear when we disagreed with you and also when we agreed with you. Part of it was also to give you feedback about which comments we felt were quite good so that we can all come out as better writers and reviewers. We apologize that our writing was not clear. We were serious in our remarks, there was no humor intended.

We do believe that you have provided solid points for improvement, and that particular point was also necessary to at least discuss in this paper, given that our work is being considered for NeurIPS. Sure, you recommended rejection due to some assumptions about our work, but with the current reviewer loads (which we also have) and the scaling of the conference, it makes sense that certain nuances may slip through. This is why the discussion period exists. We do not believe there is any reason to be facetious about this process, and we had not intended to do so either. We apologize once again and look forward to hearing from you.

Thank you for your detailed report and excellent summary! We truly appreciate your vote of confidence in our work and the excellent suggestions for improving our technical contributions. Thanks to you, our paper now includes several diverse, convincing control experiments. To save space, we refer to weaknesses with “W” and questions with “Q”, please excuse us.

W1 and W2 Please see our general response regarding the fit to venue and comparison to prior work. To address further concern, we performed two additional control studies. First, an ablation study by removing the adaptive estimation process showed that DCAL was significantly less effective without it (no space in PDF, happy to elaborate). The second study, shown in Fig. N5, demonstrates the evolution of weights over time. Also see Fig. N3 for additional evidence supporting the necessity of DCAL.

W3 Great idea! We added a new experiment to address this (Fig. N8). Please refer to the “New experiments” section in the general response. In short, we tried ResNet-50 for feature extraction from movie frames and cell extraction outputs, resulted in suboptimal features.

W4, Q7, and Q8 Excellent point! We evaluated manual annotation accuracy in Fig. S4, Tables S1, S2, and S3. We used four annotators per dataset, with majority vote as ground truth, to mitigate inconsistencies and annotation noise (See Fig. S3 for the evaluation process). Human annotators could revisit samples multiple times, while ActSort trains on the final annotations.

To address annotator reliability, we performed intraclass correlation analysis. Individual annotators were inconsistent (ICC = 0.56±0.06), but classifiers trained on these annotations mitigate human bias and were more consistent (ICC = 0.64±0.07). Majority votes across annotations were quite consistent (ICC = 0.79±0.05). This shows individual annotators are unreliable, but majority votes (In experiments, only one annotator rates!) approximate ground truths well.

W5 We compare the classifier’s prediction with single annotators across the entire dataset, NOT just a small subset. Also, as evident from the limit of 100% annotation, the classifiers do outperform humans on trained samples.

W6 We used logistic regression for the cell classifier to ensure real-time prediction speed, which converges to global convex optimum. The regularization effect is analyzed in Fig. S7, and a new hyperparameters analysis on the classifier threshold is in Fig. N4. AL convergence is depicted in Figs. 4, 5, S6, S8, S9, S10, and Tables S5, S6, S7. We now include a control for using ImageNet for pre-training general features (Fig. N8), whic the engineered features demonstrated significantly higher AUC than using deep learning.

W7 Good point! The process is dependent on cell numbers, not movie size. We added: “...(data compression) resulting in approximately 270+-90 MB/1,000 cells (mean +- std over 5 mice) data sizes.” A TB-scale movie with 10,000 cells (for example, the movie from Ebrahimi et al., 2022) can be compressed to less than 3GB. The five imaging datasets had different imaging conditions (1p-2p), apparatus, and frame rates (20, 30, and 50), demonstrating feature robustness, which is illustrated by the fact that ActSort works on both 2p and 1p movies without modification.

W8 and W9 These are outside our scope. We use linear classifiers with instant training, making batch size of one (a desirable property to mimic real-work cell sorting) attainable. Other data types have not achieved the recently attained 1 million neurons mark in calcium imaging, thus not of interest here.

W10 This key conclusion motivates ActSort. Human annotators are inconsistent, especially with large datasets. Thus, 4 annotations per dataset is a key contribution of our benchmark and evaluation. For instance, historical benchmarks like Neurofinder (2p, 100s of neurons, 1 annotator) later showed experts missed many cells in Suite2p.

Q1 We added an illustration explaining the absolute discriminability index (Fig. N1). We also added statistical quantification of traditional and new feature effect sizes (Fig. N2). We also updated the math for all features accordingly.

Q2 The label distribution of the 160,000 cell candidates is shown in Figs. S6, S8, and S10. We tested our algorithm on imbalanced datasets with more true positives (Figs. S6 and S10) and artificially inflated datasets with many false positives (Fig. S8). Our results are consistent, demonstrating our approach’s robustness to imbalance.

Q3 The Random query algorithm has no hyperparameters. CAL, DAL, and DCAL use the same cell (and/or label) classifiers to predict cell probability. We included a sweep over regularization parameters (Fig. S7) and a new experiment on classifier threshold sweeping (Fig. N4).

Q4 This indicates 1) humans are inconsistent over time, 2) humans get tired after sorting 10k cells, and 3) the active learning algorithm selects representative samples. Points 1 and 2 are supported by new intraclass correlation results, and point 3 is illustrated in Fig. 3. In short, linear classifiers primarily care about boundary samples; once those are found, classification is mostly optimal.

Q5 Other algorithms would be harder to judge (e.g., random forest, neural network) and would not allow instant training, which is needed for real-time human annotation. Convexity of LR ensures reproducibility, unlike random forest classifiers or neural networks.

Q6 We apologize for the confusion. We should not have called this pretraining and will update the text as “prelabeling”. Details of the fine-tuning process are in Algorithm S1 and Appendix C.2.

Thank you for your detailed review. We implemented your feedback whenever applicable, which improved our manuscript tremendously! If you have additional concerns, we look forward to discussing them further. If your concerns are addressed, would you consider increasing your scores?

We sincerely thank you for your time and effort in reviewing our manuscript and providing insightful feedback. We have addressed your concerns regarding where ActSort stands on within the entire calcium imaging processing pipeline and explained our fit into the NeurIPS venue in the General Response. Note that we omit citations from text excerpts due to character limits.

Prior work: Great point! We should have done a better job in explaining that 1p movies are non-standard and lack specificity. As a result, there is no published baseline for the problem we are aiming to solve in the field. We are introducing the first benchmark on these types of movies. The current method for the curation process of removing false positives rely entirely on human laborers or other pipelines use simple thresholding based on what we call traditional features (CAIMAN, ICA, EXTRACT), or classifiers defined on these features (Suite2p). Please also refer to “The concerns about comparison to prior work” section in the general response for more detailed explanations.

Imperfections in the data such as motion correction, cell extraction, stitching, cell identity You are absolutely right about all these concerns. To recap, there are two distinct problems when processing calcium imaging movies.

The first problem is the cell extraction, in which the motion should be corrected, cells should be properly identified, cells suspected of duplication should be stitched or removed, and/or cell identity should be faithfully tracked across sections. Field has been working on these problems over two decades.

ActSort comes as a quality control (second problem) pipeline on several existing pipelines designed to address these aforementioned problems. Traditionally, quality control on these outputs was done by experimentalists manually, by going over each cell with custom softwares. However, with thousands and millions of neurons being recorded in a single session nowadays, automated quality control became necessary. Yet, we should have addressed this in the discussion for the broader ML community, which is what we do now as follows:

“The quality control process focuses on identifying the true positive samples correctly while also correctly rejecting the true negative samples that are misidentified by the cell extraction algorithm. [...] However, since ActSort is a quality control algorithm, it requires that movies are motion-corrected and cells are extracted with the experimenter's favorite cell extraction algorithm, therefore, any mistake made in the cell extraction process will propagate to the cell sorting process.” and “One limitation of ActSort is that it relies on correct cell extraction by the cell extraction algorithm used by the experimenter. Future additions to ActSort could mitigate some of the common errors, such as the ability to merge segmented or duplicated cells.”

If the preprocessing step misses some of the cells there’s no recovery You make an excellent point! Although EXTRACT and ActSort all fall into the calcium processing pipeline, they have different purposes. ActSort is a standalone pipeline for QUALITY control instead of cell finding. We modified the discussion section based on your suggestions as: “ActSort is a novel standalone pipeline for quality control that can be added as a further step after using any cell extraction algorithm such as EXTRACT, Suite2p, ICA, CAIMAN, and many others (Pachitariu et al., 2016, Ren et al., 2021, Giovannucci et al., 2019, ).”

Using foundation models Wonderful question! Let us illustrate the shortcomings of foundation models on two examples on different, but related, problems:

Cell extraction: A foundational model, Cellpose, was developed for segmentation from images but is not used for cell extraction from videos. The main reason is that brain recordings, particularly the 1p imaging videos, are very diverse in their backgrounds, imaging and experimental conditions, and cell shapes/types/sizes. Therefore, even for cell extraction, the foundational models have not been successful to date.

Event extraction: Cascade (Rupprecht et al., 2021) utilized deep models to extract events from calcium activity traces in two-photon (but not one-photon) movies. The model was trained from scratch on a massive public dataset, and did not generalize to 1p movies due to aforementioned reasons.

Similarly, for cell sorting, a new model may need to be trained as more public benchmarks (our being the first with such scale) become available.

Control with a deep classifier What a great idea! To address this, we added an experiment using ResNet-50 for extracting features and classification using 1p calcium imaging snapshots. Specifically, we provided the extracted cell profiles and the average cropped movie snapshot, averaged over frames with cells’ activities, to ResNet-50 and collected 2,000 features from the final layer. We repeated our experiment in Fig. 2 for this dataset (Fig N8). The engineered features demonstrated significantly higher AUC than using deep learning.

Technical novelty We agree that the simplicity of the introduced method may seem not technical enough at first, but we kindly disagree. Please refer to the “Regarding fit to the venue and our contributions” section in the general response for further discussion. In short, we believe the impact of our technical contributions, no matter how simple, (now that we have also shown the suboptimality of a classifier trained on ResNet-50 features) is significant, as neither CAL or DAL can reach the success of DCAL, nor DCAL without adaptive updates reaches the same levels (new experiment, data not shown due to page issues, but happy to elaborate further).

Thank you for your helpful suggestions. Specifically, the deep net classifier was a very significant addition to our work, thanks to your feedback! If you believe we addressed your concerns, would you consider increasing your scores?

Thank you very much for your kind and constructive words! Your feedback was very helpful and helped us make our work better!

We appreciate your time and effort in reviewing our manuscript and providing invaluable feedback. You caught everything in the paper, your suggestions were extremely helpful, and your review led to new experiments that increased the readability and impact of our paper! To us, this was an extremely well written, helpful, and insightful review. Thank you!

Accessibility for experimentalists Thank you for highlighting this critical point while evaluating our work! We have prepared a user manual and lecture videos (with tutorials) for ActSort, but due to the anonymity requirements, we cannot share them yet. We are committed to providing a user-friendly introduction to ActSort, which will hopefully ease the experimental neuroscientists into the software. Currently, ActSort has a small user base of 20+ researchers globally (mainly collaborators), and we are actively collecting their feedback to further improve the user experience.

Discussion section The reviewer is absolutely correct. Thanks to the additional page provided after the revisions, we now added a new discussion section. We are happy to share the full text if requested, but for brevity, we will list the discussion bullet points below:

• ActSort is an active learning accelerated cell sorting pipeline for large-scale 1p and 2p calcium imaging datasets, which is compatible with any cell extraction algorithm (CAIMAN, Suite2p, EXTRACT, ICA, etc.).
• Since ActSort is a quality control algorithm, any mistake made in the cell extraction process (for example, missing cells and/or motion artifacts) would automatically propagate to the cell sorting process. Though we have plans to incorporate merging of duplicate cells in the future, the current version does not support this. Thus, we require that movies are motion corrected and cells are properly extracted with your favorite cell extraction algorithm.
• ActSort makes no assumption on the type of behavioral experiment, the Ca2+ indicator, the imaging conditions; and is robust to variations thanks to its standardized features.
• Though currently validated for calcium imaging experiments, ActSort can be minimally modified (perhaps with the addition and subtraction of new features) to be applied to newly emerging technologies such as voltage imaging and/or multi-class datasets (for example, including dendrites).
• Our optimally compressed data format has implications for sharing Ca2+ imaging datasets publicly. To date, only post processed activity traces were shared, but with our format, now movie snapshots can be shared too, allowing end users to check the quality of the data.

If you have additional comments, please let us know!

Clarification for query algorithms Great suggestion! We added an explanation to equation (3) as “The uncertainty-score $c_i^{(t)}$ represents the uncertainty regarding the sample $i$’s position relative to the decision boundary. The higher the score, the closer it is to the decision boundary. The discriminative-score $d_i^{(t)}$ represents the uncertainty regarding whether the sample $i$ faithfully represents the full dataset. Higher scores indicate the sample is unique and underrepresented by the labeled dataset.”

We added an explanation to equation (5) as “ $w$ approaches to the predefined weight $w_0$ if the labels classifier $p_\phi^{(t)}$ correctly differentiates between labeled and unlabeled data, indicating unique samples in the unlabeled dataset. $w \to 0$ if there are no underrepresented samples in the unlabeled dataset, resulting in the query algorithm selecting only the boundary cells as the CAL algorithm does.

To address your concerns about adaptive updates, we added a new figure (Fig. N5). In the beginning, when only a few samples are sorted, there is an initial drop in the weight values. This drop occurs because the label classifier fails to differentiate between labeled and unlabeled samples. This indicates that initially, the CAL component dominates the sample selection process. As the process continues, the DAL component starts to take over, selecting unique and underrepresented samples from the unlabeled dataset, and finally converging back to CAL again.

Questions about Fig. 2: Thank you for pointing out the confusion regarding the absolute discriminability index. We added an illustration to explain its definition in the PDF (see Fig. N1). The absolute discriminability index is a metric used to quantify the effect size between two distributions. If the two distributions can be easily told apart, we will obtain a higher d' value. The two distributions here are feature values conditioned on whether a sample is a cell or not. Additionally, we included a statistical analysis comparing the effect sizes of traditional features and new features (Fig. N2).

The illustration on Fig. 2 B makes the difference seem minimal. The AUC changes from 0.94 to 0.97. We will update the figure to have a zoom in version. We added a sentence explaining that the increase in true negative rate did not sacrifice the accuracy of true positive rate - “Re-analyzing the dataset from Fig. 2 B, we found that the new features increased the rejection accuracy from 67% to 75% (Fig. 2 B) without decreasing the accuracy of accepting true-positives (97% to 97%), leading to more effective separation between cell and not cell samples in our benchmarks (Fig. 2 A). ”

Question about Fig 3D: Wonderful suggestion! To address this point, we changed the feature distance measurement from euclidean distance to cosine distance, which has a much better dynamical range (Fig. N3).

Typos: Thanks for pointing out our typos! We will revise the writing! The Fig 3A legend is placed under Figure B-D. We moved the legend above to make it more accessible for the reader as you suggested.

Thanks to your feedback, we were able to improve the clarity of our work. We hope that you are satisfied with our responses and consider supporting our submission with a very strong acceptance!

Dear Reviewer,

Thank you for reading our rebuttal and your continued engagement. Please find below our responses (citations omitted and certain sentences shortened):

Discussion Section:

"In this work, we introduced ActSort, a user-friendly pipeline consisting of 3 modules: a preprocessing module, a cell selection module, and an active learning module.

The preprocessing module efficiently reduces the data size associated with $10,000$ neurons down to few GBs, which includes not only cells' Ca$^{2+}$ activity traces, but also spatial profiles, and movie snapshots during \cm events -allowing end users to check the quality of the data-, together with the engineered features. This compact representation allows the ActSort pipeline to be run locally in laptops, despite original movie sizes of up to TBs. The joint compressing of the movie snapshots and cell extraction information offers another key contribution to the Neuroscience community with implications for sharing imaging datasets publicly.

The cell selection module features a custom design with an easy-to-use interface that displays temporal, spatial, and spatiotemporal footprints, and incorporates a closed-loop online cell classifier training system. During the annotation process, the software provides real-time feedback by displaying predicted cell probabilities and the progress made by the human annotator as well as the fraction of unlabelled cells that ActSort is confident about.

The active learning module works in the background, strategically selecting candidates for human annotations, and trains cell classifiers with annotated candidates. To make our pipeline easily accessible and used by the Neuroscience community, with this work, we are providing a user manual and video tutorials.

Previous work on quality control for two-photon movies can be used, though less accurately, on one-photon movies. One-photon datasets face challenges like low resolution, high background noise, and reduced specificity. Our solution, ActSort, addresses both brain-wide one-photon and two-photon datasets through standardized features that are robust across various behavioral experiments, \cm indicators, imaging conditions, and techniques.

ActSort can be added as a quality control step after any cell extraction algorithm, such as EXTRACT, Suite2p, ICA, CAIMAN, and others. The quality control process focuses on correctly identifying true positives and rejecting true negatives misidentified by the extraction algorithm. ActSort surpasses human performance in true positive and true negative rates by annotating less than 3% of samples. However, since ActSort is a quality control algorithm, it relies on motion-corrected movies and accurate cell extraction, as any mistakes in extraction propagate to sorting.

To support the development of active learning algorithms in systems neuroscience, we introduce the first publicly available benchmark for cell extraction quality control on both one-photon and two-photon large-scale calcium imaging, comprising five datasets: three one-photon and two two-photon \cm imaging datasets, with approximately 40,000 cells and 160,000 annotations (each dataset annotated independently by four annotators). This dataset is unparalleled in the public domain.

One limitation of ActSort is that it relies on correct cell extraction by the cell extraction algorithm used by the experimenter. Future additions to ActSort could mitigate some of the common errors, such as the ability to merge segmented or duplicated cells. Future directions also include the exploration of oversampling techniques without hurting the true positive rate, more sophisticated cell classifier architectures, and batch sampling by the query algorithm.

Furthermore, while ActSort was validated with 1p and 2p datasets, it is a standing question whether it could be applied to newly emerging technologies such as voltage imaging and/or multi-class datasets (for example, including dendrites) with the current features, or with slight modifications such as addition and subtraction of new features, which can be explored in future work.

Another important aspect requiring further exploration is the expertise level of the annotators. Specifically, the six annotators had different levels of expertise in working with \cm imaging datasets. Hence, a potential moderation relationship can exist depending on the expertise of the annotator, which is left as a future work."

Fig. 3A visualization: Yes! We (approximately) visualize the region comprising of the boundary cells, as well as the boundary itself.

Fig N3: We find cosine similarity to be more interpretable, and it has a better dynamical range. We would love to hear your thoughts though?

Fig. N5: Yes, agreed. We will update the figure to be log-scale in the fraction of annotations.

Once again, thank you for your time, consideration, and support. We look forward to hearing your thoughts on our updated discussion!

Thank you very much for your support of our work and additional feedback. Indeed, we will go over our rebuttal one more time before we finalize this work and make ALL the changes we committed. We think all suggestions were helpful and necessary. We will share the changes we performed to address your remaining concerns, but we do not expect a response from the reviewer in case they feel content with the rebuttal.

We took out the first three paragraphs from the discussion, not repeating the strengths of ActSort. For the questions you raised, please find the relevant full (unshortened) paragraphs below:

Existing work

"ActSort comes as a standalone quality control software, which can be used to probe the outputs of cell extraction pipelines such as EXTRACT \cite{inan2021fast}, Suite2p \cite{pachitariu2016suite2p}, ICA \cite{mukamel2009automated}, CAIMAN \cite{giovannucci2019caiman}, and others \cite{zhou2018efficient,cnmf,chen2023hardware,roi}. These cell extraction algorithms take the raw \cm movie as their inputs, correct the brain motion, perform spatial and temporal transformations to standardize the \cm imaging movies, identify putative cells' spatial profiles and temporal activities, and often perform primitive quality controls to output the final set of neural activities.

Historically, additional quality controls on the cell extraction outputs would be performed with manual annotation, which was feasible for the small neural recordings with hundreds of neurons \cite{marzahl2020fast, salvi2019automated,amidei2020identifying, wang2021annotation,schaekermann2019understanding,corder2019amygdalar}. Yet, with the advent of large scale \cm imaging techniques, now recording up to one million cells \cite{manley2024simultaneous}, manual review became unrealistic. Instead, the field of experimental neuroscience direly needs automated quality control mechanisms that would correctly identify the true cell candidates while rejecting true negatives misidentified by the extraction algorithms.

As discussed above, ActSort is the first scalable and generalizable solution in this direction. Yet, previous works (as parts of existing cell extraction pipelines \cite{pachitariu2016suite2p,giovannucci2019caiman,inan2021fast}) had tackled this problem in specific instances: Suite2p designed cell classifiers based on some basic features to increase the precision of the algorithm \cite{pachitariu2016suite2p}, CAIMAN pre-trained a deep classifier for two-photon movies \cite{giovannucci2019caiman} (though not applicable to 1p \cm imaging movies \cite{caiman_demo_pipeline_cnmfE}), whereas EXTRACT performed thresholding on a set of quality metrics \cite{inan2021fast}. Notably, these existing automated methods with pre-trained cell classifiers often found success only for high quality 2p \cm imaging movies \cite{pachitariu2016suite2p,giovannucci2019caiman}, and even then underperformed human annotators \cite{giovannucci2019caiman}. One-photon \cm imaging datasets, on the other hand, are quite diverse in their imaging (miniscope vs mesoscope) and experimental (head-fixed vs freely behaving) conditions and face additional challenges due to low resolution and high background noise. With ActSort, we sought a generalizable solution that does not target a specific modality or require re-training, but uses interpretable features that are robust across various behavioral experiments, \cm indicators, imaging conditions, and techniques.

To provide a baseline for existing methods in our benchmarks, we designed a feature set called "traditional features" (Fig. \ref{fig:fig2}), including features used by classifiers in these prior works (See Appendix \ref{sec:feature_engineering}). Moreover, these methods did not use active learning, instead annotated only randoms subsets. Thus, in our experiments, these prior methods (or a plausible upper bound to be more exact) are represented as the random sampling query algorithm, which uses the full feature set to allow fair comparisons to active learning approaches."

(Continued below)

Future work:

"Though we performed extensive analysis to highlight the efficiency and effectiveness of ActSort, our work is merely a first step for what may hopefully become a fruitful collaborative subfield comprising of experimental neuroscientists and active learning researchers. There are several future directions that future work could improve upon, which we will briefly summarize below.

Firstly, in this work, we used linear classifiers for rapid online training during cell sorting and reproducibility of annotation results. This was mainly rooted in the fact that pre-training deep networks required substantial data and standardization across various \cm imaging movies. We believe that with additional public datasets that may follow our lead, this direction can become reality (as was the case for a different, yet relevant, problem of spike extraction \cite{rupprecht2021database}). Our results in this work have set a strong baseline for such future deep-learning approaches.

One limitation of ActSort is that it comes as a quality control pipeline to existing cell extraction approaches. Therefore, any mistakes in the cell extraction step would automatically propagate to cell sorting. Yet, some of these mistakes can be mitigated or highlighted post hoc. For instance, future ActSort versions could involve options for merging segmented or duplicated cells, or identifying motion in activity traces and thereby notifying the user to improve their preprocessing steps.

Another important aspect requiring further exploration is the expertise level of the annotators. To date, each \cm imaging movie is often annotated by a single annotator, who unfortunately can tire after long hours and become inconsistent as we have discussed above. This was the reason behind our choice to have the movies in our benchmark be annotated by multiple researchers. Yet, the human annotators had different levels of expertise in working with \cm imaging datasets. Hence, a potential moderation relationship can exist depending on the expertise of the annotator. To fully explore this relationship requires further research with more annotators per dataset, by having the same annotators sort the same cells in different times, and/or testing sorting effectiveness before and after a standardized sorting training.

Finally, in this work, we validated ActSort for one-photon and two-photon \cm imaging datasets and for sorting binary classes. Yet, our framework is generally applicable to other modalities, barring additional feature engineering performed by domain experts in those fields, or with a pre-trained deep network. For instance, by replacing the logistic regression with a multinomial version and approximating DAL scores with entropy instead of decoder scores, our work readily applies to multi-class datasets that may jointly include, e.g., dendritic and somatic activities. With correct features, the framework we introduced here should also be helpful for the newly emerging voltage imaging technologies, especially as the number of cells will inevitably increase in such movies with the technological advances."

With these additions, our work is now exactly 10 pages. We thank you very much for your time and support; and wish you a great week!