scPilot: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discovery

We sincerely thank Reviewer VK46 for their positive evaluation and for recognizing the key contributions of our work. We are especially grateful for the acknowledgment of:

1. Our Novel Domain and Foundational Benchmark: For highlighting that we are addressing a "problem domain... not frequently investigated" and that our scBench benchmark is a "noteworthy contribution" in the "important... effort to create scientific reasoning models".
2. Our Core Methodological Innovation: For identifying the specific novelty of our framework in its "explicit requirement to force the LLM to record its claims in a transparent trace" and its novel application to this domain.

(a) Novelty and Clarification on Reasoning Trace

We thank you for pointing out the need for more clarity on the reasoning trace. Figure 3 is a high-level schematic, and you're correct that the full trace contains additional, detailed natural language claims and justifications for each step. The value of this transparent trace is best shown through concrete examples of the reasoning process in action.

We kindly refer the reviewer to previous response (d) to Reviewer RBqf, response (c) (d) to Reviewer mMx1.

Cell-Type Annotation: Multi-Gene Logic**

scPilot's reasoning trace for this task records a propose -> filter -> analyze workflow that allows for sophisticated biological interpretation.

• How it Works: The trace first records the LLM's initial claim (proposing potential cell types). It then logs the tool call and its output (e.g., noting that the key marker for Plasma cells, SDC1, is absent in the data). Finally, the trace includes the LLM's explicit justification for the final annotation, such as using the combination of NKG7, GNLY, and CD3D expression to distinguish NK cells from CD8 T-cells—a nuance often missed by simpler methods.
• Value: This trace provides a fully auditable record of the claims, evidence, and justifications, allowing a scientist to manually check and understand each decision.

Trajectory Inference: Self-Auditing with Tool Diagnostics**

The reasoning trace for trajectory inference showcases an even more complex workflow involving self-correction.

• How it Works: The trace logs the initial tree construction, the call to py-Monocle for a diagnostic report, and then the LLM's detailed justification for why it is making specific repairs to the tree (e.g., correcting the root, fixing the hepatic developmental ladder).
• Value: This provides an interpretable record of how scPilot uses evidence from traditional tools to improve its own results, leading to a more accurate and biologically faithful output (e.g., reducing graph-edit distance by 6).

GRN Prediction: Contextual Reasoning**

For GRN prediction, the reasoning trace captures how scPilot integrates multiple data types to make a context-aware decision.

• How it Works: The trace shows the framework assessing a potential TF-gene relationship by combining functional data from the Gene Ontology, gene expression context from the input data, and information from known regulatory pathway databases.
• Value: This allows for true tissue-specific reasoning, and the trace makes it clear how the final prediction was weighted by this contextual evidence, moving beyond generic associations.

We will add these detailed examples of the complete reasoning traces to the supplementary material to fully illustrate this key feature of our work.

(b) Flexibility and Extendibility of Bio-tool Library

This is an excellent question about the framework's flexibility. To clarify, the tools within the bio-tool library are curated beforehand, and the agent does not dynamically add new tools during a run. As we explained in our response to Reviewer RBqf (c), this was a deliberate design choice to ensure our scBench benchmark is rigorous and reproducible, focusing on the LLM's reasoning capability rather than on tool discovery.

Regarding its use for other bioinformatics tasks, scPilot is designed to be modular and easily extendable, though this does require manual modification by a developer. As noted by Reviewer 82g2, the modular design of the "Problem-to-Text Converter," "Bio-Tool Library," and "LLM Planner" keeps each part’s responsibility clear and facilitates extension.

To adapt scPilot for a new task, a developer would simply need to:

1. Wrap the new bio-tool in a simple Python function that formats its output into the standardized JSON that scPilot expects.
2. Register the tool by adding a description (e.g., a docstring) to the library so the LLM planner understands its purpose and how to use it.

In summary, while our scBench implementation uses a fixed toolset for evaluation integrity, the underlying scPilot framework is intentionally designed to be an extensible platform, allowing developers to apply omics-native reasoning to a wide range of future bioinformatics challenges.

(c) Detailed Benchmark Dataset Statistics - already in supplementary

Thank you for this question. Detailed statistics for all datasets used in scBench—including the number of cells, number of cell types, and graph statistics for the trajectory inference tasks—are provided in Tables 2, 3, and 4 of our supplementary material.

(d) Data Filtering and Annotation Criteria

Our benchmark datasets were curated from high-quality, peer-reviewed studies without additional computational filtering to ensure they reflect established community standards.

Data Filtering

We did not perform additional computational filtering on the datasets. Our selection process focused on using data directly from high-quality, peer-reviewed publications where the data were already processed and filtered according to the standards of the original study. Instead, all datasets for scBench were curated and processed through a uniform, four-step pipeline to ensure quality and consistency.

1. Data Standardization and Compression: Raw data from high-quality, published studies is first standardized and stored. It is then compressed into a uniform text format that the LLM can process.

2. Ground-Truth Harmonization: The ground-truth criteria are stricter than simply "manually annotated." Annotations are mapped to standard databases like the Cell Ontology, and published lineage graphs are standardized to ensure a consistent, high-quality ground truth.

3. Prompt Sketch Creation: Task-specific converters generate compact JSON "sketches" that summarize key data (e.g., cluster statistics, marker genes) to fit within a single LLM prompt.

4. Packaging: Each task is bundled with its prompt sketch, data configuration, a programmatic grader, and instructions to allow any method to be run reproducibly.

This systematic pipeline ensures that scBench is built upon well-vetted data and expert-validated, harmonized ground truths.

Ground-Truth Annotation Criteria

Our criteria for ground-truth labels were stricter than simply being "manually annotated" and were tailored to each task:

• Cell-Type Annotation: The labels were required to be expert-curated and published in a peer-reviewed study, ensuring they have been vetted by the scientific community.
• Trajectory Inference: Datasets were required to include a human-curated lineage tree as ground truth within the original publication.
• GRN Prediction: The ground truth is based on experimentally-validated TF-gene interactions from the trusted TRRUST v2 database.

This curation strategy ensures that scBench is a reliable and realistic testbed for evaluating scientific reasoning.

Hello, thank you for your review. Please see the authors response, particularly around reasoning traces and tool use, to see whether you would like to change your feedback or scores. Thank you.

We sincerely thank Reviewer mMx1 for recognizing several core contributions of our work. We are particularly grateful for the acknowledgment of:

1. A Major Foundational Contribution: Highlighting the "development of the scBench omics reasoning benchmark" as a major strength.
2. A Methodological Leap Forward: Recognizing that our framework "extends past established ‘chat with your data’ systems" to enable "multi-step reasoning directly on the omics data."
3. Thorough Validation and High Performance: Appreciating our comprehensive "analysis of error modes" and, importantly, that scPilot's "performance on diverse tasks is high."

(a) Value of Reasoning: Why scPilot (o1) performs worse sometimes

We thank the reviewer for this critical question. It highlights the value of scPilot's reasoning beyond raw accuracy.

1. Why 'Direct' Prompting Occasionally Outperforms scPilot: As detailed in our response (b) to Reviewer RBqf, the rare cases where the 'Direct' method outperforms scPilot are on datasets with ambiguous markers (e.g., Liver). Here, scPilot's deep reasoning can cause it to "overthink" complex signals—a trait mirroring expert caution that can lower a strict accuracy score. The 'Direct' method, by forcing a single "best guess," can be incidentally correct in such noisy conditions.

2. The Scientific Value of "Overthinking": The value of scPilot's reasoning lies in its transparent, scientific process, not just the final label. Its "overthinking" provides crucial insights where direct methods fail. For example, on the Liver dataset:

   • Cluster 16 (Ambiguous Identity): Faced with markers for multiple cell types (hepatoblasts, hepatocytes), scPilot's log noted the cluster "may need finer analysis," correctly identifying the biological ambiguity instead of forcing a likely incorrect label.
   • Cluster 19 (Insufficient Evidence): Lacking strong markers, scPilot flagged this cluster as un-annotatable and a "prime candidate for subgrouping," correctly identifying the limits of the data and preventing a confident error where a direct method would guess.

In conclusion, scPilot's value is its transparent, cautious, and scientifically rigorous analysis. The reasoning trace reveals why a decision is made—or not made—which is invaluable for human trust and guiding future experiments, a key advantage over opaque methods.

(b) Additional experiments to compare with appropriate STOA tools - check response (c) to the reviewer 82g2

We thank the reviewer for this crucial suggestion. As detailed in our response (c) to Reviewer 82g2, we have conducted new experiments comparing scPilot against the latest generation of LLM-based bioinformatics tools.

Expanded Comparison with State-of-the-Art (SOTA) Baselines

We expanded our evaluation to include new SOTA baselines: Biomni (a powerful LLM agent), LLM4GRN (a specialized pipeline), and BioGPT (a foundation model). Our results, detailed in the response to Reviewer 82g2, show that scPilot consistently outperforms these recent methods.

Regarding C2S-Scale and CellAgent

We made significant efforts to include these specific tools but could not perform a fair benchmark due to technical limitations:

1. C2S-Scale: This tool is not fully open-source. The only public version is a 1B model, a class which our experiments show lacks the reasoning capability for scBench, making a comparison uninformative.
2. CellAgent: Evaluation was impossible as its web demo's data upload is non-functional and its source code is unavailable, preventing reproducible analysis.

In summary, we have broadened our comparisons to include relevant and accessible SOTA frameworks. We are committed to benchmarking against tools like C2S-Scale and CellAgent in scBench as they become openly available.

(c) Value of reasoning: qualitative comparison with recent methods (Biomni)

We thank the reviewer for this excellent question. While a direct benchmark against C2S-Scale and CellAgent was not feasible due to their limited accessibility (as explained above), we have conducted a detailed qualitative and quantitative comparison against Biomni, a powerful, state-of-the-art general-purpose biomedical LLM agent (technically, Biomni comes after our submission). This comparison clearly demonstrates the value of scPilot's specialized, reasoning-first approach. (Biomni has a very similar design philosophy to CellAgent, but much powerful).

Contrasting Design: Reasoning-First vs. Tool-First

The core difference is their design philosophy. scPilot is a specialized, reasoning-first agent; Biomni is a general-purpose, tool-first agent.

The Impact on Scientific Accuracy

This design difference leads to significant accuracy gaps.

• Retina Annotation: scPilot correctly distinguishes fine-grained subtypes (e.g., ON- vs. OFF-bipolar cells). Biomni makes clear errors, mislabeling Müller glia as amacrine cells.
• Liver Trajectory: scPilot correctly identifies the Epiblast root and reconstructs faithful lineages. Biomni misplaces cell types (e.g., "Cardiac muscle") in the lineage, ignoring gene evidence.

The Value in Efficiency and Cost

scPilot's specialized approach is also vastly more efficient and cost-effective.

scPilot is up to 30x cheaper and significantly faster because its curated toolchain and reasoning-first method avoid costly, broad tool searches. It also succeeds on the complex Gene Regulatory Network (GRN) prediction task where the general-purpose agent fails. This comparison proves that while general-purpose agents are flexible, scPilot's specialized reasoning delivers higher scientific fidelity at a fraction of the cost.

(d) Additional Examples of Reasoning in Action

We are happy to provide concrete examples of scPilot's reasoning, demonstrating its value in creating interpretable and verifiable workflows. The value lies not just in quantitative performance, but in its transparent, multi-step biological analysis.

Multi-Gene Logic for Accurate and Interpretable Annotation

Instead of simple marker matching, scPilot uses a propose -> filter -> analyze loop for nuanced interpretation, which we detail with examples in Figure 3 and Supp. Figures 1-2.

• How it Works: scPilot proposes cell types from top markers, filters hypotheses against data (e.g., requires key marker presence), and then analyzes multi-gene combinations for a final call.
• Value: This process correctly distinguishes NK cells from CD8 T-cells in the PBMC3k dataset by analyzing the conjunction of NKG7, GNLY, and CD3D expression. This explicit logic, recorded in the trace, allows scientists to verify the decision and is powerful enough to find rare cell types (<1% abundance).

Self-Auditing for Deeper System-Level Understanding

In trajectory inference, scPilot uses external tool outputs to self-correct its own predictions.

• How it Works: scPilot first constructs a lineage tree, then uses a diagnostic report from a tool like py-Monocle to perform a "reconsideration" step, auditing and repairing its initial tree.
• Value: This self-correction significantly improves biological accuracy (graph-edit distance reduced by 6, spectral distance by 0.32), correcting the tree's root and repairing the canonical hepatic developmental ladder. The reasoning trace (detailed in Figure 7 and Supp. Table 6) allows direct comparison of predicted branches against known biology.

Contextual Reasoning for Tissue-Specific Predictions

For Gene Regulatory Network (GRN) prediction, scPilot integrates multiple data types for context-aware analysis.

• How it Works: The framework assesses Transcription Factor (TF)-gene relationships by combining Gene Ontology, expression context, and known regulatory pathways.
• Value: This enables true tissue-specific reasoning. For example, when predicting the Klf4 → Muc5ac regulation, scPilot leverages knowledge that this interaction is key in stomach tissue, yielding a more accurate prediction than a generic model would make (see Supp. Table 7).

These examples show scPilot’s value is in its interpretable workflows, evidence-based self-correction, and nuanced biological interpretation, all exposed for verification. We will highlight these in the supplement, scuh as figure 3, Supp. Fig1, 2, Supp. Table 6, 7.

We sincerely thank Reviewer 82g2 for their detailed and insightful summary. We are particularly grateful that the reviewer recognized the core contributions of our work on multiple levels:

1. Paradigm Shift: For highlighting scPilot's success in transforming "black box" analysis into a "traceable, iterative, chat-like workflow," creating a "bioinformatics assistant."
2. Methodological Innovation: For appreciating the originality of our "complete end-to-end demo" that unites community tools, the "neat idea" of structured data summarization, and the clarity of our "modular design" and "intuitive flowcharts."
3. Solid Evaluation and Community Value: For acknowledging our "fairly thorough experimental comparison" as "pretty solid" and reproducible, and for noting the "potential community value" of the open-sourced scBench benchmark in spurring future research.

(a) Clarification on empirical thresholds and hyperparameters with sensitivity analysis

We thank the reviewer for this excellent suggestion. In response, we have conducted a new sensitivity analysis for key hyperparameter requested (marker gene selection) and are happy to clarify the criteria used for our experimental thresholds. We will add this analysis to the final manuscript to improve transparency.

1. Sensitivity Analysis on Marker Gene Selection ($K$)

For cell-type annotation, a key hyperparameter is $K$, the number of top marker genes. We tested performance sensitivity on the PBMC3k dataset for $K=5$, 10 (our default), and 20.

Table 4. Annotation Accuracy (PBMC3k) vs. Number of Top Genes ($K$)

Performance peaks at our chosen $K=10$. More importantly, the strong results across different values demonstrate our approach is robust and not highly sensitive to this hyperparameter.

2. Clarification on Trajectory Branch Point Extraction

For trajectory branching, our method uses no tunable threshold for branch point extraction. Instead, we directly compare scPilot's predicted graph against the ground-truth lineage curated in the original publications (Pancreas, Liver, Neocortex). The evaluation is a direct comparison to an established biological standard, not a process dependent on an internal parameter.

(b) Validation of intermediate tool calls

We thank the reviewer for this insightful comment. To validate that scPilot's reasoning is grounded in its tool outputs, we performed perturbation studies on two critical components. We will add these results to the manuscript.

1. Perturbation of Gene Ontology (GO) Database in GRN Prediction

To test dependency on the GO database for Gene Regulatory Network (GRN) prediction, we shuffled its term associations with random noise.

Table 5. Impact of GO Perturbation on GRN Prediction (AUROC)

Corrupting GO data significantly degrades AUROC (e.g., $p=0.044$ for gpt-4o-mini), confirming that scPilot's reasoning relies on accurate information from this intermediate step.

2. Perturbation of py-Monocle Output in Trajectory Inference

We tested dependency on py-Monocle by providing scPilot with a corrupted report (randomized cluster relationships and pseudotime order) for the liver dataset.

Table 6. Impact of Monocle Perturbation on Trajectory Inference (o1 model)

The performance drop across metrics demonstrates that accurate outputs from tools like Monocle are critical for scPilot.

Summary: These experiments prove scPilot's success is fundamentally dependent on the integrity of the data provided by the bioinformatics tools it integrates.

(c) Additional experiments with new baselines (traditional bio tools and latest LLM+bioinformatics works)

We thank the reviewer for this crucial suggestion. In response, we conducted extensive new experiments against state-of-the-art (SOTA) baselines.

Expanded Benchmarking with SOTA Baselines

We added several new baselines:

• Updated Traditional Tool: Celltypist (1.7.1).
• SOTA LLM Methods: Biomni (biomedical agent), LLM4GRN (GRN pipeline), and BioGPT-Large (foundation model).

The results confirm scPilot's superior performance holds against these newer, challenging baselines.

Table 7. New Baselines for Cell-Type Annotation

Table 8. New Baseline for Trajectory Inference

Table 9. New Baseline for GRN TF-Gene Prediction

This comprehensive comparison confirms scPilot's superior performance and the advantage of our omics-native reasoning approach. We will include all new results in the new version of the manuscript.

(d) Additional Experiments in Error Propagation and Uncertainty Assessment

We thank the reviewer for these suggestions. In response, we highlight existing work and provide new analyses on error propagation and uncertainty.

1. Error Propagation via Input Perturbation

We highlight the input perturbation experiments from our main submission (Figure 4a), where we removed all contextual metadata (e.g., tissue type) from the input prompt for the PBMC3k annotation task.

Table 10. Results of Context Removal on Annotation Accuracy

This directly shows that removing key input context significantly drops performance, confirming rich metadata is critical for reliability.

2. Uncertainty and Reliability Assessment

We performed a new reliability analysis on the GRN prediction task (Stomach dataset). We calculated 95% confidence intervals (CIs) from 10 trials, performed 1000-bootstrap resampling, and assessed calibration using Expected Calibration Error (ECE) and Brier scores.

Table 11. Uncertainty & Calibration Metrics for GRN Prediction

The powerful o1 model is not only more accurate but also more stable (tighter CI) and reliable (lower ECE and Brier scores). These analyses provide the requested quantitative assessment of scPilot's reliability, which we will add to the manuscript.

(e) Stopping Criterion

We thank the reviewer for this question. The dialogue stopping condition is pre-defined for each task to ensure reproducibility for our scBench benchmark. Please see (e) in our rebuttal to reviewer RBqf.

The balance between "exploration" and "verification" is managed by our structured reasoning loop:

• Exploration: The LLM first generates broad hypotheses from summarized data (e.g., proposing potential cell types).
• Verification: Subsequent tool calls gather evidence to verify these hypotheses (e.g., inspecting data to confirm cell types, as in Figure 3).

This propose -> call tool -> analyze cycle channels the LLM's ideas into data-grounded inquiries, preventing ungrounded tangents and ensuring a scientifically sound workflow.

(f) Consistency and Clarity in Manual Intervention

We thank the reviewer for this opportunity to clarify. We believe there may be a misunderstanding, as our paper does not use the term "fully automatic" nor state that "manual review... is required." We clarify "automatic and reproducible" and the role of human oversight below.

1. How scPilot is Automatic and Reproducible

   • Automatic Execution: "Automatic" means scPilot runs an entire workflow end-to-end without human intervention during execution.
   • Reproducible by Design: The process is reproducible because prompts, tools, and stopping conditions are fixed. This ensures the same inputs produce the same outputs, a cornerstone of scBench.

2. The Role of Human Auditing vs. Required Intervention The confusion may stem from our emphasis on auditability. A key strength is producing a transparent reasoning trace that a scientist can review for validation, much like reviewing a colleague's work. This is fundamentally different from requiring intervention during the process. The system runs automatically; transparency is a feature for trust and insight, not a bug requiring manual patches.

In summary: scPilot is automatic in operation, with outputs that are auditable by design, not dependent on intervention.

We sincerely thank Reviewer 4XCY for highlighting our significant contributions:

1. Methodological Significance: Recognizing our novel approach of "incorporating raw gene data" as "important and essential"—a critical step that has been "ignored by most previous LLM-based frameworks.
2. Experimental Rigor: Acknowledging that our "experiments are a lot," spanning three different tasks with "detailed analysis."
3. Demonstrated Improvement: Confirming that scPilot's performance is "better than previous LLMs based models."

We appreciate the reviewer's questions, which touch upon the nature of our contribution, the breadth of our experimental baselines, and the mechanics of our data-to-text conversion. We are confident that the detailed clarifications below will fully resolve these concerns and reinforce the significance and novelty of our work.

(a) Clarification on Machine Learning Contribution and Originality

We thank the reviewer for this comment, which allows us to clarify our core contribution. While we do not propose a new, standalone ML algorithm in the traditional sense, such as a novel neural network architecture or loss function, our work introduces a novel reasoning paradigm and a foundational benchmark, scBench, which are crucial and non-trivial advancements for applying AI to science.

1. First, A Novel Reasoning Paradigm (Not a Pure Application)

Our primary contribution is Omics-Native Reasoning (ONR), the first systematic framework for an LLM to directly interact with raw omics data in an interpretable reasoning loop.

This is fundamentally different from prior work and is a non-trivial advancement:

• Unlike simple tool-calling agents that merely wrap existing bioinformatics tools, scPilot deeply interprets the numerical output of those tools to form and revise hypotheses.
• Unlike embedding-based foundation models that produce opaque "black-box" vectors, scPilot generates a fully transparent and auditable reasoning trace, addressing a key limitation in computational biology.

As the reviewer noted, creating a framework where an LLM can directly reason with raw gene data is an "important and essential" challenge, ignored by most previous work. Solving this is our core methodological contribution.

2. Second, Tangible Performance Gains from a New Paradigm

This new paradigm is not merely conceptual; it delivers substantial, measurable improvements in analytical accuracy. Our experiments show that the iterative reasoning enabled by scPilot lifts average cell-type annotation accuracy by 11% and cuts trajectory graph-edit distance by 30% compared to simpler, one-shot prompting methods. This demonstrates that changing the reasoning process is a powerful vector for performance improvement, distinct from changing the model architecture.

3. Last but not least, A Foundational Benchmark for Future ML Improvements

Crucially, our work is foundational for future machine learning algorithm improvements. Before algorithms can be improved for a task, that task must be rigorously defined and benchmarked.

1. A New Task & Benchmark: Our ONR paradigm defines the task, and scBench provides the first-ever leaderboard-style benchmark to measure performance on this complex scientific reasoning.
2. Enabling Future Models: The transparent reasoning traces generated by scPilot can serve as high-quality, structured data for fine-tuning the next generation of smaller, more efficient, open-source scientific AI models to become expert "omics-native reasoners."

As a result, our framework further provides the essential blueprint and testbed for these future ML advancements.

(b) Additional Experiments with Non-LLM Baselines

We thank the reviewer for emphasizing the importance of comparing against traditional non-LLM methods. We would like to clarify that our paper already includes these crucial baselines across all three tasks, and our results consistently show that scPilot outperforms them.

In our original submission, we benchmarked scPilot against several established non-LLM methods:

• Cell-Type Annotation: CellTypist and CellMarker 2.0 (Table 2, Line 170)
• Trajectory Inference: py-Monocle (Monocle3) (Table 6, Line 231)
• GRN Prediction: Graph Neural Networks including GCN, GraphSage, and GAT (Table 7, Line 260)

To further strengthen this analysis, we have now added the latest version of Celltypist (1.7.1) as an additional baseline. The results continue to demonstrate scPilot's superior performance.

Cell Type Annotation Accuracy Comparison

As this table and the results in the main paper show (e.g., Table 6 for trajectory, Table 7 for GRN), our LLM-driven scPilot framework consistently and significantly outperforms these state-of-the-art traditional methods.

(c) Clarification on Problem-to-Text Converter

We thank the reviewer for this question and are happy to clarify the mechanics of our Problem-to-Text Converter. Its purpose is to create a compact, information-rich "semantic sketch" of the massive raw data matrix, which can contain over 100,000 cells, so that an LLM can process it within its context window.

Key Steps:

• Clustering: The converter first groups single cells into clusters (e.g., Leiden algorithm via Scanpy). This reduces the dataset from 10⁵ cells to a manageable number of clusters (typically 10–30).

• Feature Selection: For each cluster, the converter identifies the top marker genes (e.g., top 10 genes by differential expression), which define the cluster’s identity.

• Biological Metadata: The converter includes essential metadata such as species, tissue, protocol, and timepoints, providing biological context.

• Structured Prompt Formation: The output is formatted as a structured prompt summarizing for each cluster:

  • Cluster size (number of cells)
  • Top marker genes (ranked)
  • Other relevant features (e.g., mean expression of key genes, pseudotime, etc.)
  • Any available prior annotations

Example:

“Cluster 0: 1,050 cells, Top markers: [Cd14, Lyz, Fcer1g, ...];

Cluster 1: 900 cells, Top markers: [Cd3d, Cd3e, Trac, ...]; ...

Species: Mouse; Tissue: Liver; Timepoints: E9.5, E10.5 ...”

Generalization to All Tasks:

• For cell type annotation, clusters and their marker genes are provided.
• For trajectory inference, the cluster connectivity (e.g., from py-Monocle) and key developmental genes are included.
• For GRN prediction, the converter extracts top TF–target gene pairs (from tools like SCENIC) with supporting NES scores/motifs.

The resulting compressed prompt is small enough to fit in the LLM’s context window, yet detailed enough for meaningful biological reasoning. The LLM then reasons over this digest, proposes hypotheses, and iteratively calls tools for further evidence if needed.

I appreciate authors' effort on providing detailed response. Regarding the Reasoning Paradigm in the innovation, which is claimed as the core contribution in this paper, I don't think the current manuscript successfully verifies whether the reasoning path produced by LLMs is really biologically reasonable. It would be better to provide detailed reasoning paths and examing them by biology experts to see whether it provides biologically plausible reasoning results. Probably, these can be checked by reviewers from computational biology journals. Therefore, I decided to keep my original score.

We are very grateful to Reviewer RBqf for highlighting three pillars of our contribution.

1. Novelty of approach – "enabling the LLM to directly interact with data and reason through biological problems."
2. Community value of scBench – a "standardized and rigorous platform for evaluating LLMs on complex, real-world biological tasks."
3. Transparency & auditability – our fully exposed “chain-of-thought,” which explicitly tackles the usual “black-box” critique by making every step "auditable and easier for researchers to understand and verify."

(a) Accessibility of scPilot and Additional Cost Analysis

We acknowledge the reviewer's valid concern about relying on powerful proprietary LLMs. Our novel omics-native reasoning (ONR) paradigm is inherently more demanding than standard text analysis and requires powerful LLMs, a point confirmed by our experiments on Gemma 3 27B (the best open-source model testable on 8xH100s at the time). Our experiments reveal two reasons why ONR is challenging for current open-source models:

• Insufficient Domain Knowledge: Weaker models lack the nuanced, pre-trained understanding of biology, such as raw markers and pathways, to interpret omics data correctly.
• Poor Instruction Following: Weaker models often fail to follow complex instructions and adhere to biologist-defined formats (e.g., JSON), breaking the reasoning chain.

These insights are a key contribution, offering guidance for future model development. However, the cost of using powerful models via scPilot is negligible. Our detailed cost analysis for Gemini 2.5 Pro shows that a complete run for our most complex tasks costs only a few cents, making the framework accessible.

Table 1. Cost analysis of scPilot on all three tasks (Gemini-2.5 Pro)

The negligible cost allows any lab to leverage state-of-the-art AI without expensive local GPUs. Developing cheaper, specialized open-source models for ONR is a promising future direction; in parallel, our framework provides an immediate, accessible tool for the community.

(b) Additional analysis on occasional suboptimal performance

We thank the reviewer for this sharp question. The observation that a simpler ‘Direct’ prompt can occasionally outperform scPilot is a key finding, not a flaw. These cases are rare, systematic, and informative.

scPilot overwhelmingly outperforms the baseline (wins in 87 of 108 total comparisons). The rare losses are systematic: 13 of 21 (62%) are from less-capable "mini" models. Their limited capacity for sustained logic leads them to "over-explore" and make mistakes with scPilot's extended reasoning.

For the few remaining cases with powerful models, the cause is high dataset nuances, inducing "overthinking." The only two powerful-model losses in cell-type annotation occurred on our most complex dataset, Liver. The characteristics of our datasets explain this pattern:

Table 2: Characteristics of three datasets in Cell Type Annotation

A Concrete Example of "Overthinking": In the complex Liver dataset, scPilot with the powerful o1 model scored 0.518, while the Direct method scored 0.560. A key error was confusing developmentally related hepatocytes and hepatoblasts. Because they share many markers, scPilot's deep reasoning amplified this ambiguity and wrongly merged them, whereas the simpler Direct method was incidentally correct by ignoring this nuance.

This analysis defines the boundaries for applying LLMs to complex biological data and will be added to the manuscript. Future improvements include adaptive reasoning depth and marker clarity assessments.

We provide a more detailed analysis in Reviewer mMx1's rebuttal (a) about why o1's overthinking behaviors in Liver dataset.

(c) Clarification on LLM Tool Selection

This is related to the design philosophy of scPilot. scPilot uses a pre-determined tool workflow as a deliberate design choice to rigorously evaluate the LLM's core omics-native reasoning, not its ability to use tools.

1. Our primary goal is to test the model's ability to interpret complex, numerical outputs from established bioinformatics tools and formulate a coherent scientific argument.
2. By standardizing the pipeline with gold-standard tools (e.g., Monocle, SCENIC), we isolate this core reasoning capability. This prevents our evaluation from being confounded by trivial tool-use errors, like incorrect API calls, which is a separate research problem.
3. Our comparison against a tool-calling agent, Biomni, directly demonstrates the negative impact of such tool-use errors on performance; response (c) to Reviewer

(d) Additional analysis on depth of biological knowledge and reasoning in LLMs

This is an excellent question that gets to the heart of our contribution. First, we should note that the internal "chain-of-thought" of proprietary models (o1) is not accessible.

Second, our assessment is that STOA LLMs possess a surprising depth of latent biological knowledge, and the scPilot framework is crucial for structuring this knowledge into a correct, verifiable, and scientifically rigorous reasoning process. We demonstrate this across the following qualitative analysis

1. Sophisticated Multi-Gene Logic: In cell-type annotation, scPilot moves beyond simple marker matching to distinguish NK cells from CD8 T-cells using a combination of NKG7, GNLY, and CD3D—a task where single-marker tools often fail. It can also identify extremely rare cell types (~1% abundance).

2. Deep System-Level Understanding & Self-Correction: In trajectory inference, scPilot reconstructed the liver developmental tree and, more importantly, used diagnostic outputs from Monocle to self-audit and correct its own reasoning. This improved trajectory accuracy significantly (e.g., reducing graph-edit distance by 6 and spectral distance by 0.32).

3. Contextual, Tissue-Specific Reasoning: For GRN prediction, scPilot applies tissue-specific reasoning, such as understanding Klf4's role specifically in the stomach, to make more accurate predictions.

In conclusion, our framework produces a transparent reasoning trace—a step-by-step record of claims, tool outputs, and decisions—that serves as a fully auditable line of reasoning.

(e) Clarification on Termination Condition

This is an excellent question regarding the mechanics of our framework.

For our benchmark, scBench, the termination conditions are pre-defined for each task to ensure fair, reproducible, and rigorous evaluation, rather than being determined autonomously by the LLM.

The specific conditions are tailored to the nature of each task:

• Cell-Type Annotation: The process is set to a maximum of 3 reasoning iterations. This fixed number provides a sufficient window for the LLM to perform self-correction and refine its initial hypotheses without leading to "overthinking" or excessive computational cost.

• Trajectory Analysis & GRN Prediction: These are structured as single-pass reasoning tasks. For trajectory analysis, this involves an initial tree construction followed by a controlled self-reconsideration step prompted by Monocle's output. For GRN, all evidence is provided upfront, making a single, comprehensive reasoning pass the most effective approach.

This deterministic design is crucial for a benchmark. It ensures models are evaluated under identical conditions, preventing variability that confounds results, an issue seen with the autonomous agent, Biomni.

(f) Potential Information Loss from Data Compression

We agree with the reviewer on this important point. The "Problem-to-Text Converter" is a necessary trade-off to fit large omics datasets into current LLM context windows. While this compression is inherent, our framework is both highly effective today and designed for future extension.

First, our current "semantic sketch" approach is remarkably effective at preserving critical biological signals. For instance, in our Retina analysis, scPilot successfully identified rare Horizontal Cells that constitute less than 1% of the total cell population. This demonstrates that our converter retains the necessary resolution for nuanced discovery, even for subtle signals.

This trade-off represents an exciting research frontier. We envision several strategies to enhance this component:

• Dynamic, RAG-like Retrieval: Allow the LLM to query the full data matrix for fine-grained details on demand.
• Hybrid Foundation Models: Integrate scPilot with specialized single-cell models (e.g., scGPT) that can process raw data, with our LLM acting as the high-level reasoning engine.

Our scBench provides the ideal platform to develop and test these advanced strategies. Thus, while data compression is a challenge, our work offers an effective current solution and a foundational testbed for future innovation.

Dear Reviewer RBqf,

We sincerely thank you for your thoughtful and constructive review of our paper. We were particularly encouraged that you recognized the novelty of our omics-native reasoning approach, the value of our scBench contribution, and the benefit of our framework's transparency.

We have submitted a detailed rebuttal and wanted to gently follow up to ask for your feedback. We aimed to thoroughly address the excellent questions you raised regarding the framework's design and performance.

Specifically, we provide:

• A detailed cost analysis to address the valid concern about accessibility, demonstrating that scPilot is highly affordable.
• A new analysis on the rare cases where simpler methods outperform scPilot, explaining this systematic behavior as an informative finding related to model capability and dataset complexity.
• Direct clarifications on your questions regarding tool selection, the depth of biological knowledge in LLMs, task termination conditions, and potential information loss.

We would be very grateful to learn if our response has helped clarify these points. We welcome any further discussion and are eager to hear your thoughts.

Thank you once again for your time and valuable insights.

Sincerely,

The Authors of Submission 23137