Contextualizing biological perturbation experiments through language

Thank you for your review and suggestions! We hope this response provides clarity regarding your concerns, and we look forward to discussing.

Contribution and data quality

PerturbQA is first and foremost, a carefully curated benchmark for language-based reasoning for single-cell perturbations.

Compared to existing benchmarks like PubmedQA, the ground truth is not determined by fact checking, but by experimental assays. The quality of our labels can be assessed based on statistical consistencies. Figure 4 illustrates that the Wilcoxon ranked-sum test is reasonably calibrated on our datasets (for determining differential expression), and Figures 5+6 illustrate that in two near-biological replicates, there is high agreement. We also employ conservative thresholds for selecting positives and negatives (A.2), and exclude statistically uncertain examples.

The textual descriptions extracted from knowledge graphs are manually curated by large consortia, compiling decades of research. We believe that the identifier mapping tables they maintain offer a much cleaner means of extracting information relevant to each gene, compared to search engine / embedding-based approaches more common in NLP. While we cannot guarantee that the knowledge is correct, we do include context regarding data provenance (cell line, assay type, whatever is available), as LLMs have demonstrated the ability to filter noisy information, when this is provided [1].

Finally, we do not claim that the LLM-generated summaries are correct; only that they are helpful for making the end prediction, on which the framework is evaluated.

[1] Allen-Zhu and Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. 2024.

Algorithmic novelty

Please see the general response. To summarize: applying NLP techniques out of the box results in near-random performance on PerturbQA, and they must be adapted in domain-specific ways. To demonstrate that this task is feasible, we introduced a minimal LLM-based example (SUMMER), which draws from common LLM techniques. We do not claim that this method is novel from an NLP perspective. We do claim that this LLM-reasoning based framework is novel in context of perturbation modeling.

Specific questions

why not "formalized graph structure with exactly numeric data":

Both GEARS and GAT operate over formal graph structures. GEARS leverages real-valued expression matrices, while GAT is trained on our discretized labels. Both methods underperform in Table 1, and this can be attributed to two factors (Section 3, "modeling perturbations").

• "Raw" single cell expression matrices are actually the end product of extensive preprocessing pipelines, and they are subject to high aleatoric and epistemic noise (batch effects) [2]. In fact, even two datasets generated by the same lab, of the same cell line, can be largely inconsistent at the individual gene level [3]. This is why we employed stringent criteria on PerturbQA examples (A.2), and why we choose to interpret single-cell perturbation data at the level of discrete insights, in line with best practices (Section 3, "statistical insights").

• Biological knowledge graphs are highly heterogeneous, spanning many layers of hierarchy, and are annotated with details regarding the context and provenance of each observation. When current graph-based methods translate annotations into adjacencies, the semantics of each relationship are also lost: edges like "enables," "does not enable," "is part of" are all mapped to "1." Finally, different knowledge graphs operate at differing levels of quality/noise, making their harmonization difficult. Therefore, we only use the graph structure to inform the retrieval and summarization of relevant information, rather than as a strict backbone to constrain modeling.

[2] Luecken and Theis. Current best practices in single‐cell RNA‐seq analysis: a tutorial. Mol Syst Biol (2019) 15: e8746.

[3] Nadig et al. Transcriptome-wide characterization of genetic perturbations. 2024.

formal task definitions: We have included formal definitions for each task in Section 4.1, with motivation in Section 3. Could you please let us know which specific aspects of the notation / setting you find confusing, so we can try to improve the presentation? Thank you!

GEARS and GenePT: We evaluate GenePT and GEARS on our benchmark for completeness, as they represent state-of-the-art approaches for this task. We do not consider their adaptations (for discrete classification) part of our primary technical contribution.

Thank you for your review and questions. We hope this response clarifies certain points of confusion, and we look forward to discussing with you!

Retrieval vs. discovery

"retrieval system:" Please see the general response. To summarize: Only ~3% of our testing pairs physically interact in any context, while only ~20% of pairs are directly connected by any annotation (including coarse ones). We find that the presence of physical interactions is minimally predictive of differential expression (DE).

"my suggestion is to include more datasets [...] For example, in scGPT, the evaluation for perturbation is across 3 different perturbation datasets"

In scGPT, the evaluation was conducted over 3 experiments in the same K562 cell line, including the K562 essential screen that we also incorporate. The other two are older (2016, 2019) and much smaller, as single-cell CRISPR technologies have become more scalable and efficient in the past 5 years.

In contrast, our experiments cover 4 cell lines, derived from 5 experiments, which are the largest publicly-available Perturb-seq screens [1,2]. Biologically, cell lines are very distinct, and they are derived from different cancer tumors / other conditions (K562 myelogenous leukemia, RPE non-cancerous, HepG2 liver cancer, Jurkat acute T cell leukemia). In particular, different genes are expressed, and among the same genes, the marginal distributions (of expression) and gene-gene relationships may differ. To ensure the quality and consistency of our benchmark, we have chosen to focus on these 5 screens, which are larger and published more recently.

[1]. Replogle et al. Mapping information-rich genotype-phenotype landscapes with genome-scale Perturb-seq. Cell. 2022.

[2] Nadig et al. Transcriptome-wide characterization of genetic perturbations. 2024.

Combinatorial perturbations

The lack of combinatorial perturbations is less a limitation of the method, but of evaluation.

To craft a "proof of concept" for combinatorial perturbations, it would be relatively straightforward to summarize the neighborhoods of each participating gene, and prompt for any synergy/lack thereof.

However, existing datasets for combinatorial perturbations are limited. The most commonly analyzed are Norman et al. [3] and Wessels et al. [4], each containing around 100 perturbation pairs. These two experiments operate in different modalities (CRISPR activation vs. inhibition), and since there are no alternatives for comparison, it is difficult to quantify the quality of these data. A major goal of this work was to craft a trustworthy benchmark for perturbation modeling, so we chose to focus on single perturbations instead. Therefore we consider combinations currently out of scope, but an opportunity for future work when better datasets are available.

[3] Norman et al. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science. 2019.

[4] Wessels et al. Efficient combinatorial targeting of RNA transcripts in single cells with Cas13 RNA Perturb-seq. Nat Methods. 2023.

Thank you for your response. Personally, my main concern is how this method influences perturbation analysis from both a logical and practical perspective. I believe the perturbation problem is a complex process in the biological domain. As you mentioned, even the identified perturbation pairs may not accurately represent the actual perturbations. With this in mind, I am unsure whether the knowledge derived is truly reliable for the perturbation task.

One advantage of using pretrained models in perturbation analysis is their ability to study RNA data without needing to verify the knowledge's accuracy. In practice, I also think the pretrained single-cell models need less data to finetune, or infer, or predict the perturbation combination. Additionally, it’s worth noting that scGPT only extracts matching genes from GEARS for specific datasets rather than utilizing GEARS’ structure. Therefore, the results based on the GEARS backbone may not fully align with the implementation.

I appreciate this novel perspective on the perturbation task, particularly how it combines single-cell problems with LLMs, and I will adjust my score because of this novelty.

Thank you for your review and suggestions! We hope this response clarifies some of your confusion, and we look forward to discussing.

Literature review: We are happy to expand our literature review and include a wider range of work on the NLP side. Thank you for the recommendations!

Technical contribution: Please see the general response. To summarize: Our primary contribution is PerturbQA, a carefully crafted benchmark, derived from experimental assays and open knowledge graphs. Applying NLP techniques out of the box results in near-random performance on PerturbQA, and they must be adapted in domain-specific ways. To demonstrate that this task is feasible, we introduced a minimal LLM-based example (SUMMER), which draws from common LLM techniques. We do not claim that this method is novel from an NLP perspective. We do claim that this LLM-reasoning based framework is novel in context of perturbation modeling.

Additional baselines: Please see the general response. To summarize: As Table 1 demonstrates, directly applying NLP methods out-of-the-box on PerturbQA is comparable to random guessing. It is non-trivial to adapt these methods for molecular biology reasoning, especially because the vast majority of biological literature is behind paywalls (vs. in machine learning, where everything is open).

Data leakage: Please see the general response. To summarize: Our "no CoT" and "no retrieve" ablations performed near-random, so it appears that the base Llama model has very poor understanding of experimental outcomes. Furthermore, the gene pairs are also minimally represented in the knowledge graphs, and the presence of a known physical interaction is not predictive.

Thank you for the effort put into the rebuttal.

However, my primary concerns regarding the discussion of graph literature remain unresolved, as these works were neither compared nor even discussed in the updated manuscript.

As a result, I will maintain my negative score.

Thank you for your review and suggestions! We hope this response clarifies some of your questions, and we look forward to discussing with you.

Combinatorial perturbations

The lack of combinatorial perturbations is less a limitation of the method, but of evaluation.

To craft a "proof of concept" for combinatorial perturbations, it would be relatively straightforward to summarize the neighborhoods of each participating gene, and prompt for any synergy/lack thereof.

However, existing datasets for combinatorial perturbations are limited. The most commonly analyzed are Norman et al. [1] and Wessels et al. [2], each containing around 100 perturbation pairs. These two experiments operate in different modalities (CRISPR activation vs. inhibition), and since there are no alternatives for comparison, it is difficult to quantify the quality of these data. A major goal of this work was to craft a trustworthy benchmark for perturbation modeling, so we chose to focus on single perturbations instead. Therefore we consider combinations currently out of scope, but an opportunity for future work when better datasets are available.

[1] Norman et al. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science. 2019.

[2] Wessels et al. Efficient combinatorial targeting of RNA transcripts in single cells with Cas13 RNA Perturb-seq. Nat Methods. 2023.

Domain specific evaluation

Due to the open-ended nature of the gene set task, automated evaluation methods are limited in their ability to reflect practical utility. Since this paper focuses on providing value to biologists, we recruited a domain specialist (molecular biologist, trained in wet lab and computational biology) for this task (See C.1).

Overall, the LLM-generated summary was equal or more informative than the classical gene set enrichment results in 92% of cases, and agrees with the independent annotator in 72% of cases.

1. In 21/25 cases, the biologist reported that the LLM-generated summary was more informative. In 2/25 cases, they contained the same amount of information; and in 2/25 cases, the gene set contained more information.

2. In 18/25 cases, the biologist reported that the LLM summary captured the same biology as the original human annotation (our ground truth labels).

3. In the 2 cases where the gene set contained more information, a list of specific protein complexes were discovered, e.g.

   EIF2AK4 (GCN2) binds tRNA, Aminoacyl-tRNA  binds to the ribosome at the A-site, 80S:Met-tRNAi:mRNA:SECISBP2:Sec-tRNA(Sec):EEFSEC:GTP is hydrolysed to 80S:Met-tRNAi:mRNA:SECISBP2:Sec and EEFSEC:GDP by EEFSEC, UPF1 binds an mRNP with a termination codon preceding an Exon Junction Complex, Translocation of ribosome by 3 bases in the 3' direction, Translation of ROBO3.2 mRNA initiates NMD.
   

   However, this information-rich output is hard to interpret, compared to the LLM output, which the annotator marked as agreeing with the label of "translation."

   Ribosomal Protein Components Involved in Translation: This gene set is comprised of components of the large and small ribosomal subunits, which are essential for protein synthesis and translation. These genes are involved in the assembly and function of the ribosome, facilitating the translation of messenger RNA into protein.
   

4. In the 7/25 cases where the LLM summary differed from the human annotation, the LLM annotation tended to miss some high specific terms, e.g. "targets of nonsense-mediated decay" was generalized to "stress response," and "dysregulated lncRNA antisense transcripts" was generalized to "nuclear gene regulation." Related terms tend to be sparsely annotated in Gene Ontology, so this indicates that it would be useful to tune the granularity of generations in the future, or to generate multiple candidates for specific descriptions.

Thank you for your review and recommendations! We hope this response provides clarity regarding your questions, and we look forward to discussing.

Experimental insight

Thank you for the recommendation! We have updated the draft with several additional analyses regarding the qualitative aspects of our framework (currently in Appendix C), including evaluation from a domain expert (further below). Here is a summary.

• Clusters that elude manual annotation tend to be smaller or exhibit lower agreement. On these, gene set over-representation analysis focuses on highly specific gene sets, which each cover subsets of these clusters, rather than the whole. The LLM takes the opposite approach, and its summaries tend to "lift" the description to higher levels of hierarchy (Table 8). The two strategies provide orthogonal information, though the LLM outputs may be more readable.

• We analyzed 300 generations (3 trials of 100 DE examples) to understand common failure modes (detailed examples in C.3). Errors and inconsistencies primarily resulted from deductions backed by overly-generic information. For example, the LLM may list an excessively broad set of influences, e.g. "mitochondrial function, protein synthesis, or transcriptional regulation," which affect nearly everything.

  In several instances, the LLM was also confused between concepts which may be loosely connected, but not in the same context. For example, Gene A is upstream of stress signaling, e.g. "oxidative stress," which is related to the mitochondria. However, Gene A is not responsible for healthy mitochondria function, and should not respond similarly to genes that are.

Human evaluation

We are happy to provide human evaluation and will update the manuscript accordingly (See C.1). We recruited a domain specialist (molecular biologist, trained in wet lab and computational biology) for this task.

Overall, the LLM-generated summary was equal or better to the classical gene set enrichment results in 92% of cases, and agrees with the independent annotator in 72% of cases.

1. In 21/25 cases, the biologist reported that the LLM-generated summary was more informative. In 2/25 cases, they contained the same amount of information; and in 2/25 cases, the gene set contained more information.
2. In 18/25 cases, the biologist reported that the LLM summary captured the same biology as the original human annotation (our ground truth labels).

Error analysis of human annotation:

1. In the 2 cases where the gene set contained more information, a list of specific protein complexes were discovered, e.g.

   EIF2AK4 (GCN2) binds tRNA, Aminoacyl-tRNA  binds to the ribosome at the A-site, 80S:Met-tRNAi:mRNA:SECISBP2:Sec-tRNA(Sec):EEFSEC:GTP is hydrolysed to 80S:Met-tRNAi:mRNA:SECISBP2:Sec and EEFSEC:GDP by EEFSEC, UPF1 binds an mRNP with a termination codon preceding an Exon Junction Complex, Translocation of ribosome by 3 bases in the 3' direction, Translation of ROBO3.2 mRNA initiates NMD.
   

   However, this information-rich output is hard to interpret, compared to the LLM output, which the annotator marked as agreeing with the label of "translation."

   Ribosomal Protein Components Involved in Translation: This gene set is comprised of components of the large and small ribosomal subunits, which are essential for protein synthesis and translation. These genes are involved in the assembly and function of the ribosome, facilitating the translation of messenger RNA into protein.
   

2. In the 7/25 cases where the LLM summary differed from the human annotation, the LLM annotation tended to miss some high specific terms, e.g. "targets of nonsense-mediated decay" was generalized to "stress response," and "dysregulated lncRNA antisense transcripts" was generalized to "nuclear gene regulation." Related terms tend to be sparsely annotated in Gene Ontology, so this indicates that it would be useful to tune the granularity of generations in the future, or to generate multiple candidates for specific descriptions.