PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction

We thank the reviewer for their detailed and constructive feedback, as well as for recognizing the strengths of our work, including the comprehensiveness of our benchmark and its potential to guide future research. Below, we respond to the specific concerns raised.

GEARS

We agree that including GEARS in Fig. 2 provides a more complete picture of model performance and have updated the figure accordingly (see link).

GEARS uses a gene coexpression graph and a gene ontology graph to model gene relationships and perturbations, respectively, thus injecting biological priors into the model architecture. On the other hand, the SPECTRA graph is built on cell-to-cell similarity, which enables us to measure model robustness to distribution shift by simulating increasingly different train-test sets. GEARS is therefore not aware of the relationship between cells, but aware of the relationship between genes, which therefore doesn’t interfere with the way we assess robustness. We have clarified this in the manuscript to avoid any ambiguity.

Regarding the implementation, GEARS was faithfully reproduced following its original design $1$ . Briefly, it considers a perturbation dataset of $N$ cells, where each cell is described by a gene expression vector $g \\in \\mathbb{R}^K$ and an associated perturbation set P \= (P\_1, \\ldots, P\_M). The model learns a function $f$ that maps a novel perturbation set to its post-perturbation gene expression profile. Specifically, GEARS:

• Uses a GNN encoder $f\_{\\text{pert}}: \\mathcal{Z} \\rightarrow \\mathbb{R}^d$ to embed perturbations,

• Uses a second GNN encoder $f\_{\\text{gene}}: \\mathcal{Z} \\rightarrow \\mathbb{R}^d$ to embed genes,

• Combines these embeddings via a compositional module,

• Applies a cross-gene decoder $f\_{\\text{dec}}: {\\mathbb{R}^d}^K \\rightarrow \\mathbb{R}^K$ to predict the post-perturbation gene expression vector.

Training is conducted end-to-end using the autofocus direction-aware loss, with default hyperparameters from the GEARS paper (e.g., hidden size 64, one GNN layer for both GO and co-expression graphs). The only modification we made was to the GEARS dataloader: we adapted the `prepare_splits` function to use SPECTRA-defined training and test splits via a custom mapping (in `set2conditions`).

Figures 3b and 4

In Fig. 4 we further analyse the outliers seen in Fig. 3b. There, we explore the distribution of the perturbation effect on the top 20 DEGs (dashed line), and how such perturbation effect is predicted by the models (data points). Dense clustering near the dashed line (e.g., Fig. 4a) indicates high agreement with ground truth, while more dispersed predictions (e.g., Fig. 4c) suggest low reliability. This analysis suggests that the magnitude of the perturbation effect is not the only factor affecting performance, but that the distribution of such effect also matters. Perturbations with lower overall effect, but with an atypical distribution will also challenge the models. These plots emphasize how the structure of the ground truth perturbation distribution affects model accuracy, a point we highlight in the revised figure caption. We have revised the Figure to improve clarity, using smaller, non-overlapping data points to better distinguish their values. The revised figure is available at this link.

We hope these clarifications and updates address the reviewer’s concerns. We thank the reviewer again for their thoughtful comments, which helped us strengthen the clarity and completeness of our work.

$1$ https://doi.org/10.1038/s41587-023-01905-6

We thank the reviewer for their thoughtful and positive evaluation of our work. We are glad that the reviewer found our contribution timely and the benchmark, metrics, and manuscript clear and useful for the field. Below, we address the specific concerns raised.

Causal intervention

We acknowledge that setting gene expression to zero may prevent the model from distinguishing between biological zeros and knockout interventions. However, there will be differences in coexpression accompanying a natural zero, which will not occur for induced zeros. We acknowledge this limitation in our paper and agree that future foundation models would benefit from incorporating representations that explicitly encode the nature of the intervention. A potential avenue of exploration would be to include perturbation tokens during pretraining and not only during fine-tuning.

Perturbation representation

While simulating perturbations via gene knockout may seem counterintuitive for CRISPRa datasets like Norman, we chose this strategy to address the challenges inherent in standardizing perturbation presentation across diverse scFMs. Different scFMs represent perturbations in different ways - for instance, scGPT encodes gene expression as numerical values, whereas Geneformer uses rank-order representations.

Therefore, upregulation is difficult to simulate consistently: parametric models like scGPT require arbitrary magnitude choices, with no biologically grounded way to determine appropriate magnitude, while rank-based models like Geneformer lack a principled way to re-rank genes. On the other hand, knockout offers a model-agnostic, unbiased representation that avoids model-specific inconsistencies, enabling fairer comparisons across scFMs.

Nonetheless, we performed an experiment using an alternative perturbation representation, which simulates gain-of-function by doubling the perturbed gene’s expression in the Norman control expression data. We used scBERT to generate embeddings for the in silico upregulated input representation (scBERT+). We observed minimal performance differences (see Table 1 and the Figure in this link) compared to the knockout representation (scBERT-), supporting our approach. The difficulty of simulating realistic perturbations reflects a broader challenge in scFM methodology and while a full exploration of perturbation strategies is beyond the scope of this work, we hope our findings motivate further research in this area.

Table 1: MSE ± standard deviation for Norman single gene embeddings generated with scBERT using an in silico knockout (scBERT -) vs. an upregulation strategy (scBERT+)

Dataset and pre-processing choices

We agree that dataset diversity is critical for generalizability. We selected Norman and Replogle as they represent two of the most widely used, high-quality single-cell perturbation datasets currently available. Moreover, both Norman and Replogle contain two separate datasets each, so we are effectively exploring four datasets. Importantly, these all allow us to benchmark under controlled conditions and introduce distribution shifts via SPECTRA. Our platform is designed to be extensible, and we are actively exploring incorporation of additional datasets, including large-scale resources such as the recently published Tahoe-100M $1$ .

Regarding preprocessing, we applied a consistent pipeline that follows standard best practices in single-cell analysis across all models, including highly variable gene (HVG) selection, transcript count normalization, and log transformation. These steps are aligned with those used in the original scFM papers, differing only in hyperparameter value choices. We ensure internal consistency in preprocessing across all model evaluations to enable fair comparisons. While we acknowledge that preprocessing choices can affect results, this is a well-recognized, broader limitation of single-cell data analysis, which we explicitly discuss in our manuscript.

We hope our responses clarify the rationale behind our design choices and reinforce the value of our benchmark as a foundation for future work. We believe that making the limitations explicit and quantifiable is a critical step for advancing the development of biologically meaningful scFMs.

$1$ https://www.biorxiv.org/content/10.1101/2025.02.20.639398v1

We thank the reviewer for their constructive feedback on the work presented. We are committed to maintaining the benchmark as a live and extensible resource. Because we have prioritized reproducibility and modularity in our design, incorporating new models and datasets is straightforward. We are actively monitoring developments in the field, and one exciting recent advancement is Tahoe-100M $1$ , currently the largest perturbational single-cell dataset. We intend to extend our benchmark to include such resources in future iterations. As scFMs continue to evolve, especially with respect to how they represent perturbations, we plan to integrate these innovations to support broader perturbation types beyond knockouts.

Below we address the reviewer’s concerns:

Perturbation representation

We acknowledge that nullifying gene expression may not align with biological reality, especially in gain-of-function settings such as CRISPRa. We refer you to our detailed answer on this point in the rebuttal to Reviewer pVrS. You can also see the results of a preliminary experiment with upregulated embeddings at this link.

We thank you for the suggestion of incorporating attribution-based methods. If we understand correctly, calculating the effects of infinitesimal perturbations is a post-hoc explainability tool, which we would be interested in incorporating in our framework. However, it is unclear to us how we would use it for the initial representation of perturbations. We would be happy to discuss this point further.

Limitations of averaging and unpaired observations in scRNA-seq

Current methods do not allow for true paired observation of pre- and post-perturbation states for the same cell, due to the destructive nature of measurement. As the reviewer has noted, this introduces ambiguity into how perturbation effects are estimated.

Indeed, due to the context-dependent nature of biology, it is very likely that there are other factors that affect gene expression. This trade-off is an inherent limitation of current experimental techniques and affects all existing perturbation modeling studies. It is unclear to what extent this distorts the results without access to paired samples, or a robust estimate of paired samples. We believe the development of experimental protocols enabling paired measurements would significantly advance this line of work.

To attempt to mitigate this, we adopt a pseudo-bulking strategy that averages across cells to reduce noise and obtain more robust perturbation signatures. Specifically, for each condition, we randomly sample 500 control cells and average their expression profiles, pairing them with the mean expression of all perturbed cells within the same perturbation. This approach helps suppress cell-to-cell variability in the perturbed population, thereby making the overall perturbation effect more apparent. However, as the reviewer rightly points out, the unpaired nature of the control introduces uncertainty about whether the starting state truly mirrors that of the perturbed cells pre-perturbation.

Thank you for pointing out the typo in the appendix, we have now corrected it.

We thank the reviewer for recognizing the clarity and relevance of our work. We appreciate the suggestions regarding perturbation representation and data limitations, both of which highlight important areas for continued exploration in this field. We hope our responses address the reviewer’s concerns and demonstrate the care with which we have designed our framework to support the community in developing and evaluating biologically meaningful single-cell foundation models.

$1$ https://www.biorxiv.org/content/10.1101/2025.02.20.639398v1

We thank the reviewer for the thorough evaluation and constructive feedback. Below, we address each of the points raised.

Perturbation representation

We address the limitation of simulating perturbations via gene knockouts in our response to Reviewer pVrS and refer you to it. You can also see the results of a preliminary experiment with upregulated embeddings at this link.

Fine-tuning

One of the main goals of our benchmarking approach is to assess the zero-shot information content of pre-trained embeddings. If the models encode meaningful biological information for perturbation prediction, this would be apparent without task-specific adaptation. If performance gains only appear through fine-tuning, this challenges the premise that these models inherently learn generalizable representations.

MLP probes are commonly used as an approach to answer the question of information content of embeddings in NLP and CV, as they evaluate representation quality while removing confounding effects pertaining to task-specific prediction heads, which introduce inductive biases $1, 2, 3$ . We also note that previous work has investigated the performance of fine-tuned versions of scGPT and scFoundation $ 4]. This study found that simple linear baselines still outperformed these models, suggesting that fine-tuning does not fully address their limitations. Because fine-tuning performance is addressed in other studies and because it fundamentally goes against our approach of establishing existing information content, we do not include fine-tuning in our study design. The findings we present therefore highlight that zero-shot scFM embeddings do not contain useful biological information pertaining to the perturbation prediction task, which in and of itself is an important finding.

Discrepancy with scGPT results

Several factors may explain the discrepancies between obtained results: scGPT evaluates model performance after fine-tuning on perturbation data, reporting high performance, whereas we focus on evaluating zero-shot information content. We also evaluate robustness under distribution-shift, which is not considered in scGPT. We view our study as complementary to scGPT rather than contradictory. It highlights current limitations in zero-shot scFMs and clarifies the gap between pre-training and real-world application. We believe that these complementary perspectives can help guide future improvements so that models better capture perturbation effects.

Evaluation

MSE was selected as the primary metric due to its strong biological grounding and demonstrated effectiveness in capturing perturbation effect, as opposed to Pearson Correlation [5 $ . Furthermore, we aimed to provide a toolbox of comprehensive and complimentary metrics, which include: AUSPC (distribution shift robustness), E-distance (perturbation magnitude), and contextual alignment (pre-training relevance). Together, these provide a robust framework for evaluating scFMs that model transcriptomic perturbation outcomes. The comprehensiveness of our evaluation framework has also been recognised as one of the strengths of our paper by other reviewers.

We would also like to note that GEARS is not considered a simple baseline, rather a SOTA model which takes biological priors into account and is developed specifically for perturbation effect prediction. The fact GEARS outperforms the zero-shot scFMs supports our conclusions that a biologically grounded architecture, which incorporates strong inductive biases, is more useful than pretraining for this task.

We hope these clarifications demonstrate the careful thought that has gone into our experimental design and underscores the broader significance of our findings. By evaluating zero-shot capabilities across scFMs using a rigorous and biologically motivated framework, we provide a valuable benchmark and identify key limitations in current approaches. We believe this work will help guide future research toward more robust, generalizable, and biologically meaningful models.

$1$ https://arxiv.org/abs/2103.00020

$2$ https://arxiv.org/abs/1905.06316

$3$ https://doi.org/10.1038/s42256-024-00949-w

$4$ https://www.biorxiv.org/content/10.1101/2024.09.16.613342v4

$5$ https://www.biorxiv.org/content/10.1101/2023.12.26.572833v1