Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Prediction

Thank you for your detailed comments. We truly appreciate that you found the proposed approach practical and interesting, with convincing results.

We have addressed all the points raised below, including providing additional analysis, figures, and edits to the manuscript.

The rationale for the adapter and the efficiency

Thank you for your observation. While parameter efficiency can generally improve memory and computing requirements, in our setting, we are primarily focused on supporting data efficiency.

Predicting outcomes of molecular perturbations is critical in drug discovery; however, it is challenged by very limited data available for this task, making it a few-shot (or zero-shot) problem in many important settings. For example, the sciplex3 dataset used in this study (and previous studies) includes only 3 cell types and 188 drugs. It is true that the original model has 53M parameters. However, due to the limited data available for this task, it is not ideal to fine-tune all these parameters, as it can be very easy to overfit the small dataset and forget the important biological knowledge learned during pre-training. Therefore, the need for a parameter-efficient approach in our study primarily allows fine-tuning on small chemical perturbation datasets, avoiding overfitting while preserving important background biological knowledge encoded in the pre-trained weights. This approach helps us improve generalization performance, especially in few-shot and zero-shot settings.

To better clarify this point, we added the following sentence in the revised introduction: “Our parameter-efficient approach allows fine-tuning on small chemical perturbation datasets, avoiding overfitting while preserving important background biological knowledge encoded in the pre-trained weights.”

Performance of full model finetuning

Thank you for your feedback. As mentioned in the previous point, we believe the main reason the full model fine-tuning performs worse is that it involves fine-tuning a large number of parameters on a very small amount of data. Fine-tuning such a large model with limited data can lead to overfitting and suboptimal performance, especially in few-shot and zero-shot generalization tasks. As we have shown in Table 1, full model finetuning indeed achieves comparable results to scDCA in the unseen drug setting and good results in the unseen drug-cell-line combo setting. However, the results of full model finetuning are significantly worse in the unseen cell line (few-shot and zero-shot) settings. We attribute the improved results of scDCA in such challenging settings to the ability to retain biological information about unseen cell lines during finetuning.

The rationale behind the bottleneck design isn't well explained

The rationale behind the bottleneck design is similar to that of previous adapter-based finetuning papers [1]. Essentially, this design allows us to significantly reduce the number of parameters by compressing the original embedding size to a much smaller one.

[1] Houlsby et al., Parameter-Efficient Transfer Learning for NLP, NeurIPS (2019).

Missing important experimental details

Thank you for your helpful comment. Following the suggestion, we have added extensive details on the scDCA model architecture and training procedure, including hyperparameters, in Section A.2 in the Appendix. We believe that these details, combined with the code release, will ensure the full reproducibility of the work. Furthermore, we have added training time and memory usage in Section A.11 in the Appendix. As shown, scDCA has reduced training time and memory requirements compared to full model fine-tuning (even though, as previously mentioned, this is not the main goal of the method).

Several broken citations need fixing

Thank you for your observation. We have tried to fix all those that we could find.

The statement about residual connections facilitating drug-cell type interactions lacks citation

Thank you for your helpful suggestion. What we meant with that statement is that residual connections were useful to (marginally) improve model performance. We recognize the sentence was inaccurate and confusing, therefore:

• We removed that statement from the main text, and
• We added an ablation study to Section A.3 in the Appendix to show the effect of the residual network in our architecture. The result from this experiment demonstrates an improvement when using residual connections.

I think the authors for taking the time to respond to my review. I have gone through the paper one more time and while I still believe the paper should be accepted, I have decided to not raise my score. The fine tuning approach strikes me as overly complex, and I do not find the author's justification of overfitting avoidance convincing. Overcomplicated modeling lowers the scientific contribution of this work since it makes it hard to use on novel task as well as replicate. Since the evaluation is restricted to a single dataset I do not find the complexity justified.

Thank you for your detailed comments. We truly appreciate that you found the proposed approach original, promising, and relevant. We have addressed all the points raised below, including providing additional analysis, figures, and edits to the manuscript.

“In the introduction a sentence is formulated as follows…”

Thank you for the comment. Our goal with this sentence was to compare the domain of genetic perturbations, where the “alphabet” of possible perturbation is fixed as the set of genes (with their combinations), to the domain of chemical perturbations, where the set of perturbations is the set of (virtually infinite) compounds. The reason for this comparison is that several methods developed for genetic perturbations (e.g., Roohani et al., 2023) are not directly applicable to the prediction of chemical perturbations, because they make genetic-specific assumptions (such as the perturbation relationship graph used in Roohani et al., 2023).

Since this comparison early in the introduction led to some confusion, we now defer this discussion to Section 2 (Related Work).The rewritten paragraph now reads as follows:

“Several methods focus on predicting the effects of genetic perturbations, where perturbagens correspond to genes and their combinations (e.g., Roohani et al., 2023). These approaches are not always directly applicable to the prediction of chemical perturbations, which include the vast chemical space (estimated to encompass up to $10^60$ drug-like compounds (Bohacek et al., 1996)”.

Model’s performance has not been tested on held-out compounds.

In fact, one of the different settings that we consider specifically tests the model held-out compounds. In more detail, we would like to highlight that we have tested the proposed model across four different settings. These include (1) unseen drug (corresponding to held-out compounds), (2) unseen drug-covariate [renamed unseen drug-cell-line in the revised manuscript], (3) unseen cell type (few-shot), and (4) unseen cell-type (zero-shot). Figure 2 summarizes the different settings. The first two settings are exactly the same as those used in chemCPA. Specifically, in the unseen drug experiment, we held out 30% of the compounds for testing and trained on the remaining compounds. We have improved the illustration in Figure 2 to further clarify the different experiments conducted in the paper.

Overuse of the variable c.

Thank you for bringing this to our attention. We have fixed this in the manuscript by replacing the "c" that refers to the gene-specific attributes in the description of scGPT with "cond".

“Line 261-262: since it is a claim, it would be nice to have some citations”

Thank you for the comment. Following the suggestion, we have included references across multiple domains: language [1], multimodal modeling [2], as well as applied domains of molecular property prediction [3], and medical images [4].

[1] Wei et al., Finetuned Language Models are Zero-Shot Learners. ICLR 2022.

[2] Shen et al., How Much Can CLIP Benefit Vision-and-Language Tasks?. ICLR 2022

[3] Wang et al., Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence. 2022

[4] Zhang et al., A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine. 2024.

“When presenting the feature-based approach in 3.2, it is not clear what kind of model you compare the scDCA approach within Appendix A.2. Since you are doing perturbation prediction, do”

It seems that the rest of your question is incomplete. Could you please provide the complete question so that we can address it accurately? To provide some initial clarification, in Section 3.2, we mention that a common way to fine-tune foundation models is through feature-based fine-tuning. In A.4 (previously A.2), we demonstrate that this approach yields inferior results when compared to our proposed method. We clarified this in the revised version.

Equation 4 and the average embeddings of the controls

This is correct, $X_c^0$ is indeed an average. That is, we provide the average of the control embeddings to scDCA for the results presented in the paper. This approach is simpler and more computationally efficient than considering the full distribution. While an alternative method that accounts for the heterogeneity of the control data could potentially capture more nuances, it might also introduce additional noise into the model. Given that our method already yields the best results across tasks compared to baselines, we did not observe a significant need to modify this aspect of our approach.

Thank you for your insightful comments and for your recognition of our work’s significance and performance.

Comparison with other efficient fine-tuning approaches

Thank you for your feedback. It is true that efficient fine-tuning approaches that address data scarcity have recently been proposed. However, our study tackles a specific multi-modal challenge: fine-tuning a model pre-trained on data from X, where X is gene expression data in our application, through paired data (X, Y, X’), where Y are molecular structures, and (X, X’) correspond to gene expression before/after the perturbation in our application. The proposed approach can effectively handle this multi-modal finetuning setting, ensuring that both gene expression data and molecular information are integrated seamlessly and addressing both data scarcity and multi-modality simultaneously. Previous works have not addressed this specific setting. Therefore, comparisons with other fine-tuning strategies would require non-trivial method development.

Additionally, in this study, we are not primarily focused on proposing a new fine-tuning strategy. Rather, we are interested in extending and customizing a consolidated approach (adapter-based finetuning) to enable an important application (molecular perturbation prediction). Given our main goal, we focused on the comparisons with other molecular perturbation prediction methods recently proposed in the literature.

We appreciate your suggestion and will consider evaluating additional parameter-efficient fine-tuning approaches in future work to further enhance our methodology.

There is no clear explanation on canonical projection.

Thank you for your input. The goal of the projection applied to the molecular embedding is to transform it into the same dimension $d_{bottleneck}$ used for the bottleneck layer. We realize the term “canonical projection” could be confusing if not formally defined, so we removed it in the revised version of the manuscript, simply using “projection”.

The diagram in figure 1 does not match with the projections in equation 5

To better clarify and summarize the architecture, we have added details to Section A.2 in the Appendix, including all the steps and components of the scDCA architecture. While Figure 1 is accurate, we omit some details from it to improve clarity. In Figure 1, the projection after the molecule embedding transforms it into the same dimension $d_{bottleneck}$ used for the bottleneck layer.

The quality of figure 7

Thank you for pointing this out. We have fixed the figure quality in the revised version.

Why other FMs are not considered in this work? We acknowledge the importance of demonstrating the results on other architectures, which will be the subject of future research. However, the main contribution of this work is to demonstrate the ability of a fine-tuned single-cell FM to perform molecular perturbation prediction, which we showed across multiple settings, analysis, and against state-of-the-art baselines.
Importantly, our proposed model is designed to be applicable to all transformer-based foundation models, not just scGPT. Indeed, the only architectural requirement of the proposed approach is a transformer-based architecture, where our conditional adapter is inserted at each transformer block.

Is it possible to do an ablation study, specifically to show the effect of Resnet more clearly?

Following the suggestion, we have conducted this experiment and included it in Table 2 in Section A.3 in the Appendix. As we have illustrated, res_net improves the performance of our model in all settings.

Is it possible to explain why in figure 4 for some genes the predicted expressions are out of the true expression regions?

Thank you for your question. Given that these are model predictions, it is possible that they are less accurate for specific genes associated with specific perturbations. Additionally, the inherent biological variability in gene expression data can contribute to these differences.

Our primary goal with this figure is to show that, in many cases, the model’s predictions successfully capture log-fold changes up to the standard deviation of the underlying single-cell distribution. Even if this is not always true, we showed that, overall, the proposed model outperforms all the considered baseline.

Thank you for your efforts and the time you dedicated to addressing my concerns and questions. While some of my concerns have been resolved, I regret that I am unable to increase my score for the submission. Regarding your rebuttal, particularly as part of the focus is on efficient fine-tuning, I continue to feel that the study lacks a comparison with other efficient fine-tuning methods. I tried to elaborate on my perspective below.

I am unclear why a comparison with other efficient fine-tuning approaches is considered non-trivial, especially given that you have already fine-tuned scGPT. If fine-tuning scGPT is feasible, implementing other efficient fine-tuning algorithms should not present significant challenges. Moreover, without a comparison with other efficient fine-tuning methods, the significance of the adapter module is not sufficiently emphasized. I hope you will consider addressing this point in your future work.

Thank you once again.

Thank you for your detailed comments. We truly appreciate that you found the proposed approach original, novel, well-explained, and, especially, a first step toward an impactful application. We have addressed all the points raised below, including providing additional analysis, figures, and edits to the manuscript.

Improving Figure 2

Thank you for pointing out this matter. We have taken your suggestions into account and improved Figure 2 accordingly. Cell types in this figure are for illustrative purposes, and each square represents a data point (for reference, the cell types used in this study are included in Table 8 of the Appendix). Our main goal for this figure is to introduce the 4 different generalization tasks. We changed cell type icons to avoid confusion, and we added labels to clearly identify the axes. Additionally, we enhanced the caption.

Baseline using the concatenated output of scGPT and the molecular embedding

Thank you for your comments and suggestions. Following the suggestion, we have conducted an experiment where the output of scGPT is concatenated with the output of the molecular embedding to predict the effect on gene expression. The results of this experiment have been added to Table 2 in Section A.3 in the Appendix. Our findings indicate that scDCA outperformed this baseline on all tasks. Additionally, it is worth noting that our approach is also superior in terms of the number of parameters, using less than 1% of the parameters compared to the proposed model. These results are different from the experiment shown in Section A.4 (previously A.2). Specifically, we included this table because, as mentioned in Section 3.2, one common approach to fine-tuning foundation models (FMs) is feature-based fine-tuning. Our results demonstrate that this approach does not perform well in our context, highlighting the superiority of our proposed method.

"By featurizing cell types based on their initial gene expression, our framework can generalize to unseen cell types…”

Thank you for your comment. We agree that this capability really emerges from leveraging a single-cell foundation model (scGPT), which has been pre-trained across multiple cell types and conditions. At the same time, the goal of the proposed approach is exactly that of efficiently fine-tuning the FM such that the information about cell types is preserved and effectively leveraged for molecular perturbation prediction. We agree that the introduced framework does not add anything to the pre-trained model from the perspective of cell types. However, being able to preserve such information, while allowing the incorporation of molecular representations and the ability to flexibly integrate them into the pre-trained knowledge is not trivial, and it is exactly the problem that the proposed framework addresses. As we have shown, other finetuning strategies (Appendix A.3 and A.4) do not lead to the same results as the proposed approach that we attribute to the introduced elements.

We further clarified this point by adding a sentence in the main text after the one you highlighted: “Importantly, the ability to generalize to new cell lines arises from leveraging knowledge available in the pre-trained model and scDCA preserving and integrating it with molecular representations during finetuning.”

Statistical tests for Figure 3

Thank you for your helpful suggestion. We have added the results of statistical tests in Appendix A.8 (Tables 4, 5, 6, 7). Overall, we found that all the comparisons are significant at a 95% nominal confidence level.

Difficulty of different tasks in Table 1

Thank you for your observation. While the different performance levels between few-shot cell-type prediction and unseen drug-covariate [renamed unseen drug-cell-line in the revised manuscript] prediction may seem surprising, these are fundamentally different tasks, each with its own unique challenges and complexities. In particular, in both tasks, many drug-cell-line pairs are observed. However, for the drug-cell-line combination task, we split the data 50% train/50% test (280 train, 281 test combos) while for the few-shot task, the drug measurements in the held out cell line are split 10% train/90% test (392 train, 169 test combos). This creates specific challenges for the model in the few-shot cell-type generalization, with a much smaller percentage of drugs observed in the held out cell line (~50% vs 10%). There might also be additional sources of variability that make the direct comparison challenging, such as the impact of the different top-20 DEGs for each drug-cell-line combination, which can lead to task-dependent difficulty. Therefore, direct comparisons of the prediction results between these tasks are not appropriate. In this study, our main goals are to assess (1) the robustness of the proposed method across different tasks, and (2) the superiority of the proposed method against all the baselines across different tasks. We believe that our results clearly demonstrate both goals. While our results can also offer some insights into task differences, we believe that direct comparisons are not conclusive. We clarified this aspect by adding a sentence in Section 4.2: “While we observe scDCA results to be robust across the different tasks, direct comparison of the performance across tasks is challenging, given differences in splitting strategies and the unique features of each task.” We also added a clarification about the different splitting ratios to the Appendix, Section A.1.

What is Figure 7? What does DEG stand for? What is it showing?

Thank you for your question. DEGs stands for differentially expressed genes. In the presence of a new drug, not all genes may exhibit changes in expression levels, and a subset of genes may exhibit significant changes. Reporting the results on all genes may not capture the drug-related signal effectively. Therefore, similarly to previous studies [1,2], we focus on the DEGs, as these are the genes that show the most significant changes and are therefore more interesting in assessing the model’s performance. While we focus on a specific number (20) of DEGs in the main results, for completeness, Figure 8 (previously Figure 7) illustrates the performance of our model in predicting DEGs for different sizes. The size of the DEG set indicates the difficulty of the task, with smaller sets representing more challenging prediction scenarios. We have clarified this in the figure caption and the corresponding text in the paper. [1] L. Hetzel et al. Predicting cellular responses to novel drug perturbations at a single-cell resolution, NeurIPS, 2022. [2] Y. Roohani et al. Predicting transcriptional outcomes of novel multigene perturbations with gears. Nature Biotechnology, 2024.

Clarifying target-cluster results (Figure 5)

Thank you for your question. The goal of this experiment is to understand the heterogeneity of prediction quality. To achieve this, we stratify the results according to clusters defined by target annotations. More specifically, the design of the experiment is as follows: We first obtain compound annotations from ChEMBL. We conduct a compound hold-out (unseen drug) experiment, ensuring that all annotated compounds are reserved for the test set. For evaluation, we subset the test set compounds to one target cluster at a time. We then report the R2 values for each target cluster. By grouping the R2 values for each target cluster, we aim to assess the robustness of scDCA predictions across different targets. This approach allows us to evaluate the prediction quality heterogeneity for compounds targeting different biological pathways or mechanisms. We clarified this in Section A.6.1 in the Appendix.

Generalization to new cell types and similarity between cell types

In fact, the sciplex3 dataset is composed of the cell lines A549 (lung adenocarcinoma), K562 (myelogenous leukemia) and MCF7 (breast adenocarcinoma). Since there are only these three cell lines present in the sciplex3 dataset, for cell-type generalization tasks, we trained the model in a leave-one-out manner (which corresponds to three splits in total) We added the results for scDCA and baselines for each split separately in Table 8 of Appendix A.9. The lowest performing cell line holdout for scDCA is in fact K562, but this trend is not evident for the baselines (BioLord and SAMS_VAE). In every split, scDCA still outperforms the baselines. In general, we would expect the difficulty of the unseen cell line task to vary according to the degree of biological similarity of the cell lines, but the sciplex3 dataset only offers limited data to further investigate this question. We have added this as a limitation to the conclusion.

“For the unseen drugs how similar are the drugs to each other?”

Thank you for your insightful comment. Following the suggestion, we analyzed the similarity between training and test molecules, focusing on the generalization to new molecules. For each test molecule, we computed its maximum similarity to any training molecule. We used standard functions, leveraging Tanimoto similarity on Morgan fingerprints (radius=2, num_bits=2048) as implemented in RDKit library. We show the distribution of maximum similarities in Figure 9 in A.10. As shown, the vast majority of molecules have low similarity to training structures, with 88% of them having a similarity ≤ 0.4 to any training molecule (0.4 is often considered the threshold to define structural novelty [1]).

[1] Dalke, The chemfp project. J Cheminform 11, 76 (2019).

Overlap between scGPT pretraining set and test set

Thank you for raising this concern. Given that scGPT does not release the full list of datasets used for pretraining, we have thoroughly examined the scGPT GitHub repository, focusing on the functions used to build the dataset that scGPT was trained on. To the best of our knowledge, scGPT was trained on a set of tissues with no cell lines or perturbed gene expression. Therefore, we believe that there is no overlap between the dataset used for training scGPT and the perturbed dataset we used for evaluating our method. Additionally, we noticed that scGPT itself in the original paper fine-tunes the model to predict genetic perturbations using cell line datasets.

I am confused by the terminology "covariate"?

We thank the reviewer for bringing this to our attention. We initially used this term because it was used in prior work, such as chemCPA. The term "covariate" is used broadly for any kind of additional information about a single cell such as different cell types, cell states, diseases, etc. Indeed, in our paper, we mainly used it to refer to a cell type. However, we understand that this may cause confusion, especially since it is not handled as a categorical covariate in scDCA, but instead entirely encoded in the gene expression values of the control cells. For this reason, we changed the terminology in the paper to cell line/cell type wherever this is appropriate. This is currently implemented in the text; for time reasons, we will update the figures later as part of the overall revision/rebuttal process.

Thank you for your detailed and positive comments. We truly appreciate that you found the proposed approach innovative and the paper well-structured. We have addressed all the points raised below, including providing additional analysis, figures, and edits to the manuscript.

Extension to additional dataset (LINCS L1000)

Thank you for your comment. We believe that the evaluation of perturbation models, especially in the single-cell setting, is challenged by the limited size of publicly available data, which we noted in the revised conclusion of the manuscript.

We would like to emphasize that related works such as ChemCPA and BioLORD have also only shown results for the sciplex3 dataset. We are committed to finding related datasets that could be applied to our model. However, as you mentioned, the LINCS L1000 dataset is known to be very noisy, and results could be harder to interpret.

We are actively exploring other potential datasets, including LINCS, that could be used to evaluate our model more comprehensively. As more high-quality single-cell perturbation datasets become available, we plan to extend our evaluations to include these datasets to further validate and enhance our model.

Additional discussion on the biological insights

Thank you for your feedback. In Figure 5 in the paper, we cluster molecules based on their targets (extracted from ChEMBL) and demonstrate that scDCA is robust across all target clusters. Details of this analysis are included in Section A.6 of the Appendix. Even with its limitations, we believe that this analysis provides valuable insights by illustrating that our model can generalize across different biological pathways and mechanisms of action (MOA), something that was not investigated in previous works. In future work, we are going to focus further on exploring these biological insights, for example, understanding features driving the predictions and important gene-molecule interactions. Finally, we would like to emphasize that the primary goal of this paper is methodological, focusing on developing a method that could effectively fine-tune single-cell FMs for the task of molecular perturbation prediction. The biological applications and insights derived from this method are promising and will be a significant area of future research.

How feasible do you think it is to extend this model to such real-world scenarios?

Thank you for your comment. We acknowledge that translating in vitro transcriptional responses to in vivo or clinical settings is indeed a significant challenge due to the increased complexity of cellular environments in real-world scenarios. While our current model demonstrates strong performance with in vitro single-cell RNA sequencing data, extending it to in vivo or clinical settings will require adopting our model to the complex in vivo or clinical settings, access to diverse in vivo datasets, and collaborations with clinical researchers. Despite these challenges, we believe that our methodological advancements provide a solid foundation for future research aimed at translating these predictions to in vivo and clinical settings. In particular, we believe that bridging domain-specific foundation models and efficiently fine-tuning them, especially in multi-modal settings, will be critical to support these real-world scenarios, which are often characterized by extremely limited and highly multi-modal data. We are committed to exploring these directions and enhancing the model's applicability to real-world scenarios.