scMPT: towards applying large language models to complement single-cell foundation models

This paper provides a hybrid architecture that integrates the cell representation from pre-trained scGPT and LLM-based text encoders. In detail, the scRNA-seq data is forwarded to scGPT [1] to obtain the cell embedding. Another path converts the data to a so-called cell sentence (this tech was introduced in GenePT [2]) and passes it to an encoder to get the textual feature. The whole framework is simple to understand. Finally, the author conduct experiments to discuss the impact of applying the LLM-based text encoder. Although the performance on cell type annotation seems sound, the other contributions to the biological field are generally few. Besides those, the whole framework just concats the output of scGPT and GenePT, which then passes through a dense layer. The technique contribution is quite weak.

[1] Haotian Cui, et. al. scGPT: toward building a foundation model for single-cell multi-omics using generative AI
[2] Yiqun Chen, et al. GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT

The paper introduces a novel model (scMPT) that leverages synergies between single-cell foundation models and large language models (LLMs) to improve the interpretation of single-cell data. The study investigates the integration of biological insights into the performance of LLMs, which is not well understood, with a focus on using LLMs complementarily rather than as alternatives to existing foundation models.

The work presents scMPT, leveraging the strengths of both models, integrating gene expression data from scGPT and biotextual knowledge from LLM to improve the accuracy of cell classification and stability across different datasets. It is shown that scMPT outperforms independent models by capturing complementary insights, particularly in identifying marker genes critical for cell type prediction. This approach opens new avenues for applying LLMs as complementary tools for biological data analysis.

The paper studies several interesting problems of single-cell foundation models for cell type annotation. Specifically, the authors present an in-depth analysis of the interpretability of text-based annotation methods, a performance comparison between scGPT and text-based methods, and an ensemble method scMPT for fusing both methods.

The most interesting finding from the paper is that the Ember-V1 effectively leverages the syntactical similarity between cell sentences, instead of solely relying on gene semantics and inherent domain knowledge.

Despite this intriguing finding, the overall contribution and novelty of this paper are very weak. The paper mainly applies existing model interpretability methods to existing cell-type annotation methods and proposes a straightforward ensemble of both methods. Meanwhile, the proposed scMPT's performance mostly lies between GPT4o and scGPT, which is not particularly surprising. The highlighted markers from the interpretability experiment do not provide further biological or practical insights.

In summary, I would consider this paper to be an excellent technical report instead of a scientific paper. I would suggest the authors dig deep into the ablation experiments and propose some alternative methods that specifically address the weak points of existing methods exposed in the preliminary study.

The manuscript describes several explorations using language models trained on single-cell transcriptomics data. A distinction is drawn between those pre-trained on strictly transcriptomics data ("single-cell foundation models") and those pre-trained on a more general corpus ("large language models"). Insight is sought into the appropriate usage and differential capabilities of these two classes of models.

First, a set of ten pre-trained models are used in a to encode a set of text representations of transcriptomics data ("cell sentences") and the performance of the resulting embeddings evaluated in cell classification and transcriptome clustering tasks. It is found that Ember-V1 largely outperforms other general-purpose pre-trained models, including those used in previous publications, and that the dedicated pre-trained model scGPT outperforms the generalist models as a class.

In a second experiment, the cell classification performance of several models is evaluated with the inclusion of a list of potential cell types included in the prompt. The generalist models show some capability on the task under these experimental conditions, in contrast to previously reported results wherein the potential cell types were not provided.

Third, two methods, integrated gradients and LIME, are used to identify which information in the cell sentences is being utilized during the cell classification task. A set of marker genes are found amongst the top 10 importance-ranked genes.

Next, a set of experiments are run in which the gene names, frequency of occurrence, or both provided to the Ember-V1 model are altered. The resulting embeddings are once again evaluated by performance on cell classification and transcriptome clustering tasks as in the first experiment. Embeddings generated under both obfuscations retain better-than-random performance, suggesting that the cell sentence representation used contains additional information in its syntax/structure.

Finally, a generalist model (Ember-V1) and a specialist model (scGPT) are joined into a single model (scMPT). The larger models' weights are frozen and trainable MLP segments added for joining their embeddings. The joined model is found to perform well on the cell classification task, and the isolated addition of the MLP segment to the output of either Ember-V1 or scGPT is found to improve cell classification performance over utilizing KNN classification with their embeddings.

I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.