SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design

Thank you for your review and recognition of the novelty of our approach.

Organization of the Methodology

• We will take your comments into consideration for improving the clarity of the paper, such as by improving connections between sections 5-7 to make the paper more accessible.

• Regarding a): Sec. 4 is actually quite different to existing work on BERT for DDI: it shows that language models such as BERT, without features, (i.e., using random tokens as input) are performant on the drug synergy task. This result is initially unintuitive, and it supports our motivation to use language models for in-context learning of drug synergies on unknown drugs/cells. Would you please be able to point out the work you are referring to?

Patient's personalized dataset

• Thank you for pointing this out; we will add more details in the camera-ready. One of the settings we explore is “unknown cell lines”. These often come from a specific patient’s tumor cell biopsy (as in the benchmark datasets we use).
• We discuss a “standardized clinical assay for drug synergy prediction” for creating a personalized dataset. It would work by:

1. A biopsy would be taken from a patient tumor.
2. The biopsy would be tested in the proposed assay against 10-20 drug combos (selected by §6). This would give us a patient-specific personalized dataset.
3. This dataset can be used for patient-specific drug synergy predictions via SynerGPT.

Figure 2 and retrieval methods

• Our method presented in §7 is extending SynerGPT to a new task: inverse drug design via retrieval where the query is synergy tuples as input. Figure 2 shows an example; as more synergy tuples are provided to the model in-context, it is able to predict structures closer to the ground truth.
• This is the first exploration of this novel task; other retrieval methods are not designed to work with drug synergy tuples as input, so they cannot be used to compare. However, we 1) used one of the strongest current representations for molecules, MegaMolBARTv2, and 2) compared multiple different losses, including data augmentation (Appendix F Figure 4 shows results).

TPAMI Paper

• Thank you for pointing out this recent paper! We will cite and discuss it in our related work. We tested their method on our dataset splits and found it to have a ROC-AUC of 81.3 compared to 83.8 from SynerGPT (Table 2).
• A recent review (arxiv.org/abs/2404.02484v2) found it to be outperformed by our baselines GraphSynergy and DeepDDS.

Thank you for your review of our paper, and for noting its “innovative approach”, “strong empirical results”, and “potential impact”.

Limited Evaluation:

• We evaluate on cancer because there are sufficiently large, relevant datasets that have been created. This is because cell lines are frequently derived from cancer cells, whereas other diseases can be more difficult to create datasets for. We follow standard evaluation practices within this research area on datasets consisting of many types of cancer.

Lack of Detail:

• Would you be able to specify which points in the technical description are unclear?

Concerns Over Generalizability:

• While we cannot quantify performance on drugs/diseases that were not included in the dataset, we do create evaluation splits which exclusively contain drugs and cell lines unseen during the training process. Table 2 shows that in this setting we are able to improve model performance via in-context learning.
• Since SynerGPT works via in-context learning, it can be adapted to other tasks fairly easily if they can be represented as a sequence.

Data Preprocessing:

• The data is preprocessed following the methodology in Rozemberczki et al. 2022. The procedure for data splitting in section 6 is as follows: We select either $m$ drugs or cell lines in the “unknown drug” and “unknown cell line” setting, respectively. To construct our data split, we remove all “unknown” synergy tuples so that the remaining dataset only contains tuples with known drugs/cells (this is our training set $D^{Tr}$). The remaining unknown drug/cell tuples are randomly partitioned into three equal sized sets: a context bank $D^{c}$ , a validation set $D^{v}$ , and a test set $D^{Te}$.

Drug Discovery Pipelines:

• This is an excellent question! We will add discussion about this into the revised paper along these lines:

1. Improving efficiency of drug synergy testing: In silico predictions can help reduce expensive real-world testing.
2. Designing new drugs for synergistic effects: Our method can help screen large datasets consisting of millions to billions of possible drug structures for possible synergies.
3. Patient-specific drug synergy prediction: One of the most interesting aspects of our methodology is predicting drug synergies specifically for a patient.

• The cell lines in our dataset already represent genetic diversity in terms of the body tissue type, ethnicity, and biological sex of the originating organism (human).

Thank you for your thoughtful review. We appreciate your recognition of our paper’s contributions to demonstrating the “capabilities of language models to provide a new, efficient, low-cost, and easily implementable solution for drug synergistic research.”

This paper fails to adequately consider the intrinsic features of drugs and cell lines.[...] there are doubts about whether this method can be effective across a broader range of drugs and cell lines.

• This is a good point. In Appendix Table 6, however, we do consider a version of the SynerGPT model architecture trained with features of drugs and cell lines (called “Features”), but we find that results are not improved. Further, in Appendix A we note that while we show that strong performance is possible without features, future work will still likely want to integrate external database features into drug synergy prediction; however, they will likely need to be integrated in a more thoughtful manner in order to ensure an actual benefit. Regardless, in our experimental results we show that our method can improve performance via in-context learning for unseen drugs and cell lines, so we believe it will still be effective.

This paper only simply considers synergy between two drugs; no experiments are conducted regarding synergy among more drugs.

• This is an unfortunate limitation of existing datasets, but our method is easily extensible to n-tuple synergies when relevant datasets become available.

In Section 4, the paper concludes that “replacing drug and cell names with random tokens resulted in no drop in performance.” [...] the training of GPTs?

• Yes, this is the case for Section 5. As shown in Table 2, a SynerGPT model trained with random tokens evaluated without in-context learning (“Zero-Shot SynerGPT”) has roughly the same performance as a GPT-2 model trained with names (“GPT-2”).

In Section 5, the inputs of SynerGPT, such as drugs and cell lines, are obtained from a learnable embedding layer. [...] how much difference in evaluation results will there be with the current SynerGPT?

• We directly fine-tune GPT-2 using real drug and cell line names in the few-shot setting (“GPT-2” in Table 2). We find roughly 3% lower ROC-AUC than SynerGPT. Details are in Appendix B.2.

Thank you again for your comments, which will help us strengthen the writing of our paper.