How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval

We thank the reviewer for providing detailed feedback for our paper. Below, we aim to address your concerns point-by-point

Concern #1: In Table 2 and Table 4, there is a significant performance increase for DCL, CWCL, SigLip, and S2L compared to other baselines. The source of these improvements is unclear.

We assess the effectiveness of S2L, SigLip, and DCL losses by analyzing the gradient flow of the InfoNCE and DCL formulations. In particular, we analyze the cases where inactive molecules cause the gradient to vanish, inhibiting training using InfoNCE and variants thereof.

Decoupled contrastive learning (DCL) loss is an effective alternative to the InfoNCE loss due to the modification by removing the positive term from the denominator:

$$\mathcal{L}\_{\text{DCL}} = - \frac{1}{N}\sum\_{i=1}^N \left[ \log \frac{ \exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_i} \rangle / \tau) }{ \sum\_{k = 1, k \neq i}^{N} \exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_k} \rangle / \tau) + \cancel{\exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_i} \rangle / \tau)} } \right].$$

The authors show that when computing the gradient of InfoNCE, the term $q_{B,i}$ which they name the negative-positive coupling (NPC) term modulates the gradient of each sample and in cases where the term becomes small gradient flow is inhibited:

$$q\_{B,i} = 1 - \frac{ \exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_i} \rangle / \tau) }{ \sum\_{k = 1, k \neq i}^{N} \exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_k} \rangle / \tau) + \exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_i} \rangle / \tau) }.$$

NPC can be small when the positive samples are too close to one another, when there is a small number of negative samples, or when the negative samples are too simple to discriminate versus the positive pair. By removing $\exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_i} \rangle / \tau)$ from the denominator, DCL simplifies the training thus removing the NPC term from the gradient calculation.

We hypothesize that in our case inactive molecules tend to be simple negatives, thus inhibiting the gradient flow that would otherwise be helpful to the model training. Training the model with a loss that separates the positive gradient term (attracting samples from two modalities) from the negative (repelling negative pairs) achieves higher overall performance. Similarly to DCL, SigLIP and S2L losses are computed for each pair of samples independently, thus separating the positive and negative loss terms. Thus the gradient calculation for informative samples is unaffected by the NPC term, resulting in higher overall performance.

Concern #2: The paper primarily compares MolPhenix with CLOOME and a few other general domain objectives… numerous related studies specifically within the molecular-phenotype contrastive learning domain.

We thank the reviewer for an in depth exploration of relevant works. We note that the second paper linked is an early presentation of CLOOME (2023) to which we extensively compare (2nd referenced paper in reviewer suggested list). In addition we utilized a number of strong baselines from the Image-Text multi-modality literature such as CWCL (2023), SigLIP (2023), and DCL (2021).

We agree with the reviewer that we can strengthen the pheno-molecular multi-modal related works section. We focussed our assessment on works that innovate on the methodological components but agree it is important to capture a broader perspective. We note that these additional works support the importance of MolPhenix as it demonstrates the importance of learning an effective joint embedding of phenomic experiments and molecular structures. MolPhenix contributions can be used in further improvement of utilizing phenomics and cell-profiliing experiments for cancer cell line drug response prediction (4th referenced paper in reviewer suggested list) and gene knockout predictions (6th referenced paper in reviewer suggested list). As the authors of “Cross-modal graph contrastive learning with cellular images” conclude in their paper: “There are still some challenges that need to be addressed, such as inherent data noise and batch effect. These could be resolved by designing specific encoders for cellular images, optimizing cross-modal fusion mechanisms, and introducing heterogeneous cross-modal data”. Our work aims to introduce general methods for extracting additional information from phenomic data such as k-patch averaging, S2L inter-sample similarity aware loss, and concentration encoding.

We have updated our related Molecular-Phenomic Contrastive Learning section to reference and discuss the listed works.

Concern #3: Could the authors also provide results for top N accuracy, where N = 1, 10, and 100?

In the Image-Text domain it is common to provide N=1,10,100 0-shot class retrieval results. Since the number of classes between studies for a given dataset is consistent, top-K results always correspond to a semantically meaningful metric.

However, in our case we believe these statistics might be misleading since different works evaluate on datasets of different sizes, thus making the N=1 recall task of varying difficulty in a dataset of 100 molecules VS 100,000. Other published works resort to a similar strategy, for example “Cross-modal graph contrastive learning with cellular images” in Table 1 report Hit@1, 5, 10 which actually correspond to percentages due to the subsampling of the overall data to 100 data points. Similarly, “Molecule-Morphology Contrastive Pretraining for Transferable Molecular Representation” report top 1, 5, and 10% retrieval in Figure 2.

We note that in the external dataset - held out dose evaluation setting, the dataset consists of 1639 molecules making the top1% include just 16 molecules total.

We thank you for a detailed and thorough assessment of this work. Please let us know if there are any additional points we can discuss, to improve your assessment of our work.

We thank the reviewer for the positive, insightful comments and the detailed feedback. We believe your perspective will help shape this into a stronger work. Below, we aim to discuss each of the suggestions and discussion points individually.

Discussion point #1: “Generating phenotypes or molecules should be possible”

We agree with the reviewer that this is a really exciting future direction for the work. As a first step in establishing pheno-molecular multi-modal learning, we decided to focus on the retrieval task due to its straight forward quantitative evaluation advantage. Our goal in this work was to establish a set of design decisions and guidelines that we can quantitatively investigate for effective latent space stratification.

High pheno-molecular retrieval rates would indicate to us, that the model learns effective stratification of the latent space, opening up the door for future experiments with molecular and phenomics generation. This is a critical research direction with applications in drug-discovery such as identifying phenotypically similar molecular analogs with applications to generic molecule research.

There is a unique set of challenges in evaluating high quality, diverse molecule designs. The state of the art for assessing molecule quality are still biological experiments, which we are excited to explore in future works.

Discussion #2: The authors focus on retrieval instead of evaluation of the latent space.

We certainly agree with the reviewer that MolPhenix embeddings should be able to provide additional interesting assessments as to the quality of the model latent space. To that end we have multiple experiments described in Appendixes E1 and E2 which we could not include in the main text due to space constraints. In addition, we conduct experiments assessing the quality of the learned molecular representation by experimenting on 35 downstream tasks, discussed in more detail in the general response (Table 4 attached PDF).

Some brief details on appendix experiments: We were interested in assessing whether our molecular encoder captures pheno-activity throughout training. We found that by specifying only the molecular structure and the corresponding concentration, we are able to predict molecular activity levels with an ROC-AUC of 0.9. Visualization of the model latent space can be found in supplementary Figure 7. These results open up the door for in-silico activity screening that have previously been infeasible. In addition, these can be used as in-silico dose-response curve evaluations, finding an activity - dosage trade-off for a previously unscreened molecule.

In appendix E2 we evaluate MolPhenix’s ability to identify previously known concordant perturbations between small molecules and genetic knockouts. From a database of known relationships, we encode the molecular perturbation with the MolPhenix molecular encoder and assess whether it is able to match them with a corresponding embedding of a genetic knockout. To create an embedding of a genetic perturbation, we embed results from a phenomic experiment of cell lines with a corresponding gene knocked out. We find that this in-silico perturbation concordance experiment is able to provide strong results relative to a fully experimental baseline.

We believe these findings are important initial experiments demonstrating downstream utility applications of MolPhenix learned embeddings.

Concern #3: The authors train on a private dataset which makes the paper almost not reproducible. However, due to detailed model descriptions, the guidelines could be evaluated on other datasets.

We are unfortunately unable to release the training dataset for MolPhenix but aim to disseminate generalizable findings that can be helpful to other scientists working in this domain. To that end we provide pseudo-code and algorithmic descriptions for S2L loss. Additionally, our algorithm implementation is kept in PyTorch-like syntax for easier reproducibility. In addition, we evaluate our models on a large openly accessible dependent RxRx3 dataset which can be used for evaluation of other models and benchmarking other design choices by the community.

Concern #4: Please elaborate on generative or mechanistic opportunities of the model

This is an important point that we hope that we’ve addressed in sufficient detail in discussion points 1 and 2.

Concern #5: You argue MolPhenix cannot be applied to images. Please elaborate since instead of ph-1 a simple non domain-specific image feature extractor could be used.

We thank the reviewer for bringing up this important point, and would like to clarify that any sufficiently expressive image feature encoder can be used. In the general reviewer response, for example, we demonstrate that we can use an alternative encoder that is trained in a supervised fashion to predict identity of genetic perturbations (as an alternative to θPh-1). We hope that these experiments in addition to use of an ensemble of publicly accessible fingerprints provide sufficient evidence that the guidelines are generalizable across a number of public and private encoders.

We have updated Table 1 caption to be a more clear description of our training pipeline: “We note that MolPhenix’s main components such as S2L and embeddings averaging relies on having a pre-trained uni-modal phenomics model.”

Concern #6: Please comment on which model weights of ph-1, Mol-1 and MolPhenix are publicly available or you will be releasing?

We note that the code and training data for Mol-1 are available, which we will provide references to in the final version of the paper. In addition, a public version of θph-1 will be available for inference.

Thank you for your positive review and your interest in our work. We will be happy to further discuss and answer any additional points of interest.

We thank the reviewer for providing detailed feedback on our paper. We aim to address the feedback point by point below:

Concern #1: The clarity of this work could be significantly improved

We thank the reviewer for this constructive feedback, we believe this work is best assessed with the required context of molecular biology, so we aim to enhance the accessibility of our paper by adding a biological terms explanation in the appendix. In particular, we will define the following terms: phenomics, cell morphology, cell line (ARPE-19), molecular concentration, phenomolecular retrieval, cell staining, molecular perturbations, inactive molecule perturbations, batch effects, molecular fingerprints, initial cell state assumption.

While we agree the clarity of the paper can always be further improved, we note that the presentation of our paper was a highlight for other reviewers: YXwu “The paper is well-structured and clearly written”, ELHC “The paper is very well written”, Vz3t “The paper is well-written, and the motivation is clear.” Getting high ratings in soundness (3, 3, 4, 3) and presentation (3, 3, 4, 3). Additional information on related works can be found in section 2, and we have expanded the “Related Work” section on molecular-phenomic approaches.

Please let us know if this clarifies the accessibility of the paper, or if there is any additional terminology that we can add to the glossary.

Concern #2: Comparisons with a broader range of state-of-the-art methods in … multi-modal representation learning in genomics and proteomics

We’ve restricted the scope of this paper to studying phenomics and molecular modalities. While expanding to other biological modalities such as genomics or proteomics would undoubtedly be very interesting, it requires a significant commitment to curating the data and innovation in biological sequence model space. This is a direction we leave for future exploration.

The reviewer could be interested in a 0-shot evaluation that we perform to investigate MolPhenix’s ability to generalize to genetic knock-out perturbations (Appendix D2). We assess the model’s 0-shot generalization capabilities that are known to have similar molecular effects to learned molecular perturbations. The evaluation demonstrates that MolPhenix learns the landscape of genetic perturbations, allowing the model to recover known biological pairs.

On the methodological front, we compare to SOTA in the Image-Text multi-modal training consisting of DCL(2021), CWCL (2023), and SigLip (2023) - recent methods that have demonstrated significant success. In addition, we conduct thorough evaluations benchmarking recently published CLOOME (November 2023) model.

Concern #3: Significance of the work could be more explicitly articulated and demonstrated

Although the main text of our paper is mostly focused on pheno-molecular retrieval, we have some initial experiments in the appendix assessing the model’s ability to perform other biologically relevant tasks. We perform pheno-activity experiments, demonstrating that the learned embedding is predictive of molecule - concentration tuple morphology impact (Appendix E1). This opens the door to potential in-silico activity pre-sceening and in-silico dose-response curve construction. In addition, we perform 0-shot biological activity experiments mentioned earlier in Appendix D2.

In the general reviewer response, we include new experiments showcasing the effectiveness of the learned latent space by conducting a KNN evaluation of the MolPhenix latent space. We assess the learned embedding on 35 molecular property prediction tasks across the Polaris and TDC datasets (see Table 4 in the attached PDF). Our findings indicate that MolPhenix, when trained with Fingerprint embeddings, consistently outperforms standalone input fingerprints, effectively clustering molecules according to their molecular properties.

Concern #4: “Combining GNNs and fingerprints”

This is a valuable evaluation for the completeness of our work. Please find the results for these experiments in the general reviewer response (Table 2 and 3 in the attached PDF). We find that combining pre-trained GNN and molecular fingerprints further enhances retrieval performance.

Concern #5: “Report and compare metrics for top-5 and top-10 recall”

We also report top-5% accuracy metrics in supplementary tables 8, 9, 10 and 11. We choose to report hits in top-K% since we have variable sized test sets. Top K metrics do not control for the difficulty of this task unless artificially subsampled to a pre-determined size. This is proportionally equivalent to Top-K% evaluation metrics, but has the downside of being stochastic contingent on composition of the batch.

Kindly consider adjusting our overall score if our response addressed your primary concerns. We would be happy to answer any additional questions you may have in order for you to support acceptance of our work.

We thank the reviewer for providing detailed feedback on the paper.

Concern #1: Additional justification for the S2L loss

In this section we provide some additional intuition for the S2L loss and further relate it to previous works. We first assess the conceptual similarities between InfoNCE and CWCL loss and justify a similar extrapolation for the relationship between S2L and SigLIP losses.

InfoNCE can be considered a special case of the CWCL loss, where $w_{ij}$ is set to 0 for all pairs of $i$ and $j$ unless $i=j$. Conceptually, this is equivalent to stating that all the negative pairs are equally distant from the reference sample. We will consider a uni directional loss CWCL, for identifying $\mathcal{X}$ from $\mathcal{M}$:

$$\mathcal{L}\_{\text{CWCL}, \: \mathcal{M} \rightarrow \mathcal{X}} = - \frac{1}{N}\sum\_{i=1}^N \left[ \frac{1}{\sum\_{j=1}^{N} \mathbf{w}^{\mathcal{X}}\_{i,j}} \sum\_{j=1}^{N} \mathbf{w}^{\mathcal{X}}\_{i,j} \log \frac{ \exp{\left( \langle \mathbf{z}\_{x\_i},\mathbf{z}\_{m\_j} \rangle / \tau \right) }} { \sum\_{k=1}^{N} \exp{ \left( \langle \mathbf{z}\_{x\_j}, \mathbf{z}\_{m\_k} \rangle / \tau \right) }} \right].$$

If we set $w_{ij} = 0$ when $i \neq j$ and $1$ otherwise then the term $\Sigma_{j=1}^{N} w_{i,j}^\mathcal{X}$ evaluates to 1 and the above expression simplifies to:

$$\mathcal{L}\_{\text{InfoNCE}} = - \frac{1}{N}\sum\_{i=1}^N \left[ \log \frac{ \exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_i} \rangle / \tau) }{ \sum\_{k = 1}^{N} \exp(\langle \mathbf{z}\_{x\_i}, \mathbf{z}\_{m\_k} \rangle / \tau) } \right].$$

In the case of CWCL, a non 0 $w_{i,j}$ determined by a within modality similarity function informed by a pre-trained model

Similarly SigLip can be considered a special case of S2L when $w_{i,j}^{\mathcal{X}} = 0$ when $i \neq j$ and $w_{i,j}^{\mathcal{X}} = 1$ in the case $i=j$. This is the formulation of S2L

$$\mathcal{L}\_{\text{S2L}} = - \frac{1}{N} \sum\_{i=1}^N \sum\_{j=1}^N \log \left[ \frac{\mathbf{w}^{\mathcal{X}}\_{i,j}} {1 + \exp \left( -\alpha \mathbf{z}\_{\mathbf{x}\_{i}}.\mathbf{z}\_{\mathbf{m}\_{j}} + b \right) } + \frac{(1 - \mathbf{w}^{\mathcal{X}}\_{i,j})} {1 + \exp \left( \alpha \mathbf{z}\_{\mathbf{x}\_{i}}.\mathbf{z}\_{\mathbf{m}\_{j}} + b \right) } \right].$$

It can be simplified to SigLIP by setting $w_{i,j}^{\mathcal{X}}$ to $1$ when $y_{i,j} = 1$ thus setting the term $\frac{(1 - \mathbf{w}^{\mathcal{X}}\_{i,j})}{1 + \exp \left( \alpha \mathbf{z}\_{\mathbf{x}\_{i}} \cdot \mathbf{z}\_{\mathbf{m}\_{j}} + b \right)}$, corresponding to $i \neq j$, we set $w_{i,j}^{\mathcal{X}}$ to $0$ thus negating the first part of the $\mathcal{L}_{S2L}$ loss, evaluating to:

$$\mathcal{L}\_{\text{SigLIP}} = - \frac{1}{N} \sum\_{i=1}^N \sum\_{j=1}^N \left[ \log \frac{1}{1 + \exp{ \left( \mathbf{y}\_{i,j}(-\alpha \: \langle \mathbf{z}\_{\mathbf{x}\_{i}}, \mathbf{z}\_{\mathbf{m}\_{j}} \rangle + b) \right) }} \right].$$

Having a $0 \leq w_{i,j}^{\mathcal{X}} \leq 1$ allows us to inform the training by going between discrete negative labels to continuous informed by some prior information. This information is given by a pre-trained encoder $\theta_{Phi}$, in our case but can be informed by any pre-trained model.

Concern #2: “Inadequate ablation of pre-trained components”

To investigate the impact of pre-trained encoders, we perform additional experiments evaluating a supervised phenomic image encoder and highlight the ablation study of molecular fingerprints described in Figure 5.

Instead of Ph1, we trained Molphenix framework using AdaBN, a CNN-based supervised phenomic encoder, with an analogous implementation discussed in [1]. We find that the general trends between Ph-1 and AdaBN are consistent with a slight decrease in overall performance. These findings provide additional support to the generality of the proposed guidelines.

In addition, we leveraged from Mol1, which is a MPNN based GNN model with 1B parameters - an expressive capable molecular encoder [2, 3]. We note that combining Mol-1 embeddings with ECFP, MACCS, and Morgan fingerprints can provide Molphenix with richer information and yields overall higher MolPhenix performance (Table 2 and 3 of PDF in general response). We also performed an ablation study over publicly available fingerprint encoding methods, which are an effective baseline for molecular representations. We demonstrate that MolPhenix demonstrates strong retrieval performance with the use of fingerprints as an alternative to pre-trained GNN (Figure 5).

Concern #3: “Oversimplified treatment of batch effects”

The reviewer is accurate in noting that batch effect area is a smaller portion of the overall paper contributions as it is a rich area of research, especially in the biological sciences. Our intention was to highlight the ability of averaging phenomic encoder embeddings which is infeasible when working with samples directly in the image space. We perform an ablation, studying the effects of taking a random number of embeddings when performing averaging and find a small improvement in retrieval performance (Figure 5).

Concern #4, Q1 & Q2:

Training data employed is comprised of 1.3M pairs of perturbations, however, RXRX3 is composed of 1,674 known chemical entities at 8 concentrations each and is primarily used as a validation dataset. JUMP-CP [4] is a promising open source dataset which is being released in increments is a promising future phenomic data resource. With the availability of JUMP-CP it will be possible to train a fully open source analog of Molphenix. A beta version of Ph-1 is publicly available and will be linked in the updated paper. In our ablations demonstrated in Figure 5 we show that molecular fingerprinting methods are a strong baseline and are comparable with Mol-1 performance.

Kindly let us know if our response above addressed your concerns. We will be happy to further discuss and answer any questions you may have.

We thank the reviewer for a thorough and rigorous examination of our paper. Below we aim to address the questions and clarifications to further improve the work.

Concern #1: For the captions of Table 2~5, it is better to mention the experimental setting.

Thank you for your feedback. We changed the captions to clarify and simplify the experimental setting and explanation of results in Tables 2, 3, 4 and 5. For example, following is the updated caption for Table 2 “ Evaluation on cumulative concentrations: Top-1% recall accuracy with use of the proposed MolPhenix guidelines evaluating impact of training loss on retrieval. We omit explicit concentration from this experiment.”.

Concern #2: “Specific advantages to choosing the current [GNN].”

Molphenix architecture is flexible, allowing that the proposed components be replaced by other phenomic or molecular pretrained models. However, we leveraged from Mol-1, which is a MPNN based GNN model with 1B parameters which allows us to maximize architecture expressivity while minimizing the risk of overfitting [2, 3]. We also note that combining Mol-1 molecular embeddings with ECFP, MACCS, and Morgan fingerprints can provide Molphenix with richer molecular information and yields overall higher performance of MolPhenix in both cumulative and held-out concentration scenarios. Results for active and all molecules retrieval of Molphenix trained on the discussed combinational molecular embeddings are available in table 2 and 3 of our global response (attached pdf). To evaluate the impact of GNN encoder we also perform a fingerprint ablation assessing the impact of fingerprint expressivity on retrieval indicated in Figure 5 of the paper.

Concern #3: “discussions regarding the results of different concentration encoding choices”

We note in the last sentence of section 5.2 in the paper that one-hot encoding shows significant improvements in a cumulative concentration setting, however it is limited to unseen dosage. In evaluation on held-out concentration, the model isn’t required to discriminate between molecules with different concentrations, thus explicitly providing dosage isn’t directly useful. In this experiment, sigmoid embedding can be thought of as a continuous way of separating “high” and “low” concentrations, showing effectiveness in this setting. We believe that the best encoding choice is dependable on the type of in-silico application. For example, if the objective is to identify unseen molecules with same morphological impacts or if the objective to simulate the impact of a molecule for un-tested dosage. We thank the reviewer for providing this feedback and we will add this additional discussion to the final version of the paper.

Thank you for your positive feedback, and we will be happy to further discuss and answer any questions you may have.