We thank the reviewer for carefully examining our work and providing their feedback. We’re thrilled to hear that you found the motivation of our work to be “clear and closely related to real-world scientific applications” and that you recognize our results “show significant improvements over prior works”. In the space below we respond to the reviewer’s specific concerns. Due to space constraints, our response to the reviewer's questions/minor concerns can be found in the response box for CsPQ. Please let us know if we can further clarify any of these points.

Weaknesses/major concerns:

Re: How do we know that high-content screening (HCS) data has false positives and false negatives?

We thank the reviewer for raising this and for allowing us to clarify these points. First, we emphasize that: in general, cellular perturbation screens - including high-content screening (HCS) datasets - are well-known to be substantially impacted by false positive and false negative pairs.

False positive pairs often arise due to technical limitations with cellular perturbation technologies. For instance, in genetic perturbation screens, CRISPR guide RNAs are used to perturb specific genes. However, due to off-target guide effects (i.e., guide has the wrong effect) [1] or low guide efficiency (i.e., guide does not have an effect) [2], guide RNAs often fail to induce the intended gene knockdown. Indeed, previous work [3] has estimated that only 5%-10% of guide RNAs have their intended effect >50% of the time. depending on the specific guide. Similarly, in chemical screens, exposure effects on cells may vary depending on cells’ positions on experimental plates, leading some cells to be weakly or not at all affected by the intended perturbation. To mitigate the impact of false positive pairs, we thus performed per-perturbation pooling of cellular profiles using the ABMIL pooling operator, which adaptively attends to the samples most representative of a perturbation.

In response to your comment, to better understand the impact of accounting for false positives by focusing on the most representative samples within a perturbation, we assessed the impact of removing the most highly attended samples by ABMIL (i.e., those most likely to reflect effective perturbation) for the Bray et al. cross-modality retrieval tasks. We found that doing so led to significant performance degradations (see table below), indicating that, for each perturbation, only a relatively small number of cells may exhibit clear perturbation effects.

False negative pairs arise due to the fact that many cellular images within an HCS dataset share the exact same perturbation label. For example, in the RxRx3-core dataset considered in our manuscript, each perturbation is associated with ~92 cellular images. Analogous to findings with CLIP-like models applied in other domains [Wang et al., Zhao et al., Srinivasa et al.], this fact can introduce another source of instability during model training, as a model trained with the standard CLIP loss may exhibit pathological behavior when forced to push apart cells with highly similar morphological profiles and identical perturbation labels. Indeed, as demonstrated in Table 2 in the manuscript, accounting for false negatives by training on per-perturbation pooled profiles (as opposed to individual cells) resulted in substantially improved performance.

Moreover, as shown in previous works introducing new HCS datasets [4, 5], groups of related perturbations (e.g. gene knockouts targeting genes in the same higher-level gene pathway), may have near-identical morphological effects. This fact further exacerbates the issue of false negatives, as cells with semantically similar image profiles yet distinct perturbations may be spuriously pushed apart in the embedding space. To mitigate this effect, we employed the CWCL loss [6], which assigns non‐zero weights, $w_{i,j}$ in eq (5), to semantically similar negative pairs, within CellCLIP rather than the standard contrastive loss, which treats all false negatives as unrelated (zero weight). As shown in our gene-gene relationship recovery results in Table S.4, using CWCL resulted in better preservation of known relationships between perturbations (recall = 0.595) compared to the standard contrastive loss (recall = 0.553).

Re: Is the superior performance of natural language for perturbation encoding just due to increased model size?

We thank the reviewer for raising this question. In response, we conducted an additional set of experiments (see table below) using BERT models with 64M and 11.3M from [7] as the text encoder for CellCLIP for the Bray et al. cross-modal retrieval task. Note that the MPNN+ graph-neural-network based encoder used in our MolPhenix baseline had ~10 M parameters. While we observed a slight decrease in cross-modal retrieval performance compared to the full BERT model, we found that even the 11.3M parameter BERT model substantially outperformed the MPNN+ model.

These results suggest that the improvements we observe in cross‑modal retrieval are not simply artifacts due to a larger model size, but rather stem from the use of our text-based encoding strategy.

Re: Improvements to clarity.

We thank the reviewer for flagging these clarity issues. We provide our responses to clarify the specific points raised by the reviewer below. In the camera-ready final manuscript we will use the additional 10th page to clarify these points.

• Re: How do the components of our approach (lines 62-74) address the challenges identified in lines 46-61?

  To clarify, the challenges identified in lines 46-61 are:
  
  a. Limited availability of image encoding models for Cell Painting microscopy data

  b. Microscopy image channels each capturing semantically distinct information

  c. False negative pairs due to Cell Painting data being “many-to-one” (i.e., many cells share the same perturbation label)

  d. False positive pairs due to variable perturbation efficacy

  e. Difficulty encoding multiple classes of perturbations with a single perturbation encoder
  
  Our approach addresses these challenges as follows:
  
  1) CrossChannelFormer allows us to reuse openly available state-of-the-art vision embedding models trained on natural images for Cell Painting data, while accounting for the distinct semantic meaning encoded in different Cell Painting image channels (challenges a and b).
  2) Pooling cell image profiles mitigates the impact of both false negative pairs and false positive pairs (challenges c and d).
  3) Using a natural-language-based perturbation encoder allows us to encode multiple classes of perturbations with a single model (challenge e).
  
  As demonstrated by the ablation experiments in our manuscript, each of these modifications led to improved performance on downstream tasks.

• Re: Why does pooling address both false negatives and false positives?

  Thank you for pointing out this area of confusion. To clarify:
  
  Pooling addresses the issue of false negatives by preventing cells with the same perturbation label from spuriously being considered as negative pairs. If these cells were incorrectly considered as negative pairs, the model may be forced to separate embeddings of cells with highly similar morphological profiles, which could lead to instability during training. By instead operating on pooled profiles, negative pairs will always correspond to distinct perturbations, potentially resulting in better performance.
  
  At the same time, pooling addresses the issue of false positives by preventing representations of images assigned the same perturbation label, but which had divergent outcomes (e.g. one cell had a successful perturbation, while the other did not due to CRISPR guide efficiency issues) from being erroneously pushed together. By instead pooling morphological profiles, we may sidestep such issues stemming from variable perturbation efficacy, resulting in improved model performance.

• Re: “The CWCL objective in Equ. (5) seems problematic”

  This was indeed a typo. Thank you for your attention to detail, and we will fix this in the final manuscript.

References:
[1] https://www.nature.com/articles/nmeth.4293
[2] https://www.nature.com/articles/nbt.3026
[3] https://www.nature.com/articles/s41586-018-0686-x
[4] https://www.nature.com/articles/s41592-024-02241-6
[5] https://www.biorxiv.org/content/10.1101/2023.08.13.553051v1.full
[6] https://arxiv.org/abs/2309.14580
[7] https://arxiv.org/pdf/1908.08962

---

Please see the rebuttal box for reviewer CsPQ for our responses to questions/minor concerns.

I appreciate the authors' efforts in providing a comprehensive response and additional experimental results. My concerns have been satisfactorily addressed, and accordingly, I have revised my scores and now recommend acceptance.

As an additional (non-mandatory) point of curiosity that does not affect my final decision:

• The authors highlight that SMILES strings are the most influential component in the text prompts. In light of this, I wonder whether KV-PLM [1]—a BERT-based model that is continually pre-trained on biomedical texts and SMILES strings—might serve as a more suitable choice for the text encoder.

Refs.

[1] A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals.

Dear reviewer CsPQ,

Thank you for the kind words and for voting for acceptance. We’re thrilled to hear that you believe our approach “comes with significant advantages over many similar approaches”, and that our paper “is relevant and very well executed”.

If there is any other information we can provide/experiments we could run during the discussion period that would allow you to further raise your score, please let us know.

Thanks again,

Due to space constraints, we have dedicated the remainder of the space to responding to Reviewer Hzy8's questions/minor concerns.

---

Questions/minor concerns from Reviewer Hzy8:

Re: How do the different components of the prompt affect model performance?

We thank the reviewer for raising this question. In response, we performed an ablation study where we retrained and re-evaluated CellCLIP after removing different components of our prompt (e.g. cell type, drug name, SMILES string, etc.) on the cross-modal retrieval task for the Bray et al. dataset. We found that removing the SMILES string led to a large degradation in performance, and that removing cell type information led to a smaller, but consistent, degradation in performance. On the other hand, removing drug name did not negatively affect the model’s performance, likely due to any relevant information already being captured in the richer SMILES representation.

Re: Why did we specifically use BERT for text encoding, and why not use a domain-specific model (e.g. BioMedBERT)?

We decided to use BERT (e.g. as opposed to the encoder from the original CLIP model) because we required an encoder with sufficient input context length to enable sufficiently rich and detailed textual inputs. For example, the text encoder from the original CLIP model is limited to a 77‑token context window. This limited window is insufficient for representing SMILES strings (the average length of a SMILES string is 98 characters in the Bray et al. dataset) together with cell information and other metadata. On the other hand, BERT supports up to 512 tokens. Moreover, several works [1, 2, 3] have shown promising performance with BERT-based text encoder for vision-language alignment tasks. In response to the reviewer’s second question regarding the performance of domain-specific models, we conducted an additional set of experiments using the domain-specific BioMedBERT as our text encoder for the Bray et al. cross-modality retrieval tasks. We found (see table below) that the performance of BioMedBERT was largely inferior to that of the standard BERT model across various choices of vision encoder. This may be because many of the compounds in Bray et al. are experimental and thus not represented in BioMedBERT's training corpus, which consists of PubMed and PubMed Central articles.

Re: Do we need so many layers for CrossChannelFormer?

Thank you for raising this question. In response, we performed an additional set of experiments investigating the impact of the number of layers in CrossChannelFormer on performance for the Bray et al. cross-modal retrieval task. Although the number of tokens (i.e., wavelength channels) in the HCS data is fewer than 10, we found that increasing the number of layers in CrossChannelFormer is necessary for achieving strong model performance.

Re: Do we see the benefits of transfer learning when pretraining on Bray et al. and fine-tuning on CP-JUMP1?

We thank the author for raising this question. In the initial iterations of our CP-JUMP1 experiments we attempted to train CellCLIP directly on CP-JUMP1 (i.e., without a pretraining step). However, we found that doing so resulted in extremely poor performance, with CellCLIP’s mean average precision (mAP) being close to zero. We believe this poor performance was due to the relatively small number of perturbations in this dataset (811 unique perturbations, with about ~500 left for training after performing a train-val-test split).

In contrast, when we first pretrained the model on the larger Bray et al. dataset and then fine-tuned on CP-JUMP1, we observed substantial improvements in downstream performance across both CellCLIP and all baseline models. These results show clear benefits of transfer learning in this setting.

References:
[1] "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation" https://arxiv.org/abs/2201.12086
[2] "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models" https://arxiv.org/abs/2301.12597
[3] "BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs" https://arxiv.org/abs/2303.00915

We thank the reviewer for carefully examining our work and providing their feedback. We’re happy to hear that the reviewer recognizes the “clear need to develop effective models” in this area, and we’re thrilled that the reviewer found that our results “suggests the advantages of [our] strategy”. Below, we respond to the reviewer’s concerns. Please let us know if we can further clarify these points.

Re: Optimizing the choice of prompt

We thank the reviewer for raising this important point. While we acknowledge that CellCLIP’s performance may vary with prompt format, it is designed as a general, flexible framework that allows users to represent perturbations using diverse metadata tailored to their needs; we note that the prompt template used in our paper is just one example. Although the current prompting strategy consistently achieves state-of-the-art performance across all evaluated datasets, we agree that prompt engineering could offer room for further improvement. In response to the reviewer’s suggestion, we explored alternative prompting strategies. These provided insights into component-level contributions, but none of them outperformed the original. Results are detailed below:

First, we performed an ablation study where we re-trained and re-evaluated CellCLIP after removing different components of our original prompt template (e.g. cell type, drug name, SMILES string, etc.) on the cross-modal retrieval task for the Bray et al. dataset. We found that removing the SMILES string led to a large degradation in performance, and that removing cell type information led to a smaller degradation. On the other hand, removing drug names did not negatively affect the model’s performance, likely due to relevant information already being captured in the richer SMILES representation.

Second, following the reviewer’s advice, we experimented with incorporating descriptions from an external source into the prompt template for CellCLIP. Specifically, we added DrugBank descriptions for any compounds that could be found in the database (correspondsing to approximately 10% of the compounds in the Bray et al. (2017) dataset). We note that the match rate was low, as many compounds in this dataset are small experimental molecules. After incorporating available DrugBank descriptions into the prompts, we found no improvement in performance compared to our original template. While it is possible that retrieval performance could benefit from more comprehensive metadata, this was not feasible given the limited annotation coverage in DrugBank. Nonetheless, these results suggest that our original prompting strategy is already highly effective.

Re: Experiments with different text encoders (e.g. from OpenAI) for the perturbation encoder

We thank the reviewer for this suggestion. In response, we conducted an additional set of experiments where we used OpenAI’s text-embeddings-3-small model with our prompt template to generate perturbation embeddings, followed by a 4-layer MLP as the perturbation encoder for CellCLIP on the Bray et al. cross-modal retrieval tasks. We chose this embedding model specifically because embeddings from OpenAI’s decoder-only models, such as ChatGPT, are not accessible to users. We found that this resulted in substantially worse performance compared to our original choice of BERT. This suggests that general-purpose embeddings, even from powerful embedding models, are insufficient for domain-specific tasks without fine-tuning of the base text encoder.

Re: How to select the best number of images used in each encoder?

For all experiments presented in our manuscript, when pooling images within a perturbation we always used all available images for that perturbation. As shown in our submission, we found that this strategy consistently yielded strong performance compared to baseline methods.

Re: Could we just use the mean or other simpler aggregators?

As part of our initial submission we included results from benchmarking the impact of different choices of aggregation operator on CellCLIP’s performance for the cross-modal retrieval tasks for the Bray et al. dataset. We found (Table S.2) that attention-based pooling led to superior performance compared to simpler pooling operations like the mean or median.

Re: Hyperparameter tuning for CellCLIP and baselines.

The results reported in our manuscript correspond to the best‑performing models selected through extensive hyperparameter searches for both CellCLIP and baseline models. The details of these sweeps can be found in Appendix E.2, which we reproduce below for the reviewer's convenience.

• Benchmarking results presented in Table 1: Here the best performing CellCLIP and baseline models were selected via a grid search over the following hyperparameter ranges:

  • Learning rate: {7 × 10⁻⁵, 5 × 10⁻⁵, 1 × 10⁻⁵, 1 × 10⁻⁴, 2 × 10⁻⁴, 3 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³, 1 × 10⁻²}
  • Warmup steps: {200, 500, 800, 1000, 1500}
  • Number of cosine restarts: {1, 2, 3, 4}
  • Temperature τ: {14.3, 20, 30}

• CellCLIP ablation results (Table 2): For each row in the table we performed a hyperparameter sweep over:

  • Learning rate: {0.0005, 0.001, 0.005}
  • Cosine annealing restart cycles: {5, 6, 7, 8, 9, 10}
  • Temperature τ: {14.3, 20, 30}
  • Epochs: {50, 60, 70}

• Sister perturbation matching & replicate detection results (Table 3): Here we swept over the following hyperparameter ranges for all multimodal models. (For both of the unimodal CA‑MAE and OpenPhenom‑S/16 models specifically, as these models’ training data already included CP-JUMP1, we did not perform any additional training steps with these models):

  • Learning rate: {1 × 10⁻⁵, 1 × 10⁻⁴, 2 × 10⁻⁴, 3 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³, 1 × 10⁻²}
  • Epochs: {5, 10, 20, 30, 40, 50, 60, 70}
  • Batch size: {128, 256, 512}
  • Warmup steps: {500, 800, 1000}

• Zero-shot gene-gene relationship recovery results (Table 3): Because this task involved zero‑shot retrieval, for all multimodal models we reused the models pretrained on Bray 2017 with no additional fine tuning. For unimodal models we did not perform any additional training steps.

Re: Is it possible to pre-train an image encoder on Cell Painting data rather than use an encoder pretrained on natural images?

In our original submission (Appendix  A.1), we also experimented with using a specialized image encoder model pretrained on Cell Painting data as the vision encoder for CellCLIP. In particular, we used OpenPhenom [1], the current state-of-the-art openly available pretrained Cell Painting image encoder. However, we found that using OpenPhenom resulted in consistently worse performance compared to natural image encoders. Indeed, OpenPhenom (178.05 M parameters) was even outperformed by much smaller natural image encoders (e.g. DINOv2 small with 22.06 M parameters). This result can be found in Table S.1 in our submission, which we reproduce below for the reviewer’s convenience.

We believe that this result may be due to the relatively small amount of publicly available Cell Painting data available for training compared to natural images. For example, OpenPhenom was trained on ~3 M Cell Painting images collected from current publicly available datasets. This is far smaller than the typical size of datasets used to train natural image foundation models (e.g. ~142 M images for DINOV2 [2]), which likely reduces the potential benefits of pretraining/transfer learning.

Re: Can the authors evaluate the stability of their perturbation embeddings? Also, is there a way to evaluate the performance across studies?

To assess the stability of our method, all results in our experiments corresponded to trials with three random seeds. Moreover, to ensure the generality of our method our experiments contained results from multiple datasets (Bray et al., CPJUMP1, and RxRx3 Core). Please let us know if you would like further information or if we have misunderstood your question.

References:

1: https://huggingface.co/recursionpharma/OpenPhenom

2: DINOv2: Learning Robust Visual Features without Supervision, https://arxiv.org/pdf/1908.08962