A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features

We thank the reviewer DZgQ for their review, and for acknowledging the robustness of our experimental design, and the innovation behind our approach. We respond below to the comments of the reviewer :

• “The visualization is not clear, and the y-axis of Figure 1 is not labeled.” : We thank the reviewer for their feedback. For the final manuscript, we will ensure that the visualizations are clear and more zoomed in, and we will add the label to the y-axis of Figure 1 (Recall of known relationships).

• “The specific definition of weakly paired data is not clearly provided.” : We thank the reviewer for pointing this out. While we have provided short definitions for weakly paired data (L32, L138, L214), we agree that this work would benefit from a clear and self contained definition of weakly paired data (pairs of samples from two modalities that are not paired at the sample level, but share the same labels and conditions, in our case the common condition being the same cell type and same perturbation, even if the cell samples are different in other factors). We will provide a clearly defined definition of weakly paired data in the final manuscript at the Introduction section, as well as in the Experimental setup section (section 4).

• “Biological fidelity of PEA: How can it be proven that randomized batch correction does not disrupt key biological signals?.” : We have already proven and established this in the submitted manuscript, as we proposed two different and complementary ways of evaluating empirically whether the learnt representation using the augmentations is biologically : i) Both in Figure 2.b and Table 1, we evaluate PEA on the transcriptomics interpretability task, which is a published benchmark for predicting gene expression counts of real unseen perturbations, as the reviewer has proposed, through both reconstruction tasks and conservation of structural integrity measures of the data. ii) In figure 3, we focus on downstream tasks, and show that for discovering target relationships, our distilled transcriptomics representations uncover the same relationships the original transcriptomics data uncovers, while improving it further by uncovering imaging and other new relationships. This shows that we have evaluated the comprehensiveness of our transcriptomics representations thoroughly, and have shown that it conserves the original transcriptomics information even after distillation.

• “Hyperparameter experiments for adapter temperature, learning rate, batch size, etc., are not conducted.” : We thank the reviewer for pointing this out. We have added in the final manuscript an overview of the hyperparameter experiments and their results (Results section), which we also show in this anonymized link : https://imgur.com/a/0Oekn3T.

• “Computational efficiency: What is the training time for Semi-Clipped on a dataset with 1.3 million samples? This information affects the evaluation of the method's practicality.” : We thank the reviewer for pointing this out. While there is no publicly available weakly paired (transcriptomics-imaging) dataset of more than 300k samples, if we stack our existing datasets on themselves to reach the size of 1.3 million samples, Semi-Clipped then requires a 19 hour training on one single H100 GPU. Our distillation method requires extremely minimal computing resources, as it makes use of trainable adapters and frozen backbones, while the additional PEA data augmentation only multiplies the distillation training time by x1.3. We will update our final manuscript with detailed benchmarking of the time and computing resources of our approach.

• “Negative results disclosure: Have other modality combinations (e.g., proteomics + transcriptomics) been attempted?” : We thank the reviewer for proposing this idea. While other modalities can potentially add interesting information, these different modalities (e.g proteomics) suffer from even lower availability of perturbational data than transcriptomics, and even less than that in the case of weakly paired data; thus, no, other combinations have not been attempted.

We thank the reviewer T8LP for all their comments, and for their acknowledgement of the quality of the paper and the contribution. We aim to acknowledge and answer below the questions and suggestions of the reviewer :

• “Additional reference” : We thank the reviewer for pointing us toward this publication. As tabular data shares many common points with transcriptomics data, this merits an additional discussion on this paper in the final manuscript.

• “there a few cases where figures are not well labelled” : We agree with the reviewer, and while this was a limitation of the 8-pages format,, we will take advantage of the extra space in the final manuscript to ensure that all labels and captions are self contained and self explanatory.

• “Add y-labels to Figure 1, and Figure 2.” : We add y-labels to both figures in the final manuscript.

• “Figure 3 took some time to parse, could this data be more clearly represented in a tabular format?” : We thank the reviewer for their suggestion, this could clarify further the impact of our method. We have added to the final manuscript a new table that quantifies for each approach the amount of gain and loss of relationships through the distillation compared to other approaches. The added table is available in this anonymized link (https://imgur.com/a/qZB7q9l). We see that Semi-Clipped recalls the highest amount of known relationships, preserves the original transcriptomic information the best, and minimizes the loss of relationships the most.

• “Can you foresee the impact of PEA beyond transcriptomics data used in a preclinical setting, where datasets do not typically have a well defined notion of unperturbed controls?” : PEA’s augmentation principle is especially promising for biological data in batches with control, such as proteomics and metabolomics, where batch correction techniques are well established to mitigate technical variability. Although its impact may be more limited in contexts lacking a clear notion of unperturbed controls or where robust domain adaptation methods are already well established, PEA can still broaden the scope of existing solutions. PEA’s strategy of leveraging small and stochastic technical variations could extend and be adapted to specific post-processing methodologies in areas such as signal processing (ECG, wearable devices) or astronomy. We will add a discussion on this question to the conclusion of the final manuscript.

We thank the reviewer rL4s for their review, and for acknowledging the robustness demonstration of our claims. We will respond below to all their comments :

• “The reviewer is concerned that Semi-Cipped outperforms Clip in a context dependent way” : We thank the reviewer for pointing this out. To improve the demonstration of our claims, we add to Figure 1 of the final manuscript the performance of scGPT + Tx Adapter + Image Adapter. The revised figure can be found in this anonymized link (https://imgur.com/a/CDmKd3v). We show through this result that Semi-Clipped still outperforms Clip even in the scGPT context. To further illustrate this, we will add Clip as an additional approach in Figure 2.a in the final manuscript, to showcase with more clarity how Semi-Clipped compares to Vanilla Clip and other approaches.

• “There's a potential distribution shift issue since scVI was pretrained on single-cell data, while this study uses arrayed bulk sequencing data” : We thank the reviewer for pointing this out, and apologize for the confusion in the submitted manuscript. We do not have the distribution shift issue, as we pretrain scVI on arrayed bulk sequencing data (HUVEC-CMPD dataset), which is also used for the distillation training. While scVI was originally developed for single cell data, recent published benchmarks (e.g Bendidi et al. 2024) have shown that scVI still outperforms all models even when trained on Bulk sequencing data, which motivated our choice. We have updated the final manuscript to better reflect and clarify this point, given the additional page of main content available.

• “The paper adequately covers relevant literature but contains numerous citation errors” : We sincerely thank the reviewer for pointing this out. We have now fixed all errors in all the paper’s citations, this will be reflected in the final manuscript.

We thank reviewer rGLV for their review. We address reviewer’s comments below, in order to improve the clarity of the contribution of our submission :

• "This paper doesn't show empirically that the learnt representation of transcriptomics is relatively comprehensive." : We respectfully disagree. In the submitted manuscript, we proposed two different and complementary ways of evaluating empirically whether the learnt representation of transcriptomics through our approach is completely comprehensive : i) Both in Figure 2 and Table 1, we evaluate on the transcriptomics interpretability task, which is a published benchmark for evaluating whether the representation of transcriptomics is comprehensive and includes all information of the original transcriptomics data, through both reconstruction tasks and structural measures of data. ii) In figure 3, we focus on downstream tasks, and show that for discovering target relationships, our distilled transcriptomics representations uncover the same relationships the original transcriptomics data uncovers, while improving it further by uncovering imaging and other new relationships. Additionally, reviewer T8LP describes our evaluation as “that SemiClipped, which freezes the microscopy imaging encoder, recalls the most relationships overall and from those recalled by the unimodal transcriptomic model.”. This shows that we have evaluated the comprehensiveness of our transcriptomics representations thoroughly, and have shown that it conserves the original transcriptomics information even after distillation.

• “t-SNE for representation distribution” : We add in this anonymous link (https://imgur.com/a/GCi2FYj) a UMAP projection of our learnt features of the distillation model (bottom), compared to the transcriptomics features only (top). We compare both models at batch effect reduction (left, Mixed clusters is better) and perturbation separation (right, separated clusters is better). While both approaches are good at reducing batch effect, the distillation approach is largely better at separating different perturbations. This is also reflected in the NMI metric for clustering the different perturbations with a K-Means (0.128 for the transcriptomics only, 0.481 for the distillation). We had previously hesitated to add UMAP/t-SNE views due to their anecdotal nature. We will however add this figure to the final manuscript for enhanced clarity.

• "Lack of ablation study when teacher representations are not frozen." : We respectfully disagree. In the submitted manuscript, we show in Figure 1 that unfreezing the teacher representations through adding an Image Adapter leads to a drop in performance, likely due to modality drift. We add an additional evaluation of scGPT used with an Image Adapter in our response to reviewer rL4s for more comprehensiveness.

• “Proposed reference” : We thank the reviewer for the reference, we will include it in the final manuscript in the related works section (Biologically Relevant Representations).

• “The methodology contribution is limited” : We wish to point out that in addition to being the first to prove that distillation from Imaging to Transcriptomics is possible, we have proposed PEA, a completely novel and new approach for data augmentation on omics data that preserves biological information for multimodal learning, tackling one of the main problems of the omics field : Limited paired data and lack of biology preserving data augmentations. We also want to point out that reviewer rL4s described our claim and approach as “PEA augmentation is interesting and generally improves the performance”, while reviewer T8LP describes our approach as “This is a strong paper, with high quality authorship, clear and concise results, and well thought out experiments. The novel augmentation method for transcriptomics data could have impact alone.“ and “By leveraging these modalities together, the growing power of microscopy models can be leveraged to create strong transcriptomic prediction models.”. Reviewer DZgQ described our work as “For the first time, batch correction is restructured as data augmentation, addressing the scarcity of biological multimodal data while ensuring performance improvement with interpretability.”. This being an application track submission for ICML, where submissions are judged by their real world relevance and the robustness of the claims and experiments, we hope this clarifies the reviewer’s assumptions about our work.