S-MolSearch: 3D Semi-supervised Contrastive Learning for Bioactive Molecule Search

We appreciate the thoughtful questions and feedback provided by you. We have carefully considered your queries and provide detailed responses below.

Consideration of Screening Time

We have conducted additional experiments to measure the screening time of S-MolSearch compared to traditional methods. The results, as shown in the table below, indicate that S-MolSearch has a significant advantage in search time consumption compared to traditional molecular search methods even if the molecular embeddings are not precomputed.

Notably, if the molecule database is fixed and all molecules are pre-encoded and stored in vector database such as FAISS, S-MolSearch is expected to achieve a search speed of tens of millions of molecules per second.

Lack of Results for Experimental Metrics Such as AUROC or BEDROC:

We have expanded our experimental evaluation to include AUROC and BEDROC metrics. These additional results, found in table 1 and 2 in the addtional PDF, provide a more comprehensive comparison of S-MolSearch with existing models and highlight its robust performance across different evaluation criteria. The results show that S-MolSearch demonstrates superior performance on AUROC and BEDROC.

On DUD-E, S-MolSearch trained on data with a 0.4 similarity achieves AUROC and BEDROC results of 84.61% and 54.22%, respectively, surpassing all baselines. S-MolSearch trained on data with a 0.9 similarity achieves AUROC and BEDROC results of 92.56% and 75.37%, respectively, exceeding the best baseline by 50% in BEDROC.

On Lit-PCBA, S-MolSearc$h_{0.4}$ achieves AUROC and BEDROC results of 57.34% and 7.58%, respectively, surpassing all baselines in BEDROC and being comparable to the best baseline in AUROC. S-MolSearch$_{0.9}$ achieves AUROC and BEDROC results of 61.78% and 8.48%, respectively, achieving the best results in both AUROC and BEDROC.

Decline in Performance as the % of EF Increases:

We provide the specific calculation formula for EF in appendix section B. The upper limit of EF decreases with the increasing top x%. The theoretical maximum value of EF is calculated as 100 divided by x. For example, the maximum value of EF 1% is 100, and the maximum value of EF 5% is 20.

From the perspective of virtual screening tasks, this can be attributed to the increased difficulty in distinguishing between active and inactive molecules as more molecules are considered, which tends to dilute the enrichment factor. Increasing the screening percentage implies that a more diverse array of active molecules needs to be identified, which is often more challenging.

Impact of Soft Labels on Different Datasets

We think this is caused by differences in the benchmarks. The molecules in DUD-E have more analogues and decoy biases, making it crucial for the model to use soft labels to effectively distinguish between closely related molecules. The LIT-PCBA dataset ensures diversity in data representation, offers a broad distribution across chemical space, and effectively minimizes inherent biases, so the impact of soft labels is less pronounced.

Contributions of the Semi-Supervised Approach in Table 4 Compared to Fine-Tuning:

We would like to clarify once again that the fine-tuning we refer to in Table 4 involves initial pre-training with contrastive learning on an unsupervised dataset, followed by contrastive learning-based fine-tuning using active supervised data. Compared to S-MolSearch, we believe that the fine-tuning approach does not optimally integrate the information from both the unsupervised and supervised datasets. The performance improvements observed in Table 4 can be primarily attributed to the integration of unlabeled data through our semi-supervised approach, which enhances the model's ability to generalize beyond the limitations of fine-tuning.

Extending the Method from Zero-Shot to Few-Shot Setting

Extending S-MolSearch from a zero-shot to a few-shot setting is feasible. We explored two few-shot settings. In terms of data division, we randomly selected 70% of the data from each target in DUD-E as the training set for few-shot learning. The remaining 30% of active molecules and all inactive molecules were used as test data. With the training dataset, we employed two settings. The first setting(random) considers the query molecule as part of the active molecule set bound to the target, then combines it with data bound to the same target as positive pairs and data bound to different targets as negative pairs. The second setting(fix query) fixes one molecule in the positive pair as the query molecule and selects another molecule from the active molecules to form positive pairs. The construction method for negative pairs is the same as in the first setting. ZS represents zero-shot in this table, and FS represents few-shot. The results are all obtained by S-MolSearch trained on data with a 0.4 similarity.

Because there is no universal setup for few-shot in this scenario, we do not compare it with other methods.

Thank you very much for reading. We hope that our responses adequately address your concerns.

Reference:

[1] Taminau J, Thijs G, De Winter H. Pharao: pharmacophore alignment and optimization.

[2] Atwi R, Wang Y, Sciabola S, et al. ROSHAMBO: Open-Source Molecular Alignment and 3D Similarity Scoring.

We appreciate your thorough review and constructive feedback. Below, we address each of your comments and questions, aiming to clarify and enhance the understanding of our work.

Memory Consumption Concerns:

We measure memory consumption under different scenarios, as shown in the table below. Here, supervised learning corresponds to using one encoder, and S-MolSearch corresponds to using two encoders. The table shows that increasing the number of encoders leads to a linear increase in memory consumption. Since Uni-Mol is relatively lightweight, this does not significantly impact regular use.

Considering the potential applications involving large-scale data or large molecules in the future, memory consumption poses a challenge. In future work, we plan to explore update strategies, for example, similar to [1], to decrease memory consumption.

Utilization of 3D Structures:

We believe that utilizing the 3D information of molecules is advantageous in ligand-based virtual screening scenarios. While we do not introduce a new geometric deep learning technique, we ensure the preservation of geometric information by leveraging Uni-Mol. We want to point out that one of the primary contributions of this work is the combined use of both 3D molecular information and binding affinity information for virtual screening, as described in lines 77-79, rather than providing a new backbone for molecule pretraining.

Clarity in Section 3.4:

We realize that our current phrasing may lead to some misunderstandings. In fact, the optimal transport form we introduce is derived from [2], which presents a smooth and sparse form of optimal transport. This form not only effectively manages the computational cost but also maintains consistency with traditional optimal transport problems.

In S-MolSearch, $\Gamma$ can be interpreted as a joint probability distribution under given marginal probabilities. Through the design of the cost matrix $C$, we ensure that $\Gamma_{i,j}$ is positively correlated with $x_{i}x_{j}$. The additional $L2$ norm regularization ensures the sparsity of $\Gamma$. This implies that for labels with low confidence from the supervised model, the pseudo-labels are heavily punished. Overall, the introduced smooth optimal transport form guarantees that signals from the supervised model are more effectively transferred to the unsupervised model, while handling high uncertainty with appropriate regularization. We have revised the wording in the manuscript and added relevant citations accordingly.

Component Efficacy in Table 3:

We conduct an intuitively analysis to provide insights into why soft labels are pivotal for DUD-E, while pretraining is more crucial for LIT-PCBA. We believe this is related to the inherent biases of each dataset.

Regarding the pivotal role of soft labels for DUD-E [3], the molecular distribution in DUD-E has more analogues and decoy biases, making it crucial for the model to use soft labels to effectively distinguish between closely related molecules. The LIT-PCBA dataset ensures diversity in data representation, offers a broad distribution across chemical space, and effectively minimizes inherent biases, so the impact of soft labels is less pronounced. The importance of pretraining for LIT-PCBA [4] arises because the broader molecular distribution seen during pretraining allows the model to capture a wider variety of molecular features, thereby enhancing performance on this dataset.

As for the performance of original Uni-Mol, results in the tables demonstrate that the addition of our proposed components significantly enhances performance. The improvements underscore the value of integrating soft labels and pretraining into our framework, particularly in achieving better results across different benchmarks.

Minor Points and Typos:

Thank you for pointing these out. L162: We emphasize $M_{sup}$ in Figure 2 to help readers understand our work more clearly. Formula 2 and L168: 1N is an N-dimensional vector of all ones. We add this in line 170. And we correct other points and typos in the manuscript

Limitations:

Pre-trained models are versatile and can be utilized for a variety of tasks and Uni-Mol demonstrates strong performance across several applications. In virtual screening tasks, the model benefits from exposure to a broader molecular space, which we believe makes the use of pre-trained models reasonable and advantageous.

We also conducted ablation studies on pre-training, as shown in Table 3, which demonstrate that pre-training provides improvements.

Thank you very much for reading. We hope that our responses address your concerns and demonstrate the enhancements made to our work.

Reference:

[1] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning.

[2] Blondel M, Seguy V, Rolet A. Smooth and sparse optimal transport[C]//International conference on artificial intelligence and statistics. PMLR, 2018: 880-889.

[2] Mysinger M M, Carchia M, Irwin J J, et al. Directory of useful decoys, enhanced (DUD-E): better ligands and decoys for better benchmarking.

[3] Tran-Nguyen V K, Jacquemard C, Rognan D. LIT-PCBA: an unbiased data set for machine learning and virtual screening.

Thank you very much for supporting our work and careful review! We have considered each of your questions, and we provide detailed responses below.

Inference Process with Encoders

During inference, only the encoder gψg_{\psi}gψ is used to generate the molecular embeddings. The encoder fθf_{\theta}fθ is primarily used during the training phase for generating pseudo-labels. We have clarified this in the revised manuscript in Section 3.1, line131.

Handling New Proteins Without Reference Molecules

Our current work is based on known query molecules. If the query molecules are unknown, some existing techniques might be helpful for this situation. One possible approach is to construct a pseudo-ligand based on the shape of the protein pocket, as demonstrated in [1]. Another feasible approach is to combine S-MolSearch with structure-based methods, such as [2]. We believe this can serve as an excellent direction for further enhancing our future work.

Thank you for their helpful suggestions. We hope that our responses adequately address your concerns.

Reference:

[1] Gao B, Jia Y, Mo Y, et al. Self-supervised pocket pretraining via protein fragment-surroundings alignment.

[2] Zhang X, Gao H, Wang H, et al. Planet: A multi-objective graph neural network model for protein-ligand binding affinity prediction.

Thank you very much for supporting our work and careful review! We appreciate the detailed review and constructive feedback. We have addressed each of your comments and questions below, aiming to clarify and enhance the understanding of our work.

Qualitative Examples and Similarities

We agree that providing qualitative examples would enhance the understanding of our model's capabilities. We have added two examples of molecules identified as similar to query molecules in DUD-E. These molecules are all active molecules. The results indicate that molecules with high Tanimoto similarity tend to have high embedding similarity. These qualitative examples can be found in Figure 1 in the attached PDF.

Clarification on Subscripts in Section 3.4

The “sup” subscript in Section 3.4 indeed corresponds to the “label” subscript in Proposition 1. The two subscripts have the same meaning, representing supervised learning on labeled data. For clarity, we have standardized the subscript to "sup" throughout the manuscript.

Minor Comments

Thank you for pointing out the typographical errors. We have corrected the typo in line 151 and added the missing context in line 183 (now it's in line 186). These corrections are reflected in the revised manuscript.

About Negative Pairs

In our view, both of your questions pertain to the potential occurrence of false negative pairs. False negatives may arise in scenarios where a single ligand binds to multiple targets or multiple ligands bind to the same target.

The way we construct the training data, as described in lines 60-64 of the manuscript, considers molecules binding to the same target as positive pairs, while molecules binding to different targets are considered negative pairs. This approach can mitigate the occurrence of false negatives to some extent.

In the raw ChEMBL data, approximately 21% of the ligands can bind to two or more targets. To further analyze the situation of false negatives during training, we count the false negative pairs that appear in each batch of the sampled training data over one epoch and find that false negative pairs account for an average of 0.76% of all negative pairs. This is a relatively small number, and considering the robustness of contrastive learning during training, we believe these false negative pairs will not significantly impact the current results [1].

Nonetheless, the presence of false negatives remains a concern. In the future, we plan to address this issue by designing more robust contrastive learning objectives and constructing more refined datasets to minimize the occurrence of false negatives.

We believe our response effectively addresses your concerns. If you have additional questions or need further clarification, we are open to further discussion.

Reference:

[1] Wu J, Chen J, Wu J, et al. Understanding contrastive learning via distributionally robust optimization.