Thank you very much for addressing most of the concerns.

I still have two points:

1. What I meant by the docking experiment is to evaluate and report the performance of Glide, Gold and Vina on the PDBBind dataset (what is the percentage of predicted poses within 2A RMSD of the ground truth, averaged over PDBBind) and then also reporting this performance only over the datapoints where at least 2 of the softwares agreed. But this is not crucial for your paper, it just felt potentially interesting for the docking community, so decide freely whether to address this.

2. To me the sequence similarity thresholds of 0.6 and 0.9 seem to high. Could you please explain your choice? You can definitely have structural homologues with less than 0.6 sequence similarity. In my experience 0.3 or 0.4 are more common thresholds (e.g. AF2 uses them). Introducing a benchmark which would tolerate such leakage might slow down the progress on actually useful methods.

2. Regarding Sequence Identity Threshold

That is a very good question. There are several reasons for us to choose such a threshold.

From the experimental results, we observe that our task is not highly sensitive to the dataset split. Unlike previous bioactivity prediction tasks, which are sensitive to split levels due to model overfitting to the protein, our task effectively mitigates this issue. As shown in our analysis, models in earlier tasks could achieve high performance by overfitting to protein-only features, even though the task inherently requires both protein and molecule information. In contrast, our task emphasizes the relationship between a protein and multiple molecules, making it more difficult for models to overfit to the protein alone. Furthermore, the performance difference between the 0.9 sequence identity threshold and the 0.6 threshold is minor, which is why we have not further reduced the sequence identity threshold.

Additionally, the results highlight the challenging nature of our task. The grouped Pearson correlation of the models is only around 0.2 to 0.3, or even around zero, compared to values as high as 0.8 in previous tasks like Atom3D LBA for the same models. This demonstrates that data leakage or overfitting is not a major concern in our current split. Instead, the critical challenge for the bioactivity prediction community is to develop effective models capable of accurately predicting bioactivity and ranking small molecules correctly. Given this, retaining more data in the training set is more beneficial, as it allows models to learn from a larger dataset with more pocket-molecule pairs available.

Furthermore, during the preprocessing step, we have already applied structural similarity filtering using Fast Local Alignment of Protein Pockets (FLAPP) [1], a program that used to calculate the structure similarity between two protein pockets, on the entire dataset to remove structurally similar pockets. The distribution of structure similarity scores between the train and test sets is shown here: https://anonymous.4open.science/r/SIU_ICLR-ADF1/flapp_scores.png. This distribution demonstrates that most pockets have a structural similarity of less than 0.2, indicating a clear distinction between train and test sets.

Our primary goal is not to establish a benchmark but to introduce a new task with a novel dataset and evaluation methods that address the issues in previous tasks. For researchers aiming to further benchmark the extreme generalization ability of models, particularly if they believe overfitting has become an issue for our task, we provide a stricter version of the dataset split. In this stricter version, pockets from the training set with a structural similarity greater than 0.2 to any pocket in the test set are excluded. This is a highly stringent criterion and has been documented in the dataset description. A basic experiment results based on this split is also provided in Appendix G table 7.

[1] Sankar, Santhosh, Naren Chandran Sakthivel, and Nagasuma Chandra. "Fast local alignment of protein pockets (FLAPP): a system-compiled program for large-scale binding site alignment." Journal of Chemical Information and Modeling 62.19 (2022): 4810-4819.

Regarding filter passing percentage

Thank you for considering our method interesting. We can provide the percentage of poses that passed our consensus filter, calculated from a random 10% subset of our data. For the pose generated by Glide, the pass rate is 36%.

We apologize for not being able to provide the actual number, as we have deleted some parts of the original output from the docking software programs to reduce storage costs. Additionally, we provide further analysis on the PDBbind dataset in the following section.

Regarding docking benchmark

Thank you for pointing out that we should benchmark our proposed voting mechanism on a docking dataset to evaluate its efficiency. We conducted this evaluation on the PDB refined dataset, which contains 5318 protein-small molecule complex structures. We compared the three docking softwares, and we calculated the average RMSD for poses that passed the voting filter and those that did not for each software. The results are shown in Table 1:

Table 1: Comparison of Docking Software Using Voting Mechanism

From these results, it is evident that for each docking software, the molecules passing the filter exhibit a significantly lower average RMSD to the true co-crystal pose compared to those that do not. This demonstrates the effectiveness of our method in ensuring the quality of the final poses.

We also used the same dataset to analyze which two docking softwares tend to agree with each other more. The results are presented in Table 2:

Table 2: Agreement Between Docking Software

Regarding the models

The MLP is trained to predict a single scalar value for a protein-ligand pair. The input of such MLP is the concatenation of the CLS pocket embedding from the pocket encoder, and the CLS molecule embedding from the molecule encoder. We have added it to the paper.

Thank you for your valuable review. We greatly appreciate your recognition of our contribution in identifying issues with the existing task and our efforts in carefully constructing a reliable dataset for the task. We aim to address any unclarities in our paper and respond to your questions thoroughly.

Regarding unclarities in the text

Thanks for bringing up the unclarities in the text, we've checked all of these and updated with the correct or more detailed terms. We meant "converted to their negative logarithm" by "anti-logged". For initial 3D conformation generation of small molecules we use the Glide LigPrep module with default settings. For Figure 4, we've updated the figure itself to provide the general information for each panel of the figure and changed the caption to "Figure 4: Differences in assay values across label types and protein targets. (A) The mean assay values vary among representative label types, as shown by a heatmap of pairwise t-test p-values. Smaller p-values (lighter colors) indicate significant differences. (B) Violin plots illustrate the distributions of label values for different label types. (C) Differences in mean assay values among ten protein targets. (D) The distributions of label values for different protein targets." The updated figure can also be found at https://anonymous.4open.science/r/SIU_ICLR-C475/distributions_update.png

Regarding data split

We apologize for any lack of clarity in our description. For the 0.6 and 0.9 non-homology levels, we set sequence identity thresholds at 60% and 90%, respectively. Specifically, the 0.6 non-homology level ensures that no sequences in the training and test sets share more than 60% sequence identity. Sequence identity is calculated based on alignment scores generated using the pairwise2.align function in BioPython.

Regarding papyrus dataset

Thank you for bringing this paper to our attention. Papyrus is indeed an excellent dataset that also aims to provide a high-quality resource for bioactivity prediction tasks. We have incorporated the suggested discussion into the "Related Work" section.

Degarding description to facilitate dataset usage

Thank you for pointing out that a more detailed description to facilitate dataset usage is needed. We have added it to the appendix and here is the content:

split files

Splits for the training and test sets of our task are provided. However, we do not impose a fixed split for the dataset, allowing users the flexibility to perform their own splits. The updated version of the dataset includes four different predefined splitting strategies:

1. 90% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 90% to the test set are removed from the training set.
2. 60% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set.
3. 60% Sequence Identity + Structural Similarity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set. Additionally, protein pockets with a structural similarity greater than 20% to the test set are also excluded from the training set.
4. 10-Fold Cross-Validation Split: The dataset is divided into 10 clusters. Any pair of proteins with a sequence identity greater than 60% are placed within the same cluster. This split can be used for 10-fold cross-validation.

These options provide users with the flexibility to evaluate models under various levels of sequence and structural similarity constraints.

A pickle file that contains processed data and label for the dataset

It is a dictionary that contains the processed information of the dataset. Each key is a UniProt ID, and corresponding value is a list of dictionaries. Each dictionary is a data point and has following keys:

{

source data : PDB ID and UniProt ID information

label : a dictionary for different labels, including Ki, kd, ic50, ec50. Each label is a scalar value.

ik : InChI key of molecule

smi : SMILES of molecule

}

structure files

It is a dictionary of the following format:

DIR

├── uniprot id

│ ├── pdb id

│ │ ├── pocket pdb file

│ │ ├── inchikey1

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ├── inchikey2

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ...

│ ...

Users can use this to process their own data for their own models.

In response to "weakness"

We sincerely thank you for your thoughtful and constructive review. Your acknowledgment of its potential to be widely adopted by the community and its value in redefining this critical task is deeply encouraging. Our motivation has always been to contribute meaningfully to the community by addressing existing limitations and improving the definition of this task.

We appreciate the concerns raised regarding the responsibility of this work. We have taking these concerns raised by the reviewer very seriously. we recognized that certain aspects of our methodology and rationale might not have been fully articulated. To address this, we are committed to providing detailed explanations with additional clarity in the updated version of our paper to alleviate any concerns.

We also value your feedback on areas where improvements can be made and will incorporate these refinements to further strengthen our work. Your insights help us ensure that this work serves as a reliable and impactful resource for the community, and we are grateful for the opportunity to make it even better.

Thank you once again for your support and for highlighting the potential contributions of this work.

Regarding retaining only consensus filtered data points

Thank you for your valuable comment regarding the potential bias introduced by retaining only data points with consistent docking results. We appreciate the opportunity to clarify our approach and its rationale.

(1) The primary goal of our pipeline is to ensure the quality of the conformations used in our dataset. By focusing on cases where docking methods show consensus, we aim to reduce the bias introduced by any single docking software and improve the general quality of our structural data (this is further discussed in the following of our reply), which is crucial for reliable bioactivity prediction tasks.

(2) While we acknowledge the concern about a potential bias toward "easier" complexes, it is important to note that the difficulty of docking is not directly correlated with the difficulty of the bioactivity prediction task, especially when framed as a machine learning problem. Docking failure or lack of consensus may arise from various factors, including limitations in docking algorithms, which do not necessarily reflect the inherent complexity of the bioactivity prediction task whose goal is to predict the corresponding bioactivity label.

(3) The data points in our dataset are derived from well-established databases like ChEMBL and BindingDB. However, these databases inherently reflect research trends and human reporting biases, rather than the true distribution of bioactivity labels in nature. Therefore, our SIU dataset, like any derived from these sources, could not represent the real-world distribution.

Regarding Cross docked performance

We conducted an experiment in a cross-docking scenario, focusing on one target with 20 PDB structures with co-crystal ligands within the same binding site. After aligning these complex structures, we performed cross-docking on them and calculated the RMSD between the docked ligand poses in all holo pockets and the corresponding co-crystal ligand poses. The results are as follows:

As shown in the table, there is a significant improvement across various metrics for the docking poses that pass the filter compared to those that fail, highlighting the effectiveness of our consensus algorithm in ensuring the quality of docking poses.

Regarding the docking quality

Thank you for pointing this out. We recognize that our intended message was to emphasize that our pipeline achieves better quality compared to individual docking methods, as discussed earlier. We acknowledge the challenges faced by the docking community, but we would like to provide a few clarifications:

(1) We did not use "apo" docking in our work.

(2) The intention of our work is not to propose a new docking algorithm but rather to create a dataset that facilitates redefining the bioactivity prediction task. In this effort, our focus was on selecting robust and widely accepted mainstream docking methods to ensure acceptance by potential users.

(3) We implemented rigorous quality control measures, such as the consensus method described earlier, to enhance the overall reliability of the generated dataset.

Regarding the term tradition/conventional approach

Thanks for pointing this out. We realized that the terms the reviewer brought about are too genernal and might be misleading. We've added a determiner "machine learning" to these sentence to make it more clear.

By predicting binding free energy, FEP+ might indeed be considered a bioactivity prediction method in some sense. However, the primary focus of our paper is on defining an unbiased machine learning bioactivity prediction task. The issue with FEP+ methods is that, although they might have high prediction accuracy, they can only process one target at a time, and there are also limitations regarding the small molecules that can be predicted (e.g. should have high similarity with one another). In one word, handling one pocket at a time should be considered as the inherent limitation of such methods, they are not actively doing such grouping with the intention of addressing biased problems when evaluating the model itself.

Furthermore, when evaluating models, isn't it still necessary to calculate metrics as some average across different targets? Of course, if we report the metrics for each individual target, it would demonstrate that some models might perform better on specific targets. However, how do we compare the general performance of a model? This further brings up another issue with the FEP+ dataset: it includes too few targets, and for each target, there are very few small molecules. While this might work as a held-out test for machine learning models, these datasets address a completely different aspect from the motivation we aim to explore in our paper.

Response to "questions"

Regarding the definition of success ratio

The success ratio is defined as the generated pose has an RMSD < 2 with the ground truth ligand pose.

Regarding the label types contained in our test set

For the test set, we ensured that each label type used has a sufficient number of data points. Additionally, we treat different label types independently, meaning that Pearson and Spearman correlations are calculated separately for each label type. Additionally, as illustrated in the table in Appendix C, the number of small molecule-pocket pairs is substantial.

Regarding the data availability

The dataset will be released upon the acceptance of the paper. The license will be CC-BY-4.0. The released data includes:

Split files

Splits for the training and test sets of our task are provided. However, we do not impose a fixed split for the dataset, allowing users the flexibility to perform their own splits. The updated version of the dataset includes four different predefined splitting strategies:

1. 90% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 90% to the test set are removed from the training set.
2. 60% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set.
3. 60% Sequence Identity + Structural Similarity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set. Additionally, protein pockets with a structural similarity greater than 20% to the test set are also excluded from the training set.
4. 10-Fold Cross-Validation Split: The dataset is divided into 10 clusters. Any pair of proteins with a sequence identity greater than 60% are placed within the same cluster. This split can be used for 10-fold cross-validation.

These options provide users with the flexibility to evaluate models under various levels of sequence and structural similarity constraints.

A pickle file that contains processed data and label for the dataset

It is a dictionary that contains the processed information of the dataset. Each key is a uniprot id, and corresponding value is a list of dictionaries. Each dictionary is a data point and has following keys:

{

source data : PDB id and uniprot id information

label : a dictionary for different labels, icluding ki, kd, ic50, ec50. Each label is a scalar value.

ik : inchikey of molecule

smi : smiles of molecule

}

structure files

It is a dictionary of the following format:

DIR

├── uniprotid

│ ├── pdb id

│ │ ├── pocket pdb file

│ │ ├── inchikey1

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ├── inchikey2

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ...

│ ...

Users can use this to process their own data for their own models.

Thank you for their detailed rebuttal, please try to include a summary of these explanations in the paper to improve its clarity. I recommend acceptance of this paper even in its current form but I will reiterate my recommendation to the authors to reconsider the way they construct the test set and provide the training data: the evaluation of bioactivity prediction task should not have to be dependent on a specific way of generating 3D structures, I expect many methods for this task that will arise in the coming years will not require a structural input and therefore this framing of the benchmark will severely limit its applicability.

Thank you very much for your valuable feedback and for recognizing our paper’s contribution in identifying issues with the previous task, as well as the significance of the SIU dataset and metrics. We are committed to addressing your concerns and questions to further clarify and improve our work.

Response to major concerns

Regarding the dataset availability.

We will open-source the whole dataset upon acceptance. The licence will be CC-BY-4.0. It will include the .pdb files for all the protein target structures and the .sdf files for all the small molecule structures. The labels and metadata are stored in pickle files in dictionary format. Here is a description of the dataset:

split files

Splits for the training and test sets of our task are provided. However, we do not impose a fixed split for the dataset, allowing users the flexibility to perform their own splits. The updated version of the dataset includes four different predefined splitting strategies:

1. 90% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 90% to the test set are removed from the training set.
2. 60% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set.
3. 60% Sequence Identity + Structural Similarity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set. Additionally, protein pockets with a structural similarity greater than 20% to the test set are also excluded from the training set.
4. 10-Fold Cross-Validation Split: The dataset is divided into 10 clusters. Any pair of proteins with a sequence identity greater than 60% are placed within the same cluster. This split can be used for 10-fold cross-validation.

These options provide users with the flexibility to evaluate models under various levels of sequence and structural similarity constraints.

A pickle file that contains processed data and label for the dataset

It is a dictionary that contains the processed information of the dataset. Each key is a UniProt ID, and corresponding value is a list of dictionaries. Each dictionary is a data point and has following keys:

{

source data : PDB ID and UniProt ID information

label : a dictionary for different labels, including Ki, kd, ic50, ec50. Each label is a scalar value.

ik : InChI key of molecule

smi : SMILES of molecule

}

structure files

It is a dictionary of the following format:

DIR

├── uniprot id

│ ├── pdb id

│ │ ├── pocket pdb file

│ │ ├── inchikey1

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ├── inchikey2

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ...

│ ...

Users can use this to process their own data for their own models.

Response to minor comments

Regarding the Abstract

Thanks for the suggestion, the corresponding sentence has been changed to "To address these issues, we redefine the bioactivity prediction task by introducing the SIU dataset-a million-scale Structural small molecule-protein Interaction dataset for Unbiased bioactivity prediction task, which is 50 times larger than the widely used PDBbind."

GNN architecture

The GNN model is adopted from the ATOM3D dataset[1]. Each atom is a node. It employs five layers of graph convolutions as defined by Kipf and Welling [2], with each layer followed by batch normalization and ReLU activation. The architecture concludes with two fully connected layers incorporating dropout.

[1] Townshend, R. J., Vögele, M., Suriana, P., Derry, A., Powers, A., Laloudakis, Y., ... & Dror, R. O. (2020). Atom3d: Tasks on molecules in three dimensions. arXiv preprint arXiv:2012.04035.

[2] Kipf, T. N., & Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.

Unclarity in figure 1C

Apologies for any confusion caused by the figure. The figure represents a single target and various molecules. The x-axis corresponds to the true label values of each molecule, sorted in ascending order, while the y-axis represents the predicted values for each molecule. The red line indicates y=x, which serves as the ideal reference line. Ideally, the blue dots should align closely with the red line. However, as shown in the figure, this is not the case. This suggests that the model is primarily predicting the mean value of the target for all molecules, rather than capturing the variance in the data.

Regarding the cutoff

It is updated in the paper.

Regarding poses in Figure 2

By "poses", we are specifically referring to ligand poses. We filtered the ligand poses generated by the docking software and stored the pocket structures and ligand poses separately to reduce storage costs. The pocket structures were not involved in the consensus filtering process, and the RMSD calculations were performed only between ligand poses generated by different software programs. However, the pocket structures are included in the final dataset. To address the confusion, we have redrawn Figure 2 for improved clarity. Specifically, we have updated the term "poses" to "ligand poses" and added the shapes representing pocket structures to indicate their inclusion in the final dataset. The updated figure can also be found at: https://anonymous.4open.science/r/SIU_ICLR-ADF1/update_data_process.png

Regarding "identified by PDB IDs"

Thanks for pointing this out. However, we have not made such a statement. We understand that the distinction between PDB ID and UniProt ID, as well as between pockets and protein targets, can sometimes be confusing. In our paper, we clarify that multiple pockets may arise from the same protein target, which is identified by its UniProt ID, not by its PDB ID. We apologize for any confusion and have highlighted this distinction more clearly in the revised paper.

clarification on RMSD

Thanks for pointing this out. In Figure 3, all RMSD refer to the the ones between poses generated by different docking softwares to quantify consensus.

"A successful docking pose was defined as one with an RMSD of less than 2 Å compared to the experimental structure." This one is for the RMSD between the docking pose and ground truth PDB. Other RMSD are refered to the ones between poses generated by different docking softwares. We have updated the paper to make those more clear.

Regarding the validation fold construction

Currently, we use a random split to sample a portion of the training set to construct the validation set. This approach is based on our belief that defining the validation set is also a design choice left to the dataset users. For now, we focus on defining the standard train and test sets.

However, following your advice, we have also provided a 10-fold version of our dataset. In this version, the dataset is divided into 10 clusters, where any pair of proteins with a sequence identity greater than 60% is placed in the same cluster. This split can be used for 10-fold cross-validation. The results for this dataset are presented in Appendix G, Table 6, of the updated paper.

Regarding test set choice

As detailed in Appendix C, we explain the process of constructing the test set. The selected targets represent a wide range of protein classes, including G-Protein Coupled Receptors (GPCRs), kinases, cytochromes, nuclear receptors, ion channels, epigenetic targets, and others, ensuring broad coverage of the bioactivity landscape. For instance, “C11B1_HUMAN” belongs to the cytochrome P450 family, which plays a role in drug metabolism. “RARG_HUMAN” is a nuclear receptor family member, with drugs like bexarotene used in certain cancers. “NMDE1_HUMAN” represents the NMDA receptor, a key glutamate receptor implicated in neurological disorders, with memantine being an approved NMDA receptor antagonist for moderate to severe Alzheimer’s disease. Including these diverse targets enhances the relevance and applicability of our dataset for drug discovery.

The curated test set consists of 10 protein targets, covering a diverse range of protein classes while maintaining an even distribution of small molecule-pocket pair counts. Each target contains approximately 2,000 pocket-molecule pairs, resulting in a total of more than 20,000 pairs. This exceeds the size of the entire PDBbind dataset, demonstrating the comprehensiveness of our test set.

Furthermore, generalization ability is not the primary concern for our task, as evidenced by the results. This is a challenging problem where models do not exhibit overfitting, reflecting the difficulty of the task.

In response to your advice, we have also provided a 10-fold version of our dataset. In this version, the dataset is divided into 10 clusters, with any pair of proteins sharing a sequence identity greater than 60% placed in the same cluster. This split enables 10-fold cross-validation, and the results for this dataset are presented in Appendix G, Table 6, of the updated paper.

Regarding Redefining the Bioactivity Prediction Task

Thank you for pointing unclarity of input poses. it indeed highlights a challenge we faced during the construction of our pipeline. We established the selection priority as “Glide > GOLD > Vina” when a docking pose passed our consensus filter (i.e., two poses from different software with RMSD < 2 Å). All selected poses were saved in our final dataset and can be use as data augmentation during training.

When utilizing the data, we recognized that these details also serve as justification for our methodology, and we have included this information in the appendix section for transparency and reproducibility.

Regarding RMSE and MAE, we note that these metrics do not exhibit the bias issues associated with Pearson and Spearman correlation coefficients. Therefore, we retain the standard definitions for calculating RMSE and MAE in our evaluation.

Regarding Figure 3 Interpretation

We apologize for any confusion caused by Figure 3 and have updated the caption to provide greater clarity. Below is a detailed explanation of the figure:

• Figure (a) illustrates an experiment designed to determine the optimal cutoff value for defining the consensus filter. The x-axis represents different cutoff RMSD values between poses generated by various docking methods. The success ratio is defined as the percentage of docking poses that pass the filter that can be aligned with the actual pose of the ligand. The remaining ratio indicates the percentage of docking poses that meet the filter criteria.

• Figures (B, C, and D) provide representative cases that demonstrate the differences in poses generated by various docking methods. These figures illustrate how poses from different methods compare and emphasize the role of the consensus filter in selecting reliable poses.

We hope this detailed explanation resolves any confusion and makes the intent of Figure 3 clearer.

Thank you very much for your valuable feedback and for recognizing our paper’s contribution in identifying issues with the previous task, as well as the significance of the SIU dataset and metrics. We are committed to addressing your concerns and questions to further clarify and improve our work.

Response to major concenrs

Regarding Data availablity

Our dataset contains a significant number of structure files and is quite large, making it impractical to host on an anonymous GitHub repository at this time. However, we plan to open-source the entire dataset upon acceptance, under the CC-BY-4.0 license. The dataset will include .pdb files for all protein target structures and .sdf files for all small molecule structures. Labels and metadata will be provided in pickle files, stored in dictionary format. Below is a detailed description of the dataset:

split files

Splits for the training and test sets of our task are provided. However, we do not impose a fixed split for the dataset, allowing users the flexibility to perform their own splits. The updated version of the dataset includes four different predefined splitting strategies:

1. 90% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 90% to the test set are removed from the training set.
2. 60% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set.
3. 60% Sequence Identity + Structural Similarity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set. Additionally, protein pockets with a structural similarity greater than 20% to the test set are also excluded from the training set.
4. 10-Fold Cross-Validation Split: The dataset is divided into 10 clusters. Any pair of proteins with a sequence identity greater than 60% are placed within the same cluster. This split can be used for 10-fold cross-validation.

These options provide users with the flexibility to evaluate models under various levels of sequence and structural similarity constraints.

A pickle file that contains processed data and label for the dataset

It is a dictionary that contains the processed information of the dataset. Each key is a UniProt ID, and corresponding value is a list of dictionaries. Each dictionary is a data point and has following keys:

{

source data : PDB ID and UniProt ID information

label : a dictionary for different labels, including Ki, kd, ic50, ec50. Each label is a scalar value.

ik : InChI key of molecule

smi : SMILES of molecule

}

structure files

It is a dictionary of the following format:

DIR

├── uniprot id

│ ├── pdb id

│ │ ├── pocket pdb file

│ │ ├── inchikey1

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ├── inchikey2

│ │ │ ├── pose1 sdf file

│ │ │ ├── pose2 sdf file

│ │ │ ├── pose3 sdf file

│ │ ...

│ ...

Users can use this to process their own data for their own models.

Dear reviewer,

Thank you very much for your detailed and insightful review of our paper and rebuttals. We greatly appreciate the time and effort you have dedicated to providing thoughtful feedback.

Regarding test set choice

Thank you for your valuable advice. The test set presented in the main text likely represents an experiment with biochemical generalizability, emphasizing targets of greater interest to our biochemical experts. This approach defines a biologically meaningful task that addresses previous limitations and issues in bioactivity prediction tasks. While we acknowledge its limitations, we have created a 10-fold version, allowing the model to be trained on 9 folds and tested on the remaining fold, which prioritizes using a setting that is more reliable for machine learning outcomes.

The results from the 10-fold evaluation reemphasize our main conclusion that the proposed metrics are more challenging and that the current evaluation method is not appropriate, aligns with our primary goal to provide a more meaningful and unbiased bioactivity prediction task.

Regarding Figure 3 Interpretation

Thank you for your valuable comment. We apologize for the oversight regarding not including the PDB ID. We have now updated the caption of Figure 3 and also the Figure 3 itself in the revised version of the paper to include the PDB ID.

In this section, our goal is to clearly demonstrate the effect of RMSD on the alignment of ligand docking poses, which also reflects the effectiveness of our structural data quality control process. To illustrate this, we randomly selected a PDB structure to visualize ligand poses docked by different docking software in our pipeline. The PDB ID used is 3PB7, which represents the crystal structure of the catalytic domain of human Golgi-resident glutaminyl cyclase. The small moleucle docked here is 6-[4-[(4-phenoxyphenyl)methyl]-1,2,4-triazol-3-yl]-1H-benzimidazole listed by both ChEMBL and BindingDB as an inhibitor of glutaminyl cyclase with a InChI key of CVGBPSGVCJCEFD-UHFFFAOYSA-N.

Regarding adding a reference of Table 7 to Appendix G

We sincerely thank you for taking the time to review the updated Appendix and for providing such insightful feedback. We deeply regret the oversight in its initial organization. In response to your suggestions, we have included the reference to all Tables , ensured all figures and tables are properly referenced and easily accessible, and thoroughly reorganized the Appendix to present the content in a more logical and coherent sequence. Additionally, we have included a list of contents to enhance readability and facilitate understanding.

Your constructive comments have been instrumental in improving the quality of our submission, and we are truly grateful for your patience, guidance, and support.

Regarding discussion of Table 7

In Table 7 (currently Table 9), the results pertain to the Uni-Mol model.

These results are actually comparable to those obtained using the original split version. A Pearson correlation within the range of 0.3 to 0.35 suggests a weak to moderate correlation, indicating the models are not accurate. This is noteworthy because, in the Atom3D LBA task, models with similar settings can achieve a Pearson correlation close to 0.8. This highlights the challenging nature of our task and suggests that the metrics are not inflated or overfitted, even when the data split is less strict. Consequently, it is reasonable that the metrics do not decline when the split becomes stricter.

Additionally, it is notable that the conventional Pearson correlation (without grouping by pocket) for different label types drops obviously compared to the 0.6 version. However, the Pearson* metrics are not declined. This observation aligns with expectations: when correlations are calculated without grouping by PDB ID, the results are more susceptible to overfitting and potential data leakage. In contrast, for Pearson*, overfitting on the protein side does not necessarily lead to improved results.

Regardinig providing an anonymous GitHub page

Thank you for the suggestion. Our dataset exceeds 6 GB even when compressed as a ZIP file, which makes it challenging to host on an anonymized GitHub repository at this time. However, we can provide a version that includes example files.

This is the example files of our dataset: https://anonymous.4open.science/r/SIU_ICLR-E619/example_files.zip

summary

Thank you once again for your insightful and constructive feedback! We truly appreciate your valuable input and thoughtful suggestions that help us refine our paper.

Best regards,

Authors